What is technical SEO?

Crawlability: robots.txt and sitemaps

Two files shape how Google crawls your site. The robots.txt file, at the root of your domain, tells crawlers which parts of the site they may request. Google says it is used "primarily to manage crawler traffic to your site," and warns that it "is not a mechanism for keeping a web page out of Google." To keep a page out of search results, you use a noindex tag or put the page behind a password. Blocking a page in robots.txt only stops the crawl.

A sitemap.xml file lists the pages you consider important so Google can "crawl your site more efficiently." It's a hint, not a guarantee: a sitemap "doesn't guarantee that all the items in your sitemap will be crawled and indexed." Google says a small site (about 500 pages or fewer) that links well between its own pages may not need one, but a new site with few links pointing to it is one of the cases where a sitemap helps. Since it costs nothing to keep one up to date, we add one to every site we build.

You may also hear about "crawl budget." Google's guide on it is written for sites with roughly a million or more pages, or 10,000 or more pages that change daily. For a typical local business site, it isn't something to worry about.

Core Web Vitals and page speed

Core Web Vitals are three measurements of how a page feels to use. Largest Contentful Paint (LCP) measures how fast the main content loads, Interaction to Next Paint (INP) measures how quickly the page responds when someone taps or clicks, and Cumulative Layout Shift (CLS) measures how much the layout jumps around while loading. Google's targets are LCP within 2.5 seconds, INP under 200 milliseconds, and CLS under 0.1.

Google says "Core Web Vitals are used by our ranking systems" and recommends good scores "for success with Search." Keep the weight in perspective, though. Google also says it "always seeks to show the most relevant content, even if the page experience is sub-par," and that good scores don't guarantee top rankings. A fast page helps, but it won't rescue a page that doesn't answer the search.

Structured data and schema markup

Structured data is code, usually written in a format called JSON-LD, that labels what a page contains: a business with an address and phone number, an article with an author, a list of questions and answers. Google describes it as "a standardized format for providing information about a page and classifying the page content," and recommends JSON-LD where your setup allows. Its main payoff is eligibility for rich results, the enhanced listings that show extra details under a search result.

Three limits are worth knowing. Google "does not guarantee that your structured data will show up in search results." The markup "must be a true representation of the page content," so marking up things visitors can't see breaks Google's guidelines. And for AI features like AI Overviews, Google says "there's no special schema.org structured data that you need to add." Structured data helps Google read a page. It doesn't replace what's on it.

Canonical tags and duplicate content

The same page can often be reached at more than one address, such as the http and https versions of a site, or a product list with sorting or filter options added to the URL. Choosing the main version is called canonicalization, which Google describes as selecting "the representative" URL for a piece of content. A canonical tag (<link rel="canonical" href="..."> in the page head) names your preferred URL.

Google calls the tag "a strong signal," but "a hint, not a rule": it "may choose a different page as canonical than you do." Naming one URL lets Google combine signals, such as links, that would otherwise be split across the duplicates. Google recommends a self-referencing canonical, where each main page points to itself. This isn't about avoiding a penalty. Google says "some duplicate content on a site is normal and it's not a violation of Google's spam policies." The point is to make sure links and attention land on one address.