Technical SEO Guide: The Complete Handbook for Developers

Most technical SEO guides are written for marketers who need to brief a developer. This one is written for the developer doing the work. Every concept below comes with the actual robots.txt line, HTTP header, or JSON-LD block you would ship, plus a way to verify it yourself with curl or your browser's devtools instead of taking a guide's word for it.

This is the pillar guide for our technical SEO content. It covers the full surface area (crawlability, indexability, site architecture, Core Web Vitals, JavaScript rendering, structured data, international SEO, log file analysis, and AI crawler access), and each major section below is a stable anchor other posts on this site link into directly.

Table of Contents

• What Is Technical SEO (and How It Differs from On-Page and Off-Page SEO)
• Crawlability: robots.txt, Crawl Budget, and Blocking Search Engines by Accident
• Indexability: noindex, Canonical Tags, and Duplicate Content
• Site Architecture and Internal Linking
• XML Sitemaps: What Belongs in One (and What Doesn't)
• Structured Data and Schema Markup
• Core Web Vitals and Page Experience
• JavaScript Rendering: What Crawlers Actually See
• HTTPS, Security Headers, and Mobile-Friendliness
• International SEO: hreflang and Multi-Region Sites
• Log File Analysis: What's Actually Crawling Your Site
• AI Crawlers and Technical SEO in 2026
• A Practical Technical SEO Checklist (Priority Order)
• Frequently Asked Questions

What Is Technical SEO (and How It Differs from On-Page and Off-Page SEO)

Technical SEO is the set of optimizations that affect whether search engines can crawl, render, and index your site at all, as opposed to what they think about the content once they've indexed it. Split search engine optimization into three buckets and the difference gets clearer:

• Technical SEO covers crawlability, indexability, site speed, rendering, and structured markup, the infrastructure layer.
• On-page SEO covers content quality, keyword relevance, headings, and internal linking within a page's copy.
• Off-page SEO covers backlinks, brand mentions, and external signals of authority.

The reason technical SEO comes first in most audits is simple: a page with perfect content and a strong backlink profile still ranks nowhere if a crawler can't reach it, can't render its content, or gets told (correctly or by accident) not to index it. Technical SEO is the floor everything else is built on.

Crawlability: robots.txt, Crawl Budget, and Blocking Search Engines by Accident

Crawlability is whether a search engine's bot can even request your pages. The most common way developers accidentally tank crawlability is a misconfigured robots.txt, usually shipped from a staging environment and never fixed for production. A robots.txt that blocks everything looks like this:

User-agent: *
Disallow: /

That single Disallow: / line, left over from a staging config, has quietly deindexed more production sites than any algorithm update. Check yours directly at https://yourdomain.com/robots.txt, or verify it from the command line:

curl -s https://yourdomain.com/robots.txt

A working robots.txt for a typical marketing site looks closer to this, allowing the important crawlers while pointing them to your sitemap:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://yourdomain.com/sitemap.xml

Beyond robots.txt, crawl budget matters more as a site grows. Crawl budget is roughly how many pages a search engine is willing to crawl on your site in a given period, based on your site's perceived importance and crawl health. You don't need to think about crawl budget on a 50-page site. You need to think about it hard on a 500,000-page programmatic site, where wasting crawl budget on thin, duplicate, or parameter-variant URLs means your genuinely valuable pages get crawled less often. Common crawl budget wasters: infinite faceted-search URL combinations, session IDs in URLs, and paginated archives with no clear stopping point.

Indexability: noindex, Canonical Tags, and Duplicate Content

A page can be perfectly crawlable and still never show up in search results if it's marked not to be indexed. Two mechanisms control this, and mixing them up is a common source of bugs.

The noindex directive, set either as a meta tag or an HTTP header, tells a search engine "you may crawl this page, but do not put it in your index":

<meta name="robots" content="noindex, follow">

Or as a response header, useful for non-HTML resources like PDFs:

X-Robots-Tag: noindex

The canonical tag is a different signal entirely. It doesn't hide a page, it tells a search engine which URL is the "real" one when the same content is reachable at multiple URLs (with and without a trailing slash, with tracking parameters, across HTTP and HTTPS, and so on):

<link rel="canonical" href="https://yourdomain.com/products/widget">

A frequent bug: a page has both a noindex tag and a self-referencing canonical tag pointing at itself. That's contradictory instructions, and worse, a page can end up with a canonical tag pointing at a different URL that itself is noindexed, which tells the search engine to consolidate signals into a page that has explicitly opted out of being indexed. Audit canonical and noindex together, not separately.

Site Architecture and Internal Linking

Site architecture is how your pages connect to each other, and it directly affects both crawl efficiency and how link equity flows through your site. The general principle: important pages should be reachable in as few clicks from the homepage as possible, and no page should be an orphan (a page with zero internal links pointing to it, discoverable only through the sitemap).

A flat, hub-and-spoke structure (homepage links to category pages, category pages link to individual items, items link back to their category and to closely related items) crawls and ranks better than a deep, linear structure where a page is eight clicks from the homepage. If your site has thousands of pages, pagination, category hubs, and a genuinely useful footer or "related content" module are what keep deep pages from becoming orphans.

Breadcrumb navigation earns its keep here too, both for users and for crawlers. A breadcrumb trail (Home > Category > Subcategory > Page) gives every page at least one more contextual internal link back up the hierarchy, and paired with BreadcrumbList structured data (covered below), it also gives search engines an explicit, machine-readable statement of where a page sits in your site's structure, which is more reliable than inferring hierarchy from URL patterns alone.

XML Sitemaps: What Belongs in One (and What Doesn't)

An XML sitemap is a direct list of URLs you want crawled, submitted to search engines so they don't have to rely purely on discovering pages through links. A minimal sitemap entry looks like this:

<url>
  <loc>https://yourdomain.com/products/widget</loc>
  <lastmod>2026-08-15</lastmod>
</url>

Two rules matter more than the exact XML formatting. First, a sitemap should only contain canonical, indexable, 200-status URLs, no redirects, no noindexed pages, no 404s. Including non-canonical or noindexed URLs in your sitemap sends a search engine a contradictory signal and wastes crawl budget on pages you've already told it not to index. Second, the sitemap protocol caps a single sitemap file at 50,000 URLs or 50MB uncompressed, whichever comes first; larger sites need a sitemap index file that references multiple child sitemaps, commonly split by content type (products, blog posts, category pages) so you can also track indexing health per section in Search Console.

Structured Data and Schema Markup

Structured data is markup, almost always JSON-LD today, that describes your content in a machine-readable format search engines and AI systems can parse directly, instead of inferring meaning from your HTML. A basic Organization schema block looks like this:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Your Company",
  "url": "https://yourdomain.com",
  "logo": "https://yourdomain.com/logo.png"
}
</script>

The highest-value schema types for most sites are Organization or LocalBusiness (brand identity), Article or BlogPosting (content pages), FAQPage (question-and-answer content, a strong lever for both featured snippets and AI citation), BreadcrumbList (navigation context), and Product with Review or AggregateRating for e-commerce. Two implementation rules matter more than which types you use: the markup must accurately describe content that's actually visible on the page (marking up fake reviews or a fake FAQPage that doesn't correspond to real on-page content is a policy violation that can trigger a manual action), and JSON-LD should be validated, not assumed correct, since a single malformed comma silently breaks the entire block.

Don't guess whether your markup is valid. Google's own Rich Results Test (available at search.google.com/test/rich-results) parses a live URL or a pasted code snippet and tells you exactly which fields it found, which are missing, and whether the block qualifies for a rich result at all, which is a faster feedback loop than waiting to see if a rich snippet shows up in search days later.

Core Web Vitals and Page Experience

Core Web Vitals are Google's three standardized metrics for real-world page experience, each with published good, needs-improvement, and poor thresholds:

Metric	What it measures	Good	Needs improvement	Poor
LCP (Largest Contentful Paint)	Loading, time until the largest visible element renders	2.5s or less	2.5s to 4s	Over 4s
INP (Interaction to Next Paint)	Responsiveness, delay between a user interaction and the next visual update	200ms or less	200ms to 500ms	Over 500ms
CLS (Cumulative Layout Shift)	Visual stability, how much content unexpectedly shifts as the page loads	0.1 or less	0.1 to 0.25	Over 0.25

These are field metrics, measured from real user visits (Chrome User Experience Report data), not a single lab test run. That distinction matters for debugging: a page can score well in a local Lighthouse run and still show a poor field LCP, because Lighthouse tests one simulated connection and device profile, while the field data reflects your actual visitors' real devices and network conditions, which for many sites skew toward slower mobile connections than a developer's own machine.

The most common developer-side fixes, in rough order of impact: set explicit width and height attributes (or aspect-ratio in CSS) on images and embeds to prevent layout shift, preload the LCP element (usually a hero image or heading) rather than lazy-loading it, defer or remove render-blocking JavaScript and CSS that isn't needed for the initial paint, and break up long JavaScript tasks so the main thread stays free to respond to input.

JavaScript Rendering: What Crawlers Actually See

Googlebot renders JavaScript, but it does so in a second wave, separate from the initial crawl and indexing pass, and other crawlers (including several AI crawlers) render JavaScript poorly or not at all. This matters most for content that only appears after a client-side fetch, common in single-page applications and heavily client-rendered React or Vue sites.

The most reliable way to check what a crawler actually sees, rather than guessing, is to fetch the page the way a bot would, without executing JavaScript:

curl -s -A "Googlebot" https://yourdomain.com/your-page | less

If your primary content, title tag, and links aren't present in that raw HTML response, a crawler that doesn't execute JavaScript won't see them either. The reliable fix is server-side rendering or static generation for anything you need indexed (Next.js's generateStaticParams and server components, for example, ship real HTML in the initial response), reserving pure client-side rendering for content that genuinely doesn't need to be crawled, like an authenticated dashboard.

HTTPS, Security Headers, and Mobile-Friendliness

HTTPS has been a baseline ranking signal for years at this point, and Chrome actively flags plain HTTP pages as "Not Secure" in the address bar, which affects user trust independent of any ranking impact. Beyond the certificate itself, check that HTTP genuinely 301-redirects to HTTPS (not 200s both ways, which creates duplicate content), and that you aren't mixing HTTP and HTTPS resources on the same page (mixed content), which browsers will block or warn about.

Verify your redirect chain directly:

curl -sI http://yourdomain.com/ | grep -i location

Mobile-friendliness matters because Google has used mobile-first indexing since 2019, meaning the mobile version of your page, not the desktop version, is what gets crawled and evaluated primarily. If your mobile page is missing content, links, or structured data that your desktop page has, you're effectively hiding that content from the ranking algorithm, not just from mobile visitors.

International SEO: hreflang and Multi-Region Sites

hreflang tags tell search engines which language and regional variant of a page to show a given searcher, preventing the wrong-language version from ranking in the wrong market and, more subtly, preventing search engines from treating your language variants as duplicate content competing against each other. A correct hreflang block, placed either in the HTML head or the sitemap, looks like this for a page with English, Korean, and a language-neutral default:

<link rel="alternate" hreflang="en" href="https://yourdomain.com/en/page/" />
<link rel="alternate" hreflang="ko" href="https://yourdomain.com/ko/page/" />
<link rel="alternate" hreflang="x-default" href="https://yourdomain.com/en/page/" />

Two rules cause most real-world hreflang bugs. First, hreflang must be reciprocal: if the English page points to the Korean page, the Korean page must point back to the English page, or the annotation is ignored. Second, every page in a hreflang set, including the page itself, must be listed, self-referencing hreflang included. A single-locale site with no actual translated content doesn't need hreflang tags at all; adding them without genuinely distinct localized content behind each URL doesn't help, and pointing multiple hreflang tags at pages that redirect to the same content defeats the purpose entirely.

Log File Analysis: What's Actually Crawling Your Site

Search Console and third-party crawl tools tell you what a search engine reports about your site. Server log files tell you what actually happened, every single request, including bot traffic that never shows up in an analytics tool because analytics scripts don't fire for a bot that doesn't execute JavaScript. Filtering your access logs for a known crawler's user agent shows you exactly which URLs it's requesting and how often:

grep "Googlebot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Log analysis answers questions no other tool can: is a crawler wasting requests on parameter variants or old redirected URLs, is it hitting your most important pages at all, and is it respecting your robots.txt (bad bots frequently don't). For large sites, this is often the single highest-signal technical SEO exercise available, because it's the only place you see ground truth crawl behavior instead of a sampled or self-reported summary.

AI Crawlers and Technical SEO in 2026

The crawler landscape has genuinely expanded beyond traditional search engines. GPTBot, ClaudeBot, PerplexityBot, and several others now crawl the web to power AI-generated answers, and whether they can access your content is now its own technical SEO concern, distinct from but built on the exact same foundations covered above: crawlability, clean HTML in the initial response, and structured data. A robots.txt that explicitly allows the major AI crawlers looks like this:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Blocking these crawlers is a legitimate choice for some sites (content licensing concerns are real), but it should be a deliberate decision, not an accident inherited from a boilerplate robots.txt template that blocks everything unfamiliar by default. The same technical fundamentals that make a page crawlable and indexable by Google also make it visible to AI systems: server-rendered content, clean semantic HTML, and accurate structured data all directly support both. For the content and positioning side of this (how to actually get cited inside an AI-generated answer, not just crawled), see our guide to getting cited by ChatGPT and our breakdown of how SEO, AEO, and GEO actually differ.

A Practical Technical SEO Checklist (Priority Order)

Fix these roughly in order. Earlier items block everything below them from mattering.

1. Confirm robots.txt isn't blocking the whole site (check for a stray Disallow: / left over from staging).
2. Confirm your homepage and key pages return a real 200 status, not a soft 404 or an unexpected redirect.
3. Check noindex and canonical tags aren't contradicting each other, and that nothing important is accidentally noindexed.
4. Verify your sitemap only contains canonical, indexable, 200-status URLs.
5. Fetch key pages without JavaScript and confirm your title, main content, and links are present in the raw HTML.
6. Check Core Web Vitals field data in Search Console or CrUX, not just a single Lighthouse run.
7. Confirm HTTPS is enforced everywhere with no mixed content and a clean HTTP-to-HTTPS redirect.
8. Validate structured data and confirm it matches what's actually visible on the page.
9. If multi-locale, verify hreflang is reciprocal and self-referencing.
10. Check server logs (or bot-specific analytics) for how major crawlers, including AI crawlers, are actually behaving on your site.

Frequently Asked Questions

What is technical SEO?
Technical SEO is the set of optimizations that determine whether search engines and AI crawlers can crawl, render, and index a website at all, covering things like crawlability, site speed, structured data, and JavaScript rendering, as distinct from on-page content quality or off-page backlink authority.

How is technical SEO different from on-page SEO?
Technical SEO is about infrastructure: can a crawler reach and understand your pages. On-page SEO is about content: once a crawler can see a page, is the content relevant, well-structured, and worth ranking. A page can have perfect on-page content and still fail to rank if it has a technical SEO problem blocking it from being crawled or indexed.

What's the single highest-priority technical SEO fix for most sites?
Confirming the site is actually crawlable and indexable, no stray robots.txt block, no accidental noindex tags, no contradictory canonical tags. Every other technical SEO improvement is wasted effort if search engines can't reach or index the page in the first place.

Do I need to worry about crawl budget on a small site?
Generally no. Crawl budget becomes a real constraint on large sites, typically tens of thousands of pages or more, especially programmatic or faceted-search sites that can generate near-infinite URL combinations. A small marketing site with a few hundred clean, indexable pages rarely runs into crawl budget limits.

How do I check what a search engine actually sees on my page?
Fetch the page with curl using a search engine's user agent string and look at the raw HTML response, before any JavaScript executes. If your main content, title, or links aren't present in that raw response, a crawler that doesn't render JavaScript won't see them either.

Should I block AI crawlers like GPTBot or ClaudeBot?
It depends on your goals. Blocking them is a legitimate choice if you have content licensing concerns, but it should be a deliberate decision, since these crawlers are what let your content get cited in AI-generated answers. Many robots.txt templates block unfamiliar user agents by default, which can block AI crawlers by accident rather than by choice.