flowingkhaosSign in
SEO in the age of AI: get your pages indexed, from the APIstep 1 of 7 · 0 of 17 carriedSign in to keep it
Step 1 of 7 · weight 1 · about 20 minutes

Read it, then pass the check at the end to open step 2.

Early game: what Google actually does with a page before it indexes it

Updated

Here is how to get google to index your site, in one sentence: Google indexes a page when it can fetch it, reads one canonical for it, and reaches it through links from pages it already crawls, and the only reliable way to find which of those three is failing is to read every sitemap URL through the Search Console API, one call per URL, and repair what comes back before you click Request Indexing. This workshop is that sentence as six steps, with the numbers from my two sites and their dates.

Before you start

What you need

Tool or account Version Why this workshop needs it Free
Search Console, owner or full access to your property current every read and every request yes
A Google Cloud service account added as a user on that property current the API calls in step 2 yes
curl any the live checks in step 3 yes
Bun or Node Bun 1.2 or later to run the read script in step 2 yes

Get it on your machine

brew install oven-sh/bun/bun && bun --version
powershell -c "irm bun.sh/install.ps1 | iex"; bun --version
curl -fsSL https://bun.sh/install | bash && ~/.bun/bin/bun --version

Expected: 1.2.x or later.

Known gotchas

On Linux ~/.bun/bin is not on PATH until you open a new shell; on Windows the PowerShell execution policy can refuse the installer, so run it from an elevated prompt; on macOS the first curl may prompt for the Xcode command line tools.

The four stages

Google fetches the page. Then it renders it, running whatever script the page needs to show its text. Then it picks one canonical, the address it will treat as the page’s real one. Then, if it chooses to, it indexes. seo-kreativ’s guide is the clearest technical framing I found of crawling as a separate stage from indexing, and I agree with the split; where it stops is at diagnosis, which is where a workshop has to start. Conductor’s guide reports the same stages and adds that a page Google cannot reach through links is a page it may never fetch, which is the stage most of my stuck pages were failing at.

Check your own first stage now. On 2026-10-14 I fetched clevrstart’s home page as Googlebot and as a browser: identical HTML, status 200, the text server-rendered, 2,398 words on the English home page and 2,609 on the French one, no x-robots noindex header. If your page passes that, the fetch and render stages are not your problem, and the dashboard, which cannot tell you which stage failed, is not your tool.

The gate · a reading check closes this stepweight 1 of Σw 14

Check what you read in “Early game: what Google actually does with a page before it indexes it”

Three questions drawn from a larger pool, so no two papers are quite the same. Two right passes it and opens the next step. The questions are public; scoring them needs an account, and the key never leaves the server.

Loading the check…

All 7 stepsTake the check

The columns, fixes and workshops continue in the community: SEO, AI, and making money with code.

Join on Whop

Luke Sidney writes and builds everything here, alone, in the open. lukesidney.me