olli/olli.puhakka.dev
Watch
1
0
Fork
You've already forked olli.puhakka.dev
 0
Code Issues 5 Pull requests Projects Releases Packages 2 Wiki Activity Actions

Point search crawlers at the pages, keep AI training crawlers out #27

Merged
olli merged 3 commits from feat/seo into dev 2026-10-15 15:53:39 +03:00
Conversation 0 Commits 3 Files changed 8 +323 -8
olli-agent commented 2026-10-15 15:37:04 +03:00
Collaborator
Copy link

Makes the site findable by search engines once it has been crawled, by doing what their own documentation asks and nothing more.

  • SEO.md: what Google, Bing, DuckDuckGo and the AI search engines (OpenAI, Anthropic, Perplexity) document about crawling and indexing, with sources checked 2026-10-15, and the decisions taken from it.
  • public/robots.txt: every page open to search crawlers, /api/ closed, and the AI training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) turned away. Their companies' search crawlers are separate user agents and stay allowed.
  • public/sitemap.xml: the four pages, named in robots.txt. No lastmod, since nothing would keep a hand-written date accurate.
  • Canonical links on the four pages.
  • ProfilePage JSON-LD on the CV, matching what the CV shows.
  • One job title, Senior software engineer, on the landing page and the CV (titles, descriptions, eyebrow lines, JSON-LD), in English and Finnish. The landing pitch and lede now describe a generalist instead of a solution architect.

Blocking Google-Extended also keeps the site out of Gemini's grounding outside Search. Google Search and its AI Overviews are governed by Googlebot and are unaffected.

Finnish pages with hreflang are tracked in #26.

After merge to main: verify the domain in Google Search Console and Bing Webmaster Tools through DNS, and submit the sitemap.

Makes the site findable by search engines once it has been crawled, by doing what their own documentation asks and nothing more. - **`SEO.md`**: what Google, Bing, DuckDuckGo and the AI search engines (OpenAI, Anthropic, Perplexity) document about crawling and indexing, with sources checked 2026-10-15, and the decisions taken from it. - **`public/robots.txt`**: every page open to search crawlers, `/api/` closed, and the AI training crawlers (`GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `CCBot`) turned away. Their companies' search crawlers are separate user agents and stay allowed. - **`public/sitemap.xml`**: the four pages, named in robots.txt. No `lastmod`, since nothing would keep a hand-written date accurate. - **Canonical links** on the four pages. - **`ProfilePage` JSON-LD** on the CV, matching what the CV shows. - **One job title**, Senior software engineer, on the landing page and the CV (titles, descriptions, eyebrow lines, JSON-LD), in English and Finnish. The landing pitch and lede now describe a generalist instead of a solution architect. Blocking `Google-Extended` also keeps the site out of Gemini's grounding outside Search. Google Search and its AI Overviews are governed by Googlebot and are unaffected. Finnish pages with `hreflang` are tracked in #26. After merge to `main`: verify the domain in Google Search Console and Bing Webmaster Tools through DNS, and submit the sitemap.
The site should be findable once crawled, without tricks. SEO.md
collects what Google, Bing, DuckDuckGo and the AI search engines
document about crawling and indexing, with sources, so the changes
that follow can be checked against them, and re-checked when the
crawlers change.
Point crawlers at the pages, and keep AI training crawlers out
All checks were successful
CI / lint (push) Successful in 2s
Details
CI / server (push) Successful in 30s
Details
CI / image (push) Successful in 4s
Details
CI / verify-staging (push) Has been skipped
Details
fd08f69c4e 
Search engines found the pages only through links, and could not tell
whom the CV is about. robots.txt now names a sitemap of the four pages,
each page states its canonical address, and the CV carries ProfilePage
markup that matches what it shows.

The site is not offered as AI training material, so robots.txt turns
away the training crawlers by name. Their companies' search crawlers
are separate user agents and stay allowed, so the site can still be
found and cited in AI search.
Use one job title, Senior software engineer, on every page
All checks were successful
Release / find-image (push) Successful in 1s
Details
Release / verify-production (push) Successful in 11s
Details
CI / lint (push) Successful in 2s
Details
CI / server (push) Successful in 28s
Details
CI / image (push) Successful in 5s
Details
CI / verify-staging (push) Has been skipped
Details
a47c4caac8 
The landing page said solution architect, the CV senior software and AI
platform engineer, and its description something else again. Search
engines tie pages to a person partly through such details, and readers
compare them with LinkedIn. There is no official title to follow, so
the title names the trade, and the landing page's lede says the rest:
a generalist who designs the system, writes the code and runs the
servers. Solution architect also reads as a pre-sales role, which the
site is not.
olli merged commit a47c4caac8 into dev 2026-10-15 15:53:39 +03:00
olli deleted branch feat/seo 2026-10-15 15:53:39 +03:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
olli/olli.puhakka.dev!27
No description provided.