---
title: "Block Non-Human Crawlers with Go"
url: https://daily.dev/posts/block-non-human-crawlers-with-go-jrbnokeax
source_url: https://blainsmith.com/articles/block-non-human-crawlers-with-go
type: article
source: "Blain Smith"
published: 2026-05-24T07:43:41.057Z
updated: 2026-05-24T07:44:20.264Z
tags: ["golang", "crawling"]
reading_time: 4
upvotes: 0
comments: 0
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Block Non-Human Crawlers with Go

**[Blain Smith](https://daily.dev/sources/blainsmith)** · 4 min read · 0 upvotes · 0 comments

## Summary

A Go implementation of cookie-based bot blocking middleware that prevents non-human crawlers (AI bots, LLM scrapers) from accessing web content. The middleware wraps any http.Handler and works by presenting a simple checkbox form to visitors without a valid cookie. It hashes the User-Agent string using a configurable hash.Hash (e.g., HMAC-SHA256), sets a cookie upon form submission, and only allows requests through when the cookie value matches the hashed User-Agent. It skips non-HTML requests to avoid breaking images or RSS feeds. The author intentionally keeps it as copy-paste code rather than a published Go module, citing simplicity and low maintenance overhead.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://blainsmith.com/articles/block-non-human-crawlers-with-go>

## Similar posts on daily.dev

- [Blocking LLM crawlers, without JavaScript](https://daily.dev/posts/blocking-llm-crawlers-without-javascript-5rpssvlpe) · Lobsters · 4 upvotes · 0 comments
- [A modern approach to preventing CSRF in Go– Alex Edwards](https://daily.dev/posts/a-modern-approach-to-preventing-csrf-in-go-alex-edwards-5b32watfz) · Lobsters · 1 upvotes · 0 comments
- [humanely dealing with humungus crawlers](https://daily.dev/posts/humanely-dealing-with-humungus-crawlers-ztr9qecyk) · Lobsters · 0 upvotes · 0 comments
- [How I protect my forgejo instance from AI Web Crawlers](https://daily.dev/posts/how-i-protect-my-forgejo-instance-from-ai-web-crawlers-jmcp9dojw) · Lobsters · 1 upvotes · 0 comments

---

Tags: [#golang](https://daily.dev/tags/golang), [#crawling](https://daily.dev/tags/crawling)

[View this post on daily.dev](https://daily.dev/posts/block-non-human-crawlers-with-go-jrbnokeax)