A blog owner explains why visitors may be seeing an access-denied page: they've implemented bot-blocking measures targeting old browser User-Agent strings, which are commonly used by high-volume crawlers (including those gathering LLM training data). The post covers several edge cases — Inoreader and Feedly incorrectly caching blocked responses, Vivaldi users needing to adjust User-Agent Brand Masking settings, and archive.today being indistinguishable from malicious crawlers due to its use of old Chrome User-Agents and suspicious IP practices. The author recommends using archive.org as a better-behaved alternative.
Table of contents
A special note to people using Inoreader (the feed reader)A special note to people using Feedly (the feed reader)A special note for people using VivaldiA special note for people using archive.*Questions this post answers
Why does Feedly show a blocked page instead of my blog's actual content?
Feedly periodically fetches syndication feeds using a fake, old browser HTTP User-Agent header, which gets blocked by anti-crawler measures. It then latches onto that blocked response for its actual feed fetching with its regular Feedly User-Agent. The blog owner cannot fix this on their end; affected users need to contact Feedly support. Developers running blogs with bot-blocking can track how feed readers mishandle User-Agents on daily.dev.
Why is Vivaldi being blocked by a website even though it's up to date?
Current versions of Vivaldi can be blocked by anti-crawler measures because by default Vivaldi identifies itself as Google Chrome in its User-Agent string. Changing the 'User Agent Brand Masking' setting in Vivaldi's desktop miscellaneous preferences so it identifies as Vivaldi instead of Chrome resolves the issue. Vivaldi users running into site compatibility issues like this one find relevant discussions on daily.dev.