About the linkchecker
I used an automated link checker to try to reduce the number of broken links that appear on this site. You've probably found the URL of this page in your server logs and visited to find out more.
I am a compulsive outlinker, and this site now contains over a thousand outlinks, not including differences from fragment identifiers. It means that visitors to this site will have lots of other places to explore, but it's especially important for me to try to reduce linkrot.
The linkchecking bot will find the 404 errors — as well as other errors — the idea being that when I have time, I'll then manually go through and try to change the pages to point to available resources, or simply remove the link altogether. I consider pages that require subscription as broken links too. The HTTP status code 402 Payment Required seems the most appropriate way for a server to signal this, but I haven't seen it yet.
My link checker should be respecting the following rules:
- It should visit a URL no more often than once a week. If a URL returns a status code beginning with a 2 (indicating success), it should visit that URL no more often than once a fortnight.
- It should respect the
robots.txtfile on your site and, in particular, politely respect when it's not wanted or when it's rate-limited, including the access of the
- It should omit visits to any domain name whose owner has requested that I exclude it from the link-checking process.
Please get in touch with me if my linkchecker bot is misbehaving or you would like me to blacklist your site, and I'll take swift action to fix it.