Top 13 robots-txt open source projects

Polite, slim and concurrent web crawler.

Gatsby plugin that automatically creates robots.txt for your site

grobotstxt is a native Go port of Google's robots.txt parser and matcher library.

Parser for robots.txt for node.js

A webpack plugin to generate a robots.txt file

🧑🏻👩🏻 "We are people, not machines" - An initiative to know the creators of a website. Contains the information about humans to the web building - A Nuxt Module to statically integrate and generate a humans.txt author file - Based on the HumansTxt Project.

.Net Core Plugin Manager, extend web applications using plugin technology enabling true SOLID and DRY principles when developing applications

Java sitemap generator. This library generates a web sitemap, can ping Google, generate RSS feed, robots.txt and more with friendly, easy to use Java 8 functional style of programming

Ultimate Website Sitemap Parser

NodeJS robots.txt parser with support for wildcard (*) matching.

Provides robots.txt middleware for .NET core

🤖 robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API

A pure-Python robots.txt parser with support for modern conventions.