37 Open source projects that are alternatives of or similar to robots.txt

A pure-Python robots.txt parser with support for modern conventions.

NodeJS robots.txt parser with support for wildcard (*) matching.

Reliable and modern Redis Streams based task queue for Node.js 🤖

Grammar-based Clojure(script) parser

Java Predicate, supports SQL-like syntax

A useful Antlr4 tool with full TypeScript support

对免费代理IP网站进行爬取，收集汇总为自己的代理池。关键是验证代理的有效性、匿名性、去重复

A programming language for FPGAs.

Generate an accelerator extension that makes your Antlr parser in Python super-fast!

Java Parser for JavaScript/TypeScript (based on antlr4ts)

LightJason - AgentSpeak(L++) for Java

项目已经移动至：https://github.com/BaiduSpider/BaiduSpider ！！ 一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

An example of using Redis Streams, RedisGears and RedisAI for Realtime Video Analytics (i.e. filtering cats)

Polite, slim and concurrent web crawler.

Gatsby plugin that automatically creates robots.txt for your site

grobotstxt is a native Go port of Google's robots.txt parser and matcher library.

Parser for robots.txt for node.js

A webpack plugin to generate a robots.txt file

🧑🏻👩🏻 "We are people, not machines" - An initiative to know the creators of a website. Contains the information about humans to the web building - A Nuxt Module to statically integrate and generate a humans.txt author file - Based on the HumansTxt Project.

.Net Core Plugin Manager, extend web applications using plugin technology enabling true SOLID and DRY principles when developing applications

Java sitemap generator. This library generates a web sitemap, can ping Google, generate RSS feed, robots.txt and more with friendly, easy to use Java 8 functional style of programming

Ultimate Website Sitemap Parser

Provides robots.txt middleware for .NET core

ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.

Here lies the code for the Snow* programming language, currently being rewritten.

Kotlin Language Support – AST Library

An Antlr4 implementation of xqDoc for XQuery

Scripting language for derps

Visual Studio debugging visualizer, and .NET visualization controls, for ANTLR4 parse trees

A language to generate command blocks for Minecraft 1.9 and higher

Коллекция парсеров языка 1С (BSL) в формате ANTLR4.

模拟登陆基本采用的是直接登录或者使用selenium+webdriver的方式，有的网站直接登录难度很大，比如qq空间，bilibili等如果采用selenium就相对轻松一些。

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

使用Pyspider框架的豆瓣爬虫

scrapy 常用爬网必备工具包

some small project and some articles

robots.txt parser for Rust.