493 Open source projects that are alternatives of or similar to crawlerdetect

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

📃 The main project repository

The Universal Device Detection library will parse any User Agent and detect the browser, operating system, device used (desktop, tablet, mobile, tv, cars, console, etc.), brand and model.

A Laravel wrapper for CrawlerDetect - the web crawler detection library

Detect device, and render view according to detected device type.

资讯爬虫App

gathertool是golang脚本化开发库，目的是提高对应场景程序开发的效率；轻量级爬虫库，接口测试&压力测试库，DB操作库等。

scala结合actor编写的分布式网络爬虫

零基础学习python爬虫

该项目是一个使用celery作为主体框架的爬虫应用，能够灵活的添加爬虫任务，并且同时运行多站点的爬虫工作，所有组件都能够原生支持规模并发和分布式，加上celery原生的分布式调用，实现大规模并发。

Detect browser language

Generator of User-Agent header

[W.I.P] An ecosystem of crawlers for detecting: leaks, sensitive data exposure and attempts exfiltration of data

⭐ 图虫网爬虫

犇犇爬虫

A web crawler for Go

scrapy spider for scraping imdb {movie_id: [recommended, ...]}

基于Electron爬虫程序

List of major web + mobile browser user agent strings. +1 Bonus script to scrape :)

这可能是爬百度文库最全的项目了

✨ Mojolicious - Perl real-time web framework

A spiders' program for weixin which made by Express & cheerio

Do some browser detection with Ruby. Includes ActionController integration.

Integrate user-agent detection in an idiomatic React way

Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data

⚡ Light weight Golang spider framework | 轻量的 Golang 爬虫框架

Parse and process User Agents like a secret one

🕷️用于爬取B站前top100的小视频

A spider on Dcard. Strong and speedy.

裁判文书网爬虫

ECMAScript feature and API detection

🚦 Block malicious crawlers with Cloudflare Firewall Rules

🕷️ A simple crawler (spider) writen in php just for fun, with zero dependencies

项目已经移动至：https://github.com/BaiduSpider/BaiduSpider ！！ 一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

Grabs all of the audio files from all of the Blinkist books

超高速异步协程Python爬虫

文章采集工具 Article collection tool

A spider tapestry weaver

蜘蛛纸牌 for mac

一个简单的基于DHT协议的BT磁力链接爬虫

This taobao spider has been archived

Angular module to detect OS / Browser / Device

Generate an object for testing if a request is sent, request is Mikeal's request.

😎 Google chrome browser extension

Grapy, a fast high-level web crawling framework for Python 3.3 or later base on asyncio.

A precise user agent parser and device detector written in TypeScript

Your web font utility belt. It can subset web fonts. It can find unicode-ranges for you automatically. It makes julienne fries.

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

The Universal Device Detection library will parse any User Agent and detect the browser, operating system, device used (desktop, tablet, mobile, tv, cars, console, etc.), brand and model.

bet365 bot: bet365的比赛实时比分数据、实时赔率

基于nodejs的网络聊天室、爬虫，vue音乐播放器，及php后台开发的管理系统等项目

Browser sniffing gone too far — A useragent parser library for PHP

🐕 Fetch+ is a convenient Fetch API replacement with first-class middleware support.

妹子图

Iota is a web scraper which can find all of the images and links/suburls on a webpage

Proxying Python Requests

A User-agent analyze project.

Crawling GitHub Trending Pages every day

图片爬取下载工具，极速爬取下载 站酷https://www.zcool.com.cn/, CNU 视觉 http://www.cnu.cc/ 设计师/用户 上传的 图片/照片/插画。

python 爬虫(amazon, confluence ...)