506 Open source projects that are alternatives of or similar to Crawlerforreader

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

Java 網路資料爬蟲包

基于appium的app自动遍历工具

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用nomad管理docker集群； 使用EFK查询docker日志

GraphQuery is a query language and execution engine tied to any backend service.

jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.

本仓库收集整理爬虫相关资源，开发语言以Java为主

A simple and flexible web crawler framework for java.

读书笔记《自己动手写网络爬虫》，自己敲的代码。主要记录了网络爬虫的基本实现，网页去重的算法，网页指纹算法，文本信息挖掘

🇰🇷 Realtime integrated information analysis service

Easily create XML sitemaps for your website.

微信公众号爬虫，公众号历史文章，文章评论，文章阅读及在看数据，可视化web页面，可部署于Windows服务器。基于Python3之flask/mysql/redis/mitmproxy/pywin32等实现，高效微信爬虫，微信公众号爬虫，历史文章，文章评论，数据更新。

Source code for the IndieWeb search engine.

Web Application Security Scanner Framework

🐛一只登录正方教务管理系统，爬取数据的小爬虫

No description or website provided.

磁力網站U3C3介紹以及域名更新

Az összes számlád még egy helyen :)

🍰 一个可视化的爬虫平台

An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

This is a Multi-thread crawler for Tumblr.

Easy way to brute-force web directory.

Crawling CV conference papers with Python.

Just a crawler based on tg-cli for Telegram. Deprecated by now, please use telegram-export.

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Scraper example built on Scala, Akka and Jsoup

Simple yet powerful automation stuffs.

an async ETL tool written in Python.

An R web crawler and scraper

A fluent and functional approach to querying HTML

The fastest dork scanner written in Go.

Crawl a website and take screenshots

line-bot-tutorial use python flask

TumblTwo, an Improved Fork of TumblOne, a Tumblr Downloader.

一个灵活、友好的爬虫框架

一个轻量级、快速、多线程、多管道、灵活配置的网络爬虫。

Update Version of weibo_terminator, This is Workflow Version aim at Get Job Done!

Videodl: A lightweight video downloader written by pure python.

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

tw.pycon.org/2017/events/talk/314386410792550475/

The simple, easy to use command line web crawler.

爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例

无cookie版微博爬虫，可以连续爬取一个或多个新浪微博用户信息、用户微博及其微博评论转发。

🍭 epub generator for (light)novels (轻) 小说 epub 生成器，支持站点：轻之国度、轻小说文库

🌀 crawl bilibili user info and video info for data analysis | BiliBili爬虫

🐤️ Lost Ark wait notifier

A fast tool to fetch URLs from HTML attributes by crawl-in.

Fetching hotel data from TripAdvisor.

🤖 Prerendering for JavaScript powered websites. Great solution for PWAs (Progressive Web Apps), SPAs (Single Page Applications), and other websites based on top of front-end JavaScript frameworks

eXist Native XML Database and Application Platform

Crawlera middleware for Scrapy

Recursive and multi-threaded broken link checker

PTT 網路版爬蟲

A Python DNS crawler to find identical domain names under different TLDs.

模拟登录各类网站，操作 API 完成各种不可描述的事情

Practical pinned gist which show your latest medium status 📌

Google search results crawler, get google search results that you need - php