7594 Open source projects that are alternatives of or similar to Easy Scraping Tutorial

一个灵活、友好的爬虫框架

A framework for creating semi-automatic web content extractors

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

Pythonic HTML Parsing for Humans™

Crawl sites for RSS, Atom, and JSON feeds.

Screen scraping and web crawling framework

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

Crawlera middleware for Scrapy

Distributed crawler powered by Headless Chrome

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Scrapy, a fast high-level web crawling & scraping framework for Python.

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用nomad管理docker集群； 使用EFK查询docker日志

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

一个获取知乎用户主页信息的多线程Python爬虫程序。

Python scraper for Language Pods such as Japanesepod101.com 👹 🗾 🍣 Compatible with Japanese, Chinese, French, German, Italian, Korean, Portuguese, Russian, Spanish and many more! ✨

No description or website provided.

Elegant Scraper and Crawler Framework for Golang

Declarative web scraping

Crawly, a high-level web crawling & scraping framework for Elixir.

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

[Unmaintained] A simple and clean video/music/image downloader 👾

HTTP API for Scrapy spiders

ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

A Scrapy extension to log items coverage when the spider shuts down

收集各种免费的 Python 爬虫项目

Script using requests module to register accounts to Slam Jam Socialism raffles.

Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.

🕸 List of mini projects that involve web scraping 🕸

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

Tiktok Downloader/Scraper using requests & bs4

🕷️some python spiders with BeautifulSoup or scarpy

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

Continuous scalable web crawler built on top of Flink and crawler-commons

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

An interactive Command-Line Interface Build in NodeJS for downloading a single or multiple images to disk from URL

HTTP-based DSL for for validating RESTful APIs

🐤️ Lost Ark wait notifier

PTT 網路版爬蟲

API for scrapping common meme sites

A Scraper made 100% in Python using BeautifulSoup and Tor. It can be used to scrape both normal and onion links. Happy Scraping :)

Building a model to recognize incentives for landscape restoration in environmental policies from Latin America, the US and India. Bringing NLP to the world of policy analysis through an extensible framework that includes scraping, preprocessing, active learning and text analysis pipelines.

Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

爬取 www.mzitu.com 全站图片，截至目前共5162个图集，16.5万多张美女图片，使用 asyncio 和 aiohttp 实现的异步版本只需要不到2小时就能爬取完成。按日期创建图集目录，保存更合理。控制台只显示下载的进度条，详细信息保存在日志文件中。支持异常处理，不会终止爬虫程序。失败的请求，下次再执行爬虫程序时会自动下载

Web crawling and scraping framework for Golang

We will try to find your visible basic footprint from social media as much as possible - 😤 more sites is comming soon

The simple, easy to use command line web crawler.

基于搜狗微信搜索的微信公众号爬虫接口

Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy

Türkçe olarak hazırlanmış Python programlama dili ile ilgili içeriklerin derlendiği sayfa.

Scrapy Extension for monitoring spiders execution.

python爬虫

Simple yet powerful automation stuffs.

📇 Extract social media profiles and more with regular expressions

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

🎭 Twisted Deferred Thread backend for Requests.

a reliable high-level web crawling & scraping framework for Node.js.

Build and deploy a machine learning app from scratch 🚀

Collection of useful data science topics along with code and articles