1004 Open source projects that are alternatives of or similar to Xcrawler

百度贴吧生日爬虫，可抓取贴吧内吧友生日，并且在对应日期自动发送祝福

Scraping the Un–scrapable™

A scraper for videos that are trending on YouTube (https://www.youtube.com/feed/trending)

Fast concurrent image scraper

bingip2hosts is a Bing.com web scraper that discovers websites by IP address

PTT 表特版爬蟲圖片下載器

Aiohttp web server API, which scrapes Google and returns scrape results as response. Supports proxies, multiple geos and number of results.

No description or website provided.

Instagram application for Django.

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

A scraper for obtaining information on the workings of the Belgian Federal Parliament.

爬虫管理系统，支持集群，弹性伸缩。支持运行feapder、scrapy、selenium、playwright等各种框架及脚本

Python script to download messages from a Facebook page to a CSV file

A collection of tools to (semi-)automatically collect and analyze data from online discussions on Facebook groups and pages.

搜索你的知乎收藏：可以直观地浏览你的所有收藏夹的内容，并进行全文搜索

该项目是一个使用celery作为主体框架的爬虫应用，能够灵活的添加爬虫任务，并且同时运行多站点的爬虫工作，所有组件都能够原生支持规模并发和分布式，加上celery原生的分布式调用，实现大规模并发。

DorkScout - Golang tool to automate google dork scan against the entiere internet or specific targets

A django admin site for scrapy

A LinkedIn Scraper to scrape up to 10k LinkedIn profiles from company profile links and save their e-mail addresses if available!

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

Continuously search imageboards threads for images/webms and download them

Golang module to detect bots and crawlers via the user agent

Non-official Python wrapper and CLI tool for Trainline

Interactive CLI Web Crawler

Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

A Python Bot for Scraping Conversations from Twitter

Current and historical coronavirus covid-19 confirmed, recovered, deaths and active case counts segmented by country and region. Includes csv, json and sqlite data along with an interactive website explorer.

Web scraping using python, requests and selenium

python 爬虫(amazon, confluence ...)

A tutorial-based introduction to web scraping with Python.

Aviation charts

PTT 圖片下載器 (抓取整個看板的圖片，並用文章標題作為資料夾的名稱 ) (使用Scrapy)

download document from book118 for free

bet365 bot: bet365的比赛实时比分数据、实时赔率

A set of spiders and scrapers to extract location information from places that post their location on the internet.

这是一个用Python写的小说爬虫软件

V2EX爬虫

A fluent and functional approach to querying HTML

Unofficial client to access your Itaú bank data

🕸 List of mini projects that involve web scraping 🕸

妹子图

Read an Amazon wishlist programmatically with Python

Scraping Visually Similar Images from Pinterest

PHP Metacritic API - Mirrored by my GitLab

蜘蛛纸牌 for mac

Crawl a website and take screenshots

Your web font utility belt. It can subset web fonts. It can find unicode-ranges for you automatically. It makes julienne fries.

一个基于Glang语言开发的站群系统（蜘蛛池系统）

scraper for facebook, gab, google and tiktok

A Python package for acquiring both historical and current data of cryptocurrencies

Cryptocurrency historical price data library in Python. Data from https://coinmarketcap.com.

Easily create XML sitemaps for your website.

TumblTwo, an Improved Fork of TumblOne, a Tumblr Downloader.

client for https://ammobin.ca

🎬 An attempt at the most complete IMDb API

一些爬虫开发的技巧和案例

抖音接口、抖音API、抖音数据爬虫、抖音直播数据、抖音直播Api、抖音视频Api、抖音爬虫、抖音去水印、抖音视频下载、抖音视频解析、抖音直播监控、抖音数据采集

검색어 기준으로 네이버뉴스와 댓글을 수집하는 파이썬 코드

A console application that can scrape the Azur Lane wiki and export the data to Json files