1587 Open source projects that are alternatives of or similar to wget-lua

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

[Unmaintained] A simple and clean video/music/image downloader 👾

Elegant Scraper and Crawler Framework for Golang

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

Crawly, a high-level web crawling & scraping framework for Elixir.

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

An open source web crawling platform

A Facebook crawler

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

A bash script to spider a site, follow links, and fetch urls (with built-in filtering) into a generated text file.

Extract structured data from web sites. Web sites scraping.

Declarative web scraping

Distributed crawler powered by Headless Chrome

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

A Scraper, Downloader, & Recorder for static open directories.

ProxyCrawl Python library for scraping and crawling

No description or website provided.

Creating Scrapy scrapers via the Django admin interface

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Anime-dl is a command-line program to download anime from CrunchyRoll and Funimation.

simple multi-level scraper json input/output for Cheerio

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

A PHP Scraping Class

Universal scrapping tool, which allows you to extract data using multiple environments

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

项目已经移动至：https://github.com/BaiduSpider/BaiduSpider ！！ 一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

SEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type from searchresults for given keywords. Detect Ads or make automated screenshots. You can also fetch text content of urls provided in searchresults or by your own. It's usefull for SEO and business related research tasks.

👾 Fast and simple video download library and CLI tool written in Go

reborn of https://bitbucket.org/rflechner/scrapysharp

This class can retrieve search results from Google.

PHP Scraper - an highly opinionated web-interface for PHP

💡 Download the complete source code of any website (including all assets). [ Javascripts, Stylesheets, Images ] using Node.js

⚡ Light weight Golang spider framework | 轻量的 Golang 爬虫框架

Tutorial for web scraping / crawling with Node.js.

Pahe.ph [Pahe.in] Movies Website Scraper

**[ARCHIVED]** website changes tracker 🔍

A dockerized, queued high fidelity web archiver based on Squidwarc

gathertool是golang脚本化开发库，目的是提高对应场景程序开发的效率；轻量级爬虫库，接口测试&压力测试库，DB操作库等。

Instagram Photo Downloader

Advanced python library to scrap Twitter (tweets, users) from unofficial API

👨‍👩‍👦 Social account detection and extraction in Python, e.g. for crawling/scraping.

Your will to enroll in Udemy course is here, but the money isn't? Search no more! This python program searches for your desired course in more than [insert big number here] websites, compares the last updated date, and gives you the download link of the latest one back, but you also have the choice to see the other ones as well!

📥 Downloader for lezhin comics

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

scrapers for building your own image databases

Simply scrape / download all the media from an fansly account

Grabs all of the audio files from all of the Blinkist books

A Scrapy extension to log items coverage when the spider shuts down

Bulk Downloader for Reddit

Easily allows for scraping a VSCO

图片爬取下载工具，极速爬取下载 站酷https://www.zcool.com.cn/, CNU 视觉 http://www.cnu.cc/ 设计师/用户 上传的 图片/照片/插画。

Continuously search imageboards threads for images/webms and download them

带图形界面的哔哩哔哩漫画下载工具

Easily crawl news portals or blog sites using Storm Crawler.

🧊 一个可爱且任性的 B 站视频下载器（bilili V2）

Meter Information Collection System

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音|TikTok数据爬取工具，支持API调用，在线批量解析及下载。