1004 Open source projects that are alternatives of or similar to Newspaper

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

我的爬虫合集

Node.js scraper to get data from Google Play

An R web crawler and scraper

This repository is no longer maintained.

蓝天采集器是一款免费的数据采集发布爬虫软件，采用php+mysql开发，可部署在云服务器，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

Extract structured data from web sites. Web sites scraping.

Scraping scripts for various websites.

Fetches PubMed article IDs (PMIDs) from email inbox, then crawls PubMed, Google Scholar and Sci-Hub for respective PDF files.

A high performance web crawler in Elixir.

Tools for downloading agendas, minutes and other documents produced by local government

Tổng hợp script crawl dữ liệu từ các mạng xã hội & website tiếng Việt

Go process used to crawl websites

This repository contains all the code I use in my YouTube tutorials.

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

Powerful web scraping framework for Crystal

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Headless Chrome For Java （Java 爬虫）

An Instagram bot developed using the Selenium Framework

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.

👾 Fast and simple video download library and CLI tool written in Go

crawler framework, distributed crawler extractor

Golang pkg to quickly return a preview of a webpage (title/description/images)

Scrape everything you can from MyAnimeList.net

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

LinkedIn Scraper (currently working 2020)

news-please - an integrated web crawler and information extractor for news that just works.

Python爬虫，京东自动登录，在线抢购商品

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

📰微信小程序--头条新闻

Fawkes is a tool to search for targets vulnerable to SQL Injection. Performs the search using Google search engine.

获取免费socks/https/http代理的网站集合

A scalable web crawler framework for Java.

🌐 Web scraping made easy with the visual 🗺 mind map editor to JSON

Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure 💫 多语言执行器，分布式爬虫

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

The home of the Haxe Roundup's (Work in Progress)

MM131网站图片爬取 🚨

字体混淆服务

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用nomad管理docker集群； 使用EFK查询docker日志

爬虫, http代理, 模拟登陆!

A Moodle downloader that downloads course content fast from Moodle (eg. lecture pdfs)

OWASP D4N155 - Intelligent and dynamic wordlist using OSINT

A Devtools driver for web automation and scraping

Python based framework to retreive Global Database of Events, Language, and Tone (GDELT) version 1.0 and version 2.0 data.

A Simple News App built with Flutter.

iOS ZhiHuDaily client, implemented in Swift

A dungeon crawler

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

This repository contains code and bonus content which will be added from time to time for the book "Learning Social Media Analytics with R" by Packt

前端娱乐圈大事记 http://qianduanyule.club/

Digger is a powerful and flexible web crawler implemented by pure golang

GUI based offensive penetration testing tool (Open Source)

comic updater, mangafied

Pre-moderated news aggregator

Async Python 3.6+ web scraping micro-framework based on asyncio

APIs for loginning some websites by using requests.

Automate the creation of Python Selenium Scripts by drawing coloured boxes on webpage elements