1249 Open source projects that are alternatives of or similar to robotstxt

A python web scraper for public email lists.

Creating Scrapy scrapers via the Django admin interface

Be nice on the web

News extraction and scraping. Article Parsing

CRU CL v. 2.0 Climatology Client for R

A web spider for shodan.io without using the Developer API.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

一个基于webmagic框架二次开发的java爬虫框架实战，已实现能爬取腾讯，搜狐，今日头条（单独集成功能）等资讯内容，配合elasticsearch框架用法，实现了自动爬虫，已投入线上生产使用。

OSINT Swiss Army Knife

A collection of awesome web crawler,spider in different languages

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

✈️ ☀️ R package for accessing ASOS data via the Iowa Environment Mesonet ☁️ ✈️

Humane encryption

Get Aliexpress product details in JSON

Use Unpaywall with R

High performance crawler for Nodejs

Scrape Instagram content and stories anonymously, using a new technique based on the har file (No Token + No public API).

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

API for fetching data from news websites.

Create agents that monitor and act on your behalf. Your agents are standing by!

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

A web crawler (for bug hunting) that gathers more than you can imagine.

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

Elegant Scraper and Crawler Framework for Golang

📦 High level wrapper around the redland package for common rdf applications

⛔ ARCHIVED ⛔ R package for the Geoparser.io API

A R Client for the Bittrex Crypto-Currency Exchange

📦 R package to simulate neutral landscape models 🏔

R interface to rlite https://github.com/seppo0010/rlite

An open source webapp for scraping: towards a public service for webscraping

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

⛔ ARCHIVED ⛔ Accesses Air Quality Data from the Open Data Platform OpenAQ

bingip2hosts is a Bing.com web scraper that discovers websites by IP address

Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.

An R web crawler and scraper

PHP Metacritic API - Mirrored by my GitLab

快速、简洁且强大的PHP爬虫框架

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

Crawly, a high-level web crawling & scraping framework for Elixir.

🌐 R package for the OpenCage API -- both forward and reverse geocoding 🌐

Scraping scripts for various websites.

🌎 🏨 Scrape Booking.com 🏨 🌎

A web crawler for Go

A high performance web crawler in Elixir.

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

This repository contains all the code I use in my YouTube tutorials.

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

图片爬取下载工具，极速爬取下载 站酷https://www.zcool.com.cn/, CNU 视觉 http://www.cnu.cc/ 设计师/用户 上传的 图片/照片/插画。

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音|TikTok数据爬取工具，支持API调用，在线批量解析及下载。

nlrx NetLogo R

rdefra: Interact with the UK AIR Pollution Database from DEFRA

Grabbing SUPPlementary DATA in R

API wrapper around postcodes.io - free UK postcode lookup and geocoder

R package for downloading weather data from Environment and Climate Change Canada

A Facebook crawler

爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer

Grabs all of the audio files from all of the Blinkist books

Access and search medRxiv and bioRxiv preprint data