824 Open source projects that are alternatives of or similar to Arachnid

Creates an XML-Sitemap by crawling a given site.

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites

Universal scrapping tool, which allows you to extract data using multiple environments

Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Scrapy, a fast high-level web crawling & scraping framework for Python.

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

OWASP D4N155 - Intelligent and dynamic wordlist using OSINT

A framework for creating semi-automatic web content extractors

[Unmaintained] A simple and clean video/music/image downloader 👾

Distributed crawler powered by Headless Chrome

Dynamic meta tags in your AngularJS single page application

Crawly, a high-level web crawling & scraping framework for Elixir.

A scalable web crawler framework for Java.

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Easily create XML sitemaps for your website.

Simple but useful Python web scraping tutorial code.

Elegant Scraper and Crawler Framework for Golang

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

一个灵活、友好的爬虫框架

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Free Web Scraping Tool with Java

Declarative web scraping

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

java framework for prerender

No description or website provided.

Script that generates a sitemap by crawling a given URL

Crawlera middleware for Scrapy

🤖 Prerendering for JavaScript powered websites. Great solution for PWAs (Progressive Web Apps), SPAs (Single Page Applications), and other websites based on top of front-end JavaScript frameworks

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

SEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type from searchresults for given keywords. Detect Ads or make automated screenshots. You can also fetch text content of urls provided in searchresults or by your own. It's usefull for SEO and business related research tasks.

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

The REST API for Jikan

A Tumblr Backup Application

Image Dataset Tool (idt) is a cli tool designed to make the otherwise repetitive and slow task of creating image datasets into a fast and intuitive process.

Statamic Peak is an opinionated starter kit for all your Statamic sites.

以Node.js基于express以及爬虫实现的视频资源后端

golang light-weight image crawler

SEO utilities including a unique field type, sitemap & redirect manager

simple multi-level scraper json input/output for Cheerio

Python Lambda Chrome Automation (naming pending)

🍿爬虫代理IP池(proxy pool) python🍟一个还ok的IP代理池

Official Algolia Plugin for Netlify. Index your website to Algolia when deploying your project to Netlify with the Algolia Crawler

稳健高效的评分制-针对性- IP代理池 + API服务，可以自己插入采集器进行代理IP的爬取，针对你的爬虫的一个或多个目标网站分别生成有效的IP代理数据库，支持MongoDB 4.0 使用 Python3.7（Scored IP proxy pool ,customise proxy data crawler can be added anytime）

Headless Chrome For Java （Java 爬虫）

Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok

An API to scrape American court websites for metadata.

Seo Manager Package for Laravel ( with Localization )

📋 A simple website launch checklist to keep track of the most important enrichment possibilities for a website.

pip安装的天眼查爬虫API，指定的单个/多个企业工商信息一键保存为Excel/JSON格式。A Battery-included Scraper API of Tianyancha, the best Chinese business data and investigation platform.

Get (almost) original messages from google group archives. Your data is yours.

Anime-dl is a command-line program to download anime from CrunchyRoll and Funimation.

Github 仓库及用户分析爬虫

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

crawler framework, distributed crawler extractor

Lightweight package to query popular search engines and scrape for result titles, links and descriptions

京东口罩库存监控爬虫(非selenium)，扫码登录、查价、加购、下单、秒杀

Getting started with Puppeteer and Chrome Headless for Web Scraping

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

SEO Tools for Laravel