1381 Open source projects that are alternatives of or similar to Dotnetcrawler

A scalable web crawler framework for Java.

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用nomad管理docker集群； 使用EFK查询docker日志

爬虫, http代理, 模拟登陆!

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

This repository contains all the code I use in my YouTube tutorials.

OWASP D4N155 - Intelligent and dynamic wordlist using OSINT

네이버 뉴스 수집을 위한 도구

ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个

Universal scrapping tool, which allows you to extract data using multiple environments

Scrapy Extension for monitoring spiders execution.

Some scrapy and web.py exmaples

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

👨‍👩‍👦 Social account detection and extraction in Python, e.g. for crawling/scraping.

A simple, cross platform, modularized ecommerce system built on .NET Core

swiss army knife for hackers

Tutorial for web scraping / crawling with Node.js.

An open source web crawling platform

Easily crawl news portals or blog sites using Storm Crawler.

Simple and friendly Bot for Instagram, using Selenium and Scrapy with Python.

Be nice on the web

CHeSF is the Chrome Headless Scraping Framework, a very very alpha code to scrape javascript intensive web pages

Web crawling and scraping framework for Golang

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

一个基于 .NET Core构建的简单、跨平台、模块化的商城系统

Web scraper project to collect the useful Hong Kong weather data from HKO website

Crawl sites for RSS, Atom, and JSON feeds.

Screen scraping and web crawling framework

Code for the second edition Web Scraping with Python book by Packt Publications

Scrapy-based Crawlers for news of Taiwan

Continuous scalable web crawler built on top of Flink and crawler-commons

API for scrapping common meme sites

An interactive Command-Line Interface Build in NodeJS for downloading a single or multiple images to disk from URL

Schedules tweets using TweetDeck

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

a reliable high-level web crawling & scraping framework for Node.js.

Extract structured data from web sites. Web sites scraping.

The main functionality is to extract all the emails from one or several URLs - La funcionalidad principal es extraer todos los correos electrónicos de una o varias Url

Linkedin Scraper using Selenium Web Driver, Chromium headless, Docker and Scrapy

A Facebook crawler

Scrapy middleware to handle javascript pages using selenium

基于搜狗微信搜索的微信公众号爬虫接口

Build and deploy a machine learning app from scratch 🚀

A multi-thread crawler framework with many builtin image crawlers provided.

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

Life is short, *****.

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

Redis-based components for Scrapy.

🥫 The simple, fast, and modern web scraping library

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Free Web Scraping Tool with Java

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

A python web scraper for public email lists.

Scrapy as a Web Crawler for Azure Search Samples

A curated list of awesome puppeteer resources.

Distributed crawling framework for documents and structured data.

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

A Scrapy Spider for downloading PDF files from a webpage.

Public, free to use, repository with diggers configs for scraping / extracting data from various e-commerce websites and online stores

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架