527 Open source projects that are alternatives of or similar to Python3Webcrawler

News, full-text, and article metadata extraction in Python 3. Advanced docs:

A tool for pixiv.net. 人人可用的P站爬虫

No description or website provided.

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

MM131网站图片爬取 🚨

基金投资管理回测引擎

JavaScript + BeautifulSoup = JSSoup

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

微博超级话题爬虫，微博词频统计+情感分析+简单分类，新增肺炎超话爬取数据

京东商品价格监控：监控用户设定商品价格，降价邮件/微信提醒。技术：Python爬虫/IP代理池/JS接口爬取/Selenium页面爬取

Golang 实现的 IP 代理池, 涉及到的技术点: go gorm proxy proxypool ip crawler 爬虫 mysql viper cobra

Multithreading download all HD photos / pictures from someone's Sina Weibo album.

根据关键词抓取微博数据，再生成词云

自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向

Free Web Scraping Tool with Java

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

API of DouYin for Humans used to Crawl Popular Videos and Musics

字体混淆服务

A search application to explore, discover and share online files

🐤 小程序API兼容插件，一次编写，多端运行。支持：微信小程序、支付宝小程序、百度智能小程序、字节跳动小程序

A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

Distributed crawler powered by Headless Chrome

以Node.js基于express以及爬虫实现的视频资源后端

Redis-based components for Scrapy.

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料

A MXNet implementation of Baidu's DeepSpeech architecture

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

（已跑路）百度贴吧吧务管理工具，自动扫描帖子并处理违规帖

a new crawler based on python with more function including Network fingerprint search

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

🐨实时监控1000家中国企业的新闻动态

获取免费socks/https/http代理的网站集合

对免费代理IP网站进行爬取，收集汇总为自己的代理池。关键是验证代理的有效性、匿名性、去重复

哔咔漫画收藏夹下载程序

LinkedIn Scraper (currently working 2020)

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用nomad管理docker集群； 使用EFK查询docker日志

Crawl BookCorpus

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Leetcode Contest Ranking Searcher

Html网页正文提取

爬取菜鸟教程网站并转PDF__python_crawer_by_chrome

百度 莱茨狗 php 抓取,提交,增加99%识别率API

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

Open-source Enterprise Grade Search Engine Software

Get (almost) original messages from google group archives. Your data is yours.

OSINT Swiss Army Knife

Viewers for statistics and dashboarding of Domain Search Engine data

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

a terminal-translate tool

This package is a complete tool for creating a large dataset of images (specially designed -but not only- for machine learning enthusiasts). It can crawl the web, download images, rename / resize / covert the images and merge folders..

Dynamic meta tags in your AngularJS single page application

Get the lyrics for the song currently playing on Spotify

🎬 电影资源爬虫,电影图片抓取脚本,Flask|Nginx|wsgi

链家网爬虫

A free and unlimited API for Baidu Translate

百度统计API- python版

百度网盘搜索工具（.NET4.0 & Visual Studio 2017）

一个基于百度地图api的demo，采用Material Design设计界面，有全景图、室内地图等功能。