891 Open source projects that are alternatives of or similar to Pspider

网易云爬虫解决方案

汤不热 python 多线程爬虫

pylinkvalidator is a standalone and pure python link validator and crawler that traverses a web site and reports errors (e.g., 500 and 404 errors) encountered.

Get the lyrics for the song currently playing on Spotify

A set of useful libraries for Edge Apps. Run locally, write tests, and integrate it into your deployment process. Move fast and maybe don't break things? Because, gosh darnit, you're an adult.

Crawl websites for accessibility issues from the command line.

Sina stock options crawler with CSV output 新浪上证ETF期权数据爬虫

Automatically constructing corpus for automatic speech recognition from YouTube videos

A tool that generates an initial coala config file for you!

QQZone mood spider and analysis. QQ空间多线程爬虫和数据挖掘。提供线上服务，扫码登陆即可自动爬取和分析数据，还有网易云年度报告风格的数据展示；使用docker-compose打包程序，方便部署；额外提供QQ空间抽奖小程序。

A php crawler that finds emails on the internets

ES5 shim for ES6 Reflect and Proxy objects

Tổng hợp script crawl dữ liệu từ các mạng xã hội & website tiếng Việt

爬取菜鸟教程网站并转PDF__python_crawer_by_chrome

爬虫代理IP池服务，可供其他爬虫程序通过restapi获取

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

Creating Scrapy scrapers via the Django admin interface

What do people have in their dotfiles?

An automated website accessibility scanner and cli

📺 B 站全站视频信息爬虫

新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频

Simple, clear and fast Web Crawler framework build on python3.6+, powered by asyncio.

Distributed Computing for AI Made Simple

POOPAK - TOR Hidden Service Crawler

Simple CORPORA list crawler

scrapy专利爬虫（停止维护）

Distributed Image Search Engine Crawler

Unmaintained 🐳 ☕️ 🕷 Scala crawler(spider) framework, inspired by scrapy, created by @gaocegege

OWASP D4N155 - Intelligent and dynamic wordlist using OSINT

Python: An all-in-one Web Crawler, Web Parser and Web Scrapping library!

Used the notion of threads and parallelism to make a ATM Simulator.

扒网站工具，看好哪个网站，指定好URL，自动扒下来做成模版。所见网站，皆可为我所用！

vulnx 🕷️ is an intelligent bot auto shell injector that detect vulnerabilities in multiple types of cms { `wordpress , joomla , drupal , prestashop .. `}

Discover hidden deepweb pages

C++ lockless queue.

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python爬虫，返回一定格式的信息，下载，使用flask提供简易api。抖音无水印、皮皮虾、快手、网易云音乐、qq音乐、咪咕音乐、荔枝FM音频、知乎视频、最右语音、视频、微博......

爬取amazon/bestbuy/costco/6pm 的商品详情

简简单单spider

Kindle Book Maker with KindleGen, Make Book from RSS/single URL/directory and so on.

A python3 crawler for crawling Pixiv ranking top and any illustrator all artworks

It covers the blockade principle of most anti-climbing strategies and corresponding solutions.👽👽👽👽（涵盖了大部分的反爬策略的封锁原理以及对应的解决方案。）

获取最新可备案域名列表爬虫

Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.

NetCloud Web Spider

GUI of AlipayOrdersSupervisor, implemented in Java and Swing

Pholcus is a distributed high-concurrency crawler software written in pure golang

Crawler (Bot) searching for credential leaks on different paste sites.

A GUI client of schannel powered by therecipe/qt and golang

The Resources for "Natural Language to Logical Form" ; "自然语言转逻辑形式"研究资料收集。

capture pictures from website like sina, lofter, huaban and so on

一个简单的豆瓣信息爬虫😄

Implements the crawler package into Symfony

[ Closed ] 🎨 image collector, which supports custom acquisition source configuration and is compatible with MacOS and Windows operating systems.

Blazing fast admin panel finder with asyncio and aiohttp

整合第三方网站的视频和弹幕资源, 为白嫖党提供最佳看番追剧体验

开源矢量地理数据获取与预处理工具(POI/AOI/行政区/路网/土地利用)

Advanced Developer Async Programming Toolkit

A python3 package for writing spider more easily.

Project thu thập điểm chuẩn đại học 2014 - 2018 và phân tích dữ liệu