398 Open source projects that are alternatives of or similar to Scan T

Open-source Enterprise Grade Search Engine Software

Recursive and multi-threaded broken link checker

快速、简洁且强大的PHP爬虫框架

我的爬虫合集

Crawl BookCorpus

python requests + Django+ nodejs koa+ mysql to crawl eastmoney fund and stock data,for data analysis and visualiaztion .

🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web预览

🇰🇷 Realtime integrated information analysis service

OSINT Swiss Army Knife

Source code for the IndieWeb search engine.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

🐛一只登录正方教务管理系统，爬取数据的小爬虫

A framework for creating semi-automatic web content extractors

No description or website provided.

台灣上市上櫃股票爬蟲 Taiwan Stock Exchange Crawler

Az összes számlád még egy helyen :)

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

🍰 一个可视化的爬虫平台

Android 本地网络小说爬虫，基于jsoup及xpath

Easy way to brute-force web directory.

Html网页正文提取

Crawling CV conference papers with Python.

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

🌀 crawl bilibili user info and video info for data analysis | BiliBili爬虫

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

Fetching hotel data from TripAdvisor.

The fastest dork scanner written in Go.

PTT 網路版爬蟲

🐨实时监控1000家中国企业的新闻动态

Practical pinned gist which show your latest medium status 📌

Crawl GitHub APIs and store the discovered orgs, repos, commits, ...

We will try to find your visible basic footprint from social media as much as possible - 😤 more sites is comming soon

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

SSE 50 index options crawler 上证50期权数据爬虫

微信公众号爬虫，公众号历史文章，文章评论，文章阅读及在看数据，可视化web页面，可部署于Windows服务器。基于Python3之flask/mysql/redis/mitmproxy/pywin32等实现，高效微信爬虫，微信公众号爬虫，历史文章，文章评论，数据更新。

Auto Crawler Ptt Beauty Image Use Python Schedule

爬取菜鸟教程网站并转PDF__python_crawer_by_chrome

An interactive Command-Line Interface Build in NodeJS for downloading a single or multiple images to disk from URL

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

a reliable high-level web crawling & scraping framework for Node.js.

A Powerful Spider(Web Crawler) System in Python.

Crawlera middleware for Scrapy

Google search results crawler, get google search results that you need

LinkedIn Scraper (currently working 2020)

直接通过链家 API 抓取数据的极速爬虫，宇宙最快~~ 🚀

📰 A responsive interface of Hacker News with summaries and thumbnails.

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（nedb/mongodb），提供数据可视化和用户交互的实现方案

台灣股票即時爬蟲。Taiwan Stock Exchange Real Time Crawler

码云仓库链接:AJay13/ECommerceCrawlers Github 仓库链接:DropsDevopsOrg/ECommerceCrawlers 项目展示平台链接:http://wechat.doonsec.com

Easily create XML sitemaps for your website.

本仓库收集整理爬虫相关资源，开发语言以Java为主

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

Web Application Security Scanner Framework

A collection of awesome web crawler,spider in different languages

Declarative web scraping

Python的基础练习代码与各种爬虫代码

What do people have in their dotfiles?

Crawler (Bot) searching for credential leaks on different paste sites.

Update Version of weibo_terminator, This is Workflow Version aim at Get Job Done!