Top web-crawling open source projects

Apify Js

Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

✭ 3,154

javascript automation npm javascript-library puppeteer scraping headless-chrome web-scraping crawling web-crawling rpa apify

Data-mining-python-script

It contain various script on web crawling/ data mining of social web(RSS,facebook,twitter,Linkedin)

✭ 24

python rss data-mining facebook twitter linkedin web-crawling

tweetsOLAPing

implementing an end-to-end tweets ETL/Analysis pipeline.

✭ 24

python tweets analysis twitter-api multithreading api-client datawarehousing datawarehouse web-crawling ssis google-api-client etl-pipeline tweets-classification cube-analysis powerbi-report ssas-multidimensional multi-dimensional-analysis tweets-scraper

jseval

Evaluate JavaScript on a URL through headless Chrome browser.

flink-crawler

Continuous scalable web crawler built on top of Flink and crawler-commons

✭ 48

java crawler spider web-crawler crawling flink web-crawling

JAW

JAW: A Graph-based Security Analysis Framework for JavaScript and Client-side CSRF

✭ 26

javascript python neo4j static-analysis csrf client-side property-graph vulnerability-detection web-crawling

OLX Scraper

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

✭ 15

python data-science data machine-learning scraper mongodb nosql web-crawler pymongo web-scraper artificial-intelligence web-scraping scrapping scrapy scraping-websites web-crawling olx web-crawler-python nosql-mongodb

Seen

A lightweight crawling/spider framework for everyone(support JavaScript!).✨

✭ 13

python easy-to-use lightweight-framework web-crawling spider-framework javasciprt support-javascript

Udacity-Data-Analyst-Nanodegree

Repository for the projects needed to complete the Data Analyst Nanodegree.

zcrawl

An open source web crawling platform

✭ 21

go shell scraping crawling crawlers web-crawling webcrawling

Katastrophe

Command Line Tool to download torrents

✭ 85

python screenshot torrent bittorrent command-line kickass-torrents deluge web-crawling

ioweb

Web Scraping Framework

✭ 31

python Makefile crawler framework scraping web-scraping webscraping web-crawling webcrawling webmining web-scraping-python

Amazon-Flipkart-Price-Comparison-Engine

Compares price of the product entered by the user from e-commerce sites Amazon and Flipkart 💰 📊

✭ 41

python amazon tkinter web-crawling flipkart web-crawler-python ecommerce-sites-amazon corresponding-prices

Top 13 web-crawling open source projects