563 Open source projects that are alternatives of or similar to PythonScrapyBasicSetup

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

PHP Scraper - an highly opinionated web-interface for PHP

A Node.js library to easily manage and rotate a pool of web browsers, using any of the popular browser automation libraries like Puppeteer, Playwright, or SecretAgent.

Web Scraping Framework

Spin up Tor containers and then proxy HTTP requests via these Tor instances

Scrapy project for scraping data from IMDB with Movie Dataset including 58,623 movies' data.

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

Papercut is a scraping/crawling library for Node.js built on top of JSDOM. It provides basic selector features together with features like Page Caching and Geosearch.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

A library to read a YML file with Xpath or CSS Selectors and extract data from HTML pages using them

📓 Minicurso de raspagem de dados web com Python ministrado na Semana de Tecnologia da FATEC Jundiaí

A Python Tool For google Hacking

Simple Query Scraping with CSS and Go Reflection (MOVED to Gitlab)

Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

PHP Library for detecting CMS

A Scraper made 100% in Python using BeautifulSoup and Tor. It can be used to scrape both normal and onion links. Happy Scraping :)

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

Scape top GitHub repositories and users based on keywords

A framework for creating semi-automatic web content extractors

Node.js package to bypass CloudFlare's anti-bot JavaScript challenges

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

A web application that generates timetables for university students at the University of Toronto

Image Dataset Tool (idt) is a cli tool designed to make the otherwise repetitive and slow task of creating image datasets into a fast and intuitive process.

simple multi-level scraper json input/output for Cheerio

Building a Concurrent Web Scraper with Python and Selenium

Use SQL on various data sources

Anime-dl is a command-line program to download anime from CrunchyRoll and Funimation.

A browser testing and web crawling library for PHP and Symfony

⏪ Tools to Work with the Various Internet Archive Wayback Machine APIs

The REST API for Jikan

Generate a network graph of connections from your WhatsApp groups data

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

List of major web + mobile browser user agent strings. +1 Bonus script to scrape :)

An API to scrape American court websites for metadata.

Pahe.ph [Pahe.in] Movies Website Scraper

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Distributed crawling framework for documents and structured data.

Linkedin Learning videos downloader

Pythonic HTML Parsing for Humans™

This class can retrieve search results from Google.

📄 Python tool to turn Notion.so pages into lightweight, customizable static websites

The web browser that's built for scraping.

Extract data or evaluate value from HTML/XML documents using XPath

Jsoup Annotations POJO

SEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type from searchresults for given keywords. Detect Ads or make automated screenshots. You can also fetch text content of urls provided in searchresults or by your own. It's usefull for SEO and business related research tasks.

OnionFruit™ Connect - Tor access client with country selection, bridge configuration, pluggable transports and experimental DNS support

.NET agile and extensible web searching API

Private and self-hosted file sharing over the Tor network written in golang

Social media scraping / data collection tool for the Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs

Pick the most common user-agents on the Internet 👻

Scraping statistics, predicting NBA player performance with neural networks and boosting algorithms, and optimising lineups for Draft Kings with genetic algorithm. Capstone Project for Machine Learning Engineer Nanodegree by Udacity.

Get info from any web service or page

compose scripts for tor-based projects

reborn of https://bitbucket.org/rflechner/scrapysharp

📖 This tool is to download course from educative.io for offline usage. It uses your login credentials and download the course.

🕷 Google client for SERPS

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites

Your will to enroll in Udemy course is here, but the money isn't? Search no more! This python program searches for your desired course in more than [insert big number here] websites, compares the last updated date, and gives you the download link of the latest one back, but you also have the choice to see the other ones as well!

🔥🐻 The Speakeasy Scripting Engine Which Combines Speed, Safety, and Simplicity

Tiny little ruby on rails website that crawls though your public github repos to find out what your favourite languages are.