275 Open source projects that are alternatives of or similar to Memorious

An open source web crawling platform

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

Tutorial for web scraping / crawling with Node.js.

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

A test suite of common scraper detection techniques. See how detectable your scraper stack is.

Elegant Scraper and Crawler Framework for Golang

Crawly, a high-level web crawling & scraping framework for Elixir.

Web crawling and scraping framework for Golang

No description or website provided.

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

Grawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them in a file.

Scrapy, a fast high-level web crawling & scraping framework for Python.

Scrapy Extension for monitoring spiders execution.

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

Crawl sites for RSS, Atom, and JSON feeds.

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

一个灵活、友好的爬虫框架

A Scrapy extension to log items coverage when the spider shuts down

A curated list of awesome puppeteer resources.

ProxyCrawl Python library for scraping and crawling

Screen scraping and web crawling framework

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

Extract structured data from web sites. Web sites scraping.

Declarative web scraping

Easily crawl news portals or blog sites using Storm Crawler.

Simple but useful Python web scraping tutorial code.

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

👨‍👩‍👦 Social account detection and extraction in Python, e.g. for crawling/scraping.

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

Distributed crawler powered by Headless Chrome

[Unmaintained] A simple and clean video/music/image downloader 👾

Go process used to crawl websites

Image Dataset Tool (idt) is a cli tool designed to make the otherwise repetitive and slow task of creating image datasets into a fast and intuitive process.

Scraping statistics, predicting NBA player performance with neural networks and boosting algorithms, and optimising lineups for Draft Kings with genetic algorithm. Capstone Project for Machine Learning Engineer Nanodegree by Udacity.

Lightweight package to query popular search engines and scrape for result titles, links and descriptions

Simple Query Scraping with CSS and Go Reflection (MOVED to Gitlab)

Get info from any web service or page

Download a large list of files concurrently

simple multi-level scraper json input/output for Cheerio

An Instagram bot developed using the Selenium Framework

📖 This tool is to download course from educative.io for offline usage. It uses your login credentials and download the course.

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

Universal scrapping tool, which allows you to extract data using multiple environments

An API to scrape American court websites for metadata.

🕷 Google client for SERPS

Your will to enroll in Udemy course is here, but the money isn't? Search no more! This python program searches for your desired course in more than [insert big number here] websites, compares the last updated date, and gives you the download link of the latest one back, but you also have the choice to see the other ones as well!

Apache Nutch is an extensible and scalable web crawler

News, full-text, and article metadata extraction in Python 3. Advanced docs:

🔥🐻 The Speakeasy Scripting Engine Which Combines Speed, Safety, and Simplicity

Anime-dl is a command-line program to download anime from CrunchyRoll and Funimation.

6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디자인, 매크로, 크롤링까지 업무 자동화와 관련된 다양한 분야 예제가 제공됩니다.

scan for webcams on the internet

네이버 뉴스 수집을 위한 도구

Crawler for linguistic corpora

Jsoup Annotations POJO

cdp4j - Chrome DevTools Protocol for Java

Getting started with Puppeteer and Chrome Headless for Web Scraping

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head

htmlSQL is a experimental PHP library which allows you to access HTML values by an SQL like syntax.