GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → Misterhex → WebCrawler

Misterhex / WebCrawler

Licence: other
Just a simple web crawler which return crawled links as IObservable using reactive extension and async await.

Programming Languages

C#
18002 projects

Labels

crawlerdotnetweb-crawlerreactive-extension

Projects that are alternatives of or similar to WebCrawler

Awesome Web Scraper
A collection of awesome web scaper, crawler.
Stars: ✭ 147 (+167.27%)
Mutual labels:  web-crawler
Market-Trend-Prediction
This is a project of build knowledge graph course. The project leverages historical stock price, and integrates social media listening from customers to predict market Trend On Dow Jones Industrial Average (DJIA).
Stars: ✭ 57 (+3.64%)
Mutual labels:  web-crawler
json-web-crawler
Use JSON to list all elements (with css 3 and jquery selector) that you want to crawl.
Stars: ✭ 17 (-69.09%)
Mutual labels:  web-crawler
Crawler Commons
A set of reusable Java components that implement functionality common to any web crawler
Stars: ✭ 173 (+214.55%)
Mutual labels:  web-crawler
Kochat
Opensource Korean chatbot framework
Stars: ✭ 204 (+270.91%)
Mutual labels:  web-crawler
ant
A web crawler for Go
Stars: ✭ 264 (+380%)
Mutual labels:  web-crawler
Proxy
A simple tool for fetching usable proxies from several websites.
Stars: ✭ 124 (+125.45%)
Mutual labels:  web-crawler
RxMVC-Swift
Unidirectional MVC with ReactiveX
Stars: ✭ 87 (+58.18%)
Mutual labels:  reactive-extension
Strong Web Crawler
基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。
Stars: ✭ 238 (+332.73%)
Mutual labels:  web-crawler
StackOverflow-Crawler
It is a web crawler which crawls the stackoverfolw website (http://stackoverflow.com/) and finds the most popular technologies at current point of time by getting the tags info of the newest questions asked on the website.
Stars: ✭ 25 (-54.55%)
Mutual labels:  web-crawler
Zhihu Crawler People
A simple distributed crawler for zhihu && data analysis
Stars: ✭ 182 (+230.91%)
Mutual labels:  web-crawler
Antch
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
Stars: ✭ 198 (+260%)
Mutual labels:  web-crawler
Raspagem-de-dados-para-iniciantes
Raspagem de dados para iniciante usando Scrapy e outras libs básicas
Stars: ✭ 113 (+105.45%)
Mutual labels:  web-crawler
Abot
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Stars: ✭ 1,961 (+3465.45%)
Mutual labels:  web-crawler
leek
Distributed task redisqueue(最简单python分布式函数调度框架)
Stars: ✭ 60 (+9.09%)
Mutual labels:  web-crawler
Collector Http
Norconex HTTP Collector is a flexible web crawler for collecting, parsing, and manipulating data from the Internet (or Intranet) to various data repositories such as search engines.
Stars: ✭ 130 (+136.36%)
Mutual labels:  web-crawler
doc crawler.py
Explore a website recursively and download all the wanted documents (PDF, ODT…)
Stars: ✭ 22 (-60%)
Mutual labels:  web-crawler
learncpp-download
Scrape bot, to get you an offline copy of tutorials
Stars: ✭ 23 (-58.18%)
Mutual labels:  web-crawler
bolsa
Biblioteca feita em Python com o objetivo de facilitar o acesso a dados de seus investimentos na bolsa de valores(B3/CEI) através do Portal CEI.
Stars: ✭ 46 (-16.36%)
Mutual labels:  web-crawler
WeReadScan
扫描“微信读书”已购图书并下载本地PDF的爬虫
Stars: ✭ 273 (+396.36%)
Mutual labels:  web-crawler
View All Similar Projects ➔

WebCrawler

Build Status NuGet version

Just a simple web crawler which return crawled links as IObservable using reactive extension, async await and polly.

dotnet add package MisterHex.WebCrawling --version 2.0.3

Usage

Crawler crawler = new Crawler();
IObservable<Uri> observable = crawler.Crawl(new Uri("https://dotnet.microsoft.com"));
observable.Subscribe(onNext: Console.WriteLine, onCompleted: () => Console.WriteLine("Crawling completed"));
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].