GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → speed → Newcrawler

speed / Newcrawler

Free Web Scraping Tool with Java

Programming Languages

javascript
184084 projects - #8 most used programming language

Labels

dockercrawlerspiderscraping

Projects that are alternatives of or similar to Newcrawler

Linkedin Profile Scraper
🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.
Stars: ✭ 171 (-70.97%)
Mutual labels:  crawler, spider, scraping
Crawly
Crawly, a high-level web crawling & scraping framework for Elixir.
Stars: ✭ 440 (-25.3%)
Mutual labels:  crawler, spider, scraping
Awesome Python Primer
自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向
Stars: ✭ 57 (-90.32%)
Mutual labels:  crawler, spider, scraping
Colly
Elegant Scraper and Crawler Framework for Golang
Stars: ✭ 15,535 (+2537.52%)
Mutual labels:  crawler, spider, scraping
Geziyor
Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.
Stars: ✭ 1,246 (+111.54%)
Mutual labels:  crawler, spider, scraping
Gopa
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Stars: ✭ 277 (-52.97%)
Mutual labels:  crawler, spider, scraping
Go jobs
带你了解一下Golang的市场行情
Stars: ✭ 526 (-10.7%)
Mutual labels:  crawler, spider
Gosint
OSINT Swiss Army Knife
Stars: ✭ 401 (-31.92%)
Mutual labels:  crawler, spider
Haipproxy
💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
Stars: ✭ 4,993 (+747.71%)
Mutual labels:  crawler, spider
Learnpython
Python的基础练习代码与各种爬虫代码
Stars: ✭ 451 (-23.43%)
Mutual labels:  crawler, spider
Scrapple
A framework for creating semi-automatic web content extractors
Stars: ✭ 464 (-21.22%)
Mutual labels:  crawler, scraping
Douyin
API of DouYin for Humans used to Crawl Popular Videos and Musics
Stars: ✭ 580 (-1.53%)
Mutual labels:  crawler, spider
Xxl Crawler
A distributed web crawler framework.（分布式爬虫框架XXL-CRAWLER）
Stars: ✭ 561 (-4.75%)
Mutual labels:  crawler, spider
Bilili
🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器
Stars: ✭ 379 (-35.65%)
Mutual labels:  crawler, spider
Html2article
Html网页正文提取
Stars: ✭ 441 (-25.13%)
Mutual labels:  crawler, spider
Signature algorithm
各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)
Stars: ✭ 380 (-35.48%)
Mutual labels:  crawler, spider
Spider Flow
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
Stars: ✭ 365 (-38.03%)
Mutual labels:  crawler, spider
Headless Chrome Crawler
Distributed crawler powered by Headless Chrome
Stars: ✭ 5,129 (+770.8%)
Mutual labels:  crawler, scraping
Awesome Crawler
A collection of awesome web crawler,spider in different languages
Stars: ✭ 4,793 (+713.75%)
Mutual labels:  crawler, spider
Fbcrawl
A Facebook crawler
Stars: ✭ 536 (-9%)
Mutual labels:  crawler, spider
View All Similar Projects ➔

NewCrawler

Free Web Scraping Tool

NewCrawler Quick Start

www.newcrawler.com

Linux

Installing software packages on Centos / Fedora servers:

x86

curl -fsSL https://raw.githubusercontent.com/speed/newcrawler/master/install_i586.sh | sh

x64

curl -fsSL https://raw.githubusercontent.com/speed/newcrawler/master/install_x86_64.sh | sh

Installing software packages on Ubuntu / Debian servers:

x86

curl -fsSL https://raw.githubusercontent.com/speed/newcrawler/master/install_Debian_i586.sh | sh

x64

curl -fsSL https://raw.githubusercontent.com/speed/newcrawler/master/install_Debian_x86_64.sh | sh

Installing NewCrawler and Chrome software packages on Centos / Fedora servers:

x86

curl -fsSL https://raw.githubusercontent.com/speed/newcrawler/master/install_NewCrawler_Chrome_MySQL_x86_64.sh | sh

	# OS Version 、 NewCrawler Directory
	
	[[email protected] ~]# rpm -q centos-release
	centos-release-7-0.1406.el7.centos.2.5.x86_64

	[[email protected] ~]# ls
	install.sh  newcrawler

	[[email protected] ~]# ls newcrawler
	db  jetty  jre  phantomjs  start.sh  stop.sh  war

Modify the database to MySQL or use the default file database

#edit 'war/WEB-INF/classes/datanucleus.properties'

javax.jdo.option.ConnectionURL=jdbc:mysql://127.0.0.1:3306/newcrawler?characterEncoding=UTF-8
javax.jdo.option.ConnectionUserName=root
javax.jdo.option.ConnectionPassword=123456

Windows

x86

https://github.com/speed/windows-32bit-jetty-jre

x64

https://github.com/speed/windows-64bit-jetty-jre

Google App Engine

https://github.com/speed/newcrawler-gae-shell

Docker

docker pull newcrawler/spider

docker run -itd -p --net=host 8500:8500 --name=newcrawler newcrawler/spider

docker logs -f newcrawler

Docker aliyun

docker run -itd -p --net=host 8500:8500 --name=newcrawler registry.cn-shenzhen.aliyuncs.com/speed/spider

Startup NewCrawler

sh newcrawler/start.sh &

http://127.0.0.1:8500

Shutdown NewCrawler

sh newcrawler/stop.sh

Upgrade NewCrawler

sh newcrawler/upgrade.sh

Install Chrome

https://github.com/speed/selenium

ScreenShot

NewCrawler Cluster

ScreenShot

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].