medcl / Gopa Abandoned
Licence: other
GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）
Stars: ✭ 98
Labels
Projects that are alternatives of or similar to Gopa Abandoned
Gopa
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Stars: ✭ 277 (+182.65%)
Mutual labels: crawler, spider, lightweight
Crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
Stars: ✭ 8,392 (+8463.27%)
Mutual labels: crawler, spider
Awesome Python Primer
自学入门 Python 优质中文资源索引，包含 书籍 / 文档 / 视频，适用于 爬虫 / Web / 数据分析 / 机器学习 方向
Stars: ✭ 57 (-41.84%)
Mutual labels: crawler, spider
Nodespider
[DEPRECATED] Simple, flexible, delightful web crawler/spider package
Stars: ✭ 33 (-66.33%)
Mutual labels: crawler, spider
Photon
Incredibly fast crawler designed for OSINT.
Stars: ✭ 8,332 (+8402.04%)
Mutual labels: crawler, spider
Crawler
A high performance web crawler in Elixir.
Stars: ✭ 781 (+696.94%)
Mutual labels: crawler, spider
Beanbun
Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。
Stars: ✭ 1,096 (+1018.37%)
Mutual labels: crawler, spider
Arachnid
Powerful web scraping framework for Crystal
Stars: ✭ 68 (-30.61%)
Mutual labels: crawler, spider
Zhihu Crawler
zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目
Stars: ✭ 890 (+808.16%)
Mutual labels: crawler, spider
Gospider
Gospider - Fast web spider written in Go
Stars: ✭ 785 (+701.02%)
Mutual labels: crawler, spider
Avbook
AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
Stars: ✭ 8,133 (+8198.98%)
Mutual labels: crawler, spider
Grab Site
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Stars: ✭ 680 (+593.88%)
Mutual labels: crawler, spider
Creeper
🐾 Creeper - The Next Generation Crawler Framework (Go)
Stars: ✭ 762 (+677.55%)
Mutual labels: crawler, spider
Gopa
[狗爬], A Spider Written in Go.
Build Gopa
Mac/Linux: Run make build to build the Gopa
Windows: go with this wiki - How to build GOPA on windows
Download
Start Gopa
After download/build the binary file, run ./gopa to start the Gopa
Run ./gopa -h to get the full list of commandline options
- -log option : logging level,can be set to
trace,debug,info,warn,error,default isinfo - -daemon option : run in background as daemon
- -pprof option : enable and setup pprof/expvar service, eg: localhost:6060 , the endpoint will be: http://localhost:6060/debug/pprof/ and http://localhost:6060/debug/vars
- -cpuprofile option : write cpu profile to this file
- -memprofile option : write memory profile to this file
Stop Gopa
It's safety to press ctrl+c stop the current running Gopa, Gopa will handle the rest,saving the checkpoint,
you may restore the job later,the world is still in your hand.
If you are running Gopa as daemon, you can stop it like this:
kill -QUIT `pgrep gopa`
UI
Visit http://127.0.0.1:9001/ for more details.
License
Released under the Apache License, Version 2.0 .
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].