All Projects → nicksherron → proxi

nicksherron / proxi

Licence: Apache-2.0 license
Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.

Programming Languages

go
31211 projects - #10 most used programming language
shell
77523 projects
Makefile
30231 projects
Dockerfile
14818 projects

Projects that are alternatives of or similar to proxi

torchestrator
Spin up Tor containers and then proxy HTTP requests via these Tor instances
Stars: ✭ 32 (+0%)
Mutual labels:  scraping, scrapy, proxypool, proxy-list
Proxy
A simple tool for fetching usable proxies from several websites.
Stars: ✭ 124 (+287.5%)
Mutual labels:  web-crawler, proxypool, proxy-list
Proxybroker
Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭
Stars: ✭ 2,767 (+8546.88%)
Mutual labels:  http-proxy, proxypool, proxy-list
OLX Scraper
📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.
Stars: ✭ 15 (-53.12%)
Mutual labels:  web-crawler, scrapy
Crawlab Lite
Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
Stars: ✭ 122 (+281.25%)
Mutual labels:  web-crawler, scrapy
Awesome Web Scraper
A collection of awesome web scaper, crawler.
Stars: ✭ 147 (+359.38%)
Mutual labels:  web-crawler, scrapy
Gopa
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Stars: ✭ 277 (+765.63%)
Mutual labels:  web-crawler, scraping
LiveProxies
Asynchronous proxy checker
Stars: ✭ 17 (-46.87%)
Mutual labels:  http-proxy, proxy-list
double-agent
A test suite of common scraper detection techniques. See how detectable your scraper stack is.
Stars: ✭ 123 (+284.38%)
Mutual labels:  scraping, scrapy
Raspagem-de-dados-para-iniciantes
Raspagem de dados para iniciante usando Scrapy e outras libs básicas
Stars: ✭ 113 (+253.13%)
Mutual labels:  web-crawler, scrapy
InstaBot
Simple and friendly Bot for Instagram, using Selenium and Scrapy with Python.
Stars: ✭ 32 (+0%)
Mutual labels:  scraping, scrapy
Terpene Profile Parser For Cannabis Strains
Parser and database to index the terpene profile of different strains of Cannabis from online databases
Stars: ✭ 63 (+96.88%)
Mutual labels:  web-crawler, scrapy
HttpProxy
JAVA实现的IP代理池,支持HTTP与HTTPS两种方式
Stars: ✭ 37 (+15.63%)
Mutual labels:  http-proxy, proxypool
scrapy-distributed
A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.
Stars: ✭ 38 (+18.75%)
Mutual labels:  scraping, scrapy
Antch
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
Stars: ✭ 198 (+518.75%)
Mutual labels:  web-crawler, scraping
Crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
Stars: ✭ 8,392 (+26125%)
Mutual labels:  web-crawler, scrapy
RARBG-scraper
With Selenium headless browsing and CAPTCHA solving
Stars: ✭ 38 (+18.75%)
Mutual labels:  scraping, scrapy
Dotnetcrawler
DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c
Stars: ✭ 100 (+212.5%)
Mutual labels:  scraping, scrapy
Seleniumcrawler
An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site
Stars: ✭ 117 (+265.63%)
Mutual labels:  scraping, scrapy
scrapy-fieldstats
A Scrapy extension to log items coverage when the spider shuts down
Stars: ✭ 17 (-46.87%)
Mutual labels:  scraping, scrapy

proxi

Go Report Card

Proxi runs scheduled jobs for finding and checking proxies with a rest api for querying results. It can find and check over 25k proxies in under 5 minutes.

Download options

Homebrew

Homebrew or Linuxbrew

brew install nicksherron/proxi/proxi

Docker

docker run -d -p 4444:4444 nicksherron/proxi server --init

Go

Go version 1.11 or higher is required to go get proxi since it uses go modules.

GO111MODULE=on go get -u github.com/nicksherron/proxi

Releases

Binaries for Windows, Linux and Darwin can be found under releases. If your os/architecture is not supported, submit an issue and more than likely it can be included.

Usage

More documentation is to come but the commands are pretty straight forward and all have help info with default options shown.

proxi server --init

Will start the server and begin downloading/checking proxies. If you use the default listen address, you can go to http://localhost:4444/swagger/index.html in your browser to view the swagger api docs.

To get a proxy from the server just run

proxi get 

or

curl localhost:4444/get

sreenshot

$ proxi -h

Usage:
  proxi [command]

Available Commands:
  delete      Delete a proxy from the db.
  find        Find the record for a proxy
  get         Return one or more proxies from db that passed checks.
  help        Help about any command
  refresh     Re-download and check proxies.
  server      Download then check proxies and start rest api server for querying results.
  stats       Check server stats
  version     Print the version number and build info

Flags:
  -h, --help   help for proxi

Use "proxi [command] --help" for more information about a command.

Contributing

Pull request welcome !

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].