All Projects → deptagency → octopus

deptagency / octopus

Licence: MIT License
Recursive and multi-threaded broken link checker

Programming Languages

javascript
184084 projects - #8 most used programming language

Projects that are alternatives of or similar to octopus

action-my-broken-link-checker
A GitHub Action for checking broken links
Stars: ✭ 32 (+68.42%)
Mutual labels:  links, checker
eastmoney
python requests + Django+ nodejs koa+ mysql to crawl eastmoney fund and stock data,for data analysis and visualiaztion .
Stars: ✭ 56 (+194.74%)
Mutual labels:  crawler
qr
🔲 Generate QR Codes straight in your terminal!
Stars: ✭ 34 (+78.95%)
Mutual labels:  links
html-query
A fluent and functional approach to querying HTML
Stars: ✭ 48 (+152.63%)
Mutual labels:  crawler
dijnet-bot
Az összes számlád még egy helyen :)
Stars: ✭ 17 (-10.53%)
Mutual labels:  crawler
Log4j-RCE-Scanner
Remote command execution vulnerability scanner for Log4j.
Stars: ✭ 200 (+952.63%)
Mutual labels:  checker
WebCrawler
一个轻量级、快速、多线程、多管道、灵活配置的网络爬虫。
Stars: ✭ 39 (+105.26%)
Mutual labels:  crawler
jsonlint
Lightweight command-line tool for validating JSON
Stars: ✭ 27 (+42.11%)
Mutual labels:  checker
tg crawler
Just a crawler based on tg-cli for Telegram. Deprecated by now, please use telegram-export.
Stars: ✭ 71 (+273.68%)
Mutual labels:  crawler
ZhengFang System Spider
🐛一只登录正方教务管理系统,爬取数据的小爬虫
Stars: ✭ 21 (+10.53%)
Mutual labels:  crawler
snapcrawl
Crawl a website and take screenshots
Stars: ✭ 37 (+94.74%)
Mutual labels:  crawler
TumblTwo
TumblTwo, an Improved Fork of TumblOne, a Tumblr Downloader.
Stars: ✭ 57 (+200%)
Mutual labels:  crawler
codes-scratch-crawler
读书笔记《自己动手写网络爬虫》,自己敲的代码。主要记录了网络爬虫的基本实现,网页去重的算法,网页指纹算法,文本信息挖掘
Stars: ✭ 44 (+131.58%)
Mutual labels:  crawler
proxy-checker
The simple script, which checks good and bad proxies
Stars: ✭ 42 (+121.05%)
Mutual labels:  checker
weibo-scraper
Simple Weibo Scraper
Stars: ✭ 50 (+163.16%)
Mutual labels:  crawler
bulldog
The http checker
Stars: ✭ 13 (-31.58%)
Mutual labels:  checker
bots-zoo
No description or website provided.
Stars: ✭ 59 (+210.53%)
Mutual labels:  crawler
indieweb-search
Source code for the IndieWeb search engine.
Stars: ✭ 16 (-15.79%)
Mutual labels:  crawler
PY-Login
模拟登录各类网站,操作 API 完成各种不可描述的事情
Stars: ✭ 26 (+36.84%)
Mutual labels:  crawler
MyCrawler
我的爬虫合集
Stars: ✭ 55 (+189.47%)
Mutual labels:  crawler

Octopus

Octopus
Quickly crawl a whole website for broken links

Install

npm install --global @deptagency/octopus

or

yarn global add @deptagency/octopus

Usage

octopus <domain> [options]

Options

Option Description Default
--ignore-query Ignore a query string false
--ignore-external Ignore all external links false
--ignore-nofollow Ignore rel=nofollow links false
--include-images Check <img> elements false
--slack-webhook Slack incoming webhook url none
--timeout Time to wait for response 5000
--silent Run without printing progress line false
--help Output help text

Examples

octopus www.deptagency.com
octopus www.awg-mode.de --ignore-external
octopus www.hardeck.de --ignore-query=isEnergyEfficiencyChartOpen --ignore-query=followSearch
octopus www.golfino.com --silent --slack-webhook=https://hooks.slack.com/services/XXX/XXX/XXX

Icon

Made by Freepik from www.flaticon.com

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].