github / Lightcrawler
Licence: isc
Crawl a website and run it through Google lighthouse
Stars: ✭ 1,339
Programming Languages
javascript
184084 projects - #8 most used programming language
Projects that are alternatives of or similar to Lightcrawler
Pychromeless
Python Lambda Chrome Automation (naming pending)
Stars: ✭ 219 (-83.64%)
Mutual labels: crawler, chrome
Squidwarc
Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head
Stars: ✭ 125 (-90.66%)
Mutual labels: crawler, chrome
Headless Chrome Crawler
Distributed crawler powered by Headless Chrome
Stars: ✭ 5,129 (+283.05%)
Mutual labels: crawler, chrome
Catblock
CatBlock - An ad-blocking extension (with an ability to replace ads by pictures) for all major desktop browsers!
Stars: ✭ 84 (-93.73%)
Mutual labels: chrome
Geziyor
Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.
Stars: ✭ 1,246 (-6.95%)
Mutual labels: crawler
Taiwan News Crawlers
Scrapy-based Crawlers for news of Taiwan
Stars: ✭ 83 (-93.8%)
Mutual labels: crawler
Ktspeechcrawler
Automatically constructing corpus for automatic speech recognition from YouTube videos
Stars: ✭ 92 (-93.13%)
Mutual labels: crawler
The Tab Of Words
A minimal Chrome / Firefox extension to help you learn Japanese words in each new tab.
Stars: ✭ 94 (-92.98%)
Mutual labels: chrome
Scrapoxy
Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!
Stars: ✭ 1,322 (-1.27%)
Mutual labels: crawler
Docker Protractor Headless
Protractor end to end testing for AngularJS - dockerised and headless with real Chrome.
Stars: ✭ 90 (-93.28%)
Mutual labels: chrome
Roam Research Private Api
Private API to enable API access for Roam Research. Now you can connect Roam to your other projects.
Stars: ✭ 88 (-93.43%)
Mutual labels: chrome
lightcrawler
Crawl a website and run it through Google lighthouse
npm install --save-dev lightcrawler
lightcrawler --url https://atom.io/ --config lightcrawler-config.json
where lightcrawler-config.json looks something like this:
{
"extends": "lighthouse:default",
"settings": {
"crawler": {
"maxDepth": 2,
"maxChromeInstances": 5
},
"onlyCategories": [
"Accessibility",
"Performance",
"Best Practices"
],
"onlyAudits": [
"accesskeys",
"aria-allowed-attr",
"external-anchors-use-rel-noopener",
"geolocation-on-start",
"no-document-write",
"no-mutation-events",
"no-old-flexbox",
"time-to-interactive",
"user-timings",
"viewport",
"without-javascript"
]
}
}
Enjoy!
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].