felipecsl / Wombat
Licence: mit
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
Stars: ✭ 1,220
Projects that are alternatives of or similar to Wombat
Crawler
A high performance web crawler in Elixir.
Stars: ✭ 781 (-35.98%)
Mutual labels: crawler, scraper
Headless Chrome Crawler
Distributed crawler powered by Headless Chrome
Stars: ✭ 5,129 (+320.41%)
Mutual labels: crawler, scraper
Nintendo Switch Eshop
Crawler for Nintendo Switch eShop
Stars: ✭ 463 (-62.05%)
Mutual labels: crawler, scraper
Avbook
AV 电影管理系统, avmoo , javbus , javlibrary 爬虫,线上 AV 影片图书馆,AV 磁力链接数据库,Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
Stars: ✭ 8,133 (+566.64%)
Mutual labels: crawler, scraper
Spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Stars: ✭ 656 (-46.23%)
Mutual labels: crawler, scraper
Pypergrabber
Fetches PubMed article IDs (PMIDs) from email inbox, then crawls PubMed, Google Scholar and Sci-Hub for respective PDF files.
Stars: ✭ 14 (-98.85%)
Mutual labels: crawler, scraper
Scrapedin
LinkedIn Scraper (currently working 2020)
Stars: ✭ 453 (-62.87%)
Mutual labels: crawler, scraper
Goscraper
Golang pkg to quickly return a preview of a webpage (title/description/images)
Stars: ✭ 72 (-94.1%)
Mutual labels: crawler, scraper
Awesome Crawler
A collection of awesome web crawler,spider in different languages
Stars: ✭ 4,793 (+292.87%)
Mutual labels: crawler, scraper
Crawly
Crawly, a high-level web crawling & scraping framework for Elixir.
Stars: ✭ 440 (-63.93%)
Mutual labels: crawler, scraper
Freshonions Torscraper
Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion
Stars: ✭ 348 (-71.48%)
Mutual labels: crawler, scraper
Lulu
[Unmaintained] A simple and clean video/music/image downloader 👾
Stars: ✭ 789 (-35.33%)
Mutual labels: crawler, scraper
Social Scraper
Tổng hợp script crawl dữ liệu từ các mạng xã hội & website tiếng Việt
Stars: ✭ 47 (-96.15%)
Mutual labels: crawler, scraper
Wombat
Web scraper with an elegant DSL that parses structured data from web pages.
Usage:
gem install wombat
Scraping a page:
The simplest way to use Wombat is by calling Wombat.crawl
and passing it a block:
require 'wombat'
Wombat.crawl do
base_url "https://www.github.com"
path "/"
headline xpath: "//h1"
subheading css: "p.alt-lead"
what_is({ css: ".one-fourth h4" }, :list)
links do
explore xpath: '/html/body/header/div/div/nav[1]/a[4]' do |e|
e.gsub(/Explore/, "Love")
end
features css: '.nav-item-opensource'
business css: '.nav-item-business'
end
end
The code above is gonna return the following hash:
{
"headline"=>"How people build software",
"subheading"=>"Millions of developers use GitHub to build personal projects, support their businesses, and work together on open source technologies.",
"what_is"=>[
"For everything you build",
"A better way to work",
"Millions of projects",
"One platform, from start to finish"
],
"links"=>{
"explore"=>"Love",
"features"=>"Open source",
"business"=>"Business"
}
}
This is just a sneak peek of what Wombat can do. For the complete documentation, please check the links below:
Wiki
API Documentation
Changelog
Contributing to Wombat
- Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
- Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
- Fork the project
- Start a feature/bugfix branch
- Commit and push until you are happy with your contribution
- Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
- Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
Contributors
- Felipe Lima (@felipecsl)
- List of all contributors
Copyright
Copyright (c) 2019 Felipe Lima. See LICENSE.txt for further details.
License
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].