davidesantangelo / Webinspector
Licence: mit
Ruby gem to inspect completely a web page. It scrapes a given URL, and returns you its meta, links, images more.
Stars: ✭ 288
Programming Languages
ruby
36898 projects - #4 most used programming language
Projects that are alternatives of or similar to Webinspector
fiction-dl
A content downloader, capable of retrieving works of (fan)fiction from the web and saving them in a few common file formats.
Stars: ✭ 22 (-92.36%)
Mutual labels: scraper
slyblime
Interactive Lisp IDE with REPL, Inspector, Debugger and more for Sublime Text 4.
Stars: ✭ 35 (-87.85%)
Mutual labels: inspector
Smart-Inspector
Fluent re-take on Unity Inspector UX. Packed with QoL improvements.
Stars: ✭ 680 (+136.11%)
Mutual labels: inspector
kaa.si-cli
Stream anime from kaa.si and sync with anilist
Stars: ✭ 12 (-95.83%)
Mutual labels: scraper
espressione
Espressione is a Ruby community-driven common regular expression patterns gem
Stars: ✭ 13 (-95.49%)
Mutual labels: rubygem
shortuuid.rb
Convert UUIDs & numbers into space efficient and URL-safe Base62 strings, or any other alphabet.
Stars: ✭ 38 (-86.81%)
Mutual labels: rubygem
Tty Logger
A readable, structured and beautiful logging for the terminal
Stars: ✭ 280 (-2.78%)
Mutual labels: rubygem
Unity Editor Toolbox
Tools, custom attributes, drawers, hierarchy overlay, and other extensions for the Unity Editor.
Stars: ✭ 273 (-5.21%)
Mutual labels: inspector
nepse-data
Extracting data of Nepal Stock Exchange Ltd. (NEPSE)
Stars: ✭ 15 (-94.79%)
Mutual labels: scraper
lightnovel epub
🍭 epub generator for (light)novels (轻) 小说 epub 生成器,支持站点:轻之国度、轻小说文库
Stars: ✭ 89 (-69.1%)
Mutual labels: scraper
Instagram-Scraper-2021
Scrape Instagram content and stories anonymously, using a new technique based on the har file (No Token + No public API).
Stars: ✭ 57 (-80.21%)
Mutual labels: scraper
Tty Markdown
Convert a markdown document or text into a terminal friendly output.
Stars: ✭ 275 (-4.51%)
Mutual labels: rubygem
SearchScraperAPI
Aiohttp web server API, which scrapes Google and returns scrape results as response. Supports proxies, multiple geos and number of results.
Stars: ✭ 31 (-89.24%)
Mutual labels: scraper
fb-scraper
Scrape a Facebook profile and turn it into a JSON file
Stars: ✭ 18 (-93.75%)
Mutual labels: scraper
Dry Configurable
A simple mixin to make Ruby classes configurable
Stars: ✭ 280 (-2.78%)
Mutual labels: rubygem
Java Spider
一个基于webmagic框架二次开发的java爬虫框架实战,已实现能爬取腾讯,搜狐,今日头条(单独集成功能)等资讯内容,配合elasticsearch框架用法,实现了自动爬虫,已投入线上生产使用。
Stars: ✭ 276 (-4.17%)
Mutual labels: scraper
Weibo terminator workflow
Update Version of weibo_terminator, This is Workflow Version aim at Get Job Done!
Stars: ✭ 259 (-10.07%)
Mutual labels: scraper
Webinspector
Ruby gem to inspect completely a web page. It scrapes a given URL, and returns you its title, description, meta, links, images and more.
See it in action!
You can try WebInspector live at this little demo: https://scrappet.herokuapp.com
Installation
Add this line to your application's Gemfile:
gem 'webinspector'
And then execute:
$ bundle
Or install it yourself as:
$ gem install webinspector
Usage
Initialize a WebInspector instance for an URL, like this:
page = WebInspector.new('http://davidesantangelo.com')
Accessing response status and headers
You can check the status and headers from the response like this:
page.response.status # 200
page.response.headers # { "server"=>"apache", "content-type"=>"text/html; charset=utf-8", "cache-control"=>"must-revalidate, private, max-age=0", ... }
Accessing inpsected data
You can see the data like this:
page.url # URL of the page
page.scheme # Scheme of the page (http, https)
page.host # Hostname of the page (like, davidesantangelo.com, without the scheme)
page.port # Port of the page
page.title # title of the page from the head section, as string
page.description # description of the page
page.links # every link found
page.images # every image found
page.meta # metatags of the page
Accessing meta tags
page.meta # metatags of the page
page.meta['description'] # meta description
page.meta['keywords'] # meta keywords
Find words (as array)
page.find(["word1, word2"]) # return {"word1"=>3, "word2"=>1}
Contributors
- Steven Shelby (@stevenshelby)
- Sam Nissen (@samnissen)
License
The webinspector GEM is released under the MIT License.
Contributing
- Fork it ( https://github.com/[my-github-username]/webinspector/fork )
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create a new Pull Request
develop
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].