All Projects → neighborhood999 → fiveN1-rent-scraper

neighborhood999 / fiveN1-rent-scraper

Licence: MIT license
🏠 a.k.a 591 rent scraper(591 租屋網爬蟲)

Programming Languages

go
31211 projects - #10 most used programming language
Makefile
30231 projects

Projects that are alternatives of or similar to fiveN1-rent-scraper

Jvppeteer
Headless Chrome For Java (Java 爬虫)
Stars: ✭ 193 (+278.43%)
Mutual labels:  scraper
Ruiji.net
crawler framework, distributed crawler extractor
Stars: ✭ 220 (+331.37%)
Mutual labels:  scraper
Instagram Proxy Api
CORS compliant API to access Instagram's public data
Stars: ✭ 245 (+380.39%)
Mutual labels:  scraper
Querylist
🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
Stars: ✭ 2,392 (+4590.2%)
Mutual labels:  scraper
Media Scraper
Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok
Stars: ✭ 206 (+303.92%)
Mutual labels:  scraper
Annie
👾 Fast and simple video download library and CLI tool written in Go
Stars: ✭ 16,369 (+31996.08%)
Mutual labels:  scraper
Unfurl
Scraper for oEmbed, Twitter Cards and Open Graph metadata - fast and Promise-based ⚡️
Stars: ✭ 193 (+278.43%)
Mutual labels:  scraper
TwitterScraper
Scrape a User's Twitter data! Bypass the 3,200 tweet API limit for a User!
Stars: ✭ 80 (+56.86%)
Mutual labels:  scraper
Goose Parser
Universal scrapping tool, which allows you to extract data using multiple environments
Stars: ✭ 211 (+313.73%)
Mutual labels:  scraper
Scrape Linkedin Selenium
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
Stars: ✭ 239 (+368.63%)
Mutual labels:  scraper
Weibo terminater
Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator
Stars: ✭ 2,295 (+4400%)
Mutual labels:  scraper
Tianyancha
pip安装的天眼查爬虫API,指定的单个/多个企业工商信息一键保存为Excel/JSON格式。A Battery-included Scraper API of Tianyancha, the best Chinese business data and investigation platform.
Stars: ✭ 206 (+303.92%)
Mutual labels:  scraper
Skrape.it
A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.
Stars: ✭ 231 (+352.94%)
Mutual labels:  scraper
Jsonframe Cheerio
simple multi-level scraper json input/output for Cheerio
Stars: ✭ 196 (+284.31%)
Mutual labels:  scraper
Heroku ebooks
A script to generate Markov chains and to post to an _ebooks account on Twitter using Heroku
Stars: ✭ 251 (+392.16%)
Mutual labels:  scraper
Node Ytdl Core
YouTube video downloader in javascript.
Stars: ✭ 3,004 (+5790.2%)
Mutual labels:  scraper
Scrapysharp
reborn of https://bitbucket.org/rflechner/scrapysharp
Stars: ✭ 226 (+343.14%)
Mutual labels:  scraper
tv grab fr telerama
XMLTV Grabber using telerama api data
Stars: ✭ 36 (-29.41%)
Mutual labels:  scraper
Polite
Be nice on the web
Stars: ✭ 253 (+396.08%)
Mutual labels:  scraper
Getsy
A simple browser/client-side web scraper.
Stars: ✭ 238 (+366.67%)
Mutual labels:  scraper

fiveN1 Rent Scraper

logo

CI Go Report Card GoDoc

a.k.a 591 rent scraper.

Easy scraping 591 rent information.

Installation

$ make install

Usage

package main

import (
	"log"

	rent "github.com/neighborhood999/fiveN1-rent-scraper"
)

func main() {
	options := rent.NewOptions()
	url, err := rent.GenerateURL(options)
	if err != nil {
		log.Fatalf("\x1b[91;1m%s\x1b[0m", err)
	}

	f := rent.NewFiveN1(url)
	if err := f.Scrape(1); err != nil {
		log.Fatal(err)
	}

	json := rent.ConvertToJSON(f.RentList)
	log.Println(string(json))
}

And output json:

{
  "1": [
    {
      "preview":
        "https://hp1.591.com.tw/house/active/2013/11/04/138356961541004002_765x517.water3.jpg",
      "title": "松山路套房 * 交通方便近捷運後山埤站 *",
      "url": "https://rent.591.com.tw/rent-detail-5857738.html",
      "address": "信義區 - 松山路 119 巷",
      "rentType": "沒有格局說明",
      "optionType": "獨立套房",
      "ping": "6",
      "floor": "樓層:3/4",
      "price": "9,000 元 / 月",
      "isNew": true
    }
  ]
}

Index number is the representation page number, every index contain 30 items. 🏠

Options

Create and generate URL by options:

options := rent.NewOptions()
url, err := rent.GenerateURL(options)
if err != nil {
	log.Fatal(err)
}

log.Println(url)

You may set more options for your requirement. Reference below Options struct:

type Options struct {
	Region      int    `url:"region"`                // 地區 - 預設:`1`
	Section     string `url:"section,omitempty"`     // 鄉鎮 - 可選擇多個區域,例如:`section=7,4`
	Kind        int    `url:"kind"`                  // 租屋類型 - `0`:不限、`1`:整層住家、`2`:獨立套房、`3`:分租套房、`4`:雅房、`8`:車位,`24`:其他
	RentPrice   string `url:"rentprice,omitempty"`   // 租金 - `2`:5k - 10k、`3`:10k - 20k、`4`: 20k - 30k;或者可以輸入價格範圍,例如:`0,10000`
	Area        string `url:"area,omitempty"`        // 坪數格式 - `10,20`(10 到 20 坪)
	Order       string `url:"order"`                 // 貼文時間 - 預設使用刊登時間:`posttime`,或是使用價格排序:`money`
	OrderType   string `url:"orderType"`             // 排序方式 - `desc` 或 `asc`
	Sex         int    `url:"sex,omitempty"`         // 性別 - `0`:不限、`1`:男性、`2`:女性
	HasImg      string `url:"hasimg,omitempty"`      // 過濾是否有「房屋照片」 - ``:空值(不限)、`1`:是
	NotCover    string `url:"not_cover,omitempty"`   // 過濾是否為「頂樓加蓋」 - ``:空值(不限)、`1`:是
	Role        string `url:"role,omitempty"`        // 過濾是否為「屋主刊登」 - ``:空值(不限)、`1`:是
	Shape       string `url:"shape,omitempty"`       // 房屋類型 - `1`:公寓、`2`:電梯大樓、`3`:透天厝、`4`:別墅
	Pattern     string `url:"pattern,omitempty"`     // 格局單選 - `0`:不限、`1`:一房、`2``:兩房、`3`:三房、`4`:四房、`5`:五房以上
	PatternMore string `url:"patternMore,omitempty"` // 格局多選 - 參考「格局單選」,可以選多種格局,例如:`1,2,3,4,5`
	Floor       string `url:"floor,omitempty"`       // 樓層 - `0,0`:不限、`0,1`:一樓、`2,6`:二樓到六樓、`6,12`:六樓到十二樓、`12,`:十二樓以上
	Option      string `url:"option,omitempty"`      // 提供設備 - `tv`:電視、`cold`:冷氣、`icebox`:冰箱、`hotwater`:熱水器、`naturalgas`:天然瓦斯、`four`:第四台、`broadband`:網路、`washer`:洗衣機、`bed`:床、`wardrobe`:衣櫃、`sofa`:沙發。可選擇多個設備,例如:option=tv,cold
	Other       string `url:"other,omitempty"`       // 其他條件 - `cartplace`:有車位、`lift`:有電梯、`balcony_1`:有陽台、`cook`:可開伙、`pet`:可養寵物、`tragoods`:近捷運、`lease`:可短期租賃。可選擇多個條件,例如:other=cartplace,cook
	FirstRow    int    `url:"firstRow"`
}

For example:

options := rent.NewOptions()
options.Kind = 3
options.HasImg = "1"
options.NotCover = "1"
options.Role = "1"

rent.GenerateURL(options)

Multiple Pages

If you want to get more rent pieces of information, setting page amount in Scrape method:

f := rent.NewFiveN1(url)
f.Scrape(5) // page.1 to page.5

When amount > 1, it will start goroutine automatically and correspond to the page number to scrape.

Code Lists

LICENSE

MIT © Peng Jie

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].