GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → GoTrained → Scrapy Craigslist

GoTrained / Scrapy Craigslist

Web Scraping Craigslist's Engineering Jobs in NY with Scrapy

Programming Languages

python
139335 projects - #7 most used programming language

Labels

scrapyweb-scrapingweb-scraper

Projects that are alternatives of or similar to Scrapy Craigslist

Faster Than Requests
Faster requests on Python 3
Stars: ✭ 639 (+1083.33%)
Mutual labels:  scrapy, web-scraping, web-scraper
Scrapple
A framework for creating semi-automatic web content extractors
Stars: ✭ 464 (+759.26%)
Mutual labels:  scrapy, web-scraping, web-scraper
OLX Scraper
📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.
Stars: ✭ 15 (-72.22%)
Mutual labels:  web-scraper, web-scraping, scrapy
Linkedin-Client
Web scraper for grabing data from Linkedin profiles or company pages (personal project)
Stars: ✭ 42 (-22.22%)
Mutual labels:  web-scraper, web-scraping
City Scrapers
Scrape, standardize and share public meetings from local government websites
Stars: ✭ 220 (+307.41%)
Mutual labels:  scrapy, web-scraping
scrapy-wayback-machine
A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
Stars: ✭ 92 (+70.37%)
Mutual labels:  web-scraping, scrapy
Awesome Web Scraper
A collection of awesome web scaper, crawler.
Stars: ✭ 147 (+172.22%)
Mutual labels:  scrapy, web-scraper
top-github-scraper
Scape top GitHub repositories and users based on keywords
Stars: ✭ 40 (-25.93%)
Mutual labels:  web-scraper, web-scraping
IMDB-Scraper
Scrapy project for scraping data from IMDB with Movie Dataset including 58,623 movies' data.
Stars: ✭ 37 (-31.48%)
Mutual labels:  web-scraping, scrapy
restaurant-finder-featureReviews
Build a Flask web application to help users retrieve key restaurant information and feature-based reviews (generated by applying market-basket model – Apriori algorithm and NLP on user reviews).
Stars: ✭ 21 (-61.11%)
Mutual labels:  web-scraping, scrapy
Basketball reference web scraper
NBA Stats API via Basketball Reference
Stars: ✭ 279 (+416.67%)
Mutual labels:  web-scraping, web-scraper
Scrapy Fake Useragent
Random User-Agent middleware based on fake-useragent
Stars: ✭ 520 (+862.96%)
Mutual labels:  scrapy, web-scraping
Scrapy Training
Scrapy Training companion code
Stars: ✭ 157 (+190.74%)
Mutual labels:  scrapy, web-scraping
Netflix Clone
Netflix like full-stack application with SPA client and backend implemented in service oriented architecture
Stars: ✭ 156 (+188.89%)
Mutual labels:  scrapy, web-scraping
Juno crawler
Scrapy crawler to collect data on the back catalog of songs listed for sale.
Stars: ✭ 150 (+177.78%)
Mutual labels:  scrapy, web-scraping
scraping-ebay
Scraping Ebay's products using Scrapy Web Crawling Framework
Stars: ✭ 79 (+46.3%)
Mutual labels:  web-scraping, scrapy
Scrape Linkedin Selenium
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
Stars: ✭ 239 (+342.59%)
Mutual labels:  web-scraping, web-scraper
Scrapyd Cluster On Heroku
Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉
Stars: ✭ 106 (+96.3%)
Mutual labels:  scrapy, web-scraping
Php Curl Class
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
Stars: ✭ 2,903 (+5275.93%)
Mutual labels:  web-scraping, web-scraper
Spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Stars: ✭ 656 (+1114.81%)
Mutual labels:  web-scraping, web-scraper
View All Similar Projects ➔

Scrapy-Craigslist

Scrapy tutorial to build a Craigslist crawler that scrapes engineering jobs in New York.

Scrapy Tutorial: Craigslist

Check this blog post for the full Scrapy tutorial:

http://python.gotrained.com/scrapy-tutorial-web-scraping-craigslist

Usage

In Terminal or CMD, navigate to the main Scrapy project folder, and run one of the spiders:

  • Scraping Craigslist's Engineering Job Titles from One Page

scrapy crawl titles -o job-titles.csv

  • Scraping Craigslist's Engineering Job Titles, Addresses, and URLs from One Page

scrapy crawl jobsone -o job-one-page.csv

  • Scraping Craigslist's Engineering Job Titles, Addresses, and URLs from All Pages

scrapy crawl jobsall -o job-all-pages.csv

  • Scraping Craigslist's Engineering Job Details from Each Job's Web Page

scrapy crawl jobscontent -o job-all-pages-content.csv

Scrapy Online Course

Check this Scrapy tutorial to learn much more:

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].