Full Stack Web Application
Purpose
This is a demo project for showing how one full stack web application should be like.
Introduction
features:
- scrawler get data into database
- webserver push data to browser periodically
Demo
Architecture
+-----------+ +------------+ +------------+ +--------------+
| | frontend | | | | data | |
| Browser +<----------------+ Vue Server| | RestAPI +<--------------+ Scrapy |
| | | | | | | |
+------+----+ +------------+ +------+-----+ +--------------+
^ |
| |
| |data
| |
| |
| v
| +-------------+ +------+------+
| data | | data | |
+---------------------+ Websocket +<-------------+ MongoDB |
| | | |
+-------------+ +-------------+
Workflow
- scratch quotes from scrapy offical demo site(quotes), save to database(MongoDB), and show them through web.
- Front end is vue.js project, including WebSocket client(socketio) component.
- Back end is websocket server, which query data form MongoDB through ORM mongoose every period(1s), pushing quotes data to front end.
- data picker use scrapy to crawl quotes to save to MongoDB.
Technology Stack
Scrapy + webservice[restful] + websocket + VueJS + MongoDB
| Component | Description |
|---|---|
| Scrapy | scratch data from web |
| webservice | serve restapi for data storation and fetch |
| websocket | push data to browser |
| VueJS | frontend |
| MongoDB | DB |
Install
./bin/install.sh
Run
# run all services
./bin/start.sh
# stop all services
./bin/stop.sh
# start scrapy
./bin/start_scrapy.sh