3241 Open source projects that are alternatives of or similar to Scrapy_demo

Sample Event Sourcing implementation with .NET Core

Dynamic configurable crawl (动态可配置化爬虫)

Learn Clean Architecture with .NET Core 3.0 🔥

NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider

Instagram Clone - Cloning Instagram for learning purpose

10 photo website spiders, 10 个国外图库的 scrapy 爬虫代码

免费 IP 代理池。Scrapy 爬虫框架插件

A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.

🎊 Design and implement of lightweight crawler framework.

V2EX爬虫

一个基于Scrapy的数据采集爬虫代码库

A django admin site for scrapy

**Unofficial / Community** Kafka Connect MongoDB Sink Connector - Find the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector

百度贴吧爬虫(基于scrapy和mysql)

Uno Crawler por https://douban.com

More than 130+ pages in this beautiful app and more than 45 developers has contributed to it.

Generating spiders dynamically to crawl and check those free proxy ip on the internet with scrapy.

🎊 Design and implement of lightweight crawler framework.

Confluent Platform Demo including Apache Kafka, ksqlDB, Control Center, Replicator, Confluent Schema Registry, Security

Course demo code and other hand-out materials for our data-driven web apps in Flask course

PTT 圖片下載器 (抓取整個看板的圖片，並用文章標題作為資料夾的名稱 ) (使用Scrapy)

大数据生态解决方案数据平台：基于大数据、数据平台、微服务、机器学习、商城、自动化运维、DevOps、容器部署平台、数据平台采集、数据平台存储、数据平台计算、数据平台开发、数据平台应用搭建的大数据解决方案。

In this repository, I have developed the entire server-side principal architecture for real-time stock market prediction with Machine Learning. I have used Tensorflow.js for constructing ml model architecture, and Kafka for real-time data streaming and pipelining.

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

spring、spring-boot、spring-cloud 常用整合用例

The Ultimate Boilerplate for Products.

A simple demo app using node and mongodb for beginners

MEAN Stack (Angular 5) CRUD Web Application Example

Efficient DMA timeout mechanism for peripheral DMA configured in circular mode demonstrated on a STM32 microcontroller.

web spider built by puppeteer, support task-queue and task-scheduling by decorators，support nedb / mongodb, support data visualization; 基于puppeteer的web爬虫框架，提供灵活的任务队列管理调度方案，提供便捷的数据保存方案（nedb/mongodb），提供数据可视化和用户交互的实现方案

Streaming reference architecture for ETL with Kafka and Kafka-Connect. You can find more on http://lenses.io on how we provide a unified solution to manage your connectors, most advanced SQL engine for Kafka and Kafka Streams, cluster monitoring and alerting, and more.

两只蠢萌京东的分布式爬虫.

🐂基于springcloud +dubbo构建的模拟秒杀项目，模块化设计，集成了分库分表、elasticsearch🔍、gateway、mybatis-plus、spring-session等常用开源组件

👽 OneBlog，一个简洁美观、功能强大并且自适应的Java博客

深圳地铁大数据客流分析系统🚇🚄🌟

👾Scripts and samples to support Confluent Demos and Talks. ⚠️Might be rough around the edges ;-) 👉For automated tutorials and QA'd code, see https://github.com/confluentinc/examples/

A python web scraper for public email lists.

Python + Scrapy + MongoDB . 5 million data per day !!!💥 The world's largest website.

JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816

Go语言实例教程从入门到进阶，包括基础库使用、设计模式、面试易错点、工具类、对接第三方等

爬取百度贴吧、TapTap、appstore、微博官方博主上的游戏评论(基于redis_scrapy)，过滤器采用了bloomfilter。

Securing MEAN Stack (Angular 5) Web Application using Passport Authentication

The purpose of this tiny project is to put things together with the know how that i learned from the course big data expert from formacionhadoop.com The idea is to show how to play with apache spark streaming, kafka,mongo, spark machine learning algorithms.

Example project to guide users through how to create game with Cocos Creator

A multi-thread crawler framework with many builtin image crawlers provided.

AlipaySpider on Scrapy(use chrome driver); 支付宝爬虫(基于Scrapy)

An Event Sourcing service template with DDD, TDD and SOLID. It has High Cohesion and Loose Coupling, it's a good start for your next Microservice application.

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

Testing Using Python Demo. 使用Python测试脚本demo。

Example end to end data engineering project.

cesium demo，cesium 研究过程中资料整理，包括： 1、示例，提供常规的cesium示例以及综合性的业务示例 2、工具，整理研究过程中写的一些数据获取、处理、分析的工具，例如网络3dtiles数据爬虫工具、b3dm 转 gltf工具 3、插件，根据研究过程中cesium感觉使用不方便的地方，添加插件，包括针对显示控件的汉化中文语言插件等

Example Django application running on CircleCI

Scrapy pipeline to store chunked items into Amazon S3 or Google Cloud Storage bucket.

admin ui for scrapy/open source scrapinghub

This repo contains my notes on working with Go and computer systems.

👉一个基于spring boot 实现的java股票爬虫(仅支持A股)，如果你❤️请⭐️ . V2升级版正在开发中！

前端性能监控系统,消息队列,高可用,集群等相关架构

Spring Boot 2.x code examples

支持mysql、MongoDB数据变更订阅分发

Unmaintained 🐳 ☕️ 🕷 Scala crawler(spider) framework, inspired by scrapy, created by @gaocegege