All Projects → drafish → dc-sdk-js

drafish / dc-sdk-js

Licence: other
一个基于浏览器环境的数据采集SDK

Programming Languages

javascript
184084 projects - #8 most used programming language
typescript
32286 projects
HTML
75241 projects
CSS
56736 projects
Vue
7211 projects

Projects that are alternatives of or similar to dc-sdk-js

datalake-etl-pipeline
Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations
Stars: ✭ 39 (-25%)
Mutual labels:  data-pipeline
network-pipeline
Network traffic data pipeline for real-time predictions and building datasets for deep neural networks
Stars: ✭ 36 (-30.77%)
Mutual labels:  data-pipeline
React Native Firebase
🔥 A well-tested feature-rich modular Firebase implementation for React Native. Supports both iOS & Android platforms for all Firebase services.
Stars: ✭ 9,674 (+18503.85%)
Mutual labels:  web-sdk
opentrials-airflow
Configuration and definitions of Airflow for OpenTrials
Stars: ✭ 18 (-65.38%)
Mutual labels:  data-pipeline
richflow
A Node.js and JavaScript synchronous data pipeline processing, data sharing and stream processing library. Actionable & Transformable Pipeline data processing.
Stars: ✭ 17 (-67.31%)
Mutual labels:  data-pipeline
pipeline
OONI data processing pipeline
Stars: ✭ 36 (-30.77%)
Mutual labels:  data-pipeline
machine-learning-data-pipeline
Pipeline module for parallel real-time data processing for machine learning models development and production purposes.
Stars: ✭ 22 (-57.69%)
Mutual labels:  data-pipeline
AirflowETL
Blog post on ETL pipelines with Airflow
Stars: ✭ 20 (-61.54%)
Mutual labels:  data-pipeline
augraphy
Augmentation pipeline for rendering synthetic paper printing, faxing, scanning and copy machine processes
Stars: ✭ 49 (-5.77%)
Mutual labels:  data-pipeline
Data Engineering Howto
A list of useful resources to learn Data Engineering from scratch
Stars: ✭ 2,056 (+3853.85%)
Mutual labels:  data-pipeline
ATOM
Automated Tool for Optimized Modelling
Stars: ✭ 85 (+63.46%)
Mutual labels:  data-pipeline
rivery cli
Rivery CLI
Stars: ✭ 16 (-69.23%)
Mutual labels:  data-pipeline
trembita
Model complex data transformation pipelines easily
Stars: ✭ 44 (-15.38%)
Mutual labels:  data-pipeline
practical-data-engineering
Real estate dagster pipeline
Stars: ✭ 110 (+111.54%)
Mutual labels:  data-pipeline
scicloj.ml
A Clojure machine learning library
Stars: ✭ 152 (+192.31%)
Mutual labels:  data-pipeline
Data-pipeline-project
Data pipeline project
Stars: ✭ 18 (-65.38%)
Mutual labels:  data-pipeline
serverless-data-pipeline-sam
Serverless Data Pipeline powered by Kinesis Firehose, API Gateway, Lambda, S3, and Athena
Stars: ✭ 78 (+50%)
Mutual labels:  data-pipeline
datajob
Build and deploy a serverless data pipeline on AWS with no effort.
Stars: ✭ 101 (+94.23%)
Mutual labels:  data-pipeline
aws-pdf-textract-pipeline
🔍 Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript
Stars: ✭ 141 (+171.15%)
Mutual labels:  data-pipeline
Snowplow
The enterprise-grade behavioral data engine (web, mobile, server-side, webhooks), running cloud-natively on AWS and GCP
Stars: ✭ 5,935 (+11313.46%)
Mutual labels:  data-pipeline

dc-sdk-js ——一个基于浏览器环境的数据采集SDK

功能点

  • 设备信息采集
  • 监听页面加载离开
  • 监听hash变化
  • 监听页面点击
  • 监听标签页隐藏显示
  • 轮询发送热力图采集数据
  • 轮询采集页面特定信息(以手机号为例)

以上功能都是精简后的,只作为demo展示,实际使用需要自身的业务场景酌情添加代码。

解决开发痛点

  • 用webpack打包代码,支持模块化开发
  • 用core-js提供pollyfill环境,支持es6语法
  • 用eslint+standard规范代码风格,提升代码可读性和可维护性
  • 提供了完善的测试页面,方便开发调试

浏览器支持

所有现代浏览器 和 IE8+.

技术栈思维导图

avatar

项目结构

.
+-- server            //服务端测试demo
+-- spa-angular       //angular测试demo
+-- spa-react         //react测试demo
+-- spa-vue           //vue测试demo
+-- config            //webpack配置
+-- src               //项目源码
|   +-- common        //通用模块
|   +-- config        //配置模块
|   +-- core          //核心模块,业务代码都在这个目录下
|       +-- interval  //轮询相关业务代码
|       +-- listener  //监听相关业务代码
|       +-- index.js  //入口文件
|       +-- params.js //封装了对请求参数的相关操作
|       +-- sender.js //封装了ajax和websocket请求
+-- test              //单元测试
+-- package.json

接入方式

(function() {
  var collect = document.createElement('script');
  collect.type = 'text/javascript';
  collect.async = true;
  collect.src =  '/index.js';
  var s = document.getElementsByTagName('script')[0];
  s.parentNode.insertBefore(collect, s);
})();

var _XT = [];

_XT.push(['token','custom token']);
_XT.push(['channelCode','custom channelCode']);
_XT.push(['serverUrl', 'http://localhost:8081/api'])
_XT.push(['wsUrl', 'ws://localhost:8081'])

配置说明

字段名称 类型 说明
serverUrl string http服务端地址
wsUrl string websocket服务端地址
token string 用于服务端鉴权
channelCode string 用于区分不同的业务方
heatmapUrls array 热力图采集页面配置
heatMapDelay string 热力图采集数据发送时间间隔
phoneListDelay string 手机号轮询采集时间间隔
iframe string iframe地址,用于跨域共享数据

采集字段说明

字段名称 类型 说明
deviceType string 设备类型
language string 浏览器语言类型
screenHigh string 屏幕高度
screenWide string 屏幕宽度
type string 采集事件类型
currentTime number 当前时间戳
title string 当前页面标题
url string 当前页面URL
referrer string 来源页URL
data object 采集事件的额外信息
channelCode string 用于区分不同的业务方
token string 用于服务端鉴权
deviceId string 设备ID
userId string 用户ID

安装依赖

npm install

开发环境启动

npm run dev

生产环境构建

npm run build

服务端启动

npm run server

单页应用构建

## angular
cd spa-angular
npm run build

## react
cd spa-react
npm run build

## vue
cd spa-vue
npm run build

服务端测试demo

为方便开发环境调试,我用node写了一个简易的服务端项目,其中实现了以下功能点:

  • http服务
  • websocket服务
  • 服务端渲染测试demo
  • 单页应用测试demo(包括vue、react、angular)

参考项目

License

MIT

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].