All Projects → wx-chevalier → DistributedCompute-Series

wx-chevalier / DistributedCompute-Series

Licence: other
分布式计算:分布式数据处理(流计算/批处理)、消息队列、数据仓库

Programming Languages

HTML
75241 projects

Projects that are alternatives of or similar to DistributedCompute-Series

Cuneiform
Cuneiform distributed programming language
Stars: ✭ 175 (+629.17%)
Mutual labels:  distributed-computing
Holochain
The new, performant, and simplified version of Holochain on Rust (sometimes called Holochain RSM for Refactored State Model)
Stars: ✭ 205 (+754.17%)
Mutual labels:  distributed-computing
Transducers.jl
Efficient transducers for Julia
Stars: ✭ 226 (+841.67%)
Mutual labels:  distributed-computing
Mlcomp
Distributed DAG (Directed acyclic graph) framework for machine learning with UI
Stars: ✭ 183 (+662.5%)
Mutual labels:  distributed-computing
Geminigraph
A computation-centric distributed graph processing system.
Stars: ✭ 196 (+716.67%)
Mutual labels:  distributed-computing
Gosiris
An actor framework for Go
Stars: ✭ 222 (+825%)
Mutual labels:  distributed-computing
Hydra Express
A module which wraps Hydra and ExpressJS into a library for building distributed applications - such as microservices
Stars: ✭ 166 (+591.67%)
Mutual labels:  distributed-computing
Amadeus
Harmonious distributed data analysis in Rust.
Stars: ✭ 240 (+900%)
Mutual labels:  distributed-computing
Qix
Machine Learning、Deep Learning、PostgreSQL、Distributed System、Node.Js、Golang
Stars: ✭ 13,740 (+57150%)
Mutual labels:  distributed-computing
Catalyst
Accelerated deep learning R&D
Stars: ✭ 2,804 (+11583.33%)
Mutual labels:  distributed-computing
Js Spark
Realtime calculation distributed system. AKA distributed lodash
Stars: ✭ 187 (+679.17%)
Mutual labels:  distributed-computing
Sparktorch
Train and run Pytorch models on Apache Spark.
Stars: ✭ 195 (+712.5%)
Mutual labels:  distributed-computing
Dispy
Distributed and Parallel Computing Framework with / for Python
Stars: ✭ 222 (+825%)
Mutual labels:  distributed-computing
Ml Systems
papers on scalable and efficient machine learning systems
Stars: ✭ 182 (+658.33%)
Mutual labels:  distributed-computing
Backend.ai
Backend.AI is a streamlined, container-based computing cluster orchestrator that hosts diverse programming languages and popular computing/ML frameworks, with pluggable heterogeneous accelerator support including CUDA and ROCM.
Stars: ✭ 233 (+870.83%)
Mutual labels:  distributed-computing
Klyng
A message-passing distributed computing framework for node.js
Stars: ✭ 167 (+595.83%)
Mutual labels:  distributed-computing
Awesome Parallel Computing
A curated list of awesome parallel computing resources
Stars: ✭ 212 (+783.33%)
Mutual labels:  distributed-computing
taskinator
A simple orchestration library for running complex processes or workflows in Ruby
Stars: ✭ 25 (+4.17%)
Mutual labels:  distributed-computing
Selinon
An advanced distributed task flow management on top of Celery
Stars: ✭ 237 (+887.5%)
Mutual labels:  distributed-computing
Nakedtensor
Bare bone examples of machine learning in TensorFlow
Stars: ✭ 2,443 (+10079.17%)
Mutual labels:  distributed-computing

Contributors Forks Stargazers Issues license: CC BY-NC-SA 4.0


Logo

在线阅读 >>

速览手册 · 代码案例 · 参考资料 · English Version

Distributed System Series(分布式系统·实践笔记)

现实世界中的数据系统往往颇为复杂。大型应用程序经常需要以多种方式访问和处理数据,没有一个数据库可以同时满足所有这些不同的需求。因此应用程序通常组合使用多种组件:数据存储,索引,缓存,分析系统,等等,并实现在这些组件中移动数据的机制。许多现有数据系统中都采用这种数据处理方式:你发送请求指令,一段时间后(我们期望)系统会给出一个结果。数据库,缓存,搜索索引,Web 服务器以及其他一些系统都以这种方式工作。

像这样的在线(online)系统,无论是浏览器请求页面还是调用远程 API 的服务,我们通常认为请求是由人类用户触发的,并且正在等待响应。他们不应该等太久,所以我们非常关注系统的响应时间。值得说明的是,这不是构建系统的唯一方式,其他方法也有其优点。我们来看看三种不同类型的系统:

  • 服务(在线系统):服务等待客户的请求或指令到达。每收到一个,服务会试图尽快处理它,并发回一个响应。响应时间通常是服务性能的主要衡量指标,可用性通常非常重要(如果客户端无法访问服务,用户可能会收到错误消息)。

  • 批处理系统(离线系统):一个批处理系统有大量的输入数据,跑一个作业(job)来处理它,并生成一些输出数据,这往往需要一段时间(从几分钟到几天),所以通常不会有用户等待作业完成。相反,批量作业通常会定期运行(例如,每天一次)。批处理作业的主要性能衡量标准通常是吞吐量(处理特定大小的输入所需的时间)。

  • 流处理系统(准实时系统):流处理介于在线和离线(批处理)之间,所以有时候被称为准实时(near-real-time)或准在线(nearline)处理。像批处理系统一样,流处理消费输入并产生输出(并不需要响应请求)。但是,流式作业在事件发生后不久就会对事件进行操作,而批处理作业则需等待固定的一组输入数据。这种差异使流处理系统比起批处理系统具有更低的延迟。

分布式计算概念图

Nav | 关联导航

  • 如果你想了解微服务/云原生等分布式系统的应用实践,可以参阅;如果你想了解数据库相关,可以参阅 Database-Series;如果你想了解虚拟化与云计算相关,可以参阅 Cloud-Series;如果你想了解 Linux 与操作系统相关,可以参阅 Linux-Series

About

Copyright & More | 延伸阅读

笔者所有文章遵循 知识共享 署名-非商业性使用-禁止演绎 4.0 国际许可协议,欢迎转载,尊重版权。您还可以前往 NGTE Books 主页浏览包含知识体系、编程语言、软件工程、模式与架构、Web 与大前端、服务端开发实践与工程架构、分布式基础架构、人工智能与深度学习、产品运营与创业等多类目的书籍列表:

NGTE Books

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].