GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → scribd → Objinsync

scribd / Objinsync

Licence: mit
Continuously synchronize directories from remote object store to local filesystem

Programming Languages

go
31211 projects - #10 most used programming language

Labels

s3airflow

Projects that are alternatives of or similar to Objinsync

Udacity Data Engineering
Udacity Data Engineering Nano Degree (DEND)
Stars: ✭ 89 (+206.9%)
Mutual labels:  s3, airflow
Goodreads etl pipeline
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Stars: ✭ 793 (+2634.48%)
Mutual labels:  s3, airflow
jobAnalytics and search
JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.
Stars: ✭ 25 (-13.79%)
Mutual labels:  airflow, s3
Dataengineeringproject
Example end to end data engineering project.
Stars: ✭ 82 (+182.76%)
Mutual labels:  s3, airflow
astro
Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
Stars: ✭ 79 (+172.41%)
Mutual labels:  airflow, s3
Aws Toolkit Vscode
AWS Toolkit for Visual Studio Code, an extension for working with AWS services including AWS Lambda.
Stars: ✭ 823 (+2737.93%)
Mutual labels:  s3
S3 Deploy Website
Deploy website to S3/CloudFront from Python
Stars: ✭ 26 (-10.34%)
Mutual labels:  s3
Winscp
WinSCP is a popular free SFTP and FTP client for Windows, a powerful file manager that will improve your productivity. It supports also Amazon S3, FTPS, SCP and WebDAV protocols. Power users can automate WinSCP using .NET assembly.
Stars: ✭ 794 (+2637.93%)
Mutual labels:  s3
Pgbackrest
Reliable PostgreSQL Backup & Restore
Stars: ✭ 766 (+2541.38%)
Mutual labels:  s3
Docker Airflow
Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting. https://abhioncbr.github.io/docker-airflow/
Stars: ✭ 29 (+0%)
Mutual labels:  airflow
Awslib scala
An idiomatic Scala wrapper around the AWS Java SDK
Stars: ✭ 20 (-31.03%)
Mutual labels:  s3
S3bp
Read and write Python objects to S3, caching them on your hard drive to avoid unnecessary IO.
Stars: ✭ 24 (-17.24%)
Mutual labels:  s3
Phila Airflow
Stars: ✭ 16 (-44.83%)
Mutual labels:  airflow
Elyra
Elyra extends JupyterLab Notebooks with an AI centric approach.
Stars: ✭ 839 (+2793.1%)
Mutual labels:  airflow
Airflow Maintenance Dags
A series of DAGs/Workflows to help maintain the operation of Airflow
Stars: ✭ 914 (+3051.72%)
Mutual labels:  airflow
Hasura Backend Plus
🔑Auth and 📦Storage for Hasura. The quickest way to get Auth and Storage working for your next app based on Hasura.
Stars: ✭ 776 (+2575.86%)
Mutual labels:  s3
Github To S3 Lambda Deployer
⚓️ GitHub webhook extension for uploading static pages to AWS S3 directly after commiting to master via Lambda written in Node.js
Stars: ✭ 23 (-20.69%)
Mutual labels:  s3
Sherlock Holmes Partying In The Jungle
Parses AWS events payloads into a plain JavaScript object
Stars: ✭ 12 (-58.62%)
Mutual labels:  s3
Cluster Pack
A library on top of either pex or conda-pack to make your Python code easily available on a cluster
Stars: ✭ 23 (-20.69%)
Mutual labels:  s3
Automating Your Data Pipeline With Apache Airflow
Automating Your Data Pipeline with Apache Airflow
Stars: ✭ 19 (-34.48%)
Mutual labels:  airflow
View All Similar Projects ➔

ObjInSync

CI/CD

Daemon to continuously and incrementally synchronize a directory from remote object store to a local directory.

Usage

objinsync pull --exclude '**/__pycache__/**' s3://bucket/keyprefix ./localdir

When running in daemon mode (without --once flag), a health check endpoint is served at :8087/health and a prometheus metrics endponit is served at :8087/metrics. You can use --status-addr to override the binding address.

Objinsync also comes with builtin Sentry integration. To enable it, set the SENTRY_DSN environment variable.

You can also run objinsync in pull once mode, which behaves just like aws s3 sync:

objinsync pull --once s3://bucket/keyprefix ./localdir

To use with Minio instead of S3, you can set --s3-endpoint and --disable-ssl flags for pull command as you see fit.

Enable debug logs by setting the DEBUG environment variable DEBUG=1 objinsync pull ...

Installation

Simply download the prebuilt single binary from release page or use go get command:

go get github.com/scribd/objinsync

An officially maintained public docker image is available at https://hub.docker.com/repository/docker/scribdinc/objinsync.

Development

Run tests

make test

Run from source

AWS_REGION=us-east-2 go run main.go pull s3://qph-test-airflow-airflow-code/airflow_home/dags ./dags

To cut a release, push tag to remote in the format of vx.x.x.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].