All Projects → dreipol → django-collectfaster

dreipol / django-collectfaster

Licence: BSD-3-Clause License
Parallel file copying for Django's collectstatic.

Programming Languages

python
139335 projects - #7 most used programming language
Makefile
30231 projects

Projects that are alternatives of or similar to django-collectfaster

Fastflow
FastFlow pattern-based parallel programming framework (formerly on sourceforge)
Stars: ✭ 137 (+756.25%)
Mutual labels:  parallelization
paradiseo
An evolutionary computation framework to (automatically) build fast parallel stochastic optimization solvers
Stars: ✭ 73 (+356.25%)
Mutual labels:  parallelization
cre
common runtime environment for distributed programming languages
Stars: ✭ 20 (+25%)
Mutual labels:  parallelization
Future.apply
🚀 R package: future.apply - Apply Function to Elements in Parallel using Futures
Stars: ✭ 159 (+893.75%)
Mutual labels:  parallelization
Clusterrunner
ClusterRunner makes it easy to parallelize test suites across your infrastructure in the fastest and most efficient way possible.
Stars: ✭ 181 (+1031.25%)
Mutual labels:  parallelization
parallelizer
Simplifies the parallelization of function calls.
Stars: ✭ 62 (+287.5%)
Mutual labels:  parallelization
Cucable Plugin
Maven plugin that simplifies running Cucumber Scenarios in parallel.
Stars: ✭ 110 (+587.5%)
Mutual labels:  parallelization
cypress-circleci-reporter
Cypress test reporter for CircleCI
Stars: ✭ 37 (+131.25%)
Mutual labels:  parallelization
Mug
A small Java 8 util library, complementary to Guava (BiStream, Substring, MoreStreams, Parallelizer).
Stars: ✭ 236 (+1375%)
Mutual labels:  parallelization
open-ops
Open Optimizing Parallelizing System
Stars: ✭ 21 (+31.25%)
Mutual labels:  parallelization
Console Parallelization
Enables the parallelization of Symfony Console commands.
Stars: ✭ 166 (+937.5%)
Mutual labels:  parallelization
Alan
Autoscalable Programming Language
Stars: ✭ 181 (+1031.25%)
Mutual labels:  parallelization
aloisius
A Python library to create/update/delete AWS CloudFormation stacks in parallel
Stars: ✭ 16 (+0%)
Mutual labels:  parallelization
Parl
A high-performance distributed training framework for Reinforcement Learning
Stars: ✭ 2,348 (+14575%)
Mutual labels:  parallelization
selenified
The Selenified Test Framework provides mechanisms for simply testing applications at multiple tiers while easily integrating into DevOps build environments. Selenified provides traceable reporting for both web and API testing, wraps and extends Selenium calls to more appropriately handle testing errors, and supports testing over multiple browser…
Stars: ✭ 38 (+137.5%)
Mutual labels:  parallelization
Swifter
A package which efficiently applies any function to a pandas dataframe or series in the fastest available manner
Stars: ✭ 1,844 (+11425%)
Mutual labels:  parallelization
corebench
corebench - run your benchmarks against high performance computing servers with many CPU cores
Stars: ✭ 29 (+81.25%)
Mutual labels:  parallelization
TeeTime
The Java reference implementation of the Pipe-and-Filter framework TeeTime
Stars: ✭ 17 (+6.25%)
Mutual labels:  parallelization
future.callr
🚀 R package future.callr: A Future API for Parallel Processing using 'callr'
Stars: ✭ 52 (+225%)
Mutual labels:  parallelization
parallel
PARALLEL: Stata module for parallel computing
Stars: ✭ 97 (+506.25%)
Mutual labels:  parallelization

django-collectfaster

Latest Version Downloads

This package extends Django's collectstatic management command with a --faster argument that activates the parallel file copying. The speed improvement is especially helpful for remote storage backends like S3.

Quickstart

Install django-collectfaster:

pip install django-collectfaster

Configure installed apps in your settings.py and make sure collectfaster is listed before django.contrib.staticfiles:

INSTALLED_APPS = (
    ...,
    'collectfaster',
    'django.contrib.staticfiles',
    'storages',
    ...,
)

If you are using S3 with django-storages you probably already have this configured in your settings.py:

AWS_S3_HOST = 's3-eu-west-1.amazonaws.com'
AWS_STORAGE_BUCKET_NAME = '<your_aws_bucket_name>'

Set the storage backends for your static and media files in the settings.py:

STATICFILES_STORAGE = 'collectfaster.backends.boto.S3StaticStorage'
DEFAULT_FILE_STORAGE = 'collectfaster.backends.boto.S3MediaStorage'
# STATICFILES_STORAGE = 'collectfaster.backends.boto3.S3Boto3StaticStorage'
# DEFAULT_FILE_STORAGE = 'collectfaster.backends.boto3.S3Boto3MediaStorage'

You should split your static and media files on your S3 in different folders and configure it in the settings.py:

STATICFILES_LOCATION = 'static'
MEDIAFILES_LOCATION = 'media'

Set the STATIC_URL at least on your production settings:

STATIC_URL = 'https://%s/%s/%s/' % (AWS_S3_HOST, AWS_STORAGE_BUCKET_NAME, STATICFILES_LOCATION)

Usage

Collect your static files parallel:

python manage.py collectstatic --faster

Set the amount of workers to 30:

python manage.py collectstatic --faster --workers=30

Spawn workers using multiprocessing instead of gevent:

python manage.py collectstatic --faster --use-multiprocessing

Credits

Tools used in rendering this package:

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].