👩💻 Conventions
Our Priceloop Engineering conventions for Scala, Python, Git Workflow etc.
📖 Contents
- Scala Styling Guide
- Python Styling Guide
- Naming Conventions
- Git Commit Structure
- Git Branch Flow
- Secrets
- Language
- License
Scala Styling Guide
We use Scalafmt to format our code. Use the following config in your projects:
version = 3.3.1
runner.dialect = scala213source3
style = defaultWithAlign
maxColumn = 120
includeCurlyBraceInSelectChains = false
danglingParentheses.preset = true
lineEndings = preserve
trailingCommas = always
continuationIndent {
callSite = 2
defnSite = 2
}
newlines {
penalizeSingleSelectMultiArgList = false
alwaysBeforeElseAfterCurlyIf = true
}
project {
git = true
excludeFilters = [
target/
]
}
binPack {
parentConstructors = true
}
rewrite {
rules = [RedundantParens, SortImports, PreferCurlyFors]
}
align {
preset = most
arrowEnumeratorGenerator = true
tokens."+" = [":=", "+=", "++="]
openParenCallSite = false
}
Python Styling Guide
Use the following pre-commit hooks:
- isort
- pylint following Google styling guide
- black formater with the following flags:
-l 110 -S -t
- VSCode settings.json
"python.formatting.blackArgs": ["-l", "110", "-S", "-t"]
-
flake8
-
docstrings: Google conventions
-
Quotes:
double
-
mypy
-
Use pre-commit with the following hooks:
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.2.3
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: debug-statements
- id: requirements-txt-fixer
- id: flake8
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v0.720
hooks:
- id: mypy
args: [--allow-redefinition, --ignore-missing-imports]
- repo: local
hooks:
- id: isort
name: "Sort imports"
language: system
types: [file, python]
entry: isort
- id: pylint
name: "PyLint"
language: system
types: [file, python]
files: .
exclude: test_*.py
entry: python3 -m pylint
- id: black
name: "Black"
language: system
pass_filenames: false
entry: black .
args: [--safe, --quiet, "--line-length=110", "--skip-string-normalization"]
Naming Conventions
Git Branch Naming
Your branch name should follow the format type-scope.
Types:
- feat
- fix
- chore
Storage buckets, i.e. s3, gcs
ai.priceloop.{{bucket_purpose}}.{{environment}}
e.g. ai.priceloop.data.prod
{{environment}} = {prod, stage}
{{bucket_purpose}}
can potentially be broken down to
{{layer}}.{{bucket_purpose}}
, e.g.
ai.priceloop.ml.metadata
ai.priceloop.frontend.assets
ai.priceloop.dataplatform.metadata
Namespace
-
Long: priceloop-production / priceloop-staging
-
Short: priceloop-prod / priceloop-stage
Git Commit Structure
- In general, follow the Conventional Commit guidelines
- Be descriptive
- Use all lower-case
- Limit punctuations (no dots, no commas)
- Short (under 70 characters is best)
Git Branch Flow
We follow the following PRs flow with user facing feature(s):
develop → { git checkout -b } → feature branch → {PR} develop → {PR} master
CI/CD
We use Github Actions as the CI solution. Every repository with the codebase for user facing features must contain the following CI configuration/workflow definition files:
.github
└── workflows
├── develop.yaml
├── master.yaml
└── pr_branches_check.yaml
Where
develop.yaml
- defines the CI pipeline to deploy to the stage envmaster.yaml
- defines the CI pipeline to deploy to the prod envpr_branches_check.yaml
- defines the check to validate if PR is compliant with the branch flow
pr_branches_check.yaml
name: PR Branches Check
on:
pull_request:
types:
- opened
- reopened
jobs:
develop_to_master:
if: github.base_ref == 'master'
runs-on: ubuntu-latest
steps:
- name: Check if PR to "master" is from "develop"
if: ${{ github.head_ref != 'develop' }}
run: |
echo "Wrong source branch, only develop->master PR's allowed"
exit 1
any_to_develop:
if: github.base_ref == 'develop'
runs-on: ubuntu-latest
steps:
- name: Check if PR is to "develop"
if: ${{ github.head_ref == 'refs/heads/master' }}
run: |
echo "Wrong source branch, only "!master"->develop PR's allowed"
exit 1
GitHub Branches Protection Rules
Every Github repository containing user facing features must have the protection rules for the master
and develop
branches.
Secrets
- Metadata considered as secrets:
- AWS keys (
AWS_ACCESS_KEY_ID
,AWS_SECRET_ACCESS_KEY
) - Access tokens (any oauth2 tokens which have long life time)
- Access credentials, i.e. combination of password and user name
- Webhook URLs
- AWS keys (
- Metadata which can be stored to git:
- Bucket name, path to object
- Database name
- Resource ID, URI ← it may be a subject of change/dispute
Note: in case of doubt whether the info is a secret/internal or can be public, treat it at a higher security level, i.e. as a secret. It costs nothing to downgrade security level, but it's quite expensive to upgrade it
How to share secrets
To share between peers:
- Please use gpg to encrypt opaque text secret.
- Slack DM is acceptable, but remember to delete the secret message after it was received!
We use AWS Secret Manager and github secrets for secrets to be used programmatically.
Language
- Stick to American English e.g. optimize instead of optimise or behavior instead of behaviour. In doubt check Google
😃
License
- For private repos, no License necessary
- For public repos, we should use Apache 2.0 License