darklang / Dark
Programming Languages
Dark
This is the main repo for Dark, a combined language, editor, and infrastructure to make it easy to build backends.
This repo is intended to help Dark users solve their needs by fixing bugs, expanding features, or otherwise contributing. Dark is source available, not open source.
Contributing
We are committed to make Dark easy to contribute to. Our contributor docs will help guide you through your first PR, find good projects to contribute to, and learn about the code base.
Getting Started
Install dependencies
To build and run the server you must have the following installed (and running):
- Mac only: Homebrew for Mac (https://brew.sh/)
- Mac only: Docker for Mac (https://docs.docker.com/docker-for-mac/install/)
- Mac only: The latest version of bash
brew install bash - fswatch
brew install fswatch/apt install fswatch - PIP
brew install python/apt install python3-pip - live reload
pip3 install livereload
On Windows, you can run Dark in WSL2 (Windows Subsystem for Linux):
- You must be on at least Windows 10 Version 2004, and you must run WSL 2 (docker does not work in WSL 1)
- Follow the WSL 2 installation instructions
- Follow the Docker for WSL 2 installation instructions
- Install the depedencies above in your Linux distro
Docker
The app and its dependencies are all held within the container. While code is edited on your machine, the application is compiled and run inside of the container.
Ensure that docker:
- set to use 4 CPUs, 4.0 GiB of Memory, and 4.0 GiB of Swap (under the Advanced preferences tab).
Ignore the other tabs (for example you don't need to enable Kubernetes).
Dnsmasq
A local DNS server is needed to access the application via a .localhost TLD. The following is a quick start, adapted from this guide.
Install dnsmasq:
brew install dnsmasq / apt install dnsmasq
Follow brew's post-install instructions:
brew info dnsmasq
(probably sudo brew services start dnsmasq)
Add the following to (brew --prefix)/etc/dnsmasq.conf
address=/localhost/127.0.0.1
Restart dnsmasq:
sudo brew services restart dnsmasq / sudo /etc/init.d/dnsmasq restart
Configure OSX to use dnsmasq (not needed on linux):
sudo mkdir -p /etc/resolver
sudo tee /etc/resolver/localhost >/dev/null <<EOF
nameserver 127.0.0.1
EOF
Test it:
# Make sure you haven't broken your DNS.
ping -c 1 www.google.com
# Check that .localhost names work
dig [email protected]
Building and running for the first time
- Run
scripts/builder --compile --watch --test - Wait until the terminal says "Finished initial compile" - this means the
build server is ready. The
builderscript will sit open, waiting for file changes in order to recompile - If you see "initial compile failed", it may be a memory issue. Ensure you have docker configured to provide 4GB or more of memory, then rerun the builder script. (Sometimes just rerunning will work, too).
- Open your browser to http://darklang.localhost:8000/a/dark/, username "dark", password "what"
- Edit code normally - on each save to your filesystem, the app will be rebuilt and the browser will reload as necessary
Read the contributor docs
If you've gotten this far, you're now ready to contribute your first PR.
Testing
Unit tests run when you specify --test to scripts/builder. You can run them as a once off using:
-
scripts/runtests# client scripts/run-backend-testsscripts/run-rust-tests containers/strollerscripts/run-rust-tests containers/queue-scheduler
Integration tests:
scripts/run-in-docker ./integration-tests/run.sh
You can also run integration tests on your (host) machine, which gives you some debugging ability, and typically runs faster:
./integration-tests/run.sh
There are good debugging options for integration testing. See integration-tests/README.
Running unix commands in the container
scripts/run-in-docker bash
Accessing the local db
scripts/run-in-docker psql -d devdb
Config files
Config files are in config/. Simple rule: anything that runs inside the container must use a DARK_CONFIG value set in config/, and cannot use any other env var.
Debugging the client
You can enable the FluidDebugger by mousing over the Gear in the left-sidebar. There is also "Enable debugger" which enables a legacy debugger that nobody uses and doesn't work well.
If you're using Chrome, enable Custom Formatters to see ReScript values in Chrome Dev Tools instead of their JS representation. From within Chrome Dev Tools, click "⠇", "Settings", "Preferences", "Enable Custom Formatters".
Debugging dotnet
REPL (fsi)
You can get a REPL with all of the Dark libraries loaded by running:
scripts/dotnet-fsi
Segfaults and crashes
When dotnet crashes, you can debug it by running:
- `lldb -- [your command]
In LLDB, you can use dotnet's SOS plugin to read the stack, values, etc. See https://docs.microsoft.com/en-us/dotnet/framework/tools/sos-dll-sos-debugging-extension for instructions. The plugin is automatically loaded in lldb in the dev container.
Production Services
The app is split into backend/ and client/. Part of the backend is used in
the client (jsanalysis), and one directory is shared (libshared). These are
compiled to create libraries and binaries.
These are put into containers, whose definitions are in containers/. We also
have some containers which are defined entirely in their directory (typically,
these have a self-contained codebase).
The containers are used in services/. A service is typically a number of
yaml files defining a kubernetes deployment, made up of one or more
containers, which use binaries from the backend.
Some services do not use Dark's containers (eg, when we deploy 3rdparty code, such as "let's encrypt"). Some just have a single container (eg queue-scheduler and postgres-honeytail).