permanent-seeder
Install
npm i -g @geut/permanent-seeder
Alternatively you can git clone this repo and build it from source:
# install deps
npm i
npm run bootstrap
# build the dashboard
cd packages/dashboard
cp .env.example .env
npm run build
# start the seeder
cd ../cli
./bin/run stop
./bin/run start
Usage
First, create a base config file, you can tweak it later ;-)
$ permanent-seeder config:init
Then, start the Permanent Seeder daemon
$ permanent-seeder start
Commands
$ permanent-seeder [COMMAND] [--OPTIONS]
Config
$ permanent-seeder config:[init|get]
init: creates the base config file for the Permanent Seeder. This is a.tomlfile that will live in~/permanent-seeder/settings.toml.get: returns the settings from the CLI. Useful when you are changing values and want to be sure they are pick up by the Permanent Seeder.
Default settings:
## Permanent seeder path (will be completed on config:init)
path = 'permanent-seeder'
## Enable stats recording
save_stats = true
## [optional] Preferred swarm port
# swarm_port =
## [optional] Api service settings
# [api]
# port = 3001
## [optional] HTTPS certificate and key
# [api.https]
# cert = "/path/to/cert.crt"
# key = "/path/to/cert.key"
## keys.endpoints = array of configs per endpoint
[[keys.endpoints]]
# Where to fetch keys
url = 'http://localhost:3000'
# Frequency between fetchs (in minutes)
frequency = 5
# Hook to parse response
hook = 'endpoint-hook.js'
## [optional] To add another endpoint, uncomment and complete next lines:
# [[keys.endpoints]]
# url =
# frequency =
# hookStart
$ permanent-seeder start
Bootstrap a Permanent Seeder instance that will keep up running in the background. If you change settings, you will need to call start command again.
Status
$ permanent-seeder status
It will return instance status. If it is running and some basic stats.
Stop
$ permanent-seeder stop
Stops the current instance.
Dashboard
$ permanent-seeder dashboard
Opens the dashboard app in a browser. If you want to manually access the dashboard, it can be found in: localhost:3001
http://localhost:3001. If you deploy the Permanent Seeder on a server and wants to access the dashboard from the outside, you may need to setup a reverse proxy.
Sample nginx reverse proxy config
upstream dashboard-nodejs {
server 127.0.0.1:3001;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
location / {
proxy_pass http://dashboard-nodejs;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location /socket.io/ {
proxy_pass http://dashboard-nodejs;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
Environment Variables
We use the .env.example file as a placeholder for some environment variables that we use internally, like PUBLIC_URL.
If you build the Permanent Seeder from source then you may want to customize some things.
There are two variables that can be used:
PUBLIC_URL: used to define the root path of the dashboard app. DEFAULT = '/'REACT_APP_API_VERSION: used to indicate the api version to use. DEFAULT = 'v1'
For development there are a few others variables that you can tweak, like the SOCKET_URL and API_URL. Checkout the full .env.example:
REACT_APP_API_VERSION=v1
PUBLIC_URL=/
## Extend eslint config
EXTEND_ESLINT=true
## Optional ENVs only for development
#REACT_APP_SOCKET_URL=
#REACT_APP_API_URL=Key Management 🔑
$ permanent-seeder key:[add|remove|remove-all]
add: Insert a new key Permanent Seeder db, it will start downloading and seeding ASAP.remove: Removes a single key from the seeder db and also stops seeding it (e.g.: no more announcing to other peers)remove-all: Removes and unnanounce all the keys in the db.
Logs
$ permanent-seeder logs --[live|all|error]
live: like doing atail -fof the logs.all: Show all the logs stored.error: Display only error logs.
repl
$ permanent-seeder repl
Useful to inspect the Permanent Seeder under the hood.
Keys Endpoint
The Permanent Seeder can fetch keys from an external endpoint, i.e: perform a GET request against a particular endpoint. This can be useful if you maintain a service that stores hyperdrive's keys. If that is the case, then the Permament Seeder can fetch those keys regularly. You can think of this like a cron job.
Whilst, we internally expect an Array<{url}>, you can customize and parse the fetch response the way you need it.
To do this, you will need to modify $HOME/permanent-seeder/endpoint-hook.js.
That hook will be called after fetch the response.
You can also tweak the fetch frequency (defined in minutes) and the endpoint url. These options can be found in the settings.toml file:
[[keys.endpoints]]
url = "http://localhost:3000"
frequency = 5
hook = "$HOME/permanent-seeder/endpoint-hook.js"Delete Keys Endpoint
The Permanent Seeder also allows for another endpoint, this one for deleted keys. With this action the Permanent Seeder can fetch an array of keys similarly to what we defined previously for the keys endpoint (adding).
The setting looks like this:
[[keys.remove]]
url = "http://localhost:60620/deletes.json"
hook = "/Users/deka/permanent-seeder/endpoint-hook.js"Again, we internally expect an Array<{url}>, but you can customize and parse the fetch response the way you need it.
To do this, you can tweak the hook response. By default the "delete keys endpoint" is using the same default hook used with the keys endpoint.
Finally, the fetch frequency is managed by the keys endpoint frequency. In other words, they run with the same frequency interval.
Update Policy
Here are some rules that we follow to know when to update the keys.
Actions that result in update:
- add endpoint is defined and returns a list of keys. delete endpoint is not defined.
✅ - add endpoint is defined and returns a list of keys. delete endpoint is defined and returns a list of keys.
✅
Actions that does not result in update:
- add endpoint is defined and returns an empty list. Does not matter if delete endpoint is defined or not.
⛔ - add endpoint is defined but throws an error
⛔ - add endpoint is defined and returns a list of keys. delete endpoint is defined but throws an error.
⛔
Design
The Permanent Seeder is a CLI tool that can starts a daemon which will seed hyperdrive's keys that you pass into it. Using the CLI you can add, remove keys, check the status and inspect logs.
It also contains a dashboard that you can use to have a visual reference of what is going on with your hyperdrives.
As you can see the project does a couple of things. To do this we decided to use a microservices based approach. We choose to use moleculer as the structural framework behind the Permanent Seeder. This enables multiples processes to communicate each other and at the same time each of these will have a single responsibility/scope. This also give us some room to scale things up if needed.
Routes
Below we define some internal API routes that can be used to obtain some information about the PS running instance as well as some basic drive queries.
GET /api/<API_VERSION>/drives/:key
Returns summary data associated to a specific drive key. Data may contain download information, peers info, seeding status and basic index.json information and version.
GET /api/<API_VERSION>/drives/:key/size
Returns only the download information.
GET /api/<API_VERSION>/drives/:key/peers
Returns only the peers information.
GET /api/<API_VERSION>/drives/:key/stats
Returns only information about the drive's files and its download status.
GET /api/<API_VERSION>/drives/:key/info
Returns only the index.json information part.
GET /api/<API_VERSION>/drives/:key/seedingStatus
Returns only the seeding status label.
GET /api/<API_VERSION>/drives/:key/files/:file
Download the file passed as argument from key drive.
GET /api/v1/stats/host
Returns information about the host. CPU load, memory and directory usage.
GET /api/v1/stats/network
Returns network information. Include things like online status, swarm information (hole-punching, bootstrapping info and network address), along with transferred bytes.
GET /api/<API_VERSION>/raw/:key
Returns the raw data associated with the seeded drive (key). The returned data may include various lifecycle events with their timestamps. This endpoint will return values as long as save_stats config property is enabled. See config.
Filter by Version
You can also query data using the drive version.
GET /api/<API_VERSION>/drives/:key/versions
Returns all the versions saved for key drive
GET /api/<API_VERSION>/drives/:key/:version/info
Returns only the info field for this specific key and version
GET /api/<API_VERSION>/drives/:key/:version/stats
Returns only the stats (files) field for this specific key and version
GET /api/<API_VERSION>/drives/:key/:version/files/:file
Download the file passed as argument from a specific drive's checkout (key+version).
Release
For releasing new versions we are using GitHub Actions
- First, we create a new release branch:
git co -b username/release-${newVersion} - Then, we need to create the new version. This new version will trigger a changelog update (using chan). To do this just run:
npm run releaseIt will prompt you what is the next version. - Create the release pull-request. The title should indicate this:
Release: newVersion⚠️ This is an important step, otherwise the publish workflow won't see the PR.
- Finally, when the release PR is approved, a custom action will be triggered publishing the new version to npm, creating a new tag for it and creating a brand new github release
🆕 📦
Issues
Contributing
Built in collaboration with Liberate Science
Acknowledgments
License
MIT © A GEUT project