Alternatives and detailed information of dracut-crypt-sshd

dracut-crypt-sshd

Deprecation notice: there is a much improved and generalized version of this module, maintained at dracut-crypt-ssh/dracut-crypt-ssh. This project here is mostly for historical reasons.

Dracut initramfs module to start Dropbear sshd on early boot to enter encryption passphrase from across the internets or just connect and debug whatever stuff there.

Idea is to use the thing on remote VDS servers, where full-disk encryption is still desirable (if only to avoid data leaks when disks will be decomissioned and sold by VDS vendor) but rather problematic due to lack of KVM or whatever direct console access.

Authenticates users strictly by provided authorized_keys ("dropbear_acl" option) file.

See dropbear(8) manpage for full list of supported restrictions there (which are fairly similar to openssh).

Obligatory warning

Please think about your threat model first, and security/usability trade-off second.

This module is very unlikely to help at all against malicious hosting provider or whatever three-letter-agency that will coerce it into cooperation, should it take interest in your poor machine - they can just extract keys from RAM image (especially if it's a virtualized container), backdoor kernel/initramfs and force a reboot, or do whatever else to get encryption keys via hardware/backdoor.

It can help, as mentioned, against attacks on the data after you're done with it completely (i.e. shut the machine/container in question down for good), or against rather clumsy and incompetent "power off first and think second" attacks.

If this benefit is worth the hassle of this extra ssh'ing, some maintenance overhead and the possibility of loosing the LUKS key/header (and all access to data with it), only then (I think) this module might be useful to you.

Please also check out "Bad Things" section below before use.

Usage

First of all, it needs dropbear (at least its sshd, I tested only version built without pam support, both static and shared should work) and gcc installed (to build auth.c tool).

Copy or symlink 60dropbear-sshd into /usr/lib/dracut/modules.d/.
Add dracutmodules+="dropbear-sshd" to dracut.conf (will pull in "network" module as dependency).
Check out supported dracut.conf options below. With no extra options, ad-hoc server rsa key will be generated (and its fingerprint/bbcode will be printed to dracut log), /root/.ssh/authorized_keys will be used for ACL.
See dracut.cmdline(7) manpage for info on how to setup "network" module (otherwise sshd is kinda useless).

Simpliest way might be just passing ip=dhcp rd.neednet=1 on cmdline, if dhcp can assign predictable ip and pass proper routes.

Example of luks (uuid starts with "7a476ea0") + lvm (vg named "lvmcrypt", with lv there having fs with "root" label) + static-net (see manpage above for syntax) grub.cfg entry (wrapped for readability):
```
  menuentry "My Linux" {
    linux /vmlinuz ro root=LABEL=root
      rd.luks.uuid=7a476ea0 rd.lvm.vg=lvmcrypt rd.neednet=1
      ip=88.195.61.177::88.195.61.161:255.255.255.224:myhost:enp0s9:off
    initrd /dracut.xz
  }
```
Run dracut to build initramfs with the thing.

On boot, sshd will be started with:

Port: ${dropbear_port} (dracut.conf) or 2222 (default).
User (to allow login as-): root
Host key: ${dropbear_rsa_key} (dracut.conf) or generated (fingerprint echoed during generation and to console on sshd start). DSA keys are not supported (and shouldn't generally be used with ssh).
Client key(s): ${dropbear_acl} (dracut.conf) or /root/.ssh/authorized_keys
Password auth and port forwarding explicitly disabled.

Dropbear should echo a few info messages on start (unless rd.quiet or similar options are used) and print host ssh key fingerprint to console, as well as any logging (e.g. errors, if any) messages.

Do check the fingerprints either by writing them down on key generation, console or through network perspectives at least.

To login:

% ssh -p2222 [email protected]

Shell is /bin/sh, which should be dash in most dracut builds, but can probably be replaced with ash (busybox) or bash (heavy) using appropriate modules.

Once inside:

% console_peek   # to see what's on the console (e.g. which dev prompt is for)
...
% console_auth    # queries passphrase and sends it to console
Passphrase:
%

Boot should continue after last command, which should send entered passphrase to cryptsetup, waiting for it on the console, assuming its correctness.

sshd should be killed during dracut "cleanup" phase, once main os init is about to run. Connection won't be closed, but nothing should work there, as initramfs gets destroyed.

dracut.conf parameters

dropbear_port
dropbear_rsa_key
dropbear_acl

See above.

Common issues and non-issues

Dropbear sshd failed to start

Only means what it says, see output of dropbear before it died - it should print some specific errors which led to it exiting like that.

Failed reading '-', disabling DSS

Will always be printed and should be ignored - DSA keys are not generated/used in these scripts, and probably shouldn't be.

Host hangs in initramfs, but can't be pinged (e.g. ping my.host.tld) from outside.

Either no network configuration parameters were passed to dracut, or it failed to configure at least one IP address.

Don't forget rd.neednet=1 on cmdline, as dracut will ignore specified network settings without nfs (or whatever net-) root otherwise.

Read up dracut.cmdline(7), "Network" section and/or see why/if dracut failed to configure net as requested with rd.debug. See also "Debugging tips" section below.

Host pings, but ssh can't connect.

Try nc -v <host> <port>, or "ncat" instead of "nc" there. "ncat" can be found in "nmap" package, "nc" usually comes pre-installed.

If it hangs without printing "Connected to ..." line - can be some firewall before host or dropbear failed to start/listen.

If there's no "SSH-2.0-dropbear_..." after "Connected to ..." line - some issue with dropbear.

lastlog_perform_login: Couldn't stat /var/log/lastlog: No such file or directory

Pops up when logging in, can be safely ignored.

Debugging tips

If (or rather "when") something goes wrong and you can't access just-booted machine over network and can't get to console (hence sshd in initramfs), don't panic - it's fixable if machine can be rebooted into some rescue system remotely.

Usually it's some dhcp+tftp netboot thing from co-located machine (good idea to setup/test in advance) plus whoever is there occasionally pushing the power button, or maybe some fancy hw/interface for that (e.g. hetzner "rescue" interface).

To see what was going on during initramfs, open "modules.d/99base/rdsosreport.sh" in dracut, append this (to the end):

set -x
netstat -lnp
netstat -np
netstat -s
netstat -i
ip addr
ip ro
set +x

exec >/dev/null 2>&1
mkdir /tmp/myboot
mount /dev/sda2 /tmp/myboot
cp /run/initramfs/rdsosreport.txt /tmp/myboot/
umount /tmp/myboot
rmdir /tmp/myboot

Be sure to replace /dev/sda2 with whatever device is used for /boot, rebuild dracut and add rd.debug to cmdline (e.g. in grub.cfg's "linux" line).

Upon next reboot, wait for at least a minute, since dracut should give up on trying to boot the system first, then it will store full log of all the stuff modules run ("set -x") and their output in "/boot/rdsosreport.txt".

Naturally, to access that, +1 reboot into some "rescue" system might be needed.

In case of network-related issues - e.g. if "rdsosreport.txt" file gets created with "rd.debug", but host can't be pinged/connected-to for whatever reason - either enable "debug" dracut module or add dracut_install netstat ip line to install() section of "modules.d/60dropbear-sshd/module-setup.sh" and check "rdsosreport.txt" or console output for whatever netstat + ip commands above (for "rdsosreport.sh") show - there can be no default route, whatever interface naming mixup, no traffic (e.g. unrelated connection issue), etc.

Bad Things

Has quite a few rather unnecessary assumptions about environment baked-in (e.g. /lib/libnss_files.so.2 path), and does gcc -std=gnu99 -O2 -Wall "$moddir"/auth.c -o "${tmp_file}" for that "console_auth" binary on dracut run.

Better way to handle these and compile whatever binaries would be allowing for proper ./configure && make && make install process, which I don't know enough about to support.

Luckily, there are people that do, check out these links for better integrated versions of a similar thing:
Only tested with customized source-based distro (Exherbo), no idea how easy it is to use with generic debian or ubuntu.
check() in module_setup.sh should probably not be empty no-op.
Should probably have set -e or something alike (dracut-specific?) in install().
No idea how to sanely run ssh-keygen (openssh) from a script, maybe use openssl instead?

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

mk-fg / dracut-crypt-sshd

Programming Languages

Labels

Projects that are alternatives of or similar to dracut-crypt-sshd