Setting up my ns4 instance ########################## :date: 2022-02-26 :summary: Setting up ns4.shore.co.il on online.net This is one of the posts that is more of me writing what I did so I can do it again later and isn't geared toward the general public, you've been warned. A little bit of background. The instance in question is my bare-metal Dedibox instance. I use it for the following: #. `Secondary DNS instance `_ #. `Docker registry `_ #. Some static web sites. #. `GitLab CI runner `_ #. Remote `Workbench `_ instance. A bare-metal instance is very performant, allows me to run KVM and VirtualBox VMs and I feel a little better on the security front. Although I can due a remote KVM install of Debian, so far I've been using the online installation. This makes me wonder how stock is the installation and what I'll encounter if I do an in-place upgrade of Debian to a new release. The only data I care about are the images in the registry and they are backed up weekly to a machine in my house that I have off-site backups for. ``/var`` and ``/home`` are subvolumes in a btrfs volume in a LUKS encrypted partition. I enter the encryption key using KVM over IP. I have some private information but nothing I can't just git clone back to existence. I acknowledge that it's not as secure as can be but it's good enough for me and I judge the tradeoffs to be valid. Everything listed above is deployed using GitLab CI pipelines but there's a chicken and egg situation here that some of the services depend on each other and at the very least I can't deploy the GitLab runner using GitLab pipelines. So here's what I did. OS installation --------------- OS installation takes ~15 minutes but afterwards KVM over IP is disabled for another hour. I work around this limitation by opening the KVM session first and then go ahead with the OS installation. In the online.net console, install Debian using the wizard. It's pretty standard stuff but the partitioning default to a RAID-1 setup which I don't want. Instead have ``/boot/efi`` at 512 MB, ``/boot`` at 2048 MB and ``/`` using the rest on the first drive (the order is important as changes don't boot for some reason I'm not interested enough to find). Because I must have a partition on each drive, keep ``/data`` on the second using all of the space. Bootstrapping ------------- Remove the existing SSH host key: .. code:: shell ssh-keygen -R ns4.shore.co.il ssh-keygen -R "$(dig ns4.shore.co.il)" Connect using KVM as root and setup sudo. .. code:: shell apt update apt install -y sudo echo 'nimrod ALL=(ALL) NOPASSWD: ALL' | tee /etc/sudoers.d/nimrod Now SSH as normal and check sudo (``sudo whoami``). Now remove the passwords and the SSH public key from the root account. .. code:: shell sudo passwd --delete nimrod sudo passwd --delete root sudo rm -r /root/.ssh Setting up the encrypted partition. .. code:: shell sudo umount /data sudo rm --dir /data sudo sed -i '#/data#d' /etc/fstab sudo apt install -y btrfs-progs cryptsetup parted sudo parted /dev/sdb --script mklabel gpt mkpart primary 0% 100% sudo cryptsetup luksFormat /dev/sdb1 sudo cryptsetup luksOpen /dev/sdb1 data sudo mkfs -t btrfs /dev/mapper/data sudo mount /dev/mapper/data /mnt sudo btrfs subvolume create /mnt/builds sudo btrfs subvolume create /mnt/home sudo btrfs subvolume create /mnt/var sudo mkdir /builds UUID="$(lsblk --output UUID --list --noheadings --nodeps /dev/sdb1)" echo "data UUID=$UUID none luks,discard" | sudo tee -a /etc/crypttab echo "/dev/mapper/data /builds btrfs defaults,compress=zstd,discard,relatime,subvol=builds" | sudo tee -a /etc/fstab echo "/dev/mapper/data /home btrfs defaults,compress=zstd,discard,relatime,subvol=home" | sudo tee -a /etc/fstab echo "/dev/mapper/data /var btrfs defaults,compress=zstd,discard,relatime,subvol=var" | sudo tee -a /etc/fstab sudo cp --archive /home/* /mnt/home/ sudo cp --archive /var/* /mnt/var/ sudo umount /mnt sudo cryptsetup luksClose data sudo systemc reboot --now The instance will now reboot, enter the encryption key in the KVM console. From the ``homelab`` repository: .. code:: shell cd ./Ansible ansible-playbook bootstrap.yaml -l ns4 ansible-playbook debian_server.yaml -l ns4 Services -------- From the ``secondary-dns-docker`` repository: .. code:: shell eval $(docker remote ns4.shore.co.il) docker-compose build docker-compose pull docker-compose up -d Same as above, from the ``web-proxy-docker`` repository, branch ``ns4``. Some secret environment variables are set in the GitLab pipeline, so for them to exist re-run the deploy job in GitLab afterwards. From the ``homelab`` repository: .. code:: shell cd ./Ansible ansible-playbook renew-certs.yaml -t ns4 Setup the Docker registry, but the ``cron`` container uses an image from the registry. Start just the registry container and afterwards run the GitLab deploy job. From the ``registry-docker`` repository: .. code:: shell eval $(docker remote ns4.shore.co.il) docker-compose pull registry docker-compose up -d registry docker build -t registry.shore.co.il/registry-backup ./backup Copying the registry backup to ns4 (it's going to take a long while). .. code:: shell scp -3Cr host01.shore.co.il:/var/backups/registry ns4.shore.co.il:/home/nimrod/ On ns4: .. code:: shell docker run --rm -u nobody -v /home/nimrod/registry:/registry:ro registry.shore.co.il/registry-backup restore /registry registry.shore.co.il rm -r /home/nimrod/registry From the ``gitlab-runner-docker`` repository (delete the old runner in the `GitLab admin area `_): .. code:: shell eval $(docker remote ns4.shore.co.il) docker-compose pull ./deploy ns4 From here on out it's just running GitLab jobs and pipelines for the static web sites. etc. **Remember to run the deploy jobs for web-proxy-docker and registry-docker**. Remote workbench ---------------- Follow the instructions in the `rcfiles repository `_ and finish it off with ``wb -u``. Closing thoughts ---------------- Along with my previous `post on my OpenBSD machine `_ covers the more involved parts of `my infrastructure `_. Overall, I think that I'm in a pretty good state here. My setup is cost conscious, relies very little on external services and is more pets than cattle. The coverage of infrastructure code is high and the parts that aren't covered are done rarely (like reinstalling OpenBSD which I do about once a year or reinstalling Debian which I do about every 2 years) and are either trivial or nicely documented. Going forward, I'm in the process of merging the different \*-docker repositories in to the `homelab repository `_. Once done, I may copy the blog posts there and maintain them ongoing as changes are done.