diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 14a94c45964889971a498d0356a6658070be65c2..e154303f284888863bcfd9acbde1febbfb1f55d5 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -13,7 +13,7 @@ repos: rev: v0.14.3 hooks: - id: detect-secrets - exclude: Pipfile\.lock + exclude: Pipfile\.lock|\.rst$ - repo: https://github.com/adrienverge/yamllint rev: v1.25.0 diff --git a/content/setting-up-ns4.rst b/content/setting-up-ns4.rst new file mode 100644 index 0000000000000000000000000000000000000000..8fc68a0d4437bdde68681acd653f3fd417cbca7a --- /dev/null +++ b/content/setting-up-ns4.rst @@ -0,0 +1,196 @@ +Setting up my ns4 instance +########################## +:date: 2022-02-26 +:summary: Setting up ns4.shore.co.il on online.net + +This is one of the posts that is more of me writing what I did so I can do it +again later and isn't geared toward the general public, you've been warned. A +little bit of background. The instance in question is my bare-metal Dedibox +instance. I use it for the following: + +#. `Secondary DNS instance + <https://git.shore.co.il/shore/secondary-dns-docker>`_ +#. `Docker registry <https://git.shore.co.il/shore/registry-docker>`_ +#. Some static web sites. +#. `GitLab CI runner <https://git.shore.co.il/shore/gitlab-runner-docker>`_ +#. Remote `Workbench <https://git.shore.co.il/shore/workbench>`_ instance. + +A bare-metal instance is very performant, allows me to run KVM and VirtualBox +VMs and I feel a little better on the security front. Although I can due a +remote KVM install of Debian, so far I've been using the online installation. +This makes me wonder how stock is the installation and what I'll encounter if I +do an in-place upgrade of Debian to a new release. The only data I care about +are the images in the registry and they are backed up weekly to a machine in my +house that I have off-site backups for. ``/var`` and ``/home`` are subvolumes +in a btrfs volume in a LUKS encrypted partition. I enter the encryption key +using KVM over IP. I have some private information but nothing I can't just git +clone back to existence. I acknowledge that it's not as secure as can be but +it's good enough for me and I judge the tradeoffs to be valid. + +Everything listed above is deployed using GitLab CI pipelines but there's a +chicken and egg situation here that some of the services depend on each other +and at the very least I can't deploy the GitLab runner using GitLab pipelines. +So here's what I did. + +OS installation +--------------- + +OS installation takes ~15 minutes but afterwards KVM over IP is disabled for +another hour. I work around this limitation by opening the KVM session first +and then go ahead with the OS installation. In the online.net console, install +Debian using the wizard. It's pretty standard stuff but the partitioning +default to a RAID-1 setup which I don't want. Instead have ``/boot/efi`` at 512 +MB, ``/boot`` at 2048 MB and ``/`` using the rest on the first drive (the order +is important as changes don't boot for some reason I'm not interested enough to +find). Because I must have a partition on each drive, keep ``/data`` on the +second using all of the space. + +Bootstrapping +------------- + +Remove the existing SSH host key: + +.. code:: shell + + ssh-keygen -R ns4.shore.co.il + ssh-keygen -R "$(dig ns4.shore.co.il)" + +Connect using KVM as root and setup sudo. + +.. code:: shell + + apt update + apt install -y sudo + echo 'nimrod ALL=(ALL) NOPASSWD: ALL' | tee /etc/sudoers.d/nimrod + +Now SSH as normal and check sudo (``sudo whoami``). Now remove the passwords and +the SSH public key from the root account. + +.. code:: shell + + sudo passwd --delete nimrod + sudo passwd --delete root + sudo rm -r /root/.ssh + +Setting up the encrypted partition. + +.. code:: shell + + sudo umount /data + sudo rm --dir /data + sudo sed -i '#/data#d' /etc/fstab + sudo apt install -y btrfs-progs cryptsetup parted + sudo parted /dev/sdb --script mklabel gpt mkpart primary 0% 100% + sudo cryptsetup luksFormat /dev/sdb1 + sudo cryptsetup luksOpen /dev/sdb1 data + sudo mkfs -t btrfs /dev/mapper/data + sudo mount /dev/mapper/data /mnt + sudo btrfs subvolume create /mnt/builds + sudo btrfs subvolume create /mnt/home + sudo btrfs subvolume create /mnt/var + sudo mkdir /builds + UUID="$(lsblk --output UUID --list --noheadings --nodeps /dev/sdb1)" + echo "data UUID=$UUID none luks,discard" | sudo tee -a /etc/crypttab + echo "/dev/mapper/data /builds btrfs defaults,compress=zstd,discard,relatime,subvol=builds" | sudo tee -a /etc/fstab + echo "/dev/mapper/data /home btrfs defaults,compress=zstd,discard,relatime,subvol=home" | sudo tee -a /etc/fstab + echo "/dev/mapper/data /var btrfs defaults,compress=zstd,discard,relatime,subvol=var" | sudo tee -a /etc/fstab + sudo cp --archive /home/* /mnt/home/ + sudo cp --archive /var/* /mnt/var/ + sudo umount /mnt + sudo cryptsetup luksClose data + sudo systemc reboot --now + +The instance will now reboot, enter the encryption key in the KVM console. From +the ``homelab`` repository: + +.. code:: shell + + + cd ./Ansible + ansible-playbook bootstrap.yaml -l ns4 + ansible-playbook debian_server.yaml -l ns4 + +Services +-------- + +From the ``secondary-dns-docker`` repository: + +.. code:: shell + + eval $(docker remote ns4.shore.co.il) + docker-compose build + docker-compose pull + docker-compose up -d + +Same as above, from the ``web-proxy-docker`` repository, branch ``ns4``. Some +secret environment variables are set in the GitLab pipeline, so for them to +exist re-run the deploy job in GitLab afterwards. + +From the ``homelab`` repository: + +.. code:: shell + + cd ./Ansible + ansible-playbook renew-certs.yaml -t ns4 + +Setup the Docker registry, but the ``cron`` container uses an image from the +registry. Start just the registry container and afterwards run the GitLab +deploy job. From the ``registry-docker`` repository: + +.. code:: shell + + eval $(docker remote ns4.shore.co.il) + docker-compose pull registry + docker-compose up -d registry + docker build -t registry.shore.co.il/registry-backup ./backup + +Copying the registry backup to ns4 (it's going to take a long while). + +.. code:: shell + + scp -3Cr host01.shore.co.il:/var/backups/registry ns4.shore.co.il:/home/nimrod/ + +On ns4: + +.. code:: shell + + docker run --rm -u nobody -v /home/nimrod/registry:/registry:ro registry.shore.co.il/registry-backup restore /registry registry.shore.co.il + rm -r /home/nimrod/registry + + +From the ``gitlab-runner-docker`` repository (delete the old runner in the +`GitLab admin area <https://git.shore.co.il/admin/runners>`_): + +.. code:: shell + + eval $(docker remote ns4.shore.co.il) + docker-compose pull + ./deploy ns4 + +From here on out it's just running GitLab jobs and pipelines for the static web +sites. etc. **Remember to run the deploy jobs for web-proxy-docker and +registry-docker**. + +Remote workbench +---------------- + +Follow the instructions in the `rcfiles repository +<https://git.shore.co.il/nimrod/rcfiles/>`_ and finish it off with ``wb -u``. + +Closing thoughts +---------------- + +Along with my previous `post on my OpenBSD machine +<https://www.shore.co.il/blog/openbsd-on-sg-2440/>`_ covers the more involved +parts of `my infrastructure +<https://www.shore.co.il/blog/homelab-early-2021/>`_. Overall, I think that I'm +in a pretty good state here. My setup is cost conscious, relies very little on +external services and is more pets than cattle. The coverage of infrastructure +code is high and the parts that aren't covered are done rarely (like +reinstalling OpenBSD which I do about once a year or reinstalling Debian which +I do about every 2 years) and are either trivial or nicely documented. + +Going forward, I'm in the process of merging the different \*-docker +repositories in to the `homelab repository +<https://git.shore.co.il/shore/homelab>`_. Once done, I may copy the blog posts +there and maintain them ongoing as changes are done.