Post on the procedure for setting up ns4.

Also, ignore the posts from detecting secrets.

.pre-commit-config.yaml

@@ -13,7 +13,7 @@ repos:
     rev: v0.14.3
     hooks:
       - id: detect-secrets
-        exclude: Pipfile\.lock
+        exclude: Pipfile\.lock|\.rst$
 
   - repo: https://github.com/adrienverge/yamllint
     rev: v1.25.0

content/setting-up-ns4.rst

+Setting up my ns4 instance
+##########################
+:date: 2022-02-26
+:summary: Setting up ns4.shore.co.il on online.net
+
+This is one of the posts that is more of me writing what I did so I can do it
+again later and isn't geared toward the general public, you've been warned. A
+little bit of background. The instance in question is my bare-metal Dedibox
+instance. I use it for the following:
+
+#. `Secondary DNS instance
+   <https://git.shore.co.il/shore/secondary-dns-docker>`_
+#. `Docker registry <https://git.shore.co.il/shore/registry-docker>`_
+#. Some static web sites.
+#. `GitLab CI runner <https://git.shore.co.il/shore/gitlab-runner-docker>`_
+#. Remote `Workbench <https://git.shore.co.il/shore/workbench>`_ instance.
+
+A bare-metal instance is very performant, allows me to run KVM and VirtualBox
+VMs and I feel a little better on the security front. Although I can due a
+remote KVM install of Debian, so far I've been using the online installation.
+This makes me wonder how stock is the installation and what I'll encounter if I
+do an in-place upgrade of Debian to a new release. The only data I care about
+are the images in the registry and they are backed up weekly to a machine in my
+house that I have off-site backups for. ``/var`` and ``/home`` are subvolumes
+in a btrfs volume in a LUKS encrypted partition. I enter the encryption key
+using KVM over IP. I have some private information but nothing I can't just git
+clone back to existence. I acknowledge that it's not as secure as can be but
+it's good enough for me and I judge the tradeoffs to be valid.
+
+Everything listed above is deployed using GitLab CI pipelines but there's a
+chicken and egg situation here that some of the services depend on each other
+and at the very least I can't deploy the GitLab runner using GitLab pipelines.
+So here's what I did.
+
+OS installation
+---------------
+
+OS installation takes ~15 minutes but afterwards KVM over IP is disabled for
+another hour. I work around this limitation by opening the KVM session first
+and then go ahead with the OS installation. In the online.net console, install
+Debian using the wizard. It's pretty standard stuff but the partitioning
+default to a RAID-1 setup which I don't want. Instead have ``/boot/efi`` at 512
+MB, ``/boot`` at 2048 MB and ``/`` using the rest on the first drive (the order
+is important as changes don't boot for some reason I'm not interested enough to
+find). Because I must have a partition on each drive, keep ``/data`` on the
+second using all of the space.
+
+Bootstrapping
+-------------
+
+Remove the existing SSH host key:
+
+.. code:: shell
+
+   ssh-keygen -R ns4.shore.co.il
+   ssh-keygen -R "$(dig ns4.shore.co.il)"
+
+Connect using KVM as root and setup sudo.
+
+.. code:: shell
+
+   apt update
+   apt install -y sudo
+   echo 'nimrod ALL=(ALL) NOPASSWD: ALL' | tee /etc/sudoers.d/nimrod
+
+Now SSH as normal and check sudo (``sudo whoami``). Now remove the passwords and
+the SSH public key from the root account.
+
+.. code:: shell
+
+   sudo passwd --delete nimrod
+   sudo passwd --delete root
+   sudo rm -r /root/.ssh
+
+Setting up the encrypted partition.
+
+.. code:: shell
+
+   sudo umount /data
+   sudo rm --dir /data
+   sudo sed -i '#/data#d' /etc/fstab
+   sudo apt install -y btrfs-progs cryptsetup parted
+   sudo parted /dev/sdb --script mklabel gpt mkpart primary 0% 100%
+   sudo cryptsetup luksFormat /dev/sdb1
+   sudo cryptsetup luksOpen /dev/sdb1 data
+   sudo mkfs -t btrfs /dev/mapper/data
+   sudo mount /dev/mapper/data /mnt
+   sudo btrfs subvolume create /mnt/builds
+   sudo btrfs subvolume create /mnt/home
+   sudo btrfs subvolume create /mnt/var
+   sudo mkdir /builds
+   UUID="$(lsblk --output UUID --list --noheadings --nodeps /dev/sdb1)"
+   echo "data UUID=$UUID none luks,discard" | sudo tee -a /etc/crypttab
+   echo "/dev/mapper/data /builds btrfs defaults,compress=zstd,discard,relatime,subvol=builds" | sudo tee -a /etc/fstab
+   echo "/dev/mapper/data /home btrfs defaults,compress=zstd,discard,relatime,subvol=home" | sudo tee -a /etc/fstab
+   echo "/dev/mapper/data /var btrfs defaults,compress=zstd,discard,relatime,subvol=var" | sudo tee -a /etc/fstab
+   sudo cp --archive /home/* /mnt/home/
+   sudo cp --archive /var/* /mnt/var/
+   sudo umount /mnt
+   sudo cryptsetup luksClose data
+   sudo systemc reboot --now
+
+The instance will now reboot, enter the encryption key in the KVM console. From
+the ``homelab`` repository:
+
+.. code:: shell
+
+
+   cd ./Ansible
+   ansible-playbook bootstrap.yaml -l ns4
+   ansible-playbook debian_server.yaml -l ns4
+
+Services
+--------
+
+From the ``secondary-dns-docker`` repository:
+
+.. code:: shell
+
+   eval $(docker remote ns4.shore.co.il)
+   docker-compose build
+   docker-compose pull
+   docker-compose up -d
+
+Same as above, from the ``web-proxy-docker`` repository, branch ``ns4``. Some
+secret environment variables are set in the GitLab pipeline, so for them to
+exist re-run the deploy job in GitLab afterwards.
+
+From the ``homelab`` repository:
+
+.. code:: shell
+
+   cd ./Ansible
+   ansible-playbook renew-certs.yaml -t ns4
+
+Setup the Docker registry, but the ``cron`` container uses an image from the
+registry. Start just the registry container and afterwards run the GitLab
+deploy job. From the ``registry-docker`` repository:
+
+.. code:: shell
+
+   eval $(docker remote ns4.shore.co.il)
+   docker-compose pull registry
+   docker-compose up -d registry
+   docker build -t registry.shore.co.il/registry-backup ./backup
+
+Copying the registry backup to ns4 (it's going to take a long while).
+
+.. code:: shell
+
+   scp -3Cr host01.shore.co.il:/var/backups/registry ns4.shore.co.il:/home/nimrod/
+
+On ns4:
+
+.. code:: shell
+
+   docker run --rm -u nobody -v /home/nimrod/registry:/registry:ro registry.shore.co.il/registry-backup restore /registry registry.shore.co.il
+   rm -r /home/nimrod/registry
+
+
+From the ``gitlab-runner-docker`` repository (delete the old runner in the
+`GitLab admin area <https://git.shore.co.il/admin/runners>`_):
+
+.. code:: shell
+
+   eval $(docker remote ns4.shore.co.il)
+   docker-compose pull
+   ./deploy ns4
+
+From here on out it's just running GitLab jobs and pipelines for the static web
+sites. etc. **Remember to run the deploy jobs for web-proxy-docker and
+registry-docker**.
+
+Remote workbench
+----------------
+
+Follow the instructions in the `rcfiles repository
+<https://git.shore.co.il/nimrod/rcfiles/>`_ and finish it off with ``wb -u``.
+
+Closing thoughts
+----------------
+
+Along with my previous `post on my OpenBSD machine
+<https://www.shore.co.il/blog/openbsd-on-sg-2440/>`_ covers the more involved
+parts of `my infrastructure
+<https://www.shore.co.il/blog/homelab-early-2021/>`_. Overall, I think that I'm
+in a pretty good state here. My setup is cost conscious, relies very little on
+external services and is more pets than cattle. The coverage of infrastructure
+code is high and the parts that aren't covered are done rarely (like
+reinstalling OpenBSD which I do about once a year or reinstalling Debian which
+I do about every 2 years) and are either trivial or nicely documented.
+
+Going forward, I'm in the process of merging the different \*-docker
+repositories in to the `homelab repository
+<https://git.shore.co.il/shore/homelab>`_. Once done, I may copy the blog posts
+there and maintain them ongoing as changes are done.