From 1bb7fac616e1d7e1bc7ed68313c926baf4f3e7dc Mon Sep 17 00:00:00 2001 From: Adar Nimrod <nimrod@shore.co.il> Date: Wed, 6 Dec 2017 22:30:33 +0200 Subject: [PATCH] New post on bundling a binary blob in a shell script. --- content/shell_binary_bundle.rst | 85 +++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 content/shell_binary_bundle.rst diff --git a/content/shell_binary_bundle.rst b/content/shell_binary_bundle.rst new file mode 100644 index 0000000..e7cdb51 --- /dev/null +++ b/content/shell_binary_bundle.rst @@ -0,0 +1,85 @@ +Bundling a binary file into a shell script +########################################## + +:date: 2017-12-06 +:summary: Bundling a binary file into a shell script + +When creating an auto-scaling group in EC2 I often try to package the deployment +script into the user data. Installing some packaged software is easy to do but +bundling configuration files that are needed is less straightforward. +If the files are not confidential in any way, I either clone a Git repository +or download a tarball from our static assets domain. But this leads to a +dependency on external services and a slightly more complex deployment +procedure. A few days ago I was faced with the same options again but it didn't +sit right with me to do all this for a couple of files that are a few K's in +size totally. I remembered that some software have installation scripts that +bundle the binary blob inside the script. + +First version +------------- + +I searched and found an article in the `Linux Journal +<http://www.linuxjournal.com/content/add-binary-payload-your-shell-scripts>`_ +that seemed to show what I wanted to (and seems to be copied everywhere). You +could download a single file that was a shell script with the binary blob +inside. Your usage will be close to this + +.. code:: shell + + wget http://hostname.tld/bundle + sh bundle + +or this + +.. code:: shell + + wget http://hostname.tld/bundle + chmod +x bundle + ./bundle + +Which is fine. However the code was a bit longer than it should have been and +I felt it could be done better. A little more research and I found an answer in +`Stack Overflow <https://stackoverflow.com/a/10491738>`_ that mentioned +:code:`uuencode` and :code:`uudecode`. Reading the man page I saw it was closer +to what I wanted. The code I wrote is available on my `cgit instance +<https://www.shore.co.il/git/bundle/about/?id=first_implementation>`_. + +The implementation works as follows. The bundle has the script at the start of +the file with the encoded binary at the end. The shell executes the script part +(which ends with exit as to not continue any further, causing errors) and +:code:`uudecode` only starts processing after it sees the relevant header. The +script feeds itself to :code:`uudecode` (:code:`uudecode "$0"`) which decodes +the binary and outputs it to disk which the script can then use. The code has +both the build instruction in the :code:`Makefile` and usage example in the +:code:`bats` tests. + +Second version +-------------- + +However something kept nagging me. I wanted a simple invocation method like so: + +.. code:: shell + + curl http://hostname.tld/bundle | sh + +And in the case of the user data in EC2, I could simply use the bundle. +Otherwise I would need to host it somewhere and in the user data I would +download and run the bundle. Which means that if the bundle was unavailable the +instance would fail to provision. + +Everything I found assumed that the file was present in the file system for +:code:`uudecode` to decode. If it was piped there was no file that +:code:`uudecode` could then decode. I kept mauling over it and a came up with +a short, clean solution to this problem, which is available `here +<https://www.shore.co.il/git/bundle/about/?id=second_implementation>`_, again +with build instruction and test examples. + +This time I used AWK to replace a single line in the script with the file, +encoded using :code:`uuencode` but this time in base64 (to keep the script valid +without any characters with special meanings). That is piped to :code:`uudecode` +which decodes and saves it to disk. The script can then continue with the +binary blob present. + +This method is less space efficient and the build procedure is less obvious. But +the ability to use resulting script as the user data (or piping the output from +:code:`curl` to :code:`sh`) is worth it in my opinion. -- GitLab