The confirmation ran before the dry-run check, so a dry run over SSH was
refused for having no terminal to confirm on -- and the refusal came out
instead of the report it was asked for. Nothing was going to be deployed
either way: it was asking whether to go ahead with something that was not
going to happen.
Print the report, stop there for a dry run, and gate only the real thing
on the confirmation. The hold list still refuses a held commit under
--dry-run, since that is an answer a dry run should give.
--help printed a fixed line range, which the header edit above would have
clipped. Print the leading comment block itself instead.
IHASMAIL_IMMUTABLE=1 runs the container the way the README's "Running
immutably" section describes: read-only root filesystem, no volume, sessions
held in memory. Until now that shape could be run by hand but not deployed --
the run line mounted the data volume unconditionally, so a redeploy would have
quietly put a mutable container back.
The switch is one variable and nothing else. IMMUTABLE=1 is passed to the
server too, which checks the claim rather than believing it, so a half-applied
switch refuses to start instead of looking fine until the next redeploy signs
everyone out. SESSION_FILE is cleared with -e rather than by editing the
environment file, because -e wins over --env-file; that keeps going back a
matter of changing the same one variable:
IHASMAIL_IMMUTABLE=0 ./ihasmail-deploy.sh --yes
which reproduces the previous run line exactly. The named volume is never
touched in either mode, so the sessions that were in it when the switch was
thrown are still there to come back to.
Every deploy tags an image with its version, which is what makes a
rollback a `docker run` rather than a rebuild -- and what has quietly
put 7.7 GB of them on the host, 4.3 GB of it reclaimable. `docker image
prune` will not help: they are tagged, which is the whole point of them.
The script that creates them now removes them, keeping the newest
IHASMAIL_KEEP_VERSIONS (three by default, 0 to keep everything).
Two things it is careful about, both learned from what could go wrong
rather than from something that did:
- it runs only after the new container reports healthy, so a rollback
target is never dropped while the thing meant to replace it is still
unproven.
- it excludes the image the container is actually running, by asking
docker what that is rather than assuming it sorts newest. After a
rollback it does not: the running image is an old one, and naive
keep-the-newest-N would delete the image in use. Docker would refuse,
but being refused is not the same as not having tried.
Exercised against a stubbed docker on PATH, so the pipeline and xargs
are the real ones: keeps 3 and removes the oldest, keeps 1 and removes
three, keeps everything at 0, never lists :current, and spares the
running image in a rollback where it is the fourth-newest.
Also records delete-all-spam as confirmed live on 0.16.19 today: Junk
Mail emptied and Deleted Items stayed empty afterwards, which is the
half a mock cannot prove and the half the feature exists for.
Moving the deploy script into the repo put it inside the checkout it
resets, and bash does not read a script all at once -- it reads as it
goes, by byte offset. `git reset --hard` replacing the file underneath a
running shell makes it stop wherever it had reached.
Silently, and with exit status 0. A three-line demonstration:
echo "line A"
cat > "$0" <<'NEW'
echo "REWRITTEN"
NEW
echo "line B"
prints "line A" and nothing else, and exits 0. In a deploy that means
building the image, then stopping before the container is replaced, and
reporting success -- so the old container keeps serving while everything
says the new one shipped.
It only bites when a deploy carries a change to the deploy script
itself, which is rare enough to be baffling when it happens and exactly
the sort of quiet failure this project keeps paying for.
The script now re-execs from a copy outside the tree before touching
git, so the file being run cannot change while it runs, and removes the
copy on exit. Running it from outside the checkout -- as the host did
before this moved into the repo -- skips all of that.
The trap is an `if` rather than `[ -n ... ] && trap`, which would leave
the not-re-exec'd path resting on errexit ignoring a failed left operand
of &&. It does ignore it, but a deploy script is a poor place to depend
on knowing that.
Verified: with the guard, a script that overwrites itself mid-run
completes every later line and cleans up its copy; without it, the lines
after the rewrite never run.
The live deploy script has never been under version control, which makes
it the one part of the pipeline that can silently fall out of step with
the repo -- as it just did: it builds without --build-arg
IHASMAIL_VERSION, so every deployment would report 2.16.0 no matter what
was actually built.
This is that script with the host taken out of it. Every path, name,
port and volume is a variable with a default that describes the shape of
a deployment rather than any particular one, so what is published is the
logic and none of the topology. It follows Caddyfile.example and
nginx.example.conf, which are here for the same reason.
Nothing sensitive was in the original either -- it never reads the
environment file, only hands the path to docker run --env-file -- but
the absolute paths named a user and a directory layout, and there is no
reason for those to be public to buy version control over the guards.
Two things it does that the original did not:
- passes the version to the build, which is the whole reason this came
up. A Docker tag may not contain "+", which a version for a commit
that arrived outside a pull request does (2.16.57+g1fa6578), so the
tag turns it into "-" while the build is told the real form. About
and /api/health still report it correctly.
- tags each build with its own version as well as :current, so rolling
back is running the previous tag rather than rebuilding it. The
failure path lists what is there to go back to.
Exercised against a throwaway clone, container, volume and image:
the hold guard refuses a held commit that production does not already
carry and lets one through that it does; the confirmation guard refuses
to run over a pipe without --yes; a dry run stops before building; and a
full run built, replaced the container and reported healthy at
2.16.57+g0549a09 -- from an image tagged 2.16.57-g0549a09, which is the
sanitising working.
jcoffey-dev