Skip to content

Upgrade Shipwick ​

You upgrade a Shipwick server by running the installer again, on the server or from your laptop over SSH, and a CLI elsewhere with shipwick upgrade. This page covers what the installer changes, what it leaves alone, what an upgrade does from each release and what it costs, how to choose a version and how to go back, what happens to your applications and to a running deployment while the agent restarts, how the server says that a newer release exists, what the upgrades to 1.1, 1.0, 0.10, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4 and 0.3 do on first start and what you may want to turn on afterwards, and how to upgrade the CLI on laptops and in CI.

Read the changelog first ​

From 1.0 on, a minor release adds and takes nothing away: a deploy.yaml that worked keeps working, the API keeps every path and field it had, and the database changes only by a migration the agent applies when it starts, which the changelog names under Changed. Read the entry before an upgrade all the same: it says what is new, whether a migration stands in the way of going back, and what you may want to turn on afterwards. What a version number promises, in full, is in Versions and compatibility.

A notice when a newer release exists ​

Since 0.7 the agent asks GitHub once a day which release is the latest, and GET /server carries the answer. The dashboard and shipwick server status say when the server is behind, without a connection of their own:

text
Agent           v0.8.1  v1.1.0 is available  (on the server, run the installer again: curl -fsSL https://get.shipwick.com | sh)

In the dashboard the notice is on the server's Status tab, with the command that upgrades; see Use the dashboard.

  • The question says nothing about the server. It is one request to github.com/shipwick/shipwick/releases/latest, without a version or an identifier in it. See What leaves the server unasked.
  • It goes through the proxy and the certificate authorities the agent was given. A server that cannot reach GitHub notices nothing: the request gives up after 15 seconds, nothing is logged above debug level, and the next attempt is a day later like any other.
  • A restart asks nothing. The last answer is kept in update-check.json in the data directory.
  • SHIPWICK_UPDATE_CHECK=off stops the question. The agent says at startup, once, that it asks and how to stop it. See Agent configuration.
  • Pre-releases are never announced, and nothing is installed: the notice names the command, and running it is yours. An agent that runs a pre-release ahead of the latest release, or a development build, announces nothing either.

The notice names the installer. On a server installed from a package, the newer package is what to install.

Upgrade the server ​

On the server, as root:

bash
curl -fsSL https://get.shipwick.com | sh
text
✓ Docker 29.8.0 with Compose 5.5.1
  The release's signature was not checked: cosign is not installed. Every file is checked against the release's checksums.
✓ Installed /opt/shipwick/compose.yml
✓ Keeping the existing /opt/shipwick/.env (your API token is unchanged)
✓ Started the Shipwick services
✓ The agent is healthy
✓ Removed 2 image(s) of earlier Shipwick releases
✓ Installed the shipwick CLI to /usr/local/bin/shipwick
✓ shipwick on this server is signed in

Shipwick is running.

  Your API token is unchanged: SHIPWICK_AGENT_TOKEN in /opt/shipwick/.env
  From your laptop or CI:   shipwick login --url https://agent.example.com
  On this server:           shipwick ps
  Dashboard:                https://dashboard.example.com
  Open ports 80 and 443 (and 443/udp) — and nothing else — in your firewall.
  DNS records must point straight at this server (at Cloudflare: DNS only). To keep
  Cloudflare's proxy on, add SHIPWICK_CLOUDFLARE_API_TOKEN to /opt/shipwick/.env
  and run:   cd /opt/shipwick && docker compose up -d
  Upgrade later by running this installer again.

A server does not upgrade by itself. The compose file of each release pins the Shipwick images — the agent, the dashboard and, since 0.5, the proxy — to that release's version, and since 1.1 the socket proxy's image to a version of its own, so a server runs the version it installed until you run the installer again.

From your laptop, the same over SSH:

bash
shipwick server install [email protected]

It runs the installer on the server; the token is unchanged and not printed again, and the saved context keeps the one it has. Since 0.9.1 the hostnames the server was installed with are read from its .env, so they need not be given again. (The 0.9.0 CLI, run this way without them, started the agent without its hostnames and moved the context to the tunnel address: cd /opt/shipwick && docker compose up -d on the server and shipwick login put both back.) --version v0.4.1 picks a release. A server with no connection is upgraded from a bundle of the newer release: see Install on a server with no way out. See Install on a server.

What the installer does on an upgrade:

  1. Downloads the new release's compose file and verifies it against the release's checksums.txt — and, since 0.8.0, checksums.txt against the release's signature first, where cosign (2.5 or later) is installed; see Verify a release. Only a verified file replaces /opt/shipwick/compose.yml. A checksum mismatch installs nothing and leaves the running installation as it was. Since 0.8 the installer also asks Compose to read the new file before it replaces the one in use: a Compose plugin older than 2.23.1 cannot, and the installer stops there with the version it needs.
  2. Keeps /opt/shipwick/.env exactly as it is. The root token does not change, and it is not printed again. The hostnames do not change, and the installer does not ask for them. Anything else you put there — SHIPWICK_WEBHOOK_URL, a mount for images built from source — stays too.
  3. Pulls the images that the new compose file names, and recreates the containers whose definition changed.
  4. Waits for the agent to report healthy.
  5. Removes the agent, dashboard and proxy images of earlier releases, up to a few hundred megabytes each. The images that are running, and anything else on the server, are kept. This step exists since 0.5; see Upgrading from 0.4 to 0.5.
  6. Replaces shipwick on the server with the release's version, verified the same way.
  7. With a hostname for the API, puts the token from .env back into the CLI context that points at this server, and leaves every other context, and which one is current, as they were. See The CLI on the server.

compose.yml is the installer's, compose.override.yml is yours

The installer writes a fresh /opt/shipwick/compose.yml on every run. Keep your own changes — publishing the API on loopback, mounting docker login credentials — in /opt/shipwick/compose.override.yml. Compose merges the two files, and the installer never touches the override or .env.

What an upgrade does ​

From every release since 0.1.0 the upgrade to this version is one step, the installer, and needs nothing before or after it. Each of those releases was installed the way it was released, given applications with its own CLI — a volume with data, secrets, tokens, jobs and backups as far as it had them — and upgraded. What happened on every one of them:

  • Shipwick's containers are replaced — three, and since 1.1 the socket proxy's as a fourth; the applications' are not. The replicas after the upgrade are the containers the old release started, and they are served without a deployment.
  • Requests go unanswered while the proxy's container is replaced. Connections are refused for two to three seconds: with about ten requests a second to an application of two replicas, each on a connection of its own, between 9 and 21 requests failed per upgrade and none before or after (23 upgrades, at least one from each release, in a Docker daemon inside a container on a developer's machine). The agent is away a few seconds longer; nothing is supervised or deployed in that time, and nothing stops.
  • The agent brings its database up to date when it starts. The history, the tokens, the secrets, the scheduled jobs and the backups of the old release are there afterwards, and a rollback goes back to a deployment the old release made. Volumes are Docker's and are not touched.
  • .env and compose.override.yml stay as they are; compose.yml is replaced. Images named in .env (SHIPWICK_AGENT_IMAGE and the others) stay too: an upgrade does not replace an image of your own.
  • The old release's CLI keeps working against the new agent for what it knew: ps, deploy, redeploy, rollback, its token. The commands and deploy.yaml keys that came later need shipwick upgrade.

What differs by where the upgrade starts:

FromWhat the upgrade does besidesWhat you do
1.0The agent gives up the Docker socket. A fourth container, the socket proxy, holds it and answers the agent the calls on its list; the agent reaches Docker through a socket the two share (The Docker socket). No migrationNothing. A server that had compose.socket-proxy.yml as its compose.override.yml keeps running, and the installer says to take the service out of the override: its list would hold a newer agent to an older one. See Upgrading from 1.0 to 1.1
0.9, 0.10The same. 1.0 added no migration and renamed nothing; it is the release from which Versions and compatibility holdsNothing
0.8The agent's database gains the scope of secrets (migration 22); every existing secret becomes the server'sNothing. To be able to go back to 0.8, copy the agent's data first; see Go back to the release before
0.7And the proxy's image gains Shipwick's own way of finding replicas, and the agent loads a configuration that uses itNothing
0.5, 0.6The sameNothing
0.3, 0.4And the proxy changes from caddy:2-alpine to Shipwick's build of CaddyNothing
0.2And the agent creates encryption.key next to shipwick.db and encrypts the env values of earlier deployments, onceFrom then on, back encryption.key up with shipwick.db: without it the database cannot be read
0.1And the proxy joins a second network, shipwick-services; the replicas that run are given their application's name on it, without a restartThe same. agent, caddy, dashboard and localhost are no longer names an application can be deployed under

The walk from every release was made for 0.8; the rows for the releases since are from their changelogs, and the server with the old proxy overlay was walked for 1.1. Not walked this way: the packages, a bundle, and an installation with a compose file of its own. For those, see If you installed without the installer.

Choose a version ​

By default the installer installs the latest release. Pre-releases, such as release candidates, are never picked by default. To install a specific version:

bash
curl -fsSL https://get.shipwick.com | SHIPWICK_VERSION=v0.4.1 sh

Everything comes from that one release — the compose file, the images and the CLI — so the three always belong together.

Go back to the release before ​

Going back is the installer with that version:

bash
curl -fsSL https://get.shipwick.com | SHIPWICK_VERSION=v0.7.0 sh

Two things can stand in the way, and since 0.8 the installer deals with both.

The proxy's saved configuration. The proxy starts from the configuration it saved last, and one saved by a newer release may name a part the older proxy does not have; the older proxy then does not start. The installer removes that file, and the agent loads the routes again within seconds. Going back costs the same seconds without the proxy as going up. This is what going back from 0.8 to 0.7 takes, and all it takes: 0.8 adds nothing to the agent's database. Going back from 0.9 to 0.8 meets the second thing as well: 0.9 adds to the database.

text
! The proxy's saved configuration was written by a newer release and was removed; the agent loads the routes again.

Without the installer — the packages, or a compose file of your own — remove the file yourself before the older proxy starts, or it will not start:

bash
docker compose run --rm --no-deps --entrypoint rm caddy -f /config/caddy/autosave.json

A database a newer release has added to. An agent refuses such a database: it does not guess at what it does not know. Whether a release adds to the database is in its changelog, under Changed, and in the sections below. When it has, the installer finds the older agent refusing, starts the newer release again, and says so:

text
✗ The agent of this release cannot read the database: the release that ran here has added to it.
  That release was started again; the proxy was replaced twice, and requests went unanswered for some seconds.
  Going back takes the agent's data as it was before the upgrade: docs/handbook.md, "Upgrading".

Measured the same way as an upgrade, twice: 43 and 50 requests failed, within five and seven seconds. Before 0.8 the installer left the older agent restarting forever with nothing to load the routes, and every hostname went unanswered until someone installed the newer release again.

To be able to go back across such a release, copy the agent's data before you upgrade. The agent is stopped for the copy, the applications are not:

bash
cd /opt/shipwick
docker compose stop agent
docker run --rm -v shipwick_agent-data:/data -v /root:/to busybox \
  tar -czf /to/shipwick-before-upgrade.tar.gz -C /data shipwick.db encryption.key
docker compose start agent

Put it back before the installer of the older release runs:

bash
cd /opt/shipwick
docker compose stop agent
docker run --rm -v shipwick_agent-data:/data -v /root:/from:ro busybox sh -c '
  rm -f /data/shipwick.db-wal /data/shipwick.db-shm &&
  tar -xzf /from/shipwick-before-upgrade.tar.gz -C /data'
curl -fsSL https://get.shipwick.com | SHIPWICK_VERSION=v0.7.0 sh

The older agent then knows what the copy knows.

  • An application deployed since is brought back to the deployment the copy has as its last: the containers of the later one are removed and those of the earlier one started.
  • What was stored since — secrets, tokens, history — is not in the copy.
  • The volumes are as the newer release left them.

What happens to running applications ​

Running applications do not depend on the agent being up. Restarting or upgrading the agent does not restart any application container, and applications keep running while the agent is down.

While the agent is down:

  • Nothing supervises the applications. A replica that crashes during that time is not restarted until the agent is back.
  • The API does not answer, so shipwick and the dashboard cannot show or do anything. A shipwick deploy that is waiting on a deployment reports that the agent is not responding, and goes on when it is back.

When the new agent starts, before it serves requests, it reconciles what it finds:

  1. A deployment that was in flight when the old agent stopped is resumed, since 0.5. The agent picks it up where it was: containers it had created are kept, replicas that were already serving keep serving, and only what was not done is done. The deployment's events say Resumed after the agent restarted. The exception is a pre_deploy command that was running when the agent stopped: it is not run a second time, the deployment fails and says so, nothing of the running version has been touched, and you deploy again once you have looked at what the command left behind. An agent before 0.5 marked every such deployment FAILED. See Architecture.
  2. Containers that belong to a known application but neither to its active deployment nor to one that resumes are removed.
  3. Containers of applications the agent's database does not know are never touched.

Supervision then resumes. The supervisor's state is held in memory, so every replica starts with a clean slate: its backoff history is gone, and its health is unknown until it has been probed again. shipwick status shows such a replica as checking. Unknown counts as healthy, so an application does not flap to DOWN because the agent was restarted. Only the restart counter is kept, for display.

A job that was running when the old agent stopped is marked interrupted and its container removed; the job runs again at its next scheduled time. Firings that fell while the agent was down are not caught up.

After a server reboot, the agent brings every application back up according to its restart policy. What an agent that is killed in the middle of each kind of deployment does is in When things break.

Upgrading from 1.0 to 1.1 ​

1.1 changes how the agent reaches Docker, and nothing you have to relearn: no migration, nothing renamed, no key of deploy.yaml, field of the API or command of the CLI taken away. The agent applies no schema migration: the database of 1.0 is the database of 1.1.

The agent gives up the Docker socket. The release's compose file runs a fourth container, socket-proxy, which holds /var/run/docker.sock and answers the agent the calls on its list — a method and a path each — with 403 for every other; the agent reaches Docker through a filtered socket the two share, and has no other way to it. The installer replaces compose.yml, as it always has, so the proxy arrives with the upgrade and there is nothing to do; its image, wollomatic/socket-proxy, is pulled from Docker Hub and held against the release's digests like Shipwick's own. The proxy narrows what a fault in the agent can reach; it does not stop whoever holds an admin token from creating a privileged container, so an admin token is still root on the server. What it takes away and what it leaves: The Docker socket.

If you had put the proxy in by hand, with the compose.socket-proxy.yml of 0.8 to 1.0 saved as /opt/shipwick/compose.override.yml, the server keeps running — Compose merges the two definitions — and the installer says so:

text
! /opt/shipwick/compose.override.yml defines the socket-proxy service as well. The socket proxy is part of every installation now,
  and the override's list of allowed calls replaces this release's. Remove the service and the docker-proxy volume
  from the override, keep any mounts of your own, and run:   cd /opt/shipwick && docker compose up -d

Do that: the override's list of calls replaces the release's, and an agent that makes one call more is refused it by the older list, which shows as Forbidden in a failed operation and as a blocked request line in docker logs shipwick-socket-proxy-1. Mounts of your own — a Docker config.json, a CA file — stay in the agent's volumes in the override, as before. An override that holds compose.direct-socket.yml instead, the overlay that gives the agent the socket itself, is left alone, and the choice holds.

The agent says how it reaches Docker: docker.socket in GET /server is "proxy" or "direct", shipwick server status ends its Docker line with behind the socket proxy or holding the Docker socket, and shipwick doctor has a hint for the second. An agent installed from a package keeps the socket itself, as a process of the host, and says so; see Install from a package.

What an operator may want to set:

How
Another image of the socket proxySHIPWICK_SOCKET_PROXY_IMAGE in .env, which the installer keeps across upgrades and shipwick server configure writes. See Agent configuration
See every request the proxy lets throughSHIPWICK_SOCKET_PROXY_LOG_LEVEL=DEBUG in .env
The agent on the socket itselfcompose.direct-socket.yml as compose.override.yml. See The socket itself

On laptops and in CI, upgrade the CLI as described below. A 1.0 shipwick keeps working against a 1.1 agent; the Docker line of server status and the hint of doctor need 1.1, and server bundle carries the socket proxy's image only from 1.1 on.

Upgrading from 0.10 to 1.0 ​

1.0 changes nothing you have to relearn: no migration, nothing renamed, no key of deploy.yaml, field of the API or command of the CLI added or taken away. All three containers are recreated, because their images changed, and requests go unanswered while the proxy's container is replaced, as on every upgrade. The agent applies no schema migration: the database of 0.10 is the database of 1.0, and going back is the installer with SHIPWICK_VERSION=v0.10.0, with nothing to copy first.

One thing behaves differently without being asked: the look through the proxy after a deployment accepts the proxy's answer only under a certificate that names the hostname asked for. The chain behind it is still not verified, since on a development machine the issuer is Caddy's own authority and nothing is sent over the connection. See A redirect loop is caught.

Two things are fixed without a change in behavior. The proxy image is built with golang.org/x/net 0.60.0, which closes five HTTP/2 server vulnerabilities of 0.59.0 (GO-2026-6603, -6610, -6611, -6612 and -6617); Caddy 2.11.6 itself still names 0.59.0, and the module Shipwick adds to it is what raises the version. The dashboard's lockfile takes the updates its dependency ranges already allowed and drops simple-git, which closes the npm audit findings that did not need another Nuxt; node-forge under Nuxt's development server remains, since only a Nuxt downgrade would remove it.

What 1.0 adds is a promise. From here on deploy.yaml, the API and the files on the server change only with a migration and a documented upgrade; a minor release removes and renames nothing, and what is to go is marked deprecated for a minor first; a 1.x CLI works against any 1.x agent. The whole of it is in Versions and compatibility.

On laptops and in CI, upgrade the CLI as described below. A 0.10 shipwick keeps working against a 1.0 agent; shipwick server status says that the two differ, and which to upgrade.

Upgrading from 0.9 to 0.10 ​

0.10 changes names and closes gaps; applications run as before, and nothing changes for a deploy.yaml that worked before, with two exceptions named below. All three containers are recreated, because their images changed, and requests go unanswered while the proxy's container is replaced, as on every upgrade. The agent applies no schema migration: the database of 0.9 is the database of 0.10.

Several things behave differently without being asked:

  • Five fields of the API are renamed, so that the same thing has the same name everywhere: step → step_seconds of the metrics history, a traffic point's t → at, pull.last_export → pull.last_export_id of GET /standby, and deployment → sequence on a log archive entry and a searched line. A client of your own that read the old name reads nothing; a 0.9 shipwick does not fail, it shows what it shows for nothing: shipwick standby prints export #0 imported …, shipwick logs --list names an entry replica 2 instead of #3 replica 2. A list in an answer is [] when there is nothing in it, never absent: aliases, redirects, unlimited_memory, proxy.networks_missing, unenforced_limits. See the API reference.
  • A body over its limit is 413 everywhere — a redeploy or rollback body over 4 KB, a secret, registry, certificate or sign-in body over its own — where it was 400; the message names the limit.
  • logging.options.splunk-token is a secret: masked as ******** wherever a configuration is handed out, encrypted at rest, writable as ${NAME}. A deployment recorded by 0.9 still holds the token in clear until the application is deployed again: run shipwick deploy or shipwick redeploy once for each application that ships logs to Splunk, and rotate the token if it may have been read.
  • An image under shipwick.local must be the application's own, in deploy.yaml and in a redeploy: a deploy token limited to one application could run another application's built image with this application's environment. The refusal names the application the image belongs to.
  • A basic-auth username with braces is refused, since the proxy read it as a placeholder, and every username reaches the proxy escaped like a header value. The build block refuses a key it does not know, as every other block does; dockerfil: x used to be dropped without a word.
  • The API's and the dashboard's hostnames send Strict-Transport-Security, and the CLI follows no redirect from the agent's URL. See Security.
  • The installer compares the images it pulled with the release's signed digests and resolves latest once; see Install on a server. Behind the socket proxy the agent makes one more call, POST /images/<id>/tag: take the compose.socket-proxy.yml of this version, or an upload the agent refuses leaves that call blocked. The authorization policy refuses a volume with driver options inline on a container's mount, and MaskedPaths/ReadonlyPaths lists; take the docker-authz.rego of this version too. See Give the agent less than the Docker socket.
  • The agent spends half the processor time it did on watching applications; the figures are in Health checks and supervision.

Three flags are renamed in the CLI: shipwick traffic --requests follows with --follow only (-f was --follow here and --file everywhere else but logs), shipwick audit --lines is --tail (-n is unchanged), and shipwick server install --context is --name, since the global --context was shadowed by it. shipwick backups rm, shipwick registry logout and shipwick access revoke ask before they act; --yes skips the question, and a script needs it. shipwick server status says when the CLI and the agent are of different minor versions, and which to upgrade. shipwick server configure and the installer accept SHIPWICK_BACKUP_DIR.

On laptops and in CI, upgrade the CLI as described below. A 0.9 shipwick keeps working against a 0.10 agent for everything it knows, with the three places above where it reads a renamed field as nothing.

Upgrading from 0.8 to 0.9 ​

0.9 adds to what the agent stores and to what it says; applications run as before, and nothing changes for a deploy.yaml that worked before. All three containers are recreated, because their images changed. Requests go unanswered while the proxy's container is replaced, as on every upgrade; certificates and configuration are on volumes and stay, and the applications' containers are not touched.

On its first start the new agent applies one schema migration to shipwick.db, number 22: a secret belongs to one application or to the server, and the table is remade with that as its key. Every existing secret becomes the server's, with its ciphertext untouched, and every application resolves it as before. An older agent refuses a database whose schema is newer than it supports, so to be able to go back, copy the agent's data before you upgrade: see Go back to the release before.

Several things behave differently without being asked:

  • The application is asked its own hostname once after a deployment, through the proxy, and the answer is recorded as a probe event of the deployment, after completed_at. A redirect to a hostname of redirects is a warning under the summary of shipwick deploy and in shipwick status, with what to swap; any other answer is recorded and not judged. See A redirect loop is caught.
  • The proxy's networks are checked every half minute, and a proxy that is not on all three is said in the agent's log, in GET /server, in shipwick server status and in shipwick doctor, with the command that makes the container again.
  • A followed log carries a heartbeat every 20 seconds while the application prints nothing, and a last line that says why the agent ended the stream. shipwick logs -f takes a lost stream up where it left off. A client of your own that reads the stream skips both lines by their field; the dashboard already does. See Following logs.
  • The Cloudflare token is checked when the agent starts, by creating and removing a TXT record in every zone it sees; the result is in the log, in GET /server and in shipwick doctor. See Put Cloudflare in front of the server.
  • A certificate that cannot be obtained says why, in shipwick status, in the application's events, in GET /applications/:name and in shipwick doctor, read out of the proxy's log. See Certificate status.
  • An export names the keys it uses. An export written by 0.9 or later is refused by an agent that does not know one of them, before anything is taken in. Every export written before 0.9 is still read. See Move to a new server.
  • The supervisor asks Docker for its containers once per pass, whatever the number of applications, and hostnames are looked up eight at a time; the server's idle CPU falls accordingly. See Health checks and supervision.
  • shipwick doctor says more, and some of it differently: a port that times out is told from one that refuses, and the provider's firewall is named for the timeout; an agent that does not answer is told as a timeout, a refusal or a failed TLS handshake; a redirect loop, a proxy missing a network, a server near the kernel's table of neighbours, the Cloudflare token and the certificates' failures get lines. Behind Cloudflare's proxy the line reads Behind Cloudflare only: ports and record targets are not checked from outside.
  • Cosign 2.5 or later is what reads the release's signature; 2.4 does not, and the texts that said 2.4 say 2.5.

What is new is opt-in. What an operator may want to turn on or use:

How
Give an application a secret of its ownshipwick secret set NAME --app my-api. See Keep secrets on the server
Change a setting of the agent from your machineshipwick server configure [email protected] KEY=value. See Change a setting later
Install the CLI on Windows with one lineirm https://get.shipwick.com/windows | iex. See Install the CLI
Try a release candidate, and go backshipwick upgrade --version v1.1.0-rc.1. See Install the CLI
Refuse privileged containers to every client of the Docker daemonDocker's authorization plugin with configs/docker-authz.rego. See Behind an authorization plugin
Admit Cloudflare's addresses and nothing elseSee Admit Cloudflare only

On laptops and in CI, upgrade the CLI as described below; secret set --app, server configure, upgrade --version, the probe's line under a deployment's summary, the reconnecting logs -f and the new lines of doctor need it. A 0.8 shipwick keeps working against a 0.9 agent for everything it knows; its logs -f prints an empty line for each heartbeat. A 0.9 shipwick against a 0.8 agent says so where the agent lacks something: secret set --app stores nothing and says that the agent keeps no secrets per application; secret rm --app lists first and sends nothing when no application has a secret of its own; logs -f that lost its stream reconnects and says which minutes may be missing, since the older agent cannot begin at a time; deploy does not wait for a probe the agent does not make.

Upgrading from 0.7 to 0.8 ​

0.8 changes how the proxy finds replicas and who can reach the API; applications run as before, and nothing changes for a deploy.yaml that worked before. All three containers are recreated, because their images changed. Requests go unanswered for the two to three seconds in which the proxy's container is replaced; certificates and configuration are on volumes and stay, and the applications' containers are not touched.

The agent's database is as 0.7 left it: 0.8 applies no schema migration, so going back to 0.7 is the installer with that version.

What the upgrade itself changes:

  • The proxy's image gains Shipwick's own way of finding replicas, and the agent loads a configuration that uses it. The agent is restarted before the proxy is replaced; in between it serves through the older proxy the way it did, and offers the proxy its own way again once a minute. See Replicas are found by name.
  • The API moves to a network of its own, shipwick-control, for the agent, the proxy and the dashboard. The agent no longer listens on the network the applications are on, and the dashboard is no longer reachable from application containers either. The installer sets this up with the applications left running. See Who can reach the API.

Five things behave differently without being asked:

  • An application's container that calls the API directly is refused, 403 APPLICATION_CALLER, before its token is looked at. An application that calls the API on purpose does so at the API's hostname, https://<SHIPWICK_AGENT_DOMAIN>.
  • An address rests before a container of another application takes it: half a second after a replica the agent stopped, 2.5 seconds after one that stopped by itself. A deployment of one application is as fast as before; three applications of two replicas each, deployed together, took 7.3 to 7.8 seconds where one alone took about 6.
  • shipwick config gives back the values that were written in the file, for an application deployed again by 0.8 of both shipwick and the agent. What is written in deploy.yaml in plain sight is therefore readable by whoever may deploy the application; a password belongs in ${NAME}. Until that deployment every value is masked as before. See Get deploy.yaml back from the server.
  • A Docker daemon that does not answer is 503 RUNTIME_UNAVAILABLE, also for a daemon that is not running, which used to be a 500; a request the server's disk has no room for is 507 DISK_FULL. After 30 seconds without an answer the agent raises the docker alert. See When things break.
  • shipwick doctor says more: the running applications without a memory limit, a server without swap, limits the Docker daemon does not enforce, an API that is open to applications, a proxy that is not Shipwick's image of this version. None of them is counted as a problem.

Three arrangements need a look after the upgrade, and shipwick doctor names each:

ArrangementAfter the upgrade
The agent from a packageRun the compose command again: docker compose --env-file /etc/shipwick/agent.env -f /usr/share/shipwick/compose.yml up -d. It moves the proxy and the dashboard onto the control network; until then the agent refuses both. SHIPWICK_LISTEN_ADDR on the Docker bridge is needed for the dashboard only: the proxy now reaches the agent by itself
A compose file of your ownAdd the shipwick-control network as the release's compose.production.yml has it, and leave SHIPWICK_LISTEN_ADDR unset for the agent's container. Until then the agent listens as it did, and the API is open to applications
The API's port published in compose.override.yml, on Docker before 28The agent keeps listening on the network the applications are on, so that the port works. Upgrade Docker, or give the API a hostname and remove the port

A proxy image of your own (SHIPWICK_CADDY_IMAGE in .env) is not replaced by the upgrade. Build it again from the new Dockerfile.caddy; until then the agent serves through it the way it did before 0.8, and shipwick doctor says so. See A proxy image of your own.

What is new is opt-in. What an operator may want to turn on or use:

How
Lock an application's containers down furthersecurity in deploy.yaml: a read-only root filesystem, fewer capabilities, no container that runs as root. See Containers locked down further
Deploy one application out of a shipwick.yamlshipwick deploy api --image …. See Deploy from CI
Verify the release before it is installedInstall cosign on the server, and set SHIPWICK_REQUIRE_SIGNATURE=1 for the installer to make it a condition. See Verify a release
Give every application a memory limit, and the server swapresources.memory in deploy.yaml, a swap file on the server. See Prepare a server
Give the agent less than the Docker socketA socket proxy, or rootless Docker. Since 1.1 the socket proxy is part of every installation; see The Docker socket

On laptops and in CI, upgrade the CLI as described below; the names for deploy and validate, the plain values of config and the new lines of doctor need it. A 0.7 shipwick keeps working against a 0.8 agent for everything it knows. A 0.8 shipwick against a 0.7 agent deploys as before, with one difference: a deploy.yaml with a security block is refused by the older agent, as every key it does not know, and shipwick deploy says that the server is older than the file and how to upgrade it.

Upgrading from 0.6 to 0.7 ​

0.7 adds to what the agent stores and keeps on its disk; applications run as before, and nothing changes for a deploy.yaml that worked before. All three containers are recreated, because their images changed. The proxy is away for the moment that takes, and established connections through it are cut; certificates and configuration are on volumes and stay, and the applications behind the proxy keep running.

On its first start the new agent applies three schema migrations to shipwick.db, numbers 19 to 21: the index of the log archive, the references to stored secrets that are remembered with each deployment, and what a session's name rests on — the claim it was read from and the tenant it came from. An older agent refuses a database whose schema is newer than it supports, so do not downgrade afterwards. Nothing is encrypted anew.

Existing tokens, rules and sessions keep working. A session from before the upgrade was named by the email claim, and stays valid while the agent names people that way.

Four things behave differently without being asked:

  • The agent keeps the last output of containers that end, from the first one that ends after the upgrade: 14 days, at most 1 GB, as gzip files under logs/ in its data directory. They are not encrypted, like the logs Docker keeps. SHIPWICK_LOG_RETENTION_SIZE=0 keeps nothing. See Find out why it died.
  • The agent asks GitHub once a day whether a newer release exists. The request says nothing about the server; SHIPWICK_UPDATE_CHECK=off stops it. See above.
  • "********" is refused as an env value or a basic-auth password, by deploy and validate alike, with 400 INVALID_CONFIG and the fields to fill in. What the API masks can no longer be deployed as a value by pasting it back. A deploy.yaml that really holds that text as a value has to change.
  • GET /audit says whether older entries match, as more next to data, so a page that is exactly full no longer looks like there is another. ACCESS_NOT_GRANTED carries the person's name as details.name; details.email holds the same and stays for older clients.

One thing takes a deployment to arrive: shipwick config gives back a reference to a stored secret only for deployments made by 0.7. An application deployed before has all its secret values shown as masks until it is deployed again from its file.

What is new is opt-in. What an operator may want to turn on or use:

How
Read what a container printed before it diedNothing to turn on: shipwick logs my-api --previous. See Find out why it died
Keep more or less of that outputSHIPWICK_LOG_RETENTION_DAYS and SHIPWICK_LOG_RETENTION_SIZE in /opt/shipwick/.env
Get a lost deploy.yaml backshipwick config my-api -o deploy.yaml. See Get deploy.yaml back from the server
Move a token's end, or change its applications, without a new valueshipwick token update ci --expires 90d. See Change a token
End a dump that hangs at its limitbackups.before_in: container in deploy.yaml; pg_dump then needs -h localhost. See A before that is ended at its limit
Filter the audit trail by action and result, and export itshipwick audit --action token. --outcome refused,failed, --format csv. See See who changed what
Let accounts without an address sign inSHIPWICK_OIDC_NAME_CLAIM, and rules written name:. See Accounts without an address
Let the accounts of several Microsoft Entra tenants sign inSHIPWICK_OIDC_TENANTS with Entra's organizations issuer. See Accounts of several Microsoft Entra tenants
Keep an ECR or Artifact Registry credential currentThe cloud CLI's token piped into shipwick registry login. See Registries with tokens that expire
Run the agent as a systemd serviceThe .deb or .rpm of the release. It is a separate installation, reached by export and import. See Install from a package

Each variable is one the new compose file passes through and that is empty until you set it.

A bundle for a server with no connection now proves its images: a release from 0.7.0 on publishes their digests, shipwick server bundle checks the archive against them, and the installer checks what Docker loaded. See Install on a server with no way out.

In the dashboard, an application's Logs tab reads the log archive, Change the configuration opens the application's deploy.yaml, a token is edited on the Access page, the audit trail is filtered and exported, an export is downloaded and a file imported on the server's page, and every page works with the keyboard alone. See Use the dashboard.

On laptops and in CI, upgrade the CLI as described below; the new commands and flags need it. On Windows on Arm, shipwick upgrade picks the Arm build, also when the shipwick that runs is the x64 one. A 0.6 shipwick keeps working against a 0.7 agent for everything it knows. A 0.7 shipwick against a 0.6 agent says so where the agent lacks something:

Asked forThe CLI says
logs --previous, --list, --id, --search and the othersthe agent is older than this shipwick and keeps no log archive: it shows the output of running containers only
configthe agent is older than this shipwick and does not write an application's deploy.yaml
audit --action, --outcome, --actor-kindthe agent is older than this shipwick: it filters the audit trail by application, actor and time only, and would have ignored --action, --outcome and --actor-kind
audit --format, --outputthe agent is older than this shipwick and cannot export its audit trail

token update against a 0.6 agent changes nothing, and says that the agent is older.

Upgrading from 0.5 to 0.6 ​

0.6 adds to what the agent stores and to what it tells the proxy; applications run as before, and nothing changes for a deploy.yaml that worked before. All three containers are recreated, because their images changed. The proxy is away for the moment that takes, and established connections through it are cut; certificates and configuration are on volumes and stay, and the applications behind the proxy keep running.

On its first start the new agent:

  1. Applies five schema migrations to shipwick.db, numbers 14 to 18: what the agent remembers about imports and a promotion across its own restarts, the applications and the expiry of a token, the audit trail, the access rules, and the sessions of people who signed in. An older agent refuses a database whose schema is newer than it supports, so do not downgrade afterwards. Nothing is encrypted anew.
  2. Loads Caddy's configuration once. The new agent renders the proxy's configuration with one more setting — requests to replicas never go through a proxy of the environment — so the configuration it finds differs from the one it renders, and it is loaded once. A connection that is being established at that instant is reset, as with any change of the configuration; it does not happen again afterwards.

Existing tokens keep working: none is limited and none expires. The audit trail starts with the first change made after the upgrade.

Three things behave differently without being asked:

  • A hostname under redirects of an application with a path redirects to the domain and that path, https://example.com/api/users for a request for /users, instead of to the same path on the domain.
  • A standby remembers what it imported across restarts of its agent, and no longer imports the newest export once more after every restart.
  • An import validates every application's configuration by all the rules a deploy.yaml is held to.

What is new is opt-in. What an operator may want to turn on:

How
Limit CI's token to what it deploys, and give it an endCreate its replacement with shipwick token create ci-2 --role deploy --app my-api --expires 90d, hand it over, revoke the old one. See Create tokens for CI and teammates
Let people sign in to the dashboard with the company's accountsSHIPWICK_OIDC_ISSUER, SHIPWICK_OIDC_CLIENT_ID and SHIPWICK_OIDC_CLIENT_SECRET in /opt/shipwick/.env, then shipwick access grant. See Sign in with your company's accounts
Read who changed whatNothing to turn on: shipwick audit. See See who changed what
Stop Node replicas at once instead of after their grace periodinit: true in deploy.yaml. See An init process
Give a long dump more than an hourbackups.before_timeout in deploy.yaml. See Back up on a schedule
Show several servers in one dashboardSHIPWICK_AGENTS in /opt/shipwick/.env. See Several servers in one dashboard
Go out through a proxy, trust an authority of your ownHTTPS_PROXY, SHIPWICK_CA_FILE, SHIPWICK_DNS_RESOLVERS, SHIPWICK_ACME_DIRECTORY. See Run behind a corporate proxy or without internet

Each variable is one the new compose file passes through and that is empty until you set it. Backups larger than 5 GB now reach the bucket without anything being set: an archive of more than 64 MiB is sent in parts.

In the dashboard, an application's page and the server's page are tabs with addresses of their own, and tokens are under Access. The old addresses — /tokens, /secrets, /registries — redirect. See Use the dashboard.

On laptops and in CI, upgrade the CLI as described below; the new commands need it. A 0.5 shipwick keeps working against a 0.6 agent for everything it knows; it promotes a standby in one request that is held until the end. A 0.6 shipwick against a 0.5 agent says so where the agent lacks something: the agent is older than this shipwick and keeps no audit trail, the agent is older than this shipwick: it knows neither --app nor --expires, and created nothing.

Upgrading from 0.4 to 0.5 ​

0.5 changes the proxy's image and adds to what the agent stores; applications run as before. All three containers are recreated. The agent and the dashboard because their images changed, and the Caddy container because it is replaced with Shipwick's own build of Caddy, ghcr.io/shipwick/caddy: Caddy with the Cloudflare DNS module and nothing else added, pinned to the release like the other two. The proxy is away for the moment that takes, and established connections through it are cut; certificates and configuration are on volumes and stay, and the applications behind the proxy keep running. On its first start, before it serves requests, the new agent applies four schema migrations to shipwick.db, numbers 10 to 13: the registry credentials, the supplied certificates, the traffic samples, and the backups the agent takes. An older agent refuses a database whose schema is newer than it supports, so do not downgrade afterwards. Nothing is encrypted anew; registry passwords and certificate keys are encrypted with the same encryption.key as the environment values, and the key can now be rotated.

Two things the upgrade itself does differently from this release on:

  • A deployment that is running while the agent restarts is resumed, not failed, as described above. The agent that is stopped must already be 0.5 to leave it that way: during the upgrade from 0.4 itself, the old agent still marks a deployment in flight as FAILED when it shuts down, so do not start this upgrade in the middle of a deployment.
  • The installer removes the images of earlier releases. The removal was added in 0.4 and repaired in 0.4.1, but nothing called it; it now runs once the upgraded agent is healthy. Run the installer again on a server that is up to date to reclaim the space; it changes nothing else there.

With a hostname for the API, the installer also signs in the shipwick on the server: it saves the URL and the token as a context of the user who runs it, when no other server is saved there. See The CLI on the server.

Nothing changes for a deploy.yaml that worked before, with one difference in timing: replacing a replica no longer waits for the old one to exit, so a deployment is done when every new replica serves, and the replaced containers get their SIGTERM and their grace period in the background. What is new is opt-in. In deploy.yaml: path, proxy, static.fallback, backups, deploy.stop_timeout, wildcard hostnames. On the agent, each a variable in /opt/shipwick/.env that the new compose file passes through and that is empty until you set it: SHIPWICK_CLOUDFLARE_API_TOKEN, the SHIPWICK_BACKUP_* variables, SHIPWICK_EXPORT_SCHEDULE and SHIPWICK_STANDBY_SCHEDULE, and the two alert thresholds. See Agent configuration. Until SHIPWICK_BACKUP_PASSPHRASE is set, the agent's own state is not backed up, and shipwick doctor says so.

Private images keep working as they did: a mount of the server's docker login credentials in compose.override.yml is still read for a registry without a stored credential. shipwick registry login stores the credential on the agent instead, and the mount is then no longer needed; see Pull from private registries.

On laptops and in CI, upgrade the CLI as described below; the new commands need it. A 0.5 CLI against a 0.4 agent deploys as before: it asks the agent about the domain only before it builds, and sends the whole image; shipwick server rotate-key answers the agent is older than this shipwick and cannot rotate its key.

Upgrading from 0.3 to 0.4 ​

0.4 adds to what the agent stores and to what the proxy mounts; applications run as before. The agent and dashboard containers are recreated because their images changed. The Caddy container is recreated once as well, because the new compose file gives it the caddy-static volume for the folders of static applications: established connections through the proxy are cut at that moment, and the applications behind it keep running.

On its first start, before it serves requests, the new agent applies two schema migrations to shipwick.db, numbers 8 and 9: the secrets table, and what a static deployment serves (its digest, file count and size) on every deployment. An older agent refuses a database whose schema is newer than it supports, so do not downgrade afterwards. Nothing is encrypted anew; the stored secrets are encrypted with the same encryption.key as the environment values.

Nothing changes for a deploy.yaml that worked before. What is new is opt-in: build: ., static:, health.start_period, shipwick.yaml, secrets on the server, and the API's rate limit on failed authentications, which a valid token never meets. A 0.3 shipwick keeps working against a 0.4 agent for everything it knows; build, static, shipwick.yaml, secret, volumes, server install, doctor and open need the new CLI, and a 0.4 CLI against a 0.3 agent says The agent does not know this operation when it asks for one of them. The GitHub Action follows the latest release by default.

Upgrading from 0.2 to 0.3 ​

0.3 changes what the agent stores, not how applications run. No application container is recreated by the upgrade; only the agent, Caddy and dashboard containers are, because their images changed, and applications keep serving throughout.

On its first start, before it serves requests, the new agent:

  1. Applies four schema migrations to shipwick.db, numbers 4 to 7: the tokens table, the token name on every deployment (by), the metrics history samples, and the runs of jobs and one-off commands. Migrations run inside a transaction and are the reason the agent must not be downgraded afterwards: an older agent refuses a database whose schema is newer than it supports.
  2. Creates encryption.key in its data directory — the agent-data volume, /var/lib/shipwick inside the container — with mode 0600, and logs created the key that encrypts env values in the database; back it up together with shipwick.db. Every deployment's env values, of every deployment made by earlier releases too, are then encrypted in place, once. From now on the database is readable only together with the key.
  3. Finds the running replicas, gives them their names and reconciles them, as after every restart.

Back up encryption.key

Add encryption.key to whatever backs up shipwick.db. A copy of the database without the key reveals no secrets, which is the point, but the agent cannot read it either: started against a database whose values were encrypted with another key, it refuses to start, names the deployment and variable it could not read, and asks you to restore the key file from your backup or to set SHIPWICK_ENCRYPTION_KEY to the key the database was written with. Since 0.5 the agent backs up both itself once SHIPWICK_BACKUP_PASSPHRASE is set, and the key can be rotated.

The root token does not change; the token in /opt/shipwick/.env is the same one, now with the name root and the admin role. Tokens for CI and teammates are created from it afterwards, and can be given less: see Create tokens for CI and teammates. The new compose file passes SHIPWICK_WEBHOOK_URL and SHIPWICK_WEBHOOK_SECRET from .env to the agent, both empty until you set them; see Get notified. The dashboard's History charts start filling as soon as the new agent samples; there is no history from before the upgrade.

On laptops and in CI, upgrade the CLI as described next. A 0.2 shipwick keeps working against a 0.3 agent for everything it knows; the new commands need the new CLI.

Upgrading from 0.1 to 0.2 ​

0.2 changes how the proxy finds replicas: by a name on a second network instead of by container name. The upgrade recreates the Caddy container once, because it joins that network. Established connections through the proxy are cut at that moment; nothing else is, and applications keep running throughout. The replicas the new agent finds on startup are given their names before Caddy is told to look for them, and Caddy's configuration then changes once. See Routing and HTTPS.

Upgrade the CLI elsewhere ​

The server's copy of shipwick is upgraded with the server. On a laptop, the CLI upgrades itself:

bash
shipwick upgrade
text
✓ Upgraded shipwick v0.3.1 → v1.1.0
  /usr/local/bin/shipwick

The server runs v1.1.0, the latest release.

The latest release is downloaded from GitHub and verified against its checksums.txt before the binary is swapped; nothing changes if the checksum does not match. Since 0.8, where cosign (2.5 or later) is installed, checksums.txt is verified against the release's signature first, and a release from 0.8.0 on that has no signature is refused; without cosign the command says that the signature was not checked. See Verify a release. The command then compares the server's version with the release. When the server is behind, it says so and prints the installer line to run there; a server it cannot reach is one dim line, not an error. shipwick upgrade --check reports and changes nothing.

Since 0.9, shipwick upgrade --version v1.1.0-rc.1 installs exactly that release, a pre-release included, in place of whatever runs now, verified the same way: the way to try a release candidate, and the way back from one. The server is then compared with that release, and the installer line names it (SHIPWICK_VERSION=v1.1.0-rc.1). A pre-release ahead of the latest release counts as current afterwards. A development build, dev or a build made past a tag, is neither upgraded nor told that a release is available; upgrade and doctor say that it is a development build.

A shipwick from Homebrew is left to Homebrew — the command prints Upgrade with: brew upgrade shipwick — because the package manager would otherwise be confused by a file it did not put there. On Windows, shipwick upgrade replaces shipwick.exe the same way and leaves the old binary behind as shipwick.old.exe until the next run removes it; running the Windows installer again upgrades as well.

In CI, pin the version in the installer line instead, and move it when you mean to:

bash
curl -fsSL https://get.shipwick.com | SHIPWICK_VERSION=v0.4.1 sh -s -- --cli

The GitHub Action takes the same pin as version: v0.4.1. If shipwick is newer than the agent and uses an operation the agent does not have, it says so: The agent does not know this operation — it is probably older than this shipwick. shipwick server status shows both versions, and shipwick doctor compares both with the latest release.

If you installed without the installer ​

If you run the compose file by hand, replace it with the compose.production.yml attached to the new release, which pins the images to that version — since 0.5 the proxy's too, Shipwick's own build of Caddy — and passes the optional variables through, keep your .env, and run:

bash
docker compose -f compose.production.yml pull
docker compose -f compose.production.yml up -d

Since 0.8 a compose file of your own also needs the shipwick-control network, as the release's file has it; see Upgrading from 0.7 to 0.8. This kind of installation was not walked through the upgrade from every release the way the installer's was.

If you built the images from source, rebuild them from the new checkout — since 0.5 there are three, the proxy's with docker build -t ghcr.io/shipwick/caddy -f Dockerfile.caddy . — and run sh scripts/install.sh from it again.

If the agent was installed from a package, install the newer package and run its compose command again — coming from 0.7 that command is not optional; see Install from a package.

If the agent runs as a plain binary, replace it with the new build and restart it. The first start of 0.3.0 creates encryption.key in SHIPWICK_DATA_DIR, or reads SHIPWICK_ENCRYPTION_KEY if you set it; the first starts of 0.4.0, 0.5, 0.6, 0.7 and 0.9 apply the migrations described above; 0.8, 0.10 and 1.0 apply none.

What's next ​