The toolbox container tool has been updated from version 0.0.99 to 0.0.99.3 in Endless OS 6.0. Unfortunately, existing toolbox containers created in earlier versions of Endless OS will no longer be able to start after this upgrade. You will need to remove all existing containers, and recreate them, if desired.
This problem is characterised by the following error upon attempting to enter a container:
$ toolbox enter eos-toolbox-latest
Error: invalid entry point PID of container eos-toolbox-latest
This page details how to recover data and configuration from the now-incompatible containers (if desired), and how to remove them.
Before you remove your old containers, you may wish to access or save specific data or configuration from inside. To do this, mount the container's storage using the its name.
podman unshare bash
root=$(podman mount <name>)
where <name> is the container name as shown in toolbox list. Now the storage can be accessed at the path in the $root variable. With the container storage available, you can copy any files to a location in your home directory. For example:
cp -r $root/var ~/toolbox-var
When done, unmount the container storage and exit the bash shell:
podman unmount <name>
exit
If copying files out of the old container is insufficient for your recovery needs, it is possible to apply a workaround that enables you to enter your old containers as normal. It is recommended to only do this for temporary recovery purposes before deleting and recreating the container, because we don't know if this workaround will continue to be effective in future.
To do this, the dynamic linker needs to be copied into the container at the expected path. Determine the path the toolbox program expects for the dynamic linker:
readelf -p.interp /usr/bin/toolbox
On most computers this will show /run/host/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2. On the host, this file can be found with the /run/host prefix removed. Setup access to the container storage as in the process above, then copy this to the path with /run/host within the container storage. For example:
podman unshare bash
root=$(podman mount <name>)
mkdir -p $root/run/host/lib/x86_64-linux-gnu
cp /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 $root/run/host/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
podman unmount <name>
exit
You can now enter the container as usual:
toolbox enter <name>
After performing any desired data recovery as above, you can proceed to remove the old & incompatible containers.
First, stop and list existing containers:
podman stop --all
toolbox list
To remove a single container:
toolbox rm <name>
To remove all containers:
toolbox rm --all
Finally, proceed to recreate your containers. Once you do this, they will be set up in a more future-proof fashion, which hopefully means you will not be troubled by such a compatibility break in future.
This section provides detailed background context for those curious about why this unfortunate situation has occurred. It is not necessary to read or understand these low-level details for the recovery process detailed above.
toolbox performs its magic by executing the main toolbox binary from the host system within the container you are entering. However, problems naturally emerge when there are incompatibilities between the libc used by the host's toolbox and the one present in the container.
Toolbox has attempted at least 3 approaches to overcome this challenge (1, 2, 3) and the final configuration is that the host's dynamic linker (ld.so) needs to be available under /run/host/lib in the container.
While containers created with previous versions of Endless OS do mount
much of the host system /run/host, the lib directory is unavailable. Unfortunately, it is not easy to add more host mounts to existing podman containers, so we resort to removing and recreating the container.
When you create or recreate containers with the newer version of toolbox, the entirity of the host filesystem is made available under /run/host, which means that this class of problem should be avoided in future.