Great work, thank you! :)

Great work, but there are some issues:

Line 114 of the above autocompose.py ends with:

 and (ports_value !=$

That's a syntax error. I have changed that to

  and (ports_value != 'no'):

to match the original IGNORE_VALUES.

Also need to pin package requests<2.29. See docker/docker-py#3113

Whenever a named volume was used, there was an error in the generated docker-compose.yml files:

• Named volume used, but no declaration found

Besides of that, instead of using sed and awk to tweak docker's output I would suggest letting docker do the work itself by creating the list of container IDs with

docker container list -a --format '{{.ID}}'

Similarly, you can obtain the list of image IDs directly with

docker image ls --format '{{.ID}}' 

However, there can be duplicates, if the same image exists with several tags (for example latest and an explicit version tag). In that case, the duplicates should be eliminated by piping the output through

sort -u

That applies to the original version as well.

Instead of IDs, the autocompose.py script also accepts a container name. A list of container names can be obtained with

docker container list -a --format '{{.Names}}'

This allows to organize the generated docker-compose files in folders named after the container name. Whether or not this is better understandable than the container ids depends on the names given when running the container. If one has mostly the autogenerated names à la affectionate_newton then may be the ID is better. YMMV.

I really appreciate having access to this gist! Thank you.

I'm still working through a couple of issues, and would appreciate any help/expertise.

1. Tagging of the restore images didn't work for me. First, the tag file is empty unless I remove grep -v "" from the command to create the tag file. I do not understand what that was supposed to do?
2. I also had to execute tag to then add the tags and repo sources to the images. No issue once I figured out that, but maybe this note will help someone else out. Or, does this mean something else isn't correct?
3. Most of our original containers had been created with docker-compose. For those, is it necessary to use the the created docker-compose.yml files on the first run? I did, but then had one container not start (turned out I was out of space again) and so I nuked them. All but one of our containers starts with a single compose file, so I just used that and the generated file to kick start the one app. It seems to be working, but not sure if I'm misunderstanding something here?

BTW, this process took a LOT of storage space. I had to attach a second volume for holding the images, backup and other files.

3. Most of our original containers had been created with docker-compose.
   For those, is it necessary to use the the created docker-compose.yml files on the first run?

The purpose of the generated docker-compose.yml files is to be able to re-create the containers using the new storage driver. As @marcelrv gives the instruction "# Fix possible errors", that means that the generated files can have errors which you would need to fix. If you already have a working docker-compose.yml then IMHO it is always the better choice to use it. I have successfully migrated all my containers and this gist was a great help. But I have used it only for everything that was created "by hand". For services already defined with docker-compose.yml files, I just continue to use them with the new storage driver. That does work.

BTW, this process took a LOT of storage space. I had to attach a second volume for holding the images, backup and other files.

Yes. IMHO using a separate drive for the backup is a good idea anyway. I have even used a second computer for a dry run before I scratched everything on the "master" server.

For services already defined with docker-compose.yml files, I just continue to use them with the new storage driver. That does work.

Hahahaha! Wow. I wish I had realized that earlier. Would've saved me a lot of time :-)

Thank you for reply. I have to do one more server, and I'm pretty sure all of the containers are from existing compose files.

Try... Catch... Finally... Eventually...

I came to working steps / script...

Follow the above scenario, but this works, without grep

cat dockersave/list.txt | sed '1d' | awk '{ print "docker tag "$3" "$1":"$2 }'>> dockersave/tag
and, after a step to load images, you have to run

bash dockersave/tag

Which is not mentioned anywhere above...

Also, if you have old version of docker-compose you should

cd /usr/local/bin && sudo rm -rf docker-compose

and then download newest docker-compose, place it to /usr/local/bin and chmod +x it
You cam get it from here: https://github.com/docker/compose/releases
Also don't forget to remove backed-up directory under /var/lib/docker.bk

Also after this, you might want/need to completley remove docker, but don't delete/purge images and config files, and then upgrade docker...

We were able to sucessfully upgrade from ubuntu 14 to ubuntu 22 :) what a jump...

Please correct me if I'm wrong, but...

The steps that appear to have worked for me were no more complicated than:

• sudo systemctl stop docker.service;
• Edit /etc/docker/daemon.json to include the "storage-driver" : "overlay2" tuple;
• sudo systemctl start docker.service;

After this:

• journalctl -u docker.service --since '10 minutes ago' no longer reported warnings about using aufs, like:

  ... dockerd[...]: time="..." level=info msg="[graphdriver] using prior storage driver: aufs"
  ... dockerd[...]: time="..." level=warning msg="[graphdriver] WARNING: the aufs storage-driver is deprecated, and will be removed in a future release"
  ... dockerd[...]: time="..." level=info msg="Docker daemon" commit=b0f5bc3 graphdriver(s)=aufs version=20.10.7
  

  ... and instead, reports:

  ... dockerd[...]: time="..." level=info msg="Docker daemon" commit=b0f5bc3 graphdriver(s)=overlay2 version=20.10.7
  

• Morever, both /var/lib/docker/ and /var/lib/docker/image/ now each have overlay2/ sub-directories that they didn't have before; (their repective aufs/ sub-directories are still there, too, btw);
• docker image ls, docker container ls and docker service ls all report the same data as they did before...
  ... except for:
  • docker image ls which now list <none> for tags;
• Not to mention that the deployed stack also still appears to work as before;

Do note that this all is on a host in swarm mode (i.e., it had been docker swarm init ... and docker stack deploy --compose-file docker-compose.yml ...-ed at some point);

It would appear that all images were automatically converted from aufs to overlay2 by this, except for the replication of their tags, and hence that the only remaining housekeeping expected to-do, is to get those reattached.

Additionally, I think I should rm -r /var/lib/docker{,/image}/aufs once truly convinced that they're no longer needed.

For restoring and tagging the images on the same step, a restore script can be created as follows:

cat tag | awk '{ OFS="";print "docker load < ",$3, ".tar; ", $0}' > restore_images.sh

This assumes that tag contents look like this:

#tag contents
docker tag e34e831650c1 ubuntu:latest
docker tag f3d89a2abe0d postgres:15

The restore_images.sh script will look like:

docker load < e34e831650c1.tar; docker tag e34e831650c1 ubuntu:latest
docker load < f3d89a2abe0d.tar; docker tag f3d89a2abe0d postgres:15

Then run bash restore_images.sh.