Last active
September 2, 2024 13:23
-
-
Save marcelrv/e7dac93863945dccc46f70915c84b5db to your computer and use it in GitHub Desktop.
Revisions
-
marcelrv revised this gist
Jan 28, 2022 . 1 changed file with 1 addition and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -9,7 +9,7 @@ The only way I found that is somewhat automated is the the docker-compose way.. Which is still not 100% and require manual fixing of stupid errors of the docker-compose tool (mainly things that ere not interpreted right, like dates & userIds that need to manually surrounded by quotes etc) Also I needed to add `network_mode: bridge` or `network_mode: host` to the configs as I did not want a network definition for each container which seems to be the docker-compose default way ## Backup the whole crap before doing heart surgery ``` @@ -89,4 +89,3 @@ docker container stop `docker container ls -aq` docker container rm `docker container ls -aq` ``` -
marcelrv renamed this gist
Oct 5, 2021 . 1 changed file with 0 additions and 0 deletions.There are no files selected for viewing
File renamed without changes. -
marcelrv renamed this gist
Oct 5, 2021 . 1 changed file with 0 additions and 0 deletions.There are no files selected for viewing
File renamed without changes. -
marcelrv revised this gist
Oct 5, 2021 . No changes.There are no files selected for viewing
-
marcelrv revised this gist
Oct 5, 2021 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -36,7 +36,7 @@ for c in $CIDS; do mkdir ~/dockercontainers/$c ; ~/autocompose.py $c > dockercon # check if there are no errors in the created files for c in $CIDS; do cd ~/dockercontainers/$c ; echo ======== $c ======= ;docker-compose config ; done #Fix possible errors... e.g. adding quotes around strings docker-compose does not swallow (e.g. dates) ``` -
marcelrv revised this gist
Oct 5, 2021 . 1 changed file with 160 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,160 @@ #! /usr/bin/env python # Copy of https://github.com/Red5d/docker-autocompose dirty hacked with some fixes for dates and # possiblity to create docker command lines from the existing containers by using the -o2 parameter import sys, argparse, pyaml, docker from collections import OrderedDict def main(): parser = argparse.ArgumentParser(description='Generate docker-compose yaml definition from running container.') parser.add_argument('-v', '--version', type=int, default=3, help='Compose file version (1 or 3)') parser.add_argument('-o', '--output', type=int, default=0, help='Type of output (0 or 1,2)') parser.add_argument('cnames', nargs='*', type=str, help='The name of the container to process.') args = parser.parse_args() struct = {} networks = [] for cname in args.cnames: cfile, networks,cmdline = generate(cname) struct.update(cfile) if args.output >0: print ("\r\n"+ cmdline + "\r\n") if args.output <2: render(struct, args, networks) def render(struct, args, networks): # Render yaml file if args.version == 1: pyaml.p(OrderedDict(struct)) else: pyaml.p(OrderedDict({'version': '"3"', 'services': struct, 'networks': networks})) def generate(cname): c = docker.from_env() try: cid = [x.short_id for x in c.containers.list(all=True) if cname == x.name or x.short_id in cname][0] except IndexError: print("That container is not available.") sys.exit(1) cattrs = c.containers.get(cid).attrs # Build yaml dict structure cfile = {} cfile[cattrs['Name'][1:]] = {} ct = cfile[cattrs['Name'][1:]] cmdLine ="docker run --name " + str( cattrs['Name'][1:]) values = { 'cap_add': cattrs['HostConfig']['CapAdd'], 'cap_drop': cattrs['HostConfig']['CapDrop'], 'cgroup_parent': cattrs['HostConfig']['CgroupParent'], 'container_name': cattrs['Name'][1:], 'devices': [], 'dns': cattrs['HostConfig']['Dns'], 'dns_search': cattrs['HostConfig']['DnsSearch'], 'environment': cattrs['Config']['Env'], 'extra_hosts': cattrs['HostConfig']['ExtraHosts'], 'image': cattrs['Config']['Image'], 'labels': cattrs['Config']['Labels'], 'links': cattrs['HostConfig']['Links'], #'log_driver': cattrs['HostConfig']['LogConfig']['Type'], #'log_opt': cattrs['HostConfig']['LogConfig']['Config'], #'logging': {'driver': cattrs['HostConfig']['LogConfig']['Type'], 'options': cattrs['HostConfig']['LogConfig']['Config']}, 'networks': {x for x in cattrs['NetworkSettings']['Networks'].keys() if x != 'bridge'}, 'security_opt': cattrs['HostConfig']['SecurityOpt'], 'ulimits': cattrs['HostConfig']['Ulimits'], 'volumes': cattrs['HostConfig']['Binds'], 'volume_driver': cattrs['HostConfig']['VolumeDriver'], 'volumes_from': cattrs['HostConfig']['VolumesFrom'], 'entrypoint': cattrs['Config']['Entrypoint'], 'user': cattrs['Config']['User'], 'working_dir': cattrs['Config']['WorkingDir'], 'domainname': cattrs['Config']['Domainname'], 'hostname': cattrs['Config']['Hostname'], 'ipc': cattrs['HostConfig']['IpcMode'], 'mac_address': cattrs['NetworkSettings']['MacAddress'], 'privileged': cattrs['HostConfig']['Privileged'], 'restart': cattrs['HostConfig']['RestartPolicy']['Name'], 'read_only': cattrs['HostConfig']['ReadonlyRootfs'], 'stdin_open': cattrs['Config']['OpenStdin'], 'tty': cattrs['Config']['Tty'] } # Populate devices key if device values are present if cattrs['HostConfig']['Devices']: values['devices'] = [x['PathOnHost']+':'+x['PathInContainer'] for x in cattrs['HostConfig']['Devices']] networks = {} if values['networks'] == set(): del values['networks'] values['network_mode'] = 'bridge' else: networklist = c.networks.list() for network in networklist: if network.attrs['Name'] in values['networks']: networks[network.attrs['Name']] = {'external': (not network.attrs['Internal'])} # Check for command and add it if present. if cattrs['Config']['Cmd'] != None: values['command'] = " ".join(cattrs['Config']['Cmd']), # Check for exposed/bound ports and add them if needed. try: expose_value = list(cattrs['Config']['ExposedPorts'].keys()) ports_value = [cattrs['HostConfig']['PortBindings'][key][0]['HostIp']+':'+cattrs['HostConfig']['PortBindings'][key][0]['HostPort']+':'+key for key in cattrs['HostConfig']['PortBindings']] # If bound ports found, don't use the 'expose' value. if (ports_value != None) and (ports_value != "") and (ports_value != []) and (ports_value != 'null') and (ports_value != {}) and (ports_value != "default") and (ports_value != 0) and (ports_value !=$ for index, port in enumerate(ports_value): if port[0] == ':': ports_value[index] = port[1:] # print ( "-p " , str (ports_value[index])) cmdLine = cmdLine + " -p " + str (ports_value[index]) values['ports'] = ports_value else: values['expose'] = expose_value except (KeyError, TypeError): # No ports exposed/bound. Continue without them. ports = None # fix some dates from causing errors. try: if values['labels']['org.label-schema.build-date'] != None: d = values['labels']['org.label-schema.build-date'] values['labels']['org.label-schema.build-date'] = "'" + str( d ) + "'" except KeyError as ke: pass # Iterate through values to finish building yaml dict. for key in values: value = values[key] if (value != None) and (value != "") and (value != []) and (value != 'null') and (value != {}) and (value != "default") and (value != 0) and (value != ",") and (value != "no"): ct[key] = value if cattrs['HostConfig']['Binds']: for key in cattrs['HostConfig']['Binds']: cmdLine = cmdLine + ' -v ' + str(key) if cattrs['Config']['Env']: for key in values['environment']: cmdLine = cmdLine + ' -e ' + str(key.split('=')[0]) if len(key.split('='))>1: cmdLine = cmdLine + '="' + str(key.split('=')[1]) + '"' cmdLine = cmdLine + " " + str ( cattrs['Config']['Image']) return cfile, networks, cmdLine if __name__ == "__main__": main() -
marcelrv revised this gist
Oct 5, 2021 . 1 changed file with 1 addition and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -60,6 +60,7 @@ sudo systemctl stop docker sudo nano /etc/docker/daemon.json ``` if the file is already there, add line `"storage-driver" : "overlay2" ` if the file is empty, you can go with ` echo '{ "storage-driver" : "overlay2" }' > /etc/docker/daemon.json` `sudo systemctl start docker` -
marcelrv revised this gist
Oct 5, 2021 . 1 changed file with 4 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -6,7 +6,9 @@ note. Based on https://www.sylvaincoudeville.fr/2019/12/docker-migrer-le-stockag NOT WORKING!!!! IF FOLLOWING THE ABOVE.. SOMEHOW THE CONTAINERS DO NOT RE-APPEAR! The only way I found that is somewhat automated is the the docker-compose way.. Which is still not 100% and require manual fixing of stupid errors of the docker-compose tool (mainly things that ere not interpreted right, like dates & userIds that need to manually surrounded by quotes etc) Also I needed to add `network_mode: bridge` to the configs as I did not want a network definition for each container which seems to be the docker-compose default way ## Backup the whole crap before doing heart surgery @@ -16,7 +18,7 @@ sudo cp -au /var/lib/docker /var/lib/docker.bk sudo systemctl start docker ``` ## Export Containers Export all container configs ``` @@ -38,7 +40,7 @@ for c in $CIDS; do cd ~/dockercontainers/$c ; echo ======== $c ======= ;docker-c ``` ## Store all images (I think not needed if no persistent references are used) ``` cd ~ -
marcelrv revised this gist
Oct 5, 2021 . 1 changed file with 2 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,7 +1,8 @@ # Docker migrate to overlay2 from aufs script Crazy that this is pretty much forced change without proper transition script note. Based on https://www.sylvaincoudeville.fr/2019/12/docker-migrer-le-stockage-aufs-vers-overlay2/ NOT WORKING!!!! IF FOLLOWING THE ABOVE.. SOMEHOW THE CONTAINERS DO NOT RE-APPEAR! The only way I found that is somewhat automated is the the docker-compose way.. -
marcelrv revised this gist
Oct 5, 2021 . 1 changed file with 5 additions and 5 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -20,12 +20,12 @@ Export all container configs ``` cd ~ mkdir ~/dockercontainers sudo docker container list -a > ~/dockercontainers/containers.txt cat ~/dockercontainers/containers.txt CIDS=$(docker container list -a | sed '1d' | awk '{print $1}') for c in $CIDS; do mkdir ~/dockercontainers/$c ; docker run --rm -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/red5d/docker-autocompose $c > ~/dockercontainers/$c/docker-compose.yml ; done # or if local (with fixes) for c in $CIDS; do mkdir ~/dockercontainers/$c ; ~/autocompose.py $c > dockercontainers/$c/docker-compose.yml ; done -
marcelrv revised this gist
Oct 5, 2021 . 1 changed file with 9 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -77,3 +77,12 @@ for c in $IDS; do docker load -i $c; done ## Restore containers `for c in $CIDS; do cd ~/dockercontainers/$c ; docker-compose up -d --no-deps --build ; done` When something goes horrably wrong you can remove the containers and try again ``` docker container stop `docker container ls -aq` docker container rm `docker container ls -aq` ```
-
marcelrv revised this gist
Oct 5, 2021 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -24,7 +24,7 @@ mkdir dockercontainers sudo docker container list > dockercontainers/containers.txt cat dockercontainers/containers.txt CIDS=$(docker container list -all | sed '1d' | awk '{print $1}') for c in $CIDS; do mkdir dockercontainers/$c ; docker run --rm -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/red5d/docker-autocompose $c > ~/dockercontainers/$c/docker-compose.yml ; done # or if local (with fixes) -
marcelrv revised this gist
Oct 5, 2021 . 1 changed file with 5 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -60,6 +60,11 @@ if the file is already there, add line `"storage-driver" : "overlay2" ` if the file is empty, you can go with ` echo '{ "storage-driver" : "overlay2" }' > /etc/docker/daemon.json` `sudo systemctl start docker` What also works is edit the file first and than just restart (so no need to stop first) `sudo systemctl restart docker` ## Restore images ``` -
marcelrv revised this gist
Oct 5, 2021 . 1 changed file with 12 additions and 12 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,11 +1,12 @@ # Docker migrate to overlay2 from aufs script Crazy that this is pretty much forced change without proper transition script Based on https://www.sylvaincoudeville.fr/2019/12/docker-migrer-le-stockage-aufs-vers-overlay2/ NOT WORKING!!!! IF FOLLOWING THE ABOVE.. SOMEHOW THE CONTAINERS DO NOT RE-APPEAR! The only way I found that is somewhat automated is the the docker-compose way.. Which is still not 100% and require manual fixing of stupid errors of the docker-compose tool (mainly things that ere not interpreted right, like dates & userIds that need to manually surrounded by quotes etc) Also I needed to add `network_mode: bridge` to the configs as I did not want a network definition for each container which seems to be the docker-compose default way ## Backup the whole crap before doing heart surgery ``` @@ -23,18 +24,20 @@ mkdir dockercontainers sudo docker container list > dockercontainers/containers.txt cat dockercontainers/containers.txt CIDS=$(docker container list | sed '1d' | awk '{print $1}') for c in $CIDS; do mkdir dockercontainers/$c ; docker run --rm -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/red5d/docker-autocompose $c > ~/dockercontainers/$c/docker-compose.yml ; done # or if local (with fixes) for c in $CIDS; do mkdir ~/dockercontainers/$c ; ~/autocompose.py $c > dockercontainers/$c/docker-compose.yml ; done # check if there are no errors in the created files for c in $CIDS; do cd ~/dockercontainers/$c ; echo ======== $c ======= ;docker-compose config ; done #Fix possible errors... ``` Store all images (I think not needed if no persistent references are used) ``` cd ~ @@ -47,9 +50,6 @@ for c in $IDS; do docker save -o dockersave/$c.tar $c; done cat dockersave/list.txt | sed '1d' | grep -v "" | awk '{ print "docker tag "$3" "$1":"$2 }'>> dockersave/tag ``` ## Change driver ``` @@ -71,4 +71,4 @@ for c in $IDS; do docker load -i $c; done ## Restore containers `for c in $CIDS; do cd ~/dockercontainers/$c ; docker-compose up -d --no-deps --build ; done` -
marcelrv revised this gist
Oct 5, 2021 . 1 changed file with 7 additions and 4 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -3,9 +3,9 @@ Crazy that this is pretty much forced change withour proper transition script Based on https://www.sylvaincoudeville.fr/2019/12/docker-migrer-le-stockage-aufs-vers-overlay2/ NOT WORKING!!!! IF FOLLOWING THE ABOVE.. SOMEHOW THE CONTAINERS DO NOT RE-APPEAR! TRY the docker-compose way ## Backup the whole crap before doing heart surgery ``` @@ -24,10 +24,13 @@ sudo docker container list > dockercontainers/containers.txt cat dockercontainers/containers.txt IDS=$(docker container list | sed '1d' | awk '{print $1}') for c in $IDS; do mkdir dockercontainers/$c ; docker run --rm -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/red5d/docker-autocompose $c > ~/dockercontainers/$c/docker-compose.yml ; done # or if local (with fixes) for c in $IDS; do mkdir ~/dockercontainers/$c ; ~/autocompose.py $c > dockercontainers/$c/docker-compose.yml ; done # check if there are no errors in the created files for c in $IDS; do cd ~/dockercontainers/$c ; echo ======== $c ======= ;docker-compose config ; done ``` -
marcelrv revised this gist
Oct 5, 2021 . 1 changed file with 4 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -25,6 +25,10 @@ cat dockercontainers/containers.txt IDS=$(docker container list | sed '1d' | awk '{print $1}') for c in $IDS; do mkdir dockercontainers/$c ; docker run --rm -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/red5d/docker-autocompose $c > dockercontainers/$c/docker-compose.yml ; done # or if local (with fixes) for c in $IDS; do mkdir dockercontainers/$c ; ~/autocompose.py $c > dockercontainers/$c/docker-compose.yml ; done ``` Store all images -
marcelrv revised this gist
Oct 4, 2021 . 1 changed file with 7 additions and 4 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -15,8 +15,7 @@ sudo systemctl start docker ``` ## Migrate Export all container configs ``` cd ~ @@ -25,8 +24,7 @@ sudo docker container list > dockercontainers/containers.txt cat dockercontainers/containers.txt IDS=$(docker container list | sed '1d' | awk '{print $1}') for c in $IDS; do mkdir dockercontainers/$c ; docker run --rm -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/red5d/docker-autocompose $c > dockercontainers/$c/docker-compose.yml ; done ``` Store all images @@ -61,4 +59,9 @@ if the file is empty, you can go with ` echo '{ "storage-driver" : "overlay2" }' cd dockersave/ IDS=$(ls *.tar) for c in $IDS; do docker load -i $c; done ``` ## Restore containers for c in $IDS; do cd ~/dockercontainers/$c ; docker-compose up -d --no-deps --build ; done -
marcelrv created this gist
Oct 4, 2021 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,64 @@ # Docker migrate to overlay2 from aufs script Crazy that this is pretty much forced change withour proper transition script Based on https://www.sylvaincoudeville.fr/2019/12/docker-migrer-le-stockage-aufs-vers-overlay2/ NOT WORKING!!!! SOMEHOW THE CONTAINERS DO NOT RE-APPEAR! ## Backup the whole crap before doing heart surgery ``` sudo systemctl stop docker sudo cp -au /var/lib/docker /var/lib/docker.bk sudo systemctl start docker ``` ## Migrate Store all containers ``` cd ~ mkdir dockercontainers sudo docker container list > dockercontainers/containers.txt cat dockercontainers/containers.txt IDS=$(docker container list | sed '1d' | awk '{print $1}') for c in $IDS; do docker commit $c dockercontainers/$c.tar ; done cat dockersave/list.txt | sed '1d' | grep -v "" | awk '{ print "docker tag "$3" "$1":"$2 }'>> dockersave/tag ``` Store all images ``` cd ~ mkdir dockersave sudo docker images > dockersave/list.txt cat dockersave/list.txt IDS=$(docker images | sed '1d' | awk '{print $3}') for c in $IDS; do docker save -o dockersave/$c.tar $c; done cat dockersave/list.txt | sed '1d' | grep -v "" | awk '{ print "docker tag "$3" "$1":"$2 }'>> dockersave/tag ``` ## Change driver ``` sudo systemctl stop docker sudo nano /etc/docker/daemon.json ``` if the file is already there, add line `"storage-driver" : "overlay2" ` if the file is empty, you can go with ` echo '{ "storage-driver" : "overlay2" }' > /etc/docker/daemon.json` `sudo systemctl start docker` ## Restore images ``` cd dockersave/ IDS=$(ls *.tar) for c in $IDS; do docker load -i $c; done