One of the most replied posts in the Docker official forum is 'Upgrading data in data containers'

One of the most replied posts in the Docker official forum is "Upgrading data within a data container"

matlehmann
I have a container with data, which has persistent data in a volume (such as /var/data). The container contains persistent data for the software in another container.
For new versions of the software, said permanent data needs to be upgraded (structural or layout changes, etc.). As a result, I want to make another data container (in /var/data) with the upgraded data in the same location and still keep the old data container with its data intact.
This way, I can use the old data container and the old version of the software in case something goes wrong.
But, how can I do this? The steps required to achieve the desired result were not obvious to me.
I can run a command to upgrade the container like docker run -i -t --name temp --volumes-from data -v /upgraded image /upgrade_script.sh
But later, how do I restore the upgraded data to its original location without overwriting old data? If I run docker run -i -t --volumes-from temp image cp /upgraded /var/data it overwrites my old data. Do I have to use a host mounted volume for the upgraded data, or is there some better solution?

sam
Just a guess as here generally I prefer to use direct host mounted volumes and I'm struggling to find the utility of data containers.
But...can you commit your data container and then save the image etc?
sven
Oh, please also consider the suggestion of using docker commit snapshot container SAM Awesome
keeb
I actually use data containers like UNIX pipes; I feel like they fit more naturally in the paradigm
docker run -name some_pipe_storage some_container_which_generates_data
docker run --volumes-from some_pipe_storage something_that_operates_on_data
syntax, quite cumbersome. Very powerful yet primitive.

sven
There is some interesting work going on with docker as a volume management tool - I think they are heading towards 1.4 and I will do some investigating. (There would be a docker volume list and something to manipulate)
I would probably make a backup_data volume inside the container and then run a data migration image, I would like to run the image connected to data and backup_data - the first thing it would probably do It copies the data to backup_data, and then it does the data migration.
You can then run the old and new ones, connected to each respective databackend (with possibly attaching a read-only backup?)
Doing this, the host setup should be almost the same if you are using it, either directly or via Style representation for data containers.

matlehmann
Your suggestion is my first idea line but it doesn't meet my expectation because after the process the migrated data and original data will be according to different paths and I don't see any way This situation changes because non-host volumes cannot be remounted to a different path. The path to a volume from a data container is static - even for volume containers inherited by "--volumes-from".
This is different for the host volume as I can change the mount locations of these with each docker run call.
I think it is very necessary for you to talk about these volume management tools. To me, this data container docker idiom feels more like a workaround.
Excuse me, are you stating "docker commit is awesome" because I can't see it, however. At least with the use case in hand. As far as I know, a docker commit to me contains a new image of the current state of the container. This will include all the OS data I'm interested in except the persistent data.

sven
oh crud. You are right, the volume path is currently static. So you need one step
1. You have existing data container in /data
2. Migrate to temporary data container located in /migrate (if you have original installation)
3. Migrate data to /migrate to new installation Upgraded data container/data (the second migration image does not require the original data capacity to be installed
@cpuguy83 may be able to tell you more about the new tool smile
WRT docker commit - when you commit, you are not in a single include For everything in the image layer, you are making a new image layer that contains all the changes made in the container (used when the image container is started)
So if you use the container instead of the volume, persist the data and roll it off. For things like logging, you can use docker commit to snapshot/backup just your persistent data - and docker's exports might let you store those layers

cpuguy83
Yeah, I wouldn't rely on "commit" because of that. You will be limited to 127 commits unless you flatten the image out.
@matlehmann See github.com/cpuguy83/docker-volumes
This is far from a perfect solution, but works pretty well in the meantime

matlehmann
@Sven Thanks for your reply and for more information. I still don't understand step 3 of "/migrating data to the newly upgraded data container installed in /data (the second migration image does not require the original data capacity to be installed" as it currently is (with docker1.2 and (no special volume command), I don't see how I could have a container with another container's volume - part of them mounted and part of them not mounted. From what I've seen, it's either all or nothing, or "- -volumes-from other_container" or not. So if the container migrated from step 2 has the original data mounted, the container in step 3 it is already mounted, also therefore the copy operation from /mifrated to /data will overwrite the original data. Still me Lost something?
Thanks for the tip about the "commit" command, I need to think about this a little more

matlehmann
@keeb It's a good pattern, but as far as I can see it doesn't work. To solve the problem I'm talking about. All these "pipe" containers will still have the volume of "some_pipe_storage" without being able to create a different container with different data at a given path without overwriting the original data. . But maybe I'm missing your point?

sven
Well, let's see if I can make this explanation with an example:
Let's say someone has created some Docker images, webappv10, webappv11, webapp_migratorv10_to_v11.
Initially, you will already be running a 1.0 based system
docker run -v /data --name datav10 busybox true
docker run -p 80:80 --volumes-from datav10 --name webv10 webappv10
Then to upgrade, ask for your Data gave me an upgrade, would you do step 2 (as you pointed out, we can't have two volumes in the same directory)
docker run -v /migration --name datav10-to-v11 busybox true
docker run --volumes -from datav10-to-v11 --volumes-from datav10 --name migration webapp_migratorv10_to_v11
Then step 3, migrate the copied data to a new data container, and prepare the data in the /data directory using
docker run -v /data --volumes-from datav10-to-v11 --name datav11 busybox cp -r /migration /data
Then run the version 1.1 web application
docker run -p 80:80 --volumes-from datav11 --name webv11 webappv11
And for extra credit, you'd better script it all.
The update to datav10 to V11 volume container step increase is due to the discussion below

matlehmann
@sven Thank you again for your detailed answer. I appreciate your thoughts and time.
However, the procedure you outlined doesn't work. It calls volumes with "-v" overturning the assumption that a volume inherits the same path as "--volumes-from". I just tested it again to make sure, but that's not the case. That's why docker run -v / data --volumes-from migration --name datav11 busybox cp -r /migration /data will overwrite my original data container datav10.

sam
It has a special reason why you like a data container in a simple volume (i.e. easier to understand/process by feel)

sven
I'm confused. Carefully crafted the steps so that there are never 2 volumes from the contains/data dils statement we use to migrate a buffer. Let's copy it to the new data v11 container
@sam - there is such a little conceptual difference between a bind mounted volume and a volume container - you are still doing the same 3 steps - the biggest for me. The difference is that the bind mount only works locally, and assumes you have the disk space for it (not that I did), whereas the volume container approach assumes your Docker-data partition is large enough to run the Docker container, and works the same locally Remote.
If you change docker run -v /data ... lines to docker run -v /local/data:/data ..., then you use bind mounts, which represents

matlehmann
@SAM I'm now switching to using bind-mounted (or whatever the official term is "-v /host:/container") volumes instead of using data containers because of the disadvantages listed in this thread I started with. Using a data container, as the idiom is used and recommended by all on the internet, seems to be the "official way"

matlehmann
@sven
?"datav10" has ?"/data" (via -v /data)

?"migration" has ?"/data" (via --volumes from datav10)
?"/migration" (via `-v /migration')

?"datav11" has ?"/data" (via -v /data)
?"/migration" (via --volumes from migration)
?"/data" (via --volumes-from migration)

Thus, we have the "/data" container" datav11" two volumes definition - to me it looks like the --volumes-from win one.

sam
@sven I guess I'm just trying to figure out why you are storing the data in AUFS, it seems to be the wrong file system to use for this problem. BTRFS would be ok, but aufs seems an odd choice for log files, what Postgres databases are used for, etc. Am I misunderstanding the mechanics of data containers?

@matlehmann We use "volume" extensively.
We store host-mounted volumes in off-the-shelf containers for easy rotation and persistence of all logs. (Here is another option that would stream out of the container, but the mechanism is non-trivial, think about say NGINX container, do you use logs?)
We store some configuration on a mounted GlusterFS volume, so we can Absorbing stuff across the farm

matlehmann
@sven This will be a topic for another thread: I'd really like to hear about your GlusterFS setup. Is there a ready-to-use image that can be used as a representative volume within GlusterFS or how do you do it?

sam
@supermathie would be the best for setting up details for our GlusterFS, but it is all set up in a very traditional way and we don't use trusted docker containers to power up Gluster.

sven
OH (*&^, you are right.
I think I was so smart to remove the step.
You need to put the /migrate folder into its own data container so that avoids the last step you have to pay attention to
I've updated the example step by step to reflect this

sven
Yes, there is something to be resolved with WRT filesystems - I'm running mostly BTRFS with docker, I think. The installation works too - but I would still do the linking to them and then use the same process as above for the data volumes

matlehmann@sven I haven't tried this but it makes sense. An awkward crazy chicken dance, but it might work... Eagerly awaiting the roll order

sven Yes, I'm focusing on the clumsy chicken - looking forward to those versions, and the grilled chicken legs to devour

.

This article is translated from the Docker official forum: https://forums.docker.com/t/upgrade-data-within-data-container/205/20

The above introduces the most replied post in the Docker official forum "Upgrading data in the data container", including the relevant content. I hope it will be helpful to friends who are interested in PHP tutorials.