Docker has a feature known as Volumes, that allow developers to persist data in use with containers. They are entirely managed by the Docker Engine making them seamless to the end-user. Docker volumes are a very important and useful concept and in this tutorial, we’ll learn all about Docker volumes, how to create volumes, how to list volumes, and how to delete volumes. We’ll also see how to share a volume among containers by spinning up several containers which all make use of the same volume for data sharing.
About Docker Volumes
To get started with Docker Volumes, we can simply look at the commands available under the docker volume management command. Like other things in Docker, we can create, inspect, list, prune, and remove volumes.
docker-volumes> docker volume Usage: docker volume COMMAND Manage volumes Commands: create Create a volume inspect Display detailed information on one or more volumes ls List volumes prune Remove all unused local volumes rm Remove one or more volumes
Create A Docker Volume
The best way to learn about volumes in Docker is to jump right into it and create one. Here we create a volume with the name of my-volume.
docker-volumes> docker volume create my-volume my-volume
Listing Docker Volumes
Using docker volume ls shows us our new volume.
docker-volumes> docker volume ls DRIVER VOLUME NAME local my-volume
Inspecting A Docker Volume
docker-volumes> docker volume inspect my-volume
[
{
"CreatedAt": "2020-10-20T17:35:06Z",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/my-volume/_data",
"Name": "my-volume",
"Options": {},
"Scope": "local"
}
]
Real World Docker Volume Example
For this test we will make use of Jenkins, an open-source automation server that enables software engineers to reliably build, test, and deploy their software using continuous integration and continuous delivery. Jenkins is a server-based system that runs in servlet containers such as Apache Tomcat. The goal here is to show that we can store data the Jenkins server generates while working in one container and then leverage a volume to use that data in subsequent containers running an instance of Jenkins.
Pull The Jenkins Image From Docker Hub
docker-volumes> docker pull jenkins Using default tag: latest latest: Pulling from library/jenkins 55cbf04beb70: Pull complete ... 1322ea3e7bfd: Pull complete Digest: sha256:eeb4850eb65f2d92500e421b430ed1ec58a7ac909e91f518926e02473904f668 Status: Downloaded newer image for jenkins:latest docker.io/library/jenkins:latest
Run Jenkins In A Container
Now that we have the image in our local repository, we can easily spin up an instance of the Jenkins server in a container. You’ll want to be at least a little bit familiar with running containers since the command used here makes use of several options including –name, -v, and -p. Check some of the prior Docker tutorials if you need a refresher. For this tutorial, the main concept to be aware of is that we are attaching a volume of my-volume to this container on startup (using -v my-volume:/var/jenkins_home).
docker-volumes> docker container run --name my-jenkins -v my-volume:/var/jenkins_home -p 8080:8080 -p 50000:50000 jenkins
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
...
...
INFO:
*************************************************************
*************************************************************
*************************************************************
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:
039b2e1d3609465ead266da3a660eaf1
This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
*************************************************************
*************************************************************
*************************************************************
Oct 20, 2020 5:53:51 PM hudson.model.UpdateSite updateData
INFO: Obtained the latest update center data file for UpdateSource default
...
...
INFO: Jenkins is fully up and running
--> setting agent port for jnlp
--> setting agent port for jnlp... done
The Jenkins server is now running. We can visit http://localhost:8080 to see it working. In addition, we can copy and paste the generated password from the above output to log in right away.
On an initial install of Jenkins, there is some setup involved. We choose “Select plugins to install”
Then on the Getting Started page just choose None in the upper left, then install at the bottom right.
On this page just continue as admin.
When Jenkins setup is complete, click on the “Start using Jenkins” button.
Docker Persistent Data With Volumes
The goal here is to demonstrate how data from this instance of Jenkins in this container will be available to future containers running Jenkins that make use of the volume we have created. In the following few steps, we set up a new job in Jenkins.
The job name can be whatever you like, we simply choose ExampleJob, select Freestyle project, then click ok.
Just as an example, we can add a build step by adding a shell command.
Now we have a fully operational Example Job in Jenkins server.
Create A New Jenkins Container
In this next step, we’ll create a second container and again run an instance of Jenkins in it. Since we already have one instance running which uses ports 8080:8080 and 50000:50000, we need to specify new ports so they don’t collide. Also, note that this container has a new and different name of my-OTHER-jenkins, however, the volume is the same(-v my-volume:/var/jenkins_home).
docker-volumes> docker container run --name my-OTHER-jenkins -v my-volume:/var/jenkins_home -p 9090:8080 -p 60000:50000 jenkins
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
...
...
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@54b6b3cf: defining beans [filter,legacy]; root of factory hierarchy
Oct 20, 2020 6:18:31 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
--> setting agent port for jnlp
--> setting agent port for jnlp... done
Now open a new browser tab and visit http://localhost:9090, and voila! We see another instance of Jenkins. Do notice that we are presented with a login screen right away. There are no setup steps required. This shows us that our volume in Docker is working correctly. All of the work we did to set up the first instance of Jenkins is shared with this new instance of Jenkins thanks to the volume we had created named my-volume.
Let’s just confirm that we have two Jenkins containers running, and sure enough, there they are.
docker volumes> docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ca63090c2877 jenkins "/bin/tini -- /usr/l…" 7 minutes ago Up 7 minutes 0.0.0.0:9090->8080/tcp, 0.0.0.0:60000->50000/tcp my-OTHER-jenkins e87acec5997f jenkins "/bin/tini -- /usr/l…" 32 minutes ago Up 32 minutes 0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp my-jenkins
Let’s list the volumes we have as well. The my-volume volume is working perfectly for us.
docker volumes> docker volume ls DRIVER VOLUME NAME local my-volume
Had we not created the volume earlier, when we spun up a second instance of Jenkins it would have asked us for all of the setup information all over again. This volume will continue to work as we set up and tear down containers.
Let’s remove all running containers.
docker volumes> docker rm $(docker ps -a -q) Error response from daemon: You cannot remove a running container ca63090c2877cef6cdc6c5e2338471fc347f5f5be48766a997feee7cf6a2ec39. Stop the container before attempting removal or force remove Error response from daemon: You cannot remove a running container e87acec5997fe4eeb73c4bf30ba23d416dd6d2723e97f7270f84d7afdffdd439. Stop the container before attempting removal or force remove
Whoops. We need to stop the containers before we remove them.
docker volumes> docker stop $(docker ps -a -q) ca63090c2877 e87acec5997f
Now we can remove all containers since they are stopped.
docker volumes> docker rm $(docker ps -a -q) ca63090c2877 e87acec5997f
Listing the containers shows they are all gone.
docker volumes> docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
A Third Container
Imagine we just finished for the day, and now we come back tomorrow and want to run Jenkins once again. We can spin up yet another container, and it should continue to have access to the data that was generated originally and stored in my-volume.
docker volumes> docker container run --name jenkins-THE-THIRD -v my-volume:/var/jenkins_home -p 8080:8080 -p 50000:50000 jenkins
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
Oct 20, 2020 6:29:44 PM Main deleteWinstoneTempContents
...
...
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@5517b224: defining beans [filter,legacy]; root of factory hierarchy
Oct 20, 2020 6:29:48 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
--> setting agent port for jnlp
--> setting agent port for jnlp... done
We can open up the Jenkins server and once again, we are right back into our original workflow with the same ExampleJob still intact.
Confirming this is in fact an entirely new container.
All of this was made possible by our humble volume, my-volume.
docker volumes> docker volume ls DRIVER VOLUME NAME local my-volume
Specify VOLUME In Docker File
In the example above we created and specified the volume to use entirely from the command line. You can also specify volumes in a Docker File using the VOLUME command. For example, within the official MySql Docker file is the following line.
VOLUME /var/lib/mysql
You can see that line in the Docker file here. This is the default location of the MySql databases and this command within the Docker File tells Docker that when a new container is started from it to create a new volume location and assign it to this directory in the container. This means any files that are put in there in the container, will outlive the container until the volume itself is manually deleted. Volumes need to be manually deleted if you don’t want to use them anymore. After all, that is their purpose, to store data for future use. So the need to manually delete volumes is a nice insurance policy. Let’s see this in action.
With our Docker environment cleaned up at the moment, we can list the volumes and see there are none.
docker-volumes> docker volume ls DRIVER VOLUME NAME
Let’s now run a docker container like so.
docker-volumes> docker container run -d --name my-database -e MYSQL_ALLOW_EMPTY_PASSWORD=True mysql 9308a2d6d73195a517f039107b13d858334e9304a5a169d8b08abd8bdb99dbb1
Sure enough, we have a running MySql container.
Now let’s notice something interesting. If we list out the volumes again, there is one present. This is due to the VOLUME command in the Dockerfile for MySql.
docker-volumes> docker volume ls DRIVER VOLUME NAME local 4ee2bfdd76937afa4e431a1a931cb30e8aaff292eb5f5fda8560b915f4846cd6
You can see the volume as well in the image if you inspect using docker image inspect mysql. Contained within the large amount of output you see will be the following snippet showing the volume location.
"Volumes": {
"/var/lib/mysql": {}
},
So we can tell that the config that came from the Dockerfile assigned a volume to that path when the image was built.
Taking a close look at this, we can also inspect the running container using docker container inspect my-database. Contained within the output once again is a section that displays the Mounts.
"Mounts": [
{
"Type": "volume",
"Name": "4ee2bfdd76937afa4e431a1a931cb30e8aaff292eb5f5fda8560b915f4846cd6",
"Source": "/var/lib/docker/volumes/4ee2bfdd76937afa4e431a1a931cb30e8aaff292eb5f5fda8560b915f4846cd6/_data",
"Destination": "/var/lib/mysql",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
What the above represents is the running container getting its own unique location on the host to store that data at Source. In the background this location is mapped, or mounted, to the Destination in the container of /var/lib/mysql. So the MySql engine simply believes it is writing data to /var/lib/mysql, however, the data is actually being stored at that long path in the Source on the host. So if you populate that database with a large amount of data, the size of the container stays the same.
You can also inspect the volume itself with docker volume inspect
docker-volumes> docker volume inspect 4ee2bfdd76937afa4e431a1a931cb30e8aaff292eb5f5fda8560b915f4846cd6
[
{
"CreatedAt": "2020-10-21T22:19:22Z",
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/4ee2bfdd76937afa4e431a1a931cb30e8aaff292eb5f5fda8560b915f4846cd6/_data",
"Name": "4ee2bfdd76937afa4e431a1a931cb30e8aaff292eb5f5fda8560b915f4846cd6",
"Options": null,
"Scope": "local"
}
]
Named Volumes Are Better
The section above showed how MySql containers store their data using volumes. You might have noticed, it’s not that user friendly with those super long ids for the mount points. A more friendly way to do this is by using named volumes when you run a container using the syntax -v volume-name:/path/in/container. Let’s redo the above with this approach.
docker-volumes> docker container run -d --name my-database -e MYSQL_ALLOW_EMPTY_PASSWORD=True -v my-volume:/var/lib/mysql mysql 44f34578e6e8eb359725320ded595171e299509b3c6c1184e1e0f4c7ea615c52 docker-volumes> docker volume ls DRIVER VOLUME NAME local my-volume
Notice the more user-friendly name for the volume. We can see the same when inspecting the container and the volume itself.
docker-volumes> docker container inspect my-database
"Mounts": [
{
"Type": "volume",
"Name": "my-volume",
"Source": "/var/lib/docker/volumes/my-volume/_data",
"Destination": "/var/lib/mysql",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
docker-volumes> docker volume inspect my-volume
[
{
"CreatedAt": "2020-10-21T22:57:25Z",
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/my-volume/_data",
"Name": "my-volume",
"Options": null,
"Scope": "local"
}
]
Now here is something else to consider. With the first approach, anytime you start a new container, a new volume gets created with a random id. So if you start 10 containers over the course of a few days, you’re also going to have 10 different volumes with random ids and you won’t know what goes where. With the named volume approach, you can spin up as many containers you like and make use of the same volume, keeping your data consistent and making keeping track of your volumes much easier.
Let’s add a new table to the first MySql container.
docker-volumes> docker exec -it my-database bash root@44f34578e6e8:/# mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8 Server version: 8.0.22 MySQL Community Server - GPL Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> create database NEWDATABASE; Query OK, 1 row affected (0.02 sec) mysql> show databases; +--------------------+ | Database | +--------------------+ | NEWDATABASE | | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 5 rows in set (0.01 sec)
Now we can create a new MySql container, and use the same volume.
docker-volumes> docker container run -d --name my-OTHER-database -e MYSQL_ALLOW_EMPTY_PASSWORD=True -v my-volume:/var/lib/mysql mysql ad1094d0d7b696059f652b1d6d949c5f83b1d554d43dd38f14750d67f6cfc5fa docker-volumes> docker volume ls DRIVER VOLUME NAME local my-volume
Still, just one volume listed above, with a user-friendly name to boot.
Let’s see if the new container has the same databases.
docker-volumes> docker exec -it my-OTHER-database bash root@ad1094d0d7b6:/# mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8 Server version: 8.0.22 MySQL Community Server - GPL Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | NEWDATABASE | | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 5 rows in set (0.00 sec)
Sure enough, that NEWDATABASE is still there! This would not have been the case if we did not specify a named volume.
Additional Docker Volume Resources
- Docker Storage Volumes (docs.docker.com)
- Docker Data Volumes Tutorial (rominirani.com)
- How To Share Data Between The Docker Container And The Host (digitalocean.com)
- Docker Volumes Tutorial (buildvirtual.net)
- Get Started With Docker Volumes (infoworld.com)
- Docker Volume How To (decodingdevops.com)
- Docker_Volumes_Mounting (linuxhint.com)
- Understanding Volumes Docker (blog.container-solutions.com)
- Understanding And Managing Docker Container Volumes (ionos.com)
- Data Containers And Named Volumes (boxboat.com)
- Docker Volumes Introduction (stephenafamo.com)
Docker Volume Summary
Containers are by design immutable and temporary. We spin them up, use them, and then tear them down. This is what is known as immutable infrastructure meaning containers can be redeployed, but never changed. This is good, but we need a way to deal with state, unique data, and databases. Docker provides two ways to handle persisting data with Volumes being the topic we covered in this tutorial. Volumes make a special location outside of the container UFS to handle this. Next up we’ll look at Bind Mounts in Docker.
- Can use the VOLUME command in a Dockerfile
- May use with command line like:
docker run -v my-volume:/path/in/container - Bypasses Union File System and stores in alt location on host
- Includes it’s own management commands under
docker volume - Connect to none, one, or multiple containers at once
- Not subject to commit, save, or export commands
- You can use with an assigned name, or just its unique id
