vegibit

What Is A Docker Volume

Click to share! ⬇️

What Is A Docker Volume

Docker has a feature known as Volumes, that allow developers to persist data in use with containers. They are entirely managed by the Docker Engine making them seamless to the end-user. Docker volumes are a very important and useful concept and in this tutorial, we’ll learn all about Docker volumes, how to create volumes, how to list volumes, and how to delete volumes. We’ll also see how to share a volume among containers by spinning up several containers which all make use of the same volume for data sharing.

About Docker Volumes

To get started with Docker Volumes, we can simply look at the commands available under the docker volume management command. Like other things in Docker, we can create, inspect, list, prune, and remove volumes.

docker-volumes> docker volume

Usage:  docker volume COMMAND

Manage volumes

Commands:
  create      Create a volume
  inspect     Display detailed information on one or more volumes
  ls          List volumes
  prune       Remove all unused local volumes
  rm          Remove one or more volumes

Create A Docker Volume

The best way to learn about volumes in Docker is to jump right into it and create one. Here we create a volume with the name of my-volume.

docker-volumes> docker volume create my-volume
my-volume

Listing Docker Volumes

Using docker volume ls shows us our new volume.

docker-volumes> docker volume ls
DRIVER              VOLUME NAME
local               my-volume

Inspecting A Docker Volume

docker-volumes> docker volume inspect my-volume
[
    {
        "CreatedAt": "2020-10-20T17:35:06Z",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/my-volume/_data",
        "Name": "my-volume",
        "Options": {},
        "Scope": "local"
    }
]

Real World Docker Volume Example

For this test we will make use of Jenkins, an open-source automation server that enables software engineers to reliably build, test, and deploy their software using continuous integration and continuous delivery. Jenkins is a server-based system that runs in servlet containers such as Apache Tomcat. The goal here is to show that we can store data the Jenkins server generates while working in one container and then leverage a volume to use that data in subsequent containers running an instance of Jenkins.

Pull The Jenkins Image From Docker Hub

docker-volumes> docker pull jenkins
Using default tag: latest
latest: Pulling from library/jenkins
55cbf04beb70: Pull complete
...
1322ea3e7bfd: Pull complete
Digest: sha256:eeb4850eb65f2d92500e421b430ed1ec58a7ac909e91f518926e02473904f668
Status: Downloaded newer image for jenkins:latest
docker.io/library/jenkins:latest

Run Jenkins In A Container

Now that we have the image in our local repository, we can easily spin up an instance of the Jenkins server in a container. You’ll want to be at least a little bit familiar with running containers since the command used here makes use of several options including –name, -v, and -p. Check some of the prior Docker tutorials if you need a refresher. For this tutorial, the main concept to be aware of is that we are attaching a volume of my-volume to this container on startup (using -v my-volume:/var/jenkins_home).

docker-volumes> docker container run --name my-jenkins -v my-volume:/var/jenkins_home -p 8080:8080 -p 50000:50000 jenkins
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
...
...
INFO:

*************************************************************
*************************************************************
*************************************************************

Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

039b2e1d3609465ead266da3a660eaf1

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

*************************************************************
*************************************************************
*************************************************************

Oct 20, 2020 5:53:51 PM hudson.model.UpdateSite updateData
INFO: Obtained the latest update center data file for UpdateSource default
...
...
INFO: Jenkins is fully up and running
--> setting agent port for jnlp
--> setting agent port for jnlp... done

The Jenkins server is now running. We can visit http://localhost:8080 to see it working. In addition, we can copy and paste the generated password from the above output to log in right away.

docker jenkins getting started

On an initial install of Jenkins, there is some setup involved. We choose “Select plugins to install”

docker customize jenkins

Then on the Getting Started page just choose None in the upper left, then install at the bottom right.

install jenkins docker

On this page just continue as admin.

jenkins continue as admin

When Jenkins setup is complete, click on the “Start using Jenkins” button.

jenkins in docker is ready

Docker Persistent Data With Volumes

The goal here is to demonstrate how data from this instance of Jenkins in this container will be available to future containers running Jenkins that make use of the volume we have created. In the following few steps, we set up a new job in Jenkins.

jenkins docker create new job

The job name can be whatever you like, we simply choose ExampleJob, select Freestyle project, then click ok.

jenkins enter job name

Just as an example, we can add a build step by adding a shell command.

jenkins apply build step

Now we have a fully operational Example Job in Jenkins server.

jenkins example job docker

Create A New Jenkins Container

In this next step, we’ll create a second container and again run an instance of Jenkins in it. Since we already have one instance running which uses ports 8080:8080 and 50000:50000, we need to specify new ports so they don’t collide. Also, note that this container has a new and different name of my-OTHER-jenkins, however, the volume is the same(-v my-volume:/var/jenkins_home).

docker-volumes> docker container run --name my-OTHER-jenkins -v my-volume:/var/jenkins_home -p 9090:8080 -p 60000:50000 jenkins
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
...
...
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@54b6b3cf: defining beans [filter,legacy]; root of factory hierarchy
Oct 20, 2020 6:18:31 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
--> setting agent port for jnlp
--> setting agent port for jnlp... done

shared volume with second jenkins server

Now open a new browser tab and visit http://localhost:9090, and voila! We see another instance of Jenkins. Do notice that we are presented with a login screen right away. There are no setup steps required. This shows us that our volume in Docker is working correctly. All of the work we did to set up the first instance of Jenkins is shared with this new instance of Jenkins thanks to the volume we had created named my-volume.

jenkins data shared to new container

Let’s just confirm that we have two Jenkins containers running, and sure enough, there they are.

docker volumes> docker container ls
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                              NAMES
ca63090c2877        jenkins             "/bin/tini -- /usr/l…"   7 minutes ago       Up 7 minutes        0.0.0.0:9090->8080/tcp, 0.0.0.0:60000->50000/tcp   my-OTHER-jenkins
e87acec5997f        jenkins             "/bin/tini -- /usr/l…"   32 minutes ago      Up 32 minutes       0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp   my-jenkins

Let’s list the volumes we have as well. The my-volume volume is working perfectly for us.

docker volumes> docker volume ls
DRIVER              VOLUME NAME
local               my-volume

Had we not created the volume earlier, when we spun up a second instance of Jenkins it would have asked us for all of the setup information all over again. This volume will continue to work as we set up and tear down containers.

Let’s remove all running containers.

docker volumes> docker rm $(docker ps -a -q)
Error response from daemon: You cannot remove a running container ca63090c2877cef6cdc6c5e2338471fc347f5f5be48766a997feee7cf6a2ec39. Stop the container before attempting removal or force remove
Error response from daemon: You cannot remove a running container e87acec5997fe4eeb73c4bf30ba23d416dd6d2723e97f7270f84d7afdffdd439. Stop the container before attempting removal or force remove

Whoops. We need to stop the containers before we remove them.

docker volumes> docker stop $(docker ps -a -q)
ca63090c2877
e87acec5997f

Now we can remove all containers since they are stopped.

docker volumes> docker rm $(docker ps -a -q)
ca63090c2877
e87acec5997f

Listing the containers shows they are all gone.

docker volumes> docker container ls
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

A Third Container

Imagine we just finished for the day, and now we come back tomorrow and want to run Jenkins once again. We can spin up yet another container, and it should continue to have access to the data that was generated originally and stored in my-volume. 

docker volumes> docker container run --name jenkins-THE-THIRD -v my-volume:/var/jenkins_home -p 8080:8080 -p 50000:50000 jenkins
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
Oct 20, 2020 6:29:44 PM Main deleteWinstoneTempContents
...
...
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@5517b224: defining beans [filter,legacy]; root of factory hierarchy
Oct 20, 2020 6:29:48 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
--> setting agent port for jnlp
--> setting agent port for jnlp... done

We can open up the Jenkins server and once again, we are right back into our original workflow with the same ExampleJob still intact.

jenkins third container docker

Confirming this is in fact an entirely new container.

jenkins the third

All of this was made possible by our humble volume, my-volume.

docker volumes> docker volume ls
DRIVER              VOLUME NAME
local               my-volume

Specify VOLUME In Docker File

In the example above we created and specified the volume to use entirely from the command line. You can also specify volumes in a Docker File using the VOLUME command. For example, within the official MySql Docker file is the following line.

VOLUME /var/lib/mysql

You can see that line in the Docker file here. This is the default location of the MySql databases and this command within the Docker File tells Docker that when a new container is started from it to create a new volume location and assign it to this directory in the container. This means any files that are put in there in the container, will outlive the container until the volume itself is manually deleted. Volumes need to be manually deleted if you don’t want to use them anymore. After all, that is their purpose, to store data for future use. So the need to manually delete volumes is a nice insurance policy. Let’s see this in action.

With our Docker environment cleaned up at the moment, we can list the volumes and see there are none.

docker-volumes> docker volume ls
DRIVER              VOLUME NAME

Let’s now run a docker container like so.

docker-volumes> docker container run -d --name my-database -e MYSQL_ALLOW_EMPTY_PASSWORD=True mysql
9308a2d6d73195a517f039107b13d858334e9304a5a169d8b08abd8bdb99dbb1

Sure enough, we have a running MySql container.
docker mysql container running

Now let’s notice something interesting. If we list out the volumes again, there is one present. This is due to the VOLUME command in the Dockerfile for MySql.

docker-volumes> docker volume ls
DRIVER              VOLUME NAME
local               4ee2bfdd76937afa4e431a1a931cb30e8aaff292eb5f5fda8560b915f4846cd6

You can see the volume as well in the image if you inspect using docker image inspect mysql. Contained within the large amount of output you see will be the following snippet showing the volume location.

            "Volumes": {
                "/var/lib/mysql": {}
            },

So we can tell that the config that came from the Dockerfile assigned a volume to that path when the image was built.

Taking a close look at this, we can also inspect the running container using docker container inspect my-database. Contained within the output once again is a section that displays the Mounts.

"Mounts": [
    {
        "Type": "volume",
        "Name": "4ee2bfdd76937afa4e431a1a931cb30e8aaff292eb5f5fda8560b915f4846cd6",
        "Source": "/var/lib/docker/volumes/4ee2bfdd76937afa4e431a1a931cb30e8aaff292eb5f5fda8560b915f4846cd6/_data",
        "Destination": "/var/lib/mysql",
        "Driver": "local",
        "Mode": "",
        "RW": true,
        "Propagation": ""
    }
],

What the above represents is the running container getting its own unique location on the host to store that data at Source. In the background this location is mapped, or mounted, to the Destination in the container of /var/lib/mysql. So the MySql engine simply believes it is writing data to /var/lib/mysql, however, the data is actually being stored at that long path in the Source on the host. So if you populate that database with a large amount of data, the size of the container stays the same.

You can also inspect the volume itself with docker volume inspect and it will show you that same Mountpoint on the host.

docker-volumes> docker volume inspect 4ee2bfdd76937afa4e431a1a931cb30e8aaff292eb5f5fda8560b915f4846cd6
[
    {
        "CreatedAt": "2020-10-21T22:19:22Z",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/4ee2bfdd76937afa4e431a1a931cb30e8aaff292eb5f5fda8560b915f4846cd6/_data",
        "Name": "4ee2bfdd76937afa4e431a1a931cb30e8aaff292eb5f5fda8560b915f4846cd6",
        "Options": null,
        "Scope": "local"
    }
]

Named Volumes Are Better

The section above showed how MySql containers store their data using volumes. You might have noticed, it’s not that user friendly with those super long ids for the mount points. A more friendly way to do this is by using named volumes when you run a container using the syntax -v volume-name:/path/in/container. Let’s redo the above with this approach.

docker-volumes> docker container run -d --name my-database -e MYSQL_ALLOW_EMPTY_PASSWORD=True -v my-volume:/var/lib/mysql mysql
44f34578e6e8eb359725320ded595171e299509b3c6c1184e1e0f4c7ea615c52

docker-volumes> docker volume ls
DRIVER              VOLUME NAME
local               my-volume

Notice the more user-friendly name for the volume. We can see the same when inspecting the container and the volume itself.

docker-volumes> docker container inspect my-database
"Mounts": [
    {
        "Type": "volume",
        "Name": "my-volume",
        "Source": "/var/lib/docker/volumes/my-volume/_data",
        "Destination": "/var/lib/mysql",
        "Driver": "local",
        "Mode": "z",
        "RW": true,
        "Propagation": ""
    }
],
docker-volumes> docker volume inspect my-volume
[
    {
        "CreatedAt": "2020-10-21T22:57:25Z",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/my-volume/_data",
        "Name": "my-volume",
        "Options": null,
        "Scope": "local"
    }
]

Now here is something else to consider. With the first approach, anytime you start a new container, a new volume gets created with a random id. So if you start 10 containers over the course of a few days, you’re also going to have 10 different volumes with random ids and you won’t know what goes where. With the named volume approach, you can spin up as many containers you like and make use of the same volume, keeping your data consistent and making keeping track of your volumes much easier.

Let’s add a new table to the first MySql container.

docker-volumes> docker exec -it my-database bash
root@44f34578e6e8:/# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.22 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database NEWDATABASE;
Query OK, 1 row affected (0.02 sec)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| NEWDATABASE        |
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.01 sec)

Now we can create a new MySql container, and use the same volume.

docker-volumes> docker container run -d --name my-OTHER-database -e MYSQL_ALLOW_EMPTY_PASSWORD=True -v my-volume:/var/lib/mysql mysql
ad1094d0d7b696059f652b1d6d949c5f83b1d554d43dd38f14750d67f6cfc5fa

docker-volumes> docker volume ls
DRIVER              VOLUME NAME
local               my-volume

Still, just one volume listed above, with a user-friendly name to boot.

Let’s see if the new container has the same databases.

docker-volumes> docker exec -it my-OTHER-database bash
root@ad1094d0d7b6:/# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.22 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| NEWDATABASE        |
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

Sure enough, that NEWDATABASE is still there! This would not have been the case if we did not specify a named volume.

Additional Docker Volume Resources

Docker Volume Summary

Containers are by design immutable and temporary. We spin them up, use them, and then tear them down. This is what is known as immutable infrastructure meaning containers can be redeployed, but never changed. This is good, but we need a way to deal with state, unique data, and databases. Docker provides two ways to handle persisting data with Volumes being the topic we covered in this tutorial. Volumes make a special location outside of the container UFS to handle this. Next up we’ll look at Bind Mounts in Docker.

  • Can use the VOLUME command in a Dockerfile
  • May use with command line like: docker run -v my-volume:/path/in/container
  • Bypasses Union File System and stores in alt location on host
  • Includes it’s own management commands under docker volume
  • Connect to none, one, or multiple containers at once
  • Not subject to commit, save, or export commands
  • You can use with an assigned name, or just its unique id
Click to share! ⬇️

How To Run Nginx Using DockerHow To Run Nginx Using Docker How To Run MongoDB In A Docker ContainerHow To Run MongoDB In A Docker Container Docker Multiple ContainersDocker Multiple Containers Monitoring Docker ContainersMonitoring Docker Containers How To Connect To A Docker ContainerHow To Connect To A Docker Container Docker Networking TutorialDocker Networking Tutorial