Monitoring Docker Containers

Monitoring Docker Containers

This tutorial will focus on seeing what is going on inside the containers you are running. There are several docker commands to do this with the most common ones being docker container top, docker container inspect, and docker container stats. The top command is a process list in one container. The inspect command is quite interesting as it provides the details of how a container started and how it is configured. The stats command can be used to look at all running container’s stats or one at a time in a real-time stream output to the terminal. Let’s look at some of these commands in action.


Spinning Up A Couple Containers

To test out the monitoring commands for docker containers, first, we need to spin up a few containers to work with. We’ve seen how to run multiple containers, so let’s bring a couple up again. We can launch an Nginx container, a MySQL container, and then list them in their running state.

$ docker container run -d --name nginx nginx
d4f849b858a9df134f7f907c6e929c2ef1e89b86df8f851c8f209f6d6ac7a9e1

$ docker container run -d --name mysql -e MYSQL_RANDOM_ROOT_PASSWORD=true mysql
e9263551f40fb30deb0b09434bf26697fce3eaf3370141d01c209d997c73885b

$ docker container ls
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                 NAMES
e9263551f40f        mysql               "docker-entrypoint.s…"   26 seconds ago      Up 25 seconds       3306/tcp, 33060/tcp   mysql
d4f849b858a9        nginx               "/docker-entrypoint.…"   49 seconds ago      Up 48 seconds       80/tcp                nginx

Docker Container Top

In Linux, the top command is used to show the Linux processes. It provides a dynamic real-time view of the running system. We can do a similar thing in Docker.

$ docker container top nginx
PID                 USER                TIME                COMMAND
1493                root                0:00                nginx: master process nginx -g daemon off;
1551                101                 0:00                nginx: worker process

$ docker container top mysql
PID                 USER                TIME                COMMAND
1569                999                 0:01                mysqld

Docker Container Inspect

The inspect command can be run on one or more images or containers. Either the name or id can be used to identify an image or container. Here we inspect the Nginx container in our environment. The output is in JSON format and contains literally all of the data about how the container was started.

$ docker container inspect nginx
[
    {
        "Id": "d4f849b858a9df134f7f907c6e929c2ef1e89b86df8f851c8f209f6d6ac7a9e1",
        "Created": "2020-10-01T14:57:13.6068591Z",
        "Path": "/docker-entrypoint.sh",
        "Args": [
            "nginx",
            "-g",
            "daemon off;"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 1493,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-10-01T14:57:14.3647252Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:7e4d58f0e5f3b60077e9a5d96b4be1b974b5a484f54f9393000a99f3b6816e3d",
        "ResolvConfPath": "/var/lib/docker/containers/d4f849b858a9df134f7f907c6e929c2ef1e89b86df8f851c8f209f6d6ac7a9e1/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/d4f849b858a9df134f7f907c6e929c2ef1e89b86df8f851c8f209f6d6ac7a9e1/hostname",
        "HostsPath": "/var/lib/docker/containers/d4f849b858a9df134f7f907c6e929c2ef1e89b86df8f851c8f209f6d6ac7a9e1/hosts",
        "LogPath": "/var/lib/docker/containers/d4f849b858a9df134f7f907c6e929c2ef1e89b86df8f851c8f209f6d6ac7a9e1/d4f849b858a9df134f7f907c6e929c2ef1e89b86df8f851c8f209f6d6ac7a9e1-json.log",
        "Name": "/nginx",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                30,
                149
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/6a6f3b5f81d2b454cbcc7651b3c40c90f8da991e9b55add4cfa8de2e3a57cd4d-init/diff:/var/lib/docker/overlay2/b873da3ee35fed4ed5b663d42705f29857e467ef7fc70d4f01b34e8593c90a2d/diff:/var/lib/docker/overlay2/c37b3ca864167a02502bc473d368823bdf2fdddf7ef3ff0fda126aae98ccf5c3/diff:/var/lib/docker/overlay2/d0b819fd09a7a62a2fc9858b4a015cd5594a240caefce77cad9a6b0f596f0dca/diff:/var/lib/docker/overlay2/5d3260b5420b271959560369b58594ca2be7fd8ae7bd74822c24c765c8ebeafc/diff:/var/lib/docker/overlay2/d874cfda684973a0d7f77e451987cd726723eab53aabf2ef285000ff32101e3d/diff",
                "MergedDir": "/var/lib/docker/overlay2/6a6f3b5f81d2b454cbcc7651b3c40c90f8da991e9b55add4cfa8de2e3a57cd4d/merged",
                "UpperDir": "/var/lib/docker/overlay2/6a6f3b5f81d2b454cbcc7651b3c40c90f8da991e9b55add4cfa8de2e3a57cd4d/diff",
                "WorkDir": "/var/lib/docker/overlay2/6a6f3b5f81d2b454cbcc7651b3c40c90f8da991e9b55add4cfa8de2e3a57cd4d/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "d4f849b858a9",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NGINX_VERSION=1.19.2",
                "NJS_VERSION=0.4.3",
                "PKG_RELEASE=1~buster"
            ],
            "Cmd": [
                "nginx",
                "-g",
                "daemon off;"
            ],
            "Image": "nginx",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "maintainer": "NGINX Docker Maintainers <docker-maint@nginx.com>"
            },
            "StopSignal": "SIGTERM"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "ca6953bd7a7a11a50091be348a1414f9891d28fef9dcc241cc96332651fa0132",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "80/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/ca6953bd7a7a",
            "SecondaryIPAddresses": null,
            "EndpointID": "2044cf6b4ded03757b71a40d21b727f0287ac1b1c1f5edbe1f767d5fffa67456",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "0099dce04d4e730d272d5c774970a65704c8df6869dfadba2c2363e532c4caf9",
                    "EndpointID": "2044cf6b4ded03757b71a40d21b727f0287ac1b1c1f5edbe1f767d5fffa67456",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

Filtering Inspect Output

Since the output is so large, you can pipe the results to either Select-String in Powershell, or grep in bash. For example:

$ docker container inspect nginx | Select-String -Pattern IPAddress

            "SecondaryIPAddresses": null,
            "IPAddress": "172.17.0.2",
                    "IPAddress": "172.17.0.2",

Docker Container Stats

When using the stats command, you can view all running containers together, or specify just one container you want to look at. The output is live streaming data that updates in real-time with the cpu, memory, io, and so on.

$ docker container stats
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
e9263551f40f        mysql               0.13%               332MiB / 6.087GiB     5.33%               866B / 0B           0B / 0B             38
d4f849b858a9        nginx               0.00%               3.973MiB / 6.087GiB   0.06%               1.05kB / 0B         0B / 0B             2
$ docker container stats nginx
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
d4f849b858a9        nginx               0.00%               3.973MiB / 6.087GiB   0.06%               1.05kB / 0B         0B / 0B             2
$ docker container stats mysql
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
e9263551f40f        mysql               1.98%               332MiB / 6.087GiB   5.33%               866B / 0B           0B / 0B             38

Learn More

Docker Monitoring Commands Summary

We now know how to use the docker container top command to list all the processes that will be running in a single container. We also learned about docker container inspect to see how the container was run and what its options and configuration are. The docker container stats command showed gave us a live updating output to the terminal of CPU usage, Memory usage, Net IO, and Block IO.

docker container top Display the running processes of a container
docker container inspect Display detailed information on one or more containers
docker container stats Display a live stream of container(s) resource usage statistics