Docker containers inherit their image’s filesystem upon initial creation. The container’s filesystem can diverge from the image’s content after creation if changes are made to the container’s writable layer.
The ability to inspect changes in the filesystem can help you debug issues and investigate missing data. Files written to the container’s filesystem will be lost when that container’s restarted or destroyed. In some cases you might be unintentionally performing writes that should be directed to a persistent volume instead.
In this article, we’ll share some techniques for identifying modifications made to container filesystems. You can use this information to track container activity and investigate discrepancies between different environments.
Using Docker Diff
The Docker CLI has a built-in command for this purpose. Running
docker diff will enumerate all the changes made to files and directories within a particular container. It accepts the ID or name of the container you want to inspect.
Start a simple container for demonstration purposes:
$ docker run -it --name alpine alpine:latest sh
This will start a new container running the Alpine image. Your terminal window will be attached to the container. Keep this window open.
docker diff in a second terminal session:
$ docker diff alpine
No output is produced as the fresh container still exactly matches the Alpine image’s filesystem.
Switch back to your first terminal that’s still attached to the container. Run some commands that add, delete, or change files:
/ # touch demo / # touch example
Return to your first shell window. Repeat the
docker diff command to see the changes that have been made:
$ docker diff alpine A /example C /root C /root/.ash_history A /demo
Now the two new files are visible in the command’s output. Running
touch also added entries to the container’s shell history. The modification to
/root/.ash_history is reported too, as well as the consequent change to the
Interpreting Docker Diff Output
Each line of output produced by
docker diff is prefixed by either
D depending on the type of change that occurred:
A– The file or directory has been added to the container’s filesystem. It doesn’t exist in the original image.
C– The content of the file or directory is different to the version in the image.
D– A path that’s present in the image has been deleted from the container.
This information enables you to quickly look for discrepancies that could explain missing data or unexpected behavior.
Any filesystem changes that don’t show up in the output will have affected paths that are mapped to a Docker volume. These files are stored outside the container, independently of its filesystem, so they’re not surfaced by
docker diff .
Viewing Content in the Original Image
It can be helpful to compare the file version that’s included in the original image. You can do this by creating a new container and then copying out the file. Use
docker create instead of
docker run so the container is created but not started. This will ensure that modifications caused by the image’s entrypoint script don’t affect the result.
$ docker create --name temp-container my-image:latest $ docker cp temp-container:/path/to/file copied-file $ docker rm temp-container
An alternative way is to run the container with a custom entrypoint that emits the file’s content to the standard output stream. Include the
--rm flag so the container’s automatically removed after the command completes:
$ docker run --rm --entrypoint cat alpine:latest /path/to/file > copied-file
Comparing Differences Between Two Containers
Sometimes you might need to compare the differences between two different containers running the same image. First use
docker diff to get the individual change lists:
$ docker diff first-container > first $ docker diff second-container > second
Now you can use standard Unix tools such as the
diff command to find the differences in the two change lists:
diff first second 3c3 < A /a-new-file --- > A /demo
The diff shows that
/a-new-file has been removed in the second container while
/demo has been added.
Another way to compare containers is to use
docker export and the tar command to access a comprehensive file list:
$ docker export demo-container | tar tv -rwxr-xr-x 0/0 0 2022-06-21 14:21 .dockerenv drwxr-xr-x 0/0 0 2022-05-23 17:51 bin/ lrwxrwxrwx 0/0 0 2022-05-23 17:51 bin/arch -> /bin/busybox lrwxrwxrwx 0/0 0 2022-05-23 17:51 bin/ash -> /bin/busybox lrwxrwxrwx 0/0 0 2022-05-23 17:51 bin/base64 -> /bin/busybox lrwxrwxrwx 0/0 0 2022-05-23 17:51 bin/bbconfig -> /bin/busybox ...
Repeat this command for both your containers. You can then use
diff to compare the two lists. The enhanced output of tar can help you track down when files changed as modification times and permissions are displayed.
The filesystems of Docker containers should closely resemble the image they were created from. Discrepancies can occur when software within the container writes content directly to the filesystem. Users can be another source of differences, when mechanisms such as
docker exec are used to run commands within a running container.
docker diff utility provides a convenient way to view these changes. Although you can’t see why a file was added or removed, the ability to retrieve a list of modifications can be invaluable in understanding why particular behavior is being observed.
In this article we’ve focused on examining the changes made to running containers. You can use different tools such as
docker history and Dive to inspect the changes made to layers within a Docker image. The two approaches can be used in tandem, letting you work out the exact point a file was added or modified within a container’s layer stack.