Asked  6 Months ago    Answers:  5   Viewed   33 times

I managed to find the containers under directory /var/lib/docker/containers, but I can't find the images.

What are the directories and files under /var/lib/docker?

 Answers

21

The contents of the /var/lib/docker directory vary depending on the driver Docker is using for storage.

By default this will be aufs but can fall back to overlay, overlay2, btrfs, devicemapper or zfs depending on your kernel support. In most places this will be aufs but the RedHats went with devicemapper.

You can manually set the storage driver with the -s or --storage-driver= option to the Docker daemon.

  • /var/lib/docker/{driver-name} will contain the driver specific storage for contents of the images.
  • /var/lib/docker/graph/<id> now only contains metadata about the image, in the json and layersize files.

In the case of aufs:

  • /var/lib/docker/aufs/diff/<id> has the file contents of the images.
  • /var/lib/docker/repositories-aufs is a JSON file containing local image information. This can be viewed with the command docker images.

In the case of devicemapper:

  • /var/lib/docker/devicemapper/devicemapper/data stores the images
  • /var/lib/docker/devicemapper/devicemapper/metadata the metadata
  • Note these files are thin provisioned "sparse" files so aren't as big as they seem.
Tuesday, June 1, 2021
 
revive
answered 6 Months ago
34

Ok there's a couple of points that need to be addressed here.

Lets start with what a docker volume is(Try to not think about your macbook or the vagrant machine at this point. Just be mindful of the fact that the dockers use a different filesystem, where ever it may reside at this point ): Maybe imagine it like this, in and of itself every volume in Docker is just a part of the internal file system docker uses. The containers can use theses volumes, like they were "small harddrives" that can be mounted by them and also shared between them (or mounted by two of them at the same time, like mounting a super fast version of some ftp server to two clients or whatever :P ).

In principal you can declare these volumes ( still not thinking about your computer/vagrant itself, just the dockers ;) ) via the Dockerfile's VOLUME instruction. Standard example, run one webserver container like so:

FROM: nginx
VOLUME /www

Now everything that goes into /www can in theory be mounted and unmounted from a container and also mounted to multiple containers. Now Nginx alone is boring, so we want to have php run over the files that nginx stores to produce some more fun content. => We need to mount that volume into some php-fpm container. Ergo in our compose file we'd do this

web:
  image: nginx
php:
  image: php-fpm
  volumes_from:
    - web

=> voila! every folder declared by a VOLUME directive in the nginx/web container will be visible in the php one. Important point to note here, whatever is in nginx's /www, will override whatever php has in /www. If you put the :ro, php can't even write to that folder :)

Now moving close to your issue, there's a second way to declare volumes, that does not require them being declared in the Dockerfile. This can be done by mounting volumes from the host (in this case your vagrant/boo2docker thingy). Let's discuss this as though we're running on a native Linux first.

If you were to put something like:

volumes:
 - /home/myuser/folder:/folder

in your docker-compose.yml, then this will mean that /home/myuser/folder will now be mounted into the docker. It will override whatever the docker has in /folder and just like the /www also be accessible from the thing that declared it. Now the Linux machine the docker daemon is running on.

So much for the theory :), in fact you probably just need the following advice to get your stuff going :):

The way boot2docker/docker-machine/kitematic and all these things deal with the issue is simply, that they first of all just mount a volume in the vagrant machine to the docker containers, and them simply also mount this thing into your Mac file system, hoping it will all work out :P

Now for the practical problem all of us using this (or just trying to help their coworkers into the world of sweet sweet Docker :P) on Mac are facing is permissions. I mean think about it ( root or some other user handles files in the container,the user vagrant might handle files in the vagrant host and then your Mac user "skalfyfan" handles those files in Mac. They all have different user id's and whatnot => many problems ensue with that, and somewhat depending on what you're actually running in Docker. Mysql and Apache are especially painful, because they do not run as root within the container. This means, they often have trouble writing to the Mac file system.

Before trying the second approach below, simply try putting your container volumes under you Mac home directory. This will resolve issues with MySQL in most cases as I have found over time. Btw: No need to declare full paths to volumes ./folder is fine and read relative to the place your docker-compose.yml resides!

Just put the compose-yml in your Mac users folder, that's all that matters. No chmod 777 -R :P will help you here, it just needs to be under your home folder :)

Still some apps ( Apache for example ) will still give you a hard time. The fact that the user id of whatever runs in the container differs from your Mac user id will make your life hell. In order to get around this, you need to adjust the user id as well as the user group in a way that doesn't conflict with your Mac's permissions. The group you want on a Mac is staff, a UID that works would be for example 1000. Hence you could put this at the end of your Dockerfile:

RUN usermod -u 1000 www-data
RUN usermod -G staff www-data

or

RUN usermod -u 1000 mysql
RUN usermod -G staff mysql

So as you have now learnt:

In theory I think I should be required to COPY my local mac file system folder ./src to my VM first, and then I do this volume declaration. It seems docker-compose is magically doing it all at once though?

Right on, it does that :)

Lastly, we can see that I'm creating a data-only container to persist my mysql data. I've declared: volumes: - /var/lib/mysql

This one you got wrong :) As explained, if you don't give a host folder, then Docker will persist this path. But only for this container and all will stay within the docker file system. Nothing is written to the host at all! This will always only happen if you give a host folder before the container folder!

Hope this helped :)

Wednesday, August 11, 2021
 
rkeet
answered 4 Months ago
20

All you need is a docker volume in order to persist the log files. So in the same directory as your docker-compose.yml create a logs directory, then define a volume mount. When defining a mount remember the syntax is <host_machine_directy>:<container_directory>.

Give the following volume a try and let me know what you get back.

version: '3'
services:
  myapp:
    build: .
    image: myapp
    ports:
      - "9001:9001"
    volumes:
      - ./logs:/home/logs

Also worth noting that persistence goes both ways with this approach. Any changes made to the files from within the container are reflected back onto the host. Any changes from the host, are also reflected inside the container.

Thursday, August 12, 2021
 
Aniket Thakur
answered 4 Months ago
36

Okay! I found out the reason for issue.

DOCKER BUILD PROCESS

When we build a docker image, while creating a image several other intermediate images are generated in the process. We never see them in docker images because with the generation of next intermediate image the earlier image is removed.And in the end we have only one which is the final image.

The tag we provide using -t or --tag is for the final build, and obviously no intermediate container is tagged with the same.

ISSUE EXPLANATION

When we try to build a docker image with Dockerfile sometimes the process is not successfully completed with a similar message like Successfully built image with IMAGEID

So it is so obvious that the build which has failed will not be listed in docker images

Now, the image with tag <none> is some other image (intermediate). This creates a confusion that the image exists but without a tag, but the image is actually not what the final build should be, hence not tagged.

Saturday, September 25, 2021
 
Bitwise
answered 2 Months ago
91

That Docker Hub history view doesn't show the actual Dockerfile; instead, it shows content essentially extracted from the docker history of the image. That doesn't preserve the specific details you're looking for: it doesn't remember the names of base images, or the build-context file names of things that get ADDed or COPYed in.

Chasing through GitHub and Docker Hub links, the golang:*-buster Dockerfile is built FROM buildpack-deps:...-scm; buildpack-deps:buster-scm is FROM buildpack-deps:buster-curl; that is FROM debian:buster; and that has a very simple Dockerfile (quoted here in its entirety):

FROM scratch
ADD rootfs.tar.xz /
CMD ["bash"]

FROM scratch starts from a completely totally empty image; that is the base of the Docker image tree (and what tells docker history and similar tools to stop). The ADD line unpacks a tar file of a Debian system image.

If you look at docker history or the Docker Hub history view you cite, you should be able to see these same steps happening. The ADD file:4b0... in / corresponds to the ADD rootfs.tar.gz /, and the second line is the CMD ["bash"]. It is not split up by Dockerfile or image, and the original filenames from ADD aren't saved. (You couldn't reproduce the image anyways without the contents of the rootfs.tar.gz, so it's merely slightly helpful to know its filename but not essential.)

The ADD file:hash in /path syntax is not standard Dockerfile syntax (the word in in particular is not part of it). I'm not sure there's a reliable way to translate from the host file or URL to the hash, but building the image and looking at its docker history would tell you (assuming you've got a perfect match for the file metadata). There's no way to get back to the original filename or syntax, and definitely no way to get back to the file contents.

Monday, October 11, 2021
 
huhushow
answered 2 Months ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :  
Share