tahoe-lafs/trac

connect tahoe-lafs repo to Docker Hub #2421

New issue

Open

opened 2015-05-14 16:59:48 +00:00 by daira · 17 comments

daira commented

2015-05-14 16:59:48 +00:00

amontero wrote:

Before deeming the Docker Hub build as official we should link the DH automated build with the Github repo. As per https://docs.docker.com/docker-hub/builds/ it requires that the DH user that sets this up also to have rw access to the GH repo.
Since I've created 'tahoelafs' DH account as an organization, somebody with appropiate perms just ping me with their DH account and I'll promptly add him/her to organization owners.
Also, somebody more knowledgeable would point me which branches/tags to publish as Docker images. Once a Hub image is linked to the sources, you can make it generate several different tagged Docker images each one built from different branches or tags.

amontero wrote: > Before deeming the Docker Hub build as official we should link the DH automated build with the Github repo. As per <https://docs.docker.com/docker-hub/builds/> it requires that the DH user that sets this up also to have rw access to the GH repo. > Since I've created 'tahoelafs' DH account as an organization, somebody with appropiate perms just ping me with their DH account and I'll promptly add him/her to organization owners. > Also, somebody more knowledgeable would point me which branches/tags to publish as Docker images. Once a Hub image is linked to the sources, you can make it generate several different tagged Docker images each one built from different branches or tags.

daira added the

labels 2015-05-14 16:59:48 +00:00

daira added this to the undecided milestone 2015-05-14 16:59:48 +00:00

warner was assigned by daira

2015-05-14 16:59:48 +00:00

daira commented

2015-05-14 17:02:24 +00:00

Author

I've created the docker-tahoe github user. I have the password for that account. Brian has the permissions needed to give the docker-tahoe user temporary access to set the webhook. (This works exactly like the hook for Travis-CI; we want to revoke excess permissions after the hook has been set.)

To start with we want an image for the allmydata-tahoe-1.10.0 tag.

warner commented

2015-05-20 07:47:24 +00:00

I've created the "dockertahoe" user on hub.docker.com. amontero: please add it to the "tahoelafs" DH organization when you get a chance.

amontero commented

2015-05-27 02:04:31 +00:00

Owner

Done. Try creating (replacing) current tahoelafs/base automated build with the one you will be able to link to github. I will later check the generated image tags against the git tags and branches.

warner commented

2015-06-07 21:40:58 +00:00

Ok, I think I've got things wired up. https://registry.hub.docker.com/u/dockertahoe/tahoe-lafs/ shows a new Docker build each time we push a change to github master. My remaining questions:

Can we have this live in Dockerhub's tahoe-lafs/tahoe-lafs instead of dockertahoe/tahoe-lafs ? I can see that the Dockerhub "dockertahoe" account is a member of the Dockerhub "tahoe-lafs" organization, but I don't see any way to put images in that location.
The current build (of github "master") has a "Tag Name" of "latest". I assume we should offer a container with the current git trunk, and a different container with the most recent release. What is the Docker naming convention here? Does "latest" imply unreleased software? Should we use "latest" to point at e.g. 1.10.1, and "dev" or something to point at current git master?

I think I need amontero to answer these two.

Ok, I think I've got things wired up. <https://registry.hub.docker.com/u/dockertahoe/tahoe-lafs/> shows a new Docker build each time we push a change to github master. My remaining questions: * Can we have this live in Dockerhub's tahoe-lafs/tahoe-lafs instead of dockertahoe/tahoe-lafs ? I can see that the Dockerhub "dockertahoe" account is a member of the Dockerhub "tahoe-lafs" organization, but I don't see any way to put images in that location. * The current build (of github "master") has a "Tag Name" of "latest". I assume we should offer a container with the current git trunk, and a different container with the most recent release. What is the Docker naming convention here? Does "latest" imply unreleased software? Should we use "latest" to point at e.g. 1.10.1, and "dev" or something to point at current git master? I think I need amontero to answer these two.

warner commented

2015-06-07 21:47:24 +00:00

Note for the future: the Tahoe setup.py "what is my version?" code doesn't appear to succeed in the Dockerhub build process. The build log makes me think that the .git directory was not copied into the container: git describe ran without error, but probably returned a short git hash instead of a tag name. I don't know how we're going to get useful version strings into the container using the Dockerfile "ADD" command. We probably need to do a git clone inside the container instead of copying files in from the outside; maybe there's some way to copy a git hash into the container and then do a clone of that specific revision. Or perhaps the ADD command has an option to copy the .git directory too.

Note for the future: the Tahoe setup.py "what is my version?" code doesn't appear to succeed in the Dockerhub build process. The build log makes me think that the `.git` directory was not copied into the container: `git describe` ran without error, but probably returned a short git hash instead of a tag name. I don't know how we're going to get useful version strings into the container using the Dockerfile "ADD" command. We probably need to do a `git clone` inside the container instead of copying files in from the outside; maybe there's some way to copy a git hash into the container and then do a clone of that specific revision. Or perhaps the ADD command has an option to copy the .git directory too.

warner commented

2015-06-07 21:59:33 +00:00

It looks like ADD might omit .git to reduce the size of the generated container image. http://docs.docker.com/articles/dockerfile_best-practices/ emphasizes small images, and recommends COPY over ADD, but doesn't mention automatic .git-ignoring anywhere. In the longer term, we may be able to clean up the Dockerfile to just pip install . and then delete the whole source tree (once it's installed to /usr/local, we don't need the sources anymore, nor do we need the .git metadata).

It looks like ADD might omit .git to reduce the size of the generated container image. <http://docs.docker.com/articles/dockerfile_best-practices/> emphasizes small images, and recommends COPY over ADD, but doesn't mention automatic .git-ignoring anywhere. In the longer term, we may be able to clean up the Dockerfile to just `pip install .` and then delete the whole source tree (once it's installed to /usr/local, we don't need the sources anymore, nor do we need the .git metadata).

daira commented

2015-06-08 11:00:06 +00:00

Author

Replying to warner:

Note for the future: the Tahoe setup.py "what is my version?" code doesn't appear to succeed in the Dockerhub build process. The build log makes me think that the .git directory was not copied into the container: git describe ran without error, but probably returned a short git hash instead of a tag name.

That can also happen in a shallow clone that has insufficient depth of history to reach back to the latest release tag. The Travis-CI config has that problem, and uses git pull --depth=1000 to work around it.

Replying to [warner](/tahoe-lafs/trac/issues/2421#issuecomment-399211): > Note for the future: the Tahoe setup.py "what is my version?" code doesn't appear to succeed in the Dockerhub build process. The build log makes me think that the `.git` directory was not copied into the container: `git describe` ran without error, but probably returned a short git hash instead of a tag name. That can also happen in a shallow clone that has insufficient depth of history to reach back to the latest release tag. The Travis-CI config has that problem, and uses `git pull --depth=1000` to work around it.

warner commented

2015-06-09 07:55:40 +00:00

Ah, good idea, although we've got the 1.10.1a1 tag just a few (~10) commits back, so I'm not sure what's going on here. I'll pay attention to the upcoming beta1 tag to see if the tagged release is able to figure it out or not.

amontero commented

2015-06-12 15:19:11 +00:00

Owner

Replying to warner:

Ok, I think I've got things wired up. https://registry.hub.docker.com/u/dockertahoe/tahoe-lafs/ shows a new Docker build each time we push a change to github master. My remaining questions:

Can we have this live in Dockerhub's tahoe-lafs/tahoe-lafs instead of dockertahoe/tahoe-lafs ? I can see that the Dockerhub "dockertahoe" account is a member of the Dockerhub "tahoe-lafs" organization, but I don't see any way to put images in that location.

To achieve this, I've checked better and it happens that you need owner access to the organization repo (already granted). Simply delete the existing dockertahoe/tahoe-lafs and recreate it again, but selecting 'tahoelafs' at the appropiate dropdown in the new automated build creation form.

The current build (of github "master") has a "Tag Name" of "latest". I assume we should offer a container with the current git trunk, and a different container with the most recent release. What is the Docker naming convention here? Does "latest" imply unreleased software? Should we use "latest" to point at e.g. 1.10.1, and "dev" or something to point at current git master?

':latest' image tag is expected to point to latest stable release git tag.
A ':devel' or something image tag could be set up to build from HEAD git branch.
And a ':1.10.0' image tag pointing to 1.10.0 git tag and so on for other versions to allow people to base their derived images from specific image versions.
But 'latest' and 'devel' tags are by no means a hard policy, just common practice. So, we can discuss here if you have other needs or intentions.

I think I need amontero to answer these two.

Added to this, the name of the image could be tahoelafs/tahoe-lafs as you created or tahoelafs/base as I did. My intention with this was to have an unconfigured image with just the binaries and later create further images such as tahoelafs/introducer, tahoelafs/node and so on, all of them based from tahoelafs/base.
It is recommended by Docker best practices to take advantage of Docker image layers as much as possible to save download time and size on disk.

Replying to [warner](/tahoe-lafs/trac/issues/2421#issuecomment-399210): > Ok, I think I've got things wired up. <https://registry.hub.docker.com/u/dockertahoe/tahoe-lafs/> shows a new Docker build each time we push a change to github master. My remaining questions: > > * Can we have this live in Dockerhub's tahoe-lafs/tahoe-lafs instead of dockertahoe/tahoe-lafs ? I can see that the Dockerhub "dockertahoe" account is a member of the Dockerhub "tahoe-lafs" organization, but I don't see any way to put images in that location. To achieve this, I've checked better and it happens that you need owner access to the organization repo (already granted). Simply delete the existing dockertahoe/tahoe-lafs and recreate it again, but selecting 'tahoelafs' at the appropiate dropdown in the new automated build creation form. > > * The current build (of github "master") has a "Tag Name" of "latest". I assume we should offer a container with the current git trunk, and a different container with the most recent release. What is the Docker naming convention here? Does "latest" imply unreleased software? Should we use "latest" to point at e.g. 1.10.1, and "dev" or something to point at current git master? > ':latest' image tag is expected to point to latest stable release git tag. A ':devel' or something image tag could be set up to build from HEAD git branch. And a ':1.10.0' image tag pointing to 1.10.0 git tag and so on for other versions to allow people to base their derived images from specific image versions. But 'latest' and 'devel' tags are by no means a hard policy, just common practice. So, we can discuss here if you have other needs or intentions. > I think I need amontero to answer these two. Added to this, the name of the image could be tahoelafs/tahoe-lafs as you created or tahoelafs/base as I did. My intention with this was to have an unconfigured image with just the binaries and later create further images such as tahoelafs/introducer, tahoelafs/node and so on, all of them based from tahoelafs/base. It is recommended by Docker best practices to take advantage of Docker image layers as much as possible to save download time and size on disk.

amontero commented

2015-06-15 19:26:31 +00:00

Owner

[20:58] I think you can't run "tahoe create" from the Dockerfile: that'd embed the same key material in everybody's instance
[20:58] yes. scripting will be definitely needed
[20:58] it's the same issue as deciding exactly when to create ssh keys
[20:58] where should I place helper BASH scripts?
[20:59] if they're going into the tahoe source tree, I'd use misc/build_helpers/docker/

Automated build for Docker image tahoelafs/base connected to Gh and tags for 1.10.1 release set in advance. Things to do: -Figure out git tag version at build time to provide appropiate 'tahoe --version' output. -Test image as soon as 1.10.1 is tagged. Run `docker run --rm -it tahoelafs/base bash` to spin up an intereactive (-it), delete-on-exit (--rm) container to drop in a shell. You might wan to expose (-p) ports from the container to the outside. -Create role-specific images (t/introducer, t/node, etc.). Scripting will be needed. >[20:58] <warner> I think you can't run "tahoe create" from the Dockerfile: that'd embed the same key material in everybody's instance >[20:58] <pataquets> yes. scripting will be definitely needed >[20:58] <warner> it's the same issue as deciding exactly when to create ssh keys >[20:58] <pataquets> where should I place helper BASH scripts? >[20:59] <warner> if they're going into the tahoe source tree, I'd use misc/build_helpers/docker/

amontero commented

2016-05-01 17:43:05 +00:00

Owner

I think I'm experiencing this problem: https://stackoverflow.com/questions/34112585/docker-hub-automatic-build-on-tag-with-regex-match-not-triggered

So far, I've manually added 1.11 image tags and they built successfully.

After the 1.11 release, I've tried unsuccessfully to configure automatic builds to trigger automatically on new tags. Theoretically, you can now enter regexp branch/tag names to match and automatically build new images on Docker Hub. See <https://hub.docker.com/r/tahoelafs/base/~/settings/automated-builds/> help text. I think I'm experiencing this problem: <https://stackoverflow.com/questions/34112585/docker-hub-automatic-build-on-tag-with-regex-match-not-triggered> So far, I've manually added 1.11 image tags and they built successfully.

amontero commented

2016-09-18 00:18:51 +00:00

Owner

@warner: Today I used docker search tahoe and the image dockertahoe/tahoe-lafs shows in the list. Shouldn't it be deprecated?

@warner: Today I used `docker search tahoe` and the image `dockertahoe/tahoe-lafs` shows in the list. Shouldn't it be deprecated?

amontero commented

2016-12-19 19:58:32 +00:00

Owner

@warner: ping. fancy to delete the previous Docker image to avoid confusion (see previous comment)?

warner commented

2016-12-20 05:34:52 +00:00

hey, thanks for the reminder. remind me of the plan: we want folks to use https://hub.docker.com/r/tahoelafs/, but shut down https://hub.docker.com/u/dockertahoe/ ?

hey, thanks for the reminder. remind me of the plan: we want folks to use <https://hub.docker.com/r/tahoelafs/>, but shut down <https://hub.docker.com/u/dockertahoe/> ?

amontero commented

2016-12-23 12:16:01 +00:00

Owner

Yes.
Checked the dockertahoe/tahoe-lafs image and:

Has less pulls than tahoelafs/base (OK)
Has been starred 2 times (!)
I think that better yet, after deleting the automated build image, we should create a new empty, scratch image (non-automated build) with just a short notice in its description pointig to the current tahoelafs/base image.

People doing a pull will get an 'image not found' error, but they will get to a relevant info page if they go to check Docker Hub.

Yes. Checked the dockertahoe/tahoe-lafs image and: * Has less pulls than tahoelafs/base (OK) * Has been starred 2 times (!) I think that better yet, after deleting the automated build image, we should create a new empty, scratch image (non-automated build) with just a short notice in its description pointig to the current tahoelafs/base image. People doing a pull will get an 'image not found' error, but they will get to a relevant info page if they go to check Docker Hub.

amontero commented

2017-05-25 18:16:34 +00:00

Owner

Bump. Any help needed?

amontero commented

2017-05-29 10:03:45 +00:00

Owner

@warner: I hope you don't mind me assigning the issue yo you. AFAIK, you're the only person with rights to do it and will be better for it to show on your radar. Ping me if I can be of any help.