boutique VC in boutique language raises bar for contributions #1226

New issue

Closed

opened 2010-10-14 00:39:14 +00:00 by gdt · 11 comments

gdt commented

2010-10-14 00:39:14 +00:00

Owner

Were tahoe in git, I would have just cloned it and been fine, and I suspect I'm not alone. hg or bzr I would have figured out (but I haven't had to yet). I gather there is some aesthetic quality to darcs having a theory of patches, but I believe that other modern DVCSes would be adequate, because tahoe-lafs does not appear to have odd/complicated requirements. I think there is great merit to using tools that are in common use and that can be built/run using language environments that are mainstream enough so that systems without them can be called broken. (For example, if I said I couldn't compile C programs or run perl I would expect to get laughed at.)

It's hard to tell how many people have been discouraged from contributing because of this, but I think consideration of a move to a mainstream tool is warranted.

I tried to be able to use darcs. If all went well, this would require building it from a packaging system and then learning enough to be able to get the sources and then create and send patches. I found that pkgsrc has 2.0.2, which is old, but Zooko said it should work. But then ghc didn't build. This is all arguably lame of pkgsrc, but it shows that the number of people who care about ghc and darcs is low --- most things not only build but are reasonably current. My Copious Spare Time cannot support maintaining a boutique language so that I can build a boutique version control system so that I can maybe send a patch. Were tahoe in git, I would have just cloned it and been fine, and I suspect I'm not alone. hg or bzr I would have figured out (but I haven't had to yet). I gather there is some aesthetic quality to darcs having a theory of patches, but I believe that other modern DVCSes would be adequate, because tahoe-lafs does not appear to have odd/complicated requirements. I think there is great merit to using tools that are in common use and that can be built/run using language environments that are mainstream enough so that systems without them can be called broken. (For example, if I said I couldn't compile C programs or run perl I would expect to get laughed at.) It's hard to tell how many people have been discouraged from contributing because of this, but I think consideration of a move to a mainstream tool is warranted.

tahoe-lafs added the

labels 2010-10-14 00:39:14 +00:00

tahoe-lafs added this to the undecided milestone 2010-10-14 00:39:14 +00:00

daira commented

2010-10-14 01:16:22 +00:00

It's more than lame that pkgsrc (NetBSD's packaging system) fails to build/install GHC, which is the most commonly used implementation of a major programming language. Complain! (NetBSD is a "tier 2" platform for GHC development according to this, and there is no binary distribution of GHC downloadable for NetBSD.)

That said, we know that using darcs does raise the bar for contributions, so this is a valid ticket.

It's more than lame that pkgsrc (NetBSD's packaging system) fails to build/install GHC, which is the most commonly used implementation of a major programming language. [Complain!](http://www.haskell.org/ghc/docs/6.12.2/html/users_guide/mailing-lists-GHC.html) (NetBSD is a "tier 2" platform for GHC development according to [this](http://hackage.haskell.org/trac/ghc/wiki/Platforms), and there is [no binary distribution of GHC downloadable for NetBSD](http://www.haskell.org/ghc/download_ghc_6_12_3.html).) That said, we know that using darcs does raise the bar for contributions, so this is a valid ticket.

zooko commented

2010-10-14 02:48:05 +00:00

One of the big sticking points for me is that we have a lot of infrastructure built around darcs that I would not want to lose. It certainly can all be replaced by equivalent infrastructure using other tools, but that is a lot of work and I am only modestly motivated to do all that work myself at this point.

One promising way forward is to explore becoming multi-lingual in revision control tools. Brian Warner—a core developer who at last count had written far more Tahoe-LAFS code than anyone else—nowadays uses git exclusively and uses some sort of darcs<->git bridge hack to get his changes back into darcs. Maybe we could explore polishing up that hack and making it a supported way to operate. Paul "rockstar" Hummer and I are working on a darcs<->bzr bridge hack so that we can use the launchpad/ubuntu/canonical automation for building nightlies and uploading them into Ubuntu and stuff.

So basically I'm saying: 1. I agree that this is an important issue, 2. I agree in principle that we ought to allow contributors to use at least one of the big three (git, bzr, hg), 3. It is a major job to convert all of our tools over, which I would be willing to help a little with, but not willing to do the whole thing by myself, 4. Let's explore multi-lingual-in-revision-control techniques and see how they work.

Thanks for your many good bug reports, Greg.

Agreed that it is a valid ticket. I really like darcs and I really hate using git (although I really love git's data model. :-)), but I would be happy to use one of the big three revision control tools for Tahoe-LAFS solely because of the reason that is the focus of this ticket: lower the hurdle for Tahoe-LAFS contributors. One of the big sticking points for me is that we have a lot of infrastructure built around darcs that I would not want to lose. It certainly *can* all be replaced by equivalent infrastructure using other tools, but that is a lot of work and I am only modestly motivated to do all that work myself at this point. One promising way forward is to explore becoming multi-lingual in revision control tools. Brian Warner—a core developer who at last count had written far more Tahoe-LAFS code than anyone else—nowadays uses git exclusively and uses some sort of darcs<->git bridge hack to get his changes back into darcs. Maybe we could explore polishing up that hack and making it a supported way to operate. Paul "rockstar" Hummer and I are working on a darcs<->bzr bridge hack so that we can use the launchpad/ubuntu/canonical automation for building nightlies and uploading them into Ubuntu and stuff. So basically I'm saying: 1. I agree that this is an important issue, 2. I agree in principle that we *ought* to allow contributors to use at least one of the big three (git, bzr, hg), 3. It is a major job to convert all of our tools over, which I would be willing to help a little with, but not willing to do the whole thing by myself, 4. Let's explore multi-lingual-in-revision-control techniques and see how they work. Thanks for your many good bug reports, Greg.

zooko commented

2010-10-14 03:32:02 +00:00

Two more notes:

I love darcs and it makes me sad to imagine removing it entirely from my beloved tahoe-lafs development process. Hopefully I can still use it...
In the meantime, Greg and everyone, if you have trouble using darcs on your platform, you can always submit patches in unified diff format.

Two more notes: 1. I love darcs and it makes me sad to imagine removing it entirely from my beloved tahoe-lafs development process. Hopefully I can still use it... 2. In the meantime, Greg and everyone, if you have trouble using darcs on your platform, you can always submit patches in unified diff format.

gdt commented

2010-10-14 12:24:39 +00:00

Author

Owner

Replying to davidsarah:

It's more than lame that pkgsrc (NetBSD's packaging system) fails to build/install GHC, which is the most commonly used implementation of a major programming language. Complain! (NetBSD is a "tier 2" platform for GHC development according to this, and there is no binary distribution of GHC downloadable for NetBSD.)

I don't see it as a major language - this is the first time I've wanted to use something written in it, and I suspect a lot of people that I know have just barely heard of it. But my point is that I'm not going to start hacking haskell in order to get to where I can learn darcs. If it were a mainstream language, say in the top 10, then it would already have worked fine in pkgsrc, which I think has 8000 working packages. I certainly agree that in an ideal world someone would be keeping after having ghc work on NetBSD on 15 different cpu types and having it be current in pkgsrc, but not enough people have the combination of spare time and motivation.

Thanks for agreeing that this is a legitmate issue. I realize there are costs to changing and I'm not sure that's the right thing to do - I am just asking that it be seriously considered. If there's a bidirectional gateway somehow that's probably entirely good enough.

Replying to [davidsarah](/tahoe-lafs/trac/issues/1226#issuecomment-381599): > It's more than lame that pkgsrc (NetBSD's packaging system) fails to build/install GHC, which is the most commonly used implementation of a major programming language. [Complain!](http://www.haskell.org/ghc/docs/6.12.2/html/users_guide/mailing-lists-GHC.html) (NetBSD is a "tier 2" platform for GHC development according to [this](http://hackage.haskell.org/trac/ghc/wiki/Platforms), and there is [no binary distribution of GHC downloadable for NetBSD](http://www.haskell.org/ghc/download_ghc_6_12_3.html).) I don't see it as a major language - this is the first time I've wanted to use something written in it, and I suspect a lot of people that I know have just barely heard of it. But my point is that I'm not going to start hacking haskell in order to get to where I can learn darcs. If it were a mainstream language, say in the top 10, then it would already have worked fine in pkgsrc, which I think has 8000 working packages. I certainly agree that in an ideal world someone would be keeping after having ghc work on NetBSD on 15 different cpu types and having it be current in pkgsrc, but not enough people have the combination of spare time and motivation. Thanks for agreeing that this is a legitmate issue. I realize there are costs to changing and I'm not sure that's the right thing to do - I am just asking that it be seriously considered. If there's a bidirectional gateway somehow that's probably entirely good enough.

warner commented

2010-10-14 15:36:00 +00:00

The only thing that's been keeping me from publishing this repo to github is that there's one weird character in one comment which got translated incorrectly (there's a stray "\xc2" in Zooko's commit on 17-Dec-2007 which my tool converted into an "X", so the resulting comment says "welcXome", and I'd like it to say "welcome"). I've been working on the conversion tool (darcs-fast-export) this week in an attempt to re-do the conversion and fix that bug, but the process gets darcs into a "your repository is corrupted, please repair" state, and I haven't gotten past that yet.

Hopefully this weekend I'll finish the re-conversion and publish the result. (one side-effect will be the generation of a table mapping darcs patchids to git revisionids). That will make it easier to talk about switching general development (and trac, etc) over to git. I think the process is doable, and desireable, but it will take some work (some scripts to crawl through the trac DB and replace wikilinks to their new revisionids, small updates to the buildbot config to issue git commands instead of darcs, medium updates to setup.py and some build infrastructure, etc). Once the master github branch is up, I can share my experiments in this direction more easily.

I've got a (private, so far) bidirectional darcs/git bridge running on hanford, which I manually cycle to get changes from one side to the other. Then I do all my work in git locally, using feature branches and rebasing and all the git goodness. The only thing that's been keeping me from publishing this repo to github is that there's one weird character in one comment which got translated incorrectly (there's a stray "\xc2" in Zooko's commit on 17-Dec-2007 which my tool converted into an "X", so the resulting comment says "welcXome", and I'd like it to say "welcome"). I've been working on the conversion tool (darcs-fast-export) this week in an attempt to re-do the conversion and fix that bug, but the process gets darcs into a "your repository is corrupted, please repair" state, and I haven't gotten past that yet. Hopefully this weekend I'll finish the re-conversion and publish the result. (one side-effect will be the generation of a table mapping darcs patchids to git revisionids). That will make it easier to talk about switching general development (and trac, etc) over to git. I think the process is doable, and desireable, but it will take some work (some scripts to crawl through the trac DB and replace wikilinks to their new revisionids, small updates to the buildbot config to issue git commands instead of darcs, medium updates to setup.py and some build infrastructure, etc). Once the master github branch is up, I can share my experiments in this direction more easily.

warner commented

2010-10-15 00:31:45 +00:00

github repo created and published: http://github.com/warner/tahoe-lafs . Fork away. It will be updated manually, whenever I remember to run the script.

I gave up on fixing that one comment, so history will forever record zooko as typing "welcoXme" instead of "welco\xc2". Oh well :).

I'll push some of my working branches up too.

github repo created and published: <http://github.com/warner/tahoe-lafs> . Fork away. It will be updated manually, whenever I remember to run the script. I gave up on fixing that one comment, so history will forever record zooko as typing "welcoXme" instead of "welco\xc2". Oh well :). I'll push some of my working branches up too.

francois commented

2010-10-16 00:25:22 +00:00

Author

Owner

Thanks Brian, this git mirror is very useful!

The version computation magic in setup.py probably needs some improvements in order to works with both darcs and git.

This is what git clone git://github.com/warner/tahoe-lafs.git followed by make test shows on my laptop.

[FAIL]: allmydata.test.test_client.Basic.test_versions

Traceback (most recent call last):
  File "/home/francois/WORK/dev/tahoe-lafs/src/allmydata/test/test_client.py", line 159, in test_versions
    self.failIfEqual(str(allmydata.__version__), "unknown")
twisted.trial.unittest.FailTest: 'unknown' == 'unknown'
===============================================================================
[FAIL]: allmydata.test.test_runner.BinTahoe.test_path

Traceback (most recent call last):
  File "/home/francois/WORK/dev/tahoe-lafs/src/allmydata/test/test_runner.py", line 90, in _cb
    "We don't know our version, because this distribution didn't come "
twisted.trial.unittest.FailTest: We don't know our version, because this distribution didn't come with a _version.py and 'setup.py darcsver' hasn't been run.
===============================================================================
[FAIL]: allmydata.test.test_system.SystemTest.test_filesystem

Traceback (most recent call last):
  File "/home/francois/WORK/dev/tahoe-lafs/src/allmydata/test/test_system.py", line 756, in _check
    self.failUnless("%s: %s" % (allmydata.__appname__, allmydata.__version__) in res)
twisted.trial.unittest.FailTest: None
-------------------------------------------------------------------------------

Thanks Brian, this git mirror is very useful! The version computation magic in *setup.py* probably needs some improvements in order to works with both darcs and git. This is what **git clone git://github.com/warner/tahoe-lafs.git** followed by **make test** shows on my laptop. ``` [FAIL]: allmydata.test.test_client.Basic.test_versions Traceback (most recent call last): File "/home/francois/WORK/dev/tahoe-lafs/src/allmydata/test/test_client.py", line 159, in test_versions self.failIfEqual(str(allmydata.__version__), "unknown") twisted.trial.unittest.FailTest: 'unknown' == 'unknown' =============================================================================== [FAIL]: allmydata.test.test_runner.BinTahoe.test_path Traceback (most recent call last): File "/home/francois/WORK/dev/tahoe-lafs/src/allmydata/test/test_runner.py", line 90, in _cb "We don't know our version, because this distribution didn't come " twisted.trial.unittest.FailTest: We don't know our version, because this distribution didn't come with a _version.py and 'setup.py darcsver' hasn't been run. =============================================================================== [FAIL]: allmydata.test.test_system.SystemTest.test_filesystem Traceback (most recent call last): File "/home/francois/WORK/dev/tahoe-lafs/src/allmydata/test/test_system.py", line 756, in _check self.failUnless("%s: %s" % (allmydata.__appname__, allmydata.__version__) in res) twisted.trial.unittest.FailTest: None ------------------------------------------------------------------------------- ```

francois commented

2010-10-19 17:15:00 +00:00

Author

Owner

Replying to francois:

The version computation magic in setup.py probably needs some improvements in order to works with both darcs and git.

Oh, Brian has already mentionned this version problem in a post on tahoe-dev.

Replying to [francois](/tahoe-lafs/trac/issues/1226#issuecomment-381605): > The version computation magic in *setup.py* probably needs some improvements in order to works with both darcs and git. Oh, Brian has already mentionned this version problem in [a post on tahoe-dev](http://tahoe-lafs.org/pipermail/tahoe-dev/2010-October/005408.html).