tahoe-lafs/trac

connecting to sftp frontend using sshfs fails from linux client #645

New issue

Closed

opened 2009-02-26 15:00:53 +00:00 by azazel · 19 comments

azazel commented

2009-02-26 15:00:53 +00:00

Owner

Strange error, i've tried 3 linux clients, and to specify two commandline variations:

sshfs -C -p 8022 server: local_dir

and

sshfs -C -p 8022 server:/ local_dir

boh fail in setting up the mount with errors like:

server:.: Operation not permitted

on the client and tracebacks like this on the server:

2009-02-26 15:37:57+0100 [session (0) on SSHService ssh-connection on SSHServerTransport,0,88.1
49.141.217]SSHChannel GETATTRS . 1
2009-02-26 15:37:57+0100 [session (0) on SSHService ssh-connection on SSHServerTransport,0,88.1
49.141.217]SSHChannel Unhandled Error
Traceback (most recent call last):
File "/home/azazel/wip/tahoe/darcs/tahoe/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/se
ssion.py", line 105, in dataReceived
self.client.transport.write(data)
File "/home/azazel/wip/tahoe/darcs/tahoe/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/se
ssion.py", line 156, in write
self.proto.dataReceived(data)
File "/home/azazel/wip/tahoe/darcs/tahoe/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/fi
letransfer.py", line 53, in dataReceived
f(data)
File "/home/azazel/wip/tahoe/darcs/tahoe/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/fi
letransfer.py", line 321, in packet_STAT
d = defer.maybeDeferred(self.client.getAttrs, path, followLinks)
--- ---
File "/home/azazel/wip/tahoe/darcs/tahoe/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/internet/def
er.py", line 106, in maybeDeferred
result = f(*args, **kw)
File "/home/azazel/wip/tahoe/lib/python2.5/site-packages/allmydata_tahoe-1.3.0_r3745-py2.5.egg/a
llmydata/frontends/sftpd.py", line 305, in getAttrs
d = self._get_node_and_metadata_for_path(self._convert_sftp_path(path))
File "/home/azazel/wip/tahoe/lib/python2.5/site-packages/allmydata_tahoe-1.3.0_r3745-py2.5.egg/a
llmydata/frontends/sftpd.py", line 317, in _convert_sftp_path
assert pathstring[0] == "/"
exceptions.AssertionError:

The same sshfs command does work when connecting to the allmydata's production sftp service.

Strange error, i've tried 3 linux clients, and to specify two commandline variations: > sshfs -C -p 8022 server: local_dir and > sshfs -C -p 8022 server:/ local_dir boh fail in setting up the mount with errors like: > server:.: Operation not permitted on the client and tracebacks like this on the server: 2009-02-26 15:37:57+0100 [session (0) on SSHService ssh-connection on SSHServerTransport,0,88.1 49.141.217]SSHChannel GETATTRS . 1 2009-02-26 15:37:57+0100 [session (0) on SSHService ssh-connection on SSHServerTransport,0,88.1 49.141.217]SSHChannel Unhandled Error Traceback (most recent call last): File "/home/azazel/wip/tahoe/darcs/tahoe/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/se ssion.py", line 105, in dataReceived self.client.transport.write(data) File "/home/azazel/wip/tahoe/darcs/tahoe/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/se ssion.py", line 156, in write self.proto.dataReceived(data) File "/home/azazel/wip/tahoe/darcs/tahoe/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/fi letransfer.py", line 53, in dataReceived f(data) File "/home/azazel/wip/tahoe/darcs/tahoe/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/fi letransfer.py", line 321, in packet_STAT d = defer.maybeDeferred(self.client.getAttrs, path, followLinks) --- <exception caught here> --- File "/home/azazel/wip/tahoe/darcs/tahoe/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/internet/def er.py", line 106, in maybeDeferred result = f(*args, **kw) File "/home/azazel/wip/tahoe/lib/python2.5/site-packages/allmydata_tahoe-1.3.0_r3745-py2.5.egg/a llmydata/frontends/sftpd.py", line 305, in getAttrs d = self._get_node_and_metadata_for_path(self._convert_sftp_path(path)) File "/home/azazel/wip/tahoe/lib/python2.5/site-packages/allmydata_tahoe-1.3.0_r3745-py2.5.egg/a llmydata/frontends/sftpd.py", line 317, in _convert_sftp_path assert pathstring[0] == "/" exceptions.AssertionError: The same sshfs command does work when connecting to the allmydata's production sftp service.

tahoe-lafs added the

labels 2009-02-26 15:00:53 +00:00

tahoe-lafs added this to the undecided milestone 2009-02-26 15:00:53 +00:00

azazel commented

2009-02-26 15:31:50 +00:00

Author

Owner

I'm sorry for the ugly formatting of my last post, i 've clicked submit instead of preview. Attached there's a patch that fix this bug. I'm not a test expert so if anyone can suggest an idea about how implement a test for this please do it.

azazel commented

2009-02-26 15:32:37 +00:00

Author

Owner

Attachment fix-for-bug-645-correct-path-handling-logic-so-that-it-works-from-sshfs.dpatch (22108 bytes) added

**Attachment** fix-for-bug-_645_-correct-path-handling-logic-so-that-it-works-from-sshfs.dpatch (22108 bytes) added

fix-for-bug-_645_-correct-path-handling-logic-so-that-it-works-from-sshfs.dpatch

22 KiB

warner commented

2009-02-27 00:33:26 +00:00

Well, I don't think that will hurt anything, so if it makes sshfs connect correctly, I'll apply it.

As for tests, wow, I haven't really thought about that yet. Well, Twisted has an SSH client implementation too (we're using the server code here), so maybe it'd be possible to exercise some of the code that way. These tests would have to be skipped gracefully if conch or pycrypto were unavailable.

Well, I don't think that will hurt anything, so if it makes sshfs connect correctly, I'll apply it. As for tests, wow, I haven't really thought about that yet. Well, Twisted has an SSH client implementation too (we're using the server code here), so maybe it'd be possible to exercise some of the code that way. These tests would have to be skipped gracefully if conch or pycrypto were unavailable.

warner commented

2009-02-27 00:45:25 +00:00

Applied, in changeset:3035dfb8ed70ae4b. Thanks!

warner added the

r/fixed

label 2009-02-27 00:45:25 +00:00

warner closed this issue

2009-02-27 00:45:25 +00:00

rheimbuch commented

2009-03-05 22:10:17 +00:00

Author

Owner

Path changeset:3035dfb8ed70ae4b causes the contents of the 'root' directory to be displayed in all sub-directories. Example:

sftp> ls /
/foobar    /test-dir  
sftp> ls /foobar
/foobar/foobar      /foobar/test-dir    
sftp> ls /foobar/foobar
/foobar/foobar/foobar     /foobar/foobar/test-dir   
sftp>

I get this behavior in both the sftp command line and sshfs.Reverting the patch seems to fix the it, but breaks sshfs compatability. The problem seems to be with the returning an empty path array when the path == "."

Path changeset:3035dfb8ed70ae4b causes the contents of the 'root' directory to be displayed in all sub-directories. Example: ``` sftp> ls / /foobar /test-dir sftp> ls /foobar /foobar/foobar /foobar/test-dir sftp> ls /foobar/foobar /foobar/foobar/foobar /foobar/foobar/test-dir sftp> ``` I get this behavior in both the sftp command line and sshfs.Reverting the patch seems to fix the it, but breaks sshfs compatability. The problem seems to be with the returning an empty path array when the path == "."

tahoe-lafs removed the

r/fixed

label 2009-03-05 22:10:17 +00:00

rheimbuch reopened this issue

2009-03-05 22:10:17 +00:00

azazel commented

2009-03-05 23:26:59 +00:00

Author

Owner

I don't get the same behavior, here is mine:

sftp> ls /
/documenti     /personale      /test
sftp> ls /test                                              
/test/Archives  /test/Latest                                
sftp> version                                                                   
SFTP protocol version 3
azazel@lizard:~$ dpkg -S sftp
openssh-client: /usr/bin/sftp
azazel@lizard:~$ dpkg -l openssh-client
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name                   Version                Description
+++-======================-======================-===========
ii  openssh-client         1:4.6p1-2

Please add details of your environment to this ticket, so that i can reproduce it, maybe.

I don't get the same behavior, here is mine: ``` sftp> ls / /documenti /personale /test sftp> ls /test /test/Archives /test/Latest sftp> version SFTP protocol version 3 azazel@lizard:~$ dpkg -S sftp openssh-client: /usr/bin/sftp azazel@lizard:~$ dpkg -l openssh-client Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad) ||/ Name Version Description +++-======================-======================-=========== ii openssh-client 1:4.6p1-2 ``` Please add details of your environment to this ticket, so that i can reproduce it, maybe.

rheimbuch commented

2009-03-06 19:16:19 +00:00

Author

Owner

Just to clarify: I get the correct behavior using the 1.3.0 release. I get recursive display of the root directory when I build the trunk with changeset changeset:3035dfb8ed70ae4b.

The original run was on a Ubuntu Hardy LTS server:

rheimbuch@swoop:~$ uname -a
Linux swoop 2.6.24-19-generic #1 SMP Wed Jun 18 14:15:37 UTC 2008 x86_64 GNU/Linux

rheimbuch@swoop:~$ dpkg -l openssh-client
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Installed/Config-f/Unpacked/Failed-cfg/Half-inst/t-aWait/T-pend
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name                   Version                Description
+++-======================-======================-============================================================
ii  openssh-client         1:4.7p1-8ubuntu1.2     secure shell client, an rlogin/rsh/rcp replacement

To reproduce I repeated on my laptop with OSX 10.5.6 with fresh installs of the 1.3.0 release and the trunk with changeset changeset:3035dfb8ed70ae4b. v1.3.0 works correctly through sftp. The
trunk build has the broken behavior:

ryan-heimbuchs-macbook41:Volumes ryan$ cd
ryan-heimbuchs-macbook41:~ ryan$ sftp -oPort=8022 test2@localhost
Connecting to localhost...
test2@localhost's password: 
sftp> ls
test1  test2  
sftp> ls test1
test1/test1  test1/test2  
sftp> ls test2
test2/test1  test2/test2  
sftp> exit

ryan-heimbuchs-macbook41:~ ryan$ ssh -V
OpenSSH_5.1p1, OpenSSL 0.9.7l 28 Sep 2006

ryan-heimbuchs-macbook41:~ ryan$ which ssh
/usr/bin/ssh

ryan-heimbuchs-macbook41:~ ryan$ Projects/tahoe/trunk/bin/tahoe -V
allmydata-tahoe: 1.3.0-r3759, foolscap: 0.3.2, pycryptopp: 0.5.12, zfec: 1.4.4, Twisted: 8.2.0, Nevow: 0.9.32, zope.interface: 3.3.0, python: 2.5.1, platform: Darwin-9.6.0-i386-32bit, simplejson: 2.0.9, argparse: 0.8.0, pyOpenSSL: 0.6, pyutil: 1.3.30, zbase32: 1.1.1, setuptools: 0.6c12dev

Just to clarify: I get the correct behavior using the 1.3.0 release. I get recursive display of the root directory when I build the trunk with changeset changeset:3035dfb8ed70ae4b. The original run was on a Ubuntu Hardy LTS server: ``` rheimbuch@swoop:~$ uname -a Linux swoop 2.6.24-19-generic #1 SMP Wed Jun 18 14:15:37 UTC 2008 x86_64 GNU/Linux rheimbuch@swoop:~$ dpkg -l openssh-client Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Installed/Config-f/Unpacked/Failed-cfg/Half-inst/t-aWait/T-pend |/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad) ||/ Name Version Description +++-======================-======================-============================================================ ii openssh-client 1:4.7p1-8ubuntu1.2 secure shell client, an rlogin/rsh/rcp replacement ``` To reproduce I repeated on my laptop with OSX 10.5.6 with fresh installs of the 1.3.0 release and the trunk with changeset changeset:3035dfb8ed70ae4b. v1.3.0 works correctly through sftp. The trunk build has the broken behavior: ``` ryan-heimbuchs-macbook41:Volumes ryan$ cd ryan-heimbuchs-macbook41:~ ryan$ sftp -oPort=8022 test2@localhost Connecting to localhost... test2@localhost's password: sftp> ls test1 test2 sftp> ls test1 test1/test1 test1/test2 sftp> ls test2 test2/test1 test2/test2 sftp> exit ryan-heimbuchs-macbook41:~ ryan$ ssh -V OpenSSH_5.1p1, OpenSSL 0.9.7l 28 Sep 2006 ryan-heimbuchs-macbook41:~ ryan$ which ssh /usr/bin/ssh ryan-heimbuchs-macbook41:~ ryan$ Projects/tahoe/trunk/bin/tahoe -V allmydata-tahoe: 1.3.0-r3759, foolscap: 0.3.2, pycryptopp: 0.5.12, zfec: 1.4.4, Twisted: 8.2.0, Nevow: 0.9.32, zope.interface: 3.3.0, python: 2.5.1, platform: Darwin-9.6.0-i386-32bit, simplejson: 2.0.9, argparse: 0.8.0, pyOpenSSL: 0.6, pyutil: 1.3.30, zbase32: 1.1.1, setuptools: 0.6c12dev ```

zooko commented

2009-03-07 04:32:43 +00:00

Could we test this code without going all the way through the SSH interface, just by invoking the appropriate Python methods of the server? Going through SSH would be fine, too, but we do need tests for this.

warner commented

2009-03-09 00:21:35 +00:00

I suppose the first step would be to build up a catalog of how ssh clients behave. I wrote the convert_sftp_path method empirically, watching what a few clients I had available to me did. With the required inputs defined, then yeah, a non-round-trip test which creates an sftpd.SFTPHandler instance and invokes methods like openFile would be a good test. (it would still need to be skipped gracefully if conch were not available).

I suppose the first step would be to build up a catalog of how ssh clients behave. I wrote the `convert_sftp_path` method empirically, watching what a few clients I had available to me did. With the required inputs defined, then yeah, a non-round-trip test which creates an `sftpd.SFTPHandler` instance and invokes methods like `openFile` would be a good test. (it would still need to be skipped gracefully if conch were not available).

zooko commented

2009-04-07 16:42:23 +00:00

Oh, hey, this is a regression from 1.3.0? Should we revert changeset:3035dfb8ed70ae4b before 1.4.0 release?

zooko commented

2009-04-10 16:40:25 +00:00

The next step should be for someone (Alberto?) to write unit tests.

Moving this ticket to Milestone 1.4.1 and planning to revert the patch within the next 24 hours or so.

So my understanding of this issue is that Alberto's patch makes it start working for Alberto's use case but stop working for Ryan's use case, and that there are no unit tests. I'm going to revert this patch for Tahoe-1.4.0 release on the principle that Alberto's use case didn't work in Tahoe-1.3.0 but Ryan's did, and a regression is worse than "it still doesn't work". The next step should be for someone (Alberto?) to write unit tests. Moving this ticket to Milestone 1.4.1 and planning to revert the patch within the next 24 hours or so.

zooko added this to the 1.4.1 milestone 2009-04-10 16:40:25 +00:00

zooko commented

2009-12-13 05:32:44 +00:00

azazel: the next step on this would be for you to write a unit test. I would be willing to show you how to do so.

zooko modified the milestone from 1.6.0 to eventually

2009-12-13 05:32:44 +00:00

daira commented

2010-02-06 22:22:38 +00:00

From the patch:

if pathstring == "" or  ".":

This is clearly wrong, since "." is truthy. It was probably supposed to be

if pathstring == "" or pathstring == ".":

However, I'm confused as to why sshfs worked when the path was always set to [].

From the patch: ``` if pathstring == "" or ".": ``` This is clearly wrong, since "." is truthy. It was probably supposed to be ``` if pathstring == "" or pathstring == ".": ``` However, I'm confused as to why sshfs worked when the path was always set to [].

daira added

p/major

and removed

p/minor

labels 2010-02-06 22:22:38 +00:00

daira modified the milestone from eventually to 1.7.0

2010-02-06 22:22:38 +00:00

daira commented

2010-02-07 01:39:35 +00:00

Attachment potential-fix-for-sftpd-path-handling-darcspatch.txt (51367 bytes) added

Corrected potential fix for sftpd path handling (ticket #645)

**Attachment** potential-fix-for-sftpd-path-handling-darcspatch.txt (51367 bytes) added Corrected potential fix for sftpd path handling (ticket #645)

potential-fix-for-sftpd-path-handling-darcspatch.txt

50 KiB

daira commented

2010-02-07 01:40:16 +00:00

Attachment potential-fix-for-sftpd-path-handling-udiff.txt (960 bytes) added

Corrected potential fix for sftpd path handling (ticket #645) as unified diff

**Attachment** potential-fix-for-sftpd-path-handling-udiff.txt (960 bytes) added Corrected potential fix for sftpd path handling (ticket #645) as unified diff

potential-fix-for-sftpd-path-handling-udiff.txt

960 B

daira commented

2010-02-07 03:36:24 +00:00

Attachment size-assertions-darcspatch.txt (51226 bytes) added

Assertions for size attribute, to track down the bug described in http://allmydata.org/pipermail/tahoe-dev/2010-February/003831.html

**Attachment** size-assertions-darcspatch.txt (51226 bytes) added Assertions for size attribute, to track down the bug described in <http://allmydata.org/pipermail/tahoe-dev/2010-February/003831.html>

size-assertions-darcspatch.txt

50 KiB

daira commented

2010-05-12 06:06:29 +00:00

This is intended to be fixed by the new SFTP implementation in #1037.

daira commented

2010-05-16 16:14:17 +00:00

azazel: please check out the ticket1037 branch using

darcs get --lazy http://allmydata.org/source/tahoe-lafs/ticket1037 tahoe-sftp

and try this again.

azazel: please check out the ticket1037 branch using ``` darcs get --lazy http://allmydata.org/source/tahoe-lafs/ticket1037 tahoe-sftp ``` and try this again.

daira commented

2010-05-21 00:57:54 +00:00

This bug has essentially been obsoleted by the new SFTP implementation. bj0 has been testing with sshfs on Linux, and josipl with sshfs on OS X; neither have seen the bugs mentioned here.

Rows
Columns