make argv $0 be 'tahoe', not 'twistd' #174

New issue

Closed

opened 2007-10-11 10:29:47 +00:00 by warner · 25 comments

warner commented

2007-10-11 10:29:47 +00:00

To accomplish this, we'd need to invoke twistd directly (as a regular python function call), rather than using os.system() to spawn off a new process. This is tricky. I've done it in buildbot, so we can probably steal code from there, but it has ramifications for both win32 (#27) and for tailing-the-logfile (#71).

We've changed the name of the .tac file to contain the word "tahoe" (#156), so at least 'ps ax |grep tahoe' results in something useful now, but it would be nice to complete this effort.

To make it easier to identify a running tahoe node with standard tools like 'ps' and 'top', it would be awfully nice if argv[0] (aka $0) were 'tahoe'. At the moment, the way we spawn twistd means that a 'allmydata-tahoe start' command actually results in a process with an argv of `/usr/bin/python /usr/bin/twistd -y tahoe-client.tac --logfile logs/twistd.log` which is kind of gross. To accomplish this, we'd need to invoke twistd directly (as a regular python function call), rather than using os.system() to spawn off a new process. This is tricky. I've done it in buildbot, so we can probably steal code from there, but it has ramifications for both win32 (#27) and for tailing-the-logfile (#71). We've changed the name of the .tac file to contain the word "tahoe" (#156), so at least 'ps ax |grep tahoe' results in something useful now, but it would be nice to complete this effort.

warner added the

labels 2007-10-11 10:29:47 +00:00

warner added this to the eventually milestone 2007-10-11 10:29:47 +00:00

warner commented

2007-10-12 00:52:52 +00:00

Author

So, I stumbled across a feature in Application that might work well enough
for our purposes here. By adding service.IProcess(application).processName = "tahoe"
to our .tac file, we go from an argv that looks like:

/usr/bin/python /usr/bin/twistd -y tahoe-client.tac --logfile logs/twistd.log

to one that looks like:

tahoe /usr/bin/twistd --originalname -y tahoe-client.tac --logfile logs/twistd.log

It looks kind of weird now, but 'tahoe' shows up as argv0, which is probably a good thing.

I'll do some more experimentation.

So, I stumbled across a feature in Application that might work well enough for our purposes here. By adding `service.IProcess(application).processName = "tahoe"` to our .tac file, we go from an argv that looks like: `/usr/bin/python /usr/bin/twistd -y tahoe-client.tac --logfile logs/twistd.log` to one that looks like: `tahoe /usr/bin/twistd --originalname -y tahoe-client.tac --logfile logs/twistd.log` It looks kind of weird now, but 'tahoe' shows up as argv0, which is probably a good thing. I'll do some more experimentation.

warner commented

2007-10-12 01:05:29 +00:00

Author

Unfortunately, 'top' still shows the process as 'python', instead of 'tahoe'.

warner commented

2008-02-11 19:28:14 +00:00

Author

Also, we'd like the node's basedir to show up in the argv list, so that 'ps ax |grep tahoe' would tell us which node is which.

warner changed title from ~~make argv $0 be 'tahoe', not 'twistd'~~ to make argv $0 be 'tahoe', not 'twistd', and add BASEDIR to argv

2008-02-11 19:28:14 +00:00

keturn commented

2008-03-16 00:37:33 +00:00

Owner

we had to write a C extension to modify argv[0], so I'm curious if you find a cleaner way.

warner modified the milestone from eventually to undecided

2008-06-01 21:02:58 +00:00

zooko commented

2008-08-18 22:17:57 +00:00

Hm, I was just playing with dupfilefind and I noticed that its entry in top is named "dupfilefind" instead of "python", even though its entry in "ps eax" is:

 8571 pts/5    D+     9:43 /usr/bin/python /usr/bin/dupfilefind --profiles .

Hm, I was just playing with [dupfilefind](http://allmydata.org/trac/dupfilefind) and I noticed that its entry in top is named "dupfilefind" instead of "python", even though its entry in "ps eax" is: ``` 8571 pts/5 D+ 9:43 /usr/bin/python /usr/bin/dupfilefind --profiles . ```

tahoe-lafs added

and removed

labels 2008-09-22 20:59:18 +00:00

zandr commented

2009-02-24 21:51:11 +00:00

Owner

Upon reflection, I think I want the nickname in argv for nodes that have them, and the base class (cpu-watcher, stats-gatherer, et al) for those that don't.

This was inspired by the behavior of Munin, which shows the hostname of the target of each munin-update process.

Upon reflection, I think I want the nickname in argv for nodes that have them, and the base class (cpu-watcher, stats-gatherer, et al) for those that don't. This was inspired by the behavior of Munin, which shows the hostname of the target of each munin-update process.

zooko commented

2009-07-29 17:55:14 +00:00

Since the Milestone is "undecided" and nobody is actively working on this ticket, it doesn't seem right to have its priority set to "critical".

zooko added

p/major

and removed

p/critical

labels 2009-07-29 17:55:14 +00:00

zooko commented

2009-07-29 17:56:09 +00:00

If it is still of critical importance to a user of Tahoe-LAFS, such as Zandr, then please say so and we'll see if someone actually wants to spend time fixing it.

If it *is* still of critical importance to a user of Tahoe-LAFS, such as Zandr, then please say so and we'll see if someone actually wants to spend time fixing it.

zooko commented

2009-07-29 18:02:12 +00:00

<exarkun> zooko: well, uh, how about setting processName to tahoe?

``` <exarkun> zooko: well, uh, how about setting processName to tahoe? ```

zooko commented

2009-07-29 18:03:26 +00:00

<hagna> zooko, exarkun: see <http://jcalderone.livejournal.com/30438.html>

``` <hagna> zooko, exarkun: see <http://jcalderone.livejournal.com/30438.html> ```

daira added

t/enhancement

and removed

t/defect

labels 2010-01-07 00:30:20 +00:00

warner commented

2010-11-27 09:35:52 +00:00

Author

This might have been mostly fixed with the recent import+call+twistd.run() change. At least argv[0] should now be 'tahoe' instead of twistd. If you passed an explicit basedir (i.e. tahoe start FOO), then I think FOO will appear in your ps args.

Anyone want to check?

This might have been mostly fixed with the recent import+call+twistd.run() change. At least argv[0] should now be 'tahoe' instead of twistd. If you passed an explicit basedir (i.e. `tahoe start FOO`), then I think FOO will appear in your ps args. Anyone want to check?

francois commented

2010-11-27 16:04:05 +00:00

Owner

Replying to warner:

Anyone want to check?

The cmdline as shown by ps ax looks fine but the process name displayed by top remains python.

$ ps ax | egrep [t]ahoe
21233 ?        S      0:03 /usr/bin/python /home/francois/dev/tahoe-upstream/support/bin/tahoe start /home/francois/.tahoe
$ top -b -n 1 -p 21233
top - 16:33:40 up 1 day, 23:01,  7 users,  load average: 0.56, 0.46, 0.39
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.3%us,  1.6%sy,  0.0%ni, 92.4%id,  0.7%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2056044k total,  1985072k used,    70972k free,    54296k buffers
Swap:  6024336k total,    45356k used,  5978980k free,   506808k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                        
21233 francois  20   0  210m  83m 2620 S    0  4.1   0:03.00 python                                                                         

$ cat /proc/21233/status | grep Name
Name:	python
$

Replying to [warner](/tahoe-lafs/trac/issues/174#issuecomment-363328): > Anyone want to check? The cmdline as shown by `ps ax` looks fine but the process name displayed by `top` remains `python`. ``` $ ps ax | egrep [t]ahoe 21233 ? S 0:03 /usr/bin/python /home/francois/dev/tahoe-upstream/support/bin/tahoe start /home/francois/.tahoe $ top -b -n 1 -p 21233 top - 16:33:40 up 1 day, 23:01, 7 users, load average: 0.56, 0.46, 0.39 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 5.3%us, 1.6%sy, 0.0%ni, 92.4%id, 0.7%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 2056044k total, 1985072k used, 70972k free, 54296k buffers Swap: 6024336k total, 45356k used, 5978980k free, 506808k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21233 francois 20 0 210m 83m 2620 S 0 4.1 0:03.00 python $ cat /proc/21233/status | grep Name Name: python $ ```

warner commented

2010-11-27 20:18:17 +00:00

Author

Hm, I guess most interpreted-language programs (those with #! lines) will show up that way. Should we change it? That is, do folks think it's useful to know which version/instance of python is being used to run your node? I suppose if it helps you reconstruct the running program, it might be, although then there may be other ambiguities involved (i.e. if you just run "tahoe start", such that it searches your $PATH for "tahoe", will "ps" show you which one it found?).

I imagine we might be able to have the process rearrange its sys.argv array to try and change what ps sees, but I'm not 100% sure that it'd be a good idea..

daira commented

2010-11-27 20:47:06 +00:00

Here's what twistd does to change argv[0]: http://twistedmatrix.com/trac/browser/trunk/twisted/scripts/_twistd_unix.py?rev=27324#L173

(name is given by twisted.application.service.IProcess(application).processName.)

This requires execv and so is only done on Unix. (I don't think there's any way to change how the Processes tab of Task Manager lists processes on Windows.)

Here's what `twistd` does to change `argv[0]`: <http://twistedmatrix.com/trac/browser/trunk/twisted/scripts/_twistd_unix.py?rev=27324#L173> (`name` is given by `twisted.application.service.IProcess(application).processName`.) This requires `execv` and so is only done on Unix. (I don't think there's any way to change how the Processes tab of Task Manager lists processes on Windows.)

daira commented

2010-11-27 20:49:39 +00:00

Reducing priority to minor, since the commandline now does give all significant information.

daira added

p/minor

and removed

p/major

labels 2010-11-27 20:49:39 +00:00

warner commented

2010-11-27 20:54:19 +00:00

Author

Replying to davidsarah:

Here's what twistd does to change argv[0]: http://twistedmatrix.com/trac/browser/trunk/twisted/scripts/_twistd_unix.py?rev=27324#L173

Huh. That suggests that there isn't any way to affect argv from python (i.e.
sys.argv is a one-time copy of the C data structure, and changes are not
propagated back). Or at least that the Twisted devs decided it wasn't worth
trying to use such a feature.

I'm not sure if that extra exec() is going to cause us problems. We're
planning to set up a fairly specific environment before calling
twistd.run(), and I don't think that re-execing the current contents of
argv are likely to keep that same state around. It'll need some
experimentation.

Replying to [davidsarah](/tahoe-lafs/trac/issues/174#issuecomment-363331): > Here's what `twistd` does to change `argv[0]`: <http://twistedmatrix.com/trac/browser/trunk/twisted/scripts/_twistd_unix.py?rev=27324#L173> Huh. That suggests that there isn't any way to affect argv from python (i.e. sys.argv is a one-time copy of the C data structure, and changes are not propagated back). Or at least that the Twisted devs decided it wasn't worth trying to use such a feature. I'm not sure if that extra `exec()` is going to cause us problems. We're planning to set up a fairly specific environment before calling `twistd.run()`, and I don't think that re-execing the current contents of argv are likely to keep that same state around. It'll need some experimentation.

daira commented

2010-11-28 00:42:53 +00:00

Attachment fix-174.darcs.patch (26398 bytes) added

bin/tahoe-script.template: On non-Windows, invoke support/bin/tahoe directly as a script (rather than via python), so that 'top' for example will show it as 'tahoe'. On Windows, simplify some code that set argv[0], which is never used. fixes #174

**Attachment** fix-174.darcs.patch (26398 bytes) added bin/tahoe-script.template: On non-Windows, invoke support/bin/tahoe directly as a script (rather than via python), so that 'top' for example will show it as 'tahoe'. On Windows, simplify some code that set argv[0], which is never used. fixes #174

fix-174.darcs.patch

26 KiB

daira commented

2010-11-28 00:54:56 +00:00

I agree that an extra exec isn't worth it. However I think that on Unix we can invoke the support/bin/tahoe script directly, rather than via python, which should achieve the same effect. (That script's shebang line should point to the same Python interpreter as the bin/tahoe script, since both should be the full path to the Python that ran setup.py build.) Please repeat the checks in comment:15 to confirm.

I don't think we need a test for this, since the existing tests should catch any regressions. (It would be difficult to test, since sys.argv[0] is the same for a script invoked directly or via python.)

I agree that an extra `exec` isn't worth it. However I think that on Unix we can invoke the `support/bin/tahoe` script directly, rather than via `python`, which should achieve the same effect. (That script's shebang line should point to the same Python interpreter as the `bin/tahoe` script, since both should be the full path to the Python that ran `setup.py build`.) Please repeat the checks in comment:15 to confirm. I don't think we need a test for this, since the existing tests should catch any regressions. (It would be difficult to test, since `sys.argv[0]` is the same for a script invoked directly or via `python`.)

warner was assigned by daira

2010-11-28 00:54:56 +00:00

daira modified the milestone from undecided to 1.9.0

2010-11-28 03:39:30 +00:00

warner commented

2010-11-28 07:57:33 +00:00

Author

davidsarah: applying your patch on my OS-X box and then using 'tahoe start' to launch a couple of nodes gives me 'ps aux' output that looks the same as it did with current trunk:

185:warner@host-4-50% ps aux |grep tahoe
warner   50886   0.0  0.4  2480848  17332   ??  S    11:22PM   0:00.15 /usr/bin/python /Users/warner2/stuff/tahoe/t2-174/support/bin/tahoe start ../MY-TESTNET/node-3

"top" on OS-X just shows "Python" in the COMMAND column. (OS-X's top is a bit weird. Actually, all OS-X unix tools are a bit weird. sysv vs bsd, I guess.)

Is that what was supposed to happen?

(if it did the right thing, I suppose I'd r+ the patch, OTOH I get a deep sense of dread when seeing that weird .template file get changed, and it raises my "this file shouldn't even exist" hackles. I'd be much more keen to r+ a patch that deleted it altogether :-)

For the record, I'll be happy if the 'ps aux' output includes "tahoe", since "ps aux |grep tahoe" is what I always do to find out if a tahoe process is currently running. And I'm also happy if the argv array includes the target directory. Both of these conditions appear to be true right now.

davidsarah: applying your patch on my OS-X box and then using 'tahoe start' to launch a couple of nodes gives me 'ps aux' output that looks the same as it did with current trunk: ``` 185:warner@host-4-50% ps aux |grep tahoe warner 50886 0.0 0.4 2480848 17332 ?? S 11:22PM 0:00.15 /usr/bin/python /Users/warner2/stuff/tahoe/t2-174/support/bin/tahoe start ../MY-TESTNET/node-3 ``` "top" on OS-X just shows "Python" in the COMMAND column. (OS-X's top is a bit weird. Actually, all OS-X unix tools are a bit weird. sysv vs bsd, I guess.) Is that what was supposed to happen? (if it did the right thing, I suppose I'd r+ the patch, OTOH I get a deep sense of dread when seeing that weird .template file get changed, and it raises my "this file shouldn't even exist" hackles. I'd be much more keen to r+ a patch that deleted it altogether :-) For the record, I'll be happy if the 'ps aux' output includes "tahoe", since "`ps aux |grep tahoe`" is what I always do to find out if a tahoe process is currently running. And I'm also happy if the argv array includes the target directory. Both of these conditions appear to be true right now.

daira commented

2010-11-28 17:02:56 +00:00

Replying to warner:

davidsarah: applying your patch on my OS-X box and then using 'tahoe start' to launch a couple of nodes gives me 'ps aux' output that looks the same as it did with current trunk:
185:warner@host-4-50% ps aux |grep tahoe
warner   50886   0.0  0.4  2480848  17332   ??  S    11:22PM   0:00.15 /usr/bin/python /Users/warner2/stuff/tahoe/t2-174/support/bin/tahoe start ../MY-TESTNET/node-3
"top" on OS-X just shows "Python" in the COMMAND column. (OS-X's top is a bit weird. Actually, all OS-X unix tools are a bit weird. sysv vs bsd, I guess.)

Is that what was supposed to happen?

No, it was supposed to change the command line to just "/Users/warner2/stuff/tahoe/t2-174/support/bin/tahoe start ../MY-TESTNET/node-3". Note that python setup.py build is needed, I should have mentioned that.

Replying to [warner](/tahoe-lafs/trac/issues/174#issuecomment-363339): > davidsarah: applying your patch on my OS-X box and then using 'tahoe start' to launch a couple of nodes gives me 'ps aux' output that looks the same as it did with current trunk: > > ``` > 185:warner@host-4-50% ps aux |grep tahoe > warner 50886 0.0 0.4 2480848 17332 ?? S 11:22PM 0:00.15 /usr/bin/python /Users/warner2/stuff/tahoe/t2-174/support/bin/tahoe start ../MY-TESTNET/node-3 > ``` > > "top" on OS-X just shows "Python" in the COMMAND column. (OS-X's top is a bit weird. Actually, all OS-X unix tools are a bit weird. sysv vs bsd, I guess.) > > Is that what was supposed to happen? No, it was supposed to change the command line to just "`/Users/warner2/stuff/tahoe/t2-174/support/bin/tahoe start ../MY-TESTNET/node-3`". Note that `python setup.py build` is needed, I should have mentioned that.

daira commented

2011-01-01 21:31:44 +00:00

The patch doesn't help on OS X. Can someone try it on Linux?

daira changed title from ~~make argv $0 be 'tahoe', not 'twistd', and add BASEDIR to argv~~ to make argv $0 be 'tahoe', not 'twistd'

2011-01-01 21:31:44 +00:00

francois commented

2011-01-02 22:16:13 +00:00

Owner

Replying to davidsarah:

The patch doesn't help on OS X. Can someone try it on Linux?

This patch was tried on my Ubuntu 10.10 box. The output of the ps command did not changed but the output of the top command did changed.

In both cases.

francois@korn:~$ ps aux |grep tahoe
francois  6778  1.3  4.0 213176 82868 ?        S    22:36   0:01 /usr/bin/python /home/francois/dev/tahoe-upstream/support/bin/tahoe start
francois  6873  0.0  0.0   9212   844 pts/4    S+   22:37   0:00 grep tahoe

Without the patch.

francois@korn:~/dev/tahoe$ top -b -n 1 | egrep "python|tahoe"
 7208 francois  20   0  208m  81m 2620 S    0  4.0   0:00.97 python

With it.

francois@korn:~/dev/tahoe$ top -b -n 1 | egrep "python|tahoe"
 7354 francois  20   0  208m  80m 2580 S    2  4.0   0:00.92 tahoe

Replying to [davidsarah](/tahoe-lafs/trac/issues/174#issuecomment-363341): > The patch doesn't help on OS X. Can someone try it on Linux? This patch was tried on my Ubuntu 10.10 box. The output of the `ps` command did not changed but the output of the `top` command did changed. In both cases. ``` francois@korn:~$ ps aux |grep tahoe francois 6778 1.3 4.0 213176 82868 ? S 22:36 0:01 /usr/bin/python /home/francois/dev/tahoe-upstream/support/bin/tahoe start francois 6873 0.0 0.0 9212 844 pts/4 S+ 22:37 0:00 grep tahoe ``` Without the patch. ``` francois@korn:~/dev/tahoe$ top -b -n 1 | egrep "python|tahoe" 7208 francois 20 0 208m 81m 2620 S 0 4.0 0:00.97 python ``` With it. ``` francois@korn:~/dev/tahoe$ top -b -n 1 | egrep "python|tahoe" 7354 francois 20 0 208m 80m 2580 S 2 4.0 0:00.92 tahoe ```

daira commented

2011-01-03 00:38:41 +00:00

francois said on irc:

yes I read the code and haven't found any obvious flaw in it

francois said on irc: > yes I read the code and haven't found any obvious flaw in it

warner was unassigned by daira

2011-01-03 00:38:41 +00:00

daira self-assigned this 2011-01-03 00:38:41 +00:00

daira commented

2011-01-03 01:13:20 +00:00

In changeset:9815852a09582776:

bin/tahoe-script.template: On non-Windows, invoke support/bin/tahoe directly as a script (rather than via python), so that 'top' for example will show it as 'tahoe'. On Windows, simplify some code that set argv[0], which is never used. fixes #174

In changeset:9815852a09582776: ``` bin/tahoe-script.template: On non-Windows, invoke support/bin/tahoe directly as a script (rather than via python), so that 'top' for example will show it as 'tahoe'. On Windows, simplify some code that set argv[0], which is never used. fixes #174 ```

daira added the

r/fixed

label 2011-01-03 01:13:20 +00:00

daira closed this issue

2011-01-03 01:13:20 +00:00

daira commented

2011-01-06 01:28:43 +00:00

In changeset:1190ce614303b6fb:

NEWS: 'top' for node processes, WUI formatting, removal of GUI apps, documentation updates, foolscap dependency. refs #174, #1219, #1225

In changeset:1190ce614303b6fb: ``` NEWS: 'top' for node processes, WUI formatting, removal of GUI apps, documentation updates, foolscap dependency. refs #174, #1219, #1225 ```

daira modified the milestone from 1.9.0 to 1.8.1

2011-07-19 01:40:41 +00:00