document munin plugins and make them discoverable #966

New issue

Open

opened 2010-02-21 04:59:16 +00:00 by zooko · 26 comments

zooko commented

2010-02-21 04:59:16 +00:00

allmydata.com maintains some munin plugins to help you monitor the health and performance of your Tahoe-LAFS nodes. You can find them in source:misc/munin. But how are users supposed to discover this fact, and how do they learn how to use these munin plugins?

zooko added the

labels 2010-02-21 04:59:16 +00:00

zooko added this to the undecided milestone 2010-02-21 04:59:16 +00:00

freestorm commented

2010-05-30 20:58:02 +00:00

Owner

Attachment munin_stats-gatherer.darcs.txt (52046 bytes) added

**Attachment** munin_stats-gatherer.darcs.txt (52046 bytes) added

munin_stats-gatherer.darcs.txt

51 KiB

zooko commented

2010-08-01 14:11:35 +00:00

Author

Please use a more informative patch description, like the example patches shown here:

http://tahoe-lafs.org/trac/tahoe-lafs/wiki/Patches

Please use a more informative patch description, like the example patches shown here: <http://tahoe-lafs.org/trac/tahoe-lafs/wiki/Patches>

freestorm commented

2010-08-01 20:22:49 +00:00

Owner

Attachment move_munin_plugins_files.dpatch (5808 bytes) added

Split munin plugins into 2 directories

**Attachment** move_munin_plugins_files.dpatch (5808 bytes) added Split munin plugins into 2 directories

move_munin_plugins_files.dpatch

5.7 KiB

freestorm commented

2010-08-01 20:24:06 +00:00

Owner

Please ignore munin_stats-gatherer.darcs.txt patch

freestorm commented

2010-08-05 22:45:41 +00:00

Owner

Is [tahoe_estimate_files]source:misc/operations_helpers/munin/tahoe_estimate_files relevant?

I can't get it working with my config

I set:

node_dirs = [os.path.expanduser("/home/tahoe-storage/")]
N = 10
num_servers = 5

correction is always 0.0, so

files = unique_strings * (32*32/len(sections)) * correction

will always return files.value 0

Is [tahoe_estimate_files]source:misc/operations_helpers/munin/tahoe_estimate_files relevant? I can't get it working with my config I set: ``` node_dirs = [os.path.expanduser("/home/tahoe-storage/")] N = 10 num_servers = 5 ``` *correction* is always 0.0, so ``` files = unique_strings * (32*32/len(sections)) * correction ``` will always return *files.value 0*

freestorm commented

2010-08-06 21:49:47 +00:00

Owner

I need help from somebody to review the future README file for stats-gatherer plugins.

Please read and edit this wiki page

When file is going to be ready, I will create the patch.

I need help from somebody to review the future README file for stats-gatherer plugins. Please read and edit [this wiki page](http://tahoe-lafs.org/trac/tahoe-lafs/wiki/Munin_Stats-Gatherer_Readme) When file is going to be ready, I will create the patch.

warner commented

2010-08-07 00:56:43 +00:00

tahoe_estimate_files assumes an allmydata-scale grid, in which the number of servers is somewhat larger than the number of shares (N), such that each server will have either 1 or 0 shares. With your values of N=10 and num_servers=5, each server will have 2 shares, and we'd need a different equation. I think for your environment you'd want to set correction to 1.0 to get it right (if every file appears on every server, there's no undercounting to be corrected for, so the chance/no_chance/correction stuff is irrelevant).

tahoe_estimate_files assumes an allmydata-scale grid, in which the number of servers is somewhat larger than the number of shares (N), such that each server will have either 1 or 0 shares. With your values of N=10 and num_servers=5, each server will have 2 shares, and we'd need a different equation. I think for your environment you'd want to set `correction` to `1.0` to get it right (if every file appears on every server, there's no undercounting to be corrected for, so the `chance`/`no_chance`/`correction` stuff is irrelevant).

freestorm commented

2010-08-07 09:42:25 +00:00

Owner

Thanks for your reply Warner,

I will modify [tahoe_estimate_files]source:misc/operations_helpers/munin/tahoe_estimate_files to add plugin settings in [tahoe-conf]source:misc/operations_helpers/munin/tahoe-conf like other plugins

Thanks for your reply Warner, I will modify [tahoe_estimate_files]source:misc/operations_helpers/munin/tahoe_estimate_files to add plugin settings in [tahoe-conf]source:misc/operations_helpers/munin/tahoe-conf like other plugins

freestorm commented

2010-08-11 20:14:23 +00:00

Owner

Attachment munin_plugins_for_stats-gatherer.dpatch (115403 bytes) added

Patch for Munin Stats-Gatherer plugins

**Attachment** munin_plugins_for_stats-gatherer.dpatch (115403 bytes) added Patch for Munin Stats-Gatherer plugins

munin_plugins_for_stats-gatherer.dpatch

113 KiB

freestorm commented

2010-08-11 20:19:19 +00:00

Owner

Patch munin_plugins_for_stats-gatherer.dpatch is ready for review.

It contain modifications for Munin Stats-Gatherer plugins only

Another patch will contain "local_plugins" modifications

freestorm commented

2010-09-12 13:14:49 +00:00

Owner

Munin plugin tahoe_overhead speak about a PHP script to collect total-deep-size from another server.

Somebody know where this PHP script can be found?

Munin plugin [tahoe_overhead](http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/misc/operations_helpers/munin/tahoe_overhead) speak about a PHP script to collect *total-deep-size* from another server. Somebody know where this PHP script can be found?

secorp commented

2010-09-12 15:30:03 +00:00

Owner

Attachment deep_manifest.php (2023 bytes) added

**Attachment** deep_manifest.php (2023 bytes) added

deep_manifest.php

2 KiB

secorp commented

2010-09-12 15:30:16 +00:00

Owner

Attachment deep_size.php (1588 bytes) added

**Attachment** deep_size.php (1588 bytes) added

deep_size.php

1.6 KiB

secorp commented

2010-09-12 15:31:37 +00:00

Owner

The last two attachments are the working copies of the php scripts. The checked in versions are in the www project on the old dev Allmydata box. We should really get this out into the public too ... I thought we had done this already but can't seem to find the trac instance we set up for it.

freestorm commented

2010-09-13 19:22:33 +00:00

Owner

Replying to secorp:

The last two attachments are the working copies of the php scripts. The checked in versions are in the www project on the old dev Allmydata box. We should really get this out into the public too ... I thought we had done this already but can't seem to find the trac instance we set up for it.

Thanks for the files secorp

I looked them, but to fully understand them, I need the DB structure, .inc/includes.php and .inc/ui/dbconnection.php.

Actually I will mark tahoe_overhead as boken, because we need to close #669 before continue with those PHP scripts.

Replying to [secorp](/tahoe-lafs/trac/issues/966#issuecomment-376665): > The last two attachments are the working copies of the php scripts. The checked in versions are in the www project on the old dev Allmydata box. We should really get this out into the public too ... I thought we had done this already but can't seem to find the trac instance we set up for it. Thanks for the files secorp I looked them, but to fully understand them, I need the DB structure, *.inc/includes.php* and *.inc/ui/dbconnection.php*. Actually I will mark [tahoe_overhead](http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/misc/operations_helpers/munin/tahoe_overhead) as boken, because we need to close #669 before continue with those PHP scripts.

warner commented

2010-09-13 21:07:05 +00:00

those PHP scripts are deeply obsolete, because they depended upon the DB that managed customer accounts at allmydata.com . I'd suggest adding a comment that says "to use this script, you will need some source of 'tahoe deep-size' results for all rootcaps that reference data in your grid. This tool was written for the allmydata.com environment, which maintained such a value."

freestorm commented

2010-10-02 14:47:15 +00:00

Owner

I wrote the README file for munin local plugins.

It would be nice if someone can take a look a this wiki page to correct my spelling errors.

As far as this page is good I will write the Patch.

I wrote the README file for munin local plugins. It would be nice if someone can take a look a this wiki [page](http://tahoe-lafs.org/trac/tahoe-lafs/wiki/Munin_local_plugins_README) to correct my spelling errors. As far as this page is good I will write the Patch.

terrell commented

2010-10-02 20:55:49 +00:00

Owner

i've looked over the linked page in comment:376668 above and made some whitespace and formatting/spelling changes.

i'm not sure of the content itself - there are underscores at the end of two of the plugin names...

the two examples at the end use a plugin that isn't listed in the table.

i've looked over the linked page in [comment:376668](/tahoe-lafs/trac/issues/966#issuecomment-376668) above and made some whitespace and formatting/spelling changes. i'm not sure of the content itself - there are underscores at the end of two of the plugin names... the two examples at the end use a plugin that isn't listed in the table.

terrell commented

2010-10-02 21:13:41 +00:00

Owner

Replying to freestorm:

I need help from somebody to review the future README file for stats-gatherer plugins.

Please read and edit this wiki page

When file is going to be ready, I will create the patch.

i've retouched this page as well.

Replying to [freestorm](/tahoe-lafs/trac/issues/966#issuecomment-376660): > I need help from somebody to review the future README file for stats-gatherer plugins. > > Please read and edit [this wiki page](http://tahoe-lafs.org/trac/tahoe-lafs/wiki/Munin_Stats-Gatherer_Readme) > When file is going to be ready, I will create the patch. > > i've retouched this page as well.

freestorm commented

2010-10-02 21:31:37 +00:00

Owner

Replying to terrell:

i've looked over the linked page in comment:376668 above and made some whitespace and formatting/spelling changes.

i'm not sure of the content itself - there are underscores at the end of two of the plugin names...

the two examples at the end use a plugin that isn't listed in the table.

Thanks for your help.
I'm going to edit the two example at the end of file.

The underscore at the end of some plugin is normal.

But I'm not sure that we need to put the README in wiki format.
The final README format will be plain text.
I use the wiki to review those files before creating the patch

Replying to [terrell](/tahoe-lafs/trac/issues/966#issuecomment-376669): > i've looked over the linked page in [comment:376668](/tahoe-lafs/trac/issues/966#issuecomment-376668) above and made some whitespace and formatting/spelling changes. > > i'm not sure of the content itself - there are underscores at the end of two of the plugin names... > > the two examples at the end use a plugin that isn't listed in the table. Thanks for your help. I'm going to edit the two example at the end of file. The underscore at the end of some plugin is normal. But I'm not sure that we need to put the README in wiki format. The final README format will be plain text. I use the wiki to review those files before creating the patch

freestorm commented

2010-10-18 20:02:38 +00:00

Owner

Attachment local_munin_plugins.darcs.patch (108918 bytes) added

Munin local plugins

**Attachment** local_munin_plugins.darcs.patch (108918 bytes) added Munin local plugins

local_munin_plugins.darcs.patch

106 KiB

freestorm commented

2010-10-18 20:05:12 +00:00

Owner

Patch munin_plugins_for_stats-gatherer.dpatch is ready for review.

It contain modifications for Munin Stats-Gatherer plugins only

Patch local_munin_plugins.darcs.patch is ready for review.

It contain modifications for Munin Local plugins.

Patch [munin_plugins_for_stats-gatherer.dpatch](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/munin_plugins_for_stats-gatherer.dpatch) is ready for review. It contain modifications for Munin Stats-Gatherer plugins only Patch [local_munin_plugins.darcs.patch ](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/local_munin_plugins.darcs.patch) is ready for review. It contain modifications for Munin Local plugins.

zooko commented

2010-10-19 02:59:56 +00:00

Author

We could consider including this in 1.8.1 because it doesn't touch core code and therefore can't introduce a regression.

zooko modified the milestone from undecided to 1.8.1

2010-10-19 02:59:56 +00:00

francois commented

2010-10-30 11:06:15 +00:00

Owner

Hi Fred, I have reviewed your two patches, see my comments in-line.

Replying to freestorm:

Patch munin_plugins_for_stats-gatherer.dpatch is ready for review.

line 100: These plugins need a Tahoe-LAFS Helper

This statement is wrong, a stats-gatherer is required, not a helper. I guess you can remove this line because this is already stated previously in the document.

You should probably warn users in the NEWS file that tahoe_* plugins have been renamed tahoe_stats_*.
It seems that there is weird indentation between lines 773 and 776.

Patch local_munin_plugins.darcs.patch is ready for review.

It contain modifications for Munin Local plugins.

A small typo on line 27, Taoe-LAFS instead of Tahoe-LAFS
The directory mentionned in line 38 should be probably be specified relative to the top directory, ie. contrib/stats-gatherer_plugins/ instead of stats-gatherer_plugins. Same comment for all directories.

That's all, thanks for updating this documentation, I'm sure that new users of the munin plugins are going to have much less trouble getting started ;)

Hi Fred, I have reviewed your two patches, see my comments in-line. Replying to [freestorm](/tahoe-lafs/trac/issues/966#issuecomment-376672): > Patch [munin_plugins_for_stats-gatherer.dpatch](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/munin_plugins_for_stats-gatherer.dpatch) is ready for review. * [line 100](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/munin_plugins_for_stats-gatherer.dpatch#L100): `These plugins need a Tahoe-LAFS Helper` This statement is wrong, a stats-gatherer is required, not a helper. I guess you can remove this line because this is already stated previously in the document. * You should probably warn users in the `NEWS` file that `tahoe_*` plugins have been renamed `tahoe_stats_*`. * It seems that there is weird indentation between lines [773](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/munin_plugins_for_stats-gatherer.dpatch#L773) and [776](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/munin_plugins_for_stats-gatherer.dpatch#L776). > Patch [local_munin_plugins.darcs.patch ](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/local_munin_plugins.darcs.patch) is ready for review. > It contain modifications for Munin Local plugins. * A small typo on [line 27](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/local_munin_plugins.darcs.patch#L27), `Taoe-LAFS` instead of `Tahoe-LAFS` * The directory mentionned in [line 38](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/local_munin_plugins.darcs.patch#L38) should be probably be specified relative to the top directory, ie. `contrib/stats-gatherer_plugins/` instead of `stats-gatherer_plugins`. Same comment for all directories. That's all, thanks for updating this documentation, I'm sure that new users of the munin plugins are going to have much less trouble getting started ;)

freestorm commented

2010-10-31 10:13:51 +00:00

Owner

Replying to [francois]comment:20:

Hi François,

I have some questions about your comments.

Hi Fred, I have reviewed your two patches, see my comments in-line.

Replying to freestorm:

Patch munin_plugins_for_stats-gatherer.dpatch is ready for review.

line 100: These plugins need a Tahoe-LAFS Helper

This statement is wrong, a stats-gatherer is required, not a helper. I guess you can remove this line because this is already stated previously in the document.

I wrote this because if you don't have a Tahoe-LAFS Helper, it make no sense to run these plugins.

The helpers plugins would return 0 or NULL if it not exist a Helper on the grid
(I need to make further tests to be sure)

You should probably warn users in the NEWS file that tahoe_* plugins have been renamed tahoe_stats_*.

It seems that there is weird indentation between lines 773 and 776.

Patch local_munin_plugins.darcs.patch is ready for review.

It contain modifications for Munin Local plugins.

A small typo on line 27, Taoe-LAFS instead of Tahoe-LAFS

The directory mentionned in line 38 should be probably be specified relative to the top directory, ie. contrib/stats-gatherer_plugins/ instead of stats-gatherer_plugins. Same comment for all directories.
Need I refer all directories from $TAHOE_SOURCE?

In the current Patch, all directories are mentionned from these files:

README
stats-gatherer_plugins/README
local_plugins/README

That's all, thanks for updating this documentation, I'm sure that new users of the munin plugins are going to have much less trouble getting started ;)

Replying to [francois]comment:20: Hi François, I have some questions about your comments. > Hi Fred, I have reviewed your two patches, see my comments in-line. > > Replying to [freestorm](/tahoe-lafs/trac/issues/966#issuecomment-376672): > > > Patch [munin_plugins_for_stats-gatherer.dpatch](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/munin_plugins_for_stats-gatherer.dpatch) is ready for review. > > * [line 100](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/munin_plugins_for_stats-gatherer.dpatch#L100): `These plugins need a Tahoe-LAFS Helper` > > This statement is wrong, a stats-gatherer is required, not a helper. I guess you can remove this line because this is already stated previously in the document. I wrote this because if you don't have a Tahoe-LAFS Helper, it make no sense to run these plugins. The helpers plugins would return 0 or NULL if it not exist a Helper on the grid (I need to make further tests to be sure) > > * You should probably warn users in the `NEWS` file that `tahoe_*` plugins have been renamed `tahoe_stats_*`. > > * It seems that there is weird indentation between lines [773](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/munin_plugins_for_stats-gatherer.dpatch#L773) and [776](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/munin_plugins_for_stats-gatherer.dpatch#L776). > > > Patch [local_munin_plugins.darcs.patch ](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/local_munin_plugins.darcs.patch) is ready for review. > > It contain modifications for Munin Local plugins. > > * A small typo on [line 27](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/local_munin_plugins.darcs.patch#L27), `Taoe-LAFS` instead of `Tahoe-LAFS` > > * The directory mentionned in [line 38](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/966/local_munin_plugins.darcs.patch#L38) should be probably be specified relative to the top directory, ie. `contrib/stats-gatherer_plugins/` instead of `stats-gatherer_plugins`. Same comment for all directories. Need I refer all directories from $TAHOE_SOURCE? In the current Patch, all directories are mentionned from these files: * README * stats-gatherer_plugins/README * local_plugins/README > > That's all, thanks for updating this documentation, I'm sure that new users of the munin plugins are going to have much less trouble getting started ;)

francois commented

2010-11-01 09:14:18 +00:00

Owner

Replying to [freestorm]comment:22:

I wrote this because if you don't have a Tahoe-LAFS Helper, it make no sense to run these plugins.

The helpers plugins would return 0 or NULL if it not exist a Helper on the grid
(I need to make further tests to be sure)

Oops, my bad, my comment was indeed incorrect, please discard it.

Need I refer all directories from $TAHOE_SOURCE?

In the current Patch, all directories are mentionned from these files:

README

stats-gatherer_plugins/README

local_plugins/README

Yes, it looks much clearer to me to always use the top directory as reference.

Replying to [freestorm]comment:22: > I wrote this because if you don't have a Tahoe-LAFS Helper, it make no sense to run these plugins. > The helpers plugins would return 0 or NULL if it not exist a Helper on the grid > (I need to make further tests to be sure) Oops, my bad, my comment was indeed incorrect, please discard it. > Need I refer all directories from $TAHOE_SOURCE? > In the current Patch, all directories are mentionned from these files: > * README > * stats-gatherer_plugins/README > * local_plugins/README > Yes, it looks much clearer to me to always use the top directory as reference.

zooko modified the milestone from 1.8.1 to soon

2010-11-02 02:32:25 +00:00