improve munin stats (stats_gatherer)

[cl]

Hi,

I have been asked by Zooko to open this ticket, so here it goes:

The statistics gathered by the stats_gatherer can be nicely graphed using munin (file misc/operations_helpers/munin/tahoe_stats). While that file is not terribly difficult to modify/extend/adjust, the default one is missing a few rather important fields (storage used/available, for instance).

I am attaching the configuration that we use in VG2. Most important changes are:

• added support for a few other useful stats (most importantly, storage used/free, download stats)
• different graph types where it makes sense
• example of setting warning/critical limits (storage free)
• an "introductory comment" which is hopefully helpful to make it easier to get started
• slightly different node naming scheme, and alpabetically sorted output of the nodes.

This might still need a bit of tweaking (especially the last bullet, node naming, already considers an idiosyncrasy of VG2, namely that VG2 has the convention of including the maintainer's mail address in the node name. This *should* only kick in if node names contain an "@", though). However, I think that it might serve as a slightly enhanced version of the current script which could be considered for inclusion in future releases.

[lenzenmi]

I'm using the revised file, and it seems to work well.

One thing I have noticed is that the tahoe_server_disk_used graph shows the total space used on the filesystem where tahoe is running, not just the space used by tahoe shares.

I took a quick look at the stats.pickle file, and the only disk usage values I see are:

free_for_root disk_total disk_free_for_non_root disk_avail disk_used

None of which give you the stat for the space consumed by the grid. I know the share total is available from each server, and can be gathered by the node specific tahoe_storagespace munin plugin, but this needs to be installed on each machine that runs a server.

I suppose this should probably go into a new ticket, but perhaps we could add the tahoe_storagespace stat to the stats.pickle file that's collected by the stats-server?

As a workaround, new installs of tahoe could be made to their own filesystem, so the disk_used and tahoe_storagespace numbers are equivalent.

[exarkun]

Going to remove the stats gatherer and related code: ticket:3549