[tahoe-dev] the provisioning tool in tahoe

[Jimmy Tang]

On Fri, Dec 9, 2011 at 12:28 AM, Brian Warner <warner at lothar.com> wrote:

> Take a look at:
>
>  https://github.com/warner/tahoe-lafs/tree/deprovisioning
>
> in the misc/operations_helpers/provisioning/ directory. Run "run.py" in
> that directory, and it should pop open a browser with both tools. I've
> removed the tools from their old place (and updated the tests).
>
> Jimmy: if you could review that and let me know if that works for you,
> I'll land it on trunk later this week.
>

This might just be OSX/Macports being odd by from initial testing on
my snowleopard and lion installs, it fails with a dependancy problem
(btw I primarily usually use the SUMO build as its just more
convenient)

export PYTHONPATH=/Users/jtang/develop/tahoe-lafs-jt/support/lib/python2.7/site-packages
python run.py
/Users/jtang/develop/tahoe-lafs-jt/support/lib/python2.7/site-packages/zope.interface-3.8.0-py2.7-macosx-10.6-x86_64.egg/zope/interface/interface.py:688:
UserWarning: Hashing uninitialized InterfaceClass instance
  warnings.warn('Hashing uninitialized InterfaceClass instance')
Traceback (most recent call last):
  File "run.py", line 11, in <module>
    import web_reliability, provisioning
  File "/Users/jtang/develop/tahoe-lafs-jt/misc/operations_helpers/provisioning/web_reliability.py",
line 4, in <module>
    import reliability # requires NumPy
  File "/Users/jtang/develop/tahoe-lafs-jt/misc/operations_helpers/provisioning/reliability.py",
line 5, in <module>
    from numpy import array, matrix, dot
ImportError: No module named numpy

I was hoping that it would just pick up whatever was installed in the
support directory. I then just installed numpy from macports
(py27-numpy) and it just works after that. I will need to flush my
environment settings and test a bit more though. I will probably test
on rhel5 as well to make sure it works (for me at least)

without setting PYTHONPATH to what the build downloaded and setup...

python run.py
Traceback (most recent call last):
  File "run.py", line 7, in <module>
    from twisted.application import strports
ImportError: No module named twisted.application

It would be nice to preset the PYTHONPATH for the user like how the
main tahoe script does, but I guess people can live with it not being
set if it is documentend.

Also OSX's macports is a bit borked with nevow not being packaged up
for py27 etc..., it's a bit messy for my liking.

>
> I think this is what I based it on:
>
>  http://storagemojo.com/2007/02/19/googles-disk-failure-experience/
>  http://www.usenix.org/event/fast07/tech/full_papers/pinheiro/pinheiro_html/
>  http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.163.4740
>
>> I think having a few 'standard' MTBF's that manufacturers appear to be
>> selling drives at might be worth putting in as user selectable values
>> as well.
>
> Yeah! It's a pity that there are so few useful statistics and studies
> out there. Very few of the players who could do the research have an
> incentive to be honest about the results :).

Excellent I will be taking a look at those publications.

>
>> BTW I'm finding the provisioning tool to be fairly useful outside of
>> tahoe for estimating sizes and costs for a storage system :P
>
> Great!
>
> One thing I'd keep in mind, the "average space consumed per user" in a
> real deployment is (hopefully!) likely to be much much less than the
> space given to each user. My Dropbox client tells me that I'm only using
> 13% of my 2.0GB quota, and they're absolutely depending upon most users
> being like me: if everyone used their full quota, they'd go out of
> business even faster :). So measuring real-world usage, given a certain
> user demographic and use patterns, is pretty important for real
> provisioning work.
>

Heh, I'm deploying in an academic scenario where the users expect
cheap storage and will fill it :P especially for numerical computing
and mri scanning.


Thanks,
Jimmy
-- 
http://www.sgenomics.org/~jtang/