[tahoe-dev] some questions about tahoe-lafs

[Zooko Wilcox-OHearn]

On Thu, Jul 18, 2013 at 2:22 PM, Paul Rabahy <prabahy at gmail.com> wrote:
> https://github.com/LeastAuthority/tahoe-lafs/tree/eaa6e22358f1fcf924b3b26094029fad14d72df7/src/allmydata/storage/backends/cloud
> Shows some work being done to support googlestorage, msazure, openstack, and s3. I'm not sure how far along this is or if it has plans to get merged into the mainline client, but there is some work on it.

Hi! Thanks for noticing the RAIC, a.k.a. 1819-cloud-merge branch! I
think RAIC (Redundant Array of Independent Clouds) is a powerful
technology for people who want to use "The Cloud" ¹ without giving up
control of their data.

¹ https://tahoe-lafs.org/~zooko/theCloud.ogg

Here are the relevant facts about this source code. I'm in a hurry
because LeastAuthority.com is working to announce a commercial service
ASAP. (This commercial service is *not* based on the RAIC branch — it
is just based on Amazon S3 — more about that below.)

Since I'm in a hurry, I'm going to try to spew out all the relevant
facts bullet-point style instead of writing a readable story.

• We call that code ¹, "Redundant Array of Independent Clouds (RAIC)"
or "the 1819-cloud-merge branch".

¹ https://github.com/LeastAuthority/tahoe-lafs/tree/1819-cloud-merge

• That code was written by LeastAuthority.com.

• We plan to roll out a commercial service based on this code. Our
customers' ciphertext will be erasure-coded across four (or more)
cloud storage services, such that even if any one or two of the cloud
storage services is malfunctioning, 100% of the customer's files will
still be available. This will give our customers provider-independent
*availability*, which complements the provider-independent
confidentiality and integrity that LAFS already implements. But, we're
launching a simpler commercial service first, which stores the
ciphertext solely on Amazon S3.

• The RAIC code does several things:

   1. Defines a generalized interface to the Tahoe-LAFS "backend" that
stores the ciphertext.

   2. Implements several plugins that satisfy this interface: disk
backend (equivalent to what is in Tahoe-LAFS storage server now),
cloud backend (more on that below), mock backend (an in-memory backend
for testing) and null backend (a /dev/null backend for testing).

   3. Implements a "cloud backend" which splits an encrypted,
erasure-coded "share" into a set of fixed-sized, immutable "objects"
which are suitable for being stored on any cloud storage service.

   4. Moves all of the lease-tracking and accounting from being stored
inside the share files to being stored in a sqlite database on each
storage server. This makes it much easier and more efficient to use a
cloud storage service to hold the ciphertext share files. It also
opens up the possibility of a lot of accounting and leasing
improvements in general, even if you're using the normal old disk
backend. This was a necessary step for Brian's Accounting project.
(This is called the "leasedb" code.)

   5. Defines a generalized interface to cloud storage services.

   6. Implements four cloud storage connectors that satisfy this
interface: Amazon S3, OpenStack/Rackspace/HewlettPackard, Microsoft
Azure, and Google Storage For Developers. Note: this includes
implementing the authentication, authorization, and accounting bits by
which each of these services control and track who is accessing their
service.

• The development of this code was funded by a DARPA grant under
DARPA-RA-11-52 "Cyber Fast Track".

• It was a research grant, which DARPA gave us solely because they
thought the LAFS technology was awesome. It doesn't give DARPA any
copyright, patent, or trademark rights over the resulting source code.
It doesn't obligate LeastAuthority.com to do anything other than write
the source code, which we've now done.

• That contract is now complete, and we have no ongoing contract or
other relationship with DARPA at this time.

• The code is complete and it works! It was developed using the
standard test-driven development process that we use when we
contribute patches to the Tahoe-LAFS codebase. It was mostly written
by Daira, with help from Itamar Turner-Trauring. It was designed by
Daira and me, and it passes all our unit tests, plus it was vetted by
DARPA kicking its tires before they paid us. It comes with
documentation about the design, the implementation, and
how-to-set-it-up.

• The design is good. It is portable to any cloud storage service (it
requires only a limited feature set from the cloud storage service),
it allows streaming uploads, arbitrarily large files, etc. See the
attached document that explains the design. (That document is an
edited version of the proposal that we submitted to DARPA.)

• The licensing rights to the original Tahoe-LAFS source code, that we
started with, are owned by The Tahoe-LAFS Software Foundation, which
is led by Peter Secor. I'm not an officer of that Foundation, nor is
any other employee of LeastAuthority.com. We don't have any direct
control over the Foundation.

• The 1819-cloud-merge backend is a "derived work" of Tahoe-LAFS under
US and European copyright law. That means we require a licence before
we are allowed to redistribute, sell, perform, etc. our RAIC source
code. (This is the same reason that any Free/Open Source licence like
the GNU General Public Licence can impose legal obligations on
extensions or forks of GPL'ed software.)

• LeastAuthority.com has the right, granted by the Tahoe-LAFS
Free/Open Source licences ², to distribute and use our
1819-cloud-merge branch without open-sourcing it for up to 12 months
after we first distributed it or operated it as a hosted service.
Therefore, we do not have to release the RAIC work as open-source
right now, but we do have to do so within 12 months of having
published it (or we would be in violation of the copyright owned by
the Tahoe-LAFS Software Foundation and the individual contributors).

² https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/about.rst#license

• We've decided to immediately contribute to the community the core of
it, which includes parts 1, 2, 3, 4, and 5 from the list above. We've
contributed all of the code that does those parts, excluding part 6,
to the public under the same terms as Tahoe-LAFS itself. You can get
the code here: ³. As far as I'm concerned, that code should be merged
into Tahoe-LAFS trunk ASAP, and as far as I know Brian and Daira
agree. This would be a perfect starting point for anyone who wants to
add their own cloud storage connectors, since parts 1-5 handle most of
the complexity for you.

³ https://github.com/LeastAuthority/tahoe-lafs/tree/1819-cloud-merge-opensource

• We are withholding from open source part 6 — the connectors to
connect the cloud backend to S3, Rackspace, Azure, and Google — but
we've granted a licence to use those connectors for non-commercial
use. Of course, as explained above, part 6 will also become Free and
Open Source no later than 12 months after we first distributed or
hosted it.



Okay, so here's what you can do with this code:

• If you want to implement a connector to a cloud service, you can use
the parts that are already open source as the basis for your code.

• If you don't mind using code that is not-yet-Open-Source but is free
for non-commercial use, then you can use the full 1819-cloud-merge
branch now.

• If you want to use it for commercial use, write to me and I'll sell
you a licence and support contract.

• If you want to use this but only when it is Free and Open Source,
then wait a few months.

• If you want to pay for RAIC as a service, then write to us at
LeastAuthority.com to sign up for the alpha test of it! (In the
meantime, you can also sign up for the current service which is the
simpler just-Amazon-S3 storage.)


Thanks for your support, everyone!

Regards,

Zooko
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cft-leastauthority-01-filt.pdf
Type: application/pdf
Size: 859262 bytes
Desc: not available
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20130721/6f1ace83/attachment-0001.pdf>