[tahoe-dev] on backward-compatibility and new features

[Zooko Wilcox-O'Hearn]

[N.B. I pressed Brian about this in conversation on IRC yesterday,  
arguing that it wouldn't hurt to make users explicitly enable  
immutable-dirs-in-backups before they could use them.]

On Thursday, 2009-11-19, at 22:01 , Brian Warner wrote:

>  * "tahoe backup" is incredibly faster with immutable directories
...
>  * the directories it creates are repairable
...
>  * The only directory that has a DIR2-CHK objects placed into it is  
> the top-level target of the "tahoe backup" command
...
>  * I don't believe that "tahoe backup" is the tool of choice for  
> sharing files
...
> So yeah, if we were talking about changing "tahoe mkdir" to create  
> new caps, then I'd agree with a conservative approach and add a  
> config knob to slowly introduce the new feature over time. But for  
> "tahoe backup", I think the benefits outweigh the downsides.

Okay, good arguments!  Let's do it your way!


Now about the more general principle for future reference:

>> I would like people to get used to the idea that they can always  
>> upgrade to the new version of Tahoe-LAFS and continue sharing with  
>> their more conservative friends without having to even think  
>> about  these issues.
>
> Ok, but then how is it possible to ever introduce new features?

By an explicit action on the part of the user.  This could be setting  
a configuration option or (better) just clicking on a different  
button in the WUI or invoking a different command in the CLI or  
adding a "--newfangled-awesomeness" option in the CLI.  Also, with  
sufficient passage of time we -- the Tahoe-LAFS developers -- can  
observe that people have in practice upgraded to a sufficiently new  
version that we can make the new feature default on in the next  
release of Tahoe-LAFS after that.  (See below about that.)

> A good concrete example would be Elk Point..

N.B.  Elk Point won't necessarly be the next cap format.  There is a  
project to invent a New Cap format [1], and Elk Point is one  
candidate for that design.  I really like Elk Point -- a lot! -- but  
we might end up implementing a different candidate (at least at  
first), or we might end up implementing a subset of Elk Point at  
first for reasons of simplicity -- ease of implementation,  
verification, testing, deployment.

The next step on New Cap Design project is, I think, to document the  
alternatives in a comparable level of detail to the current Elk Point  
docs -- svg diagrams, What Could Go Wrong [2], etc.

But yes, a good concrete example would be Elk Point or some other New  
Cap format.

> This could mean that some snazzy new features might be implemented  
> and shipped but go effectively unused for a year or more, depending  
> upon the release schedule and our overall patience with old versions.

Be careful about this assumption that seems to be implicit in this  
statement -- that new features go unused if they are not turned on by  
default.  If the feature isn't desirable enough that the user learns  
about it and explicitly indicates the willingness to use it (at the  
cost of sharing files with her stick-in-the-mud friends), then  
perhaps that means it's just not that important.  They can have it by  
default a few years later.  :-)  But I suspect that good new features  
*would* get used before they are turned on by default.

For example, suppose New Caps come out in Tahoe-LAFS v1.7, are faster  
to create and use, offer better security, and generate smaller,  
prettier, and more usable caps.  Suppose that the "create a new  
directory" button continues to produce old-caps just like it did in  
Tahoe-LAFS v1.6.  But, suppose that there is a checkbox next to it  
that says "new cap format".  Users can easily (without even reading  
the NEWS.txt) figure out how the button behaves differently with and  
without the checkbox, figure out that their old-version-using friends  
can't read the new caps, and then decide for themselves whether to  
keep using old-caps or to persuade their friends to upgrade.


Let me restate my use case now that you've persuaded me that  
immutable dirs in backupdb is okay.  It's not that new features have  
to be disabled by default, it's that:

A user wants to know if she can upgrade to the new release of Tahoe- 
LAFS, briefly glance at the NEWS.txt file, and continue to  
interoperate with her old-version-using friends, without  
significantly altering her workflow.

I still think this use case is important, because if we miss it by  
too much then our user base splits into two camps and we have to  
either support both or abandon the stick-in-the-muds.  I think it is  
possible to really screw up here by overestimating the amount of work  
users are willing to undertake in order to upgrade.  For example, the  
darcs project made this mistake and now they have two repository  
formats to support and so far nobody has figured out how to get the  
stick-in-the-muds (which includes us) to make the jump to the new  
format.  For another example, the Python project made a move like  
this (the backwards-incompatible Python 3), and I'm still waiting to  
find out how many years, if ever, before stick-in-the-muds like us  
make the jump to Python 3.

But, I think you're right that there are some cases, such as  
immutable-dirs in backupdb, where the current user base and their  
current workflows don't run afoul of the backwards-compatibility issue.


By the way, I checked and allmydata.com customers who use the  
allmydata.com Windows client are still using Tahoe-LAFS v1.1.0, which  
came out in June of 2008, plus a few critical bugfixes to the WUI  
which were backported from newer releases of Tahoe-LAFS.  (A few of  
the "power user" customers of allmydata.com use Tahoe-LAFS v1.4.1 or  
v1.5.0 with the allmydata.com grid, on Linux and on Mac OS X.)

I'm fairly sure that none of those allmydata.com-Windows-client-using  
users wants to interoperate with modern Tahoe-LAFS users, so I'm not  
worried about that, but it is interesting to note.  I wouldn't be  
surprised if the addition of immutable directories is the final straw  
that prompts allmydata.com to make the jump to a modern version of  
Tahoe-LAFS, bringing them up to Tahoe-LAFS v1.6.  (It would improve  
client performance and reduce load on the allmydata.com storage grid  
considerably.)

The other pool of potential stick-in-the-muds is Ubuntu users.  The  
current version of Ubuntu -- 9.10 (Karmic Koala) -- comes with Tahoe- 
LAFS v1.5.  I think we'll be able to get Tahoe-LAFS v1.6 into Ubuntu  
10.04 if we pay attention to their schedule [3].  If that happens  
then the people who use Tahoe-LAFS as packaged by Ubuntu will  
probably upgrade to Tahoe-LAFS v1.6 because Ubuntu 10.04 is an LTS  
(Long-Term-Support) release.  After that, a lot of them will not  
upgrade to a newer Ubuntu for many years.  Whatever version of Tahoe- 
LAFS ships in Ubuntu 10.04 will remain in widespread use for at least  
five years.


>> This is also why "forward-compatibility" features are so important  
>> to me.  They can sometimes ease these transitions.
>
> Yes! I'm very glad you pushed for #708 (tolerate unknown nodes),  
> because having it implemented in v1.5 made DIR2-CHK much easier to  
> deploy in v1.6, and makes me feel more confident about things like  
> Elk Point.

Yes!  That worked out well!  Now think about what features you would  
like to add to Tahoe-LAFS v2.0, and then think about what we can do  
in v1.6 or v1.7 to provide forward-compatibility with those  
features.  :-)

Regards,


Zooko

[1] http://allmydata.org/trac/tahoe/wiki/NewCapDesign
[2] http://allmydata.org/trac/tahoe/wiki/NewCaps/WhatCouldGoWrong
[3] https://wiki.ubuntu.com/LucidReleaseSchedule? 
action=show&redirect=LucidLynxSchedule
>