missing freeform-default.css -- nevow install error?

[zooko]

When I click on an html file on win32 (namely my slide-1.html), then I get an unresponsive UI and these kinds of log messages:

2007-10-13 22:43:32.875Z [Uninitialized] Stopping factory <foolscap.negotiate.TubConnectorClientFactory object [from jbzjax2i] [to kzl7rg4a] at 0x0186E650>

2007-10-13 22:43:33.078Z [HTTPChannel,0,127.0.0.1] 127.0.0.1 - - [13/Oct/2007:22:43:32 +0000] "GET / HTTP/1.1" 200 2259 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7"

2007-10-13 22:43:33.092Z [HTTPChannel,0,127.0.0.1] Unhandled Error

	Traceback (most recent call last):

	  File "c:\Python25\Lib\site-packages\twisted\internet\defer.py", line 191, in addCallback

	    callbackKeywords=kw)

	  File "c:\Python25\Lib\site-packages\twisted\internet\defer.py", line 182, in addCallbacks

	    self._runCallbacks()

	  File "c:\Python25\Lib\site-packages\twisted\internet\defer.py", line 317, in _runCallbacks

	    self.result = callback(self.result, *args, **kw)

	  File "c:\python25\lib\site-packages\nevow-0.9.18-py2.5.egg\nevow\appserver.py", line 158, in gotPageContext

	    pageContext.tag.renderHTTP, pageContext

	--- <exception caught here> ---

	  File "c:\Python25\Lib\site-packages\twisted\internet\defer.py", line 107, in maybeDeferred

	    result = f(*args, **kw)

	  File "c:\python25\lib\site-packages\nevow-0.9.18-py2.5.egg\nevow\static.py", line 273, in renderHTTP

	    self.fp.restat()

	  File "c:\Python25\Lib\site-packages\twisted\python\filepath.py", line 383, in restat

	    self.statinfo = stat(self.path)

	exceptions.WindowsError: [Error 2] The system cannot find the file specified: 'c:\\Python25\\lib\\site-packages\\nevow-0.9.18-py2.5.egg\\formless\\freeform-default.css'

	

2007-10-13 22:43:33.125Z [HTTPChannel,0,127.0.0.1] 127.0.0.1 - - [13/Oct/2007:22:43:32 +0000] "GET /webform_css HTTP/1.1" 500 5841 "http://localhost:8123/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7"

[warner]

hm, is that .css file not getting installed properly? Nevow ought to install that when its 'setup.py install' is run, but maybe the eggstuff is failing?

Weird that this should screw up the UI, though.. I'd have expected it to just report a 404. It's just a little CSS file, and in fact I think we aren't using anything from it.

[zooko]

You are right that this doesn't actually screw up the UI.

The UI hung earlier for reasons unknown -- waiting for a storage server, perhaps? I can't reproduce the UI hanging now.

The error messages continues to appear in logs whenever I load a page.

[zooko]

The path that it gets installed to differs from the one it uses to stat the file:

          File "c:\python25\lib\site-packages\nevow-0.9.18-py2.5.egg\nevow\static.py", line 273, in renderHTTP
            self.fp.restat()
          File "c:\Python25\Lib\site-packages\twisted\python\filepath.py", line 383, in restat
            self.statinfo = stat(self.path)
        exceptions.WindowsError: [Error 2] The system cannot find the file specified: 'c:\\Python25\\lib\\site-packages\\nevow-0.9.18-py2.5.egg\\formless\\freeform-default.css'
        

compared to

$ find /c/Python25/ -iname '*freeform-default*'
/c/Python25/Lib/site-packages/formless/freeform-default.css
/c/Python25/Lib/site-packages/nevow-0.9.18-py2.5.egg/Lib/site-packages/formless/freeform-default.css

[zooko]

As far as I can tell, this installation glitch doesn't interfere with Tahoe functionality. Brian: if you think that this is unlikely to cause any actual problems for users (on Windows) in the current version, will you please bump this ticket out of Milestone v0.6.1?

[warner]

ok, if the unresponsive UI thing was unrelated, then I'm ok with pushing this out

[warner]

I looked at this briefly, it looked like the problem was that the Nevow setup.py was manually assembling the target directory for it's package_data (using something like "/usr/lib/python%s" % sys.version), and the actual install process (setuptools?) was using /usr/lib/python/ (without the version number).

There's a comment in the Nevow setup.py saying "this sucks, we should use pkg_resources instead". Maybe we should write a patch for them? I think they were trying to maintain python2.3 compatibility at one point, though.

[zooko]

Does that difference that you've observed also explain the fact that on Windows Nevow was attempting to stat c:\Python25\lib\site-packages\nevow-0.9.18-py2.5.egg\formless\freeform-default.css but the file was in C:\Python25\Lib\site-packages\nevow-0.9.18-py2.5.egg\Lib\site-packages\formless\freeform-default.css ?

The standard distutils behavior on Windows is to use C:\PythonXY\Lib\site-packages and on other systems to use $PREFIX\lib\pythonX.Y\site-packages.

How do we contact the Nevow folks -- through ​their trac?

[warner]

I suspect it does, but I didn't take careful notes when I noticed that stuff. I was trying to install nevow on an OS-X box into a non-.egg directory when I saw it, and I think I fixed the problem by patching the nevow setup.py .

And yes, the divmod Trac is probably the most persistent/public way to get them to fix it. The most immediate way would be to hassle them on the #twisted or #twisted-web irc channel, but in general trac tickets are preferred.

[zooko]

We're focussing on an imminent v0.7.0 (see ​the roadmap) which hopefully has #197 -- Small Distributed Mutable Files and also a fix for #199 -- bad SHA-256. So I'm bumping less urgent tickets to v0.7.1.

[zooko]

Hm. I did a quick query on the Nevow trac for "pkg_resources", and found these patches which apparently make Nevow use pkg_resources. The weird part is that these patches were apparently committed to Nevow trunk two years ago:

​http://divmod.org/trac/changeset/2238 ​http://divmod.org/trac/changeset/2262 ​http://divmod.org/trac/changeset/2263

[zooko]

Okay, the Nevow 0.9.29 release comes with a ​setup.py} and also with a [http://divmod.org/trac/browser/trunk/Nevow/setup_egg.py setup_egg.py. The latter uses setuptools to decide where to put the package_data so that pkg_resources can later find it. (It is worth comparing those two files visually -- the stanza at the top of nevow's setup.py which starts with "if sys.platform" is the place where that script is deciding where to put these files.

If we use the setuptools version then this bug is fixed -- Nevow can find its data files at runtime. I tested the fix on Cygwin, Windows, and Mac.

The problem is that our packaging policy forbids us to patch 3rd-party libraries that we use, and there isn't a way to automate the use of the setup_egg.py without patching the Nevow tarball (or else writing a script to build Nevow differently from the way we build other dependent libs, which is the moral equivalent of patching the Nevow tarball).

Attached is a patch for Nevow that makes Nevow's setup.py choose whether to use the custom-Nevow setup script or the setuptools setup script at build time, based on whether setuptools can be imported. This patch is not yet tested. I intend to add a unit test (to allmydata.org Tahoe) which fails if Nevow can't find its resources files at run-time, and then to provisionally patch Nevow with this patch. This violates our policy of not patching 3rd-party libraries, but I don't know of a better way to fix this issue.

[zooko]

Here's a version of the patch for Nevow-0.9.18, which is the version that we currently use. I also wrote a unit test that simply calls

        webform.defaultCSS.openForReading()

This incurs an exception -- can't find the default css file -- without the patch and passes with the patch.

[zooko]

Submitted a patch to divmod: ​http://www.divmod.org/trac/ticket/2527

[zooko]

​http://www.divmod.org/trac/ticket/2527

Has finally been fixed on Nevow trunk. The next release of Nevow will no longer have this problem. I guess I should wait til then to close this ticket.

[zooko]

This is fixed as of Nevow v0.9.33.