Missing `allmydata/web/static/img` subdirectory when installed from `pip`.

[nejucomo]

Synopsis:

The front page ​welcome.xhtml has ​an image link for /img/logo.png which does not exist in a pip installed tahoe. In fact, the entire allmydata/web/static/img subdirectory is absent:

$ cd $VIRTUAL_ENV
$ find . -name '*.png'
./lib/python2.7/site-packages/allmydata/web/static/icon.png
$ ls ./lib/python2.7/site-packages/allmydata/web/static/
css  d3-2.4.6.min.js  d3-2.4.6.time.min.js  download_status_timeline.js  icon.png  jquery-1.6.1.min.js  tahoe.css
$ ./bin/tahoe --version
allmydata-tahoe: 1.10.0
foolscap: 0.6.4
pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958
zfec: 1.4.24
Twisted: 13.2.0
Nevow: 0.10.0
zope.interface: unknown
python: 2.7.5+
platform: Linux-debian_jessie/sid-x86_64-64bit_ELF
pyOpenSSL: 0.14
simplejson: 3.3.3
pycrypto: 2.6.1
pyasn1: 0.1.7
mock: 1.0.1
setuptools: 0.9.8

Warning: dependency 'cryptography' (version '0.3') found by pkg_resources not found by import.
Warning: dependency 'six' (version '1.6.1') found by pkg_resources not found by import.
Warning: dependency 'cffi' (version '0.8.2') found by pkg_resources not found by import.
Warning: dependency 'pycparser' (version '2.10') found by pkg_resources not found by import.

For debugging purposes, the PYTHONPATH was
  '/home/n/lib/python:/usr/lib/pyshared/python2.7/'
install_requires was
  ['setuptools >= 0.6c6', 'zfec >= 1.1.0', 'simplejson >= 1.4', 'zope.interface == 3.6.0, == 3.6.1, == 3.6.2, >= 3.6.5', 'Twisted >= 11.0.0', 'foolscap >= 0.6.3', 'pyOpenSSL', 'Nevow >= 0.6.]
sys.path after importing pkg_resources was
  /home/n/virtualenvs/lafs-wapi-dev/bin:
  /home/n/lib/python:
  /usr/lib/pyshared/python2.7:
  /home/n/virtualenvs/lafs-wapi-dev/lib/python2.7:
  /home/n/virtualenvs/lafs-wapi-dev/lib/python2.7/plat-x86_64-linux-gnu:
  /home/n/virtualenvs/lafs-wapi-dev/lib/python2.7/lib-tk:
  /home/n/virtualenvs/lafs-wapi-dev/lib/python2.7/lib-old:
  /home/n/virtualenvs/lafs-wapi-dev/lib/python2.7/lib-dynload:
  /usr/lib/python2.7:
  /usr/lib/python2.7/plat-x86_64-linux-gnu:
  /usr/lib/python2.7/lib-tk:
  /home/n/virtualenvs/lafs-wapi-dev/local/lib/python2.7/site-packages

Details:

The image link was added in ​changeset 709be93a29e20026e61a436dac3fa1c160e9cef2, which also contains the image.

I checked to see if this was a case of build misconfiguration, and it appears that the image should be included in the ​MANIFEST.in.

I also verified that an sdist tarball generated from master has the file:

$ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
$ git rev-parse HEAD
0ef593947755a8b81edc73d033219724268faf80
$ rm -rf ./dist/
$ python ./setup.py sdist | grep logo
warning: no previously-included files matching '*~' found anywhere in distribution
hard linking src/allmydata/web/static/img/logo.png -> allmydata-tahoe-1.10.0.post76/src/allmydata/web/static/img
$ tar -tzf ./dist/allmydata-tahoe-1.10.0.post76.tar.gz | grep logo.png
allmydata-tahoe-1.10.0.post76/src/allmydata/web/static/img/logo.png

At this point I am uncertain how the pip package allmydata-tahoe is generated. It seems like a potential next step is to create a packaging test within the buildbot configuration to verify that this file is included in the allmydata-tahoe package distribution.

[daira]

I see the problem. allmydata.web.static.img should be included in the list of packages here: setup.py#L447. I will fix it.

[daira]

Note that MANIFEST.in is a generated file. The .img files are supposed to be included as a result of this line:

recursive-include src *.xhtml *.js *.png *.css

[daira]

The sdist installed by pip is created using python setup.py sdist --formats=gztar (or similar).

[daira]

​https://github.com/tahoe-lafs/tahoe-lafs/pull/87

[daira]

daira1: dstufft: the only difference between the sdist that is installed correctly by pip (including the src/allmydata/web/static/img directory) and the sdist that is not installed correctly
daira1: is that the former has 'allmydata.web.static.img', in the 'packages=' argument to setuptools.setup(...)
daira1: however the 'package_data=' argument contains "allmydata.web.static.img": ["*.png"] in both cases
daira1: is this a bug in pip and/or setuptools?
dstufft: can you link me to the setup.py?
dstufft: or the two setup.py's
daira1: yes, just a sec
daira1: with 'allmydata.web.static.img': https://github.com/tahoe-lafs/tahoe-lafs/blob/1931505518726256ff2e8becb5634910afbcf766/setup.py
daira1: without: https://github.com/tahoe-lafs/tahoe-lafs/blob/master/setup.py
daira1: diff: https://github.com/tahoe-lafs/tahoe-lafs/commit/1931505518726256ff2e8becb5634910afbcf766
daira1: and here's the ticket: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2209
daira1: I was under the impression that packages= was only supposed to contain source packages? I could be wrong about that
dstufft: daira1: allmydata.web.static.img is a package, it has a __init__.py
daira1: it has an __init__.py only to work around a setuptools bug
dstufft: in order for that to work without allmydata.web.static.img in packages=[...] you'd need to set package_data={"allmydaa.web.static": ["img/*.png"]}
daira1: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1960
dstufft: in other words, the keys for package_data has to be installed packages
daira1: ah, I see
daira1: currently we have:
daira1:       package_data={"allmydata.web": ["*.xhtml"],
daira1:                     "allmydata.web.static": ["*.js", "*.png", "*.css"],
daira1:                     "allmydata.web.static.css": ["*.css"],
daira1:                     "allmydata.web.static.img": ["*.png"],
daira1:                     },
daira1: so that should be:
daira1:       package_data={"allmydata.web": ["*.xhtml"],
daira1:                     "allmydata.web.static": ["*.js", "*.png", "*.css", "img/*.png", "css/*.css"],
daira1:                     },
daira1: ?
dstufft: Yea I think so
dstufft: I'm not 100% sure on that, but I think so
daira1: I'll try it
daira1: thanks
dstufft: np
dstufft: let me know!
***daira1 nods
daira1: hmm, or maybe:
daira1:       package_data={"allmydata.web": ["*.xhtml", "static/*.js", "static/*.png",
daira1:                                       "static/*.css", "static/img/*.png", "static/css/*.css"],
daira1:                     },

The pull request uses the latter. I smoke-tested it with pip and also using:

make clean
python setup.py build
sudo python setup.py install

[daira]

I also checked that this doesn't regress #1960. (zooko was exactly correct in 1960#comment:10.)

[zooko]

Here's a relevant patch: b09b400c8a38ac7b1e1a8d788376db8cce666660/trunk, pull request: ​https://github.com/tahoe-lafs/tahoe-lafs/commit/b09b400c8a38ac7b1e1a8d788376db8cce666660 and buildbot run: ​https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Kyle%20OpenBSD%20amd64/builds/263

So can we close this ticket now, Daira?

[zooko]

Replying to daira:

Note that MANIFEST.in is a generated file.

No, MANIFEST is generated. MANIFEST.in is a source file.

[daira]

Replying to zooko:

Replying to daira:

Note that MANIFEST.in is a generated file.

No, MANIFEST is generated. MANIFEST.in is a source file.

You're right, sorry.