test framework for web apps that use JavaScript

[davidsarah]

If we add dependencies on JavaScript to Tahoe, such as the 'Tahoe Explorer' frontend as proposed in #1000, we will need a test framework for the JS code. Ideally, it should be able to test interactions between JavaScript and Python, and automatically run tests in multiple web browsers.

[zooko]

I looked into what node.js uses since I consider node.js to be inspired by Twisted (at least by Twisted's event-based concurrency) and Twisted has an excellent tradition of testing. Here's what I found: ​http://debuggable.com/posts/unit-testing-with-node-js:4b647d40-34e4-435a-a880-3b04cbdd56cb

[davidsarah]

I've been investigating ​Windmill, which seems to have similar functionality to Selenium in terms of web browser testing, but which uses Python as its scripting language (in addition to JavaScript). It looks like a very good fit, because we need to be able to script interactions between Python code and a web app (for example, setting up a temporary grid + directory structure in Python and then testing the web app against it, or hooking Python objects to make sure they are receiving the intended requests from the app).

I was initially concerned that Windmill might be overkill. However, if we need to be able to test Tahoe Explorer (#1000) as well as ​Josip Lisec's Music player and ​Hasini Gunasinghe's Blog for GSoC (if those projects are accepted), then I think we may end up needing a fair amount of the functionality that it provides over and above simpler JS test frameworks like JSUnit or node.js.

Windmill is Apache 2.0-licensed.

[davidsarah]

Dummy test that opens Firefox, IE, Safari and Chrome using Windmill

[davidsarah]

Simple hack to run Windmill tests from a trial test suite

[davidsarah]

ignore me (file deleted)

[josipl]

I agree with David-Sarah that Windmill, in the long run, is better option.

It probably is a bit overkill for simpler tests, but assertion module from node.js just can't cover all of the functionality - with node.js one can't test the actual UI part since there is no DOM API provided by default. ​Module which provides DOM API exists but is far from ​complete and it has dependencies on other modules/tools, and thus I'm not sure if introducing all these dependencies is worth the trouble.

On the other hand, Windmill runs the tests directly in the browser and thus allows testing every aspect of app's functionality. Also, ​Windmill's IDE seems like an rather excellent tool.

[davidsarah]

Replying to josipl:

... with node.js one can't test the actual UI part since there is no DOM API provided by default. ​Module which provides DOM API exists but is far from ​complete and it has dependencies on other modules/tools, and thus I'm not sure if introducing all these dependencies is worth the trouble.

Also, that would only be testing against that particular implementation of the DOM, rather than the implementations used by Firefox, IE, Safari, and Chrome.

I'm going to attach a patch that adds a dependency on Windmill to source:_auto_deps.py . Please test that this doesn't break anything on any platform (it doesn't on win32), but don't commit to trunk yet.

[davidsarah]

Oh, please test that after applying this patch and installing Tahoe using python setup.py install, that python -c "import windmill; help(windmill)" works. It should print a package description.

[davidsarah]

Add dependency on Windmill (fixed patch)

[davidsarah]

The patch has been applied, and hasn't caused any build failures so far (but not all of the buildslaves are on-line at the moment).

[zooko]

Now that more builders are working, we can see that this patch breaks the "Desert Island" test:

​http://tahoe-lafs.org/buildbot/builders/clean/builds/2262/steps/test-desert-island/logs/stdio

This build step ends with:

python misc/check-build.py make.out no-downloads

Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/
Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/
Reading http://pypi.python.org/simple/windmill/
Reading http://www.getwindmill.com/
Reading http://windmill.osafoundation.org/
Downloading http://pypi.python.org/packages/source/w/windmill/windmill-1.3.tar.gz#md5=1f5a8fca648beeaacbfc8f0e01d41c8c
Failed: build tried to download files

To make it pass this test, add a copy of windmill into the file that it fetches at the beginning of the test:

test -f tahoe-deps.tar.gz || wget http://allmydata.org/source/tahoe/deps/tahoe-deps.tar.gz

[davidsarah]

Since we don't actually use Windmill yet, and it's 3.2 MiB, I think we should just comment out the dependency for 1.7. We've already tested that it installs on all supported platforms, which was the intention of applying the patch.

[davidsarah]

Replying to davidsarah:

Since we don't actually use Windmill yet, and it's 3.2 MiB, I think we should just comment out the dependency for 1.7.

Done in 43b092ffeb7cae9e.

[josipl]

I'm attaching a modified version of tilting.py which was discussed the other day in IRC along with another dummy test.

TiltingMixin's setUp method now starts an introducer and storage nodes. Results of JavaScript tests are being reported in a manner which test suite can easier pickup.

test_dummy.py file contains a dummy test which copies code of an imaginary application to public_html directory of a started storage node and then starts JavaScript tests.

There are some unresolved issues with code: I couldn't find a way to disable Windmill from printing to stdout (someone on Windmill's IRC channel suggested that it should be simply used as a Nose plugin). The other problem, is the way waiting for introducer.furl file is implemented - using a loop and sleep(), this will be fixed once Python API for creating/starting/stopping nodes gets implemented.

[josipl]

Dummy test which should run JS tests and open Firefox.

[zooko]

Note that such a dependency should probably be added to {{tests_require}}} in setup.py and not to install_requires in _auto_deps.py.