the "update-apt" step for pycryptopp buildbots always says that a task is already queued

[zooko]

Here is a builder whose job is to update the apt repository:

​http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/

It seems like every time the update-apt step is run, it says:

update already queued, deferring to them

stderr_dk on IRC says:

<stderr_dk> I wonder where
	    http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/builds/7
	    were uploaded...
<stderr_dk> It says "python-pycryptopp_0.5.25-r755-1_i386.deb:
	    uploaded"
<stderr_dk> But the latest version in
	    http://tahoe-lafs.org/debian/dists/lenny/main/binary-i386/ is
	    0.5.17 from 01-Jun-2010.
<stderr_dk> Maybe the .deb is uploaded to the wrong place? Maybe
	    that's part of the problem?

I think the reason it isn't served up by ​http://tahoe-lafs.org is that the script which always thinks it is already queued is supposed to scp it to tahoe-lafs.org.

[zooko]

I found the script on dev.allmydata.org and ran it manually:

buildslave@dev:~/tahoe-debs$ make
update already queued, deferring to them

inspecting the Makefile I see that it is looking for a file named queue.stamp:

-rw-------  1 buildslave buildslave        0 2010-06-09 01:39 queue.stamp

[zooko]

Reading the Makefile I don't see any flaw in its handling of queue.stamp. I guess what happened was that the system rebooted or in some other way the make process exited without finishing its clean-up of those .stamp files. I'm going to manually remove them.

[zooko]

That fixed it! But there is a new failure: ​http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/builds/10/steps/update%20apt%20repo/logs/stdio

make[3]: Leaving directory `/home/buildslave/tahoe-debs'
~/bin/mirror-debs-to-org.sh
pushing .debs to tahoe-lafs.org
Host key verification failed.
make[2]: Leaving directory `/home/buildslave/tahoe-debs'
make[1]: Leaving directory `/home/buildslave/tahoe-debs'
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7]
make[2]: *** [run-update-and-mirror] Error 255
make[1]: *** [run-update-and-check-for-queued] Error 2
make: *** [update] Error 2
program finished with exit code 2

[zooko]

I suspect it might have just been waiting for someone to type "yes" to ssh. I did so manually. I guess this means we should be configuring our ssh to not prompt that. There is a way to do that.

[zooko]

Aha, and that led to ​http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/builds/11/steps/update%20apt%20repo/logs/stdio

update in progress, queueing a new one

So I guess that timeout from scp waiting for someone to type "yes" is what led to the leftover stale queue.stamp file...

[zooko]

Okay that worked: ​http://tahoe-lafs.org/buildbot-pycryptopp/builders/Arthur%20debian-lenny-c7-i386%20syslib/builds/12/steps/update%20apt%20repo/logs/stdio

Now ​http://tahoe-lafs.org/debian/dists/lenny/main/binary-i386/?C=M;O=D has the latest files.

To fix this ticket: make it so failures of the upload code don't leave behind an immortal stale lock file (probably by using the debian/ubuntu package lockfile-progs and perhaps the bash trap builtin or else a subprocess).

Alternate fix for this ticket: decide that we can live with that script doing that whenever it dies and close this ticket as "fixed".

[davidsarah]

The deb builders have been decommissioned.