Opened at 2012-01-22T00:08:44Z
Last modified at 2012-04-02T18:00:41Z
#1664 new defect
webapi fails to handle all TCP disconnects: "Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this."
Reported by: | nejucomo | Owned by: | nobody |
---|---|---|---|
Priority: | major | Milestone: | soon |
Component: | code-frontend-web | Version: | 1.9.0 |
Keywords: | webapi error disconnect twisted | Cc: | |
Launchpad Bug: |
Description (last modified by nejucomo)
Symptoms: While running a gateway which has haproxy as a webapi client which I frequently restart, I see exceptions like this:
2012-01-21 14:35:15-0800 [-] Unhandled error in Deferred: 2012-01-21 14:35:15-0800 [-] Unhandled Error Traceback (most recent call last): File "/home/n/virtualenvs/default/lib/python2.7/site-packages/nevow/flat/twist.py", line 42, in cb _drive(iterable, finished) File "/home/n/virtualenvs/default/lib/python2.7/site-packages/nevow/flat/twist.py", line 26, in _drive finished.callback('') File "/home/n/virtualenvs/default/lib/python2.7/site-packages/twisted/internet/defer.py", line 362, in callback self._startRunCallbacks(result) File "/home/n/virtualenvs/default/lib/python2.7/site-packages/twisted/internet/defer.py", line 458, in _startRunCallbacks self._runCallbacks() --- <exception caught here> --- File "/home/n/virtualenvs/default/lib/python2.7/site-packages/twisted/internet/defer.py", line 545, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/home/n/virtualenvs/default/lib/python2.7/site-packages/nevow/appserver.py", line 181, in _cbFinishRender self.finishRequest( True ) File "/home/n/virtualenvs/default/lib/python2.7/site-packages/nevow/appserver.py", line 176, in finishRequest server.Request.finish(self) File "/home/n/virtualenvs/default/lib/python2.7/site-packages/twisted/web/http.py", line 866, in finish "Request.finish called on a request after its connection was lost; " exceptions.RuntimeError: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.
Resolution: Handle early/exceptional TCP disconnects.
Related Issues:
- It looks like a closed ticket purports to fix this issue, at least for some cases: #1366 -also see: #1278
- haproxy may behave unlike browsers. For instance, it may treat the connection to the gateway as Keep-Alive, even when the browser clients do not. I am using default configuration for this kind of behavior, and in generaly I expect haproxy to "play well" with many different http server implementations.
Change History (6)
comment:1 Changed at 2012-01-22T00:11:53Z by nejucomo
- Description modified (diff)
comment:2 Changed at 2012-03-12T19:27:24Z by davidsarah
- Component changed from unknown to code-frontend-web
- Keywords webapi error disconnect twisted added
comment:3 Changed at 2012-04-01T23:07:29Z by davidsarah
- Milestone changed from undecided to 1.11.0
comment:4 follow-up: ↓ 6 Changed at 2012-04-02T03:24:25Z by nejucomo
comment:5 Changed at 2012-04-02T03:31:35Z by nejucomo
I am somewhat confident this behavior is benign; the web server will continue processing other requests without being affected. For this reason I'd lower the priority.
There's only two possible drawbacks I now brainstorm:
- The exception may scare users.
- The process generating the response may be wasting resources; especially if this is a very long-running request which consumes many resources.
There *could* be a bug if a failure for the client to receive a request result leads to corruption. I doubt that is the case, at least for the RESTful storage APIs.
comment:6 in reply to: ↑ 4 Changed at 2012-04-02T18:00:41Z by davidsarah
Replying to nejucomo:
For instance, this query shows twisted made the same mistake internally in three different components [...]
Actually at least five; the query didn't catch https://twistedmatrix.com/trac/ticket/4901 or https://twistedmatrix.com/trac/ticket/5132 (found by searching for just notifyFinish).
After some more experience with twisted.web, this seems to be a very common bug. For instance, this query shows twisted made the same mistake internally in three different components:
Twisted trac query for "Request.finish called on a request after its connection was lost"