httpd-38602 (resource leak)



Failure report:

How it is diagnosed (reproduced or source analysis)?

We cannot reproduce the failure so we rely on source analysis.


Resource leak (socket).

In this failure, Apache httpd server is used as a proxy server..

JBoss (backend) <=> Apache (proxy) <=> client

                                                             <= req1

                            <= req2

Apache is leaking sockets: it keeps sockets open with the JBoss backend, without reusing them.

Root cause:

The root cause is that client’s request header (req1) includes ‘Keep-alive’, but when apache is forwarding the request to JBoss, the req2 doesn’t include a ‘keep-alive’ header anymore, thus result the response from JBoss to Apache does not include ‘Connection: close’

The fix is to simply set ‘keep-alive’ in the header of req2.

Being said, let’s look at the patch:

-- httpd/httpd/trunk/modules/proxy/mod_proxy_http.c        378031
+++ httpd/httpd/trunk/modules/proxy/mod_proxy_http.c        378032
@@ -981,9 +981,18 @@ apr_status_t ap_proxy_http_request(apr_p

/* Yes I hate gotos.  This is the subrequest shortcut */
-    /* Handle Connection: header */
-    if (!force10 && p_conn->close) {
+    /*
+     * Handle Connection: header if we do HTTP/1.1 request:
+     * If we plan to close the backend connection sent Connection: close
+     * otherwise sent Connection: Keep-Alive.
+     */
+    if (!force10) {
+        if (p_conn->close) {
        buf = apr_pstrdup(p, "Connection: close" CRLF);
+        }
+        else {
+            buf = apr_pstrdup(p, "Connection: Keep-Alive" CRLF);
+        }
        ap_xlate_proto_to_ascii(buf, strlen(buf));

     /* This is the socket allocation function. If we can put log message to

     adaptively sample this function, we can clearly diagnose the failure!*/
         e = apr_bucket_pool_create(buf, strlen(buf), p, c->bucket_alloc);
        APR_BRIGADE_INSERT_TAIL(header_brigade, e);

So the key here is to know the value of ‘force10’ and ‘p_conn->close’. The original logic is that under http-1.1 (force10 == 0), and if connection is ‘closed’, it would send a ‘connection’ header. Otherwise, there is no “Connection: “ header. However, the fix is that when http-1.1 (force10 == 0), and p_conn->close != 1, it always sets “Connection: Keep-Alive”.

Is there Error Message?


Can Errlog insert a log message?

Yes. This is the resource leak patter. We can insert message to adaptively sample the socket allocation, and from the context clearly know the socket is leaking.