pgsql-4677

Version:

8.3.6

How it is diagnosed:

Reproduced. Simply execute the SQL statements listed in the bug report.

Links:

Bug report: http://archives.postgresql.org/pgsql-bugs/2009-02/msg00153.php

Patch: http://archives.postgresql.org/pgsql-committers/2009-04/msg00127.php

Symptom:

Postgres EXHAUSTS memory quickly. Eventually the memory would be freed after the end of execution of a user-defined function, but during the execution of this function, memory leaks quickly and can cause the server to crash!

 

During our inhouse failure reproduction, sometimes this will result the entire Linux box to go down. Sometimes, the OS would simply kill the process.

Root cause:
The root cause was mainly triggered by if there was an ‘try...exception...’ block within a transaction of the user-defined function, and the user-defined function (f in this bug report) is executed both in and out this try...exception... block, the memory allocated for the outer function will not be freed until the entire user function was finished. So in the buggy input, where there is a loop, it means the allocated memory for the second call to ‘f’ will not be freed until the finish of the loop:

CREATE or replace FUNCTION info.memory_growth(i_max integer)
 RETURNS integer AS
$BODY$
DECLARE
 i integer;
BEGIN
 i = 0;

 WHILE  i < i_max  LOOP

   BEGIN
     PERFORM info.f();
   EXCEPTION
     WHEN OTHERS THEN
       --
   END;
   
    PERFORM info.f(); -- The memory allocated to this function ‘f’ will not be freed until the end of all loop iterations!

   i = i + 1;
 END LOOP;

 RETURN i;
END;$BODY$
 LANGUAGE 'plpgsql' VOLATILE;

after running
select info.memory_growth(100000);

exec_stmt_block (...) {

    … ...

    if (block->exceptions)

    {

       …

      PG_TRY();

        {

            palloc...

        }

     }

}

It is key to know where the leaked palloc is called from!

Fix:

 plpgsql_subxact_cb(SubXactEvent event, SubTransactionId mySubid,
*************** plpgsql_subxact_cb(SubXactEvent event, S
*** 5251,5266 ****
         if (event == SUBXACT_EVENT_START_SUB)
                 return;
 
        if (simple_estate_stack != NULL &&
                simple_estate_stack->xact_subxid == mySubid)
         {
                SimpleEstateStackEntry *next;
 
---                if (event == SUBXACT_EVENT_COMMIT_SUB)
                        FreeExecutorState(simple_estate_stack->xact_eval_estate);                                 next = simple_estate_stack->next;
                pfree(simple_estate_stack);
                simple_estate_stack = next;
         }
 }

Is there any log message?:

No.

One thing is that, even the memory was exhausted, we still did not have an error message. In fact, in postgres, they do have the following logic:

static void *

AllocSetAlloc(MemoryContext context, Size size)

{

   … …

      block = (AllocBlock) malloc(blksize);

        while (block == NULL && blksize > 1024 * 1024)

        {

            blksize >>= 1;

            if (blksize < required_size)

                break;

            block = (AllocBlock) malloc(blksize);

        }

        if (block == NULL)

        {

            MemoryContextStats(TopMemoryContext);

            ereport(ERROR,

                    (errcode(ERRCODE_OUT_OF_MEMORY),

                     errmsg("out of memory"),

                     errdetail("Failed on request of size %lu.",

                               (unsigned long) size)));

        }

}

malloc would return NULL only if the virtual address space is exhausted. But in reality, on 64 bit machines, this is unlikely to happen, since the virtual address space is 256 TB. It is very likely that Linux would kill the process before it exhausts the virtual memory address space. In fact, Linux would start to kill process if it detects both physical memory and swap space is exhausted.

In some cases, when linux kills the postgres server process due to memory exhaustion, the server can restart itself and print the following message at the start up:

LOG:  background writer process (PID 12116) was terminated by signal 9: Killed

LOG:  terminating any other active server processes

WARNING:  terminating connection because of crash of another server process

DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

HINT:  In a moment you should be able to reconnect to the database and repeat your command.

But sometimes the entire linux box would simply freeze and crash. Also, as we can see, with the above messages are hardly useful in diagnosing memory exhaustion: we do not even know it is killed because of memory exhaustion, not to mention we do not know which malloc caused the leak.

Can we automatically anticipate?

Yes. Adaptive sampling of resource allocation function malloc. This would be sufficient to handle this failure.