Ultimate Guide to the LDG Architecture & its Pitfalls

As you probably know, except democoding I do some application development from time to time, namely zView is my kind of adopted child for a while :) And if you are a zView user, you might have noticed it uses so called LDG plugins to provide support for various image formats. People on the MiNT mailing list motivated me to fix a few illegal memory accesses (so zView can run flawlessly on FreeMiNT with memory protection) and the discussion had brought also a question of getting rid of the LDG plugin architecture as it’s common source of crashes and system instability (at least to the extent of people’s claims).

Generally speaking, LDG seems to be favoured by French developers but ignored by rest of the world, just because of its bugs and non-system approach. So one nice day I told myself to take a look, hopefully my findings will be interesting also for you. At this place I must thank Olivier Landemarre and specially Arnaud Bercegeay for nice discussions, it’s really much easier if you can talk directly to the developers :) By the way, don’t take this only as a guide to LDG but more like a source for thinking in case you decide to solve the shared library problem in FreeMiNT.

Introduction

So let’s start with little introduction... what actually LDG (http://ldg.sourceforge.net) is? LDG stands for “Librairies Dynamiques GEM” in French, “GEM Dynamical Libraries” in English. It has been developed by Olivier Landemarre, Dominique Béréziat and Arnaud Bercegeay for a few years, so it’s quite a mature product, especially after its 2.0 version. Its name could suggest its main purpose is to provide a mechanism for dynamic libraries and that is, ladies and gentlemen, the first mistake you do in your LDG evaluation.

The purpose is, surprisingly, to provide a mechanism to easily code plugins and extensions for your client application (where an LDG plugin serves as a server -- providing some functionality useful for the application) without a need for recompilation. To be honest, had I known about LDG in 2006, I’d definitely have used it for mxPlay (http://mxplay.sourceforge.net). The option to make the plugins shareable (i.e. available not only to your application but to others as well) appeared much later.

What makes it so useful in this case? If you ever coded a plugin, you stood against a task how to make your plugin visible to the rest of the world, i.e. you had to define some API for function calls, a way to pass arguments and receive return values, … If you are fine with plugins done completely in assembler, it’s relatively easy, you define a header, for example:

dc.l my_init_function

dc.l my_deinit_function

dc.l my_function_called_every_iteration

and then you load the plugin with Pexec( PE_LOAD, … ), get address of the text segment from the base page pointer and cast it to your C structure, for example:

struct MyPlugin

{

  int (*init)( void );

  int (*deinit)( void );

  int (*process)( int param );

};

The troubles are obvious:

The LDG way

LDG solves first two problems and partially also the third one. How so? I’m not going to supply developer’s documentation here but let’s make fast overview. As a server (LDG plugin) you do:

  1. In your plugin you define similar header as in our assembler example but with the difference you are free to use C language. The header contains pointer to each function, its name and description (!)
  2. This header is referenced by a pointer in another header which says what file extension it handles, if its shareable (re-entrant) and some other functions.
  3. All you need to do is to call ldg_init() with a pointer to this structure and that’s it.

As a client (application) you do:

  1. Application looks for all plugins in the LDG path (taken from cookie jar, environment variable, application directory or even some absolute one, defined by programmer).
  2. Using function ldg_open() application loads the plugin.
  3. Using function ldg_find() we actually look for a name (!) of the function we are interested in (usually “init”, then for example “decode_image”).
  4. Application stores function pointers for later usage (usually in some array).

And that’s it! As soon as you want to use some of your plugins, you just iterate through the array of loaded plugins, compare the file extension(s) it supports with the current file you want to process and pass execution to the plugin. And this is of course only one possible scenario, you can for example make variable number of functions in each plugin, add meaningful description to each function and directly hook them into application menu bar. It’s definitely not restricted only for file extension-based processing or image processing, not at all!

As long as you keep your plugins “local”, i.e. not shareable, you are free of any complications, you can do whatever you want with them. You can even load and unload them every time you (don’t) need them to spare some memory. This could conclude our story and motivate you to create some nice LDG plugins and/or apps... however, I was curious about the other case, what if I do want to make them shareable and re-entrant?

Reentrancy

What does it actually mean? Reentrancy is a term bound to multitasking environment where execution of one application (process) can be interrupted and execution of another application (process) is possible. To user it seems like applications (processes) are running in the same time.

If you want to make your function or library reentrant, you must ensure it can be interrupted in the middle of its execution and then safely called again ("re-entered") before its previous invocations complete executing what is exactly the case for shareable library.

In plain English it means you must make sure if your function is running, system interrupts its execution and the function is called from another thread/process that it will run correctly afterwards and of course, it finishes its execution correctly as soon as the initial context is active again.

In practice, you must avoid accessing system resources given to one process from another one. Typical system resources are memory and file descriptors. In a system with real support for dynamic libraries most situations do not pose a problem because only the text (code) segment is shared and every instance of library has its own data segment (bss and heap memory, too). Why it is so good you will see in a short while.

Shareable LDG plugins

Since neither FreeMiNT, MagiC nor TOS offers support for dynamic-link libraries (for full featured support you’d need a proper virtual memory implementation in the kernel), it opens door for various home cooked solutions. One example are MagiC’s SLBs (FreeMiNT and TOS with BetaDOS support them, too), another one is the LDG. Every approach has its advantages and disadvantages but since we focus on LDG here, we’ll discuss the LDG approach in detail.

From the user/programmer point of view, nothing has changed, whole sharing mechanism is hidden inside LDG internals, one just must set a flag in the LDG plugin header as shareable and put the LDG manager into AUTO folder (without it it’s impossible to have shared plugins). It’s not that important  how exactly does it work, the more important thing is how plugins are loaded and accessed:

  1. Client application requests an LDG plugin to load.
  2. Client side of the LDG library (the library which handles LDG plugins, libldg.a)  loads & executes the LDG plugin (the result is we get an address of the LDG plugin header discussed above).
  3. Server side of the LDG library is initialized via plugin’s main() function (which is naturally executed with the Pexec() call made by the client side) and, this is important, the main() function terminates shortly thereafter.
  4. Another application requests the same LDG plugin to load.
  5. Client side of the LDG library realizes the plugin is already in memory and returns only a pointer to the LDG plugin header, i.e. plugin’s main() is not called again.

Now, what does it mean for our application code? We’ve got an LDG handle from ldg_open() and we can use this handle for functions like the mentioned ldg_find(). In reality, what we get from ldg_find() is a function pointer to the LDG plugin.

Without the LDG manager, every application loads its own LDG plugin using mentioned Pexec() mechanism (points 1-3). So the plugin is loaded into the address space of the parent application and cannot be accessed by other processes. However, if the LDG manager is present, its ldg_open() (accessible from the cookie jar, don’t worry, this is done for you automatically) version loads the plugin into shared memory and make it accessible for all processes. This is the first difference against classic shared libraries -- the plugin code is accessible for everyone, not only the process which asked for the library.

So far, so good. Some badass application could possibly destroy/corrupt our plugin code but since we are Atarians, we can live with that ;-) So what’s the problem with the shared code? Surprisingly it’s … shared data.

I’ve already mentioned that in real operating systems the data segment (and bss + heap memory) is not shared, i.e. each process using the library has its own copy (it’s not 100% like that but in fact, it is ;-)). On the other hand, we’ve just learned about the LDG sharing mechanism -- we get something like pointer to whole application image. That means not only the text (code) segment is in memory only once but the data, too.

This introduces whole lot of new problems, consider this one:

static FILE* pFs;

static char* pBuffer;

static const int size = 10000;

// init function called upon a start of the library

void init( const char* filename )

{

        pFs = fopen( filename, “rb” );

        pBuffer = (char*)malloc( size );

}

void loadPicture()

{

        fread( pBuffer, size, 1, pFs );

}

This, particularly not dazzling, piece of code is fine if you link it as a static library and call init() and loadPicture() afterwards. But what happens if you link it as an LDG plugin? Imagine you call init() from process P1 and loadPicture() from process P2. Formally its fine, pFs and pBuffer are initialized but as soon as P2 calls loadPicture() we are in trouble because pFs and pBuffer belong to P1 (P1 loaded and accessed the code so from the code execution point of view it’s like init() is part of P1). Consider this example:

static const int size = 10000;

void loadPicture( const char* filename )

{

FILE* pFs;

char* pBuffer;

        pFs = fopen( filename, “rb” );

        pBuffer = (char*)malloc( size );

        fread( pBuffer, size, 1, pFs );

        ...

}

No matter what we do, we are always on the safe side -- pFs and pBuffer are placed on stack and are initialized per process, i.e. P1 receives its own file descriptor and memory pointer, the same for P2. Please note it’s no problem to use size as its read-only memory location, it makes no harm to access it from both processes, it’s a truly shared memory location. Why stack do not pose a problem? Because stack is of course separate for each process -- so everytime P1 or P2 enters the loadPicture() function, new variables are created. And then of course, from the same process context, the file descriptor and memory pointer is returned.

Just for the sake of completeness, let’s compare this behavior to a classic dynamic-link library. Our first example would fail too but for a different reason. If we call init() from P1, we initialize the static variables in P1’s address space. Then, if we call loadPicture() from P2, we would refer to the original (uninitialized!) state of variables because what? Because each process received the initial copy of the data segment. The latter example works fine because size is copied and does not change over the time of execution.

Hidden troubles

One could think it’s still quite good  -- we are now educated, we’ve been taught what to avoid, how not to code LDG plugin functions so … it’s only the bright future ahead of us, isn’t it? No, it is not :)

First we must make ourselves clear about what plugins we are talking about. The easier option is not to use gcc, better said, its libc (mintlib in our case). So in whole plugin we use only functions like Fopen() (and not fopen), Cconws() (and not printf), Mxalloc() (and not malloc) etc and a really simple startup code. Most likely plugin like that will be compiled by PureC (of course, without using its C libraries as well, we link only the startup code!) or completely in assembler or with gcc but avoiding its libraries using -nostdlib command line switch (and supplying our own startup code like here at Saulot’s website: http://bus-error.nokturnal.pl/tiki-read_article.php?articleId=12). This provides us with a version of the plugin which doesn’t do anything behind our back because it calls only our and/or system functions.

Common problem regardless which approach you choose is the fact each process kind of abuses plugin’s functions and pretends those functions and data belong only to that process. That results in very bad things even if you are careful and allocate all shared buffers as global memory. Imagine a shared buffer between two processes (i.e. a typical situation when two applications request usage of one plugin). P1 does something to the buffer, then P2, then P1 again … and then, out of nowhere, the application which P1 belongs to is terminated. In case we are unlucky and the shared buffer in question was allocated by P1, we are back to the ground zero because the buffer is released by OS as the process which allocated the buffer is now finished. And no, the fact we allocated the buffer as shared (global) memory doesn’t help a bit.

Using TOS functions is not only uncomfortable (and it makes porting of libraries very hard) but might be also very ineffective -- applications/libraries typically do a lot of allocation requests, in case we rewrite every malloc() as Mxalloc(), your memory would become very fragmented after a while. So we take the path of gcc/mintlib … what does it mean for LDG shared plugins?

First problem is that user malloc() calls aren’t the only source of memory allocation, at least as far as mintlib is concerned. In other words, there are some malloc() calls done behind your back. Try to grep mintlib source code for those calls, you’ll be surprised on how many places the allocation is done. Of course, most of the calls is harmless because free() is called in the same function, so other process wont use the allocated memory. However, there are calls like in fopen() -- the allocated memory is used as long as the file isn’t closed again and this memory is accessed even if you are careful and don’t use the file descriptor from another process.

Here I take a little detour, to explain how it’s possible we have a problem like this. You might think this totally contradicts what I’ve written above, that every time we call a function, OS thinks it’s part of the process which called the function. This is true, as far as the text (code) segment is concerned. We must realize one thing -- on Atari, you always link the C library statically. That means, if we call malloc() (which is of course a mintlib function), we call it as a function linked to the plugin, not as a function linked to the process which called the function (that would be case for Mxalloc(), it’s a system call). And here comes the fun. As I’ve mentioned above, malloc() isn’t just an alias for Mxalloc(), it takes whole bunch of memory from OS and then it “distributes” for the application which calls malloc() -- only after the allocated buffer is gone it asks OS for another one. So what happens if two processes access the plugin and do two, totally independent malloc()’s ? We can perfectly use your latest loadPicture() as an example. As we can see, there’s no fclose() nor free() called. So what happens if it is called by P1 and then by P2? P1 calls loadPicture(), that calls plugin’s (!) fopen() (and as we know, it does a hidden malloc() there but it doesn’t matter, we do an explicit malloc() shortly afterwards). So plugin’s mintlib asks OS for some memory. Let’s say OS gives it four times bigger buffer. P1 gets a pointer, allocates buffer & loads the picture, no problem. Then P2 calls the same function. According to my previous explanation, it should be fine, we call it from the P2 context, we use only stack allocated variables but … we are unaware of the fopen() / malloc() internals, i.e. we don’t know the next malloc() wont take any memory from OS (in that case we would get memory associated to the P2 context) but from the preallocated buffer! And to whom the buffer belongs? Yeah, to the P1, because P1 got the buffer from OS! So unless all your plugin functions return all the memory back to OS after each call, you’re in big trouble. And please note, there’s no guarantee the malloc() implementation will behave like that, i.e. as soon as you release the last memory block, it will return whole preallocated block to OS, it might keep it for itself till the whole process (that is, P1 in our case) is terminated!

Second problem is the fact we are actually working with dead code, from the OS point of view. As I have mentioned, our main() function actually lives a very short life, in 99% cases it serves only as a way to initialize LDG. That means as soon as the function ends, the process (the LDG plugin process) is officially terminated. Every file descriptor is closed, every memory allocated is released back to OS. So any function inside the LDG plugin which depends on mintlib initialization code (for example the list of environment variables -- remember, we again call the plugin’s code for getenv() and this function might depend on a buffer allocated by pre-main() code which is now released because the process had ended) wont work anymore.

Third problem has a little from all of the above. As we know yet, after the initialization phase, we are working with dead code. One implication has been already mentioned, another is following: we can’t really know what happens to functions like malloc() in case the process is terminated. Even though all memory was released to OS after main() had ended, we don’t know in what state malloc() is right now. It might depend on some variable, buffer, OS call, … made by the plugin’s pre-main() code. Or might not. It might be the case for mintlib only. Or might not.  And if you think about it, it’s not only a shared plugin problem, it touches also non-shared ones. You would be surprised what kind of side effects this has in FreeMiNT (everything seems fine), MagiC (real magic here, even things like the application filename count!) or TOS (immediate crash).

Solutions

Still here? :) Well, solutions. It might look like there’s no way how to solve that mess, i.e. that LDG is doomed forever. However, it isn’t that bad. Let’s take it case by case.

  1. Shared buffers

Those who are familiar with C++ or any other higher programming language know what “counted pointer” is. It’s a tool for memory allocation where memory is released as late as the last “client” claims it doesn’t need it anymore. So three processes can share a buffer but the buffer isn’t released when the process which allocated the buffer is terminated but when the last one is. Of course, this kind of memory allocation must be supported by OS. On Atari, FreeMiNT and MagiC provide a very nice interface via /dev/shm.

  1. Accessing other process’ memory

You can make your functions in that way they always allocate and release all the memory in each function (what is practically impossible) or use a custom malloc() wrapper which transforms all calls into global memory requests. Another option is to implement a custom wrapper which “distributes” the memory chunks per process, i.e. the new malloc() call would check the process ID and allocate memory for it from OS.

  1. Accessing already released memory

This is solvable using custom startup code (which doesn’t allocate any buffers) but of course, it may reduce number of functions which you can use (for example the mentioned getenv() call). Or you can make all mintlib functions context based per process (more on this later).

  1. Calling functions in unknown state

Practically the same solution here. Don’t use plugin’s C library, use the library of the process which called the plugin function (this may lead to some incompatibilities in case mintlib API changes).

libshare

Of course, I’m not the first person on this planet who realized those problems. Authors of the LDG were ;) Arnaud Bercegeay had developed for this purpose a library called libshare (http://arnaud.bercegeay.free.fr/libshare). It partially solves the problems we’ve been talking about, it works in conjunction with mintlib because the C library of PureC is more or less safe to use for shared plugins.

Shared buffers are, as I’ve said, unsolvable without an OS support. libshare contains some experimental code which poses as a wrapper to malloc() but it’s untested. And of course, it limits the availability for FreeMiNT and MagiC only as TOS has no way how to keep a memory after a process is terminated (ok, this is not 100% true -- we can use the Ptermres() function but in that case, we couldn’t release the memory EVER). So one would need to code a “driver” for TOS which imitates /dev/shm behavior. LDG manager is a good candidate for that.

Accessing other process’s memory is solved quite straightforwardly, libshare takes advantage of gcc’s weak_alias directive which makes replacement of some C library calls possible. So every malloc() call is replaced with a function, which allocates memory as global, i.e. shared. So fopen() and friends are again safe to use but for the prize even all buffers (not only code, as I’ve described in previous chapters) are now global.

Dead code problem is solved only partially -- as libshare replaces the standard malloc() call, we are now safe because it doesn’t depend on any startup code anymore. However, it doesn’t replace every function (naturally) so for example the mentioned getenv() call still poses a risk. Plus functions like printf() don’t work either, same reason.

There’s one nearly ideal solution: usage of contexts, similar as gemlib does. That means every process would be given something as this pointer and every C library function would be redefined:

#include "libshare.h"

my_ldg_function( CONTEXT *c, int param1, int param2...)

{

  ...

  getenv("blabla");

  ...

  fp=fopen("aaa","r");

  ...

}

with in libshare.h

#define getenv c->getenv

#define fopen c->fopen

so the plugin would be linked without any C library dependency, every function would be called from the process context. This would fix all issues -- there wouldn’t be any dead code anymore, malloc(), printf(), getenv() etc calls would be called from the right context, simply -- perfect.

Interesting thing would be compilation of libraries like libjpeg which a programmer would want to use in his plugin -- these libraries use C library declarations but in the end, they don’t link with the library itself (library can’t be linked to another library). This poses a little problem for us since we must solve all external dependencies statically. I.e. our plugin, which wants to use  libjpeg functions, must provide a way how to tell the library it shouldn’t use plugin’s mintlib but the context calls instead. But this is easy -- we replace all include directives for stdio.h and friends with libshare.h, as in our example above. Or, if we don’t want to mess with the library (we would have to have two version of the library then -- original one for normal usage and this modified one for LDG plugins) we can use a wrapper inside libshare like:

CONTEXT* c;        // global context initialized by the server process

FILE * fopen ( const char * filename, const char * mode )

{

    return c->fopen( filename, mode );

}

In that case we can freely use stdio.h and friends but instead of linking to plugin’s mintlib we would link it to libshare. Of course, we must be super careful to have function signatures right else stack would get corrupted pretty soon. But since C and POSIX standards don’t change every day... it’s quite safe.

But you must be careful when it comes to shared plugins (again!). You must ensure each process is able to supply own context pointer -- you can’t call a context of P2 using a call from P1! Also termination of P1 in that case would lead to disaster. Simply put, you must ensure your context pointers are always paired with the correct processes.

Trouble is no one did this yet ;-) I’ve coded a proof-of-concept implementation for zView and its two plugins and the results are amazing, godpaint.ldg is like 20x smaller, jpg.ldg is smaller by ~100 KB. Yeah kids, mintlib is huge and bloated in combination with ineffective a.out format.

Real world usage

This is all nice, you say, but what does it mean? Shall I use the LDG or not? It is safe? Isn’t? Here is a compilation of my advices:

  1. PureC (or gcc with custom startup code) + only system calls in whole plugin + non-shared flag = the safest combination. My experience shows it’s fine to use also normal mintlib startup code but your binary will grow about 50 KBs for no reason.

  1. Using non-shared plugins must not be that ineffective. You can load the plugin, let it do its work and then unload again.

  1. If you don’t want to use any external libraries like libjpeg or you are going to code in PureC, respectivelly, just keep in mind the shared buffer issue. Don’t make your buffers shared between two applications, you never know which one terminates in the most inappropriate time. In most situations, this will never happen. You are not likely to let two application load one picture. Maybe some parallel fractal calculators would be a problem :)

  1. If you do want to use external libraries or you are simply lazy to use system calls only, feel free to use mintlib. Only thing you need to do is to link it with libshare and add  libshare_init() and libshare_exit() to your plugin code.

  1. Don’t forget libshare wont solve the shared buffers problem.

You see? libshare is pretty clever.

Conclusion

Ok, I’m pretty tired, I’m sure you too :) I hope it’s clearer to you what are the most visible pitfalls when it comes to LDG programming and that LDG plugins aren’t any worse than normal applications. It’s true there were some crashes caused by the LDG “kernel” (library) but they are fixed now.

For the future, my wish is someone would take a look on the two problems I mentioned: shared memory manager (implemented in the LDG manager, where applicable /dev/shm would be used else custom OS wrapper would be installed) and context-based libc wrapper. With this done, we would get pretty good workaround for the lack of dynamic-link libraries.

Miro Kropacek aka MiKRO / Mystic Bytes

18.12.2011

Bratislava / Slovakia

http://mikro.atari.org