Titan

Macro-Powered FFI for LuaJIT

About Me

• Wrote the Haxe Lua target
• Working on ML/Data Science at Salesforce.com
• Part of Haxe Community since 2006(!)
• Living in beautiful Seattle!
• Looking to start some more Seattle area Haxe meetups, message me if interested!

Titan

• Second largest moon in solar system
• Promising planet for colonization!
• The only other dense atmosphere in solar system
• Abundant hydrocarbons and nitrogen
• Days are 360 hours long
• Oceans of liquid methane instead of water
• Surface Temperature 98.29 K (−179 °C, or −290 °F)
• Atmosphere mostly nitrogen

​

Titan and Lua - Moons and Adaptation

• Lua is Portuguese for “Moon” (Brazilian)
• Lua has had success in quickly adapting fast C code with its interpreter
• LuaJIT improves greatly on the speed of the interpreter
• LuaJIT has a great FFI
• How can we adapt LuaJIT FFI with Haxe and adapt to foreign environments/runtimes?

​

What is an FFI? Why should I care?

• Foreign Function Interfaces (FFI) enable functions written in one language to be called from another.
• Typically people are interested in calling fast/compiled functions from slow/interpreted languages.
• This can greatly speed up performance in certain cases, but involves a lot of abstractions between language concepts.

Haxe Code

Foreign Function Interface

Compiled C/C++ Code

What are Problems with FFI?

• Code now consists of three abstractions:
• How Haxe interprets the function
• How the FFI interprets the function
• How C interprets the function
• The surface area for bugs is greatly increased
• Shipping code now often involves shipping compiled binaries
• Difficult to experiment with in general

LuaJIT FFI

• LuaJIT incurs very low overhead for FFI
• Lua itself was designed as a scripting layer for ANSI C
• LuaJIT FFI written by Mike Pall (LuaJIT Author)

Source : https://github.com/dyu/ffi-overhead

FFI Overhead (cumulative MS)

LuaJIT FFI Usage

• Load the FFI library
• Add a C declaration for the function (in double brackets).
• Call the named C function
• That’s it!
• No really… move to the next slide

LuaJIT FFI Idioms

LuaJIT maps common C value/accessor constructs to and from Lua idioms.

​

Some of these are necessarily awkward due to lack of direct Lua support (Pointer arithmetic, dereference)

LuaJIT FFI Conversions

C to Lua

Lua to C

LuaJIT FFI headaches

• Standard value types (numerics, booleans, etc) are easy to convert, but lose resolution (single numeric type)
• Table/Array conversion is “quirky”
• 0 vs 1 based indexing scheme is detected rather than specified
• Value “padding” based on first value
• Null values terminate array
• Structs and methods for cdefs need signatures.
• Normally this is copy pasted, but could require manual tweaking
• Awkward to insert C syntax as an inline string in a Lua source file
• Directly importing C headers, and “doing the right thing” would be a big win… but it’s not that easy.

C Headers and C Macros

• C Headers are complicated by C Macros/pragmas
• It is difficult to know what exactly is exported by a header unless you try to compile it.
• There’s many other parts of a header file that are useless to an FFI implementation.

Parsing with Clang

• Clang has a special “ast” mode:
• $> clang -ast-dump some.h
• Parser follows compiler behaviors, so it’s possible to follow pragma/metadata directives
• Easy to filter for relevant AST nodes (FunctionDecl, struct)
• Fast, and broadly available

​

Titan Workflow

Titan uses a compiler macro to define the header location*

*Folder support pending

Titan only supports the Lua target with the LuaJIT runtime

​

Titan Workflow

In the class file, all imported C method definitions are placed in the “Titan” namespace, and are accessible as static methods.

Structs are abstracts with constructors.

Titan Development Patterns

Titan loads all C FFI interfaces to a single namespace (Titan), since all C methods are loaded into a single namespace as well.

Organizing the FFI methods into modules, etc. is up to the developer. E.g. a cross-platform function for “sleep” might look like this =>

Don’t forget to include the compiled libraries when you distribute your code!

​

​

What can you do with Titan?

Titan works well with single-file libraries, such as:

• Nuklear
• Imgui
• Klib
• mmx

Large multi-file libraries are possible, but painful for various platform specific reasons

​

​

nuklear

imgui

Recap

​

• C only has one* namespace, so FFI either needs to have a single namespace, or do some fancy extern/exporting
• JIT-style interpreters + FFI can be fast. In some cases faster than C!
• Clang can enable the Haxe compiler to “see” the interface exactly as a C compiler would (resolving macros/pragmas/etc)
• LuaJIT has a fantastic FFI C interface, but this is mainly due to the design of Lua (Scripting for Ansi C), and the skill of the LuaJIT creator (Mike Pall)

* Technically 4 namespaces : tags, types, labels, members of typed unions

When can I use Titan?

Titan is still not quite finished…

Some remaining issues:

1. Finding good GC patterns
2. Handling more complex libraries with additional dependencies
3. Managing the process for manual configuration vs. automatic code generation

​

​

​

​

The End!

​

Any questions?

​