DLLs with JNative
From Processing
Here is an example of calling native functions (in this case, a vendor-supplied Windows .DLL that talks to the vendor's gadget) from within Processing, using a native wrapper library called JNative. This hack assumes you have no access to the DLL's source code and cannot recompile it with your own modifications. Using JNative you can accomplish this without access to the DLL source, and without writing or compiling any native code.
Functionally JNative seems to be great, but the documentation is a bit sparse for a newb trying to use it for the first time (the homepage's offer of documentation in your choice of "English" or "Javadoc" is more apt than the author probably intended). This tutorial shows both simple native function calls and a more advanced feature (callbacks). The JNative site already does an all right job of explaining some other advanced stuff, like creating and passing structures back and forth to native code.
Applications
Some examples of where you might want to (or be forced to) tap into native code are: 1) to run a big math-heavy calculation as quickly as possible, 2) to interface with a specific piece of hardware via its vendor-supplied drivers/DLLs, 3) to 'borrow' handy routines from a native program (compression algorithms, video decoders etc.)
In this example I use Processing to initialize and receive a response from a 2.4GHz ANT radio attached to the USB port, through the manufacturer's supplied interface DLL, whose file name is ANT_DLL.DLL.
Supplies Needed
- JNative
- Processing (tested on version 0144 Beta, which uses javac as the compiler)
- The DLL's API documentation (or header file showing the exported functions)
This hack assumes you at least have access to a published API or header files which list the DLL's exported functions, what arguments they take, and what calling convention they use (such as "cdecl" or "stdcall"). A calling convention is an agreement between the called and calling functions about how the arguments are actually passed, in which order, etc. In any case you must pass the exact number of bytes the function is expecting, and receive the exact number of bytes it returns, so doublecheck those API docs and make sure you get them right! (There are advanced ways to get/guess this information without having the API docs, but that kind of hacking is pretty well beyond the scope of this tutorial!)
Ideally, you have a header file containing lines similar to the following:
__declspec(dllexport) int gadget_init(unsigned int stuff, unsigned char morestuff);
Install JNative as a Processing library
Unzip the JNative zip file to a temporary location. Under your Processing directory (e.g. \processing-0144\), create the directory \libraries\JNative\library and place JNative.jar and JNativeCpp.dll in this directory. The \libraries\xxxxx\library\ directory structure is how Processing identifies its installed libraries; if you placed it exactly as shown, "JNative" will now appear in the //Sketch -> Import Library// menu next time you start Processing.
Create Java files for your sketch
Once you've created a new sketch, create a .java file whose name exactly matches the name of the class(es) you will be creating. It's CaSe-sEnsiTive! For example, my DLL is named ANT_DLL.DLL, so I will create a Java class called ANT_DLL to encapsulate all the DLL functions. (Don't know Java? Neither do I! Don't worry about it, just read on for a bit.) To do this, you will add a new tab(s) to your Processing sketch using the arrow button at top-right, one for each Java class you create (e.g. if you have to work with more than one DLL, or implement callback classes), and type in the appropriate filename (ClassName.java). If you need to create more classes, repeat this step as necessary. A class is just a collection of Java functions and variables that you can call from elsewhere, wrap up in bigger classes, etc. (There's more to it, but that's a topic for your first Java programming course...)
Don't forget to //"Import Library -> JNative"// for each tab. This will add a long list of lines like these to the beginning of your code:
Then add the following at the very beginning, before all of those lines:
package org.xvolks.jnative.util;
Write the classes (and keep Processing from messing with them)
We are now working without a net, in that naked Java world that Processing tries to shield you from. The .java file extension is special; it tells Processing not to mess with it. As a result some of the messy details (classes, inheritance, etc.) are no longer taken care of for you. So you have to...
Add the following line to the beginning of your .java file:
import processing.core.*;
Create an empty class as shown (named according to your own DLL of course):
Note these secret incantations (import processing.core.*, "extends PApplet") apply to any naked Java classes you add to a Processing sketch (it has nothing to do with JNative).
Now actually write the class - this you pull pretty much directly from JNative's examples, following the API of your DLL. Note your vendor may have forgotten that compilers often mangle (or "decorate") the function names when building the DLL, so the function they prototyped as "doStuff()" may actually appear in the DLL as _doStuff, or doStuff@16, etc. If you get any "NativeException: Function doStuff not found" type errors using the vendor-supplied info, use a DLL export viewer such as http://www.nirsoft.net/utils/dll_export_viewer.html to find the real (mangled) names.
Basic example: write the actual native interface class
public class ANT_DLL extends PApplet { public static final String DLL_NAME = "ANT_DLL.DLL"; //Cache the JNative object between calls. private static JNative nANT_Init; // Native-side function names private static JNative nANT_AssignResponseFunction; //one for each DLL function you will use... // The actual DLL export, most likely from API documentation / header file: // __declspec(dllexport) BOOL ANT_Init(UCHAR ucUSBDeviceNum, USHORT usBaudrate); // Java-side function name public static int ANT_Init(byte ucUSBDeviceNum, short usBaudrate) throws NativeException, IllegalAccessException { if(nANT_Init == null) { nANT_Init = new JNative(DLL_NAME, "_ANT_Init"); // The actual "decorated" name in the DLL //BOOL is in fact an INT nANT_Init.setRetVal(Type.INT); // tell JNative's DLL what kind of return data to expect } nANT_Init.setParameter(0, ucUSBDeviceNum); // tell JNative's DLL what data to pass to the native function, in what order. nANT_Init.setParameter(1, usBaudrate); // ... nANT_Init.invoke(); // Finally, execute the native function return nANT_Init.getRetValAsInt(); // get native call's return value } }
In case you didn't notice, that's a whole lot of typing just to implement one lousy function. Hopefully you don't have a huge number of them :-) Luckily you only need to write this .java file once, then can re-use it in future projects.
Now the final step...
Call the function from your main Processing sketch
import org.xvolks.test.bug.*; // [... etc. ...] void setup() { JNative.setLoggingEnabled(true); // show what JNative is doing! try { int blah=org.xvolks.jnative.util.ANT_DLL.ANT_Init((byte)0,(short)57600); // Call the first native function. This initializes the USB radio via the manufacturer's DLL. print(blah); // should return '1' upon success. This was easy so far! } catch (NativeException e) // traps in case anything naughty happens { e.printStackTrace(); } catch (IllegalAccessException e) { e.printStackTrace(); } } void draw() { }
More advanced example... callbacks using JNative
Normally, you control the flow of execution explicitly. You call the function in the DLL, it runs, and your program keeps right on chugging. But sometimes, you need a function call to be triggered by something external to your program's normal execution flow. I.e. rather than you calling a function in the DLL, also allow the DLL to call functions in *your* code from out of nowhere (In Soviet Russia...). This is known as a callback.
Implementing a callback is straightforward when you're dealing with all native code--just pass the DLL a pointer to the function, and to any buffers, etc.-- but trying to do this cross-platform presents challenges. Java and the native DLL cannot access each others' memory, and Java is meant to be a high-level language where you should never have to know or care where variables or functions are located in physical memory (i.e. pointers). More to the point, since you should never need this information, Java does not allow you to get it in the first place. Even if you did get pointers to things in Javaland, they wouldn't be valid anyway since the two can't access each others' memory. JNative provides builtin functions to deal with these problems: for example, one to get function pointers which are meaningful on the native side, one to allocate memory blocks (e.g. buffers) on the native side to mirror the ones in your Java callback function, and one to copy data between the native memory block and the Java-side memory block it mirrors.
First, create a class to implement the callback function
Create a new .java tab with the name of the class. In my case I named the callback class ResponseFunc, so create ResponseFunc.java. Write your callback function (what actually handles the callback), plus the required getCallbackAddress() function as shown below.
In my case the callback function also has to receive data from a native memory buffer, so I create the identical buffer on the Java side (jResponseBuffer) and use it as shown. Creating the matching buffer on the native side will be shown in a later step.
package org.xvolks.jnative.util; import processing.core.*; import org.xvolks.test.bug.*; //[ ... ] public class ResponseFunc implements Callback { private byte[] jResponseBuffer = new byte[16]; // Java-side buffer to hold the data copied over from native-side buffer private int myAddress = -1; // Here is the function that will actually be registered, and called when the event occurs public int callback(long[] args) // in my case, the native function doesn't pass any args, so 'args' is not used below. { try { jResponseBuffer=org.xvolks.jnative.util.ANT_DLL.pResponseBuffer.getMemory(); // bring the data buffer contents from the native side over to the Java side System.out.println("Callbacked!\n"); System.out.printf("ResponseBuffer contains %02X %02X %02X %02X %02X\n", jResponseBuffer[0], jResponseBuffer[1], jResponseBuffer[2], jResponseBuffer[3], jResponseBuffer[4]); // etc. // Here we would actually do something useful with the data... } catch (NativeException e) { e.printStackTrace(); } finally // compiler says I need this { } return 1; } // Support function to create an instance of the Java-side callback function ("callback(long[] args)" above) and get an "address" we can pass to the native function who registers the callback. // I have absolutely no idea how this works under-the-hood, but it does. public int getCallbackAddress() throws NativeException { if(myAddress == -1) { myAddress = JNative.createCallback(0, this); // Create a new instance of the callback function and get its address (expressed as an Int) // ^^^ First value specifies how many arguments to pass to your callback function. 2nd argument is the Callback object (function) that will actually be called. //'this' is a special keyword that specifies the name of *this* class (starting with the 'public class ResponseFunc implements Callback' above). // the function named 'callback' in this class is what will actually be called when a callback occurs. } return myAddress; // Int containing the equivalent of a function pointer(?) } }
Creating native-side memory buffers; registering the callback and native buffer
As mentioned earlier, the DLL sitting on the native side cannot write to a Java memory buffer, so you can't pass data around as easily as between a pair of C functions in the same program. So we modify the ANT_DLL class a bit to use JNative's "MemoryBlockFactory" to allocate a buffer (memory block) on the native side.
Before the DLL can call any callbacks in your code, you have to //register// the callback. If your DLL does callbacks, it will have a function that allows you to pass in the address of your callback function and any associated stuff. In this case it is ANT_AssignResponseFunction(...).
The ANT_DLL class now looks like:
public class ANT_DLL extends PApplet { public static Pointer pResponseBuffer; // Here we will store the native pointer to a native byte buffer static { try { pResponseBuffer = new Pointer(MemoryBlockFactory.createMemoryBlock(16)); // Create a 16-byte buffer on the native side. When calling the callback function, the native DLL will write the incoming radio data here. } catch (NativeException e) { e.printStackTrace(); throw new RuntimeException(e); } } public static final String DLL_NAME = "ANT_DLL.DLL"; // [... cut some stuff already shown in the above example ...] // This native DLL function sets up the callback, by passing in the native function pointer and a pointer to a byte buffer to write the data into. public static void ANT_AssignResponseFunction(int pResponseFunc, Pointer pResponseBuffer) throws NativeException, IllegalAccessException { if(nANT_AssignResponseFunction == null) { nANT_AssignResponseFunction = new JNative(DLL_NAME, "_ANT_AssignResponseFunction"); //setRetVal...don't need because it's void? } nANT_AssignResponseFunction.setParameter(0, pResponseFunc); // address of callback fcn nANT_AssignResponseFunction.setParameter(1, pResponseBuffer); nANT_AssignResponseFunction.invoke(); } }
Use these functions in the main sketch
Here the sketch is amended to now 1) create an instance of ResponseFunc (our callback), 2) get its address, 3) pass this address and the pResponseBuffer pointer to the native DLL using its AssignResponseFunction(...).
void setup() { JNative.setLoggingEnabled(true); // show what JNative is doing! try { int blah=org.xvolks.jnative.util.ANT_DLL.ANT_Init((byte)0,(short)57600); // Call the first native function. This initializes the USB radio via the manufacturer's DLL. print(blah); // should return '1' upon success. This was easy so far! print("\n"); // Now for the messier part. When a radio event occurs (e.g. we get a transmission, or radio acknowledges we did something to it), it will asynchronously generate a callback to a user-defined function to handle the incoming transmission. // Normally, we get a pointer to this handler function's memory address, and pass it as an argument to a 'register callback' function, and a pointer to a byte buffer to place the incoming data into. // But now, our handler function is way out here in Java-land. So that requires we can find out the memory address of a Java function (not going to happen!) and somehow let a C function write into a Java byte array (not going to happen!) // Luckily, Jnative has some tricks up its sleeve to approximate these functionalities. print("getAvCallbacks " + JNative.getAvailableCallbacks()); // find out how many callbacks JNative supports at once (says 1000 on my machine) // OK, now create an instance of the (Java) callback function that will be called when USB gadget gets a packet. // Then get a native-side pointer to the function, and pass it to the DLL to assign the callback function pointer. ResponseFunc r = new ResponseFunc(); // create an instance of the Java callback function that will handle the data int ResponseFuncAddr=0; try { ResponseFuncAddr=r.getCallbackAddress(); // JNative's trick: I have no idea how it works, but JNative gets an address for this function that can be passed into the native side. } catch (NativeException e) { e.printStackTrace(); } org.xvolks.jnative.util.ANT_DLL.ANT_AssignResponseFunction(ResponseFuncAddr, org.xvolks.jnative.util.ANT_DLL.pResponseBuffer); // The vendor function that registers the callback. We pass it a pointer to the callback function (data handler), and a pointer to a native buffer to stuff the data in. org.xvolks.jnative.util.ANT_DLL.ANT_AssignChannel((byte)0, (byte)64, (byte)0); //The moment of truth: We create a radio channel, and the radio should do some stuff and (a short time later) trigger our ResponseFunc callback, with response data indicating if the channel created successfully. } catch (NativeException e) // traps in case anything naughty happens { e.printStackTrace(); } catch (IllegalAccessException e) { e.printStackTrace(); } } void draw() { }
Run it and I get the following on the console:
Sep 6, 2008 1:45:29 AM, [DEBUG]Using org.xvolks.jnative.pointers.memory.HeapMemoryBlock memory reservation strategy Sep 6, 2008 1:45:29 AM, [DEBUG]Creating Lib info (name = ANT_DLL.DLL, handle = b2d0000, inUseFor = 1) 1 getAvCallbacks 1000 Sep 6, 2008 1:45:29 AM, [DEBUG]registering callback a47e8 Sep 6, 2008 1:45:29 AM, [DEBUG]Resusing Lib info (name = ANT_DLL.DLL, handle = b2d0000, inUseFor = 2) Sep 6, 2008 1:45:29 AM, [DEBUG]Resusing Lib info (name = ANT_DLL.DLL, handle = b2d0000, inUseFor = 3) Sep 6, 2008 1:45:29 AM, [DEBUG]in Java callback #a47e8 with 0 arguments Callbacked! ResponseBuffer contains 00 42 15 00 00
Success! [00 42 15...] is the expected response from the radio (technically it's indicating a radio error, because I ran the Processing code several times already and haven't written the channel CLOSE function yet. But as you see above, we've already done all the hard stuff.)
Some miscellaneous notes/shortcuts
IIRC, under any calling convention each parameter is actually passed as the full width of a CPU register (or a multiple of same, for e.g. LONGs), even if the actual data type is shorter (e.g. C 'char' or 'bool'). On most machines this is 32 bits. So you may safely pass an Integer type (32 bits) where the native function calls for a bool, char, short, etc., without crashing the machine. Useful to get around Java's lack of most 'unsigned' types by just passing in a slightly larger type (e.g. int for unsigned char). Just be sure not to pass in any value larger than the maximum size for the *native* type!
Downloads
FIXME: Add an example that doesn't depend on obscure hardware :-) (JNative homepage has some examples of tapping into Windows API functions)