Tag Archives: JNI

Java as a library, part 2

In the previuos blog I’ve made a Java class Grep that exports a method named doGrep(). I also supplied a Java test driver to test the Grep class. In this blog I will write the ALGOL program that takes the search text and directory as a parameter and calls the Java doGrep() method to do the actual work.

The utility JifGen has been run to generate the ALGOL include file that holds the definitions of the Java methods (GREP_INCL.ALG_M). I copied this ALGOL symbol to my MCP system as

My ALGOL program looks like this:

NEXT+   ....*....1....*....2....*....3....*....4....*....5....*....6....*....7..
00002000% grep                                                                  
00003000% Parameters:                                                           
00004000%   text:    String parameter of the text to search for.                
00005000%   path:    String parameter of file or directory to search.           
00006000%   recurse: Boolean, if TRUE all subdirectories will also be           
00007000%            searched.                                                  
00009000% Note: The Apache Software Licence applies to this code.               
00010000% (http://www.apache.org/licenses/LICENSE-2.0.html)                     
00012000$ tads                                                                  
00013000$$ set level 2                                                          
00014000procedure grep(arguments);                                              
00015000array arguments[*];                                                     
00020000    ARRAY JNIEnv[0] = envEtc;                                           
00021000    ARRAY JavaVM[-JNI_MIN_ENV_ARRAY_SIZE] = envEtc;                     
00022000    ebcdic value array jvmOptions ("-cp ./grep.jar " 48"00");           
00023000    ebcdic value array mcpOptions (48"00");                             
00024000    integer jvmRslt;                                                    
00025000    double returnArray;                                                 
00026000    ebcdic array param[0:299];                                          
00027000    ebcdic array ax[0:119];                                             
00028000    integer errorCode;                                                  
00029000    pointer pparam;                                                     
00029200    file lp(kind = printer);                                            
00029400    ebcdic array line[0:132];                                           
00031000$$  INCLUDE GREPJNI="GREP/SRC/INCLUDE/GREP_INCL"                        
00033000    procedure listHits(returnArray);                                    
00034000    value returnArray;                                                  
00035000    double returnArray;                                                 
00036000    begin                                                               
00038000        ebcdic array found[0:300];                                      
00039000        integer foundLen;                                               
00040000        double arrayElement;                                            
00041000        integer returnLength;                                           
00042000        integer i;                                                      
00044000        returnLength := JNI_GET_ARRAY_LENGTH(JNIEnv, returnArray);      
00046000        for i := 0 step 1 until (returnLength - 1) do                   
00047000        begin                                                           
00048000            replace found by " " for 300;                               
00049000            foundLen := 300;                                            
00050000            arrayElement := JNI_GET_OBJECT_ARRAY_ELEMENT                
00051000                (JNIEnv, returnArray, i);                               
00052000            JNI_GET_STRING_EBCDIC_CHARS                                 
00053000                (JNIEnv, arrayElement, found, 0, foundLen);             
00054000            if foundLen > 0 then                                        
00055000            begin                                                       
00056000                write(lp, *, found);                                    
00057000            end;                                                        
00058000        end;                                                            
00059000    end of listHits;                                                    
00061000    if (size(arguments) * 6) > 300 then                                 
00062000    begin                                                               
00063000        resize(param, size(arguments) * 6, DISCARD);                    
00064000    end;                                                                
00065000    replace pparam:param[0] by pointer(arguments)                       
00066000        for size(arguments) * 6 until = 48"00";                         
00068000    % Initialize the Java Virtual Machine                               
00069000    jvmRslt := JNI_CREATE_JAVA_VM_STAR(JavaVM, JNIEnv,                  
00070000                                       JNI_VERSION_1_6,                 
00071000                                       jvmOptions,                      
00072000                                       mcpOptions);                     
00072100    if jvmRslt < 0 then                                                 
00072200    begin                                                               
00072300        replace line by " " for 132;                                    
00072400        replace line by "Result from JNI_CREATE_JAVA_VM_STAR: ",        
00072500            jvmRslt for * digits;                                       
00072600        write(lp, 132, line);                                           
00072700    end                                                                 
00072800    else                                                                
00072900    begin                                                               
00073000        INITIALIZE_GREP_;                                               
00074000        returnArray := GREP__GREP_DO_GREP(param,                        
00075000            0, offset(pparam));                                         
00076000        if JNI_Check_Exception(JNIEnv) then                             
00077000        begin                                                           
00078000            errorCode := JNI_GET_LAST_ERROR(JNIEnv);                    
00079000            replace ax by "JNI Exception: Error code = ",               
00080000                errorCode for * digits, 48"00";                         
00081000            display(ax);                                                
00082000        end;                                                            
00083000        listHits(returnArray);                                          
00083500        JNI_DESTROY_JAVA_VM(JavaVM);                                    
00084000    end;                                                                
00085000end of grep.                                                            

Line 17000 includes the variables and procedure declarations that are necessary to communicate via JNI. It needs to be included before the include of the generated GREP/SRC/INCLUDE/GREP_INCL. The lines 19000 – 21000 declare some arrays that the generated ALGOL file needs. The comments at the top of the generated include file remind you of this.

Lines 22000 and 23000 define parameter strings that need to be supplied to the JNI_CREATE_JAVA_VM_STAR procedure (line 69000). This procedure will start a Java Virtual Machine on the JProcessor. The array jvmOptions sets the CLASSPATH of the JVM. In this program I have hardcoded the CLASSPATH relative to the CURRENTDIRECTORY attribute of the program. I a production system you would probably get these parameters from a configuration file.

The INIALIZE_GREP_ in line 73000 results in a call (via GREP/SRC/INCLUDE/GREP_INCL) of the JAVAJNILIB library procdeure JNI_FIND_CLASS_E. This procedure returns a handle to the Grep class.

Since I've made the doGrep() method a static method, I don't need to create an instance of the Grep class. I find this makes the ALGOL code less complicated. In a production system that I built, I defined a single class to interface between Java and ALGOL. A call to a static method in this class creates a single instance of a Manager class, the Manager class in turn, instantiates other objects as needed. This scheme can work because a JVM is dedicated to one program stack.

In line 74000, the actual doGrep method is called. The parameter param is an EBCDIC array that contains the arguments for doGrep. The next parameter, 0, is the offset into the param array where the actual arguments start. The last parameter gives the length, in characters, of the arguments string. The GREP__GREP_DO_GREP procedure in the generated include file wraps the actual, more complex, JNI call.

I started a test using the WFL below:

NEXT+   ....*....1....*....2....*....3....*....4....*....5....*....6....*....7..
00000100BEGIN JOB WFL/GREP;                                                     
00000300RUN GREP/GREPDRIVER(                                                    
00000350        "-r grep  /-/user/usercode/ruud/grep/src");                     
00000400    LIBRARY JAVAVM(LIBACCESS = BYTITLE,                                 
00000500        TITLE=*DIR/JRE6/BIN/JAVAPROXYLIB ON DISK);                      
00000600    CURRENTDIRECTORY="/-/USER/USERCODE/RUUD/GREP";                      
00000800END JOB 

The CURRENTDIRECTORY equation is, apparently, just for the GREP/GREPDRIVER program, it is not propagated to the Java Virtual Machine. That means that the path to directory to be searched by Grep must be an absolute path, it cannot be relative to the CURRENTDIRECTORY.

I also got a message “JAVAHOMEPATH not provided; JRE location extracted from proxy name (/-/DISK/DIR/JRE6).”. This message comes from the program *SYSTEM/JAVAFILE, which is used by the JVM to access the MCP file system. Adding a file equation FILE JAVAHOMEPATH(PATHNAME="/-/DISK/DIR/JRE6"); did not get rid of the message. But using the path from the PROXYLIB is just fine.

When I ran my test WFL, much to my surprise, I didn't get any results back. I figured the JNI_CREATE_JAVA_VM_STAR failed, that's why I added the code in lines 72300 – 72600. But there was no problem there, nor was there a problem in the GREP__GREP_DO_GREP call. Wouldn't it be nice if I could debug my Java code now. Well, I could, the way to do that was obvious: Just insert the debug arguments
-Xdebug -Xrunjdwp:transport=dt_socket,address=9000,server=y,suspend=y
at the beginning of the jvmOptions array and I could connect my Netbeans to the JVM under test. Stepping through the doGrep() Java code, the problem jumped right at me. The param.split(“ “) method splits the parameter string using a single space as a token separator, but my test job has two spaces between “grep” and “/-/user/....”. I have now changed the regular expression in the param.split to “ +”, telling the split to use one or more spaces to separate the string.

Java as a library Part 1

In the Unix/Linux world there is this utility called ‘grep’. It is a very handy tool that allows you to search for a string of text in one or more files of a directory. The original grep can handle wildcards in file names and regular expressions in the target text. My goal is to experiment with the Java Native Interface (JNI) between ALGOL and Java and to do so, I will implement a simplified version of grep. The simple version of grep still shows the power of MCP Java. Java is used to search for file names in directories and to scan the files for the target text. Using Java to find a target text in a string is all standard stuff, but I’m especially pleased with the Java ‘listFiles’ method, which is so much easier to use than setting up GETSTATUS calls from ALGOL.

I have set up a Netbeans project named ‘grep’. The class I am going to call from ALGOL is Grep. This class contains one public method, doGrep. The sourec code for the Grep class is here.

Only the method doGrep has been declared public. JifGen will generate an ALGOL call for this public method.

I am catching all exceptions in the Java program, because I have not yet quite figured out how to recover from all sorts of Java exceptions on the ALGOL end. It seems that simply re-initializing the JVM is not good enough. I suppose this can be the subject of another blog post.

To test the Grep class in my MCP environment, I included a test driver class, Greptest, in the project. GrepTest just contains a main method that calls Grep.doGrep(). I can now test and debug the Grep class while it runs on the MCP and actually reads MCP files and directories. To easily start the tests in debug mode, I made a small WFL with all the necessary parameters:

NEXT+   ....*....1....*....2....*....3....*....4....*....5....*....6....*....7..
00000100BEGIN JOB DEBUG;                                                        
00000300RUN *DIR/JRE6/BIN/JAVA                                                  
00000420    ("  -Xdebug -Xrunjdwp:transport=dt_socket,address=9000,"            
00000440    &"server=y,suspend=y"                                               
00000490    &" -jar grep.jar -r  grep /-/user/usercode/ruud/grep/src");         
00000500    CURRENTDIRECTORY = "/-/USER/USERCODE/RUUD/GREP";                    
00000700END JOB                                                                 

The arguments “-r grep /-/user/usercode/ruud/grep/src” tell grep to recursively look through the files in directory (RUUD)GREP/SRC/= ON USER searching for the text “grep”. The output of this run is below:

/-/user/usercode/ruud/grep/src/java/Grep.java:package com.mcpjava.mcp.grep;
/-/user/usercode/ruud/grep/src/java/Grep.java:                hits = grep(args[0].trim(), args[1].trim(), false);
/-/user/usercode/ruud/grep/src/java/Grep.java:                    hits = grep(args[1].trim(), args[2].trim(), true);
/-/user/usercode/ruud/grep/src/java/Grep.java:                    hits = grep(args[1].trim(), args[2].trim(), false);
/-/user/usercode/ruud/grep/src/java/Grep.java:    private static ArrayList grep(String text, String path, boolean recurse) {
/-/user/usercode/ruud/grep/src/java/Grep.java:                        ArrayList rslt = grep(text, f.getAbsolutePath(), recurse);
/-/user/usercode/ruud/grep/src/java/GrepTest.java:package com.mcpjava.mcp.grep;
/-/USER/USERCODE/RUUD/GREP/SRC/INCLUDE/GREP_INCL:11700:% com.mcpjava.mcp.grep.Grep
/-/USER/USERCODE/RUUD/GREP/SRC/INCLUDE/GREP_INCL:12200:  80"com/mcpjava/mcp/grep/Grep" 48"00"
/-/USER/USERCODE/RUUD/GREP/SRC/INCLUDE/GREP_INCL:13100:  % com/mcpjava/mcp/grep/Grep
/-/USER/USERCODE/RUUD/GREP/SRC/INCLUDE/GREP_INCL:13500:% ALGOL interfaces for class:com.mcpjava.mcp.grep.Grep
/-/USER/USERCODE/RUUD/GREP/SRC/INCLUDE/GREP_INCL:13900:% Class:       com_mcpjava_mcp_grep_Grep
/-/USER/USERCODE/RUUD/GREP/SRC/INCLUDE/GREP_INCL:18600:% Class:       com_mcpjava_mcp_grep_Grep
/-/USER/USERCODE/RUUD/GREP/SRC/GREPDRIVER:11000:procedure grep(arguments);
/-/USER/USERCODE/RUUD/GREP/SRC/GREPDRIVER:16520:    ebcdic value array jvmOptions ("-cp ./grep.jar" 48"00");

Once I’m satisfied with the Grep class, I can generate the ALGOL interface. On my development PC I have a folder named \libs\mcp in which I put all the jar files that are part of the MCP Java release. The JifGen utility is included in the mcptools.jar file. A Netbeans “Clean and Build” creates a jar file in the “dist” folder of the project. The jar file, by default, has the name of the Netbeans project, in my case “grep.jar”. I directed the Command Prompt window to the Netbeans directory for the grep project and laboriously typed in the following command to generate the ALGOL include file:

java -cp /libs/mcp/mcp.jar;/libs/mcp/mcptools.jar;/"program files"/java/jdk1.6.0_24/lib/tools.jar com.unisys.mcp.tools.JifGen -algol -classpath dist/grep.jar -d incldir -output GREP_ -verbose com.mcpjava.mcp.grep.Grep

JifGen has now generated an ALGOL include file named GREP_INCL.ALG_M in directory incldir. An ALGOL procedure has been genereated for each public method in Java class Grep. It generated a procedure DOUBLE PROCEDURE GREP__GREP_DO_GREP which reflects the method Grep.doGrep on the Java side. The default class constructor procedure, DOUBLE PROCEDURE GREP__GREP_GREP, is also generated, but since my callable method is declared static, I don’t need to construct a Grep object.
On my first attempt I had left the “”program files”/java/jdk1.6.0_24/lib/tools.jar” out. This resulted in a Java Exception:
Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/javadoc/Doclet
I reread the Unisys JifGen Javadoc and yes, it’s documented there; apart from mcp.jar and mcptools.jar you also need to include the Java SDK (Oracle) provided tools.jar (that is where the com.sun.javadoc.Doclet is located).
Initially, I also had some trouble getting my classpaths right, but you have to realize that the
-cp /libs/mcp/mcp.jar;/libs/mcp/mcptools.jar;/"program files"/java/jdk1.6.0_24/lib/tools.jar
is to get JifGen going and the
-classpath dist/grep.jar
near the end is to tell JifGen where it can find the classes it must generate ALGOL for. JifGen does not need to see the Java sources because it uses Java Reflection to get the info from the compiled classes. This is the generated GREP_INCL.ALG_M.

In the previous blog I mentioned that the generated file needed an addition to call JNI_CANCEL_PARAMETER_LIST in case of an exception. No change needed anymore, in the latest MCPJava release JifGen has already been fixed!
In the next post I will write an ALGOL program that takes the string parameter, calls the Java’s Grep.doGrep() method with the parameter and unravels the result.

Check for JNI exceptions!

Recently, I had a problem on a production system with an ALGOL to Java JNI (Java Native Interface) connection. Since this problem is still fresh in my head, I will post about it now. In a later blog I’ll go through the steps I took to set up the JNI connection in more detail.

At this customer site, a separate database is run for each brand they are marketing. Each brand has a connection to Websphere-MQ. A Java program on the JProcessor connects to MQ, the ALGOL program calls methods in the Java program. The whole Java environment (JVM) is like a library to the ALGOL program.

The ALGOL program contains the controlling loop. It calls a Java method to fetch a message from the Message Queue. The ALGOL program processeses the message and calls Java to put a reply message on the queue. It then fetches the next message etc. If are no messages in the queue, the Java method blocks until a message arrives.

But, when we setting up a new brand, and consequently another copy of the ALGOL-Java program, we could not get it to work. The new copy of the ALGOL program kept failing with a “REQUESTED MEMORY SIZE GREATER THAN 65535 WORDS” message. I usually put an ON ANYFAULT, PROGRAMDUMP(ALL) statement in my ALGOL programs, so the fault produced a program dump. The fault appeared to come from the SYSTEM/JAVAJNILIB. This librray tried to resize an array beyond the 65535 words limit as soon as the new environment tried to call a Java method. To test if we had hit a limit somewhere, we stopped one of the running environments and started the new one. Now it worked and restarting the stopped, existing environment failed.

At this point I contacted Unisys. After all, the dump showed that it was the SYSTEM/JAVAJNILIB that had all of it’s 65535 data buffers filled up with parameters from ALGOL to Java. The Unisys people jumped onto the problem right away and after some mails back and forth they came up with three problem areas: The JAVAJNILIB could do with some better failure detection/recovery, the generated include file should call JNI_CANCEL_PARAMETER_LIST if creating a parameter list to Java fails and, most importantly, my ALGOL program should check for a JVM exception after each call to a Java method. Because the ALGOL program did not check for Java exceptions, it kept firing off method calls to a JVM that was no longer functional, thus, at high speed, eating up all the data buffers in the JAVAJNILIB.
Here is the code snippet that does a method call from ALGOL to Java:

00105900    JNILIB_MCPJNICONNECTION_COMMAND(command,                            
00106000        0, offset(pc), result, rsltOffset, rsltSz);                     
00106010    if JNI_Check_Exception(JNIEnv) then                                 
00106020    begin                                                               
00106030      errorCode := JNI_GET_LAST_ERROR(JNIEnv);                          
00106052      replace axmsg by "JNI Exception: Error code = ",                  
00106054         errorCode for * digits, NULL;                                  
00106056      display(axmsg);                                                   
00106058      myself.status := value(TERMINATED);                               
00106060    end;

After the call to the Java method in line 105900, JNI_Check_Exception is called in line 106010. If that call returns a true, the JNI error code is determined. The exception is displayed and the program is terminated. In my case, I could leave it to COMS to restart the program.
A Java utility “JifGen” is provided to generate wrapper procedures that do the low level JNI calls. The current JifGen does not generate the JNI_CANCEL_PARAMETER_LIST in, but you can add that bit of code yourself. This the procedure JNILIB_MCPJNICONNECTION_COMMAND that is called from the user program:

00016600% Class:     nl_acme_mcp_mq_MCPJNIConnection                          
00016700% Method:    command                                                    
00016800% Signature: (Ljava/lang/String;)Ljava/lang/String;                     
00016900% ID: 2                                                                 
00017100PROCEDURE JNILIB_MCPJNICONNECTION_COMMAND                               
00017300  VALUE         ARG0_OFFSET, ARG0_SIZE, RESULT_OFFSET ;                 
00017400  EBCDIC ARRAY  ARG0[0];                % Input                         
00017500  INTEGER       ARG0_OFFSET;                                            
00017600  INTEGER       ARG0_SIZE;              % Input                         
00017700  EBCDIC ARRAY  RESULT[0];              % Output                        
00017800  INTEGER       RESULT_OFFSET;                                          
00017900  INTEGER       RESULT_SIZE;            % Output                        
00018100  LABEL xit;                                                            
00018200  OWN DOUBLE methodID;                                                  
00018300  REAL hParams;                                                         
00018500  IF JNI_IsNull(methodID) THEN                                          
00018600  BEGIN                                                                 
00018700    methodID := JNI_GET_STATIC_METHOD_ID_E(                             
00018800            JNIEnv,                                                     
00018900            CLASS_MCPJNIConnection_REF,                                 
00019000            JNILIB_StringPool, 50,       % command                      
00019100            JNILIB_StringPool, 58 );     %                              
00019200    %  (Ljava/lang/String;)Ljava/lang/String;                           
00019300    IF JNI_Check_Exception( JNIenv ) THEN GO TO xit;                    
00019400  END IsNull;                                                           
00019600  hParams := JNI_CREATE_PARAMETER_LIST( JNIEnv, methodID,               
00019700   JNILIB_StringPool, 58, 600 );           %                            
00019800  %  (Ljava/lang/String;)Ljava/lang/String;                             
00019900  IF hParams EQL 0 THEN GO TO xit;                                      
00020100   ARG0_SIZE);                                                          
00020200  IF JNI_Check_Exception( JNIenv ) THEN                                 
00020220  begin                                                                 
00020240   JNI_CANCEL_PARAMETER_LIST(JNIEnv, hParams);                          
00020260   GO TO xit;                                                           
00020280  end;                                                                  
00020300  JNI_CALL_STATIC_STRING_EBCDIC_METHOD_P(                               
00020400    JNIEnv, CLASS_MCPJNIConnection_REF, methodID, hParams, RESULT,      
00020500     RESULT_OFFSET, RESULT_SIZE);                                       
00020700  xit:                                                                  

The code generated by GifGen just does the GO TO xit. I added the JNI_CANCEL_PARAMETER_LIST in line 20240.

Allthough a UCF was submitted for the JAVAJNILIB and the JifGen, these two points are not really an issue, as long as you make sure you check for exceptions after each call to the JVM.


  • The Java Native Interface Programmer’s Guide and Specification, Sheng Liang, 1999
  • The Unisys Javadoc for JNI on the MCP system in *JRE/jni