Foil - a Foreign Object Interface for Lisp

Copyright (c) Rich Hickey and Eric Thorsen. All rights reserved.

The use and distribution terms for this software are covered by the Common Public License 1.0, which can be found in the file CPL.TXT at the root of this distribution. By using this software in any fashion, you are agreeing to be bound by the terms of this license. You must not remove this notice, or any other, from this software.

Contents

Description and Rationale

Foil consists of a protocol and a set of libraries that facilitate access to popular object runtimes, such as the JVM and the CLI/CLR, and their libraries, from Lisp. A protocol is defined which abstracts out the common features provided by Java-like environments - object construction, method, field, and property access, object lifetime management etc. The protocol defines a set of features as well as an s-expression based stream format for communication. Runtime server applications are provided that utilize Java and C# libraries to implement the object runtime side of the protocol for Java and the CLI. Source for the applications is provided so that custom hosts can be built. A library for Common Lisp is provided that implements the consumer side of the protocol, and offers seamless access to the foreign objects in a Lisp-like manner. The design of Foil owes much to jfli, an in-process solution to the same problem for Java, and it remains extremely similar in its Lisp interface. Several factors motivated the significant difference in the Foil design - its use of an out-of-process instance of the foreign runtime: The major tradeoff in stream-based access to out-of-proc runtimes is a significant drop in per-call performance. However, even with jfli, which was very fast, the overhead of reflection per call could be high in certain scenarios, since the APIs of these platforms tend to be very 'chatty'. Foil includes a marshalling system that allows for efficient transfer of large and composite objects with minimal call overhead, in a manner that doesn't pollute the Lisp code on the consumer side.

Foil provides all the facilities of jfli and more -

Features of jfli that are retained/enhanced:
Some of the additions are:

Download and Communication

Foil is hosted on SourceForge SourceForge.net Logo

We are going to try using SourceForge facilities for all communication regarding Foil, so please use the project tracker and forums there.

Quick Start

Build and start the Java or CLI runtime server of your choice.
(compile-file "/dev/foil/foil.lisp")
(load "/dev/foil/foil")
(use-package :foil)
;this is specific to LispWorks' sockets support
(require "comm")
;create a foreign VM
(setf *fvm* (make-instance 'foreign-vm
                           :stream
                           (comm:open-tcp-stream "localhost" 13579)))

;;;;;;;;; if Java ;;;;;;;;;;;
;create a wrapper for dialog class
(def-foil-class "javax.swing.JOptionPane")
;use the wrapper
(use-package "javax.swing")
;show it
(joptionpane.showmessagedialog nil "Hello World from Lisp")

;;;;;;;;; if CLI ;;;;;;;;;;;;
;create a wrapper for dialog class
(def-foil-class "System.Windows.Forms.MessageBox")
;use the wrapper
(use-package "System.Windows.Forms")
;show it
(messagebox.show "Hello World from Lisp")

Typically you wouldn't define single class wrappers by hand, but would instead pre-generate wrappers for entire packages using get-library-classnames and dump-wrapper-defs-to-file. See foil-java.lisp and foil-cli.lisp for examples.

Lisp API Reference

Other than 2 wrapper-generator helpers, all of the code for the Lisp side of foil resides in foil.lisp, and is runtime server independent. Foil is designed to be very portable, and is 99% pure Common Lisp. A complete port requires some facility for weak hash tables and object finalization.

Foreign VMs

Foil is built upon the notion of interactions with one or more foreign VMs, instances of the JVM or CLR, running the Foil libraries, in another process on the same or another machine. The connection to a specific VM is via one or more bidirectional streams. Note that the instantiation of the foreign VM and the establishment of the streams is outside the scope of this Lisp API. It is presumed you might utilize one of the supplied runtime servers, creating stream connections via sockets or pipes with the API provided by your Lisp implementation. Many scenarios are possible, including embedding the Foil support libraries into your existing Java or C# application, multiple streams to different threads in the same VM, etc.

A foreign VM is represented by an instance of the foreign-vm class. Each instance has a primary default stream over which communication will occur. The special variable *fvm* represents the default VM to which any unqualified Foil calls will be directed, and can be bound in a specific context, thus allowing for multiple VMs. Note - instance property/method calls will always be routed to the VM hosting that instance.

Foreign VMs maintain their own foreign reference pools, type caches etc, and objects from one VM cannot be passed to another, even if they are both Java or CLI. However, in multi-thread, multi-stream scenarios, references are valid across threads in the same VM, and the runtime server implementations are thread safe.

A simple startup scenario would look like this:

- First, outside of Lisp, start the Java or CLI Foil runtime server supplied with Foil, running on port 13579, then:

   (load "/dev/foil/foil")
   (use-package :foil)
   (require "comm") ;LispWorks-specific socket library
   (setf *fvm* (make-instance 'foreign-vm
                           :stream
                           (comm:open-tcp-stream "localhost" 13579)))
    ;use Foil  

Foreign References

Foil programs invariably create instances of objects in the foreign VM. Those objects are tracked by Lisp in instances of the fref class. The Foil API and the runtime server cooperate to ensure that objects referenced by Lisp are kept alive in the foreign runtime, and that when no longer referenced by Lisp, they become available for collection in the foreign VM. frefs maintain their source VM, an ID and revision counter for this purpose. In addition, frefs can cache hash codes, types, and values that have been marshalled. Only a single fref will be created for each remote object, thus any 2 frefs that reference the same object are themselves eq.

Wrapper Generation

Object Creation

In addition to the generated ctor wrappers (classname.new described above), the following, built upon the same, add some additional capabilites and ease of use:

Object Services

These functions provide access to basic facilities provided by all runtimes (usually through syntax or the Object class), but should be used instead, as they are portable and can be more efficient, caching and resolving some things locally on the Lisp side.

Vectors

Miscellaneous

Arguments and Boxing

In most cases argument matching and conversion should be transparent. Lisp strings can be passed where Strings are required, Lisp numbers where int float etc are required. t and nil can be passed where booleans are required etc. nil can be passed for null. Some Foil APIs (e.g. make-new-vector) require type arguments, and unless specified otherwise, any of the following are acceptable:

Note that this only applies to Foil APIs, if a foreign runtime API takes a Class/Type argument, you must supply a fref referring to an actual Class/Type instance.

Occasionally it may be necessary to provide a hint as to the intended type of a numeric argument in order to force resolution to a particular overload.

  • Function (box type val)

    Type must be a primitive designator keyword - :boolean|:byte|:char|:double|:float|:int|:long|:short
    Produces an object that when passed to a Foil-generated function will be interpreted by the runtime as that type. Note that silent truncation may occur.

  • It is also possible to create vectors in-line as arguments, which will avoid multiple round-trips vs. calling make-new-vector in place. Note this is only good for ephemeral vectors, as there is no way to retain a reference to the newly-created vector.

  • Function (box-vector type &rest vals)

    Produces an object that when passed to a Foil-generated function will be an array of type with the supplied values.

  • Example:
    CL-USER 102 > (vref (box-vector string. "a" "b" "c" "d") 2) ;only one round-trip
    "c"

    Class/Type Helpers

    Proxies and Callbacks

    Proxies allow the creation of foreign objects that implement one or more interfaces (or in the case of CLI, one or more interfaces or a single delegate) in Lisp, and thus callbacks from the foreign VM to Lisp. Foil supports Lisp calling foreign runtime calling Lisp... to an arbitrary (probably stack-limited) depth.

    Marshalling

    Foil supports an extensible marshalling system which allows the values of reference/composite types to be returned in addition to, or even instead of, the references themselves. Used appropriately, this can substantially reduce the number of round trips between processes and avoid significant 'chatter' overhead.

    Marshalling comes into play whenever a reference type is returned from a Foil function. With certain settings, it is possible to return any or all of a reference, its hash code, its type, and its value (and the same for any of its value's reference members), to a specific depth. The nature and depth of the marshalling is governed by two special variables on the Lisp side - *marshalling-flags* and *marshalling-depth*.

    The format of marshalled values is determined by the runtime servers, and both the Java and CLI servers provided with Foil have facilities for adding new marshallers for specific types. The way an object's value is marshalled is a function of its type.

    Class or Types will always marshall the string representing the packageQualifiedTypeName, ignoring *marshalling-depth*

    By default, the following marshalling will be performed when requested, i.e. *marshalling-depth* > 0

    Marshalling example:
    CL-USER 79 > (setf string-class (get-type-for-name "java.lang.String"))
    #}1
    
    CL-USER 88 > (class.getpackage string-class)
    #}12
    
    CL-USER 90 > (pprint (with-marshalling (1 +MARSHALL-NO-IDS+)
                           (class.getpackage string-class)))
    
    ((:IMPLEMENTATIONTITLE . "Java Runtime Environment")
     (:IMPLEMENTATIONVENDOR . "Sun Microsystems, Inc.")
     (:IMPLEMENTATIONVERSION . "1.4.2_05")
     (:NAME . "java.lang")
     (:SEALED)
     (:SPECIFICATIONTITLE . "Java Platform API Specification")
     (:SPECIFICATIONVENDOR . "Sun Microsystems, Inc.")
     (:SPECIFICATIONVERSION . "1.4")) 

    Runtime Servers

    Foil includes 2 complete implementations of the runtime server portion of the protocol, one for Java/JVM, the other for C#/CLI. The implementations are 100% Java/Managed, and use only standard libraries. Both will run the protocol over one or more TCP sockets specified on the command line, or, if none specified, via standard IO.

    Project files are included for Eclipse and Visual Studio. All of the Java code is in com/richhickey/foil and the stand-alone server is in RuntimeServer.Main. The CLI implementation is in 2 projects, one for the Foil library itself - FoilCLI, and the other for the stand-alone server - FoilCLISvr.

    After building, you can invoke the Java server as follows:

    java -cp . com.richhickey.foil.RuntimeServer 13579

    Make sure the classpath includes the libraries and .jars you will want to use via Foil.

    After building, you can invoke the CLI server as follows:

    foilclisvr 13479

    Protocol

    The foil protocol describes the on-stream interface between a Lisp instance and a runtime instance, and should not be confused with the foil library which provides the interface to the protocol for Common Lisp. A user of Foil will not need to know the protocol, but if you intend to add support for another runtime environment (Python anyone?) or host language (Scheme anyone?), hopefully this section will help. Note that the protocol docs are not formal, and mostly consists of notes to myself and Eric. This will be improved when I get time. For the moment, should there be any omissions or inaccuracies here, the Lisp and Java implementations should be considered canonic.

    Connection Services

    Foil is a stream-based protocol. However, no protocol is provided for the establishment of the streams - that is an implementation detail of the runtime and Lisp libraries. It is suggested that any foil runtime implementation provide at least a stand-alone executable server that implements the protocol over its standard IO ports, as well as being able to run over a TCP/IP socket. Many other scenarios are possible, including multi-socket servers, pre-existing Lisp and runtime instances discovering each other etc. The remainder of the protocol description presumes a bi-directional stream has been established. Sendable messages:

    Returnable messages:

    Invocation Services

    Obtaining callable references (crefs)

    (:cref member-type tref|"packageQualifiedTypeName" "memberName")

    Where member-type is an integer representing one of:

    Note that both Java and the CLI support overloading, so a single member name might map to multiple overloads. The resolution of the overloading must occur in the runtime server at the time of invocation, i.e. any of the overloads may be called through the same cref.

    Returns -> A reference to a callable thing is returned in the standard return format (see below).

    (:ret #{:ref ...})

    Creating new object instances

    (:new tref marshall-flags marshall-value-depth-limit (args ...) property-inits ...)

    where property-inits is a set of :keyword-style-name value pairs

    Calling a callable

    (:call cref marshall-flags marshall-value-depth-limit target args ...)

    Example:

    (:call #}101 1 0 2 #}17 "fred")

    Where cref is an cref that has been obtained via :cref, or, only in the case of calls to Lisp, a symbol that names a function.
    marshalling-flags is an integer representing a bitwise-or'ing of:

    a marshall-value-depth-limit of 0 means no reference values are marshalled, a setting of 1 means that reference values will be marshalled for the return value (if it is a reference), but not any nested references. When > 1 nested reference types will marshall to that depth of nesting.
    If marshalling-flags is 0, no references will be returned (only values) and if depth is also 0 then nil will be returned.
    target is the object upon which to invoke the method/field/property - pass nil if static
    args are zero or more args as per below.

    Return Format


    one of:

    Argument and Return Values

    Primitives and Value Types

    Boxed Primitives

    Occasionally it may be necessary to provide a hint as to the intended type of a numeric argument in order to force resolution to a particular overload.

    #{:box typename value}

    Where typename is one of :byte :int :long :short :float :double

    N.B. silent truncation may occur

    Return values should never be boxed

    vector literals

    A vector can be specified in-line as an argument #{:vector "packageQualifiedTypeName"|tref|:int(etc) value ...}

    References

    Reference types are returned with the following tagged syntax:
    #{:ref id rev :type a-ref :hash an-int :val marshalled-value} Note that it is possible to return marshalled values of reference objects without maintaining the reference object on the hosting side (by setting the marshall-id flag to 0 and having marshall-value-depth-limit > 0

    A reference (obtained previously) is passed back to its host like this:

    #}123

    The host will look up the object with that id and pass it along to the call.

    Exception Reporting

    All exceptions are reported via a return of the form:

    (:err "error description" "stack trace")

    if an exception occurred while processing the request. Unless the exception originated in the reflection API, it is preferred that the stack trace be of the inner (reflection-invoked) call.

    Object support services

    Object references

    Object lifetime management

    (:free refid refrev ...) -> nil

    a list of id/rev pairs is passed. Allows one or more refs to be GC-ed on the hosting side. It is an error to refer to these refids again.

    Object marshalling

    It is anticipated that runtime servers will provide for user-installable marshallers, associated with types, that will render the value of an object of that type on a stream in a form readable by Lisp. By default at least the following marshallers should be provided:

    In addition to marshalling returns during calls, the value of an object reference can be explicitly marshalled:

    (:marshall ref marshall-flags marshall-value-depth-limit) -> Lisp-readable-value

    Hash values
    (:hash ref) -> int

    Object equality
    (:equals ref ref) -> t|nil, per Object.Equals

    ToString
    (:str ref) -> "string value"

    Reflection Services

    Note, when trefs are returned by these reflection calls, the :val field of the reference is always (default) marshalled, i.e. set to the packageQualifiedTypeName as a string.

    Obtaining a reference to a Type/Class object

    (:tref "packageQualifiedTypeName") -> tref

    Object type

    (:type-of ref) -> tref

    (:is-a ref tref) -> t|nil

    (:bases tref|"packageQualifiedTypeName")

    -> (:ret ("packageQualifiedTypeName" ...))    ;most-derived to least-derived
    

    (:members tref|"packageQualifiedTypeName")

    -> (:ret ( (:ctors doc-string ...)
            (:methods ((:name string)
                       (:static bool)
                       (:doc doc-string)) ...)
            (:fields ((:name string)
                      (:static bool)
                      (:doc doc-string)) ...)
            (:properties ((:name string)
                          (:static bool)
                          (:get-doc doc-string)
                          (:set-doc doc-string)) ...)))

    Proxies

    (:proxy marshall-flags marshall-value-depth-limit interface-trefs ...)
    -> (:ret proxy-ref)

    Creates a proxy object that implements the given interface(s). When any of the object's methods are called, sends a :proxy-call message of the form:

    (:proxy-call method-symbol proxy-ref args ...)

    where the proxy-ref is the same one originally returned from the :proxy message, and the args are marshalled with the flags and depth requested in the :proxy message. method-symbol has the form

    |package.name|::classname.methodname

    note this means that the Lisp names are not independent, hmmm...

    Vectors

    Creating an vector:

    Summary

    I'd like to thank my good friend Eric Thorsen for his hard work on the CLI/C# port.

    I hope you find Foil useful. It is my sincere intent that it enhance the utility and interoperability of Common Lisp. I welcome comments and code contributions.

    Rich Hickey, February, 2005