Welcome to IFCX : Internet Foundation Classes eXtreme!

Jim White's Blog

November 23, 2013 12:21 AM
Bundling Local JARs for your Groovy scripts the Grape Way
A FAQ which often does not receive a satisfactory answer is how to add local JAR files to a Groovy script without having to fiddle with the dreaded CLASSPATH environment variable. Folks see that things are quite nice when using Grape with a Maven repository and wish they could do the same with local JAR files. When creating IFCX Wings I implemented Groovy For OpenOffice (pre-Grape) which also used Apache Ivy for dynamically extending the classpath. One difference though is that I directly supported including JAR files which are not in a repository but are simply referenced by URL, whether local (file) or remote (http/https). It turns out though that Grape can be used with local JAR files in fairly nice fashion by keeping them in a Maven-format repository in the local filesystem. Here's how to do that in pretty friendly manner.

Use Maven to put the JAR files in a Local Repository

Maven will install files in a local repository for you without fussing around with a POM file. We'll use the mvn deploy-file command and a local file URL that is relative to where our script is located which will be file:repo in this example. Notice that the file URL has no leading slash and so is relative to the working directory. Naturally you'll need to use an appropriate groupId and artifactId (please be a good netizen and use a groupId that the reverse of a domain name that is assigned to you or your organization). For this example I'm demonstrating a little script that uses Apache Lucene and I'll use my groupId so that Grape will only use my copy. One flaw in this scheme at the moment is that transitive dependencies are not managed although it should be possible to work that out and I'll look into it later. The upshot of that is we have to list each JAR file individually, which shouldn't really be an issue since we're already just wrangling JAR files by hand anyhow.

Layout of my Working Directory

$ find *
lib
lib/lucene-analyzers-common-4.5.1.jar
lib/lucene-core-4.5.1.jar
whats_new.groovy

Top of my Groovy Script

#!/usr/bin/env groovy

@GrabResolver(name='script', root='file:repo')
@Grab(group='org.ifcx.lucene', module='lucene-core', version='4.5.1')
@Grab(group='org.ifcx.lucene', module='lucene-analyzers-common', version='4.5.1')

import org.apache.lucene.analysis.standard.StandardAnalyzer
import org.apache.lucene.document.Document
import org.apache.lucene.document.Field
import org.apache.lucene.index.DirectoryReader
...
The full script (using the default Maven Central repo) can be see in this gist https://gist.github.com/jimwhite/7609525.

Create the Local Maven Repository

We use the command mvn deploy:deploy-file -Durl=file:repo -Dfile=/path/to/jar -DgroupId=name -DartifactId=name -Dversion=ver to copy the JAR file to a new Maven repository in the local directory repo.
$ mvn deploy:deploy-file -Durl=file:repo -Dfile=lib/lucene-core-4.5.1.jar -DgroupId=org.ifcx.lucene -DartifactId=lucene-core -Dpackaging=jar -Dversion=4.5.1
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-deploy-plugin:2.7:deploy-file (default-cli) @ standalone-pom ---
Uploading: file:repo/org/ifcx/lucene/lucene-core/4.5.1/lucene-core-4.5.1.jar
Uploaded: file:repo/org/ifcx/lucene/lucene-core/4.5.1/lucene-core-4.5.1.jar (2244 KB at 16025.9 KB/sec)
Uploading: file:repo/org/ifcx/lucene/lucene-core/4.5.1/lucene-core-4.5.1.pom
Uploaded: file:repo/org/ifcx/lucene/lucene-core/4.5.1/lucene-core-4.5.1.pom (399 B at 389.6 KB/sec)
Downloading: file:repo/org/ifcx/lucene/lucene-core/maven-metadata.xml
Uploading: file:repo/org/ifcx/lucene/lucene-core/maven-metadata.xml
Uploaded: file:repo/org/ifcx/lucene/lucene-core/maven-metadata.xml (306 B at 298.8 KB/sec)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.820s
[INFO] Finished at: Sat Nov 23 00:42:08 PST 2013
[INFO] Final Memory: 3M/81M
[INFO] ------------------------------------------------------------------------

$ mvn deploy:deploy-file -Durl=file:repo -Dfile=lib/lucene-analyzers-common-4.5.1.jar -DgroupId=org.ifcx.lucene -DartifactId=lucene-analyzers-common -Dpackaging=jar -Dversion=4.5.1
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-deploy-plugin:2.7:deploy-file (default-cli) @ standalone-pom ---
Uploading: file:repo/org/ifcx/lucene/lucene-analyzers-common/4.5.1/lucene-analyzers-common-4.5.1.jar
Uploaded: file:repo/org/ifcx/lucene/lucene-analyzers-common/4.5.1/lucene-analyzers-common-4.5.1.jar (1549 KB at 12196.3 KB/sec)
Uploading: file:repo/org/ifcx/lucene/lucene-analyzers-common/4.5.1/lucene-analyzers-common-4.5.1.pom
Uploaded: file:repo/org/ifcx/lucene/lucene-analyzers-common/4.5.1/lucene-analyzers-common-4.5.1.pom (411 B at 401.4 KB/sec)
Downloading: file:repo/org/ifcx/lucene/lucene-analyzers-common/maven-metadata.xml
Uploading: file:repo/org/ifcx/lucene/lucene-analyzers-common/maven-metadata.xml
Uploaded: file:repo/org/ifcx/lucene/lucene-analyzers-common/maven-metadata.xml (318 B at 155.3 KB/sec)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.745s
[INFO] Finished at: Sat Nov 23 00:42:28 PST 2013
[INFO] Final Memory: 3M/81M
[INFO] ------------------------------------------------------------------------

A New Repository is Born

$ find *
lib
lib/lucene-analyzers-common-4.5.1.jar
lib/lucene-core-4.5.1.jar
repo
repo/org
repo/org/ifcx
repo/org/ifcx/lucene
repo/org/ifcx/lucene/lucene-analyzers-common
repo/org/ifcx/lucene/lucene-analyzers-common/4.5.1
repo/org/ifcx/lucene/lucene-analyzers-common/4.5.1/lucene-analyzers-common-4.5.1.jar
repo/org/ifcx/lucene/lucene-analyzers-common/4.5.1/lucene-analyzers-common-4.5.1.jar.md5
repo/org/ifcx/lucene/lucene-analyzers-common/4.5.1/lucene-analyzers-common-4.5.1.jar.sha1
repo/org/ifcx/lucene/lucene-analyzers-common/4.5.1/lucene-analyzers-common-4.5.1.pom
repo/org/ifcx/lucene/lucene-analyzers-common/4.5.1/lucene-analyzers-common-4.5.1.pom.md5
repo/org/ifcx/lucene/lucene-analyzers-common/4.5.1/lucene-analyzers-common-4.5.1.pom.sha1
repo/org/ifcx/lucene/lucene-analyzers-common/maven-metadata.xml
repo/org/ifcx/lucene/lucene-analyzers-common/maven-metadata.xml.md5
repo/org/ifcx/lucene/lucene-analyzers-common/maven-metadata.xml.sha1
repo/org/ifcx/lucene/lucene-core
repo/org/ifcx/lucene/lucene-core/4.5.1
repo/org/ifcx/lucene/lucene-core/4.5.1/lucene-core-4.5.1.jar
repo/org/ifcx/lucene/lucene-core/4.5.1/lucene-core-4.5.1.jar.md5
repo/org/ifcx/lucene/lucene-core/4.5.1/lucene-core-4.5.1.jar.sha1
repo/org/ifcx/lucene/lucene-core/4.5.1/lucene-core-4.5.1.pom
repo/org/ifcx/lucene/lucene-core/4.5.1/lucene-core-4.5.1.pom.md5
repo/org/ifcx/lucene/lucene-core/4.5.1/lucene-core-4.5.1.pom.sha1
repo/org/ifcx/lucene/lucene-core/maven-metadata.xml
repo/org/ifcx/lucene/lucene-core/maven-metadata.xml.md5
repo/org/ifcx/lucene/lucene-core/maven-metadata.xml.sha1
whats_new.groovy

We're Ready to Rock

$ ./whats_new.groovy index_dir ~/Downloads/ucsd_2.0_content/editor_source/command.text 
SEGMENT PROCEDURE EDITCORE;

(* Core procedures.  Execute these commands until either a set environment
   comes along or a quit command. *)
...
There are 263 new lines of 263 total.

Note that the script was run with the working directory being the one containing the Groovy script. That is essential because the root URL we used in the @GrabResolver annotation is relative to the working directory. That isn't quite as inconvenient as it might seem since the URL will only be used if the artifacts are not in the Groovy Grapes cache (~/.groovy/grapes), which typically will only be the first time the script is run by a user. The caching though does mean that if the JAR changes then the version number needs to change and the script must again be run from the working directory.

Update

As a result of addressing this issue I've created an improvement for Grape that makes it work really nicely with these Maven repositories collocated with the Groovy scripts that use them. Follow GROOVY-6451 to see when this feature is ready to use. I'll also post here as well of course.

Go Forth in Groovy Style

That's all there is to it. The working directory can be delivered using your favorite method such as tar and ssh or what-have-you. The lib directory doesn't need to be included of course and is only shown here for completeness - the path given to mvn deploy:deploy-file could easily have been elsewhere (including your ~/.m2 or ~/.groovy/grapes cache).
By Jim White  Permalink -Comments
December 22, 2008 1:04 AM
Bottom of the Ninth, or, Just how "Open" is Java anyways?
A recent post on the OCJUG list asked to what degree is Java "open" and when will everyone agree that it is "open" to their satisfaction.

The fact that "Java" is a trademark (the significance of which was made obvious when Sun changed their stock symbol from SUNW to JAVA) means that what most folks think Java is will never be as open or free as some folks want.

Furthermore, Sun has a dual license for their Java distribution which includes OpenJDK. So if you want to contribute to the most popular JDK distribution, you have to agree that Sun can also use your efforts under terms that are not necessarily free or open (although they do promise to always make your contribution available under a FSF and/or OSI approved license).

http://openjdk.java.net/contribute/

So OpenJDK has two strikes against it.

Here comes the next pitch, aaannd OpenJDK is GPL licensed, a home run!

I was amazed and pleased that Sun went all the way and used GPL for Java. After all, I'd been saying that Java would eventually be OSS, even after the ISO talk was replaced by the JCP.

That means this pillar of the Great Java Renaissance of 2006 is strong as it possibly can be. One of those strengths is that anyone who isn't satisfied with the degree of openness can simply fork with total abandon and with the best license for software freedom. That is a reality and one of the first projects spawned by OpenJDK is IcedTea, which is a free and open implementation of Java.

As for folks saying Java being OSS doesn't matter. They are quite mistaken. Already we've seen significant developments such as SoyLatte which counters Apple's weak support of Java, and research work on Java being truly free and open in the Da Vinci Machine Project. This change in research work is important because in the past such work, if done with Sun's JDK, used a license that meant the resulting code rarely ever left the university.

That the OpenJDK is an effective OSS project is clear because all of these efforts are now part of the OpenJDK project itself, rather than forking or otherwise choosing to stand alone or with another group.

That is only the beginning and, by being real Open Source Software, we can rest assured that Java will grow in strength over the next decade just as it did the first. Of course we can also be assured that there are plenty of folks that will disagree with just about every aspect of all this jazz.

By Jim White  Permalink -Comments
December 12, 2008 10:58 AM
Grid computing with Groovy
Alex Tkachman posted a question about Groovy style for checkpointed calculations.

Here's my first try:

// Proof of concept for Groovy checkpointed calculation with closures.
// @author Jim White <jim@pagemsiths.com>
// http://www.ifcx.org/

Session session = new Session()

// def initState = new Expando(data:'datalocation')
def initState = [data:'datalocation']

def endState = eachWithCheckpoints(session, 'state1', initState, [
      { stepCount = longInitialConditionsCalculation(data) }
    , { accumulator = 0
        iterateWithCheckpoints(session, 'state2', it.state
           , 0..stepCount
           , { accumulator += 2 * it.step })
      }
    , { result = "We did $stepCount iterations and the answer is $accumulator!" }
])

def longInitialConditionsCalculation(d) { 10 }

println endState.result

class Session {
    Object loadCheckpoint(String k) { null }
    void saveCheckpoint(String k, Serializable s) { }
}

def eachWithCheckpoints(Session session, String key, def initialState, List<Closure> closures) {
    def control = session.loadCheckpoint(key)

    if (control.is(null)) {
        control = [step:0, state:initialState]
    }

    while (control.step < closures.size()) {
        Closure clos = closures[control.step]

        clos.delegate = control.state
        clos.resolveStrategy = Closure.DELEGATE_FIRST

        clos.call(control)

        control.step += 1

        session.saveCheckpoint(key, control)
    }

    control.state
}

def iterateWithCheckpoints(Session session, String key, def initialState, Range range, Closure clos) {
    def control = session.loadCheckpoint(key)

    if (control.is(null)) {
        // Leave reversed range case as an exercise for the reader...
        assert !range.isReverse()

        control = [step:range.from, state:initialState]
    }

    clos.delegate = control.state
    clos.resolveStrategy = Closure.DELEGATE_FIRST

    while (control.step <= range.to) {
        clos.call(control)

        control.step += 1

        session.saveCheckpoint(key, control)
    }

    control.state
}
==>
We did 10 iterations and the answer is 110!

The idea is either you're doing a sequence of different steps or iterating the same step some number of times. Notice that the state can be any serializable thing. Map or Expando is handy, but some bean or other class would be fine.

Obviously there are further refinements possible such as reversed and stepped ranges, and also a more functional style is possible. Either a builder or controller class would streamline things a bit by hiding some of the details of the eachWithCheckpoints/iterateWithCheckpoints calls.

If Groovy had serializable iterators for the control step, that would make iterateWithCheckpoints nicer and let it change from taking just a simple integer range to a list.

Naturally where there this is headed is cloud-powered click-and-it-goes Wings via your web browser using OOHTML.

By Jim White  Permalink -Comments
December 9, 2008 5:25 PM
TREE-META
I've been interested in TREE-META since I discovered it while at UC Irvine. We used a version for the UCSD p-System on the Teraks (desktop LSI-11/23). It had a pleasant and compact syntax that was more convenient than the LISP that took it's place for me.

For several years I've been on the lookout for information on TREE-META but haven't been able to find out a great deal. After about the third time listening to The Mother of All Demos I heard them mention that TREE-META was the language used to implement the "special purpose languages", which we call "domain-specific" today.

I've been unable to locate a copy of any version the program or the tech report describing the language, although I've seen hints than some folks have used it fairly recently.

In commemoration of the 40th anniversary of the MoAD (also at Wired), I've created a Wikipedia stub about TREE-META as well as one here for non-WP suitable material. The WP article is very rough, but I'm hoping that others with information will bring it forward and collect it there.

By Jim White  Permalink -Comments

Blog Archive

News

September 15, 2008 4:20 PM
OCaml, Scheme, and Adenine support for IFCX Wings, an in-progress console, and OOo My!
IFCX Wings v0.7 is now available from the usual place (wow, spiffy new site design for SF.net!). It features an in-progress (during script execution) console that pops up during long running scripts. Very handy to see that something is happening while Ivy downloads your JAR dependencies. Default support for OCaml, Scheme, and Adenine has also been added.

I've tested the Wings.odt with OOo 3.0beta2 on Windows XP and Mac OS X Leopard and it seems to work fine except that on Mac there is no support for AWT (and consequently no Swing). It also works with OOo 2.4 on Windows and NeoOffice 2.2 on Mac. Earlier versions are mostly OK too, but don't go toooo far (like 2.0) or GroovyForOpenOffice won't work (and you need it for Wings).

Note that you must install Groovy For OpenOffice for the WingsEval macro to work. Otherwise you can just open the Wings document and read it, but the code won't be executable.

The keyboard shortcut for the Wings macro seems to work with all those versions and platforms too. The Wings.odt file is signed so that you can have some reasonable expectation that we're using the same bits. Of course if yours is not signed by my CA Cert showing JAMES PAUL WHITE and my email address then there may be a problem.

OCaml (Objective Caml) is a popular version of ML sporting OO features. OCaml-Java is the spiffy implementation for the JVM by Xavier Clerc used in Wings.

Adenine is the language of the MIT Haystack Semantic Desktop and is essentially a LISP for graphs. It features a RDF data model, Pythonic syntax, and is implemented for the JVM. I have extricated it's implementation from Eclipse and packaged it with a JSR-223 engine adapter so that Wings can support it by default. The Adenine Tutorial converted to IFCX Wings is available from SVN.

The Scheme support is using the SISC implementation (even though I'm long time Kawahead) because it has a JSR-223 engine already and it is also better for pedagogic purposes since it has full continuations.

There is also exciting news regarding IFCX.org and the OpenOffice.org Community Innovation Program, but I'll hold off on the details of that until November.

By Jim White  Permalink -Comments
May 12, 2008 11:00 AM
IFCX Wings & G4OO updated, plus we've got Scala!
The v0.6 release of IFCX Groovy For OpenOffice and Wings (version numbers bumped to align for the moment), demonstrated at CommunityOne and JavaOne last week, are available for download in the usual place.

A big THANK YOU to Brendan Humphreys and Peter Moore of Atlassian for the invitation and support to present Wings in a lightning talk at CommunityOne. I did have a nasty demo devil bite that prevented a live demonstration, so next time I'll have a Time Machine backup with me...

This release of Wings should be useful on at least an experimental level for the adventurous. The examples include plenty of Groovy of course, but also Ruby/JRuby, Python/Jython, and Haskell/Jaskell.

Apache Ivy integration enables JAR dependencies to be loaded dynamically and languages are pluggable via the JSR 223: Scripting for the Java Platform. JDK 1.5 is all that is required, although JDK 1.6 or 1.7 are fine of course.

G4OO v0.6 features Groovy 1.5.6 and Apache Ivy 2.0-SNAPSHOT bundled with the Ivy RoundUp Builder Resolver.

Using that pluggable language scheme, and thanks to a sprint at the Scala lift off last Saturday with the invaluable assistance of Lex Spoon and Toomas Romer, WE HAVE SCALA! You can download it from the usual place.

By Jim White  Permalink -Comments

News Archive

Recent Changes

2013-11-24
JimWhite_blogentry_231113_1 13:13:08
2013-11-23
Main 00:04:12