la.scala._

capWord Competition

Fri, 11 Sep 2009 19:22:30 -0400

This competition was run on #scala. The task is to write a short def capWord which checks whether a given string is a Capital word. Some entries follow.

@paulp I:

def capWord(s: String) = (s.tail map (_.isLowerCase))
.reduceLeft(_ && _) && s.head.isUpperCase

— and he notes: of course that fails on “P”. but maybe that’s not a word either. more of a letter.

@paulp II:

def capWord(s: String) = !s.isEmpty && 
(s.head isUpperCase) && (s.tail forall (_.isLowerCase))

— and he notes: I won’t deal with null out of spite.

@RShulz:

def capWord(word: String): Boolean = 
{ word.count(_.isUpperCase) == 1 && word(0).isUpperCase }

— and he notes: (Count exists in my mind.)

@dcsobral I, the winning entry (no head or tail, and I like it):

def capWord(w: String) = w.toList match {
  case h :: t if ('A' to 'Z' contains h) && 
    (t forall ('a' to 'z' contains _)) => true
  case _ => false
}

@dcsobral II, the fold entry:

def capWord(w: String) = w.tail.foldLeft(w.head isUpperCase)
( (flag, l) => flag && l.isLowerCase )

@dcsobral III, fanciful entry:

def capWord(w: String) = (
    (
    w
    map (_ isUpperCase)
    map (if (_) 1 else 0)
    zipWithIndex
    )
  map Function.tupled((a,b) => a * (b+1))
  reduceLeft (_+_)
  ) == 1

@SethTisue, mindboggling entry:

def capWord(s: String) = 
(Math.log(s.map(c => 
    if(c.isUpperCase) 1 else 0).mkString.toDouble) 
/ Math.log(10)).toString.endsWith(".0")

@paulp III, the mondblowingboggling entry:

def capWord(s: String) = s.head.isUpperCase && 
(s.tail map (c => 
    io.Source.fromURL(new java.net.URL("http://www.simplyscala.com/interp?code='%s'.isLowerCase"
.format(c.toChar.toString))).mkString) 
forall (_ contains "true"))

— and he notes: I could use threads to speed that up a little.

Salmon Run and other NLP Toolkits

Fri, 11 Sep 2009 19:02:46 -0400

There’re several important NLP toolkits in Java and soon Scala, as well as blogs about them. Here’s some.

Salmon Run Excellent soup-to-nuts usage guides to many JVM technologies.

LingPipe We’re using it for N-gram models.

OpenNLP My UPenn PhD classmate Tom Morton’s framework.

DRMacIver It’s not a project, it’s a person! Awesome tools.

Bzip2

Fri, 11 Sep 2009 18:57:12 -0400

Needed to open bz2 files in the same way as zip ones. Old bzip2 streams for Java have a bug where you have to manually swallow two first characters from the underlying file stream, which are BZ, before you coudl give it to the bzip2 stream. The modern wrapper from Apache Commons relieves you from this tedious check.

Apache Commons Compress

Value Serialization

Fri, 11 Sep 2009 18:52:05 -0400

So I needed to serialize a bunch of Scala Maps, and it’s supersimple: just tack a

@serialize

on top of the declaration, be it a class, or just a value! And then the usual Java serialization applies!

Command-line options

Fri, 11 Sep 2009 18:48:23 -0400

DRMacIver has a super-lovely li’l class, optional.Application, from which you can derive your main object, with parameters for main being extracted from the cmdline.

Graph Storage

Fri, 11 Sep 2009 18:46:49 -0400

When representing data as a graph, it’s important to get efficient storage and traversing platform for it. Some good JVM ones I found:

JGraphT

Neo4j

Any RDF storage can be used for triplets — edges with a weight/relationship, e.g.

AllegroGraph RDF Store

Other RDF stores, such as the one underlying DBLP++.

Autovivification, anyone?

Fri, 17 Jul 2009 15:41:37 -0400

I’m storing a bunch of counts for twits in a map of maps, like

import scala.collection.mutable.{Map=>UMap}

class Repliers {
  type RepCount = UMap[UserID,(TwitCount,TwitCount)]
  var reps: UMap[UserID, RepCount] = UMap.empty

  // TODO here goes our favorite non-autovivification,
  // a question is, whether this can be made into auto-
  def addTwit(twit: Twit): Unit = {
    twit.reply match {
      case None =>
      case Some(reply) =>
        val uid = twit.uid
        val ruid = reply.replyUser
        val tinc = if (reply.replyTwit.isEmpty) 0 else 1
        val u = reps.get(uid)
           u match {
             case Some(repcount) => repcount.get(ruid) match {
               case Some((numUser,numTwit)) => {
                 val x = reps(uid)(ruid)
                 reps(uid)(ruid) = (x._1 + 1, x._2 + tinc)
               }
               case _ => reps(uid)(ruid) = (1, tinc)
             }
             case _ => reps(uid) = UMap(ruid->(1, tinc))
           }
    }
  }
// ...
}

How do we auto-vivify the maps, by a kind of more laconic upsert (insert or update)?

Using Berkeley DB DPL for Implementations of Abstract Scala Classes

Fri, 17 Jul 2009 15:33:08 -0400

I have a class which is a spec for storage and may be implemented via PostgreSQL, Berkeley DB, db4o, jdbm, etc. Annotating it is not possible as it’s already published and backend-agnostic, so, for BDB, we create a mirror class with Java types and fill it from the original Scala class. This allows to use nullable values instead of Options to simplify indexing, and allows to serialize types Berkeley DB knows about, such as Date, instead of Joda’s DateTime. I have constructors which take the Scala original and fill in the annotated BDB Entity fields, applying conversions such as Long=>java.lang.Long, and toScalaStuff functions to return Scala ones back, with reverse conversions. Here’s how one class is mirrored in BDB:

The spec:

case class ReplyTwit (
  tid: TwitID,
  replyTwit: Option[TwitID],
  replyUser: UserID
  )

BDB body:

@Entity
class ReplyTwitBDB {
  @PrimaryKey
  var tid: java.lang.Long = null
  @SecondaryKey{val relate=MANY_TO_ONE}
  var replyTwit: java.lang.Long = null
  @SecondaryKey{val relate=MANY_TO_ONE}
  var replyUser: java.lang.Integer = null

  def this(
    _tid: TwitID,
    _replyTwit: Option[TwitID],
    _replyUser: UserID
    ) = { this()
      tid = _tid
      replyTwit = _replyTwit match {
        case Some(x) => x
        case _ => null
      }
      replyUser = _replyUser
    }
  def this(t: ReplyTwit) = this(t.tid,t.replyTwit,t.replyUser)
  def toReplyTwit: ReplyTwit = {
    val replyTwitLongOpt = replyTwit match {
      case null => None
      case x => Some(x.longValue)
    }
    ReplyTwit(tid.longValue,replyTwitLongOpt,replyUser.intValue)
  }

  override def toString:String = {
    var s = "Reply tid:%d ru:%d" format (tid,replyUser)
    if (replyTwit != null) s += " rt:"+replyTwit
    s
  }
}

It may be a bit verbose, but it works, and is used in db-bdb.scala in tfitter. I might also implement the original BDB Bind API with my own serialization, but then I’d have to manage secondary indices manually as databases. The DPL also allows me to evolve the class layouts with versioning and narrowing/widening between them.

Back from the Concrete into the Abstract

Fri, 17 Jul 2009 15:10:51 -0400

Once I implemented a PostgreSQL interface for my TwitterDB abstract class, I did another backend with Berkeley DB, Java Edition. When doing it, I found that one method is almost identical, so I decided to push it back into the abstract class. But how’d I do construct backend-specific objects? One way is to pass them back in as function parameters. So here’s the result — in the abstract class:

case class SubParams (
  makeTwit:  TwitID => TwitDB, 
  makeUser:  UserID => UserDB,
  txnBegin:     () => Unit,
  txnCommit:    () => Unit,
  txnRollback:  () => Unit
)


// curry make/txn params
def insertUserTwitCurry(subParams: SubParams)(ut: UserTwit)
    : Unit = {
    import System.err

    val SubParams(makeTwit,makeUser,txnBegin,txnCommit,
        txnRollback) = subParams

    val UserTwit(user,twit) = ut
    val uid = user.uid
    val tid = twit.tid
    try {

      val t = makeTwit(tid) // TwitPG(tid)

      txnBegin
      t put twit // will cause exception if present and rollback
      val u = makeUser(uid) // UserPG(uid)
      u.updateUserForTwit(ut)
      txnCommit
    } catch {
      case e => {
        err.println(e)
        err.println("ROLLBACK uid="+uid+" tid="+tid)
        txnRollback
      }
    }
}

I needed to pass a several functions to a method as parameters, and created a class, subParams, to hold them. I pass case class constructors, such as TwitPG, directly as functions there, and it works.

Interestingly, one can supply the same def blah: Unit function via call by name into a def, but have to group together in a case class with ()=>Unit, you can’t have an x: => Unit parameters to a case class.

Here’s how I implement my abstract function in the PostgreSQL and Berkeley DB backends:

// PostgreSQL has no commit begin, passing Unit=>Unit do-nothing 
val subParams = SubParams(TwitPG,UserPG,()=>(),conn.commit _,
    conn.rollback _)
def insertUserTwitB = insertUserTwitCurry(subParams)(_)

// Berkleley DB
val subParams = SubParams(TwitBDB,UserBDB,txnBegin _,
    txnCommit _,txnRollback _)
def insertUserTwitB = insertUserTwitCurry(subParams)(_)

sneaking closing actions into an iterator

Thu, 16 Jul 2009 21:32:05 -0400

I created an iterator to wrap a Berkeley DB cursor. One thing with iterating through it is closing the cursor when we reach the end of the iteration. So I simply stuck the closing into the hasNext call-through to the underlying Java iterator!

  class TwIteratorBDB extends TwIterator {
    val cursor = twitPrimaryIndex.entities
    val javaIter = cursor.iterator
    // TODO a way to stick closing actions into our iterator,
    // are there any official or better ways?
    def hasNext: Boolean = { val has = javaIter.hasNext
      if (!has) cursor.close
      has
    }
    def next: Twit = javaIter.next.toTwit
  }

continually, ultimately, takeWhileFinally

Thu, 16 Jul 2009 21:31:48 -0400

I needed to generate a Stream out of a Berkeley DB cursor. One thing you need to do when you finish reading from a cursor is to close it. So we needed to have a Stream with a closing action. @paulp to the rescue!

First, we ‘d came up with this cute Stream generator, without cursor closing yet:

  def cursorStream[T](cursor: EntityCursor[T]): Stream[T] =
        continually(cursor.next _) map (_.apply()) 
        takeWhile (_ != null)

Rumor has it you can say just map(_.apply) or evenmap(_())`

continually is in 2.8 but not in 2.7, so I made it up as // paulp has it in scala.util.control.Exception in 2.8, // replacing Stream.Cons with Stream.cons works in 2.7: def continually[A](elem: => A): Stream[A] = Stream.cons(elem, continually(elem))

OK, but now we need to close the cursor when finished with the stream. @paulp suggested ultimately(…) which he had written together with s.u.c.E in 2.8 as well. I copied it into 2.7 as

  // def ultimately[T](body: => Unit): Catch[T] = 
  //        noCatch andFinally body
  def ultimately[T](fin: => Unit)(body: => T): T = 
    try { body } finally { fin }

So one would have though this might work:

  // WRONG: closes right after head, tail-lazy Stream!
  def cursorStream[T](cursor: EntityCursor[T]): Stream[T] =
    ultimately(cursor.close) { 
      continually(cursor.next _) map (_.apply()) 
        takeWhile (_ != null) force.toStream }

— but it doesn’t, doing fin right after the first head read.

@paulp promised 22 other ways to do it otehrwise, e.g.

  // WRONG: error on append (() => cursor.close), 
  // same wrong Unit type, not T, 
   // for Stream(cursor.close _):
  (continually(cursor.next _) takeWhile (_ != null)) 
    append (() => cursor.close) map (_.apply())

(continually(cursor.next _) takeWhile (_ != null)) ++ 
  Stream(cursor.close _) map (_())

— both of which don’t typecheck as they try to pin a Unit on top on a T Stream. I’d gladly explore the remaining 20 ideas, as this got me really obsessed with brevity and the strange new world of s.u.c.E-like combinators! noCatch andThen, man!

@dibblego briefly mentioned

val r = st takeWhile p; fin; r

— but retracted it in the face of side effects (inherent in dealing with cursor).

And I ended up with the combinator I really needed, takeWhileFinally,

  def takeWhileFinally[T](st: Stream[T], p: T => Boolean, 
    fin: => Unit): Stream[T] =
    st match {
      case Stream.cons(head,tail) if p(head) => 
        Stream.cons(head, takeWhileFinally(tail,p,fin))
      case _ => fin; Stream.empty
    }

— which keeps pushing fin to the closing predicate check. In terms of it, the working cursorStream looks like

  def cursorStream[T](cursor: EntityCursor[T]): Stream[T] =
    takeWhileFinally(continually(cursor.next _) map (_.apply()), 
      (_:T) != null, cursor.close)

I also learned that Jorge Ortiz doesn’t like Stream anymore to the point of advocating its banishment, although his Euler project solutions in Scala from 12/2007 employ it with wild abandon, causing Scala 2.8 stack overflow on fib right away when trying those.

Further Maven Findings: CLI plugin, errors

Tue, 23 Jun 2009 22:45:44 -0400

There’s a maven-cli-plugin by Don Brown, the author of the maven-yamlpom-plugin as well. The CLI one allows to start a maven shell, maven2>, and enter commands like compile. The following findigs apply to Scala:

when running as mvn cli:execute, compile means Java compile and does nothing.
need to run in phase mode, mvn cli:execute-phase
then got errors for maven-resource-plugin having wrong description, apparently twdata’s, where yaml and cli plugins come from. This error persists in maven 2.1.0 but is not there for 2.0.10, so I ended up doing the CLI plugin under 2.0.10.

Still, whenever I push arrow up the first time, half the time I get “pipe closed” error and the shell quits. So much for reusing maven metadata.

Some other maven fun: when I installed, by tab completing oversight, a tar.gz instead of a jar into it, with -Dpackaging=jar, it did nothing to warn me, and mvn compile couldn’t find the classes which were supposed to be in that jar; also was showing downloading every time.

On Mac OSX, jar-with-dependencies works fine attached to package phase; on CentOS, same pom was doing weird thing under both maven 2.0.10 and 2.1.0, cooking the jar seemingly OK and then erroring with “cannot open zip file.”

Now since we got a team here at Dartmouth, I’d like to make my m2 repo shared. So I moved my ~/.m2 to /opt/m2. Then I created a new ~/.m2 and placed the following settings.xml there:


    /opt/m2/repository

Interestingly, the original settings.xml, with their defined, are now in /opt/m2/settings.xml — and seem to work.

I also found, in $M2_HOME, there’s a bin/m2.conf with a line,

set maven.home default ${user.home}/m2

but replacing it with an e.g.

set maven.home default /opt/m2

— doesn’t lead to using that instead of ~/.m2 by itself.

Learning About Ranges -- the Hard Way

Tue, 23 Jun 2009 22:37:18 -0400

I have a 3-part actor pipeline, ReadLines feeding Parsers feeding Inserters. The Parsers hold both other ends as parameters. Originally they were connected to each other as

paste

ReadLines started
Inserter 0 started, object com.tfitter.Status$Inserter@384ab40a
Inserter 1 started, object com.tfitter.Status$Inserter@636f2067
Parser 0 started, object com.tfitter.Status$JSONExtractor@7d98d9cf, talking to Inserter 0 [com.tfitter.Status$Inserter@71cd427a]
Parser 1 started, object com.tfitter.Status$JSONExtractor@60328c40, talking to Inserter 1 [com.tfitter.Status$Inserter@1695ef17]
.parser 0 [com.tfitter.Status$JSONExtractor@7d98d9cf] sends its inserter 0 [com.tfitter.Status$Inserter@71cd427a] twit 2173617511
parser 1 [com.tfitter.Status$JSONExtractor@60328c40] sends its inserter 1 [com.tfitter.Status$Inserter@1695ef17] twit 2173617601
parser 0 [com.tfitter.Status$JSONExtractor@7d98d9cf] sends its inserter 0 [com.tfitter.Status$Inserter@71cd427a] twit 2173617606
parser 0 [com.tfitter.Status$JSONExtractor@7d98d9cf] sends its inserter 0 [com.tfitter.Status$Inserter@71cd427a] twit 2173617603
parser 1 [com.tfitter.Status$JSONExtractor@60328c40] sends its inserter 1 [com.tfitter.Status$Inserter@1695ef17] twit 2173617604
parser 0 [com.tfitter.Status$JSONExtractor@7d98d9cf] sends its inserter 0 [com.tfitter.Status$Inserter@71cd427a] twit 2173617605
parser 1 [com.tfitter.Status$JSONExtractor@60328c40] sends its inserter 1 [com.tfitter.Status$Inserter@1695ef17] twit 2173617613
parser 0 [com.tfitter.Status$JSONExtractor@7d98d9cf] sends its inserter 0 [com.tfitter.Status$Inserter@71cd427a] twit 2173617609
parser 1 [com.tfitter.Status$JSONExtractor@60328c40] sends its inserter 1 [com.tfitter.Status$Inserter@1695ef17] twit 2173617607
end of input, informing parser 1
Parser 1 exiting.
parser 0 [com.tfitter.Status$JSONExtractor@7d98d9cf] sends its inserter 0 [com.tfitter.Status$Inserter@71cd427a] twit 2173617614
end of input, informing parser 0
ReadLines exiting
Parser 0 exiting.

The inserters which started were one, and those to whom parsers send messages — different. It turns out that (0 until numThreads) is not a List but rather a Range, and map applied to it did something quite different — so when those inserters, not a list, were iterated over, we got new inserters assigned to parsers instead of the originals. Or, as @paulp said, I’ve learned about the ranges the hard way.

I found it simultaneously by appending the type annotation to inserters as shown below — that led to adding .toList onto the Range.

Here’s the right way:

val readLines = new ReadLines(args(0),numThreads,showingProgress)

// I didn't realize I'm not using a List but get a Range...
// added .toList below
val inserters: List[Inserter] = (0 until numThreads).toList map (
    new Inserter(_))

val parsers: List[JSONExtractor] = inserters map (
    ins => new JSONExtractor(ins.id,readLines,ins))

Finally, I started the processes in the end of the main:

readLines.start
inserters map (_.start)
parsers map (_.start)

Since I had main defined as

def main(args: Array[String]) = { ... }

I was getting an error that main cannot be found! Apparently this is because the last map returns not a Unit. In OCaml there’s a different iterator in addition to List.map returning a unit type — List.iter, and similarly we have foreach. Additionally, we can exploit a typical beginners’ error, omitting = between the def and the body, to ensure main discarding any result and thus always returning Unit. Altogether, this looks like:

def main(args: Array[String])  { 
    // no = for main: Array[String] => Unit!

    readLines.start
    inserters foreach (_.start)
    parsers foreach (_.start)
}

On the other hand, for the first time since Ada’s rendezvous, an elegant parallel app works well for me, taking up to 600% CPU on an 8-core box. Compared with single-threaded MacBook Pro at 38 minutes, that animal finishes in 7 minutes. This even makes it unnecessary to rewrite the whole thing in OCaml.

case objects, parens, extends abstract

Tue, 23 Jun 2009 21:30:57 -0400

For passing messages between actors in my system, I declared things from memory as

abstract case class ParserMessage()
case class Parse(s: String) extends ParserMessage
case class EndOfInput // extends ParserMessage // caused class not found exception!

One funny thing occurred is that EndOfInput was not found in the receiving actor until I cut off its extends. This was probably caused by abstract case class — needed to cut off case per @jorgeortiz85. Scala 2.8 said that case classes without parameters are deprecated — I had to do EndOfInput() or use a case object, which I did. Before that declaring with parens and matching without them also caused “class not found” error.

So now the right way:

abstract class ParserMessage() // hmm, still need those ()?
case class Parse(s: String) extends ParserMessage
case object EndOfInput extends ParserMessage

default parameters breaking old code

Tue, 23 Jun 2009 14:31:07 -0400

Fun facts from @DRMacIver and @paulp, formatted and elided for blog.

DRMacIver: Fun fact: the reason Stream changed from a trait to a class was because doing so made its memory usage change from unbounded to constant in certain circumstances,
paulp: You can’t overload foo and foo(), they are the same in definition. You can define overloads in the repl, but like this:

def foo = 5 ; def foo() = 10

— of course that one won’t work.

def foo() = 5 ; def foo(x: String) = 5 + x.length.

paulp: You’d think you could unify these as

def foo(x: String = "") = 5 + x.length

but you will break things if you do. (I will leave what breaks as puzzle.)

DRMacIver: The big thing I want to unify and know that we can’t is mkString paulp: indeed mkString is a perfect example. DRMacIver: As clearly it should be unified as

def mkString(start : String = "", def connect : String = "", def end : String = "") : String = ...
paulp: and would break the most code, maybe.
DRMacIver: but the problem is that if you do that then mkString(“,”) does rather the wrong thing
paulp: DRMacIver: not only that. mkString now won’t work. You’d have to say mkString().
DRMacIver: I consider that a boring and acceptable breakage
paulp: it’s awfully widespread.
DRMacIver: Particularly given that rules for ()s are going to be tightened up anyway at some point paulp: then we’d best do it now.
DRMacIver: It’s awfully widespread and awfully easy to fix. I think it is something slated for 2.8. But it’s probably one of those things that are easy to forget about… paulp: what is so wrong with mkString(connect, start, end) then?
DRMacIver: Existing code compiles but does the wrong thing. Which is a nasty thing to do
paulp: DRMacIver: mark it with a @transition annotation, enable a -Xtransition flag, and complain about everything using it unless they re-mark it with @itscool (or stop compiling with -Xtransition.) I dunno, should be some annoyance-based approach out there.
paulp: alexyk: it’s a “feature” I believe that you must supply a parameter list if there are any defaults. I don’t put in feature in quotes because I think it’s a bad idea, it’s probably a good idea.

anyToMap versions

Tue, 23 Jun 2009 14:20:25 -0400

I wanted to get an anyToMap in a shape most reminiscent of OCaml/Haskell, similar to a partial function consisting of a series of case matches in react. Here’s some looks.

// original and
// @djspiewak
def anyToMap(a: Any) = a match {
  case m: Map[_, _] => m
  case _ => throw new RuntimeException("Baad")
}

// alternatively
def anyToMap(a: Any) = a.asInstanceOf[Map[_, _]]

// verbose
def anyToMap = { a:Any =>a match {
  case m: Map[_, _] => m
  case _ => throw new RuntimeException("Baad")
}}

And the winner is…

// @jesnor
val anyToMap : Any => Map[_, _] = 
    { case m: Map[_,_] => m; case _ => error("Baad") }

Also, as @paulp reminds us, we know that

error("Baad") == throw new RuntimeException("Baad")

The Price of Trunk

Tue, 16 Jun 2009 22:59:40 -0400

I need to parse JSON streams, and was going to do it with scala-json. Now that is an Ant+Ivy production, untangled below, and it depends on @robey’s `configgy. I was wondering why — it has no config… of course it bit me in the arse.

There’s a method call, s.regexSub, in def quote(s: String). Obviously there’s no regexSub in String nor in scala.runtime.RichString, and thus I was searching for a while. The thing has an import on top:

import net.lag._

— and sure enough, configgy has a file, extensions.scala, with said regexSub and unquoteC defined on a ConfiggyString. There’s also a top-level object extensions with an

implicit def stringToConfiggyString(s: String): ConfiggyString =
    new ConfiggyString(s)

OK, got that. However, configgy is also Ant+Ivy and depends on Scala 2.7.3. Am going into ivysettings.xml and try to replace 2.7.3 with 2.8.0-SNAPSHOT. However, as opposed to Maven, Ivy cannot find 2.8.0-SNAPSHOT — in repo-snapshots, on scala-tools.org, there’s a directory 2.8.0-SNAPSHOT/, but no scala-...-2.8.0-SNAPSHOT.jar.

Fine, I edit ivysettings.xml and add a maven-local repo, and since I do have a Scala 2.8 in ~/.m2/, it starts compiling — only to error out on collections and stuff. So much for configgy for now — could only email @robey for 2.8 update…

For now I copied extensions.scala to scala-json. For testing, it’s using specs, and it turns out specs doesn’t compile under 2.8 either. This calls for an agonizing choice between BDD and 2.8, same as for others. Then I simply uncomment . Now it compiles, issuing some unchecked and deprecated warnings. Those can be revealed in maven with the following configuration to maven-scala-plugin:


    org.scala-tools
    maven-scala-plugin
    
    
        
            
                add-source
                compile
                
                
            
        
    
     
         
         
         
        -deprecation 
        -unchecked 
         
        -optimise

OK, now I got my scala-json! Using @paulp’s cool :jar command in REPL, we can test it right away, without restarting REPL:

scala> :jar /Users/alexyk/.m2/repository/com/twitter/scala-json/1.0/scala-json-1.0.jar
Added /Users/alexyk/.m2/repository/com/twitter/scala-json/1.0/scala-json-1.0.jar to your classpath.

scala> import com.twitter.commons.Json
import com.twitter.commons.Json

scala> Json.build(Json.build(List(1, 2))).toString
res0: String = [1,2]

mvn install:install-file # woes

Tue, 16 Jun 2009 16:14:47 -0400

Trying to install an ivy-generated jar for configgy-1.3 into the local .m2 repo. Ant+Ivy end up with a dist/ directory, e.g. dist/configgy-1.3/configgy-1.3.{jar,pom} and more. When I try to do, manually,

 mvn install:install-file -Dfile=configgy-1.3.jar -DgroupId=net.lag
   -DartifactId=configgy -Dversion=1.3 -Dpackaging=jar

maven complains there’s no artifactId or etc. or packaging — it was originally -Dpackaging=jar missing, like it doesn’t know what, huh? You messin’ with me, mah-ven?

Thanks to @mharrah’s good graces, here’s a way to use Ant+Ivy to install the fruit of its labors into maven’s own repo:

Change “local” on line 85 of ant/package.xml to "maven-local" Add this after on line 22 of ivy/ivysettings.xml:

Run ant package

Let the Week Begin with Another Java Build-Tool ACI!

Mon, 15 Jun 2009 17:27:58 -0400

Those who read this blog probably know what ACI means. (See previous posts if you don’t.) One thing about Java-based world is that it produces ACI in a frequency similar to MSFT: there’s a new self-inflicted Java build tool ACI every day! Here’s how.

I didn’t rest on my newly obtained Maven-fu laurels more than a day when I wanted to get scala-json off the github and into my mavenized project. Well, scala-json is not mavenized, but rather is antivyized — it uses the Ant+Ivy combo to build, as other things from Twitter HQ do nowadays.

Since it relied on older versions of everything, I had to immediately go in and prevent downloading of obsolete Scala source jars — even Maven didn’t want those sources, but this did.

The build breaks at docs for the lack of a Scala method (presumably due to the Scala version update). It still produces some commons-1.0 packages — both a zip and a jar.

Now I end up with a dist subdir with a commons-1.0.zip and commons-1.0/, the latter containing the commons-1.0.jar. Strangely, the zip contains all the prerequisites but no json, and the jar is the json. But it’s referred in the commons-1.0.pom sitting alongside by artifactId org.twitter and name commons.

How do you install an Ant+Ivy-built jar into your maven repo?

Encrypting and Replacing Passwords with Maven

Mon, 15 Jun 2009 17:03:41 -0400

I’ve had an interesting ACI experience (lana.reverse-cranial inversion, similar to MSFT’s inversion of / into \) trying to hide my Twitter password in a Maven app from it being deployed to GitHub. At first I absent-mindedly hard-coded the password into code and uploaded it, which called for a need to learn how to obliterate a file completely from a git repo. Then temporarily, I replaced the password line with

twitterPassword = readLine

Which of course is still not good enough, even though you can call such a program with echo pwd | program and keep in the commandline for repetition with GNU readline. One real persistent way to do it is read from a file. The below is the same thing with ACI-inspired Maven and Java resources.

Basic Maven resource filtering is described in Maven Getting Started Guide — Filtering Resources. I created my src/main/resources/application.resources file with just two lines,

${twitter.user}
${twitter.password}

The actual passwords are in another file I created not to be version-tracked, src/main/filters/filter.properties, with just two lines for the actual things:

twitter.user=actualUserName
twitter.password=actualPassword

Then I stuck the following into the POM:


    src/main/scala

    
        src/main/filters/filter.properties
    

    
      
        src/main/resources
        true
      
    ...

Now we access this resource as follows, from Scala’s main:

  def main(args: Array[String]) = {

    def getProperties = 
      this.getClass getResourceAsStream "/application.properties"
    var st = getProperties

    // in order to reset, do ss = ss.reset // !
    var ss = BufferedSource.fromInputStream(st, "UTF-8", 512, { 
      () =>  Source.fromInputStream(getProperties) })

    val linesIter = ss.getLines
    val twitterUser=linesIter.next.trim
    // can use some simple steganography here
    val twitterPassword=linesIter.next.trim

    println("user=>"+twitterUser+", password=>"+twitterPassword)
  }

In order to be able to reread from the stream, one wishes to make it resettable with ss.reset — and it turned out to be necessary to use BufferedSource here. You have to reassign ss.reset to something on which you’ll re-read the source, e.g. declare it a var and reassign to itself with ss = ss.reset (hence that var).

Since I own scala.la domain, my packages are all operas, starting with la.scala. Thus I discovered I can’t refer to things as scala.io.BufferedSource — I either have to prefix it with _root_., or import things, as I’ve done here. Alas, one can’t rename top-level scala with something like import _root_.{scala=>lisp}: it compolains, somewhat confusingly, that _root_ cannot be imported.

It would be nice to encrypt all passwords similarly to password encryption in Mavem 2.1.0 settings.xml. However, it’s used mostly for deployment plugin. Were you to refer to the passwords in filtered resources, they still will be plain text in the end. One thing I suggested on the Maven Users list is

the password to be encrypted in the same way as Maven 2.1.0 does it; then
the password decrypted internally during compilation
the source file with the password to be interpolated with the decrypted password during compilation
upon successful compilation into a class file or failed compile, the expanded source to be discarded from target/

I also asked, on the Maven Users list, whether, and how, one can refer to elements from settings.xml:

Also, I tried to refer to the password as defined in settings.xml,


    
        twitter
        blah

…

— but the dotted notation seems incapable of referring to the username from the block with the id of “twitter”. Is there a way to do it with ${settings:...what?...}

Nobody from Sonatype replied to this, which a bit clouds the idea of “Sonatype the Maven company” supporting their product properly.

Some more references to settings.xml are here:

Maven Settings

General Settings

One thing worth checking, suggested by @nuttycom, is setting a repo policy there to not auto-download-and-update snapshots, preventing the sutuation where my local Scala build is partially supplanted by a broken Maven snapshot (where scala-compiler was failing for a while).

Upd. Here’s how one can filter the source code — just have to tell the compile phase to work off the filtered code, not the originals — by Bram de Kruijff:


  ...
  
    ...
      
      target/filtered-sources/java
  ...
  
...