Simple build tools

[local ~/projects]$ sbt
Project does not exist, create new project? (y/N/s) y
Name: sample
Organization: com.twitter
Version [1.0]: 1.0-SNAPSHOT
Scala version [2.7.7]: 2.8.1
sbt version [0.7.4]:      
Getting Scala 2.7.7 ...
:: retrieving :: org.scala-tools.sbt#boot-scala
    confs: [default]
    2 artifacts copied, 0 already retrieved (9911kB/221ms)
Getting org.scala-tools.sbt sbt_2.7.7 0.7.4 ...
:: retrieving :: org.scala-tools.sbt#boot-app
    confs: [default]
    15 artifacts copied, 0 already retrieved (4096kB/167ms)
[success] Successfully initialized directory structure.
Getting Scala 2.8.1 ...
:: retrieving :: org.scala-tools.sbt#boot-scala
    confs: [default]
    2 artifacts copied, 0 already retrieved (15118kB/386ms)
[info] Building project sample 1.0-SNAPSHOT against Scala 2.8.1
[info]    using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7

You can see that it has created a snapshot version of the project in a better form.

The project layout

Project - Project definition file
- project/build/.scala main project definition file
- project/build.properties Project, sbt, and Scala version definitions
src/main Your application code appears here, indicating the language of the code in the subdirector src/main/scala src/main/java
src/main/resources files you want to add to the jar package (e.g. log configuration)
src/test like src/main, just for testing
lib_managed jar files on which your project depends. Filled by sbt update
target - the target path of the generator (e.g. automatically generated thrift code, class files, jar packs)

Add some code

We'll create a simple JSON parser for a simple tweet message. Add the following code to src/main/scala/com/twitter/sample/SimpleParser.scala

package com.twitter.sample

case class SimpleParsed(id: Long, text: String)

class SimpleParser {

  val tweetRegex = "\"id\":(.*),\"text\":\"(.*)\"".r

  def parse(str: String) = {
    tweetRegex.findFirstMatchIn(str) match {
      case Some(m) => {
        val id = str.substring(m.start(1), m.end(1)).toInt
        val text = str.substring(m.start(2), m.end(2))
        Some(SimpleParsed(id, text))
      }
      case _ => None
    }
  }
}

This code is ugly and has bugs, but should be able to compile through.

Tests in the console

SBT can be used neither as a command-line script nor as a build console. We'll primarily use it as a build console, but most commands can be passed as parameters to SBT to run independently, for example

sbt test

Note That if a command requires parameters, you need to use quotation marks to include the entire parameter path, for example

sbt 'test-only com.twitter.sample.SampleSpec'

It's a strange way.

Anyway, to get our code working, start SBT

[local ~/projects/sbt-sample]$ sbt
[info] Building project sample 1.0-SNAPSHOT against Scala 2.8.1
[info]    using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7
>

SBT allows you to start a Scala REPL and load all project dependencies. It compiles the source code for the project before starting the console, giving us a workstable for a quick test parser.

> console
[info] 
[info] == compile ==
[info]   Source analysis: 0 new/modified, 0 indirectly invalidated, 0 removed.
[info] Compiling main sources...
[info] Nothing to compile.
[info]   Post-analysis: 3 classes.
[info] == compile ==
[info] 
[info] == copy-test-resources ==
[info] == copy-test-resources ==
[info] 
[info] == test-compile ==
[info]   Source analysis: 0 new/modified, 0 indirectly invalidated, 0 removed.
[info] Compiling test sources...
[info] Nothing to compile.
[info]   Post-analysis: 0 classes.
[info] == test-compile ==
[info] 
[info] == copy-resources ==
[info] == copy-resources ==
[info] 
[info] == console ==
[info] Starting scala interpreter...
[info] 
Welcome to Scala version 2.8.1.final (Java HotSpot(TM) 64-Bit Server VM, Java 1.6.0_22).
Type in expressions to have them evaluated.
Type :help for more information.

scala>

Our code compiles through and provides a typical Scala prompt. We'll create a new parser, a tweet to make sure it "works"

scala> import com.twitter.sample._            
import com.twitter.sample._

scala> val tweet = """{"id":1,"text":"foo"}"""
tweet: java.lang.String = {"id":1,"text":"foo"}

scala> val parser = new SimpleParser          
parser: com.twitter.sample.SimpleParser = com.twitter.sample.SimpleParser@71060c3e

scala> parser.parse(tweet)                    
res0: Option[com.twitter.sample.SimpleParsed] = Some(SimpleParsed(1,"foo"}))

scala>

Add dependencies

Our simple parser works fine with this very small input set, but we need to add more tests and make it wrong. T he first step is to add the specs test library and a real JSON parser to our project. To do this, we must go beyond the default SBT project layout to create a project.

SBT considers the Scala file in the project/build directory to be a project definition. Add the following to this file project/build/SampleProject.scala

import sbt._

class SampleProject(info: ProjectInfo) extends DefaultProject(info) {
  val jackson = "org.codehaus.jackson" % "jackson-core-asl" % "1.6.1"
  val specs = "org.scala-tools.testing" % "specs_2.8.0" % "1.6.5" % "test"
}

A project definition is an SBT class. In the example above, we extended defaultProject for SBT.

Here is the dependency declared by val. S BT uses reflection to scan all val dependencies in a project and establish a dependency tree at build time. The syntax used here may be new, but the nature and Maven dependency are the same

<dependency>
  <groupId>org.codehaus.jackson</groupId>
  <artifactId>jackson-core-asl</artifactId>
  <version>1.6.1</version>
</dependency>
<dependency>
  <groupId>org.scala-tools.testing</groupId>
  <artifactId>specs_2.8.0</artifactId>
  <version>1.6.5</version>
  <scope>test</scope>
</dependency>

We can now download our project dependent. Run sbt update on the command line instead of sbt console

[local ~/projects/sbt-sample]$ sbt update
[info] Building project sample 1.0-SNAPSHOT against Scala 2.8.1
[info]    using SampleProject with sbt 0.7.4 and Scala 2.7.7
[info] 
[info] == update ==
[info] :: retrieving :: com.twitter#sample_2.8.1 [sync]
[info]  confs: [compile, runtime, test, provided, system, optional, sources, javadoc]
[info]  1 artifacts copied, 0 already retrieved (2785kB/71ms)
[info] == update ==
[success] Successful.
[info] 
[info] Total time: 1 s, completed Nov 24, 2010 8:47:26 AM
[info] 
[info] Total session time: 2 s, completed Nov 24, 2010 8:47:26 AM
[success] Build completed successfully.

You'll see that sbt retrieves the specs library. A lib_managed directory has also been added and lib_managed/scala_2.8.1/test is included in specs_2.8.0-1.6.5.jar

Add a test

Now that you have a test library, you can write the following test src/test/scala/com/twitter/sample/SimpleParserSpec.scala file

package com.twitter.sample

import org.specs._

object SimpleParserSpec extends Specification {
  "SimpleParser" should {
    val parser = new SimpleParser()
    "work with basic tweet" in {
      val tweet = """{"id":1,"text":"foo"}"""
      parser.parse(tweet) match {
        case Some(parsed) => {
          parsed.text must be_==("foo")
          parsed.id must be_==(1)
        }
        case _ => fail("didn't parse tweet")
      }
    }
  }
}

Run test in the SBT console

> test
[info] 
[info] == compile ==
[info]   Source analysis: 0 new/modified, 0 indirectly invalidated, 0 removed.
[info] Compiling main sources...
[info] Nothing to compile.
[info]   Post-analysis: 3 classes.
[info] == compile ==
[info] 
[info] == test-compile ==
[info]   Source analysis: 0 new/modified, 0 indirectly invalidated, 0 removed.
[info] Compiling test sources...
[info] Nothing to compile.
[info]   Post-analysis: 10 classes.
[info] == test-compile ==
[info] 
[info] == copy-test-resources ==
[info] == copy-test-resources ==
[info] 
[info] == copy-resources ==
[info] == copy-resources ==
[info] 
[info] == test-start ==
[info] == test-start ==
[info] 
[info] == com.twitter.sample.SimpleParserSpec ==
[info] SimpleParserSpec
[info] SimpleParser should
[info]   + work with basic tweet
[info] == com.twitter.sample.SimpleParserSpec ==
[info] 
[info] == test-complete ==
[info] == test-complete ==
[info] 
[info] == test-finish ==
[info] Passed: : Total 1, Failed 0, Errors 0, Passed 1, Skipped 0
[info]  
[info] All tests PASSED.
[info] == test-finish ==
[info] 
[info] == test-cleanup ==
[info] == test-cleanup ==
[info] 
[info] == test ==
[info] == test ==
[success] Successful.
[info] 
[info] Total time: 0 s, completed Nov 24, 2010 8:54:45 AM
>

Our test passed! N ow we can add more. R unning trigger actions is one of the best features that SBT provides. A dding a wavy line at the beginning of the action starts a loop and re-runs the action as the source file changes. Let's ~test and see what happens.

[info] == test ==
[success] Successful.
[info] 
[info] Total time: 0 s, completed Nov 24, 2010 8:55:50 AM
1. Waiting for source changes... (press enter to interrupt)

Now, let's add the following test case

 "reject a non-JSON tweet" in {
      val tweet = """"id":1,"text":"foo""""
      parser.parse(tweet) match {
        case Some(parsed) => fail("didn't reject a non-JSON tweet")
        case e => e must be_==(None)
      }
    }

    "ignore nested content" in {
      val tweet = """{"id":1,"text":"foo","nested":{"id":2}}"""
      parser.parse(tweet) match {
        case Some(parsed) => {
          parsed.text must be_==("foo")
          parsed.id must be_==(1)
        }
        case _ => fail("didn't parse tweet")
      }
    }

    "fail on partial content" in {
      val tweet = """{"id":1}"""
      parser.parse(tweet) match {
        case Some(parsed) => fail("didn't reject a partial tweet")
        case e => e must be_==(None)
      }
    }

After we save the file, SBT detects a change, runs the test, and notifies us that there is a problem with the parser

[info] == com.twitter.sample.SimpleParserSpec ==
[info] SimpleParserSpec
[info] SimpleParser should
[info]   + work with basic tweet
[info]   x reject a non-JSON tweet
[info]     didn't reject a non-JSON tweet (Specification.scala:43)
[info]   x ignore nested content
[info]     'foo","nested":{"id' is not equal to 'foo' (SimpleParserSpec.scala:31)
[info]   + fail on partial content

So let's rework the real JSON parser

package com.twitter.sample

import org.codehaus.jackson._
import org.codehaus.jackson.JsonToken._

case class SimpleParsed(id: Long, text: String)

class SimpleParser {

  val parserFactory = new JsonFactory()

  def parse(str: String) = {
    val parser = parserFactory.createJsonParser(str)
    if (parser.nextToken() == START_OBJECT) {
      var token = parser.nextToken()
      var textOpt:Option[String] = None
      var idOpt:Option[Long] = None
      while(token != null) {
        if (token == FIELD_NAME) {
          parser.getCurrentName() match {
            case "text" => {
              parser.nextToken()
              textOpt = Some(parser.getText())
            }
            case "id" => {
              parser.nextToken()
              idOpt = Some(parser.getLongValue())
            }
            case _ => // noop
          }
        }
        token = parser.nextToken()
      }
      if (textOpt.isDefined && idOpt.isDefined) {
        Some(SimpleParsed(idOpt.get, textOpt.get))
      } else {
        None
      }
    } else {
      None
    }
  }
}

This is a simple Jackson parser. W hen we save, SBT recompiles the code and runs the tests. The code is getting better and better!

info] SimpleParser should
[info]   + work with basic tweet
[info]   + reject a non-JSON tweet
[info]   x ignore nested content
[info]     '2' is not equal to '1' (SimpleParserSpec.scala:32)
[info]   + fail on partial content
[info] == com.twitter.sample.SimpleParserSpec ==

Oh. W e need to check the nested objects. Let's add some ugly guards at the token read loop.

  def parse(str: String) = {
    val parser = parserFactory.createJsonParser(str)
    var nested = 0
    if (parser.nextToken() == START_OBJECT) {
      var token = parser.nextToken()
      var textOpt:Option[String] = None
      var idOpt:Option[Long] = None
      while(token != null) {
        if (token == FIELD_NAME && nested == 0) {
          parser.getCurrentName() match {
            case "text" => {
              parser.nextToken()
              textOpt = Some(parser.getText())
            }
            case "id" => {
              parser.nextToken()
              idOpt = Some(parser.getLongValue())
            }
            case _ => // noop
          }
        } else if (token == START_OBJECT) {
          nested += 1
        } else if (token == END_OBJECT) {
          nested -= 1
        }
        token = parser.nextToken()
      }
      if (textOpt.isDefined && idOpt.isDefined) {
        Some(SimpleParsed(idOpt.get, textOpt.get))
      } else {
        None
      }
    } else {
      None
    }
  }

... The test passed!

Package and publish

Now we can run the package command to generate a jar file. H owever, we may want to share our jar pack with other groups. To do this, we'll build on StandardProject, which gives us a good start.

The first step is to introduce StandardProject as an SBT plug-in. P lug-ins are a way to introduce dependencies to your build, not to your project. T hese dependencies are defined project/plugins/Plugins.scala file. Add the following code to the Plugins.scala file.

import sbt._

class Plugins(info: ProjectInfo) extends PluginDefinition(info) {
  val twitterMaven = "twitter.com" at "http://maven.twttr.com/"
  val defaultProject = "com.twitter" % "standard-project" % "0.7.14"
}

Note that we have specified a Maven repository and a dependency. This is because this standard project library is hosted by twitter and is not in the repository that SBT checks by default.

We will also update the project definition to extend StandardProject, including the SVN release feature, and the warehouse definition we want to publish. Modify SampleProject.scala

import sbt._
import com.twitter.sbt._

class SampleProject(info: ProjectInfo) extends StandardProject(info) with SubversionPublisher {
  val jackson = "org.codehaus.jackson" % "jackson-core-asl" % "1.6.1"
  val specs = "org.scala-tools.testing" % "specs_2.8.0" % "1.6.5" % "test"

  override def subversionRepository = Some("http://svn.local.twitter.com/maven/")
}

Now if we run the publish operation, we'll see the following output

[info] == deliver ==
IvySvn Build-Version: null
IvySvn Build-DateTime: null
[info] :: delivering :: com.twitter#sample;1.0-SNAPSHOT :: 1.0-SNAPSHOT :: release :: Wed Nov 24 10:26:45 PST 2010
[info]  delivering ivy file to /Users/mmcbride/projects/sbt-sample/target/ivy-1.0-SNAPSHOT.xml
[info] == deliver ==
[info] 
[info] == make-pom ==
[info] Wrote /Users/mmcbride/projects/sbt-sample/target/sample-1.0-SNAPSHOT.pom
[info] == make-pom ==
[info] 
[info] == publish ==
[info] :: publishing :: com.twitter#sample
[info] Scheduling publish to http://svn.local.twitter.com/maven/com/twitter/sample/1.0-SNAPSHOT/sample-1.0-SNAPSHOT.jar
[info]  published sample to com/twitter/sample/1.0-SNAPSHOT/sample-1.0-SNAPSHOT.jar
[info] Scheduling publish to http://svn.local.twitter.com/maven/com/twitter/sample/1.0-SNAPSHOT/sample-1.0-SNAPSHOT.pom
[info]  published sample to com/twitter/sample/1.0-SNAPSHOT/sample-1.0-SNAPSHOT.pom
[info] Scheduling publish to http://svn.local.twitter.com/maven/com/twitter/sample/1.0-SNAPSHOT/ivy-1.0-SNAPSHOT.xml
[info]  published ivy to com/twitter/sample/1.0-SNAPSHOT/ivy-1.0-SNAPSHOT.xml
[info] Binary diff deleting com/twitter/sample/1.0-SNAPSHOT
[info] Commit finished r977 by 'mmcbride' at Wed Nov 24 10:26:47 PST 2010
[info] Copying from com/twitter/sample/.upload to com/twitter/sample/1.0-SNAPSHOT
[info] Binary diff finished : r978 by 'mmcbride' at Wed Nov 24 10:26:47 PST 2010
[info] == publish ==
[success] Successful.
[info] 
[info] Total time: 4 s, completed Nov 24, 2010 10:26:47 AM

This way (after a while), you can binaries.local.twitter.com the jar package we published on the server.

Add a task

The task is the Scala function. The easiest way to add a task is to introduce a val-defined task method into your project definition, such as

lazy val print = task {log.info("a test action"); None}

You can also add dependency and description in this way

lazy val print = task {log.info("a test action"); None}.dependsOn(compile) describedAs("prints a line after compile")

Refresh the project and perform a print operation, and we'll see the following output

> print
[info] 
[info] == print ==
[info] a test action
[info] == print ==
[success] Successful.
[info] 
[info] Total time: 0 s, completed Nov 24, 2010 11:05:12 AM
>

So it worked. T his works well if you're just defining a task in a project. H owever, if you define a plug-in, it is inflexible. I might want to

lazy val print = printAction
def printAction = printTask.dependsOn(compile) describedAs("prints a line after compile")
def printTask = task {log.info("a test action"); None}

This allows the consumer to override the task itself, the dependency and/or the description of the task, or the action itself. M ost SBT built-in actions follow this pattern. As an example, we can print the current timestamp by modifying the built-in packaging task

lazy val printTimestamp = task { log.info("current time is " + System.currentTimeMillis); None}
override def packageAction = super.packageAction.dependsOn(printTimestamp)

There are many examples of how to adjust SBT's default StandardProject and how to add custom tasks.

Quick reference

Common commands

Action - Shows the actions available in this item
Update - Download dependency
Compile - Compile the source file
test - Run the test
package - Create a publishable jar file
publish-local - Install the built jar package in the local ivy cache
Publish - Push your jar into a remote library (if configured)

More commands

test-failed - Run all failed specification tests
test-quick - Runs any failed and/or dependent update specifications
clean-cache - Removes all kinds of things from the SBT cache. Just like sbt's clean command
clean-lib - lib_managed everything under the folder

Simple build tools

Table of contents