May 14, 2021 Scala
SBT is a modern build tool. Although it was written by Scala and provided a lot of Scala convenience, it is a common build tool.
Why SBT?
Refer to scala-sbt's documentation for the latest SBT installation methods
java -Xmx512M -jar sbt-launch.jar "$@"
[local ~/projects]$ sbt
Project does not exist, create new project? (y/N/s) y
Name: sample
Organization: com.twitter
Version [1.0]: 1.0-SNAPSHOT
Scala version [2.7.7]: 2.8.1
sbt version [0.7.4]:
Getting Scala 2.7.7 ...
:: retrieving :: org.scala-tools.sbt#boot-scala
confs: [default]
2 artifacts copied, 0 already retrieved (9911kB/221ms)
Getting org.scala-tools.sbt sbt_2.7.7 0.7.4 ...
:: retrieving :: org.scala-tools.sbt#boot-app
confs: [default]
15 artifacts copied, 0 already retrieved (4096kB/167ms)
[success] Successfully initialized directory structure.
Getting Scala 2.8.1 ...
:: retrieving :: org.scala-tools.sbt#boot-scala
confs: [default]
2 artifacts copied, 0 already retrieved (15118kB/386ms)
[info] Building project sample 1.0-SNAPSHOT against Scala 2.8.1
[info] using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7
You can see that it has created a snapshot version of the project in a better form.
project/build/.scala
main project definition file
project/build.properties
Project, sbt, and Scala version definitions
src/main
Your application code appears here, indicating the language of the code in the subdirector
src/main/scala
src/main/java
src/main/resources
files you want to add to the jar package (e.g. log configuration)
src/test
like src/main, just for testing
lib_managed
jar files on which your project depends.
Filled by sbt update
target
- the target path of the generator (e.g. automatically generated thrift code, class files, jar packs)
We'll create a simple JSON parser for a simple tweet message.
Add the following code to
src/main/scala/com/twitter/sample/SimpleParser.scala
package com.twitter.sample
case class SimpleParsed(id: Long, text: String)
class SimpleParser {
val tweetRegex = "\"id\":(.*),\"text\":\"(.*)\"".r
def parse(str: String) = {
tweetRegex.findFirstMatchIn(str) match {
case Some(m) => {
val id = str.substring(m.start(1), m.end(1)).toInt
val text = str.substring(m.start(2), m.end(2))
Some(SimpleParsed(id, text))
}
case _ => None
}
}
}
This code is ugly and has bugs, but should be able to compile through.
SBT can be used neither as a command-line script nor as a build console. We'll primarily use it as a build console, but most commands can be passed as parameters to SBT to run independently, for example
sbt test
Note That if a command requires parameters, you need to use quotation marks to include the entire parameter path, for example
sbt 'test-only com.twitter.sample.SampleSpec'
It's a strange way.
Anyway, to get our code working, start SBT
[local ~/projects/sbt-sample]$ sbt
[info] Building project sample 1.0-SNAPSHOT against Scala 2.8.1
[info] using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7
>
SBT allows you to start a Scala REPL and load all project dependencies. It compiles the source code for the project before starting the console, giving us a workstable for a quick test parser.
> console
[info]
[info] == compile ==
[info] Source analysis: 0 new/modified, 0 indirectly invalidated, 0 removed.
[info] Compiling main sources...
[info] Nothing to compile.
[info] Post-analysis: 3 classes.
[info] == compile ==
[info]
[info] == copy-test-resources ==
[info] == copy-test-resources ==
[info]
[info] == test-compile ==
[info] Source analysis: 0 new/modified, 0 indirectly invalidated, 0 removed.
[info] Compiling test sources...
[info] Nothing to compile.
[info] Post-analysis: 0 classes.
[info] == test-compile ==
[info]
[info] == copy-resources ==
[info] == copy-resources ==
[info]
[info] == console ==
[info] Starting scala interpreter...
[info]
Welcome to Scala version 2.8.1.final (Java HotSpot(TM) 64-Bit Server VM, Java 1.6.0_22).
Type in expressions to have them evaluated.
Type :help for more information.
scala>
Our code compiles through and provides a typical Scala prompt. We'll create a new parser, a tweet to make sure it "works"
scala> import com.twitter.sample._
import com.twitter.sample._
scala> val tweet = """{"id":1,"text":"foo"}"""
tweet: java.lang.String = {"id":1,"text":"foo"}
scala> val parser = new SimpleParser
parser: com.twitter.sample.SimpleParser = com.twitter.sample.SimpleParser@71060c3e
scala> parser.parse(tweet)
res0: Option[com.twitter.sample.SimpleParsed] = Some(SimpleParsed(1,"foo"}))
scala>
Our simple parser works fine with this very small input set, but we need to add more tests and make it wrong. T he first step is to add the specs test library and a real JSON parser to our project. To do this, we must go beyond the default SBT project layout to create a project.
SBT considers the Scala file in the
project/build
directory to be a project definition.
Add the following to this file
project/build/SampleProject.scala
import sbt._
class SampleProject(info: ProjectInfo) extends DefaultProject(info) {
val jackson = "org.codehaus.jackson" % "jackson-core-asl" % "1.6.1"
val specs = "org.scala-tools.testing" % "specs_2.8.0" % "1.6.5" % "test"
}
A project definition is an SBT class. In the example above, we extended defaultProject for SBT.
Here is the dependency declared by val. S BT uses reflection to scan all val dependencies in a project and establish a dependency tree at build time. The syntax used here may be new, but the nature and Maven dependency are the same
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>org.scala-tools.testing</groupId>
<artifactId>specs_2.8.0</artifactId>
<version>1.6.5</version>
<scope>test</scope>
</dependency>
We can now download our project dependent. Run sbt update on the command line instead of sbt console
[local ~/projects/sbt-sample]$ sbt update
[info] Building project sample 1.0-SNAPSHOT against Scala 2.8.1
[info] using SampleProject with sbt 0.7.4 and Scala 2.7.7
[info]
[info] == update ==
[info] :: retrieving :: com.twitter#sample_2.8.1 [sync]
[info] confs: [compile, runtime, test, provided, system, optional, sources, javadoc]
[info] 1 artifacts copied, 0 already retrieved (2785kB/71ms)
[info] == update ==
[success] Successful.
[info]
[info] Total time: 1 s, completed Nov 24, 2010 8:47:26 AM
[info]
[info] Total session time: 2 s, completed Nov 24, 2010 8:47:26 AM
[success] Build completed successfully.
You'll see that sbt retrieves the specs library.
A lib_managed directory has also been added and
lib_managed/scala_2.8.1/test
is included in
specs_2.8.0-1.6.5.jar
Now that you have a test library, you can write the following test
src/test/scala/com/twitter/sample/SimpleParserSpec.scala
file
package com.twitter.sample
import org.specs._
object SimpleParserSpec extends Specification {
"SimpleParser" should {
val parser = new SimpleParser()
"work with basic tweet" in {
val tweet = """{"id":1,"text":"foo"}"""
parser.parse(tweet) match {
case Some(parsed) => {
parsed.text must be_==("foo")
parsed.id must be_==(1)
}
case _ => fail("didn't parse tweet")
}
}
}
}
Run test in the SBT console
> test
[info]
[info] == compile ==
[info] Source analysis: 0 new/modified, 0 indirectly invalidated, 0 removed.
[info] Compiling main sources...
[info] Nothing to compile.
[info] Post-analysis: 3 classes.
[info] == compile ==
[info]
[info] == test-compile ==
[info] Source analysis: 0 new/modified, 0 indirectly invalidated, 0 removed.
[info] Compiling test sources...
[info] Nothing to compile.
[info] Post-analysis: 10 classes.
[info] == test-compile ==
[info]
[info] == copy-test-resources ==
[info] == copy-test-resources ==
[info]
[info] == copy-resources ==
[info] == copy-resources ==
[info]
[info] == test-start ==
[info] == test-start ==
[info]
[info] == com.twitter.sample.SimpleParserSpec ==
[info] SimpleParserSpec
[info] SimpleParser should
[info] + work with basic tweet
[info] == com.twitter.sample.SimpleParserSpec ==
[info]
[info] == test-complete ==
[info] == test-complete ==
[info]
[info] == test-finish ==
[info] Passed: : Total 1, Failed 0, Errors 0, Passed 1, Skipped 0
[info]
[info] All tests PASSED.
[info] == test-finish ==
[info]
[info] == test-cleanup ==
[info] == test-cleanup ==
[info]
[info] == test ==
[info] == test ==
[success] Successful.
[info]
[info] Total time: 0 s, completed Nov 24, 2010 8:54:45 AM
>
Our test passed! N
ow we can add more. R
unning trigger actions is one of the best features that SBT provides. A
dding a wavy line at the beginning of the action starts a loop and re-runs the action as the source file changes.
Let's
~test
and see what happens.
[info] == test ==
[success] Successful.
[info]
[info] Total time: 0 s, completed Nov 24, 2010 8:55:50 AM
1. Waiting for source changes... (press enter to interrupt)
Now, let's add the following test case
"reject a non-JSON tweet" in {
val tweet = """"id":1,"text":"foo""""
parser.parse(tweet) match {
case Some(parsed) => fail("didn't reject a non-JSON tweet")
case e => e must be_==(None)
}
}
"ignore nested content" in {
val tweet = """{"id":1,"text":"foo","nested":{"id":2}}"""
parser.parse(tweet) match {
case Some(parsed) => {
parsed.text must be_==("foo")
parsed.id must be_==(1)
}
case _ => fail("didn't parse tweet")
}
}
"fail on partial content" in {
val tweet = """{"id":1}"""
parser.parse(tweet) match {
case Some(parsed) => fail("didn't reject a partial tweet")
case e => e must be_==(None)
}
}
After we save the file, SBT detects a change, runs the test, and notifies us that there is a problem with the parser
[info] == com.twitter.sample.SimpleParserSpec ==
[info] SimpleParserSpec
[info] SimpleParser should
[info] + work with basic tweet
[info] x reject a non-JSON tweet
[info] didn't reject a non-JSON tweet (Specification.scala:43)
[info] x ignore nested content
[info] 'foo","nested":{"id' is not equal to 'foo' (SimpleParserSpec.scala:31)
[info] + fail on partial content
So let's rework the real JSON parser
package com.twitter.sample
import org.codehaus.jackson._
import org.codehaus.jackson.JsonToken._
case class SimpleParsed(id: Long, text: String)
class SimpleParser {
val parserFactory = new JsonFactory()
def parse(str: String) = {
val parser = parserFactory.createJsonParser(str)
if (parser.nextToken() == START_OBJECT) {
var token = parser.nextToken()
var textOpt:Option[String] = None
var idOpt:Option[Long] = None
while(token != null) {
if (token == FIELD_NAME) {
parser.getCurrentName() match {
case "text" => {
parser.nextToken()
textOpt = Some(parser.getText())
}
case "id" => {
parser.nextToken()
idOpt = Some(parser.getLongValue())
}
case _ => // noop
}
}
token = parser.nextToken()
}
if (textOpt.isDefined && idOpt.isDefined) {
Some(SimpleParsed(idOpt.get, textOpt.get))
} else {
None
}
} else {
None
}
}
}
This is a simple Jackson parser. W hen we save, SBT recompiles the code and runs the tests. The code is getting better and better!
info] SimpleParser should
[info] + work with basic tweet
[info] + reject a non-JSON tweet
[info] x ignore nested content
[info] '2' is not equal to '1' (SimpleParserSpec.scala:32)
[info] + fail on partial content
[info] == com.twitter.sample.SimpleParserSpec ==
Oh. W e need to check the nested objects. Let's add some ugly guards at the token read loop.
def parse(str: String) = {
val parser = parserFactory.createJsonParser(str)
var nested = 0
if (parser.nextToken() == START_OBJECT) {
var token = parser.nextToken()
var textOpt:Option[String] = None
var idOpt:Option[Long] = None
while(token != null) {
if (token == FIELD_NAME && nested == 0) {
parser.getCurrentName() match {
case "text" => {
parser.nextToken()
textOpt = Some(parser.getText())
}
case "id" => {
parser.nextToken()
idOpt = Some(parser.getLongValue())
}
case _ => // noop
}
} else if (token == START_OBJECT) {
nested += 1
} else if (token == END_OBJECT) {
nested -= 1
}
token = parser.nextToken()
}
if (textOpt.isDefined && idOpt.isDefined) {
Some(SimpleParsed(idOpt.get, textOpt.get))
} else {
None
}
} else {
None
}
}
... The test passed!
Now we can run the package command to generate a jar file. H owever, we may want to share our jar pack with other groups. To do this, we'll build on StandardProject, which gives us a good start.
The first step is to introduce StandardProject as an SBT plug-in. P
lug-ins are a way to introduce dependencies to your build, not to your project. T
hese dependencies are defined
project/plugins/Plugins.scala
file.
Add the following code to the Plugins.scala file.
import sbt._
class Plugins(info: ProjectInfo) extends PluginDefinition(info) {
val twitterMaven = "twitter.com" at "http://maven.twttr.com/"
val defaultProject = "com.twitter" % "standard-project" % "0.7.14"
}
Note that we have specified a Maven repository and a dependency. This is because this standard project library is hosted by twitter and is not in the repository that SBT checks by default.
We will also update the project definition to extend StandardProject, including the SVN release feature, and the warehouse definition we want to publish.
Modify
SampleProject.scala
import sbt._
import com.twitter.sbt._
class SampleProject(info: ProjectInfo) extends StandardProject(info) with SubversionPublisher {
val jackson = "org.codehaus.jackson" % "jackson-core-asl" % "1.6.1"
val specs = "org.scala-tools.testing" % "specs_2.8.0" % "1.6.5" % "test"
override def subversionRepository = Some("http://svn.local.twitter.com/maven/")
}
Now if we run the publish operation, we'll see the following output
[info] == deliver ==
IvySvn Build-Version: null
IvySvn Build-DateTime: null
[info] :: delivering :: com.twitter#sample;1.0-SNAPSHOT :: 1.0-SNAPSHOT :: release :: Wed Nov 24 10:26:45 PST 2010
[info] delivering ivy file to /Users/mmcbride/projects/sbt-sample/target/ivy-1.0-SNAPSHOT.xml
[info] == deliver ==
[info]
[info] == make-pom ==
[info] Wrote /Users/mmcbride/projects/sbt-sample/target/sample-1.0-SNAPSHOT.pom
[info] == make-pom ==
[info]
[info] == publish ==
[info] :: publishing :: com.twitter#sample
[info] Scheduling publish to http://svn.local.twitter.com/maven/com/twitter/sample/1.0-SNAPSHOT/sample-1.0-SNAPSHOT.jar
[info] published sample to com/twitter/sample/1.0-SNAPSHOT/sample-1.0-SNAPSHOT.jar
[info] Scheduling publish to http://svn.local.twitter.com/maven/com/twitter/sample/1.0-SNAPSHOT/sample-1.0-SNAPSHOT.pom
[info] published sample to com/twitter/sample/1.0-SNAPSHOT/sample-1.0-SNAPSHOT.pom
[info] Scheduling publish to http://svn.local.twitter.com/maven/com/twitter/sample/1.0-SNAPSHOT/ivy-1.0-SNAPSHOT.xml
[info] published ivy to com/twitter/sample/1.0-SNAPSHOT/ivy-1.0-SNAPSHOT.xml
[info] Binary diff deleting com/twitter/sample/1.0-SNAPSHOT
[info] Commit finished r977 by 'mmcbride' at Wed Nov 24 10:26:47 PST 2010
[info] Copying from com/twitter/sample/.upload to com/twitter/sample/1.0-SNAPSHOT
[info] Binary diff finished : r978 by 'mmcbride' at Wed Nov 24 10:26:47 PST 2010
[info] == publish ==
[success] Successful.
[info]
[info] Total time: 4 s, completed Nov 24, 2010 10:26:47 AM
This way (after a while), you can
binaries.local.twitter.com
the jar package we published on the server.
The task is the Scala function. The easiest way to add a task is to introduce a val-defined task method into your project definition, such as
lazy val print = task {log.info("a test action"); None}
You can also add dependency and description in this way
lazy val print = task {log.info("a test action"); None}.dependsOn(compile) describedAs("prints a line after compile")
Refresh the project and perform a print operation, and we'll see the following output
> print
[info]
[info] == print ==
[info] a test action
[info] == print ==
[success] Successful.
[info]
[info] Total time: 0 s, completed Nov 24, 2010 11:05:12 AM
>
So it worked. T his works well if you're just defining a task in a project. H owever, if you define a plug-in, it is inflexible. I might want to
lazy val print = printAction
def printAction = printTask.dependsOn(compile) describedAs("prints a line after compile")
def printTask = task {log.info("a test action"); None}
This allows the consumer to override the task itself, the dependency and/or the description of the task, or the action itself. M ost SBT built-in actions follow this pattern. As an example, we can print the current timestamp by modifying the built-in packaging task
lazy val printTimestamp = task { log.info("current time is " + System.currentTimeMillis); None}
override def packageAction = super.packageAction.dependsOn(printTimestamp)
There are many examples of how to adjust SBT's default StandardProject and how to add custom tasks.
Common commands
More commands