Gitlab CI caching for sbt projects

Compiling and testing sbt projects on hosted CI platfroms like Travis and Gitlab CI is dominated by downloading the required dependencies from Maven Central.

Oftentimes you have four to five minutes of downloading and less than one minute of compiling and testing.

Some offer a caching feature which is supposed to speed up these always-the-same download tasks by caching the dependencies in an S3 bucket.

Caching dependencies for Scala in Travis is relatively straight forward however I couldn’t find an example to do this for Gitlab CI. After a lot of trial and error I figured it out.

Here is the .gitlab-ci.yml:

# some parts originally from https://github.com/randm-ch/units-of-information/blob/master/.gitlab-ci.yml

image: "hseeberger/scala-sbt"

variables:
  SBT_VERSION: "0.13.9"
  SBT_OPTS: "-Dsbt.global.base=sbt-cache/.sbtboot -Dsbt.boot.directory=sbt-cache/.boot -Dsbt.ivy.home=sbt-cache/.ivy"

cache:
  key: "$CI_BUILD_REF_NAME" # contains either the branch or the tag, so it's caching per branch
  untracked: true
  paths:
    - "sbt-cache/cache/.ivy"
    - "sbt-cache/.boot"
    - "sbt-cache/.sbtboot"
    - "sbt-cache/target"

stages:
  - test

test:
  script:
    - sbt test

Sometimes it takes a few times for it to actually work so just commit it and try it a few times.

Comments »


An in depth guide to deploying to Maven Central with sbt

So, you’ve written a library or program in Java or Scala (or any other JVM language) and want to deploy it so that others can use it without configuring extra repos? Just deploy it to Maven Central! This repository of code artifacts and their metadata is the place to publish your code as virtually all JVM build systems pull their dependencies from there.

I published my first artifact to Maven Central with Maven (the tool, not the repository format) in 2011 and back then it was an extremely complicated process.

In 2016 I did it again with sbt and mvn and was pleasantly surprised about how much the process had been streamlined.

In this blog post I’m going into detail about how to deploy an artifact with sbt.

Sonatype

Only very few projects deploy to Maven Central directly. Most small projects these days deploy to Maven Central via the Sonatype Repository. This in turn then syncs with Maven Central.

There is an official guide on how to do that with sbt but we are going to use a couple of plugins so we don’t have to use the confusing UI of the Sonatype Repository. (At least it was confusing when I used it last in ~2011.)

sbt setup

Add the following plugins to your project/plugins.sbt:

addSbtPlugin("org.xerial.sbt" % "sbt-sonatype" % "1.1")

addSbtPlugin("com.jsuereth" % "sbt-pgp" % "1.0.0")

and the following keys to your project settings in your main build.sbt:

// POM settings for Sonatype
homepage := Some(url("https://github.com/username/projectname")),
scmInfo := Some(ScmInfo(url("https://github.com/username/projectname"),
                            "git@github.com:username/projectname.git")),
developers += Developer("username",
                        "User Name",
                        "mail@username.de",
                        url("https://github.com/username")),
licenses += ("Apache-2.0", url("http://www.apache.org/licenses/LICENSE-2.0")),
pomIncludeRepository := (_ => false),

Obviously replace the placeholder values appropriately.

GPG keys

In order to be on Maven Central your artifacts have to be signed with PGP. If you already have a PGP key, great! If not create one with:

sbt pgp-cmd gen-key

Then deploy this key to a key server

pgp-cmd send-key ${keyname} hkp://pool.sks-keyservers.net

There are a few key servers around the internet and every few hours they syncronise with each other so eventually all keys will be on all servers. Sonatype and Maven/Ivy clients will validate the signature on the artifact with your public key to make sure it is exactly what you have published.

Sonatype account

You need to make an account at the Sonatype repo: https://issues.sonatype.org/secure/Signup!default.jspa

They usually takes a few hours to respond and will ask you if you own the domain of your group id. If you don’t then maybe use the domain of the provider where you host the code such as com.github.username as your group id.

After a while the ticket will be closed and you’ll be told that you now can publish into your group id.

Configure credentials

Create a file $HOME/.sbt/(sbt-version)/sonatype.sbt

Set Sonatype account information (user name and password) in the global sbt settings. To protect your password, never commit this file to your SCM.

credentials += Credentials("Sonatype Nexus Repository Manager",
        "oss.sonatype.org",
        "(Sonatype user name)",
        "(Sonatype password)")

Publishing

Once you have the permission to upload you can simply run

sbt publishSigned

This will sign and upload your artifact to Sonatype’s staging repository. If you want you can test this release first by adding the repo to your build.sbt.

If you’re happy with the release and want to push to Maven Central run

sbt sonatypeRelease

This promotes the release to be ready for synching to Maven Central. It usually takes a few hours before it will show up in http://search.maven.org/ but the actual sync job runs every hour or so.

Now head over to the Maven search to check out your artifact.

Comments »


First impressions of ScalaJS

My last couple of projects have mostly been written in Scala and I’ve really started too love the language as I became better and better at it. Once you’ve fully experienced the joy of proper type safety, you can’t go back.

However, these days Scala is confined mostly to the server. Since my current contract also involves a rather large frontend component I also occassionally have to write JavaScript. Don’t get me wrong, I’m not some hapless backend code slinger dipping his toes into the browser world. Years ago I used the language a lot and I’m quite competent at it. However after writing a lot of Scala, JS’s weak types really grind on you: they make refactoring really difficult and you discover many problems, which a compiler would tell you much earlier, at runtime. :/

Help is at hand: EPFL (where Martin Odersky is a professor) is developing ScalaJS, which compiles Scala to JavaScript. This weekend I took it for a spin and here are my observations. (Mind you, this isn’t a thorough review but rather a rough-and-ready stream of consciousness.)

Nice integration with sbt

You can keep using your build system and IDE. Everything worked out-of-the-box.

Compiler

The compilation process is not slower than the compiler that targets the JVM. However, the Scala compiler is quite slow but you get used to it because it really helps you during development. Those sweet, sweet types are great!

The compiler produces unreadable code which will make debugging harder. I’ve read that source maps are available, but I haven’t been able to make them work yet. Source maps do work and can help you debugging!

Community

It’s rather small, but that shouldn’t surprise you since ScalaJS has only come out of beta a few weeks ago. You should expect to write stuff yourself because it doesn’t exist yet.

Libraries and frameworks

This is currently a weak spot. Obviously, you can use the fantastic Scala core library which to me is one of the main appeals. I haven’t found out how easy it is to get third party libraries like, say, Joda-Time working.

On the DOM side, there is the nice scalatags which makes writing HTML typesafe but other than there isn’t all that much around.

The community seems to be split between people wanting to wrap native JS frameworks like React and others wanting to write stuff from scratch in pure Scala.

I think we will see a lot of experimentation of approaches and many new frameworks popping up and eventually dieing. This isn’t something that worries me a lot since this is the way that really make a community thrive. Yes, it does cause fragmentation to a certain degree but I feel that it’s a price worth paying for finding the best ideas and practises.

File size

In short: it’s acceptable. With fast optimisation my dummy app clocked in at ~600kb. A full optimisation brought that down to around 140kb. Really large apps probably will be a few MB, but that’s not something unheard off with apps written in native JS, too.

In summary

I’m very excited about ScalaJS and I think it’s very promising. It gets a lot of things right and people are working on the things that aren’t so great yet.

I might sound like a fanboy, but most things coming out of the Scala ecosystem these days are very high quality and this is no exception.

If you can tolerate a little technological immaturity and don’t mind being an early adopter, I would seriously consider ScalaJS for you next frontend project.

Comments »


Setting sbt memory options

If you’ve been working on a bigger Scala project you will probably have experienced that SBT consumers a rather large amount of memory and can crash with a OutOfMemoryError Metaspace. There is a lot of advise floating around the web that tells you to set your memory options in $SBT_OPTS.

I learned that this is only partially true.

What I found out about sbt-launcher version 0.13.8 (the most recent version at the time of writing) is that setting Xmx and friends in $SBT_OPTS does indeed result in them being added to the command line arguments but then they are overwritten.

The resulting command line invocation looks like this:

/usr/bin/java -XX:+CMSClassUnloadingEnabled -Xmx1500M -XX:MaxMetaspaceSize=512m -Xms1024m -Xmx1024m -XX:ReservedCodeCacheSize=128m -XX:MaxMetaspaceSize=256m -jar /usr/local/Cellar/sbt/0.13.8/libexec/sbt-launch.jar

You can see that Xmx is set twice because the sbt launcher script blindly prepends the contents of $SBT_OPTS to the incovation.

If you put the setting into $JAVA_OPTS then the sbt launcher intelligently parses them but I wanted the memory settings to apply to sbt only and not other Java applications.

The solution I came up with in the end is to use the -mem paramter of sbt directly.

I’ve set the following alias

alias sbt="sbt -mem 1500"

This sets the heap space to 1.5 GB and the MaxMetaspace to 512 MB.

Comments »


Make Maven treat warning as errors

I like compiler warnings and I think a nicely linted code base really improves code hygiene and keeps standards high. Rather than relying on some post-compile script or a tool like Sonar I prefer the compiler to throw an error when it encouters something that we defined as a code smell. In other words, I want it to treat warnings as errors.

javac can do exactly that with the -Werror flag but using it in Maven took me a little while to find out. To enable it in your Maven builds, use the following XML:

...
<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-compiler-plugin</artifactId>
      <version>3.1</version>
      <configuration>
        <source>1.7</source>
        <target>1.7</target>
        <compilerArguments>
          <Werror />
          <Xlint:all />
        </compilerArguments>
      </configuration>
    </plugin>
  </plugins>
</build>
...

Comments »