Leonard Ehrenfried’s notebook

Itron Openway mit Volkszähler

2020-08-31T00:00:00+02:00

Vor einer Weile hat mich mein Vater gebeten ihm zu helfen herauszufinden wie viel Strom seiner neuen Photovoltaikanlage er an seinen Energieanbieter verschenkt, wenn er sie ins Netz einspeist.

Das hat mich in die Welt der Home Automation eingeführt und ich will meine Ergebnisse hier ein wenig dokumentieren.

Hardware

Mein Vater hat von der EnBW einen Zähler bekommen, der auf der Vorderseite als “Itron Openway 3Hz” bezeichnet wird. Leider konnte ich online recht wenig über dieses Modell finden und vermute das ist irgendeine Sonderedition.

Software

Manuel Wortmann hat einen sehr guten Post über sein Setup geschrieben und ich bin dieser Anleitung gefolgt.

Hier die Kurzversion:

Weidmann Optokopf kaufen
Raspberry Pi kaufen
Volkszähler-Image auf RasPi installieren
D0-Schnittstelle (optische Datenausgabe) am Zähler aktivieren
vzlogger konfigurieren

Datenausgabe über optische Schnittstelle

Um Daten aus dem Itron auszulesen muss die optische Schnittstelle mittels Taschenlampe und PIN-Eingabe aktiviert werden. Ziel ist es das Gerät in den “Info On” Modus, was dann auch im Display steht, zu setzen. Das ist eine sehr fummelige Angelegenheit und ist im Handbuch des Herstellers und in der des Weidmann Optokopfes beschrieben.

Eigenheiten

Um den den Channel anzulegen, habe ich auch nicht den MySQL-Client benutzt, sondern die Volkszähler-UI. Hierbei ist es wichtig als Typen “El. Energie (Leistungswerte))” auszuwählen.

Die Konfiguration des vzlogger für den Itron Openway ist ein wenig anders als beim von Manuel beschriebenen Modell.

Die Datenausgabe des Stromzählers muss nicht initialisiert werden. Wenn das Gerät auf “Info On”-Modus ist, werden die Lichtumpulse alle Sekunde ausgegeben.
Die Daten sind im sml-Protokoll kodiert.

Das Gerät liefert zwei Werte pro Datenpaket:

den klassischen Zählerstand, also eine immerzu steigende Zahl
den aktuellen Verbrauch

Mein Vater war an letzerem interessiert und um diese Daten via vzlogger an die Volkszähler-Middleware zu schicken, habe ich folgende vzlogger.conf-Datei benutzt:

{
  "retry": 0,
  "daemon": true,
  "verbosity": 0,
  "log": "/var/log/vzlogger.log",
  "local": {
    "enabled": false,
    "port": 8080,
    "index": true,
    "timeout": 0,
    "buffer": 0
  },
  "meters": [
    {
      "enabled": true,
      "device": "/dev/ttyUSB0",
      "protocol": "sml",
      "aggtime": -1,
      "aggfixedinterval": false,
      "baudrate": 9600,
      "parity": "8n1",
      "duplicates": 3600,
      "channels": [
        {
          "uuid": "${von der UI generierte UUID}",
          "middleware": "http://localhost/middleware.php",
          "identifier": "1-0:16.7.0",
          "aggmode": "none"
        }
      ]
    }
  ]
}

In dieser Konfiguration wird der Zählerstand nicht weiterverabeitet. Solltest Du an diesem interessiert sein, dann erhöhe die verbosity auf 15 und schau’ Dir die Werte und deren ID im Log an. Dafür könntest Du einen zweiten Channel anlegen und auch diese Werte aufzeichnen.

Foto des fertigen Setups

A reproducible benchmark for sbt

2017-11-12T00:00:00+01:00

A few weeks ago sbt version 1.0.0 has been released. One of the big ticket items in this release has been an improved incremental compiler that needs to compile less and therefore is faster. Slow builds are a common complaint for Scala developers so this generated a lot of excitement.

In reality however, build times in version 1.0.0 have not seen any improvement but rather a significant slow down. While the incremental compiler itself is faster, other areas of sbt have seen a sudden worseing of performance. This has been reflected in a flurry of Github tickets and discussions on the Gitter channel.

The sbt core developers have been understandably reluctant to merge in spurious pull requests by the community without evidence that they really improve performance beyond a simple micro benchmark.

The Scala compiler, for example, has a solid performance benchmark suite that is run on every commit and gives immediate feedback about proposed changes.

Something along these lines is needed for sbt, too.

Since there doesn’t seem to be anything around yet, I took upon me to write one.

Methodology

I took an sbt project provided by Sam Halliday that had pathologically bad performance.

The problem with this project is its very large classpath and many submodules - it contains almost no source code itself! Before compilation even starts a large amount of time is spent on checking if classpath entries have changed.

Around this test repository wrote a little bit of tooling for executing and timing sbt commands and running them with various sbt versions.

The benchmark sets sbt up to download all the required dependencies before the actual task under benchmark in order to ensure fairness. It then benchmark tasks with various sbt versions and measures execution times.

Results

The timing results are as follows:

command	sbt 0.13.16	sbt 1.0.3
startup	37 seconds	45 seconds
compile once	89 seconds	129 seconds
compile twice	107 seconds	189 seconds

The results are clear: sbt 1.0.0 has seen a substantial slowdown - at least for this particular use case.

In case you’re wondering, these benchmarks were run on a 2015 Macbook Pro with a 2.9 GHz Intel Core i5 and 16 GB of RAM.

Future work

I’m planning on tracking and comparing the performance of future sbt versions. In particular, 1.0.4 is going to be released very soon and has some performance improvement patches.

The benchmark can also be extended with more test case to give insights into a wide range of performance scenarios. Contributions are very welcome!

Snake case keys in Circe

2017-11-10T00:00:00+01:00

How to combine Circe’s semi-automatic derivation with snake case keys took me a little while to find out.

You will need to add the circe module: "io.circe" %% "circe-generic-extras" % "0.8.0".

Here is the code:

import io.circe.generic.extras.Configuration
import io.circe.generic.extras.semiauto._
import io.circe.{Decoder, Encoder}

case class Snaky(someProperty: String)

object Snaky {
  implicit val customConfig: Configuration = Configuration.default.withSnakeCaseKeys.withDefaults
  implicit val snakyEncoder: Encoder[Snaky] = deriveEncoder
  implicit val snakyDecoder: Decoder[Snaky] = deriveDecoder
}

Running a task on startup in sbt 1.0

2017-10-27T00:00:00+02:00

If you want to run a task on startup in sbt 0.13 you’d probably do something like this:

onLoad in Global := (Command.process("project server", _: State)) compose (onLoad in Global).value

This changes into the the subproject server after sbt has loaded.

In sbt 1.0 Command.process has been removed and to achieve the same thing, you now have to do this:

onLoad in Global ~= (_ andThen ("project server" :: _))

Gitlab CI caching for sbt projects

2017-05-01T00:00:00+02:00

Compiling and testing sbt projects on hosted CI platfroms like Travis and Gitlab CI is dominated by downloading the required dependencies from Maven Central.

Oftentimes you have four to five minutes of downloading and less than one minute of compiling and testing.

Some offer a feature which is supposed to speed up these always-the-same download tasks by caching the dependencies in an S3 bucket.

Caching dependencies for Scala in Travis is relatively straight forward however I couldn’t find an example to do this for Gitlab CI. After a bit of trial and error I figured it out.

Here is the .gitlab-ci.yml:

# some parts originally from https://github.com/randm-ch/units-of-information/blob/master/.gitlab-ci.yml

image: "hseeberger/scala-sbt:8u141-jdk_2.12.3_1.0.2"

variables:
  SBT_OPTS: "-Dsbt.global.base=sbt-cache/sbtboot -Dsbt.boot.directory=sbt-cache/boot -Dsbt.ivy.home=sbt-cache/ivy"
  COURSIER_CACHE: sbt-cache/coursier #only needed if you use coursier


cache:
  # if you want to have a separate cache per branch, uncomment the next line
  # key: "$CI_BUILD_REF_NAME"
  untracked: true
  paths:
    - "sbt-cache/ivy/cache"
    - "sbt-cache/boot"
    - "sbt-cache/sbtboot"
    - "sbt-cache/target"
    - "sbt-cache/coursier" #only needed if you use coursier

stages:
  - test

test:
  script:
    - sbt test

Sometimes it takes a few times for it to actually work so just commit it and try it more than once.

Edit 2017-09-20: I updated the directories and the base image slightly.

An in depth guide to deploying to Maven Central with sbt

2017-01-15T00:00:00+01:00

So, you’ve written a library or program in Java or Scala (or any other JVM language) and want to deploy it so that others can use it without configuring extra repos? Just deploy it to Maven Central! This repository of code artifacts and their metadata is the place to publish your code as virtually all JVM build systems pull their dependencies from there.

I published my first artifact to Maven Central with Maven (the tool, not the repository format) in 2011 and back then it was an extremely complicated process.

In 2016 I did it again with sbt and mvn and was pleasantly surprised about how much the process had been streamlined.

In this blog post I’m going into detail about how to deploy an artifact with sbt.

Sonatype

Only very few projects deploy to Maven Central directly. Most small projects these days deploy to Maven Central via the Sonatype Repository. This in turn then syncs with Maven Central.

There is an official guide on how to do that with sbt but we are going to use a couple of plugins so we don’t have to use the confusing UI of the Sonatype Repository. (At least it was confusing when I used it last in ~2011.)

sbt setup

Add the following plugins to your project/plugins.sbt:

addSbtPlugin("org.xerial.sbt" % "sbt-sonatype" % "2.3")

addSbtPlugin("com.jsuereth" % "sbt-pgp" % "1.1.1")

and the following keys to your project settings in your main build.sbt:

// POM settings for Sonatype
organization := "com.example"
homepage := Some(url("https://github.com/username/projectname"))
scmInfo := Some(ScmInfo(url("https://github.com/username/projectname")
                            "git@github.com:username/projectname.git"))
developers := List(Developer("username",
                             "User Name",
                             "mail@username.de",
                             url("https://github.com/username")))
licenses += ("Apache-2.0", url("http://www.apache.org/licenses/LICENSE-2.0"))
publishMavenStyle := true

// Add sonatype repository settings
publishTo := Some(
  if (isSnapshot.value)
    Opts.resolver.sonatypeSnapshots
  else
    Opts.resolver.sonatypeStaging
)

Obviously replace the placeholder values appropriately.

GPG keys

In order to be on Maven Central your artifacts have to be signed with PGP. If you already have a PGP key, great! If not create one with:

sbt pgp-cmd gen-key

Then deploy this key to a key server

pgp-cmd send-key ${keyname} hkp://pool.sks-keyservers.net

There are a few key servers around the internet and every few hours they syncronise with each other so eventually all keys will be on all servers. Sonatype and Maven/Ivy clients will validate the signature on the artifact with your public key to make sure it is exactly what you have published.

Sonatype account

You need to make an account at the Sonatype repo: https://issues.sonatype.org/secure/Signup!default.jspa

They usually takes a few hours to respond and will ask you if you own the domain of your group id. If you don’t then maybe use the domain of the provider where you host the code such as com.github.username as your group id.

After a while the ticket will be closed and you’ll be told that you now can publish into your group id.

Configure credentials

Create a file $HOME/.sbt/(sbt-version)/sonatype.sbt

Set Sonatype account information (user name and password) in the global sbt settings. To protect your password, never commit this file to your SCM.

credentials += Credentials("Sonatype Nexus Repository Manager",
        "oss.sonatype.org",
        "(Sonatype user name)",
        "(Sonatype password)")

Publishing

Once you have the permission to upload you can simply run

sbt publishSigned

This will sign and upload your artifact to Sonatype’s staging repository. If you want you can test this release first by adding the repo to your build.sbt.

If you’re happy with the release and want to push to Maven Central run

sbt sonatypeRelease

This promotes the release to be ready for synching to Maven Central. It usually takes a few hours before it will show up in http://search.maven.org/ but the actual sync job runs every hour or so.

Now head over to the Maven search to check out your artifact.

First impressions of ScalaJS

2015-08-08T00:00:00+02:00

My last couple of projects have mostly been written in Scala and I’ve really started too love the language as I became better and better at it. Once you’ve fully experienced the joy of proper type safety, you can’t go back.

However, these days Scala is confined mostly to the server. Since my current contract also involves a rather large frontend component I also occassionally have to write JavaScript. Don’t get me wrong, I’m not some hapless backend code slinger dipping his toes into the browser world. Years ago I used the language a lot and I’m quite competent at it. However after writing a lot of Scala, JS’s weak types really grind on you: they make refactoring really difficult and you discover many problems, which a compiler would tell you much earlier, at runtime. :/

Help is at hand: EPFL (where Martin Odersky is a professor) is developing ScalaJS, which compiles Scala to JavaScript. This weekend I took it for a spin and here are my observations. (Mind you, this isn’t a thorough review but rather a rough-and-ready stream of consciousness.)

Nice integration with sbt

You can keep using your build system and IDE. Everything worked out-of-the-box.

Compiler

The compilation process is not slower than the compiler that targets the JVM. However, the Scala compiler is quite slow but you get used to it because it really helps you during development. Those sweet, sweet types are great!

The compiler produces unreadable code which will make debugging harder. ~~I’ve read that source maps are available, but I haven’t been able to make them work yet.~~ Source maps do work and can help you debugging!

Community

It’s rather small, but that shouldn’t surprise you since ScalaJS has only come out of beta a few weeks ago. You should expect to write stuff yourself because it doesn’t exist yet.

Libraries and frameworks

This is currently a weak spot. Obviously, you can use the fantastic Scala core library which to me is one of the main appeals. I haven’t found out how easy it is to get third party libraries like, say, Joda-Time working.

On the DOM side, there is the nice scalatags which makes writing HTML typesafe but other than there isn’t all that much around.

The community seems to be split between people wanting to wrap native JS frameworks like React and others wanting to write stuff from scratch in pure Scala.

I think we will see a lot of experimentation of approaches and many new frameworks popping up and eventually dieing. This isn’t something that worries me a lot since this is the way that really make a community thrive. Yes, it does cause fragmentation to a certain degree but I feel that it’s a price worth paying for finding the best ideas and practises.

File size

In short: it’s acceptable. With fast optimisation my dummy app clocked in at ~600kb. A full optimisation brought that down to around 140kb. Really large apps probably will be a few MB, but that’s not something unheard off with apps written in native JS, too.

In summary

I’m very excited about ScalaJS and I think it’s very promising. It gets a lot of things right and people are working on the things that aren’t so great yet.

I might sound like a fanboy, but most things coming out of the Scala ecosystem these days are very high quality and this is no exception.

If you can tolerate a little technological immaturity and don’t mind being an early adopter, I would seriously consider ScalaJS for you next frontend project.

Setting sbt memory options

2015-06-04T00:00:00+02:00

If you’ve been working on a bigger Scala project you will probably have experienced that SBT consumers a rather large amount of memory and can crash with a OutOfMemoryError Metaspace. There is a lot of advise floating around the web that tells you to set your memory options in $SBT_OPTS.

I learned that this is only partially true.

What I found out about sbt-launcher version 0.13.8 (the most recent version at the time of writing) is that setting Xmx and friends in $SBT_OPTS does indeed result in them being added to the command line arguments but then they are overwritten.

The resulting command line invocation looks like this:

/usr/bin/java -XX:+CMSClassUnloadingEnabled -Xmx1500M -XX:MaxMetaspaceSize=512m -Xms1024m -Xmx1024m -XX:ReservedCodeCacheSize=128m -XX:MaxMetaspaceSize=256m -jar /usr/local/Cellar/sbt/0.13.8/libexec/sbt-launch.jar

You can see that Xmx is set twice because the sbt launcher script blindly prepends the contents of $SBT_OPTS to the incovation.

If you put the setting into $JAVA_OPTS then the sbt launcher intelligently parses them but I wanted the memory settings to apply to sbt only and not other Java applications.

The solution I came up with in the end is to use the -mem paramter of sbt directly.

I’ve set the following alias

alias sbt="sbt -mem 1500"

This sets the heap space to 1.5 GB and the MaxMetaspace to 512 MB.

Make Maven treat warning as errors

2014-02-28T00:00:00+01:00

I like compiler warnings and I think a nicely linted code base really improves code hygiene and keeps standards high. Rather than relying on some post-compile script or a tool like Sonar I prefer the compiler to throw an error when it encouters something that we defined as a code smell. In other words, I want it to treat warnings as errors.

javac can do exactly that with the -Werror flag but using it in Maven took me a little while to find out. To enable it in your Maven builds, use the following XML:

...
<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-compiler-plugin</artifactId>
      <version>3.1</version>
      <configuration>
        <source>1.7</source>
        <target>1.7</target>
        <compilerArguments>
          <Werror />
          <Xlint:all />
        </compilerArguments>
      </configuration>
    </plugin>
  </plugins>
</build>
...

Find out who the maintainer of a Debian/Ubuntu package is

2014-02-11T00:00:00+01:00

Run the following command:

dpkg-query -W -f='${Maintainer}' coreutils

Obviously, replace coreutils with the package you’re interested in.

Removing 200s from an Apache access log

2013-10-25T00:00:00+02:00

At work we use Splunk to do log analysis of our frontend Apache which acts as a simple proxy to the application servers. I quite like Splunk but we were hitting our quota quite frequently when we started to include our access log to the indexed files.

We noticed that the vast majority of our entries in the access log had 200 response statuses. We’re actually not that interested in all these 200s and it would greatly reduce our Splunk usage if we could filter them all out. What we could have done is to just use the normal access log and then have a cronjob grep out all the 400s and 500s but that didn’t seem very elegant. I wanted a solution without an intermediate step.

Apache has a feature called conditional logging, however it can’t be used to filter by response code.

On the other hand I found out about piped logs. The idea is that you just pipe the log to another process which in turn can do any log processing and filtering you want.

I chose a combination of grep and Apache’s rotatelogs. To use it, put the following in your Apache configuration:

LogFormat "%s %h %l %u %t \"%r\" %b" splunk
CustomLog "|stdbuf -o0 /bin/grep --invert-match '^200' | /usr/sbin/rotatelogs /var/log/apache2/splunk-access.log 86400" splunk

The first line defines a log format with the nickname splunk. In this format we put the response code (%s) at the beginning of the log file so we can grep for it easily.

The second one is where the action starts. The pipe (|) indicates that it is a piped log. Next we use stdbuf -o0 to disable stdin buffering which makes it a pain to test this setup. You can skip this in production if you want to.

Next we hand over to grep and remove all lines that start with the string 200.

Finally we pass the data on to rotatelogs which rotates the log once a day and gzipps those older 24 hours. Read the rotatelogs manual for many more configuration settings.

ack2 package for Ubuntu 12.04

2013-10-15T00:00:00+02:00

Ubuntu 12.04 Precise Pengolin ships with version 1.96 of ack which has been superseded by the much improved version 2. Read the ack homepage if you want to find out what is new in ack 2.

Since I’m now using Ubuntu on my work computer I have built a Debian/Ubuntu package and uploaded it to a PPA. This makes installing ack2 really easy on Ubuntu 12.04.

If you want to install it do the following:

sudo add-apt-repository ppa:leonard-ehrenfried/ack2
sudo apt-get update
sudo apt-get install ack-grep

Afterwards, ack will be available as ack-grep (there is another program in the Ubuntu repositories using the name ack).

Since I share my .bashrc between OS X and Linux I’ve aliased it as follows:

if [ `uname` == "Linux" ]
then
  alias ack="ack-grep"
fi

Unit testing Javascript UIs

2013-08-19T00:00:00+02:00

NB: Javascript is quite popular on the server-side now as well. This article however concerns itself exclusively with JS in the browser.

In the last 5 years Javascript has come a very long way. When I started out as a (browser) Javascript developer people smiled when I called myself that. I was patronised and thought of as a lesser programmer, a pixel pusher. To be fair there was some truth to it as widgets often broke.

Times have changed. Interactive websites are at the heart of a lot of businesses and no longer an afterthought tacked on after the “real” development has been completed.

But still, there is a large holdout who think of browser programming with Javascript as a foolish activity and not a serious developer’s job.

Today I chatted to a friend about this and have started to develop at theory why that might be. I think it has partially something to do with how Javascript testing was done in the browser.

JS and HTML

On traditional web apps it worked mostly like this:

The server spits out HTML which is the baseline and users should be able to use the app/website even with JS disabled
Javascript then takes the exitisting HTML and applies some transformation on it, rendering it more interactive.

This meant that the widgets the frontend developers built were really hard to test, due to the fact that HTML and JS were largely separated. The widget expected a certain kind of HTML to operate on and if this structure wasn’t there, mostly the JS widget would just not work.

Activating one of these widgets would often work something like this

$(".my-widget").datepicker("activate");

and this would assume a DOM structure which would look something like this

<div class="my-widget">
  <input type="text">
  <div class="dropdown"></div>
</div>

Components

But a new style of programming the DOM, fuelled by the rise of client-side MVC frameworks like Backbone, came into being: the UI became parcelled up into components, views or whatever you might want to call them. This meant that a frontend developer wouldn’t have to build a certain HTML structure and then call some jQuery plugin on it.

Instead, the HTML isn’t rendered on the server but the component you are trying to build comes with its own HTML. Rather than the above you would do something like this:

var view = new DatepickerView();
var rendered = view.render();
$(".my-widget").append(rendered);

Can you spot the difference here? The component itself brings its own HTML to the table instead of manipulating some globally available DOM.

How is this different?

If we write our UI widgets in this style testing is rather easy. Before, if you wanted to really test your JS you had to test both the HTML the server produced as well as the widget code that operated on it. That would mean to somehow also build up the entire context of the server-side templating language and render a full response.

Whilst before you were never quite sure that the HTML you produced was matching the expectations the JS had, it is now rather easy to assert against the widget.

rendered.find("button").click();
expect(view.clickCount).toBe(1);

This new-found engineering rigour means that JS code can be as easily tested as server side code, if not easier. What has made the difference is the coupling of JS with the HTML it operates on (and produces). In my opinion, that’s the qualitative distinction.

In fact, if all your server produces is JSON asserting against that becomes easy too. You no longer have to do some DOM gymnastics to find out if your server has generated the correct response: just parse the JSON and assert against that.

Conclusion

Nowadays, JS-heavy projects are thought of as API clients to the server that produces JSON. This has enabled a style of frontend development that takes raw data as its input and not data mixed with a sprinkling of presentation layer. I think that this is a good thing. Frontend developers are now writing programs, apps and no longer just spice up your <select> elements.

How to test a Play framework project on Travis CI

2013-07-02T00:00:00+02:00

Setting up a Travis CI build job for a Play project isn’t quite as straightforward as I had hoped. There are some guides floating around the web telling you to download the binary distribution and execute that.

I, however, found that it is easier, quicker and cleaner to use the sbt command that comes preinstalled with the Travis VMs if you set the language to scala. (The play command is just a thin wrapper around sbt anyway.)

What it boils down to is this .travis.yml:

language: scala
scala:
   - 2.10.0
script: sbt test

You will also need to make sure that you have specified at least version 0.12.3 of sbt in your build.properties:

sbt.version=0.12.3

I found that with earlier sbt versions I would get a weird error whereby a slf4j-api dependency couldn’t be resolved.

Using the play command

Today I also got a second variation working by downloading the Play zip distribution and running that.

language: scala
env:
  - PLAY_VERSION=2.1.1
before_script:
  - wget http://downloads.typesafe.com/play/${PLAY_VERSION}/play-${PLAY_VERSION}.zip
  - unzip -q play-${PLAY_VERSION}.zip
script: play-${PLAY_VERSION}/play test

This takes a little longer but runs the tests in the way a developer is advised by the Play documentation to do it. I think either way is fine.

You must make absolutely sure that you are using the same Play version in your travis.yml and the play-sbt-plugin in your build.sbt.

Example project

commercetools/sphere-snowflake is an example Github repository of a Play project being tested on Travis.

Starting WEBBrick with logging disabled

2013-06-19T00:00:00+02:00

Today I had to start a little web server that prints out the HTTP requests received and I used Ruby’s WEBBrick for that.

The trouble is that it has its own logging built in that interfered with what I was trying to log.

Here is how you disable logging completely:

dev_null = WEBrick::Log::new("/dev/null", 7)
server = WEBrick::HTTPServer.new({ :Port => 9000, :Logger => dev_null, :AccessLog => dev_null })
server.start

Editing Scala with vim

2013-04-05T00:00:00+02:00

Even though it hurts my brain a little bit, learning Scala has been a pleasurable part of my job lately. It definitely isn’t for the faint-hearted but it has taught me a lot about thinking precisely about the topics of immutability and shared state. The functional mindset forces you to think mainly about input and output of functions and this training, I feel, makes me a better programmer even when programming in a language other than Scala.

Tools

I’m not really an IDE lover, quite the opposite, but I have started taking a shine to the comforts that Intelli-J is offering when writing Java.

I was happy to see that Intelli-J has a plugin for Scala development and have been using that for the last couple of months. Sadly, the plugin is really, really slow which makes it pretty unusable to me.

There is no question that the language is beautiful but the tools aren’t.

Enter vim

I’m already using vim for everything other than Java so I hunted around the web in order to find plugins. I’ve been quite satisfied with the following combination.

###vim-scala Obviously, you will need syntax highlighting and this plugin provides that.

###neocomplcache

The built-in vim autocomplete isn’t great for Scala but the neocomplcache plugin makes typing out long Scala class or package names easier. There is no semantic analysis behind it so all this plugin does is string matching but I was very surprised to realise that this takes you a long way. To me at least it is adequate.

ctags

In order to enable jump-to-source navigation you need to configure ctags to index scala files. You can do that by adding the following to your ~/.ctags:

--langdef=scala
--langmap=scala:.scala
--regex-scala=/^[ \t]*((abstract|final|sealed|implicit|lazy)[ \t]*)*(private[^ ]*|protected)?[ \t]*class[ \t]+([a-zA-Z0-9_]+)/\4/c,classes/
--regex-scala=/^[ \t]*((abstract|final|sealed|implicit|lazy)[ \t]*)*(private[^ ]*|protected)?[ \t]*object[ \t]+([a-zA-Z0-9_]+)/\4/c,objects/
--regex-scala=/^[ \t]*((abstract|final|sealed|implicit|lazy)[ \t]*)*(private[^ ]*|protected)?[ \t]*((abstract|final|sealed|implicit|lazy)[ \t]*)*case class[ \t]+([a-zA-Z0-9_]+)/\6/c,case classes/
--regex-scala=/^[ \t]*((abstract|final|sealed|implicit|lazy)[ \t]*)*(private[^ ]*|protected)?[ \t]*case object[ \t]+([a-zA-Z0-9_]+)/\4/c,case objects/
--regex-scala=/^[ \t]*((abstract|final|sealed|implicit|lazy)[ \t]*)*(private[^ ]*|protected)?[ \t]*trait[ \t]+([a-zA-Z0-9_]+)/\4/t,traits/
--regex-scala=/^[ \t]*type[ \t]+([a-zA-Z0-9_]+)/\1/T,types/
--regex-scala=/^[ \t]*((abstract|final|sealed|implicit|lazy|private[^ ]*(\[[a-z]*\])*|protected)[ \t]*)*def[ \t]+([a-zA-Z0-9_]+)/\4/m,methods/
--regex-scala=/^[ \t]*((abstract|final|sealed|implicit|lazy|private[^ ]*|protected)[ \t]*)*val[ \t]+([a-zA-Z0-9_]+)/\3/l,constants/
--regex-scala=/^[ \t]*((abstract|final|sealed|implicit|lazy|private[^ ]*|protected)[ \t]*)*var[ \t]+([a-zA-Z0-9_]+)/\3/l,variables/
--regex-scala=/^[ \t]*package[ \t]+([a-zA-Z0-9_.]+)/\1/p,packages/

After having done that you need to run ctags from the root of your project directory with something like this:

ctags -R . --exclude=target --exclude=vendor

By default vim will only look for the tags file in the directory of the file open in the current buffer. If you want vim do move up the directory hierarchy until it has found the file add this to your ~/.vimrc:

set tags=tags;/

Now, if you move the cursor over a type name and press Ctrl-] you will jump to the definition of the type. That’s pretty nifty!

If you want to go back to where you were before the jump press Ctrl-T.

Sorting of imports

Based on feedback from a collegue who was complaining that my imports always look messy I have contributed a command to the vim-scala plugin which automatically sorts your import statements.

You can invoke the command with :SortScalaImports and there are two modes on how your imports could be sorted. By default the command goes through the import groups (which are defined as separated by a newline) and orders them alphabetically. This is useful for when you prefer to sort your import into groups yourself like this:

// Utility imports
import com.me
import com.them

// Concurrency imports
import akka....
import scala...
import java...
import spray....

// Domain imports
import com.me.data....
import com.me.data....

The second mode can be enabled by setting the following in your .vimrc:

let g:scala_sort_across_groups=1

This will take all of your imports and puts them into 3 different groups:

Java/Scala core libraries
3rd party libraries
First party code, a.k.a your own

What is considered first party code can be configured by setting the a regex. I have set it to this, which is for a standard Play app:

let g:scala_first_party_namespaces='\(controllers\|views\|models\|util\|de.\)'

The result of the sort looks like this:

import java.text.SimpleDateFormat
import java.util.{ Currency, Locale, UUID, Calendar }
import scala.collection.JavaConversions._
import scala.util.Random

import play.api._
import play.api.mvc._

import controllers.Secured._
import de.mycompany.useful.library.Class
import util._

My 2009 dissertation using GATE

2013-01-29T00:00:00+01:00

Not quite 4 years ago I left university after having graduated with a degree in computer science. I did fairly well on my final dissertation and have been meaning to publish it for quite some time but never actually came round to it.

The reason that publishing this popped back into my mind was that last week a colleague and I got chatting about a friend of his. The friend is also wanting to write a thesis on natural language processing and I said that I can pass on mine.

My dissertation is titled Automatic extraction and categorisation of bicycle items for sale on an internet bulletin board and its focus is on natural language processing using the software GATE which is/was developed at the University of Sheffield.

If you want to read it you can head over to the document on Google Drive. From there you can download the original PDF on the top left corner.

Handlebars array access

2012-11-27T00:00:00+01:00

I keep forgetting and regoogling this and apparently it isn’t documented anywhere I could find it, so I’m going to write it down hoping that Google will help spread the word.

So, how do I access an array in a Handlebar template?

The syntax is the following: {{ array.[index] }}

An example from an app I’m building right now

<div class="img-container pull-left">
  {{ #if imageURLs.length }}
    <img src="{{ imageURLs.[0] }}"/>
  {{ else }}
    <img src="http://placehold.it/130x130" class="img-polaroid">
  {{ /if }}
</div>

Tools for the discerning front end developer

2012-11-25T00:00:00+01:00

Pro-forma preamble

Working as a web developer and doing a lot of frontend work, over the years I have collected a rather nice tool belt (if I may say so) of things that make me more productive. Today I’d like to share some of these.

Javascript

Grunt

One of my more recent discoveries is a JavaScript build tool called grunt. Even though I guess you could use it to build node modules, it is really meant for frontend coding tasks. If you’ve ever written your own scripts to glue together jshint, uglify, file concatenation and rsync and wondered if there is a better way, grunt can help you. It is ideal not only for those things but has plugins for virtually everything like:

CoffeeScript, including on-the-fly compilation if a file is modified
CSS Compression
LessCSS compilation
Live Reload of a website without browser plugins (more about this in the next section)

Yes, there is a bewildering array of JS build tools (Jake, Cake, Yeoman, Buildr, custom ant scripts… the list goes on) but I found grunt so far the most pleasurable to use. Go try it out.

Grunt Live Reload plugin

Frankly, this is like magic. What this enables you to do is to hit save in your favourite text editor and have this automatically trigger a refresh of the page you are working on. And all of that without even installing a browser extension. The way this works is by injecting a little snippet of JavaScript into the page which in turn opens a Websocket connection to the development server which watches for changes on the file system. Here is an example grunt.js file to achieve this:

module.exports = function (grunt) {
  grunt.initConfig({
    lint: {
      all: ['js/*.js']
    },
    less: {
      style: {
        src: 'style.less',
        dest: 'style.css'
      }
    },
    watch: {
      files: ['*.less', "*.html", "*.js"],
      tasks: 'less reload'
    },
    reload: {
      port: 6001,
      proxy: {
        host: 'localhost',
        port: 8000
      }
    },
  });
  grunt.loadNpmTasks('grunt-contrib-less');
  grunt.loadNpmTasks('grunt-reload');
  grunt.registerTask('default', 'server reload watch');
};

Who knew that development could be fun with tools like this?

CSS reload bookmarklet

If you don’t need the whole live reload setup or are maybe using a web framework which doesn’t play nice with grunt you can use Paul Irish’s CSS reload bookmarklet. This simply iterates over all the stylesheets in the document and adds a fake parameter with a timestamp value to their URL causing them to be reloaded. The timestamp also takes care of any caching issues.

Chrome doesn’t allow bookmarklets to be invoked with a keyboard shortcut but I found the following trick to be able to kinda-simulate a shortcut: You need to bind the bookmarklet to a custom search, which in turn can be triggered by a one-letter keyword.

In order to do this, create a new search, paste the bookmarklet Javascript URL into the URL field, give it a name and a one letter keyword (I have given it the letter ‘r’). Now you can jump to the URL bar with ⌘ + L, then type R and hit enter and voila, your CSS is reloaded without you having to refresh the entire page.

cdnjs

This free, community-curated CDN hosts Javascript libraries like Backbone, moment.js and various popular jQuery plugins. Using this has the double advantage of not polluting your source tree with third-party libraries as well as speeding up your page load by spreading the load onto multiple servers.

An added bonus: They support SPDY for extra-parallel download for enabled browsers. You can also add a library you need to the CDN by submitting a pull request. I’ve done that a few times and Ryan and Thomas have always been quick and very friendly.

Bootswatch

If you can’t stand the default look of Bootstrap anymore go and check out this nice selection of themes for Bootstrap.

Bootstrap Form Builder

If you’re building a lot of forms with Bootstrap, you probably want to check out this little app. It gives you a nice UI for building them.

Bootsnipp

If it isn’t forms specifically you’re building, Bootsnipp gives you small snippets of readymade Bootstrap components and examples of what you could build with Bootstrap.

BootstrapCDN

Are you tired of re-adding Bootstrap’s CSS file to your repositories over and over or using Github as a static asset server? As the name suggests this free CDN hosts Bootstrap for you. My favourite feature are the hosted versions of Bootswatch.

Misc.

Fontello

Remember the days when you had to pester your designer for small icon PNGs (most likely in plain as well as mouseover/active variations)? Thank God we have icon fonts now! The only downside is that selecting an icon character and inserting it into your HTML is quite a bit of work and may involve special programs for font managements.

Fontello is a web app that has made it its mission to take away the pain from using icon fonts. They have collected a hand full of very popular icon fonts (Entypo, for example), have given each icon character a simple, descriptive name and make this all available as single webfont file (in all formats you’ll ever need) plus a handy stylesheet. This lets you create icons by simply applying a CSS class to an element, just as it is done in Twitter Bootstrap.

If you suggest this in your team, your designer will love you as this frees her/him up to do actual design. Other benefits are that you can style your icons with CSS and have a lot less HTTP requests during page load.

JSON formatting

jq is pretty nice tool for querying a JSON document but if all you need is a pretty printer you can pipe your JSON through a built in python module with the following one-liner:

echo '{"foo": "lorem", "bar": "ipsum"}' | python -mjson.tool

I have this aliased in my .profile to jsonpretty with a

alias jsonpretty='python -mjson.tool'

ack - better than grep

If the project-wide full text search of $FAVOURITE_IDE sucks, and lets be honest they all do, and grep is just a little too awkward to use then take a look at ack. It has changed the way I explore and navigate source trees. Ack is designed to search through large source code trees, searches recursively by default and excludes stuff you definitely don’t want to search like git and temp folders.

Even though it is currently still in alpha state, I’m using version 2.0 for a couple of months now without any issues. Ack 2 removes the limitation of having to keep a whitelist of files to be searched and includes all text files by default. Another thing which wasn’t possible in ack 1 but is in 2 is including file names without dots (i.e. Gemfile). If you want to install ack 2.0 alpha through homebrew, I have prepared a recipe for you. Just do a

brew install https://raw.github.com/leonardehrenfried/ack2-compiled/master/ack2.rb

Testing regular expressions

To me it was a real eye opener when I discovered visual regex tools. They shorten the feedback loop drastically and make reasoning about them a lot easier (even though it still often takes a few moments to fully wrap your head around what is happening.)

My current favourite for doing that using the JS regex syntax is an obscure tool called Hifi Regex Tester. Even though its usability isn’t too great (too many colours!) it is the only one that I have found which displays match groups.

If you can live without match groups, refiddle feels a lot more modern and tidied up.

Your favourites

If you liked this little list, how about you letting me know about other productivity enhancers?

Slides for my talk at CocoaHeads Berlin

2012-10-18T00:00:00+02:00

Yesterday I gave a talk at Cocoaheads Berlin, which is the iOS/OS X developers user group here. The talk was titled “CocoaPods - Sane library management for Xcode”.

If you are interested in reading my HTML slide deck, I’ve made them available online.

Here is the same thing in a little embedded viewer: