A developer's approach to build, host and deploy native Linux packages

Leonard Ehrenfried - Freelance Developer


About me and my current project

  • Freelance developer here in Berlin
  • Current client: Startup where I wrote a Scala web service
  • No dedicated operations staff up until 2 weeks ago
  • Did it all by myself
  • Always very ops-minded and had the fortune of working together with really talented operations people previously

Today's story

How I built a pretty decent deployment pipeline and infrastructure on the cheap without a dedicated DevOps

How to deploy

  • For a while we did git pull && sbt stage
  • Obviously worst of all solutions
  • Scala compilation is slooooow
  • Scales horribly: each node has to compile separately, mismatched versions and third-party libs
  • We needed to improve that as we neared the launch

Enter DevOps

  • Obvious first step: Build server (Jenkins)
  • But how to copy onto nodes?
  • web-service.tar.gz?


  • Every Linux distro has already an excellent mechanism for putting files on a computer
  • apt-get, yum, pacman ...
  • Very good solution to the depoyment problem
  • Used by previous two customers/employers
  • But... quite complicated to set up
  • Problems: Building debs, maintaining APT repository

Building debs, pt 1

  • Fantastic sbt plugin: sbt-native-packager
  • Extremely easy to set up: took about 30 minutes to turn Scala project into a deb
  • No messing about with Debian files: It just works!
  • Very good defaults

Building debs, pt 2

  • App main content is at: /usr/share/$appname
  • Jars (compiled code + deps): /usr/share/$appname/lib
  • Startup script: /usr/share/$appname/bin/$appname
  • Conf files: /usr/share/$appname/conf (symlinked to /etc/$appname)
  • Upstart script: /etc/init/$appname.conf


  • Has actually been around for a while but I never used it until the current project
  • Seriously simple startup scripts
  • Godsend for anyone who had the misfortune to write/debug spagetti init.d scripts

Upstart conf example

description "Relayr web service for administering users, devices, transmitters and apps"
author "Relayr Team <team@relayr.io>"

start on runlevel [2345]
start on started [networking]
stop on runlevel [016]
stop on stopping [networking]

respawn limit 5 60

chdir /usr/share/registration
setuid registration

exec ./bin/registration

How to control Upstart jobs

  • Very much like service
  • sudo start $appname
  • sudo status $appname
  • sudo stop $appname
  • If no init.d service found service will try the upstart jobs

Ok, so now we have a deb. Now what?

Private Debian repository: the story so far

  • Jenkins builds deb
  • Copies deb to repo server
  • Repo server re-indexes complete repository with dpkg-scanpackages and friends
  • Very IO intensive, which cloud servers are bad at
  • Sloooow: At a previous project the last step alone took ~10 minutes
  • Optional: Use s3cmd to upload to S3 (also slow for large repos)
  • Complicated beast: brought seasoned DevOps to tears
  • reprepo can do incremental repositories but not multiple versions

Enter deb-s3

  • Since we are an AWS shop hosting on S3 is a no-brainer - we don't need yet another piece of infrastructure to build, maintain, monitor
  • deb-s3 lets you skip the repo server step
  • Sets up repository structure if it doesn't exist
  • Parses existing Packages.gz and adds a package as you go along
  • From deb file and empty bucket to full repository server in less than 1 minute
  • Fully incremental without full scan of repository
  • Supports multiple versions (for rollback) in one repository, and signing with GPG

Installing a deb on a local node

  • apt-get speaks http/https but to enable it on S3 you will have to make a bucket public :|
  • Use apt-transport-s3 to make apt speak the S3 protocol
  • Not in the Ubuntu repositories but PPA is available
  • Access is managed with ordinary AWS secret keys
  • sudo apt-get install $appname=1.3.4

Demo time!

The End


Hire me! http://leonard.io