A developer's approach to build, host and deploy native Linux packages
Leonard Ehrenfried - Freelance Developer
http://leonard.io
About me and my current project
- Freelance developer here in Berlin
- Current client: Startup where I wrote a Scala web service
- No dedicated operations staff up until 2 weeks ago
- Did it all by myself
- Always very ops-minded and had the fortune of working together with really talented operations people previously
Today's story
How I built a pretty decent deployment pipeline and infrastructure
on the cheap without a dedicated DevOps
How to deploy
- For a while we did
git pull && sbt stage
- Obviously worst of all solutions
- Scala compilation is slooooow
- Scales horribly: each node has to compile separately, mismatched versions and third-party libs
- We needed to improve that as we neared the launch
Enter DevOps
- Obvious first step: Build server (Jenkins)
- But how to copy onto nodes?
web-service.tar.gz?
Packaging
- Every Linux distro has already an excellent mechanism for putting files on a computer
apt-get, yum, pacman ...- Very good solution to the depoyment problem
- Used by previous two customers/employers
- But... quite complicated to set up
- Problems: Building debs, maintaining APT repository
Building debs, pt 1
- Fantastic sbt plugin:
sbt-native-packager
- Extremely easy to set up: took about 30 minutes to turn Scala project into a deb
- No messing about with Debian files: It just works!
- Very good defaults
Building debs, pt 2
- App main content is at:
/usr/share/$appname
- Jars (compiled code + deps):
/usr/share/$appname/lib
- Startup script:
/usr/share/$appname/bin/$appname
- Conf files:
/usr/share/$appname/conf (symlinked to /etc/$appname)
- Upstart script:
/etc/init/$appname.conf
Upstart
- Has actually been around for a while but I never used it until the current project
- Seriously simple startup scripts
- Godsend for anyone who had the misfortune to write/debug spagetti
init.d scripts
Upstart conf example
description "Relayr web service for administering users, devices, transmitters and apps"
author "Relayr Team <team@relayr.io>"
start on runlevel [2345]
start on started [networking]
stop on runlevel [016]
stop on stopping [networking]
respawn
respawn limit 5 60
chdir /usr/share/registration
setuid registration
exec ./bin/registration
How to control Upstart jobs
- Very much like
service
sudo start $appnamesudo status $appnamesudo stop $appname- If no init.d service found
service will try the upstart jobs
Ok, so now we have a deb. Now what?
Private Debian repository: the story so far
- Jenkins builds deb
- Copies deb to repo server
- Repo server re-indexes complete repository with
dpkg-scanpackages and friends
- Very IO intensive, which cloud servers are bad at
- Sloooow: At a previous project the last step alone took ~10 minutes
- Optional: Use
s3cmd to upload to S3 (also slow for large repos)
- Complicated beast: brought seasoned DevOps to tears
reprepo can do incremental repositories but not multiple versions
Enter deb-s3
- Since we are an AWS shop hosting on S3 is a no-brainer - we don't need yet another
piece of infrastructure to build, maintain, monitor
deb-s3 lets you skip the repo server step- Sets up repository structure if it doesn't exist
- Parses existing
Packages.gz and adds a package as you go along
- From deb file and empty bucket to full repository server in less than 1 minute
- Fully incremental without full scan of repository
- Supports multiple versions (for rollback) in one repository, and signing with GPG
Installing a deb on a local node
apt-get speaks http/https but to enable it on S3 you will have to make a bucket public :|- Use
apt-transport-s3 to make apt speak the S3 protocol
- Not in the Ubuntu repositories but PPA is available
- Access is managed with ordinary AWS secret keys
sudo apt-get install $appname=1.3.4