1. Intro/Background
2. Continuous Deployment architecture/approach
3. Walkthough: Check-in → Compile → Package → Deploy
4. Lessons learned

• Integration
• Deployment
• Delivery

• $startup
• 2 week sprints
• me: Senior Systems Engineer, Cloud Team
  • responsible for AWS stack, from architecture to ops
  • automate
  • work with Engineering & Q/A

Stack

• AWS (EC2, S3)
• RHEL (previously CentOS 5.x, now SL 6.1)
• JVM, MongoDB
• Frontend

Automation

• Puppet
• MCollective
• Shell + Ruby

Environment

• Self-contained set of servers, all environments are equal. Staging is like all other environments, just labeled Staging. It's true name is w2012017-105536.

General theme

• "God created the world"/"Bigbang"
• project "weltenmeister"

Have your environment whenever you want! It's just one click in Jenkins away.

Default environment is 5 servers (1x m1.small, 2x c1.medium, 2x m1.large)

Timings

• 10-15min from check-in to deploy (CI)
• In 90min from mkfs.ext4 to running environment

• Jenkins
  • knows when to compile, knows dependencies between projects
• Bigbang
  • stable interface to manage environments
• Selenium
  • acceptance tests

• create/update
  • universe()
  • world()
  • publish() + packages()
• shutdown
  • apocalypse()
  • collapse()
• manage
  • bless() + curse()
  • baptize()
  • ...

• design
  • dev's responsibility to keep .rpm in good shape (log-config, monitoring, etc)
  • Project's build-system (sbt, ant, maven) compiles, tests, packages .rpm
• implementation
  • project has build.properties
  • replace @@property@@ in .spec
  • Maven snapshots: RPM release number is YYYYDDMM-hhmmss
  • %post hook will integrate service into environment (details follow)

• design
  • Jenkins knows pattern (*.jar, *.rpm) or directories only
  • use S3 to serve packages
• implementation
  • publish to Nexus (sbt deploy)
  • find ./target -type f -name '*.rpm' | xargs bigbang publish
  • use createrepo's baseurl feature: upload once, reuse multiple times
  • s3cmd sync
  • yum-s3 plugin to enable fetching from non-public S3

• design
  • install specific version of service (downgrades and upgrades)
  • allow multiple installs at once
• implementation
  • MCollective packages agent: clean yum cache, retry on error, check version, ...
  • %post install scripts
    • config-files are Ruby templates. Value source: facter + env.json
    • auto-register at HA-Proxy and Monit: drop files in /etc/haproxy.d/ and /etc/monit.d/

Shared staging, testing env? - No, have your own dedicated environment in 90 minutes.

Architecture

• do one thing right - build small tools
• embrace share nothing

Design/Implementation

• be "cloud ready"
• use proper programming language, shell for subtasks only
• avg. 50 deploys/day

Packaging

• use packages, it's easy
• avoid git pull, .gem, .zip when possible
• Amazon S3 as package repo

Tooling

• Puppet/MCollective/ActiveMQ
• MCollective Packages Agent
  https://github.com/jbraeuer/mcollective-plugins.git,
  PuppetLabs MCollective Wiki
• Fog (http://fog.io)
• Monit (http://mmonit.com/monit/)

• French Linux Magazin
  • Romain PELISSE on Yum+S3, Reliable Scripting
• https://github.com/jbraeuer
  • yum-s3-plugin, apt-s3
  • MCollective packages agent