Continuous Deployment
- the tech parts -

Jens Bräuer
<braeuer.jens@googlemail.com>

Presenter Notes

TOC

  1. Intro/Background
  2. Continuous Deployment architecture/approach
  3. Walkthough: Check-in → Compile → Package → Deploy
  4. Lessons learned

Presenter Notes

Continuous ...

  • Integration
  • Deployment
  • Delivery

Presenter Notes

Background/Setting

  • $startup
  • 2 week sprints
  • me: Senior Systems Engineer, Cloud Team
    • responsible for AWS stack, from architecture to ops
    • automate
    • work with Engineering & Q/A

Presenter Notes

Software

Stack

  • AWS (EC2, S3)
  • RHEL (previously CentOS 5.x, now SL 6.1)
  • JVM, MongoDB
  • Frontend

Automation

  • Puppet
  • MCollective
  • Shell + Ruby

Presenter Notes

Disclaimer
This is not perfect (yet :-)

Presenter Notes

Wording

Environment

  • Self-contained set of servers, all environments are equal. Staging is like all other environments, just labeled Staging. It's true name is w2012017-105536.

General theme

  • "God created the world"/"Bigbang"
  • project "weltenmeister"

Presenter Notes

Continuous Deployment

Environment

Have your environment whenever you want! It's just one click in Jenkins away.

Default environment is 5 servers (1x m1.small, 2x c1.medium, 2x m1.large)

Timings

  • 10-15min from check-in to deploy (CI)
  • In 90min from mkfs.ext4 to running environment

Presenter Notes

Separation of concerns

BigBang-API

  • Jenkins
    • knows when to compile, knows dependencies between projects
  • Bigbang
    • stable interface to manage environments
  • Selenium
    • acceptance tests

Presenter Notes

Bigbang API

  • create/update
    • universe()
    • world()
    • publish() + packages()
  • shutdown
    • apocalypse()
    • collapse()
  • manage
    • bless() + curse()
    • baptize()
    • ...

Presenter Notes

CD walkthrough

Presenter Notes

CD walkthrough (compile)

cicompile

  • design
    • dev's responsibility to keep .rpm in good shape (log-config, monitoring, etc)
    • Project's build-system (sbt, ant, maven) compiles, tests, packages .rpm
  • implementation
    • project has build.properties
    • replace @@property@@ in .spec
    • Maven snapshots: RPM release number is YYYYDDMM-hhmmss
    • %post hook will integrate service into environment (details follow)

Presenter Notes

CD walkthrough (publish)

cipublish

  • design
    • Jenkins knows pattern (*.jar, *.rpm) or directories only
    • use S3 to serve packages
  • implementation
    • publish to Nexus (sbt deploy)
    • find ./target -type f -name '*.rpm' | xargs bigbang publish
    • use createrepo's baseurl feature: upload once, reuse multiple times
    • s3cmd sync
    • yum-s3 plugin to enable fetching from non-public S3

Presenter Notes

CD walkthrough (install)

ciinstall

  • design
    • install specific version of service (downgrades and upgrades)
    • allow multiple installs at once
  • implementation
    • MCollective packages agent: clean yum cache, retry on error, check version, ...
    • %post install scripts
      • config-files are Ruby templates. Value source: facter + env.json
      • auto-register at HA-Proxy and Monit: drop files in /etc/haproxy.d/ and /etc/monit.d/

Presenter Notes

Lessons learned

Presenter Notes

Lessons learned

Shared staging, testing env? - No, have your own dedicated environment in 90 minutes.

Architecture

  • do one thing right - build small tools
  • embrace share nothing

Design/Implementation

  • be "cloud ready"
  • use proper programming language, shell for subtasks only
  • avg. 50 deploys/day

Presenter Notes

Lessons learned (cont.)

Packaging

  • use packages, it's easy
  • avoid git pull, .gem, .zip when possible
  • Amazon S3 as package repo

Tooling

Presenter Notes

Questions?
Feedback?
How do you do this?

Presenter Notes

Thank you!

Presenter Notes