Your browser doesn't support the features required by impress.js, so you are presented with a simplified version of this presentation.

For the best experience please use the latest Chrome, Safari or Firefox browser. Upcoming version 10 of Internet Explorer should also handle it.

Jenkins Scaling

best practices and tips

Agenda

Scale?
Master
Slaves
Job configuration
Central logging
Archiving

Scale?

Tens of projects
Multiple branches
Various technologies (Java, JS, Ruby, Python)
No single machine can handle all of the builds
Elasticity (jobs come and go)
A lot of developers accessing master

M a s t e r

Master do's and dont's

Keep webui responsive
Running/Manaing jobs/slaves take resources
Throw a beefy VM at it (IO less of an issue)
Don't run jobs on it
Manage/Backup configuration programmaticly (puppet)

# see http://tinyurl.com/746w35l  
location ~ ^/static/[0-9a-fA-F] ... $ {
 # rewrite all static files into the root:
 # /static/1234/css/some.css -> /css/some.css
 rewrite "^/static/[0-9a-fA-F] ... ;
}

Slaves

Should be quick to setup and throw away
plan for high IO usage! (not single hypervisor)
Monitor Cpu and Disk
Segregate heavy jobs

Configure Jenkins master and its plugins
Configure build tools and sdk's
Install logging/monitoring agents

Config

uration

Job configuration smells

Build logic leaks to job
Too many steps
Shell steps
Not under scm

Managing branches from UI results in duplication, create a template project and clone, use programatic API

job(name:'liquibase'){

  slave(name:'jenkins-slave')  

  scm {
    url('git://github.com/...')
  } 

  steps {
    gradle(version: '1.0', tasks: 'build')
  }
}


  
    
    
    build
    
    
    1.0
    false

% gradle tasks

  Other tasks
  -----------
  buildJob
  createJob
  deleteJob
  listjobs

Tracking

Logstash

input {
 file {
  type => "jenkins"
  path => "/var/log/jenkins/jenkins.log"
 }

 file {
  type => "jenkins-slave"
  path => "/var/.../slave-jenkins-slave.log"
 }
}

output {
  stdout { }

  gelf {
     host => "graylog2"
  }
}

i = Inotify.new
LOGGER = GELF::Logger.new("graylog2" ..)

t = Thread.new do
 i.each_event do |ev|
   path = "#{ARGV[0]}/#{ev.name}"
   if(File.directory?(path) && ...)
     log_job("#{path}/log") if File.exists? ..
   end
 end
end

i.add_watch(ARGV[0], Inotify::CREATE ...)
t.join

Archiving

What to lookout for

Jenkins isn't optimal at this
Back it up
Fast download (deployment source)
Metadata will be required at some point

Alternatives

Artifactory (it rocks!)
Serve over http (nginx)
Serve deltas (Rsync server)
AWS S3

Vagrant

Vagrant::Config.run do |config|

 config.vm.define :master do |master|
   # master configuration
 end

  config.vm.define :slave do |slave|
   # slave configuration
  end

end