Performance analysis

of docker



Jeremy Eder, Red Hat

@jeremyeder


Including work from many...many Red Hatters



Agenda

  • Who am I ?
  • Why is Docker interesting (to me)?
  • Approach to Performance Analysis
  • Test Harness Integration
  • Performance Features of Docker
  • Some test results
  • Demo

who am i ?

  • Long time Linux user
  • Former Datacenter Infrastructure "Ops Guy"
  • Performance Engineer for Red Hat
    • Red Hat Summit Present er
  • Mostly worked on networking things
  • Now LXC/Docker and other strange things...

WHY DOCKER IS INTERESTING (TO ME)


It's easy...
It's light-weight...
It's not virtualization...

It's fast...

But how fast ?

performance analysis basics


approach to perf analysis

  • Test for limits
  • Quantify overhead of container subsystems
  • See what breaks, document/fix
  • Assemble and document tuning advice
  • Adapt tests to "Docker-friendly" patterns


interesting stuff to test
  • All the benchmarks
  • Traditional apps in containers
    • Databases
    • Business Analytics
  • Web stacks
    • Everything that OpenShift can run
  • Use containers to scale-test other products
  • Tracking OpenStack+Docker

PERFORMANCE-IMPACTING FEATURES


PERFORMANCE-IMPACTING FEATURES: volumes


  • Customize storage characteristics per-volume
    • Can't do that with any COW FS
  • Bind mount storage from data container (or host)
    • --volume or --volumes-from  
  • Within 5% across several workloads
    •

Is anyone planning NOT to use volumes?

PERFORMANCE-IMPACTING FEATURES: device mapper

Why use Device Mapper
  • It's upstream
  • Significant production exposure, default in RHEL
  • Mature, well documented industry standard
  • Red Hat employs several of the maintainers 
    • --storage-driver=devicemapper

    DEVICE MAPPER STORAGE OPTIONS PATCH

    • dm.basesize: Size of the base dm device (10G by default)
    • dm.loopdatasize: Initial size of the data looback file
    • dm.loopmetadatasize: Initial size of the metadata loopback file
    • dm.fs: The filesystem to use for the base image (xfs or ext4)
    • dm.datadev: Set raw block device to use for data
    • dm.metadatadev: Set raw block device to use for metadata
    • dm.blocksize:  Custom blocksize for the thin pool.  Default is 64K.


    Alex Larsson, Mike Snitzer, Red Hat LVM Team

    PERFORMANCE-IMPACTING FEATURES:

    --net=host


    • Expose host network stack to a container
    • Skips creation of NEWNET namespace
    • Performs better than a bridge or OVS
      • Bare metal throughput and latency
      • Works around a UDP frag issue
    • Adds complexity to port allocation

    PERFORMANCE-IMPACTING FEATURES: cgroups


    • libcontainer-based Docker CLI currently supports
      • CPU Shares
      • CPUsets
      • Memory Limits
      • nsinit


    • LXC backend has more flexibility
    • libcontainer catching up
    • libcontainer default since v0.9


    DEMO

    PERFORMANCE-IMPACTING FEATURES: sysctls


    • Kernel defaults are suitable for most deployments
    • Tuning required for high density
      • SELinux
      • Max PIDs/open files
      • netfilter conntrack
      • netdev_max_backlog


    Can't I just push a button and make it go fast ???

    PERFORMANCE-IMPACTING FEATURES: tuned


    "Tune-D" is a tuning profile delivery mechanism

    Red Hat ships tuned profiles that improve performance for many workloads...hopefully yours!

    custom Tuned Profiles


      Tuned Profiles throughout our Product Line


    avc cache tuning

    performance monitoring

    "If you can't measure it,  you can't improve it" William Thompson

    TEST INTEGRATION

     

    TEST integration:  running tests

    benchmark integration demo:  sysbench

    docker as a lightweight test utility

    • Red Hat scale tests all it's products
    • Challenging as we scale into many thousands
    • We think containers can replace older simulation techniques
      • Lighter, faster, less hardware required

    initial test results

    sysbench oltp (mariadb)


    cyclictest

     SOME TEST results...network latency



    network throughput

    linpack

    database


    other one-liners


    • O_DIRECT still works
    • Hugepages still matter (+ ~10%)
    • NUMA affinity still matters (+ ~10-25%)
    • PCI locality still matters (- ~30% latency)
    • No support for SCHED_FIFO tasks yet, boo :-(
      •

    THE END

    Performance Analysis of Docker

    By Jeremy Eder

    Performance Analysis of Docker

    • 6,404
    Loading comments...