feat(metrics): Add documentation and contrib scripts

release-0.4
Brian Waldon 2014-01-17 16:15:54 -08:00
parent 14c96306a0
commit 3e7c2dff96
14 changed files with 1234 additions and 28 deletions

View File

@ -0,0 +1,69 @@
# Debugging etcd
Diagnosing issues in a distributed application is hard.
etcd will help as much as it can - just enable these debug features using the CLI flag `-trace=*` or the config option `trace=*`.
## Logging
Log verbosity can be increased to the max using either the `-vvv` CLI flag or the `very_very_verbose=true` config option.
The only supported logging mode is to stdout.
## Metrics
etcd itself can generate a set of metrics.
These metrics represent many different internal data points that can be helpful when debugging etcd servers.
#### Metrics reference
Each individual metric name is prefixed with `etcd.<NAME>`, where \<NAME\> is the configured name of the etcd server.
* `timer.appendentries.handle`: amount of time a peer takes to process an AppendEntriesRequest from the POV of the peer itself
* `timer.peer.<PEER>.heartbeat`: amount of time a peer heartbeat operation takes from the POV of the leader that initiated that operation for peer \<PEER\>
* `timer.command.<COMMAND>`: amount of time a given command took to be processed through the local server's raft state machine. This does not include time waiting on locks.
#### Fetching metrics over HTTP
Once tracing has been enabled on a given etcd server, all metric data is available at the server's `/debug/metrics` HTTP endpoint (i.e. `http://127.0.0.1:4001/debug/metrics`).
Executing a GET HTTP command against the metrics endpoint will yield the current state of all metrics in the etcd server.
#### Sending metrics to Graphite
etcd supports [Graphite's Carbon plaintext protocol](https://graphite.readthedocs.org/en/latest/feeding-carbon.html#the-plaintext-protocol) - a TCP wire protocol designed for shipping metric data to an aggregator.
To send metrics to a Graphite endpoint using this protocol, use of the `-graphite-host` CLI flag or the `graphite_host` config option (i.e. `graphite_host=172.17.0.19:2003`).
See an [example graphite deploy script](https://github.com/coreos/etcd/contrib/graphite).
#### Generating additional metrics with Collectd
[Collectd](http://collectd.org/documentation.shtml) gathers metrics from the host running etcd.
While these aren't metrics generated by etcd itself, it can be invaluable to compare etcd's view of the world to that of a separate process running next to etcd.
See an [example collectd deploy script](https://github.com/coreos/etcd/contrib/collectd).
## Profiling
etcd exposes profiling information from the Go pprof package over HTTP.
The basic browseable interface is served by etcd at the `/debug/pprof` HTTP endpoint (i.e. `http://127.0.0.1:4001/debug/pprof`).
For more information on using profiling tools, see http://blog.golang.org/profiling-go-programs.
**NOTE**: In the following examples you need to ensure that the `./bin/etcd` is identical to the `./bin/etcd` that you are targetting (same git hash, arch, platform, etc).
#### Heap memory profile
```
go tool pprof ./bin/etcd http://127.0.0.1:4001/debug/pprof/heap
```
#### CPU profile
```
go tool pprof ./bin/etcd http://127.0.0.1:4001/debug/pprof/profile
```
#### Blocked goroutine profile
```
go tool pprof ./bin/etcd http://127.0.0.1:4001/debug/pprof/block
```

View File

@ -1,28 +0,0 @@
## Profiling
etcd exposes profiling information from the Go pprof package over HTTP.
The basic browseable interface can be found at `http://127.0.0.1:4001/debug/pprof`.
**NOTE**: In the following examples you need to ensure that the `./bin/etcd` is
identical to the `./bin/etcd` that you are targetting (same git hash, arch,
platform, etc).
### Heap memory profile
```
go tool pprof ./bin/etcd http://127.0.0.1:4001/debug/pprof/heap
```
### CPU profile
```
go tool pprof ./bin/etcd http://127.0.0.1:4001/debug/pprof/profile
```
### Blocked goroutine profile
```
go tool pprof ./bin/etcd http://127.0.0.1:4001/debug/pprof/block
```
For more information on using the tools see http://blog.golang.org/profiling-go-programs

View File

@ -0,0 +1,9 @@
FROM stackbrew/ubuntu:raring
RUN apt-get update && apt-get install -y collectd
RUN adduser --system --group --no-create-home collectd
ADD collectd.conf /etc/collectd/collectd.conf.tmpl
ADD collectd-wrapper /bin/collectd-wrapper
RUN chown -R collectd:collectd /etc/collectd
CMD ["collectd-wrapper"]

20
contrib/collectd/README Normal file
View File

@ -0,0 +1,20 @@
We're going to use Docker to build a chroot env that can be run with systemd-nspawn since I cannot figure out how to run
a container using docker in the global network namespace.
1. Build the collectd image using docker
docker build -t collectd .
2. Run the container (since we have to run it to export it...)
COLLECTD_CONTAINER=`docker run -name collectd-tmp -d collectd`
3. Export then kill the container
docker export collectd-tmp > /tmp/collectd.tar
4. Kill the temporary container
docker kill $COLLECTD_CONTAINER
5. Unpack the tar archive
mkdir -p /tmp/collectd && tar -xvf /tmp/collectd.tar -C /tmp/collectd/
6. Run collectd with systemd-nspawn - replace the COLLECTD_* env vars with your parameters!
sudo systemd-run --unit collectd systemd-nspawn -D /tmp/collectd /bin/bash -c "COLLECTD_GRAPHITE_HOSTNAME=172.31.13.241 COLLECTD_LOCAL_HOSTNAME=node1 /bin/collectd-wrapper"

View File

@ -0,0 +1,16 @@
#!/bin/bash
cat /etc/collectd/collectd.conf.tmpl > /etc/collectd/collectd.conf
cat << EOF >> /etc/collectd/collectd.conf
Hostname "${COLLECTD_LOCAL_HOSTNAME}"
<Plugin write_graphite>
<Carbon>
Host "${COLLECTD_GRAPHITE_HOSTNAME}"
Port "2003"
</Carbon>
</Plugin>
EOF
collectd -C /etc/collectd/collectd.conf -f

View File

@ -0,0 +1,898 @@
# Config file for collectd(1).
#
# Some plugins need additional configuration and are disabled by default.
# Please read collectd.conf(5) for details.
#
# You should also read /usr/share/doc/collectd-core/README.Debian.plugins
# before enabling any more plugins.
#Hostname "localhost"
#FQDNLookup true
#BaseDir "/var/lib/collectd"
#PluginDir "/usr/lib/collectd"
#TypesDB "/usr/share/collectd/types.db" "/etc/collectd/my_types.db"
#Interval 10
#Timeout 2
#ReadThreads 5
LoadPlugin logfile
#LoadPlugin syslog
<Plugin logfile>
LogLevel "info"
File STDOUT
Timestamp true
PrintSeverity false
</Plugin>
#<Plugin syslog>
# LogLevel info
#</Plugin>
#LoadPlugin amqp
#LoadPlugin apache
#LoadPlugin apcups
#LoadPlugin ascent
#LoadPlugin battery
#LoadPlugin bind
#LoadPlugin conntrack
#LoadPlugin contextswitch
LoadPlugin cpu
#LoadPlugin cpufreq
#LoadPlugin csv
#LoadPlugin curl
#LoadPlugin curl_json
#LoadPlugin curl_xml
#LoadPlugin dbi
LoadPlugin df
#LoadPlugin disk
#LoadPlugin dns
#LoadPlugin email
#LoadPlugin entropy
#LoadPlugin ethstat
#LoadPlugin exec
#LoadPlugin filecount
#LoadPlugin fscache
#LoadPlugin gmond
#LoadPlugin hddtemp
#LoadPlugin interface
#LoadPlugin ipmi
#LoadPlugin iptables
#LoadPlugin ipvs
#LoadPlugin irq
#LoadPlugin java
#LoadPlugin libvirt
#LoadPlugin load
#LoadPlugin madwifi
#LoadPlugin mbmon
#LoadPlugin md
#LoadPlugin memcachec
#LoadPlugin memcached
LoadPlugin memory
#LoadPlugin multimeter
#LoadPlugin mysql
#LoadPlugin netlink
#LoadPlugin network
#LoadPlugin nfs
#LoadPlugin nginx
#LoadPlugin notify_desktop
#LoadPlugin notify_email
#LoadPlugin ntpd
#LoadPlugin numa
#LoadPlugin nut
#LoadPlugin olsrd
#LoadPlugin openvpn
#<LoadPlugin perl>
# Globals true
#</LoadPlugin>
#LoadPlugin pinba
#LoadPlugin ping
#LoadPlugin postgresql
#LoadPlugin powerdns
#LoadPlugin processes
#LoadPlugin protocols
#<LoadPlugin python>
# Globals true
#</LoadPlugin>
#LoadPlugin rrdcached
#LoadPlugin rrdtool
#LoadPlugin sensors
#LoadPlugin serial
#LoadPlugin snmp
#LoadPlugin swap
#LoadPlugin table
#LoadPlugin tail
LoadPlugin tcpconns
#LoadPlugin teamspeak2
#LoadPlugin ted
#LoadPlugin thermal
#LoadPlugin tokyotyrant
#LoadPlugin unixsock
#LoadPlugin uptime
#LoadPlugin users
#LoadPlugin uuid
#LoadPlugin varnish
#LoadPlugin vmem
#LoadPlugin vserver
#LoadPlugin wireless
LoadPlugin write_graphite
#LoadPlugin write_http
#LoadPlugin write_mongodb
#<Plugin amqp>
# <Publish "name">
# Host "localhost"
# Port "5672"
# VHost "/"
# User "guest"
# Password "guest"
# Exchange "amq.fanout"
# RoutingKey "collectd"
# Persistent false
# StoreRates false
# </Publish>
#</Plugin>
#<Plugin apache>
# <Instance "foo">
# URL "http://localhost/server-status?auto"
# User "www-user"
# Password "secret"
# VerifyPeer false
# VerifyHost false
# CACert "/etc/ssl/ca.crt"
# Server "apache"
# </Instance>
#
# <Instance "bar">
# URL "http://some.domain.tld/status?auto"
# Host "some.domain.tld"
# Server "lighttpd"
# </Instance>
#</Plugin>
#<Plugin apcups>
# Host "localhost"
# Port "3551"
#</Plugin>
#<Plugin ascent>
# URL "http://localhost/ascent/status/"
# User "www-user"
# Password "secret"
# VerifyPeer false
# VerifyHost false
# CACert "/etc/ssl/ca.crt"
#</Plugin>
#<Plugin "bind">
# URL "http://localhost:8053/"
#
# ParseTime false
#
# OpCodes true
# QTypes true
# ServerStats true
# ZoneMaintStats true
# ResolverStats false
# MemoryStats true
#
# <View "_default">
# QTypes true
# ResolverStats true
# CacheRRSets true
#
# Zone "127.in-addr.arpa/IN"
# </View>
#</Plugin>
#<Plugin csv>
# DataDir "/var/lib/collectd/csv"
# StoreRates false
#</Plugin>
#<Plugin curl>
# <Page "stock_quotes">
# URL "http://finance.google.com/finance?q=NYSE%3AAMD"
# User "foo"
# Password "bar"
# VerifyPeer false
# VerifyHost false
# CACert "/etc/ssl/ca.crt"
# MeasureResponseTime false
# <Match>
# Regex "<span +class=\"pr\"[^>]*> *([0-9]*\\.[0-9]+) *</span>"
# DSType "GaugeAverage"
# Type "stock_value"
# Instance "AMD"
# </Match>
# </Page>
#</Plugin>
#<Plugin curl_json>
## See: http://wiki.apache.org/couchdb/Runtime_Statistics
# <URL "http://localhost:5984/_stats">
# Instance "httpd"
# <Key "httpd/requests/count">
# Type "http_requests"
# </Key>
#
# <Key "httpd_request_methods/*/count">
# Type "http_request_methods"
# </Key>
#
# <Key "httpd_status_codes/*/count">
# Type "http_response_codes"
# </Key>
# </URL>
## Database status metrics:
# <URL "http://localhost:5984/_all_dbs">
# Instance "dbs"
# <Key "*/doc_count">
# Type "gauge"
# </Key>
# <Key "*/doc_del_count">
# Type "counter"
# </Key>
# <Key "*/disk_size">
# Type "bytes"
# </Key>
# </URL>
#</Plugin>
#<Plugin "curl_xml">
# <URL "http://localhost/stats.xml">
# Host "my_host"
# Instance "some_instance"
# User "collectd"
# Password "thaiNg0I"
# VerifyPeer true
# VerifyHost true
# CACert "/path/to/ca.crt"
#
# <XPath "table[@id=\"magic_level\"]/tr">
# Type "magic_level"
# InstancePrefix "prefix-"
# InstanceFrom "td[1]"
# ValuesFrom "td[2]/span[@class=\"level\"]"
# </XPath>
# </URL>
#</Plugin>
#<Plugin dbi>
# <Query "num_of_customers">
# Statement "SELECT 'customers' AS c_key, COUNT(*) AS c_value \
# FROM customers_tbl"
# MinVersion 40102
# MaxVersion 50042
# <Result>
# Type "gauge"
# InstancePrefix "customer"
# InstancesFrom "c_key"
# ValuesFrom "c_value"
# </Result>
# </Query>
#
# <Database "customers_db">
# Driver "mysql"
# DriverOption "host" "localhost"
# DriverOption "username" "collectd"
# DriverOption "password" "secret"
# DriverOption "dbname" "custdb0"
# SelectDB "custdb0"
# Query "num_of_customers"
# Query "..."
# </Database>
#</Plugin>
#<Plugin df>
# Device "/dev/sda1"
# Device "192.168.0.2:/mnt/nfs"
# MountPoint "/home"
# FSType "ext3"
# IgnoreSelected false
# ReportByDevice false
# ReportReserved false
# ReportInodes false
#</Plugin>
#<Plugin disk>
# Disk "hda"
# Disk "/sda[23]/"
# IgnoreSelected false
#</Plugin>
#<Plugin dns>
# Interface "eth0"
# IgnoreSource "192.168.0.1"
# SelectNumericQueryTypes false
#</Plugin>
#<Plugin email>
# SocketFile "/var/run/collectd-email"
# SocketGroup "collectd"
# SocketPerms "0770"
# MaxConns 5
#</Plugin>
#<Plugin ethstat>
# Interface "eth0"
# Map "rx_csum_offload_errors" "if_rx_errors" "checksum_offload"
# Map "multicast" "if_multicast"
# MappedOnly false
#</Plugin>
#<Plugin exec>
# Exec user "/path/to/exec"
# Exec "user:group" "/path/to/exec"
# NotificationExec user "/path/to/exec"
#</Plugin>
#<Plugin filecount>
# <Directory "/path/to/dir">
# Instance "foodir"
# Name "*.conf"
# MTime "-5m"
# Size "+10k"
# Recursive true
# IncludeHidden false
# </Directory>
#</Plugin>
#<Plugin gmond>
# MCReceiveFrom "239.2.11.71" "8649"
#
# <Metric "swap_total">
# Type "swap"
# TypeInstance "total"
# DataSource "value"
# </Metric>
#
# <Metric "swap_free">
# Type "swap"
# TypeInstance "free"
# DataSource "value"
# </Metric>
#</Plugin>
#<Plugin hddtemp>
# Host "127.0.0.1"
# Port 7634
#</Plugin>
#<Plugin interface>
# Interface "eth0"
# IgnoreSelected false
#</Plugin>
#<Plugin ipmi>
# Sensor "some_sensor"
# Sensor "another_one"
# IgnoreSelected false
# NotifySensorAdd false
# NotifySensorRemove true
# NotifySensorNotPresent false
#</Plugin>
#<Plugin iptables>
# Chain "table" "chain"
#</Plugin>
#<Plugin irq>
# Irq 7
# Irq 8
# Irq 9
# IgnoreSelected true
#</Plugin>
#<Plugin java>
# JVMArg "-verbose:jni"
# JVMArg "-Djava.class.path=/usr/share/collectd/java/collectd-api.jar"
#
# LoadPlugin "org.collectd.java.GenericJMX"
# <Plugin "GenericJMX">
# # See /usr/share/doc/collectd/examples/GenericJMX.conf
# # for an example config.
# </Plugin>
#</Plugin>
#<Plugin libvirt>
# Connection "xen:///"
# RefreshInterval 60
# Domain "name"
# BlockDevice "name:device"
# InterfaceDevice "name:device"
# IgnoreSelected false
# HostnameFormat name
# InterfaceFormat name
#</Plugin>
#<Plugin madwifi>
# Interface "wlan0"
# IgnoreSelected false
# Source "SysFS"
# WatchSet "None"
# WatchAdd "node_octets"
# WatchAdd "node_rssi"
# WatchAdd "is_rx_acl"
# WatchAdd "is_scan_active"
#</Plugin>
#<Plugin mbmon>
# Host "127.0.0.1"
# Port 411
#</Plugin>
#<Plugin md>
# Device "/dev/md0"
# IgnoreSelected false
#</Plugin>
#<Plugin memcachec>
# <Page "plugin_instance">
# Server "localhost"
# Key "page_key"
# <Match>
# Regex "(\\d+) bytes sent"
# ExcludeRegex "<lines to be excluded>"
# DSType CounterAdd
# Type "ipt_octets"
# Instance "type_instance"
# </Match>
# </Page>
#</Plugin>
#<Plugin memcached>
# Socket "/var/run/memcached.sock"
# or:
# Host "127.0.0.1"
# Port "11211"
#</Plugin>
#<Plugin mysql>
# <Database db_name>
# Host "database.serv.er"
# Port "3306"
# User "db_user"
# Password "secret"
# Database "db_name"
# MasterStats true
# </Database>
#
# <Database db_name2>
# Host "localhost"
# Socket "/var/run/mysql/mysqld.sock"
# SlaveStats true
# SlaveNotifications true
# </Database>
#</Plugin>
#<Plugin netlink>
# Interface "All"
# VerboseInterface "All"
# QDisc "eth0" "pfifo_fast-1:0"
# Class "ppp0" "htb-1:10"
# Filter "ppp0" "u32-1:0"
# IgnoreSelected false
#</Plugin>
#<Plugin network>
# # client setup:
# Server "ff18::efc0:4a42" "25826"
# <Server "239.192.74.66" "25826">
# SecurityLevel Encrypt
# Username "user"
# Password "secret"
# Interface "eth0"
# </Server>
# TimeToLive "128"
#
# # server setup:
# Listen "0.0.0.0" "25826"
# <Listen "239.192.74.66" "25826">
# SecurityLevel Sign
# AuthFile "/etc/collectd/passwd"
# Interface "eth0"
# </Listen>
# MaxPacketSize 1024
#
# # proxy setup (client and server as above):
# Forward true
#
# # statistics about the network plugin itself
# ReportStats false
#
# # "garbage collection"
# CacheFlush 1800
#</Plugin>
#<Plugin nginx>
# URL "http://localhost/status?auto"
# User "www-user"
# Password "secret"
# VerifyPeer false
# VerifyHost false
# CACert "/etc/ssl/ca.crt"
#</Plugin>
#<Plugin notify_desktop>
# OkayTimeout 1000
# WarningTimeout 5000
# FailureTimeout 0
#</Plugin>
#<Plugin notify_email>
# SMTPServer "localhost"
# SMTPPort 25
# SMTPUser "my-username"
# SMTPPassword "my-password"
# From "collectd@main0server.com"
# # <WARNING/FAILURE/OK> on <hostname>.
# # Beware! Do not use not more than two placeholders (%)!
# Subject "[collectd] %s on %s!"
# Recipient "email1@domain1.net"
# Recipient "email2@domain2.com"
#</Plugin>
#<Plugin ntpd>
# Host "localhost"
# Port 123
# ReverseLookups false
#</Plugin>
#<Plugin nut>
# UPS "upsname@hostname:port"
#</Plugin>
#<Plugin olsrd>
# Host "127.0.0.1"
# Port "2006"
# CollectLinks "Summary"
# CollectRoutes "Summary"
# CollectTopology "Summary"
#</Plugin>
#<Plugin openvpn>
# StatusFile "/etc/openvpn/openvpn-status.log"
# ImprovedNamingSchema false
# CollectCompression true
# CollectIndividualUsers true
# CollectUserCount false
#</Plugin>
#<Plugin perl>
# IncludeDir "/my/include/path"
# BaseName "Collectd::Plugins"
# EnableDebugger ""
# LoadPlugin Monitorus
# LoadPlugin OpenVZ
#
# <Plugin foo>
# Foo "Bar"
# Qux "Baz"
# </Plugin>
#</Plugin>
#<Plugin pinba>
# Address "::0"
# Port "30002"
# <View "name">
# Host "host name"
# Server "server name"
# Script "script name"
# <View>
#</Plugin>
#<Plugin ping>
# Host "host.foo.bar"
# Host "host.baz.qux"
# Interval 1.0
# Timeout 0.9
# TTL 255
# SourceAddress "1.2.3.4"
# Device "eth0"
# MaxMissed -1
#</Plugin>
#<Plugin postgresql>
# <Query magic>
# Statement "SELECT magic FROM wizard WHERE host = $1;"
# Param hostname
#
# <Result>
# Type gauge
# InstancePrefix "magic"
# ValuesFrom "magic"
# </Result>
# </Query>
#
# <Query rt36_tickets>
# Statement "SELECT COUNT(type) AS count, type \
# FROM (SELECT CASE \
# WHEN resolved = 'epoch' THEN 'open' \
# ELSE 'resolved' END AS type \
# FROM tickets) type \
# GROUP BY type;"
#
# <Result>
# Type counter
# InstancePrefix "rt36_tickets"
# InstancesFrom "type"
# ValuesFrom "count"
# </Result>
# </Query>
#
# <Database foo>
# Host "hostname"
# Port 5432
# User "username"
# Password "secret"
#
# SSLMode "prefer"
# KRBSrvName "kerberos_service_name"
#
# Query magic
# </Database>
#
# <Database bar>
# Interval 60
# Service "service_name"
#
# Query backend # predefined
# Query rt36_tickets
# </Database>
#</Plugin>
#<Plugin powerdns>
# <Server "server_name">
# Collect "latency"
# Collect "udp-answers" "udp-queries"
# Socket "/var/run/pdns.controlsocket"
# </Server>
# <Recursor "recursor_name">
# Collect "questions"
# Collect "cache-hits" "cache-misses"
# Socket "/var/run/pdns_recursor.controlsocket"
# </Recursor>
# LocalSocket "/opt/collectd/var/run/collectd-powerdns"
#</Plugin>
#<Plugin processes>
# Process "name"
# ProcessMatch "foobar" "/usr/bin/perl foobar\\.pl.*"
#</Plugin>
#<Plugin protocols>
# Value "/^Tcp:/"
# IgnoreSelected false
#</Plugin>
#<Plugin python>
# ModulePath "/path/to/your/python/modules"
# LogTraces true
# Interactive true
# Import "spam"
#
# <Module spam>
# spam "wonderful" "lovely"
# </Module>
#</Plugin>
#<Plugin rrdcached>
# DaemonAddress "unix:/var/run/rrdcached.sock"
# DataDir "/var/lib/rrdcached/db/collectd"
# CreateFiles true
# CollectStatistics true
#</Plugin>
<Plugin rrdtool>
DataDir "/var/lib/collectd/rrd"
# CacheTimeout 120
# CacheFlush 900
# WritesPerSecond 30
# RandomTimeout 0
#
# The following settings are rather advanced
# and should usually not be touched:
# StepSize 10
# HeartBeat 20
# RRARows 1200
# RRATimespan 158112000
# XFF 0.1
</Plugin>
#<Plugin sensors>
# SensorConfigFile "/etc/sensors3.conf"
# Sensor "it8712-isa-0290/temperature-temp1"
# Sensor "it8712-isa-0290/fanspeed-fan3"
# Sensor "it8712-isa-0290/voltage-in8"
# IgnoreSelected false
#</Plugin>
# See /usr/share/doc/collectd/examples/snmp-data.conf.gz for a
# comprehensive sample configuration.
#<Plugin snmp>
# <Data "powerplus_voltge_input">
# Type "voltage"
# Table false
# Instance "input_line1"
# Scale 0.1
# Values "SNMPv2-SMI::enterprises.6050.5.4.1.1.2.1"
# </Data>
# <Data "hr_users">
# Type "users"
# Table false
# Instance ""
# Shift -1
# Values "HOST-RESOURCES-MIB::hrSystemNumUsers.0"
# </Data>
# <Data "std_traffic">
# Type "if_octets"
# Table true
# InstancePrefix "traffic"
# Instance "IF-MIB::ifDescr"
# Values "IF-MIB::ifInOctets" "IF-MIB::ifOutOctets"
# </Data>
#
# <Host "some.switch.mydomain.org">
# Address "192.168.0.2"
# Version 1
# Community "community_string"
# Collect "std_traffic"
# Inverval 120
# </Host>
# <Host "some.server.mydomain.org">
# Address "192.168.0.42"
# Version 2
# Community "another_string"
# Collect "std_traffic" "hr_users"
# </Host>
# <Host "some.ups.mydomain.org">
# Address "192.168.0.3"
# Version 1
# Community "more_communities"
# Collect "powerplus_voltge_input"
# Interval 300
# </Host>
#</Plugin>
#<Plugin swap>
# ReportByDevice false
#</Plugin>
#<Plugin table>
# <Table "/proc/slabinfo">
# Instance "slabinfo"
# Separator " "
# <Result>
# Type gauge
# InstancePrefix "active_objs"
# InstancesFrom 0
# ValuesFrom 1
# </Result>
# <Result>
# Type gauge
# InstancePrefix "objperslab"
# InstancesFrom 0
# ValuesFrom 4
# </Result>
# </Table>
#</Plugin>
#<Plugin "tail">
# <File "/var/log/exim4/mainlog">
# Instance "exim"
# <Match>
# Regex "S=([1-9][0-9]*)"
# DSType "CounterAdd"
# Type "ipt_bytes"
# Instance "total"
# </Match>
# <Match>
# Regex "\\<R=local_user\\>"
# ExcludeRegex "\\<R=local_user\\>.*mail_spool defer"
# DSType "CounterInc"
# Type "counter"
# Instance "local_user"
# </Match>
# </File>
#</Plugin>
<Plugin tcpconns>
LocalPort "4001"
LocalPort "7001"
</Plugin>
#<Plugin teamspeak2>
# Host "127.0.0.1"
# Port "51234"
# Server "8767"
#</Plugin>
#<Plugin ted>
# Device "/dev/ttyUSB0"
# Retries 0
#</Plugin>
#<Plugin thermal>
# ForceUseProcfs false
# Device "THRM"
# IgnoreSelected false
#</Plugin>
#<Plugin tokyotyrant>
# Host "localhost"
# Port "1978"
#</Plugin>
#<Plugin unixsock>
# SocketFile "/var/run/collectd-unixsock"
# SocketGroup "collectd"
# SocketPerms "0660"
# DeleteSocket false
#</Plugin>
#<Plugin uuid>
# UUIDFile "/etc/uuid"
#</Plugin>
#<Plugin varnish>
# <Instance>
# CollectCache true
# CollectBackend true
# CollectConnections true
# CollectSHM true
# CollectESI false
# CollectFetch false
# CollectHCB false
# CollectSMA false
# CollectSMS false
# CollectSM false
# CollectTotals false
# CollectWorkers false
# </Instance>
#
# <Instance "myinstance">
# CollectCache true
# </Instance>
#</Plugin>
#<Plugin vmem>
# Verbose false
#</Plugin>
#<Plugin write_graphite>
# <Carbon>
# Host "127.0.01"
# Port "2003"
# Prefix "collectd"
# Postfix "collectd"
# StoreRates false
# AlwaysAppendDS false
# EscapeCharacter "_"
# </Carbon>
#</Plugin>
#<Plugin write_http>
# <URL "http://example.com/collectd-post">
# User "collectd"
# Password "secret"
# VerifyPeer true
# VerifyHost true
# CACert "/etc/ssl/ca.crt"
# Format "Command"
# StoreRates false
# </URL>
#</Plugin>
#<Plugin write_mongodb>
# <Node "example">
# Host "localhost"
# Port "27017"
# Timeout 1000
# StoreRates false
# <Node>
#</Plugin>
Include "/etc/collectd/filters.conf"
Include "/etc/collectd/thresholds.conf"

View File

@ -0,0 +1,31 @@
from stackbrew/ubuntu:precise
run echo 'deb http://us.archive.ubuntu.com/ubuntu/ precise universe' >> /etc/apt/sources.list
run apt-get -y update
# Install required packages
run apt-get -y install python-cairo python-django python-twisted python-django-tagging python-simplejson python-pysqlite2 python-support python-pip gunicorn supervisor nginx-light
run pip install whisper
run pip install --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/lib" carbon
run pip install --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/webapp" graphite-web
# Add system service config
add ./nginx.conf /etc/nginx/nginx.conf
add ./supervisord.conf /etc/supervisor/conf.d/supervisord.conf
# Add graphite config
add ./initial_data.json /var/lib/graphite/webapp/graphite/initial_data.json
add ./local_settings.py /var/lib/graphite/webapp/graphite/local_settings.py
add ./carbon.conf /var/lib/graphite/conf/carbon.conf
add ./storage-schemas.conf /var/lib/graphite/conf/storage-schemas.conf
run mkdir -p /var/lib/graphite/storage/whisper
run touch /var/lib/graphite/storage/graphite.db /var/lib/graphite/storage/index
run chown -R www-data /var/lib/graphite/storage
run chmod 0775 /var/lib/graphite/storage /var/lib/graphite/storage/whisper
run chmod 0664 /var/lib/graphite/storage/graphite.db
run cd /var/lib/graphite/webapp/graphite && python manage.py syncdb --noinput
expose :80
expose :2003
cmd ["/usr/bin/supervisord"]

7
contrib/graphite/README Normal file
View File

@ -0,0 +1,7 @@
Running graphite under Docker is straightforward:
1. Build the graphite image using Docker
docker build -t graphite .
2. Run a graphite container. Be sure to replace the $IP field with the IP address at which you wish to expose your graphite web service.
docker run -p $IP:8080:80 -p $IP:2003:2003 -d graphite

View File

@ -0,0 +1,62 @@
[cache]
LOCAL_DATA_DIR = /var/lib/graphite/storage/whisper/
# Specify the user to drop privileges to
# If this is blank carbon runs as the user that invokes it
# This user must have write access to the local data directory
USER =
# Limit the size of the cache to avoid swapping or becoming CPU bound.
# Sorts and serving cache queries gets more expensive as the cache grows.
# Use the value "inf" (infinity) for an unlimited cache size.
MAX_CACHE_SIZE = inf
# Limits the number of whisper update_many() calls per second, which effectively
# means the number of write requests sent to the disk. This is intended to
# prevent over-utilizing the disk and thus starving the rest of the system.
# When the rate of required updates exceeds this, then carbon's caching will
# take effect and increase the overall throughput accordingly.
MAX_UPDATES_PER_SECOND = 1000
# Softly limits the number of whisper files that get created each minute.
# Setting this value low (like at 50) is a good way to ensure your graphite
# system will not be adversely impacted when a bunch of new metrics are
# sent to it. The trade off is that it will take much longer for those metrics'
# database files to all get created and thus longer until the data becomes usable.
# Setting this value high (like "inf" for infinity) will cause graphite to create
# the files quickly but at the risk of slowing I/O down considerably for a while.
MAX_CREATES_PER_MINUTE = inf
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
#PICKLE_RECEIVER_INTERFACE = 0.0.0.0
#PICKLE_RECEIVER_PORT = 2004
#CACHE_QUERY_INTERFACE = 0.0.0.0
#CACHE_QUERY_PORT = 7002
LOG_UPDATES = False
# Enable AMQP if you want to receve metrics using an amqp broker
# ENABLE_AMQP = False
# Verbose means a line will be logged for every metric received
# useful for testing
# AMQP_VERBOSE = False
# AMQP_HOST = localhost
# AMQP_PORT = 5672
# AMQP_VHOST = /
# AMQP_USER = guest
# AMQP_PASSWORD = guest
# AMQP_EXCHANGE = graphite
# Patterns for all of the metrics this machine will store. Read more at
# http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol#Bindings
#
# Example: store all sales, linux servers, and utilization metrics
# BIND_PATTERNS = sales.#, servers.linux.#, #.utilization
#
# Example: store everything
# BIND_PATTERNS = #

View File

@ -0,0 +1,20 @@
[
{
"pk": 1,
"model": "auth.user",
"fields": {
"username": "admin",
"first_name": "",
"last_name": "",
"is_active": true,
"is_superuser": true,
"is_staff": true,
"last_login": "2011-09-20 17:02:14",
"groups": [],
"user_permissions": [],
"password": "sha1$1b11b$edeb0a67a9622f1f2cfeabf9188a711f5ac7d236",
"email": "root@example.com",
"date_joined": "2011-09-20 17:02:14"
}
}
]

View File

@ -0,0 +1 @@
TIME_ZONE = 'UTC'

View File

@ -0,0 +1,69 @@
daemon off;
user www-data;
worker_processes 1;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
server_tokens off;
server_names_hash_bucket_size 32;
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
gzip on;
gzip_disable "msie6";
server {
listen 80 default_server;
server_name _;
open_log_file_cache max=1000 inactive=20s min_uses=2 valid=1m;
location / {
proxy_pass http://127.0.0.1:8000;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header Host $host;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
}
add_header Access-Control-Allow-Origin "*";
add_header Access-Control-Allow-Methods "GET, OPTIONS";
add_header Access-Control-Allow-Headers "origin, authorization, accept";
location /content {
alias /var/lib/graphite/webapp/content;
}
location /media {
alias /usr/share/pyshared/django/contrib/admin/media;
}
}
}

View File

@ -0,0 +1,7 @@
[carbon]
pattern = ^carbon\..*
retentions = 1m:31d,10m:1y,1h:5y
[default]
pattern = .*
retentions = 10s:8d,1m:31d,10m:1y,1h:5y

View File

@ -0,0 +1,25 @@
[supervisord]
nodaemon = true
environment = GRAPHITE_STORAGE_DIR='/var/lib/graphite/storage',GRAPHITE_CONF_DIR='/var/lib/graphite/conf'
[program:nginx]
command = /usr/sbin/nginx
stdout_logfile = /var/log/supervisor/%(program_name)s.log
stderr_logfile = /var/log/supervisor/%(program_name)s.log
autorestart = true
[program:carbon-cache]
user = www-data
command = /var/lib/graphite/bin/carbon-cache.py --debug start
stdout_logfile = /var/log/supervisor/%(program_name)s.log
stderr_logfile = /var/log/supervisor/%(program_name)s.log
autorestart = true
[program:graphite-webapp]
user = www-data
directory = /var/lib/graphite/webapp
environment = PYTHONPATH='/var/lib/graphite/webapp'
command = /usr/bin/gunicorn_django -b127.0.0.1:8000 -w2 graphite/settings.py
stdout_logfile = /var/log/supervisor/%(program_name)s.log
stderr_logfile = /var/log/supervisor/%(program_name)s.log
autorestart = true