Compare commits

..

15 Commits

Author SHA1 Message Date
Yicheng Qin
d6523fe463 bump to v0.4.7 2015-02-10 15:59:33 -08:00
Yicheng Qin
c25127a699 Merge pull request #2262 from yichengq/047
server: forbid /v2/stats/leader on follower
2015-02-09 22:29:48 -08:00
Yicheng Qin
9f031e6218 server: forbid /v2/stats/leader on follower 2015-02-09 14:50:34 -08:00
Yicheng Qin
e55724e959 Merge pull request #2260 from yichengq/047
server: refresh commit index when someone rejoins
2015-02-09 14:35:00 -08:00
Yicheng Qin
29af192e3d server: refresh commit index when someone rejoins
Update commit index when rejoin happens, because 2.0 doesn't accept
more than one uncommitted config entry.
2015-02-09 14:28:30 -08:00
Yicheng Qin
2fc79912c2 Merge pull request #2194 from yichengq/o3
server: standby exits when detecting v2 is running
2015-01-30 09:32:19 -08:00
Yicheng Qin
ebb8d781b5 server: standby exits when detecting v2 is running 2015-01-29 16:26:47 -08:00
Xiang Li
2e30b3c17f Merge pull request #2130 from xiang90/047-version
server: add internal version
2015-01-22 15:23:38 -08:00
Xiang Li
9a2d82854e server: add internal version 2015-01-22 15:23:15 -08:00
Xiang Li
b077dcf6c4 Merge pull request #2125 from xiang90/047-handler
next-version-handler
2015-01-22 11:40:54 -08:00
Yicheng Qin
2b572cb6e8 Merge pull request #2126 from yichengq/o1
etcd: register usable versions when bootstrap
2015-01-22 11:32:23 -08:00
Xiang Li
f36d55f062 next-version-handler 2015-01-22 11:20:52 -08:00
Yicheng Qin
9f70568a02 etcd: register usable versions when bootstrap 2015-01-22 11:08:58 -08:00
Xiang Li
1ca7d1e064 Merge pull request #2124 from xiang90/047-version
server: add version monitoring
2015-01-22 10:38:27 -08:00
Xiang Li
4f1f003d04 server: add version monitoring 2015-01-22 10:23:15 -08:00
784 changed files with 79083 additions and 29437 deletions

View File

@@ -1 +0,0 @@
.git

5
.gitignore vendored
View File

@@ -1,11 +1,6 @@
/coverage
/gopath
/go-bindata
/machine*
/bin
.vagrant
*.etcd
/etcd
*.swp
/hack/insta-discovery/.env
*.test

View File

@@ -1,10 +0,0 @@
language: go
go:
- 1.3
install:
- go get code.google.com/p/go.tools/cmd/cover
- go get code.google.com/p/go.tools/cmd/vet
script:
- ./test

View File

@@ -21,13 +21,12 @@ This is a rough outline of what a contributor's workflow looks like:
- Make sure your commit messages are in the proper format (see below).
- Push your changes to a topic branch in your fork of the repository.
- Submit a pull request to coreos/etcd.
- Your PR must receive a LGTM from two maintainers found in the MAINTAINERS file.
Thanks for your contributions!
### Code style
The coding style suggested by the Golang community is used in etcd. See the [style doc](https://code.google.com/p/go-wiki/wiki/CodeReviewComments) for details.
The coding style suggested by the Golang community is used in etcd. See [style doc](https://code.google.com/p/go-wiki/wiki/Style) for details.
Please follow this style to make etcd easy to review, maintain and develop.

View File

@@ -1,2 +1,12 @@
FROM golang:onbuild
EXPOSE 4001 7001 2379 2380
FROM ubuntu:12.04
# Let's install go just like Docker (from source).
RUN apt-get update -q
RUN DEBIAN_FRONTEND=noninteractive apt-get install -qy build-essential curl git
RUN curl -s https://storage.googleapis.com/golang/go1.3.src.tar.gz | tar -v -C /usr/local -xz
RUN cd /usr/local/go/src && ./make.bash --no-clean 2>&1
ENV PATH /usr/local/go/bin:$PATH
ADD . /opt/etcd
RUN cd /opt/etcd && ./build
EXPOSE 4001 7001
ENTRYPOINT ["/opt/etcd/bin/etcd"]

View File

@@ -1,33 +0,0 @@
## Administration
### Data Directory
#### Lifecycle
When first started, etcd stores its configuration into a data directory specified by the data-dir configuration parameter.
Configuration is stored in the write ahead log and includes: the local member ID, cluster ID, and initial cluster configuration.
The write ahead log and snapshot files are used during member operation and to recover after a restart.
If a members data directory is ever lost or corrupted then the user should remove the etcd member from the cluster via the [members API][members-api].
A user should avoid restarting an etcd member with a data directory from an out-of-date backup.
Using an out-of-date data directory can lead to inconsistency as the member had agreed to store information via raft then re-joins saying it needs that information again.
For maximum safety, if an etcd member suffers any sort of data corruption or loss, it must be removed from the cluster.
Once removed the member can be re-added with an empty data directory.
[members-api]: https://github.com/coreos/etcd/blob/master/Documentation/0.5/other_apis.md#members-api
#### Contents
The data directory has two sub-directories in it:
1. wal: write ahead log files are stored here. For details see the [wal package documentation][wal-pkg]
2. snap: log snapshots are stored here. For details see the [snap package documentation][snap-pkg]
[wal-pkg]: http://godoc.org/github.com/coreos/etcd/wal
[snap-pkg]: http://godoc.org/github.com/coreos/etcd/snap
### Cluster Lifecycle
If you are spinning up multiple clusters for testing it is recommended that you specify a unique initial-cluster-token for the different clusters.
This can protect you from cluster corruption in case of mis-configuration because two members started with different cluster tokens will refuse members from each other.

File diff suppressed because it is too large Load Diff

View File

@@ -1,204 +0,0 @@
# Clustering Guide
This guide will walk you through configuring a three machine etcd cluster with
the following details:
|Name |Address |
|-------|---------------|
|infra0 |10.0.1.10 |
|infra1 |10.0.1.11 |
|infra2 |10.0.1.12 |
## Static
As we know the cluster members, their addresses and the size of the cluster
before starting we can use an offline bootstrap configuration. Each machine
will get either the following command line or environment variables:
```
ETCD_INITIAL_CLUSTER=”infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379”
ETCD_INITIAL_CLUSTER_STATE=new
```
```
-initial-cluster infra0=http://10.0.1.10:2379,http://10.0.1.11:2379,infra2=http://10.0.1.12:2379 \
-initial-cluster-state new
```
If you are spinning up multiple clusters (or creating and destroying a single cluster) with same configuration for testing purpose, it is highly recommended that you specify a unique `initial-cluster-token` for the different clusters.
By doing this, etcd can generate unique cluster IDs and member IDs for the clusters even if they otherwise have the exact same configuration. This can protect you from cross-cluster-interaction, which might corrupt your clusters.
On each machine you would start etcd with these flags:
```
$ etcd -name infra0 -initial-advertise-peer-urls https://10.0.1.10:2379 initial-cluster-token etcd-cluster-1\
-initial-cluster infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379 \
-initial-cluster-state new
$ etcd -name infra1 -initial-advertise-peer-urls https://10.0.1.11:2379 initial-cluster-token etcd-cluster-1\
-initial-cluster infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379 \
-initial-cluster-state new
$ etcd -name infra2 -initial-advertise-peer-urls https://10.0.1.12:2379 initial-cluster-token etcd-cluster-1\
-initial-cluster infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379 \
-initial-cluster-state new
```
The command line parameters starting with `-initial-cluster` will be ignored on
subsequent runs of etcd. You are free to remove the environment variables or
command line flags after the initial bootstrap process. If you need to make
changes to the configuration later see our guide on runtime configuration.
### Error Cases
In the following case we have not included our new host in the list of
enumerated nodes. If this is a new cluster, the node must be added to the list
of initial cluster members.
```
$ etcd -name infra1 -initial-advertise-peer-urls http://10.0.1.11:2379 \
-initial-cluster infra0=http://10.0.1.10:2379 \
-initial-cluster-state new
etcd: infra1 not listed in the initial cluster config
exit 1
```
In this case we are attempting to map a node (infra0) on a different address
(127.0.0.1:2379) than its enumerated address in the cluster list
(10.0.1.10:2379). If this node is to listen on multiple addresses, all
addresses must be reflected in the “initial-cluster” configuration directive.
```
$ etcd -name infra0 -initial-advertise-peer-urls http://127.0.0.1:2379 \
-initial-cluster infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379 \
-initial-cluster-state=new
etcd: infra0 has different advertised URLs in the cluster and advertised peer URLs list
exit 1
```
If you configure a peer with a different set of configuration and attempt to
join this cluster you will get a cluster ID mismatch and etcd will exit.
```
$ etcd -name infra3 -initial-advertise-peer-urls http://10.0.1.13:2379 \
-initial-cluster infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra3=http://10.0.1.13:2379 \
-initial-cluster-state=new
etcd: conflicting cluster ID to the target cluster (c6ab534d07e8fcc4 != bc25ea2a74fb18b0). Exiting.
exit 1
```
## Discovery
In a number of cases you might not know the IPs of your cluster peers ahead of
time. This is common when utilizing cloud providers or when your network uses
DHCP. In these cases you can use an existing etcd cluster to bootstrap a new
one. We call this process “discovery”.
### Custom etcd discovery service
Discovery uses an existing cluster to bootstrap itself. If you are using your
own etcd cluster you can create a URL like so:
```
$ curl -X PUT https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83/_config/size -d value=3
```
By setting the size key to the URL, you create a discovery URL with expected-cluster-size of 3.
The URL you will use in this case will be
`https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83`
and the etcd members will use the
`https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83`
directory for registration as they start.
Now we start etcd with those relevant flags for each member:
```
$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2379 -discovery https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83
$ etcd -name infra1 -initial-advertise-peer-urls http://10.0.1.11:2379 -discovery https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83
$ etcd -name infra2 -initial-advertise-peer-urls http://10.0.1.12:2379 -discovery https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83
```
This will cause each member to register itself with the custom etcd discovery service and begin
the cluster once all machines have been registered.
### Public discovery service
If you do not have access to an existing cluster you can use the public discovery
service hosted at discovery.etcd.io. You can create a private discovery URL using the
"new" endpoint like so:
```
$ curl https://discovery.etcd.io/new?size=3
https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
```
This will create the cluster with an initial expected size of 3 members. If you
do not specify a size a default of 3 will be used.
```
ETCD_DISCOVERY=https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
```
```
-discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
```
Now we start etcd with those relevant flags for each member:
```
$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2379 -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
$ etcd -name infra1 -initial-advertise-peer-urls http://10.0.1.11:2379 -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
$ etcd -name infra2 -initial-advertise-peer-urls http://10.0.1.12:2379 -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
```
This will cause each member to register itself with the discovery service and begin
the cluster once all members have been registered.
You can use the environment variable `ETCD_DISCOVERY_PROXY` to cause etcd to use an HTTP proxy to connect to the discovery service.
### Error and Warning Cases
#### Discovery Server Errors
```
$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2379 -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
etcd: error: the cluster doesnt have a size configuration value in https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de/_config
exit 1
```
#### User Errors
```
$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2379 -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
etcd: error: the cluster using discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de has already started with all 5 members
exit 1
```
#### Warnings
This is a harmless warning notifying you that the discovery URL will be
ignored on this machine.
```
$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2379 -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
etcd: warn: ignoring discovery URL: etcd has already been initialized and has a valid log in /var/lib/etcd
```
# 0.4 to 0.5+ Migration Guide
In etcd 0.5 we introduced the ability to listen on more than one address and to
advertise multiple addresses. This makes using etcd easier when you have
complex networking, such as private and public networks on various cloud
providers.
To make understanding this feature easier, we changed the naming of some flags,
but we support the old flags to make the migration from the old to new version
easier.
|Old Flag |New Flag |Migration Behavior |
|-----------------------|-----------------------|---------------------------------------------------------------------------------------|
|-peer-addr |-initial-advertise-peer-urls |If specified, peer-addr will be used as the only peer URL. Error if both flags specified.|
|-addr |-advertise-client-urls |If specified, addr will be used as the only client URL. Error if both flags specified.|
|-peer-bind-addr |-listen-peer-urls |If specified, peer-bind-addr will be used as the only peer bind URL. Error if both flags specified.|
|-bind-addr |-listen-client-urls |If specified, bind-addr will be used as the only client bind URL. Error if both flags specified.|
|-peers |none |Deprecated. The -initial-cluster flag provides a similar concept with different semantics. Please read this guide on cluster startup.|
|-peers-file |none |Deprecated. The -initial-cluster flag provides a similar concept with different semantics. Please read this guide on cluster startup.|

View File

@@ -1,31 +0,0 @@
## Glossary
This document defines the various terms used in etcd documentation, command line and source code.
### Node
Node is an instance of raft state machine.
It has a unique identification, and records other nodes' progress internally when it is the leader.
### Member
Member is an instance of etcd. It hosts a node, and provides service to clients.
### Cluster
Cluster consists of several members.
The node in each member follows raft consensus protocol to replicate logs. Cluster receives proposals from members, commits them and apply to local store.
### Peer
Peer is another member of the same cluster.
### Client
Client is a caller of the cluster's HTTP API.
### Machine (deprecated)
The alternative of Member in etcd before 0.5

View File

@@ -1,70 +0,0 @@
## Members API
### GET /v2/members
Return an HTTP 200 OK response code and a representation of all members in the etcd cluster:
```
Example Request: GET
http://localhost:2379/v2/members
Response formats: JSON
Example Response:
```
```json
{
"members": [
{
"id":"272e204152",
"name":"infra1",
"peerURLs":[
"http://10.0.0.10:2379"
],
"clientURLs":[
"http://10.0.0.10:2380"
]
},
{
"id":"2225373f43",
"name":"infra2",
"peerURLs":[
"http://127.0.0.11:2379"
],
"clientURLs":[
"http://127.0.0.11:2380"
]
},
]
}
```
### POST /v2/members
Add a member to the cluster.
Returns an HTTP 201 response code and the representation of added member with a newly generated a memberID when successful. Returns a string describing the failure condition when unsuccessful.
If the POST body is malformed an HTTP 400 will be returned. If the member exists in the cluster or existed in the cluster at some point in the past an HTTP 500(TODO: fix this) will be returned. If the cluster fails to process the request within timeout an HTTP 500 will be returned, though the request may be processed later.
```
Example Request: POST
http://localhost:2379/v2/members
Body:
{"peerURLs":["http://10.0.0.10:2379"]}
Respose formats: JSON
Example Response:
```
```json
{
"id":"3777296169",
"peerURLs":[
"http://10.0.0.10:2379"
],
}
```
### DELETE /v2/members/:id
Remove a member from the cluster.
Returns empty when successful. Returns a string describing the failure condition when unsuccessful.
If the member does not exist in the cluster an HTTP 500(TODO: fix this) will be returned. If the cluster fails to process the request within timeout an HTTP 500 will be returned, though the request may be processed later.
```
Response formats: JSON
Example Request: DELETE
http://localhost:2379/v2/members/272e204152
Example Response: Empty
```

View File

@@ -1,139 +0,0 @@
## Runtime Reconfiguration
etcd comes with support for incremental runtime reconfiguration, which allows users to update the membership of the cluster at run time.
## Reconfiguration Use Cases
Let us walk through the four use cases for re-configuring a cluster: replacing a member, increasing or decreasing cluster size, and restarting a cluster from a majority failure.
### Replace a Member
The most common use case of cluster reconfiguration is to replace a member because of a permanent failure of the existing member: for example, hardware failure, loss of network address, or data directory corruption.
It is important to replace failed members as soon as the failure is detected.
If etcd falls below a simple majority of members it can no longer accept writes: e.g. in a 3 member cluster the loss of two members will cause writes to fail and the cluster to stop operating.
### Increase Cluster Size
To make your cluster more resilient to machine failure you can increase the size of the cluster.
For example, if the cluster consists of three machines, it can tolerate one failure.
If we increase the cluster size to five, it can tolerate two machine failures.
Increasing the cluster size can also provide better read performance.
When a client accesses etcd, the normal read gets the data from the local copy of each member (members always shares the same view of the cluster at the same index, which is guaranteed by the sequential consistency of etcd).
Since clients can read from any member, increasing the number of members thus increases overall read throughput.
### Decrease Cluster Size
To improve the write performance of a cluster, you might want to trade off resilience by removing members.
etcd replicates the data to the majority of members of the cluster before committing the write.
Decreasing the cluster size means the etcd cluster has to do less work for each write, thus increasing the write performance.
### Restart Cluster from Majority Failure
If the majority of your cluster is lost, then you need to take manual action in order to recover safely.
The basic steps in the recovery process include creating a new cluster using the old data, forcing a single member to act as the leader, and finally using runtime configuration to add members to this new cluster.
TODO: https://github.com/coreos/etcd/issues/1242
## Cluster Reconfiguration Operations
Now that we have the use cases in mind, let us lay out the operations involved in each.
Before making any change, the simple majority (quorum) of etcd members must be available.
This is essentially the same requirement as for any other write to etcd.
All changes to the cluster are done one at a time:
To replace a single member you will make an add then a remove operation
To increase from 3 to 5 members you will make two add operations
To decrease from 5 to 3 you will make two remove operations
All of these examples will use the `etcdctl` command line tool that ships with etcd.
If you want to use the member API directly you can find the documentation [here](https://github.com/coreos/etcd/blob/master/Documentation/0.5/other_apis.md).
### Remove a Member
First, we need to find the target member:
```
$ etcdctl member list
6e3bd23ae5f1eae0: name=node2 peerURLs=http://localhost:7002 clientURLs=http://127.0.0.1:4002
924e2e83e93f2560: name=node3 peerURLs=http://localhost:7003 clientURLs=http://127.0.0.1:4003
a8266ecf031671f3: name=node1 peerURLs=http://localhost:7001 clientURLs=http://127.0.0.1:4001
```
Let us say the member ID we want to remove is a8266ecf031671f3.
We then use the `remove` command to perform the removal:
```
$ etcdctl member remove a8266ecf031671f3
EOF
```
The target member will stop itself at this point and print out the removal in the log:
```
etcd: this member has been permanently removed from the cluster. Exiting.
```
Removal of the leader is safe, but the cluster will be out of progress for a period of election timeout because it needs to elect the new leader.
### Add a Member
Adding a member is a two step process.
First we have to tell the cluster that it should expect a new member to join, then we will need to start the member with the correct configuration.
Using `etcdctl`, lets tell the cluster about the new member:
```
$ etcdctl member add infra3 http://10.0.1.13:2379
added member 9bf1b35fc7761a23 to cluster
ETCD_NAME="infra3"
ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379,infra3=http://10.0.1.13:2379"
ETCD_INITIAL_CLUSTER_STATE=existing
```
Now start the new etcd process with the relevant flags for the new member:
```
$ export ETCD_NAME="infra3"
$ export ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379,infra3=http://10.0.1.13:2379"
$ export ETCD_INITIAL_CLUSTER_STATE=existing
$ etcd -listen-client-urls http://10.0.1.13:2379 -advertise-client-urls http://10.0.1.13:2379 -listen-peer-urls http://10.0.1.13:2380 -initial-advertise-peer-urls http://10.0.1.13:2380
```
The new member will run as a part of the cluster and immediately begin catching up with the rest of the cluster.
If you are adding multiple members the best practice is to configure the new member, then start the process, then configure the next, and so on.
A common case is increasing a cluster from 1 to 3: if you add one member to a 1-node cluster, the cluster cannot make progress before the new member starts because it needs two members as majority to agree on the consensus.
#### Error Cases
In the following case we have not included our new host in the list of enumerated nodes.
If this is a new cluster, the node must be added to the list of initial cluster members.
```
$ etcd -name infra3 \
-initial-cluster infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379 \
-initial-cluster-state existing
etcdserver: assign ids error: the member count is unequal
exit 1
```
In this case we give a different address (10.0.1.14:2379) to the one that we used to join the cluster (10.0.1.13:2379).
```
$ etcd -name infra4 \
-initial-cluster infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379,infra4=http://10.0.1.14:2379 \
-initial-cluster-state existing
etcdserver: assign ids error: unmatched member while checking PeerURLs
exit 1
```
When we start etcd using the data directory of a removed member, etcd will exit automatically if it connects to any alive member in the cluster:
```
$ etcd
etcd: this member has been permanently removed from the cluster. Exiting.
exit 1
```

View File

@@ -126,10 +126,10 @@ curl -L http://127.0.0.1:4001/v2/keys/message -XPUT -d value="Hello etcd"
"value": "Hello etcd"
},
"prevNode": {
"createdIndex": 2,
"createdIndex": 2
"key": "/message",
"value": "Hello world",
"modifiedIndex": 2
"modifiedIndex": 2,
}
}
```
@@ -206,7 +206,7 @@ If the TTL has expired, the key will have been deleted, and you will be returned
"cause": "/foo",
"errorCode": 100,
"index": 6,
"message": "Key not found"
"message": "Key Not Found"
}
```
@@ -224,7 +224,7 @@ curl -L http://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar -d ttl= -d prevExis
"key": "/foo",
"modifiedIndex": 6,
"value": "bar"
},
}
"prevNode": {
"createdIndex": 5,
"expiration": "2013-12-04T12:01:21.874888581-08:00",
@@ -383,7 +383,7 @@ curl -L http://127.0.0.1:4001/v2/keys/dir -XPUT -d ttl=30 -d dir=true
"createdIndex": 17,
"dir": true,
"expiration": "2013-12-11T10:37:33.689275857-08:00",
"key": "/dir",
"key": "/newdir",
"modifiedIndex": 17,
"ttl": 30
}
@@ -417,7 +417,7 @@ curl 'http://127.0.0.1:4001/v2/keys/dir/asdf?consistent=true&wait=true'
"dir":true,
"modifiedIndex": 17,
"expiration": "2013-12-11T10:39:35.689275857-08:00"
}
},
}
```
@@ -457,7 +457,7 @@ The error code explains the problem:
"cause": "/foo",
"errorCode": 105,
"index": 39776,
"message": "Key already exists"
"message": "Already exists"
}
```
@@ -474,7 +474,7 @@ This will try to compare the previous value of the key and the previous value we
"cause": "[two != one]",
"errorCode": 101,
"index": 8,
"message": "Compare failed"
"message": "Test Failed"
}
```
@@ -643,22 +643,16 @@ We should see the response as an array of items:
```json
{
"action": "get",
    "node": {
     "key": "/",
       "dir": true,
       "nodes": [
         {
             "key": "/foo_dir",
             "dir": true,
             "modifiedIndex": 2,
             "createdIndex": 2
          },
          {
             "key": "/foo",
             "value": "two",
             "modifiedIndex": 1,
             "createdIndex": 1
          }
"node": {
"dir": true,
"key": "/",
"nodes": [
{
"createdIndex": 2,
"dir": true,
"key": "/foo_dir",
"modifiedIndex": 2
}
]
}
}
@@ -673,33 +667,27 @@ curl -L http://127.0.0.1:4001/v2/keys/?recursive=true
```json
{
    "action": "get",
    "node": {
        "key": "/",
        "dir": true,
        "nodes": [
            {
                "key": "/foo_dir",
                "dir": true,
                "nodes": [
                    {
                        "key": "/foo_dir/foo",
                        "value": "bar",
                        "modifiedIndex": 2,
                        "createdIndex": 2
                    }
                ],
                "modifiedIndex": 2,
                "createdIndex": 2
            },
            {
                "key": "/foo",
                "value": "two",
                "modifiedIndex": 1,
                "createdIndex": 1
            }
        ]
    }
"action": "get",
"node": {
"dir": true,
"key": "/",
"nodes": [
{
"createdIndex": 2,
"dir": true,
"key": "/foo_dir",
"modifiedIndex": 2,
"nodes": [
{
"createdIndex": 2,
"key": "/foo_dir/foo",
"modifiedIndex": 2,
"value": "bar"
}
]
}
]
}
}
```

View File

@@ -1,43 +1,46 @@
# Client libraries support matrix for etcd
As etcd features support is really uneven between client libraries, a compatibility matrix can be important.
We will consider in detail only the features of clients supporting the v2 API. Clients still supporting the v1 API *only* are listed below.
## v1-only clients
Clients supporting only the API version 1
- [justinsb/jetcd](https://github.com/justinsb/jetcd) Java
- [transitorykris/etcd-py](https://github.com/transitorykris/etcd-py) Python
- [russellhaering/txetcd](https://github.com/russellhaering/txetcd) Python
- [iconara/etcd-rb](https://github.com/iconara/etcd-rb) Ruby
- [jpfuentes2/etcd-ruby](https://github.com/jpfuentes2/etcd-ruby) Ruby
- [aterreno/etcd-clojure](https://github.com/aterreno/etcd-clojure) Clojure
- [marshall-lee/etcd.erl](https://github.com/marshall-lee/etcd.erl) Erlang
## v2 clients
The v2 API has a lot of features, we will categorize them in a few categories:
- **Language**: The language in which the client library was written.
- **HTTPS Auth**: Support for SSL-certificate based authentication
- **Reconnect**: If the client is able to reconnect automatically to another server if one fails.
- **Mod/Lock**: Support for the locking module
- **Mod/Leader**: Support for the leader election module
- **GET,PUT,POST,DEL Features**: Support for all the modifiers when calling the etcd server with said HTTP method.
### Supported features matrix
| Client| [go-etcd](https://github.com/coreos/go-etcd) | [jetcd](https://github.com/diwakergupta/jetcd) | [python-etcd](https://github.com/jplana/python-etcd) | [python-etcd-client](https://github.com/dsoprea/PythonEtcdClient) | [node-etcd](https://github.com/stianeikeland/node-etcd) | [nodejs-etcd](https://github.com/lavagetto/nodejs-etcd) | [etcd-ruby](https://github.com/ranjib/etcd-ruby) | [etcd-api](https://github.com/jdarcy/etcd-api) | [cetcd](https://github.com/dwwoelfel/cetcd) | [clj-etcd](https://github.com/rthomas/clj-etcd) | [etcetera](https://github.com/drusellers/etcetera)| [Etcd.jl](https://github.com/forio/Etcd.jl) | [p5-etcd](https://metacpan.org/release/Etcd)
| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **HTTPS Auth** | Y | Y | Y | Y | Y | Y | - | - | - | - | - | - | - |
| **Reconnect** | Y | - | Y | Y | - | - | - | Y | - | - | - | - | - |
| **Mod/Lock** | - | - | Y | Y | - | - | - | - | - | - | - | Y | - |
| **Mod/Leader** | - | - | - | Y | - | - | - | - | - | - | - | Y | - |
| **GET Features** | F | B | F | F | F | F | F | B | F | G | F | F | F |
| **PUT Features** | F | B | F | F | F | F | F | G | F | G | F | F | F |
| **POST Features** | F | - | F | F | - | F | F | - | - | - | F | F | F |
| **DEL Features** | F | B | F | F | F | F | F | B | G | B | F | F | F |
**Legend**
**F**: Full support **G**: Good support **B**: Basic support
**Y**: Feature supported **-**: Feature not supported
Sorted alphabetically on language/name
|Client |**Language**|**HTTPS Auth**|**Re-connect**|**GET**|**PUT**|**POST**|**DEL**|**Mod Lock**|**Mod Leader**|
| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|[etcd-api](https://github.com/jdarcy/etcd-api) |C |-|Y|B|G|-|B|-|-|
|[etcdcpp](https://github.com/edwardcapriolo/etcdcpp) |C++ |-|-|F|F|G|-|-|-|
|[cetcd](https://github.com/dwwoelfel/cetcd) |Clojure|-|-|F|F|-|G|-|-|
|[clj-etcd](https://github.com/rthomas/clj-etcd) |Clojure|-|-|G|G|-|B|-|-|
|[etcd-clojure](https://github.com/aterreno/etcd-clojure) |Clojure|-|-|F|F|F|F|-|-|
|[go-etcd](https://github.com/coreos/go-etcd) |go |Y|Y|F|F|F|F|-|-|
|[boon etcd client](https://github.com/boonproject/boon/blob/master/etcd/README.md) |java |Y|Y|F|F|F|F|-|F|
|[etcd4j](https://github.com/jurmous/etcd4j) |java |Y|Y|F|F|F|F|-|-|
|[jetcd](https://github.com/diwakergupta/jetcd) |java |Y|-|B|B|-|B|-|-|
|[jetcd](https://github.com/justinsb/jetcd) |java |-|-|B|B|-|B|-|-|
|[Etcd.jl](https://github.com/forio/Etcd.jl) |Julia |-|-|F|F|F|F|Y|Y|
|[etcetera](https://github.com/drusellers/etcetera) |.net |-|-|F|F|F|F|-|-|
|[node-etcd](https://github.com/stianeikeland/node-etcd) |nodejs |Y|-|F|F|-|F|-|-|
|[nodejs-etcd](https://github.com/lavagetto/nodejs-etcd) |nodejs |Y|-|F|F|F|F|-|-|
|[p5-etcd](https://metacpan.org/release/Etcd) |perl |-|-|F|F|F|F|-|-|
|[python-etcd](https://github.com/jplana/python-etcd) |python |Y|Y|F|F|F|F|Y|-|
|[python-etcd-client](https://github.com/dsoprea/PythonEtcdClient)|python |Y|Y|F|F|F|F|Y|Y|
|[txetcd](https://github.com/russellhaering/txetcd) |python |-|-|G|G|F|G|-|-|
|[etcd-ruby](https://github.com/ranjib/etcd-ruby) |ruby |-|-|F|F|F|F|-|-|

View File

@@ -13,11 +13,11 @@ Please note - at least 3 nodes are required for [cluster availability][optimal-c
## Using discovery.etcd.io
### Create a Discovery URL
### Create a Token
To use the discovery API, you must first create a unique discovery URL for your etcd cluster. Visit [https://discovery.etcd.io/new](https://discovery.etcd.io/new) to create a new discovery URL.
To use the discovery API, you must first create a token for your etcd cluster. Visit [https://discovery.etcd.io/new](https://discovery.etcd.io/new) to create a new token.
You can inspect the list of peers by viewing `https://discovery.etcd.io/<cluster id>`.
You can inspect the list of peers by viewing `https://discovery.etcd.io/<token>`.
### Start etcd With the Discovery Flag
@@ -26,10 +26,10 @@ Specify the `-discovery` flag when you start each etcd instance. The list of exi
Here's a full example:
```
URL=$(curl https://discovery.etcd.io/new)
./etcd -name instance1 -peer-addr 10.1.2.3:7001 -addr 10.1.2.3:4001 -discovery $URL
./etcd -name instance2 -peer-addr 10.1.2.4:7001 -addr 10.1.2.4:4001 -discovery $URL
./etcd -name instance3 -peer-addr 10.1.2.5:7001 -addr 10.1.2.5:4001 -discovery $URL
TOKEN=$(curl https://discovery.etcd.io/new)
./etcd -name instance1 -peer-addr 10.1.2.3:7001 -addr 10.1.2.3:4001 -discovery $TOKEN
./etcd -name instance2 -peer-addr 10.1.2.4:7001 -addr 10.1.2.4:4001 -discovery $TOKEN
./etcd -name instance3 -peer-addr 10.1.2.5:7001 -addr 10.1.2.5:4001 -discovery $TOKEN
```
## Running Your Own Discovery Endpoint
@@ -37,10 +37,10 @@ URL=$(curl https://discovery.etcd.io/new)
The discovery API communicates with a separate etcd cluster to store and retrieve the list of peers. CoreOS provides [https://discovery.etcd.io](https://discovery.etcd.io) as a free service, but you can easily run your own etcd cluster for this purpose. Here's an example using an etcd cluster located at `10.10.10.10:4001`:
```
URL="http://10.10.10.10:4001/v2/keys/testcluster"
./etcd -name instance1 -peer-addr 10.1.2.3:7001 -addr 10.1.2.3:4001 -discovery $URL
./etcd -name instance2 -peer-addr 10.1.2.4:7001 -addr 10.1.2.4:4001 -discovery $URL
./etcd -name instance3 -peer-addr 10.1.2.5:7001 -addr 10.1.2.5:4001 -discovery $URL
TOKEN="testcluster"
./etcd -name instance1 -peer-addr 10.1.2.3:7001 -addr 10.1.2.3:4001 -discovery http://10.10.10.10:4001/v2/keys/$TOKEN
./etcd -name instance2 -peer-addr 10.1.2.4:7001 -addr 10.1.2.4:4001 -discovery http://10.10.10.10:4001/v2/keys/$TOKEN
./etcd -name instance3 -peer-addr 10.1.2.5:7001 -addr 10.1.2.5:4001 -discovery http://10.10.10.10:4001/v2/keys/$TOKEN
```
If you're interested in how to discovery API works behind the scenes, read about the [Discovery Protocol](https://github.com/coreos/etcd/blob/master/Documentation/discovery-protocol.md).

View File

@@ -167,9 +167,3 @@ Etcd can also do internal server-to-server communication using SSL client certs.
To do this just change the `-*-file` flags to `-peer-*-file`.
If you are using SSL for server-to-server communication, you must use it on all instances of etcd.
### Bootstrapping a new cluster by name
An etcd server is uniquely defined by the peer addresses it listens to. Suppose, however, that you wish to start over, while maintaining the data from the previous cluster -- that is, to pretend that this machine has never joined a cluster before.
You can use `--initial-cluster-name` to generate a new unique ID for each node, as a shared token that every node understands. Nodes also take this into account for bootstrapping the new cluster ID, so it also provides a way for a machine to listen on the same interfaces, disconnect from one cluster, and join a different cluster.

View File

@@ -75,8 +75,8 @@ cors = []
cpu_profile_file = ""
data_dir = "."
discovery = "http://etcd.local:4001/v2/keys/_etcd/registry/examplecluster"
http_read_timeout = 10.0
http_write_timeout = 10.0
http_read_timeout = 10
http_write_timeout = 10
key_file = ""
peers = []
peers_file = ""
@@ -84,7 +84,7 @@ max_cluster_size = 9
max_result_buffer = 1024
max_retry_attempts = 3
name = "default-name"
snapshot = true
snapshot = false
verbose = false
very_verbose = false
@@ -112,8 +112,8 @@ sync_interval = 5.0
* `ETCD_CPU_PROFILE_FILE`
* `ETCD_DATA_DIR`
* `ETCD_DISCOVERY`
* `ETCD_HTTP_READ_TIMEOUT`
* `ETCD_HTTP_WRITE_TIMEOUT`
* `ETCD_CLUSTER_HTTP_READ_TIMEOUT`
* `ETCD_CLUSTER_HTTP_WRITE_TIMEOUT`
* `ETCD_KEY_FILE`
* `ETCD_PEERS`
* `ETCD_PEERS_FILE`

View File

@@ -0,0 +1,12 @@
# Development tools
## Vagrant
For fast start you can use Vagrant. `vagrant up` will make etcd build and running on virtual machine. Required Vagrant version is 1.5.0.
Next lets set a single key and then retrieve it:
```
curl -L http://127.0.0.1:4001/v2/keys/mykey -XPUT -d value="this is awesome"
curl -L http://127.0.0.1:4001/v2/keys/mykey
```

View File

@@ -3,10 +3,8 @@
**Tools**
- [etcdctl](https://github.com/coreos/etcdctl) - A command line client for etcd
- [etcd-backup](https://github.com/fanhattan/etcd-backup) - A powerful command line utility for dumping/restoring etcd - Supports v2
- [etcd-dump](https://npmjs.org/package/etcd-dump) - Command line utility for dumping/restoring etcd.
- [etcd-fs](https://github.com/xetorthio/etcd-fs) - FUSE filesystem for etcd
- [etcd-browser](https://github.com/henszey/etcd-browser) - A web-based key/value editor for etcd using AngularJS
**Go libraries**
@@ -14,12 +12,8 @@
**Java libraries**
- [boonproject/etcd](https://github.com/boonproject/boon/blob/master/etcd/README.md) - Supports v2, Async/Sync and waits
- [justinsb/jetcd](https://github.com/justinsb/jetcd)
- [diwakergupta/jetcd](https://github.com/diwakergupta/jetcd) - Supports v2
- [jurmous/etcd4j](https://github.com/jurmous/etcd4j) - Supports v2
- [AdoHe/etcd4j](http://github.com/AdoHe/etcd4j) - Supports v2 (enhance for real production cluster)
**Python libraries**
@@ -42,9 +36,6 @@
- [jdarcy/etcd-api](https://github.com/jdarcy/etcd-api) - Supports v2
**C++ libraries**
- [edwardcapriolo/etcdcpp](https://github.com/edwardcapriolo/etcdcpp) - Supports v2
**Clojure libraries**
- [aterreno/etcd-clojure](https://github.com/aterreno/etcd-clojure)
@@ -101,6 +92,3 @@ A detailed recap of client functionalities can be found in the [clients compatib
- [GoogleCloudPlatform/kubernetes](https://github.com/GoogleCloudPlatform/kubernetes) - Container cluster manager.
- [mailgun/vulcand](https://github.com/mailgun/vulcand) - HTTP proxy that uses etcd as a configuration backend.
- [duedil-ltd/discodns](https://github.com/duedil-ltd/discodns) - Simple DNS nameserver using etcd as a database for names and records.
- [skynetservices/skydns](https://github.com/skynetservices/skydns) - RFC compliant DNS server
- [xordataexchange/crypt](https://github.com/xordataexchange/crypt) - Securely store values in etcd using GPG encryption
- [spf13/viper](https://github.com/spf13/viper) - Go configuration library, reads values from ENV, pflags, files, and etcd with optional encryption

View File

@@ -1,50 +1,34 @@
# Etcd security model
# Reading and Writing over HTTPS
Etcd supports SSL/TLS as well as authentication through client certificates, both for clients to server as well as peer (server to server / cluster) communication.
## Transport Security with HTTPS
To get up and running you first need to have a CA certificate and a signed key pair for your node. It is recommended to create and sign a new key pair for every node in a cluster.
For convenience the [etcd-ca](https://github.com/coreos/etcd-ca) tool provides an easy interface to certificate generation, alternatively this site provides a good reference on how to generate self-signed key pairs:
Etcd supports SSL/TLS and client cert authentication for clients to server, as well as server to server communication.
First, you need to have a CA cert `clientCA.crt` and signed key pair `client.crt`, `client.key`.
This site has a good reference for how to generate self-signed key pairs:
http://www.g-loaded.eu/2005/11/10/be-your-own-ca/
Or you could use [etcd-ca](https://github.com/coreos/etcd-ca) to generate certs and keys.
## Basic setup
For testing you can use the certificates in the `fixtures/ca` directory.
Etcd takes several certificate related configuration options, either through command-line flags or environment variables:
**Client-to-server communication:**
`--cert-file=<path>`: Certificate used for SSL/TLS connections **to** etcd. When this option is set, you can reach etcd through HTTPS - for example at `https://127.0.0.1:4001`
`--key-file=<path>`: Key for the certificate. Must be unencrypted.
`--ca-file=<path>`: When this is set etcd will check all incoming HTTPS requests for a client certificate signed by the supplied CA, requests that don't supply a valid client certificate will fail.
**Peer (server-to-server / cluster) communication:**
The peer options work the same way as the client-to-server options:
`--peer-cert-file=<path>`: Certificate used for SSL/TLS connections between peers. This will be used both for listening on the peer address as well as sending requests to other peers.
`--peer-key-file=<path>`: Key for the certificate. Must be unencrypted.
`--peer-ca-file=<path>`: When set, etcd will check all incoming peer requests from the cluster for valid client certificates signed by the supplied CA.
If either a client-to-server or peer certificate is supplied the key must also be set. All of these configuration options are also available through the environment variables, `ETCD_CA_FILE`, `ETCD_PEER_CA_FILE` and so on.
## Example 1: Client-to-server transport security with HTTPS
For this you need your CA certificate (`ca.crt`) and signed key pair (`server.crt`, `server.key`) ready. If you just want to test the functionality, there are example certificates provided in the [etcd git repository](https://github.com/coreos/etcd/tree/master/fixtures/ca) (namely `server.crt` and `server.key.insecure`).
Assuming you have these files ready, let's configure etcd to use them to provide simple HTTPS transport security.
Let's configure etcd to use this keypair:
```sh
etcd -name machine0 -data-dir machine0 -cert-file=/path/to/server.crt -key-file=/path/to/server.key
./etcd -f -name machine0 -data-dir machine0 -cert-file=./fixtures/ca/server.crt -key-file=./fixtures/ca/server.key.insecure
```
This should start up fine and you can now test the configuration by speaking HTTPS to etcd:
There are a few new options we're using:
* `-f` - forces a new machine configuration, even if an existing configuration is found. (WARNING: data loss!)
* `-cert-file` and `-key-file` specify the location of the cert and key files to be used for for transport layer security between the client and server.
You can now test the configuration using HTTPS:
```sh
curl --cacert /path/to/ca.crt https://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar -v
curl --cacert ./fixtures/ca/server-chain.pem https://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar -v
```
You should be able to see the handshake succeed. Because we use self-signed certificates with our own certificate authorities you need to provide the CA to curl using the `--cacert` option. Another possibility would be to add your CA certificate to the trusted certificates on your system (usually in `/etc/ssl/certs`).
You should be able to see the handshake succeed.
**OSX 10.9+ Users**: curl 7.30.0 on OSX 10.9+ doesn't understand certificates passed in on the command line.
Instead you must import the dummy ca.crt directly into the keychain or add the `-k` flag to curl to ignore errors.
@@ -52,28 +36,42 @@ If you want to test without the `-k` flag run `open ./fixtures/ca/ca.crt` and fo
Please remove this certificate after you are done testing!
If you know of a workaround let us know.
## Example 2: Client-to-server authentication with HTTPS client certificates
For now we've given the etcd client the ability to verify the server identity and provide transport security. We can however also use client certificates to prevent unauthorized access to etcd.
The clients will provide their certificates to the server and the server will check whether the cert is signed by the supplied CA and decide whether to serve the request.
You need the same files mentioned in the first example for this, as well as a key pair for the client (`client.crt`, `client.key`) signed by the same certificate authority.
```sh
etcd -name machine0 -data-dir machine0 -ca-file=/path/to/ca.crt -cert-file=/path/to/server.crt -key-file=/path/to/server.key
```
...
SSLv3, TLS handshake, Finished (20):
...
```
Notice that the addition of the `-ca-file` option automatically enables client certificate checking.
And also the response from the etcd server:
Now try the same request as above to this server:
```sh
curl --cacert /path/to/ca.crt https://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar -v
```json
{
"action": "set",
"key": "/foo",
"modifiedIndex": 3,
"value": "bar"
}
```
The request should be rejected by the server:
## Authentication with HTTPS Client Certificates
We can also do authentication using CA certs.
The clients will provide their cert to the server and the server will check whether the cert is signed by the CA and decide whether to serve the request.
```sh
./etcd -f -name machine0 -data-dir machine0 -ca-file=./fixtures/ca/ca.crt -cert-file=./fixtures/ca/server.crt -key-file=./fixtures/ca/server.key.insecure
```
```-ca-file``` is the path to the CA cert.
Try the same request to this server:
```sh
curl --cacert ./fixtures/ca/server-chain.pem https://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar -v
```
The request should be rejected by the server.
```
...
@@ -81,10 +79,10 @@ routines:SSL3_READ_BYTES:sslv3 alert bad certificate
...
```
To make it succeed, we need to give the CA signed client certificate to the server:
We need to give the CA signed cert to the server.
```sh
curl --cacert /path/to/ca.crt --cert /path/to/client.crt --key /path/to/client.key -L https://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar -v
curl --key ./fixtures/ca/server2.key.insecure --cert ./fixtures/ca/server2.crt --cacert ./fixtures/ca/server-chain.pem -L https://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar -v
```
You should able to see:
@@ -110,28 +108,7 @@ And also the response from the server:
}
```
## Example 3: Transport security & client certificates in a cluster
Etcd supports the same model as above for **peer communication**, that means the communication between etcd nodes in a cluster.
Assuming we have our `ca.crt` and two nodes with their own keypairs (`node1.crt` & `node1.key`, `node2.crt` & `node2.key`) signed by this CA, we launch etcd as follows:
```sh
DISCOVERY_URL=... # from https://discovery.etcd.io/new
# Node1
etcd -name node1 -data-dir node1 -ca-file=/path/to/ca.crt -cert-file=/path/to/node1.crt -key-file=/path/to/node1.key -peer-addr ${node1_public_ip}:7001 -discovery ${DISCOVERY_URL}
# Node2
etcd -name node1 -data-dir node2 -ca-file=/path/to/ca.crt -cert-file=/path/to/node2.crt -key-file=/path/to/node2.key -peer-addr ${node2_public_ip}:7001 -discovery ${DISCOVERY_URL}
```
The etcd nodes will form a cluster and all communication between nodes in the cluster will be encrypted and authenticated using the client certificates. You will see in the output of etcd that the addresses it connects to use HTTPS.
## Frequently Asked Questions
### I'm seeing a SSLv3 alert handshake failure when using SSL client authentication?
### Why SSLv3 alert handshake failure when using SSL client auth?
The `crypto/tls` package of `golang` checks the key usage of the certificate public key before using it.
To use the certificate public key to do client auth, we need to add `clientAuth` to `Extended Key Usage` when creating the certificate public key.
@@ -152,8 +129,3 @@ When creating the cert be sure to reference it in the `-extensions` flag:
```
openssl ca -config openssl.cnf -policy policy_anything -extensions ssl_client -out certs/machine.crt -infiles machine.csr
```
### With peer certificate authentication I receive "certificate is valid for 127.0.0.1, not $MY_IP"
Make sure that you sign your certificates with a Subject Name your node's public IP address. The `etcd-ca` tool for example provides an `--ip=` option for its `new-cert` command.
If you need your certificate to be signed for your node's FQDN in its Subject Name then you could use Subject Alternative Names (short IP SNAs) to add your IP address. This is not [currently supported](https://github.com/coreos/etcd-ca/issues/29) by `etcd-ca` but can be done [with openssl](http://wiki.cacert.org/FAQ/subjectAltName).

View File

@@ -3,8 +3,6 @@
The default settings in etcd should work well for installations on a local network where the average network latency is low.
However, when using etcd across multiple data centers or over networks with high latency you may need to tweak the heartbeat interval and election timeout settings.
The network isn't the only source of latency. Each request and response may be impacted by slow disks on both the leader and follower. Each of these timeouts represents the total time from request to successful response from the other machine.
### Time Parameters
The underlying distributed consensus protocol relies on two separate time parameters to ensure that nodes can handoff leadership if one stalls or goes offline.
@@ -43,7 +41,7 @@ Or you can set the values within the configuration file:
```toml
[peer]
heartbeat_interval = 100
election_timeout = 500
election_timeout = 100
```
The values are specified in milliseconds.

36
Godeps/Godeps.json generated
View File

@@ -1,36 +0,0 @@
{
"ImportPath": "github.com/coreos/etcd",
"GoVersion": "go1.3.1",
"Packages": [
"./..."
],
"Deps": [
{
"ImportPath": "code.google.com/p/go.net/context",
"Comment": "null-144",
"Rev": "ad01a6fcc8a19d3a4478c836895ffe883bd2ceab"
},
{
"ImportPath": "code.google.com/p/gogoprotobuf/proto",
"Rev": "7fd1620f09261338b6b1ca1289ace83aee0ec946"
},
{
"ImportPath": "github.com/codegangsta/cli",
"Comment": "1.2.0-26-gf7ebb76",
"Rev": "f7ebb761e83e21225d1d8954fde853bf8edd46c4"
},
{
"ImportPath": "github.com/coreos/go-etcd/etcd",
"Comment": "v0.2.0-rc1-127-g6fe04d5",
"Rev": "6fe04d580dfb71c9e34cbce2f4df9eefd1e1241e"
},
{
"ImportPath": "github.com/jonboulle/clockwork",
"Rev": "72f9bd7c4e0c2a40055ab3d0f09654f730cce982"
},
{
"ImportPath": "github.com/stretchr/testify/assert",
"Rev": "9cc77fa25329013ce07362c7742952ff887361f2"
}
]
}

5
Godeps/Readme generated
View File

@@ -1,5 +0,0 @@
This directory tree is generated automatically by godep.
Please do not edit.
See https://github.com/tools/godep for more information.

2
Godeps/_workspace/.gitignore generated vendored
View File

@@ -1,2 +0,0 @@
/pkg
/bin

View File

@@ -1,431 +0,0 @@
// Copyright 2014 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package context defines the Context type, which carries deadlines,
// cancelation signals, and other request-scoped values across API boundaries
// and between processes.
//
// Incoming requests to a server should create a Context, and outgoing calls to
// servers should accept a Context. The chain of function calls between must
// propagate the Context, optionally replacing it with a modified copy created
// using WithDeadline, WithTimeout, WithCancel, or WithValue.
//
// Programs that use Contexts should follow these rules to keep interfaces
// consistent across packages and enable static analysis tools to check context
// propagation:
//
// Do not store Contexts inside a struct type; instead, pass a Context
// explicitly to each function that needs it. The Context should be the first
// parameter, typically named ctx:
//
// func DoSomething(ctx context.Context, arg Arg) error {
// // ... use ctx ...
// }
//
// Do not pass a nil Context, even if a function permits it. Pass context.TODO
// if you are unsure about which Context to use.
//
// Use context Values only for request-scoped data that transits processes and
// APIs, not for passing optional parameters to functions.
//
// The same Context may be passed to functions running in different goroutines;
// Contexts are safe for simultaneous use by multiple goroutines.
//
// See http://blog.golang.org/context for example code for a server that uses
// Contexts.
package context
import (
"errors"
"fmt"
"sync"
"time"
)
// A Context carries a deadline, a cancelation signal, and other values across
// API boundaries.
//
// Context's methods may be called by multiple goroutines simultaneously.
type Context interface {
// Deadline returns the time when work done on behalf of this context
// should be canceled. Deadline returns ok==false when no deadline is
// set. Successive calls to Deadline return the same results.
Deadline() (deadline time.Time, ok bool)
// Done returns a channel that's closed when work done on behalf of this
// context should be canceled. Done may return nil if this context can
// never be canceled. Successive calls to Done return the same value.
//
// WithCancel arranges for Done to be closed when cancel is called;
// WithDeadline arranges for Done to be closed when the deadline
// expires; WithTimeout arranges for Done to be closed when the timeout
// elapses.
//
// Done is provided for use in select statements:
//
// // DoSomething calls DoSomethingSlow and returns as soon as
// // it returns or ctx.Done is closed.
// func DoSomething(ctx context.Context) (Result, error) {
// c := make(chan Result, 1)
// go func() { c <- DoSomethingSlow(ctx) }()
// select {
// case res := <-c:
// return res, nil
// case <-ctx.Done():
// return nil, ctx.Err()
// }
// }
//
// See http://blog.golang.org/pipelines for more examples of how to use
// a Done channel for cancelation.
Done() <-chan struct{}
// Err returns a non-nil error value after Done is closed. Err returns
// Canceled if the context was canceled or DeadlineExceeded if the
// context's deadline passed. No other values for Err are defined.
// After Done is closed, successive calls to Err return the same value.
Err() error
// Value returns the value associated with this context for key, or nil
// if no value is associated with key. Successive calls to Value with
// the same key returns the same result.
//
// Use context values only for request-scoped data that transits
// processes and API boundaries, not for passing optional parameters to
// functions.
//
// A key identifies a specific value in a Context. Functions that wish
// to store values in Context typically allocate a key in a global
// variable then use that key as the argument to context.WithValue and
// Context.Value. A key can be any type that supports equality;
// packages should define keys as an unexported type to avoid
// collisions.
//
// Packages that define a Context key should provide type-safe accessors
// for the values stores using that key:
//
// // Package user defines a User type that's stored in Contexts.
// package user
//
// import "code.google.com/p/go.net/context"
//
// // User is the type of value stored in the Contexts.
// type User struct {...}
//
// // key is an unexported type for keys defined in this package.
// // This prevents collisions with keys defined in other packages.
// type key int
//
// // userKey is the key for user.User values in Contexts. It is
// // unexported; clients use user.NewContext and user.FromContext
// // instead of using this key directly.
// var userKey key = 0
//
// // NewContext returns a new Context that carries value u.
// func NewContext(ctx context.Context, u *User) context.Context {
// return context.WithValue(userKey, u)
// }
//
// // FromContext returns the User value stored in ctx, if any.
// func FromContext(ctx context.Context) (*User, bool) {
// u, ok := ctx.Value(userKey).(*User)
// return u, ok
// }
Value(key interface{}) interface{}
}
// Canceled is the error returned by Context.Err when the context is canceled.
var Canceled = errors.New("context canceled")
// DeadlineExceeded is the error returned by Context.Err when the context's
// deadline passes.
var DeadlineExceeded = errors.New("context deadline exceeded")
// An emptyCtx is never canceled, has no values, and has no deadline.
type emptyCtx int
func (emptyCtx) Deadline() (deadline time.Time, ok bool) {
return
}
func (emptyCtx) Done() <-chan struct{} {
return nil
}
func (emptyCtx) Err() error {
return nil
}
func (emptyCtx) Value(key interface{}) interface{} {
return nil
}
func (n emptyCtx) String() string {
switch n {
case background:
return "context.Background"
case todo:
return "context.TODO"
}
return "unknown empty Context"
}
const (
background emptyCtx = 1
todo emptyCtx = 2
)
// Background returns a non-nil, empty Context. It is never canceled, has no
// values, and has no deadline. It is typically used by the main function,
// initialization, and tests, and as the top-level Context for incoming
// requests.
func Background() Context {
return background
}
// TODO returns a non-nil, empty Context. Code should use context.TODO when
// it's unclear which Context to use or it's is not yet available (because the
// surrounding function has not yet been extended to accept a Context
// parameter). TODO is recognized by static analysis tools that determine
// whether Contexts are propagated correctly in a program.
func TODO() Context {
return todo
}
// A CancelFunc tells an operation to abandon its work.
// A CancelFunc does not wait for the work to stop.
// After the first call, subsequent calls to a CancelFunc do nothing.
type CancelFunc func()
// WithCancel returns a copy of parent with a new Done channel. The returned
// context's Done channel is closed when the returned cancel function is called
// or when the parent context's Done channel is closed, whichever happens first.
func WithCancel(parent Context) (ctx Context, cancel CancelFunc) {
c := newCancelCtx(parent)
propagateCancel(parent, &c)
return &c, func() { c.cancel(true, Canceled) }
}
// newCancelCtx returns an initialized cancelCtx.
func newCancelCtx(parent Context) cancelCtx {
return cancelCtx{
Context: parent,
done: make(chan struct{}),
}
}
// propagateCancel arranges for child to be canceled when parent is.
func propagateCancel(parent Context, child canceler) {
if parent.Done() == nil {
return // parent is never canceled
}
if p, ok := parentCancelCtx(parent); ok {
p.mu.Lock()
if p.err != nil {
// parent has already been canceled
child.cancel(false, p.err)
} else {
if p.children == nil {
p.children = make(map[canceler]bool)
}
p.children[child] = true
}
p.mu.Unlock()
} else {
go func() {
select {
case <-parent.Done():
child.cancel(false, parent.Err())
case <-child.Done():
}
}()
}
}
// parentCancelCtx follows a chain of parent references until it finds a
// *cancelCtx. This function understands how each of the concrete types in this
// package represents its parent.
func parentCancelCtx(parent Context) (*cancelCtx, bool) {
for {
switch c := parent.(type) {
case *cancelCtx:
return c, true
case *timerCtx:
return &c.cancelCtx, true
case *valueCtx:
parent = c.Context
default:
return nil, false
}
}
}
// A canceler is a context type that can be canceled directly. The
// implementations are *cancelCtx and *timerCtx.
type canceler interface {
cancel(removeFromParent bool, err error)
Done() <-chan struct{}
}
// A cancelCtx can be canceled. When canceled, it also cancels any children
// that implement canceler.
type cancelCtx struct {
Context
done chan struct{} // closed by the first cancel call.
mu sync.Mutex
children map[canceler]bool // set to nil by the first cancel call
err error // set to non-nil by the first cancel call
}
func (c *cancelCtx) Done() <-chan struct{} {
return c.done
}
func (c *cancelCtx) Err() error {
c.mu.Lock()
defer c.mu.Unlock()
return c.err
}
func (c *cancelCtx) String() string {
return fmt.Sprintf("%v.WithCancel", c.Context)
}
// cancel closes c.done, cancels each of c's children, and, if
// removeFromParent is true, removes c from its parent's children.
func (c *cancelCtx) cancel(removeFromParent bool, err error) {
if err == nil {
panic("context: internal error: missing cancel error")
}
c.mu.Lock()
if c.err != nil {
c.mu.Unlock()
return // already canceled
}
c.err = err
close(c.done)
for child := range c.children {
// NOTE: acquiring the child's lock while holding parent's lock.
child.cancel(false, err)
}
c.children = nil
c.mu.Unlock()
if removeFromParent {
if p, ok := parentCancelCtx(c.Context); ok {
p.mu.Lock()
if p.children != nil {
delete(p.children, c)
}
p.mu.Unlock()
}
}
}
// WithDeadline returns a copy of the parent context with the deadline adjusted
// to be no later than d. If the parent's deadline is already earlier than d,
// WithDeadline(parent, d) is semantically equivalent to parent. The returned
// context's Done channel is closed when the deadline expires, when the returned
// cancel function is called, or when the parent context's Done channel is
// closed, whichever happens first.
//
// Canceling this context releases resources associated with the deadline
// timer, so code should call cancel as soon as the operations running in this
// Context complete.
func WithDeadline(parent Context, deadline time.Time) (Context, CancelFunc) {
if cur, ok := parent.Deadline(); ok && cur.Before(deadline) {
// The current deadline is already sooner than the new one.
return WithCancel(parent)
}
c := &timerCtx{
cancelCtx: newCancelCtx(parent),
deadline: deadline,
}
propagateCancel(parent, c)
d := deadline.Sub(time.Now())
if d <= 0 {
c.cancel(true, DeadlineExceeded) // deadline has already passed
return c, func() { c.cancel(true, Canceled) }
}
c.mu.Lock()
defer c.mu.Unlock()
if c.err == nil {
c.timer = time.AfterFunc(d, func() {
c.cancel(true, DeadlineExceeded)
})
}
return c, func() { c.cancel(true, Canceled) }
}
// A timerCtx carries a timer and a deadline. It embeds a cancelCtx to
// implement Done and Err. It implements cancel by stopping its timer then
// delegating to cancelCtx.cancel.
type timerCtx struct {
cancelCtx
timer *time.Timer // Under cancelCtx.mu.
deadline time.Time
}
func (c *timerCtx) Deadline() (deadline time.Time, ok bool) {
return c.deadline, true
}
func (c *timerCtx) String() string {
return fmt.Sprintf("%v.WithDeadline(%s [%s])", c.cancelCtx.Context, c.deadline, c.deadline.Sub(time.Now()))
}
func (c *timerCtx) cancel(removeFromParent bool, err error) {
c.cancelCtx.cancel(removeFromParent, err)
c.mu.Lock()
if c.timer != nil {
c.timer.Stop()
c.timer = nil
}
c.mu.Unlock()
}
// WithTimeout returns WithDeadline(parent, time.Now().Add(timeout)).
//
// Canceling this context releases resources associated with the deadline
// timer, so code should call cancel as soon as the operations running in this
// Context complete:
//
// func slowOperationWithTimeout(ctx context.Context) (Result, error) {
// ctx, cancel := context.WithTimeout(ctx, 100*time.Millisecond)
// defer cancel() // releases resources if slowOperation completes before timeout elapses
// return slowOperation(ctx)
// }
func WithTimeout(parent Context, timeout time.Duration) (Context, CancelFunc) {
return WithDeadline(parent, time.Now().Add(timeout))
}
// WithValue returns a copy of parent in which the value associated with key is
// val.
//
// Use context Values only for request-scoped data that transits processes and
// APIs, not for passing optional parameters to functions.
func WithValue(parent Context, key interface{}, val interface{}) Context {
return &valueCtx{parent, key, val}
}
// A valueCtx carries a key-value pair. It implements Value for that key and
// delegates all other calls to the embedded Context.
type valueCtx struct {
Context
key, val interface{}
}
func (c *valueCtx) String() string {
return fmt.Sprintf("%v.WithValue(%#v, %#v)", c.Context, c.key, c.val)
}
func (c *valueCtx) Value(key interface{}) interface{} {
if c.key == key {
return c.val
}
return c.Context.Value(key)
}

View File

@@ -1,553 +0,0 @@
// Copyright 2014 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package context
import (
"fmt"
"math/rand"
"runtime"
"strings"
"sync"
"testing"
"time"
)
// otherContext is a Context that's not one of the types defined in context.go.
// This lets us test code paths that differ based on the underlying type of the
// Context.
type otherContext struct {
Context
}
func TestBackground(t *testing.T) {
c := Background()
if c == nil {
t.Fatalf("Background returned nil")
}
select {
case x := <-c.Done():
t.Errorf("<-c.Done() == %v want nothing (it should block)", x)
default:
}
if got, want := fmt.Sprint(c), "context.Background"; got != want {
t.Errorf("Background().String() = %q want %q", got, want)
}
}
func TestTODO(t *testing.T) {
c := TODO()
if c == nil {
t.Fatalf("TODO returned nil")
}
select {
case x := <-c.Done():
t.Errorf("<-c.Done() == %v want nothing (it should block)", x)
default:
}
if got, want := fmt.Sprint(c), "context.TODO"; got != want {
t.Errorf("TODO().String() = %q want %q", got, want)
}
}
func TestWithCancel(t *testing.T) {
c1, cancel := WithCancel(Background())
if got, want := fmt.Sprint(c1), "context.Background.WithCancel"; got != want {
t.Errorf("c1.String() = %q want %q", got, want)
}
o := otherContext{c1}
c2, _ := WithCancel(o)
contexts := []Context{c1, o, c2}
for i, c := range contexts {
if d := c.Done(); d == nil {
t.Errorf("c[%d].Done() == %v want non-nil", i, d)
}
if e := c.Err(); e != nil {
t.Errorf("c[%d].Err() == %v want nil", i, e)
}
select {
case x := <-c.Done():
t.Errorf("<-c.Done() == %v want nothing (it should block)", x)
default:
}
}
cancel()
time.Sleep(100 * time.Millisecond) // let cancelation propagate
for i, c := range contexts {
select {
case <-c.Done():
default:
t.Errorf("<-c[%d].Done() blocked, but shouldn't have", i)
}
if e := c.Err(); e != Canceled {
t.Errorf("c[%d].Err() == %v want %v", i, e, Canceled)
}
}
}
func TestParentFinishesChild(t *testing.T) {
// Context tree:
// parent -> cancelChild
// parent -> valueChild -> timerChild
parent, cancel := WithCancel(Background())
cancelChild, stop := WithCancel(parent)
defer stop()
valueChild := WithValue(parent, "key", "value")
timerChild, stop := WithTimeout(valueChild, 10000*time.Hour)
defer stop()
select {
case x := <-parent.Done():
t.Errorf("<-parent.Done() == %v want nothing (it should block)", x)
case x := <-cancelChild.Done():
t.Errorf("<-cancelChild.Done() == %v want nothing (it should block)", x)
case x := <-timerChild.Done():
t.Errorf("<-timerChild.Done() == %v want nothing (it should block)", x)
case x := <-valueChild.Done():
t.Errorf("<-valueChild.Done() == %v want nothing (it should block)", x)
default:
}
// The parent's children should contain the two cancelable children.
pc := parent.(*cancelCtx)
cc := cancelChild.(*cancelCtx)
tc := timerChild.(*timerCtx)
pc.mu.Lock()
if len(pc.children) != 2 || !pc.children[cc] || !pc.children[tc] {
t.Errorf("bad linkage: pc.children = %v, want %v and %v",
pc.children, cc, tc)
}
pc.mu.Unlock()
if p, ok := parentCancelCtx(cc.Context); !ok || p != pc {
t.Errorf("bad linkage: parentCancelCtx(cancelChild.Context) = %v, %v want %v, true", p, ok, pc)
}
if p, ok := parentCancelCtx(tc.Context); !ok || p != pc {
t.Errorf("bad linkage: parentCancelCtx(timerChild.Context) = %v, %v want %v, true", p, ok, pc)
}
cancel()
pc.mu.Lock()
if len(pc.children) != 0 {
t.Errorf("pc.cancel didn't clear pc.children = %v", pc.children)
}
pc.mu.Unlock()
// parent and children should all be finished.
check := func(ctx Context, name string) {
select {
case <-ctx.Done():
default:
t.Errorf("<-%s.Done() blocked, but shouldn't have", name)
}
if e := ctx.Err(); e != Canceled {
t.Errorf("%s.Err() == %v want %v", name, e, Canceled)
}
}
check(parent, "parent")
check(cancelChild, "cancelChild")
check(valueChild, "valueChild")
check(timerChild, "timerChild")
// WithCancel should return a canceled context on a canceled parent.
precanceledChild := WithValue(parent, "key", "value")
select {
case <-precanceledChild.Done():
default:
t.Errorf("<-precanceledChild.Done() blocked, but shouldn't have")
}
if e := precanceledChild.Err(); e != Canceled {
t.Errorf("precanceledChild.Err() == %v want %v", e, Canceled)
}
}
func TestChildFinishesFirst(t *testing.T) {
cancelable, stop := WithCancel(Background())
defer stop()
for _, parent := range []Context{Background(), cancelable} {
child, cancel := WithCancel(parent)
select {
case x := <-parent.Done():
t.Errorf("<-parent.Done() == %v want nothing (it should block)", x)
case x := <-child.Done():
t.Errorf("<-child.Done() == %v want nothing (it should block)", x)
default:
}
cc := child.(*cancelCtx)
pc, pcok := parent.(*cancelCtx) // pcok == false when parent == Background()
if p, ok := parentCancelCtx(cc.Context); ok != pcok || (ok && pc != p) {
t.Errorf("bad linkage: parentCancelCtx(cc.Context) = %v, %v want %v, %v", p, ok, pc, pcok)
}
if pcok {
pc.mu.Lock()
if len(pc.children) != 1 || !pc.children[cc] {
t.Errorf("bad linkage: pc.children = %v, cc = %v", pc.children, cc)
}
pc.mu.Unlock()
}
cancel()
if pcok {
pc.mu.Lock()
if len(pc.children) != 0 {
t.Errorf("child's cancel didn't remove self from pc.children = %v", pc.children)
}
pc.mu.Unlock()
}
// child should be finished.
select {
case <-child.Done():
default:
t.Errorf("<-child.Done() blocked, but shouldn't have")
}
if e := child.Err(); e != Canceled {
t.Errorf("child.Err() == %v want %v", e, Canceled)
}
// parent should not be finished.
select {
case x := <-parent.Done():
t.Errorf("<-parent.Done() == %v want nothing (it should block)", x)
default:
}
if e := parent.Err(); e != nil {
t.Errorf("parent.Err() == %v want nil", e)
}
}
}
func testDeadline(c Context, wait time.Duration, t *testing.T) {
select {
case <-time.After(wait):
t.Fatalf("context should have timed out")
case <-c.Done():
}
if e := c.Err(); e != DeadlineExceeded {
t.Errorf("c.Err() == %v want %v", e, DeadlineExceeded)
}
}
func TestDeadline(t *testing.T) {
c, _ := WithDeadline(Background(), time.Now().Add(100*time.Millisecond))
if got, prefix := fmt.Sprint(c), "context.Background.WithDeadline("; !strings.HasPrefix(got, prefix) {
t.Errorf("c.String() = %q want prefix %q", got, prefix)
}
testDeadline(c, 200*time.Millisecond, t)
c, _ = WithDeadline(Background(), time.Now().Add(100*time.Millisecond))
o := otherContext{c}
testDeadline(o, 200*time.Millisecond, t)
c, _ = WithDeadline(Background(), time.Now().Add(100*time.Millisecond))
o = otherContext{c}
c, _ = WithDeadline(o, time.Now().Add(300*time.Millisecond))
testDeadline(c, 200*time.Millisecond, t)
}
func TestTimeout(t *testing.T) {
c, _ := WithTimeout(Background(), 100*time.Millisecond)
if got, prefix := fmt.Sprint(c), "context.Background.WithDeadline("; !strings.HasPrefix(got, prefix) {
t.Errorf("c.String() = %q want prefix %q", got, prefix)
}
testDeadline(c, 200*time.Millisecond, t)
c, _ = WithTimeout(Background(), 100*time.Millisecond)
o := otherContext{c}
testDeadline(o, 200*time.Millisecond, t)
c, _ = WithTimeout(Background(), 100*time.Millisecond)
o = otherContext{c}
c, _ = WithTimeout(o, 300*time.Millisecond)
testDeadline(c, 200*time.Millisecond, t)
}
func TestCanceledTimeout(t *testing.T) {
c, _ := WithTimeout(Background(), 200*time.Millisecond)
o := otherContext{c}
c, cancel := WithTimeout(o, 400*time.Millisecond)
cancel()
time.Sleep(100 * time.Millisecond) // let cancelation propagate
select {
case <-c.Done():
default:
t.Errorf("<-c.Done() blocked, but shouldn't have")
}
if e := c.Err(); e != Canceled {
t.Errorf("c.Err() == %v want %v", e, Canceled)
}
}
type key1 int
type key2 int
var k1 = key1(1)
var k2 = key2(1) // same int as k1, different type
var k3 = key2(3) // same type as k2, different int
func TestValues(t *testing.T) {
check := func(c Context, nm, v1, v2, v3 string) {
if v, ok := c.Value(k1).(string); ok == (len(v1) == 0) || v != v1 {
t.Errorf(`%s.Value(k1).(string) = %q, %t want %q, %t`, nm, v, ok, v1, len(v1) != 0)
}
if v, ok := c.Value(k2).(string); ok == (len(v2) == 0) || v != v2 {
t.Errorf(`%s.Value(k2).(string) = %q, %t want %q, %t`, nm, v, ok, v2, len(v2) != 0)
}
if v, ok := c.Value(k3).(string); ok == (len(v3) == 0) || v != v3 {
t.Errorf(`%s.Value(k3).(string) = %q, %t want %q, %t`, nm, v, ok, v3, len(v3) != 0)
}
}
c0 := Background()
check(c0, "c0", "", "", "")
c1 := WithValue(Background(), k1, "c1k1")
check(c1, "c1", "c1k1", "", "")
if got, want := fmt.Sprint(c1), `context.Background.WithValue(1, "c1k1")`; got != want {
t.Errorf("c.String() = %q want %q", got, want)
}
c2 := WithValue(c1, k2, "c2k2")
check(c2, "c2", "c1k1", "c2k2", "")
c3 := WithValue(c2, k3, "c3k3")
check(c3, "c2", "c1k1", "c2k2", "c3k3")
c4 := WithValue(c3, k1, nil)
check(c4, "c4", "", "c2k2", "c3k3")
o0 := otherContext{Background()}
check(o0, "o0", "", "", "")
o1 := otherContext{WithValue(Background(), k1, "c1k1")}
check(o1, "o1", "c1k1", "", "")
o2 := WithValue(o1, k2, "o2k2")
check(o2, "o2", "c1k1", "o2k2", "")
o3 := otherContext{c4}
check(o3, "o3", "", "c2k2", "c3k3")
o4 := WithValue(o3, k3, nil)
check(o4, "o4", "", "c2k2", "")
}
func TestAllocs(t *testing.T) {
bg := Background()
for _, test := range []struct {
desc string
f func()
limit float64
gccgoLimit float64
}{
{
desc: "Background()",
f: func() { Background() },
limit: 0,
gccgoLimit: 0,
},
{
desc: fmt.Sprintf("WithValue(bg, %v, nil)", k1),
f: func() {
c := WithValue(bg, k1, nil)
c.Value(k1)
},
limit: 1,
gccgoLimit: 3,
},
{
desc: "WithTimeout(bg, 15*time.Millisecond)",
f: func() {
c, _ := WithTimeout(bg, 15*time.Millisecond)
<-c.Done()
},
limit: 8,
gccgoLimit: 13,
},
{
desc: "WithCancel(bg)",
f: func() {
c, cancel := WithCancel(bg)
cancel()
<-c.Done()
},
limit: 5,
gccgoLimit: 8,
},
{
desc: "WithTimeout(bg, 100*time.Millisecond)",
f: func() {
c, cancel := WithTimeout(bg, 100*time.Millisecond)
cancel()
<-c.Done()
},
limit: 8,
gccgoLimit: 25,
},
} {
limit := test.limit
if runtime.Compiler == "gccgo" {
// gccgo does not yet do escape analysis.
// TOOD(iant): Remove this when gccgo does do escape analysis.
limit = test.gccgoLimit
}
if n := testing.AllocsPerRun(100, test.f); n > limit {
t.Errorf("%s allocs = %f want %d", test.desc, n, int(limit))
}
}
}
func TestSimultaneousCancels(t *testing.T) {
root, cancel := WithCancel(Background())
m := map[Context]CancelFunc{root: cancel}
q := []Context{root}
// Create a tree of contexts.
for len(q) != 0 && len(m) < 100 {
parent := q[0]
q = q[1:]
for i := 0; i < 4; i++ {
ctx, cancel := WithCancel(parent)
m[ctx] = cancel
q = append(q, ctx)
}
}
// Start all the cancels in a random order.
var wg sync.WaitGroup
wg.Add(len(m))
for _, cancel := range m {
go func(cancel CancelFunc) {
cancel()
wg.Done()
}(cancel)
}
// Wait on all the contexts in a random order.
for ctx := range m {
select {
case <-ctx.Done():
case <-time.After(1 * time.Second):
buf := make([]byte, 10<<10)
n := runtime.Stack(buf, true)
t.Fatalf("timed out waiting for <-ctx.Done(); stacks:\n%s", buf[:n])
}
}
// Wait for all the cancel functions to return.
done := make(chan struct{})
go func() {
wg.Wait()
close(done)
}()
select {
case <-done:
case <-time.After(1 * time.Second):
buf := make([]byte, 10<<10)
n := runtime.Stack(buf, true)
t.Fatalf("timed out waiting for cancel functions; stacks:\n%s", buf[:n])
}
}
func TestInterlockedCancels(t *testing.T) {
parent, cancelParent := WithCancel(Background())
child, cancelChild := WithCancel(parent)
go func() {
parent.Done()
cancelChild()
}()
cancelParent()
select {
case <-child.Done():
case <-time.After(1 * time.Second):
buf := make([]byte, 10<<10)
n := runtime.Stack(buf, true)
t.Fatalf("timed out waiting for child.Done(); stacks:\n%s", buf[:n])
}
}
func TestLayersCancel(t *testing.T) {
testLayers(t, time.Now().UnixNano(), false)
}
func TestLayersTimeout(t *testing.T) {
testLayers(t, time.Now().UnixNano(), true)
}
func testLayers(t *testing.T, seed int64, testTimeout bool) {
rand.Seed(seed)
errorf := func(format string, a ...interface{}) {
t.Errorf(fmt.Sprintf("seed=%d: %s", seed, format), a...)
}
const (
timeout = 200 * time.Millisecond
minLayers = 30
)
type value int
var (
vals []*value
cancels []CancelFunc
numTimers int
ctx = Background()
)
for i := 0; i < minLayers || numTimers == 0 || len(cancels) == 0 || len(vals) == 0; i++ {
switch rand.Intn(3) {
case 0:
v := new(value)
ctx = WithValue(ctx, v, v)
vals = append(vals, v)
case 1:
var cancel CancelFunc
ctx, cancel = WithCancel(ctx)
cancels = append(cancels, cancel)
case 2:
var cancel CancelFunc
ctx, cancel = WithTimeout(ctx, timeout)
cancels = append(cancels, cancel)
numTimers++
}
}
checkValues := func(when string) {
for _, key := range vals {
if val := ctx.Value(key).(*value); key != val {
errorf("%s: ctx.Value(%p) = %p want %p", when, key, val, key)
}
}
}
select {
case <-ctx.Done():
errorf("ctx should not be canceled yet")
default:
}
if s, prefix := fmt.Sprint(ctx), "context.Background."; !strings.HasPrefix(s, prefix) {
t.Errorf("ctx.String() = %q want prefix %q", s, prefix)
}
t.Log(ctx)
checkValues("before cancel")
if testTimeout {
select {
case <-ctx.Done():
case <-time.After(timeout + timeout/10):
errorf("ctx should have timed out")
}
checkValues("after timeout")
} else {
cancel := cancels[rand.Intn(len(cancels))]
cancel()
select {
case <-ctx.Done():
default:
errorf("ctx should be canceled")
}
checkValues("after cancel")
}
}

View File

@@ -1,26 +0,0 @@
// Copyright 2014 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package context_test
import (
"fmt"
"time"
"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
)
func ExampleWithTimeout() {
// Pass a context with a timeout to tell a blocking function that it
// should abandon its work after the timeout elapses.
ctx, _ := context.WithTimeout(context.Background(), 100*time.Millisecond)
select {
case <-time.After(200 * time.Millisecond):
fmt.Println("overslept")
case <-ctx.Done():
fmt.Println(ctx.Err()) // prints "context deadline exceeded"
}
// Output:
// context deadline exceeded
}

View File

@@ -1,6 +0,0 @@
language: go
go: 1.1
script:
- go vet ./...
- go test -v ./...

View File

@@ -1,21 +0,0 @@
Copyright (C) 2013 Jeremy Saenz
All Rights Reserved.
MIT LICENSE
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

View File

@@ -1,287 +0,0 @@
[![Build Status](https://travis-ci.org/codegangsta/cli.png?branch=master)](https://travis-ci.org/codegangsta/cli)
# cli.go
cli.go is simple, fast, and fun package for building command line apps in Go. The goal is to enable developers to write fast and distributable command line applications in an expressive way.
You can view the API docs here:
http://godoc.org/github.com/codegangsta/cli
## Overview
Command line apps are usually so tiny that there is absolutely no reason why your code should *not* be self-documenting. Things like generating help text and parsing command flags/options should not hinder productivity when writing a command line app.
**This is where cli.go comes into play.** cli.go makes command line programming fun, organized, and expressive!
## Installation
Make sure you have a working Go environment (go 1.1 is *required*). [See the install instructions](http://golang.org/doc/install.html).
To install `cli.go`, simply run:
```
$ go get github.com/codegangsta/cli
```
Make sure your `PATH` includes to the `$GOPATH/bin` directory so your commands can be easily used:
```
export PATH=$PATH:$GOPATH/bin
```
## Getting Started
One of the philosophies behind cli.go is that an API should be playful and full of discovery. So a cli.go app can be as little as one line of code in `main()`.
``` go
package main
import (
"os"
"github.com/codegangsta/cli"
)
func main() {
cli.NewApp().Run(os.Args)
}
```
This app will run and show help text, but is not very useful. Let's give an action to execute and some help documentation:
``` go
package main
import (
"os"
"github.com/codegangsta/cli"
)
func main() {
app := cli.NewApp()
app.Name = "boom"
app.Usage = "make an explosive entrance"
app.Action = func(c *cli.Context) {
println("boom! I say!")
}
app.Run(os.Args)
}
```
Running this already gives you a ton of functionality, plus support for things like subcommands and flags, which are covered below.
## Example
Being a programmer can be a lonely job. Thankfully by the power of automation that is not the case! Let's create a greeter app to fend off our demons of loneliness!
``` go
/* greet.go */
package main
import (
"os"
"github.com/codegangsta/cli"
)
func main() {
app := cli.NewApp()
app.Name = "greet"
app.Usage = "fight the loneliness!"
app.Action = func(c *cli.Context) {
println("Hello friend!")
}
app.Run(os.Args)
}
```
Install our command to the `$GOPATH/bin` directory:
```
$ go install
```
Finally run our new command:
```
$ greet
Hello friend!
```
cli.go also generates some bitchass help text:
```
$ greet help
NAME:
greet - fight the loneliness!
USAGE:
greet [global options] command [command options] [arguments...]
VERSION:
0.0.0
COMMANDS:
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS
--version Shows version information
```
### Arguments
You can lookup arguments by calling the `Args` function on `cli.Context`.
``` go
...
app.Action = func(c *cli.Context) {
println("Hello", c.Args()[0])
}
...
```
### Flags
Setting and querying flags is simple.
``` go
...
app.Flags = []cli.Flag {
cli.StringFlag{
Name: "lang",
Value: "english",
Usage: "language for the greeting",
},
}
app.Action = func(c *cli.Context) {
name := "someone"
if len(c.Args()) > 0 {
name = c.Args()[0]
}
if c.String("lang") == "spanish" {
println("Hola", name)
} else {
println("Hello", name)
}
}
...
```
#### Alternate Names
You can set alternate (or short) names for flags by providing a comma-delimited list for the `Name`. e.g.
``` go
app.Flags = []cli.Flag {
cli.StringFlag{
Name: "lang, l",
Value: "english",
Usage: "language for the greeting",
},
}
```
#### Values from the Environment
You can also have the default value set from the environment via `EnvVar`. e.g.
``` go
app.Flags = []cli.Flag {
cli.StringFlag{
Name: "lang, l",
Value: "english",
Usage: "language for the greeting",
EnvVar: "APP_LANG",
},
}
```
That flag can then be set with `--lang spanish` or `-l spanish`. Note that giving two different forms of the same flag in the same command invocation is an error.
### Subcommands
Subcommands can be defined for a more git-like command line app.
```go
...
app.Commands = []cli.Command{
{
Name: "add",
ShortName: "a",
Usage: "add a task to the list",
Action: func(c *cli.Context) {
println("added task: ", c.Args().First())
},
},
{
Name: "complete",
ShortName: "c",
Usage: "complete a task on the list",
Action: func(c *cli.Context) {
println("completed task: ", c.Args().First())
},
},
{
Name: "template",
ShortName: "r",
Usage: "options for task templates",
Subcommands: []cli.Command{
{
Name: "add",
Usage: "add a new template",
Action: func(c *cli.Context) {
println("new task template: ", c.Args().First())
},
},
{
Name: "remove",
Usage: "remove an existing template",
Action: func(c *cli.Context) {
println("removed task template: ", c.Args().First())
},
},
},
},
}
...
```
### Bash Completion
You can enable completion commands by setting the `EnableBashCompletion`
flag on the `App` object. By default, this setting will only auto-complete to
show an app's subcommands, but you can write your own completion methods for
the App or its subcommands.
```go
...
var tasks = []string{"cook", "clean", "laundry", "eat", "sleep", "code"}
app := cli.NewApp()
app.EnableBashCompletion = true
app.Commands = []cli.Command{
{
Name: "complete",
ShortName: "c",
Usage: "complete a task on the list",
Action: func(c *cli.Context) {
println("completed task: ", c.Args().First())
},
BashComplete: func(c *cli.Context) {
// This will complete if no args are passed
if len(c.Args()) > 0 {
return
}
for _, t := range tasks {
fmt.Println(t)
}
},
}
}
...
```
#### To Enable
Source the `autocomplete/bash_autocomplete` file in your `.bashrc` file while
setting the `PROG` variable to the name of your program:
`PROG=myprogram source /.../cli/autocomplete/bash_autocomplete`
## Contribution Guidelines
Feel free to put up a pull request to fix a bug or maybe add a feature. I will give it a code review and make sure that it does not break backwards compatibility. If I or any other collaborators agree that it is in line with the vision of the project, we will work with you to get the code into a mergeable state and merge it into the master branch.
If you are have contributed something significant to the project, I will most likely add you as a collaborator. As a collaborator you are given the ability to merge others pull requests. It is very important that new code does not break existing code, so be careful about what code you do choose to merge. If you have any questions feel free to link @codegangsta to the issue in question and we can review it together.
If you feel like you have contributed to the project but have not yet been added as a collaborator, I probably forgot to add you. Hit @codegangsta up over email and we will get it figured out.
## About
cli.go is written by none other than the [Code Gangsta](http://codegangsta.io)

View File

@@ -1,246 +0,0 @@
package cli
import (
"fmt"
"io/ioutil"
"os"
"time"
)
// App is the main structure of a cli application. It is recomended that
// and app be created with the cli.NewApp() function
type App struct {
// The name of the program. Defaults to os.Args[0]
Name string
// Description of the program.
Usage string
// Version of the program
Version string
// List of commands to execute
Commands []Command
// List of flags to parse
Flags []Flag
// Boolean to enable bash completion commands
EnableBashCompletion bool
// Boolean to hide built-in help command
HideHelp bool
// An action to execute when the bash-completion flag is set
BashComplete func(context *Context)
// An action to execute before any subcommands are run, but after the context is ready
// If a non-nil error is returned, no subcommands are run
Before func(context *Context) error
// The action to execute when no subcommands are specified
Action func(context *Context)
// Execute this function if the proper command cannot be found
CommandNotFound func(context *Context, command string)
// Compilation date
Compiled time.Time
// Author
Author string
// Author e-mail
Email string
}
// Tries to find out when this binary was compiled.
// Returns the current time if it fails to find it.
func compileTime() time.Time {
info, err := os.Stat(os.Args[0])
if err != nil {
return time.Now()
}
return info.ModTime()
}
// Creates a new cli Application with some reasonable defaults for Name, Usage, Version and Action.
func NewApp() *App {
return &App{
Name: os.Args[0],
Usage: "A new cli application",
Version: "0.0.0",
BashComplete: DefaultAppComplete,
Action: helpCommand.Action,
Compiled: compileTime(),
}
}
// Entry point to the cli app. Parses the arguments slice and routes to the proper flag/args combination
func (a *App) Run(arguments []string) error {
// append help to commands
if a.Command(helpCommand.Name) == nil && !a.HideHelp {
a.Commands = append(a.Commands, helpCommand)
a.appendFlag(HelpFlag)
}
//append version/help flags
if a.EnableBashCompletion {
a.appendFlag(BashCompletionFlag)
}
a.appendFlag(VersionFlag)
// parse flags
set := flagSet(a.Name, a.Flags)
set.SetOutput(ioutil.Discard)
err := set.Parse(arguments[1:])
nerr := normalizeFlags(a.Flags, set)
if nerr != nil {
fmt.Println(nerr)
context := NewContext(a, set, set)
ShowAppHelp(context)
fmt.Println("")
return nerr
}
context := NewContext(a, set, set)
if err != nil {
fmt.Printf("Incorrect Usage.\n\n")
ShowAppHelp(context)
fmt.Println("")
return err
}
if checkCompletions(context) {
return nil
}
if checkHelp(context) {
return nil
}
if checkVersion(context) {
return nil
}
if a.Before != nil {
err := a.Before(context)
if err != nil {
return err
}
}
args := context.Args()
if args.Present() {
name := args.First()
c := a.Command(name)
if c != nil {
return c.Run(context)
}
}
// Run default Action
a.Action(context)
return nil
}
// Another entry point to the cli app, takes care of passing arguments and error handling
func (a *App) RunAndExitOnError() {
if err := a.Run(os.Args); err != nil {
os.Stderr.WriteString(fmt.Sprintln(err))
os.Exit(1)
}
}
// Invokes the subcommand given the context, parses ctx.Args() to generate command-specific flags
func (a *App) RunAsSubcommand(ctx *Context) error {
// append help to commands
if len(a.Commands) > 0 {
if a.Command(helpCommand.Name) == nil && !a.HideHelp {
a.Commands = append(a.Commands, helpCommand)
a.appendFlag(HelpFlag)
}
}
// append flags
if a.EnableBashCompletion {
a.appendFlag(BashCompletionFlag)
}
// parse flags
set := flagSet(a.Name, a.Flags)
set.SetOutput(ioutil.Discard)
err := set.Parse(ctx.Args().Tail())
nerr := normalizeFlags(a.Flags, set)
context := NewContext(a, set, ctx.globalSet)
if nerr != nil {
fmt.Println(nerr)
if len(a.Commands) > 0 {
ShowSubcommandHelp(context)
} else {
ShowCommandHelp(ctx, context.Args().First())
}
fmt.Println("")
return nerr
}
if err != nil {
fmt.Printf("Incorrect Usage.\n\n")
ShowSubcommandHelp(context)
return err
}
if checkCompletions(context) {
return nil
}
if len(a.Commands) > 0 {
if checkSubcommandHelp(context) {
return nil
}
} else {
if checkCommandHelp(ctx, context.Args().First()) {
return nil
}
}
if a.Before != nil {
err := a.Before(context)
if err != nil {
return err
}
}
args := context.Args()
if args.Present() {
name := args.First()
c := a.Command(name)
if c != nil {
return c.Run(context)
}
}
// Run default Action
if len(a.Commands) > 0 {
a.Action(context)
} else {
a.Action(ctx)
}
return nil
}
// Returns the named command on App. Returns nil if the command does not exist
func (a *App) Command(name string) *Command {
for _, c := range a.Commands {
if c.HasName(name) {
return &c
}
}
return nil
}
func (a *App) hasFlag(flag Flag) bool {
for _, f := range a.Flags {
if flag == f {
return true
}
}
return false
}
func (a *App) appendFlag(flag Flag) {
if !a.hasFlag(flag) {
a.Flags = append(a.Flags, flag)
}
}

View File

@@ -1,423 +0,0 @@
package cli_test
import (
"fmt"
"os"
"testing"
"github.com/codegangsta/cli"
)
func ExampleApp() {
// set args for examples sake
os.Args = []string{"greet", "--name", "Jeremy"}
app := cli.NewApp()
app.Name = "greet"
app.Flags = []cli.Flag{
cli.StringFlag{Name: "name", Value: "bob", Usage: "a name to say"},
}
app.Action = func(c *cli.Context) {
fmt.Printf("Hello %v\n", c.String("name"))
}
app.Run(os.Args)
// Output:
// Hello Jeremy
}
func ExampleAppSubcommand() {
// set args for examples sake
os.Args = []string{"say", "hi", "english", "--name", "Jeremy"}
app := cli.NewApp()
app.Name = "say"
app.Commands = []cli.Command{
{
Name: "hello",
ShortName: "hi",
Usage: "use it to see a description",
Description: "This is how we describe hello the function",
Subcommands: []cli.Command{
{
Name: "english",
ShortName: "en",
Usage: "sends a greeting in english",
Description: "greets someone in english",
Flags: []cli.Flag{
cli.StringFlag{
Name: "name",
Value: "Bob",
Usage: "Name of the person to greet",
},
},
Action: func(c *cli.Context) {
fmt.Println("Hello,", c.String("name"))
},
},
},
},
}
app.Run(os.Args)
// Output:
// Hello, Jeremy
}
func ExampleAppHelp() {
// set args for examples sake
os.Args = []string{"greet", "h", "describeit"}
app := cli.NewApp()
app.Name = "greet"
app.Flags = []cli.Flag{
cli.StringFlag{Name: "name", Value: "bob", Usage: "a name to say"},
}
app.Commands = []cli.Command{
{
Name: "describeit",
ShortName: "d",
Usage: "use it to see a description",
Description: "This is how we describe describeit the function",
Action: func(c *cli.Context) {
fmt.Printf("i like to describe things")
},
},
}
app.Run(os.Args)
// Output:
// NAME:
// describeit - use it to see a description
//
// USAGE:
// command describeit [arguments...]
//
// DESCRIPTION:
// This is how we describe describeit the function
}
func ExampleAppBashComplete() {
// set args for examples sake
os.Args = []string{"greet", "--generate-bash-completion"}
app := cli.NewApp()
app.Name = "greet"
app.EnableBashCompletion = true
app.Commands = []cli.Command{
{
Name: "describeit",
ShortName: "d",
Usage: "use it to see a description",
Description: "This is how we describe describeit the function",
Action: func(c *cli.Context) {
fmt.Printf("i like to describe things")
},
}, {
Name: "next",
Usage: "next example",
Description: "more stuff to see when generating bash completion",
Action: func(c *cli.Context) {
fmt.Printf("the next example")
},
},
}
app.Run(os.Args)
// Output:
// describeit
// d
// next
// help
// h
}
func TestApp_Run(t *testing.T) {
s := ""
app := cli.NewApp()
app.Action = func(c *cli.Context) {
s = s + c.Args().First()
}
err := app.Run([]string{"command", "foo"})
expect(t, err, nil)
err = app.Run([]string{"command", "bar"})
expect(t, err, nil)
expect(t, s, "foobar")
}
var commandAppTests = []struct {
name string
expected bool
}{
{"foobar", true},
{"batbaz", true},
{"b", true},
{"f", true},
{"bat", false},
{"nothing", false},
}
func TestApp_Command(t *testing.T) {
app := cli.NewApp()
fooCommand := cli.Command{Name: "foobar", ShortName: "f"}
batCommand := cli.Command{Name: "batbaz", ShortName: "b"}
app.Commands = []cli.Command{
fooCommand,
batCommand,
}
for _, test := range commandAppTests {
expect(t, app.Command(test.name) != nil, test.expected)
}
}
func TestApp_CommandWithArgBeforeFlags(t *testing.T) {
var parsedOption, firstArg string
app := cli.NewApp()
command := cli.Command{
Name: "cmd",
Flags: []cli.Flag{
cli.StringFlag{Name: "option", Value: "", Usage: "some option"},
},
Action: func(c *cli.Context) {
parsedOption = c.String("option")
firstArg = c.Args().First()
},
}
app.Commands = []cli.Command{command}
app.Run([]string{"", "cmd", "my-arg", "--option", "my-option"})
expect(t, parsedOption, "my-option")
expect(t, firstArg, "my-arg")
}
func TestApp_Float64Flag(t *testing.T) {
var meters float64
app := cli.NewApp()
app.Flags = []cli.Flag{
cli.Float64Flag{Name: "height", Value: 1.5, Usage: "Set the height, in meters"},
}
app.Action = func(c *cli.Context) {
meters = c.Float64("height")
}
app.Run([]string{"", "--height", "1.93"})
expect(t, meters, 1.93)
}
func TestApp_ParseSliceFlags(t *testing.T) {
var parsedOption, firstArg string
var parsedIntSlice []int
var parsedStringSlice []string
app := cli.NewApp()
command := cli.Command{
Name: "cmd",
Flags: []cli.Flag{
cli.IntSliceFlag{Name: "p", Value: &cli.IntSlice{}, Usage: "set one or more ip addr"},
cli.StringSliceFlag{Name: "ip", Value: &cli.StringSlice{}, Usage: "set one or more ports to open"},
},
Action: func(c *cli.Context) {
parsedIntSlice = c.IntSlice("p")
parsedStringSlice = c.StringSlice("ip")
parsedOption = c.String("option")
firstArg = c.Args().First()
},
}
app.Commands = []cli.Command{command}
app.Run([]string{"", "cmd", "my-arg", "-p", "22", "-p", "80", "-ip", "8.8.8.8", "-ip", "8.8.4.4"})
IntsEquals := func(a, b []int) bool {
if len(a) != len(b) {
return false
}
for i, v := range a {
if v != b[i] {
return false
}
}
return true
}
StrsEquals := func(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i, v := range a {
if v != b[i] {
return false
}
}
return true
}
var expectedIntSlice = []int{22, 80}
var expectedStringSlice = []string{"8.8.8.8", "8.8.4.4"}
if !IntsEquals(parsedIntSlice, expectedIntSlice) {
t.Errorf("%v does not match %v", parsedIntSlice, expectedIntSlice)
}
if !StrsEquals(parsedStringSlice, expectedStringSlice) {
t.Errorf("%v does not match %v", parsedStringSlice, expectedStringSlice)
}
}
func TestApp_BeforeFunc(t *testing.T) {
beforeRun, subcommandRun := false, false
beforeError := fmt.Errorf("fail")
var err error
app := cli.NewApp()
app.Before = func(c *cli.Context) error {
beforeRun = true
s := c.String("opt")
if s == "fail" {
return beforeError
}
return nil
}
app.Commands = []cli.Command{
cli.Command{
Name: "sub",
Action: func(c *cli.Context) {
subcommandRun = true
},
},
}
app.Flags = []cli.Flag{
cli.StringFlag{Name: "opt"},
}
// run with the Before() func succeeding
err = app.Run([]string{"command", "--opt", "succeed", "sub"})
if err != nil {
t.Fatalf("Run error: %s", err)
}
if beforeRun == false {
t.Errorf("Before() not executed when expected")
}
if subcommandRun == false {
t.Errorf("Subcommand not executed when expected")
}
// reset
beforeRun, subcommandRun = false, false
// run with the Before() func failing
err = app.Run([]string{"command", "--opt", "fail", "sub"})
// should be the same error produced by the Before func
if err != beforeError {
t.Errorf("Run error expected, but not received")
}
if beforeRun == false {
t.Errorf("Before() not executed when expected")
}
if subcommandRun == true {
t.Errorf("Subcommand executed when NOT expected")
}
}
func TestAppHelpPrinter(t *testing.T) {
oldPrinter := cli.HelpPrinter
defer func() {
cli.HelpPrinter = oldPrinter
}()
var wasCalled = false
cli.HelpPrinter = func(template string, data interface{}) {
wasCalled = true
}
app := cli.NewApp()
app.Run([]string{"-h"})
if wasCalled == false {
t.Errorf("Help printer expected to be called, but was not")
}
}
func TestAppVersionPrinter(t *testing.T) {
oldPrinter := cli.VersionPrinter
defer func() {
cli.VersionPrinter = oldPrinter
}()
var wasCalled = false
cli.VersionPrinter = func(c *cli.Context) {
wasCalled = true
}
app := cli.NewApp()
ctx := cli.NewContext(app, nil, nil)
cli.ShowVersion(ctx)
if wasCalled == false {
t.Errorf("Version printer expected to be called, but was not")
}
}
func TestAppCommandNotFound(t *testing.T) {
beforeRun, subcommandRun := false, false
app := cli.NewApp()
app.CommandNotFound = func(c *cli.Context, command string) {
beforeRun = true
}
app.Commands = []cli.Command{
cli.Command{
Name: "bar",
Action: func(c *cli.Context) {
subcommandRun = true
},
},
}
app.Run([]string{"command", "foo"})
expect(t, beforeRun, true)
expect(t, subcommandRun, false)
}
func TestGlobalFlagsInSubcommands(t *testing.T) {
subcommandRun := false
app := cli.NewApp()
app.Flags = []cli.Flag{
cli.BoolFlag{Name: "debug, d", Usage: "Enable debugging"},
}
app.Commands = []cli.Command{
cli.Command{
Name: "foo",
Subcommands: []cli.Command{
{
Name: "bar",
Action: func(c *cli.Context) {
if c.GlobalBool("debug") {
subcommandRun = true
}
},
},
},
},
}
app.Run([]string{"command", "-d", "foo", "bar"})
expect(t, subcommandRun, true)
}

View File

@@ -1,13 +0,0 @@
#! /bin/bash
_cli_bash_autocomplete() {
local cur prev opts base
COMPREPLY=()
cur="${COMP_WORDS[COMP_CWORD]}"
prev="${COMP_WORDS[COMP_CWORD-1]}"
opts=$( ${COMP_WORDS[@]:0:$COMP_CWORD} --generate-bash-completion )
COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
return 0
}
complete -F _cli_bash_autocomplete $PROG

View File

@@ -1,5 +0,0 @@
autoload -U compinit && compinit
autoload -U bashcompinit && bashcompinit
script_dir=$(dirname $0)
source ${script_dir}/bash_autocomplete

View File

@@ -1,19 +0,0 @@
// Package cli provides a minimal framework for creating and organizing command line
// Go applications. cli is designed to be easy to understand and write, the most simple
// cli application can be written as follows:
// func main() {
// cli.NewApp().Run(os.Args)
// }
//
// Of course this application does not do much, so let's make this an actual application:
// func main() {
// app := cli.NewApp()
// app.Name = "greet"
// app.Usage = "say a greeting"
// app.Action = func(c *cli.Context) {
// println("Greetings")
// }
//
// app.Run(os.Args)
// }
package cli

View File

@@ -1,100 +0,0 @@
package cli_test
import (
"os"
"github.com/codegangsta/cli"
)
func Example() {
app := cli.NewApp()
app.Name = "todo"
app.Usage = "task list on the command line"
app.Commands = []cli.Command{
{
Name: "add",
ShortName: "a",
Usage: "add a task to the list",
Action: func(c *cli.Context) {
println("added task: ", c.Args().First())
},
},
{
Name: "complete",
ShortName: "c",
Usage: "complete a task on the list",
Action: func(c *cli.Context) {
println("completed task: ", c.Args().First())
},
},
}
app.Run(os.Args)
}
func ExampleSubcommand() {
app := cli.NewApp()
app.Name = "say"
app.Commands = []cli.Command{
{
Name: "hello",
ShortName: "hi",
Usage: "use it to see a description",
Description: "This is how we describe hello the function",
Subcommands: []cli.Command{
{
Name: "english",
ShortName: "en",
Usage: "sends a greeting in english",
Description: "greets someone in english",
Flags: []cli.Flag{
cli.StringFlag{
Name: "name",
Value: "Bob",
Usage: "Name of the person to greet",
},
},
Action: func(c *cli.Context) {
println("Hello, ", c.String("name"))
},
}, {
Name: "spanish",
ShortName: "sp",
Usage: "sends a greeting in spanish",
Flags: []cli.Flag{
cli.StringFlag{
Name: "surname",
Value: "Jones",
Usage: "Surname of the person to greet",
},
},
Action: func(c *cli.Context) {
println("Hola, ", c.String("surname"))
},
}, {
Name: "french",
ShortName: "fr",
Usage: "sends a greeting in french",
Flags: []cli.Flag{
cli.StringFlag{
Name: "nickname",
Value: "Stevie",
Usage: "Nickname of the person to greet",
},
},
Action: func(c *cli.Context) {
println("Bonjour, ", c.String("nickname"))
},
},
},
}, {
Name: "bye",
Usage: "says goodbye",
Action: func(c *cli.Context) {
println("bye")
},
},
}
app.Run(os.Args)
}

View File

@@ -1,144 +0,0 @@
package cli
import (
"fmt"
"io/ioutil"
"strings"
)
// Command is a subcommand for a cli.App.
type Command struct {
// The name of the command
Name string
// short name of the command. Typically one character
ShortName string
// A short description of the usage of this command
Usage string
// A longer explanation of how the command works
Description string
// The function to call when checking for bash command completions
BashComplete func(context *Context)
// An action to execute before any sub-subcommands are run, but after the context is ready
// If a non-nil error is returned, no sub-subcommands are run
Before func(context *Context) error
// The function to call when this command is invoked
Action func(context *Context)
// List of child commands
Subcommands []Command
// List of flags to parse
Flags []Flag
// Treat all flags as normal arguments if true
SkipFlagParsing bool
// Boolean to hide built-in help command
HideHelp bool
}
// Invokes the command given the context, parses ctx.Args() to generate command-specific flags
func (c Command) Run(ctx *Context) error {
if len(c.Subcommands) > 0 || c.Before != nil {
return c.startApp(ctx)
}
if !c.HideHelp {
// append help to flags
c.Flags = append(
c.Flags,
HelpFlag,
)
}
if ctx.App.EnableBashCompletion {
c.Flags = append(c.Flags, BashCompletionFlag)
}
set := flagSet(c.Name, c.Flags)
set.SetOutput(ioutil.Discard)
firstFlagIndex := -1
for index, arg := range ctx.Args() {
if strings.HasPrefix(arg, "-") {
firstFlagIndex = index
break
}
}
var err error
if firstFlagIndex > -1 && !c.SkipFlagParsing {
args := ctx.Args()
regularArgs := args[1:firstFlagIndex]
flagArgs := args[firstFlagIndex:]
err = set.Parse(append(flagArgs, regularArgs...))
} else {
err = set.Parse(ctx.Args().Tail())
}
if err != nil {
fmt.Printf("Incorrect Usage.\n\n")
ShowCommandHelp(ctx, c.Name)
fmt.Println("")
return err
}
nerr := normalizeFlags(c.Flags, set)
if nerr != nil {
fmt.Println(nerr)
fmt.Println("")
ShowCommandHelp(ctx, c.Name)
fmt.Println("")
return nerr
}
context := NewContext(ctx.App, set, ctx.globalSet)
if checkCommandCompletions(context, c.Name) {
return nil
}
if checkCommandHelp(context, c.Name) {
return nil
}
context.Command = c
c.Action(context)
return nil
}
// Returns true if Command.Name or Command.ShortName matches given name
func (c Command) HasName(name string) bool {
return c.Name == name || c.ShortName == name
}
func (c Command) startApp(ctx *Context) error {
app := NewApp()
// set the name and usage
app.Name = fmt.Sprintf("%s %s", ctx.App.Name, c.Name)
if c.Description != "" {
app.Usage = c.Description
} else {
app.Usage = c.Usage
}
// set CommandNotFound
app.CommandNotFound = ctx.App.CommandNotFound
// set the flags and commands
app.Commands = c.Subcommands
app.Flags = c.Flags
app.HideHelp = c.HideHelp
// bash completion
app.EnableBashCompletion = ctx.App.EnableBashCompletion
if c.BashComplete != nil {
app.BashComplete = c.BashComplete
}
// set the actions
app.Before = c.Before
if c.Action != nil {
app.Action = c.Action
} else {
app.Action = helpSubcommand.Action
}
return app.RunAsSubcommand(ctx)
}

View File

@@ -1,49 +0,0 @@
package cli_test
import (
"flag"
"testing"
"github.com/codegangsta/cli"
)
func TestCommandDoNotIgnoreFlags(t *testing.T) {
app := cli.NewApp()
set := flag.NewFlagSet("test", 0)
test := []string{"blah", "blah", "-break"}
set.Parse(test)
c := cli.NewContext(app, set, set)
command := cli.Command{
Name: "test-cmd",
ShortName: "tc",
Usage: "this is for testing",
Description: "testing",
Action: func(_ *cli.Context) {},
}
err := command.Run(c)
expect(t, err.Error(), "flag provided but not defined: -break")
}
func TestCommandIgnoreFlags(t *testing.T) {
app := cli.NewApp()
set := flag.NewFlagSet("test", 0)
test := []string{"blah", "blah"}
set.Parse(test)
c := cli.NewContext(app, set, set)
command := cli.Command{
Name: "test-cmd",
ShortName: "tc",
Usage: "this is for testing",
Description: "testing",
Action: func(_ *cli.Context) {},
SkipFlagParsing: true,
}
err := command.Run(c)
expect(t, err, nil)
}

View File

@@ -1,315 +0,0 @@
package cli
import (
"errors"
"flag"
"strconv"
"strings"
"time"
)
// Context is a type that is passed through to
// each Handler action in a cli application. Context
// can be used to retrieve context-specific Args and
// parsed command-line options.
type Context struct {
App *App
Command Command
flagSet *flag.FlagSet
globalSet *flag.FlagSet
setFlags map[string]bool
}
// Creates a new context. For use in when invoking an App or Command action.
func NewContext(app *App, set *flag.FlagSet, globalSet *flag.FlagSet) *Context {
return &Context{App: app, flagSet: set, globalSet: globalSet}
}
// Looks up the value of a local int flag, returns 0 if no int flag exists
func (c *Context) Int(name string) int {
return lookupInt(name, c.flagSet)
}
// Looks up the value of a local time.Duration flag, returns 0 if no time.Duration flag exists
func (c *Context) Duration(name string) time.Duration {
return lookupDuration(name, c.flagSet)
}
// Looks up the value of a local float64 flag, returns 0 if no float64 flag exists
func (c *Context) Float64(name string) float64 {
return lookupFloat64(name, c.flagSet)
}
// Looks up the value of a local bool flag, returns false if no bool flag exists
func (c *Context) Bool(name string) bool {
return lookupBool(name, c.flagSet)
}
// Looks up the value of a local boolT flag, returns false if no bool flag exists
func (c *Context) BoolT(name string) bool {
return lookupBoolT(name, c.flagSet)
}
// Looks up the value of a local string flag, returns "" if no string flag exists
func (c *Context) String(name string) string {
return lookupString(name, c.flagSet)
}
// Looks up the value of a local string slice flag, returns nil if no string slice flag exists
func (c *Context) StringSlice(name string) []string {
return lookupStringSlice(name, c.flagSet)
}
// Looks up the value of a local int slice flag, returns nil if no int slice flag exists
func (c *Context) IntSlice(name string) []int {
return lookupIntSlice(name, c.flagSet)
}
// Looks up the value of a local generic flag, returns nil if no generic flag exists
func (c *Context) Generic(name string) interface{} {
return lookupGeneric(name, c.flagSet)
}
// Looks up the value of a global int flag, returns 0 if no int flag exists
func (c *Context) GlobalInt(name string) int {
return lookupInt(name, c.globalSet)
}
// Looks up the value of a global time.Duration flag, returns 0 if no time.Duration flag exists
func (c *Context) GlobalDuration(name string) time.Duration {
return lookupDuration(name, c.globalSet)
}
// Looks up the value of a global bool flag, returns false if no bool flag exists
func (c *Context) GlobalBool(name string) bool {
return lookupBool(name, c.globalSet)
}
// Looks up the value of a global string flag, returns "" if no string flag exists
func (c *Context) GlobalString(name string) string {
return lookupString(name, c.globalSet)
}
// Looks up the value of a global string slice flag, returns nil if no string slice flag exists
func (c *Context) GlobalStringSlice(name string) []string {
return lookupStringSlice(name, c.globalSet)
}
// Looks up the value of a global int slice flag, returns nil if no int slice flag exists
func (c *Context) GlobalIntSlice(name string) []int {
return lookupIntSlice(name, c.globalSet)
}
// Looks up the value of a global generic flag, returns nil if no generic flag exists
func (c *Context) GlobalGeneric(name string) interface{} {
return lookupGeneric(name, c.globalSet)
}
// Determines if the flag was actually set exists
func (c *Context) IsSet(name string) bool {
if c.setFlags == nil {
c.setFlags = make(map[string]bool)
c.flagSet.Visit(func(f *flag.Flag) {
c.setFlags[f.Name] = true
})
}
return c.setFlags[name] == true
}
// Returns a slice of flag names used in this context.
func (c *Context) FlagNames() (names []string) {
for _, flag := range c.Command.Flags {
name := strings.Split(flag.getName(), ",")[0]
if name == "help" {
continue
}
names = append(names, name)
}
return
}
type Args []string
// Returns the command line arguments associated with the context.
func (c *Context) Args() Args {
args := Args(c.flagSet.Args())
return args
}
// Returns the nth argument, or else a blank string
func (a Args) Get(n int) string {
if len(a) > n {
return a[n]
}
return ""
}
// Returns the first argument, or else a blank string
func (a Args) First() string {
return a.Get(0)
}
// Return the rest of the arguments (not the first one)
// or else an empty string slice
func (a Args) Tail() []string {
if len(a) >= 2 {
return []string(a)[1:]
}
return []string{}
}
// Checks if there are any arguments present
func (a Args) Present() bool {
return len(a) != 0
}
// Swaps arguments at the given indexes
func (a Args) Swap(from, to int) error {
if from >= len(a) || to >= len(a) {
return errors.New("index out of range")
}
a[from], a[to] = a[to], a[from]
return nil
}
func lookupInt(name string, set *flag.FlagSet) int {
f := set.Lookup(name)
if f != nil {
val, err := strconv.Atoi(f.Value.String())
if err != nil {
return 0
}
return val
}
return 0
}
func lookupDuration(name string, set *flag.FlagSet) time.Duration {
f := set.Lookup(name)
if f != nil {
val, err := time.ParseDuration(f.Value.String())
if err == nil {
return val
}
}
return 0
}
func lookupFloat64(name string, set *flag.FlagSet) float64 {
f := set.Lookup(name)
if f != nil {
val, err := strconv.ParseFloat(f.Value.String(), 64)
if err != nil {
return 0
}
return val
}
return 0
}
func lookupString(name string, set *flag.FlagSet) string {
f := set.Lookup(name)
if f != nil {
return f.Value.String()
}
return ""
}
func lookupStringSlice(name string, set *flag.FlagSet) []string {
f := set.Lookup(name)
if f != nil {
return (f.Value.(*StringSlice)).Value()
}
return nil
}
func lookupIntSlice(name string, set *flag.FlagSet) []int {
f := set.Lookup(name)
if f != nil {
return (f.Value.(*IntSlice)).Value()
}
return nil
}
func lookupGeneric(name string, set *flag.FlagSet) interface{} {
f := set.Lookup(name)
if f != nil {
return f.Value
}
return nil
}
func lookupBool(name string, set *flag.FlagSet) bool {
f := set.Lookup(name)
if f != nil {
val, err := strconv.ParseBool(f.Value.String())
if err != nil {
return false
}
return val
}
return false
}
func lookupBoolT(name string, set *flag.FlagSet) bool {
f := set.Lookup(name)
if f != nil {
val, err := strconv.ParseBool(f.Value.String())
if err != nil {
return true
}
return val
}
return false
}
func copyFlag(name string, ff *flag.Flag, set *flag.FlagSet) {
switch ff.Value.(type) {
case *StringSlice:
default:
set.Set(name, ff.Value.String())
}
}
func normalizeFlags(flags []Flag, set *flag.FlagSet) error {
visited := make(map[string]bool)
set.Visit(func(f *flag.Flag) {
visited[f.Name] = true
})
for _, f := range flags {
parts := strings.Split(f.getName(), ",")
if len(parts) == 1 {
continue
}
var ff *flag.Flag
for _, name := range parts {
name = strings.Trim(name, " ")
if visited[name] {
if ff != nil {
return errors.New("Cannot use two forms of the same flag: " + name + " " + ff.Name)
}
ff = set.Lookup(name)
}
}
if ff == nil {
continue
}
for _, name := range parts {
name = strings.Trim(name, " ")
if !visited[name] {
copyFlag(name, ff, set)
}
}
}
return nil
}

View File

@@ -1,77 +0,0 @@
package cli_test
import (
"flag"
"testing"
"time"
"github.com/codegangsta/cli"
)
func TestNewContext(t *testing.T) {
set := flag.NewFlagSet("test", 0)
set.Int("myflag", 12, "doc")
globalSet := flag.NewFlagSet("test", 0)
globalSet.Int("myflag", 42, "doc")
command := cli.Command{Name: "mycommand"}
c := cli.NewContext(nil, set, globalSet)
c.Command = command
expect(t, c.Int("myflag"), 12)
expect(t, c.GlobalInt("myflag"), 42)
expect(t, c.Command.Name, "mycommand")
}
func TestContext_Int(t *testing.T) {
set := flag.NewFlagSet("test", 0)
set.Int("myflag", 12, "doc")
c := cli.NewContext(nil, set, set)
expect(t, c.Int("myflag"), 12)
}
func TestContext_Duration(t *testing.T) {
set := flag.NewFlagSet("test", 0)
set.Duration("myflag", time.Duration(12*time.Second), "doc")
c := cli.NewContext(nil, set, set)
expect(t, c.Duration("myflag"), time.Duration(12*time.Second))
}
func TestContext_String(t *testing.T) {
set := flag.NewFlagSet("test", 0)
set.String("myflag", "hello world", "doc")
c := cli.NewContext(nil, set, set)
expect(t, c.String("myflag"), "hello world")
}
func TestContext_Bool(t *testing.T) {
set := flag.NewFlagSet("test", 0)
set.Bool("myflag", false, "doc")
c := cli.NewContext(nil, set, set)
expect(t, c.Bool("myflag"), false)
}
func TestContext_BoolT(t *testing.T) {
set := flag.NewFlagSet("test", 0)
set.Bool("myflag", true, "doc")
c := cli.NewContext(nil, set, set)
expect(t, c.BoolT("myflag"), true)
}
func TestContext_Args(t *testing.T) {
set := flag.NewFlagSet("test", 0)
set.Bool("myflag", false, "doc")
c := cli.NewContext(nil, set, set)
set.Parse([]string{"--myflag", "bat", "baz"})
expect(t, len(c.Args()), 2)
expect(t, c.Bool("myflag"), true)
}
func TestContext_IsSet(t *testing.T) {
set := flag.NewFlagSet("test", 0)
set.Bool("myflag", false, "doc")
set.String("otherflag", "hello world", "doc")
c := cli.NewContext(nil, set, set)
set.Parse([]string{"--myflag", "bat", "baz"})
expect(t, c.IsSet("myflag"), true)
expect(t, c.IsSet("otherflag"), false)
expect(t, c.IsSet("bogusflag"), false)
}

View File

@@ -1,410 +0,0 @@
package cli
import (
"flag"
"fmt"
"os"
"strconv"
"strings"
"time"
)
// This flag enables bash-completion for all commands and subcommands
var BashCompletionFlag = BoolFlag{
Name: "generate-bash-completion",
}
// This flag prints the version for the application
var VersionFlag = BoolFlag{
Name: "version, v",
Usage: "print the version",
}
// This flag prints the help for all commands and subcommands
var HelpFlag = BoolFlag{
Name: "help, h",
Usage: "show help",
}
// Flag is a common interface related to parsing flags in cli.
// For more advanced flag parsing techniques, it is recomended that
// this interface be implemented.
type Flag interface {
fmt.Stringer
// Apply Flag settings to the given flag set
Apply(*flag.FlagSet)
getName() string
}
func flagSet(name string, flags []Flag) *flag.FlagSet {
set := flag.NewFlagSet(name, flag.ContinueOnError)
for _, f := range flags {
f.Apply(set)
}
return set
}
func eachName(longName string, fn func(string)) {
parts := strings.Split(longName, ",")
for _, name := range parts {
name = strings.Trim(name, " ")
fn(name)
}
}
// Generic is a generic parseable type identified by a specific flag
type Generic interface {
Set(value string) error
String() string
}
// GenericFlag is the flag type for types implementing Generic
type GenericFlag struct {
Name string
Value Generic
Usage string
EnvVar string
}
func (f GenericFlag) String() string {
return withEnvHint(f.EnvVar, fmt.Sprintf("%s%s %v\t`%v` %s", prefixFor(f.Name), f.Name, f.Value, "-"+f.Name+" option -"+f.Name+" option", f.Usage))
}
func (f GenericFlag) Apply(set *flag.FlagSet) {
val := f.Value
if f.EnvVar != "" {
if envVal := os.Getenv(f.EnvVar); envVal != "" {
val.Set(envVal)
}
}
eachName(f.Name, func(name string) {
set.Var(f.Value, name, f.Usage)
})
}
func (f GenericFlag) getName() string {
return f.Name
}
type StringSlice []string
func (f *StringSlice) Set(value string) error {
*f = append(*f, value)
return nil
}
func (f *StringSlice) String() string {
return fmt.Sprintf("%s", *f)
}
func (f *StringSlice) Value() []string {
return *f
}
type StringSliceFlag struct {
Name string
Value *StringSlice
Usage string
EnvVar string
}
func (f StringSliceFlag) String() string {
firstName := strings.Trim(strings.Split(f.Name, ",")[0], " ")
pref := prefixFor(firstName)
return withEnvHint(f.EnvVar, fmt.Sprintf("%s '%v'\t%v", prefixedNames(f.Name), pref+firstName+" option "+pref+firstName+" option", f.Usage))
}
func (f StringSliceFlag) Apply(set *flag.FlagSet) {
if f.EnvVar != "" {
if envVal := os.Getenv(f.EnvVar); envVal != "" {
newVal := &StringSlice{}
for _, s := range strings.Split(envVal, ",") {
newVal.Set(s)
}
f.Value = newVal
}
}
eachName(f.Name, func(name string) {
set.Var(f.Value, name, f.Usage)
})
}
func (f StringSliceFlag) getName() string {
return f.Name
}
type IntSlice []int
func (f *IntSlice) Set(value string) error {
tmp, err := strconv.Atoi(value)
if err != nil {
return err
} else {
*f = append(*f, tmp)
}
return nil
}
func (f *IntSlice) String() string {
return fmt.Sprintf("%d", *f)
}
func (f *IntSlice) Value() []int {
return *f
}
type IntSliceFlag struct {
Name string
Value *IntSlice
Usage string
EnvVar string
}
func (f IntSliceFlag) String() string {
firstName := strings.Trim(strings.Split(f.Name, ",")[0], " ")
pref := prefixFor(firstName)
return withEnvHint(f.EnvVar, fmt.Sprintf("%s '%v'\t%v", prefixedNames(f.Name), pref+firstName+" option "+pref+firstName+" option", f.Usage))
}
func (f IntSliceFlag) Apply(set *flag.FlagSet) {
if f.EnvVar != "" {
if envVal := os.Getenv(f.EnvVar); envVal != "" {
newVal := &IntSlice{}
for _, s := range strings.Split(envVal, ",") {
err := newVal.Set(s)
if err != nil {
fmt.Fprintf(os.Stderr, err.Error())
}
}
f.Value = newVal
}
}
eachName(f.Name, func(name string) {
set.Var(f.Value, name, f.Usage)
})
}
func (f IntSliceFlag) getName() string {
return f.Name
}
type BoolFlag struct {
Name string
Usage string
EnvVar string
}
func (f BoolFlag) String() string {
return withEnvHint(f.EnvVar, fmt.Sprintf("%s\t%v", prefixedNames(f.Name), f.Usage))
}
func (f BoolFlag) Apply(set *flag.FlagSet) {
val := false
if f.EnvVar != "" {
if envVal := os.Getenv(f.EnvVar); envVal != "" {
envValBool, err := strconv.ParseBool(envVal)
if err == nil {
val = envValBool
}
}
}
eachName(f.Name, func(name string) {
set.Bool(name, val, f.Usage)
})
}
func (f BoolFlag) getName() string {
return f.Name
}
type BoolTFlag struct {
Name string
Usage string
EnvVar string
}
func (f BoolTFlag) String() string {
return withEnvHint(f.EnvVar, fmt.Sprintf("%s\t%v", prefixedNames(f.Name), f.Usage))
}
func (f BoolTFlag) Apply(set *flag.FlagSet) {
val := true
if f.EnvVar != "" {
if envVal := os.Getenv(f.EnvVar); envVal != "" {
envValBool, err := strconv.ParseBool(envVal)
if err == nil {
val = envValBool
}
}
}
eachName(f.Name, func(name string) {
set.Bool(name, val, f.Usage)
})
}
func (f BoolTFlag) getName() string {
return f.Name
}
type StringFlag struct {
Name string
Value string
Usage string
EnvVar string
}
func (f StringFlag) String() string {
var fmtString string
fmtString = "%s %v\t%v"
if len(f.Value) > 0 {
fmtString = "%s '%v'\t%v"
} else {
fmtString = "%s %v\t%v"
}
return withEnvHint(f.EnvVar, fmt.Sprintf(fmtString, prefixedNames(f.Name), f.Value, f.Usage))
}
func (f StringFlag) Apply(set *flag.FlagSet) {
if f.EnvVar != "" {
if envVal := os.Getenv(f.EnvVar); envVal != "" {
f.Value = envVal
}
}
eachName(f.Name, func(name string) {
set.String(name, f.Value, f.Usage)
})
}
func (f StringFlag) getName() string {
return f.Name
}
type IntFlag struct {
Name string
Value int
Usage string
EnvVar string
}
func (f IntFlag) String() string {
return withEnvHint(f.EnvVar, fmt.Sprintf("%s '%v'\t%v", prefixedNames(f.Name), f.Value, f.Usage))
}
func (f IntFlag) Apply(set *flag.FlagSet) {
if f.EnvVar != "" {
if envVal := os.Getenv(f.EnvVar); envVal != "" {
envValInt, err := strconv.ParseUint(envVal, 10, 64)
if err == nil {
f.Value = int(envValInt)
}
}
}
eachName(f.Name, func(name string) {
set.Int(name, f.Value, f.Usage)
})
}
func (f IntFlag) getName() string {
return f.Name
}
type DurationFlag struct {
Name string
Value time.Duration
Usage string
EnvVar string
}
func (f DurationFlag) String() string {
return withEnvHint(f.EnvVar, fmt.Sprintf("%s '%v'\t%v", prefixedNames(f.Name), f.Value, f.Usage))
}
func (f DurationFlag) Apply(set *flag.FlagSet) {
if f.EnvVar != "" {
if envVal := os.Getenv(f.EnvVar); envVal != "" {
envValDuration, err := time.ParseDuration(envVal)
if err == nil {
f.Value = envValDuration
}
}
}
eachName(f.Name, func(name string) {
set.Duration(name, f.Value, f.Usage)
})
}
func (f DurationFlag) getName() string {
return f.Name
}
type Float64Flag struct {
Name string
Value float64
Usage string
EnvVar string
}
func (f Float64Flag) String() string {
return withEnvHint(f.EnvVar, fmt.Sprintf("%s '%v'\t%v", prefixedNames(f.Name), f.Value, f.Usage))
}
func (f Float64Flag) Apply(set *flag.FlagSet) {
if f.EnvVar != "" {
if envVal := os.Getenv(f.EnvVar); envVal != "" {
envValFloat, err := strconv.ParseFloat(envVal, 10)
if err == nil {
f.Value = float64(envValFloat)
}
}
}
eachName(f.Name, func(name string) {
set.Float64(name, f.Value, f.Usage)
})
}
func (f Float64Flag) getName() string {
return f.Name
}
func prefixFor(name string) (prefix string) {
if len(name) == 1 {
prefix = "-"
} else {
prefix = "--"
}
return
}
func prefixedNames(fullName string) (prefixed string) {
parts := strings.Split(fullName, ",")
for i, name := range parts {
name = strings.Trim(name, " ")
prefixed += prefixFor(name) + name
if i < len(parts)-1 {
prefixed += ", "
}
}
return
}
func withEnvHint(envVar, str string) string {
envText := ""
if envVar != "" {
envText = fmt.Sprintf(" [$%s]", envVar)
}
return str + envText
}

View File

@@ -1,587 +0,0 @@
package cli_test
import (
"fmt"
"os"
"reflect"
"strings"
"testing"
"github.com/codegangsta/cli"
)
var boolFlagTests = []struct {
name string
expected string
}{
{"help", "--help\t"},
{"h", "-h\t"},
}
func TestBoolFlagHelpOutput(t *testing.T) {
for _, test := range boolFlagTests {
flag := cli.BoolFlag{Name: test.name}
output := flag.String()
if output != test.expected {
t.Errorf("%s does not match %s", output, test.expected)
}
}
}
var stringFlagTests = []struct {
name string
value string
expected string
}{
{"help", "", "--help \t"},
{"h", "", "-h \t"},
{"h", "", "-h \t"},
{"test", "Something", "--test 'Something'\t"},
}
func TestStringFlagHelpOutput(t *testing.T) {
for _, test := range stringFlagTests {
flag := cli.StringFlag{Name: test.name, Value: test.value}
output := flag.String()
if output != test.expected {
t.Errorf("%s does not match %s", output, test.expected)
}
}
}
func TestStringFlagWithEnvVarHelpOutput(t *testing.T) {
os.Setenv("APP_FOO", "derp")
for _, test := range stringFlagTests {
flag := cli.StringFlag{Name: test.name, Value: test.value, EnvVar: "APP_FOO"}
output := flag.String()
if !strings.HasSuffix(output, " [$APP_FOO]") {
t.Errorf("%s does not end with [$APP_FOO]", output)
}
}
}
var stringSliceFlagTests = []struct {
name string
value *cli.StringSlice
expected string
}{
{"help", func() *cli.StringSlice {
s := &cli.StringSlice{}
s.Set("")
return s
}(), "--help '--help option --help option'\t"},
{"h", func() *cli.StringSlice {
s := &cli.StringSlice{}
s.Set("")
return s
}(), "-h '-h option -h option'\t"},
{"h", func() *cli.StringSlice {
s := &cli.StringSlice{}
s.Set("")
return s
}(), "-h '-h option -h option'\t"},
{"test", func() *cli.StringSlice {
s := &cli.StringSlice{}
s.Set("Something")
return s
}(), "--test '--test option --test option'\t"},
}
func TestStringSliceFlagHelpOutput(t *testing.T) {
for _, test := range stringSliceFlagTests {
flag := cli.StringSliceFlag{Name: test.name, Value: test.value}
output := flag.String()
if output != test.expected {
t.Errorf("%q does not match %q", output, test.expected)
}
}
}
func TestStringSliceFlagWithEnvVarHelpOutput(t *testing.T) {
os.Setenv("APP_QWWX", "11,4")
for _, test := range stringSliceFlagTests {
flag := cli.StringSliceFlag{Name: test.name, Value: test.value, EnvVar: "APP_QWWX"}
output := flag.String()
if !strings.HasSuffix(output, " [$APP_QWWX]") {
t.Errorf("%q does not end with [$APP_QWWX]", output)
}
}
}
var intFlagTests = []struct {
name string
expected string
}{
{"help", "--help '0'\t"},
{"h", "-h '0'\t"},
}
func TestIntFlagHelpOutput(t *testing.T) {
for _, test := range intFlagTests {
flag := cli.IntFlag{Name: test.name}
output := flag.String()
if output != test.expected {
t.Errorf("%s does not match %s", output, test.expected)
}
}
}
func TestIntFlagWithEnvVarHelpOutput(t *testing.T) {
os.Setenv("APP_BAR", "2")
for _, test := range intFlagTests {
flag := cli.IntFlag{Name: test.name, EnvVar: "APP_BAR"}
output := flag.String()
if !strings.HasSuffix(output, " [$APP_BAR]") {
t.Errorf("%s does not end with [$APP_BAR]", output)
}
}
}
var durationFlagTests = []struct {
name string
expected string
}{
{"help", "--help '0'\t"},
{"h", "-h '0'\t"},
}
func TestDurationFlagHelpOutput(t *testing.T) {
for _, test := range durationFlagTests {
flag := cli.DurationFlag{Name: test.name}
output := flag.String()
if output != test.expected {
t.Errorf("%s does not match %s", output, test.expected)
}
}
}
func TestDurationFlagWithEnvVarHelpOutput(t *testing.T) {
os.Setenv("APP_BAR", "2h3m6s")
for _, test := range durationFlagTests {
flag := cli.DurationFlag{Name: test.name, EnvVar: "APP_BAR"}
output := flag.String()
if !strings.HasSuffix(output, " [$APP_BAR]") {
t.Errorf("%s does not end with [$APP_BAR]", output)
}
}
}
var intSliceFlagTests = []struct {
name string
value *cli.IntSlice
expected string
}{
{"help", &cli.IntSlice{}, "--help '--help option --help option'\t"},
{"h", &cli.IntSlice{}, "-h '-h option -h option'\t"},
{"h", &cli.IntSlice{}, "-h '-h option -h option'\t"},
{"test", func() *cli.IntSlice {
i := &cli.IntSlice{}
i.Set("9")
return i
}(), "--test '--test option --test option'\t"},
}
func TestIntSliceFlagHelpOutput(t *testing.T) {
for _, test := range intSliceFlagTests {
flag := cli.IntSliceFlag{Name: test.name, Value: test.value}
output := flag.String()
if output != test.expected {
t.Errorf("%q does not match %q", output, test.expected)
}
}
}
func TestIntSliceFlagWithEnvVarHelpOutput(t *testing.T) {
os.Setenv("APP_SMURF", "42,3")
for _, test := range intSliceFlagTests {
flag := cli.IntSliceFlag{Name: test.name, Value: test.value, EnvVar: "APP_SMURF"}
output := flag.String()
if !strings.HasSuffix(output, " [$APP_SMURF]") {
t.Errorf("%q does not end with [$APP_SMURF]", output)
}
}
}
var float64FlagTests = []struct {
name string
expected string
}{
{"help", "--help '0'\t"},
{"h", "-h '0'\t"},
}
func TestFloat64FlagHelpOutput(t *testing.T) {
for _, test := range float64FlagTests {
flag := cli.Float64Flag{Name: test.name}
output := flag.String()
if output != test.expected {
t.Errorf("%s does not match %s", output, test.expected)
}
}
}
func TestFloat64FlagWithEnvVarHelpOutput(t *testing.T) {
os.Setenv("APP_BAZ", "99.4")
for _, test := range float64FlagTests {
flag := cli.Float64Flag{Name: test.name, EnvVar: "APP_BAZ"}
output := flag.String()
if !strings.HasSuffix(output, " [$APP_BAZ]") {
t.Errorf("%s does not end with [$APP_BAZ]", output)
}
}
}
var genericFlagTests = []struct {
name string
value cli.Generic
expected string
}{
{"help", &Parser{}, "--help <nil>\t`-help option -help option` "},
{"h", &Parser{}, "-h <nil>\t`-h option -h option` "},
{"test", &Parser{}, "--test <nil>\t`-test option -test option` "},
}
func TestGenericFlagHelpOutput(t *testing.T) {
for _, test := range genericFlagTests {
flag := cli.GenericFlag{Name: test.name}
output := flag.String()
if output != test.expected {
t.Errorf("%q does not match %q", output, test.expected)
}
}
}
func TestGenericFlagWithEnvVarHelpOutput(t *testing.T) {
os.Setenv("APP_ZAP", "3")
for _, test := range genericFlagTests {
flag := cli.GenericFlag{Name: test.name, EnvVar: "APP_ZAP"}
output := flag.String()
if !strings.HasSuffix(output, " [$APP_ZAP]") {
t.Errorf("%s does not end with [$APP_ZAP]", output)
}
}
}
func TestParseMultiString(t *testing.T) {
(&cli.App{
Flags: []cli.Flag{
cli.StringFlag{Name: "serve, s"},
},
Action: func(ctx *cli.Context) {
if ctx.String("serve") != "10" {
t.Errorf("main name not set")
}
if ctx.String("s") != "10" {
t.Errorf("short name not set")
}
},
}).Run([]string{"run", "-s", "10"})
}
func TestParseMultiStringFromEnv(t *testing.T) {
os.Setenv("APP_COUNT", "20")
(&cli.App{
Flags: []cli.Flag{
cli.StringFlag{Name: "count, c", EnvVar: "APP_COUNT"},
},
Action: func(ctx *cli.Context) {
if ctx.String("count") != "20" {
t.Errorf("main name not set")
}
if ctx.String("c") != "20" {
t.Errorf("short name not set")
}
},
}).Run([]string{"run"})
}
func TestParseMultiStringSlice(t *testing.T) {
(&cli.App{
Flags: []cli.Flag{
cli.StringSliceFlag{Name: "serve, s", Value: &cli.StringSlice{}},
},
Action: func(ctx *cli.Context) {
if !reflect.DeepEqual(ctx.StringSlice("serve"), []string{"10", "20"}) {
t.Errorf("main name not set")
}
if !reflect.DeepEqual(ctx.StringSlice("s"), []string{"10", "20"}) {
t.Errorf("short name not set")
}
},
}).Run([]string{"run", "-s", "10", "-s", "20"})
}
func TestParseMultiStringSliceFromEnv(t *testing.T) {
os.Setenv("APP_INTERVALS", "20,30,40")
(&cli.App{
Flags: []cli.Flag{
cli.StringSliceFlag{Name: "intervals, i", Value: &cli.StringSlice{}, EnvVar: "APP_INTERVALS"},
},
Action: func(ctx *cli.Context) {
if !reflect.DeepEqual(ctx.StringSlice("intervals"), []string{"20", "30", "40"}) {
t.Errorf("main name not set from env")
}
if !reflect.DeepEqual(ctx.StringSlice("i"), []string{"20", "30", "40"}) {
t.Errorf("short name not set from env")
}
},
}).Run([]string{"run"})
}
func TestParseMultiInt(t *testing.T) {
a := cli.App{
Flags: []cli.Flag{
cli.IntFlag{Name: "serve, s"},
},
Action: func(ctx *cli.Context) {
if ctx.Int("serve") != 10 {
t.Errorf("main name not set")
}
if ctx.Int("s") != 10 {
t.Errorf("short name not set")
}
},
}
a.Run([]string{"run", "-s", "10"})
}
func TestParseMultiIntFromEnv(t *testing.T) {
os.Setenv("APP_TIMEOUT_SECONDS", "10")
a := cli.App{
Flags: []cli.Flag{
cli.IntFlag{Name: "timeout, t", EnvVar: "APP_TIMEOUT_SECONDS"},
},
Action: func(ctx *cli.Context) {
if ctx.Int("timeout") != 10 {
t.Errorf("main name not set")
}
if ctx.Int("t") != 10 {
t.Errorf("short name not set")
}
},
}
a.Run([]string{"run"})
}
func TestParseMultiIntSlice(t *testing.T) {
(&cli.App{
Flags: []cli.Flag{
cli.IntSliceFlag{Name: "serve, s", Value: &cli.IntSlice{}},
},
Action: func(ctx *cli.Context) {
if !reflect.DeepEqual(ctx.IntSlice("serve"), []int{10, 20}) {
t.Errorf("main name not set")
}
if !reflect.DeepEqual(ctx.IntSlice("s"), []int{10, 20}) {
t.Errorf("short name not set")
}
},
}).Run([]string{"run", "-s", "10", "-s", "20"})
}
func TestParseMultiIntSliceFromEnv(t *testing.T) {
os.Setenv("APP_INTERVALS", "20,30,40")
(&cli.App{
Flags: []cli.Flag{
cli.IntSliceFlag{Name: "intervals, i", Value: &cli.IntSlice{}, EnvVar: "APP_INTERVALS"},
},
Action: func(ctx *cli.Context) {
if !reflect.DeepEqual(ctx.IntSlice("intervals"), []int{20, 30, 40}) {
t.Errorf("main name not set from env")
}
if !reflect.DeepEqual(ctx.IntSlice("i"), []int{20, 30, 40}) {
t.Errorf("short name not set from env")
}
},
}).Run([]string{"run"})
}
func TestParseMultiFloat64(t *testing.T) {
a := cli.App{
Flags: []cli.Flag{
cli.Float64Flag{Name: "serve, s"},
},
Action: func(ctx *cli.Context) {
if ctx.Float64("serve") != 10.2 {
t.Errorf("main name not set")
}
if ctx.Float64("s") != 10.2 {
t.Errorf("short name not set")
}
},
}
a.Run([]string{"run", "-s", "10.2"})
}
func TestParseMultiFloat64FromEnv(t *testing.T) {
os.Setenv("APP_TIMEOUT_SECONDS", "15.5")
a := cli.App{
Flags: []cli.Flag{
cli.Float64Flag{Name: "timeout, t", EnvVar: "APP_TIMEOUT_SECONDS"},
},
Action: func(ctx *cli.Context) {
if ctx.Float64("timeout") != 15.5 {
t.Errorf("main name not set")
}
if ctx.Float64("t") != 15.5 {
t.Errorf("short name not set")
}
},
}
a.Run([]string{"run"})
}
func TestParseMultiBool(t *testing.T) {
a := cli.App{
Flags: []cli.Flag{
cli.BoolFlag{Name: "serve, s"},
},
Action: func(ctx *cli.Context) {
if ctx.Bool("serve") != true {
t.Errorf("main name not set")
}
if ctx.Bool("s") != true {
t.Errorf("short name not set")
}
},
}
a.Run([]string{"run", "--serve"})
}
func TestParseMultiBoolFromEnv(t *testing.T) {
os.Setenv("APP_DEBUG", "1")
a := cli.App{
Flags: []cli.Flag{
cli.BoolFlag{Name: "debug, d", EnvVar: "APP_DEBUG"},
},
Action: func(ctx *cli.Context) {
if ctx.Bool("debug") != true {
t.Errorf("main name not set from env")
}
if ctx.Bool("d") != true {
t.Errorf("short name not set from env")
}
},
}
a.Run([]string{"run"})
}
func TestParseMultiBoolT(t *testing.T) {
a := cli.App{
Flags: []cli.Flag{
cli.BoolTFlag{Name: "serve, s"},
},
Action: func(ctx *cli.Context) {
if ctx.BoolT("serve") != true {
t.Errorf("main name not set")
}
if ctx.BoolT("s") != true {
t.Errorf("short name not set")
}
},
}
a.Run([]string{"run", "--serve"})
}
func TestParseMultiBoolTFromEnv(t *testing.T) {
os.Setenv("APP_DEBUG", "0")
a := cli.App{
Flags: []cli.Flag{
cli.BoolTFlag{Name: "debug, d", EnvVar: "APP_DEBUG"},
},
Action: func(ctx *cli.Context) {
if ctx.BoolT("debug") != false {
t.Errorf("main name not set from env")
}
if ctx.BoolT("d") != false {
t.Errorf("short name not set from env")
}
},
}
a.Run([]string{"run"})
}
type Parser [2]string
func (p *Parser) Set(value string) error {
parts := strings.Split(value, ",")
if len(parts) != 2 {
return fmt.Errorf("invalid format")
}
(*p)[0] = parts[0]
(*p)[1] = parts[1]
return nil
}
func (p *Parser) String() string {
return fmt.Sprintf("%s,%s", p[0], p[1])
}
func TestParseGeneric(t *testing.T) {
a := cli.App{
Flags: []cli.Flag{
cli.GenericFlag{Name: "serve, s", Value: &Parser{}},
},
Action: func(ctx *cli.Context) {
if !reflect.DeepEqual(ctx.Generic("serve"), &Parser{"10", "20"}) {
t.Errorf("main name not set")
}
if !reflect.DeepEqual(ctx.Generic("s"), &Parser{"10", "20"}) {
t.Errorf("short name not set")
}
},
}
a.Run([]string{"run", "-s", "10,20"})
}
func TestParseGenericFromEnv(t *testing.T) {
os.Setenv("APP_SERVE", "20,30")
a := cli.App{
Flags: []cli.Flag{
cli.GenericFlag{Name: "serve, s", Value: &Parser{}, EnvVar: "APP_SERVE"},
},
Action: func(ctx *cli.Context) {
if !reflect.DeepEqual(ctx.Generic("serve"), &Parser{"20", "30"}) {
t.Errorf("main name not set from env")
}
if !reflect.DeepEqual(ctx.Generic("s"), &Parser{"20", "30"}) {
t.Errorf("short name not set from env")
}
},
}
a.Run([]string{"run"})
}

View File

@@ -1,224 +0,0 @@
package cli
import (
"fmt"
"os"
"text/tabwriter"
"text/template"
)
// The text template for the Default help topic.
// cli.go uses text/template to render templates. You can
// render custom help text by setting this variable.
var AppHelpTemplate = `NAME:
{{.Name}} - {{.Usage}}
USAGE:
{{.Name}} {{if .Flags}}[global options] {{end}}command{{if .Flags}} [command options]{{end}} [arguments...]
VERSION:
{{.Version}}{{if or .Author .Email}}
AUTHOR:{{if .Author}}
{{.Author}}{{if .Email}} - <{{.Email}}>{{end}}{{else}}
{{.Email}}{{end}}{{end}}
COMMANDS:
{{range .Commands}}{{.Name}}{{with .ShortName}}, {{.}}{{end}}{{ "\t" }}{{.Usage}}
{{end}}{{if .Flags}}
GLOBAL OPTIONS:
{{range .Flags}}{{.}}
{{end}}{{end}}
`
// The text template for the command help topic.
// cli.go uses text/template to render templates. You can
// render custom help text by setting this variable.
var CommandHelpTemplate = `NAME:
{{.Name}} - {{.Usage}}
USAGE:
command {{.Name}}{{if .Flags}} [command options]{{end}} [arguments...]{{if .Description}}
DESCRIPTION:
{{.Description}}{{end}}{{if .Flags}}
OPTIONS:
{{range .Flags}}{{.}}
{{end}}{{ end }}
`
// The text template for the subcommand help topic.
// cli.go uses text/template to render templates. You can
// render custom help text by setting this variable.
var SubcommandHelpTemplate = `NAME:
{{.Name}} - {{.Usage}}
USAGE:
{{.Name}} command{{if .Flags}} [command options]{{end}} [arguments...]
COMMANDS:
{{range .Commands}}{{.Name}}{{with .ShortName}}, {{.}}{{end}}{{ "\t" }}{{.Usage}}
{{end}}{{if .Flags}}
OPTIONS:
{{range .Flags}}{{.}}
{{end}}{{end}}
`
var helpCommand = Command{
Name: "help",
ShortName: "h",
Usage: "Shows a list of commands or help for one command",
Action: func(c *Context) {
args := c.Args()
if args.Present() {
ShowCommandHelp(c, args.First())
} else {
ShowAppHelp(c)
}
},
}
var helpSubcommand = Command{
Name: "help",
ShortName: "h",
Usage: "Shows a list of commands or help for one command",
Action: func(c *Context) {
args := c.Args()
if args.Present() {
ShowCommandHelp(c, args.First())
} else {
ShowSubcommandHelp(c)
}
},
}
// Prints help for the App
var HelpPrinter = printHelp
// Prints version for the App
var VersionPrinter = printVersion
func ShowAppHelp(c *Context) {
HelpPrinter(AppHelpTemplate, c.App)
}
// Prints the list of subcommands as the default app completion method
func DefaultAppComplete(c *Context) {
for _, command := range c.App.Commands {
fmt.Println(command.Name)
if command.ShortName != "" {
fmt.Println(command.ShortName)
}
}
}
// Prints help for the given command
func ShowCommandHelp(c *Context, command string) {
for _, c := range c.App.Commands {
if c.HasName(command) {
HelpPrinter(CommandHelpTemplate, c)
return
}
}
if c.App.CommandNotFound != nil {
c.App.CommandNotFound(c, command)
} else {
fmt.Printf("No help topic for '%v'\n", command)
}
}
// Prints help for the given subcommand
func ShowSubcommandHelp(c *Context) {
HelpPrinter(SubcommandHelpTemplate, c.App)
}
// Prints the version number of the App
func ShowVersion(c *Context) {
VersionPrinter(c)
}
func printVersion(c *Context) {
fmt.Printf("%v version %v\n", c.App.Name, c.App.Version)
}
// Prints the lists of commands within a given context
func ShowCompletions(c *Context) {
a := c.App
if a != nil && a.BashComplete != nil {
a.BashComplete(c)
}
}
// Prints the custom completions for a given command
func ShowCommandCompletions(ctx *Context, command string) {
c := ctx.App.Command(command)
if c != nil && c.BashComplete != nil {
c.BashComplete(ctx)
}
}
func printHelp(templ string, data interface{}) {
w := tabwriter.NewWriter(os.Stdout, 0, 8, 1, '\t', 0)
t := template.Must(template.New("help").Parse(templ))
err := t.Execute(w, data)
if err != nil {
panic(err)
}
w.Flush()
}
func checkVersion(c *Context) bool {
if c.GlobalBool("version") {
ShowVersion(c)
return true
}
return false
}
func checkHelp(c *Context) bool {
if c.GlobalBool("h") || c.GlobalBool("help") {
ShowAppHelp(c)
return true
}
return false
}
func checkCommandHelp(c *Context, name string) bool {
if c.Bool("h") || c.Bool("help") {
ShowCommandHelp(c, name)
return true
}
return false
}
func checkSubcommandHelp(c *Context) bool {
if c.GlobalBool("h") || c.GlobalBool("help") {
ShowSubcommandHelp(c)
return true
}
return false
}
func checkCompletions(c *Context) bool {
if c.GlobalBool(BashCompletionFlag.Name) && c.App.EnableBashCompletion {
ShowCompletions(c)
return true
}
return false
}
func checkCommandCompletions(c *Context, name string) bool {
if c.Bool(BashCompletionFlag.Name) && c.App.EnableBashCompletion {
ShowCommandCompletions(c, name)
return true
}
return false
}

View File

@@ -1,19 +0,0 @@
package cli_test
import (
"reflect"
"testing"
)
/* Test Helpers */
func expect(t *testing.T, a interface{}, b interface{}) {
if a != b {
t.Errorf("Expected %v (type %v) - Got %v (type %v)", b, reflect.TypeOf(b), a, reflect.TypeOf(a))
}
}
func refute(t *testing.T, a interface{}, b interface{}) {
if a == b {
t.Errorf("Did not expect %v (type %v) - Got %v (type %v)", b, reflect.TypeOf(b), a, reflect.TypeOf(a))
}
}

View File

@@ -1,55 +0,0 @@
package etcd
import (
"fmt"
"io/ioutil"
"log"
"strings"
)
var logger *etcdLogger
func SetLogger(l *log.Logger) {
logger = &etcdLogger{l}
}
func GetLogger() *log.Logger {
return logger.log
}
type etcdLogger struct {
log *log.Logger
}
func (p *etcdLogger) Debug(args ...interface{}) {
msg := "DEBUG: " + fmt.Sprint(args...)
p.log.Println(msg)
}
func (p *etcdLogger) Debugf(f string, args ...interface{}) {
msg := "DEBUG: " + fmt.Sprintf(f, args...)
// Append newline if necessary
if !strings.HasSuffix(msg, "\n") {
msg = msg + "\n"
}
p.log.Print(msg)
}
func (p *etcdLogger) Warning(args ...interface{}) {
msg := "WARNING: " + fmt.Sprint(args...)
p.log.Println(msg)
}
func (p *etcdLogger) Warningf(f string, args ...interface{}) {
msg := "WARNING: " + fmt.Sprintf(f, args...)
// Append newline if necessary
if !strings.HasSuffix(msg, "\n") {
msg = msg + "\n"
}
p.log.Print(msg)
}
func init() {
// Default logger uses the go default log.
SetLogger(log.New(ioutil.Discard, "go-etcd", log.LstdFlags))
}

View File

@@ -1,28 +0,0 @@
package etcd
import (
"testing"
)
type Foo struct{}
type Bar struct {
one string
two int
}
// Tests that logs don't panic with arbitrary interfaces
func TestDebug(t *testing.T) {
f := &Foo{}
b := &Bar{"asfd", 3}
for _, test := range []interface{}{
1234,
"asdf",
f,
b,
} {
logger.Debug(test)
logger.Debugf("something, %s", test)
logger.Warning(test)
logger.Warningf("something, %s", test)
}
}

View File

@@ -1,396 +0,0 @@
package etcd
import (
"errors"
"fmt"
"io"
"io/ioutil"
"math/rand"
"net/http"
"net/url"
"path"
"strings"
"sync"
"time"
)
// Errors introduced by handling requests
var (
ErrRequestCancelled = errors.New("sending request is cancelled")
)
type RawRequest struct {
Method string
RelativePath string
Values url.Values
Cancel <-chan bool
}
// NewRawRequest returns a new RawRequest
func NewRawRequest(method, relativePath string, values url.Values, cancel <-chan bool) *RawRequest {
return &RawRequest{
Method: method,
RelativePath: relativePath,
Values: values,
Cancel: cancel,
}
}
// getCancelable issues a cancelable GET request
func (c *Client) getCancelable(key string, options Options,
cancel <-chan bool) (*RawResponse, error) {
logger.Debugf("get %s [%s]", key, c.cluster.Leader)
p := keyToPath(key)
// If consistency level is set to STRONG, append
// the `consistent` query string.
if c.config.Consistency == STRONG_CONSISTENCY {
options["consistent"] = true
}
str, err := options.toParameters(VALID_GET_OPTIONS)
if err != nil {
return nil, err
}
p += str
req := NewRawRequest("GET", p, nil, cancel)
resp, err := c.SendRequest(req)
if err != nil {
return nil, err
}
return resp, nil
}
// get issues a GET request
func (c *Client) get(key string, options Options) (*RawResponse, error) {
return c.getCancelable(key, options, nil)
}
// put issues a PUT request
func (c *Client) put(key string, value string, ttl uint64,
options Options) (*RawResponse, error) {
logger.Debugf("put %s, %s, ttl: %d, [%s]", key, value, ttl, c.cluster.Leader)
p := keyToPath(key)
str, err := options.toParameters(VALID_PUT_OPTIONS)
if err != nil {
return nil, err
}
p += str
req := NewRawRequest("PUT", p, buildValues(value, ttl), nil)
resp, err := c.SendRequest(req)
if err != nil {
return nil, err
}
return resp, nil
}
// post issues a POST request
func (c *Client) post(key string, value string, ttl uint64) (*RawResponse, error) {
logger.Debugf("post %s, %s, ttl: %d, [%s]", key, value, ttl, c.cluster.Leader)
p := keyToPath(key)
req := NewRawRequest("POST", p, buildValues(value, ttl), nil)
resp, err := c.SendRequest(req)
if err != nil {
return nil, err
}
return resp, nil
}
// delete issues a DELETE request
func (c *Client) delete(key string, options Options) (*RawResponse, error) {
logger.Debugf("delete %s [%s]", key, c.cluster.Leader)
p := keyToPath(key)
str, err := options.toParameters(VALID_DELETE_OPTIONS)
if err != nil {
return nil, err
}
p += str
req := NewRawRequest("DELETE", p, nil, nil)
resp, err := c.SendRequest(req)
if err != nil {
return nil, err
}
return resp, nil
}
// SendRequest sends a HTTP request and returns a Response as defined by etcd
func (c *Client) SendRequest(rr *RawRequest) (*RawResponse, error) {
var req *http.Request
var resp *http.Response
var httpPath string
var err error
var respBody []byte
var numReqs = 1
checkRetry := c.CheckRetry
if checkRetry == nil {
checkRetry = DefaultCheckRetry
}
cancelled := make(chan bool, 1)
reqLock := new(sync.Mutex)
if rr.Cancel != nil {
cancelRoutine := make(chan bool)
defer close(cancelRoutine)
go func() {
select {
case <-rr.Cancel:
cancelled <- true
logger.Debug("send.request is cancelled")
case <-cancelRoutine:
return
}
// Repeat canceling request until this thread is stopped
// because we have no idea about whether it succeeds.
for {
reqLock.Lock()
c.httpClient.Transport.(*http.Transport).CancelRequest(req)
reqLock.Unlock()
select {
case <-time.After(100 * time.Millisecond):
case <-cancelRoutine:
return
}
}
}()
}
// If we connect to a follower and consistency is required, retry until
// we connect to a leader
sleep := 25 * time.Millisecond
maxSleep := time.Second
for attempt := 0; ; attempt++ {
if attempt > 0 {
select {
case <-cancelled:
return nil, ErrRequestCancelled
case <-time.After(sleep):
sleep = sleep * 2
if sleep > maxSleep {
sleep = maxSleep
}
}
}
logger.Debug("Connecting to etcd: attempt ", attempt+1, " for ", rr.RelativePath)
if rr.Method == "GET" && c.config.Consistency == WEAK_CONSISTENCY {
// If it's a GET and consistency level is set to WEAK,
// then use a random machine.
httpPath = c.getHttpPath(true, rr.RelativePath)
} else {
// Else use the leader.
httpPath = c.getHttpPath(false, rr.RelativePath)
}
// Return a cURL command if curlChan is set
if c.cURLch != nil {
command := fmt.Sprintf("curl -X %s %s", rr.Method, httpPath)
for key, value := range rr.Values {
command += fmt.Sprintf(" -d %s=%s", key, value[0])
}
c.sendCURL(command)
}
logger.Debug("send.request.to ", httpPath, " | method ", rr.Method)
req, err := func() (*http.Request, error) {
reqLock.Lock()
defer reqLock.Unlock()
if rr.Values == nil {
if req, err = http.NewRequest(rr.Method, httpPath, nil); err != nil {
return nil, err
}
} else {
body := strings.NewReader(rr.Values.Encode())
if req, err = http.NewRequest(rr.Method, httpPath, body); err != nil {
return nil, err
}
req.Header.Set("Content-Type",
"application/x-www-form-urlencoded; param=value")
}
return req, nil
}()
if err != nil {
return nil, err
}
resp, err = c.httpClient.Do(req)
defer func() {
if resp != nil {
resp.Body.Close()
}
}()
// If the request was cancelled, return ErrRequestCancelled directly
select {
case <-cancelled:
return nil, ErrRequestCancelled
default:
}
numReqs++
// network error, change a machine!
if err != nil {
logger.Debug("network error: ", err.Error())
lastResp := http.Response{}
if checkErr := checkRetry(c.cluster, numReqs, lastResp, err); checkErr != nil {
return nil, checkErr
}
c.cluster.switchLeader(attempt % len(c.cluster.Machines))
continue
}
// if there is no error, it should receive response
logger.Debug("recv.response.from ", httpPath)
if validHttpStatusCode[resp.StatusCode] {
// try to read byte code and break the loop
respBody, err = ioutil.ReadAll(resp.Body)
if err == nil {
logger.Debug("recv.success ", httpPath)
break
}
// ReadAll error may be caused due to cancel request
select {
case <-cancelled:
return nil, ErrRequestCancelled
default:
}
if err == io.ErrUnexpectedEOF {
// underlying connection was closed prematurely, probably by timeout
// TODO: empty body or unexpectedEOF can cause http.Transport to get hosed;
// this allows the client to detect that and take evasive action. Need
// to revisit once code.google.com/p/go/issues/detail?id=8648 gets fixed.
respBody = []byte{}
break
}
}
// if resp is TemporaryRedirect, set the new leader and retry
if resp.StatusCode == http.StatusTemporaryRedirect {
u, err := resp.Location()
if err != nil {
logger.Warning(err)
} else {
// Update cluster leader based on redirect location
// because it should point to the leader address
c.cluster.updateLeaderFromURL(u)
logger.Debug("recv.response.relocate ", u.String())
}
resp.Body.Close()
continue
}
if checkErr := checkRetry(c.cluster, numReqs, *resp,
errors.New("Unexpected HTTP status code")); checkErr != nil {
return nil, checkErr
}
resp.Body.Close()
}
r := &RawResponse{
StatusCode: resp.StatusCode,
Body: respBody,
Header: resp.Header,
}
return r, nil
}
// DefaultCheckRetry defines the retrying behaviour for bad HTTP requests
// If we have retried 2 * machine number, stop retrying.
// If status code is InternalServerError, sleep for 200ms.
func DefaultCheckRetry(cluster *Cluster, numReqs int, lastResp http.Response,
err error) error {
if numReqs >= 2*len(cluster.Machines) {
return newError(ErrCodeEtcdNotReachable,
"Tried to connect to each peer twice and failed", 0)
}
code := lastResp.StatusCode
if code == http.StatusInternalServerError {
time.Sleep(time.Millisecond * 200)
}
logger.Warning("bad response status code", code)
return nil
}
func (c *Client) getHttpPath(random bool, s ...string) string {
var machine string
if random {
machine = c.cluster.Machines[rand.Intn(len(c.cluster.Machines))]
} else {
machine = c.cluster.Leader
}
fullPath := machine + "/" + version
for _, seg := range s {
fullPath = fullPath + "/" + seg
}
return fullPath
}
// buildValues builds a url.Values map according to the given value and ttl
func buildValues(value string, ttl uint64) url.Values {
v := url.Values{}
if value != "" {
v.Set("value", value)
}
if ttl > 0 {
v.Set("ttl", fmt.Sprintf("%v", ttl))
}
return v
}
// convert key string to http path exclude version
// for example: key[foo] -> path[keys/foo]
// key[/] -> path[keys/]
func keyToPath(key string) string {
p := path.Join("keys", key)
// corner case: if key is "/" or "//" ect
// path join will clear the tailing "/"
// we need to add it back
if p == "keys" {
p = "keys/"
}
return p
}

View File

@@ -1,58 +0,0 @@
clockwork
=========
a simple fake clock for golang
# Usage
Replace uses of the `time` package with the `clockwork.Clock` interface instead.
For example, instead of using `time.Sleep` directly:
```
func my_func() {
time.Sleep(3 * time.Second)
do_something()
}
```
inject a clock and use its `Sleep` method instead:
```
func my_func(clock clockwork.Clock) {
clock.Sleep(3 * time.Second)
do_something()
}
```
Now you can easily test `my_func` with a `FakeClock`:
```
func TestMyFunc(t *testing.T) {
c := clockwork.NewFakeClock()
// Start our sleepy function
my_func(c)
// Ensure we wait until my_func is sleeping
c.BlockUntil(1)
assert_state()
// Advance the FakeClock forward in time
c.Advance(3)
assert_state()
}
```
and in production builds, simply inject the real clock instead:
```
my_func(clockwork.NewRealClock())
```
See [example_test.go](example_test.go) for a full example.
# Credits
Inspired by @wickman's [threaded fake clock](https://gist.github.com/wickman/3840816), and the [Golang playground](http://blog.golang.org/playground#Faking time)

View File

@@ -1,161 +0,0 @@
package clockwork
import (
"sync"
"time"
)
// Clock provides an interface that packages can use instead of directly
// using the time module, so that chronology-related behavior can be tested
type Clock interface {
After(d time.Duration) <-chan time.Time
Sleep(d time.Duration)
Now() time.Time
}
// FakeClock provides an interface for a clock which can be
// manually advanced through time
type FakeClock interface {
Clock
// Advance advances the FakeClock to a new point in time, ensuring any existing
// sleepers are notified appropriately before returning
Advance(d time.Duration)
// BlockUntil will block until the FakeClock has the given number of
// sleepers (callers of Sleep or After)
BlockUntil(n int)
}
// NewRealClock returns a Clock which simply delegates calls to the actual time
// package; it should be used by packages in production.
func NewRealClock() Clock {
return &realClock{}
}
// NewFakeClock returns a FakeClock implementation which can be
// manually advanced through time for testing.
func NewFakeClock() FakeClock {
return &fakeClock{
l: sync.RWMutex{},
}
}
type realClock struct{}
func (rc *realClock) After(d time.Duration) <-chan time.Time {
return time.After(d)
}
func (rc *realClock) Sleep(d time.Duration) {
time.Sleep(d)
}
func (rc *realClock) Now() time.Time {
return time.Now()
}
type fakeClock struct {
sleepers []*sleeper
blockers []*blocker
time time.Time
l sync.RWMutex
}
// sleeper represents a caller of After or Sleep
type sleeper struct {
until time.Time
done chan time.Time
}
// blocker represents a caller of BlockUntil
type blocker struct {
count int
ch chan struct{}
}
// After mimics time.After; it waits for the given duration to elapse on the
// fakeClock, then sends the current time on the returned channel.
func (fc *fakeClock) After(d time.Duration) <-chan time.Time {
fc.l.Lock()
defer fc.l.Unlock()
now := fc.time
done := make(chan time.Time, 1)
if d.Nanoseconds() == 0 {
// special case - trigger immediately
done <- now
} else {
// otherwise, add to the set of sleepers
s := &sleeper{
until: now.Add(d),
done: done,
}
fc.sleepers = append(fc.sleepers, s)
// and notify any blockers
fc.blockers = notifyBlockers(fc.blockers, len(fc.sleepers))
}
return done
}
// notifyBlockers notifies all the blockers waiting until the
// given number of sleepers are waiting on the fakeClock. It
// returns an updated slice of blockers (i.e. those still waiting)
func notifyBlockers(blockers []*blocker, count int) (newBlockers []*blocker) {
for _, b := range blockers {
if b.count == count {
close(b.ch)
} else {
newBlockers = append(newBlockers, b)
}
}
return
}
// Sleep blocks until the given duration has passed on the fakeClock
func (fc *fakeClock) Sleep(d time.Duration) {
<-fc.After(d)
}
// Time returns the current time of the fakeClock
func (fc *fakeClock) Now() time.Time {
fc.l.Lock()
defer fc.l.Unlock()
return fc.time
}
// Advance advances fakeClock to a new point in time, ensuring channels from any
// previous invocations of After are notified appropriately before returning
func (fc *fakeClock) Advance(d time.Duration) {
fc.l.Lock()
defer fc.l.Unlock()
end := fc.time.Add(d)
var newSleepers []*sleeper
for _, s := range fc.sleepers {
if end.Sub(s.until) >= 0 {
s.done <- end
} else {
newSleepers = append(newSleepers, s)
}
}
fc.sleepers = newSleepers
fc.blockers = notifyBlockers(fc.blockers, len(fc.sleepers))
fc.time = end
}
// BlockUntil will block until the fakeClock has the given number of sleepers
// (callers of Sleep or After)
func (fc *fakeClock) BlockUntil(n int) {
fc.l.Lock()
// Fast path: current number of sleepers is what we're looking for
if len(fc.sleepers) == n {
fc.l.Unlock()
return
}
// Otherwise, set up a new blocker
b := &blocker{
count: n,
ch: make(chan struct{}),
}
fc.blockers = append(fc.blockers, b)
fc.l.Unlock()
<-b.ch
}

View File

@@ -1,106 +0,0 @@
package clockwork
import (
"testing"
"time"
)
func TestFakeClockAfter(t *testing.T) {
fc := &fakeClock{}
zero := fc.After(0)
select {
case <-zero:
default:
t.Errorf("zero did not return!")
}
one := fc.After(1)
two := fc.After(2)
six := fc.After(6)
ten := fc.After(10)
fc.Advance(1)
select {
case <-one:
default:
t.Errorf("one did not return!")
}
select {
case <-two:
t.Errorf("two returned prematurely!")
case <-six:
t.Errorf("six returned prematurely!")
case <-ten:
t.Errorf("ten returned prematurely!")
default:
}
fc.Advance(1)
select {
case <-two:
default:
t.Errorf("two did not return!")
}
select {
case <-six:
t.Errorf("six returned prematurely!")
case <-ten:
t.Errorf("ten returned prematurely!")
default:
}
fc.Advance(1)
select {
case <-six:
t.Errorf("six returned prematurely!")
case <-ten:
t.Errorf("ten returned prematurely!")
default:
}
fc.Advance(3)
select {
case <-six:
default:
t.Errorf("six did not return!")
}
select {
case <-ten:
t.Errorf("ten returned prematurely!")
default:
}
fc.Advance(100)
select {
case <-ten:
default:
t.Errorf("ten did not return!")
}
}
func TestNotifyBlockers(t *testing.T) {
b1 := &blocker{1, make(chan struct{})}
b2 := &blocker{2, make(chan struct{})}
b3 := &blocker{5, make(chan struct{})}
b4 := &blocker{10, make(chan struct{})}
b5 := &blocker{10, make(chan struct{})}
bs := []*blocker{b1, b2, b3, b4, b5}
bs1 := notifyBlockers(bs, 2)
if n := len(bs1); n != 4 {
t.Fatalf("got %d blockers, want %d", n, 4)
}
select {
case <-b2.ch:
case <-time.After(time.Second):
t.Fatalf("timed out waiting for channel close!")
}
bs2 := notifyBlockers(bs1, 10)
if n := len(bs2); n != 2 {
t.Fatalf("got %d blockers, want %d", n, 2)
}
select {
case <-b4.ch:
case <-time.After(time.Second):
t.Fatalf("timed out waiting for channel close!")
}
select {
case <-b5.ch:
case <-time.After(time.Second):
t.Fatalf("timed out waiting for channel close!")
}
}

View File

@@ -1,49 +0,0 @@
package clockwork
import (
"sync"
"testing"
"time"
)
// my_func is an example of a time-dependent function, using an
// injected clock
func my_func(clock Clock, i *int) {
clock.Sleep(3 * time.Second)
*i += 1
}
// assert_state is an example of a state assertion in a test
func assert_state(t *testing.T, i, j int) {
if i != j {
t.Fatalf("i %d, j %d", i, j)
}
}
// TestMyFunc tests my_func's behaviour with a FakeClock
func TestMyFunc(t *testing.T) {
var i int
c := NewFakeClock()
var wg sync.WaitGroup
wg.Add(1)
go func() {
my_func(c, &i)
wg.Done()
}()
// Wait until my_func is actually sleeping on the clock
c.BlockUntil(1)
// Assert the initial state
assert_state(t, i, 0)
// Now advance the clock forward in time
c.Advance(1 * time.Hour)
// Wait until the function completes
wg.Wait()
// Assert the final state
assert_state(t, i, 1)
}

View File

@@ -1,3 +0,0 @@
Blake Mizerany <blake.mizerany@gmail.com> (@bmizerany)
Brandon Philips <brandon.philips@coreos.com> (@philips)
Xiang Li <xiang.li@coreos.com> (@xiangli-cmu)

View File

@@ -1,5 +0,0 @@
# Use goreman to run `go get github.com/mattn/goreman`
etcd1: bin/etcd -name infra1 -listen-client-urls http://127.0.0.1:4001 -advertise-client-urls http://127.0.0.1:4001 -listen-peer-urls http://127.0.0.1:7001 -initial-advertise-peer-urls http://127.0.0.1:7001 -initial-cluster-token etcd-cluster-1 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003' -initial-cluster-state new
etcd2: bin/etcd -name infra2 -listen-client-urls http://127.0.0.1:4002 -advertise-client-urls http://127.0.0.1:4002 -listen-peer-urls http://127.0.0.1:7002 -initial-advertise-peer-urls http://127.0.0.1:7002 -initial-cluster-token etcd-cluster-1 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003' -initial-cluster-state new
etcd3: bin/etcd -name infra3 -listen-client-urls http://127.0.0.1:4003 -advertise-client-urls http://127.0.0.1:4003 -listen-peer-urls http://127.0.0.1:7003 -initial-advertise-peer-urls http://127.0.0.1:7003 -initial-cluster-token etcd-cluster-1 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003' -initial-cluster-state new
proxy: bin/etcd -proxy=on -bind-addr 127.0.0.1:8080 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003'

View File

@@ -1,17 +1,6 @@
# etcd
[![Build Status](https://travis-ci.org/coreos/etcd.png?branch=master)](https://travis-ci.org/coreos/etcd)
[![Docker Repository on Quay.io](https://quay.io/repository/coreos/etcd-git/status "Docker Repository on Quay.io")](https://quay.io/repository/coreos/etcd-git)
### WARNING ###
The current `master` branch of etcd is under heavy development in anticipation of the forthcoming 0.5.0 release.
It is strongly recommended that users work with the latest 0.4.x release (0.4.6), which can be found on the [releases](https://github.com/coreos/etcd/releases) page.
Unless otherwise noted, the etcd documentation refers to configuring and running 0.4.x releases.
## README version 0.4.6
README version 0.4.7
A highly-available key value store for shared configuration and service discovery.
etcd is inspired by [Apache ZooKeeper][zookeeper] and [doozer][doozer], with a focus on being:
@@ -41,7 +30,30 @@ The latest release and setup instructions are available at [GitHub][github-relea
[github-release]: https://github.com/coreos/etcd/releases/
### Running etcd
### Building
You can build etcd from source:
```sh
git clone https://github.com/coreos/etcd
cd etcd
./build
```
This will generate a binary called `./bin/etcd`.
_NOTE_: you need go 1.2+. Please check your installation with
```
go version
```
See the [development tools documentation][development-tools] for alternative build methods like using Vagrant.
[development-tools]: https://github.com/coreos/etcd/blob/master/Documentation/development-tools.md
### Running
First start a single-machine cluster of etcd:
@@ -84,7 +96,7 @@ You have successfully started an etcd on a single machine and written a key to t
- Mailing list: [etcd-dev](https://groups.google.com/forum/?hl=en#!forum/etcd-dev)
- IRC: #[coreos](irc://irc.freenode.org:6667/#coreos) on freenode.org
- Planning/Roadmap: [milestones](https://github.com/coreos/etcd/milestones)
- Planning/Roadmap: [milestones](https://github.com/coreos/etcd/issues/milestones)
- Bugs: [issues](https://github.com/coreos/etcd/issues)
## Contributing
@@ -108,17 +120,15 @@ curl -L http://127.0.0.1:4001/version
#### API Versioning
The `v2` API responses should not change after the 0.2.0 release but new features will be added over time.
Clients are encouraged to use the `v2` API. The `v1` API will not change.
The `v1` API has been deprecated and will not be supported.
The `v2` API responses should not change after the 0.2.0 release but new features will be added over time.
During the pre-v1.0.0 series of releases we may break the API as we fix bugs and get feedback.
#### 32-bit systems
#### 32bit systems
etcd has known issues on 32-bit systems due to a bug in the Go runtime. See #[358][358] for more information.
[358]: https://github.com/coreos/etcd/issues/358
etcd has known issues on 32bit systems. We plan to address these issues after the 0.4 release.
### License

27
Vagrantfile vendored Normal file
View File

@@ -0,0 +1,27 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
#
Vagrant.require_version '>= 1.5.0'
Vagrant.configure("2") do |config|
config.vm.box = "precise64"
config.vm.box_url = "http://files.vagrantup.com/precise64.box"
config.vm.network :forwarded_port, host: 4001, guest: 4001
config.vm.network :forwarded_port, host: 7001, guest: 7001
# Fix docker not being able to resolve private registry in VirtualBox
config.vm.provider :virtualbox do |vb, override|
vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
vb.customize ["modifyvm", :id, "--natdnsproxy1", "on"]
end
config.vm.provision "docker" do |d|
d.build_image "/vagrant", args: '-t etcd'
d.run "etcd", args: "-p 4001:4001 -p 7001:7001", demonize: true
end
# plugin conflict
if Vagrant.has_plugin?("vagrant-vbguest")
config.vbguest.auto_update = false
end
end

79
bench/bench.go Normal file
View File

@@ -0,0 +1,79 @@
package main
import (
"flag"
"log"
"strconv"
"time"
"github.com/coreos/etcd/third_party/github.com/coreos/go-etcd/etcd"
)
var (
// output debug to log
verbose *bool
)
func write(endpoint string, requests int, end chan int) {
client := etcd.NewClient([]string{endpoint})
for i := 0; i < requests; i++ {
key := strconv.Itoa(i)
_, err := client.Set(key, key, 0)
if err != nil {
println(err.Error())
}
}
end <- 1
}
func watch(endpoint string, key string) {
client := etcd.NewClient([]string{endpoint})
receiver := make(chan *etcd.Response)
go client.Watch(key, 0, true, receiver, nil)
if *verbose {
log.Printf("watching: %s", key)
}
received := 0
for {
<-receiver
received++
}
}
func main() {
endpoint := flag.String("endpoint", "http://127.0.0.1:4001", "etcd HTTP endpoint")
rWrites := flag.Int("write-requests", 50000, "number of writes")
cWrites := flag.Int("concurrent-writes", 500, "number of concurrent writes")
watches := flag.Int("watches", 500, "number of concurrent watches")
verbose = flag.Bool("verbose", false, "output debug info")
flag.Parse()
log.Printf("Benchmarking %v", *endpoint)
log.Printf("%v writes with %v concurrent writers and %v watches", *rWrites, *cWrites, *watches)
t := time.Now()
for i := 0; i < *watches; i++ {
key := strconv.Itoa(i)
go watch(*endpoint, key)
}
wChan := make(chan int, *cWrites)
for i := 0; i < *cWrites; i++ {
go write(*endpoint, (*rWrites / *cWrites), wChan)
}
for i := 0; i < *cWrites; i++ {
<-wChan
if *verbose {
log.Printf("Completed %d writes", (*rWrites / *cWrites))
}
}
log.Printf("Took %v", time.Now().Sub(t))
}

23
build
View File

@@ -1,15 +1,18 @@
#!/bin/bash -e
#!/bin/sh -e
ORG_PATH="github.com/coreos"
REPO_PATH="${ORG_PATH}/etcd"
if [ ! -h gopath/src/github.com/coreos/etcd ]; then
mkdir -p gopath/src/github.com/coreos/
ln -s ../../../.. gopath/src/github.com/coreos/etcd
fi
export GOBIN=${PWD}/bin
export GOPATH=${PWD}/gopath
export GOFMTPATH="./bench ./config ./discovery ./etcd ./error ./http ./log main.go ./metrics ./mod ./server ./store ./tests"
rm -f $GOPATH/src/${REPO_PATH}
mkdir -p $GOPATH/src/${ORG_PATH}
ln -s ${PWD} $GOPATH/src/${REPO_PATH}
# Don't surprise user by formatting their codes by stealth
if [ "--fmt" = "$1" ]; then
gofmt -s -w -l $GOFMTPATH
fi
eval $(go env)
go build -o bin/etcd ${REPO_PATH}
go build -o bin/etcdctl ${REPO_PATH}/etcdctl
go install github.com/coreos/etcd
go install github.com/coreos/etcd/bench

View File

@@ -1,53 +0,0 @@
/*
Copyright 2014 CoreOS, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package client
import (
"net/http"
"net/url"
"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
)
func newHTTPClusterClient(tr *http.Transport, eps []string) (*httpClusterClient, error) {
c := httpClusterClient{
endpoints: make([]*httpClient, len(eps)),
}
for i, ep := range eps {
u, err := url.Parse(ep)
if err != nil {
return nil, err
}
c.endpoints[i] = &httpClient{
transport: tr,
endpoint: *u,
}
}
return &c, nil
}
type httpClusterClient struct {
endpoints []*httpClient
}
func (c *httpClusterClient) do(ctx context.Context, act httpAction) (int, []byte, error) {
//TODO(bcwaldon): introduce retry logic so all endpoints are attempted
return c.endpoints[0].do(ctx, act)
}

View File

@@ -1,97 +0,0 @@
/*
Copyright 2014 CoreOS, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package client
import (
"io/ioutil"
"net/http"
"net/url"
"time"
"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
)
var (
ErrTimeout = context.DeadlineExceeded
DefaultRequestTimeout = 5 * time.Second
)
// transport mimics http.Transport to provide an interface which can be
// substituted for testing (since the RoundTripper interface alone does not
// require the CancelRequest method)
type transport interface {
http.RoundTripper
CancelRequest(req *http.Request)
}
type httpAction interface {
httpRequest(url.URL) *http.Request
}
type httpActionDo interface {
do(context.Context, httpAction) (int, []byte, error)
}
type roundTripResponse struct {
resp *http.Response
err error
}
type httpClient struct {
transport transport
endpoint url.URL
timeout time.Duration
}
func (c *httpClient) do(ctx context.Context, act httpAction) (int, []byte, error) {
req := act.httpRequest(c.endpoint)
rtchan := make(chan roundTripResponse, 1)
go func() {
resp, err := c.transport.RoundTrip(req)
rtchan <- roundTripResponse{resp: resp, err: err}
close(rtchan)
}()
var resp *http.Response
var err error
select {
case rtresp := <-rtchan:
resp, err = rtresp.resp, rtresp.err
case <-ctx.Done():
c.transport.CancelRequest(req)
// wait for request to actually exit before continuing
<-rtchan
err = ctx.Err()
}
// always check for resp nil-ness to deal with possible
// race conditions between channels above
defer func() {
if resp != nil {
resp.Body.Close()
}
}()
if err != nil {
return 0, nil, err
}
body, err := ioutil.ReadAll(resp.Body)
return resp.StatusCode, body, err
}

View File

@@ -1,151 +0,0 @@
/*
Copyright 2014 CoreOS, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package client
import (
"errors"
"io/ioutil"
"net/http"
"net/url"
"reflect"
"strings"
"testing"
"time"
"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
)
type fakeTransport struct {
respchan chan *http.Response
errchan chan error
startCancel chan struct{}
finishCancel chan struct{}
}
func newFakeTransport() *fakeTransport {
return &fakeTransport{
respchan: make(chan *http.Response, 1),
errchan: make(chan error, 1),
startCancel: make(chan struct{}, 1),
finishCancel: make(chan struct{}, 1),
}
}
func (t *fakeTransport) RoundTrip(*http.Request) (*http.Response, error) {
select {
case resp := <-t.respchan:
return resp, nil
case err := <-t.errchan:
return nil, err
case <-t.startCancel:
// wait on finishCancel to simulate taking some amount of
// time while calling CancelRequest
<-t.finishCancel
return nil, errors.New("cancelled")
}
}
func (t *fakeTransport) CancelRequest(*http.Request) {
t.startCancel <- struct{}{}
}
type fakeAction struct{}
func (a *fakeAction) httpRequest(url.URL) *http.Request {
return &http.Request{}
}
func TestHTTPClientDoSuccess(t *testing.T) {
tr := newFakeTransport()
c := &httpClient{transport: tr}
tr.respchan <- &http.Response{
StatusCode: http.StatusTeapot,
Body: ioutil.NopCloser(strings.NewReader("foo")),
}
code, body, err := c.do(context.Background(), &fakeAction{})
if err != nil {
t.Fatalf("incorrect error value: want=nil got=%v", err)
}
wantCode := http.StatusTeapot
if wantCode != code {
t.Fatalf("invalid response code: want=%d got=%d", wantCode, code)
}
wantBody := []byte("foo")
if !reflect.DeepEqual(wantBody, body) {
t.Fatalf("invalid response body: want=%q got=%q", wantBody, body)
}
}
func TestHTTPClientDoError(t *testing.T) {
tr := newFakeTransport()
c := &httpClient{transport: tr}
tr.errchan <- errors.New("fixture")
_, _, err := c.do(context.Background(), &fakeAction{})
if err == nil {
t.Fatalf("expected non-nil error, got nil")
}
}
func TestHTTPClientDoCancelContext(t *testing.T) {
tr := newFakeTransport()
c := &httpClient{transport: tr}
tr.startCancel <- struct{}{}
tr.finishCancel <- struct{}{}
_, _, err := c.do(context.Background(), &fakeAction{})
if err == nil {
t.Fatalf("expected non-nil error, got nil")
}
}
func TestHTTPClientDoCancelContextWaitForRoundTrip(t *testing.T) {
tr := newFakeTransport()
c := &httpClient{transport: tr}
donechan := make(chan struct{})
ctx, cancel := context.WithCancel(context.Background())
go func() {
c.do(ctx, &fakeAction{})
close(donechan)
}()
// This should call CancelRequest and begin the cancellation process
cancel()
select {
case <-donechan:
t.Fatalf("httpClient.do should not have exited yet")
default:
}
tr.finishCancel <- struct{}{}
select {
case <-donechan:
//expected behavior
return
case <-time.After(time.Second):
t.Fatalf("httpClient.do did not exit within 1s")
}
}

View File

@@ -1,294 +0,0 @@
/*
Copyright 2014 CoreOS, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package client
import (
"encoding/json"
"errors"
"fmt"
"net/http"
"net/url"
"path"
"strconv"
"strings"
"time"
"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
)
var (
DefaultV2KeysPrefix = "/v2/keys"
)
var (
ErrUnavailable = errors.New("client: no available etcd endpoints")
ErrNoLeader = errors.New("client: no leader")
ErrKeyNoExist = errors.New("client: key does not exist")
ErrKeyExists = errors.New("client: key already exists")
)
func NewKeysAPI(tr *http.Transport, eps []string, to time.Duration) (KeysAPI, error) {
return newHTTPKeysAPIWithPrefix(tr, eps, to, DefaultV2KeysPrefix)
}
func NewDiscoveryKeysAPI(tr *http.Transport, eps []string, to time.Duration) (KeysAPI, error) {
return newHTTPKeysAPIWithPrefix(tr, eps, to, "")
}
func newHTTPKeysAPIWithPrefix(tr *http.Transport, eps []string, to time.Duration, prefix string) (*httpKeysAPI, error) {
c, err := newHTTPClusterClient(tr, eps)
if err != nil {
return nil, err
}
kAPI := httpKeysAPI{
client: c,
prefix: prefix,
timeout: to,
}
return &kAPI, nil
}
type KeysAPI interface {
Create(key, value string, ttl time.Duration) (*Response, error)
Get(key string) (*Response, error)
Watch(key string, idx uint64) Watcher
RecursiveWatch(key string, idx uint64) Watcher
}
type Watcher interface {
Next() (*Response, error)
}
type Response struct {
Action string `json:"action"`
Node *Node `json:"node"`
PrevNode *Node `json:"prevNode"`
}
type Nodes []*Node
type Node struct {
Key string `json:"key"`
Value string `json:"value"`
Nodes Nodes `json:"nodes"`
ModifiedIndex uint64 `json:"modifiedIndex"`
CreatedIndex uint64 `json:"createdIndex"`
}
func (n *Node) String() string {
return fmt.Sprintf("{Key: %s, CreatedIndex: %d, ModifiedIndex: %d}", n.Key, n.CreatedIndex, n.ModifiedIndex)
}
type httpKeysAPI struct {
client httpActionDo
prefix string
timeout time.Duration
}
func (k *httpKeysAPI) Create(key, val string, ttl time.Duration) (*Response, error) {
create := &createAction{
Prefix: k.prefix,
Key: key,
Value: val,
}
if ttl >= 0 {
uttl := uint64(ttl.Seconds())
create.TTL = &uttl
}
ctx, cancel := context.WithTimeout(context.Background(), k.timeout)
code, body, err := k.client.do(ctx, create)
cancel()
if err != nil {
return nil, err
}
return unmarshalHTTPResponse(code, body)
}
func (k *httpKeysAPI) Get(key string) (*Response, error) {
get := &getAction{
Prefix: k.prefix,
Key: key,
Recursive: false,
}
ctx, cancel := context.WithTimeout(context.Background(), k.timeout)
code, body, err := k.client.do(ctx, get)
cancel()
if err != nil {
return nil, err
}
return unmarshalHTTPResponse(code, body)
}
func (k *httpKeysAPI) Watch(key string, idx uint64) Watcher {
return &httpWatcher{
client: k.client,
nextWait: waitAction{
Prefix: k.prefix,
Key: key,
WaitIndex: idx,
Recursive: false,
},
}
}
func (k *httpKeysAPI) RecursiveWatch(key string, idx uint64) Watcher {
return &httpWatcher{
client: k.client,
nextWait: waitAction{
Prefix: k.prefix,
Key: key,
WaitIndex: idx,
Recursive: true,
},
}
}
type httpWatcher struct {
client httpActionDo
nextWait waitAction
}
func (hw *httpWatcher) Next() (*Response, error) {
//TODO(bcwaldon): This needs to be cancellable by the calling user
code, body, err := hw.client.do(context.Background(), &hw.nextWait)
if err != nil {
return nil, err
}
resp, err := unmarshalHTTPResponse(code, body)
if err != nil {
return nil, err
}
hw.nextWait.WaitIndex = resp.Node.ModifiedIndex + 1
return resp, nil
}
// v2KeysURL forms a URL representing the location of a key.
// The endpoint argument represents the base URL of an etcd
// server. The prefix is the path needed to route from the
// provided endpoint's path to the root of the keys API
// (typically "/v2/keys").
func v2KeysURL(ep url.URL, prefix, key string) *url.URL {
ep.Path = path.Join(ep.Path, prefix, key)
return &ep
}
type getAction struct {
Prefix string
Key string
Recursive bool
}
func (g *getAction) httpRequest(ep url.URL) *http.Request {
u := v2KeysURL(ep, g.Prefix, g.Key)
params := u.Query()
params.Set("recursive", strconv.FormatBool(g.Recursive))
u.RawQuery = params.Encode()
req, _ := http.NewRequest("GET", u.String(), nil)
return req
}
type waitAction struct {
Prefix string
Key string
WaitIndex uint64
Recursive bool
}
func (w *waitAction) httpRequest(ep url.URL) *http.Request {
u := v2KeysURL(ep, w.Prefix, w.Key)
params := u.Query()
params.Set("wait", "true")
params.Set("waitIndex", strconv.FormatUint(w.WaitIndex, 10))
params.Set("recursive", strconv.FormatBool(w.Recursive))
u.RawQuery = params.Encode()
req, _ := http.NewRequest("GET", u.String(), nil)
return req
}
type createAction struct {
Prefix string
Key string
Value string
TTL *uint64
}
func (c *createAction) httpRequest(ep url.URL) *http.Request {
u := v2KeysURL(ep, c.Prefix, c.Key)
params := u.Query()
params.Set("prevExist", "false")
u.RawQuery = params.Encode()
form := url.Values{}
form.Add("value", c.Value)
if c.TTL != nil {
form.Add("ttl", strconv.FormatUint(*c.TTL, 10))
}
body := strings.NewReader(form.Encode())
req, _ := http.NewRequest("PUT", u.String(), body)
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
return req
}
func unmarshalHTTPResponse(code int, body []byte) (res *Response, err error) {
switch code {
case http.StatusOK, http.StatusCreated:
res, err = unmarshalSuccessfulResponse(body)
default:
err = unmarshalErrorResponse(code)
}
return
}
func unmarshalSuccessfulResponse(body []byte) (*Response, error) {
var res Response
err := json.Unmarshal(body, &res)
if err != nil {
return nil, err
}
return &res, nil
}
func unmarshalErrorResponse(code int) error {
switch code {
case http.StatusNotFound:
return ErrKeyNoExist
case http.StatusPreconditionFailed:
return ErrKeyExists
case http.StatusInternalServerError:
// this isn't necessarily true
return ErrNoLeader
default:
}
return fmt.Errorf("unrecognized HTTP status code %d", code)
}

View File

@@ -1,368 +0,0 @@
/*
Copyright 2014 CoreOS, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package client
import (
"errors"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"reflect"
"testing"
)
func TestV2KeysURLHelper(t *testing.T) {
tests := []struct {
endpoint url.URL
prefix string
key string
want url.URL
}{
// key is empty, no problem
{
endpoint: url.URL{Scheme: "http", Host: "example.com", Path: "/v2/keys"},
prefix: "",
key: "",
want: url.URL{Scheme: "http", Host: "example.com", Path: "/v2/keys"},
},
// key is joined to path
{
endpoint: url.URL{Scheme: "http", Host: "example.com", Path: "/v2/keys"},
prefix: "",
key: "/foo/bar",
want: url.URL{Scheme: "http", Host: "example.com", Path: "/v2/keys/foo/bar"},
},
// key is joined to path when path is empty
{
endpoint: url.URL{Scheme: "http", Host: "example.com", Path: ""},
prefix: "",
key: "/foo/bar",
want: url.URL{Scheme: "http", Host: "example.com", Path: "/foo/bar"},
},
// Host field carries through with port
{
endpoint: url.URL{Scheme: "http", Host: "example.com:8080", Path: "/v2/keys"},
prefix: "",
key: "",
want: url.URL{Scheme: "http", Host: "example.com:8080", Path: "/v2/keys"},
},
// Scheme carries through
{
endpoint: url.URL{Scheme: "https", Host: "example.com", Path: "/v2/keys"},
prefix: "",
key: "",
want: url.URL{Scheme: "https", Host: "example.com", Path: "/v2/keys"},
},
// Prefix is applied
{
endpoint: url.URL{Scheme: "https", Host: "example.com", Path: "/foo"},
prefix: "/bar",
key: "/baz",
want: url.URL{Scheme: "https", Host: "example.com", Path: "/foo/bar/baz"},
},
}
for i, tt := range tests {
got := v2KeysURL(tt.endpoint, tt.prefix, tt.key)
if tt.want != *got {
t.Errorf("#%d: want=%#v, got=%#v", i, tt.want, *got)
}
}
}
func TestGetAction(t *testing.T) {
ep := url.URL{Scheme: "http", Host: "example.com/v2/keys"}
wantURL := &url.URL{
Scheme: "http",
Host: "example.com",
Path: "/v2/keys/foo/bar",
}
wantHeader := http.Header{}
tests := []struct {
recursive bool
wantQuery string
}{
{
recursive: false,
wantQuery: "recursive=false",
},
{
recursive: true,
wantQuery: "recursive=true",
},
}
for i, tt := range tests {
f := getAction{
Key: "/foo/bar",
Recursive: tt.recursive,
}
got := *f.httpRequest(ep)
wantURL := wantURL
wantURL.RawQuery = tt.wantQuery
err := assertResponse(got, wantURL, wantHeader, nil)
if err != nil {
t.Errorf("#%d: %v", i, err)
}
}
}
func TestWaitAction(t *testing.T) {
ep := url.URL{Scheme: "http", Host: "example.com/v2/keys"}
wantURL := &url.URL{
Scheme: "http",
Host: "example.com",
Path: "/v2/keys/foo/bar",
}
wantHeader := http.Header{}
tests := []struct {
waitIndex uint64
recursive bool
wantQuery string
}{
{
recursive: false,
waitIndex: uint64(0),
wantQuery: "recursive=false&wait=true&waitIndex=0",
},
{
recursive: false,
waitIndex: uint64(12),
wantQuery: "recursive=false&wait=true&waitIndex=12",
},
{
recursive: true,
waitIndex: uint64(12),
wantQuery: "recursive=true&wait=true&waitIndex=12",
},
}
for i, tt := range tests {
f := waitAction{
Key: "/foo/bar",
WaitIndex: tt.waitIndex,
Recursive: tt.recursive,
}
got := *f.httpRequest(ep)
wantURL := wantURL
wantURL.RawQuery = tt.wantQuery
err := assertResponse(got, wantURL, wantHeader, nil)
if err != nil {
t.Errorf("#%d: %v", i, err)
}
}
}
func TestCreateAction(t *testing.T) {
ep := url.URL{Scheme: "http", Host: "example.com/v2/keys"}
wantURL := &url.URL{
Scheme: "http",
Host: "example.com",
Path: "/v2/keys/foo/bar",
RawQuery: "prevExist=false",
}
wantHeader := http.Header(map[string][]string{
"Content-Type": []string{"application/x-www-form-urlencoded"},
})
ttl12 := uint64(12)
tests := []struct {
value string
ttl *uint64
wantBody string
}{
{
value: "baz",
wantBody: "value=baz",
},
{
value: "baz",
ttl: &ttl12,
wantBody: "ttl=12&value=baz",
},
}
for i, tt := range tests {
f := createAction{
Key: "/foo/bar",
Value: tt.value,
TTL: tt.ttl,
}
got := *f.httpRequest(ep)
err := assertResponse(got, wantURL, wantHeader, []byte(tt.wantBody))
if err != nil {
t.Errorf("#%d: %v", i, err)
}
}
}
func assertResponse(got http.Request, wantURL *url.URL, wantHeader http.Header, wantBody []byte) error {
if !reflect.DeepEqual(wantURL, got.URL) {
return fmt.Errorf("want.URL=%#v got.URL=%#v", wantURL, got.URL)
}
if !reflect.DeepEqual(wantHeader, got.Header) {
return fmt.Errorf("want.Header=%#v got.Header=%#v", wantHeader, got.Header)
}
if got.Body == nil {
if wantBody != nil {
return fmt.Errorf("want.Body=%v got.Body=%v", wantBody, got.Body)
}
} else {
if wantBody == nil {
return fmt.Errorf("want.Body=%v got.Body=%s", wantBody, got.Body)
} else {
gotBytes, err := ioutil.ReadAll(got.Body)
if err != nil {
return err
}
if !reflect.DeepEqual(wantBody, gotBytes) {
return fmt.Errorf("want.Body=%s got.Body=%s", wantBody, gotBytes)
}
}
}
return nil
}
func TestUnmarshalSuccessfulResponse(t *testing.T) {
tests := []struct {
body string
res *Response
expectError bool
}{
// Neither PrevNode or Node
{
`{"action":"delete"}`,
&Response{Action: "delete"},
false,
},
// PrevNode
{
`{"action":"delete", "prevNode": {"key": "/foo", "value": "bar", "modifiedIndex": 12, "createdIndex": 10}}`,
&Response{Action: "delete", PrevNode: &Node{Key: "/foo", Value: "bar", ModifiedIndex: 12, CreatedIndex: 10}},
false,
},
// Node
{
`{"action":"get", "node": {"key": "/foo", "value": "bar", "modifiedIndex": 12, "createdIndex": 10}}`,
&Response{Action: "get", Node: &Node{Key: "/foo", Value: "bar", ModifiedIndex: 12, CreatedIndex: 10}},
false,
},
// PrevNode and Node
{
`{"action":"update", "prevNode": {"key": "/foo", "value": "baz", "modifiedIndex": 10, "createdIndex": 10}, "node": {"key": "/foo", "value": "bar", "modifiedIndex": 12, "createdIndex": 10}}`,
&Response{Action: "update", PrevNode: &Node{Key: "/foo", Value: "baz", ModifiedIndex: 10, CreatedIndex: 10}, Node: &Node{Key: "/foo", Value: "bar", ModifiedIndex: 12, CreatedIndex: 10}},
false,
},
// Garbage in body
{
`garbage`,
nil,
true,
},
}
for i, tt := range tests {
res, err := unmarshalSuccessfulResponse([]byte(tt.body))
if tt.expectError != (err != nil) {
t.Errorf("#%d: expectError=%t, err=%v", i, tt.expectError, err)
}
if (res == nil) != (tt.res == nil) {
t.Errorf("#%d: received res==%v, but expected res==%v", i, res, tt.res)
continue
} else if tt.res == nil {
// expected and succesfully got nil response
continue
}
if res.Action != tt.res.Action {
t.Errorf("#%d: Action=%s, expected %s", i, res.Action, tt.res.Action)
}
if !reflect.DeepEqual(res.Node, tt.res.Node) {
t.Errorf("#%d: Node=%v, expected %v", i, res.Node, tt.res.Node)
}
}
}
func TestUnmarshalErrorResponse(t *testing.T) {
unrecognized := errors.New("test fixture")
tests := []struct {
code int
want error
}{
{http.StatusBadRequest, unrecognized},
{http.StatusUnauthorized, unrecognized},
{http.StatusPaymentRequired, unrecognized},
{http.StatusForbidden, unrecognized},
{http.StatusNotFound, ErrKeyNoExist},
{http.StatusMethodNotAllowed, unrecognized},
{http.StatusNotAcceptable, unrecognized},
{http.StatusProxyAuthRequired, unrecognized},
{http.StatusRequestTimeout, unrecognized},
{http.StatusConflict, unrecognized},
{http.StatusGone, unrecognized},
{http.StatusLengthRequired, unrecognized},
{http.StatusPreconditionFailed, ErrKeyExists},
{http.StatusRequestEntityTooLarge, unrecognized},
{http.StatusRequestURITooLong, unrecognized},
{http.StatusUnsupportedMediaType, unrecognized},
{http.StatusRequestedRangeNotSatisfiable, unrecognized},
{http.StatusExpectationFailed, unrecognized},
{http.StatusTeapot, unrecognized},
{http.StatusInternalServerError, ErrNoLeader},
{http.StatusNotImplemented, unrecognized},
{http.StatusBadGateway, unrecognized},
{http.StatusServiceUnavailable, unrecognized},
{http.StatusGatewayTimeout, unrecognized},
{http.StatusHTTPVersionNotSupported, unrecognized},
}
for i, tt := range tests {
want := tt.want
if reflect.DeepEqual(unrecognized, want) {
want = fmt.Errorf("unrecognized HTTP status code %d", tt.code)
}
got := unmarshalErrorResponse(tt.code)
if !reflect.DeepEqual(want, got) {
t.Errorf("#%d: want=%v, got=%v", i, want, got)
}
}
}

View File

@@ -1,165 +0,0 @@
/*
Copyright 2014 CoreOS, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package client
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"net/url"
"path"
"time"
"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
"github.com/coreos/etcd/etcdserver/etcdhttp/httptypes"
"github.com/coreos/etcd/pkg/types"
)
var (
DefaultV2MembersPrefix = "/v2/members"
)
func NewMembersAPI(tr *http.Transport, eps []string, to time.Duration) (MembersAPI, error) {
c, err := newHTTPClusterClient(tr, eps)
if err != nil {
return nil, err
}
mAPI := httpMembersAPI{
client: c,
timeout: to,
}
return &mAPI, nil
}
type MembersAPI interface {
List() ([]httptypes.Member, error)
Add(peerURL string) (*httptypes.Member, error)
Remove(mID string) error
}
type httpMembersAPI struct {
client httpActionDo
timeout time.Duration
}
func (m *httpMembersAPI) List() ([]httptypes.Member, error) {
req := &membersAPIActionList{}
ctx, cancel := context.WithTimeout(context.Background(), m.timeout)
code, body, err := m.client.do(ctx, req)
cancel()
if err != nil {
return nil, err
}
if err := assertStatusCode(http.StatusOK, code); err != nil {
return nil, err
}
var mCollection httptypes.MemberCollection
if err := json.Unmarshal(body, &mCollection); err != nil {
return nil, err
}
return []httptypes.Member(mCollection), nil
}
func (m *httpMembersAPI) Add(peerURL string) (*httptypes.Member, error) {
urls, err := types.NewURLs([]string{peerURL})
if err != nil {
return nil, err
}
req := &membersAPIActionAdd{peerURLs: urls}
ctx, cancel := context.WithTimeout(context.Background(), m.timeout)
code, body, err := m.client.do(ctx, req)
cancel()
if err != nil {
return nil, err
}
if err := assertStatusCode(http.StatusCreated, code); err != nil {
return nil, err
}
var memb httptypes.Member
if err := json.Unmarshal(body, &memb); err != nil {
return nil, err
}
return &memb, nil
}
func (m *httpMembersAPI) Remove(memberID string) error {
req := &membersAPIActionRemove{memberID: memberID}
ctx, cancel := context.WithTimeout(context.Background(), m.timeout)
code, _, err := m.client.do(ctx, req)
cancel()
if err != nil {
return err
}
return assertStatusCode(http.StatusNoContent, code)
}
type membersAPIActionList struct{}
func (l *membersAPIActionList) httpRequest(ep url.URL) *http.Request {
u := v2MembersURL(ep)
req, _ := http.NewRequest("GET", u.String(), nil)
return req
}
type membersAPIActionRemove struct {
memberID string
}
func (d *membersAPIActionRemove) httpRequest(ep url.URL) *http.Request {
u := v2MembersURL(ep)
u.Path = path.Join(u.Path, d.memberID)
req, _ := http.NewRequest("DELETE", u.String(), nil)
return req
}
type membersAPIActionAdd struct {
peerURLs types.URLs
}
func (a *membersAPIActionAdd) httpRequest(ep url.URL) *http.Request {
u := v2MembersURL(ep)
m := httptypes.MemberCreateRequest{PeerURLs: a.peerURLs}
b, _ := json.Marshal(&m)
req, _ := http.NewRequest("POST", u.String(), bytes.NewReader(b))
req.Header.Set("Content-Type", "application/json")
return req
}
func assertStatusCode(want, got int) (err error) {
if want != got {
err = fmt.Errorf("unexpected status code %d", got)
}
return err
}
// v2MembersURL add the necessary path to the provided endpoint
// to route requests to the default v2 members API.
func v2MembersURL(ep url.URL) *url.URL {
ep.Path = path.Join(ep.Path, DefaultV2MembersPrefix)
return &ep
}

View File

@@ -1,113 +0,0 @@
/*
Copyright 2014 CoreOS, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package client
import (
"net/http"
"net/url"
"reflect"
"testing"
"github.com/coreos/etcd/pkg/types"
)
func TestMembersAPIActionList(t *testing.T) {
ep := url.URL{Scheme: "http", Host: "example.com"}
act := &membersAPIActionList{}
wantURL := &url.URL{
Scheme: "http",
Host: "example.com",
Path: "/v2/members",
}
got := *act.httpRequest(ep)
err := assertResponse(got, wantURL, http.Header{}, nil)
if err != nil {
t.Error(err.Error())
}
}
func TestMembersAPIActionAdd(t *testing.T) {
ep := url.URL{Scheme: "http", Host: "example.com"}
act := &membersAPIActionAdd{
peerURLs: types.URLs([]url.URL{
url.URL{Scheme: "https", Host: "127.0.0.1:8081"},
url.URL{Scheme: "http", Host: "127.0.0.1:8080"},
}),
}
wantURL := &url.URL{
Scheme: "http",
Host: "example.com",
Path: "/v2/members",
}
wantHeader := http.Header{
"Content-Type": []string{"application/json"},
}
wantBody := []byte(`{"peerURLs":["https://127.0.0.1:8081","http://127.0.0.1:8080"]}`)
got := *act.httpRequest(ep)
err := assertResponse(got, wantURL, wantHeader, wantBody)
if err != nil {
t.Error(err.Error())
}
}
func TestMembersAPIActionRemove(t *testing.T) {
ep := url.URL{Scheme: "http", Host: "example.com"}
act := &membersAPIActionRemove{memberID: "XXX"}
wantURL := &url.URL{
Scheme: "http",
Host: "example.com",
Path: "/v2/members/XXX",
}
got := *act.httpRequest(ep)
err := assertResponse(got, wantURL, http.Header{}, nil)
if err != nil {
t.Error(err.Error())
}
}
func TestAssertStatusCode(t *testing.T) {
if err := assertStatusCode(400, 404); err == nil {
t.Errorf("assertStatusCode failed to detect conflict in 400 vs 404")
}
if err := assertStatusCode(400, 400); err != nil {
t.Errorf("assertStatusCode found conflict in 400 vs 400: %v", err)
}
}
func TestV2MembersURL(t *testing.T) {
got := v2MembersURL(url.URL{
Scheme: "http",
Host: "foo.example.com:4002",
Path: "/pants",
})
want := &url.URL{
Scheme: "http",
Host: "foo.example.com:4002",
Path: "/pants/v2/members",
}
if !reflect.DeepEqual(want, got) {
t.Fatalf("v2MembersURL got %#v, want %#v", got, want)
}
}

500
config/config.go Normal file
View File

@@ -0,0 +1,500 @@
package config
import (
"flag"
"fmt"
"io/ioutil"
"math/rand"
"net"
"net/url"
"os"
"path/filepath"
"reflect"
"strconv"
"strings"
"time"
"github.com/coreos/etcd/third_party/github.com/BurntSushi/toml"
"github.com/coreos/etcd/log"
ustrings "github.com/coreos/etcd/pkg/strings"
"github.com/coreos/etcd/server"
)
// The default location for the etcd configuration file.
const DefaultSystemConfigPath = "/etc/etcd/etcd.conf"
// A lookup of deprecated flags to their new flag name.
var newFlagNameLookup = map[string]string{
"C": "peers",
"CF": "peers-file",
"n": "name",
"c": "addr",
"cl": "bind-addr",
"s": "peer-addr",
"sl": "peer-bind-addr",
"d": "data-dir",
"m": "max-result-buffer",
"r": "max-retry-attempts",
"maxsize": "cluster-active-size",
"clientCAFile": "ca-file",
"clientCert": "cert-file",
"clientKey": "key-file",
"serverCAFile": "peer-ca-file",
"serverCert": "peer-cert-file",
"serverKey": "peer-key-file",
"snapshotCount": "snapshot-count",
"peer-heartbeat-timeout": "peer-heartbeat-interval",
"max-cluster-size": "cluster-active-size",
}
// Config represents the server configuration.
type Config struct {
SystemPath string
Addr string `toml:"addr" env:"ETCD_ADDR"`
BindAddr string `toml:"bind_addr" env:"ETCD_BIND_ADDR"`
CAFile string `toml:"ca_file" env:"ETCD_CA_FILE"`
CertFile string `toml:"cert_file" env:"ETCD_CERT_FILE"`
CPUProfileFile string
CorsOrigins []string `toml:"cors" env:"ETCD_CORS"`
DataDir string `toml:"data_dir" env:"ETCD_DATA_DIR"`
Discovery string `toml:"discovery" env:"ETCD_DISCOVERY"`
Force bool
KeyFile string `toml:"key_file" env:"ETCD_KEY_FILE"`
HTTPReadTimeout float64 `toml:"http_read_timeout" env:"ETCD_HTTP_READ_TIMEOUT"`
HTTPWriteTimeout float64 `toml:"http_write_timeout" env:"ETCD_HTTP_WRITE_TIMEOUT"`
Peers []string `toml:"peers" env:"ETCD_PEERS"`
PeersFile string `toml:"peers_file" env:"ETCD_PEERS_FILE"`
MaxResultBuffer int `toml:"max_result_buffer" env:"ETCD_MAX_RESULT_BUFFER"`
MaxRetryAttempts int `toml:"max_retry_attempts" env:"ETCD_MAX_RETRY_ATTEMPTS"`
RetryInterval float64 `toml:"retry_interval" env:"ETCD_RETRY_INTERVAL"`
Name string `toml:"name" env:"ETCD_NAME"`
Snapshot bool `toml:"snapshot" env:"ETCD_SNAPSHOT"`
SnapshotCount int `toml:"snapshot_count" env:"ETCD_SNAPSHOTCOUNT"`
ShowHelp bool
ShowVersion bool
Verbose bool `toml:"verbose" env:"ETCD_VERBOSE"`
VeryVerbose bool `toml:"very_verbose" env:"ETCD_VERY_VERBOSE"`
VeryVeryVerbose bool `toml:"very_very_verbose" env:"ETCD_VERY_VERY_VERBOSE"`
Peer struct {
Addr string `toml:"addr" env:"ETCD_PEER_ADDR"`
BindAddr string `toml:"bind_addr" env:"ETCD_PEER_BIND_ADDR"`
CAFile string `toml:"ca_file" env:"ETCD_PEER_CA_FILE"`
CertFile string `toml:"cert_file" env:"ETCD_PEER_CERT_FILE"`
KeyFile string `toml:"key_file" env:"ETCD_PEER_KEY_FILE"`
HeartbeatInterval int `toml:"heartbeat_interval" env:"ETCD_PEER_HEARTBEAT_INTERVAL"`
ElectionTimeout int `toml:"election_timeout" env:"ETCD_PEER_ELECTION_TIMEOUT"`
}
strTrace string `toml:"trace" env:"ETCD_TRACE"`
GraphiteHost string `toml:"graphite_host" env:"ETCD_GRAPHITE_HOST"`
Cluster struct {
ActiveSize int `toml:"active_size" env:"ETCD_CLUSTER_ACTIVE_SIZE"`
RemoveDelay float64 `toml:"remove_delay" env:"ETCD_CLUSTER_REMOVE_DELAY"`
SyncInterval float64 `toml:"sync_interval" env:"ETCD_CLUSTER_SYNC_INTERVAL"`
}
}
// New returns a Config initialized with default values.
func New() *Config {
c := new(Config)
c.SystemPath = DefaultSystemConfigPath
c.Addr = "127.0.0.1:4001"
c.HTTPReadTimeout = server.DefaultReadTimeout
c.HTTPWriteTimeout = server.DefaultWriteTimeout
c.MaxResultBuffer = 1024
c.MaxRetryAttempts = 3
c.RetryInterval = 10.0
c.Snapshot = true
c.SnapshotCount = 10000
c.Peer.Addr = "127.0.0.1:7001"
c.Peer.HeartbeatInterval = defaultHeartbeatInterval
c.Peer.ElectionTimeout = defaultElectionTimeout
rand.Seed(time.Now().UTC().UnixNano())
// Make maximum twice as minimum.
c.RetryInterval = float64(50+rand.Int()%50) * defaultHeartbeatInterval / 1000
c.Cluster.ActiveSize = server.DefaultActiveSize
c.Cluster.RemoveDelay = server.DefaultRemoveDelay
c.Cluster.SyncInterval = server.DefaultSyncInterval
return c
}
// Loads the configuration from the system config, command line config,
// environment variables, and finally command line arguments.
func (c *Config) Load(arguments []string) error {
var path string
f := flag.NewFlagSet("etcd", -1)
f.SetOutput(ioutil.Discard)
f.StringVar(&path, "config", "", "path to config file")
f.Parse(arguments)
// Load from system file.
if err := c.LoadSystemFile(); err != nil {
return err
}
// Load from config file specified in arguments.
if path != "" {
if err := c.LoadFile(path); err != nil {
return err
}
}
// Load from the environment variables next.
if err := c.LoadEnv(); err != nil {
return err
}
// Load from command line flags.
if err := c.LoadFlags(arguments); err != nil {
return err
}
// Loads peers if a peer file was specified.
if err := c.LoadPeersFile(); err != nil {
return err
}
return nil
}
// Loads from the system etcd configuration file if it exists.
func (c *Config) LoadSystemFile() error {
if _, err := os.Stat(c.SystemPath); os.IsNotExist(err) {
return nil
}
return c.LoadFile(c.SystemPath)
}
// Loads configuration from a file.
func (c *Config) LoadFile(path string) error {
_, err := toml.DecodeFile(path, &c)
return err
}
// LoadEnv loads the configuration via environment variables.
func (c *Config) LoadEnv() error {
if err := c.loadEnv(c); err != nil {
return err
}
if err := c.loadEnv(&c.Peer); err != nil {
return err
}
if err := c.loadEnv(&c.Cluster); err != nil {
return err
}
return nil
}
func (c *Config) loadEnv(target interface{}) error {
value := reflect.Indirect(reflect.ValueOf(target))
typ := value.Type()
for i := 0; i < typ.NumField(); i++ {
field := typ.Field(i)
// Retrieve environment variable.
v := strings.TrimSpace(os.Getenv(field.Tag.Get("env")))
if v == "" {
continue
}
// Set the appropriate type.
switch field.Type.Kind() {
case reflect.Bool:
value.Field(i).SetBool(v != "0" && v != "false")
case reflect.Int:
newValue, err := strconv.ParseInt(v, 10, 0)
if err != nil {
return fmt.Errorf("Parse error: %s: %s", field.Tag.Get("env"), err)
}
value.Field(i).SetInt(newValue)
case reflect.String:
value.Field(i).SetString(v)
case reflect.Slice:
value.Field(i).Set(reflect.ValueOf(ustrings.TrimSplit(v, ",")))
case reflect.Float64:
newValue, err := strconv.ParseFloat(v, 64)
if err != nil {
return fmt.Errorf("Parse error: %s: %s", field.Tag.Get("env"), err)
}
value.Field(i).SetFloat(newValue)
}
}
return nil
}
// Loads configuration from command line flags.
func (c *Config) LoadFlags(arguments []string) error {
var peers, cors, path string
f := flag.NewFlagSet(os.Args[0], flag.ContinueOnError)
f.SetOutput(ioutil.Discard)
f.BoolVar(&c.ShowHelp, "h", false, "")
f.BoolVar(&c.ShowHelp, "help", false, "")
f.BoolVar(&c.ShowVersion, "version", false, "")
f.BoolVar(&c.Force, "f", false, "")
f.BoolVar(&c.Force, "force", false, "")
f.BoolVar(&c.Verbose, "v", c.Verbose, "")
f.BoolVar(&c.VeryVerbose, "vv", c.VeryVerbose, "")
f.BoolVar(&c.VeryVeryVerbose, "vvv", c.VeryVeryVerbose, "")
f.StringVar(&peers, "peers", "", "")
f.StringVar(&c.PeersFile, "peers-file", c.PeersFile, "")
f.StringVar(&c.Name, "name", c.Name, "")
f.StringVar(&c.Addr, "addr", c.Addr, "")
f.StringVar(&c.Discovery, "discovery", c.Discovery, "")
f.StringVar(&c.BindAddr, "bind-addr", c.BindAddr, "")
f.StringVar(&c.Peer.Addr, "peer-addr", c.Peer.Addr, "")
f.StringVar(&c.Peer.BindAddr, "peer-bind-addr", c.Peer.BindAddr, "")
f.StringVar(&c.CAFile, "ca-file", c.CAFile, "")
f.StringVar(&c.CertFile, "cert-file", c.CertFile, "")
f.StringVar(&c.KeyFile, "key-file", c.KeyFile, "")
f.StringVar(&c.Peer.CAFile, "peer-ca-file", c.Peer.CAFile, "")
f.StringVar(&c.Peer.CertFile, "peer-cert-file", c.Peer.CertFile, "")
f.StringVar(&c.Peer.KeyFile, "peer-key-file", c.Peer.KeyFile, "")
f.Float64Var(&c.HTTPReadTimeout, "http-read-timeout", c.HTTPReadTimeout, "")
f.Float64Var(&c.HTTPWriteTimeout, "http-write-timeout", c.HTTPReadTimeout, "")
f.StringVar(&c.DataDir, "data-dir", c.DataDir, "")
f.IntVar(&c.MaxResultBuffer, "max-result-buffer", c.MaxResultBuffer, "")
f.IntVar(&c.MaxRetryAttempts, "max-retry-attempts", c.MaxRetryAttempts, "")
f.Float64Var(&c.RetryInterval, "retry-interval", c.RetryInterval, "")
f.IntVar(&c.Peer.HeartbeatInterval, "peer-heartbeat-interval", c.Peer.HeartbeatInterval, "")
f.IntVar(&c.Peer.ElectionTimeout, "peer-election-timeout", c.Peer.ElectionTimeout, "")
f.StringVar(&cors, "cors", "", "")
f.BoolVar(&c.Snapshot, "snapshot", c.Snapshot, "")
f.IntVar(&c.SnapshotCount, "snapshot-count", c.SnapshotCount, "")
f.StringVar(&c.CPUProfileFile, "cpuprofile", "", "")
f.StringVar(&c.strTrace, "trace", "", "")
f.StringVar(&c.GraphiteHost, "graphite-host", "", "")
f.IntVar(&c.Cluster.ActiveSize, "cluster-active-size", c.Cluster.ActiveSize, "")
f.Float64Var(&c.Cluster.RemoveDelay, "cluster-remove-delay", c.Cluster.RemoveDelay, "")
f.Float64Var(&c.Cluster.SyncInterval, "cluster-sync-interval", c.Cluster.SyncInterval, "")
// BEGIN IGNORED FLAGS
f.StringVar(&path, "config", "", "")
// BEGIN IGNORED FLAGS
// BEGIN DEPRECATED FLAGS
f.StringVar(&peers, "C", "", "(deprecated)")
f.StringVar(&c.PeersFile, "CF", c.PeersFile, "(deprecated)")
f.StringVar(&c.Name, "n", c.Name, "(deprecated)")
f.StringVar(&c.Addr, "c", c.Addr, "(deprecated)")
f.StringVar(&c.BindAddr, "cl", c.BindAddr, "(deprecated)")
f.StringVar(&c.Peer.Addr, "s", c.Peer.Addr, "(deprecated)")
f.StringVar(&c.Peer.BindAddr, "sl", c.Peer.BindAddr, "(deprecated)")
f.StringVar(&c.Peer.CAFile, "serverCAFile", c.Peer.CAFile, "(deprecated)")
f.StringVar(&c.Peer.CertFile, "serverCert", c.Peer.CertFile, "(deprecated)")
f.StringVar(&c.Peer.KeyFile, "serverKey", c.Peer.KeyFile, "(deprecated)")
f.StringVar(&c.CAFile, "clientCAFile", c.CAFile, "(deprecated)")
f.StringVar(&c.CertFile, "clientCert", c.CertFile, "(deprecated)")
f.StringVar(&c.KeyFile, "clientKey", c.KeyFile, "(deprecated)")
f.StringVar(&c.DataDir, "d", c.DataDir, "(deprecated)")
f.IntVar(&c.MaxResultBuffer, "m", c.MaxResultBuffer, "(deprecated)")
f.IntVar(&c.MaxRetryAttempts, "r", c.MaxRetryAttempts, "(deprecated)")
f.IntVar(&c.SnapshotCount, "snapshotCount", c.SnapshotCount, "(deprecated)")
f.IntVar(&c.Peer.HeartbeatInterval, "peer-heartbeat-timeout", c.Peer.HeartbeatInterval, "(deprecated)")
f.IntVar(&c.Cluster.ActiveSize, "max-cluster-size", c.Cluster.ActiveSize, "(deprecated)")
f.IntVar(&c.Cluster.ActiveSize, "maxsize", c.Cluster.ActiveSize, "(deprecated)")
// END DEPRECATED FLAGS
if err := f.Parse(arguments); err != nil {
return err
}
// Print deprecation warnings on STDERR.
f.Visit(func(f *flag.Flag) {
if len(newFlagNameLookup[f.Name]) > 0 {
fmt.Fprintf(os.Stderr, "[deprecated] use -%s, not -%s\n", newFlagNameLookup[f.Name], f.Name)
}
})
// Convert some parameters to lists.
if peers != "" {
c.Peers = ustrings.TrimSplit(peers, ",")
}
if cors != "" {
c.CorsOrigins = ustrings.TrimSplit(cors, ",")
}
return nil
}
// LoadPeersFile loads the peers listed in the peers file.
func (c *Config) LoadPeersFile() error {
if c.PeersFile == "" {
return nil
}
b, err := ioutil.ReadFile(c.PeersFile)
if err != nil {
return fmt.Errorf("Peers file error: %s", err)
}
c.Peers = ustrings.TrimSplit(string(b), ",")
return nil
}
// DataDirFromName sets the data dir from a machine name and issue a warning
// that etcd is guessing.
func (c *Config) DataDirFromName() {
c.DataDir = c.Name + ".etcd"
log.Warnf("Using the directory %s as the etcd curation directory because a directory was not specified. ", c.DataDir)
return
}
// NameFromHostname sets the machine name from the hostname. This is to help
// people get started without thinking up a name.
func (c *Config) NameFromHostname() {
host, err := os.Hostname()
if err != nil && host == "" {
log.Fatal("Node name required and hostname not set. e.g. '-name=name'")
}
c.Name = host
}
// Reset removes all server configuration files.
func (c *Config) Reset() error {
if err := os.RemoveAll(filepath.Join(c.DataDir, "log")); err != nil {
return err
}
if err := os.RemoveAll(filepath.Join(c.DataDir, "conf")); err != nil {
return err
}
if err := os.RemoveAll(filepath.Join(c.DataDir, "snapshot")); err != nil {
return err
}
if err := os.RemoveAll(filepath.Join(c.DataDir, "standby_info")); err != nil {
return err
}
return nil
}
// Sanitize cleans the input fields.
func (c *Config) Sanitize() error {
var err error
var url *url.URL
// Sanitize the URLs first.
if c.Addr, url, err = sanitizeURL(c.Addr, c.EtcdTLSInfo().Scheme()); err != nil {
return fmt.Errorf("Advertised URL: %s", err)
}
if c.BindAddr, err = sanitizeBindAddr(c.BindAddr, url); err != nil {
return fmt.Errorf("Listen Host: %s", err)
}
if c.Peer.Addr, url, err = sanitizeURL(c.Peer.Addr, c.PeerTLSInfo().Scheme()); err != nil {
return fmt.Errorf("Peer Advertised URL: %s", err)
}
if c.Peer.BindAddr, err = sanitizeBindAddr(c.Peer.BindAddr, url); err != nil {
return fmt.Errorf("Peer Listen Host: %s", err)
}
// Only guess the machine name if there is no data dir specified
// because the info file should have our name
if c.Name == "" && c.DataDir == "" {
c.NameFromHostname()
}
if c.DataDir == "" && c.Name != "" && !c.ShowVersion && !c.ShowHelp {
c.DataDirFromName()
}
return nil
}
// EtcdTLSInfo retrieves a TLSInfo object for the etcd server
func (c *Config) EtcdTLSInfo() *server.TLSInfo {
return &server.TLSInfo{
CAFile: c.CAFile,
CertFile: c.CertFile,
KeyFile: c.KeyFile,
}
}
// PeerRaftInfo retrieves a TLSInfo object for the peer server.
func (c *Config) PeerTLSInfo() *server.TLSInfo {
return &server.TLSInfo{
CAFile: c.Peer.CAFile,
CertFile: c.Peer.CertFile,
KeyFile: c.Peer.KeyFile,
}
}
// MetricsBucketName generates the name that should be used for a
// corresponding MetricsBucket object
func (c *Config) MetricsBucketName() string {
return fmt.Sprintf("etcd.%s", c.Name)
}
// Trace determines if any trace-level information should be emitted
func (c *Config) Trace() bool {
return c.strTrace == "*"
}
func (c *Config) ClusterConfig() *server.ClusterConfig {
return &server.ClusterConfig{
ActiveSize: c.Cluster.ActiveSize,
RemoveDelay: c.Cluster.RemoveDelay,
SyncInterval: c.Cluster.SyncInterval,
}
}
// sanitizeURL will cleanup a host string in the format hostname[:port] and
// attach a schema.
func sanitizeURL(host string, defaultScheme string) (string, *url.URL, error) {
// Blank URLs are fine input, just return it
if len(host) == 0 {
return host, &url.URL{}, nil
}
p, err := url.Parse(host)
if err != nil {
return "", nil, err
}
// Make sure the host is in Host:Port format
_, _, err = net.SplitHostPort(host)
if err != nil {
return "", nil, err
}
p = &url.URL{Host: host, Scheme: defaultScheme}
return p.String(), p, nil
}
// sanitizeBindAddr cleans up the BindAddr parameter and appends a port
// if necessary based on the advertised port.
func sanitizeBindAddr(bindAddr string, aurl *url.URL) (string, error) {
// If it is a valid host:port simply return with no further checks.
bhost, bport, err := net.SplitHostPort(bindAddr)
if err == nil && bhost != "" {
return bindAddr, nil
}
// SplitHostPort makes the host optional, but we don't want that.
if bhost == "" && bport != "" {
return "", fmt.Errorf("IP required can't use a port only")
}
// bindAddr doesn't have a port if we reach here so take the port from the
// advertised URL.
_, aport, err := net.SplitHostPort(aurl.Host)
if err != nil {
return "", err
}
return net.JoinHostPort(bindAddr, aport), nil
}

798
config/config_test.go Normal file
View File

@@ -0,0 +1,798 @@
package config
import (
"io/ioutil"
"os"
"testing"
"github.com/coreos/etcd/third_party/github.com/BurntSushi/toml"
"github.com/coreos/etcd/third_party/github.com/stretchr/testify/assert"
)
// Ensures that a configuration can be deserialized from TOML.
func TestConfigTOML(t *testing.T) {
content := `
addr = "127.0.0.1:4002"
ca_file = "/tmp/file.ca"
cert_file = "/tmp/file.cert"
cors = ["*"]
cpu_profile_file = "XXX"
data_dir = "/tmp/data"
discovery = "http://example.com/foobar"
key_file = "/tmp/file.key"
bind_addr = "127.0.0.1:4003"
peers = ["coreos.com:4001", "coreos.com:4002"]
peers_file = "/tmp/peers"
max_cluster_size = 10
max_result_buffer = 512
max_retry_attempts = 5
name = "test-name"
http_read_timeout = 2.34
snapshot = true
verbose = true
very_verbose = true
http_write_timeout = 1.23
[peer]
addr = "127.0.0.1:7002"
ca_file = "/tmp/peer/file.ca"
cert_file = "/tmp/peer/file.cert"
key_file = "/tmp/peer/file.key"
bind_addr = "127.0.0.1:7003"
[cluster]
active_size = 5
remove_delay = 100.0
sync_interval = 10.0
`
c := New()
_, err := toml.Decode(content, &c)
assert.Nil(t, err, "")
assert.Equal(t, c.Addr, "127.0.0.1:4002", "")
assert.Equal(t, c.CAFile, "/tmp/file.ca", "")
assert.Equal(t, c.CertFile, "/tmp/file.cert", "")
assert.Equal(t, c.CorsOrigins, []string{"*"}, "")
assert.Equal(t, c.DataDir, "/tmp/data", "")
assert.Equal(t, c.Discovery, "http://example.com/foobar", "")
assert.Equal(t, c.HTTPReadTimeout, 2.34, "")
assert.Equal(t, c.HTTPWriteTimeout, 1.23, "")
assert.Equal(t, c.KeyFile, "/tmp/file.key", "")
assert.Equal(t, c.BindAddr, "127.0.0.1:4003", "")
assert.Equal(t, c.Peers, []string{"coreos.com:4001", "coreos.com:4002"}, "")
assert.Equal(t, c.PeersFile, "/tmp/peers", "")
assert.Equal(t, c.MaxResultBuffer, 512, "")
assert.Equal(t, c.MaxRetryAttempts, 5, "")
assert.Equal(t, c.Name, "test-name", "")
assert.Equal(t, c.Snapshot, true, "")
assert.Equal(t, c.Verbose, true, "")
assert.Equal(t, c.VeryVerbose, true, "")
assert.Equal(t, c.Peer.Addr, "127.0.0.1:7002", "")
assert.Equal(t, c.Peer.CAFile, "/tmp/peer/file.ca", "")
assert.Equal(t, c.Peer.CertFile, "/tmp/peer/file.cert", "")
assert.Equal(t, c.Peer.KeyFile, "/tmp/peer/file.key", "")
assert.Equal(t, c.Peer.BindAddr, "127.0.0.1:7003", "")
assert.Equal(t, c.Cluster.ActiveSize, 5, "")
assert.Equal(t, c.Cluster.RemoveDelay, 100.0, "")
assert.Equal(t, c.Cluster.SyncInterval, 10.0, "")
}
// Ensures that a configuration can be retrieved from environment variables.
func TestConfigEnv(t *testing.T) {
os.Setenv("ETCD_CA_FILE", "/tmp/file.ca")
os.Setenv("ETCD_CERT_FILE", "/tmp/file.cert")
os.Setenv("ETCD_CPU_PROFILE_FILE", "XXX")
os.Setenv("ETCD_CORS", "localhost:4001,localhost:4002")
os.Setenv("ETCD_DATA_DIR", "/tmp/data")
os.Setenv("ETCD_DISCOVERY", "http://example.com/foobar")
os.Setenv("ETCD_HTTP_READ_TIMEOUT", "2.34")
os.Setenv("ETCD_HTTP_WRITE_TIMEOUT", "1.23")
os.Setenv("ETCD_KEY_FILE", "/tmp/file.key")
os.Setenv("ETCD_BIND_ADDR", "127.0.0.1:4003")
os.Setenv("ETCD_PEERS", "coreos.com:4001,coreos.com:4002")
os.Setenv("ETCD_PEERS_FILE", "/tmp/peers")
os.Setenv("ETCD_MAX_CLUSTER_SIZE", "10")
os.Setenv("ETCD_MAX_RESULT_BUFFER", "512")
os.Setenv("ETCD_MAX_RETRY_ATTEMPTS", "5")
os.Setenv("ETCD_NAME", "test-name")
os.Setenv("ETCD_SNAPSHOT", "true")
os.Setenv("ETCD_VERBOSE", "1")
os.Setenv("ETCD_VERY_VERBOSE", "yes")
os.Setenv("ETCD_PEER_ADDR", "127.0.0.1:7002")
os.Setenv("ETCD_PEER_CA_FILE", "/tmp/peer/file.ca")
os.Setenv("ETCD_PEER_CERT_FILE", "/tmp/peer/file.cert")
os.Setenv("ETCD_PEER_KEY_FILE", "/tmp/peer/file.key")
os.Setenv("ETCD_PEER_BIND_ADDR", "127.0.0.1:7003")
os.Setenv("ETCD_CLUSTER_ACTIVE_SIZE", "5")
os.Setenv("ETCD_CLUSTER_REMOVE_DELAY", "100")
os.Setenv("ETCD_CLUSTER_SYNC_INTERVAL", "10")
c := New()
c.LoadEnv()
assert.Equal(t, c.CAFile, "/tmp/file.ca", "")
assert.Equal(t, c.CertFile, "/tmp/file.cert", "")
assert.Equal(t, c.CorsOrigins, []string{"localhost:4001", "localhost:4002"}, "")
assert.Equal(t, c.DataDir, "/tmp/data", "")
assert.Equal(t, c.Discovery, "http://example.com/foobar", "")
assert.Equal(t, c.HTTPReadTimeout, 2.34, "")
assert.Equal(t, c.HTTPWriteTimeout, 1.23, "")
assert.Equal(t, c.KeyFile, "/tmp/file.key", "")
assert.Equal(t, c.BindAddr, "127.0.0.1:4003", "")
assert.Equal(t, c.Peers, []string{"coreos.com:4001", "coreos.com:4002"}, "")
assert.Equal(t, c.PeersFile, "/tmp/peers", "")
assert.Equal(t, c.MaxResultBuffer, 512, "")
assert.Equal(t, c.MaxRetryAttempts, 5, "")
assert.Equal(t, c.Name, "test-name", "")
assert.Equal(t, c.Snapshot, true, "")
assert.Equal(t, c.Verbose, true, "")
assert.Equal(t, c.VeryVerbose, true, "")
assert.Equal(t, c.Peer.Addr, "127.0.0.1:7002", "")
assert.Equal(t, c.Peer.CAFile, "/tmp/peer/file.ca", "")
assert.Equal(t, c.Peer.CertFile, "/tmp/peer/file.cert", "")
assert.Equal(t, c.Peer.KeyFile, "/tmp/peer/file.key", "")
assert.Equal(t, c.Peer.BindAddr, "127.0.0.1:7003", "")
assert.Equal(t, c.Cluster.ActiveSize, 5, "")
assert.Equal(t, c.Cluster.RemoveDelay, 100.0, "")
assert.Equal(t, c.Cluster.SyncInterval, 10.0, "")
// Clear this as it will mess up other tests
os.Setenv("ETCD_DISCOVERY", "")
}
// Ensures that the "help" flag can be parsed.
func TestConfigHelpFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-help"}), "")
assert.True(t, c.ShowHelp)
}
// Ensures that the abbreviated "help" flag can be parsed.
func TestConfigAbbreviatedHelpFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-h"}), "")
assert.True(t, c.ShowHelp)
}
// Ensures that the "version" flag can be parsed.
func TestConfigVersionFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-version"}), "")
assert.True(t, c.ShowVersion)
}
// Ensures that the "force config" flag can be parsed.
func TestConfigForceFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-force"}), "")
assert.True(t, c.Force)
}
// Ensures that the abbreviated "force config" flag can be parsed.
func TestConfigAbbreviatedForceFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-f"}), "")
assert.True(t, c.Force)
}
// Ensures that the advertised url can be parsed from the environment.
func TestConfigAddrEnv(t *testing.T) {
withEnv("ETCD_ADDR", "127.0.0.1:4002", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.Addr, "127.0.0.1:4002", "")
})
}
// Ensures that the advertised flag can be parsed.
func TestConfigAddrFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-addr", "127.0.0.1:4002"}), "")
assert.Equal(t, c.Addr, "127.0.0.1:4002", "")
}
// Ensures that the CA file can be parsed from the environment.
func TestConfigCAFileEnv(t *testing.T) {
withEnv("ETCD_CA_FILE", "/tmp/file.ca", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.CAFile, "/tmp/file.ca", "")
})
}
// Ensures that the CA file flag can be parsed.
func TestConfigCAFileFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-ca-file", "/tmp/file.ca"}), "")
assert.Equal(t, c.CAFile, "/tmp/file.ca", "")
}
// Ensures that the CA file can be parsed from the environment.
func TestConfigCertFileEnv(t *testing.T) {
withEnv("ETCD_CERT_FILE", "/tmp/file.cert", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.CertFile, "/tmp/file.cert", "")
})
}
// Ensures that the Cert file flag can be parsed.
func TestConfigCertFileFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-cert-file", "/tmp/file.cert"}), "")
assert.Equal(t, c.CertFile, "/tmp/file.cert", "")
}
// Ensures that the Key file can be parsed from the environment.
func TestConfigKeyFileEnv(t *testing.T) {
withEnv("ETCD_KEY_FILE", "/tmp/file.key", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.KeyFile, "/tmp/file.key", "")
})
}
// Ensures that the Key file flag can be parsed.
func TestConfigKeyFileFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-key-file", "/tmp/file.key"}), "")
assert.Equal(t, c.KeyFile, "/tmp/file.key", "")
}
// Ensures that the Listen Host can be parsed from the environment.
func TestConfigBindAddrEnv(t *testing.T) {
withEnv("ETCD_BIND_ADDR", "127.0.0.1:4003", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.BindAddr, "127.0.0.1:4003", "")
})
}
// Ensures that the Listen Host file flag can be parsed.
func TestConfigBindAddrFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-bind-addr", "127.0.0.1:4003"}), "")
assert.Equal(t, c.BindAddr, "127.0.0.1:4003", "")
}
// Ensures that the Listen Host port overrides the advertised port
func TestConfigBindAddrOverride(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-addr", "127.0.0.1:4009", "-bind-addr", "127.0.0.1:4010"}), "")
assert.Nil(t, c.Sanitize())
assert.Equal(t, c.BindAddr, "127.0.0.1:4010", "")
}
// Ensures that the Listen Host port overrides the advertised port
func TestConfigBindIPv6AddrOverride(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-addr", "[::1]:4009", "-bind-addr", "[::1]:4010"}), "")
assert.Nil(t, c.Sanitize())
assert.Equal(t, c.BindAddr, "[::1]:4010", "")
}
// Ensures that the Listen Host port overrides the advertised port
func TestConfigBindIPv6WithZoneAddrOverride(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-addr", "[::1%25lo]:4009", "-bind-addr", "[::1%25lo]:4010"}), "")
assert.Nil(t, c.Sanitize())
assert.Equal(t, c.BindAddr, "[::1%25lo]:4010", "")
}
// Ensures that the Listen Host inherits its port from the advertised addr
func TestConfigBindAddrInheritPort(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-addr", "127.0.0.1:4009", "-bind-addr", "127.0.0.1"}), "")
assert.Nil(t, c.Sanitize())
assert.Equal(t, c.BindAddr, "127.0.0.1:4009", "")
}
// Ensures that the Listen Host inherits its port from the advertised addr
func TestConfigBindIPv6AddrInheritPort(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-addr", "[::1]:4009", "-bind-addr", "::1"}), "")
assert.Nil(t, c.Sanitize())
assert.Equal(t, c.BindAddr, "[::1]:4009", "")
}
// Ensures that the Listen Host inherits its port from the advertised addr
func TestConfigBindIPv6WithZoneAddrInheritPort(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-addr", "[::1%25lo]:4009", "-bind-addr", "::1%25lo"}), "")
assert.Nil(t, c.Sanitize())
assert.Equal(t, c.BindAddr, "[::1%25lo]:4009", "")
}
// Ensures that a port only argument errors out
func TestConfigBindAddrErrorOnNoHost(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-addr", "127.0.0.1:4009", "-bind-addr", ":4010"}), "")
assert.Error(t, c.Sanitize())
}
// Ensures that a bad IPv6 address will raise an error
func TestConfigBindAddrErrorOnBadIPv6Addr(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-addr", "[::1%lo]:4009"}), "")
assert.Error(t, c.Sanitize())
}
// Ensures that the peers can be parsed from the environment.
func TestConfigPeersEnv(t *testing.T) {
withEnv("ETCD_PEERS", "coreos.com:4001,coreos.com:4002", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.Peers, []string{"coreos.com:4001", "coreos.com:4002"}, "")
})
}
// Ensures that the Peers flag can be parsed.
func TestConfigPeersFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-peers", "coreos.com:4001,coreos.com:4002"}), "")
assert.Equal(t, c.Peers, []string{"coreos.com:4001", "coreos.com:4002"}, "")
}
// Ensures that the Peers File can be parsed from the environment.
func TestConfigPeersFileEnv(t *testing.T) {
withEnv("ETCD_PEERS_FILE", "/tmp/peers", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.PeersFile, "/tmp/peers", "")
})
}
// Ensures that the Peers File flag can be parsed.
func TestConfigPeersFileFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-peers-file", "/tmp/peers"}), "")
assert.Equal(t, c.PeersFile, "/tmp/peers", "")
}
// Ensures that the Max Result Buffer can be parsed from the environment.
func TestConfigMaxResultBufferEnv(t *testing.T) {
withEnv("ETCD_MAX_RESULT_BUFFER", "512", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.MaxResultBuffer, 512, "")
})
}
// Ensures that the Max Result Buffer flag can be parsed.
func TestConfigMaxResultBufferFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-max-result-buffer", "512"}), "")
assert.Equal(t, c.MaxResultBuffer, 512, "")
}
// Ensures that the Max Retry Attempts can be parsed from the environment.
func TestConfigMaxRetryAttemptsEnv(t *testing.T) {
withEnv("ETCD_MAX_RETRY_ATTEMPTS", "10", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.MaxRetryAttempts, 10, "")
})
}
// Ensures that the Max Retry Attempts flag can be parsed.
func TestConfigMaxRetryAttemptsFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-max-retry-attempts", "10"}), "")
assert.Equal(t, c.MaxRetryAttempts, 10, "")
}
// Ensures that the Name can be parsed from the environment.
func TestConfigNameEnv(t *testing.T) {
withEnv("ETCD_NAME", "test-name", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.Name, "test-name", "")
})
}
// Ensures that the Name flag can be parsed.
func TestConfigNameFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-name", "test-name"}), "")
assert.Equal(t, c.Name, "test-name", "")
}
// Ensures that a Name gets guessed if not specified
func TestConfigNameGuess(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{}), "")
assert.Nil(t, c.Sanitize())
name, _ := os.Hostname()
assert.Equal(t, c.Name, name, "")
}
// Ensures that a DataDir gets guessed if not specified
func TestConfigDataDirGuess(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{}), "")
assert.Nil(t, c.Sanitize())
name, _ := os.Hostname()
assert.Equal(t, c.DataDir, name+".etcd", "")
}
// Ensures that Snapshot can be parsed from the environment.
func TestConfigSnapshotEnv(t *testing.T) {
withEnv("ETCD_SNAPSHOT", "1", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.Snapshot, true, "")
})
}
// Ensures that the Snapshot flag can be parsed.
func TestConfigSnapshotFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-snapshot"}), "")
assert.Equal(t, c.Snapshot, true, "")
}
// Ensures that Verbose can be parsed from the environment.
func TestConfigVerboseEnv(t *testing.T) {
withEnv("ETCD_VERBOSE", "true", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.Verbose, true, "")
})
}
// Ensures that the Verbose flag can be parsed.
func TestConfigVerboseFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-v"}), "")
assert.Equal(t, c.Verbose, true, "")
}
// Ensures that Very Verbose can be parsed from the environment.
func TestConfigVeryVerboseEnv(t *testing.T) {
withEnv("ETCD_VERY_VERBOSE", "true", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.VeryVerbose, true, "")
})
}
// Ensures that the Very Verbose flag can be parsed.
func TestConfigVeryVerboseFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-vv"}), "")
assert.Equal(t, c.VeryVerbose, true, "")
}
// Ensures that the Peer Advertised URL can be parsed from the environment.
func TestConfigPeerAddrEnv(t *testing.T) {
withEnv("ETCD_PEER_ADDR", "localhost:7002", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.Peer.Addr, "localhost:7002", "")
})
}
// Ensures that the Peer Advertised URL flag can be parsed.
func TestConfigPeerAddrFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-peer-addr", "localhost:7002"}), "")
assert.Equal(t, c.Peer.Addr, "localhost:7002", "")
}
// Ensures that the Peer CA File can be parsed from the environment.
func TestConfigPeerCAFileEnv(t *testing.T) {
withEnv("ETCD_PEER_CA_FILE", "/tmp/peer/file.ca", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.Peer.CAFile, "/tmp/peer/file.ca", "")
})
}
// Ensures that the Peer CA file flag can be parsed.
func TestConfigPeerCAFileFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-peer-ca-file", "/tmp/peer/file.ca"}), "")
assert.Equal(t, c.Peer.CAFile, "/tmp/peer/file.ca", "")
}
// Ensures that the Peer Cert File can be parsed from the environment.
func TestConfigPeerCertFileEnv(t *testing.T) {
withEnv("ETCD_PEER_CERT_FILE", "/tmp/peer/file.cert", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.Peer.CertFile, "/tmp/peer/file.cert", "")
})
}
// Ensures that the Cert file flag can be parsed.
func TestConfigPeerCertFileFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-peer-cert-file", "/tmp/peer/file.cert"}), "")
assert.Equal(t, c.Peer.CertFile, "/tmp/peer/file.cert", "")
}
// Ensures that the Peer Key File can be parsed from the environment.
func TestConfigPeerKeyFileEnv(t *testing.T) {
withEnv("ETCD_PEER_KEY_FILE", "/tmp/peer/file.key", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.Peer.KeyFile, "/tmp/peer/file.key", "")
})
}
// Ensures that the Peer Key file flag can be parsed.
func TestConfigPeerKeyFileFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-peer-key-file", "/tmp/peer/file.key"}), "")
assert.Equal(t, c.Peer.KeyFile, "/tmp/peer/file.key", "")
}
// Ensures that the Peer Listen Host can be parsed from the environment.
func TestConfigPeerBindAddrEnv(t *testing.T) {
withEnv("ETCD_PEER_BIND_ADDR", "localhost:7004", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.Peer.BindAddr, "localhost:7004", "")
})
}
// Ensures that a bad flag returns an error.
func TestConfigBadFlag(t *testing.T) {
c := New()
err := c.LoadFlags([]string{"-no-such-flag"})
assert.Error(t, err)
assert.Equal(t, err.Error(), `flag provided but not defined: -no-such-flag`)
}
// Ensures that the Peer Listen Host file flag can be parsed.
func TestConfigPeerBindAddrFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-peer-bind-addr", "127.0.0.1:4003"}), "")
assert.Equal(t, c.Peer.BindAddr, "127.0.0.1:4003", "")
}
// Ensures that the cluster active size can be parsed from the environment.
func TestConfigClusterActiveSizeEnv(t *testing.T) {
withEnv("ETCD_CLUSTER_ACTIVE_SIZE", "5", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.Cluster.ActiveSize, 5, "")
})
}
// Ensures that the cluster active size flag can be parsed.
func TestConfigClusterActiveSizeFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-cluster-active-size", "5"}), "")
assert.Equal(t, c.Cluster.ActiveSize, 5, "")
}
// Ensures that the cluster remove delay can be parsed from the environment.
func TestConfigClusterRemoveDelayEnv(t *testing.T) {
withEnv("ETCD_CLUSTER_REMOVE_DELAY", "100", func(c *Config) {
assert.Nil(t, c.LoadEnv(), "")
assert.Equal(t, c.Cluster.RemoveDelay, 100.0, "")
})
}
// Ensures that the cluster remove delay flag can be parsed.
func TestConfigClusterRemoveDelayFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-cluster-remove-delay", "100"}), "")
assert.Equal(t, c.Cluster.RemoveDelay, 100.0, "")
}
func TestConfigClusterSyncIntervalFlag(t *testing.T) {
c := New()
assert.Nil(t, c.LoadFlags([]string{"-http-read-timeout", "2.34"}), "")
assert.Equal(t, c.HTTPReadTimeout, 2.34, "")
assert.Nil(t, c.LoadFlags([]string{"-http-write-timeout", "1.23"}), "")
assert.Equal(t, c.HTTPWriteTimeout, 1.23, "")
}
// Ensures that a system config field is overridden by a custom config field.
func TestConfigCustomConfigOverrideSystemConfig(t *testing.T) {
system := `addr = "127.0.0.1:5000"`
custom := `addr = "127.0.0.1:6000"`
withTempFile(system, func(p1 string) {
withTempFile(custom, func(p2 string) {
c := New()
c.SystemPath = p1
assert.Nil(t, c.Load([]string{"-config", p2}), "")
assert.Equal(t, c.Addr, "127.0.0.1:6000", "")
})
})
}
// Ensures that a custom config field is overridden by an environment variable.
func TestConfigEnvVarOverrideCustomConfig(t *testing.T) {
os.Setenv("ETCD_PEER_ADDR", "127.0.0.1:8000")
defer os.Setenv("ETCD_PEER_ADDR", "")
custom := `[peer]` + "\n" + `advertised_url = "127.0.0.1:9000"`
withTempFile(custom, func(path string) {
c := New()
c.SystemPath = ""
assert.Nil(t, c.Load([]string{"-config", path}), "")
assert.Equal(t, c.Peer.Addr, "127.0.0.1:8000", "")
})
}
// Ensures that an environment variable field is overridden by a command line argument.
func TestConfigCLIArgsOverrideEnvVar(t *testing.T) {
os.Setenv("ETCD_ADDR", "127.0.0.1:1000")
defer os.Setenv("ETCD_ADDR", "")
c := New()
c.SystemPath = ""
assert.Nil(t, c.Load([]string{"-addr", "127.0.0.1:2000"}), "")
assert.Equal(t, c.Addr, "127.0.0.1:2000", "")
}
//--------------------------------------
// DEPRECATED (v1)
//--------------------------------------
func TestConfigDeprecatedAddrFlag(t *testing.T) {
_, stderr := capture(func() {
c := New()
err := c.LoadFlags([]string{"-c", "127.0.0.1:4002"})
assert.NoError(t, err)
assert.Equal(t, c.Addr, "127.0.0.1:4002")
})
assert.Equal(t, stderr, "[deprecated] use -addr, not -c\n")
}
func TestConfigDeprecatedBindAddrFlag(t *testing.T) {
_, stderr := capture(func() {
c := New()
err := c.LoadFlags([]string{"-cl", "127.0.0.1:4003"})
assert.NoError(t, err)
assert.Equal(t, c.BindAddr, "127.0.0.1:4003", "")
})
assert.Equal(t, stderr, "[deprecated] use -bind-addr, not -cl\n", "")
}
func TestConfigDeprecatedCAFileFlag(t *testing.T) {
_, stderr := capture(func() {
c := New()
err := c.LoadFlags([]string{"-clientCAFile", "/tmp/file.ca"})
assert.NoError(t, err)
assert.Equal(t, c.CAFile, "/tmp/file.ca", "")
})
assert.Equal(t, stderr, "[deprecated] use -ca-file, not -clientCAFile\n", "")
}
func TestConfigDeprecatedCertFileFlag(t *testing.T) {
_, stderr := capture(func() {
c := New()
err := c.LoadFlags([]string{"-clientCert", "/tmp/file.cert"})
assert.NoError(t, err)
assert.Equal(t, c.CertFile, "/tmp/file.cert", "")
})
assert.Equal(t, stderr, "[deprecated] use -cert-file, not -clientCert\n", "")
}
func TestConfigDeprecatedKeyFileFlag(t *testing.T) {
_, stderr := capture(func() {
c := New()
err := c.LoadFlags([]string{"-clientKey", "/tmp/file.key"})
assert.NoError(t, err)
assert.Equal(t, c.KeyFile, "/tmp/file.key", "")
})
assert.Equal(t, stderr, "[deprecated] use -key-file, not -clientKey\n", "")
}
func TestConfigDeprecatedPeersFlag(t *testing.T) {
_, stderr := capture(func() {
c := New()
err := c.LoadFlags([]string{"-C", "coreos.com:4001,coreos.com:4002"})
assert.NoError(t, err)
assert.Equal(t, c.Peers, []string{"coreos.com:4001", "coreos.com:4002"}, "")
})
assert.Equal(t, stderr, "[deprecated] use -peers, not -C\n", "")
}
func TestConfigDeprecatedPeersFileFlag(t *testing.T) {
_, stderr := capture(func() {
c := New()
err := c.LoadFlags([]string{"-CF", "/tmp/machines"})
assert.NoError(t, err)
assert.Equal(t, c.PeersFile, "/tmp/machines", "")
})
assert.Equal(t, stderr, "[deprecated] use -peers-file, not -CF\n", "")
}
func TestConfigDeprecatedMaxRetryAttemptsFlag(t *testing.T) {
_, stderr := capture(func() {
c := New()
err := c.LoadFlags([]string{"-r", "10"})
assert.NoError(t, err)
assert.Equal(t, c.MaxRetryAttempts, 10, "")
})
assert.Equal(t, stderr, "[deprecated] use -max-retry-attempts, not -r\n", "")
}
func TestConfigDeprecatedNameFlag(t *testing.T) {
_, stderr := capture(func() {
c := New()
err := c.LoadFlags([]string{"-n", "test-name"})
assert.NoError(t, err)
assert.Equal(t, c.Name, "test-name", "")
})
assert.Equal(t, stderr, "[deprecated] use -name, not -n\n", "")
}
func TestConfigDeprecatedPeerAddrFlag(t *testing.T) {
_, stderr := capture(func() {
c := New()
err := c.LoadFlags([]string{"-s", "localhost:7002"})
assert.NoError(t, err)
assert.Equal(t, c.Peer.Addr, "localhost:7002", "")
})
assert.Equal(t, stderr, "[deprecated] use -peer-addr, not -s\n", "")
}
func TestConfigDeprecatedPeerBindAddrFlag(t *testing.T) {
_, stderr := capture(func() {
c := New()
err := c.LoadFlags([]string{"-sl", "127.0.0.1:4003"})
assert.NoError(t, err)
assert.Equal(t, c.Peer.BindAddr, "127.0.0.1:4003", "")
})
assert.Equal(t, stderr, "[deprecated] use -peer-bind-addr, not -sl\n", "")
}
func TestConfigDeprecatedPeerCAFileFlag(t *testing.T) {
_, stderr := capture(func() {
c := New()
err := c.LoadFlags([]string{"-serverCAFile", "/tmp/peer/file.ca"})
assert.NoError(t, err)
assert.Equal(t, c.Peer.CAFile, "/tmp/peer/file.ca", "")
})
assert.Equal(t, stderr, "[deprecated] use -peer-ca-file, not -serverCAFile\n", "")
}
func TestConfigDeprecatedPeerCertFileFlag(t *testing.T) {
_, stderr := capture(func() {
c := New()
err := c.LoadFlags([]string{"-serverCert", "/tmp/peer/file.cert"})
assert.NoError(t, err)
assert.Equal(t, c.Peer.CertFile, "/tmp/peer/file.cert", "")
})
assert.Equal(t, stderr, "[deprecated] use -peer-cert-file, not -serverCert\n", "")
}
func TestConfigDeprecatedPeerKeyFileFlag(t *testing.T) {
_, stderr := capture(func() {
c := New()
err := c.LoadFlags([]string{"-serverKey", "/tmp/peer/file.key"})
assert.NoError(t, err)
assert.Equal(t, c.Peer.KeyFile, "/tmp/peer/file.key", "")
})
assert.Equal(t, stderr, "[deprecated] use -peer-key-file, not -serverKey\n", "")
}
//--------------------------------------
// Helpers
//--------------------------------------
// Sets up the environment with a given environment variable set.
func withEnv(key, value string, f func(c *Config)) {
os.Setenv(key, value)
defer os.Setenv(key, "")
c := New()
f(c)
}
// Creates a temp file and calls a function with the context.
func withTempFile(content string, fn func(string)) {
f, _ := ioutil.TempFile("", "")
f.WriteString(content)
f.Close()
defer os.Remove(f.Name())
fn(f.Name())
}
// Captures STDOUT & STDERR and returns the output as strings.
func capture(fn func()) (string, string) {
// Create temp files.
tmpout, _ := ioutil.TempFile("", "")
defer os.Remove(tmpout.Name())
tmperr, _ := ioutil.TempFile("", "")
defer os.Remove(tmperr.Name())
stdout, stderr := os.Stdout, os.Stderr
os.Stdout, os.Stderr = tmpout, tmperr
// Execute function argument and then reassign stdout/stderr.
fn()
os.Stdout, os.Stderr = stdout, stderr
// Close temp files and read them.
tmpout.Close()
bout, _ := ioutil.ReadFile(tmpout.Name())
tmperr.Close()
berr, _ := ioutil.ReadFile(tmperr.Name())
return string(bout), string(berr)
}

9
config/timeout.go Normal file
View File

@@ -0,0 +1,9 @@
package config
const (
// The amount of time (in ms) to elapse without a heartbeat before becoming a candidate
defaultElectionTimeout = 200
// The frequency (in ms) by which heartbeats are sent to followers.
defaultHeartbeatInterval = 50
)

View File

@@ -0,0 +1,9 @@
FROM stackbrew/ubuntu:raring
RUN apt-get update && apt-get install -y collectd
RUN adduser --system --group --no-create-home collectd
ADD collectd.conf /etc/collectd/collectd.conf.tmpl
ADD collectd-wrapper /bin/collectd-wrapper
RUN chown -R collectd:collectd /etc/collectd
CMD ["collectd-wrapper"]

20
contrib/collectd/README Normal file
View File

@@ -0,0 +1,20 @@
We're going to use Docker to build a chroot env that can be run with systemd-nspawn since I cannot figure out how to run
a container using docker in the global network namespace.
1. Build the collectd image using docker
docker build -t collectd .
2. Run the container (since we have to run it to export it...)
COLLECTD_CONTAINER=`docker run -name collectd-tmp -d collectd`
3. Export then kill the container
docker export collectd-tmp > /tmp/collectd.tar
4. Kill the temporary container
docker kill $COLLECTD_CONTAINER
5. Unpack the tar archive
mkdir -p /tmp/collectd && tar -xvf /tmp/collectd.tar -C /tmp/collectd/
6. Run collectd with systemd-nspawn - replace the COLLECTD_* env vars with your parameters!
sudo systemd-run --unit collectd systemd-nspawn -D /tmp/collectd /bin/bash -c "COLLECTD_GRAPHITE_HOSTNAME=172.31.13.241 COLLECTD_LOCAL_HOSTNAME=node1 /bin/collectd-wrapper"

View File

@@ -0,0 +1,16 @@
#!/bin/bash
cat /etc/collectd/collectd.conf.tmpl > /etc/collectd/collectd.conf
cat << EOF >> /etc/collectd/collectd.conf
Hostname "${COLLECTD_LOCAL_HOSTNAME}"
<Plugin write_graphite>
<Carbon>
Host "${COLLECTD_GRAPHITE_HOSTNAME}"
Port "2003"
</Carbon>
</Plugin>
EOF
collectd -C /etc/collectd/collectd.conf -f

View File

@@ -0,0 +1,898 @@
# Config file for collectd(1).
#
# Some plugins need additional configuration and are disabled by default.
# Please read collectd.conf(5) for details.
#
# You should also read /usr/share/doc/collectd-core/README.Debian.plugins
# before enabling any more plugins.
#Hostname "localhost"
#FQDNLookup true
#BaseDir "/var/lib/collectd"
#PluginDir "/usr/lib/collectd"
#TypesDB "/usr/share/collectd/types.db" "/etc/collectd/my_types.db"
#Interval 10
#Timeout 2
#ReadThreads 5
LoadPlugin logfile
#LoadPlugin syslog
<Plugin logfile>
LogLevel "info"
File STDOUT
Timestamp true
PrintSeverity false
</Plugin>
#<Plugin syslog>
# LogLevel info
#</Plugin>
#LoadPlugin amqp
#LoadPlugin apache
#LoadPlugin apcups
#LoadPlugin ascent
#LoadPlugin battery
#LoadPlugin bind
#LoadPlugin conntrack
#LoadPlugin contextswitch
LoadPlugin cpu
#LoadPlugin cpufreq
#LoadPlugin csv
#LoadPlugin curl
#LoadPlugin curl_json
#LoadPlugin curl_xml
#LoadPlugin dbi
LoadPlugin df
#LoadPlugin disk
#LoadPlugin dns
#LoadPlugin email
#LoadPlugin entropy
#LoadPlugin ethstat
#LoadPlugin exec
#LoadPlugin filecount
#LoadPlugin fscache
#LoadPlugin gmond
#LoadPlugin hddtemp
#LoadPlugin interface
#LoadPlugin ipmi
#LoadPlugin iptables
#LoadPlugin ipvs
#LoadPlugin irq
#LoadPlugin java
#LoadPlugin libvirt
#LoadPlugin load
#LoadPlugin madwifi
#LoadPlugin mbmon
#LoadPlugin md
#LoadPlugin memcachec
#LoadPlugin memcached
LoadPlugin memory
#LoadPlugin multimeter
#LoadPlugin mysql
#LoadPlugin netlink
#LoadPlugin network
#LoadPlugin nfs
#LoadPlugin nginx
#LoadPlugin notify_desktop
#LoadPlugin notify_email
#LoadPlugin ntpd
#LoadPlugin numa
#LoadPlugin nut
#LoadPlugin olsrd
#LoadPlugin openvpn
#<LoadPlugin perl>
# Globals true
#</LoadPlugin>
#LoadPlugin pinba
#LoadPlugin ping
#LoadPlugin postgresql
#LoadPlugin powerdns
#LoadPlugin processes
#LoadPlugin protocols
#<LoadPlugin python>
# Globals true
#</LoadPlugin>
#LoadPlugin rrdcached
#LoadPlugin rrdtool
#LoadPlugin sensors
#LoadPlugin serial
#LoadPlugin snmp
#LoadPlugin swap
#LoadPlugin table
#LoadPlugin tail
LoadPlugin tcpconns
#LoadPlugin teamspeak2
#LoadPlugin ted
#LoadPlugin thermal
#LoadPlugin tokyotyrant
#LoadPlugin unixsock
#LoadPlugin uptime
#LoadPlugin users
#LoadPlugin uuid
#LoadPlugin varnish
#LoadPlugin vmem
#LoadPlugin vserver
#LoadPlugin wireless
LoadPlugin write_graphite
#LoadPlugin write_http
#LoadPlugin write_mongodb
#<Plugin amqp>
# <Publish "name">
# Host "localhost"
# Port "5672"
# VHost "/"
# User "guest"
# Password "guest"
# Exchange "amq.fanout"
# RoutingKey "collectd"
# Persistent false
# StoreRates false
# </Publish>
#</Plugin>
#<Plugin apache>
# <Instance "foo">
# URL "http://localhost/server-status?auto"
# User "www-user"
# Password "secret"
# VerifyPeer false
# VerifyHost false
# CACert "/etc/ssl/ca.crt"
# Server "apache"
# </Instance>
#
# <Instance "bar">
# URL "http://some.domain.tld/status?auto"
# Host "some.domain.tld"
# Server "lighttpd"
# </Instance>
#</Plugin>
#<Plugin apcups>
# Host "localhost"
# Port "3551"
#</Plugin>
#<Plugin ascent>
# URL "http://localhost/ascent/status/"
# User "www-user"
# Password "secret"
# VerifyPeer false
# VerifyHost false
# CACert "/etc/ssl/ca.crt"
#</Plugin>
#<Plugin "bind">
# URL "http://localhost:8053/"
#
# ParseTime false
#
# OpCodes true
# QTypes true
# ServerStats true
# ZoneMaintStats true
# ResolverStats false
# MemoryStats true
#
# <View "_default">
# QTypes true
# ResolverStats true
# CacheRRSets true
#
# Zone "127.in-addr.arpa/IN"
# </View>
#</Plugin>
#<Plugin csv>
# DataDir "/var/lib/collectd/csv"
# StoreRates false
#</Plugin>
#<Plugin curl>
# <Page "stock_quotes">
# URL "http://finance.google.com/finance?q=NYSE%3AAMD"
# User "foo"
# Password "bar"
# VerifyPeer false
# VerifyHost false
# CACert "/etc/ssl/ca.crt"
# MeasureResponseTime false
# <Match>
# Regex "<span +class=\"pr\"[^>]*> *([0-9]*\\.[0-9]+) *</span>"
# DSType "GaugeAverage"
# Type "stock_value"
# Instance "AMD"
# </Match>
# </Page>
#</Plugin>
#<Plugin curl_json>
## See: http://wiki.apache.org/couchdb/Runtime_Statistics
# <URL "http://localhost:5984/_stats">
# Instance "httpd"
# <Key "httpd/requests/count">
# Type "http_requests"
# </Key>
#
# <Key "httpd_request_methods/*/count">
# Type "http_request_methods"
# </Key>
#
# <Key "httpd_status_codes/*/count">
# Type "http_response_codes"
# </Key>
# </URL>
## Database status metrics:
# <URL "http://localhost:5984/_all_dbs">
# Instance "dbs"
# <Key "*/doc_count">
# Type "gauge"
# </Key>
# <Key "*/doc_del_count">
# Type "counter"
# </Key>
# <Key "*/disk_size">
# Type "bytes"
# </Key>
# </URL>
#</Plugin>
#<Plugin "curl_xml">
# <URL "http://localhost/stats.xml">
# Host "my_host"
# Instance "some_instance"
# User "collectd"
# Password "thaiNg0I"
# VerifyPeer true
# VerifyHost true
# CACert "/path/to/ca.crt"
#
# <XPath "table[@id=\"magic_level\"]/tr">
# Type "magic_level"
# InstancePrefix "prefix-"
# InstanceFrom "td[1]"
# ValuesFrom "td[2]/span[@class=\"level\"]"
# </XPath>
# </URL>
#</Plugin>
#<Plugin dbi>
# <Query "num_of_customers">
# Statement "SELECT 'customers' AS c_key, COUNT(*) AS c_value \
# FROM customers_tbl"
# MinVersion 40102
# MaxVersion 50042
# <Result>
# Type "gauge"
# InstancePrefix "customer"
# InstancesFrom "c_key"
# ValuesFrom "c_value"
# </Result>
# </Query>
#
# <Database "customers_db">
# Driver "mysql"
# DriverOption "host" "localhost"
# DriverOption "username" "collectd"
# DriverOption "password" "secret"
# DriverOption "dbname" "custdb0"
# SelectDB "custdb0"
# Query "num_of_customers"
# Query "..."
# </Database>
#</Plugin>
#<Plugin df>
# Device "/dev/sda1"
# Device "192.168.0.2:/mnt/nfs"
# MountPoint "/home"
# FSType "ext3"
# IgnoreSelected false
# ReportByDevice false
# ReportReserved false
# ReportInodes false
#</Plugin>
#<Plugin disk>
# Disk "hda"
# Disk "/sda[23]/"
# IgnoreSelected false
#</Plugin>
#<Plugin dns>
# Interface "eth0"
# IgnoreSource "192.168.0.1"
# SelectNumericQueryTypes false
#</Plugin>
#<Plugin email>
# SocketFile "/var/run/collectd-email"
# SocketGroup "collectd"
# SocketPerms "0770"
# MaxConns 5
#</Plugin>
#<Plugin ethstat>
# Interface "eth0"
# Map "rx_csum_offload_errors" "if_rx_errors" "checksum_offload"
# Map "multicast" "if_multicast"
# MappedOnly false
#</Plugin>
#<Plugin exec>
# Exec user "/path/to/exec"
# Exec "user:group" "/path/to/exec"
# NotificationExec user "/path/to/exec"
#</Plugin>
#<Plugin filecount>
# <Directory "/path/to/dir">
# Instance "foodir"
# Name "*.conf"
# MTime "-5m"
# Size "+10k"
# Recursive true
# IncludeHidden false
# </Directory>
#</Plugin>
#<Plugin gmond>
# MCReceiveFrom "239.2.11.71" "8649"
#
# <Metric "swap_total">
# Type "swap"
# TypeInstance "total"
# DataSource "value"
# </Metric>
#
# <Metric "swap_free">
# Type "swap"
# TypeInstance "free"
# DataSource "value"
# </Metric>
#</Plugin>
#<Plugin hddtemp>
# Host "127.0.0.1"
# Port 7634
#</Plugin>
#<Plugin interface>
# Interface "eth0"
# IgnoreSelected false
#</Plugin>
#<Plugin ipmi>
# Sensor "some_sensor"
# Sensor "another_one"
# IgnoreSelected false
# NotifySensorAdd false
# NotifySensorRemove true
# NotifySensorNotPresent false
#</Plugin>
#<Plugin iptables>
# Chain "table" "chain"
#</Plugin>
#<Plugin irq>
# Irq 7
# Irq 8
# Irq 9
# IgnoreSelected true
#</Plugin>
#<Plugin java>
# JVMArg "-verbose:jni"
# JVMArg "-Djava.class.path=/usr/share/collectd/java/collectd-api.jar"
#
# LoadPlugin "org.collectd.java.GenericJMX"
# <Plugin "GenericJMX">
# # See /usr/share/doc/collectd/examples/GenericJMX.conf
# # for an example config.
# </Plugin>
#</Plugin>
#<Plugin libvirt>
# Connection "xen:///"
# RefreshInterval 60
# Domain "name"
# BlockDevice "name:device"
# InterfaceDevice "name:device"
# IgnoreSelected false
# HostnameFormat name
# InterfaceFormat name
#</Plugin>
#<Plugin madwifi>
# Interface "wlan0"
# IgnoreSelected false
# Source "SysFS"
# WatchSet "None"
# WatchAdd "node_octets"
# WatchAdd "node_rssi"
# WatchAdd "is_rx_acl"
# WatchAdd "is_scan_active"
#</Plugin>
#<Plugin mbmon>
# Host "127.0.0.1"
# Port 411
#</Plugin>
#<Plugin md>
# Device "/dev/md0"
# IgnoreSelected false
#</Plugin>
#<Plugin memcachec>
# <Page "plugin_instance">
# Server "localhost"
# Key "page_key"
# <Match>
# Regex "(\\d+) bytes sent"
# ExcludeRegex "<lines to be excluded>"
# DSType CounterAdd
# Type "ipt_octets"
# Instance "type_instance"
# </Match>
# </Page>
#</Plugin>
#<Plugin memcached>
# Socket "/var/run/memcached.sock"
# or:
# Host "127.0.0.1"
# Port "11211"
#</Plugin>
#<Plugin mysql>
# <Database db_name>
# Host "database.serv.er"
# Port "3306"
# User "db_user"
# Password "secret"
# Database "db_name"
# MasterStats true
# </Database>
#
# <Database db_name2>
# Host "localhost"
# Socket "/var/run/mysql/mysqld.sock"
# SlaveStats true
# SlaveNotifications true
# </Database>
#</Plugin>
#<Plugin netlink>
# Interface "All"
# VerboseInterface "All"
# QDisc "eth0" "pfifo_fast-1:0"
# Class "ppp0" "htb-1:10"
# Filter "ppp0" "u32-1:0"
# IgnoreSelected false
#</Plugin>
#<Plugin network>
# # client setup:
# Server "ff18::efc0:4a42" "25826"
# <Server "239.192.74.66" "25826">
# SecurityLevel Encrypt
# Username "user"
# Password "secret"
# Interface "eth0"
# </Server>
# TimeToLive "128"
#
# # server setup:
# Listen "0.0.0.0" "25826"
# <Listen "239.192.74.66" "25826">
# SecurityLevel Sign
# AuthFile "/etc/collectd/passwd"
# Interface "eth0"
# </Listen>
# MaxPacketSize 1024
#
# # proxy setup (client and server as above):
# Forward true
#
# # statistics about the network plugin itself
# ReportStats false
#
# # "garbage collection"
# CacheFlush 1800
#</Plugin>
#<Plugin nginx>
# URL "http://localhost/status?auto"
# User "www-user"
# Password "secret"
# VerifyPeer false
# VerifyHost false
# CACert "/etc/ssl/ca.crt"
#</Plugin>
#<Plugin notify_desktop>
# OkayTimeout 1000
# WarningTimeout 5000
# FailureTimeout 0
#</Plugin>
#<Plugin notify_email>
# SMTPServer "localhost"
# SMTPPort 25
# SMTPUser "my-username"
# SMTPPassword "my-password"
# From "collectd@main0server.com"
# # <WARNING/FAILURE/OK> on <hostname>.
# # Beware! Do not use not more than two placeholders (%)!
# Subject "[collectd] %s on %s!"
# Recipient "email1@domain1.net"
# Recipient "email2@domain2.com"
#</Plugin>
#<Plugin ntpd>
# Host "localhost"
# Port 123
# ReverseLookups false
#</Plugin>
#<Plugin nut>
# UPS "upsname@hostname:port"
#</Plugin>
#<Plugin olsrd>
# Host "127.0.0.1"
# Port "2006"
# CollectLinks "Summary"
# CollectRoutes "Summary"
# CollectTopology "Summary"
#</Plugin>
#<Plugin openvpn>
# StatusFile "/etc/openvpn/openvpn-status.log"
# ImprovedNamingSchema false
# CollectCompression true
# CollectIndividualUsers true
# CollectUserCount false
#</Plugin>
#<Plugin perl>
# IncludeDir "/my/include/path"
# BaseName "Collectd::Plugins"
# EnableDebugger ""
# LoadPlugin Monitorus
# LoadPlugin OpenVZ
#
# <Plugin foo>
# Foo "Bar"
# Qux "Baz"
# </Plugin>
#</Plugin>
#<Plugin pinba>
# Address "::0"
# Port "30002"
# <View "name">
# Host "host name"
# Server "server name"
# Script "script name"
# <View>
#</Plugin>
#<Plugin ping>
# Host "host.foo.bar"
# Host "host.baz.qux"
# Interval 1.0
# Timeout 0.9
# TTL 255
# SourceAddress "1.2.3.4"
# Device "eth0"
# MaxMissed -1
#</Plugin>
#<Plugin postgresql>
# <Query magic>
# Statement "SELECT magic FROM wizard WHERE host = $1;"
# Param hostname
#
# <Result>
# Type gauge
# InstancePrefix "magic"
# ValuesFrom "magic"
# </Result>
# </Query>
#
# <Query rt36_tickets>
# Statement "SELECT COUNT(type) AS count, type \
# FROM (SELECT CASE \
# WHEN resolved = 'epoch' THEN 'open' \
# ELSE 'resolved' END AS type \
# FROM tickets) type \
# GROUP BY type;"
#
# <Result>
# Type counter
# InstancePrefix "rt36_tickets"
# InstancesFrom "type"
# ValuesFrom "count"
# </Result>
# </Query>
#
# <Database foo>
# Host "hostname"
# Port 5432
# User "username"
# Password "secret"
#
# SSLMode "prefer"
# KRBSrvName "kerberos_service_name"
#
# Query magic
# </Database>
#
# <Database bar>
# Interval 60
# Service "service_name"
#
# Query backend # predefined
# Query rt36_tickets
# </Database>
#</Plugin>
#<Plugin powerdns>
# <Server "server_name">
# Collect "latency"
# Collect "udp-answers" "udp-queries"
# Socket "/var/run/pdns.controlsocket"
# </Server>
# <Recursor "recursor_name">
# Collect "questions"
# Collect "cache-hits" "cache-misses"
# Socket "/var/run/pdns_recursor.controlsocket"
# </Recursor>
# LocalSocket "/opt/collectd/var/run/collectd-powerdns"
#</Plugin>
#<Plugin processes>
# Process "name"
# ProcessMatch "foobar" "/usr/bin/perl foobar\\.pl.*"
#</Plugin>
#<Plugin protocols>
# Value "/^Tcp:/"
# IgnoreSelected false
#</Plugin>
#<Plugin python>
# ModulePath "/path/to/your/python/modules"
# LogTraces true
# Interactive true
# Import "spam"
#
# <Module spam>
# spam "wonderful" "lovely"
# </Module>
#</Plugin>
#<Plugin rrdcached>
# DaemonAddress "unix:/var/run/rrdcached.sock"
# DataDir "/var/lib/rrdcached/db/collectd"
# CreateFiles true
# CollectStatistics true
#</Plugin>
<Plugin rrdtool>
DataDir "/var/lib/collectd/rrd"
# CacheTimeout 120
# CacheFlush 900
# WritesPerSecond 30
# RandomTimeout 0
#
# The following settings are rather advanced
# and should usually not be touched:
# StepSize 10
# HeartBeat 20
# RRARows 1200
# RRATimespan 158112000
# XFF 0.1
</Plugin>
#<Plugin sensors>
# SensorConfigFile "/etc/sensors3.conf"
# Sensor "it8712-isa-0290/temperature-temp1"
# Sensor "it8712-isa-0290/fanspeed-fan3"
# Sensor "it8712-isa-0290/voltage-in8"
# IgnoreSelected false
#</Plugin>
# See /usr/share/doc/collectd/examples/snmp-data.conf.gz for a
# comprehensive sample configuration.
#<Plugin snmp>
# <Data "powerplus_voltge_input">
# Type "voltage"
# Table false
# Instance "input_line1"
# Scale 0.1
# Values "SNMPv2-SMI::enterprises.6050.5.4.1.1.2.1"
# </Data>
# <Data "hr_users">
# Type "users"
# Table false
# Instance ""
# Shift -1
# Values "HOST-RESOURCES-MIB::hrSystemNumUsers.0"
# </Data>
# <Data "std_traffic">
# Type "if_octets"
# Table true
# InstancePrefix "traffic"
# Instance "IF-MIB::ifDescr"
# Values "IF-MIB::ifInOctets" "IF-MIB::ifOutOctets"
# </Data>
#
# <Host "some.switch.mydomain.org">
# Address "192.168.0.2"
# Version 1
# Community "community_string"
# Collect "std_traffic"
# Inverval 120
# </Host>
# <Host "some.server.mydomain.org">
# Address "192.168.0.42"
# Version 2
# Community "another_string"
# Collect "std_traffic" "hr_users"
# </Host>
# <Host "some.ups.mydomain.org">
# Address "192.168.0.3"
# Version 1
# Community "more_communities"
# Collect "powerplus_voltge_input"
# Interval 300
# </Host>
#</Plugin>
#<Plugin swap>
# ReportByDevice false
#</Plugin>
#<Plugin table>
# <Table "/proc/slabinfo">
# Instance "slabinfo"
# Separator " "
# <Result>
# Type gauge
# InstancePrefix "active_objs"
# InstancesFrom 0
# ValuesFrom 1
# </Result>
# <Result>
# Type gauge
# InstancePrefix "objperslab"
# InstancesFrom 0
# ValuesFrom 4
# </Result>
# </Table>
#</Plugin>
#<Plugin "tail">
# <File "/var/log/exim4/mainlog">
# Instance "exim"
# <Match>
# Regex "S=([1-9][0-9]*)"
# DSType "CounterAdd"
# Type "ipt_bytes"
# Instance "total"
# </Match>
# <Match>
# Regex "\\<R=local_user\\>"
# ExcludeRegex "\\<R=local_user\\>.*mail_spool defer"
# DSType "CounterInc"
# Type "counter"
# Instance "local_user"
# </Match>
# </File>
#</Plugin>
<Plugin tcpconns>
LocalPort "4001"
LocalPort "7001"
</Plugin>
#<Plugin teamspeak2>
# Host "127.0.0.1"
# Port "51234"
# Server "8767"
#</Plugin>
#<Plugin ted>
# Device "/dev/ttyUSB0"
# Retries 0
#</Plugin>
#<Plugin thermal>
# ForceUseProcfs false
# Device "THRM"
# IgnoreSelected false
#</Plugin>
#<Plugin tokyotyrant>
# Host "localhost"
# Port "1978"
#</Plugin>
#<Plugin unixsock>
# SocketFile "/var/run/collectd-unixsock"
# SocketGroup "collectd"
# SocketPerms "0660"
# DeleteSocket false
#</Plugin>
#<Plugin uuid>
# UUIDFile "/etc/uuid"
#</Plugin>
#<Plugin varnish>
# <Instance>
# CollectCache true
# CollectBackend true
# CollectConnections true
# CollectSHM true
# CollectESI false
# CollectFetch false
# CollectHCB false
# CollectSMA false
# CollectSMS false
# CollectSM false
# CollectTotals false
# CollectWorkers false
# </Instance>
#
# <Instance "myinstance">
# CollectCache true
# </Instance>
#</Plugin>
#<Plugin vmem>
# Verbose false
#</Plugin>
#<Plugin write_graphite>
# <Carbon>
# Host "127.0.01"
# Port "2003"
# Prefix "collectd"
# Postfix "collectd"
# StoreRates false
# AlwaysAppendDS false
# EscapeCharacter "_"
# </Carbon>
#</Plugin>
#<Plugin write_http>
# <URL "http://example.com/collectd-post">
# User "collectd"
# Password "secret"
# VerifyPeer true
# VerifyHost true
# CACert "/etc/ssl/ca.crt"
# Format "Command"
# StoreRates false
# </URL>
#</Plugin>
#<Plugin write_mongodb>
# <Node "example">
# Host "localhost"
# Port "27017"
# Timeout 1000
# StoreRates false
# <Node>
#</Plugin>
Include "/etc/collectd/filters.conf"
Include "/etc/collectd/thresholds.conf"

View File

@@ -0,0 +1,31 @@
from stackbrew/ubuntu:precise
run echo 'deb http://us.archive.ubuntu.com/ubuntu/ precise universe' >> /etc/apt/sources.list
run apt-get -y update
# Install required packages
run apt-get -y install python-cairo python-django python-twisted python-django-tagging python-simplejson python-pysqlite2 python-support python-pip gunicorn supervisor nginx-light
run pip install whisper
run pip install --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/lib" carbon
run pip install --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/webapp" graphite-web
# Add system service config
add ./nginx.conf /etc/nginx/nginx.conf
add ./supervisord.conf /etc/supervisor/conf.d/supervisord.conf
# Add graphite config
add ./initial_data.json /var/lib/graphite/webapp/graphite/initial_data.json
add ./local_settings.py /var/lib/graphite/webapp/graphite/local_settings.py
add ./carbon.conf /var/lib/graphite/conf/carbon.conf
add ./storage-schemas.conf /var/lib/graphite/conf/storage-schemas.conf
run mkdir -p /var/lib/graphite/storage/whisper
run touch /var/lib/graphite/storage/graphite.db /var/lib/graphite/storage/index
run chown -R www-data /var/lib/graphite/storage
run chmod 0775 /var/lib/graphite/storage /var/lib/graphite/storage/whisper
run chmod 0664 /var/lib/graphite/storage/graphite.db
run cd /var/lib/graphite/webapp/graphite && python manage.py syncdb --noinput
expose :80
expose :2003
cmd ["/usr/bin/supervisord"]

7
contrib/graphite/README Normal file
View File

@@ -0,0 +1,7 @@
Running graphite under Docker is straightforward:
1. Build the graphite image using Docker
docker build -t graphite .
2. Run a graphite container. Be sure to replace the $IP field with the IP address at which you wish to expose your graphite web service.
docker run -p $IP:8080:80 -p $IP:2003:2003 -d graphite

View File

@@ -0,0 +1,62 @@
[cache]
LOCAL_DATA_DIR = /var/lib/graphite/storage/whisper/
# Specify the user to drop privileges to
# If this is blank carbon runs as the user that invokes it
# This user must have write access to the local data directory
USER =
# Limit the size of the cache to avoid swapping or becoming CPU bound.
# Sorts and serving cache queries gets more expensive as the cache grows.
# Use the value "inf" (infinity) for an unlimited cache size.
MAX_CACHE_SIZE = inf
# Limits the number of whisper update_many() calls per second, which effectively
# means the number of write requests sent to the disk. This is intended to
# prevent over-utilizing the disk and thus starving the rest of the system.
# When the rate of required updates exceeds this, then carbon's caching will
# take effect and increase the overall throughput accordingly.
MAX_UPDATES_PER_SECOND = 1000
# Softly limits the number of whisper files that get created each minute.
# Setting this value low (like at 50) is a good way to ensure your graphite
# system will not be adversely impacted when a bunch of new metrics are
# sent to it. The trade off is that it will take much longer for those metrics'
# database files to all get created and thus longer until the data becomes usable.
# Setting this value high (like "inf" for infinity) will cause graphite to create
# the files quickly but at the risk of slowing I/O down considerably for a while.
MAX_CREATES_PER_MINUTE = inf
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
#PICKLE_RECEIVER_INTERFACE = 0.0.0.0
#PICKLE_RECEIVER_PORT = 2004
#CACHE_QUERY_INTERFACE = 0.0.0.0
#CACHE_QUERY_PORT = 7002
LOG_UPDATES = False
# Enable AMQP if you want to receve metrics using an amqp broker
# ENABLE_AMQP = False
# Verbose means a line will be logged for every metric received
# useful for testing
# AMQP_VERBOSE = False
# AMQP_HOST = localhost
# AMQP_PORT = 5672
# AMQP_VHOST = /
# AMQP_USER = guest
# AMQP_PASSWORD = guest
# AMQP_EXCHANGE = graphite
# Patterns for all of the metrics this machine will store. Read more at
# http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol#Bindings
#
# Example: store all sales, linux servers, and utilization metrics
# BIND_PATTERNS = sales.#, servers.linux.#, #.utilization
#
# Example: store everything
# BIND_PATTERNS = #

View File

@@ -0,0 +1,20 @@
[
{
"pk": 1,
"model": "auth.user",
"fields": {
"username": "admin",
"first_name": "",
"last_name": "",
"is_active": true,
"is_superuser": true,
"is_staff": true,
"last_login": "2011-09-20 17:02:14",
"groups": [],
"user_permissions": [],
"password": "sha1$1b11b$edeb0a67a9622f1f2cfeabf9188a711f5ac7d236",
"email": "root@example.com",
"date_joined": "2011-09-20 17:02:14"
}
}
]

View File

@@ -0,0 +1 @@
TIME_ZONE = 'UTC'

View File

@@ -0,0 +1,69 @@
daemon off;
user www-data;
worker_processes 1;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
server_tokens off;
server_names_hash_bucket_size 32;
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
gzip on;
gzip_disable "msie6";
server {
listen 80 default_server;
server_name _;
open_log_file_cache max=1000 inactive=20s min_uses=2 valid=1m;
location / {
proxy_pass http://127.0.0.1:8000;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header Host $host;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
}
add_header Access-Control-Allow-Origin "*";
add_header Access-Control-Allow-Methods "GET, OPTIONS";
add_header Access-Control-Allow-Headers "origin, authorization, accept";
location /content {
alias /var/lib/graphite/webapp/content;
}
location /media {
alias /usr/share/pyshared/django/contrib/admin/media;
}
}
}

View File

@@ -0,0 +1,7 @@
[carbon]
pattern = ^carbon\..*
retentions = 1m:31d,10m:1y,1h:5y
[default]
pattern = .*
retentions = 10s:8d,1m:31d,10m:1y,1h:5y

View File

@@ -0,0 +1,25 @@
[supervisord]
nodaemon = true
environment = GRAPHITE_STORAGE_DIR='/var/lib/graphite/storage',GRAPHITE_CONF_DIR='/var/lib/graphite/conf'
[program:nginx]
command = /usr/sbin/nginx
stdout_logfile = /var/log/supervisor/%(program_name)s.log
stderr_logfile = /var/log/supervisor/%(program_name)s.log
autorestart = true
[program:carbon-cache]
user = www-data
command = /var/lib/graphite/bin/carbon-cache.py --debug start
stdout_logfile = /var/log/supervisor/%(program_name)s.log
stderr_logfile = /var/log/supervisor/%(program_name)s.log
autorestart = true
[program:graphite-webapp]
user = www-data
directory = /var/lib/graphite/webapp
environment = PYTHONPATH='/var/lib/graphite/webapp'
command = /usr/bin/gunicorn_django -b127.0.0.1:8000 -w2 graphite/settings.py
stdout_logfile = /var/log/supervisor/%(program_name)s.log
stderr_logfile = /var/log/supervisor/%(program_name)s.log
autorestart = true

30
cover
View File

@@ -1,30 +0,0 @@
#!/bin/bash -e
#
# Generate coverage HTML for a package
# e.g. PKG=./unit ./cover
#
if [ -z "$PKG" ]; then
echo "cover only works with a single package, sorry"
exit 255
fi
COVEROUT="coverage"
if ! [ -d "$COVEROUT" ]; then
mkdir "$COVEROUT"
fi
# strip leading dot/slash and trailing slash and sanitize other slashes
# e.g. ./etcdserver/etcdhttp/ ==> etcdserver_etcdhttp
COVERPKG=${PKG/#./}
COVERPKG=${COVERPKG/#\//}
COVERPKG=${COVERPKG/%\//}
COVERPKG=${COVERPKG//\//_}
# generate arg for "go test"
export COVER="-coverprofile ${COVEROUT}/${COVERPKG}.out"
source ./test
go tool cover -html=${COVEROUT}/${COVERPKG}.out

View File

@@ -1,288 +1,148 @@
/*
Copyright 2014 CoreOS, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package discovery
import (
"errors"
"fmt"
"log"
"net/http"
"net/url"
"os"
"path"
"sort"
"strconv"
"strings"
"time"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/jonboulle/clockwork"
"github.com/coreos/etcd/client"
"github.com/coreos/etcd/pkg/types"
)
var (
ErrInvalidURL = errors.New("discovery: invalid URL")
ErrBadSizeKey = errors.New("discovery: size key is bad")
ErrSizeNotFound = errors.New("discovery: size key not found")
ErrTokenNotFound = errors.New("discovery: token not found")
ErrDuplicateID = errors.New("discovery: found duplicate id")
ErrFullCluster = errors.New("discovery: cluster is full")
ErrTooManyRetries = errors.New("discovery: too many retries")
etcdErr "github.com/coreos/etcd/error"
"github.com/coreos/etcd/log"
"github.com/coreos/etcd/third_party/github.com/coreos/go-etcd/etcd"
)
const (
// Environment variable used to configure an HTTP proxy for discovery
DiscoveryProxyEnv = "ETCD_DISCOVERY_PROXY"
// Number of retries discovery will attempt before giving up and erroring out.
nRetries = uint(3)
stateKey = "_state"
startedState = "started"
defaultTTL = 604800 // One week TTL
)
type Discoverer interface {
Discover() (string, error)
type Discoverer struct {
client *etcd.Client
name string
peer string
prefix string
discoveryURL string
}
type discovery struct {
cluster string
id types.ID
config string
c client.KeysAPI
retries uint
url *url.URL
var defaultDiscoverer *Discoverer
clock clockwork.Clock
func init() {
defaultDiscoverer = &Discoverer{}
}
// proxyFuncFromEnv builds a proxy function if the appropriate environment
// variable is set. It performs basic sanitization of the environment variable
// and returns any error encountered.
func proxyFuncFromEnv() (func(*http.Request) (*url.URL, error), error) {
proxy := os.Getenv(DiscoveryProxyEnv)
if proxy == "" {
return nil, nil
func (d *Discoverer) Do(discoveryURL string, name string, peer string, closeChan <-chan bool, startRoutine func(func())) (peers []string, err error) {
d.name = name
d.peer = peer
d.discoveryURL = discoveryURL
u, err := url.Parse(discoveryURL)
if err != nil {
return
}
// Do a small amount of URL sanitization to help the user
// Derived from net/http.ProxyFromEnvironment
proxyURL, err := url.Parse(proxy)
if err != nil || !strings.HasPrefix(proxyURL.Scheme, "http") {
// proxy was bogus. Try prepending "http://" to it and
// see if that parses correctly. If not, we ignore the
// error and complain about the original one
var err2 error
proxyURL, err2 = url.Parse("http://" + proxy)
if err2 == nil {
err = nil
// prefix is prepended to all keys for this discovery
d.prefix = strings.TrimPrefix(u.Path, "/v2/keys/")
// keep the old path in case we need to set the KeyPrefix below
oldPath := u.Path
u.Path = ""
// Connect to a scheme://host not a full URL with path
log.Infof("Discovery via %s using prefix %s.", u.String(), d.prefix)
d.client = etcd.NewClient([]string{u.String()})
if !strings.HasPrefix(oldPath, "/v2/keys") {
d.client.SetKeyPrefix("")
}
// Register this machine first and announce that we are a member of
// this cluster
err = d.heartbeat()
if err != nil {
return
}
// Start the very slow heartbeat to the cluster now in anticipation
// that everything is going to go alright now
startRoutine(func() { d.startHeartbeat(closeChan) })
// Attempt to take the leadership role, if there is no error we are it!
resp, err := d.client.Create(path.Join(d.prefix, stateKey), startedState, 0)
// Bail out on unexpected errors
if err != nil {
if clientErr, ok := err.(*etcd.EtcdError); !ok || clientErr.ErrorCode != etcdErr.EcodeNodeExist {
return nil, err
}
}
if err != nil {
return nil, fmt.Errorf("invalid proxy address %q: %v", proxy, err)
// If we got a response then the CAS was successful, we are leader
if resp != nil && resp.Node.Value == startedState {
// We are the leader, we have no peers
log.Infof("Discovery _state was empty, so this machine is the initial leader.")
return nil, nil
}
log.Printf("discovery: using proxy %q", proxyURL.String())
return http.ProxyURL(proxyURL), nil
// Fall through to finding the other discovery peers
return d.findPeers()
}
func New(durl string, id types.ID, config string) (Discoverer, error) {
u, err := url.Parse(durl)
func (d *Discoverer) findPeers() (peers []string, err error) {
resp, err := d.client.Get(path.Join(d.prefix), false, true)
if err != nil {
return nil, err
}
token := u.Path
u.Path = ""
pf, err := proxyFuncFromEnv()
if err != nil {
return nil, err
node := resp.Node
if node == nil {
return nil, fmt.Errorf("%s key doesn't exist.", d.prefix)
}
c, err := client.NewDiscoveryKeysAPI(&http.Transport{Proxy: pf}, []string{u.String()}, client.DefaultRequestTimeout)
if err != nil {
return nil, err
for _, n := range node.Nodes {
// Skip our own entry in the list, there is no point
if strings.HasSuffix(n.Key, "/"+d.name) {
continue
}
peers = append(peers, n.Value)
}
return &discovery{
cluster: token,
id: id,
config: config,
c: c,
url: u,
clock: clockwork.NewRealClock(),
}, nil
if len(peers) == 0 {
return nil, errors.New("Discovery found an initialized cluster but no reachable peers are registered.")
}
log.Infof("Discovery found peers %v", peers)
return
}
func (d *discovery) Discover() (string, error) {
// fast path: if the cluster is full, returns the error
// do not need to register itself to the cluster in this
// case.
if _, _, err := d.checkCluster(); err != nil {
return "", err
func (d *Discoverer) startHeartbeat(closeChan <-chan bool) {
// In case of errors we should attempt to heartbeat fairly frequently
heartbeatInterval := defaultTTL / 8
ticker := time.NewTicker(time.Second * time.Duration(heartbeatInterval))
defer ticker.Stop()
for {
select {
case <-ticker.C:
err := d.heartbeat()
if err != nil {
log.Warnf("Discovery heartbeat failed: %v", err)
}
case <-closeChan:
return
}
}
if err := d.createSelf(); err != nil {
// Fails, even on a timeout, if createSelf times out.
// TODO(barakmich): Retrying the same node might want to succeed here
// (ie, createSelf should be idempotent for discovery).
return "", err
}
nodes, size, err := d.checkCluster()
if err != nil {
return "", err
}
all, err := d.waitNodes(nodes, size)
if err != nil {
return "", err
}
return nodesToCluster(all), nil
}
func (d *discovery) createSelf() error {
resp, err := d.c.Create(d.selfKey(), d.config, -1)
if err != nil {
return err
}
// ensure self appears on the server we connected to
w := d.c.Watch(d.selfKey(), resp.Node.CreatedIndex)
_, err = w.Next()
func (d *Discoverer) heartbeat() error {
_, err := d.client.Set(path.Join(d.prefix, d.name), d.peer, defaultTTL)
return err
}
func (d *discovery) checkCluster() (client.Nodes, int, error) {
configKey := path.Join("/", d.cluster, "_config")
// find cluster size
resp, err := d.c.Get(path.Join(configKey, "size"))
if err != nil {
if err == client.ErrKeyNoExist {
return nil, 0, ErrSizeNotFound
}
if err == client.ErrTimeout {
return d.checkClusterRetry()
}
return nil, 0, err
}
size, err := strconv.Atoi(resp.Node.Value)
if err != nil {
return nil, 0, ErrBadSizeKey
}
resp, err = d.c.Get(d.cluster)
if err != nil {
if err == client.ErrTimeout {
return d.checkClusterRetry()
}
return nil, 0, err
}
nodes := make(client.Nodes, 0)
// append non-config keys to nodes
for _, n := range resp.Node.Nodes {
if !(path.Base(n.Key) == path.Base(configKey)) {
nodes = append(nodes, n)
}
}
snodes := sortableNodes{nodes}
sort.Sort(snodes)
// find self position
for i := range nodes {
if path.Base(nodes[i].Key) == path.Base(d.selfKey()) {
break
}
if i >= size-1 {
return nil, size, ErrFullCluster
}
}
return nodes, size, nil
func Do(discoveryURL string, name string, peer string, closeChan <-chan bool, startRoutine func(func())) ([]string, error) {
return defaultDiscoverer.Do(discoveryURL, name, peer, closeChan, startRoutine)
}
func (d *discovery) logAndBackoffForRetry(step string) {
d.retries++
retryTime := time.Second * (0x1 << d.retries)
log.Println("discovery: during", step, "connection to", d.url, "timed out, retrying in", retryTime)
d.clock.Sleep(retryTime)
}
func (d *discovery) checkClusterRetry() (client.Nodes, int, error) {
if d.retries < nRetries {
d.logAndBackoffForRetry("cluster status check")
return d.checkCluster()
}
return nil, 0, ErrTooManyRetries
}
func (d *discovery) waitNodesRetry() (client.Nodes, error) {
if d.retries < nRetries {
d.logAndBackoffForRetry("waiting for other nodes")
nodes, n, err := d.checkCluster()
if err != nil {
return nil, err
}
return d.waitNodes(nodes, n)
}
return nil, ErrTooManyRetries
}
func (d *discovery) waitNodes(nodes client.Nodes, size int) (client.Nodes, error) {
if len(nodes) > size {
nodes = nodes[:size]
}
w := d.c.RecursiveWatch(d.cluster, nodes[len(nodes)-1].ModifiedIndex+1)
all := make(client.Nodes, len(nodes))
copy(all, nodes)
for _, n := range all {
if path.Base(n.Key) == path.Base(d.selfKey()) {
log.Printf("discovery: found self %s in the cluster", path.Base(d.selfKey()))
} else {
log.Printf("discovery: found peer %s in the cluster", path.Base(n.Key))
}
}
// wait for others
for len(all) < size {
log.Printf("discovery: found %d peer(s), waiting for %d more", len(all), size-len(all))
resp, err := w.Next()
if err != nil {
if err == client.ErrTimeout {
return d.waitNodesRetry()
}
return nil, err
}
log.Printf("discovery: found peer %s in the cluster", path.Base(resp.Node.Key))
all = append(all, resp.Node)
}
log.Printf("discovery: found %d needed peer(s)", len(all))
return all, nil
}
func (d *discovery) selfKey() string {
return path.Join("/", d.cluster, d.id.String())
}
func nodesToCluster(ns client.Nodes) string {
s := make([]string, len(ns))
for i, n := range ns {
s[i] = n.Value
}
return strings.Join(s, ",")
}
type sortableNodes struct{ client.Nodes }
func (ns sortableNodes) Len() int { return len(ns.Nodes) }
func (ns sortableNodes) Less(i, j int) bool {
return ns.Nodes[i].CreatedIndex < ns.Nodes[j].CreatedIndex
}
func (ns sortableNodes) Swap(i, j int) { ns.Nodes[i], ns.Nodes[j] = ns.Nodes[j], ns.Nodes[i] }

View File

@@ -1,509 +0,0 @@
/*
Copyright 2014 CoreOS, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package discovery
import (
"errors"
"math/rand"
"net/http"
"os"
"sort"
"strconv"
"reflect"
"testing"
"time"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/jonboulle/clockwork"
"github.com/coreos/etcd/client"
)
func TestProxyFuncFromEnvUnset(t *testing.T) {
os.Setenv(DiscoveryProxyEnv, "")
pf, err := proxyFuncFromEnv()
if pf != nil {
t.Fatal("unexpected non-nil proxyFunc")
}
if err != nil {
t.Fatalf("unexpected non-nil err: %v", err)
}
}
func TestProxyFuncFromEnvBad(t *testing.T) {
tests := []string{
"%%",
"http://foo.com/%1",
}
for i, in := range tests {
os.Setenv(DiscoveryProxyEnv, in)
pf, err := proxyFuncFromEnv()
if pf != nil {
t.Errorf("#%d: unexpected non-nil proxyFunc", i)
}
if err == nil {
t.Errorf("#%d: unexpected nil err", i)
}
}
}
func TestProxyFuncFromEnv(t *testing.T) {
tests := map[string]string{
"bar.com": "http://bar.com",
"http://disco.foo.bar": "http://disco.foo.bar",
}
for in, w := range tests {
os.Setenv(DiscoveryProxyEnv, in)
pf, err := proxyFuncFromEnv()
if pf == nil {
t.Errorf("%s: unexpected nil proxyFunc", in)
continue
}
if err != nil {
t.Errorf("%s: unexpected non-nil err: %v", in, err)
continue
}
g, err := pf(&http.Request{})
if err != nil {
t.Errorf("%s: unexpected non-nil err: %v", in, err)
}
if g.String() != w {
t.Errorf("%s: proxyURL=%q, want %q", in, g, w)
}
}
}
func TestCheckCluster(t *testing.T) {
cluster := "/prefix/1000"
self := "/1000/1"
tests := []struct {
nodes []*client.Node
werr error
wsize int
}{
{
// self is in the size range
[]*client.Node{
{Key: "/1000/_config/size", Value: "3", CreatedIndex: 1},
{Key: "/1000/_config/"},
{Key: self, CreatedIndex: 2},
{Key: "/1000/2", CreatedIndex: 3},
{Key: "/1000/3", CreatedIndex: 4},
{Key: "/1000/4", CreatedIndex: 5},
},
nil,
3,
},
{
// self is in the size range
[]*client.Node{
{Key: "/1000/_config/size", Value: "3", CreatedIndex: 1},
{Key: "/1000/_config/"},
{Key: "/1000/2", CreatedIndex: 2},
{Key: "/1000/3", CreatedIndex: 3},
{Key: self, CreatedIndex: 4},
{Key: "/1000/4", CreatedIndex: 5},
},
nil,
3,
},
{
// self is out of the size range
[]*client.Node{
{Key: "/1000/_config/size", Value: "3", CreatedIndex: 1},
{Key: "/1000/_config/"},
{Key: "/1000/2", CreatedIndex: 2},
{Key: "/1000/3", CreatedIndex: 3},
{Key: "/1000/4", CreatedIndex: 4},
{Key: self, CreatedIndex: 5},
},
ErrFullCluster,
3,
},
{
// self is not in the cluster
[]*client.Node{
{Key: "/1000/_config/size", Value: "3", CreatedIndex: 1},
{Key: "/1000/_config/"},
{Key: "/1000/2", CreatedIndex: 2},
{Key: "/1000/3", CreatedIndex: 3},
},
nil,
3,
},
{
[]*client.Node{
{Key: "/1000/_config/size", Value: "3", CreatedIndex: 1},
{Key: "/1000/_config/"},
{Key: "/1000/2", CreatedIndex: 2},
{Key: "/1000/3", CreatedIndex: 3},
{Key: "/1000/4", CreatedIndex: 4},
},
ErrFullCluster,
3,
},
{
// bad size key
[]*client.Node{
{Key: "/1000/_config/size", Value: "bad", CreatedIndex: 1},
},
ErrBadSizeKey,
0,
},
{
// no size key
[]*client.Node{},
ErrSizeNotFound,
0,
},
}
for i, tt := range tests {
rs := make([]*client.Response, 0)
if len(tt.nodes) > 0 {
rs = append(rs, &client.Response{Node: tt.nodes[0]})
rs = append(rs, &client.Response{
Node: &client.Node{
Key: cluster,
Nodes: tt.nodes[1:],
},
})
}
c := &clientWithResp{rs: rs}
d := discovery{cluster: cluster, id: 1, c: c}
cRetry := &clientWithRetry{failTimes: 3}
cRetry.rs = rs
fc := clockwork.NewFakeClock()
dRetry := discovery{cluster: cluster, id: 1, c: cRetry, clock: fc}
for _, d := range []discovery{d, dRetry} {
go func() {
for i := uint(1); i <= nRetries; i++ {
fc.BlockUntil(1)
fc.Advance(time.Second * (0x1 << i))
}
}()
ns, size, err := d.checkCluster()
if err != tt.werr {
t.Errorf("#%d: err = %v, want %v", i, err, tt.werr)
}
if reflect.DeepEqual(ns, tt.nodes) {
t.Errorf("#%d: nodes = %v, want %v", i, ns, tt.nodes)
}
if size != tt.wsize {
t.Errorf("#%d: size = %v, want %d", i, size, tt.wsize)
}
}
}
}
func TestWaitNodes(t *testing.T) {
all := client.Nodes{
0: {Key: "/1000/1", CreatedIndex: 2},
1: {Key: "/1000/2", CreatedIndex: 3},
2: {Key: "/1000/3", CreatedIndex: 4},
}
tests := []struct {
nodes client.Nodes
rs []*client.Response
}{
{
all,
[]*client.Response{},
},
{
all[:1],
[]*client.Response{
{Node: &client.Node{Key: "/1000/2", CreatedIndex: 3}},
{Node: &client.Node{Key: "/1000/3", CreatedIndex: 4}},
},
},
{
all[:2],
[]*client.Response{
{Node: &client.Node{Key: "/1000/3", CreatedIndex: 4}},
},
},
{
append(all, &client.Node{Key: "/1000/4", CreatedIndex: 5}),
[]*client.Response{
{Node: &client.Node{Key: "/1000/3", CreatedIndex: 4}},
},
},
}
for i, tt := range tests {
// Basic case
c := &clientWithResp{nil, &watcherWithResp{tt.rs}}
d := &discovery{cluster: "1000", c: c}
// Retry case
retryScanResp := make([]*client.Response, 0)
if len(tt.nodes) > 0 {
retryScanResp = append(retryScanResp, &client.Response{
Node: &client.Node{
Key: "1000",
Value: strconv.Itoa(3),
},
})
retryScanResp = append(retryScanResp, &client.Response{
Node: &client.Node{
Nodes: tt.nodes,
},
})
}
cRetry := &clientWithResp{
rs: retryScanResp,
w: &watcherWithRetry{rs: tt.rs, failTimes: 2},
}
fc := clockwork.NewFakeClock()
dRetry := &discovery{
cluster: "1000",
c: cRetry,
clock: fc,
}
for _, d := range []*discovery{d, dRetry} {
go func() {
for i := uint(1); i <= nRetries; i++ {
fc.BlockUntil(1)
fc.Advance(time.Second * (0x1 << i))
}
}()
g, err := d.waitNodes(tt.nodes, 3)
if err != nil {
t.Errorf("#%d: err = %v, want %v", i, err, nil)
}
if !reflect.DeepEqual(g, all) {
t.Errorf("#%d: all = %v, want %v", i, g, all)
}
}
}
}
func TestCreateSelf(t *testing.T) {
rs := []*client.Response{{Node: &client.Node{Key: "1000/1", CreatedIndex: 2}}}
w := &watcherWithResp{rs}
errw := &watcherWithErr{errors.New("watch err")}
c := &clientWithResp{rs, w}
errc := &clientWithErr{errors.New("create err"), w}
errwc := &clientWithResp{rs, errw}
tests := []struct {
c client.KeysAPI
werr error
}{
// no error
{c, nil},
// client.create returns an error
{errc, errc.err},
// watcher.next retuens an error
{errwc, errw.err},
}
for i, tt := range tests {
d := discovery{cluster: "1000", c: tt.c}
if err := d.createSelf(); err != tt.werr {
t.Errorf("#%d: err = %v, want %v", i, err, nil)
}
}
}
func TestNodesToCluster(t *testing.T) {
nodes := client.Nodes{
0: {Key: "/1000/1", Value: "1=1.1.1.1", CreatedIndex: 1},
1: {Key: "/1000/2", Value: "2=2.2.2.2", CreatedIndex: 2},
2: {Key: "/1000/3", Value: "3=3.3.3.3", CreatedIndex: 3},
}
w := "1=1.1.1.1,2=2.2.2.2,3=3.3.3.3"
cluster := nodesToCluster(nodes)
if !reflect.DeepEqual(cluster, w) {
t.Errorf("cluster = %v, want %v", cluster, w)
}
}
func TestSortableNodes(t *testing.T) {
ns := client.Nodes{
0: {CreatedIndex: 5},
1: {CreatedIndex: 1},
2: {CreatedIndex: 3},
3: {CreatedIndex: 4},
}
// add some randomness
for i := 0; i < 10000; i++ {
ns = append(ns, &client.Node{CreatedIndex: uint64(rand.Int31())})
}
sns := sortableNodes{ns}
sort.Sort(sns)
cis := make([]int, 0)
for _, n := range sns.Nodes {
cis = append(cis, int(n.CreatedIndex))
}
if sort.IntsAreSorted(cis) != true {
t.Errorf("isSorted = %v, want %v", sort.IntsAreSorted(cis), true)
}
cis = make([]int, 0)
for _, n := range ns {
cis = append(cis, int(n.CreatedIndex))
}
if sort.IntsAreSorted(cis) != true {
t.Errorf("isSorted = %v, want %v", sort.IntsAreSorted(cis), true)
}
}
func TestRetryFailure(t *testing.T) {
cluster := "1000"
c := &clientWithRetry{failTimes: 4}
fc := clockwork.NewFakeClock()
d := discovery{
cluster: cluster,
id: 1,
c: c,
clock: fc,
}
go func() {
for i := uint(1); i <= nRetries; i++ {
fc.BlockUntil(1)
fc.Advance(time.Second * (0x1 << i))
}
}()
if _, _, err := d.checkCluster(); err != ErrTooManyRetries {
t.Errorf("err = %v, want %v", err, ErrTooManyRetries)
}
}
type clientWithResp struct {
rs []*client.Response
w client.Watcher
}
func (c *clientWithResp) Create(key string, value string, ttl time.Duration) (*client.Response, error) {
if len(c.rs) == 0 {
return &client.Response{}, nil
}
r := c.rs[0]
c.rs = c.rs[1:]
return r, nil
}
func (c *clientWithResp) Get(key string) (*client.Response, error) {
if len(c.rs) == 0 {
return &client.Response{}, client.ErrKeyNoExist
}
r := c.rs[0]
c.rs = append(c.rs[1:], r)
return r, nil
}
func (c *clientWithResp) Watch(key string, waitIndex uint64) client.Watcher {
return c.w
}
func (c *clientWithResp) RecursiveWatch(key string, waitIndex uint64) client.Watcher {
return c.w
}
type clientWithErr struct {
err error
w client.Watcher
}
func (c *clientWithErr) Create(key string, value string, ttl time.Duration) (*client.Response, error) {
return &client.Response{}, c.err
}
func (c *clientWithErr) Get(key string) (*client.Response, error) {
return &client.Response{}, c.err
}
func (c *clientWithErr) Watch(key string, waitIndex uint64) client.Watcher {
return c.w
}
func (c *clientWithErr) RecursiveWatch(key string, waitIndex uint64) client.Watcher {
return c.w
}
type watcherWithResp struct {
rs []*client.Response
}
func (w *watcherWithResp) Next() (*client.Response, error) {
if len(w.rs) == 0 {
return &client.Response{}, nil
}
r := w.rs[0]
w.rs = w.rs[1:]
return r, nil
}
type watcherWithErr struct {
err error
}
func (w *watcherWithErr) Next() (*client.Response, error) {
return &client.Response{}, w.err
}
// clientWithRetry will timeout all requests up to failTimes
type clientWithRetry struct {
clientWithResp
failCount int
failTimes int
}
func (c *clientWithRetry) Create(key string, value string, ttl time.Duration) (*client.Response, error) {
if c.failCount < c.failTimes {
c.failCount++
return nil, client.ErrTimeout
}
return c.clientWithResp.Create(key, value, ttl)
}
func (c *clientWithRetry) Get(key string) (*client.Response, error) {
if c.failCount < c.failTimes {
c.failCount++
return nil, client.ErrTimeout
}
return c.clientWithResp.Get(key)
}
// watcherWithRetry will timeout all requests up to failTimes
type watcherWithRetry struct {
rs []*client.Response
failCount int
failTimes int
}
func (w *watcherWithRetry) Next() (*client.Response, error) {
if w.failCount < w.failTimes {
w.failCount++
return nil, client.ErrTimeout
}
if len(w.rs) == 0 {
return &client.Response{}, nil
}
r := w.rs[0]
w.rs = w.rs[1:]
return r, nil
}

View File

@@ -1,22 +0,0 @@
/*
Copyright 2014 CoreOS, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
Package discovery provides an implementation of the cluster discovery that
is used by etcd.
*/
package discovery

View File

@@ -1,17 +1,17 @@
/*
Copyright 2014 CoreOS, Inc.
Copyright 2013 CoreOS Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package error
@@ -46,7 +46,6 @@ var errors = map[int]string{
EcodeIndexOrValueRequired: "Index or value is required",
EcodeIndexValueMutex: "Index and value cannot both be specified",
EcodeInvalidField: "Invalid field",
EcodeInvalidForm: "Invalid POST form",
// raft related errors
EcodeRaftInternal: "Raft Internal Error",
@@ -85,7 +84,6 @@ const (
EcodeIndexOrValueRequired = 207
EcodeIndexValueMutex = 208
EcodeInvalidField = 209
EcodeInvalidForm = 210
EcodeRaftInternal = 300
EcodeLeaderElect = 301
@@ -106,10 +104,6 @@ type Error struct {
Index uint64 `json:"index"`
}
func NewRequestError(errorCode int, cause string) *Error {
return NewError(errorCode, cause, 0)
}
func NewError(errorCode int, cause string, index uint64) *Error {
return &Error{
ErrorCode: errorCode,
@@ -133,7 +127,7 @@ func (e Error) toJsonString() string {
return string(b)
}
func (e Error) WriteTo(w http.ResponseWriter) {
func (e Error) Write(w http.ResponseWriter) {
w.Header().Add("X-Etcd-Index", fmt.Sprint(e.Index))
// 3xx is raft internal error
status := http.StatusBadRequest

426
etcd/etcd.go Normal file
View File

@@ -0,0 +1,426 @@
/*
Copyright 2013 CoreOS Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package etcd
import (
"net/http"
"os"
"path/filepath"
"runtime"
"strings"
"sync"
"time"
goetcd "github.com/coreos/etcd/third_party/github.com/coreos/go-etcd/etcd"
golog "github.com/coreos/etcd/third_party/github.com/coreos/go-log/log"
"github.com/coreos/etcd/third_party/github.com/goraft/raft"
httpclient "github.com/coreos/etcd/third_party/github.com/mreiferson/go-httpclient"
"github.com/coreos/etcd/config"
ehttp "github.com/coreos/etcd/http"
"github.com/coreos/etcd/log"
"github.com/coreos/etcd/metrics"
"github.com/coreos/etcd/server"
"github.com/coreos/etcd/store"
)
// TODO(yichengq): constant extraTimeout is a hack.
// Current problem is that there is big lag between join command
// execution and join success.
// Fix it later. It should be removed when proper method is found and
// enough tests are provided. It is expected to be calculated from
// heartbeatInterval and electionTimeout only.
const extraTimeout = time.Duration(1000) * time.Millisecond
type Etcd struct {
Config *config.Config // etcd config
Store store.Store // data store
Registry *server.Registry // stores URL information for nodes
Server *server.Server // http server, runs on 4001 by default
PeerServer *server.PeerServer // peer server, runs on 7001 by default
StandbyServer *server.StandbyServer
server *http.Server
peerServer *http.Server
mode Mode
modeMutex sync.Mutex
closeChan chan bool
readyNotify chan bool // To signal when server is ready to accept connections
onceReady sync.Once
stopNotify chan bool // To signal when server is stopped totally
}
// New returns a new Etcd instance.
func New(c *config.Config) *Etcd {
if c == nil {
c = config.New()
}
return &Etcd{
Config: c,
closeChan: make(chan bool),
readyNotify: make(chan bool),
stopNotify: make(chan bool),
}
}
// Run the etcd instance.
func (e *Etcd) Run() {
// Sanitize all the input fields.
if err := e.Config.Sanitize(); err != nil {
log.Fatalf("failed sanitizing configuration: %v", err)
}
// Force remove server configuration if specified.
if e.Config.Force {
e.Config.Reset()
}
// Enable options.
if e.Config.VeryVeryVerbose {
log.Verbose = true
raft.SetLogLevel(raft.Trace)
goetcd.SetLogger(
golog.New(
"go-etcd",
false,
golog.CombinedSink(
os.Stdout,
"[%s] %s %-9s | %s\n",
[]string{"prefix", "time", "priority", "message"},
),
),
)
} else if e.Config.VeryVerbose {
log.Verbose = true
raft.SetLogLevel(raft.Debug)
} else if e.Config.Verbose {
log.Verbose = true
}
if e.Config.CPUProfileFile != "" {
profile(e.Config.CPUProfileFile)
}
if e.Config.DataDir == "" {
log.Fatal("The data dir was not set and could not be guessed from machine name")
}
// Create data directory if it doesn't already exist.
if err := os.MkdirAll(e.Config.DataDir, 0744); err != nil {
log.Fatalf("Unable to create path: %s", err)
}
// Warn people if they have an info file
info := filepath.Join(e.Config.DataDir, "info")
if _, err := os.Stat(info); err == nil {
log.Warnf("All cached configuration is now ignored. The file %s can be removed.", info)
}
var mbName string
if e.Config.Trace() {
mbName = e.Config.MetricsBucketName()
runtime.SetBlockProfileRate(1)
}
mb := metrics.NewBucket(mbName)
if e.Config.GraphiteHost != "" {
err := mb.Publish(e.Config.GraphiteHost)
if err != nil {
panic(err)
}
}
// Retrieve CORS configuration
corsInfo, err := ehttp.NewCORSInfo(e.Config.CorsOrigins)
if err != nil {
log.Fatal("CORS:", err)
}
// Create etcd key-value store and registry.
e.Store = store.New()
e.Registry = server.NewRegistry(e.Store)
// Create stats objects
followersStats := server.NewRaftFollowersStats(e.Config.Name)
serverStats := server.NewRaftServerStats(e.Config.Name)
// Calculate all of our timeouts
heartbeatInterval := time.Duration(e.Config.Peer.HeartbeatInterval) * time.Millisecond
electionTimeout := time.Duration(e.Config.Peer.ElectionTimeout) * time.Millisecond
dialTimeout := (3 * heartbeatInterval) + electionTimeout
responseHeaderTimeout := (3 * heartbeatInterval) + electionTimeout
clientTransporter := &httpclient.Transport{
ResponseHeaderTimeout: responseHeaderTimeout + extraTimeout,
// This is a workaround for Transport.CancelRequest doesn't work on
// HTTPS connections blocked. The patch for it is in progress,
// and would be available in Go1.3
// More: https://codereview.appspot.com/69280043/
ConnectTimeout: dialTimeout + extraTimeout,
RequestTimeout: responseHeaderTimeout + dialTimeout + 2*extraTimeout,
}
if e.Config.PeerTLSInfo().Scheme() == "https" {
clientTLSConfig, err := e.Config.PeerTLSInfo().ClientConfig()
if err != nil {
log.Fatal("client TLS error: ", err)
}
clientTransporter.TLSClientConfig = clientTLSConfig
clientTransporter.DisableCompression = true
}
client := server.NewClient(clientTransporter)
// Create peer server
psConfig := server.PeerServerConfig{
Name: e.Config.Name,
Scheme: e.Config.PeerTLSInfo().Scheme(),
URL: e.Config.Peer.Addr,
SnapshotCount: e.Config.SnapshotCount,
RetryTimes: e.Config.MaxRetryAttempts,
RetryInterval: e.Config.RetryInterval,
}
e.PeerServer = server.NewPeerServer(psConfig, client, e.Registry, e.Store, &mb, followersStats, serverStats)
// Create raft transporter and server
raftTransporter := server.NewTransporter(followersStats, serverStats, e.Registry, heartbeatInterval, dialTimeout, responseHeaderTimeout)
if e.Config.PeerTLSInfo().Scheme() == "https" {
raftClientTLSConfig, err := e.Config.PeerTLSInfo().ClientConfig()
if err != nil {
log.Fatal("raft client TLS error: ", err)
}
raftTransporter.SetTLSConfig(*raftClientTLSConfig)
}
raftServer, err := raft.NewServer(e.Config.Name, e.Config.DataDir, raftTransporter, e.Store, e.PeerServer, "")
if err != nil {
log.Fatal(err)
}
raftServer.SetElectionTimeout(electionTimeout)
raftServer.SetHeartbeatInterval(heartbeatInterval)
e.PeerServer.SetRaftServer(raftServer, e.Config.Snapshot)
// Create etcd server
e.Server = server.New(e.Config.Name, e.Config.Addr, e.PeerServer, e.Registry, e.Store, &mb)
if e.Config.Trace() {
e.Server.EnableTracing()
}
e.PeerServer.SetServer(e.Server)
// Create standby server
ssConfig := server.StandbyServerConfig{
Name: e.Config.Name,
PeerScheme: e.Config.PeerTLSInfo().Scheme(),
PeerURL: e.Config.Peer.Addr,
ClientURL: e.Config.Addr,
DataDir: e.Config.DataDir,
}
e.StandbyServer = server.NewStandbyServer(ssConfig, client)
e.StandbyServer.SetRaftServer(raftServer)
// Generating config could be slow.
// Put it here to make listen happen immediately after peer-server starting.
peerTLSConfig := server.TLSServerConfig(e.Config.PeerTLSInfo())
etcdTLSConfig := server.TLSServerConfig(e.Config.EtcdTLSInfo())
if !e.StandbyServer.IsRunning() {
startPeerServer, possiblePeers, err := e.PeerServer.FindCluster(e.Config.Discovery, e.Config.Peers)
if err != nil {
log.Fatal(err)
}
if startPeerServer {
e.setMode(PeerMode)
} else {
e.StandbyServer.SyncCluster(possiblePeers)
e.setMode(StandbyMode)
}
} else {
e.setMode(StandbyMode)
}
serverHTTPHandler := &ehttp.CORSHandler{e.Server.HTTPHandler(), corsInfo}
peerServerHTTPHandler := &ehttp.CORSHandler{e.PeerServer.HTTPHandler(), corsInfo}
standbyServerHTTPHandler := &ehttp.CORSHandler{e.StandbyServer.ClientHTTPHandler(), corsInfo}
log.Infof("etcd server [name %s, listen on %s, advertised url %s]", e.Server.Name, e.Config.BindAddr, e.Server.URL())
listener := server.NewListener(e.Config.EtcdTLSInfo().Scheme(), e.Config.BindAddr, etcdTLSConfig)
e.server = &http.Server{Handler: &ModeHandler{e, serverHTTPHandler, standbyServerHTTPHandler},
ReadTimeout: time.Duration(e.Config.HTTPReadTimeout) * time.Second,
WriteTimeout: time.Duration(e.Config.HTTPWriteTimeout) * time.Second,
}
log.Infof("peer server [name %s, listen on %s, advertised url %s]", e.PeerServer.Config.Name, e.Config.Peer.BindAddr, e.PeerServer.Config.URL)
peerListener := server.NewListener(e.Config.PeerTLSInfo().Scheme(), e.Config.Peer.BindAddr, peerTLSConfig)
e.peerServer = &http.Server{Handler: &ModeHandler{e, peerServerHTTPHandler, http.NotFoundHandler()},
ReadTimeout: time.Duration(server.DefaultReadTimeout) * time.Second,
WriteTimeout: time.Duration(server.DefaultWriteTimeout) * time.Second,
}
wg := sync.WaitGroup{}
wg.Add(2)
go func() {
<-e.readyNotify
defer wg.Done()
if err := e.server.Serve(listener); err != nil {
if !isListenerClosing(err) {
log.Fatal(err)
}
}
}()
go func() {
<-e.readyNotify
defer wg.Done()
if err := e.peerServer.Serve(peerListener); err != nil {
if !isListenerClosing(err) {
log.Fatal(err)
}
}
}()
e.runServer()
listener.Close()
peerListener.Close()
wg.Wait()
log.Infof("etcd instance is stopped [name %s]", e.Config.Name)
close(e.stopNotify)
}
func (e *Etcd) runServer() {
var removeNotify <-chan bool
for {
if e.mode == PeerMode {
log.Infof("%v starting in peer mode", e.Config.Name)
go registerAvailableInternalVersions(e.Config.Name, e.Config.Addr, e.Config.EtcdTLSInfo())
// Starting peer server should be followed close by listening on its port
// If not, it may leave many requests unaccepted, or cannot receive heartbeat from the cluster.
// One severe problem caused if failing receiving heartbeats is when the second node joins one-node cluster,
// the cluster could be out of work as long as the two nodes cannot transfer messages.
e.PeerServer.Start(e.Config.Snapshot, e.Config.ClusterConfig())
removeNotify = e.PeerServer.RemoveNotify()
} else {
log.Infof("%v starting in standby mode", e.Config.Name)
e.StandbyServer.Start()
removeNotify = e.StandbyServer.RemoveNotify()
}
// etcd server is ready to accept connections, notify waiters.
e.onceReady.Do(func() { close(e.readyNotify) })
select {
case <-e.closeChan:
e.PeerServer.Stop()
e.StandbyServer.Stop()
return
case <-removeNotify:
}
if e.mode == PeerMode {
peerURLs := e.Registry.PeerURLs(e.PeerServer.RaftServer().Leader(), e.Config.Name)
e.StandbyServer.SyncCluster(peerURLs)
e.setMode(StandbyMode)
} else {
// Create etcd key-value store and registry.
e.Store = store.New()
e.Registry = server.NewRegistry(e.Store)
e.PeerServer.SetStore(e.Store)
e.PeerServer.SetRegistry(e.Registry)
e.Server.SetStore(e.Store)
e.Server.SetRegistry(e.Registry)
// Generate new peer server here.
// TODO(yichengq): raft server cannot be started after stopped.
// It should be removed when raft restart is implemented.
heartbeatInterval := time.Duration(e.Config.Peer.HeartbeatInterval) * time.Millisecond
electionTimeout := time.Duration(e.Config.Peer.ElectionTimeout) * time.Millisecond
raftServer, err := raft.NewServer(e.Config.Name, e.Config.DataDir, e.PeerServer.RaftServer().Transporter(), e.Store, e.PeerServer, "")
if err != nil {
log.Fatal(err)
}
raftServer.SetElectionTimeout(electionTimeout)
raftServer.SetHeartbeatInterval(heartbeatInterval)
e.PeerServer.SetRaftServer(raftServer, e.Config.Snapshot)
e.StandbyServer.SetRaftServer(raftServer)
e.PeerServer.SetJoinIndex(e.StandbyServer.JoinIndex())
e.setMode(PeerMode)
}
}
}
// Stop the etcd instance.
func (e *Etcd) Stop() {
close(e.closeChan)
<-e.stopNotify
}
// ReadyNotify returns a channel that is going to be closed
// when the etcd instance is ready to accept connections.
func (e *Etcd) ReadyNotify() <-chan bool {
return e.readyNotify
}
func (e *Etcd) Mode() Mode {
e.modeMutex.Lock()
defer e.modeMutex.Unlock()
return e.mode
}
func (e *Etcd) setMode(m Mode) {
e.modeMutex.Lock()
defer e.modeMutex.Unlock()
e.mode = m
}
func isListenerClosing(err error) bool {
// An error string equivalent to net.errClosing for using with
// http.Serve() during server shutdown. Need to re-declare
// here because it is not exported by "net" package.
const errClosing = "use of closed network connection"
return strings.Contains(err.Error(), errClosing)
}
type ModeGetter interface {
Mode() Mode
}
type ModeHandler struct {
ModeGetter
PeerModeHandler http.Handler
StandbyModeHandler http.Handler
}
func (h *ModeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
switch h.Mode() {
case PeerMode:
h.PeerModeHandler.ServeHTTP(w, r)
case StandbyMode:
h.StandbyModeHandler.ServeHTTP(w, r)
}
}
type Mode int
const (
PeerMode Mode = iota
StandbyMode
)

41
etcd/etcd_test.go Normal file
View File

@@ -0,0 +1,41 @@
/*
Copyright 2013 CoreOS Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package etcd
import (
"io/ioutil"
"os"
"testing"
"github.com/coreos/etcd/config"
)
func TestRunStop(t *testing.T) {
path, _ := ioutil.TempDir("", "etcd-")
defer os.RemoveAll(path)
config := config.New()
config.Name = "ETCDTEST"
config.DataDir = path
config.Addr = "localhost:0"
config.Peer.Addr = "localhost:0"
etcd := New(config)
go etcd.Run()
<-etcd.ReadyNotify()
etcd.Stop()
}

27
etcd/profile.go Normal file
View File

@@ -0,0 +1,27 @@
package etcd
import (
"os"
"os/signal"
"runtime/pprof"
"github.com/coreos/etcd/log"
)
// profile starts CPU profiling.
func profile(path string) {
f, err := os.Create(path)
if err != nil {
log.Fatal(err)
}
pprof.StartCPUProfile(f)
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt)
go func() {
sig := <-c
log.Infof("captured %v, stopping profiler and exiting..", sig)
pprof.StopCPUProfile()
os.Exit(1)
}()
}

59
etcd/upgrade.go Normal file
View File

@@ -0,0 +1,59 @@
package etcd
import (
"fmt"
"os"
"runtime"
"time"
"github.com/coreos/etcd/log"
"github.com/coreos/etcd/server"
"github.com/coreos/etcd/third_party/github.com/coreos/go-etcd/etcd"
)
var defaultEtcdBinaryDir = "/usr/libexec/etcd/internal_versions/"
func registerAvailableInternalVersions(name string, addr string, tls *server.TLSInfo) {
var c *etcd.Client
if tls.Scheme() == "http" {
c = etcd.NewClient([]string{addr})
} else {
var err error
c, err = etcd.NewTLSClient([]string{addr}, tls.CertFile, tls.KeyFile, tls.CAFile)
if err != nil {
log.Fatalf("client TLS error: %v", err)
}
}
vers, err := getInternalVersions()
if err != nil {
log.Infof("failed to get local etcd versions: %v", err)
return
}
for _, v := range vers {
for {
_, err := c.Set("/_etcd/available-internal-versions/"+v+"/"+name, "ok", 0)
if err == nil {
break
}
time.Sleep(time.Second)
}
}
log.Infof("%s: available_internal_versions %s is registered into key space successfully.", name, vers)
}
func getInternalVersions() ([]string, error) {
if runtime.GOOS != "linux" {
return nil, fmt.Errorf("unmatched os version %v", runtime.GOOS)
}
etcdBinaryDir := os.Getenv("ETCD_BINARY_DIR")
if etcdBinaryDir == "" {
etcdBinaryDir = defaultEtcdBinaryDir
}
dir, err := os.Open(etcdBinaryDir)
if err != nil {
return nil, err
}
defer dir.Close()
return dir.Readdirnames(-1)
}

View File

@@ -1,267 +0,0 @@
etcdctl
========
`etcdctl` is a command line client for [etcd][etcd].
It can be used in scripts or for administrators to explore an etcd cluster.
[etcd]: https://github.com/coreos/etcd
## Getting etcdctl
The latest release is available as a binary at [Github][github-release] along with etcd.
[github-release]: https://github.com/coreos/etcd/releases/
You can also build etcdctl from source using the build script found in the parent directory.
## Usage
### Setting Key Values
Set a value on the `/foo/bar` key:
```
$ etcdctl set /foo/bar "Hello world"
Hello world
```
Set a value on the `/foo/bar` key with a value that expires in 60 seconds:
```
$ etcdctl set /foo/bar "Hello world" --ttl 60
Hello world
```
Conditionally set a value on `/foo/bar` if the previous value was "Hello world":
```
$ etcdctl set /foo/bar "Goodbye world" --swap-with-value "Hello world"
Goodbye world
```
Conditionally set a value on `/foo/bar` if the previous etcd index was 12:
```
$ etcdctl set /foo/bar "Goodbye world" --swap-with-index 12
Goodbye world
```
Create a new key `/foo/bar`, only if the key did not previously exist:
```
$ etcdctl mk /foo/new_bar "Hello world"
Hello world
```
Create a new dir `/fooDir`, only if the key did not previously exist:
```
$ etcdctl mkdir /fooDir
```
Update an existing key `/foo/bar`, only if the key already existed:
```
$ etcdctl update /foo/bar "Hola mundo"
Hola mundo
```
Create or update a directory called `/mydir`:
```
$ etcdctl setDir /mydir
```
### Retrieving a key value
Get the current value for a single key in the local etcd node:
```
$ etcdctl get /foo/bar
Hello world
```
Get the current value for a key within the cluster:
```
$ etcdctl get /foo/bar --consistent
Hello world
```
Get the value of a key with additional metadata in a parseable format:
```
$ etcdctl -o extended get /foo/bar
Key: /foo/bar
Modified-Index: 72
TTL: 0
Etcd-Index: 72
Raft-Index: 5611
Raft-Term: 1
Hello World
```
### Listing a directory
Explore the keyspace using the `ls` command
```
$ etcdctl ls
/akey
/adir
$ etcdctl ls /adir
/adir/key1
/adir/key2
```
Add `--recursive` to recursively list subdirectories encountered.
```
$ etcdctl ls --recursive
/akey
/adir
/adir/key1
/adir/key2
```
Directories can also have a trailing `/` added to output using `-p`.
```
$ etcdctl ls -p
/akey
/adir/
```
### Deleting a key
Delete a key:
```
$ etcdctl rm /foo/bar
```
Delete an empty directory or a key-value pair
```
$ etcdctl rmdir /path/to/dir
```
or
```
$ etcdctl rm /path/to/dir --dir
```
Recursively delete a key and all child keys:
```
$ etcdctl rm /path/to/dir --recursive
```
Conditionally delete `/foo/bar` if the previous value was "Hello world":
```
$ etcdctl rm /foo/bar --with-value "Hello world"
```
Conditionally delete `/foo/bar` if the previous etcd index was 12:
```
$ etcdctl rm /foo/bar --with-index 12
```
### Watching for changes
Watch for only the next change on a key:
```
$ etcdctl watch /foo/bar
Hello world
```
Continuously watch a key:
```
$ etcdctl watch /foo/bar --forever
Hello world
.... client hangs forever until ctrl+C printing values as key change
```
Continuously watch a key, starting with a given etcd index:
```
$ etcdctl watch /foo/bar --forever --index 12
Hello world
.... client hangs forever until ctrl+C printing values as key change
```
Continuously watch a key and exec a program:
```
$ etcdctl exec-watch /foo/bar -- sh -c "env | grep ETCD"
ETCD_WATCH_ACTION=set
ETCD_VALUE=My configuration stuff
ETCD_MODIFIED_INDEX=1999
ETCD_KEY=/foo/bar
ETCD_WATCH_ACTION=set
ETCD_VALUE=My new configuration stuff
ETCD_MODIFIED_INDEX=2000
ETCD_KEY=/foo/bar
```
Continuously and recursively watch a key and exec a program:
```
$ etcdctl exec-watch --recursive /foo -- sh -c "env | grep ETCD"
ETCD_WATCH_ACTION=set
ETCD_VALUE=My configuration stuff
ETCD_MODIFIED_INDEX=1999
ETCD_KEY=/foo/bar
ETCD_WATCH_ACTION=set
ETCD_VALUE=My new configuration stuff
ETCD_MODIFIED_INDEX=2000
ETCD_KEY=/foo/barbar
```
## Return Codes
The following exit codes can be returned from etcdctl:
```
0 Success
1 Malformed etcdctl arguments
2 Failed to connect to host
3 Failed to auth (client cert rejected, ca validation failure, etc)
4 400 error from etcd
5 500 error from etcd
```
## Peers
If your etcd cluster isn't available on `http://127.0.0.1:4001` you can specify
a `--peers` flag or `ETCDCTL_PEERS` environment variable. You can list one peer,
or a comma-separated list of peers.
```
ETCDCTL_PEERS="http://10.0.28.1:4002" etcdctl set my-key to-a-value
ETCDCTL_PEERS="http://10.0.28.1:4002,http://10.0.28.2:4002,http://10.0.28.3:4002" etcdctl set my-key to-a-value
etcdctl --peers http://10.0.28.1:4002 my-key to-a-value
etcdctl --peers http://10.0.28.1:4002,http://10.0.28.2:4002,http://10.0.28.3:4002 etcdctl set my-key to-a-value
```
## Project Details
### Versioning
etcdctl uses [semantic versioning][semver].
Releases will follow lockstep with the etcd release cycle.
[semver]: http://semver.org/
### License
etcdctl is under the Apache 2.0 license. See the [LICENSE][license] file for details.
[license]: https://github.com/coreos/etcdctl/blob/master/LICENSE

View File

@@ -1,19 +0,0 @@
package command
import (
"fmt"
"os"
)
const (
SUCCESS = iota
MalformedEtcdctlArguments
FailedToConnectToHost
FailedToAuth
ErrorFromEtcd
)
func handleError(code int, err error) {
fmt.Fprintln(os.Stderr, "Error: ", err)
os.Exit(code)
}

View File

@@ -1,102 +0,0 @@
package command
import (
"errors"
"fmt"
"io"
"os"
"os/exec"
"os/signal"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/go-etcd/etcd"
)
// NewExecWatchCommand returns the CLI command for "exec-watch".
func NewExecWatchCommand() cli.Command {
return cli.Command{
Name: "exec-watch",
Usage: "watch a key for changes and exec an executable",
Flags: []cli.Flag{
cli.IntFlag{Name: "after-index", Value: 0, Usage: "watch after the given index"},
cli.BoolFlag{Name: "recursive", Usage: "watch all values for key and child keys"},
},
Action: func(c *cli.Context) {
handleKey(c, execWatchCommandFunc)
},
}
}
// execWatchCommandFunc executes the "exec-watch" command.
func execWatchCommandFunc(c *cli.Context, client *etcd.Client) (*etcd.Response, error) {
_ = io.Copy
_ = exec.Command
args := c.Args()
argsLen := len(args)
if argsLen < 2 {
return nil, errors.New("Key and command to exec required")
}
key := args[argsLen-1]
cmdArgs := args[:argsLen-1]
index := 0
if c.Int("after-index") != 0 {
index = c.Int("after-index") + 1
key = args[0]
cmdArgs = args[2:]
}
recursive := c.Bool("recursive")
if recursive != false {
key = args[0]
cmdArgs = args[2:]
}
sigch := make(chan os.Signal, 1)
signal.Notify(sigch, os.Interrupt)
stop := make(chan bool)
go func() {
<-sigch
stop <- true
os.Exit(0)
}()
receiver := make(chan *etcd.Response)
go client.Watch(key, uint64(index), recursive, receiver, stop)
for {
resp := <-receiver
cmd := exec.Command(cmdArgs[0], cmdArgs[1:]...)
cmd.Env = environResponse(resp, os.Environ())
stdout, err := cmd.StdoutPipe()
if err != nil {
fmt.Fprintf(os.Stderr, err.Error())
os.Exit(1)
}
stderr, err := cmd.StderrPipe()
if err != nil {
fmt.Fprintf(os.Stderr, err.Error())
os.Exit(1)
}
err = cmd.Start()
if err != nil {
fmt.Fprintf(os.Stderr, err.Error())
os.Exit(1)
}
go io.Copy(os.Stdout, stdout)
go io.Copy(os.Stderr, stderr)
cmd.Wait()
}
}
func environResponse(resp *etcd.Response, env []string) []string {
env = append(env, "ETCD_WATCH_ACTION="+resp.Action)
env = append(env, "ETCD_WATCH_MODIFIED_INDEX="+fmt.Sprintf("%d", resp.Node.ModifiedIndex))
env = append(env, "ETCD_WATCH_KEY="+resp.Node.Key)
env = append(env, "ETCD_WATCH_VALUE="+resp.Node.Value)
return env
}

View File

@@ -1,60 +0,0 @@
package command
import (
"errors"
"fmt"
"os"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/go-etcd/etcd"
)
// NewGetCommand returns the CLI command for "get".
func NewGetCommand() cli.Command {
return cli.Command{
Name: "get",
Usage: "retrieve the value of a key",
Flags: []cli.Flag{
cli.BoolFlag{Name: "sort", Usage: "returns result in sorted order"},
cli.BoolFlag{Name: "consistent", Usage: "send request to the leader, thereby guranteeing that any earlier writes will be seen by the read"},
},
Action: func(c *cli.Context) {
handleGet(c, getCommandFunc)
},
}
}
// handleGet handles a request that intends to do get-like operations.
func handleGet(c *cli.Context, fn handlerFunc) {
handlePrint(c, fn, printGet)
}
// printGet writes error message when getting the value of a directory.
func printGet(resp *etcd.Response, format string) {
if resp.Node.Dir {
fmt.Fprintln(os.Stderr, fmt.Sprintf("%s: is a directory", resp.Node.Key))
os.Exit(1)
}
printKey(resp, format)
}
// getCommandFunc executes the "get" command.
func getCommandFunc(c *cli.Context, client *etcd.Client) (*etcd.Response, error) {
if len(c.Args()) == 0 {
return nil, errors.New("Key required")
}
key := c.Args()[0]
consistent := c.Bool("consistent")
sorted := c.Bool("sort")
// Setup consistency on the client.
if consistent {
client.SetConsistency(etcd.STRONG_CONSISTENCY)
} else {
client.SetConsistency(etcd.WEAK_CONSISTENCY)
}
// Retrieve the value from the server.
return client.Get(key, sorted, false)
}

View File

@@ -1,191 +0,0 @@
package command
import (
"encoding/json"
"errors"
"fmt"
"net/url"
"os"
"strings"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/go-etcd/etcd"
)
type handlerFunc func(*cli.Context, *etcd.Client) (*etcd.Response, error)
type printFunc func(*etcd.Response, string)
type contextualPrintFunc func(*cli.Context, *etcd.Response, string)
// dumpCURL blindly dumps all curl output to os.Stderr
func dumpCURL(client *etcd.Client) {
client.OpenCURL()
for {
fmt.Fprintf(os.Stderr, "Curl-Example: %s\n", client.RecvCURL())
}
}
// createHttpPath attaches http scheme to the given address if needed
func createHttpPath(addr string) (string, error) {
u, err := url.Parse(addr)
if err != nil {
return "", err
}
if u.Scheme == "" {
u.Scheme = "http"
}
return u.String(), nil
}
func getPeersFlagValue(c *cli.Context) []string {
peerstr := c.GlobalString("peers")
// Use an environment variable if nothing was supplied on the
// command line
if peerstr == "" {
peerstr = os.Getenv("ETCDCTL_PEERS")
}
// If we still don't have peers, use a default
if peerstr == "" {
peerstr = "127.0.0.1:4001"
}
return strings.Split(peerstr, ",")
}
// rawhandle wraps the command function handlers and sets up the
// environment but performs no output formatting.
func rawhandle(c *cli.Context, fn handlerFunc) (*etcd.Response, error) {
sync := !c.GlobalBool("no-sync")
peers := getPeersFlagValue(c)
// If no sync, create http path for each peer address
if !sync {
revisedPeers := make([]string, 0)
for _, peer := range peers {
if revisedPeer, err := createHttpPath(peer); err != nil {
fmt.Fprintf(os.Stderr, "Unsupported url %v: %v\n", peer, err)
} else {
revisedPeers = append(revisedPeers, revisedPeer)
}
}
peers = revisedPeers
}
client := etcd.NewClient(peers)
if c.GlobalBool("debug") {
go dumpCURL(client)
}
// Sync cluster.
if sync {
if ok := client.SyncCluster(); !ok {
handleError(FailedToConnectToHost, errors.New("Cannot sync with the cluster using peers "+strings.Join(peers, ", ")))
}
}
if c.GlobalBool("debug") {
fmt.Fprintf(os.Stderr, "Cluster-Peers: %s\n",
strings.Join(client.GetCluster(), " "))
}
// Execute handler function.
return fn(c, client)
}
// handlePrint wraps the command function handlers to parse global flags
// into a client and to properly format the response objects.
func handlePrint(c *cli.Context, fn handlerFunc, pFn printFunc) {
resp, err := rawhandle(c, fn)
// Print error and exit, if necessary.
if err != nil {
handleError(ErrorFromEtcd, err)
}
if resp != nil && pFn != nil {
pFn(resp, c.GlobalString("output"))
}
}
// Just like handlePrint but also passed the context of the command
func handleContextualPrint(c *cli.Context, fn handlerFunc, pFn contextualPrintFunc) {
resp, err := rawhandle(c, fn)
if err != nil {
handleError(ErrorFromEtcd, err)
}
if resp != nil && pFn != nil {
pFn(c, resp, c.GlobalString("output"))
}
}
// handleDir handles a request that wants to do operations on a single dir.
// Dir cannot be printed out, so we set NIL print function here.
func handleDir(c *cli.Context, fn handlerFunc) {
handlePrint(c, fn, nil)
}
// handleKey handles a request that wants to do operations on a single key.
func handleKey(c *cli.Context, fn handlerFunc) {
handlePrint(c, fn, printKey)
}
func handleAll(c *cli.Context, fn handlerFunc) {
handlePrint(c, fn, printAll)
}
// printKey writes the etcd response to STDOUT in the given format.
func printKey(resp *etcd.Response, format string) {
// printKey is only for keys, error on directories
if resp.Node.Dir == true {
fmt.Fprintln(os.Stderr, fmt.Sprintf("Cannot print key [%s: Is a directory]", resp.Node.Key))
os.Exit(1)
}
printKeyOnly(resp, format)
}
// printAll prints the etcd response in the given format in its best efforts.
func printAll(resp *etcd.Response, format string) {
if resp.Node.Dir == true {
return
}
printKeyOnly(resp, format)
}
// printKeyOnly only supports to print key correctly.
func printKeyOnly(resp *etcd.Response, format string) {
// Format the result.
switch format {
case "simple":
fmt.Println(resp.Node.Value)
case "extended":
// Extended prints in a rfc2822 style format
fmt.Println("Key:", resp.Node.Key)
fmt.Println("Created-Index:", resp.Node.CreatedIndex)
fmt.Println("Modified-Index:", resp.Node.ModifiedIndex)
if resp.PrevNode != nil {
fmt.Println("PrevNode.Value:", resp.PrevNode.Value)
}
fmt.Println("TTL:", resp.Node.TTL)
fmt.Println("Etcd-Index:", resp.EtcdIndex)
fmt.Println("Raft-Index:", resp.RaftIndex)
fmt.Println("Raft-Term:", resp.RaftTerm)
fmt.Println("")
fmt.Println(resp.Node.Value)
case "json":
b, err := json.Marshal(resp)
if err != nil {
panic(err)
}
fmt.Println(string(b))
default:
fmt.Fprintln(os.Stderr, "Unsupported output format:", format)
}
}

View File

@@ -1,66 +0,0 @@
package command
import (
"fmt"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/go-etcd/etcd"
)
func NewLsCommand() cli.Command {
return cli.Command{
Name: "ls",
Usage: "retrieve a directory",
Flags: []cli.Flag{
cli.BoolFlag{Name: "sort", Usage: "returns result in sorted order"},
cli.BoolFlag{Name: "recursive", Usage: "returns all values for key and child keys"},
cli.BoolFlag{Name: "p", Usage: "append slash (/) to directories"},
},
Action: func(c *cli.Context) {
handleLs(c, lsCommandFunc)
},
}
}
// handleLs handles a request that intends to do ls-like operations.
func handleLs(c *cli.Context, fn handlerFunc) {
handleContextualPrint(c, fn, printLs)
}
// printLs writes a response out in a manner similar to the `ls` command in unix.
// Non-empty directories list their contents and files list their name.
func printLs(c *cli.Context, resp *etcd.Response, format string) {
if !resp.Node.Dir {
fmt.Println(resp.Node.Key)
}
for _, node := range resp.Node.Nodes {
rPrint(c, node)
}
}
// lsCommandFunc executes the "ls" command.
func lsCommandFunc(c *cli.Context, client *etcd.Client) (*etcd.Response, error) {
key := "/"
if len(c.Args()) != 0 {
key = c.Args()[0]
}
recursive := c.Bool("recursive")
sort := c.Bool("sort")
// Retrieve the value from the server.
return client.Get(key, sort, recursive)
}
// rPrint recursively prints out the nodes in the node structure.
func rPrint(c *cli.Context, n *etcd.Node) {
if n.Dir && c.Bool("p") {
fmt.Println(fmt.Sprintf("%v/", n.Key))
} else {
fmt.Println(n.Key)
}
for _, node := range n.Nodes {
rPrint(c, node)
}
}

View File

@@ -1,129 +0,0 @@
package command
import (
"fmt"
"net/http"
"os"
"strings"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
"github.com/coreos/etcd/client"
)
func NewMemberCommand() cli.Command {
return cli.Command{
Name: "member",
Usage: "member add, remove and list subcommands",
Subcommands: []cli.Command{
cli.Command{
Name: "list",
Usage: "enumerate existing cluster members",
Action: actionMemberList,
},
cli.Command{
Name: "add",
Usage: "add a new member to the etcd cluster",
Action: actionMemberAdd,
},
cli.Command{
Name: "remove",
Usage: "remove an existing member from the etcd cluster",
Action: actionMemberRemove,
},
},
}
}
func mustNewMembersAPI(c *cli.Context) client.MembersAPI {
peers := getPeersFlagValue(c)
for i, p := range peers {
if !strings.HasPrefix(p, "http") && !strings.HasPrefix(p, "https") {
peers[i] = fmt.Sprintf("http://%s", p)
}
}
mAPI, err := client.NewMembersAPI(&http.Transport{}, peers, client.DefaultRequestTimeout)
if err != nil {
fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}
return mAPI
}
func actionMemberList(c *cli.Context) {
if len(c.Args()) != 0 {
fmt.Fprintln(os.Stderr, "No arguments accepted")
os.Exit(1)
}
mAPI := mustNewMembersAPI(c)
members, err := mAPI.List()
if err != nil {
fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}
for _, m := range members {
fmt.Printf("%s: name=%s peerURLs=%s clientURLs=%s\n", m.ID, m.Name, strings.Join(m.PeerURLs, ","), strings.Join(m.ClientURLs, ","))
}
}
func actionMemberAdd(c *cli.Context) {
args := c.Args()
if len(args) != 2 {
fmt.Fprintln(os.Stderr, "Provide a name and a single member peerURL")
os.Exit(1)
}
mAPI := mustNewMembersAPI(c)
url := args[1]
m, err := mAPI.Add(url)
if err != nil {
fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}
newID := m.ID
newName := args[0]
fmt.Printf("Added member named %s with ID %s to cluster\n", newName, newID)
members, err := mAPI.List()
if err != nil {
fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}
conf := []string{}
for _, m := range members {
for _, u := range m.PeerURLs {
n := m.Name
if m.ID == newID {
n = newName
}
conf = append(conf, fmt.Sprintf("%s=%s", n, u))
}
}
fmt.Print("\n")
fmt.Printf("ETCD_NAME=%q\n", newName)
fmt.Printf("ETCD_INITIAL_CLUSTER=%q\n", strings.Join(conf, ","))
fmt.Printf("ETCD_INITIAL_CLUSTER_STATE=\"existing\"\n")
}
func actionMemberRemove(c *cli.Context) {
args := c.Args()
if len(args) != 1 {
fmt.Fprintln(os.Stderr, "Provide a single member ID")
os.Exit(1)
}
mAPI := mustNewMembersAPI(c)
mID := args[0]
if err := mAPI.Remove(mID); err != nil {
fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}
fmt.Printf("Removed member %s from cluster\n", mID)
}

View File

@@ -1,39 +0,0 @@
package command
import (
"errors"
"os"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/go-etcd/etcd"
)
// NewMakeCommand returns the CLI command for "mk".
func NewMakeCommand() cli.Command {
return cli.Command{
Name: "mk",
Usage: "make a new key with a given value",
Flags: []cli.Flag{
cli.IntFlag{Name: "ttl", Value: 0, Usage: "key time-to-live"},
},
Action: func(c *cli.Context) {
handleKey(c, makeCommandFunc)
},
}
}
// makeCommandFunc executes the "make" command.
func makeCommandFunc(c *cli.Context, client *etcd.Client) (*etcd.Response, error) {
if len(c.Args()) == 0 {
return nil, errors.New("Key required")
}
key := c.Args()[0]
value, err := argOrStdin(c.Args(), os.Stdin, 1)
if err != nil {
return nil, errors.New("Value required")
}
ttl := c.Int("ttl")
return client.Create(key, value, uint64(ttl))
}

View File

@@ -1,33 +0,0 @@
package command
import (
"errors"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/go-etcd/etcd"
)
// NewMakeDirCommand returns the CLI command for "mkdir".
func NewMakeDirCommand() cli.Command {
return cli.Command{
Name: "mkdir",
Usage: "make a new directory",
Flags: []cli.Flag{
cli.IntFlag{Name: "ttl", Value: 0, Usage: "key time-to-live"},
},
Action: func(c *cli.Context) {
handleDir(c, makeDirCommandFunc)
},
}
}
// makeDirCommandFunc executes the "mkdir" command.
func makeDirCommandFunc(c *cli.Context, client *etcd.Client) (*etcd.Response, error) {
if len(c.Args()) == 0 {
return nil, errors.New("Key required")
}
key := c.Args()[0]
ttl := c.Int("ttl")
return client.CreateDir(key, uint64(ttl))
}

View File

@@ -1,50 +0,0 @@
package command
import (
"errors"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/go-etcd/etcd"
)
// NewRemoveCommand returns the CLI command for "rm".
func NewRemoveCommand() cli.Command {
return cli.Command{
Name: "rm",
Usage: "remove a key",
Flags: []cli.Flag{
cli.BoolFlag{Name: "dir", Usage: "removes the key if it is an empty directory or a key-value pair"},
cli.BoolFlag{Name: "recursive", Usage: "removes the key and all child keys(if it is a directory)"},
cli.StringFlag{Name: "with-value", Value: "", Usage: "previous value"},
cli.IntFlag{Name: "with-index", Value: 0, Usage: "previous index"},
},
Action: func(c *cli.Context) {
handleAll(c, removeCommandFunc)
},
}
}
// removeCommandFunc executes the "rm" command.
func removeCommandFunc(c *cli.Context, client *etcd.Client) (*etcd.Response, error) {
if len(c.Args()) == 0 {
return nil, errors.New("Key required")
}
key := c.Args()[0]
recursive := c.Bool("recursive")
dir := c.Bool("dir")
// TODO: distinguish with flag is not set and empty flag
// the cli pkg need to provide this feature
prevValue := c.String("with-value")
prevIndex := uint64(c.Int("with-index"))
if prevValue != "" || prevIndex != 0 {
return client.CompareAndDelete(key, prevValue, prevIndex)
}
if recursive || !dir {
return client.Delete(key, recursive)
}
return client.DeleteDir(key)
}

View File

@@ -1,29 +0,0 @@
package command
import (
"errors"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/go-etcd/etcd"
)
// NewRemoveCommand returns the CLI command for "rmdir".
func NewRemoveDirCommand() cli.Command {
return cli.Command{
Name: "rmdir",
Usage: "removes the key if it is an empty directory or a key-value pair",
Action: func(c *cli.Context) {
handleDir(c, removeDirCommandFunc)
},
}
}
// removeDirCommandFunc executes the "rmdir" command.
func removeDirCommandFunc(c *cli.Context, client *etcd.Client) (*etcd.Response, error) {
if len(c.Args()) == 0 {
return nil, errors.New("Key required")
}
key := c.Args()[0]
return client.DeleteDir(key)
}

View File

@@ -1,47 +0,0 @@
package command
import (
"errors"
"os"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/go-etcd/etcd"
)
// NewSetCommand returns the CLI command for "set".
func NewSetCommand() cli.Command {
return cli.Command{
Name: "set",
Usage: "set the value of a key",
Flags: []cli.Flag{
cli.IntFlag{Name: "ttl", Value: 0, Usage: "key time-to-live"},
cli.StringFlag{Name: "swap-with-value", Value: "", Usage: "previous value"},
cli.IntFlag{Name: "swap-with-index", Value: 0, Usage: "previous index"},
},
Action: func(c *cli.Context) {
handleKey(c, setCommandFunc)
},
}
}
// setCommandFunc executes the "set" command.
func setCommandFunc(c *cli.Context, client *etcd.Client) (*etcd.Response, error) {
if len(c.Args()) == 0 {
return nil, errors.New("Key required")
}
key := c.Args()[0]
value, err := argOrStdin(c.Args(), os.Stdin, 1)
if err != nil {
return nil, errors.New("Value required")
}
ttl := c.Int("ttl")
prevValue := c.String("swap-with-value")
prevIndex := c.Int("swap-with-index")
if prevValue == "" && prevIndex == 0 {
return client.Set(key, value, uint64(ttl))
} else {
return client.CompareAndSwap(key, value, uint64(ttl), prevValue, uint64(prevIndex))
}
}

View File

@@ -1,33 +0,0 @@
package command
import (
"errors"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/go-etcd/etcd"
)
// NewSetDirCommand returns the CLI command for "setDir".
func NewSetDirCommand() cli.Command {
return cli.Command{
Name: "setdir",
Usage: "create a new or existing directory",
Flags: []cli.Flag{
cli.IntFlag{Name: "ttl", Value: 0, Usage: "key time-to-live"},
},
Action: func(c *cli.Context) {
handleDir(c, setDirCommandFunc)
},
}
}
// setDirCommandFunc executes the "setDir" command.
func setDirCommandFunc(c *cli.Context, client *etcd.Client) (*etcd.Response, error) {
if len(c.Args()) == 0 {
return nil, errors.New("Key required")
}
key := c.Args()[0]
ttl := c.Int("ttl")
return client.SetDir(key, uint64(ttl))
}

Some files were not shown because too many files have changed in this diff Show More