version: bump to v2.0.0-rc.1

In anticipation for a 2.0.0-rc.0 release update and move the
documentation.

.godir

@@ -0,0 +1 @@
+github.com/coreos/etcd

.travis.yml

@@ -1,4 +1,5 @@
 language: go
+sudo: false
 go:
   - 1.3
 

CHANGELOG

@@ -6,7 +6,7 @@ v0.4.6
 * Various documentation improvements (#907, #882)
 
 v0.4.5
-* Flush headers immediatly on `wait=true` requests (#877)
+* Flush headers immediately on `wait=true` requests (#877)
 * Add `ETCD_HTTP_READ_TIMEOUT` and `ETCD_HTTP_WRITE_TIMEOUT` (#880)
 * Add `ETCDCTL_PEERS` configuration to etcdctl (#95)
 * etcdctl takes stdin for mk (#91)

Documentation/0.5/admin_guide.md

@@ -1,33 +0,0 @@
-## Administration
-
-### Data Directory
-
-#### Lifecycle
-
-When first started, etcd stores its configuration into a data directory specified by the data-dir configuration parameter.
-Configuration is stored in the write ahead log and includes: the local member ID, cluster ID, and initial cluster configuration.
-The write ahead log and snapshot files are used during member operation and to recover after a restart.
-
-If a member’s data directory is ever lost or corrupted then the user should remove the etcd member from the cluster via the [members API][members-api].
-
-A user should avoid restarting an etcd member with a data directory from an out-of-date backup.
-Using an out-of-date data directory can lead to inconsistency as the member had agreed to store information via raft then re-joins saying it needs that information again.
-For maximum safety, if an etcd member suffers any sort of data corruption or loss, it must be removed from the cluster.
-Once removed the member can be re-added with an empty data directory.
-
-[members-api]: https://github.com/coreos/etcd/blob/master/Documentation/0.5/other_apis.md#members-api
-
-#### Contents
-
-The data directory has two sub-directories in it:
-
-1. wal: write ahead log files are stored here. For details see the [wal package documentation][wal-pkg]
-2. snap: log snapshots are stored here. For details see the [snap package documentation][snap-pkg]
-
-[wal-pkg]: http://godoc.org/github.com/coreos/etcd/wal
-[snap-pkg]: http://godoc.org/github.com/coreos/etcd/snap
-
-### Cluster Lifecycle
-
-If you are spinning up multiple clusters for testing it is recommended that you specify a unique initial-cluster-token for the different clusters.
-This can protect you from cluster corruption in case of mis-configuration because two members started with different cluster tokens will refuse members from each other.

Documentation/0.5/clustering.md

@@ -1,204 +0,0 @@
-# Clustering Guide
-
-This guide will walk you through configuring a three machine etcd cluster with
-the following details:
-
-|Name	|Address	|
-|-------|---------------|
-|infra0	|10.0.1.10	|
-|infra1	|10.0.1.11	|
-|infra2	|10.0.1.12	|
-
-## Static
-
-As we know the cluster members, their addresses and the size of the cluster
-before starting we can use an offline bootstrap configuration. Each machine
-will get either the following command line or environment variables:
-
-```
-ETCD_INITIAL_CLUSTER=”infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379”
-ETCD_INITIAL_CLUSTER_STATE=new
-```
-
-```
--initial-cluster infra0=http://10.0.1.10:2379,http://10.0.1.11:2379,infra2=http://10.0.1.12:2379 \
-	-initial-cluster-state new
-```
-
-If you are spinning up multiple clusters (or creating and destroying a single cluster) with same configuration for testing purpose, it is highly recommended that you specify a unique `initial-cluster-token` for the different clusters. 
-By doing this, etcd can generate unique cluster IDs and member IDs for the clusters even if they otherwise have the exact same configuration. This can protect you from cross-cluster-interaction, which might corrupt your clusters.
-
-On each machine you would start etcd with these flags:
-
-```
-$ etcd -name infra0 -initial-advertise-peer-urls https://10.0.1.10:2379 initial-cluster-token etcd-cluster-1\
-	-initial-cluster infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379 \
-	-initial-cluster-state new
-$ etcd -name infra1 -initial-advertise-peer-urls https://10.0.1.11:2379 initial-cluster-token etcd-cluster-1\
-	-initial-cluster infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379 \
-	-initial-cluster-state new
-$ etcd -name infra2 -initial-advertise-peer-urls https://10.0.1.12:2379 initial-cluster-token etcd-cluster-1\
-	-initial-cluster infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379 \
-	-initial-cluster-state new
-```
-
-The command line parameters starting with `-initial-cluster` will be ignored on
-subsequent runs of etcd. You are free to remove the environment variables or
-command line flags after the initial bootstrap process. If you need to make
-changes to the configuration later see our guide on runtime configuration.
-
-### Error Cases
-
-In the following case we have not included our new host in the list of
-enumerated nodes. If this is a new cluster, the node must be added to the list
-of initial cluster members.
-```
-$ etcd -name infra1 -initial-advertise-peer-urls http://10.0.1.11:2379 \
-	-initial-cluster infra0=http://10.0.1.10:2379 \
-	-initial-cluster-state new
-etcd: infra1 not listed in the initial cluster config
-exit 1
-```
-
-In this case we are attempting to map a node (infra0) on a different address
-(127.0.0.1:2379) than its enumerated address in the cluster list
-(10.0.1.10:2379). If this node is to listen on multiple addresses, all
-addresses must be reflected in the “initial-cluster” configuration directive.
-
-```
-$ etcd -name infra0 -initial-advertise-peer-urls http://127.0.0.1:2379 \
-	-initial-cluster infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379 \
-	-initial-cluster-state=new
-etcd: infra0 has different advertised URLs in the cluster and advertised peer URLs list
-exit 1
-```
-
-If you configure a peer with a different set of configuration and attempt to
-join this cluster you will get a cluster ID mismatch and etcd will exit.
-
-```
-$ etcd -name infra3 -initial-advertise-peer-urls http://10.0.1.13:2379 \
-	-initial-cluster infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra3=http://10.0.1.13:2379 \
-	-initial-cluster-state=new
-etcd: conflicting cluster ID to the target cluster (c6ab534d07e8fcc4 != bc25ea2a74fb18b0). Exiting.
-exit 1
-```
-
-
-## Discovery
-
-In a number of cases you might not know the IPs of your cluster peers ahead of
-time. This is common when utilizing cloud providers or when your network uses
-DHCP. In these cases you can use an existing etcd cluster to bootstrap a new
-one. We call this process “discovery”.
-
-### Custom etcd discovery service
-
-Discovery uses an existing cluster to bootstrap itself.  If you are using your
-own etcd cluster you can create a URL like so:
-
-```
-$ curl -X PUT https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83/_config/size -d value=3
-```
-
-By setting the size key to the URL, you create a discovery URL with expected-cluster-size of 3.
-
-The URL you will use in this case will be
-`https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83`
-and the etcd members will use the
-`https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83`
-directory for registration as they start.
-
-Now we start etcd with those relevant flags for each member:
-
-```
-$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2379 -discovery https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83
-$ etcd -name infra1 -initial-advertise-peer-urls http://10.0.1.11:2379 -discovery https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83
-$ etcd -name infra2 -initial-advertise-peer-urls http://10.0.1.12:2379 -discovery https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83
-```
-
-This will cause each member to register itself with the custom etcd discovery service and begin
-the cluster once all machines have been registered.
-
-### Public discovery service
-
-If you do not have access to an existing cluster you can use the public discovery
-service hosted at discovery.etcd.io.  You can create a private discovery URL using the
-"new" endpoint like so:
-
-```
-$ curl https://discovery.etcd.io/new?size=3
-https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
-```
-
-This will create the cluster with an initial expected size of 3 members. If you
-do not specify a size a default of 3 will be used.
-
-```
-ETCD_DISCOVERY=https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
-```
-
-```
--discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
-```
-
-Now we start etcd with those relevant flags for each member:
-
-```
-$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2379 -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
-$ etcd -name infra1 -initial-advertise-peer-urls http://10.0.1.11:2379 -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
-$ etcd -name infra2 -initial-advertise-peer-urls http://10.0.1.12:2379 -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
-```
-
-This will cause each member to register itself with the discovery service and begin
-the cluster once all members have been registered.
-
-You can use the environment variable `ETCD_DISCOVERY_PROXY` to cause etcd to use an HTTP proxy to connect to the discovery service.
-
-### Error and Warning Cases
-
-#### Discovery Server Errors
-
-```
-$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2379 -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
-etcd: error: the cluster doesn’t have a size configuration value in https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de/_config
-exit 1
-```
-
-#### User Errors
-
-```
-$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2379 -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
-etcd: error: the cluster using discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de has already started with all 5 members
-exit 1
-```
-
-#### Warnings
-
-This is a harmless warning notifying you that the discovery URL will be
-ignored on this machine.
-
-```
-$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2379 -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
-etcd: warn: ignoring discovery URL: etcd has already been initialized and has a valid log in /var/lib/etcd
-```
-
-# 0.4 to 0.5+ Migration Guide
-
-In etcd 0.5 we introduced the ability to listen on more than one address and to
-advertise multiple addresses. This makes using etcd easier when you have
-complex networking, such as private and public networks on various cloud
-providers.
-
-To make understanding this feature easier, we changed the naming of some flags,
-but we support the old flags to make the migration from the old to new version
-easier.
-
-|Old Flag		|New Flag		|Migration Behavior									|
-|-----------------------|-----------------------|---------------------------------------------------------------------------------------|
-|-peer-addr		|-initial-advertise-peer-urls 	|If specified, peer-addr will be used as the only peer URL. Error if both flags specified.|
-|-addr			|-advertise-client-urls	|If specified, addr will be used as the only client URL. Error if both flags specified.|
-|-peer-bind-addr	|-listen-peer-urls	|If specified, peer-bind-addr will be used as the only peer bind URL. Error if both flags specified.|
-|-bind-addr		|-listen-client-urls	|If specified, bind-addr will be used as the only client bind URL. Error if both flags specified.|
-|-peers			|none			|Deprecated. The -initial-cluster flag provides a similar concept with different semantics. Please read this guide on cluster startup.|
-|-peers-file		|none			|Deprecated. The -initial-cluster flag provides a similar concept with different semantics. Please read this guide on cluster startup.|

Documentation/0.5/other_apis.md

@@ -1,70 +0,0 @@
-## Members API
-
-### GET /v2/members
-Return an HTTP 200 OK response code and a representation of all members in the etcd cluster:
-```
-    Example Request: GET 
-                     http://localhost:2379/v2/members
-    Response formats: JSON
-    Example Response:
-```
-```json
-    {
-        "members": [
-                {
-                    "id":"272e204152",
-                    "name":"infra1",
-                    "peerURLs":[
-                        "http://10.0.0.10:2379"
-                    ],
-                    "clientURLs":[
-                            "http://10.0.0.10:2380"
-                    ]
-                },
-                {
-                    "id":"2225373f43",
-                    "name":"infra2",
-                    "peerURLs":[
-                        "http://127.0.0.11:2379"
-                    ],
-                    "clientURLs":[
-                        "http://127.0.0.11:2380"
-                    ]
-            },
-        ]
-    }
-```
-
-### POST /v2/members
-Add a member to the cluster.
-Returns an HTTP 201 response code and the representation of added member with a newly generated a memberID when successful. Returns a string describing the failure condition when unsuccessful. 
-
-If the POST body is malformed an HTTP 400 will be returned. If the member exists in the cluster or existed in the cluster at some point in the past an HTTP 500(TODO: fix this) will be returned. If the cluster fails to process the request within timeout an HTTP 500 will be returned, though the request may be processed later.
-```
-    Example Request: POST
-                     http://localhost:2379/v2/members
-                     Body:
-                     {"peerURLs":["http://10.0.0.10:2379"]}
-    Respose formats: JSON
-    Example Response:
-```
-```json
-    {
-        "id":"3777296169",
-        "peerURLs":[
-            "http://10.0.0.10:2379"
-        ],
-    }
-```
-
-### DELETE /v2/members/:id
-Remove a member from the cluster.
-Returns empty when successful. Returns a string describing the failure condition when unsuccessful. 
-
-If the member does not exist in the cluster an HTTP 500(TODO: fix this) will be returned. If the cluster fails to process the request within timeout an HTTP 500 will be returned, though the request may be processed later.
-```
-    Response formats: JSON
-    Example Request: DELETE
-                     http://localhost:2379/v2/members/272e204152
-    Example Response: Empty
-```

Documentation/2.0/0_4_migration_tool.md

@@ -0,0 +1,47 @@
+## etcd 0.4.x -> 2.0.0 Data Migration Tool
+
+### Upgrading from 0.4.x
+
+Between 0.4.x and 2.0, the on-disk data formats have changed. In order to allow users to convert to 2.0, a migration tool is provided.
+
+In the early 2.0.0-alpha series, we're providing this tool early to encourage adoption. However, before 2.0.0-release, etcd will autodetect the 0.4.x data dir upon upgrade and automatically update the data too (while leaving a backup, in case of emergency).
+
+### Data Migration Tips
+
+* Keep the environment variables and etcd instance flags the same (much as [the upgrade document](../upgrade.md) suggests), particularly `--name`/`ETCD_NAME`.
+* Don't change the cluster configuration. If there's a plan to add or remove machines, it's probably best to arrange for that after the migration, rather than before or at the same time.
+
+### Running the tool
+
+The tool can be run via:
+```sh
+./bin/etcd-migrate --data-dir=<PATH TO YOUR DATA>
+```
+
+It should autodetect everything and convert the data-dir to be 2.0 compatible. It does not remove the 0.4.x data, and is safe to convert multiple times; the 2.0 data will be overwritten. Recovering the disk space once everything is settled is covered later in the document.
+
+If, however, it complains about autodetecting the name (which can happen, depending on how the cluster was configured), you need to supply the name of this particular node. This is equivalent to the `--name` flag (or `ETCD_NAME` variable) that etcd was run with, which can also be found by accessing the self api, eg:
+
+```sh
+curl -L http://127.0.0.1:4001/v2/stats/self
+```
+
+Where the `"name"` field is the name of the local machine.
+
+Then, run the migration tool with
+
+```sh
+./bin/etcd-migrate --data-dir=<PATH TO YOUR DATA> --name=<NAME>
+```
+
+And the tool should migrate successfully. If it still has an error at this time, it's a failure or bug in the tool and it's worth reporting a bug.
+
+### Recovering Disk Space
+
+If the conversion has completed, the entire cluster is running on something 2.0-based, and the disk space is important, the following command will clear 0.4.x data from the data-dir:
+
+```sh
+rm -ri snapshot conf log
+```
+
+It will ask before every deletion, but these are the 0.4.x files and will not affect the working 2.0 data.

Documentation/2.0/admin_guide.md

@@ -0,0 +1,155 @@
+## Administration
+
+### Data Directory
+
+#### Lifecycle
+
+When first started, etcd stores its configuration into a data directory specified by the data-dir configuration parameter.
+Configuration is stored in the write ahead log and includes: the local member ID, cluster ID, and initial cluster configuration.
+The write ahead log and snapshot files are used during member operation and to recover after a restart.
+
+If a member’s data directory is ever lost or corrupted then the user should remove the etcd member from the cluster via the [members API][members-api].
+
+A user should avoid restarting an etcd member with a data directory from an out-of-date backup.
+Using an out-of-date data directory can lead to inconsistency as the member had agreed to store information via raft then re-joins saying it needs that information again.
+For maximum safety, if an etcd member suffers any sort of data corruption or loss, it must be removed from the cluster.
+Once removed the member can be re-added with an empty data directory.
+
+[members-api]: https://github.com/coreos/etcd/blob/master/Documentation/2.0/other_apis.md#members-api
+
+#### Contents
+
+The data directory has two sub-directories in it:
+
+1. wal: write ahead log files are stored here. For details see the [wal package documentation][wal-pkg]
+2. snap: log snapshots are stored here. For details see the [snap package documentation][snap-pkg]
+
+[wal-pkg]: http://godoc.org/github.com/coreos/etcd/wal
+[snap-pkg]: http://godoc.org/github.com/coreos/etcd/snap
+
+### Cluster Lifecycle
+
+If you are spinning up multiple clusters for testing it is recommended that you specify a unique initial-cluster-token for the different clusters.
+This can protect you from cluster corruption in case of mis-configuration because two members started with different cluster tokens will refuse members from each other.
+
+### Member Migration
+
+When there is a scheduled machine maintenance or retirement, you might want to migrate an etcd member to another machine without losing the data and changing the member ID. 
+
+The data directory contains all the data to recover a member to its point-in-time state. To migrate a member:
+
+* Stop the member process
+* Copy the data directory of the now-idle member to the new machine
+* Update the peer URLs for that member to reflect the new machine according to the [member api] [change peer url]
+* Start etcd on the new machine, using the same configuration and the copy of the data directory
+
+This example will walk you through the process of migrating the infra1 member to a new machine:
+
+|Name|Peer URL|
+|------|--------------|
+|infra0|10.0.1.10:2380|
+|infra1|10.0.1.11:2380|
+|infra2|10.0.1.12:2380|
+
+```
+$ export ETCDCTL_PEERS=http://10.0.1.10:2379,http://10.0.1.11:2379,http://10.0.1.12:2379
+```
+
+```
+$ etcdctl member list
+84194f7c5edd8b37: name=infra0 peerURLs=http://10.0.1.10:2380 clientURLs=http://127.0.0.1:2379,http://10.0.1.10:2379
+b4db3bf5e495e255: name=infra1 peerURLs=http://10.0.1.11:2380 clientURLs=http://127.0.0.1:2379,http://10.0.1.11:2379
+bc1083c870280d44: name=infra2 peerURLs=http://10.0.1.12:2380 clientURLs=http://127.0.0.1:2379,http://10.0.1.12:2379
+```
+
+#### Stop the member etcd process
+
+```
+$ ssh core@10.0.1.11
+```
+
+```
+$ sudo systemctl stop etcd
+```
+
+#### Copy the data directory of the now-idle member to the new machine
+
+```
+$ tar -cvzf node1.etcd.tar.gz /var/lib/etcd/node1.etcd 
+```
+
+```
+$ scp node1.etcd.tar.gz core@10.0.1.13:~/
+```
+
+#### Update the peer URLs for that member to reflect the new machine
+
+```
+$ curl http://10.0.1.10:2379/v2/members/b4db3bf5e495e255 -XPUT \
+-H "Content-Type: application/json" -d '{"peerURLs":["http://10.0.1.13:2380"]}'
+```
+
+#### Start etcd on the new machine, using the same configuration and the copy of the data directory
+
+```
+$ ssh core@10.0.1.13
+```
+
+```
+$ tar -xzvf node1.etcd.tar.gz -C /var/lib/etcd
+```
+
+```
+etcd -name node1 \
+-listen-peer-urls http://10.0.1.13:2380 \
+-listen-client-urls http://10.0.1.13:2379,http://127.0.0.1:2379 \
+-advertise-client-urls http://10.0.1.13:2379,http://127.0.0.1:2379
+```
+
+[change peer url]: https://github.com/coreos/etcd/blob/master/Documentation/2.0/other_apis.md#change-the-peer-urls-of-a-member
+
+### Disaster Recovery
+
+etcd is designed to be resilient to machine failures. An etcd cluster can automatically recover from any number of temporary failures (for example, machine reboots), and a cluster of N members can tolerate up to _(N/2)-1_ permanent failures (where a member can no longer access the cluster, due to hardware failure or disk corruption). However, in extreme circumstances, a cluster might permanently lose enough members such that quorum is irrevocably lost. For example, if a three-node cluster suffered two simultaneous and unrecoverable machine failures, it would be normally impossible for the cluster to restore quorum and continue functioning.
+
+To recover from such scenarios, etcd provides functionality to backup and restore the datastore and recreate the cluster without data loss.
+
+#### Backing up the datastore
+
+The first step of the recovery is to backup the data directory on a functioning etcd node. To do this, use the `etcdctl backup` command, passing in the original data directory used by etcd. For example:
+
+```sh
+    etcdctl backup \
+      --data-dir /var/lib/etcd \
+      --backup-dir /tmp/etcd_backup
+```
+
+This command will rewrite some of the metadata contained in the backup (specifically, the node ID and cluster ID), which means that the node will lose its former identity. In order to recreate a cluster from the backup, you will need to start a new, single-node cluster. The metadata is rewritten to prevent the new node from inadvertently being joined onto an existing cluster.
+
+#### Restoring a backup
+
+To restore a backup using the procedure created above, start etcd with the `-force-new-cluster` option and pointing to the backup directory. This will initialize a new, single-member cluster with the default advertised peer URLs, but preserve the entire contents of the etcd data store. Continuing from the previous example:
+
+```sh
+    etcd \
+      -data-dir=/tmp/etcd_backup \
+      -force-new-cluster \
+      ...
+```
+
+Now etcd should be available on this node and serving the original datastore.
+
+Once you have verified that etcd has started successfully, shut it down and move the data back to the previous location (you may wish to make another copy as well to be safe):
+
+```sh
+    pkill etcd
+    rm -fr /var/lib/etcd
+    mv /tmp/etcd_backup /var/lib/etcd
+    etcd \
+      -data-dir=/var/lib/etcd \
+      ...
+```
+
+#### Restoring the cluster
+
+Now that the node is running successfully, you can add more nodes to the cluster and restore resiliency. See the [runtime configuration](runtime-configuration.md) guide for more details.

Documentation/2.0/api.md

@@ -401,19 +401,19 @@ curl 'http://127.0.0.1:2379/v2/keys/dir/asdf?consistent=true&wait=true'
 
 ```json
 {
-	"action": "expire",
-	"node": {
-		"createdIndex": 8,
-		"key": "/dir",
-		"modifiedIndex": 15
-	},
-	"prevNode": {
-		"createdIndex": 8,
-		"key": "/dir",
-		"dir":true,
-		"modifiedIndex": 17,
-		"expiration": "2013-12-11T10:39:35.689275857-08:00"
-	}
+    "action": "expire",
+    "node": {
+        "createdIndex": 8,
+        "key": "/dir",
+        "modifiedIndex": 15
+    },
+    "prevNode": {
+        "createdIndex": 8,
+        "key": "/dir",
+        "dir":true,
+        "modifiedIndex": 17,
+        "expiration": "2013-12-11T10:39:35.689275857-08:00"
+    }
 }
 ```
 
@@ -639,22 +639,22 @@ We should see the response as an array of items:
 ```json
 {
     "action": "get",
-    "node": {
-        "key": "/",
-        "dir": true,
-        "nodes": [
-            {
-                "key": "/foo_dir",
-                "dir": true,
-                "modifiedIndex": 2,
-                "createdIndex": 2
-            },
-            {
-                "key": "/foo",
-                "value": "two",
-                "modifiedIndex": 1,
-                "createdIndex": 1
-            }
+    "node": {
+        "key": "/",
+        "dir": true,
+        "nodes": [
+            {
+                "key": "/foo_dir",
+                "dir": true,
+                "modifiedIndex": 2,
+                "createdIndex": 2
+            },
+            {
+                "key": "/foo",
+                "value": "two",
+                "modifiedIndex": 1,
+                "createdIndex": 1
+            }
         ]
     }
 }
@@ -669,33 +669,33 @@ curl http://127.0.0.1:2379/v2/keys/?recursive=true
 
 ```json
 {
-    "action": "get",
-    "node": {
-        "key": "/",
-        "dir": true,
-        "nodes": [
-            {
-                "key": "/foo_dir",
-                "dir": true,
-                "nodes": [
-                    {
-                        "key": "/foo_dir/foo",
-                        "value": "bar",
-                        "modifiedIndex": 2,
-                        "createdIndex": 2
-                    }
-                ],
-                "modifiedIndex": 2,
-                "createdIndex": 2
-            },
-            {
-                "key": "/foo",
-                "value": "two",
-                "modifiedIndex": 1,
-                "createdIndex": 1
-            }
-        ]
-    }
+    "action": "get",
+    "node": {
+        "key": "/",
+        "dir": true,
+        "nodes": [
+            {
+                "key": "/foo_dir",
+                "dir": true,
+                "nodes": [
+                    {
+                        "key": "/foo_dir/foo",
+                        "value": "bar",
+                        "modifiedIndex": 2,
+                        "createdIndex": 2
+                    }
+                ],
+                "modifiedIndex": 2,
+                "createdIndex": 2
+            },
+            {
+                "key": "/foo",
+                "value": "two",
+                "modifiedIndex": 1,
+                "createdIndex": 1
+            }
+        ]
+    }
 }
 ```
 

Documentation/2.0/backward_compatibility.md

@@ -0,0 +1,53 @@
+### Backward Compatibility
+
+The main goal of etcd 2.0 release is to improve cluster safety around bootstrapping and dynamic reconfiguration. To do this, we deprecated the old error-prone APIs and provide a new set of APIs.
+
+The other main focus of this release was a more reliable Raft implementation, but as this change is internal it should not have any notable effects to users.
+
+#### Command Line Flags Changes
+
+The major flag changes are to mostly related to bootstrapping. The `initial-*` flags provide an improved way to specify the required criteria to start the cluster. The advertised URLs now support a list of values instead of a single value, which allows etcd users to gracefully migrate to the new set of IANA-assigned ports (2379/client and 2380/peers) while maintaining backward compatibility with the old ports.
+
+ - `-addr` is replaced by `-advertise-client-urls`.
+ - `-bind-addr` is replaced by `-listen-client-urls`.
+ - `-peer-add` is replaced by `-initial-advertise-peer-urls`.
+ - `-peer-bind-addr` is replaced by `-listen-peer-urls`.
+ - `-peers` is replaced by `-initial-cluster`.
+ - `-peers-file` is replaced by `-initial-cluster`.
+
+The documentation of new command line flags can be found at 
+https://github.com/coreos/etcd/blob/master/Documentation/2.0/configuration.md.
+
+#### Data Dir 
+- Default data dir location has changed from {$hostname}.etcd to {name}.etcd.
+
+- The disk format within the data dir has changed. etcd 2.0 should be able to auto upgrade the old data format. Instructions on doing so manually are in the [migration tool doc][migrationtooldoc].
+
+[migrationtooldoc]: https://github.com/coreos/etcd/blob/master/Documentation/2.0/0_4_migration_tool.md
+
+#### Standby
+
+etcd 0.4’s standby mode has been deprecated by 2.0’s [proxy mode][proxymode]. 
+
+Standby mode was intended for large clusters that had a subset of the members acting in the consensus process. Overall this process was too magical and allowed for operators to back themselves into a corner.
+
+Proxy mode in 2.0 will provide similar functionality, and with improved control over which machines act as proxies due to the operator specifically configuring them. Proxies also support read only or read/write modes for increased security and durability.
+
+[proxymode]: https://github.com/coreos/etcd/blob/master/Documentation/2.0/proxy.md
+
+#### Discovery Service
+
+A size key needs to be provided inside a [discovery token][discoverytoken].
+[discoverytoken]: https://github.com/coreos/etcd/blob/master/Documentation/2.0/clustering.md#custom-etcd-discovery-service
+
+#### HTTP Admin API
+
+`v2/admin` on peer url and `v2/keys/_etcd` are unified under the new [v2/member API][memberapi] to better explain which machines are part of an etcd cluster, and to simplify the keyspace for all your use cases.
+
+[memberapi]: https://github.com/coreos/etcd/blob/master/Documentation/2.0/other_apis.md
+
+#### HTTP Key Value API
+- The follower can now transparently proxy write equests to the leader. Clients will no longer see 307 redirections to the leader from etcd.
+
+- Expiration time is in UTC instead of local time.
+

Documentation/2.0/clustering.md

@@ -0,0 +1,233 @@
+# Clustering Guide
+
+This guide will walk you through configuring a three machine etcd cluster with the following details:
+
+|Name	|Address	|
+|-------|-----------|
+|infra0	|10.0.1.10	|
+|infra1	|10.0.1.11	|
+|infra2	|10.0.1.12	|
+
+## Static
+
+As we know the cluster members, their addresses and the size of the cluster before starting, we can use an offline bootstrap configuration by setting the `initial-cluster` flag. Each machine will get either the following command line or environment variables:
+
+```
+ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380"
+ETCD_INITIAL_CLUSTER_STATE=new
+```
+
+```
+-initial-cluster infra0=http://10.0.1.10:2380,http://10.0.1.11:2380,infra2=http://10.0.1.12:2380 \
+  -initial-cluster-state new
+```
+
+Note that the URLs specified in `initial-cluster` are the _advertised peer URLs_, i.e. they should match the value of `initial-advertise-peer-urls` on the respective nodes.
+
+If you are spinning up multiple clusters (or creating and destroying a single cluster) with same configuration for testing purpose, it is highly recommended that you specify a unique `initial-cluster-token` for the different clusters. By doing this, etcd can generate unique cluster IDs and member IDs for the clusters even if they otherwise have the exact same configuration. This can protect you from cross-cluster-interaction, which might corrupt your clusters.
+
+On each machine you would start etcd with these flags:
+
+```
+$ etcd -name infra0 -initial-advertise-peer-urls https://10.0.1.10:2380 \
+  -listen-peer-urls https://10.0.1.10:2380 \
+  -initial-cluster-token etcd-cluster-1 \
+  -initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380 \
+  -initial-cluster-state new
+```
+```
+$ etcd -name infra1 -initial-advertise-peer-urls https://10.0.1.11:2380 \
+  -listen-peer-urls https://10.0.1.11:2380 \
+  -initial-cluster-token etcd-cluster-1 \
+  -initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380 \
+  -initial-cluster-state new
+```
+```
+$ etcd -name infra2 -initial-advertise-peer-urls https://10.0.1.12:2380 \
+  -listen-peer-urls https://10.0.1.12:2380 \
+  -initial-cluster-token etcd-cluster-1 \
+  -initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380 \
+  -initial-cluster-state new
+```
+
+The command line parameters starting with `-initial-cluster` will be ignored on subsequent runs of etcd. You are free to remove the environment variables or command line flags after the initial bootstrap process. If you need to make changes to the configuration later (for example, adding or removing members to/from the cluster), see the [runtime configuration](runtime-configuration.md) guide.
+
+### Error Cases
+
+In the following example, we have not included our new host in the list of enumerated nodes. If this is a new cluster, the node _must_ be added to the list of initial cluster members.
+
+```
+$ etcd -name infra1 -initial-advertise-peer-urls http://10.0.1.11:2380 \
+  -listen-peer-urls https://10.0.1.11:2380 \
+  -initial-cluster infra0=http://10.0.1.10:2380 \
+  -initial-cluster-state new
+etcd: infra1 not listed in the initial cluster config
+exit 1
+```
+
+In this example, we are attempting to map a node (infra0) on a different address (127.0.0.1:2380) than its enumerated address in the cluster list (10.0.1.10:2380). If this node is to listen on multiple addresses, all addresses _must_ be reflected in the "initial-cluster" configuration directive.
+
+```
+$ etcd -name infra0 -initial-advertise-peer-urls http://127.0.0.1:2380 \
+  -listen-peer-urls http://10.0.1.10:2380 \
+  -initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380 \
+  -initial-cluster-state=new
+etcd: error setting up initial cluster: infra0 has different advertised URLs in the cluster and advertised peer URLs list
+exit 1
+```
+
+If you configure a peer with a different set of configuration and attempt to join this cluster you will get a cluster ID mismatch and etcd will exit.
+
+```
+$ etcd -name infra3 -initial-advertise-peer-urls http://10.0.1.13:2380 \
+  -listen-peer-urls http://10.0.1.13:2380 \
+  -initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra3=http://10.0.1.13:2380 \
+  -initial-cluster-state=new
+etcd: conflicting cluster ID to the target cluster (c6ab534d07e8fcc4 != bc25ea2a74fb18b0). Exiting.
+exit 1
+```
+
+## Discovery
+
+In a number of cases, you might not know the IPs of your cluster peers ahead of time. This is common when utilizing cloud providers or when your network uses DHCP. In these cases, rather than specifying a static configuration, you can use an existing etcd cluster to bootstrap a new one. We call this process "discovery".
+
+### Lifetime of a Discovery URL
+
+A discovery URL identifies a unique etcd cluster. Instead of reusing a discovery URL, you should always create discovery URLs for new clusters.
+
+Moreover, discovery URLs should ONLY be used for the initial bootstrapping of a cluster. To change cluster membership after the cluster is already running, see the [runtime reconfiguration][runtime] guide.
+
+[runtime]: https://github.com/coreos/etcd/blob/master/Documentation/2.0/runtime-configuration.md
+
+### Custom etcd discovery service
+
+Discovery uses an existing cluster to bootstrap itself. If you are using your own etcd cluster you can create a URL like so:
+
+```
+$ curl -X PUT https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83/_config/size -d value=3
+```
+
+By setting the size key to the URL, you create a discovery URL with an expected cluster size of 3.
+
+If you bootstrap an etcd cluster using discovery service with more than the expected number of etcd members, the extra etcd processes will [fall back][fall-back] to being [proxies][proxy] by default.
+
+The URL you will use in this case will be `https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83` and the etcd members will use the `https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83` directory for registration as they start.
+
+Now we start etcd with those relevant flags for each member:
+
+```
+$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2380 \
+  -listen-peer-urls http://10.0.1.10:2380 \
+  -discovery https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83
+```
+```
+$ etcd -name infra1 -initial-advertise-peer-urls http://10.0.1.11:2380 \
+  -listen-peer-urls http://10.0.1.11:2380 \
+  -discovery https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83
+```
+```
+$ etcd -name infra2 -initial-advertise-peer-urls http://10.0.1.12:2380 \
+  -listen-peer-urls http://10.0.1.12:2380 \
+  -discovery https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83
+```
+
+This will cause each member to register itself with the custom etcd discovery service and begin the cluster once all machines have been registered.
+
+### Public discovery service
+
+If you do not have access to an existing cluster, you can use the public discovery service hosted at `discovery.etcd.io`.  You can create a private discovery URL using the "new" endpoint like so:
+
+```
+$ curl https://discovery.etcd.io/new?size=3
+https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
+```
+
+This will create the cluster with an initial expected size of 3 members. If you do not specify a size, a default of 3 will be used.
+
+If you bootstrap an etcd cluster using discovery service with more than the expected number of etcd members, the extra etcd processes will [fall back][fall-back] to being [proxies][proxy] by default.
+
+[fall-back]: proxy.md#fallback-to-proxy-mode-with-discovery-service
+[proxy]: proxy.md
+
+```
+ETCD_DISCOVERY=https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
+```
+
+```
+-discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
+```
+
+Now we start etcd with those relevant flags for each member:
+
+```
+$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2380 \
+  -listen-peer-urls http://10.0.1.10:2380 \
+  -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
+```
+```
+$ etcd -name infra1 -initial-advertise-peer-urls http://10.0.1.11:2380 \
+  -listen-peer-urls http://10.0.1.11:2380 \
+  -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
+```
+```
+$ etcd -name infra2 -initial-advertise-peer-urls http://10.0.1.12:2380 \
+  -listen-peer-urls http://10.0.1.12:2380 \
+  -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
+```
+
+This will cause each member to register itself with the discovery service and begin the cluster once all members have been registered.
+
+You can use the environment variable `ETCD_DISCOVERY_PROXY` to cause etcd to use an HTTP proxy to connect to the discovery service.
+
+### Error and Warning Cases
+
+#### Discovery Server Errors
+
+
+```
+$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2380 \
+  -listen-peer-urls http://10.0.1.10:2380 \
+  -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
+etcd: error: the cluster doesn’t have a size configuration value in https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de/_config
+exit 1
+```
+
+#### User Errors
+
+This error will occur if the discovery cluster already has the configured number of members, and `discovery-fallback` is explicitly disabled
+
+```
+$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2380 \
+  -listen-peer-urls http://10.0.1.10:2380 \
+  -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de \
+  -discovery-fallback exit
+etcd: discovery: cluster is full
+exit 1
+```
+
+#### Warnings
+
+This is a harmless warning notifying you that the discovery URL will be
+ignored on this machine.
+
+```
+$ etcd -name infra0 -initial-advertise-peer-urls http://10.0.1.10:2380 \
+  -listen-peer-urls http://10.0.1.10:2380 \
+  -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
+etcdserver: discovery token ignored since a cluster has already been initialized. Valid log found at /var/lib/etcd
+```
+
+# 0.4 to 2.0+ Migration Guide
+
+In etcd 2.0 we introduced the ability to listen on more than one address and to advertise multiple addresses. This makes using etcd easier when you have complex networking, such as private and public networks on various cloud providers.
+
+To make understanding this feature easier, we changed the naming of some flags, but we support the old flags to make the migration from the old to new version easier.
+
+|Old Flag		|New Flag		|Migration Behavior									|
+|-----------------------|-----------------------|---------------------------------------------------------------------------------------|
+|-peer-addr		|-initial-advertise-peer-urls 	|If specified, peer-addr will be used as the only peer URL. Error if both flags specified.|
+|-addr			|-advertise-client-urls	|If specified, addr will be used as the only client URL. Error if both flags specified.|
+|-peer-bind-addr	|-listen-peer-urls	|If specified, peer-bind-addr will be used as the only peer bind URL. Error if both flags specified.|
+|-bind-addr		|-listen-client-urls	|If specified, bind-addr will be used as the only client bind URL. Error if both flags specified.|
+|-peers			|none			|Deprecated. The -initial-cluster flag provides a similar concept with different semantics. Please read this guide on cluster startup.|
+|-peers-file		|none			|Deprecated. The -initial-cluster flag provides a similar concept with different semantics. Please read this guide on cluster startup.|

Documentation/2.0/configuration.md

@@ -0,0 +1,139 @@
+## Configuration Flags
+
+etcd is configurable through command-line flags and environment variables. Options set on the command line take precedence over those from the environment.
+
+The format of environment variable for flag `-my-flag` is `ENV_MY_FLAG`. It applies to all  flags.
+
+To start etcd automatically using custom settings at startup in Linux, using a [systemd][systemd-intro] unit is highly recommended.
+
+[systemd-intro]: http://freedesktop.org/wiki/Software/systemd/
+
+### Member Flags
+
+##### -name
++ Human-readable name for this member.
++ default: "default"
+
+##### -data-dir
++ Path to the data directory.
++ default: "${name}.etcd"
+
+##### -snapshot-count
++ Number of committed transactions to trigger a snapshot to disk.
++ default: "10000"
+
+##### -listen-peer-urls
++ List of URLs to listen on for peer traffic.
++ default: "http://localhost:2380,http://localhost:7001"
+
+##### -listen-client-urls
++ List of URLs to listen on for client traffic.
++ default: "http://localhost:2379,http://localhost:4001"
+
+##### -max-snapshots
++ Maximum number of snapshot files to retain (0 is unlimited)
++ default: 5
+
+##### -max-wals
++ Maximum number of wal files to retain (0 is unlimited)
++ default: 5
+
+##### -cors
++ Comma-separated white list of origins for CORS (cross-origin resource sharing).
++ default: none
+
+### Clustering Flags
+
+`-initial` prefix flags are used in bootstrapping ([static bootstrap][build-cluster], [discovery-service bootstrap][discovery] or [runtime reconfiguration][reconfig]) a new member, and ignored when restarting an existing member.
+
+`-discovery` prefix flags need to be set when using [discovery service][discovery].
+
+##### -initial-advertise-peer-urls
+
++ List of this member's peer URLs to advertise to the rest of the cluster. These addresses are used for communicating etcd data around the cluster. At least one must be routable to all cluster members.
++ default: "http://localhost:2380,http://localhost:7001"
+
+##### -initial-cluster
++ Initial cluster configuration for bootstrapping.
++ default: "default=http://localhost:2380,default=http://localhost:7001"
+
+##### initial-cluster-state
++ Initial cluster state ("new" or "existing").
++ default: "new"
+
+##### initial-cluster-token
++ Initial cluster token for the etcd cluster during bootstrap.
++ default: "etcd-cluster"
+
+##### advertise-client-urls
++ List of this member's client URLs to advertise to the rest of the cluster.
++ default: "http://localhost:2379,http://localhost:4001"
+
+##### -discovery
++ Discovery URL used to bootstrap the cluster.
++ default: none
+
+##### -discovery-fallback
++ Expected behavior ("exit" or "proxy") when discovery services fails.
++ default: "proxy"
+
+##### -discovery-proxy
++ HTTP proxy to use for traffic to discovery service.
++ default: none
+
+### Proxy Flags
+
+`-proxy` prefix flags configures etcd to run in [proxy mode][proxy].
+
+##### -proxy
++ Proxy mode setting ("off", "readonly" or "on").
++ default: "off"
+
+### Security Flags
+
+The security flags help to [build a secure etcd cluster][security].
+
+##### -ca-file
++ Path to the client server TLS CA file.
++ default: none
+
+##### -cert-file
++ Path to the client server TLS cert file.
++ default: none
+
+##### -key-file
++ Path to the client server TLS key file.
++ default: none
+
+##### -peer-ca-file
++ Path to the peer server TLS CA file.
++ default: none
+
+##### -peer-cert-file
++ Path to the peer server TLS cert file.
++ default: none
+
+##### -peer-key-file
++ Path to the peer server TLS key file.
++ default: none
+
+### Unsafe Flags
+
+Be CAUTIOUS to use unsafe flags because it will break the guarantee given by consensus protocol. For example, it may panic if other members in the cluster are still alive. Follow the instructions when using these falgs.
+
+##### -force-new-cluster
++ Force to create a new one-member cluster. It commits configuration changes in force to remove all existing members in the cluster and add itself. It needs to be set to [restore a backup][restore].
++ default: false
+
+### Miscellaneous Flags
+
+##### -version
++ Print the version and exit.
++ default: false
+
+[build-cluster]: https://github.com/coreos/etcd/blob/master/Documentation/2.0/clustering.md#static
+[reconfig]: https://github.com/coreos/etcd/blob/master/Documentation/2.0/runtime-configuration.md
+[discovery]: https://github.com/coreos/etcd/blob/master/Documentation/2.0/clustering.md#discovery
+[proxy]: https://github.com/coreos/etcd/blob/master/Documentation/2.0/proxy.md
+[security]: https://github.com/coreos/etcd/blob/master/Documentation/security.md
+[restore]: https://github.com/coreos/etcd/blob/master/Documentation/2.0/admin_guide.md#restoring-a-backup

Documentation/2.0/glossary.md

@@ -28,4 +28,4 @@ Client is a caller of the cluster's HTTP API.
 
 ### Machine (deprecated)
 
-The alternative of Member in etcd before 0.5
+The alternative of Member in etcd before 2.0

Documentation/2.0/other_apis.md

@@ -0,0 +1,119 @@
+## Members API
+
+* [List members](#list-members)
+* [Add a member](#add-a-member)
+* [Delete a member](#delete-a-member)
+* [Change the peer urls of a member](#change-the-peer-urls-of-a-member)
+
+## List members
+
+Return an HTTP 200 OK response code and a representation of all members in the etcd cluster.
+
+### Request
+
+```
+GET /v2/members HTTP/1.1
+```
+
+### Example
+
+```sh
+curl http://10.0.0.10:2379/v2/members
+```
+
+```json
+{
+    "members": [
+        {
+            "id": "272e204152",
+            "name": "infra1",
+            "peerURLs": [
+                "http://10.0.0.10:2380"
+            ],
+            "clientURLs": [
+                "http://10.0.0.10:2379"
+            ]
+        },
+        {
+            "id": "2225373f43",
+            "name": "infra2",
+            "peerURLs": [
+                "http://10.0.0.11:2380"
+            ],
+            "clientURLs": [
+                "http://10.0.0.11:2379"
+            ]
+        },
+    ]
+}
+```
+
+## Add a member
+
+Returns an HTTP 201 response code and the representation of added member with a newly generated a memberID when successful. Returns a string describing the failure condition when unsuccessful. 
+
+If the POST body is malformed an HTTP 400 will be returned. If the member exists in the cluster or existed in the cluster at some point in the past an HTTP 409 will be returned. If any of the given peerURLs exists in the cluster an HTTP 409 will be returned. If the cluster fails to process the request within timeout an HTTP 500 will be returned, though the request may be processed later.
+
+### Request
+
+```
+POST /v2/members HTTP/1.1
+
+{"peerURLs": ["http://10.0.0.10:2380"]}
+```
+
+### Example
+
+```sh
+curl http://10.0.0.10:2379/v2/members -XPOST \
+-H "Content-Type: application/json" -d '{"peerURLs":["http://10.0.0.10:2380"]}'
+```
+
+```json
+{
+    "id": "3777296169",
+    "peerURLs": [
+        "http://10.0.0.10:2380"
+    ]
+}
+```
+
+## Delete a member
+
+Remove a member from the cluster. The member ID must be a hex-encoded uint64.
+Returns 204 with empty content when successful. Returns a string describing the failure condition when unsuccessful. 
+
+If the member does not exist in the cluster an HTTP 500(TODO: fix this) will be returned. If the cluster fails to process the request within timeout an HTTP 500 will be returned, though the request may be processed later.
+
+### Request
+
+```
+DELETE /v2/members/<id> HTTP/1.1
+```
+
+### Example
+
+```sh
+curl http://10.0.0.10:2379/v2/members/272e204152 -XDELETE
+```
+
+## Change the peer urls of a member
+
+Change the peer urls of a given mamber. The member ID must be a hex-encoded uint64. Returns 204 with empty content when successful. Returns a string describing the failure condition when unsuccessful.
+
+If the POST body is malformed an HTTP 400 will be returned. If the member does not exist in the cluster an HTTP 404 will be returned. If any of the given peerURLs exists in the cluster an HTTP 409 will be returned. If the cluster fails to process the request within timeout an HTTP 500 will be returned, though the request may be processed later.
+
+#### Request
+
+```
+PUT /v2/members/<id> HTTP/1.1
+
+{"peerURLs": ["http://10.0.0.10:2380"]}
+```
+
+#### Example
+
+```sh
+curl http://10.0.0.10:2379/v2/members/272e204152 -XPUT \
+-H "Content-Type: application/json" -d '{"peerURLs":["http://10.0.0.10:2380"]}'
+```

Documentation/2.0/proxy.md

@@ -0,0 +1,32 @@
+## Proxy
+
+etcd can now run as a transparent proxy. Running etcd as a proxy allows for easily discovery of etcd within your infrastructure, since it can run on each machine as a local service. In this mode, etcd acts as a reverse proxy and forwards client requests to an active etcd cluster. The etcd proxy does not participant in the consensus replication of the etcd cluster, thus it neither increases the resilience nor decreases the write performance of the etcd cluster. 
+
+etcd currently supports two proxy modes: `readwrite` and `readonly`. The default mode is `readwrite`, which forwards both read and write requests to the etcd cluster. A `readonly` etcd proxy only forwards read requests to the etcd cluster, and returns `HTTP 501` to all write requests. 
+
+### Using an etcd proxy
+To start etcd in proxy mode, you need to provide three flags: `proxy`, `listen-client-urls`, and `initial-cluster` (or `discovery-url`). 
+
+To start a readwrite proxy, set `-proxy on`; To start a readonly proxy, set `-proxy readonly`.
+
+The proxy will be listening on `listen-client-urls` and forward requests to the etcd cluster discovered from in `initial-cluster` or `discovery url`. 
+
+#### Start an etcd proxy with a static configuration
+To start a proxy that will connect to a statically defined etcd cluster, specify the `initial-cluster` flag:
+```
+etcd -proxy on -client-listen-urls 127.0.0.1:8080 -initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380
+```
+
+#### Start an etcd proxy with the discovery service
+If you bootstrap an etcd cluster using the [discovery service][discovery-service], you can also start the proxy with the same `discovery-url`. 
+
+To start a proxy using the discovery service, specify the `discovery-url` flag. The proxy will wait until the etcd cluster defined at the `discovery-url` finishes bootstrapping, and then start to forward the requests. 
+
+```
+etcd -proxy on -client-listen-urls 127.0.0.1:8080  -discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
+```
+
+#### Fallback to proxy mode with discovery service
+If you bootstrap a etcd cluster using [discovery service][discovery-service] with more than the expected number of etcd members, the extra etcd processes will fall back to being `readwrite` proxies by default. They will forward the requests to the cluster as described above. For example, if you create a discovery url with `size=5`, and start ten etcd processes using that same discovery URL, the result will be a cluster with five etcd members and five proxies. Note that this behaviour can be disabled with the `proxy-fallback` flag.
+
+[discovery-service]: https://github.com/coreos/etcd/blob/master/Documentation/2.0/clustering.md#discovery

Documentation/2.0/runtime-configuration.md

@@ -6,12 +6,16 @@ etcd comes with support for incremental runtime reconfiguration, which allows us
 
 Let us walk through the four use cases for re-configuring a cluster: replacing a member, increasing or decreasing cluster size, and restarting a cluster from a majority failure.
 
-### Replace a Member
+### Replace a Non-recoverable Member
 
-The most common use case of cluster reconfiguration is to replace a member because of a permanent failure of the existing member: for example, hardware failure, loss of network address, or data directory corruption.
+The most common use case of cluster reconfiguration is to replace a member because of a permanent failure of the existing member: for example, hardware failure or data directory corruption.
 It is important to replace failed members as soon as the failure is detected.
 If etcd falls below a simple majority of members it can no longer accept writes: e.g. in a 3 member cluster the loss of two members will cause writes to fail and the cluster to stop operating.
 
+If you want to migrate a running member to another machine, please refer [member migration section][member migration].
+
+[member migration]: https://github.com/coreos/etcd/blob/master/Documentation/2.0/admin_guide.md#member-migration
+
 ### Increase Cluster Size
 
 To make your cluster more resilient to machine failure you can increase the size of the cluster.
@@ -49,7 +53,7 @@ To increase from 3 to 5 members you will make two add operations
 To decrease from 5 to 3 you will make two remove operations
 
 All of these examples will use the `etcdctl` command line tool that ships with etcd.
-If you want to use the member API directly you can find the documentation [here](https://github.com/coreos/etcd/blob/master/Documentation/0.5/other_apis.md).
+If you want to use the member API directly you can find the documentation [here](https://github.com/coreos/etcd/blob/master/Documentation/2.0/other_apis.md).
 
 ### Remove a Member
 
@@ -67,7 +71,7 @@ We then use the `remove` command to perform the removal:
 
 ```
 $ etcdctl member remove a8266ecf031671f3
-EOF
+Removed member a8266ecf031671f3 from cluster
 ```
 
 The target member will stop itself at this point and print out the removal in the log:
@@ -80,24 +84,28 @@ Removal of the leader is safe, but the cluster will be out of progress for a per
 
 ### Add a Member
 
-Adding a member is a two step process.
-First we have to tell the cluster that it should expect a new member to join, then we will need to start the member with the correct configuration.
+Adding a member is a two step process:
 
-Using `etcdctl`, let’s tell the cluster about the new member:
+ * Add the new member to the cluster via the [members API](https://github.com/coreos/etcd/blob/master/Documentation/2.0/other_apis.md#post-v2members) or the `etcdctl member add` command.
+ * Start the member with the correct configuration.
+
+Using `etcdctl` let's add the new member to the cluster:
 
 ```
-$ etcdctl member add infra3 http://10.0.1.13:2379
+$ etcdctl member add infra3 http://10.0.1.13:2380
 added member 9bf1b35fc7761a23 to cluster
 ETCD_NAME="infra3"
-ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379,infra3=http://10.0.1.13:2379"
+ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380,infra3=http://10.0.1.13:2380"
 ETCD_INITIAL_CLUSTER_STATE=existing
 ```
 
+> Notice that infra3 was added to the cluster using its advertised peer URL.
+
 Now start the new etcd process with the relevant flags for the new member:
 
 ```
 $ export ETCD_NAME="infra3"
-$ export ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379,infra3=http://10.0.1.13:2379"
+$ export ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380,infra3=http://10.0.1.13:2380"
 $ export ETCD_INITIAL_CLUSTER_STATE=existing
 $ etcd -listen-client-urls http://10.0.1.13:2379 -advertise-client-urls http://10.0.1.13:2379  -listen-peer-urls http://10.0.1.13:2380 -initial-advertise-peer-urls http://10.0.1.13:2380
 ```
@@ -114,18 +122,18 @@ If this is a new cluster, the node must be added to the list of initial cluster
 
 ```
 $ etcd -name infra3 \
-    -initial-cluster infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379 \
-    -initial-cluster-state existing
+  -initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380 \
+  -initial-cluster-state existing
 etcdserver: assign ids error: the member count is unequal
 exit 1
 ```
 
-In this case we give a different address (10.0.1.14:2379) to the one that we used to join the cluster (10.0.1.13:2379).
+In this case we give a different address (10.0.1.14:2380) to the one that we used to join the cluster (10.0.1.13:2380).
 
 ```
 $ etcd -name infra4 \
-    -initial-cluster infra0=http://10.0.1.10:2379,infra1=http://10.0.1.11:2379,infra2=http://10.0.1.12:2379,infra4=http://10.0.1.14:2379 \
-    -initial-cluster-state existing
+  -initial-cluster infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380,infra4=http://10.0.1.14:2380 \
+  -initial-cluster-state existing
 etcdserver: assign ids error: unmatched member while checking PeerURLs
 exit 1
 ```

Documentation/api.md

@@ -126,10 +126,10 @@ curl -L http://127.0.0.1:4001/v2/keys/message -XPUT -d value="Hello etcd"
         "value": "Hello etcd"
     },
     "prevNode": {
-    	"createdIndex": 2,
-    	"key": "/message",
-    	"value": "Hello world",
-    	"modifiedIndex": 2
+        "createdIndex": 2,
+        "key": "/message",
+        "value": "Hello world",
+        "modifiedIndex": 2
     }
 }
 ```
@@ -153,10 +153,10 @@ curl -L http://127.0.0.1:4001/v2/keys/message -XDELETE
         "modifiedIndex": 4
     },
     "prevNode": {
-    	"key": "/message",
-    	"value": "Hello etcd",
-    	"modifiedIndex": 3,
-    	"createdIndex": 3
+        "key": "/message",
+        "value": "Hello etcd",
+        "modifiedIndex": 3,
+        "createdIndex": 3
     }
 }
 ```
@@ -405,19 +405,19 @@ curl 'http://127.0.0.1:4001/v2/keys/dir/asdf?consistent=true&wait=true'
 
 ```json
 {
-	"action": "expire",
-	"node": {
-		"createdIndex": 8,
-		"key": "/dir",
-		"modifiedIndex": 15
-	},
-	"prevNode": {
-		"createdIndex": 8,
-		"key": "/dir",
-		"dir":true,
-		"modifiedIndex": 17,
-		"expiration": "2013-12-11T10:39:35.689275857-08:00"
-	}
+    "action": "expire",
+    "node": {
+        "createdIndex": 8,
+        "key": "/dir",
+        "modifiedIndex": 15
+    },
+    "prevNode": {
+        "createdIndex": 8,
+        "key": "/dir",
+        "dir":true,
+        "modifiedIndex": 17,
+        "expiration": "2013-12-11T10:39:35.689275857-08:00"
+    }
 }
 ```
 
@@ -499,10 +499,10 @@ The response should be:
         "value": "two"
     },
     "prevNode": {
-    	"createdIndex": 8,
-    	"key": "/foo",
-    	"modifiedIndex": 8,
-    	"value": "one"
+        "createdIndex": 8,
+        "key": "/foo",
+        "modifiedIndex": 8,
+        "value": "one"
     }
 }
 ```
@@ -536,10 +536,10 @@ The error code explains the problem:
 
 ```json
 {
-	"errorCode": 101,
-	"message": "Compare failed",
-	"cause": "[two != one]",
-	"index": 8
+    "errorCode": 101,
+    "message": "Compare failed",
+    "cause": "[two != one]",
+    "index": 8
 }
 ```
 
@@ -551,10 +551,10 @@ curl -L http://127.0.0.1:4001/v2/keys/foo?prevIndex=1 -XDELETE
 
 ```json
 {
-	"errorCode": 101,
-	"message": "Compare failed",
-	"cause": "[1 != 8]",
-	"index": 8
+    "errorCode": 101,
+    "message": "Compare failed",
+    "cause": "[1 != 8]",
+    "index": 8
 }
 ```
 
@@ -568,18 +568,18 @@ The successful response will look something like:
 
 ```json
 {
-	"action": "compareAndDelete",
-	"node": {
-		"key": "/foo",
-		"modifiedIndex": 9,
-		"createdIndex": 8
-	},
-	"prevNode": {
-		"key": "/foo",
-		"value": "one",
-		"modifiedIndex": 8,
-		"createdIndex": 8
-	}
+    "action": "compareAndDelete",
+    "node": {
+        "key": "/foo",
+        "modifiedIndex": 9,
+        "createdIndex": 8
+    },
+    "prevNode": {
+        "key": "/foo",
+        "value": "one",
+        "modifiedIndex": 8,
+        "createdIndex": 8
+    }
 }
 ```
 
@@ -643,22 +643,22 @@ We should see the response as an array of items:
 ```json
 {
     "action": "get",
-    "node": {
-        "key": "/",
-        "dir": true,
-        "nodes": [
-            {
-                "key": "/foo_dir",
-                "dir": true,
-                "modifiedIndex": 2,
-                "createdIndex": 2
-            },
-            {
-                "key": "/foo",
-                "value": "two",
-                "modifiedIndex": 1,
-                "createdIndex": 1
-            }
+    "node": {
+        "key": "/",
+        "dir": true,
+        "nodes": [
+            {
+                "key": "/foo_dir",
+                "dir": true,
+                "modifiedIndex": 2,
+                "createdIndex": 2
+            },
+            {
+                "key": "/foo",
+                "value": "two",
+                "modifiedIndex": 1,
+                "createdIndex": 1
+            }
         ]
     }
 }
@@ -673,68 +673,84 @@ curl -L http://127.0.0.1:4001/v2/keys/?recursive=true
 
 ```json
 {
-    "action": "get",
-    "node": {
-        "key": "/",
-        "dir": true,
-        "nodes": [
-            {
-                "key": "/foo_dir",
-                "dir": true,
-                "nodes": [
-                    {
-                        "key": "/foo_dir/foo",
-                        "value": "bar",
-                        "modifiedIndex": 2,
-                        "createdIndex": 2
-                    }
-                ],
-                "modifiedIndex": 2,
-                "createdIndex": 2
-            },
-            {
-                "key": "/foo",
-                "value": "two",
-                "modifiedIndex": 1,
-                "createdIndex": 1
-            }
-        ]
-    }
+    "action": "get",
+    "node": {
+        "key": "/",
+        "dir": true,
+        "nodes": [
+            {
+                "key": "/foo_dir",
+                "dir": true,
+                "nodes": [
+                    {
+                        "key": "/foo_dir/foo",
+                        "value": "bar",
+                        "modifiedIndex": 2,
+                        "createdIndex": 2
+                    }
+                ],
+                "modifiedIndex": 2,
+                "createdIndex": 2
+            },
+            {
+                "key": "/foo",
+                "value": "two",
+                "modifiedIndex": 1,
+                "createdIndex": 1
+            }
+        ]
+    }
 }
 ```
 
 
 ### Deleting a Directory
 
-Now let's try to delete the directory `/foo_dir`.
+Now let's try to delete the directory `/dir`
 
-You can remove an empty directory using the `DELETE` verb and the `dir=true` parameter.
+You can remove an empty directory using the `DELETE` verb and the `dir=true` parameter. Following will succeed because `/dir` was empty
 
 ```sh
-curl -L 'http://127.0.0.1:4001/v2/keys/foo_dir?dir=true' -XDELETE
+curl -L 'http://127.0.0.1:4001/v2/keys/dir?dir=true' -XDELETE
 ```
+
 ```json
 {
     "action": "delete",
     "node": {
         "createdIndex": 30,
         "dir": true,
-        "key": "/foo_dir",
+        "key": "/dir",
         "modifiedIndex": 31
     },
     "prevNode": {
-    	"createdIndex": 30,
-    	"key": "/foo_dir",
-    	"dir": true,
-    	"modifiedIndex": 30
+        "createdIndex": 30,
+        "key": "/dir",
+        "dir": true,
+        "modifiedIndex": 30
     }
 }
 ```
 
+However, deleting `/foo_dir` will result into an error because `/foo_dir` is not empty.
+
+```sh
+curl -L 'http://127.0.0.1:4001/v2/keys/foo_dir?dir=true' -XDELETE
+```
+
+```json
+{
+    "errorCode":108,
+    "message":"Directory not empty",
+    "cause":"/foo_dir",
+    "index":2
+}
+```
+
 To delete a directory that holds keys, you must add `recursive=true`.
 
 ```sh
-curl -L http://127.0.0.1:4001/v2/keys/dir?recursive=true -XDELETE
+curl -L http://127.0.0.1:4001/v2/keys/foo_dir?recursive=true -XDELETE
 ```
 
 ```json
@@ -747,10 +763,10 @@ curl -L http://127.0.0.1:4001/v2/keys/dir?recursive=true -XDELETE
         "modifiedIndex": 11
     },
     "prevNode": {
-    	"createdIndex": 10,
-    	"dir": true,
-    	"key": "/dir",
-    	"modifiedIndex": 10
+        "createdIndex": 10,
+        "dir": true,
+        "key": "/dir",
+        "modifiedIndex": 10
     }
 }
 ```

Documentation/configuration.md

@@ -31,32 +31,34 @@ The full documentation is contained in the [API docs](https://github.com/coreos/
 ### Optional
 
 * `-addr` - The advertised public hostname:port for client communication. Defaults to `127.0.0.1:4001`.
-* `-discovery` - A URL to use for discovering the peer list. (i.e `"https://discovery.etcd.io/your-unique-key"`).
-* `-http-read-timeout` - The number of seconds before an HTTP read operation is timed out.
-* `-http-write-timeout` - The number of seconds before an HTTP write operation is timed out.
 * `-bind-addr` - The listening hostname for client communication. Defaults to advertised IP.
-* `-peers` - A comma separated list of peers in the cluster (i.e `"203.0.113.101:7001,203.0.113.102:7001"`).
-* `-peers-file` - The file path containing a comma separated list of peers in the cluster.
 * `-ca-file` - The path of the client CAFile. Enables client cert authentication when present.
 * `-cert-file` - The cert file of the client.
-* `-key-file` - The key file of the client.
+* `-cluster-active-size` - The expected number of instances participating in the consensus protocol. Only applied if the etcd instance is the first peer in the cluster.
+* `-cluster-remove-delay` - The number of seconds before one node is removed from the cluster since it cannot be connected at all. Only applied if the etcd instance is the first peer in the cluster.
+* `-cluster-sync-interval` - The number of seconds between synchronization for standby-mode instance with the cluster. Only applied if the etcd instance is the first peer in the cluster.
 * `-config` - The path of the etcd configuration file. Defaults to `/etc/etcd/etcd.conf`.
 * `-cors` - A comma separated white list of origins for cross-origin resource sharing.
 * `-cpuprofile` - The path to a file to output CPU profile data. Enables CPU profiling when present.
 * `-data-dir` - The directory to store log and snapshot. Defaults to the current working directory.
+* `-discovery` - A URL to use for discovering the peer list. (i.e `"https://discovery.etcd.io/your-unique-key"`).
+* `-graphite-host` - The Graphite endpoint to which to send metrics.
+* `-http-read-timeout` - The number of seconds before an HTTP read operation is timed out.
+* `-http-write-timeout` - The number of seconds before an HTTP write operation is timed out.
+* `-key-file` - The key file of the client.
 * `-max-result-buffer` - The max size of result buffer. Defaults to `1024`.
 * `-max-retry-attempts` - The max retry attempts when trying to join a cluster. Defaults to `3`.
 * `-peer-addr` - The advertised public hostname:port for server communication. Defaults to `127.0.0.1:7001`.
 * `-peer-bind-addr` - The listening hostname for server communication. Defaults to advertised IP.
 * `-peer-ca-file` - The path of the CAFile. Enables client/peer cert authentication when present.
 * `-peer-cert-file` - The cert file of the server.
-* `-peer-key-file` - The key file of the server.
 * `-peer-election-timeout` - The number of milliseconds to wait before the leader is declared unhealthy.
 * `-peer-heartbeat-interval` - The number of milliseconds in between heartbeat requests
+* `-peer-key-file` - The key file of the server.
+* `-peers` - A comma separated list of peers in the cluster (i.e `"203.0.113.101:7001,203.0.113.102:7001"`).
+* `-peers-file` - The file path containing a comma separated list of peers in the cluster.
+* `-retry-interval` - Seconds to wait between cluster join retry attempts.
 * `-snapshot=false` - Disable log snapshots. Defaults to `true`.
-* `-cluster-active-size` - The expected number of instances participating in the consensus protocol. Only applied if the etcd instance is the first peer in the cluster.
-* `-cluster-remove-delay` - The number of seconds before one node is removed from the cluster since it cannot be connected at all. Only applied if the etcd instance is the first peer in the cluster.
-* `-cluster-sync-interval` - The number of seconds between synchronization for standby-mode instance with the cluster. Only applied if the etcd instance is the first peer in the cluster.
 * `-v` - Enable verbose logging. Defaults to `false`.
 * `-vv` - Enable very verbose logging. Defaults to `false`.
 * `-version` - Print the version and exit.
@@ -80,7 +82,6 @@ http_write_timeout = 10.0
 key_file = ""
 peers = []
 peers_file = ""
-max_cluster_size = 9
 max_result_buffer = 1024
 max_retry_attempts = 3
 name = "default-name"
@@ -107,29 +108,32 @@ sync_interval = 5.0
  * `ETCD_BIND_ADDR`
  * `ETCD_CA_FILE`
  * `ETCD_CERT_FILE`
- * `ETCD_CORS_ORIGINS`
- * `ETCD_CONFIG`
- * `ETCD_CPU_PROFILE_FILE`
+ * `ETCD_CLUSTER_ACTIVE_SIZE`
+ * `ETCD_CLUSTER_REMOVE_DELAY`
+ * `ETCD_CLUSTER_SYNC_INTERVAL`
+ * `ETCD_CORS`
  * `ETCD_DATA_DIR`
  * `ETCD_DISCOVERY`
+ * `ETCD_GRAPHITE_HOST`
  * `ETCD_HTTP_READ_TIMEOUT`
  * `ETCD_HTTP_WRITE_TIMEOUT`
  * `ETCD_KEY_FILE`
- * `ETCD_PEERS`
- * `ETCD_PEERS_FILE`
- * `ETCD_MAX_CLUSTER_SIZE`
  * `ETCD_MAX_RESULT_BUFFER`
  * `ETCD_MAX_RETRY_ATTEMPTS`
  * `ETCD_NAME`
- * `ETCD_SNAPSHOT`
- * `ETCD_VERBOSE`
- * `ETCD_VERY_VERBOSE`
  * `ETCD_PEER_ADDR`
  * `ETCD_PEER_BIND_ADDR`
  * `ETCD_PEER_CA_FILE`
  * `ETCD_PEER_CERT_FILE`
- * `ETCD_PEER_KEY_FILE`
  * `ETCD_PEER_ELECTION_TIMEOUT`
- * `ETCD_CLUSTER_ACTIVE_SIZE`
- * `ETCD_CLUSTER_REMOVE_DELAY`
- * `ETCD_CLUSTER_SYNC_INTERVAL`
+ * `ETCD_PEER_HEARTBEAT_INTERVAL`
+ * `ETCD_PEER_KEY_FILE`
+ * `ETCD_PEERS`
+ * `ETCD_PEERS_FILE`
+ * `ETCD_RETRY_INTERVAL`
+ * `ETCD_SNAPSHOT`
+ * `ETCD_SNAPSHOTCOUNT`
+ * `ETCD_TRACE`
+ * `ETCD_VERBOSE`
+ * `ETCD_VERY_VERBOSE`
+ * `ETCD_VERY_VERY_VERBOSE`

Documentation/libraries-and-tools.md

@@ -7,6 +7,7 @@
 - [etcd-dump](https://npmjs.org/package/etcd-dump) - Command line utility for dumping/restoring etcd.
 - [etcd-fs](https://github.com/xetorthio/etcd-fs) - FUSE filesystem for etcd
 - [etcd-browser](https://github.com/henszey/etcd-browser) - A web-based key/value editor for etcd using AngularJS
+- [etcd-lock](https://github.com/datawisesystems/etcd-lock) - A lock implementation for etcd
 
 **Go libraries**
 
@@ -26,6 +27,7 @@
 - [jplana/python-etcd](https://github.com/jplana/python-etcd) - Supports v2
 - [russellhaering/txetcd](https://github.com/russellhaering/txetcd) - a Twisted Python library
 - [cholcombe973/autodock](https://github.com/cholcombe973/autodock) - A docker deployment automation tool
+- [lisael/aioetcd](https://github.com/lisael/aioetcd) - (Python 3.4+) Asyncio coroutines client (Supports v2)
 
 **Node libraries**
 

Documentation/modules.md

@@ -3,7 +3,7 @@
 etcd has a number of modules that are built on top of the core etcd API.
 These modules provide things like dashboards, locks and leader election (removed).
 
-**Warning**: Modules are deprecated from v0.4 until we have a solid base we can apply them back onto.
+**Warning**: Modules and dashboard are deprecated from v0.4 until we have a solid base we can apply them back onto.
 For now, we are choosing to focus on raft algorithm and core etcd to make sure that it works correctly and fast.
 And it is time consuming to maintain these modules in this period, given that etcd's API changes from time to time.
 Moreover, the lock module has some unfixed bugs, which may mislead users.
@@ -11,6 +11,8 @@ But we also notice that these modules are popular and useful, and plan to add th
 
 ### Dashboard
 
+**Other Dashboards**: There are other dashboards available on [Github](https://github.com/henszey/etcd-browser) that can be run [in a container](https://registry.hub.docker.com/u/tomaskral/etcd-browser/).
+
 An HTML dashboard can be found at `http://127.0.0.1:4001/mod/dashboard/`.
 This dashboard is compiled into the etcd binary and uses the same API as regular etcd clients.
 

Documentation/security.md

@@ -124,7 +124,7 @@ DISCOVERY_URL=... # from https://discovery.etcd.io/new
 etcd -name node1 -data-dir node1 -ca-file=/path/to/ca.crt -cert-file=/path/to/node1.crt -key-file=/path/to/node1.key -peer-addr ${node1_public_ip}:7001 -discovery ${DISCOVERY_URL}
 
 # Node2
-etcd -name node1 -data-dir node2 -ca-file=/path/to/ca.crt -cert-file=/path/to/node2.crt -key-file=/path/to/node2.key -peer-addr ${node2_public_ip}:7001 -discovery ${DISCOVERY_URL}
+etcd -name node2 -data-dir node2 -ca-file=/path/to/ca.crt -cert-file=/path/to/node2.crt -key-file=/path/to/node2.key -peer-addr ${node2_public_ip}:7001 -discovery ${DISCOVERY_URL}
 ```
 
 The etcd nodes will form a cluster and all communication between nodes in the cluster will be encrypted and authenticated using the client certificates. You will see in the output of etcd that the addresses it connects to use HTTPS.

Godeps/Godeps.json

@@ -5,11 +5,6 @@
 		"./..."
 	],
 	"Deps": [
-		{
-			"ImportPath": "code.google.com/p/go.net/context",
-			"Comment": "null-144",
-			"Rev": "ad01a6fcc8a19d3a4478c836895ffe883bd2ceab"
-		},
 		{
 			"ImportPath": "code.google.com/p/gogoprotobuf/proto",
 			"Rev": "7fd1620f09261338b6b1ca1289ace83aee0ec946"
@@ -31,6 +26,11 @@
 		{
 			"ImportPath": "github.com/stretchr/testify/assert",
 			"Rev": "9cc77fa25329013ce07362c7742952ff887361f2"
+		},
+		{
+			"ImportPath": "golang.org/x/net/context",
+			"Comment": "null-220",
+			"Rev": "c5a46024776ec35eb562fa9226968b9d543bb13a"
 		}
 	]
 }

Godeps/_workspace/src/github.com/codegangsta/cli/app_test.go

@@ -5,7 +5,7 @@ import (
 	"os"
 	"testing"
 
-	"github.com/codegangsta/cli"
+	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
 )
 
 func ExampleApp() {

Godeps/_workspace/src/github.com/codegangsta/cli/cli_test.go

@@ -3,7 +3,7 @@ package cli_test
 import (
 	"os"
 
-	"github.com/codegangsta/cli"
+	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
 )
 
 func Example() {

Godeps/_workspace/src/github.com/codegangsta/cli/command_test.go

@@ -4,7 +4,7 @@ import (
 	"flag"
 	"testing"
 
-	"github.com/codegangsta/cli"
+	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
 )
 
 func TestCommandDoNotIgnoreFlags(t *testing.T) {

Godeps/_workspace/src/github.com/codegangsta/cli/context_test.go

@@ -5,7 +5,7 @@ import (
 	"testing"
 	"time"
 
-	"github.com/codegangsta/cli"
+	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
 )
 
 func TestNewContext(t *testing.T) {

Godeps/_workspace/src/github.com/codegangsta/cli/flag_test.go

@@ -7,7 +7,7 @@ import (
 	"strings"
 	"testing"
 
-	"github.com/codegangsta/cli"
+	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
 )
 
 var boolFlagTests = []struct {

Godeps/_workspace/src/golang.org/x/net/context/context.go

@@ -108,7 +108,7 @@ type Context interface {
 	// 	// Package user defines a User type that's stored in Contexts.
 	// 	package user
 	//
-	// 	import "code.google.com/p/go.net/context"
+	// 	import "golang.org/x/net/context"
 	//
 	// 	// User is the type of value stored in the Contexts.
 	// 	type User struct {...}
@@ -124,7 +124,7 @@ type Context interface {
 	//
 	// 	// NewContext returns a new Context that carries value u.
 	// 	func NewContext(ctx context.Context, u *User) context.Context {
-	// 		return context.WithValue(userKey, u)
+	// 		return context.WithValue(ctx, userKey, u)
 	// 	}
 	//
 	// 	// FromContext returns the User value stored in ctx, if any.
@@ -142,27 +142,28 @@ var Canceled = errors.New("context canceled")
 // deadline passes.
 var DeadlineExceeded = errors.New("context deadline exceeded")
 
-// An emptyCtx is never canceled, has no values, and has no deadline.
+// An emptyCtx is never canceled, has no values, and has no deadline.  It is not
+// struct{}, since vars of this type must have distinct addresses.
 type emptyCtx int
 
-func (emptyCtx) Deadline() (deadline time.Time, ok bool) {
+func (*emptyCtx) Deadline() (deadline time.Time, ok bool) {
 	return
 }
 
-func (emptyCtx) Done() <-chan struct{} {
+func (*emptyCtx) Done() <-chan struct{} {
 	return nil
 }
 
-func (emptyCtx) Err() error {
+func (*emptyCtx) Err() error {
 	return nil
 }
 
-func (emptyCtx) Value(key interface{}) interface{} {
+func (*emptyCtx) Value(key interface{}) interface{} {
 	return nil
 }
 
-func (n emptyCtx) String() string {
-	switch n {
+func (e *emptyCtx) String() string {
+	switch e {
 	case background:
 		return "context.Background"
 	case todo:
@@ -171,9 +172,9 @@ func (n emptyCtx) String() string {
 	return "unknown empty Context"
 }
 
-const (
-	background emptyCtx = 1
-	todo       emptyCtx = 2
+var (
+	background = new(emptyCtx)
+	todo       = new(emptyCtx)
 )
 
 // Background returns a non-nil, empty Context. It is never canceled, has no

Godeps/_workspace/src/golang.org/x/net/context/context_test.go

@@ -365,7 +365,7 @@ func TestAllocs(t *testing.T) {
 				c := WithValue(bg, k1, nil)
 				c.Value(k1)
 			},
-			limit:      1,
+			limit:      3,
 			gccgoLimit: 3,
 		},
 		{

Godeps/_workspace/src/golang.org/x/net/context/withtimeout_test.go

@@ -8,7 +8,7 @@ import (
 	"fmt"
 	"time"
 
-	"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 )
 
 func ExampleWithTimeout() {

Procfile

@@ -1,5 +1,5 @@
 # Use goreman to run `go get github.com/mattn/goreman`
-etcd1: bin/etcd -name infra1 -listen-client-urls http://127.0.0.1:4001 -advertise-client-urls http://127.0.0.1:4001 -listen-peer-urls http://127.0.0.1:7001 -initial-advertise-peer-urls http://127.0.0.1:7001 -initial-cluster-token etcd-cluster-1 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003' -initial-cluster-state new
-etcd2: bin/etcd -name infra2 -listen-client-urls http://127.0.0.1:4002 -advertise-client-urls http://127.0.0.1:4002 -listen-peer-urls http://127.0.0.1:7002 -initial-advertise-peer-urls http://127.0.0.1:7002 -initial-cluster-token etcd-cluster-1 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003' -initial-cluster-state new
-etcd3: bin/etcd -name infra3 -listen-client-urls http://127.0.0.1:4003 -advertise-client-urls http://127.0.0.1:4003 -listen-peer-urls http://127.0.0.1:7003 -initial-advertise-peer-urls http://127.0.0.1:7003 -initial-cluster-token etcd-cluster-1 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003' -initial-cluster-state new
+etcd1: bin/etcd -name infra1 -listen-client-urls http://localhost:4001 -advertise-client-urls http://localhost:4001 -listen-peer-urls http://localhost:7001 -initial-advertise-peer-urls http://localhost:7001 -initial-cluster-token etcd-cluster-1 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003' -initial-cluster-state new
+etcd2: bin/etcd -name infra2 -listen-client-urls http://localhost:4002 -advertise-client-urls http://localhost:4002 -listen-peer-urls http://localhost:7002 -initial-advertise-peer-urls http://localhost:7002 -initial-cluster-token etcd-cluster-1 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003' -initial-cluster-state new
+etcd3: bin/etcd -name infra3 -listen-client-urls http://localhost:4003 -advertise-client-urls http://localhost:4003 -listen-peer-urls http://localhost:7003 -initial-advertise-peer-urls http://localhost:7003 -initial-cluster-token etcd-cluster-1 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003' -initial-cluster-state new
 proxy: bin/etcd -proxy=on -bind-addr 127.0.0.1:8080 -initial-cluster 'infra1=http://localhost:7001,infra2=http://localhost:7002,infra3=http://localhost:7003'

README.md

@@ -3,13 +3,14 @@
 [![Build Status](https://travis-ci.org/coreos/etcd.png?branch=master)](https://travis-ci.org/coreos/etcd)
 [![Docker Repository on Quay.io](https://quay.io/repository/coreos/etcd-git/status "Docker Repository on Quay.io")](https://quay.io/repository/coreos/etcd-git)
 
-### WARNING ###
+### Release Candidate Warning ###
 
-The current `master` branch of etcd is under heavy development in anticipation of the forthcoming 0.5.0 release. 
+The current `master` branch of etcd is under development in anticipation of the forthcoming 2.0.0 release.
 
 It is strongly recommended that users work with the latest 0.4.x release (0.4.6), which can be found on the [releases](https://github.com/coreos/etcd/releases) page.
 
 Unless otherwise noted, the etcd documentation refers to configuring and running 0.4.x releases.
+Documentation related to the 2.0.0 release candidates can be found in the `Documentation/2.0` directory.
 
 ## README version 0.4.6
 

build

@@ -12,4 +12,5 @@ ln -s ${PWD} $GOPATH/src/${REPO_PATH}
 eval $(go env)
 
 go build -o bin/etcd ${REPO_PATH}
-go build -o bin/etcdctl ${REPO_PATH}/etcdctl
+CGO_ENABLED=0 go build -a -ldflags '-s' -o bin/etcdctl ${REPO_PATH}/etcdctl
+go build -o bin/etcd-migrate ${REPO_PATH}/migrate/cmd/etcd-migrate

client/cluster.go

@@ -1,53 +0,0 @@
-/*
-   Copyright 2014 CoreOS, Inc.
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
-*/
-
-package client
-
-import (
-	"net/http"
-	"net/url"
-
-	"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
-)
-
-func newHTTPClusterClient(tr *http.Transport, eps []string) (*httpClusterClient, error) {
-	c := httpClusterClient{
-		endpoints: make([]*httpClient, len(eps)),
-	}
-
-	for i, ep := range eps {
-		u, err := url.Parse(ep)
-		if err != nil {
-			return nil, err
-		}
-
-		c.endpoints[i] = &httpClient{
-			transport: tr,
-			endpoint:  *u,
-		}
-	}
-
-	return &c, nil
-}
-
-type httpClusterClient struct {
-	endpoints []*httpClient
-}
-
-func (c *httpClusterClient) do(ctx context.Context, act httpAction) (int, []byte, error) {
-	//TODO(bcwaldon): introduce retry logic so all endpoints are attempted
-	return c.endpoints[0].do(ctx, act)
-}

client/http.go

@@ -17,33 +17,128 @@
 package client
 
 import (
+	"errors"
+	"fmt"
 	"io/ioutil"
 	"net/http"
 	"net/url"
 	"time"
 
-	"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 )
 
 var (
-	ErrTimeout            = context.DeadlineExceeded
+	ErrTimeout          = context.DeadlineExceeded
+	ErrCanceled         = context.Canceled
+	ErrNoEndpoints      = errors.New("no endpoints available")
+	ErrTooManyRedirects = errors.New("too many redirects")
+
 	DefaultRequestTimeout = 5 * time.Second
+	DefaultMaxRedirects   = 10
 )
 
-// transport mimics http.Transport to provide an interface which can be
+type SyncableHTTPClient interface {
+	HTTPClient
+	Sync(context.Context) error
+	Endpoints() []string
+}
+
+type HTTPClient interface {
+	Do(context.Context, HTTPAction) (*http.Response, []byte, error)
+}
+
+type HTTPAction interface {
+	HTTPRequest(url.URL) *http.Request
+}
+
+// CancelableTransport mimics http.Transport to provide an interface which can be
 // substituted for testing (since the RoundTripper interface alone does not
 // require the CancelRequest method)
-type transport interface {
+type CancelableTransport interface {
 	http.RoundTripper
 	CancelRequest(req *http.Request)
 }
 
-type httpAction interface {
-	httpRequest(url.URL) *http.Request
+func NewHTTPClient(tr CancelableTransport, eps []string) (SyncableHTTPClient, error) {
+	return newHTTPClusterClient(tr, eps)
 }
 
-type httpActionDo interface {
-	do(context.Context, httpAction) (int, []byte, error)
+func newHTTPClusterClient(tr CancelableTransport, eps []string) (*httpClusterClient, error) {
+	c := httpClusterClient{
+		transport: tr,
+		endpoints: eps,
+		clients:   make([]HTTPClient, len(eps)),
+	}
+
+	for i, ep := range eps {
+		u, err := url.Parse(ep)
+		if err != nil {
+			return nil, err
+		}
+
+		c.clients[i] = &redirectFollowingHTTPClient{
+			max: DefaultMaxRedirects,
+			client: &httpClient{
+				transport: tr,
+				endpoint:  *u,
+			},
+		}
+	}
+
+	return &c, nil
+}
+
+type httpClusterClient struct {
+	transport CancelableTransport
+	endpoints []string
+	clients   []HTTPClient
+}
+
+func (c *httpClusterClient) Do(ctx context.Context, act HTTPAction) (resp *http.Response, body []byte, err error) {
+	if len(c.clients) == 0 {
+		return nil, nil, ErrNoEndpoints
+	}
+	for _, hc := range c.clients {
+		resp, body, err = hc.Do(ctx, act)
+		if err != nil {
+			if err == ErrTimeout || err == ErrCanceled {
+				return nil, nil, err
+			}
+			continue
+		}
+		if resp.StatusCode/100 == 5 {
+			continue
+		}
+		break
+	}
+	return
+}
+
+func (c *httpClusterClient) Endpoints() []string {
+	return c.endpoints
+}
+
+func (c *httpClusterClient) Sync(ctx context.Context) error {
+	mAPI := NewMembersAPI(c)
+	ms, err := mAPI.List(ctx)
+	if err != nil {
+		return err
+	}
+
+	eps := make([]string, 0)
+	for _, m := range ms {
+		eps = append(eps, m.ClientURLs...)
+	}
+	if len(eps) == 0 {
+		return ErrNoEndpoints
+	}
+	nc, err := newHTTPClusterClient(c.transport, eps)
+	if err != nil {
+		return err
+	}
+
+	*c = *nc
+	return nil
 }
 
 type roundTripResponse struct {
@@ -52,13 +147,12 @@ type roundTripResponse struct {
 }
 
 type httpClient struct {
-	transport transport
+	transport CancelableTransport
 	endpoint  url.URL
-	timeout   time.Duration
 }
 
-func (c *httpClient) do(ctx context.Context, act httpAction) (int, []byte, error) {
-	req := act.httpRequest(c.endpoint)
+func (c *httpClient) Do(ctx context.Context, act HTTPAction) (*http.Response, []byte, error) {
+	req := act.HTTPRequest(c.endpoint)
 
 	rtchan := make(chan roundTripResponse, 1)
 	go func() {
@@ -89,9 +183,51 @@ func (c *httpClient) do(ctx context.Context, act httpAction) (int, []byte, error
 	}()
 
 	if err != nil {
-		return 0, nil, err
+		return nil, nil, err
 	}
 
 	body, err := ioutil.ReadAll(resp.Body)
-	return resp.StatusCode, body, err
+	return resp, body, err
+}
+
+type redirectFollowingHTTPClient struct {
+	client HTTPClient
+	max    int
+}
+
+func (r *redirectFollowingHTTPClient) Do(ctx context.Context, act HTTPAction) (*http.Response, []byte, error) {
+	for i := 0; i <= r.max; i++ {
+		resp, body, err := r.client.Do(ctx, act)
+		if err != nil {
+			return nil, nil, err
+		}
+		if resp.StatusCode/100 == 3 {
+			hdr := resp.Header.Get("Location")
+			if hdr == "" {
+				return nil, nil, fmt.Errorf("Location header not set")
+			}
+			loc, err := url.Parse(hdr)
+			if err != nil {
+				return nil, nil, fmt.Errorf("Location header not valid URL: %s", hdr)
+			}
+			act = &redirectedHTTPAction{
+				action:   act,
+				location: *loc,
+			}
+			continue
+		}
+		return resp, body, nil
+	}
+	return nil, nil, ErrTooManyRedirects
+}
+
+type redirectedHTTPAction struct {
+	action   HTTPAction
+	location url.URL
+}
+
+func (r *redirectedHTTPAction) HTTPRequest(ep url.URL) *http.Request {
+	orig := r.action.HTTPRequest(ep)
+	orig.URL = &r.location
+	return orig
 }

client/http_test.go

@@ -26,9 +26,42 @@ import (
 	"testing"
 	"time"
 
-	"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 )
 
+type staticHTTPClient struct {
+	resp http.Response
+	err  error
+}
+
+func (s *staticHTTPClient) Do(context.Context, HTTPAction) (*http.Response, []byte, error) {
+	return &s.resp, nil, s.err
+}
+
+type staticHTTPAction struct {
+	request http.Request
+}
+
+type staticHTTPResponse struct {
+	resp http.Response
+	err  error
+}
+
+func (s *staticHTTPAction) HTTPRequest(url.URL) *http.Request {
+	return &s.request
+}
+
+type multiStaticHTTPClient struct {
+	responses []staticHTTPResponse
+	cur       int
+}
+
+func (s *multiStaticHTTPClient) Do(context.Context, HTTPAction) (*http.Response, []byte, error) {
+	r := s.responses[s.cur]
+	s.cur++
+	return &r.resp, nil, r.err
+}
+
 type fakeTransport struct {
 	respchan     chan *http.Response
 	errchan      chan error
@@ -65,7 +98,7 @@ func (t *fakeTransport) CancelRequest(*http.Request) {
 
 type fakeAction struct{}
 
-func (a *fakeAction) httpRequest(url.URL) *http.Request {
+func (a *fakeAction) HTTPRequest(url.URL) *http.Request {
 	return &http.Request{}
 }
 
@@ -78,14 +111,14 @@ func TestHTTPClientDoSuccess(t *testing.T) {
 		Body:       ioutil.NopCloser(strings.NewReader("foo")),
 	}
 
-	code, body, err := c.do(context.Background(), &fakeAction{})
+	resp, body, err := c.Do(context.Background(), &fakeAction{})
 	if err != nil {
 		t.Fatalf("incorrect error value: want=nil got=%v", err)
 	}
 
 	wantCode := http.StatusTeapot
-	if wantCode != code {
-		t.Fatalf("invalid response code: want=%d got=%d", wantCode, code)
+	if wantCode != resp.StatusCode {
+		t.Fatalf("invalid response code: want=%d got=%d", wantCode, resp.StatusCode)
 	}
 
 	wantBody := []byte("foo")
@@ -100,7 +133,7 @@ func TestHTTPClientDoError(t *testing.T) {
 
 	tr.errchan <- errors.New("fixture")
 
-	_, _, err := c.do(context.Background(), &fakeAction{})
+	_, _, err := c.Do(context.Background(), &fakeAction{})
 	if err == nil {
 		t.Fatalf("expected non-nil error, got nil")
 	}
@@ -113,7 +146,7 @@ func TestHTTPClientDoCancelContext(t *testing.T) {
 	tr.startCancel <- struct{}{}
 	tr.finishCancel <- struct{}{}
 
-	_, _, err := c.do(context.Background(), &fakeAction{})
+	_, _, err := c.Do(context.Background(), &fakeAction{})
 	if err == nil {
 		t.Fatalf("expected non-nil error, got nil")
 	}
@@ -126,7 +159,7 @@ func TestHTTPClientDoCancelContextWaitForRoundTrip(t *testing.T) {
 	donechan := make(chan struct{})
 	ctx, cancel := context.WithCancel(context.Background())
 	go func() {
-		c.do(ctx, &fakeAction{})
+		c.Do(ctx, &fakeAction{})
 		close(donechan)
 	}()
 
@@ -149,3 +182,304 @@ func TestHTTPClientDoCancelContextWaitForRoundTrip(t *testing.T) {
 		t.Fatalf("httpClient.do did not exit within 1s")
 	}
 }
+
+func TestHTTPClusterClientDo(t *testing.T) {
+	fakeErr := errors.New("fake!")
+	tests := []struct {
+		client   *httpClusterClient
+		wantCode int
+		wantErr  error
+	}{
+		// first good response short-circuits Do
+		{
+			client: &httpClusterClient{
+				clients: []HTTPClient{
+					&staticHTTPClient{resp: http.Response{StatusCode: http.StatusTeapot}},
+					&staticHTTPClient{err: fakeErr},
+				},
+			},
+			wantCode: http.StatusTeapot,
+		},
+
+		// fall through to good endpoint if err is arbitrary
+		{
+			client: &httpClusterClient{
+				clients: []HTTPClient{
+					&staticHTTPClient{err: fakeErr},
+					&staticHTTPClient{resp: http.Response{StatusCode: http.StatusTeapot}},
+				},
+			},
+			wantCode: http.StatusTeapot,
+		},
+
+		// ErrTimeout short-circuits Do
+		{
+			client: &httpClusterClient{
+				clients: []HTTPClient{
+					&staticHTTPClient{err: ErrTimeout},
+					&staticHTTPClient{resp: http.Response{StatusCode: http.StatusTeapot}},
+				},
+			},
+			wantErr: ErrTimeout,
+		},
+
+		// ErrCanceled short-circuits Do
+		{
+			client: &httpClusterClient{
+				clients: []HTTPClient{
+					&staticHTTPClient{err: ErrCanceled},
+					&staticHTTPClient{resp: http.Response{StatusCode: http.StatusTeapot}},
+				},
+			},
+			wantErr: ErrCanceled,
+		},
+
+		// return err if there are no endpoints
+		{
+			client: &httpClusterClient{
+				clients: []HTTPClient{},
+			},
+			wantErr: ErrNoEndpoints,
+		},
+
+		// return err if all endpoints return arbitrary errors
+		{
+			client: &httpClusterClient{
+				clients: []HTTPClient{
+					&staticHTTPClient{err: fakeErr},
+					&staticHTTPClient{err: fakeErr},
+				},
+			},
+			wantErr: fakeErr,
+		},
+
+		// 500-level errors cause Do to fallthrough to next endpoint
+		{
+			client: &httpClusterClient{
+				clients: []HTTPClient{
+					&staticHTTPClient{resp: http.Response{StatusCode: http.StatusBadGateway}},
+					&staticHTTPClient{resp: http.Response{StatusCode: http.StatusTeapot}},
+				},
+			},
+			wantCode: http.StatusTeapot,
+		},
+	}
+
+	for i, tt := range tests {
+		resp, _, err := tt.client.Do(context.Background(), nil)
+		if !reflect.DeepEqual(tt.wantErr, err) {
+			t.Errorf("#%d: got err=%v, want=%v", i, err, tt.wantErr)
+			continue
+		}
+
+		if resp == nil {
+			if tt.wantCode != 0 {
+				t.Errorf("#%d: resp is nil, want=%d", i, tt.wantCode)
+			}
+			continue
+		}
+
+		if resp.StatusCode != tt.wantCode {
+			t.Errorf("#%d: resp code=%d, want=%d", i, resp.StatusCode, tt.wantCode)
+			continue
+		}
+	}
+}
+
+func TestRedirectedHTTPAction(t *testing.T) {
+	act := &redirectedHTTPAction{
+		action: &staticHTTPAction{
+			request: http.Request{
+				Method: "DELETE",
+				URL: &url.URL{
+					Scheme: "https",
+					Host:   "foo.example.com",
+					Path:   "/ping",
+				},
+			},
+		},
+		location: url.URL{
+			Scheme: "https",
+			Host:   "bar.example.com",
+			Path:   "/pong",
+		},
+	}
+
+	want := &http.Request{
+		Method: "DELETE",
+		URL: &url.URL{
+			Scheme: "https",
+			Host:   "bar.example.com",
+			Path:   "/pong",
+		},
+	}
+	got := act.HTTPRequest(url.URL{Scheme: "http", Host: "baz.example.com", Path: "/pang"})
+
+	if !reflect.DeepEqual(want, got) {
+		t.Fatalf("HTTPRequest is %#v, want %#v", want, got)
+	}
+}
+
+func TestRedirectFollowingHTTPClient(t *testing.T) {
+	tests := []struct {
+		max      int
+		client   HTTPClient
+		wantCode int
+		wantErr  error
+	}{
+		// errors bubbled up
+		{
+			max: 2,
+			client: &multiStaticHTTPClient{
+				responses: []staticHTTPResponse{
+					staticHTTPResponse{
+						err: errors.New("fail!"),
+					},
+				},
+			},
+			wantErr: errors.New("fail!"),
+		},
+
+		// no need to follow redirect if none given
+		{
+			max: 2,
+			client: &multiStaticHTTPClient{
+				responses: []staticHTTPResponse{
+					staticHTTPResponse{
+						resp: http.Response{
+							StatusCode: http.StatusTeapot,
+						},
+					},
+				},
+			},
+			wantCode: http.StatusTeapot,
+		},
+
+		// redirects if less than max
+		{
+			max: 2,
+			client: &multiStaticHTTPClient{
+				responses: []staticHTTPResponse{
+					staticHTTPResponse{
+						resp: http.Response{
+							StatusCode: http.StatusTemporaryRedirect,
+							Header:     http.Header{"Location": []string{"http://example.com"}},
+						},
+					},
+					staticHTTPResponse{
+						resp: http.Response{
+							StatusCode: http.StatusTeapot,
+						},
+					},
+				},
+			},
+			wantCode: http.StatusTeapot,
+		},
+
+		// succeed after reaching max redirects
+		{
+			max: 2,
+			client: &multiStaticHTTPClient{
+				responses: []staticHTTPResponse{
+					staticHTTPResponse{
+						resp: http.Response{
+							StatusCode: http.StatusTemporaryRedirect,
+							Header:     http.Header{"Location": []string{"http://example.com"}},
+						},
+					},
+					staticHTTPResponse{
+						resp: http.Response{
+							StatusCode: http.StatusTemporaryRedirect,
+							Header:     http.Header{"Location": []string{"http://example.com"}},
+						},
+					},
+					staticHTTPResponse{
+						resp: http.Response{
+							StatusCode: http.StatusTeapot,
+						},
+					},
+				},
+			},
+			wantCode: http.StatusTeapot,
+		},
+
+		// fail at max+1 redirects
+		{
+			max: 1,
+			client: &multiStaticHTTPClient{
+				responses: []staticHTTPResponse{
+					staticHTTPResponse{
+						resp: http.Response{
+							StatusCode: http.StatusTemporaryRedirect,
+							Header:     http.Header{"Location": []string{"http://example.com"}},
+						},
+					},
+					staticHTTPResponse{
+						resp: http.Response{
+							StatusCode: http.StatusTemporaryRedirect,
+							Header:     http.Header{"Location": []string{"http://example.com"}},
+						},
+					},
+					staticHTTPResponse{
+						resp: http.Response{
+							StatusCode: http.StatusTeapot,
+						},
+					},
+				},
+			},
+			wantErr: ErrTooManyRedirects,
+		},
+
+		// fail if Location header not set
+		{
+			max: 1,
+			client: &multiStaticHTTPClient{
+				responses: []staticHTTPResponse{
+					staticHTTPResponse{
+						resp: http.Response{
+							StatusCode: http.StatusTemporaryRedirect,
+						},
+					},
+				},
+			},
+			wantErr: errors.New("Location header not set"),
+		},
+
+		// fail if Location header is invalid
+		{
+			max: 1,
+			client: &multiStaticHTTPClient{
+				responses: []staticHTTPResponse{
+					staticHTTPResponse{
+						resp: http.Response{
+							StatusCode: http.StatusTemporaryRedirect,
+							Header:     http.Header{"Location": []string{":"}},
+						},
+					},
+				},
+			},
+			wantErr: errors.New("Location header not valid URL: :"),
+		},
+	}
+
+	for i, tt := range tests {
+		client := &redirectFollowingHTTPClient{client: tt.client, max: tt.max}
+		resp, _, err := client.Do(context.Background(), nil)
+		if !reflect.DeepEqual(tt.wantErr, err) {
+			t.Errorf("#%d: got err=%v, want=%v", i, err, tt.wantErr)
+			continue
+		}
+
+		if resp == nil {
+			if tt.wantCode != 0 {
+				t.Errorf("#%d: resp is nil, want=%d", i, tt.wantCode)
+			}
+			continue
+		}
+
+		if resp.StatusCode != tt.wantCode {
+			t.Errorf("#%d: resp code=%d, want=%d", i, resp.StatusCode, tt.wantCode)
+			continue
+		}
+	}
+}

client/keys.go

@@ -27,7 +27,7 @@ import (
 	"strings"
 	"time"
 
-	"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 )
 
 var (
@@ -41,44 +41,37 @@ var (
 	ErrKeyExists   = errors.New("client: key already exists")
 )
 
-func NewKeysAPI(tr *http.Transport, eps []string, to time.Duration) (KeysAPI, error) {
-	return newHTTPKeysAPIWithPrefix(tr, eps, to, DefaultV2KeysPrefix)
+func NewKeysAPI(c HTTPClient) KeysAPI {
+	return &httpKeysAPI{
+		client: c,
+		prefix: DefaultV2KeysPrefix,
+	}
 }
 
-func NewDiscoveryKeysAPI(tr *http.Transport, eps []string, to time.Duration) (KeysAPI, error) {
-	return newHTTPKeysAPIWithPrefix(tr, eps, to, "")
-}
-
-func newHTTPKeysAPIWithPrefix(tr *http.Transport, eps []string, to time.Duration, prefix string) (*httpKeysAPI, error) {
-	c, err := newHTTPClusterClient(tr, eps)
-	if err != nil {
-		return nil, err
+func NewDiscoveryKeysAPI(c HTTPClient) KeysAPI {
+	return &httpKeysAPI{
+		client: c,
+		prefix: "",
 	}
-
-	kAPI := httpKeysAPI{
-		client:  c,
-		prefix:  prefix,
-		timeout: to,
-	}
-
-	return &kAPI, nil
 }
 
 type KeysAPI interface {
-	Create(key, value string, ttl time.Duration) (*Response, error)
-	Get(key string) (*Response, error)
+	Create(ctx context.Context, key, value string, ttl time.Duration) (*Response, error)
+	Get(ctx context.Context, key string) (*Response, error)
+
 	Watch(key string, idx uint64) Watcher
 	RecursiveWatch(key string, idx uint64) Watcher
 }
 
 type Watcher interface {
-	Next() (*Response, error)
+	Next(context.Context) (*Response, error)
 }
 
 type Response struct {
 	Action   string `json:"action"`
 	Node     *Node  `json:"node"`
 	PrevNode *Node  `json:"prevNode"`
+	Index    uint64
 }
 
 type Nodes []*Node
@@ -95,12 +88,11 @@ func (n *Node) String() string {
 }
 
 type httpKeysAPI struct {
-	client  httpActionDo
-	prefix  string
-	timeout time.Duration
+	client HTTPClient
+	prefix string
 }
 
-func (k *httpKeysAPI) Create(key, val string, ttl time.Duration) (*Response, error) {
+func (k *httpKeysAPI) Create(ctx context.Context, key, val string, ttl time.Duration) (*Response, error) {
 	create := &createAction{
 		Prefix: k.prefix,
 		Key:    key,
@@ -111,31 +103,27 @@ func (k *httpKeysAPI) Create(key, val string, ttl time.Duration) (*Response, err
 		create.TTL = &uttl
 	}
 
-	ctx, cancel := context.WithTimeout(context.Background(), k.timeout)
-	code, body, err := k.client.do(ctx, create)
-	cancel()
+	resp, body, err := k.client.Do(ctx, create)
 	if err != nil {
 		return nil, err
 	}
 
-	return unmarshalHTTPResponse(code, body)
+	return unmarshalHTTPResponse(resp.StatusCode, resp.Header, body)
 }
 
-func (k *httpKeysAPI) Get(key string) (*Response, error) {
+func (k *httpKeysAPI) Get(ctx context.Context, key string) (*Response, error) {
 	get := &getAction{
 		Prefix:    k.prefix,
 		Key:       key,
 		Recursive: false,
 	}
 
-	ctx, cancel := context.WithTimeout(context.Background(), k.timeout)
-	code, body, err := k.client.do(ctx, get)
-	cancel()
+	resp, body, err := k.client.Do(ctx, get)
 	if err != nil {
 		return nil, err
 	}
 
-	return unmarshalHTTPResponse(code, body)
+	return unmarshalHTTPResponse(resp.StatusCode, resp.Header, body)
 }
 
 func (k *httpKeysAPI) Watch(key string, idx uint64) Watcher {
@@ -163,18 +151,17 @@ func (k *httpKeysAPI) RecursiveWatch(key string, idx uint64) Watcher {
 }
 
 type httpWatcher struct {
-	client   httpActionDo
+	client   HTTPClient
 	nextWait waitAction
 }
 
-func (hw *httpWatcher) Next() (*Response, error) {
-	//TODO(bcwaldon): This needs to be cancellable by the calling user
-	code, body, err := hw.client.do(context.Background(), &hw.nextWait)
+func (hw *httpWatcher) Next(ctx context.Context) (*Response, error) {
+	httpresp, body, err := hw.client.Do(ctx, &hw.nextWait)
 	if err != nil {
 		return nil, err
 	}
 
-	resp, err := unmarshalHTTPResponse(code, body)
+	resp, err := unmarshalHTTPResponse(httpresp.StatusCode, httpresp.Header, body)
 	if err != nil {
 		return nil, err
 	}
@@ -199,7 +186,7 @@ type getAction struct {
 	Recursive bool
 }
 
-func (g *getAction) httpRequest(ep url.URL) *http.Request {
+func (g *getAction) HTTPRequest(ep url.URL) *http.Request {
 	u := v2KeysURL(ep, g.Prefix, g.Key)
 
 	params := u.Query()
@@ -217,7 +204,7 @@ type waitAction struct {
 	Recursive bool
 }
 
-func (w *waitAction) httpRequest(ep url.URL) *http.Request {
+func (w *waitAction) HTTPRequest(ep url.URL) *http.Request {
 	u := v2KeysURL(ep, w.Prefix, w.Key)
 
 	params := u.Query()
@@ -237,7 +224,7 @@ type createAction struct {
 	TTL    *uint64
 }
 
-func (c *createAction) httpRequest(ep url.URL) *http.Request {
+func (c *createAction) HTTPRequest(ep url.URL) *http.Request {
 	u := v2KeysURL(ep, c.Prefix, c.Key)
 
 	params := u.Query()
@@ -257,10 +244,10 @@ func (c *createAction) httpRequest(ep url.URL) *http.Request {
 	return req
 }
 
-func unmarshalHTTPResponse(code int, body []byte) (res *Response, err error) {
+func unmarshalHTTPResponse(code int, header http.Header, body []byte) (res *Response, err error) {
 	switch code {
 	case http.StatusOK, http.StatusCreated:
-		res, err = unmarshalSuccessfulResponse(body)
+		res, err = unmarshalSuccessfulResponse(header, body)
 	default:
 		err = unmarshalErrorResponse(code)
 	}
@@ -268,13 +255,18 @@ func unmarshalHTTPResponse(code int, body []byte) (res *Response, err error) {
 	return
 }
 
-func unmarshalSuccessfulResponse(body []byte) (*Response, error) {
+func unmarshalSuccessfulResponse(header http.Header, body []byte) (*Response, error) {
 	var res Response
 	err := json.Unmarshal(body, &res)
 	if err != nil {
 		return nil, err
 	}
-
+	if header.Get("X-Etcd-Index") != "" {
+		res.Index, err = strconv.ParseUint(header.Get("X-Etcd-Index"), 10, 64)
+	}
+	if err != nil {
+		return nil, err
+	}
 	return &res, nil
 }
 
@@ -287,6 +279,8 @@ func unmarshalErrorResponse(code int) error {
 	case http.StatusInternalServerError:
 		// this isn't necessarily true
 		return ErrNoLeader
+	case http.StatusGatewayTimeout:
+		return ErrTimeout
 	default:
 	}
 

client/keys_test.go

@@ -117,7 +117,7 @@ func TestGetAction(t *testing.T) {
 			Key:       "/foo/bar",
 			Recursive: tt.recursive,
 		}
-		got := *f.httpRequest(ep)
+		got := *f.HTTPRequest(ep)
 
 		wantURL := wantURL
 		wantURL.RawQuery = tt.wantQuery
@@ -166,7 +166,7 @@ func TestWaitAction(t *testing.T) {
 			WaitIndex: tt.waitIndex,
 			Recursive: tt.recursive,
 		}
-		got := *f.httpRequest(ep)
+		got := *f.HTTPRequest(ep)
 
 		wantURL := wantURL
 		wantURL.RawQuery = tt.wantQuery
@@ -213,7 +213,7 @@ func TestCreateAction(t *testing.T) {
 			Value: tt.value,
 			TTL:   tt.ttl,
 		}
-		got := *f.httpRequest(ep)
+		got := *f.HTTPRequest(ep)
 
 		err := assertResponse(got, wantURL, wantHeader, []byte(tt.wantBody))
 		if err != nil {
@@ -255,40 +255,46 @@ func assertResponse(got http.Request, wantURL *url.URL, wantHeader http.Header,
 
 func TestUnmarshalSuccessfulResponse(t *testing.T) {
 	tests := []struct {
+		indexHeader string
 		body        string
 		res         *Response
 		expectError bool
 	}{
 		// Neither PrevNode or Node
 		{
+			"1",
 			`{"action":"delete"}`,
-			&Response{Action: "delete"},
+			&Response{Action: "delete", Index: 1},
 			false,
 		},
 
 		// PrevNode
 		{
+			"15",
 			`{"action":"delete", "prevNode": {"key": "/foo", "value": "bar", "modifiedIndex": 12, "createdIndex": 10}}`,
-			&Response{Action: "delete", PrevNode: &Node{Key: "/foo", Value: "bar", ModifiedIndex: 12, CreatedIndex: 10}},
+			&Response{Action: "delete", Index: 15, PrevNode: &Node{Key: "/foo", Value: "bar", ModifiedIndex: 12, CreatedIndex: 10}},
 			false,
 		},
 
 		// Node
 		{
+			"15",
 			`{"action":"get", "node": {"key": "/foo", "value": "bar", "modifiedIndex": 12, "createdIndex": 10}}`,
-			&Response{Action: "get", Node: &Node{Key: "/foo", Value: "bar", ModifiedIndex: 12, CreatedIndex: 10}},
+			&Response{Action: "get", Index: 15, Node: &Node{Key: "/foo", Value: "bar", ModifiedIndex: 12, CreatedIndex: 10}},
 			false,
 		},
 
 		// PrevNode and Node
 		{
+			"15",
 			`{"action":"update", "prevNode": {"key": "/foo", "value": "baz", "modifiedIndex": 10, "createdIndex": 10}, "node": {"key": "/foo", "value": "bar", "modifiedIndex": 12, "createdIndex": 10}}`,
-			&Response{Action: "update", PrevNode: &Node{Key: "/foo", Value: "baz", ModifiedIndex: 10, CreatedIndex: 10}, Node: &Node{Key: "/foo", Value: "bar", ModifiedIndex: 12, CreatedIndex: 10}},
+			&Response{Action: "update", Index: 15, PrevNode: &Node{Key: "/foo", Value: "baz", ModifiedIndex: 10, CreatedIndex: 10}, Node: &Node{Key: "/foo", Value: "bar", ModifiedIndex: 12, CreatedIndex: 10}},
 			false,
 		},
 
 		// Garbage in body
 		{
+			"",
 			`garbage`,
 			nil,
 			true,
@@ -296,7 +302,9 @@ func TestUnmarshalSuccessfulResponse(t *testing.T) {
 	}
 
 	for i, tt := range tests {
-		res, err := unmarshalSuccessfulResponse([]byte(tt.body))
+		h := make(http.Header)
+		h.Add("X-Etcd-Index", tt.indexHeader)
+		res, err := unmarshalSuccessfulResponse(h, []byte(tt.body))
 		if tt.expectError != (err != nil) {
 			t.Errorf("#%d: expectError=%t, err=%v", i, tt.expectError, err)
 		}
@@ -305,14 +313,16 @@ func TestUnmarshalSuccessfulResponse(t *testing.T) {
 			t.Errorf("#%d: received res==%v, but expected res==%v", i, res, tt.res)
 			continue
 		} else if tt.res == nil {
-			// expected and succesfully got nil response
+			// expected and successfully got nil response
 			continue
 		}
 
 		if res.Action != tt.res.Action {
 			t.Errorf("#%d: Action=%s, expected %s", i, res.Action, tt.res.Action)
 		}
-
+		if res.Index != tt.res.Index {
+			t.Errorf("#%d: Index=%d, expected %d", i, res.Index, tt.res.Index)
+		}
 		if !reflect.DeepEqual(res.Node, tt.res.Node) {
 			t.Errorf("#%d: Node=%v, expected %v", i, res.Node, tt.res.Node)
 		}
@@ -350,7 +360,7 @@ func TestUnmarshalErrorResponse(t *testing.T) {
 		{http.StatusNotImplemented, unrecognized},
 		{http.StatusBadGateway, unrecognized},
 		{http.StatusServiceUnavailable, unrecognized},
-		{http.StatusGatewayTimeout, unrecognized},
+		{http.StatusGatewayTimeout, ErrTimeout},
 		{http.StatusHTTPVersionNotSupported, unrecognized},
 	}
 

client/members.go

@@ -23,9 +23,8 @@ import (
 	"net/http"
 	"net/url"
 	"path"
-	"time"
 
-	"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 	"github.com/coreos/etcd/etcdserver/etcdhttp/httptypes"
 	"github.com/coreos/etcd/pkg/types"
 )
@@ -34,41 +33,30 @@ var (
 	DefaultV2MembersPrefix = "/v2/members"
 )
 
-func NewMembersAPI(tr *http.Transport, eps []string, to time.Duration) (MembersAPI, error) {
-	c, err := newHTTPClusterClient(tr, eps)
-	if err != nil {
-		return nil, err
+func NewMembersAPI(c HTTPClient) MembersAPI {
+	return &httpMembersAPI{
+		client: c,
 	}
-
-	mAPI := httpMembersAPI{
-		client:  c,
-		timeout: to,
-	}
-
-	return &mAPI, nil
 }
 
 type MembersAPI interface {
-	List() ([]httptypes.Member, error)
-	Add(peerURL string) (*httptypes.Member, error)
-	Remove(mID string) error
+	List(ctx context.Context) ([]httptypes.Member, error)
+	Add(ctx context.Context, peerURL string) (*httptypes.Member, error)
+	Remove(ctx context.Context, mID string) error
 }
 
 type httpMembersAPI struct {
-	client  httpActionDo
-	timeout time.Duration
+	client HTTPClient
 }
 
-func (m *httpMembersAPI) List() ([]httptypes.Member, error) {
+func (m *httpMembersAPI) List(ctx context.Context) ([]httptypes.Member, error) {
 	req := &membersAPIActionList{}
-	ctx, cancel := context.WithTimeout(context.Background(), m.timeout)
-	code, body, err := m.client.do(ctx, req)
-	cancel()
+	resp, body, err := m.client.Do(ctx, req)
 	if err != nil {
 		return nil, err
 	}
 
-	if err := assertStatusCode(http.StatusOK, code); err != nil {
+	if err := assertStatusCode(resp.StatusCode, http.StatusOK); err != nil {
 		return nil, err
 	}
 
@@ -80,24 +68,30 @@ func (m *httpMembersAPI) List() ([]httptypes.Member, error) {
 	return []httptypes.Member(mCollection), nil
 }
 
-func (m *httpMembersAPI) Add(peerURL string) (*httptypes.Member, error) {
+func (m *httpMembersAPI) Add(ctx context.Context, peerURL string) (*httptypes.Member, error) {
 	urls, err := types.NewURLs([]string{peerURL})
 	if err != nil {
 		return nil, err
 	}
 
 	req := &membersAPIActionAdd{peerURLs: urls}
-	ctx, cancel := context.WithTimeout(context.Background(), m.timeout)
-	code, body, err := m.client.do(ctx, req)
-	cancel()
+	resp, body, err := m.client.Do(ctx, req)
 	if err != nil {
 		return nil, err
 	}
 
-	if err := assertStatusCode(http.StatusCreated, code); err != nil {
+	if err := assertStatusCode(resp.StatusCode, http.StatusCreated, http.StatusConflict); err != nil {
 		return nil, err
 	}
 
+	if resp.StatusCode != http.StatusCreated {
+		var httperr httptypes.HTTPError
+		if err := json.Unmarshal(body, &httperr); err != nil {
+			return nil, err
+		}
+		return nil, httperr
+	}
+
 	var memb httptypes.Member
 	if err := json.Unmarshal(body, &memb); err != nil {
 		return nil, err
@@ -106,21 +100,19 @@ func (m *httpMembersAPI) Add(peerURL string) (*httptypes.Member, error) {
 	return &memb, nil
 }
 
-func (m *httpMembersAPI) Remove(memberID string) error {
+func (m *httpMembersAPI) Remove(ctx context.Context, memberID string) error {
 	req := &membersAPIActionRemove{memberID: memberID}
-	ctx, cancel := context.WithTimeout(context.Background(), m.timeout)
-	code, _, err := m.client.do(ctx, req)
-	cancel()
+	resp, _, err := m.client.Do(ctx, req)
 	if err != nil {
 		return err
 	}
 
-	return assertStatusCode(http.StatusNoContent, code)
+	return assertStatusCode(resp.StatusCode, http.StatusNoContent)
 }
 
 type membersAPIActionList struct{}
 
-func (l *membersAPIActionList) httpRequest(ep url.URL) *http.Request {
+func (l *membersAPIActionList) HTTPRequest(ep url.URL) *http.Request {
 	u := v2MembersURL(ep)
 	req, _ := http.NewRequest("GET", u.String(), nil)
 	return req
@@ -130,7 +122,7 @@ type membersAPIActionRemove struct {
 	memberID string
 }
 
-func (d *membersAPIActionRemove) httpRequest(ep url.URL) *http.Request {
+func (d *membersAPIActionRemove) HTTPRequest(ep url.URL) *http.Request {
 	u := v2MembersURL(ep)
 	u.Path = path.Join(u.Path, d.memberID)
 	req, _ := http.NewRequest("DELETE", u.String(), nil)
@@ -141,7 +133,7 @@ type membersAPIActionAdd struct {
 	peerURLs types.URLs
 }
 
-func (a *membersAPIActionAdd) httpRequest(ep url.URL) *http.Request {
+func (a *membersAPIActionAdd) HTTPRequest(ep url.URL) *http.Request {
 	u := v2MembersURL(ep)
 	m := httptypes.MemberCreateRequest{PeerURLs: a.peerURLs}
 	b, _ := json.Marshal(&m)
@@ -150,11 +142,13 @@ func (a *membersAPIActionAdd) httpRequest(ep url.URL) *http.Request {
 	return req
 }
 
-func assertStatusCode(want, got int) (err error) {
-	if want != got {
-		err = fmt.Errorf("unexpected status code %d", got)
+func assertStatusCode(got int, want ...int) (err error) {
+	for _, w := range want {
+		if w == got {
+			return nil
+		}
 	}
-	return err
+	return fmt.Errorf("unexpected status code %d", got)
 }
 
 // v2MembersURL add the necessary path to the provided endpoint

client/members_test.go

@@ -35,7 +35,7 @@ func TestMembersAPIActionList(t *testing.T) {
 		Path:   "/v2/members",
 	}
 
-	got := *act.httpRequest(ep)
+	got := *act.HTTPRequest(ep)
 	err := assertResponse(got, wantURL, http.Header{}, nil)
 	if err != nil {
 		t.Error(err.Error())
@@ -61,7 +61,7 @@ func TestMembersAPIActionAdd(t *testing.T) {
 	}
 	wantBody := []byte(`{"peerURLs":["https://127.0.0.1:8081","http://127.0.0.1:8080"]}`)
 
-	got := *act.httpRequest(ep)
+	got := *act.HTTPRequest(ep)
 	err := assertResponse(got, wantURL, wantHeader, wantBody)
 	if err != nil {
 		t.Error(err.Error())
@@ -78,7 +78,7 @@ func TestMembersAPIActionRemove(t *testing.T) {
 		Path:   "/v2/members/XXX",
 	}
 
-	got := *act.httpRequest(ep)
+	got := *act.HTTPRequest(ep)
 	err := assertResponse(got, wantURL, http.Header{}, nil)
 	if err != nil {
 		t.Error(err.Error())
@@ -86,12 +86,12 @@ func TestMembersAPIActionRemove(t *testing.T) {
 }
 
 func TestAssertStatusCode(t *testing.T) {
-	if err := assertStatusCode(400, 404); err == nil {
+	if err := assertStatusCode(404, 400); err == nil {
 		t.Errorf("assertStatusCode failed to detect conflict in 400 vs 404")
 	}
 
-	if err := assertStatusCode(400, 400); err != nil {
-		t.Errorf("assertStatusCode found conflict in 400 vs 400: %v", err)
+	if err := assertStatusCode(404, 400, 404); err != nil {
+		t.Errorf("assertStatusCode found conflict in (404,400) vs 400: %v", err)
 	}
 }
 

discovery/discovery.go

@@ -20,9 +20,9 @@ import (
 	"errors"
 	"fmt"
 	"log"
+	"math"
 	"net/http"
 	"net/url"
-	"os"
 	"path"
 	"sort"
 	"strconv"
@@ -30,6 +30,7 @@ import (
 	"time"
 
 	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/jonboulle/clockwork"
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 	"github.com/coreos/etcd/client"
 	"github.com/coreos/etcd/pkg/types"
 )
@@ -44,21 +45,34 @@ var (
 	ErrTooManyRetries = errors.New("discovery: too many retries")
 )
 
-const (
-	// Environment variable used to configure an HTTP proxy for discovery
-	DiscoveryProxyEnv = "ETCD_DISCOVERY_PROXY"
+var (
 	// Number of retries discovery will attempt before giving up and erroring out.
-	nRetries = uint(3)
+	nRetries = uint(math.MaxUint32)
 )
 
-type Discoverer interface {
-	Discover() (string, error)
+// JoinCluster will connect to the discovery service at the given url, and
+// register the server represented by the given id and config to the cluster
+func JoinCluster(durl, dproxyurl string, id types.ID, config string) (string, error) {
+	d, err := newDiscovery(durl, dproxyurl, id)
+	if err != nil {
+		return "", err
+	}
+	return d.joinCluster(config)
+}
+
+// GetCluster will connect to the discovery service at the given url and
+// retrieve a string describing the cluster
+func GetCluster(durl, dproxyurl string) (string, error) {
+	d, err := newDiscovery(durl, dproxyurl, 0)
+	if err != nil {
+		return "", err
+	}
+	return d.getCluster()
 }
 
 type discovery struct {
 	cluster string
 	id      types.ID
-	config  string
 	c       client.KeysAPI
 	retries uint
 	url     *url.URL
@@ -66,11 +80,10 @@ type discovery struct {
 	clock clockwork.Clock
 }
 
-// proxyFuncFromEnv builds a proxy function if the appropriate environment
-// variable is set. It performs basic sanitization of the environment variable
-// and returns any error encountered.
-func proxyFuncFromEnv() (func(*http.Request) (*url.URL, error), error) {
-	proxy := os.Getenv(DiscoveryProxyEnv)
+// newProxyFunc builds a proxy function from the given string, which should
+// represent a URL that can be used as a proxy. It performs basic
+// sanitization of the URL and returns any error encountered.
+func newProxyFunc(proxy string) (func(*http.Request) (*url.URL, error), error) {
 	if proxy == "" {
 		return nil, nil
 	}
@@ -95,52 +108,51 @@ func proxyFuncFromEnv() (func(*http.Request) (*url.URL, error), error) {
 	return http.ProxyURL(proxyURL), nil
 }
 
-func New(durl string, id types.ID, config string) (Discoverer, error) {
+func newDiscovery(durl, dproxyurl string, id types.ID) (*discovery, error) {
 	u, err := url.Parse(durl)
 	if err != nil {
 		return nil, err
 	}
 	token := u.Path
 	u.Path = ""
-	pf, err := proxyFuncFromEnv()
+	pf, err := newProxyFunc(dproxyurl)
 	if err != nil {
 		return nil, err
 	}
-	c, err := client.NewDiscoveryKeysAPI(&http.Transport{Proxy: pf}, []string{u.String()}, client.DefaultRequestTimeout)
+	c, err := client.NewHTTPClient(&http.Transport{Proxy: pf}, []string{u.String()})
 	if err != nil {
 		return nil, err
 	}
+	dc := client.NewDiscoveryKeysAPI(c)
 	return &discovery{
 		cluster: token,
+		c:       dc,
 		id:      id,
-		config:  config,
-		c:       c,
 		url:     u,
 		clock:   clockwork.NewRealClock(),
 	}, nil
 }
 
-func (d *discovery) Discover() (string, error) {
-	// fast path: if the cluster is full, returns the error
-	// do not need to register itself to the cluster in this
-	// case.
-	if _, _, err := d.checkCluster(); err != nil {
+func (d *discovery) joinCluster(config string) (string, error) {
+	// fast path: if the cluster is full, return the error
+	// do not need to register to the cluster in this case.
+	if _, _, _, err := d.checkCluster(); err != nil {
 		return "", err
 	}
 
-	if err := d.createSelf(); err != nil {
+	if err := d.createSelf(config); err != nil {
 		// Fails, even on a timeout, if createSelf times out.
 		// TODO(barakmich): Retrying the same node might want to succeed here
 		// (ie, createSelf should be idempotent for discovery).
 		return "", err
 	}
 
-	nodes, size, err := d.checkCluster()
+	nodes, size, index, err := d.checkCluster()
 	if err != nil {
 		return "", err
 	}
 
-	all, err := d.waitNodes(nodes, size)
+	all, err := d.waitNodes(nodes, size, index)
 	if err != nil {
 		return "", err
 	}
@@ -148,42 +160,67 @@ func (d *discovery) Discover() (string, error) {
 	return nodesToCluster(all), nil
 }
 
-func (d *discovery) createSelf() error {
-	resp, err := d.c.Create(d.selfKey(), d.config, -1)
+func (d *discovery) getCluster() (string, error) {
+	nodes, size, index, err := d.checkCluster()
 	if err != nil {
+		if err == ErrFullCluster {
+			return nodesToCluster(nodes), nil
+		}
+		return "", err
+	}
+
+	all, err := d.waitNodes(nodes, size, index)
+	if err != nil {
+		return "", err
+	}
+	return nodesToCluster(all), nil
+}
+
+func (d *discovery) createSelf(contents string) error {
+	ctx, cancel := context.WithTimeout(context.Background(), client.DefaultRequestTimeout)
+	resp, err := d.c.Create(ctx, d.selfKey(), contents, -1)
+	cancel()
+	if err != nil {
+		if err == client.ErrKeyExists {
+			return ErrDuplicateID
+		}
 		return err
 	}
 
 	// ensure self appears on the server we connected to
 	w := d.c.Watch(d.selfKey(), resp.Node.CreatedIndex)
-	_, err = w.Next()
+	_, err = w.Next(context.Background())
 	return err
 }
 
-func (d *discovery) checkCluster() (client.Nodes, int, error) {
+func (d *discovery) checkCluster() (client.Nodes, int, uint64, error) {
 	configKey := path.Join("/", d.cluster, "_config")
+	ctx, cancel := context.WithTimeout(context.Background(), client.DefaultRequestTimeout)
 	// find cluster size
-	resp, err := d.c.Get(path.Join(configKey, "size"))
+	resp, err := d.c.Get(ctx, path.Join(configKey, "size"))
+	cancel()
 	if err != nil {
 		if err == client.ErrKeyNoExist {
-			return nil, 0, ErrSizeNotFound
+			return nil, 0, 0, ErrSizeNotFound
 		}
 		if err == client.ErrTimeout {
 			return d.checkClusterRetry()
 		}
-		return nil, 0, err
+		return nil, 0, 0, err
 	}
 	size, err := strconv.Atoi(resp.Node.Value)
 	if err != nil {
-		return nil, 0, ErrBadSizeKey
+		return nil, 0, 0, ErrBadSizeKey
 	}
 
-	resp, err = d.c.Get(d.cluster)
+	ctx, cancel = context.WithTimeout(context.Background(), client.DefaultRequestTimeout)
+	resp, err = d.c.Get(ctx, d.cluster)
+	cancel()
 	if err != nil {
 		if err == client.ErrTimeout {
 			return d.checkClusterRetry()
 		}
-		return nil, 0, err
+		return nil, 0, 0, err
 	}
 	nodes := make(client.Nodes, 0)
 	// append non-config keys to nodes
@@ -202,10 +239,10 @@ func (d *discovery) checkCluster() (client.Nodes, int, error) {
 			break
 		}
 		if i >= size-1 {
-			return nil, size, ErrFullCluster
+			return nodes[:size], size, resp.Index, ErrFullCluster
 		}
 	}
-	return nodes, size, nil
+	return nodes, size, resp.Index, nil
 }
 
 func (d *discovery) logAndBackoffForRetry(step string) {
@@ -215,31 +252,32 @@ func (d *discovery) logAndBackoffForRetry(step string) {
 	d.clock.Sleep(retryTime)
 }
 
-func (d *discovery) checkClusterRetry() (client.Nodes, int, error) {
+func (d *discovery) checkClusterRetry() (client.Nodes, int, uint64, error) {
 	if d.retries < nRetries {
 		d.logAndBackoffForRetry("cluster status check")
 		return d.checkCluster()
 	}
-	return nil, 0, ErrTooManyRetries
+	return nil, 0, 0, ErrTooManyRetries
 }
 
 func (d *discovery) waitNodesRetry() (client.Nodes, error) {
 	if d.retries < nRetries {
 		d.logAndBackoffForRetry("waiting for other nodes")
-		nodes, n, err := d.checkCluster()
+		nodes, n, index, err := d.checkCluster()
 		if err != nil {
 			return nil, err
 		}
-		return d.waitNodes(nodes, n)
+		return d.waitNodes(nodes, n, index)
 	}
 	return nil, ErrTooManyRetries
 }
 
-func (d *discovery) waitNodes(nodes client.Nodes, size int) (client.Nodes, error) {
+func (d *discovery) waitNodes(nodes client.Nodes, size int, index uint64) (client.Nodes, error) {
 	if len(nodes) > size {
 		nodes = nodes[:size]
 	}
-	w := d.c.RecursiveWatch(d.cluster, nodes[len(nodes)-1].ModifiedIndex+1)
+	// watch from the next index
+	w := d.c.RecursiveWatch(d.cluster, index+1)
 	all := make(client.Nodes, len(nodes))
 	copy(all, nodes)
 	for _, n := range all {
@@ -253,7 +291,7 @@ func (d *discovery) waitNodes(nodes client.Nodes, size int) (client.Nodes, error
 	// wait for others
 	for len(all) < size {
 		log.Printf("discovery: found %d peer(s), waiting for %d more", len(all), size-len(all))
-		resp, err := w.Next()
+		resp, err := w.Next(context.Background())
 		if err != nil {
 			if err == client.ErrTimeout {
 				return d.waitNodesRetry()

discovery/discovery_test.go

@@ -18,23 +18,26 @@ package discovery
 
 import (
 	"errors"
+	"math"
 	"math/rand"
 	"net/http"
-	"os"
+	"reflect"
 	"sort"
 	"strconv"
-
-	"reflect"
 	"testing"
 	"time"
 
 	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/jonboulle/clockwork"
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 	"github.com/coreos/etcd/client"
 )
 
-func TestProxyFuncFromEnvUnset(t *testing.T) {
-	os.Setenv(DiscoveryProxyEnv, "")
-	pf, err := proxyFuncFromEnv()
+const (
+	maxRetryInTest = 3
+)
+
+func TestNewProxyFuncUnset(t *testing.T) {
+	pf, err := newProxyFunc("")
 	if pf != nil {
 		t.Fatal("unexpected non-nil proxyFunc")
 	}
@@ -43,14 +46,13 @@ func TestProxyFuncFromEnvUnset(t *testing.T) {
 	}
 }
 
-func TestProxyFuncFromEnvBad(t *testing.T) {
+func TestNewProxyFuncBad(t *testing.T) {
 	tests := []string{
 		"%%",
 		"http://foo.com/%1",
 	}
 	for i, in := range tests {
-		os.Setenv(DiscoveryProxyEnv, in)
-		pf, err := proxyFuncFromEnv()
+		pf, err := newProxyFunc(in)
 		if pf != nil {
 			t.Errorf("#%d: unexpected non-nil proxyFunc", i)
 		}
@@ -60,14 +62,13 @@ func TestProxyFuncFromEnvBad(t *testing.T) {
 	}
 }
 
-func TestProxyFuncFromEnv(t *testing.T) {
+func TestNewProxyFunc(t *testing.T) {
 	tests := map[string]string{
 		"bar.com":              "http://bar.com",
 		"http://disco.foo.bar": "http://disco.foo.bar",
 	}
 	for in, w := range tests {
-		os.Setenv(DiscoveryProxyEnv, in)
-		pf, err := proxyFuncFromEnv()
+		pf, err := newProxyFunc(in)
 		if pf == nil {
 			t.Errorf("%s: unexpected nil proxyFunc", in)
 			continue
@@ -93,6 +94,7 @@ func TestCheckCluster(t *testing.T) {
 
 	tests := []struct {
 		nodes []*client.Node
+		index uint64
 		werr  error
 		wsize int
 	}{
@@ -106,6 +108,7 @@ func TestCheckCluster(t *testing.T) {
 				{Key: "/1000/3", CreatedIndex: 4},
 				{Key: "/1000/4", CreatedIndex: 5},
 			},
+			5,
 			nil,
 			3,
 		},
@@ -119,6 +122,7 @@ func TestCheckCluster(t *testing.T) {
 				{Key: self, CreatedIndex: 4},
 				{Key: "/1000/4", CreatedIndex: 5},
 			},
+			5,
 			nil,
 			3,
 		},
@@ -132,6 +136,7 @@ func TestCheckCluster(t *testing.T) {
 				{Key: "/1000/4", CreatedIndex: 4},
 				{Key: self, CreatedIndex: 5},
 			},
+			5,
 			ErrFullCluster,
 			3,
 		},
@@ -143,6 +148,7 @@ func TestCheckCluster(t *testing.T) {
 				{Key: "/1000/2", CreatedIndex: 2},
 				{Key: "/1000/3", CreatedIndex: 3},
 			},
+			3,
 			nil,
 			3,
 		},
@@ -154,6 +160,7 @@ func TestCheckCluster(t *testing.T) {
 				{Key: "/1000/3", CreatedIndex: 3},
 				{Key: "/1000/4", CreatedIndex: 4},
 			},
+			3,
 			ErrFullCluster,
 			3,
 		},
@@ -162,12 +169,14 @@ func TestCheckCluster(t *testing.T) {
 			[]*client.Node{
 				{Key: "/1000/_config/size", Value: "bad", CreatedIndex: 1},
 			},
+			0,
 			ErrBadSizeKey,
 			0,
 		},
 		{
 			// no size key
 			[]*client.Node{},
+			0,
 			ErrSizeNotFound,
 			0,
 		},
@@ -176,12 +185,13 @@ func TestCheckCluster(t *testing.T) {
 	for i, tt := range tests {
 		rs := make([]*client.Response, 0)
 		if len(tt.nodes) > 0 {
-			rs = append(rs, &client.Response{Node: tt.nodes[0]})
+			rs = append(rs, &client.Response{Node: tt.nodes[0], Index: tt.index})
 			rs = append(rs, &client.Response{
 				Node: &client.Node{
 					Key:   cluster,
 					Nodes: tt.nodes[1:],
 				},
+				Index: tt.index,
 			})
 		}
 		c := &clientWithResp{rs: rs}
@@ -194,12 +204,12 @@ func TestCheckCluster(t *testing.T) {
 
 		for _, d := range []discovery{d, dRetry} {
 			go func() {
-				for i := uint(1); i <= nRetries; i++ {
+				for i := uint(1); i <= maxRetryInTest; i++ {
 					fc.BlockUntil(1)
 					fc.Advance(time.Second * (0x1 << i))
 				}
 			}()
-			ns, size, err := d.checkCluster()
+			ns, size, index, err := d.checkCluster()
 			if err != tt.werr {
 				t.Errorf("#%d: err = %v, want %v", i, err, tt.werr)
 			}
@@ -209,6 +219,9 @@ func TestCheckCluster(t *testing.T) {
 			if size != tt.wsize {
 				t.Errorf("#%d: size = %v, want %d", i, size, tt.wsize)
 			}
+			if index != tt.index {
+				t.Errorf("#%d: index = %v, want %d", i, index, tt.index)
+			}
 		}
 	}
 }
@@ -282,12 +295,12 @@ func TestWaitNodes(t *testing.T) {
 
 		for _, d := range []*discovery{d, dRetry} {
 			go func() {
-				for i := uint(1); i <= nRetries; i++ {
+				for i := uint(1); i <= maxRetryInTest; i++ {
 					fc.BlockUntil(1)
 					fc.Advance(time.Second * (0x1 << i))
 				}
 			}()
-			g, err := d.waitNodes(tt.nodes, 3)
+			g, err := d.waitNodes(tt.nodes, 3, 0) // we do not care about index in this test
 			if err != nil {
 				t.Errorf("#%d: err = %v, want %v", i, err, nil)
 			}
@@ -322,7 +335,7 @@ func TestCreateSelf(t *testing.T) {
 
 	for i, tt := range tests {
 		d := discovery{cluster: "1000", c: tt.c}
-		if err := d.createSelf(); err != tt.werr {
+		if err := d.createSelf(""); err != tt.werr {
 			t.Errorf("#%d: err = %v, want %v", i, err, nil)
 		}
 	}
@@ -372,6 +385,9 @@ func TestSortableNodes(t *testing.T) {
 }
 
 func TestRetryFailure(t *testing.T) {
+	nRetries = maxRetryInTest
+	defer func() { nRetries = math.MaxUint32 }()
+
 	cluster := "1000"
 	c := &clientWithRetry{failTimes: 4}
 	fc := clockwork.NewFakeClock()
@@ -382,12 +398,12 @@ func TestRetryFailure(t *testing.T) {
 		clock:   fc,
 	}
 	go func() {
-		for i := uint(1); i <= nRetries; i++ {
+		for i := uint(1); i <= maxRetryInTest; i++ {
 			fc.BlockUntil(1)
 			fc.Advance(time.Second * (0x1 << i))
 		}
 	}()
-	if _, _, err := d.checkCluster(); err != ErrTooManyRetries {
+	if _, _, _, err := d.checkCluster(); err != ErrTooManyRetries {
 		t.Errorf("err = %v, want %v", err, ErrTooManyRetries)
 	}
 }
@@ -397,7 +413,7 @@ type clientWithResp struct {
 	w  client.Watcher
 }
 
-func (c *clientWithResp) Create(key string, value string, ttl time.Duration) (*client.Response, error) {
+func (c *clientWithResp) Create(ctx context.Context, key string, value string, ttl time.Duration) (*client.Response, error) {
 	if len(c.rs) == 0 {
 		return &client.Response{}, nil
 	}
@@ -406,7 +422,7 @@ func (c *clientWithResp) Create(key string, value string, ttl time.Duration) (*c
 	return r, nil
 }
 
-func (c *clientWithResp) Get(key string) (*client.Response, error) {
+func (c *clientWithResp) Get(ctx context.Context, key string) (*client.Response, error) {
 	if len(c.rs) == 0 {
 		return &client.Response{}, client.ErrKeyNoExist
 	}
@@ -428,11 +444,11 @@ type clientWithErr struct {
 	w   client.Watcher
 }
 
-func (c *clientWithErr) Create(key string, value string, ttl time.Duration) (*client.Response, error) {
+func (c *clientWithErr) Create(ctx context.Context, key string, value string, ttl time.Duration) (*client.Response, error) {
 	return &client.Response{}, c.err
 }
 
-func (c *clientWithErr) Get(key string) (*client.Response, error) {
+func (c *clientWithErr) Get(ctx context.Context, key string) (*client.Response, error) {
 	return &client.Response{}, c.err
 }
 
@@ -448,7 +464,7 @@ type watcherWithResp struct {
 	rs []*client.Response
 }
 
-func (w *watcherWithResp) Next() (*client.Response, error) {
+func (w *watcherWithResp) Next(context.Context) (*client.Response, error) {
 	if len(w.rs) == 0 {
 		return &client.Response{}, nil
 	}
@@ -461,7 +477,7 @@ type watcherWithErr struct {
 	err error
 }
 
-func (w *watcherWithErr) Next() (*client.Response, error) {
+func (w *watcherWithErr) Next(context.Context) (*client.Response, error) {
 	return &client.Response{}, w.err
 }
 
@@ -472,20 +488,20 @@ type clientWithRetry struct {
 	failTimes int
 }
 
-func (c *clientWithRetry) Create(key string, value string, ttl time.Duration) (*client.Response, error) {
+func (c *clientWithRetry) Create(ctx context.Context, key string, value string, ttl time.Duration) (*client.Response, error) {
 	if c.failCount < c.failTimes {
 		c.failCount++
 		return nil, client.ErrTimeout
 	}
-	return c.clientWithResp.Create(key, value, ttl)
+	return c.clientWithResp.Create(ctx, key, value, ttl)
 }
 
-func (c *clientWithRetry) Get(key string) (*client.Response, error) {
+func (c *clientWithRetry) Get(ctx context.Context, key string) (*client.Response, error) {
 	if c.failCount < c.failTimes {
 		c.failCount++
 		return nil, client.ErrTimeout
 	}
-	return c.clientWithResp.Get(key)
+	return c.clientWithResp.Get(ctx, key)
 }
 
 // watcherWithRetry will timeout all requests up to failTimes
@@ -495,7 +511,7 @@ type watcherWithRetry struct {
 	failTimes int
 }
 
-func (w *watcherWithRetry) Next() (*client.Response, error) {
+func (w *watcherWithRetry) Next(context.Context) (*client.Response, error) {
 	if w.failCount < w.failTimes {
 		w.failCount++
 		return nil, client.ErrTimeout

error/error.go

@@ -63,6 +63,16 @@ var errors = map[int]string{
 	EcodeClientInternal: "Client Internal Error",
 }
 
+var errorStatus = map[int]int{
+	EcodeKeyNotFound:  http.StatusNotFound,
+	EcodeNotFile:      http.StatusForbidden,
+	EcodeDirNotEmpty:  http.StatusForbidden,
+	EcodeTestFailed:   http.StatusPreconditionFailed,
+	EcodeNodeExist:    http.StatusPreconditionFailed,
+	EcodeRaftInternal: http.StatusInternalServerError,
+	EcodeLeaderElect:  http.StatusInternalServerError,
+}
+
 const (
 	EcodeKeyNotFound      = 100
 	EcodeTestFailed       = 101
@@ -133,22 +143,17 @@ func (e Error) toJsonString() string {
 	return string(b)
 }
 
+func (e Error) statusCode() int {
+	status, ok := errorStatus[e.ErrorCode]
+	if !ok {
+		status = http.StatusBadRequest
+	}
+	return status
+}
+
 func (e Error) WriteTo(w http.ResponseWriter) {
 	w.Header().Add("X-Etcd-Index", fmt.Sprint(e.Index))
-	// 3xx is raft internal error
-	status := http.StatusBadRequest
-	switch e.ErrorCode {
-	case EcodeKeyNotFound:
-		status = http.StatusNotFound
-	case EcodeNotFile, EcodeDirNotEmpty:
-		status = http.StatusForbidden
-	case EcodeTestFailed, EcodeNodeExist:
-		status = http.StatusPreconditionFailed
-	default:
-		if e.ErrorCode/100 == 3 {
-			status = http.StatusInternalServerError
-		}
-	}
-	w.WriteHeader(status)
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(e.statusCode())
 	fmt.Fprintln(w, e.toJsonString())
 }

error/error_test.go

@@ -0,0 +1,52 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
+package error
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"reflect"
+	"strings"
+	"testing"
+)
+
+func TestErrorWriteTo(t *testing.T) {
+	for k, _ := range errors {
+		err := NewError(k, "", 1)
+		rr := httptest.NewRecorder()
+		err.WriteTo(rr)
+
+		if err.statusCode() != rr.Code {
+			t.Errorf("HTTP status code %d, want %d", rr.Code, err.statusCode())
+		}
+
+		gbody := strings.TrimSuffix(rr.Body.String(), "\n")
+		if err.toJsonString() != gbody {
+			t.Errorf("HTTP body %q, want %q", gbody, err.toJsonString())
+		}
+
+		wheader := http.Header(map[string][]string{
+			"Content-Type": []string{"application/json"},
+			"X-Etcd-Index": []string{"1"},
+		})
+
+		if !reflect.DeepEqual(wheader, rr.HeaderMap) {
+			t.Errorf("HTTP headers %v, want %v", rr.HeaderMap, wheader)
+		}
+	}
+
+}

etcdctl/command/backup_command.go

@@ -0,0 +1,93 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
+package command
+
+import (
+	"log"
+	"math/rand"
+	"os"
+	"path"
+	"time"
+
+	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
+	"github.com/coreos/etcd/etcdserver"
+	"github.com/coreos/etcd/etcdserver/etcdserverpb"
+	"github.com/coreos/etcd/pkg/pbutil"
+	"github.com/coreos/etcd/snap"
+	"github.com/coreos/etcd/wal"
+)
+
+func NewBackupCommand() cli.Command {
+	return cli.Command{
+		Name:  "backup",
+		Usage: "backup an etcd directory",
+		Flags: []cli.Flag{
+			cli.StringFlag{Name: "data-dir", Value: "", Usage: "Path to the etcd data dir"},
+			cli.StringFlag{Name: "backup-dir", Value: "", Usage: "Path to the backup dir"},
+		},
+		Action: handleBackup,
+	}
+}
+
+// handleBackup handles a request that intends to do a backup.
+func handleBackup(c *cli.Context) {
+	srcSnap := path.Join(c.String("data-dir"), "snap")
+	destSnap := path.Join(c.String("backup-dir"), "snap")
+	srcWAL := path.Join(c.String("data-dir"), "wal")
+	destWAL := path.Join(c.String("backup-dir"), "wal")
+
+	if err := os.MkdirAll(destSnap, 0700); err != nil {
+		log.Fatalf("failed creating backup snapshot dir %v: %v", destSnap, err)
+	}
+	ss := snap.New(srcSnap)
+	snapshot, err := ss.Load()
+	if err != nil && err != snap.ErrNoSnapshot {
+		log.Fatal(err)
+	}
+	var index uint64
+	if snapshot != nil {
+		index = snapshot.Metadata.Index
+		newss := snap.New(destSnap)
+		if err := newss.SaveSnap(*snapshot); err != nil {
+			log.Fatal(err)
+		}
+	}
+
+	w, err := wal.OpenNotInUse(srcWAL, index)
+	if err != nil {
+		log.Fatal(err)
+	}
+	defer w.Close()
+	wmetadata, state, ents, err := w.ReadAll()
+	if err != nil {
+		log.Fatal(err)
+	}
+	var metadata etcdserverpb.Metadata
+	pbutil.MustUnmarshal(&metadata, wmetadata)
+	rand.Seed(time.Now().UnixNano())
+	metadata.NodeID = etcdserver.GenID()
+	metadata.ClusterID = etcdserver.GenID()
+
+	neww, err := wal.Create(destWAL, pbutil.MustMarshal(&metadata))
+	if err != nil {
+		log.Fatal(err)
+	}
+	defer neww.Close()
+	if err := neww.Save(state, ents); err != nil {
+		log.Fatal(err)
+	}
+}

etcdctl/command/error.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (

etcdctl/command/exec_watch_command.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (

etcdctl/command/get_command.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (

etcdctl/command/handle.go

@@ -1,10 +1,25 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (
 	"encoding/json"
 	"errors"
 	"fmt"
-	"net/url"
 	"os"
 	"strings"
 
@@ -24,72 +39,35 @@ func dumpCURL(client *etcd.Client) {
 	}
 }
 
-// createHttpPath attaches http scheme to the given address if needed
-func createHttpPath(addr string) (string, error) {
-	u, err := url.Parse(addr)
-	if err != nil {
-		return "", err
-	}
-
-	if u.Scheme == "" {
-		u.Scheme = "http"
-	}
-	return u.String(), nil
-}
-
-func getPeersFlagValue(c *cli.Context) []string {
-	peerstr := c.GlobalString("peers")
-
-	// Use an environment variable if nothing was supplied on the
-	// command line
-	if peerstr == "" {
-		peerstr = os.Getenv("ETCDCTL_PEERS")
-	}
-
-	// If we still don't have peers, use a default
-	if peerstr == "" {
-		peerstr = "127.0.0.1:4001"
-	}
-
-	return strings.Split(peerstr, ",")
-}
-
 // rawhandle wraps the command function handlers and sets up the
 // environment but performs no output formatting.
 func rawhandle(c *cli.Context, fn handlerFunc) (*etcd.Response, error) {
-	sync := !c.GlobalBool("no-sync")
-
-	peers := getPeersFlagValue(c)
-
-	// If no sync, create http path for each peer address
-	if !sync {
-		revisedPeers := make([]string, 0)
-		for _, peer := range peers {
-			if revisedPeer, err := createHttpPath(peer); err != nil {
-				fmt.Fprintf(os.Stderr, "Unsupported url %v: %v\n", peer, err)
-			} else {
-				revisedPeers = append(revisedPeers, revisedPeer)
-			}
-		}
-		peers = revisedPeers
+	endpoints, err := getEndpoints(c)
+	if err != nil {
+		return nil, err
 	}
 
-	client := etcd.NewClient(peers)
+	tr, err := getTransport(c)
+	if err != nil {
+		return nil, err
+	}
+
+	client := etcd.NewClient(endpoints)
+	client.SetTransport(tr)
 
 	if c.GlobalBool("debug") {
 		go dumpCURL(client)
 	}
 
 	// Sync cluster.
-	if sync {
+	if !c.GlobalBool("no-sync") {
 		if ok := client.SyncCluster(); !ok {
-			handleError(FailedToConnectToHost, errors.New("Cannot sync with the cluster using peers "+strings.Join(peers, ", ")))
+			handleError(FailedToConnectToHost, errors.New("cannot sync with the cluster using endpoints "+strings.Join(endpoints, ", ")))
 		}
 	}
 
 	if c.GlobalBool("debug") {
-		fmt.Fprintf(os.Stderr, "Cluster-Peers: %s\n",
-			strings.Join(client.GetCluster(), " "))
+		fmt.Fprintf(os.Stderr, "Cluster-Endpoints: %s\n", strings.Join(client.GetCluster(), ", "))
 	}
 
 	// Execute handler function.

etcdctl/command/ls_command.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (

etcdctl/command/member_commands.go

@@ -1,12 +1,28 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (
 	"fmt"
-	"net/http"
 	"os"
 	"strings"
 
 	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 	"github.com/coreos/etcd/client"
 )
 
@@ -35,20 +51,39 @@ func NewMemberCommand() cli.Command {
 }
 
 func mustNewMembersAPI(c *cli.Context) client.MembersAPI {
-	peers := getPeersFlagValue(c)
-	for i, p := range peers {
-		if !strings.HasPrefix(p, "http") && !strings.HasPrefix(p, "https") {
-			peers[i] = fmt.Sprintf("http://%s", p)
-		}
-	}
-
-	mAPI, err := client.NewMembersAPI(&http.Transport{}, peers, client.DefaultRequestTimeout)
+	eps, err := getEndpoints(c)
 	if err != nil {
 		fmt.Fprintln(os.Stderr, err.Error())
 		os.Exit(1)
 	}
 
-	return mAPI
+	tr, err := getTransport(c)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, err.Error())
+		os.Exit(1)
+	}
+
+	hc, err := client.NewHTTPClient(tr, eps)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, err.Error())
+		os.Exit(1)
+	}
+
+	if !c.GlobalBool("no-sync") {
+		ctx, cancel := context.WithTimeout(context.Background(), client.DefaultRequestTimeout)
+		err := hc.Sync(ctx)
+		cancel()
+		if err != nil {
+			fmt.Fprintln(os.Stderr, err.Error())
+			os.Exit(1)
+		}
+	}
+
+	if c.GlobalBool("debug") {
+		fmt.Fprintf(os.Stderr, "Cluster-Endpoints: %s\n", strings.Join(hc.Endpoints(), ", "))
+	}
+
+	return client.NewMembersAPI(hc)
 }
 
 func actionMemberList(c *cli.Context) {
@@ -57,7 +92,9 @@ func actionMemberList(c *cli.Context) {
 		os.Exit(1)
 	}
 	mAPI := mustNewMembersAPI(c)
-	members, err := mAPI.List()
+	ctx, cancel := context.WithTimeout(context.Background(), client.DefaultRequestTimeout)
+	members, err := mAPI.List(ctx)
+	cancel()
 	if err != nil {
 		fmt.Fprintln(os.Stderr, err.Error())
 		os.Exit(1)
@@ -78,7 +115,9 @@ func actionMemberAdd(c *cli.Context) {
 	mAPI := mustNewMembersAPI(c)
 
 	url := args[1]
-	m, err := mAPI.Add(url)
+	ctx, cancel := context.WithTimeout(context.Background(), client.DefaultRequestTimeout)
+	m, err := mAPI.Add(ctx, url)
+	cancel()
 	if err != nil {
 		fmt.Fprintln(os.Stderr, err.Error())
 		os.Exit(1)
@@ -88,7 +127,9 @@ func actionMemberAdd(c *cli.Context) {
 	newName := args[0]
 	fmt.Printf("Added member named %s with ID %s to cluster\n", newName, newID)
 
-	members, err := mAPI.List()
+	ctx, cancel = context.WithTimeout(context.Background(), client.DefaultRequestTimeout)
+	members, err := mAPI.List(ctx)
+	cancel()
 	if err != nil {
 		fmt.Fprintln(os.Stderr, err.Error())
 		os.Exit(1)
@@ -120,7 +161,10 @@ func actionMemberRemove(c *cli.Context) {
 
 	mAPI := mustNewMembersAPI(c)
 	mID := args[0]
-	if err := mAPI.Remove(mID); err != nil {
+	ctx, cancel := context.WithTimeout(context.Background(), client.DefaultRequestTimeout)
+	err := mAPI.Remove(ctx, mID)
+	cancel()
+	if err != nil {
 		fmt.Fprintln(os.Stderr, err.Error())
 		os.Exit(1)
 	}

etcdctl/command/mk_command.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (

etcdctl/command/mkdir_command.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (

etcdctl/command/rm_command.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (

etcdctl/command/rmdir_command.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (

etcdctl/command/set_command.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (

etcdctl/command/set_dir_command.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (

etcdctl/command/update_command.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (

etcdctl/command/update_dir_command.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (

etcdctl/command/util.go

@@ -1,10 +1,32 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (
 	"errors"
 	"io"
 	"io/ioutil"
+	"net/http"
+	"net/url"
+	"os"
 	"strings"
+
+	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/codegangsta/cli"
+	"github.com/coreos/etcd/pkg/transport"
 )
 
 var (
@@ -33,3 +55,63 @@ func argOrStdin(args []string, stdin io.Reader, i int) (string, error) {
 	}
 	return string(bytes), nil
 }
+
+func getPeersFlagValue(c *cli.Context) []string {
+	peerstr := c.GlobalString("peers")
+
+	// Use an environment variable if nothing was supplied on the
+	// command line
+	if peerstr == "" {
+		peerstr = os.Getenv("ETCDCTL_PEERS")
+	}
+
+	// If we still don't have peers, use a default
+	if peerstr == "" {
+		peerstr = "127.0.0.1:4001"
+	}
+
+	return strings.Split(peerstr, ",")
+}
+
+func getEndpoints(c *cli.Context) ([]string, error) {
+	eps := getPeersFlagValue(c)
+	for i, ep := range eps {
+		u, err := url.Parse(ep)
+		if err != nil {
+			return nil, err
+		}
+
+		if u.Scheme == "" {
+			u.Scheme = "http"
+		}
+
+		eps[i] = u.String()
+	}
+	return eps, nil
+}
+
+func getTransport(c *cli.Context) (*http.Transport, error) {
+	cafile := c.GlobalString("ca-file")
+	certfile := c.GlobalString("cert-file")
+	keyfile := c.GlobalString("key-file")
+
+	// Use an environment variable if nothing was supplied on the
+	// command line
+	if cafile == "" {
+		cafile = os.Getenv("ETCDCTL_CA_FILE")
+	}
+	if certfile == "" {
+		certfile = os.Getenv("ETCDCTL_CERT_FILE")
+	}
+	if keyfile == "" {
+		keyfile = os.Getenv("ETCDCTL_KEY_FILE")
+	}
+
+	tls := transport.TLSInfo{
+		CAFile:   cafile,
+		CertFile: certfile,
+		KeyFile:  keyfile,
+	}
+	return transport.NewTransport(tls)
+
+}

etcdctl/command/util_test.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (

etcdctl/command/watch_command.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package command
 
 import (

etcdctl/main.go

@@ -1,3 +1,19 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
 package main
 
 import (
@@ -19,8 +35,12 @@ func main() {
 		cli.BoolFlag{Name: "no-sync", Usage: "don't synchronize cluster information before sending request"},
 		cli.StringFlag{Name: "output, o", Value: "simple", Usage: "output response in the given format (`simple` or `json`)"},
 		cli.StringFlag{Name: "peers, C", Value: "", Usage: "a comma-delimited list of machine addresses in the cluster (default: \"127.0.0.1:4001\")"},
+		cli.StringFlag{Name: "cert-file", Value: "", Usage: "identify HTTPS client using this SSL certificate file"},
+		cli.StringFlag{Name: "key-file", Value: "", Usage: "identify HTTPS client using this SSL key file"},
+		cli.StringFlag{Name: "ca-file", Value: "", Usage: "verify certificates of HTTPS-enabled servers using this CA bundle"},
 	}
 	app.Commands = []cli.Command{
+		command.NewBackupCommand(),
 		command.NewMakeCommand(),
 		command.NewMakeDirCommand(),
 		command.NewRemoveCommand(),

etcdmain/etcd.go

@@ -20,10 +20,13 @@ import (
 	"flag"
 	"fmt"
 	"log"
+	"net"
 	"net/http"
+	"net/url"
 	"os"
 	"strings"
 
+	"github.com/coreos/etcd/discovery"
 	"github.com/coreos/etcd/etcdserver"
 	"github.com/coreos/etcd/etcdserver/etcdhttp"
 	"github.com/coreos/etcd/pkg/cors"
@@ -32,32 +35,58 @@ import (
 	"github.com/coreos/etcd/pkg/transport"
 	"github.com/coreos/etcd/pkg/types"
 	"github.com/coreos/etcd/proxy"
+	"github.com/coreos/etcd/rafthttp"
 	"github.com/coreos/etcd/version"
 )
 
 const (
 	// the owner can make/remove files inside the directory
 	privateDirMode = 0700
+
+	proxyFlagOff      = "off"
+	proxyFlagReadonly = "readonly"
+	proxyFlagOn       = "on"
+
+	fallbackFlagExit  = "exit"
+	fallbackFlagProxy = "proxy"
+
+	clusterStateFlagNew      = "new"
+	clusterStateFlagExisting = "existing"
 )
 
 var (
-	fs           = flag.NewFlagSet("etcd", flag.ContinueOnError)
-	name         = fs.String("name", "default", "Unique human-readable name for this node")
-	dir          = fs.String("data-dir", "", "Path to the data directory")
-	durl         = fs.String("discovery", "", "Discovery service used to bootstrap the cluster")
-	snapCount    = fs.Uint64("snapshot-count", etcdserver.DefaultSnapCount, "Number of committed transactions to trigger a snapshot")
-	printVersion = fs.Bool("version", false, "Print the version and exit")
+	fs              = flag.NewFlagSet("etcd", flag.ContinueOnError)
+	name            = fs.String("name", "default", "Unique human-readable name for this node")
+	dir             = fs.String("data-dir", "", "Path to the data directory")
+	durl            = fs.String("discovery", "", "Discovery service used to bootstrap the cluster")
+	dproxy          = fs.String("discovery-proxy", "", "HTTP proxy to use for traffic to discovery service")
+	snapCount       = fs.Uint64("snapshot-count", etcdserver.DefaultSnapCount, "Number of committed transactions to trigger a snapshot")
+	printVersion    = fs.Bool("version", false, "Print the version and exit")
+	forceNewCluster = fs.Bool("force-new-cluster", false, "Force to create a new one member cluster")
+	maxSnapFiles    = fs.Uint("max-snapshots", 5, "Maximum number of snapshot files to retain (0 is unlimited)")
+	maxWalFiles     = fs.Uint("max-wals", 5, "Maximum number of wal files to retain (0 is unlimited)")
 
 	initialCluster      = fs.String("initial-cluster", "default=http://localhost:2380,default=http://localhost:7001", "Initial cluster configuration for bootstrapping")
 	initialClusterToken = fs.String("initial-cluster-token", "etcd-cluster", "Initial cluster token for the etcd cluster during bootstrap")
-	clusterState        = new(etcdserver.ClusterState)
-
-	corsInfo  = &cors.CORSInfo{}
-	proxyFlag = new(flags.Proxy)
 
+	corsInfo      = &cors.CORSInfo{}
 	clientTLSInfo = transport.TLSInfo{}
 	peerTLSInfo   = transport.TLSInfo{}
 
+	proxyFlag = flags.NewStringsFlag(
+		proxyFlagOff,
+		proxyFlagReadonly,
+		proxyFlagOn,
+	)
+	fallbackFlag = flags.NewStringsFlag(
+		fallbackFlagExit,
+		fallbackFlagProxy,
+	)
+	clusterStateFlag = flags.NewStringsFlag(
+		clusterStateFlagNew,
+		clusterStateFlagExisting,
+	)
+
 	ignored = []string{
 		"cluster-active-size",
 		"cluster-remove-delay",
@@ -76,10 +105,15 @@ var (
 )
 
 func init() {
-	fs.Var(clusterState, "initial-cluster-state", "Initial cluster configuration for bootstrapping")
-	if err := clusterState.Set(etcdserver.ClusterStateValueNew); err != nil {
+	fs.Usage = func() {
+		fmt.Println(usageline)
+		fmt.Println(flagsline)
+	}
+
+	fs.Var(clusterStateFlag, "initial-cluster-state", "Initial cluster configuration for bootstrapping")
+	if err := clusterStateFlag.Set(clusterStateFlagNew); err != nil {
 		// Should never happen.
-		log.Panicf("unexpected error setting up clusterState: %v", err)
+		log.Panicf("unexpected error setting up clusterStateFlag: %v", err)
 	}
 
 	fs.Var(flags.NewURLsValue("http://localhost:2380,http://localhost:7001"), "initial-advertise-peer-urls", "List of this member's peer URLs to advertise to the rest of the cluster")
@@ -89,11 +123,16 @@ func init() {
 
 	fs.Var(corsInfo, "cors", "Comma-separated white list of origins for CORS (cross-origin resource sharing).")
 
-	fs.Var(proxyFlag, "proxy", fmt.Sprintf("Valid values include %s", strings.Join(flags.ProxyValues, ", ")))
-	if err := proxyFlag.Set(flags.ProxyValueOff); err != nil {
+	fs.Var(proxyFlag, "proxy", fmt.Sprintf("Valid values include %s", strings.Join(proxyFlag.Values, ", ")))
+	if err := proxyFlag.Set(proxyFlagOff); err != nil {
 		// Should never happen.
 		log.Panicf("unexpected error setting up proxyFlag: %v", err)
 	}
+	fs.Var(fallbackFlag, "discovery-fallback", fmt.Sprintf("Valid values include %s", strings.Join(fallbackFlag.Values, ", ")))
+	if err := fallbackFlag.Set(fallbackFlagProxy); err != nil {
+		// Should never happen.
+		log.Panicf("unexpected error setting up discovery-fallback flag: %v", err)
+	}
 
 	fs.StringVar(&clientTLSInfo.CAFile, "ca-file", "", "Path to the client server TLS CA file.")
 	fs.StringVar(&clientTLSInfo.CertFile, "cert-file", "", "Path to the client server TLS cert file.")
@@ -118,9 +157,8 @@ func init() {
 }
 
 func Main() {
-	fs.Usage = flags.UsageWithIgnoredFlagsFunc(fs, ignored)
-	err := fs.Parse(os.Args[1:])
-	switch err {
+	perr := fs.Parse(os.Args[1:])
+	switch perr {
 	case nil:
 	case flag.ErrHelp:
 		os.Exit(0)
@@ -133,113 +171,190 @@ func Main() {
 		os.Exit(0)
 	}
 
-	flags.SetFlagsFromEnv(fs)
-
-	if string(*proxyFlag) == flags.ProxyValueOff {
-		startEtcd()
-	} else {
-		startProxy()
+	err := flags.SetFlagsFromEnv(fs)
+	if err != nil {
+		log.Fatalf("etcd: %v", err)
 	}
 
-	// Block indefinitely
-	<-make(chan struct{})
+	shouldProxy := proxyFlag.String() != proxyFlagOff
+	var stopped <-chan struct{}
+	if !shouldProxy {
+		stopped, err = startEtcd()
+		if err == discovery.ErrFullCluster && fallbackFlag.String() == fallbackFlagProxy {
+			log.Printf("etcd: discovery cluster full, falling back to %s", fallbackFlagProxy)
+			shouldProxy = true
+		}
+	}
+	if shouldProxy {
+		err = startProxy()
+	}
+	if err != nil {
+		switch err {
+		case discovery.ErrDuplicateID:
+			log.Fatalf("etcd: member %s has previously registered with discovery service (%s), but the data-dir (%s) on disk cannot be found.",
+				*name, *durl, *dir)
+		default:
+			log.Fatalf("etcd: %v", err)
+		}
+	}
+	<-stopped
 }
 
 // startEtcd launches the etcd server and HTTP handlers for client/server communication.
-func startEtcd() {
-	cls, err := setupCluster()
+func startEtcd() (<-chan struct{}, error) {
+	apurls, err := flags.URLsFromFlags(fs, "initial-advertise-peer-urls", "peer-addr", peerTLSInfo)
 	if err != nil {
-		log.Fatalf("etcd: error setting up initial cluster: %v", err)
+		return nil, err
+	}
+	cls, err := setupCluster(apurls)
+	if err != nil {
+		return nil, fmt.Errorf("error setting up initial cluster: %v", err)
 	}
 
 	if *dir == "" {
 		*dir = fmt.Sprintf("%v.etcd", *name)
-		log.Printf("etcd: no data-dir provided, using default data-dir ./%s", *dir)
+		log.Printf("no data-dir provided, using default data-dir ./%s", *dir)
 	}
 	if err := os.MkdirAll(*dir, privateDirMode); err != nil {
-		log.Fatalf("etcd: cannot create data directory: %v", err)
+		return nil, fmt.Errorf("cannot create data directory: %v", err)
 	}
 	if err := fileutil.IsDirWriteable(*dir); err != nil {
-		log.Fatalf("etcd: cannot write to data directory: %v", err)
+		return nil, fmt.Errorf("cannot write to data directory: %v", err)
 	}
 
-	pt, err := transport.NewTransport(peerTLSInfo)
+	pt, err := transport.NewTimeoutTransport(peerTLSInfo, rafthttp.ConnReadTimeout, rafthttp.ConnWriteTimeout)
 	if err != nil {
-		log.Fatal(err)
+		return nil, err
 	}
 
 	acurls, err := flags.URLsFromFlags(fs, "advertise-client-urls", "addr", clientTLSInfo)
 	if err != nil {
-		log.Fatal(err.Error())
+		return nil, err
 	}
+
+	lpurls, err := flags.URLsFromFlags(fs, "listen-peer-urls", "peer-bind-addr", peerTLSInfo)
+	if err != nil {
+		return nil, err
+	}
+
+	if !peerTLSInfo.Empty() {
+		log.Printf("etcd: peerTLS: %s", peerTLSInfo)
+	}
+	plns := make([]net.Listener, 0)
+	for _, u := range lpurls {
+		var l net.Listener
+		l, err = transport.NewTimeoutListener(u.Host, u.Scheme, peerTLSInfo, rafthttp.ConnReadTimeout, rafthttp.ConnWriteTimeout)
+		if err != nil {
+			return nil, err
+		}
+
+		urlStr := u.String()
+		log.Print("etcd: listening for peers on ", urlStr)
+		defer func() {
+			if err != nil {
+				l.Close()
+				log.Print("etcd: stopping listening for peers on ", urlStr)
+			}
+		}()
+		plns = append(plns, l)
+	}
+
+	lcurls, err := flags.URLsFromFlags(fs, "listen-client-urls", "bind-addr", clientTLSInfo)
+	if err != nil {
+		return nil, err
+	}
+
+	if !clientTLSInfo.Empty() {
+		log.Printf("etcd: clientTLS: %s", clientTLSInfo)
+	}
+	clns := make([]net.Listener, 0)
+	for _, u := range lcurls {
+		var l net.Listener
+		l, err = transport.NewListener(u.Host, u.Scheme, clientTLSInfo)
+		if err != nil {
+			return nil, err
+		}
+
+		urlStr := u.String()
+		log.Print("etcd: listening for client requests on ", urlStr)
+		defer func() {
+			if err != nil {
+				l.Close()
+				log.Print("etcd: stopping listening for client requests on ", urlStr)
+			}
+		}()
+		clns = append(clns, l)
+	}
+
 	cfg := &etcdserver.ServerConfig{
-		Name:         *name,
-		ClientURLs:   acurls,
-		DataDir:      *dir,
-		SnapCount:    *snapCount,
-		Cluster:      cls,
-		DiscoveryURL: *durl,
-		ClusterState: *clusterState,
-		Transport:    pt,
+		Name:            *name,
+		ClientURLs:      acurls,
+		PeerURLs:        apurls,
+		DataDir:         *dir,
+		SnapCount:       *snapCount,
+		MaxSnapFiles:    *maxSnapFiles,
+		MaxWALFiles:     *maxWalFiles,
+		Cluster:         cls,
+		DiscoveryURL:    *durl,
+		DiscoveryProxy:  *dproxy,
+		NewCluster:      clusterStateFlag.String() == clusterStateFlagNew,
+		ForceNewCluster: *forceNewCluster,
+		Transport:       pt,
+	}
+	var s *etcdserver.EtcdServer
+	s, err = etcdserver.NewServer(cfg)
+	if err != nil {
+		return nil, err
 	}
-	s := etcdserver.NewServer(cfg)
 	s.Start()
 
+	if corsInfo.String() != "" {
+		log.Printf("etcd: cors = %s", corsInfo)
+	}
 	ch := &cors.CORSHandler{
 		Handler: etcdhttp.NewClientHandler(s),
 		Info:    corsInfo,
 	}
 	ph := etcdhttp.NewPeerHandler(s)
-
-	lpurls, err := flags.URLsFromFlags(fs, "listen-peer-urls", "peer-bind-addr", peerTLSInfo)
-	if err != nil {
-		log.Fatal(err.Error())
+	// Start the peer server in a goroutine
+	for _, l := range plns {
+		go func(l net.Listener) {
+			log.Fatal(serveHTTP(l, ph))
+		}(l)
 	}
-
-	for _, u := range lpurls {
-		l, err := transport.NewListener(u.Host, peerTLSInfo)
-		if err != nil {
-			log.Fatal(err)
-		}
-
-		// Start the peer server in a goroutine
-		urlStr := u.String()
-		go func() {
-			log.Print("etcd: listening for peers on ", urlStr)
-			log.Fatal(http.Serve(l, ph))
-		}()
-	}
-
-	lcurls, err := flags.URLsFromFlags(fs, "listen-client-urls", "bind-addr", clientTLSInfo)
-	if err != nil {
-		log.Fatal(err.Error())
-	}
-
 	// Start a client server goroutine for each listen address
-	for _, u := range lcurls {
-		l, err := transport.NewListener(u.Host, clientTLSInfo)
-		if err != nil {
-			log.Fatal(err)
-		}
-
-		urlStr := u.String()
-		go func() {
-			log.Print("etcd: listening for client requests on ", urlStr)
-			log.Fatal(http.Serve(l, ch))
-		}()
+	for _, l := range clns {
+		go func(l net.Listener) {
+			log.Fatal(serveHTTP(l, ch))
+		}(l)
 	}
+	return s.StopNotify(), nil
 }
 
 // startProxy launches an HTTP proxy for client communication which proxies to other etcd nodes.
-func startProxy() {
-	cls, err := setupCluster()
+func startProxy() error {
+	apurls, err := flags.URLsFromFlags(fs, "initial-advertise-peer-urls", "peer-addr", peerTLSInfo)
 	if err != nil {
-		log.Fatalf("etcd: error setting up initial cluster: %v", err)
+		return err
+	}
+	cls, err := setupCluster(apurls)
+	if err != nil {
+		return fmt.Errorf("error setting up initial cluster: %v", err)
+	}
+
+	if *durl != "" {
+		s, err := discovery.GetCluster(*durl, *dproxy)
+		if err != nil {
+			return err
+		}
+		if cls, err = etcdserver.NewClusterFromString(*durl, s); err != nil {
+			return err
+		}
 	}
 
 	pt, err := transport.NewTransport(clientTLSInfo)
 	if err != nil {
-		log.Fatal(err)
+		return err
 	}
 
 	// TODO(jonboulle): update peerURLs dynamically (i.e. when updating
@@ -248,7 +363,7 @@ func startProxy() {
 	uf := func() []string {
 		cls, err := etcdserver.GetClusterFromPeers(peerURLs)
 		if err != nil {
-			log.Printf("etcd: %v", err)
+			log.Printf("proxy: %v", err)
 			return []string{}
 		}
 		return cls.ClientURLs()
@@ -259,31 +374,31 @@ func startProxy() {
 		Info:    corsInfo,
 	}
 
-	if string(*proxyFlag) == flags.ProxyValueReadonly {
+	if proxyFlag.String() == proxyFlagReadonly {
 		ph = proxy.NewReadonlyHandler(ph)
 	}
-
 	lcurls, err := flags.URLsFromFlags(fs, "listen-client-urls", "bind-addr", clientTLSInfo)
 	if err != nil {
-		log.Fatal(err.Error())
+		return err
 	}
 	// Start a proxy server goroutine for each listen address
 	for _, u := range lcurls {
-		l, err := transport.NewListener(u.Host, clientTLSInfo)
+		l, err := transport.NewListener(u.Host, u.Scheme, clientTLSInfo)
 		if err != nil {
-			log.Fatal(err)
+			return err
 		}
 
 		host := u.Host
 		go func() {
-			log.Print("etcd: proxy listening for client requests on ", host)
+			log.Print("proxy: listening for client requests on ", host)
 			log.Fatal(http.Serve(l, ph))
 		}()
 	}
+	return nil
 }
 
-// setupCluster sets up the cluster definition for bootstrap or discovery.
-func setupCluster() (*etcdserver.Cluster, error) {
+// setupCluster sets up an initial cluster definition for bootstrap or discovery.
+func setupCluster(apurls []url.URL) (*etcdserver.Cluster, error) {
 	set := make(map[string]bool)
 	fs.Visit(func(f *flag.Flag) {
 		set[f.Name] = true
@@ -291,21 +406,18 @@ func setupCluster() (*etcdserver.Cluster, error) {
 	if set["discovery"] && set["initial-cluster"] {
 		return nil, fmt.Errorf("both discovery and bootstrap-config are set")
 	}
-	apurls, err := flags.URLsFromFlags(fs, "initial-advertise-peer-urls", "addr", peerTLSInfo)
-	if err != nil {
-		return nil, err
-	}
-
 	var cls *etcdserver.Cluster
+	var err error
 	switch {
 	case set["discovery"]:
+		// If using discovery, generate a temporary cluster based on
+		// self's advertised peer URLs
 		clusterStr := genClusterString(*name, apurls)
 		cls, err = etcdserver.NewClusterFromString(*durl, clusterStr)
 	case set["initial-cluster"]:
 		fallthrough
 	default:
 		// We're statically configured, and cluster has appropriately been set.
-		// Try to configure by indexing the static cluster by name.
 		cls, err = etcdserver.NewClusterFromString(*initialClusterToken, *initialCluster)
 	}
 	return cls, err

etcdmain/etcd_test.go

@@ -17,11 +17,20 @@
 package etcdmain
 
 import (
+	"net/url"
 	"testing"
 
 	"github.com/coreos/etcd/pkg/types"
 )
 
+func mustNewURLs(t *testing.T, urls []string) []url.URL {
+	u, err := types.NewURLs(urls)
+	if err != nil {
+		t.Fatalf("unexpected new urls error: %v", err)
+	}
+	return u
+}
+
 func TestGenClusterString(t *testing.T) {
 	tests := []struct {
 		token string
@@ -38,10 +47,7 @@ func TestGenClusterString(t *testing.T) {
 		},
 	}
 	for i, tt := range tests {
-		urls, err := types.NewURLs(tt.urls)
-		if err != nil {
-			t.Fatalf("unexpected new urls error: %v", err)
-		}
+		urls := mustNewURLs(t, tt.urls)
 		str := genClusterString(tt.token, urls)
 		if str != tt.wstr {
 			t.Errorf("#%d: cluster = %s, want %s", i, str, tt.wstr)

etcdmain/help.go

@@ -0,0 +1,80 @@
+package etcdmain
+
+var (
+	usageline = `usage: etcd [flags]
+       start an etcd server
+
+       etcd --version
+       show the version of etcd
+
+       etcd -h | --help
+       show the help information about etcd
+	`
+	flagsline = `
+member flags:
+
+	--name 'default'
+		human-readable name for this member.
+	--data-dir '${name}.etcd'
+		path to the data directory.
+	--snapshot-count '10000'
+		number of committed transactions to trigger a snapshot to disk.
+	--listen-peer-urls 'http://localhost:2380,http://localhost:7001'
+		list of URLs to listen on for peer traffic.
+	--listen-client-urls 'http://localhost:2379,http://localhost:4001'
+		list of URLs to listen on for client traffic.
+	-cors ''
+		comma-separated whitelist of origins for CORS (cross-origin resource sharing).
+
+
+clustering flags:
+
+	--initial-advertise-peer-urls 'http://localhost:2380,http://localhost:7001'
+		list of this member's peer URLs to advertise to the rest of the cluster. 
+	--initial-cluster 'default=http://localhost:2380,default=http://localhost:7001'
+		initial cluster configuration for bootstrapping.
+	--initial-cluster-state 'new'
+		initial cluster state ('new' or 'existing').
+	--initial-cluster-token 'etcd-cluster'
+		initial cluster token for the etcd cluster during bootstrap.
+	--advertise-client-urls 'http://localhost:2379,http://localhost:4001'
+		list of this member's client URLs to advertise to the rest of the cluster.
+	--discovery ''
+		discovery URL used to bootstrap the cluster.
+	--discovery-fallback 'proxy'
+		expected behavior ('exit' or 'proxy') when discovery services fails.
+	--discovery-proxy ''
+		HTTP proxy to use for traffic to discovery service.
+
+
+proxy flags:
+
+	--proxy 'off'
+		proxy mode setting ('off', 'readonly' or 'on').
+
+
+security flags:
+
+	--ca-file ''
+		path to the client server TLS CA file.
+	--cert-file ''
+		path to the client server TLS cert file.
+	--key-file ''
+		path to the client server TLS key file.
+	--peer-ca-file ''
+		path to the peer server TLS CA file.
+	--peer-cert-file ''
+		path to the peer server TLS cert file.
+	--peer-key-file ''
+		path to the peer server TLS key file.
+
+
+unsafe flags:
+
+Please be CAUTIOUS to use unsafe flags because it will break the guarantee given 
+by consensus protocol. 
+	
+	--force-new-cluster 'false'
+		force to create a new one-member cluster.
+`
+)

etcdmain/http.go

@@ -0,0 +1,21 @@
+package etcdmain
+
+import (
+	"io/ioutil"
+	"log"
+	"net"
+	"net/http"
+)
+
+// serveHTTP accepts incoming HTTP connections on the listener l,
+// creating a new service goroutine for each. The service goroutines
+// read requests and then call handler to reply to them.
+func serveHTTP(l net.Listener, handler http.Handler) error {
+	logger := log.New(ioutil.Discard, "etcdhttp", 0)
+	// TODO: add debug flag; enable logging when debug flag is set
+	srv := &http.Server{
+		Handler:  handler,
+		ErrorLog: logger, // do not log user error
+	}
+	return srv.Serve(l)
+}

etcdserver/cluster.go

@@ -27,10 +27,12 @@ import (
 	"reflect"
 	"sort"
 	"strings"
+	"sync"
 
 	etcdErr "github.com/coreos/etcd/error"
 	"github.com/coreos/etcd/pkg/flags"
 	"github.com/coreos/etcd/pkg/types"
+	"github.com/coreos/etcd/raft/raftpb"
 	"github.com/coreos/etcd/store"
 )
 
@@ -40,11 +42,19 @@ const (
 )
 
 type ClusterInfo interface {
+	// ID returns the cluster ID
 	ID() types.ID
+	// ClientURLs returns an aggregate set of all URLs on which this
+	// cluster is listening for client requests
 	ClientURLs() []string
 	// Members returns a slice of members sorted by their ID
 	Members() []*Member
+	// Member retrieves a particular member based on ID, or nil if the
+	// member does not exist in the cluster
 	Member(id types.ID) *Member
+	// IsIDRemoved checks whether the given ID has been removed from this
+	// cluster at some point in the past
+	IsIDRemoved(id types.ID) bool
 }
 
 // Cluster is a list of Members that belong to the same raft cluster
@@ -56,10 +66,12 @@ type Cluster struct {
 	// removed id cannot be reused.
 	removed map[types.ID]bool
 	store   store.Store
+	sync.Mutex
 }
 
-// NewClusterFromString returns Cluster through given cluster token and parsing
-// members from a sets of names to IPs discovery formatted like:
+// NewClusterFromString returns a Cluster instantiated from the given cluster token
+// and cluster string, by parsing members from a set of discovery-formatted
+// names-to-IPs, like:
 // mach0=http://1.1.1.1,mach0=http://2.2.2.2,mach1=http://3.3.3.3,mach2=http://4.4.4.4
 func NewClusterFromString(token string, cluster string) (*Cluster, error) {
 	c := newCluster(token)
@@ -89,33 +101,7 @@ func NewClusterFromString(token string, cluster string) (*Cluster, error) {
 func NewClusterFromStore(token string, st store.Store) *Cluster {
 	c := newCluster(token)
 	c.store = st
-
-	e, err := c.store.Get(storeMembersPrefix, true, true)
-	if err != nil {
-		if isKeyNotFound(err) {
-			return c
-		}
-		log.Panicf("get storeMembers should never fail: %v", err)
-	}
-	for _, n := range e.Node.Nodes {
-		m, err := nodeToMember(n)
-		if err != nil {
-			log.Panicf("nodeToMember should never fail: %v", err)
-		}
-		c.members[m.ID] = m
-	}
-
-	e, err = c.store.Get(storeRemovedMembersPrefix, true, true)
-	if err != nil {
-		if isKeyNotFound(err) {
-			return c
-		}
-		log.Panicf("get storeRemovedMembers should never fail: %v", err)
-	}
-	for _, n := range e.Node.Nodes {
-		c.removed[mustParseMemberIDFromKey(n.Key)] = true
-	}
-
+	c.members, c.removed = membersFromStore(c.store)
 	return c
 }
 
@@ -136,12 +122,14 @@ func newCluster(token string) *Cluster {
 	}
 }
 
-func (c Cluster) ID() types.ID { return c.id }
+func (c *Cluster) ID() types.ID { return c.id }
 
-func (c Cluster) Members() []*Member {
+func (c *Cluster) Members() []*Member {
+	c.Lock()
+	defer c.Unlock()
 	var sms SortableMemberSlice
 	for _, m := range c.members {
-		sms = append(sms, m)
+		sms = append(sms, m.Clone())
 	}
 	sort.Sort(sms)
 	return []*Member(sms)
@@ -154,25 +142,31 @@ func (s SortableMemberSlice) Less(i, j int) bool { return s[i].ID < s[j].ID }
 func (s SortableMemberSlice) Swap(i, j int)      { s[i], s[j] = s[j], s[i] }
 
 func (c *Cluster) Member(id types.ID) *Member {
-	return c.members[id]
+	c.Lock()
+	defer c.Unlock()
+	return c.members[id].Clone()
 }
 
 // MemberByName returns a Member with the given name if exists.
 // If more than one member has the given name, it will panic.
 func (c *Cluster) MemberByName(name string) *Member {
+	c.Lock()
+	defer c.Unlock()
 	var memb *Member
 	for _, m := range c.members {
 		if m.Name == name {
 			if memb != nil {
-				panic("two members with the given name exist in the cluster")
+				log.Panicf("two members with the given name %q exist", name)
 			}
 			memb = m
 		}
 	}
-	return memb
+	return memb.Clone()
 }
 
-func (c Cluster) MemberIDs() []types.ID {
+func (c *Cluster) MemberIDs() []types.ID {
+	c.Lock()
+	defer c.Unlock()
 	var ids []types.ID
 	for _, m := range c.members {
 		ids = append(ids, m.ID)
@@ -182,13 +176,17 @@ func (c Cluster) MemberIDs() []types.ID {
 }
 
 func (c *Cluster) IsIDRemoved(id types.ID) bool {
+	c.Lock()
+	defer c.Unlock()
 	return c.removed[id]
 }
 
 // PeerURLs returns a list of all peer addresses. Each address is prefixed
 // with the scheme (currently "http://"). The returned list is sorted in
 // ascending lexicographical order.
-func (c Cluster) PeerURLs() []string {
+func (c *Cluster) PeerURLs() []string {
+	c.Lock()
+	defer c.Unlock()
 	endpoints := make([]string, 0)
 	for _, p := range c.members {
 		for _, addr := range p.PeerURLs {
@@ -202,7 +200,9 @@ func (c Cluster) PeerURLs() []string {
 // ClientURLs returns a list of all client addresses. Each address is prefixed
 // with the scheme (currently "http://"). The returned list is sorted in
 // ascending lexicographical order.
-func (c Cluster) ClientURLs() []string {
+func (c *Cluster) ClientURLs() []string {
+	c.Lock()
+	defer c.Unlock()
 	urls := make([]string, 0)
 	for _, p := range c.members {
 		for _, url := range p.ClientURLs {
@@ -213,7 +213,9 @@ func (c Cluster) ClientURLs() []string {
 	return urls
 }
 
-func (c Cluster) String() string {
+func (c *Cluster) String() string {
+	c.Lock()
+	defer c.Unlock()
 	sl := []string{}
 	for _, m := range c.members {
 		for _, u := range m.PeerURLs {
@@ -224,33 +226,6 @@ func (c Cluster) String() string {
 	return strings.Join(sl, ",")
 }
 
-// ValidateAndAssignIDs validates the given members by matching their PeerURLs
-// with the existing members in the cluster. If the validation succeeds, it
-// assigns the IDs from the given members to the existing members in the
-// cluster. If the validation fails, an error will be returned.
-func (c *Cluster) ValidateAndAssignIDs(membs []*Member) error {
-	if len(c.members) != len(membs) {
-		return fmt.Errorf("member count is unequal")
-	}
-	omembs := make([]*Member, 0)
-	for _, m := range c.members {
-		omembs = append(omembs, m)
-	}
-	sort.Sort(SortableMemberSliceByPeerURLs(omembs))
-	sort.Sort(SortableMemberSliceByPeerURLs(membs))
-	for i := range omembs {
-		if !reflect.DeepEqual(omembs[i].PeerURLs, membs[i].PeerURLs) {
-			return fmt.Errorf("unmatched member while checking PeerURLs")
-		}
-		omembs[i].ID = membs[i].ID
-	}
-	c.members = make(map[types.ID]*Member)
-	for _, m := range omembs {
-		c.members[m.ID] = m
-	}
-	return nil
-}
-
 func (c *Cluster) genID() {
 	mIDs := c.MemberIDs()
 	b := make([]byte, 8*len(mIDs))
@@ -265,24 +240,83 @@ func (c *Cluster) SetID(id types.ID) { c.id = id }
 
 func (c *Cluster) SetStore(st store.Store) { c.store = st }
 
-// AddMember puts a new Member into the store.
+func (c *Cluster) Recover() {
+	c.members, c.removed = membersFromStore(c.store)
+}
+
+// ValidateConfigurationChange takes a proposed ConfChange and
+// ensures that it is still valid.
+func (c *Cluster) ValidateConfigurationChange(cc raftpb.ConfChange) error {
+	members, removed := membersFromStore(c.store)
+	id := types.ID(cc.NodeID)
+	if removed[id] {
+		return ErrIDRemoved
+	}
+	switch cc.Type {
+	case raftpb.ConfChangeAddNode:
+		if members[id] != nil {
+			return ErrIDExists
+		}
+		urls := make(map[string]bool)
+		for _, m := range members {
+			for _, u := range m.PeerURLs {
+				urls[u] = true
+			}
+		}
+		m := new(Member)
+		if err := json.Unmarshal(cc.Context, m); err != nil {
+			log.Panicf("unmarshal member should never fail: %v", err)
+		}
+		for _, u := range m.PeerURLs {
+			if urls[u] {
+				return ErrPeerURLexists
+			}
+		}
+	case raftpb.ConfChangeRemoveNode:
+		if members[id] == nil {
+			return ErrIDNotFound
+		}
+	case raftpb.ConfChangeUpdateNode:
+		if members[id] == nil {
+			return ErrIDNotFound
+		}
+		urls := make(map[string]bool)
+		for _, m := range members {
+			if m.ID == id {
+				continue
+			}
+			for _, u := range m.PeerURLs {
+				urls[u] = true
+			}
+		}
+		m := new(Member)
+		if err := json.Unmarshal(cc.Context, m); err != nil {
+			log.Panicf("unmarshal member should never fail: %v", err)
+		}
+		for _, u := range m.PeerURLs {
+			if urls[u] {
+				return ErrPeerURLexists
+			}
+		}
+	default:
+		log.Panicf("ConfChange type should be either AddNode, RemoveNode or UpdateNode")
+	}
+	return nil
+}
+
+// AddMember adds a new Member into the cluster, and saves the given member's
+// raftAttributes into the store. The given member should have empty attributes.
 // A Member with a matching id must not exist.
 func (c *Cluster) AddMember(m *Member) {
+	c.Lock()
+	defer c.Unlock()
 	b, err := json.Marshal(m.RaftAttributes)
 	if err != nil {
-		log.Panicf("marshal error: %v", err)
+		log.Panicf("marshal raftAttributes should never fail: %v", err)
 	}
 	p := path.Join(memberStoreKey(m.ID), raftAttributesSuffix)
 	if _, err := c.store.Create(p, false, string(b), false, store.Permanent); err != nil {
-		log.Panicf("add raftAttributes should never fail: %v", err)
-	}
-	b, err = json.Marshal(m.Attributes)
-	if err != nil {
-		log.Panicf("marshal error: %v", err)
-	}
-	p = path.Join(memberStoreKey(m.ID), attributesSuffix)
-	if _, err := c.store.Create(p, false, string(b), false, store.Permanent); err != nil {
-		log.Panicf("add attributes should never fail: %v", err)
+		log.Panicf("create raftAttributes should never fail: %v", err)
 	}
 	c.members[m.ID] = m
 }
@@ -290,38 +324,123 @@ func (c *Cluster) AddMember(m *Member) {
 // RemoveMember removes a member from the store.
 // The given id MUST exist, or the function panics.
 func (c *Cluster) RemoveMember(id types.ID) {
+	c.Lock()
+	defer c.Unlock()
 	if _, err := c.store.Delete(memberStoreKey(id), true, true); err != nil {
-		log.Panicf("delete peer should never fail: %v", err)
+		log.Panicf("delete member should never fail: %v", err)
 	}
 	delete(c.members, id)
 	if _, err := c.store.Create(removedMemberStoreKey(id), false, "", false, store.Permanent); err != nil {
-		log.Panicf("creating RemovedMember should never fail: %v", err)
+		log.Panicf("create removedMember should never fail: %v", err)
 	}
 	c.removed[id] = true
 }
 
+func (c *Cluster) UpdateMemberAttributes(id types.ID, attr Attributes) {
+	c.Lock()
+	defer c.Unlock()
+	c.members[id].Attributes = attr
+}
+
+func (c *Cluster) UpdateMember(nm *Member) {
+	c.Lock()
+	defer c.Unlock()
+	b, err := json.Marshal(nm.RaftAttributes)
+	if err != nil {
+		log.Panicf("marshal raftAttributes should never fail: %v", err)
+	}
+	p := path.Join(memberStoreKey(nm.ID), raftAttributesSuffix)
+	if _, err := c.store.Update(p, string(b), store.Permanent); err != nil {
+		log.Panicf("update raftAttributes should never fail: %v", err)
+	}
+	c.members[nm.ID].RaftAttributes = nm.RaftAttributes
+}
+
 // nodeToMember builds member through a store node.
 // the child nodes of the given node should be sorted by key.
 func nodeToMember(n *store.NodeExtern) (*Member, error) {
 	m := &Member{ID: mustParseMemberIDFromKey(n.Key)}
-	if len(n.Nodes) != 2 {
-		return m, fmt.Errorf("len(nodes) = %d, want 2", len(n.Nodes))
+	attrs := make(map[string][]byte)
+	raftAttrKey := path.Join(n.Key, raftAttributesSuffix)
+	attrKey := path.Join(n.Key, attributesSuffix)
+	for _, nn := range n.Nodes {
+		if nn.Key != raftAttrKey && nn.Key != attrKey {
+			return nil, fmt.Errorf("unknown key %q", nn.Key)
+		}
+		attrs[nn.Key] = []byte(*nn.Value)
 	}
-	if w := path.Join(n.Key, attributesSuffix); n.Nodes[0].Key != w {
-		return m, fmt.Errorf("key = %v, want %v", n.Nodes[0].Key, w)
+	if data := attrs[raftAttrKey]; data != nil {
+		if err := json.Unmarshal(data, &m.RaftAttributes); err != nil {
+			return nil, fmt.Errorf("unmarshal raftAttributes error: %v", err)
+		}
+	} else {
+		return nil, fmt.Errorf("raftAttributes key doesn't exist")
 	}
-	if err := json.Unmarshal([]byte(*n.Nodes[0].Value), &m.Attributes); err != nil {
-		return m, fmt.Errorf("unmarshal attributes error: %v", err)
-	}
-	if w := path.Join(n.Key, raftAttributesSuffix); n.Nodes[1].Key != w {
-		return m, fmt.Errorf("key = %v, want %v", n.Nodes[1].Key, w)
-	}
-	if err := json.Unmarshal([]byte(*n.Nodes[1].Value), &m.RaftAttributes); err != nil {
-		return m, fmt.Errorf("unmarshal raftAttributes error: %v", err)
+	if data := attrs[attrKey]; data != nil {
+		if err := json.Unmarshal(data, &m.Attributes); err != nil {
+			return m, fmt.Errorf("unmarshal attributes error: %v", err)
+		}
 	}
 	return m, nil
 }
 
+func membersFromStore(st store.Store) (map[types.ID]*Member, map[types.ID]bool) {
+	members := make(map[types.ID]*Member)
+	removed := make(map[types.ID]bool)
+	e, err := st.Get(storeMembersPrefix, true, true)
+	if err != nil {
+		if isKeyNotFound(err) {
+			return members, removed
+		}
+		log.Panicf("get storeMembers should never fail: %v", err)
+	}
+	for _, n := range e.Node.Nodes {
+		m, err := nodeToMember(n)
+		if err != nil {
+			log.Panicf("nodeToMember should never fail: %v", err)
+		}
+		members[m.ID] = m
+	}
+
+	e, err = st.Get(storeRemovedMembersPrefix, true, true)
+	if err != nil {
+		if isKeyNotFound(err) {
+			return members, removed
+		}
+		log.Panicf("get storeRemovedMembers should never fail: %v", err)
+	}
+	for _, n := range e.Node.Nodes {
+		removed[mustParseMemberIDFromKey(n.Key)] = true
+	}
+	return members, removed
+}
+
+// ValidateClusterAndAssignIDs validates the local cluster by matching the PeerURLs
+// with the existing cluster. If the validation succeeds, it assigns the IDs
+// from the existing cluster to the local cluster.
+// If the validation fails, an error will be returned.
+func ValidateClusterAndAssignIDs(local *Cluster, existing *Cluster) error {
+	ems := existing.Members()
+	lms := local.Members()
+	if len(ems) != len(lms) {
+		return fmt.Errorf("member count is unequal")
+	}
+	sort.Sort(SortableMemberSliceByPeerURLs(ems))
+	sort.Sort(SortableMemberSliceByPeerURLs(lms))
+
+	for i := range ems {
+		if !reflect.DeepEqual(ems[i].PeerURLs, lms[i].PeerURLs) {
+			return fmt.Errorf("unmatched member while checking PeerURLs")
+		}
+		lms[i].ID = ems[i].ID
+	}
+	local.members = make(map[types.ID]*Member)
+	for _, m := range lms {
+		local.members[m.ID] = m
+	}
+	return nil
+}
+
 func isKeyNotFound(err error) bool {
 	e, ok := err.(*etcdErr.Error)
 	return ok && e.ErrorCode == etcdErr.EcodeKeyNotFound

etcdserver/cluster_state.go

@@ -1,53 +0,0 @@
-/*
-   Copyright 2014 CoreOS, Inc.
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
-*/
-
-package etcdserver
-
-import (
-	"errors"
-)
-
-const (
-	ClusterStateValueNew      = "new"
-	ClusterStateValueExisting = "existing"
-)
-
-var (
-	ClusterStateValues = []string{
-		ClusterStateValueNew,
-		ClusterStateValueExisting,
-	}
-)
-
-// ClusterState implements the flag.Value interface.
-type ClusterState string
-
-// Set verifies the argument to be a valid member of ClusterStateFlagValues
-// before setting the underlying flag value.
-func (cs *ClusterState) Set(s string) error {
-	for _, v := range ClusterStateValues {
-		if s == v {
-			*cs = ClusterState(s)
-			return nil
-		}
-	}
-
-	return errors.New("invalid value")
-}
-
-func (cs *ClusterState) String() string {
-	return string(*cs)
-}

etcdserver/cluster_test.go

@@ -17,24 +17,27 @@
 package etcdserver
 
 import (
+	"encoding/json"
+	"fmt"
 	"path"
 	"reflect"
 	"testing"
 
 	"github.com/coreos/etcd/pkg/types"
+	"github.com/coreos/etcd/raft/raftpb"
 	"github.com/coreos/etcd/store"
 )
 
 func TestClusterFromString(t *testing.T) {
 	tests := []struct {
 		f    string
-		mems []Member
+		mems []*Member
 	}{
 		{
 			"mem1=http://10.0.0.1:2379,mem1=http://128.193.4.20:2379,mem2=http://10.0.0.2:2379,default=http://127.0.0.1:2379",
-			[]Member{
-				newTestMember(4322322643958477905, []string{"http://10.0.0.1:2379", "http://128.193.4.20:2379"}, "mem1", nil),
+			[]*Member{
 				newTestMember(3141198903430435750, []string{"http://10.0.0.2:2379"}, "mem2", nil),
+				newTestMember(4322322643958477905, []string{"http://10.0.0.1:2379", "http://128.193.4.20:2379"}, "mem1", nil),
 				newTestMember(12762790032478827328, []string{"http://127.0.0.1:2379"}, "default", nil),
 			},
 		},
@@ -47,9 +50,8 @@ func TestClusterFromString(t *testing.T) {
 		if c.token != "abc" {
 			t.Errorf("#%d: token = %v, want abc", i, c.token)
 		}
-		wc := newTestCluster(tt.mems)
-		if !reflect.DeepEqual(c.members, wc.members) {
-			t.Errorf("#%d: members = %+v, want %+v", i, c.members, wc.members)
+		if !reflect.DeepEqual(c.Members(), tt.mems) {
+			t.Errorf("#%d: members = %+v, want %+v", i, c.Members(), tt.mems)
 		}
 	}
 }
@@ -77,41 +79,39 @@ func TestClusterFromStringBad(t *testing.T) {
 
 func TestClusterFromStore(t *testing.T) {
 	tests := []struct {
-		mems []Member
+		mems []*Member
 	}{
 		{
-			[]Member{newTestMember(1, nil, "node1", nil)},
+			[]*Member{newTestMember(1, nil, "", nil)},
 		},
 		{
-			[]Member{},
+			nil,
 		},
 		{
-			[]Member{
-				newTestMember(1, nil, "node1", nil),
-				newTestMember(2, nil, "node2", nil),
+			[]*Member{
+				newTestMember(1, nil, "", nil),
+				newTestMember(2, nil, "", nil),
 			},
 		},
 	}
 	for i, tt := range tests {
-		st := store.New()
 		hc := newTestCluster(nil)
-		hc.SetStore(st)
+		hc.SetStore(store.New())
 		for _, m := range tt.mems {
-			hc.AddMember(&m)
+			hc.AddMember(m)
 		}
-		c := NewClusterFromStore("abc", st)
+		c := NewClusterFromStore("abc", hc.store)
 		if c.token != "abc" {
 			t.Errorf("#%d: token = %v, want %v", i, c.token, "abc")
 		}
-		wc := newTestCluster(tt.mems)
-		if !reflect.DeepEqual(c.members, wc.members) {
-			t.Errorf("#%d: members = %v, want %v", i, c.members, wc.members)
+		if !reflect.DeepEqual(c.Members(), tt.mems) {
+			t.Errorf("#%d: members = %v, want %v", i, c.Members(), tt.mems)
 		}
 	}
 }
 
 func TestClusterMember(t *testing.T) {
-	membs := []Member{
+	membs := []*Member{
 		newTestMember(1, nil, "node1", nil),
 		newTestMember(2, nil, "node2", nil),
 	}
@@ -136,7 +136,7 @@ func TestClusterMember(t *testing.T) {
 }
 
 func TestClusterMemberByName(t *testing.T) {
-	membs := []Member{
+	membs := []*Member{
 		newTestMember(1, nil, "node1", nil),
 		newTestMember(2, nil, "node2", nil),
 	}
@@ -161,7 +161,7 @@ func TestClusterMemberByName(t *testing.T) {
 }
 
 func TestClusterMemberIDs(t *testing.T) {
-	c := newTestCluster([]Member{
+	c := newTestCluster([]*Member{
 		newTestMember(1, nil, "", nil),
 		newTestMember(4, nil, "", nil),
 		newTestMember(100, nil, "", nil),
@@ -175,12 +175,12 @@ func TestClusterMemberIDs(t *testing.T) {
 
 func TestClusterPeerURLs(t *testing.T) {
 	tests := []struct {
-		mems  []Member
+		mems  []*Member
 		wurls []string
 	}{
 		// single peer with a single address
 		{
-			mems: []Member{
+			mems: []*Member{
 				newTestMember(1, []string{"http://192.0.2.1"}, "", nil),
 			},
 			wurls: []string{"http://192.0.2.1"},
@@ -188,7 +188,7 @@ func TestClusterPeerURLs(t *testing.T) {
 
 		// single peer with a single address with a port
 		{
-			mems: []Member{
+			mems: []*Member{
 				newTestMember(1, []string{"http://192.0.2.1:8001"}, "", nil),
 			},
 			wurls: []string{"http://192.0.2.1:8001"},
@@ -196,7 +196,7 @@ func TestClusterPeerURLs(t *testing.T) {
 
 		// several members explicitly unsorted
 		{
-			mems: []Member{
+			mems: []*Member{
 				newTestMember(2, []string{"http://192.0.2.3", "http://192.0.2.4"}, "", nil),
 				newTestMember(3, []string{"http://192.0.2.5", "http://192.0.2.6"}, "", nil),
 				newTestMember(1, []string{"http://192.0.2.1", "http://192.0.2.2"}, "", nil),
@@ -206,13 +206,13 @@ func TestClusterPeerURLs(t *testing.T) {
 
 		// no members
 		{
-			mems:  []Member{},
+			mems:  []*Member{},
 			wurls: []string{},
 		},
 
 		// peer with no peer urls
 		{
-			mems: []Member{
+			mems: []*Member{
 				newTestMember(3, []string{}, "", nil),
 			},
 			wurls: []string{},
@@ -230,12 +230,12 @@ func TestClusterPeerURLs(t *testing.T) {
 
 func TestClusterClientURLs(t *testing.T) {
 	tests := []struct {
-		mems  []Member
+		mems  []*Member
 		wurls []string
 	}{
 		// single peer with a single address
 		{
-			mems: []Member{
+			mems: []*Member{
 				newTestMember(1, nil, "", []string{"http://192.0.2.1"}),
 			},
 			wurls: []string{"http://192.0.2.1"},
@@ -243,7 +243,7 @@ func TestClusterClientURLs(t *testing.T) {
 
 		// single peer with a single address with a port
 		{
-			mems: []Member{
+			mems: []*Member{
 				newTestMember(1, nil, "", []string{"http://192.0.2.1:8001"}),
 			},
 			wurls: []string{"http://192.0.2.1:8001"},
@@ -251,7 +251,7 @@ func TestClusterClientURLs(t *testing.T) {
 
 		// several members explicitly unsorted
 		{
-			mems: []Member{
+			mems: []*Member{
 				newTestMember(2, nil, "", []string{"http://192.0.2.3", "http://192.0.2.4"}),
 				newTestMember(3, nil, "", []string{"http://192.0.2.5", "http://192.0.2.6"}),
 				newTestMember(1, nil, "", []string{"http://192.0.2.1", "http://192.0.2.2"}),
@@ -261,13 +261,13 @@ func TestClusterClientURLs(t *testing.T) {
 
 		// no members
 		{
-			mems:  []Member{},
+			mems:  []*Member{},
 			wurls: []string{},
 		},
 
 		// peer with no client urls
 		{
-			mems: []Member{
+			mems: []*Member{
 				newTestMember(3, nil, "", []string{}),
 			},
 			wurls: []string{},
@@ -285,40 +285,41 @@ func TestClusterClientURLs(t *testing.T) {
 
 func TestClusterValidateAndAssignIDsBad(t *testing.T) {
 	tests := []struct {
-		clmembs []Member
+		clmembs []*Member
 		membs   []*Member
 	}{
 		{
 			// unmatched length
-			[]Member{
+			[]*Member{
 				newTestMember(1, []string{"http://127.0.0.1:2379"}, "", nil),
 			},
 			[]*Member{},
 		},
 		{
 			// unmatched peer urls
-			[]Member{
+			[]*Member{
 				newTestMember(1, []string{"http://127.0.0.1:2379"}, "", nil),
 			},
 			[]*Member{
-				newTestMemberp(1, []string{"http://127.0.0.1:4001"}, "", nil),
+				newTestMember(1, []string{"http://127.0.0.1:4001"}, "", nil),
 			},
 		},
 		{
 			// unmatched peer urls
-			[]Member{
+			[]*Member{
 				newTestMember(1, []string{"http://127.0.0.1:2379"}, "", nil),
 				newTestMember(2, []string{"http://127.0.0.2:2379"}, "", nil),
 			},
 			[]*Member{
-				newTestMemberp(1, []string{"http://127.0.0.1:2379"}, "", nil),
-				newTestMemberp(2, []string{"http://127.0.0.2:4001"}, "", nil),
+				newTestMember(1, []string{"http://127.0.0.1:2379"}, "", nil),
+				newTestMember(2, []string{"http://127.0.0.2:4001"}, "", nil),
 			},
 		},
 	}
 	for i, tt := range tests {
-		cl := newTestCluster(tt.clmembs)
-		if err := cl.ValidateAndAssignIDs(tt.membs); err == nil {
+		ecl := newTestCluster(tt.clmembs)
+		lcl := newTestCluster(tt.membs)
+		if err := ValidateClusterAndAssignIDs(lcl, ecl); err == nil {
 			t.Errorf("#%d: unexpected update success", i)
 		}
 	}
@@ -326,35 +327,158 @@ func TestClusterValidateAndAssignIDsBad(t *testing.T) {
 
 func TestClusterValidateAndAssignIDs(t *testing.T) {
 	tests := []struct {
-		clmembs []Member
+		clmembs []*Member
 		membs   []*Member
 		wids    []types.ID
 	}{
 		{
-			[]Member{
+			[]*Member{
 				newTestMember(1, []string{"http://127.0.0.1:2379"}, "", nil),
 				newTestMember(2, []string{"http://127.0.0.2:2379"}, "", nil),
 			},
 			[]*Member{
-				newTestMemberp(3, []string{"http://127.0.0.1:2379"}, "", nil),
-				newTestMemberp(4, []string{"http://127.0.0.2:2379"}, "", nil),
+				newTestMember(3, []string{"http://127.0.0.1:2379"}, "", nil),
+				newTestMember(4, []string{"http://127.0.0.2:2379"}, "", nil),
 			},
 			[]types.ID{3, 4},
 		},
 	}
 	for i, tt := range tests {
-		cl := newTestCluster(tt.clmembs)
-		if err := cl.ValidateAndAssignIDs(tt.membs); err != nil {
+		lcl := newTestCluster(tt.clmembs)
+		ecl := newTestCluster(tt.membs)
+		if err := ValidateClusterAndAssignIDs(lcl, ecl); err != nil {
 			t.Errorf("#%d: unexpect update error: %v", i, err)
 		}
-		if !reflect.DeepEqual(cl.MemberIDs(), tt.wids) {
-			t.Errorf("#%d: ids = %v, want %v", i, cl.MemberIDs(), tt.wids)
+		if !reflect.DeepEqual(lcl.MemberIDs(), tt.wids) {
+			t.Errorf("#%d: ids = %v, want %v", i, lcl.MemberIDs(), tt.wids)
+		}
+	}
+}
+
+func TestClusterValidateConfigurationChange(t *testing.T) {
+	cl := newCluster("")
+	cl.SetStore(store.New())
+	for i := 1; i <= 4; i++ {
+		attr := RaftAttributes{PeerURLs: []string{fmt.Sprintf("http://127.0.0.1:%d", i)}}
+		cl.AddMember(&Member{ID: types.ID(i), RaftAttributes: attr})
+	}
+	cl.RemoveMember(4)
+
+	attr := RaftAttributes{PeerURLs: []string{fmt.Sprintf("http://127.0.0.1:%d", 1)}}
+	ctx, err := json.Marshal(&Member{ID: types.ID(5), RaftAttributes: attr})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	attr = RaftAttributes{PeerURLs: []string{fmt.Sprintf("http://127.0.0.1:%d", 5)}}
+	ctx5, err := json.Marshal(&Member{ID: types.ID(5), RaftAttributes: attr})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	attr = RaftAttributes{PeerURLs: []string{fmt.Sprintf("http://127.0.0.1:%d", 3)}}
+	ctx2to3, err := json.Marshal(&Member{ID: types.ID(2), RaftAttributes: attr})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	attr = RaftAttributes{PeerURLs: []string{fmt.Sprintf("http://127.0.0.1:%d", 5)}}
+	ctx2to5, err := json.Marshal(&Member{ID: types.ID(2), RaftAttributes: attr})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	tests := []struct {
+		cc   raftpb.ConfChange
+		werr error
+	}{
+		{
+			raftpb.ConfChange{
+				Type:   raftpb.ConfChangeRemoveNode,
+				NodeID: 3,
+			},
+			nil,
+		},
+		{
+			raftpb.ConfChange{
+				Type:   raftpb.ConfChangeAddNode,
+				NodeID: 4,
+			},
+			ErrIDRemoved,
+		},
+		{
+			raftpb.ConfChange{
+				Type:   raftpb.ConfChangeRemoveNode,
+				NodeID: 4,
+			},
+			ErrIDRemoved,
+		},
+		{
+			raftpb.ConfChange{
+				Type:   raftpb.ConfChangeAddNode,
+				NodeID: 1,
+			},
+			ErrIDExists,
+		},
+		{
+			raftpb.ConfChange{
+				Type:    raftpb.ConfChangeAddNode,
+				NodeID:  5,
+				Context: ctx,
+			},
+			ErrPeerURLexists,
+		},
+		{
+			raftpb.ConfChange{
+				Type:   raftpb.ConfChangeRemoveNode,
+				NodeID: 5,
+			},
+			ErrIDNotFound,
+		},
+		{
+			raftpb.ConfChange{
+				Type:    raftpb.ConfChangeAddNode,
+				NodeID:  5,
+				Context: ctx5,
+			},
+			nil,
+		},
+		{
+			raftpb.ConfChange{
+				Type:    raftpb.ConfChangeUpdateNode,
+				NodeID:  5,
+				Context: ctx,
+			},
+			ErrIDNotFound,
+		},
+		// try to change the peer url of 2 to the peer url of 3
+		{
+			raftpb.ConfChange{
+				Type:    raftpb.ConfChangeUpdateNode,
+				NodeID:  2,
+				Context: ctx2to3,
+			},
+			ErrPeerURLexists,
+		},
+		{
+			raftpb.ConfChange{
+				Type:    raftpb.ConfChangeUpdateNode,
+				NodeID:  2,
+				Context: ctx2to5,
+			},
+			nil,
+		},
+	}
+	for i, tt := range tests {
+		err := cl.ValidateConfigurationChange(tt.cc)
+		if err != tt.werr {
+			t.Errorf("#%d: validateConfigurationChange error = %v, want %v", i, err, tt.werr)
 		}
 	}
 }
 
 func TestClusterGenID(t *testing.T) {
-	cs := newTestCluster([]Member{
+	cs := newTestCluster([]*Member{
 		newTestMember(1, nil, "", nil),
 		newTestMember(2, nil, "", nil),
 	})
@@ -366,7 +490,7 @@ func TestClusterGenID(t *testing.T) {
 	previd := cs.ID()
 
 	cs.SetStore(&storeRecorder{})
-	cs.AddMember(newTestMemberp(3, nil, "", nil))
+	cs.AddMember(newTestMember(3, nil, "", nil))
 	cs.genID()
 	if cs.ID() == previd {
 		t.Fatalf("cluster.ID = %v, want not %v", cs.ID(), previd)
@@ -379,22 +503,22 @@ func TestNodeToMemberBad(t *testing.T) {
 			{Key: "/1234/strange"},
 		}},
 		{Key: "/1234", Nodes: []*store.NodeExtern{
-			{Key: "/1234/dynamic", Value: stringp("garbage")},
+			{Key: "/1234/raftAttributes", Value: stringp("garbage")},
 		}},
 		{Key: "/1234", Nodes: []*store.NodeExtern{
-			{Key: "/1234/dynamic", Value: stringp(`{"peerURLs":null}`)},
+			{Key: "/1234/attributes", Value: stringp(`{"name":"node1","clientURLs":null}`)},
 		}},
 		{Key: "/1234", Nodes: []*store.NodeExtern{
-			{Key: "/1234/dynamic", Value: stringp(`{"peerURLs":null}`)},
+			{Key: "/1234/raftAttributes", Value: stringp(`{"peerURLs":null}`)},
 			{Key: "/1234/strange"},
 		}},
 		{Key: "/1234", Nodes: []*store.NodeExtern{
-			{Key: "/1234/dynamic", Value: stringp(`{"peerURLs":null}`)},
-			{Key: "/1234/static", Value: stringp("garbage")},
+			{Key: "/1234/raftAttributes", Value: stringp(`{"peerURLs":null}`)},
+			{Key: "/1234/attributes", Value: stringp("garbage")},
 		}},
 		{Key: "/1234", Nodes: []*store.NodeExtern{
-			{Key: "/1234/dynamic", Value: stringp(`{"peerURLs":null}`)},
-			{Key: "/1234/static", Value: stringp(`{"name":"node1","clientURLs":null}`)},
+			{Key: "/1234/raftAttributes", Value: stringp(`{"peerURLs":null}`)},
+			{Key: "/1234/attributes", Value: stringp(`{"name":"node1","clientURLs":null}`)},
 			{Key: "/1234/strange"},
 		}},
 	}
@@ -409,7 +533,7 @@ func TestClusterAddMember(t *testing.T) {
 	st := &storeRecorder{}
 	c := newTestCluster(nil)
 	c.SetStore(st)
-	c.AddMember(newTestMemberp(1, nil, "node1", nil))
+	c.AddMember(newTestMember(1, nil, "node1", nil))
 
 	wactions := []action{
 		{
@@ -422,16 +546,6 @@ func TestClusterAddMember(t *testing.T) {
 				store.Permanent,
 			},
 		},
-		{
-			name: "Create",
-			params: []interface{}{
-				path.Join(storeMembersPrefix, "1", "attributes"),
-				false,
-				`{"name":"node1"}`,
-				false,
-				store.Permanent,
-			},
-		},
 	}
 	if g := st.Action(); !reflect.DeepEqual(g, wactions) {
 		t.Errorf("actions = %v, want %v", g, wactions)
@@ -463,32 +577,32 @@ func TestClusterMembers(t *testing.T) {
 func TestClusterString(t *testing.T) {
 	cls := &Cluster{
 		members: map[types.ID]*Member{
-			1: newTestMemberp(
+			1: newTestMember(
 				1,
 				[]string{"http://1.1.1.1:1111", "http://0.0.0.0:0000"},
 				"abc",
 				nil,
 			),
-			2: newTestMemberp(
+			2: newTestMember(
 				2,
 				[]string{"http://2.2.2.2:2222"},
 				"def",
 				nil,
 			),
-			3: newTestMemberp(
+			3: newTestMember(
 				3,
 				[]string{"http://3.3.3.3:1234", "http://127.0.0.1:7001"},
 				"ghi",
 				nil,
 			),
 			// no PeerURLs = not included
-			4: newTestMemberp(
+			4: newTestMember(
 				4,
 				[]string{},
 				"four",
 				nil,
 			),
-			5: newTestMemberp(
+			5: newTestMember(
 				5,
 				nil,
 				"five",
@@ -533,23 +647,10 @@ func TestNodeToMember(t *testing.T) {
 	}
 }
 
-func newTestCluster(membs []Member) *Cluster {
+func newTestCluster(membs []*Member) *Cluster {
 	c := &Cluster{members: make(map[types.ID]*Member), removed: make(map[types.ID]bool)}
-	for i, m := range membs {
-		c.members[m.ID] = &membs[i]
+	for _, m := range membs {
+		c.members[m.ID] = m
 	}
 	return c
 }
-
-func newTestMember(id uint64, peerURLs []string, name string, clientURLs []string) Member {
-	return Member{
-		ID:             types.ID(id),
-		RaftAttributes: RaftAttributes{PeerURLs: peerURLs},
-		Attributes:     Attributes{Name: name, ClientURLs: clientURLs},
-	}
-}
-
-func newTestMemberp(id uint64, peerURLs []string, name string, clientURLs []string) *Member {
-	m := newTestMember(id, peerURLs, name, clientURLs)
-	return &m
-}

etcdserver/config.go

@@ -18,8 +18,11 @@ package etcdserver
 
 import (
 	"fmt"
+	"log"
 	"net/http"
 	"path"
+	"reflect"
+	"sort"
 
 	"github.com/coreos/etcd/pkg/types"
 	"github.com/coreos/etcd/raft"
@@ -27,14 +30,19 @@ import (
 
 // ServerConfig holds the configuration of etcd as taken from the command line or discovery.
 type ServerConfig struct {
-	Name         string
-	DiscoveryURL string
-	ClientURLs   types.URLs
-	DataDir      string
-	SnapCount    uint64
-	Cluster      *Cluster
-	ClusterState ClusterState
-	Transport    *http.Transport
+	Name            string
+	DiscoveryURL    string
+	DiscoveryProxy  string
+	ClientURLs      types.URLs
+	PeerURLs        types.URLs
+	DataDir         string
+	SnapCount       uint64
+	MaxSnapFiles    uint
+	MaxWALFiles     uint
+	Cluster         *Cluster
+	NewCluster      bool
+	ForceNewCluster bool
+	Transport       *http.Transport
 }
 
 // VerifyBootstrapConfig sanity-checks the initial config and returns an error
@@ -43,13 +51,13 @@ func (c *ServerConfig) VerifyBootstrapConfig() error {
 	m := c.Cluster.MemberByName(c.Name)
 	// Make sure the cluster at least contains the local server.
 	if m == nil {
-		return fmt.Errorf("couldn't find local name %s in the initial cluster configuration", c.Name)
+		return fmt.Errorf("couldn't find local name %q in the initial cluster configuration", c.Name)
 	}
 	if uint64(m.ID) == raft.None {
 		return fmt.Errorf("cannot use %x as member id", raft.None)
 	}
 
-	if c.DiscoveryURL == "" && c.ClusterState != ClusterStateValueNew {
+	if c.DiscoveryURL == "" && !c.NewCluster {
 		return fmt.Errorf("initial cluster state unset and no wal or discovery URL found")
 	}
 
@@ -63,6 +71,13 @@ func (c *ServerConfig) VerifyBootstrapConfig() error {
 			urlMap[url] = true
 		}
 	}
+
+	// Advertised peer URLs must match those in the cluster peer list
+	apurls := c.PeerURLs.StringSlice()
+	sort.Strings(apurls)
+	if !reflect.DeepEqual(apurls, m.PeerURLs) {
+		return fmt.Errorf("%s has different advertised URLs in the cluster and advertised peer URLs list", c.Name)
+	}
 	return nil
 }
 
@@ -70,6 +85,28 @@ func (c *ServerConfig) WALDir() string { return path.Join(c.DataDir, "wal") }
 
 func (c *ServerConfig) SnapDir() string { return path.Join(c.DataDir, "snap") }
 
-func (c *ServerConfig) ShouldDiscover() bool {
-	return c.DiscoveryURL != ""
+func (c *ServerConfig) ShouldDiscover() bool { return c.DiscoveryURL != "" }
+
+func (c *ServerConfig) PrintWithInitial() { c.print(true) }
+
+func (c *ServerConfig) Print() { c.print(false) }
+
+func (c *ServerConfig) print(initial bool) {
+	log.Printf("etcdserver: name = %s", c.Name)
+	if c.ForceNewCluster {
+		log.Println("etcdserver: force new cluster")
+	}
+	log.Printf("etcdserver: data dir = %s", c.DataDir)
+	log.Printf("etcdserver: snapshot count = %d", c.SnapCount)
+	if len(c.DiscoveryURL) != 0 {
+		log.Printf("etcdserver: discovery URL= %s", c.DiscoveryURL)
+		if len(c.DiscoveryProxy) != 0 {
+			log.Printf("etcdserver: discovery proxy = %s", c.DiscoveryProxy)
+		}
+	}
+	log.Printf("etcdserver: advertise client URLs = %s", c.ClientURLs)
+	if initial {
+		log.Printf("etcdserver: initial advertise peer URLs = %s", c.PeerURLs)
+		log.Printf("etcdserver: initial cluster = %s", c.Cluster)
+	}
 }

etcdserver/config_test.go

@@ -16,50 +16,90 @@
 
 package etcdserver
 
-import "testing"
+import (
+	"net/url"
+	"testing"
+
+	"github.com/coreos/etcd/pkg/types"
+)
+
+func mustNewURLs(t *testing.T, urls []string) []url.URL {
+	u, err := types.NewURLs(urls)
+	if err != nil {
+		t.Fatalf("error creating new URLs from %q: %v", urls, err)
+	}
+	return u
+}
 
 func TestBootstrapConfigVerify(t *testing.T) {
 	tests := []struct {
 		clusterSetting string
-		clst           ClusterState
+		newclst        bool
+		apurls         []string
 		disc           string
 		shouldError    bool
 	}{
 		{
 			// Node must exist in cluster
 			"",
-			ClusterStateValueNew,
+			true,
+			nil,
 			"",
+
 			true,
 		},
 		{
 			// Cannot have duplicate URLs in cluster config
 			"node1=http://localhost:7001,node2=http://localhost:7001,node2=http://localhost:7002",
-			ClusterStateValueNew,
+			true,
+			nil,
 			"",
+
 			true,
 		},
 		{
 			// Node defined, ClusterState OK
 			"node1=http://localhost:7001,node2=http://localhost:7002",
-			ClusterStateValueNew,
+			true,
+			[]string{"http://localhost:7001"},
 			"",
+
 			false,
 		},
 		{
 			// Node defined, discovery OK
 			"node1=http://localhost:7001",
-			// TODO(jonboulle): replace with ClusterStateExisting once it exists
-			"",
+			false,
+			[]string{"http://localhost:7001"},
 			"http://discovery",
+
 			false,
 		},
 		{
 			// Cannot have ClusterState!=new && !discovery
 			"node1=http://localhost:7001",
-			// TODO(jonboulle): replace with ClusterStateExisting once it exists
-			ClusterState("foo"),
+			false,
+			nil,
 			"",
+
+			true,
+		},
+		{
+			// Advertised peer URLs must match those in cluster-state
+			"node1=http://localhost:7001",
+			true,
+			[]string{"http://localhost:12345"},
+			"",
+
+			true,
+		},
+		{
+			// Advertised peer URLs must match those in cluster-state
+			"node1=http://localhost:7001,node1=http://localhost:12345",
+			true,
+			[]string{"http://localhost:12345"},
+			"",
+
 			true,
 		},
 	}
@@ -69,12 +109,14 @@ func TestBootstrapConfigVerify(t *testing.T) {
 		if err != nil {
 			t.Fatalf("#%d: Got unexpected error: %v", i, err)
 		}
-
 		cfg := ServerConfig{
 			Name:         "node1",
 			DiscoveryURL: tt.disc,
 			Cluster:      cluster,
-			ClusterState: tt.clst,
+			NewCluster:   tt.newclst,
+		}
+		if tt.apurls != nil {
+			cfg.PeerURLs = mustNewURLs(t, tt.apurls)
 		}
 		err = cfg.VerifyBootstrapConfig()
 		if (err == nil) && tt.shouldError {

etcdserver/etcdhttp/client.go

@@ -29,12 +29,13 @@ import (
 	"strings"
 	"time"
 
-	"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
 	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/jonboulle/clockwork"
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 	etcdErr "github.com/coreos/etcd/error"
 	"github.com/coreos/etcd/etcdserver"
 	"github.com/coreos/etcd/etcdserver/etcdhttp/httptypes"
 	"github.com/coreos/etcd/etcdserver/etcdserverpb"
+	"github.com/coreos/etcd/etcdserver/stats"
 	"github.com/coreos/etcd/pkg/types"
 	"github.com/coreos/etcd/store"
 	"github.com/coreos/etcd/version"
@@ -93,7 +94,7 @@ type keysHandler struct {
 }
 
 func (h *keysHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
-	if !allowMethod(w, r.Method, "GET", "PUT", "POST", "DELETE") {
+	if !allowMethod(w, r.Method, "HEAD", "GET", "PUT", "POST", "DELETE") {
 		return
 	}
 	w.Header().Set("X-Etcd-Cluster-ID", h.clusterInfo.ID().String())
@@ -148,7 +149,7 @@ type membersHandler struct {
 }
 
 func (h *membersHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
-	if !allowMethod(w, r.Method, "GET", "POST", "DELETE") {
+	if !allowMethod(w, r.Method, "GET", "POST", "DELETE", "PUT") {
 		return
 	}
 	w.Header().Set("X-Etcd-Cluster-ID", h.clusterInfo.ID().String())
@@ -168,30 +169,22 @@ func (h *membersHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 			log.Printf("etcdhttp: %v", err)
 		}
 	case "POST":
-		ctype := r.Header.Get("Content-Type")
-		if ctype != "application/json" {
-			writeError(w, httptypes.NewHTTPError(http.StatusUnsupportedMediaType, fmt.Sprintf("Bad Content-Type %s, accept application/json", ctype)))
-			return
-		}
-		b, err := ioutil.ReadAll(r.Body)
-		if err != nil {
-			writeError(w, httptypes.NewHTTPError(http.StatusBadRequest, err.Error()))
-			return
-		}
 		req := httptypes.MemberCreateRequest{}
-		if err := json.Unmarshal(b, &req); err != nil {
-			writeError(w, httptypes.NewHTTPError(http.StatusBadRequest, err.Error()))
+		if ok := unmarshalRequest(r, &req, w); !ok {
 			return
 		}
-
 		now := h.clock.Now()
 		m := etcdserver.NewMember("", req.PeerURLs, "", &now)
-		if err := h.server.AddMember(ctx, *m); err != nil {
+		err := h.server.AddMember(ctx, *m)
+		switch {
+		case err == etcdserver.ErrIDExists || err == etcdserver.ErrPeerURLexists:
+			writeError(w, httptypes.NewHTTPError(http.StatusConflict, err.Error()))
+			return
+		case err != nil:
 			log.Printf("etcdhttp: error adding node %s: %v", m.ID, err)
 			writeError(w, err)
 			return
 		}
-
 		res := newMember(m)
 		w.Header().Set("Content-Type", "application/json")
 		w.WriteHeader(http.StatusCreated)
@@ -199,31 +192,52 @@ func (h *membersHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 			log.Printf("etcdhttp: %v", err)
 		}
 	case "DELETE":
-		idStr := trimPrefix(r.URL.Path, membersPrefix)
-		if idStr == "" {
-			http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
+		id, ok := getID(r.URL.Path, w)
+		if !ok {
 			return
 		}
-		id, err := types.IDFromString(idStr)
-		if err != nil {
-			writeError(w, httptypes.NewHTTPError(http.StatusBadRequest, err.Error()))
-			return
-		}
-		err = h.server.RemoveMember(ctx, uint64(id))
+		err := h.server.RemoveMember(ctx, uint64(id))
 		switch {
+		case err == etcdserver.ErrIDRemoved:
+			writeError(w, httptypes.NewHTTPError(http.StatusGone, fmt.Sprintf("Member permanently removed: %s", id)))
 		case err == etcdserver.ErrIDNotFound:
-			writeError(w, httptypes.NewHTTPError(http.StatusNotFound, fmt.Sprintf("No such member: %s", idStr)))
+			writeError(w, httptypes.NewHTTPError(http.StatusNotFound, fmt.Sprintf("No such member: %s", id)))
 		case err != nil:
 			log.Printf("etcdhttp: error removing node %s: %v", id, err)
 			writeError(w, err)
 		default:
 			w.WriteHeader(http.StatusNoContent)
 		}
+	case "PUT":
+		id, ok := getID(r.URL.Path, w)
+		if !ok {
+			return
+		}
+		req := httptypes.MemberUpdateRequest{}
+		if ok := unmarshalRequest(r, &req, w); !ok {
+			return
+		}
+		m := etcdserver.Member{
+			ID:             id,
+			RaftAttributes: etcdserver.RaftAttributes{PeerURLs: req.PeerURLs.StringSlice()},
+		}
+		err := h.server.UpdateMember(ctx, m)
+		switch {
+		case err == etcdserver.ErrPeerURLexists:
+			writeError(w, httptypes.NewHTTPError(http.StatusConflict, err.Error()))
+		case err == etcdserver.ErrIDNotFound:
+			writeError(w, httptypes.NewHTTPError(http.StatusNotFound, fmt.Sprintf("No such member: %s", id)))
+		case err != nil:
+			log.Printf("etcdhttp: error updating node %s: %v", m.ID, err)
+			writeError(w, err)
+		default:
+			w.WriteHeader(http.StatusNoContent)
+		}
 	}
 }
 
 type statsHandler struct {
-	stats etcdserver.Stats
+	stats stats.Stats
 }
 
 func (h *statsHandler) serveStore(w http.ResponseWriter, r *http.Request) {
@@ -246,8 +260,13 @@ func (h *statsHandler) serveLeader(w http.ResponseWriter, r *http.Request) {
 	if !allowMethod(w, r.Method, "GET") {
 		return
 	}
+	stats := h.stats.LeaderStats()
+	if stats == nil {
+		writeError(w, httptypes.NewHTTPError(http.StatusForbidden, "not current leader"))
+		return
+	}
 	w.Header().Set("Content-Type", "application/json")
-	w.Write(h.stats.LeaderStats())
+	w.Write(stats)
 }
 
 func serveVersion(w http.ResponseWriter, r *http.Request) {
@@ -342,7 +361,7 @@ func parseKeyRequest(r *http.Request, id uint64, clock clockwork.Clock) (etcdser
 	pV := r.FormValue("prevValue")
 	if _, ok := r.Form["prevValue"]; ok && pV == "" {
 		return emptyReq, etcdErr.NewRequestError(
-			etcdErr.EcodeInvalidField,
+			etcdErr.EcodePrevValueRequired,
 			`"prevValue" cannot be empty`,
 		)
 	}
@@ -500,6 +519,38 @@ func trimErrorPrefix(err error, prefix string) error {
 	return err
 }
 
+func unmarshalRequest(r *http.Request, req json.Unmarshaler, w http.ResponseWriter) bool {
+	ctype := r.Header.Get("Content-Type")
+	if ctype != "application/json" {
+		writeError(w, httptypes.NewHTTPError(http.StatusUnsupportedMediaType, fmt.Sprintf("Bad Content-Type %s, accept application/json", ctype)))
+		return false
+	}
+	b, err := ioutil.ReadAll(r.Body)
+	if err != nil {
+		writeError(w, httptypes.NewHTTPError(http.StatusBadRequest, err.Error()))
+		return false
+	}
+	if err := req.UnmarshalJSON(b); err != nil {
+		writeError(w, httptypes.NewHTTPError(http.StatusBadRequest, err.Error()))
+		return false
+	}
+	return true
+}
+
+func getID(p string, w http.ResponseWriter) (types.ID, bool) {
+	idStr := trimPrefix(p, membersPrefix)
+	if idStr == "" {
+		http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
+		return 0, false
+	}
+	id, err := types.IDFromString(idStr)
+	if err != nil {
+		writeError(w, httptypes.NewHTTPError(http.StatusNotFound, fmt.Sprintf("No such member: %s", idStr)))
+		return 0, false
+	}
+	return id, true
+}
+
 // getUint64 extracts a uint64 by the given key from a Form. If the key does
 // not exist in the form, 0 is returned. If the key exists but the value is
 // badly formed, an error is returned. If multiple values are present only the

etcdserver/etcdhttp/client_test.go

@@ -31,8 +31,8 @@ import (
 	"testing"
 	"time"
 
-	"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
 	"github.com/coreos/etcd/Godeps/_workspace/src/github.com/jonboulle/clockwork"
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 	etcdErr "github.com/coreos/etcd/error"
 	"github.com/coreos/etcd/etcdserver"
 	"github.com/coreos/etcd/etcdserver/etcdhttp/httptypes"
@@ -63,6 +63,18 @@ func mustNewForm(t *testing.T, p string, vals url.Values) *http.Request {
 	return req
 }
 
+// mustNewPostForm takes a set of Values and constructs a POST *http.Request,
+// with a URL constructed from appending the given path to the standard keysPrefix
+func mustNewPostForm(t *testing.T, p string, vals url.Values) *http.Request {
+	u := mustNewURL(t, path.Join(keysPrefix, p))
+	req, err := http.NewRequest("POST", u.String(), strings.NewReader(vals.Encode()))
+	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	if err != nil {
+		t.Fatalf("error creating new request: %v", err)
+	}
+	return req
+}
+
 // mustNewRequest takes a path, appends it to the standard keysPrefix, and constructs
 // a GET *http.Request referencing the resulting URL
 func mustNewRequest(t *testing.T, p string) *http.Request {
@@ -80,6 +92,9 @@ type serverRecorder struct {
 	actions []action
 }
 
+func (s *serverRecorder) Start()       {}
+func (s *serverRecorder) Stop()        {}
+func (s *serverRecorder) ID() types.ID { return types.ID(1) }
 func (s *serverRecorder) Do(_ context.Context, r etcdserverpb.Request) (etcdserver.Response, error) {
 	s.actions = append(s.actions, action{name: "Do", params: []interface{}{r}})
 	return etcdserver.Response{}, nil
@@ -88,8 +103,6 @@ func (s *serverRecorder) Process(_ context.Context, m raftpb.Message) error {
 	s.actions = append(s.actions, action{name: "Process", params: []interface{}{m}})
 	return nil
 }
-func (s *serverRecorder) Start() {}
-func (s *serverRecorder) Stop()  {}
 func (s *serverRecorder) AddMember(_ context.Context, m etcdserver.Member) error {
 	s.actions = append(s.actions, action{name: "AddMember", params: []interface{}{m}})
 	return nil
@@ -99,6 +112,11 @@ func (s *serverRecorder) RemoveMember(_ context.Context, id uint64) error {
 	return nil
 }
 
+func (s *serverRecorder) UpdateMember(_ context.Context, m etcdserver.Member) error {
+	s.actions = append(s.actions, action{name: "UpdateMember", params: []interface{}{m}})
+	return nil
+}
+
 type action struct {
 	name   string
 	params []interface{}
@@ -121,14 +139,16 @@ type resServer struct {
 	res etcdserver.Response
 }
 
+func (rs *resServer) Start()       {}
+func (rs *resServer) Stop()        {}
+func (rs *resServer) ID() types.ID { return types.ID(1) }
 func (rs *resServer) Do(_ context.Context, _ etcdserverpb.Request) (etcdserver.Response, error) {
 	return rs.res, nil
 }
-func (rs *resServer) Process(_ context.Context, _ raftpb.Message) error      { return nil }
-func (rs *resServer) Start()                                                 {}
-func (rs *resServer) Stop()                                                  {}
-func (rs *resServer) AddMember(_ context.Context, _ etcdserver.Member) error { return nil }
-func (rs *resServer) RemoveMember(_ context.Context, _ uint64) error         { return nil }
+func (rs *resServer) Process(_ context.Context, _ raftpb.Message) error         { return nil }
+func (rs *resServer) AddMember(_ context.Context, _ etcdserver.Member) error    { return nil }
+func (rs *resServer) RemoveMember(_ context.Context, _ uint64) error            { return nil }
+func (rs *resServer) UpdateMember(_ context.Context, _ etcdserver.Member) error { return nil }
 
 func boolp(b bool) *bool { return &b }
 
@@ -257,7 +277,7 @@ func TestBadParseRequest(t *testing.T) {
 		// prevValue cannot be empty
 		{
 			mustNewForm(t, "foo", url.Values{"prevValue": []string{""}}),
-			etcdErr.EcodeInvalidField,
+			etcdErr.EcodePrevValueRequired,
 		},
 		// wait is only valid with GET requests
 		{
@@ -686,6 +706,48 @@ func TestServeMembersDelete(t *testing.T) {
 	}
 }
 
+func TestServeMembersUpdate(t *testing.T) {
+	u := mustNewURL(t, path.Join(membersPrefix, "1"))
+	b := []byte(`{"peerURLs":["http://127.0.0.1:1"]}`)
+	req, err := http.NewRequest("PUT", u.String(), bytes.NewReader(b))
+	if err != nil {
+		t.Fatal(err)
+	}
+	req.Header.Set("Content-Type", "application/json")
+	s := &serverRecorder{}
+	h := &membersHandler{
+		server:      s,
+		clock:       clockwork.NewFakeClock(),
+		clusterInfo: &fakeCluster{id: 1},
+	}
+	rw := httptest.NewRecorder()
+
+	h.ServeHTTP(rw, req)
+
+	wcode := http.StatusNoContent
+	if rw.Code != wcode {
+		t.Errorf("code=%d, want %d", rw.Code, wcode)
+	}
+
+	gcid := rw.Header().Get("X-Etcd-Cluster-ID")
+	wcid := h.clusterInfo.ID().String()
+	if gcid != wcid {
+		t.Errorf("cid = %s, want %s", gcid, wcid)
+	}
+
+	wm := etcdserver.Member{
+		ID: 1,
+		RaftAttributes: etcdserver.RaftAttributes{
+			PeerURLs: []string{"http://127.0.0.1:1"},
+		},
+	}
+
+	wactions := []action{{name: "UpdateMember", params: []interface{}{wm}}}
+	if !reflect.DeepEqual(s.actions, wactions) {
+		t.Errorf("actions = %+v, want %+v", s.actions, wactions)
+	}
+}
+
 func TestServeMembersFail(t *testing.T) {
 	tests := []struct {
 		req    *http.Request
@@ -756,11 +818,39 @@ func TestServeMembersFail(t *testing.T) {
 				Header: map[string][]string{"Content-Type": []string{"application/json"}},
 			},
 			&errServer{
-				errors.New("blah"),
+				errors.New("Error while adding a member"),
 			},
 
 			http.StatusInternalServerError,
 		},
+		{
+			// etcdserver.AddMember error
+			&http.Request{
+				URL:    mustNewURL(t, membersPrefix),
+				Method: "POST",
+				Body:   ioutil.NopCloser(strings.NewReader(`{"PeerURLs": ["http://127.0.0.1:1"]}`)),
+				Header: map[string][]string{"Content-Type": []string{"application/json"}},
+			},
+			&errServer{
+				etcdserver.ErrIDExists,
+			},
+
+			http.StatusConflict,
+		},
+		{
+			// etcdserver.AddMember error
+			&http.Request{
+				URL:    mustNewURL(t, membersPrefix),
+				Method: "POST",
+				Body:   ioutil.NopCloser(strings.NewReader(`{"PeerURLs": ["http://127.0.0.1:1"]}`)),
+				Header: map[string][]string{"Content-Type": []string{"application/json"}},
+			},
+			&errServer{
+				etcdserver.ErrPeerURLexists,
+			},
+
+			http.StatusConflict,
+		},
 		{
 			// etcdserver.RemoveMember error with arbitrary server error
 			&http.Request{
@@ -768,11 +858,23 @@ func TestServeMembersFail(t *testing.T) {
 				Method: "DELETE",
 			},
 			&errServer{
-				errors.New("blah"),
+				errors.New("Error while removing member"),
 			},
 
 			http.StatusInternalServerError,
 		},
+		{
+			// etcdserver.RemoveMember error with previously removed ID
+			&http.Request{
+				URL:    mustNewURL(t, path.Join(membersPrefix, "0")),
+				Method: "DELETE",
+			},
+			&errServer{
+				etcdserver.ErrIDRemoved,
+			},
+
+			http.StatusGone,
+		},
 		{
 			// etcdserver.RemoveMember error with nonexistent ID
 			&http.Request{
@@ -793,7 +895,7 @@ func TestServeMembersFail(t *testing.T) {
 			},
 			nil,
 
-			http.StatusBadRequest,
+			http.StatusNotFound,
 		},
 		{
 			// etcdserver.RemoveMember with no ID
@@ -803,6 +905,104 @@ func TestServeMembersFail(t *testing.T) {
 			},
 			nil,
 
+			http.StatusMethodNotAllowed,
+		},
+		{
+			// parse body error
+			&http.Request{
+				URL:    mustNewURL(t, path.Join(membersPrefix, "0")),
+				Method: "PUT",
+				Body:   ioutil.NopCloser(strings.NewReader("bad json")),
+				Header: map[string][]string{"Content-Type": []string{"application/json"}},
+			},
+			&resServer{},
+
+			http.StatusBadRequest,
+		},
+		{
+			// bad content type
+			&http.Request{
+				URL:    mustNewURL(t, path.Join(membersPrefix, "0")),
+				Method: "PUT",
+				Body:   ioutil.NopCloser(strings.NewReader(`{"PeerURLs": ["http://127.0.0.1:1"]}`)),
+				Header: map[string][]string{"Content-Type": []string{"application/bad"}},
+			},
+			&errServer{},
+
+			http.StatusUnsupportedMediaType,
+		},
+		{
+			// bad url
+			&http.Request{
+				URL:    mustNewURL(t, path.Join(membersPrefix, "0")),
+				Method: "PUT",
+				Body:   ioutil.NopCloser(strings.NewReader(`{"PeerURLs": ["http://a"]}`)),
+				Header: map[string][]string{"Content-Type": []string{"application/json"}},
+			},
+			&errServer{},
+
+			http.StatusBadRequest,
+		},
+		{
+			// etcdserver.UpdateMember error
+			&http.Request{
+				URL:    mustNewURL(t, path.Join(membersPrefix, "0")),
+				Method: "PUT",
+				Body:   ioutil.NopCloser(strings.NewReader(`{"PeerURLs": ["http://127.0.0.1:1"]}`)),
+				Header: map[string][]string{"Content-Type": []string{"application/json"}},
+			},
+			&errServer{
+				errors.New("blah"),
+			},
+
+			http.StatusInternalServerError,
+		},
+		{
+			// etcdserver.UpdateMember error
+			&http.Request{
+				URL:    mustNewURL(t, path.Join(membersPrefix, "0")),
+				Method: "PUT",
+				Body:   ioutil.NopCloser(strings.NewReader(`{"PeerURLs": ["http://127.0.0.1:1"]}`)),
+				Header: map[string][]string{"Content-Type": []string{"application/json"}},
+			},
+			&errServer{
+				etcdserver.ErrPeerURLexists,
+			},
+
+			http.StatusConflict,
+		},
+		{
+			// etcdserver.UpdateMember error
+			&http.Request{
+				URL:    mustNewURL(t, path.Join(membersPrefix, "0")),
+				Method: "PUT",
+				Body:   ioutil.NopCloser(strings.NewReader(`{"PeerURLs": ["http://127.0.0.1:1"]}`)),
+				Header: map[string][]string{"Content-Type": []string{"application/json"}},
+			},
+			&errServer{
+				etcdserver.ErrIDNotFound,
+			},
+
+			http.StatusNotFound,
+		},
+		{
+			// etcdserver.UpdateMember error with badly formed ID
+			&http.Request{
+				URL:    mustNewURL(t, path.Join(membersPrefix, "bad_id")),
+				Method: "PUT",
+			},
+			nil,
+
+			http.StatusNotFound,
+		},
+		{
+			// etcdserver.UpdateMember with no ID
+			&http.Request{
+				URL:    mustNewURL(t, membersPrefix),
+				Method: "PUT",
+			},
+			nil,
+
 			http.StatusMethodNotAllowed,
 		},
 	}
@@ -943,6 +1143,43 @@ func TestServeMachines(t *testing.T) {
 	}
 }
 
+func TestGetID(t *testing.T) {
+	tests := []struct {
+		path string
+
+		wok   bool
+		wid   types.ID
+		wcode int
+	}{
+		{
+			"123",
+			true, 0x123, http.StatusOK,
+		},
+		{
+			"bad_id",
+			false, 0, http.StatusNotFound,
+		},
+		{
+			"",
+			false, 0, http.StatusMethodNotAllowed,
+		},
+	}
+
+	for i, tt := range tests {
+		w := httptest.NewRecorder()
+		id, ok := getID(tt.path, w)
+		if id != tt.wid {
+			t.Errorf("#%d: id = %d, want %d", i, id, tt.wid)
+		}
+		if ok != tt.wok {
+			t.Errorf("#%d: ok = %t, want %t", i, ok, tt.wok)
+		}
+		if w.Code != tt.wcode {
+			t.Errorf("#%d code = %d, want %d", i, w.Code, tt.wcode)
+		}
+	}
+}
+
 type dummyStats struct {
 	data []byte
 }
@@ -1120,11 +1357,11 @@ func TestBadServeKeys(t *testing.T) {
 			// etcdserver.Server error
 			mustNewRequest(t, "foo"),
 			&errServer{
-				errors.New("blah"),
+				errors.New("Internal Server Error"),
 			},
 
 			http.StatusInternalServerError,
-			"Internal Server Error",
+			`{"message":"Internal Server Error"}`,
 		},
 		{
 			// etcdserver.Server etcd error
@@ -1144,7 +1381,7 @@ func TestBadServeKeys(t *testing.T) {
 			},
 
 			http.StatusInternalServerError,
-			"Internal Server Error",
+			`{"message":"Internal Server Error"}`,
 		},
 	}
 	for i, tt := range testBadCases {
@@ -1171,6 +1408,55 @@ func TestBadServeKeys(t *testing.T) {
 	}
 }
 
+func TestServeKeysGood(t *testing.T) {
+	tests := []struct {
+		req   *http.Request
+		wcode int
+	}{
+		{
+			mustNewMethodRequest(t, "HEAD", "foo"),
+			http.StatusOK,
+		},
+		{
+			mustNewMethodRequest(t, "GET", "foo"),
+			http.StatusOK,
+		},
+		{
+			mustNewForm(t, "foo", url.Values{"value": []string{"bar"}}),
+			http.StatusOK,
+		},
+		{
+			mustNewMethodRequest(t, "DELETE", "foo"),
+			http.StatusOK,
+		},
+		{
+			mustNewPostForm(t, "foo", url.Values{"value": []string{"bar"}}),
+			http.StatusOK,
+		},
+	}
+	server := &resServer{
+		etcdserver.Response{
+			Event: &store.Event{
+				Action: store.Get,
+				Node:   &store.NodeExtern{},
+			},
+		},
+	}
+	for i, tt := range tests {
+		h := &keysHandler{
+			timeout:     time.Hour,
+			server:      server,
+			timer:       &dummyRaftTimer{},
+			clusterInfo: &fakeCluster{id: 1},
+		}
+		rw := httptest.NewRecorder()
+		h.ServeHTTP(rw, tt.req)
+		if rw.Code != tt.wcode {
+			t.Errorf("#%d: got code=%d, want %d", i, rw.Code, tt.wcode)
+		}
+	}
+}
+
 func TestServeKeysEvent(t *testing.T) {
 	req := mustNewRequest(t, "foo")
 	server := &resServer{

etcdserver/etcdhttp/http.go

@@ -51,7 +51,8 @@ func writeError(w http.ResponseWriter, err error) {
 		e.WriteTo(w)
 	default:
 		log.Printf("etcdhttp: unexpected error: %v", err)
-		http.Error(w, "Internal Server Error", http.StatusInternalServerError)
+		herr := httptypes.NewHTTPError(http.StatusInternalServerError, "Internal Server Error")
+		herr.WriteTo(w)
 	}
 }
 

etcdserver/etcdhttp/http_test.go

@@ -24,7 +24,7 @@ import (
 	"sort"
 	"testing"
 
-	"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 	etcdErr "github.com/coreos/etcd/error"
 	"github.com/coreos/etcd/etcdserver"
 	"github.com/coreos/etcd/etcdserver/etcdserverpb"
@@ -57,6 +57,7 @@ func (c *fakeCluster) Members() []*etcdserver.Member {
 	return []*etcdserver.Member(sms)
 }
 func (c *fakeCluster) Member(id types.ID) *etcdserver.Member { return c.members[uint64(id)] }
+func (c *fakeCluster) IsIDRemoved(id types.ID) bool          { return false }
 
 // errServer implements the etcd.Server interface for testing.
 // It returns the given error from any Do/Process/AddMember/RemoveMember calls.
@@ -64,20 +65,24 @@ type errServer struct {
 	err error
 }
 
+func (fs *errServer) Start()       {}
+func (fs *errServer) Stop()        {}
+func (fs *errServer) ID() types.ID { return types.ID(1) }
 func (fs *errServer) Do(ctx context.Context, r etcdserverpb.Request) (etcdserver.Response, error) {
 	return etcdserver.Response{}, fs.err
 }
 func (fs *errServer) Process(ctx context.Context, m raftpb.Message) error {
 	return fs.err
 }
-func (fs *errServer) Start() {}
-func (fs *errServer) Stop()  {}
 func (fs *errServer) AddMember(ctx context.Context, m etcdserver.Member) error {
 	return fs.err
 }
 func (fs *errServer) RemoveMember(ctx context.Context, id uint64) error {
 	return fs.err
 }
+func (fs *errServer) UpdateMember(ctx context.Context, m etcdserver.Member) error {
+	return fs.err
+}
 
 func TestWriteError(t *testing.T) {
 	// nil error should not panic

etcdserver/etcdhttp/httptypes/errors.go

@@ -18,6 +18,7 @@ package httptypes
 
 import (
 	"encoding/json"
+	"log"
 	"net/http"
 )
 
@@ -37,7 +38,7 @@ func (e HTTPError) WriteTo(w http.ResponseWriter) {
 	w.WriteHeader(e.Code)
 	b, err := json.Marshal(e)
 	if err != nil {
-		panic("unexpected json marshal error")
+		log.Panicf("marshal HTTPError should never fail: %v", err)
 	}
 	w.Write(b)
 }

etcdserver/etcdhttp/httptypes/member.go

@@ -33,6 +33,10 @@ type MemberCreateRequest struct {
 	PeerURLs types.URLs
 }
 
+type MemberUpdateRequest struct {
+	MemberCreateRequest
+}
+
 func (m *MemberCreateRequest) MarshalJSON() ([]byte, error) {
 	s := struct {
 		PeerURLs []string `json:"peerURLs"`

etcdserver/etcdhttp/peer.go

@@ -18,89 +18,33 @@ package etcdhttp
 
 import (
 	"encoding/json"
-	"io/ioutil"
 	"log"
 	"net/http"
 
-	"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
 	"github.com/coreos/etcd/etcdserver"
-	"github.com/coreos/etcd/pkg/types"
-	"github.com/coreos/etcd/raft/raftpb"
+	"github.com/coreos/etcd/rafthttp"
 )
 
 const (
-	raftPrefix        = "/raft"
 	peerMembersPrefix = "/members"
 )
 
 // NewPeerHandler generates an http.Handler to handle etcd peer (raft) requests.
 func NewPeerHandler(server *etcdserver.EtcdServer) http.Handler {
-	rh := &raftHandler{
-		stats:       server,
-		server:      server,
-		clusterInfo: server.Cluster,
-	}
-
+	rh := rafthttp.NewHandler(server, server.Cluster.ID())
+	rsh := rafthttp.NewStreamHandler(server.SenderFinder(), server.ID(), server.Cluster.ID())
 	mh := &peerMembersHandler{
 		clusterInfo: server.Cluster,
 	}
 
 	mux := http.NewServeMux()
 	mux.HandleFunc("/", http.NotFound)
-	mux.Handle(raftPrefix, rh)
+	mux.Handle(rafthttp.RaftPrefix, rh)
+	mux.Handle(rafthttp.RaftStreamPrefix+"/", rsh)
 	mux.Handle(peerMembersPrefix, mh)
 	return mux
 }
 
-type raftHandler struct {
-	stats       etcdserver.Stats
-	server      etcdserver.Server
-	clusterInfo etcdserver.ClusterInfo
-}
-
-func (h *raftHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
-	if !allowMethod(w, r.Method, "POST") {
-		return
-	}
-
-	wcid := h.clusterInfo.ID().String()
-	w.Header().Set("X-Etcd-Cluster-ID", wcid)
-
-	gcid := r.Header.Get("X-Etcd-Cluster-ID")
-	if gcid != wcid {
-		log.Printf("etcdhttp: request ignored due to cluster ID mismatch got %s want %s", gcid, wcid)
-		http.Error(w, "clusterID mismatch", http.StatusPreconditionFailed)
-		return
-	}
-
-	b, err := ioutil.ReadAll(r.Body)
-	if err != nil {
-		log.Println("etcdhttp: error reading raft message:", err)
-		http.Error(w, "error reading raft message", http.StatusBadRequest)
-		return
-	}
-	var m raftpb.Message
-	if err := m.Unmarshal(b); err != nil {
-		log.Println("etcdhttp: error unmarshaling raft message:", err)
-		http.Error(w, "error unmarshaling raft message", http.StatusBadRequest)
-		return
-	}
-	if err := h.server.Process(context.TODO(), m); err != nil {
-		switch err {
-		case etcdserver.ErrRemoved:
-			log.Printf("etcdhttp: reject message from removed member %s", types.ID(m.From).String())
-			http.Error(w, "cannot process message from removed member", http.StatusForbidden)
-		default:
-			writeError(w, err)
-		}
-		return
-	}
-	if m.Type == raftpb.MsgApp {
-		h.stats.UpdateRecvApp(types.ID(m.From), r.ContentLength)
-	}
-	w.WriteHeader(http.StatusNoContent)
-}
-
 type peerMembersHandler struct {
 	clusterInfo etcdserver.ClusterInfo
 }

etcdserver/etcdhttp/peer_test.go

@@ -17,165 +17,15 @@
 package etcdhttp
 
 import (
-	"bytes"
 	"encoding/json"
-	"errors"
-	"io"
 	"net/http"
 	"net/http/httptest"
 	"path"
-	"strings"
 	"testing"
 
 	"github.com/coreos/etcd/etcdserver"
-	"github.com/coreos/etcd/raft/raftpb"
 )
 
-func mustMarshalMsg(t *testing.T, m raftpb.Message) []byte {
-	json, err := m.Marshal()
-	if err != nil {
-		t.Fatalf("error marshalling raft Message: %#v", err)
-	}
-	return json
-}
-
-// errReader implements io.Reader to facilitate a broken request.
-type errReader struct{}
-
-func (er *errReader) Read(_ []byte) (int, error) { return 0, errors.New("some error") }
-
-func TestServeRaft(t *testing.T) {
-	testCases := []struct {
-		method    string
-		body      io.Reader
-		serverErr error
-		clusterID string
-
-		wcode int
-	}{
-		{
-			// bad method
-			"GET",
-			bytes.NewReader(
-				mustMarshalMsg(
-					t,
-					raftpb.Message{},
-				),
-			),
-			nil,
-			"0",
-			http.StatusMethodNotAllowed,
-		},
-		{
-			// bad method
-			"PUT",
-			bytes.NewReader(
-				mustMarshalMsg(
-					t,
-					raftpb.Message{},
-				),
-			),
-			nil,
-			"0",
-			http.StatusMethodNotAllowed,
-		},
-		{
-			// bad method
-			"DELETE",
-			bytes.NewReader(
-				mustMarshalMsg(
-					t,
-					raftpb.Message{},
-				),
-			),
-			nil,
-			"0",
-			http.StatusMethodNotAllowed,
-		},
-		{
-			// bad request body
-			"POST",
-			&errReader{},
-			nil,
-			"0",
-			http.StatusBadRequest,
-		},
-		{
-			// bad request protobuf
-			"POST",
-			strings.NewReader("malformed garbage"),
-			nil,
-			"0",
-			http.StatusBadRequest,
-		},
-		{
-			// good request, etcdserver.Server internal error
-			"POST",
-			bytes.NewReader(
-				mustMarshalMsg(
-					t,
-					raftpb.Message{},
-				),
-			),
-			errors.New("some error"),
-			"0",
-			http.StatusInternalServerError,
-		},
-		{
-			// good request from removed member
-			"POST",
-			bytes.NewReader(
-				mustMarshalMsg(
-					t,
-					raftpb.Message{},
-				),
-			),
-			etcdserver.ErrRemoved,
-			"0",
-			http.StatusForbidden,
-		},
-		{
-			// good request
-			"POST",
-			bytes.NewReader(
-				mustMarshalMsg(
-					t,
-					raftpb.Message{},
-				),
-			),
-			nil,
-			"1",
-			http.StatusPreconditionFailed,
-		},
-		{
-			// good request
-			"POST",
-			bytes.NewReader(
-				mustMarshalMsg(
-					t,
-					raftpb.Message{},
-				),
-			),
-			nil,
-			"0",
-			http.StatusNoContent,
-		},
-	}
-	for i, tt := range testCases {
-		req, err := http.NewRequest(tt.method, "foo", tt.body)
-		if err != nil {
-			t.Fatalf("#%d: could not create request: %#v", i, err)
-		}
-		req.Header.Set("X-Etcd-Cluster-ID", tt.clusterID)
-		rw := httptest.NewRecorder()
-		h := &raftHandler{stats: nil, server: &errServer{tt.serverErr}, clusterInfo: &fakeCluster{id: 0}}
-		h.ServeHTTP(rw, req)
-		if rw.Code != tt.wcode {
-			t.Errorf("#%d: got code=%d, want %d", i, rw.Code, tt.wcode)
-		}
-	}
-}
-
 func TestServeMembersFails(t *testing.T) {
 	tests := []struct {
 		method string

etcdserver/force_cluster.go

@@ -0,0 +1,155 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
+package etcdserver
+
+import (
+	"encoding/json"
+	"log"
+	"sort"
+
+	"github.com/coreos/etcd/pkg/pbutil"
+	"github.com/coreos/etcd/pkg/types"
+	"github.com/coreos/etcd/raft"
+	"github.com/coreos/etcd/raft/raftpb"
+	"github.com/coreos/etcd/wal"
+)
+
+func restartAsStandaloneNode(cfg *ServerConfig, index uint64, snapshot *raftpb.Snapshot) (types.ID, raft.Node, *raft.MemoryStorage, *wal.WAL) {
+	w, id, cid, st, ents := readWAL(cfg.WALDir(), index)
+	cfg.Cluster.SetID(cid)
+
+	// discard the previously uncommitted entries
+	for i, ent := range ents {
+		if ent.Index > st.Commit {
+			log.Printf("etcdserver: discarding %d uncommitted WAL entries ", len(ents)-i)
+			ents = ents[:i]
+			break
+		}
+	}
+
+	// force append the configuration change entries
+	toAppEnts := createConfigChangeEnts(getIDs(snapshot, ents), uint64(id), st.Term, st.Commit)
+	ents = append(ents, toAppEnts...)
+
+	// force commit newly appended entries
+	for _, e := range toAppEnts {
+		err := w.SaveEntry(&e)
+		if err != nil {
+			log.Fatalf("etcdserver: %v", err)
+		}
+	}
+	if len(ents) != 0 {
+		st.Commit = ents[len(ents)-1].Index
+	}
+
+	log.Printf("etcdserver: forcing restart of member %s in cluster %s at commit index %d", id, cfg.Cluster.ID(), st.Commit)
+	s := raft.NewMemoryStorage()
+	if snapshot != nil {
+		s.ApplySnapshot(*snapshot)
+	}
+	s.SetHardState(st)
+	s.Append(ents)
+	n := raft.RestartNode(uint64(id), 10, 1, s)
+	return id, n, s, w
+}
+
+// getIDs returns an ordered set of IDs included in the given snapshot and
+// the entries. The given snapshot/entries can contain two kinds of
+// ID-related entry:
+// - ConfChangeAddNode, in which case the contained ID will be added into the set.
+// - ConfChangeAddRemove, in which case the contained ID will be removed from the set.
+func getIDs(snap *raftpb.Snapshot, ents []raftpb.Entry) []uint64 {
+	ids := make(map[uint64]bool)
+	if snap != nil {
+		for _, id := range snap.Metadata.ConfState.Nodes {
+			ids[id] = true
+		}
+	}
+	for _, e := range ents {
+		if e.Type != raftpb.EntryConfChange {
+			continue
+		}
+		var cc raftpb.ConfChange
+		pbutil.MustUnmarshal(&cc, e.Data)
+		switch cc.Type {
+		case raftpb.ConfChangeAddNode:
+			ids[cc.NodeID] = true
+		case raftpb.ConfChangeRemoveNode:
+			delete(ids, cc.NodeID)
+		default:
+			log.Panicf("ConfChange Type should be either ConfChangeAddNode or ConfChangeRemoveNode!")
+		}
+	}
+	sids := make(types.Uint64Slice, 0)
+	for id := range ids {
+		sids = append(sids, id)
+	}
+	sort.Sort(sids)
+	return []uint64(sids)
+}
+
+// createConfigChangeEnts creates a series of Raft entries (i.e.
+// EntryConfChange) to remove the set of given IDs from the cluster. The ID
+// `self` is _not_ removed, even if present in the set.
+// If `self` is not inside the given ids, it creates a Raft entry to add a
+// default member with the given `self`.
+func createConfigChangeEnts(ids []uint64, self uint64, term, index uint64) []raftpb.Entry {
+	ents := make([]raftpb.Entry, 0)
+	next := index + 1
+	found := false
+	for _, id := range ids {
+		if id == self {
+			found = true
+			continue
+		}
+		cc := &raftpb.ConfChange{
+			Type:   raftpb.ConfChangeRemoveNode,
+			NodeID: id,
+		}
+		e := raftpb.Entry{
+			Type:  raftpb.EntryConfChange,
+			Data:  pbutil.MustMarshal(cc),
+			Term:  term,
+			Index: next,
+		}
+		ents = append(ents, e)
+		next++
+	}
+	if !found {
+		m := Member{
+			ID:             types.ID(self),
+			RaftAttributes: RaftAttributes{PeerURLs: []string{"http://localhost:7001", "http://localhost:2380"}},
+		}
+		ctx, err := json.Marshal(m)
+		if err != nil {
+			log.Panicf("marshal member should never fail: %v", err)
+		}
+		cc := &raftpb.ConfChange{
+			Type:    raftpb.ConfChangeAddNode,
+			NodeID:  self,
+			Context: ctx,
+		}
+		e := raftpb.Entry{
+			Type:  raftpb.EntryConfChange,
+			Data:  pbutil.MustMarshal(cc),
+			Term:  term,
+			Index: next,
+		}
+		ents = append(ents, e)
+	}
+	return ents
+}

etcdserver/force_cluster_test.go

@@ -0,0 +1,145 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
+package etcdserver
+
+import (
+	"encoding/json"
+	"reflect"
+	"testing"
+
+	"github.com/coreos/etcd/pkg/pbutil"
+	"github.com/coreos/etcd/pkg/types"
+	"github.com/coreos/etcd/raft/raftpb"
+)
+
+func TestGetIDs(t *testing.T) {
+	addcc := &raftpb.ConfChange{Type: raftpb.ConfChangeAddNode, NodeID: 2}
+	addEntry := raftpb.Entry{Type: raftpb.EntryConfChange, Data: pbutil.MustMarshal(addcc)}
+	removecc := &raftpb.ConfChange{Type: raftpb.ConfChangeRemoveNode, NodeID: 2}
+	removeEntry := raftpb.Entry{Type: raftpb.EntryConfChange, Data: pbutil.MustMarshal(removecc)}
+	normalEntry := raftpb.Entry{Type: raftpb.EntryNormal}
+
+	tests := []struct {
+		confState *raftpb.ConfState
+		ents      []raftpb.Entry
+
+		widSet []uint64
+	}{
+		{nil, []raftpb.Entry{}, []uint64{}},
+		{&raftpb.ConfState{Nodes: []uint64{1}},
+			[]raftpb.Entry{}, []uint64{1}},
+		{&raftpb.ConfState{Nodes: []uint64{1}},
+			[]raftpb.Entry{addEntry}, []uint64{1, 2}},
+		{&raftpb.ConfState{Nodes: []uint64{1}},
+			[]raftpb.Entry{addEntry, removeEntry}, []uint64{1}},
+		{&raftpb.ConfState{Nodes: []uint64{1}},
+			[]raftpb.Entry{addEntry, normalEntry}, []uint64{1, 2}},
+		{&raftpb.ConfState{Nodes: []uint64{1}},
+			[]raftpb.Entry{addEntry, removeEntry, normalEntry}, []uint64{1}},
+	}
+
+	for i, tt := range tests {
+		var snap raftpb.Snapshot
+		if tt.confState != nil {
+			snap.Metadata.ConfState = *tt.confState
+		}
+		idSet := getIDs(&snap, tt.ents)
+		if !reflect.DeepEqual(idSet, tt.widSet) {
+			t.Errorf("#%d: idset = %#v, want %#v", i, idSet, tt.widSet)
+		}
+	}
+}
+
+func TestCreateConfigChangeEnts(t *testing.T) {
+	m := Member{
+		ID:             types.ID(1),
+		RaftAttributes: RaftAttributes{PeerURLs: []string{"http://localhost:7001", "http://localhost:2380"}},
+	}
+	ctx, err := json.Marshal(m)
+	if err != nil {
+		t.Fatal(err)
+	}
+	addcc1 := &raftpb.ConfChange{Type: raftpb.ConfChangeAddNode, NodeID: 1, Context: ctx}
+	removecc2 := &raftpb.ConfChange{Type: raftpb.ConfChangeRemoveNode, NodeID: 2}
+	removecc3 := &raftpb.ConfChange{Type: raftpb.ConfChangeRemoveNode, NodeID: 3}
+	tests := []struct {
+		ids         []uint64
+		self        uint64
+		term, index uint64
+
+		wents []raftpb.Entry
+	}{
+		{
+			[]uint64{1},
+			1,
+			1, 1,
+
+			[]raftpb.Entry{},
+		},
+		{
+			[]uint64{1, 2},
+			1,
+			1, 1,
+
+			[]raftpb.Entry{{Term: 1, Index: 2, Type: raftpb.EntryConfChange, Data: pbutil.MustMarshal(removecc2)}},
+		},
+		{
+			[]uint64{1, 2},
+			1,
+			2, 2,
+
+			[]raftpb.Entry{{Term: 2, Index: 3, Type: raftpb.EntryConfChange, Data: pbutil.MustMarshal(removecc2)}},
+		},
+		{
+			[]uint64{1, 2, 3},
+			1,
+			2, 2,
+
+			[]raftpb.Entry{
+				{Term: 2, Index: 3, Type: raftpb.EntryConfChange, Data: pbutil.MustMarshal(removecc2)},
+				{Term: 2, Index: 4, Type: raftpb.EntryConfChange, Data: pbutil.MustMarshal(removecc3)},
+			},
+		},
+		{
+			[]uint64{2, 3},
+			2,
+			2, 2,
+
+			[]raftpb.Entry{
+				{Term: 2, Index: 3, Type: raftpb.EntryConfChange, Data: pbutil.MustMarshal(removecc3)},
+			},
+		},
+		{
+			[]uint64{2, 3},
+			1,
+			2, 2,
+
+			[]raftpb.Entry{
+				{Term: 2, Index: 3, Type: raftpb.EntryConfChange, Data: pbutil.MustMarshal(removecc2)},
+				{Term: 2, Index: 4, Type: raftpb.EntryConfChange, Data: pbutil.MustMarshal(removecc3)},
+				{Term: 2, Index: 5, Type: raftpb.EntryConfChange, Data: pbutil.MustMarshal(addcc1)},
+			},
+		},
+	}
+
+	for i, tt := range tests {
+		gents := createConfigChangeEnts(tt.ids, tt.self, tt.term, tt.index)
+		if !reflect.DeepEqual(gents, tt.wents) {
+			t.Errorf("#%d: ents = %v, want %v", i, gents, tt.wents)
+		}
+	}
+}

etcdserver/member.go

@@ -47,7 +47,7 @@ type Member struct {
 	Attributes
 }
 
-// newMember creates a Member without an ID and generates one based on the
+// NewMember creates a Member without an ID and generates one based on the
 // name, peer URLs. This is used for bootstrapping/adding new member.
 func NewMember(name string, peerURLs types.URLs, clusterName string, now *time.Time) *Member {
 	m := &Member{
@@ -75,15 +75,40 @@ func NewMember(name string, peerURLs types.URLs, clusterName string, now *time.T
 // It will panic if there is no PeerURLs available in Member.
 func (m *Member) PickPeerURL() string {
 	if len(m.PeerURLs) == 0 {
-		panic("member should always have some peer url")
+		log.Panicf("member should always have some peer url")
 	}
 	return m.PeerURLs[rand.Intn(len(m.PeerURLs))]
 }
 
+func (m *Member) Clone() *Member {
+	if m == nil {
+		return nil
+	}
+	mm := &Member{
+		ID: m.ID,
+		Attributes: Attributes{
+			Name: m.Name,
+		},
+	}
+	if m.PeerURLs != nil {
+		mm.PeerURLs = make([]string, len(m.PeerURLs))
+		copy(mm.PeerURLs, m.PeerURLs)
+	}
+	if m.ClientURLs != nil {
+		mm.ClientURLs = make([]string, len(m.ClientURLs))
+		copy(mm.ClientURLs, m.ClientURLs)
+	}
+	return mm
+}
+
 func memberStoreKey(id types.ID) string {
 	return path.Join(storeMembersPrefix, id.String())
 }
 
+func MemberAttributesStorePath(id types.ID) string {
+	return path.Join(memberStoreKey(id), attributesSuffix)
+}
+
 func mustParseMemberIDFromKey(key string) types.ID {
 	id, err := types.IDFromString(path.Base(key))
 	if err != nil {

etcdserver/member_test.go

@@ -18,6 +18,7 @@ package etcdserver
 
 import (
 	"net/url"
+	"reflect"
 	"testing"
 	"time"
 
@@ -62,7 +63,7 @@ func TestMemberPick(t *testing.T) {
 		urls map[string]bool
 	}{
 		{
-			newTestMemberp(1, []string{"abc", "def", "ghi", "jkl", "mno", "pqr", "stu"}, "", nil),
+			newTestMember(1, []string{"abc", "def", "ghi", "jkl", "mno", "pqr", "stu"}, "", nil),
 			map[string]bool{
 				"abc": true,
 				"def": true,
@@ -74,7 +75,7 @@ func TestMemberPick(t *testing.T) {
 			},
 		},
 		{
-			newTestMemberp(2, []string{"xyz"}, "", nil),
+			newTestMember(2, []string{"xyz"}, "", nil),
 			map[string]bool{"xyz": true},
 		},
 	}
@@ -88,3 +89,29 @@ func TestMemberPick(t *testing.T) {
 		}
 	}
 }
+
+func TestMemberClone(t *testing.T) {
+	tests := []*Member{
+		newTestMember(1, nil, "abc", nil),
+		newTestMember(1, []string{"http://a"}, "abc", nil),
+		newTestMember(1, nil, "abc", []string{"http://b"}),
+		newTestMember(1, []string{"http://a"}, "abc", []string{"http://b"}),
+	}
+	for i, tt := range tests {
+		nm := tt.Clone()
+		if nm == tt {
+			t.Errorf("#%d: the pointers are the same, and clone doesn't happen", i)
+		}
+		if !reflect.DeepEqual(nm, tt) {
+			t.Errorf("#%d: member = %+v, want %+v", i, nm, tt)
+		}
+	}
+}
+
+func newTestMember(id uint64, peerURLs []string, name string, clientURLs []string) *Member {
+	return &Member{
+		ID:             types.ID(id),
+		RaftAttributes: RaftAttributes{PeerURLs: peerURLs},
+		Attributes:     Attributes{Name: name, ClientURLs: clientURLs},
+	}
+}

etcdserver/sender.go

@@ -1,124 +0,0 @@
-/*
-   Copyright 2014 CoreOS, Inc.
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
-*/
-
-package etcdserver
-
-import (
-	"bytes"
-	"fmt"
-	"log"
-	"net/http"
-	"time"
-
-	"github.com/coreos/etcd/etcdserver/stats"
-	"github.com/coreos/etcd/pkg/types"
-	"github.com/coreos/etcd/raft/raftpb"
-)
-
-const raftPrefix = "/raft"
-
-// Sender creates the default production sender used to transport raft messages
-// in the cluster. The returned sender will update the given ServerStats and
-// LeaderStats appropriately.
-func Sender(t *http.Transport, cl *Cluster, ss *stats.ServerStats, ls *stats.LeaderStats) func(msgs []raftpb.Message) {
-	c := &http.Client{Transport: t}
-
-	return func(msgs []raftpb.Message) {
-		for _, m := range msgs {
-			// TODO: reuse go routines
-			// limit the number of outgoing connections for the same receiver
-			go send(c, cl, m, ss, ls)
-		}
-	}
-}
-
-// send uses the given client to send a message to a member in the given
-// ClusterStore, retrying up to 3 times for each message. The given
-// ServerStats and LeaderStats are updated appropriately
-func send(c *http.Client, cl *Cluster, m raftpb.Message, ss *stats.ServerStats, ls *stats.LeaderStats) {
-	to := types.ID(m.To)
-	cid := cl.ID()
-	// TODO (xiangli): reasonable retry logic
-	for i := 0; i < 3; i++ {
-		memb := cl.Member(to)
-		if memb == nil {
-			if !cl.IsIDRemoved(to) {
-				// TODO: unknown peer id.. what do we do? I
-				// don't think his should ever happen, need to
-				// look into this further.
-				log.Printf("etcdserver: error sending message to unknown receiver %s", to.String())
-			}
-			return
-		}
-		u := fmt.Sprintf("%s%s", memb.PickPeerURL(), raftPrefix)
-
-		// TODO: don't block. we should be able to have 1000s
-		// of messages out at a time.
-		data, err := m.Marshal()
-		if err != nil {
-			log.Println("sender: dropping message:", err)
-			return // drop bad message
-		}
-		if m.Type == raftpb.MsgApp {
-			ss.SendAppendReq(len(data))
-		}
-		fs := ls.Follower(to.String())
-
-		start := time.Now()
-		sent := httpPost(c, u, cid, data)
-		end := time.Now()
-		if sent {
-			fs.Succ(end.Sub(start))
-			return
-		}
-		fs.Fail()
-		// TODO: backoff
-	}
-}
-
-// httpPost POSTs a data payload to a url using the given client. Returns true
-// if the POST succeeds, false on any failure.
-func httpPost(c *http.Client, url string, cid types.ID, data []byte) bool {
-	req, err := http.NewRequest("POST", url, bytes.NewBuffer(data))
-	if err != nil {
-		// TODO: log the error?
-		return false
-	}
-	req.Header.Set("Content-Type", "application/protobuf")
-	req.Header.Set("X-Etcd-Cluster-ID", cid.String())
-	resp, err := c.Do(req)
-	if err != nil {
-		// TODO: log the error?
-		return false
-	}
-	resp.Body.Close()
-
-	switch resp.StatusCode {
-	case http.StatusPreconditionFailed:
-		// TODO: shutdown the etcdserver gracefully?
-		log.Fatalf("etcd: conflicting cluster ID with the target cluster (%s != %s)", resp.Header.Get("X-Etcd-Cluster-ID"), cid.String())
-		return false
-	case http.StatusForbidden:
-		// TODO: stop the server
-		log.Println("etcd: this member has been permanently removed from the cluster")
-		log.Fatalln("etcd: the data-dir used by this member must be removed so that this host can be re-added with a new member ID")
-		return false
-	case http.StatusNoContent:
-		return true
-	default:
-		return false
-	}
-}

etcdserver/sendhub.go

@@ -0,0 +1,162 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
+package etcdserver
+
+import (
+	"log"
+	"net/http"
+	"net/url"
+	"path"
+	"sync"
+
+	"github.com/coreos/etcd/etcdserver/stats"
+	"github.com/coreos/etcd/pkg/types"
+	"github.com/coreos/etcd/raft/raftpb"
+	"github.com/coreos/etcd/rafthttp"
+)
+
+const (
+	raftPrefix = "/raft"
+)
+
+type SendHub interface {
+	rafthttp.SenderFinder
+	Send(m []raftpb.Message)
+	Add(m *Member)
+	Remove(id types.ID)
+	Update(m *Member)
+	Stop()
+	ShouldStopNotify() <-chan struct{}
+}
+
+type sendHub struct {
+	tr         http.RoundTripper
+	cl         ClusterInfo
+	p          rafthttp.Processor
+	ss         *stats.ServerStats
+	ls         *stats.LeaderStats
+	mu         sync.RWMutex // protect the sender map
+	senders    map[types.ID]rafthttp.Sender
+	shouldstop chan struct{}
+}
+
+// newSendHub creates the default send hub used to transport raft messages
+// to other members. The returned sendHub will update the given ServerStats and
+// LeaderStats appropriately.
+func newSendHub(t http.RoundTripper, cl ClusterInfo, p rafthttp.Processor, ss *stats.ServerStats, ls *stats.LeaderStats) *sendHub {
+	return &sendHub{
+		tr:         t,
+		cl:         cl,
+		p:          p,
+		ss:         ss,
+		ls:         ls,
+		senders:    make(map[types.ID]rafthttp.Sender),
+		shouldstop: make(chan struct{}, 1),
+	}
+}
+
+func (h *sendHub) Sender(id types.ID) rafthttp.Sender {
+	h.mu.RLock()
+	defer h.mu.RUnlock()
+	return h.senders[id]
+}
+
+func (h *sendHub) Send(msgs []raftpb.Message) {
+	for _, m := range msgs {
+		to := types.ID(m.To)
+		s, ok := h.senders[to]
+		if !ok {
+			if !h.cl.IsIDRemoved(to) {
+				log.Printf("etcdserver: send message to unknown receiver %s", to)
+			}
+			continue
+		}
+
+		if m.Type == raftpb.MsgApp {
+			h.ss.SendAppendReq(m.Size())
+		}
+
+		s.Send(m)
+	}
+}
+
+func (h *sendHub) Stop() {
+	for _, s := range h.senders {
+		s.Stop()
+	}
+	if tr, ok := h.tr.(*http.Transport); ok {
+		tr.CloseIdleConnections()
+	}
+}
+
+func (h *sendHub) ShouldStopNotify() <-chan struct{} {
+	return h.shouldstop
+}
+
+func (h *sendHub) Add(m *Member) {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	if _, ok := h.senders[m.ID]; ok {
+		return
+	}
+	// TODO: considering how to switch between all available peer urls
+	peerURL := m.PickPeerURL()
+	u, err := url.Parse(peerURL)
+	if err != nil {
+		log.Panicf("unexpect peer url %s", peerURL)
+	}
+	u.Path = path.Join(u.Path, raftPrefix)
+	fs := h.ls.Follower(m.ID.String())
+	s := rafthttp.NewSender(h.tr, u.String(), m.ID, h.cl.ID(), h.p, fs, h.shouldstop)
+	h.senders[m.ID] = s
+}
+
+func (h *sendHub) Remove(id types.ID) {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	h.senders[id].Stop()
+	delete(h.senders, id)
+}
+
+func (h *sendHub) Update(m *Member) {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	// TODO: return error or just panic?
+	if _, ok := h.senders[m.ID]; !ok {
+		return
+	}
+	peerURL := m.PickPeerURL()
+	u, err := url.Parse(peerURL)
+	if err != nil {
+		log.Panicf("unexpect peer url %s", peerURL)
+	}
+	u.Path = path.Join(u.Path, raftPrefix)
+	h.senders[m.ID].Update(u.String())
+}
+
+// for testing
+func (h *sendHub) pause() {
+	for _, s := range h.senders {
+		s.Pause()
+	}
+}
+
+func (h *sendHub) resume() {
+	for _, s := range h.senders {
+		s.Resume()
+	}
+}

etcdserver/sendhub_test.go

@@ -0,0 +1,104 @@
+/*
+   Copyright 2014 CoreOS, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+*/
+
+package etcdserver
+
+import (
+	"net/http"
+	"testing"
+	"time"
+
+	"github.com/coreos/etcd/etcdserver/stats"
+	"github.com/coreos/etcd/pkg/testutil"
+	"github.com/coreos/etcd/pkg/types"
+	"github.com/coreos/etcd/raft/raftpb"
+)
+
+func TestSendHubAdd(t *testing.T) {
+	cl := newTestCluster(nil)
+	ls := stats.NewLeaderStats("")
+	h := newSendHub(nil, cl, nil, nil, ls)
+	m := newTestMember(1, []string{"http://a"}, "", nil)
+	h.Add(m)
+
+	if _, ok := ls.Followers["1"]; !ok {
+		t.Errorf("FollowerStats[1] is nil, want exists")
+	}
+	s, ok := h.senders[types.ID(1)]
+	if !ok {
+		t.Fatalf("senders[1] is nil, want exists")
+	}
+
+	h.Add(m)
+	ns := h.senders[types.ID(1)]
+	if s != ns {
+		t.Errorf("sender = %p, want %p", ns, s)
+	}
+}
+
+func TestSendHubRemove(t *testing.T) {
+	cl := newTestCluster(nil)
+	ls := stats.NewLeaderStats("")
+	h := newSendHub(nil, cl, nil, nil, ls)
+	m := newTestMember(1, []string{"http://a"}, "", nil)
+	h.Add(m)
+	h.Remove(types.ID(1))
+
+	if _, ok := h.senders[types.ID(1)]; ok {
+		t.Fatalf("senders[1] exists, want removed")
+	}
+}
+
+func TestSendHubShouldStop(t *testing.T) {
+	tr := newRespRoundTripper(http.StatusForbidden, nil)
+	cl := newTestCluster(nil)
+	ls := stats.NewLeaderStats("")
+	h := newSendHub(tr, cl, nil, nil, ls)
+	m := newTestMember(1, []string{"http://a"}, "", nil)
+	h.Add(m)
+
+	shouldstop := h.ShouldStopNotify()
+	select {
+	case <-shouldstop:
+		t.Fatalf("received unexpected shouldstop notification")
+	case <-time.After(10 * time.Millisecond):
+	}
+	h.senders[1].Send(raftpb.Message{})
+
+	testutil.ForceGosched()
+	select {
+	case <-shouldstop:
+	default:
+		t.Fatalf("cannot receive stop notification")
+	}
+}
+
+type respRoundTripper struct {
+	code int
+	err  error
+}
+
+func newRespRoundTripper(code int, err error) *respRoundTripper {
+	return &respRoundTripper{code: code, err: err}
+}
+func (t *respRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
+	return &http.Response{StatusCode: t.code, Body: &nopReadCloser{}}, t.err
+}
+
+type nopReadCloser struct{}
+
+func (n *nopReadCloser) Read(p []byte) (int, error) { return 0, nil }
+func (n *nopReadCloser) Close() error               { return nil }

etcdserver/server.go

@@ -27,20 +27,25 @@ import (
 	"os"
 	"path"
 	"regexp"
+	"sort"
 	"sync/atomic"
 	"time"
 
-	"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 	"github.com/coreos/etcd/discovery"
+	"github.com/coreos/etcd/etcdserver/etcdhttp/httptypes"
 	pb "github.com/coreos/etcd/etcdserver/etcdserverpb"
 	"github.com/coreos/etcd/etcdserver/stats"
+	"github.com/coreos/etcd/migrate"
+	"github.com/coreos/etcd/pkg/fileutil"
 	"github.com/coreos/etcd/pkg/pbutil"
 	"github.com/coreos/etcd/pkg/types"
+	"github.com/coreos/etcd/pkg/wait"
 	"github.com/coreos/etcd/raft"
 	"github.com/coreos/etcd/raft/raftpb"
+	"github.com/coreos/etcd/rafthttp"
 	"github.com/coreos/etcd/snap"
 	"github.com/coreos/etcd/store"
-	"github.com/coreos/etcd/wait"
 	"github.com/coreos/etcd/wal"
 )
 
@@ -55,15 +60,17 @@ const (
 
 	StoreAdminPrefix = "/0"
 	StoreKeysPrefix  = "/1"
+
+	purgeFileInterval = 30 * time.Second
 )
 
 var (
 	ErrUnknownMethod = errors.New("etcdserver: unknown method")
 	ErrStopped       = errors.New("etcdserver: server stopped")
-	ErrRemoved       = errors.New("etcdserver: server removed")
 	ErrIDRemoved     = errors.New("etcdserver: ID removed")
 	ErrIDExists      = errors.New("etcdserver: ID exists")
 	ErrIDNotFound    = errors.New("etcdserver: ID not found")
+	ErrPeerURLexists = errors.New("etcdserver: peerURL exists")
 	ErrCanceled      = errors.New("etcdserver: request cancelled")
 	ErrTimeout       = errors.New("etcdserver: request timed out")
 
@@ -77,27 +84,12 @@ func init() {
 	rand.Seed(time.Now().UnixNano())
 }
 
-type sendFunc func(m []raftpb.Message)
-
 type Response struct {
 	Event   *store.Event
 	Watcher store.Watcher
 	err     error
 }
 
-type Storage interface {
-	// Save function saves ents and state to the underlying stable storage.
-	// Save MUST block until st and ents are on stable storage.
-	Save(st raftpb.HardState, ents []raftpb.Entry)
-	// SaveSnap function saves snapshot to the underlying stable storage.
-	SaveSnap(snap raftpb.Snapshot)
-
-	// TODO: WAL should be able to control cut itself. After implement self-controled cut,
-	// remove it in this interface.
-	// Cut cuts out a new wal file for saving new state and entries.
-	Cut() error
-}
-
 type Server interface {
 	// Start performs any initialization of the Server necessary for it to
 	// begin serving requests. It must be called before Do or Process.
@@ -107,7 +99,9 @@ type Server interface {
 	// Stop terminates the Server and performs any necessary finalization.
 	// Do and Process cannot be called after Stop has been invoked.
 	Stop()
-	// Do takes a request and attempts to fulfil it, returning a Response.
+	// ID returns the ID of the Server.
+	ID() types.ID
+	// Do takes a request and attempts to fulfill it, returning a Response.
 	Do(ctx context.Context, r pb.Request) (Response, error)
 	// Process takes a raft message and applies it to the server's raft state
 	// machine, respecting any timeout of the given context.
@@ -120,18 +114,10 @@ type Server interface {
 	// return ErrIDRemoved if member ID is removed from the cluster, or return
 	// ErrIDNotFound if member ID is not in the cluster.
 	RemoveMember(ctx context.Context, id uint64) error
-}
 
-type Stats interface {
-	// SelfStats returns the struct representing statistics of this server
-	SelfStats() []byte
-	// LeaderStats returns the statistics of all followers in the cluster
-	// if this server is leader. Otherwise, nil is returned.
-	LeaderStats() []byte
-	// StoreStats returns statistics of the store backing this EtcdServer
-	StoreStats() []byte
-	// UpdateRecvApp updates the underlying statistics in response to a receiving an Append request
-	UpdateRecvApp(from types.ID, length int64)
+	// UpdateMember attempts to update a existing member in the cluster. It will
+	// return ErrIDNotFound if the member ID does not exist.
+	UpdateMember(ctx context.Context, updateMemb Member) error
 }
 
 type RaftTimer interface {
@@ -141,25 +127,27 @@ type RaftTimer interface {
 
 // EtcdServer is the production implementation of the Server interface
 type EtcdServer struct {
+	cfg        *ServerConfig
 	w          wait.Wait
 	done       chan struct{}
-	stopped    chan struct{}
+	stop       chan struct{}
 	id         types.ID
 	attributes Attributes
 
 	Cluster *Cluster
 
-	node  raft.Node
-	store store.Store
+	node        raft.Node
+	raftStorage *raft.MemoryStorage
+	store       store.Store
 
 	stats  *stats.ServerStats
 	lstats *stats.LeaderStats
 
-	// send specifies the send function for sending msgs to members. send
+	// sender specifies the sender to send msgs to members. sending msgs
 	// MUST NOT block. It is okay to drop messages, since clients should
 	// timeout and reissue their messages.  If send is nil, server will
 	// panic.
-	send sendFunc
+	sendhub SendHub
 
 	storage Storage
 
@@ -171,70 +159,111 @@ type EtcdServer struct {
 	// Cache of the latest raft index and raft term the server has seen
 	raftIndex uint64
 	raftTerm  uint64
+
+	raftLead uint64
+}
+
+// UpgradeWAL converts an older version of the EtcdServer data to the newest version.
+// It must ensure that, after upgrading, the most recent version is present.
+func UpgradeWAL(cfg *ServerConfig, ver wal.WalVersion) error {
+	if ver == wal.WALv0_4 {
+		log.Print("Converting v0.4 log to v0.5")
+		err := migrate.Migrate4To5(cfg.DataDir, cfg.Name)
+		if err != nil {
+			log.Fatalf("Failed migrating data-dir: %v", err)
+			return err
+		}
+	}
+	return nil
 }
 
 // NewServer creates a new EtcdServer from the supplied configuration. The
 // configuration is considered static for the lifetime of the EtcdServer.
-func NewServer(cfg *ServerConfig) *EtcdServer {
-	if err := os.MkdirAll(cfg.SnapDir(), privateDirMode); err != nil {
-		log.Fatalf("etcdserver: cannot create snapshot directory: %v", err)
-	}
-	ss := snap.New(cfg.SnapDir())
+func NewServer(cfg *ServerConfig) (*EtcdServer, error) {
 	st := store.New()
 	var w *wal.WAL
 	var n raft.Node
+	var s *raft.MemoryStorage
 	var id types.ID
-	haveWAL := wal.Exist(cfg.WALDir())
-	switch {
-	case !haveWAL && cfg.ClusterState == ClusterStateValueExisting:
-		cl, err := GetClusterFromPeers(cfg.Cluster.PeerURLs())
+	walVersion, err := wal.DetectVersion(cfg.DataDir)
+	if err != nil {
+		return nil, err
+	}
+	if walVersion == wal.WALUnknown {
+		return nil, fmt.Errorf("unknown wal version in data dir %s", cfg.DataDir)
+	}
+	haveWAL := walVersion != wal.WALNotExist
+
+	if haveWAL && walVersion != wal.WALv0_5 {
+		err := UpgradeWAL(cfg, walVersion)
 		if err != nil {
-			log.Fatal(err)
+			return nil, err
 		}
-		if err := cfg.Cluster.ValidateAndAssignIDs(cl.Members()); err != nil {
-			log.Fatalf("etcdserver: error validating IDs from cluster %s: %v", cl, err)
+	}
+
+	ss := snap.New(cfg.SnapDir())
+	switch {
+	case !haveWAL && !cfg.NewCluster:
+		us := getOtherPeerURLs(cfg.Cluster, cfg.Name)
+		existingCluster, err := GetClusterFromPeers(us)
+		if err != nil {
+			return nil, fmt.Errorf("cannot fetch cluster info from peer urls: %v", err)
 		}
-		cfg.Cluster.SetID(cl.id)
+		if err := ValidateClusterAndAssignIDs(cfg.Cluster, existingCluster); err != nil {
+			return nil, fmt.Errorf("error validating peerURLs %s: %v", existingCluster, err)
+		}
+		cfg.Cluster.SetID(existingCluster.id)
 		cfg.Cluster.SetStore(st)
-		id, n, w = startNode(cfg, nil)
-	case !haveWAL && cfg.ClusterState == ClusterStateValueNew:
+		cfg.Print()
+		id, n, s, w = startNode(cfg, nil)
+	case !haveWAL && cfg.NewCluster:
 		if err := cfg.VerifyBootstrapConfig(); err != nil {
-			log.Fatalf("etcdserver: %v", err)
+			return nil, err
+		}
+		if err := checkClientURLsEmptyFromPeers(cfg.Cluster, cfg.Name); err != nil {
+			return nil, err
 		}
 		m := cfg.Cluster.MemberByName(cfg.Name)
 		if cfg.ShouldDiscover() {
-			d, err := discovery.New(cfg.DiscoveryURL, m.ID, cfg.Cluster.String())
+			s, err := discovery.JoinCluster(cfg.DiscoveryURL, cfg.DiscoveryProxy, m.ID, cfg.Cluster.String())
 			if err != nil {
-				log.Fatalf("etcdserver: cannot init discovery %v", err)
-			}
-			s, err := d.Discover()
-			if err != nil {
-				log.Fatalf("etcdserver: %v", err)
+				return nil, err
 			}
 			if cfg.Cluster, err = NewClusterFromString(cfg.Cluster.token, s); err != nil {
-				log.Fatalf("etcdserver: %v", err)
+				return nil, err
 			}
 		}
 		cfg.Cluster.SetStore(st)
-		id, n, w = startNode(cfg, cfg.Cluster.MemberIDs())
+		cfg.PrintWithInitial()
+		id, n, s, w = startNode(cfg, cfg.Cluster.MemberIDs())
 	case haveWAL:
 		if cfg.ShouldDiscover() {
-			log.Printf("etcdserver: warn: ignoring discovery: etcd has already been initialized and has a valid log in %q", cfg.WALDir())
+			log.Printf("etcdserver: discovery token ignored since a cluster has already been initialized. Valid log found at %q", cfg.WALDir())
 		}
 		var index uint64
 		snapshot, err := ss.Load()
 		if err != nil && err != snap.ErrNoSnapshot {
-			log.Fatal(err)
+			return nil, err
 		}
 		if snapshot != nil {
-			log.Printf("etcdserver: recovering from snapshot at index %d", snapshot.Index)
-			st.Recovery(snapshot.Data)
-			index = snapshot.Index
+			if err := st.Recovery(snapshot.Data); err != nil {
+				log.Panicf("etcdserver: recovered store from snapshot error: %v", err)
+			}
+			log.Printf("etcdserver: recovered store from snapshot at index %d", snapshot.Metadata.Index)
+			index = snapshot.Metadata.Index
 		}
 		cfg.Cluster = NewClusterFromStore(cfg.Cluster.token, st)
-		id, n, w = restartNode(cfg, index, snapshot)
+		cfg.Print()
+		if snapshot != nil {
+			log.Printf("etcdserver: loaded cluster information from store: %s", cfg.Cluster)
+		}
+		if !cfg.ForceNewCluster {
+			id, n, s, w = restartNode(cfg, index+1, snapshot)
+		} else {
+			id, n, s, w = restartAsStandaloneNode(cfg, index+1, snapshot)
+		}
 	default:
-		log.Fatalf("etcdserver: unsupported bootstrap config")
+		return nil, fmt.Errorf("unsupported bootstrap config")
 	}
 
 	sstats := &stats.ServerStats{
@@ -243,24 +272,26 @@ func NewServer(cfg *ServerConfig) *EtcdServer {
 	}
 	lstats := stats.NewLeaderStats(id.String())
 
-	s := &EtcdServer{
-		store:      st,
-		node:       n,
-		id:         id,
-		attributes: Attributes{Name: cfg.Name, ClientURLs: cfg.ClientURLs.StringSlice()},
-		Cluster:    cfg.Cluster,
-		storage: struct {
-			*wal.WAL
-			*snap.Snapshotter
-		}{w, ss},
-		stats:      sstats,
-		lstats:     lstats,
-		send:       Sender(cfg.Transport, cfg.Cluster, sstats, lstats),
-		Ticker:     time.Tick(100 * time.Millisecond),
-		SyncTicker: time.Tick(500 * time.Millisecond),
-		snapCount:  cfg.SnapCount,
+	srv := &EtcdServer{
+		cfg:         cfg,
+		store:       st,
+		node:        n,
+		raftStorage: s,
+		id:          id,
+		attributes:  Attributes{Name: cfg.Name, ClientURLs: cfg.ClientURLs.StringSlice()},
+		Cluster:     cfg.Cluster,
+		storage:     NewStorage(w, ss),
+		stats:       sstats,
+		lstats:      lstats,
+		Ticker:      time.Tick(100 * time.Millisecond),
+		SyncTicker:  time.Tick(500 * time.Millisecond),
+		snapCount:   cfg.SnapCount,
 	}
-	return s
+	srv.sendhub = newSendHub(cfg.Transport, cfg.Cluster, srv, sstats, lstats)
+	for _, m := range getOtherMembers(cfg.Cluster, cfg.Name) {
+		srv.sendhub.Add(m)
+	}
+	return srv, nil
 }
 
 // Start prepares and starts server in a new goroutine. It is no longer safe to
@@ -269,6 +300,7 @@ func NewServer(cfg *ServerConfig) *EtcdServer {
 func (s *EtcdServer) Start() {
 	s.start()
 	go s.publish(defaultPublishRetryInterval)
+	go s.purgeFile()
 }
 
 // start prepares and starts server in a new goroutine. It is no longer safe to
@@ -281,71 +313,145 @@ func (s *EtcdServer) start() {
 	}
 	s.w = wait.New()
 	s.done = make(chan struct{})
-	s.stopped = make(chan struct{})
+	s.stop = make(chan struct{})
 	s.stats.Initialize()
 	// TODO: if this is an empty log, writes all peer infos
 	// into the first entry
 	go s.run()
 }
 
+func (s *EtcdServer) purgeFile() {
+	var serrc, werrc <-chan error
+	if s.cfg.MaxSnapFiles > 0 {
+		serrc = fileutil.PurgeFile(s.cfg.SnapDir(), "snap", s.cfg.MaxSnapFiles, purgeFileInterval, s.done)
+	}
+	if s.cfg.MaxWALFiles > 0 {
+		werrc = fileutil.PurgeFile(s.cfg.WALDir(), "wal", s.cfg.MaxWALFiles, purgeFileInterval, s.done)
+	}
+	select {
+	case e := <-werrc:
+		log.Fatalf("etcdserver: failed to purge wal file %v", e)
+	case e := <-serrc:
+		log.Fatalf("etcdserver: failed to purge snap file %v", e)
+	case <-s.done:
+		return
+	}
+}
+
+func (s *EtcdServer) ID() types.ID { return s.id }
+
+func (s *EtcdServer) SenderFinder() rafthttp.SenderFinder { return s.sendhub }
+
 func (s *EtcdServer) Process(ctx context.Context, m raftpb.Message) error {
 	if s.Cluster.IsIDRemoved(types.ID(m.From)) {
-		return ErrRemoved
+		log.Printf("etcdserver: reject message from removed member %s", types.ID(m.From).String())
+		return httptypes.NewHTTPError(http.StatusForbidden, "cannot process message from removed member")
+	}
+	if m.Type == raftpb.MsgApp {
+		s.stats.RecvAppendReq(types.ID(m.From).String(), m.Size())
 	}
 	return s.node.Step(ctx, m)
 }
 
 func (s *EtcdServer) run() {
 	var syncC <-chan time.Time
+	var shouldstop bool
+	shouldstopC := s.sendhub.ShouldStopNotify()
+
+	// load initial state from raft storage
+	snap, err := s.raftStorage.Snapshot()
+	if err != nil {
+		log.Panicf("etcdserver: get snapshot from raft storage error: %v", err)
+	}
 	// snapi indicates the index of the last submitted snapshot request
-	var snapi, appliedi uint64
-	var nodes []uint64
+	snapi := snap.Metadata.Index
+	appliedi := snap.Metadata.Index
+	confState := snap.Metadata.ConfState
+
+	defer func() {
+		s.node.Stop()
+		s.sendhub.Stop()
+		if err := s.storage.Close(); err != nil {
+			log.Panicf("etcdserver: close storage error: %v", err)
+		}
+		close(s.done)
+	}()
 	for {
 		select {
 		case <-s.Ticker:
 			s.node.Tick()
 		case rd := <-s.node.Ready():
 			if rd.SoftState != nil {
-				nodes = rd.SoftState.Nodes
+				atomic.StoreUint64(&s.raftLead, rd.SoftState.Lead)
 				if rd.RaftState == raft.StateLeader {
 					syncC = s.SyncTicker
+					// TODO: remove the nil checking
+					// current test utility does not provide the stats
+					if s.stats != nil {
+						s.stats.BecomeLeader()
+					}
 				} else {
 					syncC = nil
 				}
 			}
 
-			s.storage.Save(rd.HardState, rd.Entries)
-			s.storage.SaveSnap(rd.Snapshot)
-			s.send(rd.Messages)
+			// apply snapshot to storage if it is more updated than current snapi
+			if !raft.IsEmptySnap(rd.Snapshot) && rd.Snapshot.Metadata.Index > snapi {
+				if err := s.storage.SaveSnap(rd.Snapshot); err != nil {
+					log.Fatalf("etcdserver: save snapshot error: %v", err)
+				}
+				s.raftStorage.ApplySnapshot(rd.Snapshot)
+				snapi = rd.Snapshot.Metadata.Index
+				log.Printf("etcdserver: saved incoming snapshot at index %d", snapi)
+			}
 
+			if err := s.storage.Save(rd.HardState, rd.Entries); err != nil {
+				log.Fatalf("etcdserver: save state and entries error: %v", err)
+			}
+			s.raftStorage.Append(rd.Entries)
+
+			s.sendhub.Send(rd.Messages)
+
+			// recover from snapshot if it is more updated than current applied
+			if !raft.IsEmptySnap(rd.Snapshot) && rd.Snapshot.Metadata.Index > appliedi {
+				if err := s.store.Recovery(rd.Snapshot.Data); err != nil {
+					log.Panicf("recovery store error: %v", err)
+				}
+				s.Cluster.Recover()
+				appliedi = rd.Snapshot.Metadata.Index
+				log.Printf("etcdserver: recovered from incoming snapshot at index %d", snapi)
+			}
 			// TODO(bmizerany): do this in the background, but take
 			// care to apply entries in a single goroutine, and not
 			// race them.
-			// TODO: apply configuration change into ClusterStore.
 			if len(rd.CommittedEntries) != 0 {
-				appliedi = s.apply(rd.CommittedEntries, nodes)
-			}
-
-			if rd.Snapshot.Index > snapi {
-				snapi = rd.Snapshot.Index
-			}
-
-			// recover from snapshot if it is more updated than current applied
-			if rd.Snapshot.Index > appliedi {
-				if err := s.store.Recovery(rd.Snapshot.Data); err != nil {
-					panic("TODO: this is bad, what do we do about it?")
+				firsti := rd.CommittedEntries[0].Index
+				if firsti > appliedi+1 {
+					log.Panicf("etcdserver: first index of committed entry[%d] should <= appliedi[%d] + 1", firsti, appliedi)
+				}
+				var ents []raftpb.Entry
+				if appliedi+1-firsti < uint64(len(rd.CommittedEntries)) {
+					ents = rd.CommittedEntries[appliedi+1-firsti:]
+				}
+				if len(ents) > 0 {
+					if appliedi, shouldstop = s.apply(ents, &confState); shouldstop {
+						return
+					}
 				}
-				appliedi = rd.Snapshot.Index
 			}
 
+			s.node.Advance()
+
 			if appliedi-snapi > s.snapCount {
-				s.snapshot(appliedi, nodes)
+				log.Printf("etcdserver: start to snapshot (applied: %d, lastsnap: %d)", appliedi, snapi)
+				s.snapshot(appliedi, &confState)
 				snapi = appliedi
 			}
 		case <-syncC:
 			s.sync(defaultSyncTimeout)
-		case <-s.done:
-			close(s.stopped)
+		case <-shouldstopC:
+			return
+		case <-s.stop:
 			return
 		}
 	}
@@ -354,11 +460,18 @@ func (s *EtcdServer) run() {
 // Stop stops the server gracefully, and shuts down the running goroutine.
 // Stop should be called after a Start(s), otherwise it will block forever.
 func (s *EtcdServer) Stop() {
-	s.node.Stop()
-	close(s.done)
-	<-s.stopped
+	select {
+	case s.stop <- struct{}{}:
+	case <-s.done:
+		return
+	}
+	<-s.done
 }
 
+// StopNotify returns a channel that receives a empty struct
+// when the server is stopped.
+func (s *EtcdServer) StopNotify() <-chan struct{} { return s.done }
+
 // Do interprets r and performs an operation on s.store according to r.Method
 // and other fields. If r.Method is "POST", "PUT", "DELETE", or a "GET" with
 // Quorum == true, r will be sent through consensus before performing its
@@ -366,7 +479,7 @@ func (s *EtcdServer) Stop() {
 // an error.
 func (s *EtcdServer) Do(ctx context.Context, r pb.Request) (Response, error) {
 	if r.ID == 0 {
-		panic("r.ID cannot be 0")
+		log.Panicf("request ID should never be 0")
 	}
 	if r.Method == "GET" && r.Quorum {
 		r.Method = "QGET"
@@ -404,27 +517,28 @@ func (s *EtcdServer) Do(ctx context.Context, r pb.Request) (Response, error) {
 			}
 			return Response{Event: ev}, nil
 		}
+	case "HEAD":
+		ev, err := s.store.Get(r.Path, r.Recursive, r.Sorted)
+		if err != nil {
+			return Response{}, err
+		}
+		return Response{Event: ev}, nil
 	default:
 		return Response{}, ErrUnknownMethod
 	}
 }
 
-func (s *EtcdServer) SelfStats() []byte {
-	return s.stats.JSON()
-}
+func (s *EtcdServer) SelfStats() []byte { return s.stats.JSON() }
 
 func (s *EtcdServer) LeaderStats() []byte {
-	// TODO(jonboulle): need to lock access to lstats, set it to nil when not leader, ...
+	lead := atomic.LoadUint64(&s.raftLead)
+	if lead != uint64(s.id) {
+		return nil
+	}
 	return s.lstats.JSON()
 }
 
-func (s *EtcdServer) StoreStats() []byte {
-	return s.store.JsonStats()
-}
-
-func (s *EtcdServer) UpdateRecvApp(from types.ID, length int64) {
-	s.stats.RecvAppendReq(from.String(), int(length))
-}
+func (s *EtcdServer) StoreStats() []byte { return s.store.JsonStats() }
 
 func (s *EtcdServer) AddMember(ctx context.Context, memb Member) error {
 	// TODO: move Member to protobuf type
@@ -450,17 +564,33 @@ func (s *EtcdServer) RemoveMember(ctx context.Context, id uint64) error {
 	return s.configure(ctx, cc)
 }
 
+func (s *EtcdServer) UpdateMember(ctx context.Context, memb Member) error {
+	b, err := json.Marshal(memb)
+	if err != nil {
+		return err
+	}
+	cc := raftpb.ConfChange{
+		ID:      GenID(),
+		Type:    raftpb.ConfChangeUpdateNode,
+		NodeID:  uint64(memb.ID),
+		Context: b,
+	}
+	return s.configure(ctx, cc)
+}
+
 // Implement the RaftTimer interface
-func (s *EtcdServer) Index() uint64 {
-	return atomic.LoadUint64(&s.raftIndex)
-}
+func (s *EtcdServer) Index() uint64 { return atomic.LoadUint64(&s.raftIndex) }
 
-func (s *EtcdServer) Term() uint64 {
-	return atomic.LoadUint64(&s.raftTerm)
-}
+func (s *EtcdServer) Term() uint64 { return atomic.LoadUint64(&s.raftTerm) }
 
-// configure sends configuration change through consensus then performs it.
-// It will block until the change is performed or there is an error.
+// Only for testing purpose
+// TODO: add Raft server interface to expose raft related info:
+// Index, Term, Lead, Committed, Applied, LastIndex, etc.
+func (s *EtcdServer) Lead() uint64 { return atomic.LoadUint64(&s.raftLead) }
+
+// configure sends a configuration change through consensus and
+// then waits for it to be applied to the server. It
+// will block until the change is performed or there is an error.
 func (s *EtcdServer) configure(ctx context.Context, cc raftpb.ConfChange) error {
 	ch := s.w.Register(cc.ID)
 	if err := s.node.ProposeConfChange(ctx, cc); err != nil {
@@ -473,7 +603,7 @@ func (s *EtcdServer) configure(ctx context.Context, cc raftpb.ConfChange) error
 			return err
 		}
 		if x != nil {
-			log.Panicf("unexpected return type")
+			log.Panicf("return type should always be error")
 		}
 		return nil
 	case <-ctx.Done():
@@ -517,7 +647,7 @@ func (s *EtcdServer) publish(retryInterval time.Duration) {
 	req := pb.Request{
 		ID:     GenID(),
 		Method: "PUT",
-		Path:   path.Join(memberStoreKey(s.id), attributesSuffix),
+		Path:   MemberAttributesStorePath(s.id),
 		Val:    string(b),
 	}
 
@@ -546,7 +676,10 @@ func getExpirationTime(r *pb.Request) time.Time {
 	return t
 }
 
-func (s *EtcdServer) apply(es []raftpb.Entry, nodes []uint64) uint64 {
+// apply takes entries received from Raft (after it has been committed) and
+// applies them to the current state of the EtcdServer.
+// The given entries should not be empty.
+func (s *EtcdServer) apply(es []raftpb.Entry, confState *raftpb.ConfState) (uint64, bool) {
 	var applied uint64
 	for i := range es {
 		e := es[i]
@@ -558,15 +691,19 @@ func (s *EtcdServer) apply(es []raftpb.Entry, nodes []uint64) uint64 {
 		case raftpb.EntryConfChange:
 			var cc raftpb.ConfChange
 			pbutil.MustUnmarshal(&cc, e.Data)
-			s.w.Trigger(cc.ID, s.applyConfChange(cc, nodes))
+			shouldstop, err := s.applyConfChange(cc, confState)
+			s.w.Trigger(cc.ID, err)
+			if shouldstop {
+				return applied, true
+			}
 		default:
-			panic("unexpected entry type")
+			log.Panicf("entry type should be either EntryNormal or EntryConfChange")
 		}
 		atomic.StoreUint64(&s.raftIndex, e.Index)
 		atomic.StoreUint64(&s.raftTerm, e.Term)
 		applied = e.Index
 	}
-	return applied
+	return applied, false
 }
 
 // applyRequest interprets r as a call to store.X and returns a Response interpreted
@@ -592,13 +729,11 @@ func (s *EtcdServer) applyRequest(r pb.Request) Response {
 		default:
 			if storeMemberAttributeRegexp.MatchString(r.Path) {
 				id := mustParseMemberIDFromKey(path.Dir(r.Path))
-				m := s.Cluster.Member(id)
-				if m == nil {
-					log.Fatalf("fetch member %s should never fail", id)
-				}
-				if err := json.Unmarshal([]byte(r.Val), &m.Attributes); err != nil {
-					log.Fatalf("unmarshal %s should never fail: %v", r.Val, err)
+				var attr Attributes
+				if err := json.Unmarshal([]byte(r.Val), &attr); err != nil {
+					log.Panicf("unmarshal %s should never fail: %v", r.Val, err)
 				}
+				s.Cluster.UpdateMemberAttributes(id, attr)
 			}
 			return f(s.store.Set(r.Path, r.Dir, r.Val, expr))
 		}
@@ -620,80 +755,168 @@ func (s *EtcdServer) applyRequest(r pb.Request) Response {
 	}
 }
 
-func (s *EtcdServer) applyConfChange(cc raftpb.ConfChange, nodes []uint64) error {
-	if err := s.checkConfChange(cc, nodes); err != nil {
+// applyConfChange applies a ConfChange to the server. It is only
+// invoked with a ConfChange that has already passed through Raft
+func (s *EtcdServer) applyConfChange(cc raftpb.ConfChange, confState *raftpb.ConfState) (bool, error) {
+	if err := s.Cluster.ValidateConfigurationChange(cc); err != nil {
 		cc.NodeID = raft.None
 		s.node.ApplyConfChange(cc)
-		return err
+		return false, err
 	}
-	s.node.ApplyConfChange(cc)
+	*confState = *s.node.ApplyConfChange(cc)
 	switch cc.Type {
 	case raftpb.ConfChangeAddNode:
 		m := new(Member)
 		if err := json.Unmarshal(cc.Context, m); err != nil {
-			panic("unexpected unmarshal error")
+			log.Panicf("unmarshal member should never fail: %v", err)
 		}
 		if cc.NodeID != uint64(m.ID) {
-			panic("unexpected nodeID mismatch")
+			log.Panicf("nodeID should always be equal to member ID")
 		}
 		s.Cluster.AddMember(m)
-	case raftpb.ConfChangeRemoveNode:
-		s.Cluster.RemoveMember(types.ID(cc.NodeID))
-	}
-	return nil
-}
-
-func (s *EtcdServer) checkConfChange(cc raftpb.ConfChange, nodes []uint64) error {
-	if s.Cluster.IsIDRemoved(types.ID(cc.NodeID)) {
-		return ErrIDRemoved
-	}
-	switch cc.Type {
-	case raftpb.ConfChangeAddNode:
-		if containsUint64(nodes, cc.NodeID) {
-			return ErrIDExists
+		if m.ID == s.id {
+			log.Printf("etcdserver: added local member %s %v to cluster %s", m.ID, m.PeerURLs, s.Cluster.ID())
+		} else {
+			s.sendhub.Add(m)
+			log.Printf("etcdserver: added member %s %v to cluster %s", m.ID, m.PeerURLs, s.Cluster.ID())
 		}
 	case raftpb.ConfChangeRemoveNode:
-		if !containsUint64(nodes, cc.NodeID) {
-			return ErrIDNotFound
+		id := types.ID(cc.NodeID)
+		s.Cluster.RemoveMember(id)
+		if id == s.id {
+			log.Printf("etcdserver: removed local member %s from cluster %s", id, s.Cluster.ID())
+			log.Println("etcdserver: the data-dir used by this member must be removed so that this host can be re-added with a new member ID")
+			return true, nil
+		} else {
+			s.sendhub.Remove(id)
+			log.Printf("etcdserver: removed member %s from cluster %s", id, s.Cluster.ID())
+		}
+	case raftpb.ConfChangeUpdateNode:
+		m := new(Member)
+		if err := json.Unmarshal(cc.Context, m); err != nil {
+			log.Panicf("unmarshal member should never fail: %v", err)
+		}
+		if cc.NodeID != uint64(m.ID) {
+			log.Panicf("nodeID should always be equal to member ID")
+		}
+		s.Cluster.UpdateMember(m)
+		if m.ID == s.id {
+			log.Printf("etcdserver: update local member %s %v in cluster %s", m.ID, m.PeerURLs, s.Cluster.ID())
+		} else {
+			s.sendhub.Update(m)
+			log.Printf("etcdserver: update member %s %v in cluster %s", m.ID, m.PeerURLs, s.Cluster.ID())
 		}
-	default:
-		panic("unexpected ConfChange type")
 	}
-	return nil
+	return false, nil
 }
 
 // TODO: non-blocking snapshot
-func (s *EtcdServer) snapshot(snapi uint64, snapnodes []uint64) {
+func (s *EtcdServer) snapshot(snapi uint64, confState *raftpb.ConfState) {
 	d, err := s.store.Save()
 	// TODO: current store will never fail to do a snapshot
 	// what should we do if the store might fail?
 	if err != nil {
-		panic("TODO: this is bad, what do we do about it?")
+		log.Panicf("etcdserver: store save should never fail: %v", err)
 	}
-	s.node.Compact(snapi, snapnodes, d)
-	s.storage.Cut()
+	err = s.raftStorage.Compact(snapi, confState, d)
+	if err != nil {
+		// the snapshot was done asynchronously with the progress of raft.
+		// raft might have already got a newer snapshot and called compact.
+		if err == raft.ErrCompacted {
+			return
+		}
+		log.Panicf("etcdserver: unexpected compaction error %v", err)
+	}
+	log.Printf("etcdserver: compacted log at index %d", snapi)
+
+	if err := s.storage.Cut(); err != nil {
+		log.Panicf("etcdserver: rotate wal file should never fail: %v", err)
+	}
+	snap, err := s.raftStorage.Snapshot()
+	if err != nil {
+		log.Panicf("etcdserver: snapshot error: %v", err)
+	}
+	if err := s.storage.SaveSnap(snap); err != nil {
+		log.Fatalf("etcdserver: save snapshot error: %v", err)
+	}
+	log.Printf("etcdserver: saved snapshot at index %d", snap.Metadata.Index)
 }
 
+// for testing
+func (s *EtcdServer) PauseSending() {
+	hub := s.sendhub.(*sendHub)
+	hub.pause()
+}
+
+func (s *EtcdServer) ResumeSending() {
+	hub := s.sendhub.(*sendHub)
+	hub.resume()
+}
+
+// checkClientURLsEmptyFromPeers does its best to get the cluster from peers,
+// and if this succeeds, checks that the member of the given id exists in the
+// cluster, and its ClientURLs is empty.
+func checkClientURLsEmptyFromPeers(cl *Cluster, name string) error {
+	us := getOtherPeerURLs(cl, name)
+	rcl, err := getClusterFromPeers(us, false)
+	if err != nil {
+		return nil
+	}
+	id := cl.MemberByName(name).ID
+	m := rcl.Member(id)
+	if m == nil {
+		return nil
+	}
+	if len(m.ClientURLs) > 0 {
+		return fmt.Errorf("etcdserver: member with id %s has started and registered its client urls", id)
+	}
+	return nil
+}
+
+// GetClusterFromPeers takes a set of URLs representing etcd peers, and
+// attempts to construct a Cluster by accessing the members endpoint on one of
+// these URLs. The first URL to provide a response is used. If no URLs provide
+// a response, or a Cluster cannot be successfully created from a received
+// response, an error is returned.
 func GetClusterFromPeers(urls []string) (*Cluster, error) {
+	return getClusterFromPeers(urls, true)
+}
+
+// If logerr is true, it prints out more error messages.
+func getClusterFromPeers(urls []string, logerr bool) (*Cluster, error) {
+	cc := &http.Client{
+		Transport: &http.Transport{
+			ResponseHeaderTimeout: 500 * time.Millisecond,
+		},
+		Timeout: time.Second,
+	}
 	for _, u := range urls {
-		resp, err := http.Get(u + "/members")
+		resp, err := cc.Get(u + "/members")
 		if err != nil {
-			log.Printf("etcdserver: get /members on %s: %v", u, err)
+			if logerr {
+				log.Printf("etcdserver: could not get cluster response from %s: %v", u, err)
+			}
 			continue
 		}
 		b, err := ioutil.ReadAll(resp.Body)
 		if err != nil {
-			log.Printf("etcdserver: read body error: %v", err)
+			if logerr {
+				log.Printf("etcdserver: could not read the body of cluster response: %v", err)
+			}
 			continue
 		}
 		var membs []*Member
 		if err := json.Unmarshal(b, &membs); err != nil {
-			log.Printf("etcdserver: unmarshal body error: %v", err)
+			if logerr {
+				log.Printf("etcdserver: could not unmarshal cluster response: %v", err)
+			}
 			continue
 		}
 		id, err := types.IDFromString(resp.Header.Get("X-Etcd-Cluster-ID"))
 		if err != nil {
-			log.Printf("etcdserver: parse uint error: %v", err)
+			if logerr {
+				log.Printf("etcdserver: could not parse the cluster ID from cluster res: %v", err)
+			}
 			continue
 		}
 		return NewClusterFromMembers("", id, membs), nil
@@ -701,10 +924,8 @@ func GetClusterFromPeers(urls []string) (*Cluster, error) {
 	return nil, fmt.Errorf("etcdserver: could not retrieve cluster information from the given urls")
 }
 
-func startNode(cfg *ServerConfig, ids []types.ID) (id types.ID, n raft.Node, w *wal.WAL) {
+func startNode(cfg *ServerConfig, ids []types.ID) (id types.ID, n raft.Node, s *raft.MemoryStorage, w *wal.WAL) {
 	var err error
-	// TODO: remove the discoveryURL when it becomes part of the source for
-	// generating nodeID.
 	member := cfg.Cluster.MemberByName(cfg.Name)
 	metadata := pbutil.MustMarshal(
 		&pb.Metadata{
@@ -712,40 +933,79 @@ func startNode(cfg *ServerConfig, ids []types.ID) (id types.ID, n raft.Node, w *
 			ClusterID: uint64(cfg.Cluster.ID()),
 		},
 	)
+	if err := os.MkdirAll(cfg.SnapDir(), privateDirMode); err != nil {
+		log.Fatalf("etcdserver create snapshot directory error: %v", err)
+	}
 	if w, err = wal.Create(cfg.WALDir(), metadata); err != nil {
-		log.Fatal(err)
+		log.Fatalf("etcdserver: create wal error: %v", err)
 	}
 	peers := make([]raft.Peer, len(ids))
 	for i, id := range ids {
 		ctx, err := json.Marshal((*cfg.Cluster).Member(id))
 		if err != nil {
-			log.Fatal(err)
+			log.Panicf("marshal member should never fail: %v", err)
 		}
 		peers[i] = raft.Peer{ID: uint64(id), Context: ctx}
 	}
 	id = member.ID
-	log.Printf("etcdserver: start node %s in cluster %s", id, cfg.Cluster.ID())
-	n = raft.StartNode(uint64(id), peers, 10, 1)
+	log.Printf("etcdserver: start member %s in cluster %s", id, cfg.Cluster.ID())
+	s = raft.NewMemoryStorage()
+	n = raft.StartNode(uint64(id), peers, 10, 1, s)
 	return
 }
 
-func restartNode(cfg *ServerConfig, index uint64, snapshot *raftpb.Snapshot) (id types.ID, n raft.Node, w *wal.WAL) {
-	var err error
-	// restart a node from previous wal
-	if w, err = wal.OpenAtIndex(cfg.WALDir(), index); err != nil {
-		log.Fatal(err)
-	}
-	wmetadata, st, ents, err := w.ReadAll()
-	if err != nil {
-		log.Fatal(err)
+func getOtherMembers(cl ClusterInfo, self string) []*Member {
+	var ms []*Member
+	for _, m := range cl.Members() {
+		if m.Name != self {
+			ms = append(ms, m)
+		}
 	}
+	return ms
+}
 
+// getOtherPeerURLs returns peer urls of other members in the cluster. The
+// returned list is sorted in ascending lexicographical order.
+func getOtherPeerURLs(cl ClusterInfo, self string) []string {
+	us := make([]string, 0)
+	for _, m := range cl.Members() {
+		if m.Name == self {
+			continue
+		}
+		us = append(us, m.PeerURLs...)
+	}
+	sort.Strings(us)
+	return us
+}
+
+func restartNode(cfg *ServerConfig, index uint64, snapshot *raftpb.Snapshot) (types.ID, raft.Node, *raft.MemoryStorage, *wal.WAL) {
+	w, id, cid, st, ents := readWAL(cfg.WALDir(), index)
+	cfg.Cluster.SetID(cid)
+
+	log.Printf("etcdserver: restart member %s in cluster %s at commit index %d", id, cfg.Cluster.ID(), st.Commit)
+	s := raft.NewMemoryStorage()
+	if snapshot != nil {
+		s.ApplySnapshot(*snapshot)
+	}
+	s.SetHardState(st)
+	s.Append(ents)
+	n := raft.RestartNode(uint64(id), 10, 1, s)
+	return id, n, s, w
+}
+
+func readWAL(waldir string, index uint64) (w *wal.WAL, id, cid types.ID, st raftpb.HardState, ents []raftpb.Entry) {
+	var err error
+	if w, err = wal.Open(waldir, index); err != nil {
+		log.Fatalf("etcdserver: open wal error: %v", err)
+	}
+	var wmetadata []byte
+	if wmetadata, st, ents, err = w.ReadAll(); err != nil {
+		log.Fatalf("etcdserver: read wal error: %v", err)
+	}
 	var metadata pb.Metadata
 	pbutil.MustUnmarshal(&metadata, wmetadata)
 	id = types.ID(metadata.NodeID)
-	cfg.Cluster.SetID(types.ID(metadata.ClusterID))
-	log.Printf("etcdserver: restart member %s in cluster %s at commit index %d", id, cfg.Cluster.ID(), st.Commit)
-	n = raft.RestartNode(uint64(id), 10, 1, snapshot, st, ents)
+	cid = types.ID(metadata.ClusterID)
 	return
 }
 

etcdserver/server_test.go

@@ -19,6 +19,8 @@ package etcdserver
 import (
 	"encoding/json"
 	"fmt"
+	"io/ioutil"
+	"log"
 	"math/rand"
 	"path"
 	"reflect"
@@ -27,15 +29,21 @@ import (
 	"testing"
 	"time"
 
-	"github.com/coreos/etcd/Godeps/_workspace/src/code.google.com/p/go.net/context"
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 	pb "github.com/coreos/etcd/etcdserver/etcdserverpb"
+	"github.com/coreos/etcd/pkg/pbutil"
 	"github.com/coreos/etcd/pkg/testutil"
 	"github.com/coreos/etcd/pkg/types"
 	"github.com/coreos/etcd/raft"
 	"github.com/coreos/etcd/raft/raftpb"
+	"github.com/coreos/etcd/rafthttp"
 	"github.com/coreos/etcd/store"
 )
 
+func init() {
+	log.SetOutput(ioutil.Discard)
+}
+
 func TestGetExpirationTime(t *testing.T) {
 	tests := []struct {
 		r    pb.Request
@@ -87,6 +95,16 @@ func TestDoLocalAction(t *testing.T) {
 				},
 			},
 		},
+		{
+			pb.Request{Method: "HEAD", ID: 1},
+			Response{Event: &store.Event{}}, nil,
+			[]action{
+				action{
+					name:   "Get",
+					params: []interface{}{"", false, false},
+				},
+			},
+		},
 		{
 			pb.Request{Method: "BADMETHOD", ID: 1},
 			Response{}, ErrUnknownMethod, []action{},
@@ -127,6 +145,10 @@ func TestDoBadLocalAction(t *testing.T) {
 			pb.Request{Method: "GET", ID: 1},
 			[]action{action{name: "Get"}},
 		},
+		{
+			pb.Request{Method: "HEAD", ID: 1},
+			[]action{action{name: "Get"}},
+		},
 	}
 	for i, tt := range tests {
 		st := &errStoreRecorder{err: storeErr}
@@ -387,7 +409,7 @@ func TestApplyRequest(t *testing.T) {
 }
 
 func TestApplyRequestOnAdminMemberAttributes(t *testing.T) {
-	cl := newTestCluster([]Member{{ID: 1}})
+	cl := newTestCluster([]*Member{{ID: 1}})
 	srv := &EtcdServer{
 		store:   &storeRecorder{},
 		Cluster: cl,
@@ -407,8 +429,13 @@ func TestApplyRequestOnAdminMemberAttributes(t *testing.T) {
 
 // TODO: test ErrIDRemoved
 func TestApplyConfChangeError(t *testing.T) {
-	nodes := []uint64{1, 2, 3}
-	removed := map[types.ID]bool{4: true}
+	cl := newCluster("")
+	cl.SetStore(store.New())
+	for i := 1; i <= 4; i++ {
+		cl.AddMember(&Member{ID: types.ID(i)})
+	}
+	cl.RemoveMember(4)
+
 	tests := []struct {
 		cc   raftpb.ConfChange
 		werr error
@@ -422,7 +449,7 @@ func TestApplyConfChangeError(t *testing.T) {
 		},
 		{
 			raftpb.ConfChange{
-				Type:   raftpb.ConfChangeRemoveNode,
+				Type:   raftpb.ConfChangeUpdateNode,
 				NodeID: 4,
 			},
 			ErrIDRemoved,
@@ -444,12 +471,11 @@ func TestApplyConfChangeError(t *testing.T) {
 	}
 	for i, tt := range tests {
 		n := &nodeRecorder{}
-		cl := &Cluster{removed: removed}
 		srv := &EtcdServer{
 			node:    n,
 			Cluster: cl,
 		}
-		err := srv.applyConfChange(tt.cc, nodes)
+		_, err := srv.applyConfChange(tt.cc, nil)
 		if err != tt.werr {
 			t.Errorf("#%d: applyConfChange error = %v, want %v", i, err, tt.werr)
 		}
@@ -466,22 +492,67 @@ func TestApplyConfChangeError(t *testing.T) {
 	}
 }
 
+func TestApplyConfChangeShouldStop(t *testing.T) {
+	cl := newCluster("")
+	cl.SetStore(store.New())
+	for i := 1; i <= 3; i++ {
+		cl.AddMember(&Member{ID: types.ID(i)})
+	}
+	srv := &EtcdServer{
+		id:      1,
+		node:    &nodeRecorder{},
+		Cluster: cl,
+		sendhub: &nopSender{},
+	}
+	cc := raftpb.ConfChange{
+		Type:   raftpb.ConfChangeRemoveNode,
+		NodeID: 2,
+	}
+	// remove non-local member
+	shouldStop, err := srv.applyConfChange(cc, &raftpb.ConfState{})
+	if err != nil {
+		t.Fatalf("unexpected error %v", err)
+	}
+	if shouldStop != false {
+		t.Errorf("shouldStop = %t, want %t", shouldStop, false)
+	}
+
+	// remove local member
+	cc.NodeID = 1
+	shouldStop, err = srv.applyConfChange(cc, &raftpb.ConfState{})
+	if err != nil {
+		t.Fatalf("unexpected error %v", err)
+	}
+	if shouldStop != true {
+		t.Errorf("shouldStop = %t, want %t", shouldStop, true)
+	}
+}
+
 func TestClusterOf1(t *testing.T) { testServer(t, 1) }
 func TestClusterOf3(t *testing.T) { testServer(t, 3) }
 
+type fakeSender struct {
+	ss []*EtcdServer
+}
+
+func (s *fakeSender) Sender(id types.ID) rafthttp.Sender { return nil }
+func (s *fakeSender) Send(msgs []raftpb.Message) {
+	for _, m := range msgs {
+		s.ss[m.To-1].node.Step(context.TODO(), m)
+	}
+}
+func (s *fakeSender) Add(m *Member)                     {}
+func (s *fakeSender) Update(m *Member)                  {}
+func (s *fakeSender) Remove(id types.ID)                {}
+func (s *fakeSender) Stop()                             {}
+func (s *fakeSender) ShouldStopNotify() <-chan struct{} { return nil }
+
 func testServer(t *testing.T, ns uint64) {
 	ctx, cancel := context.WithCancel(context.Background())
 	defer cancel()
 
 	ss := make([]*EtcdServer, ns)
 
-	send := func(msgs []raftpb.Message) {
-		for _, m := range msgs {
-			t.Logf("m = %+v\n", m)
-			ss[m.To-1].node.Step(ctx, m)
-		}
-	}
-
 	ids := make([]uint64, ns)
 	for i := uint64(0); i < ns; i++ {
 		ids[i] = i + 1
@@ -489,23 +560,30 @@ func testServer(t *testing.T, ns uint64) {
 	members := mustMakePeerSlice(t, ids...)
 	for i := uint64(0); i < ns; i++ {
 		id := i + 1
-		n := raft.StartNode(id, members, 10, 1)
+		s := raft.NewMemoryStorage()
+		n := raft.StartNode(id, members, 10, 1, s)
 		tk := time.NewTicker(10 * time.Millisecond)
 		defer tk.Stop()
+		st := store.New()
 		cl := newCluster("abc")
-		cl.SetStore(&storeRecorder{})
+		cl.SetStore(st)
 		srv := &EtcdServer{
-			node:    n,
-			store:   store.New(),
-			send:    send,
-			storage: &storageRecorder{},
-			Ticker:  tk.C,
-			Cluster: cl,
+			node:        n,
+			raftStorage: s,
+			store:       st,
+			sendhub:     &fakeSender{ss},
+			storage:     &storageRecorder{},
+			Ticker:      tk.C,
+			Cluster:     cl,
 		}
-		srv.start()
 		ss[i] = srv
 	}
 
+	// Start the servers after they're all created to avoid races in send().
+	for i := uint64(0); i < ns; i++ {
+		ss[i].start()
+	}
+
 	for i := 1; i <= 10; i++ {
 		r := pb.Request{
 			Method: "PUT",
@@ -522,8 +600,8 @@ func testServer(t *testing.T, ns uint64) {
 
 		g, w := resp.Event.Node, &store.NodeExtern{
 			Key:           "/foo",
-			ModifiedIndex: uint64(i),
-			CreatedIndex:  uint64(i),
+			ModifiedIndex: uint64(i) + ns,
+			CreatedIndex:  uint64(i) + ns,
 			Value:         stringp("bar"),
 		}
 
@@ -556,20 +634,22 @@ func TestDoProposal(t *testing.T) {
 
 	for i, tt := range tests {
 		ctx, _ := context.WithCancel(context.Background())
-		n := raft.StartNode(0xBAD0, mustMakePeerSlice(t, 0xBAD0), 10, 1)
+		s := raft.NewMemoryStorage()
+		n := raft.StartNode(0xBAD0, mustMakePeerSlice(t, 0xBAD0), 10, 1, s)
 		st := &storeRecorder{}
 		tk := make(chan time.Time)
 		// this makes <-tk always successful, which accelerates internal clock
 		close(tk)
 		cl := newCluster("abc")
-		cl.SetStore(&storeRecorder{})
+		cl.SetStore(store.New())
 		srv := &EtcdServer{
-			node:    n,
-			store:   st,
-			send:    func(_ []raftpb.Message) {},
-			storage: &storageRecorder{},
-			Ticker:  tk,
-			Cluster: cl,
+			node:        n,
+			raftStorage: s,
+			store:       st,
+			sendhub:     &nopSender{},
+			storage:     &storageRecorder{},
+			Ticker:      tk,
+			Cluster:     cl,
 		}
 		srv.start()
 		resp, err := srv.Do(ctx, tt)
@@ -592,14 +672,16 @@ func TestDoProposal(t *testing.T) {
 func TestDoProposalCancelled(t *testing.T) {
 	ctx, cancel := context.WithCancel(context.Background())
 	// node cannot make any progress because there are two nodes
-	n := raft.StartNode(0xBAD0, mustMakePeerSlice(t, 0xBAD0, 0xBAD1), 10, 1)
+	s := raft.NewMemoryStorage()
+	n := raft.StartNode(0xBAD0, mustMakePeerSlice(t, 0xBAD0, 0xBAD1), 10, 1, s)
 	st := &storeRecorder{}
 	wait := &waitRecorder{}
 	srv := &EtcdServer{
 		// TODO: use fake node for better testability
-		node:  n,
-		store: st,
-		w:     wait,
+		node:        n,
+		raftStorage: s,
+		store:       st,
+		w:           wait,
 	}
 
 	done := make(chan struct{})
@@ -640,18 +722,23 @@ func TestDoProposalStopped(t *testing.T) {
 	ctx, cancel := context.WithCancel(context.Background())
 	defer cancel()
 	// node cannot make any progress because there are two nodes
-	n := raft.StartNode(0xBAD0, mustMakePeerSlice(t, 0xBAD0, 0xBAD1), 10, 1)
+	s := raft.NewMemoryStorage()
+	n := raft.StartNode(0xBAD0, mustMakePeerSlice(t, 0xBAD0, 0xBAD1), 10, 1, s)
 	st := &storeRecorder{}
 	tk := make(chan time.Time)
 	// this makes <-tk always successful, which accelarates internal clock
 	close(tk)
+	cl := newCluster("abc")
+	cl.SetStore(store.New())
 	srv := &EtcdServer{
 		// TODO: use fake node for better testability
-		node:    n,
-		store:   st,
-		send:    func(_ []raftpb.Message) {},
-		storage: &storageRecorder{},
-		Ticker:  tk,
+		node:        n,
+		raftStorage: s,
+		store:       st,
+		sendhub:     &nopSender{},
+		storage:     &storageRecorder{},
+		Ticker:      tk,
+		Cluster:     cl,
 	}
 	srv.start()
 
@@ -679,12 +766,17 @@ func TestSync(t *testing.T) {
 	srv := &EtcdServer{
 		node: n,
 	}
-	start := time.Now()
-	srv.sync(defaultSyncTimeout)
+	done := make(chan struct{})
+	go func() {
+		srv.sync(10 * time.Second)
+		close(done)
+	}()
 
 	// check that sync is non-blocking
-	if d := time.Since(start); d > time.Millisecond {
-		t.Errorf("CallSyncTime = %v, want < %v", d, time.Millisecond)
+	select {
+	case <-done:
+	case <-time.After(time.Second):
+		t.Fatalf("sync should be non-blocking but did not return after 1s!")
 	}
 
 	testutil.ForceGosched()
@@ -708,12 +800,17 @@ func TestSyncTimeout(t *testing.T) {
 	srv := &EtcdServer{
 		node: n,
 	}
-	start := time.Now()
-	srv.sync(0)
+	done := make(chan struct{})
+	go func() {
+		srv.sync(0)
+		close(done)
+	}()
 
 	// check that sync is non-blocking
-	if d := time.Since(start); d > time.Millisecond {
-		t.Errorf("CallSyncTime = %v, want < %v", d, time.Millisecond)
+	select {
+	case <-done:
+	case <-time.After(time.Second):
+		t.Fatalf("sync should be non-blocking but did not return after 1s!")
 	}
 
 	// give time for goroutine in sync to cancel
@@ -747,11 +844,12 @@ func TestSyncTrigger(t *testing.T) {
 	}
 	st := make(chan time.Time, 1)
 	srv := &EtcdServer{
-		node:       n,
-		store:      &storeRecorder{},
-		send:       func(_ []raftpb.Message) {},
-		storage:    &storageRecorder{},
-		SyncTicker: st,
+		node:        n,
+		raftStorage: raft.NewMemoryStorage(),
+		store:       &storeRecorder{},
+		sendhub:     &nopSender{},
+		storage:     &storageRecorder{},
+		SyncTicker:  st,
 	}
 	srv.start()
 	// trigger the server to become a leader and accept sync requests
@@ -781,17 +879,45 @@ func TestSyncTrigger(t *testing.T) {
 // snapshot should snapshot the store and cut the persistent
 // TODO: node.Compact is called... we need to make the node an interface
 func TestSnapshot(t *testing.T) {
-	n := raft.StartNode(0xBAD0, mustMakePeerSlice(t, 0xBAD0), 10, 1)
+	s := raft.NewMemoryStorage()
+	n := raft.StartNode(0xBAD0, mustMakePeerSlice(t, 0xBAD0), 10, 1, s)
 	defer n.Stop()
+
+	// Progress the node to the point where it has something to snapshot.
+	// TODO(bdarnell): this could be improved with changes in the raft internals.
+	// First, we must apply the initial conf changes so we can have an election.
+	rd := <-n.Ready()
+	s.Append(rd.Entries)
+	for _, e := range rd.CommittedEntries {
+		if e.Type == raftpb.EntryConfChange {
+			var cc raftpb.ConfChange
+			err := cc.Unmarshal(e.Data)
+			if err != nil {
+				t.Fatal(err)
+			}
+			n.ApplyConfChange(cc)
+		}
+	}
+	n.Advance()
+
+	// Now we can have an election and persist the rest of the log.
+	// This causes HardState.Commit to advance. HardState.Commit must
+	// be > 0 to snapshot.
+	n.Campaign(context.Background())
+	rd = <-n.Ready()
+	s.Append(rd.Entries)
+	n.Advance()
+
 	st := &storeRecorder{}
 	p := &storageRecorder{}
-	s := &EtcdServer{
-		store:   st,
-		storage: p,
-		node:    n,
+	srv := &EtcdServer{
+		store:       st,
+		storage:     p,
+		node:        n,
+		raftStorage: s,
 	}
 
-	s.snapshot(0, []uint64{1})
+	srv.snapshot(1, &raftpb.ConfState{Nodes: []uint64{1}})
 	gaction := st.Action()
 	if len(gaction) != 1 {
 		t.Fatalf("len(action) = %d, want 1", len(gaction))
@@ -801,43 +927,51 @@ func TestSnapshot(t *testing.T) {
 	}
 
 	gaction = p.Action()
-	if len(gaction) != 1 {
-		t.Fatalf("len(action) = %d, want 1", len(gaction))
+	if len(gaction) != 2 {
+		t.Fatalf("len(action) = %d, want 2", len(gaction))
 	}
 	if !reflect.DeepEqual(gaction[0], action{name: "Cut"}) {
 		t.Errorf("action = %s, want Cut", gaction[0])
 	}
+	if !reflect.DeepEqual(gaction[1], action{name: "SaveSnap"}) {
+		t.Errorf("action = %s, want SaveSnap", gaction[1])
+	}
 }
 
 // Applied > SnapCount should trigger a SaveSnap event
 func TestTriggerSnap(t *testing.T) {
 	ctx := context.Background()
-	n := raft.StartNode(0xBAD0, mustMakePeerSlice(t, 0xBAD0), 10, 1)
-	<-n.Ready()
-	n.ApplyConfChange(raftpb.ConfChange{Type: raftpb.ConfChangeAddNode, NodeID: 0xBAD0})
+	s := raft.NewMemoryStorage()
+	n := raft.StartNode(0xBAD0, mustMakePeerSlice(t, 0xBAD0), 10, 1, s)
 	n.Campaign(ctx)
 	st := &storeRecorder{}
 	p := &storageRecorder{}
-	s := &EtcdServer{
-		store:     st,
-		send:      func(_ []raftpb.Message) {},
-		storage:   p,
-		node:      n,
-		snapCount: 10,
-		Cluster:   &Cluster{},
+	cl := newCluster("abc")
+	cl.SetStore(store.New())
+	srv := &EtcdServer{
+		store:       st,
+		sendhub:     &nopSender{},
+		storage:     p,
+		node:        n,
+		raftStorage: s,
+		snapCount:   10,
+		Cluster:     cl,
 	}
 
-	s.start()
-	for i := 0; uint64(i) < s.snapCount-1; i++ {
-		s.Do(ctx, pb.Request{Method: "PUT", ID: 1})
-	}
+	srv.start()
+	// wait for saving nop
 	time.Sleep(time.Millisecond)
-	s.Stop()
+	for i := 0; uint64(i) < srv.snapCount-1; i++ {
+		srv.Do(ctx, pb.Request{Method: "PUT", ID: 1})
+	}
+	// wait for saving the last entry
+	time.Sleep(time.Millisecond)
+	srv.Stop()
 
 	gaction := p.Action()
 	// each operation is recorded as a Save
 	// BootstrapConfig/Nop + (SnapCount - 1) * Puts + Cut + SaveSnap = Save + (SnapCount - 1) * Save + Cut + SaveSnap
-	wcnt := 2 + int(s.snapCount)
+	wcnt := 2 + int(srv.snapCount)
 	if len(gaction) != wcnt {
 		t.Fatalf("len(action) = %d, want %d", len(gaction), wcnt)
 	}
@@ -852,15 +986,19 @@ func TestRecvSnapshot(t *testing.T) {
 	n := newReadyNode()
 	st := &storeRecorder{}
 	p := &storageRecorder{}
+	cl := newCluster("abc")
+	cl.SetStore(store.New())
 	s := &EtcdServer{
-		store:   st,
-		send:    func(_ []raftpb.Message) {},
-		storage: p,
-		node:    n,
+		store:       st,
+		sendhub:     &nopSender{},
+		storage:     p,
+		node:        n,
+		raftStorage: raft.NewMemoryStorage(),
+		Cluster:     cl,
 	}
 
 	s.start()
-	n.readyc <- raft.Ready{Snapshot: raftpb.Snapshot{Index: 1}}
+	n.readyc <- raft.Ready{Snapshot: raftpb.Snapshot{Metadata: raftpb.SnapshotMetadata{Index: 1}}}
 	// make goroutines move forward to receive snapshot
 	testutil.ForceGosched()
 	s.Stop()
@@ -869,31 +1007,36 @@ func TestRecvSnapshot(t *testing.T) {
 	if g := st.Action(); !reflect.DeepEqual(g, wactions) {
 		t.Errorf("store action = %v, want %v", g, wactions)
 	}
-	wactions = []action{action{name: "Save"}, action{name: "SaveSnap"}}
+	wactions = []action{action{name: "SaveSnap"}, action{name: "Save"}}
 	if g := p.Action(); !reflect.DeepEqual(g, wactions) {
 		t.Errorf("storage action = %v, want %v", g, wactions)
 	}
 }
 
 // TestRecvSlowSnapshot tests that slow snapshot will not be applied
-// to store.
+// to store. The case could happen when server compacts the log and
+// raft returns the compacted snapshot.
 func TestRecvSlowSnapshot(t *testing.T) {
 	n := newReadyNode()
 	st := &storeRecorder{}
+	cl := newCluster("abc")
+	cl.SetStore(store.New())
 	s := &EtcdServer{
-		store:   st,
-		send:    func(_ []raftpb.Message) {},
-		storage: &storageRecorder{},
-		node:    n,
+		store:       st,
+		sendhub:     &nopSender{},
+		storage:     &storageRecorder{},
+		node:        n,
+		raftStorage: raft.NewMemoryStorage(),
+		Cluster:     cl,
 	}
 
 	s.start()
-	n.readyc <- raft.Ready{Snapshot: raftpb.Snapshot{Index: 1}}
+	n.readyc <- raft.Ready{Snapshot: raftpb.Snapshot{Metadata: raftpb.SnapshotMetadata{Index: 1}}}
 	// make goroutines move forward to receive snapshot
 	testutil.ForceGosched()
 	action := st.Action()
 
-	n.readyc <- raft.Ready{Snapshot: raftpb.Snapshot{Index: 1}}
+	n.readyc <- raft.Ready{Snapshot: raftpb.Snapshot{Metadata: raftpb.SnapshotMetadata{Index: 1}}}
 	// make goroutines move forward to receive snapshot
 	testutil.ForceGosched()
 	s.Stop()
@@ -903,23 +1046,62 @@ func TestRecvSlowSnapshot(t *testing.T) {
 	}
 }
 
+// TestApplySnapshotAndCommittedEntries tests that server applies snapshot
+// first and then committed entries.
+func TestApplySnapshotAndCommittedEntries(t *testing.T) {
+	n := newReadyNode()
+	st := &storeRecorder{}
+	cl := newCluster("abc")
+	cl.SetStore(store.New())
+	storage := raft.NewMemoryStorage()
+	s := &EtcdServer{
+		store:       st,
+		sendhub:     &nopSender{},
+		storage:     &storageRecorder{},
+		node:        n,
+		raftStorage: storage,
+		Cluster:     cl,
+	}
+
+	s.start()
+	req := &pb.Request{Method: "QGET"}
+	n.readyc <- raft.Ready{
+		Snapshot: raftpb.Snapshot{Metadata: raftpb.SnapshotMetadata{Index: 1}},
+		CommittedEntries: []raftpb.Entry{
+			{Index: 2, Data: pbutil.MustMarshal(req)},
+		},
+	}
+	// make goroutines move forward to receive snapshot
+	testutil.ForceGosched()
+	s.Stop()
+
+	actions := st.Action()
+	if len(actions) != 2 {
+		t.Fatalf("len(action) = %d, want 2", len(actions))
+	}
+	if actions[0].name != "Recovery" {
+		t.Errorf("actions[0] = %s, want %s", actions[0].name, "Recovery")
+	}
+	if actions[1].name != "Get" {
+		t.Errorf("actions[1] = %s, want %s", actions[1].name, "Get")
+	}
+}
+
 // TestAddMember tests AddMember can propose and perform node addition.
 func TestAddMember(t *testing.T) {
 	n := newNodeConfChangeCommitterRecorder()
 	n.readyc <- raft.Ready{
-		SoftState: &raft.SoftState{
-			RaftState: raft.StateLeader,
-			Nodes:     []uint64{2345, 3456},
-		},
+		SoftState: &raft.SoftState{RaftState: raft.StateLeader},
 	}
 	cl := newTestCluster(nil)
-	cl.SetStore(&storeRecorder{})
+	cl.SetStore(store.New())
 	s := &EtcdServer{
-		node:    n,
-		store:   &storeRecorder{},
-		send:    func(_ []raftpb.Message) {},
-		storage: &storageRecorder{},
-		Cluster: cl,
+		node:        n,
+		raftStorage: raft.NewMemoryStorage(),
+		store:       &storeRecorder{},
+		sendhub:     &nopSender{},
+		storage:     &storageRecorder{},
+		Cluster:     cl,
 	}
 	s.start()
 	m := Member{ID: 1234, RaftAttributes: RaftAttributes{PeerURLs: []string{"foo"}}}
@@ -943,19 +1125,18 @@ func TestAddMember(t *testing.T) {
 func TestRemoveMember(t *testing.T) {
 	n := newNodeConfChangeCommitterRecorder()
 	n.readyc <- raft.Ready{
-		SoftState: &raft.SoftState{
-			RaftState: raft.StateLeader,
-			Nodes:     []uint64{1234, 2345, 3456},
-		},
+		SoftState: &raft.SoftState{RaftState: raft.StateLeader},
 	}
-	cl := newTestCluster([]Member{{ID: 1234}})
-	cl.SetStore(&storeRecorder{})
+	cl := newTestCluster(nil)
+	cl.SetStore(store.New())
+	cl.AddMember(&Member{ID: 1234})
 	s := &EtcdServer{
-		node:    n,
-		store:   &storeRecorder{},
-		send:    func(_ []raftpb.Message) {},
-		storage: &storageRecorder{},
-		Cluster: cl,
+		node:        n,
+		raftStorage: raft.NewMemoryStorage(),
+		store:       &storeRecorder{},
+		sendhub:     &nopSender{},
+		storage:     &storageRecorder{},
+		Cluster:     cl,
 	}
 	s.start()
 	err := s.RemoveMember(context.TODO(), 1234)
@@ -974,6 +1155,41 @@ func TestRemoveMember(t *testing.T) {
 	}
 }
 
+// TestUpdateMember tests RemoveMember can propose and perform node update.
+func TestUpdateMember(t *testing.T) {
+	n := newNodeConfChangeCommitterRecorder()
+	n.readyc <- raft.Ready{
+		SoftState: &raft.SoftState{RaftState: raft.StateLeader},
+	}
+	cl := newTestCluster(nil)
+	cl.SetStore(store.New())
+	cl.AddMember(&Member{ID: 1234})
+	s := &EtcdServer{
+		node:        n,
+		raftStorage: raft.NewMemoryStorage(),
+		store:       &storeRecorder{},
+		sendhub:     &nopSender{},
+		storage:     &storageRecorder{},
+		Cluster:     cl,
+	}
+	s.start()
+	wm := Member{ID: 1234, RaftAttributes: RaftAttributes{PeerURLs: []string{"http://127.0.0.1:1"}}}
+	err := s.UpdateMember(context.TODO(), wm)
+	gaction := n.Action()
+	s.Stop()
+
+	if err != nil {
+		t.Fatalf("UpdateMember error: %v", err)
+	}
+	wactions := []action{action{name: "ProposeConfChange:ConfChangeUpdateNode"}, action{name: "ApplyConfChange:ConfChangeUpdateNode"}}
+	if !reflect.DeepEqual(gaction, wactions) {
+		t.Errorf("action = %v, want %v", gaction, wactions)
+	}
+	if !reflect.DeepEqual(cl.Member(1234), &wm) {
+		t.Errorf("member = %v, want %v", cl.Member(1234), &wm)
+	}
+}
+
 // TODO: test server could stop itself when being removed
 
 // TODO: test wait trigger correctness in multi-server case
@@ -1021,13 +1237,13 @@ func TestPublish(t *testing.T) {
 func TestPublishStopped(t *testing.T) {
 	srv := &EtcdServer{
 		node:    &nodeRecorder{},
+		sendhub: &nopSender{},
 		Cluster: &Cluster{},
 		w:       &waitRecorder{},
 		done:    make(chan struct{}),
-		stopped: make(chan struct{}),
+		stop:    make(chan struct{}),
 	}
-	close(srv.stopped)
-	srv.Stop()
+	close(srv.done)
 	srv.publish(time.Hour)
 }
 
@@ -1039,7 +1255,7 @@ func TestPublishRetry(t *testing.T) {
 		w:    &waitRecorder{},
 		done: make(chan struct{}),
 	}
-	time.AfterFunc(500*time.Microsecond, srv.Stop)
+	time.AfterFunc(500*time.Microsecond, func() { close(srv.done) })
 	srv.publish(10 * time.Nanosecond)
 
 	action := n.Action()
@@ -1049,6 +1265,71 @@ func TestPublishRetry(t *testing.T) {
 	}
 }
 
+func TestStopNotify(t *testing.T) {
+	s := &EtcdServer{
+		stop: make(chan struct{}),
+		done: make(chan struct{}),
+	}
+	go func() {
+		<-s.stop
+		close(s.done)
+	}()
+
+	notifier := s.StopNotify()
+	select {
+	case <-notifier:
+		t.Fatalf("received unexpected stop notification")
+	default:
+	}
+	s.Stop()
+	select {
+	case <-notifier:
+	default:
+		t.Fatalf("cannot receive stop notification")
+	}
+}
+
+func TestGetOtherPeerURLs(t *testing.T) {
+	tests := []struct {
+		membs []*Member
+		self  string
+		wurls []string
+	}{
+		{
+			[]*Member{
+				newTestMember(1, []string{"http://10.0.0.1"}, "a", nil),
+			},
+			"a",
+			[]string{},
+		},
+		{
+			[]*Member{
+				newTestMember(1, []string{"http://10.0.0.1"}, "a", nil),
+				newTestMember(2, []string{"http://10.0.0.2"}, "b", nil),
+				newTestMember(3, []string{"http://10.0.0.3"}, "c", nil),
+			},
+			"a",
+			[]string{"http://10.0.0.2", "http://10.0.0.3"},
+		},
+		{
+			[]*Member{
+				newTestMember(1, []string{"http://10.0.0.1"}, "a", nil),
+				newTestMember(3, []string{"http://10.0.0.3"}, "c", nil),
+				newTestMember(2, []string{"http://10.0.0.2"}, "b", nil),
+			},
+			"a",
+			[]string{"http://10.0.0.2", "http://10.0.0.3"},
+		},
+	}
+	for i, tt := range tests {
+		cl := NewClusterFromMembers("", types.ID(0), tt.membs)
+		urls := getOtherPeerURLs(cl, tt.self)
+		if !reflect.DeepEqual(urls, tt.wurls) {
+			t.Errorf("#%d: urls = %+v, want %+v", i, urls, tt.wurls)
+		}
+	}
+}
+
 func TestGetBool(t *testing.T) {
 	tests := []struct {
 		b    *bool
@@ -1222,19 +1503,21 @@ type storageRecorder struct {
 	recorder
 }
 
-func (p *storageRecorder) Save(st raftpb.HardState, ents []raftpb.Entry) {
+func (p *storageRecorder) Save(st raftpb.HardState, ents []raftpb.Entry) error {
 	p.record(action{name: "Save"})
+	return nil
 }
 func (p *storageRecorder) Cut() error {
 	p.record(action{name: "Cut"})
 	return nil
 }
-func (p *storageRecorder) SaveSnap(st raftpb.Snapshot) {
-	if raft.IsEmptySnap(st) {
-		return
+func (p *storageRecorder) SaveSnap(st raftpb.Snapshot) error {
+	if !raft.IsEmptySnap(st) {
+		p.record(action{name: "SaveSnap"})
 	}
-	p.record(action{name: "SaveSnap"})
+	return nil
 }
+func (p *storageRecorder) Close() error { return nil }
 
 type readyNode struct {
 	readyc chan raft.Ready
@@ -1250,19 +1533,19 @@ func (n *readyNode) Propose(ctx context.Context, data []byte) error { return nil
 func (n *readyNode) ProposeConfChange(ctx context.Context, conf raftpb.ConfChange) error {
 	return nil
 }
-func (n *readyNode) Step(ctx context.Context, msg raftpb.Message) error { return nil }
-func (n *readyNode) Ready() <-chan raft.Ready                           { return n.readyc }
-func (n *readyNode) ApplyConfChange(conf raftpb.ConfChange)             {}
-func (n *readyNode) Stop()                                              {}
-func (n *readyNode) Compact(index uint64, nodes []uint64, d []byte)     {}
+func (n *readyNode) Step(ctx context.Context, msg raftpb.Message) error       { return nil }
+func (n *readyNode) Ready() <-chan raft.Ready                                 { return n.readyc }
+func (n *readyNode) Advance()                                                 {}
+func (n *readyNode) ApplyConfChange(conf raftpb.ConfChange) *raftpb.ConfState { return nil }
+func (n *readyNode) Stop()                                                    {}
+func (n *readyNode) Compact(index uint64, nodes []uint64, d []byte)           {}
 
 type nodeRecorder struct {
 	recorder
 }
 
-func (n *nodeRecorder) Tick() {
-	n.record(action{name: "Tick"})
-}
+func (n *nodeRecorder) Tick() { n.record(action{name: "Tick"}) }
+
 func (n *nodeRecorder) Campaign(ctx context.Context) error {
 	n.record(action{name: "Campaign"})
 	return nil
@@ -1280,8 +1563,10 @@ func (n *nodeRecorder) Step(ctx context.Context, msg raftpb.Message) error {
 	return nil
 }
 func (n *nodeRecorder) Ready() <-chan raft.Ready { return nil }
-func (n *nodeRecorder) ApplyConfChange(conf raftpb.ConfChange) {
+func (n *nodeRecorder) Advance()                 {}
+func (n *nodeRecorder) ApplyConfChange(conf raftpb.ConfChange) *raftpb.ConfState {
 	n.record(action{name: "ApplyConfChange", params: []interface{}{conf}})
+	return &raftpb.ConfState{}
 }
 func (n *nodeRecorder) Stop() {
 	n.record(action{name: "Stop"})
@@ -1323,6 +1608,7 @@ func (n *nodeProposalBlockerRecorder) Propose(ctx context.Context, data []byte)
 type nodeConfChangeCommitterRecorder struct {
 	nodeRecorder
 	readyc chan raft.Ready
+	index  uint64
 }
 
 func newNodeConfChangeCommitterRecorder() *nodeConfChangeCommitterRecorder {
@@ -1334,15 +1620,17 @@ func (n *nodeConfChangeCommitterRecorder) ProposeConfChange(ctx context.Context,
 	if err != nil {
 		return err
 	}
-	n.readyc <- raft.Ready{CommittedEntries: []raftpb.Entry{{Type: raftpb.EntryConfChange, Data: data}}}
+	n.index++
+	n.readyc <- raft.Ready{CommittedEntries: []raftpb.Entry{{Index: n.index, Type: raftpb.EntryConfChange, Data: data}}}
 	n.record(action{name: "ProposeConfChange:" + conf.Type.String()})
 	return nil
 }
 func (n *nodeConfChangeCommitterRecorder) Ready() <-chan raft.Ready {
 	return n.readyc
 }
-func (n *nodeConfChangeCommitterRecorder) ApplyConfChange(conf raftpb.ConfChange) {
+func (n *nodeConfChangeCommitterRecorder) ApplyConfChange(conf raftpb.ConfChange) *raftpb.ConfState {
 	n.record(action{name: "ApplyConfChange:" + conf.Type.String()})
+	return &raftpb.ConfState{}
 }
 
 type waitWithResponse struct {
@@ -1354,30 +1642,15 @@ func (w *waitWithResponse) Register(id uint64) <-chan interface{} {
 }
 func (w *waitWithResponse) Trigger(id uint64, x interface{}) {}
 
-type clusterStoreRecorder struct {
-	recorder
-}
+type nopSender struct{}
 
-func (cs *clusterStoreRecorder) Add(m Member) {
-	cs.record(action{name: "Add", params: []interface{}{m}})
-}
-func (cs *clusterStoreRecorder) Get() Cluster {
-	cs.record(action{name: "Get"})
-	return Cluster{}
-}
-func (cs *clusterStoreRecorder) Remove(id uint64) {
-	cs.record(action{name: "Remove", params: []interface{}{id}})
-}
-func (cs *clusterStoreRecorder) IsRemoved(id uint64) bool { return false }
-
-type removedClusterStore struct {
-	removed map[uint64]bool
-}
-
-func (cs *removedClusterStore) Add(m Member)             {}
-func (cs *removedClusterStore) Get() Cluster             { return Cluster{} }
-func (cs *removedClusterStore) Remove(id uint64)         {}
-func (cs *removedClusterStore) IsRemoved(id uint64) bool { return cs.removed[id] }
+func (s *nopSender) Sender(id types.ID) rafthttp.Sender { return nil }
+func (s *nopSender) Send(m []raftpb.Message)            {}
+func (s *nopSender) Add(m *Member)                      {}
+func (s *nopSender) Remove(id types.ID)                 {}
+func (s *nopSender) Update(m *Member)                   {}
+func (s *nopSender) Stop()                              {}
+func (s *nopSender) ShouldStopNotify() <-chan struct{}  { return nil }
 
 func mustMakePeerSlice(t *testing.T, ids ...uint64) []raft.Peer {
 	peers := make([]raft.Peer, len(ids))

etcdserver/stats/server.go

@@ -141,3 +141,11 @@ func (ss *ServerStats) SendAppendReq(reqSize int) {
 
 	ss.SendAppendRequestCnt++
 }
+
+func (ss *ServerStats) BecomeLeader() {
+	if ss.State != raft.StateLeader {
+		ss.State = raft.StateLeader
+		ss.LeaderInfo.Name = ss.ID
+		ss.LeaderInfo.StartTime = time.Now()
+	}
+}

etcdserver/stats/stats.go

@@ -0,0 +1,11 @@
+package stats
+
+type Stats interface {
+	// SelfStats returns the struct representing statistics of this server
+	SelfStats() []byte
+	// LeaderStats returns the statistics of all followers in the cluster
+	// if this server is leader. Otherwise, nil is returned.
+	LeaderStats() []byte
+	// StoreStats returns statistics of the store backing this EtcdServer
+	StoreStats() []byte
+}

etcdserver/storage.go

@@ -0,0 +1,45 @@
+package etcdserver
+
+import (
+	"github.com/coreos/etcd/raft/raftpb"
+	"github.com/coreos/etcd/snap"
+	"github.com/coreos/etcd/wal"
+)
+
+type Storage interface {
+	// Save function saves ents and state to the underlying stable storage.
+	// Save MUST block until st and ents are on stable storage.
+	Save(st raftpb.HardState, ents []raftpb.Entry) error
+	// SaveSnap function saves snapshot to the underlying stable storage.
+	SaveSnap(snap raftpb.Snapshot) error
+
+	// TODO: WAL should be able to control cut itself. After implement self-controlled cut,
+	// remove it in this interface.
+	// Cut cuts out a new wal file for saving new state and entries.
+	Cut() error
+	// Close closes the Storage and performs finalization.
+	Close() error
+}
+
+type storage struct {
+	*wal.WAL
+	*snap.Snapshotter
+}
+
+func NewStorage(w *wal.WAL, s *snap.Snapshotter) Storage {
+	return &storage{w, s}
+}
+
+// SaveSnap saves the snapshot to disk and release the locked
+// wal files since they will not be used.
+func (st *storage) SaveSnap(snap raftpb.Snapshot) error {
+	err := st.Snapshotter.SaveSnap(snap)
+	if err != nil {
+		return err
+	}
+	err = st.WAL.ReleaseLockTo(snap.Metadata.Index)
+	if err != nil {
+		return err
+	}
+	return nil
+}

hack/insta-discovery/Procfile

@@ -1,4 +1,6 @@
 # Use goreman to run `go get github.com/mattn/goreman`
+# One of the four etcd members falls back to a proxy
 etcd1: ../../bin/etcd -name infra1 -listen-client-urls http://127.0.0.1:4001 -advertise-client-urls http://127.0.0.1:4001 -listen-peer-urls http://127.0.0.1:7001 -initial-advertise-peer-urls http://127.0.0.1:7001
 etcd2: ../../bin/etcd -name infra2 -listen-client-urls http://127.0.0.1:4002 -advertise-client-urls http://127.0.0.1:4002 -listen-peer-urls http://127.0.0.1:7002 -initial-advertise-peer-urls http://127.0.0.1:7002
 etcd3: ../../bin/etcd -name infra3 -listen-client-urls http://127.0.0.1:4003 -advertise-client-urls http://127.0.0.1:4003 -listen-peer-urls http://127.0.0.1:7003 -initial-advertise-peer-urls http://127.0.0.1:7003
+etcd4: ../../bin/etcd -name infra4 -listen-client-urls http://127.0.0.1:4004 -advertise-client-urls http://127.0.0.1:4004 -listen-peer-urls http://127.0.0.1:7004 -initial-advertise-peer-urls http://127.0.0.1:7004

hack/insta-discovery/discovery

@@ -1,6 +1,7 @@
 #!/bin/sh
 
-disc=$(curl https://discovery.etcd.io/new?size=3)
+rm -rf infra*.etcd
+disc=$(curl -s https://discovery.etcd.io/new?size=3)
 echo ETCD_DISCOVERY=${disc} > .env
 echo "setup discovery start your cluster"
 cat .env

integration/cluster_test.go

@@ -20,24 +20,33 @@ import (
 	"fmt"
 	"io/ioutil"
 	"log"
+	"math/rand"
 	"net"
 	"net/http"
 	"net/http/httptest"
-	"net/url"
 	"os"
+	"reflect"
+	"sort"
 	"strings"
 	"testing"
 	"time"
 
+	"github.com/coreos/etcd/client"
 	"github.com/coreos/etcd/etcdserver"
 	"github.com/coreos/etcd/etcdserver/etcdhttp"
+	"github.com/coreos/etcd/etcdserver/etcdhttp/httptypes"
+	"github.com/coreos/etcd/pkg/testutil"
 	"github.com/coreos/etcd/pkg/transport"
 	"github.com/coreos/etcd/pkg/types"
+	"github.com/coreos/etcd/rafthttp"
+
+	"github.com/coreos/etcd/Godeps/_workspace/src/golang.org/x/net/context"
 )
 
 const (
-	tickDuration = 5 * time.Millisecond
-	clusterName  = "etcd"
+	tickDuration   = 10 * time.Millisecond
+	clusterName    = "etcd"
+	requestTimeout = 2 * time.Second
 )
 
 func init() {
@@ -50,97 +59,252 @@ func TestClusterOf3(t *testing.T) { testCluster(t, 3) }
 
 func testCluster(t *testing.T, size int) {
 	defer afterTest(t)
-	c := &cluster{Size: size}
+	c := NewCluster(t, size)
 	c.Launch(t)
-	for i := 0; i < size; i++ {
-		for _, u := range c.Members[i].ClientURLs {
-			var err error
-			for j := 0; j < 3; j++ {
-				if err = setKey(u, "/foo", "bar"); err == nil {
-					break
-				}
-			}
-			if err != nil {
-				t.Errorf("setKey on %v error: %v", u.String(), err)
-			}
-		}
-	}
-	c.Terminate(t)
+	defer c.Terminate(t)
+	clusterMustProgress(t, c.Members)
 }
 
-// TODO: use etcd client
-func setKey(u url.URL, key string, value string) error {
-	u.Path = "/v2/keys" + key
-	v := url.Values{"value": []string{value}}
-	req, err := http.NewRequest("PUT", u.String(), strings.NewReader(v.Encode()))
-	if err != nil {
-		return err
+func TestClusterOf1UsingDiscovery(t *testing.T) { testClusterUsingDiscovery(t, 1) }
+func TestClusterOf3UsingDiscovery(t *testing.T) { testClusterUsingDiscovery(t, 3) }
+
+func testClusterUsingDiscovery(t *testing.T, size int) {
+	defer afterTest(t)
+	dc := NewCluster(t, 1)
+	dc.Launch(t)
+	defer dc.Terminate(t)
+	// init discovery token space
+	dcc := mustNewHTTPClient(t, dc.URLs())
+	dkapi := client.NewKeysAPI(dcc)
+	ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
+	if _, err := dkapi.Create(ctx, "/_config/size", fmt.Sprintf("%d", size), -1); err != nil {
+		t.Fatal(err)
 	}
-	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
-	resp, err := http.DefaultClient.Do(req)
-	if err != nil {
-		return err
+	cancel()
+
+	c := NewClusterByDiscovery(t, size, dc.URL(0)+"/v2/keys")
+	c.Launch(t)
+	defer c.Terminate(t)
+	clusterMustProgress(t, c.Members)
+}
+
+func TestDoubleClusterSizeOf1(t *testing.T) { testDoubleClusterSize(t, 1) }
+func TestDoubleClusterSizeOf3(t *testing.T) { testDoubleClusterSize(t, 3) }
+
+func testDoubleClusterSize(t *testing.T, size int) {
+	defer afterTest(t)
+	c := NewCluster(t, size)
+	c.Launch(t)
+	defer c.Terminate(t)
+
+	for i := 0; i < size; i++ {
+		c.AddMember(t)
 	}
-	ioutil.ReadAll(resp.Body)
-	resp.Body.Close()
-	if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusCreated {
-		return fmt.Errorf("statusCode = %d, want %d or %d", resp.StatusCode, http.StatusOK, http.StatusCreated)
+	clusterMustProgress(t, c.Members)
+}
+
+func TestDecreaseClusterSizeOf3(t *testing.T) { testDecreaseClusterSize(t, 3) }
+func TestDecreaseClusterSizeOf5(t *testing.T) { testDecreaseClusterSize(t, 5) }
+
+func testDecreaseClusterSize(t *testing.T, size int) {
+	defer afterTest(t)
+	c := NewCluster(t, size)
+	c.Launch(t)
+	defer c.Terminate(t)
+
+	// TODO: remove the last but one member
+	for i := 0; i < size-2; i++ {
+		id := c.Members[len(c.Members)-1].s.ID()
+		c.RemoveMember(t, uint64(id))
+		c.waitLeader(t, c.Members)
+	}
+	clusterMustProgress(t, c.Members)
+}
+
+// clusterMustProgress ensures that cluster can make progress. It creates
+// a random key first, and check the new key could be got from all client urls
+// of the cluster.
+func clusterMustProgress(t *testing.T, membs []*member) {
+	cc := mustNewHTTPClient(t, []string{membs[0].URL()})
+	kapi := client.NewKeysAPI(cc)
+	ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
+	key := fmt.Sprintf("foo%d", rand.Int())
+	resp, err := kapi.Create(ctx, "/"+key, "bar", -1)
+	if err != nil {
+		t.Fatalf("create on %s error: %v", membs[0].URL(), err)
+	}
+	cancel()
+
+	for i, m := range membs {
+		u := m.URL()
+		cc := mustNewHTTPClient(t, []string{u})
+		kapi := client.NewKeysAPI(cc)
+		ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
+		if _, err := kapi.Watch(key, resp.Node.ModifiedIndex).Next(ctx); err != nil {
+			t.Fatalf("#%d: watch on %s error: %v", i, u, err)
+		}
+		cancel()
+	}
+}
+
+// TODO: support TLS
+type cluster struct {
+	Members []*member
+}
+
+func fillClusterForMembers(ms []*member, cName string) error {
+	addrs := make([]string, 0)
+	for _, m := range ms {
+		for _, l := range m.PeerListeners {
+			addrs = append(addrs, fmt.Sprintf("%s=%s", m.Name, "http://"+l.Addr().String()))
+		}
+	}
+	clusterStr := strings.Join(addrs, ",")
+	var err error
+	for _, m := range ms {
+		m.Cluster, err = etcdserver.NewClusterFromString(cName, clusterStr)
+		if err != nil {
+			return err
+		}
 	}
 	return nil
 }
 
-type cluster struct {
-	Size    int
-	Members []member
+// NewCluster returns an unlaunched cluster of the given size which has been
+// set to use static bootstrap.
+func NewCluster(t *testing.T, size int) *cluster {
+	c := &cluster{}
+	ms := make([]*member, size)
+	for i := 0; i < size; i++ {
+		ms[i] = mustNewMember(t, c.name(i))
+	}
+	c.Members = ms
+	if err := fillClusterForMembers(c.Members, clusterName); err != nil {
+		t.Fatal(err)
+	}
+
+	return c
+}
+
+// NewClusterUsingDiscovery returns an unlaunched cluster of the given size
+// which has been set to use the given url as discovery service to bootstrap.
+func NewClusterByDiscovery(t *testing.T, size int, url string) *cluster {
+	c := &cluster{}
+	ms := make([]*member, size)
+	for i := 0; i < size; i++ {
+		ms[i] = mustNewMember(t, c.name(i))
+		ms[i].DiscoveryURL = url
+	}
+	c.Members = ms
+	return c
 }
 
-// TODO: support TLS
 func (c *cluster) Launch(t *testing.T) {
-	if c.Size <= 0 {
-		t.Fatalf("cluster size <= 0")
+	errc := make(chan error)
+	for _, m := range c.Members {
+		// Members are launched in separate goroutines because if they boot
+		// using discovery url, they have to wait for others to register to continue.
+		go func(m *member) {
+			errc <- m.Launch()
+		}(m)
 	}
-
-	lns := make([]net.Listener, c.Size)
-	addrs := make([]string, c.Size)
-	for i := 0; i < c.Size; i++ {
-		l := newLocalListener(t)
-		// each member claims only one peer listener
-		lns[i] = l
-		addrs[i] = fmt.Sprintf("%v=%v", c.name(i), "http://"+l.Addr().String())
+	for _ = range c.Members {
+		if err := <-errc; err != nil {
+			t.Fatalf("error setting up member: %v", err)
+		}
 	}
-	clusterStr := strings.Join(addrs, ",")
+	// wait cluster to be stable to receive future client requests
+	c.waitMembersMatch(t, c.HTTPMembers())
+}
 
+func (c *cluster) URL(i int) string {
+	return c.Members[i].ClientURLs[0].String()
+}
+
+func (c *cluster) URLs() []string {
+	urls := make([]string, 0)
+	for _, m := range c.Members {
+		for _, u := range m.ClientURLs {
+			urls = append(urls, u.String())
+		}
+	}
+	return urls
+}
+
+func (c *cluster) HTTPMembers() []httptypes.Member {
+	ms := make([]httptypes.Member, len(c.Members))
+	for i, m := range c.Members {
+		ms[i].Name = m.Name
+		for _, ln := range m.PeerListeners {
+			ms[i].PeerURLs = append(ms[i].PeerURLs, "http://"+ln.Addr().String())
+		}
+		for _, ln := range m.ClientListeners {
+			ms[i].ClientURLs = append(ms[i].ClientURLs, "http://"+ln.Addr().String())
+		}
+	}
+	return ms
+}
+
+func (c *cluster) AddMember(t *testing.T) {
+	clusterStr := c.Members[0].Cluster.String()
+	idx := len(c.Members)
+	m := mustNewMember(t, c.name(idx))
+
+	// send add request to the cluster
+	cc := mustNewHTTPClient(t, []string{c.URL(0)})
+	ma := client.NewMembersAPI(cc)
+	ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
+	peerURL := "http://" + m.PeerListeners[0].Addr().String()
+	if _, err := ma.Add(ctx, peerURL); err != nil {
+		t.Fatalf("add member on %s error: %v", c.URL(0), err)
+	}
+	cancel()
+
+	// wait for the add node entry applied in the cluster
+	members := append(c.HTTPMembers(), httptypes.Member{PeerURLs: []string{peerURL}, ClientURLs: []string{}})
+	c.waitMembersMatch(t, members)
+
+	for _, ln := range m.PeerListeners {
+		clusterStr += fmt.Sprintf(",%s=http://%s", m.Name, ln.Addr().String())
+	}
 	var err error
-	for i := 0; i < c.Size; i++ {
-		m := member{}
-		m.PeerListeners = []net.Listener{lns[i]}
-		cln := newLocalListener(t)
-		m.ClientListeners = []net.Listener{cln}
-		m.Name = c.name(i)
-		m.ClientURLs, err = types.NewURLs([]string{"http://" + cln.Addr().String()})
-		if err != nil {
-			t.Fatal(err)
-		}
-		m.DataDir, err = ioutil.TempDir(os.TempDir(), "etcd")
-		if err != nil {
-			t.Fatal(err)
-		}
-		m.Cluster, err = etcdserver.NewClusterFromString(clusterName, clusterStr)
-		if err != nil {
-			t.Fatal(err)
-		}
-		m.ClusterState = etcdserver.ClusterStateValueNew
-		m.Transport, err = transport.NewTransport(transport.TLSInfo{})
-		if err != nil {
-			t.Fatal(err)
-		}
-		// TODO: need the support of graceful stop in Sender to remove this
-		m.Transport.DisableKeepAlives = true
-		m.Transport.Dial = (&net.Dialer{Timeout: 100 * time.Millisecond}).Dial
-
-		m.Launch(t)
-		c.Members = append(c.Members, m)
+	m.Cluster, err = etcdserver.NewClusterFromString(clusterName, clusterStr)
+	if err != nil {
+		t.Fatal(err)
 	}
+	m.NewCluster = false
+	if err := m.Launch(); err != nil {
+		t.Fatal(err)
+	}
+	c.Members = append(c.Members, m)
+	// wait cluster to be stable to receive future client requests
+	c.waitMembersMatch(t, c.HTTPMembers())
+}
+
+func (c *cluster) RemoveMember(t *testing.T, id uint64) {
+	// send remove request to the cluster
+	cc := mustNewHTTPClient(t, []string{c.URL(0)})
+	ma := client.NewMembersAPI(cc)
+	ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
+	if err := ma.Remove(ctx, types.ID(id).String()); err != nil {
+		t.Fatalf("unexpected remove error %v", err)
+	}
+	cancel()
+	newMembers := make([]*member, 0)
+	for _, m := range c.Members {
+		if uint64(m.s.ID()) != id {
+			newMembers = append(newMembers, m)
+		} else {
+			select {
+			case <-m.s.StopNotify():
+				m.Terminate(t)
+			case <-time.After(time.Second):
+				t.Fatalf("failed to remove member %s in one second", m.s.ID())
+			}
+		}
+	}
+	c.Members = newMembers
+	c.waitMembersMatch(t, c.HTTPMembers())
 }
 
 func (c *cluster) Terminate(t *testing.T) {
@@ -149,10 +313,58 @@ func (c *cluster) Terminate(t *testing.T) {
 	}
 }
 
+func (c *cluster) waitMembersMatch(t *testing.T, membs []httptypes.Member) {
+	for _, u := range c.URLs() {
+		cc := mustNewHTTPClient(t, []string{u})
+		ma := client.NewMembersAPI(cc)
+		for {
+			ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
+			ms, err := ma.List(ctx)
+			cancel()
+			if err == nil && isMembersEqual(ms, membs) {
+				break
+			}
+			time.Sleep(tickDuration)
+		}
+	}
+	return
+}
+
+func (c *cluster) waitLeader(t *testing.T, membs []*member) {
+	possibleLead := make(map[uint64]bool)
+	var lead uint64
+	for _, m := range membs {
+		possibleLead[uint64(m.s.ID())] = true
+	}
+
+	for lead == 0 || !possibleLead[lead] {
+		lead = 0
+		for _, m := range membs {
+			if lead != 0 && lead != m.s.Lead() {
+				lead = 0
+				break
+			}
+			lead = m.s.Lead()
+		}
+		time.Sleep(10 * tickDuration)
+	}
+}
+
 func (c *cluster) name(i int) string {
 	return fmt.Sprint("node", i)
 }
 
+// isMembersEqual checks whether two members equal except ID field.
+// The given wmembs should always set ID field to empty string.
+func isMembersEqual(membs []httptypes.Member, wmembs []httptypes.Member) bool {
+	sort.Sort(SortableMemberSliceByPeerURLs(membs))
+	sort.Sort(SortableMemberSliceByPeerURLs(wmembs))
+	for i := range membs {
+		membs[i].ID = ""
+	}
+	return reflect.DeepEqual(membs, wmembs)
+}
+
 func newLocalListener(t *testing.T) net.Listener {
 	l, err := net.Listen("tcp", "127.0.0.1:0")
 	if err != nil {
@@ -161,26 +373,113 @@ func newLocalListener(t *testing.T) net.Listener {
 	return l
 }
 
+func newListenerWithAddr(t *testing.T, addr string) net.Listener {
+	var err error
+	var l net.Listener
+	// TODO: we want to reuse a previous closed port immediately.
+	// a better way is to set SO_REUSExx instead of doing retry.
+	for i := 0; i < 5; i++ {
+		l, err = net.Listen("tcp", addr)
+		if err == nil {
+			break
+		}
+		time.Sleep(500 * time.Millisecond)
+	}
+	if err != nil {
+		t.Fatal(err)
+	}
+	return l
+}
+
 type member struct {
 	etcdserver.ServerConfig
 	PeerListeners, ClientListeners []net.Listener
 
-	s   *etcdserver.EtcdServer
-	hss []*httptest.Server
+	raftHandler *testutil.PauseableHandler
+	s           *etcdserver.EtcdServer
+	hss         []*httptest.Server
+}
+
+func mustNewMember(t *testing.T, name string) *member {
+	var err error
+	m := &member{}
+
+	pln := newLocalListener(t)
+	m.PeerListeners = []net.Listener{pln}
+	m.PeerURLs, err = types.NewURLs([]string{"http://" + pln.Addr().String()})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	cln := newLocalListener(t)
+	m.ClientListeners = []net.Listener{cln}
+	m.ClientURLs, err = types.NewURLs([]string{"http://" + cln.Addr().String()})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	m.Name = name
+
+	m.DataDir, err = ioutil.TempDir(os.TempDir(), "etcd")
+	if err != nil {
+		t.Fatal(err)
+	}
+	clusterStr := fmt.Sprintf("%s=http://%s", name, pln.Addr().String())
+	m.Cluster, err = etcdserver.NewClusterFromString(clusterName, clusterStr)
+	if err != nil {
+		t.Fatal(err)
+	}
+	m.NewCluster = true
+	m.Transport = mustNewTransport(t)
+	return m
+}
+
+// Clone returns a member with the same server configuration. The returned
+// member will not set PeerListeners and ClientListeners.
+func (m *member) Clone(t *testing.T) *member {
+	mm := &member{}
+	mm.ServerConfig = m.ServerConfig
+
+	var err error
+	clientURLStrs := m.ClientURLs.StringSlice()
+	mm.ClientURLs, err = types.NewURLs(clientURLStrs)
+	if err != nil {
+		// this should never fail
+		panic(err)
+	}
+	peerURLStrs := m.PeerURLs.StringSlice()
+	mm.PeerURLs, err = types.NewURLs(peerURLStrs)
+	if err != nil {
+		// this should never fail
+		panic(err)
+	}
+	clusterStr := m.Cluster.String()
+	mm.Cluster, err = etcdserver.NewClusterFromString(clusterName, clusterStr)
+	if err != nil {
+		// this should never fail
+		panic(err)
+	}
+	mm.Transport = mustNewTransport(t)
+	return mm
 }
 
 // Launch starts a member based on ServerConfig, PeerListeners
 // and ClientListeners.
-func (m *member) Launch(t *testing.T) {
-	m.s = etcdserver.NewServer(&m.ServerConfig)
+func (m *member) Launch() error {
+	var err error
+	if m.s, err = etcdserver.NewServer(&m.ServerConfig); err != nil {
+		return fmt.Errorf("failed to initialize the etcd server: %v", err)
+	}
 	m.s.Ticker = time.Tick(tickDuration)
-	m.s.SyncTicker = nil
+	m.s.SyncTicker = time.Tick(500 * time.Millisecond)
 	m.s.Start()
 
+	m.raftHandler = &testutil.PauseableHandler{Next: etcdhttp.NewPeerHandler(m.s)}
+
 	for _, ln := range m.PeerListeners {
 		hs := &httptest.Server{
 			Listener: ln,
-			Config:   &http.Server{Handler: etcdhttp.NewPeerHandler(m.s)},
+			Config:   &http.Server{Handler: m.raftHandler},
 		}
 		hs.Start()
 		m.hss = append(m.hss, hs)
@@ -193,25 +492,78 @@ func (m *member) Launch(t *testing.T) {
 		hs.Start()
 		m.hss = append(m.hss, hs)
 	}
+	return nil
+}
+
+func (m *member) URL() string { return m.ClientURLs[0].String() }
+
+func (m *member) Pause() {
+	m.raftHandler.Pause()
+	m.s.PauseSending()
+}
+
+func (m *member) Resume() {
+	m.raftHandler.Resume()
+	m.s.ResumeSending()
 }
 
 // Stop stops the member, but the data dir of the member is preserved.
 func (m *member) Stop(t *testing.T) {
-	panic("unimplemented")
+	m.s.Stop()
+	for _, hs := range m.hss {
+		hs.CloseClientConnections()
+		hs.Close()
+	}
+	m.hss = nil
 }
 
-// Start starts the member using preserved data dir.
-func (m *member) Start(t *testing.T) {
-	panic("unimplemented")
+// Start starts the member using the preserved data dir.
+func (m *member) Restart(t *testing.T) error {
+	newPeerListeners := make([]net.Listener, 0)
+	for _, ln := range m.PeerListeners {
+		newPeerListeners = append(newPeerListeners, newListenerWithAddr(t, ln.Addr().String()))
+	}
+	m.PeerListeners = newPeerListeners
+	newClientListeners := make([]net.Listener, 0)
+	for _, ln := range m.ClientListeners {
+		newClientListeners = append(newClientListeners, newListenerWithAddr(t, ln.Addr().String()))
+	}
+	m.ClientListeners = newClientListeners
+	return m.Launch()
 }
 
-// Terminate stops the member and remove the data dir.
+// Terminate stops the member and removes the data dir.
 func (m *member) Terminate(t *testing.T) {
 	m.s.Stop()
 	for _, hs := range m.hss {
+		hs.CloseClientConnections()
 		hs.Close()
 	}
 	if err := os.RemoveAll(m.ServerConfig.DataDir); err != nil {
 		t.Fatal(err)
 	}
 }
+
+func mustNewHTTPClient(t *testing.T, eps []string) client.HTTPClient {
+	cc, err := client.NewHTTPClient(mustNewTransport(t), eps)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return cc
+}
+
+func mustNewTransport(t *testing.T) *http.Transport {
+	tr, err := transport.NewTimeoutTransport(transport.TLSInfo{}, rafthttp.ConnReadTimeout, rafthttp.ConnWriteTimeout)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return tr
+}
+
+type SortableMemberSliceByPeerURLs []httptypes.Member
+
+func (p SortableMemberSliceByPeerURLs) Len() int { return len(p) }
+func (p SortableMemberSliceByPeerURLs) Less(i, j int) bool {
+	return p[i].PeerURLs[0] < p[j].PeerURLs[0]
+}
+func (p SortableMemberSliceByPeerURLs) Swap(i, j int) { p[i], p[j] = p[j], p[i] }

integration/member_test.go

@@ -0,0 +1,62 @@
+package integration
+
+import (
+	"io/ioutil"
+	"os"
+	"testing"
+)
+
+func TestPauseMember(t *testing.T) {
+	defer afterTest(t)
+	c := NewCluster(t, 5)
+	c.Launch(t)
+	defer c.Terminate(t)
+
+	for i := 0; i < 5; i++ {
+		c.Members[i].Pause()
+		membs := append([]*member{}, c.Members[:i]...)
+		membs = append(membs, c.Members[i+1:]...)
+		c.waitLeader(t, membs)
+		clusterMustProgress(t, membs)
+		c.Members[i].Resume()
+	}
+	c.waitLeader(t, c.Members)
+	clusterMustProgress(t, c.Members)
+}
+
+func TestRestartMember(t *testing.T) {
+	defer afterTest(t)
+	c := NewCluster(t, 3)
+	c.Launch(t)
+	defer c.Terminate(t)
+
+	for i := 0; i < 3; i++ {
+		c.Members[i].Stop(t)
+		membs := append([]*member{}, c.Members[:i]...)
+		membs = append(membs, c.Members[i+1:]...)
+		c.waitLeader(t, membs)
+		clusterMustProgress(t, membs)
+		err := c.Members[i].Restart(t)
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+	clusterMustProgress(t, c.Members)
+}
+
+func TestLaunchDuplicateMemberShouldFail(t *testing.T) {
+	size := 3
+	c := NewCluster(t, size)
+	m := c.Members[0].Clone(t)
+	var err error
+	m.DataDir, err = ioutil.TempDir(os.TempDir(), "etcd")
+	if err != nil {
+		t.Fatal(err)
+	}
+	c.Launch(t)
+	defer c.Terminate(t)
+
+	if err := m.Launch(); err == nil {
+		t.Errorf("unexpect successful launch")
+	}
+}

integration/migration_test.go

@@ -0,0 +1,34 @@
+package integration
+
+import (
+	"github.com/coreos/etcd/pkg/types"
+	"net"
+	"os/exec"
+	"testing"
+)
+
+func TestUpgradeMember(t *testing.T) {
+	defer afterTest(t)
+	m := mustNewMember(t, "integration046")
+	newPeerListeners := make([]net.Listener, 0)
+	newPeerListeners = append(newPeerListeners, newListenerWithAddr(t, "127.0.0.1:59892"))
+	m.PeerListeners = newPeerListeners
+	urls, err := types.NewURLs([]string{"http://127.0.0.1:59892"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	m.PeerURLs = urls
+	m.NewCluster = true
+	c := &cluster{}
+	c.Members = []*member{m}
+	fillClusterForMembers(c.Members, "etcd-cluster")
+	cmd := exec.Command("cp", "-r", "testdata/integration046_data/conf", "testdata/integration046_data/log", "testdata/integration046_data/snapshot", m.DataDir)
+	err = cmd.Run()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	c.Launch(t)
+	defer c.Terminate(t)
+	clusterMustProgress(t, c.Members)
+}

integration/testdata/integration046_data/conf

@@ -0,0 +1 @@
+{"commitIndex":1,"peers":[]}