Merge pull request #13962 from serathius/contributor-docs

Add contributor docs

.github/ISSUE_TEMPLATE/bug-report.yaml

@@ -6,7 +6,7 @@ body:
 - type: markdown
   attributes:
     value: |
-      Please read https://etcd.io/docs/latest/reporting_bugs/
+      Please read https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md
       If this matter is security related, please disclose it privately via security@etcd.io.
       Please fill the form below and provide as much information as possible.
       Not doing so may result in your bug not being addressed in a timely manner.

Documentation/README.md

@@ -1 +1,4 @@
-The etcd docs are hosted at [etcd.io](https://etcd.io/), and so most files formerly in this directory have moved to the [website](https://github.com/etcd-io/website/) repo.
+This directory includes etcd project internal documentation for new and existing contributors.
+
+For user and developer documentation please go to [etcdd.io](https://etcd.io/),
+which is developed in [website](https://github.com/etcd-io/website/) repo.

Documentation/contributor-guide/branch_management.md

@@ -0,0 +1,27 @@
+# Branch management
+
+## Guide
+
+* New development occurs on the [main branch][main].
+* Main branch should always have a green build!
+* Backwards-compatible bug fixes should target the main branch and subsequently be ported to stable branches.
+* Once the main branch is ready for release, it will be tagged and become the new stable branch.
+
+The etcd team has adopted a *rolling release model* and supports two stable versions of etcd.
+
+### Main branch
+
+The `main` branch is our development branch. All new features land here first.
+
+To try new and experimental features, pull `main` and play with it. Note that `main` may not be stable because new features may introduce bugs.
+
+Before the release of the next stable version, feature PRs will be frozen. A [release manager](../dev-internal/release/#release-management) will be assigned to major/minor version and will lead the etcd community in test, bug-fix and documentation of the release for one to two weeks.
+
+### Stable branches
+
+All branches with prefix `release-` are considered _stable_ branches.
+
+After every minor release ([semver.org](https://semver.org/)), we will have a new stable branch for that release, managed by a [patch release manager](../dev-internal/release/#release-management). We will keep fixing the backwards-compatible bugs for the latest two stable releases. A _patch_ release to each supported release branch, incorporating any bug fixes, will be once every two weeks, given any patches.
+
+[main]: https://github.com/etcd-io/etcd/tree/main
+

Documentation/contributor-guide/local_cluster.md

@@ -0,0 +1,149 @@
+# Set up local cluster
+
+For testing and development deployments, the quickest and easiest way is to configure a local cluster. For a production deployment, refer to the [clustering][clustering] section.
+
+## Local standalone cluster
+
+### Starting a cluster
+
+Run the following to deploy an etcd cluster as a standalone cluster:
+
+```
+$ ./etcd
+...
+```
+
+If the `etcd` binary is not present in the current working directory, it might be located either at `$GOPATH/bin/etcd` or at `/usr/local/bin/etcd`. Run the command appropriately.
+
+The running etcd member listens on `localhost:2379` for client requests.
+
+### Interacting with the cluster
+
+Use `etcdctl` to interact with the running cluster:
+
+1. Store an example key-value pair in the cluster:
+
+    ```
+      $ ./etcdctl put foo bar
+      OK
+    ```
+
+    If OK is printed, storing key-value pair is successful.
+
+2. Retrieve the value of `foo`:
+
+    ```
+    $ ./etcdctl get foo
+    bar
+    ```
+
+    If `bar` is returned, interaction with the etcd cluster is working as expected.
+
+## Local multi-member cluster
+
+### Starting a cluster
+
+A `Procfile` at the base of the etcd git repository is provided to easily configure a local multi-member cluster. To start a multi-member cluster, navigate to the root of the etcd source tree and perform the following:
+
+1. Install `goreman` to control Procfile-based applications:
+
+    ```
+    $ go get github.com/mattn/goreman
+    ```
+
+2. Start a cluster with `goreman` using etcd's stock Procfile:
+
+    ```
+    $ goreman -f Procfile start
+    ```
+
+    The members start running. They listen on `localhost:2379`, `localhost:22379`, and `localhost:32379` respectively for client requests.
+
+### Interacting with the cluster
+
+Use `etcdctl` to interact with the running cluster:
+
+1. Print the list of members:
+
+    ```
+    $ etcdctl --write-out=table --endpoints=localhost:2379 member list
+    ```
+    The list of etcd members are displayed as follows:
+
+    ```
+    +------------------+---------+--------+------------------------+------------------------+
+    |        ID        | STATUS  |  NAME  |       PEER ADDRS       |      CLIENT ADDRS      |
+    +------------------+---------+--------+------------------------+------------------------+
+    | 8211f1d0f64f3269 | started | infra1 | http://127.0.0.1:2380  | http://127.0.0.1:2379  |
+    | 91bc3c398fb3c146 | started | infra2 | http://127.0.0.1:22380 | http://127.0.0.1:22379 |
+    | fd422379fda50e48 | started | infra3 | http://127.0.0.1:32380 | http://127.0.0.1:32379 |
+    +------------------+---------+--------+------------------------+------------------------+
+    ```
+
+2. Store an example key-value pair in the cluster:
+
+    ```
+    $ etcdctl put foo bar
+    OK
+    ```
+
+    If OK is printed, storing key-value pair is successful.
+
+### Testing fault tolerance
+
+To exercise etcd's fault tolerance, kill a member and attempt to retrieve the key.
+
+1. Identify the process name of the member to be stopped.
+
+    The `Procfile` lists the properties of the multi-member cluster. For example, consider the member with the process name, `etcd2`.
+
+2. Stop the member:
+
+    ```
+    # kill etcd2
+    $ goreman run stop etcd2
+    ```
+
+3. Store a key:
+
+    ```
+    $ etcdctl put key hello
+    OK
+    ```
+
+4.  Retrieve the key that is stored in the previous step:
+
+    ```
+    $ etcdctl get key
+    hello
+    ```
+
+5. Retrieve a key from the stopped member:
+
+    ```
+    $ etcdctl --endpoints=localhost:22379 get key
+    ```
+
+    The command should display an error caused by connection failure:
+
+    ```
+    2017/06/18 23:07:35 grpc: Conn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 127.0.0.1:22379: getsockopt: connection refused"; Reconnecting to "localhost:22379"
+    Error:  grpc: timed out trying to connect
+    ```
+6. Restart the stopped member:
+
+    ```
+    $ goreman run restart etcd2
+    ```
+
+7. Get the key from the restarted member:
+
+    ```
+    $ etcdctl --endpoints=localhost:22379 get key
+    hello
+    ```
+
+    Restarting the member re-establish the connection. `etcdctl` will now be able to retrieve the key successfully. To learn more about interacting with etcd, read [interacting with etcd section][interacting].
+
+[clustering]: https://etcd.io/docs/latest/op-guide/clustering/
+[interacting]: https://etcd.io/docs/latest/dev-guide/interacting_v3/

Documentation/contributor-guide/logging.md

@@ -0,0 +1,29 @@
+# Logging Conventions
+
+etcd uses the [capnslog][capnslog] library for logging application output categorized into *levels*. A log message's level is determined according to these conventions:
+
+* Error: Data has been lost, a request has failed for a bad reason, or a required resource has been lost
+  * Examples:
+    * A failure to allocate disk space for WAL
+
+* Warning: (Hopefully) Temporary conditions that may cause errors, but may work fine. A replica disappearing (that may reconnect) is a warning.
+  * Examples:
+    * Failure to send raft message to a remote peer
+    * Failure to receive heartbeat message within the configured election timeout
+
+* Notice: Normal, but important (uncommon) log information.
+  * Examples:
+    * Add a new node into the cluster
+    * Add a new user into auth subsystem
+
+* Info: Normal, working log information, everything is fine, but helpful notices for auditing or common operations.
+  * Examples:
+    * Startup configuration
+    * Start to do snapshot
+
+* Debug: Everything is still fine, but even common operations may be logged, and less helpful but more quantity of notices.
+  * Examples:
+    * Send a normal message to a remote peer
+    * Write a log entry to disk
+
+[capnslog]: https://github.com/coreos/pkg/tree/master/capnslog

Documentation/contributor-guide/modules.md

@@ -0,0 +1,93 @@
+# Golang modules
+
+The etcd project (since version 3.5) is organized into multiple
+[golang modules](https://golang.org/ref/mod) hosted  in a [single repository](https://golang.org/ref/mod#vcs-dir).
+
+![modules graph](modules.svg)
+
+There are following modules:
+
+  - **go.etcd.io/etcd/api/v3** - contains API definitions
+  (like protos & proto-generated libraries) that defines communication protocol
+  between etcd clients and server.
+
+  - **go.etcd.io/etcd/pkg/v3** - collection of utility packages used by etcd
+  without being specific to etcd itself. A package belongs here
+  only if it could possibly be moved out into its own repository in the future.
+  Please avoid adding here code that has a lot of dependencies on its own, as
+  they automatically becoming dependencies of the client library
+  (that we want to keep lightweight).
+
+  - **go.etcd.io/etcd/client/v3** - client library used to contact etcd over
+  the network (grpc). Recommended for all new usage of etcd.
+
+  - **go.etcd.io/etcd/client/v2** - legacy client library used to contact etcd
+  over HTTP protocol. Deprecated. All new usage should depend on /v3 library.
+
+  - **go.etcd.io/etcd/raft/v3** - implementation of distributed consensus
+  protocol. Should have no etcd specific code.
+
+  - **go.etcd.io/etcd/server/v3** - etcd implementation.
+  The code in this package is etcd internal and should not be consumed
+  by external projects. The package layout and API can change within the minor versions.
+
+  - **go.etcd.io/etcd/etcdctl/v3** - a command line tool to access and manage etcd.
+
+  - **go.etcd.io/etcd/tests/v3** - a module that contains all integration tests of etcd.
+    Notice: All unit-tests (fast and not requiring cross-module dependencies)
+    should be kept in the local modules to the code under the test.
+
+  - **go.etcd.io/bbolt** - implementation of persistent b-tree.
+    Hosted in a separate repository: https://github.com/etcd-io/bbolt.
+
+
+### Operations
+
+1. All etcd modules should be released in the same versions, e.g.
+   `go.etcd.io/etcd/client/v3@v3.5.10` must depend on `go.etcd.io/etcd/api/v3@v3.5.10`.
+
+   The consistent updating of versions can by performed using:
+   ```shell script
+   % DRY_RUN=false TARGET_VERSION="v3.5.10" ./scripts/release_mod.sh update_versions
+   ```
+2. The released modules should be tagged according to https://golang.org/ref/mod#vcs-version rules,
+   i.e. each module should get its own tag.
+   The tagging can be performed using:
+   ```shell script
+   % DRY_RUN=false REMOTE_REPO="origin" ./scripts/release_mod.sh push_mod_tags
+   ```
+
+3. All etcd modules should depend on the same versions of underlying dependencies.
+   This can be verified using:
+   ```shell script
+   % PASSES="dep" ./test.sh
+   ```
+
+4. The go.mod files must not contain dependencies not being used and must
+   conform to `go mod tidy` format.
+   This is being verified by:
+   ```
+   % PASSES="mod_tidy" ./test.sh
+   ```
+
+5. To trigger actions across all modules (e.g. auto-format all files), please
+   use/expand the following script:
+   ```shell script
+   % ./scripts/fix.sh
+   ```
+
+### Future
+
+As a North Star, we would like to evaluate etcd modules towards following model:
+
+![modules graph](modules-future.svg)
+
+This assumes:
+  - Splitting etcdmigrate/etcdadm out of etcdctl binary.
+    Thanks to this etcdctl would become clearly a command-line wrapper
+    around network client API,
+    while etcdmigrate/etcdadm would support direct physical operations on the
+    etcd storage files.
+  - Splitting etcd-proxy out of ./etcd binary, as it contains more experimental code
+    so carries additional risk & dependencies.
+  - Deprecation of support for v2 protocol.

Documentation/contributor-guide/reporting_bugs.md

@@ -0,0 +1,45 @@
+# Reporting bugs
+
+If any part of the etcd project has bugs or documentation mistakes, please let us know by [opening an issue][etcd-issue]. We treat bugs and mistakes very seriously and believe no issue is too small. Before creating a bug report, please check that an issue reporting the same problem does not already exist.
+
+To make the bug report accurate and easy to understand, please try to create bug reports that are:
+
+- Specific. Include as much details as possible: which version, what environment, what configuration, etc. If the bug is related to running the etcd server, please attach the etcd log (the starting log with etcd configuration is especially important).
+
+- Reproducible. Include the steps to reproduce the problem. We understand some issues might be hard to reproduce, please includes the steps that might lead to the problem. If possible, please attach the affected etcd data dir and stack strace to the bug report.
+
+- Isolated. Please try to isolate and reproduce the bug with minimum dependencies. It would significantly slow down the speed to fix a bug if too many dependencies are involved in a bug report. Debugging external systems that rely on etcd is out of scope, but we are happy to provide guidance in the right direction or help with using etcd itself.
+
+- Unique. Do not duplicate existing bug report.
+
+- Scoped. One bug per report. Do not follow up with another bug inside one report.
+
+It may be worthwhile to read [Elika Etemad’s article on filing good bug reports][filing-good-bugs] before creating a bug report.
+
+We might ask for further information to locate a bug. A duplicated bug report will be closed.
+
+## Frequently asked questions
+
+### How to get a stack trace
+
+``` bash
+$ kill -QUIT $PID
+```
+
+### How to get etcd version
+
+``` bash
+$ etcd --version
+```
+
+### How to get etcd configuration and log when it runs as systemd service ‘etcd2.service’
+
+``` bash
+$ sudo systemctl cat etcd2
+$ sudo journalctl -u etcd2
+```
+
+Due to an upstream systemd bug, journald may miss the last few log lines when its processes exit. If journalctl says etcd stopped without fatal or panic message, try `sudo journalctl -f -t etcd2` to get full log.
+
+[etcd-issue]: https://github.com/etcd-io/etcd/issues/new
+[filing-good-bugs]: http://fantasai.inkedblade.net/style/talks/filing-good-bugs/

Documentation/contributor-guide/trige_issues.md

@@ -0,0 +1,46 @@
+# Issue triage guidelines
+
+## Purpose
+
+Speed up issue management.
+
+The `etcd` issues are listed at https://github.com/etcd-io/etcd/issues
+and are identified with labels. For example, an issue that is identified
+as a bug will eventually be set to label `area/bug `. New issues will
+start out without any labels, but typically `etcd` maintainers and active contributors
+add labels based on their findings. The detailed list of labels can be found at
+https://github.com/kubernetes/kubernetes/labels
+
+Following are few predetermined searches on issues for convenience:
+* [Bugs](https://github.com/etcd-io/etcd/labels/area%2Fbug)
+* [Help Wanted](https://github.com/etcd-io/etcd/labels/Help%20Wanted)
+* [Longest untriaged issues](https://github.com/etcd-io/etcd/issues?utf8=%E2%9C%93&q=is%3Aopen+sort%3Aupdated-asc+)
+
+## Scope
+
+These guidelines serves as a primary document for triaging an incoming issues in
+`etcd`. Everyone is welcome to help manage issues and PRs but the work and responsibilities discussed in this document are created with `etcd` maintainers and active contributors in mind.
+
+## Validate if an issue is a bug
+
+Validate if the issue is indeed a bug. If not, add a comment with findings and close trivial issue. For non-trivial issue, wait to hear back from issue reporter and see if there is any objection. If issue reporter does not reply in 30 days, close the issue. If the problem can not be reproduced or require more information, leave a comment for the issue reporter.
+
+## Inactive issues
+
+Issues that lack enough information from the issue reporter should be closed if issue reporter do not provide information in 60 days.
+
+## Duplicate issues
+
+If an issue is a duplicate, add a comment stating so along with a reference for the original issue and close it.
+
+## Issues that don't belong to etcd
+
+Sometime issues are reported that actually belongs to other projects that `etcd` use. For example, `grpc` or `golang` issues. Such issues should be addressed by asking reporter to open issues in appropriate other project. Close the issue unless a maintainer and issue reporter see a need to keep it open for tracking purpose.
+
+## Verify important labels are in place
+
+Make sure that issue has label on areas it belongs to, proper assignees are added and milestone is identified. If any of these labels are missing, add one. If labels can not be assigned due to limited privilege or correct label can not be decided, that’s fine, contact maintainers if needed.
+
+## Poke issue owner if needed
+
+If an issue owned by a developer has no PR created in 30 days, contact the issue owner and ask for a PR or to release ownership if needed.

Documentation/contributor-guide/trige_prs.md

@@ -0,0 +1,28 @@
+# PR management
+
+## Purpose
+
+Speed up PR management.
+
+The `etcd` PRs are listed at https://github.com/etcd-io/etcd/pulls
+A PR can have various labels, milestone, reviewer etc. The detailed list of labels can be found at
+https://github.com/kubernetes/kubernetes/labels
+
+Following are few example searches on PR for convenience:
+* [Open PRS for milestone etcd-v3.4](https://github.com/etcd-io/etcd/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+milestone%3Aetcd-v3.4)
+* [PRs under investigation](https://github.com/etcd-io/etcd/labels/Investigating)
+
+## Scope
+
+These guidelines serves as a primary document for managing PRs in `etcd`. Everyone is welcome to help manage PRs but the work and responsibilities discussed in this document is created with `etcd` maintainers and active contributors in mind.
+
+## Handle inactive PRs
+Poke PR owner if review comments are not addressed in 15 days. If PR owner does not reply in 90 days, update the PR with a new commit if possible. If not, inactive PR should be closed after 180 days.
+
+## Poke reviewer if needed
+
+Reviewers are responsive in a timely fashion, but considering everyone is busy, give them some time after requesting review if quick response is not provided. If response is not provided in 10 days, feel free to contact them via adding a comment in the PR or sending an email or message on the Slack.
+
+## Verify important labels are in place
+
+Make sure that appropriate reviewers are added to the PR. Also, make sure that a milestone is identified. If any of these or other important labels are missing, add them. If a correct label cannot be decided, leave a comment for the maintainers to do so as needed.

README.md

@@ -169,7 +169,7 @@ See [CONTRIBUTING](CONTRIBUTING.md) for details on submitting patches and the co
 
 ## Reporting bugs
 
-See [reporting bugs](https://etcd.io/docs/latest/reporting_bugs/) for details about reporting any issues.
+See [reporting bugs](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md) for details about reporting any issues.
 
 ## Reporting a security vulnerability
 
@@ -177,9 +177,9 @@ See [security disclosure and release process](security/README.md) for details on
 
 ## Issue and PR management
 
-See [issue triage guidelines](https://etcd.io/docs/current/triage/issues/) for details on how issues are managed.
+See [issue triage guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/triage_issues.md) for details on how issues are managed.
 
-See [PR management](https://etcd.io/docs/current/triage/prs/) for guidelines on how pull requests are managed.
+See [PR management](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/triage_prs.md) for guidelines on how pull requests are managed.
 
 ## etcd Emeritus Maintainers
 

security/README.md

@@ -6,7 +6,7 @@ Join the [etcd-dev](https://groups.google.com/forum/?hl=en#!forum/etcd-dev) grou
 
 We’re extremely grateful for security researchers and users that report vulnerabilities to the etcd Open Source Community. All reports are thoroughly investigated by a dedicated committee of community volunteers called [Product Security Committee](security-release-process.md#product-security-committee).
 
-To make a report, please email the private [security@etcd.io](mailto:security@etcd.io) list with the security details and the details expected for [all etcd bug reports](https://etcd.io/docs/latest/reporting_bugs/).
+To make a report, please email the private [security@etcd.io](mailto:security@etcd.io) list with the security details and the details expected for [all etcd bug reports](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
 
 ### When Should I Report a Vulnerability?