Compare commits

...

156 Commits

Author SHA1 Message Date
Yicheng Qin
ab5a69cb18 *: bump to v2.2.0-alpha.0 2015-08-13 10:20:05 -07:00
Yicheng Qin
976ce93539 Merge pull request #3277 from yichengq/better-log
etcdserver: specify timeout caused by leader election
2015-08-12 17:02:27 -07:00
Yicheng Qin
27170e67b9 etcdserver: specify timeout caused by leader election
Before this PR, the timeout caused by leader election returns:

```
14:45:37 etcd2 | 2015-08-12 14:45:37.786349 E | etcdhttp: got unexpected
response error (etcdserver: request timed out)
```

After this PR:

```
15:52:54 etcd1 | 2015-08-12 15:52:54.389523 E | etcdhttp: etcdserver:
request timed out, possibly due to leader down
```
2015-08-12 16:53:18 -07:00
Xiang Li
ddfe343e77 Merge pull request #3271 from yichengq/doc-discovery
docs: add discovery protocol doc
2015-08-12 13:51:32 -07:00
Yicheng Qin
a45f0ede56 docs: add discovery protocol doc
This document talks about the technical details of discovery service
protocol. It helps users to learn about how discovery service works and
what behavior to expect.
2015-08-12 13:15:21 -07:00
Alex Polvi
7bd9d9aede Merge pull request #3273 from polvi/kube-hack
add etcd on k8s example
2015-08-12 22:13:15 +03:00
Alex Polvi
cfb3522b63 add etcd on k8s example 2015-08-12 22:12:00 +03:00
Xiang Li
f468d8b51a Merge pull request #3270 from xiang90/better_err
Better error message for etcdctl
2015-08-12 10:27:42 -07:00
Xiang Li
7e04a79fb4 etcdctl: print out better error information 2015-08-12 10:09:56 -07:00
Xiang Li
5d06d4ec44 client: print url as string 2015-08-12 10:09:40 -07:00
Xiang Li
e894756144 Merge pull request #3190 from yichengq/adjust-prop-timeout
etcdserver: adjust proposal timeout based on config
2015-08-12 09:41:25 -07:00
Yicheng Qin
c3d4d11402 etcdhttp: adjust request timeout based on config
It uses heartbeat interval and election timeout to estimate the
expected request timeout.

This PR helps etcd survive under high roundtrip-time environment,
e.g., globally-deployed cluster.
2015-08-12 09:22:59 -07:00
Xiang Li
18ecc297bc Merge pull request #3254 from es-chow/log-group
set groupID in multinode as log context so it can be logged
2015-08-12 08:05:50 -07:00
es-chow
cc362ccdad raft: set logger to raft so log context such as multinode groupID can be logged 2015-08-12 22:56:00 +08:00
Yicheng Qin
5a91937367 etcdserver: adjust commit timeout based on config
It uses heartbeat interval and election timeout to estimate the
commit timeout for internal requests.

This PR helps etcd survive under high roundtrip-time environment,
e.g., globally-deployed cluster.
2015-08-11 21:09:03 -07:00
Yicheng Qin
042afcf2a3 Merge pull request #3266 from yichengq/client-readme
client: clean up README
2015-08-11 16:21:13 -07:00
Yicheng Qin
7d618c46ad client: clean up README
Address rob's comments about sentences in README.
2015-08-11 15:33:56 -07:00
Xiang Li
18a1c95f22 Merge pull request #3263 from xiang90/ctl_tr
etcdctl: add per request timeout
2015-08-11 14:17:12 -07:00
Yicheng Qin
dceacacd49 Merge pull request #3194 from yichengq/client-readme
client: add README
2015-08-11 13:35:54 -07:00
Xiang Li
e36c499d0f etcdctl: add per request timeout 2015-08-11 13:33:50 -07:00
Yicheng Qin
8a7cf56e13 client: add README
It describes some basic usage and caveat of etcd/client package.

Write it together with Xiang.
2015-08-11 12:07:24 -07:00
Yicheng Qin
83efc08137 Merge pull request #3262 from yichengq/client-deadline
client: return context.DeadlineExceeded instead of ClusterError
2015-08-11 10:42:29 -07:00
Yicheng Qin
a1ef699aeb client: return context.DeadlineExceeded instead of ClusterError
This is done to match user expectation to see context.DeadlineExceeded
when it reaches deadline.
2015-08-11 10:18:38 -07:00
Yicheng Qin
1fe52e1ec3 Merge pull request #3245 from yichengq/client_timeout
client: set timeout for each request
2015-08-11 10:10:42 -07:00
Yicheng Qin
f4c29a5f55 client: support to set timeout for each request
Add HeaderTimeout field in Config, so users could set timeout for each request.
Before this, one hanged request may block the call for long time. After
this, if the network is good, the user could set short timeout and expect
that API call can attempt next available endpoint quickly.
2015-08-11 10:01:05 -07:00
Xiang Li
a718329ad3 Merge pull request #3248 from xiang90/v3
initial v3 demo
2015-08-10 13:59:03 -07:00
Xiang Li
fb5e1ac548 Merge pull request #3256 from xiang90/update_log
update logger
2015-08-10 13:54:28 -07:00
Xiang Li
6c58333969 etcdmain: use default formatter
The default formatter would use syslog style when running
under init system, and would use pretty format otherwise.
2015-08-10 13:38:22 -07:00
Xiang Li
48e36bbb84 Godep: update capnslog dependency 2015-08-10 13:38:00 -07:00
Xiang Li
b0ea4ab3b1 doc: link to v3 api doc 2015-08-10 11:22:55 -07:00
Xiang Li
c32919e6d1 *: rename v3etcdctl to etcdctlv3 2015-08-10 11:21:37 -07:00
Xiang Li
c1e0b19f9f *: better flag 2015-08-10 09:53:17 -07:00
Xiang Li
48b1cd54f3 Merge pull request #3243 from xiang90/conf
doc: add runtime reconfiguration design doc
2015-08-09 10:56:51 -07:00
Xiang Li
89bf5824c2 Merge pull request #3159 from sofuture/master
use /usr/bin/env to find bash
2015-08-09 10:56:12 -07:00
Xiang Li
601801ced5 doc: add runtime reconfiguration design doc 2015-08-09 10:55:34 -07:00
Brandon Philips
45f3a0c547 Merge pull request #3249 from philips/get-etcd-running-under-arm64
Get etcd running under arm64
2015-08-08 20:32:33 -07:00
Jeff Zellner
1239e1ce6f test, scripts: use /usr/bin/env to find bash
use /usr/bin/env to find bash

add set -e back into scripts it was removed from
2015-08-08 20:52:53 -06:00
Brandon Philips
1b894c6b0b test: race detector doesn't work on armv7l
Test fails without this fix on armv7l:

    go test: -race is only supported on linux/amd64, freebsd/amd64, darwin/amd64 and windows/amd64
2015-08-08 18:11:41 -07:00
Brandon Philips
fb1951204c etcdserver: move atomics to make etcd work on arm64
Follow the simple rule in the atomic package:

"On both ARM and x86-32, it is the caller's responsibility to arrange
for 64-bit alignment of 64-bit words accessed atomically. The first word
in a global variable or in an allocated struct or slice can be relied
upon to be 64-bit aligned."

Tested on a system with /proc/cpuinfo reporting:

processor       : 0
model name      : ARMv7 Processor rev 1 (v7l)
Features        : swp half thumb fastmult vfp edsp thumbee neon vfpv3
tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc0d
CPU revision    : 1
2015-08-08 18:11:41 -07:00
Xiang Li
9ff7075ce8 etcdserver: use v3server interface 2015-08-08 10:39:04 -07:00
Xiang Li
523567bcc7 v3etcdctl: initial v3 ctl support 2015-08-08 05:58:58 -07:00
Xiang Li
f004b4dac7 *: etcdserver supports v3 demo 2015-08-08 05:58:29 -07:00
Xiang Li
82afadbcc6 etcdserverpb: update proto 2015-08-08 05:31:35 -07:00
Xiang Li
668a8a8367 Merge pull request #3242 from xiang90/typo
*: fix typos vaild->valid
2015-08-07 10:58:39 -07:00
Xiang Li
845c51fedd *: fix typos vaild->valid 2015-08-07 10:57:11 -07:00
Yicheng Qin
f0a5874473 Merge pull request #3241 from yichengq/sync-pin
client: Sync() pin the endpoint when member list doesn't change
2015-08-07 10:24:29 -07:00
Yicheng Qin
0ab16db728 client: Sync() pin the endpoint when member list doesn't change
This helps client to pin the same endpoint as long as cluster doesn't change.
2015-08-07 10:08:28 -07:00
Xiang Li
d7adcc3e65 Merge pull request #3239 from xiang90/improve_probing
rafthttp: use customized transport for probing
2015-08-07 09:37:32 -07:00
Xiang Li
b6580a9591 rafthttp: use customized transport for probing
We need to support TLS verification when probing.
2015-08-06 16:20:44 -07:00
Xiang Li
d2363afd52 Merge pull request #3240 from xiang90/fix_log
etcdmain: fix path printing
2015-08-06 15:56:14 -07:00
Yicheng Qin
f03f048232 Merge pull request #3184 from yichengq/fast-bootstrap
etcdserver: tick ElectionTicks before starting when bootstrap new cluster
2015-08-06 15:54:40 -07:00
Xiang Li
1b572ae2dd etcdmain: fix path printing 2015-08-06 15:53:24 -07:00
Yicheng Qin
21f5b885f2 etcdserver: fast election timeout when bootstrap cluster
The behavior accelarates the happen of the first-time leader election,
so the cluster could elect its leader fast. Technically, it could
help to reduce `electionMs - heartbeatMs` wait time for the first leader election.

Main usage:
1. Quick start for the local cluster when setting a little longer
election timeout
2. Quick start for the global cluster, which sets election timeout to
its maximum 50s.
2015-08-06 15:44:26 -07:00
Yicheng Qin
a637e86372 Merge pull request #3220 from yichengq/fix-auth-check
etcdhttp: fix access check for multiple roles in auth
2015-08-06 15:09:04 -07:00
Xiang Li
b9c6b64d61 Merge pull request #3216 from yichengq/cancel-err
client: return context canceled error correctly
2015-08-06 15:04:49 -07:00
Yicheng Qin
b965c4b415 Merge pull request #3217 from yichengq/update-migrate-example
update commands used in admin_guide.md
2015-08-06 15:00:04 -07:00
Yicheng Qin
78af793338 client: return context canceled error correctly
If the body is closed to stop watching, it will ignore the error from
reading body and return context error.

Before this PR, the cancel when watching always returns error `read tcp
127.0.0.1:57824: use of closed network connection`. After this PR, it
will return expected context canceled error.
2015-08-06 14:52:04 -07:00
Xiang Li
b04bb3e0ea Merge pull request #3229 from xiang90/f_cerr
client: return context.Canceled error when user cancels the request
2015-08-06 14:41:19 -07:00
Yicheng Qin
25ad71fbac Merge pull request #3225 from yichengq/client-record-err
client: return correct error for 50x response
2015-08-06 14:40:38 -07:00
Xiang Li
7314310aed Merge pull request #3233 from xiang90/srv_discovery
better dns discovery error and doc
2015-08-06 14:35:22 -07:00
Yicheng Qin
cfeaf3d172 client: return correct error for 50x response
etcd always returns 500/503 response when it may have no leader.
So we should log the other 50x response in a normal way.

This helps to log correctly when discovery meets 504 error. Before this
PR, it logs like this:

```
18:31:58 etcd2 | 2015/08/4 18:31:58 discovery: error #0: client: etcd
member https://discovery.etcd.io has no leader
18:31:58 etcd2 | 2015/08/4 18:31:58 discovery: waiting for other nodes:
error connecting to https://discovery.etcd.io, retrying in 4s
```

After this PR:

```
22:20:25 etcd2 | 2015/08/4 22:20:25 discovery: error #0: client: etcd
member https://discovery.etcd.io returns server error [Gateway Timeout]
22:20:25 etcd2 | 2015/08/4 22:20:25 discovery: waiting for other nodes:
error connecting to https://discovery.etcd.io, retrying in 4s
```
2015-08-06 14:25:03 -07:00
Xiang Li
e9f05e8959 doc: explain srv error 2015-08-06 14:24:58 -07:00
Yicheng Qin
2c2249dadc Merge pull request #3219 from yichengq/limit-listener
etcdmain: stop accepting client conns when it reachs limit
2015-08-06 12:17:49 -07:00
Yicheng Qin
97923ca3fc etcdmain: close client conns when it exceeds limit
This solves the problem that etcd may fatal because its critical path
cannot get file descriptor resource when the number of clients is too
big. The PR lets the client listener close client connections
immediately after they are accepted when
the file descriptor usage in the process reaches some pre-set limit, so
it ensures that the internal critical path could always get file
descriptor when it needs.

When there are tons to clients connecting to the server, the original
behavior is like this:

```
2015/08/4 16:42:08 etcdserver: cannot monitor file descriptor usage
(open /proc/self/fd: too many open files)
2015/08/4 16:42:33 etcdserver: failed to purge snap file open
default2.etcd/member/snap: too many open files
[halted]
```

Current behavior is like this:

```
2015/08/6 19:05:25 transport: accept error: closing connection,
exceed file descriptor usage limitation (fd limit=874)
2015/08/6 19:05:25 transport: accept error: closing connection,
exceed file descriptor usage limitation (fd limit=874)
2015/08/6 19:05:26 transport: accept error: closing connection,
exceed file descriptor usage limitation (fd limit=874)
2015/08/6 19:05:27 transport: accept error: closing connection,
exceed file descriptor usage limitation (fd limit=874)
2015/08/6 19:05:28 transport: accept error: closing connection,
exceed file descriptor usage limitation (fd limit=874)
2015/08/6 19:05:28 etcdserver: 80% of the file descriptor limit is
used [used = 873, limit = 1024]
```

It is available at linux system today because pkg/runtime only has linux
support.
2015-08-06 12:03:20 -07:00
Xiang Li
203e0f178b etcdmian: better error for srv discovery failure 2015-08-06 11:38:53 -07:00
Xiang Li
01c286ccb6 Merge pull request #3231 from xiang90/fallocate
pkg/fileutil: support perallocate
2015-08-06 10:25:28 -07:00
Xiang Li
39a4b6a5e5 pkg/fileutil: support perallocate 2015-08-06 10:10:58 -07:00
Xiang Li
9a8607fce1 Merge pull request #3187 from yichengq/client-keep-sync
client: add KeepSync function
2015-08-06 00:16:28 -07:00
Yicheng Qin
c53b3016ae client: add AutoSync function
AutoSync provides the way for client to syncing member list from
etcd cluster automatically.
2015-08-05 13:22:56 -07:00
Yicheng Qin
807a6f209e docs/admin_guide: decouple example from CoreOS specific details
This makes the example commands general, while keeping it easy to
understand. It also fixes some name mismatch.
2015-08-05 11:33:46 -07:00
Xiang Li
f38187bbdb client: return context.Canceled error when user cancels the request 2015-08-05 09:52:30 -07:00
Xiang Li
ff0b8723c7 Merge pull request #2688 from xiang90/versioning
etcdserver: internal request union
2015-08-05 09:27:32 -07:00
Xiang Li
58503817ec etcdserver: internal request union 2015-08-05 07:47:10 -07:00
Xiang Li
487639b2d8 Merge pull request #3222 from mitake/wal-log-error
wal: log errors in wal.Close()
2015-08-04 23:19:45 -07:00
Xiang Li
9cbeffc720 Merge pull request #3224 from xiang90/fix_ls
etcdctl: ls takes / as default key arg
2015-08-04 23:15:29 -07:00
Hitoshi Mitake
ba76e27875 wal: log errors in wal.Close()
This patch adds error logging in wal.Close() if unlocking and
destroying fail. Though it is hard to handling the errors, logging
would be helpful for trouble shooting.
2015-08-05 15:03:45 +09:00
Xiang Li
9527a97720 etcdctl: ls takes / as default key arg 2015-08-04 22:56:55 -07:00
Xiang Li
718a42f408 Merge pull request #3210 from xiang90/probing
monitoring connectivity between peers
2015-08-04 16:56:31 -07:00
Yicheng Qin
18169e896c etcdhttp: fix access check for multiple roles in auth
Check access for multiple roles should go through all roles.
2015-08-04 14:31:07 -07:00
Yicheng Qin
0650170a1b Merge pull request #3196 from eyakubovich/fix-watch-timeout
client: handle watch timing out elegantly
2015-08-04 13:52:42 -07:00
Xiang Li
1e048b5c24 rafthttp: cleanup prober when stopping the transport 2015-08-04 17:42:51 +08:00
Xiang Li
709718ed97 godeps: update probing pkg 2015-08-04 17:40:39 +08:00
Xiang Li
0fc764200d rafthttp: monitor connection 2015-08-04 17:39:40 +08:00
Xiang Li
ff5c3469c1 Merge pull request #3197 from xiang90/health
etcdctl: cluster-health supports forever flag
2015-08-03 20:48:06 -07:00
Eugene Yakubovich
6312e22b1d client: handle empty watch responses elegantly
Even though current etcd does not time out
watches, the client could be running against
an old etcd version or the server may close
polling connection for other reasons.
This patch ignores successful (as in 200)
responses with emtpy bodies instead
of producing JSON errors.
2015-08-03 11:47:21 -07:00
Xiang Li
306085db5f Godeps: add probing dependency 2015-08-03 09:07:43 +08:00
Xiang Li
f7f00b0af6 etcdctl: cluster-health supports forever flag
cluster-health command supports checking the cluster health
forever.
2015-08-01 22:29:08 +08:00
Xiang Li
3da1df2648 Merge pull request #3207 from xiang90/rm_migration
*: remove migration related stuff from 2.2
2015-08-01 19:47:17 +08:00
Xiang Li
2b8abeb093 *: remove migration related stuff from 2.2 2015-08-01 19:37:20 +08:00
Xiang Li
eee1c8b8ee Merge pull request #3200 from xiang90/d_doc
doc: unique names must be specified when using public discovery service
2015-08-01 07:34:25 +08:00
Yicheng Qin
8bd9554338 Merge pull request #3202 from yichengq/fix-etcdctl-watch
etcdctl: fix watch -after-index parsing
2015-07-31 14:41:45 -07:00
Yicheng Qin
4a89b3f8f3 Merge pull request #3116 from offscale/master
build: implemented build shell-script for Windows
2015-07-31 11:55:42 -07:00
Xiang Li
05b2d06788 Merge pull request #3199 from xiang90/sdnotify
etcdmain: support sdnotify for readiness
2015-07-31 19:04:35 +08:00
Samuel Marks
4a0d8ee4bd build: implemented build shell-script for Windows 2015-07-31 17:43:47 +10:00
Xiang Li
0cbac56fa2 etcdmain: support sdnotify for readiness 2015-07-31 13:33:18 +08:00
Xiang Li
beeecc32b0 doc: unique names must be specified when using public discovery service 2015-07-31 09:12:44 +08:00
Barak Michener
c1c5c7c99c Merge pull request #3091 from barakmich/client_auth_cov
etcdhttp: Improve test coverage surrounding auth
2015-07-30 17:00:49 -04:00
Barak Michener
dd1a8fe330 etcdhttp: Improve test coverage surrounding auth 2015-07-30 14:21:08 -04:00
Yicheng Qin
147885078c etcdctl: fix watch -after-index parsing
It uses -after-index incorrectly now:

```
$ ./bin/etcdctl --debug watch -after-index 31 foo
Cluster-Endpoints: http://localhost:2379, http://localhost:4001
cURL Command: curl -X GET
http://localhost:2379/v2/keys/foo?recursive=false&wait=true&waitIndex=33
```

After this PR:

```
$ ./bin/etcdctl --debug watch -after-index 31 foo
Cluster-Endpoints: http://localhost:2379, http://localhost:4001
cURL Command: curl -X GET
http://localhost:2379/v2/keys/foo?recursive=false&wait=true&waitIndex=32
```
2015-07-30 11:15:43 -07:00
Yicheng Qin
219ed1695b Merge pull request #3178 from yichengq/refactor-cluster-health
etcdctl: refactor the way to check cluster health
2015-07-29 18:16:26 -07:00
Xiang Li
80b794dccc Merge pull request #3185 from xiang90/add_debug_endpoint
etcdhttp: add config/local/debug endpoint
2015-07-30 08:46:07 +08:00
Xiang Li
4e31df2c2b etcdhttp: add config/local/log endpoint
PUT on the endpoint sets the GlobalDebugLevel to json level value.
The action overwrites the origianl log level setting from
users. We need to write doc to warn this.
2015-07-30 08:35:01 +08:00
Yicheng Qin
e62a3b8a62 Merge pull request #2891 from glensc/patch-1
build: use posix shell
2015-07-29 17:15:57 -07:00
Xiang Li
ff945c7404 Merge pull request #3181 from xiang90/2.2-client-error
client: return cluster error if the etcd cluster is not avaliable
2015-07-30 08:08:09 +08:00
Yicheng Qin
f1aaa7a9e3 etcdctl: refactor the way to check cluster health
This method uses raft status exposed at /debug/varz to determine the
health of the cluster. It uses whether commit index increases to
determine the cluster health, and uses whether match index increases to
determine the member health.

This could fix the bug #2711 that fails to detect follower is unhealthy
because it doesn't rely on whether message in long-polling connection is sent.

This health check is stricter than the old one, and reflects the
situation that whether followers are healthy in the view of the leader. One
example is that if the follower is receiving the snapshot, it will turns
out to be unhealthy because it doesn't move forward.

`etcdctl cluster-health` will reflect the healthy view in the raft level,
while connectivity checks reflects the healthy view in transport level.
2015-07-29 17:06:55 -07:00
Xiang Li
a47e661fff discovery: print out detailed cluster error 2015-07-29 23:06:57 +08:00
Xiang Li
5fa8652241 client: return cluster error if the etcd cluster is not avaliable
Add a new ClusterError type. It contians all encountered errors and
return ClusterNotAvailable as the error string.
2015-07-29 22:55:15 +08:00
Yicheng Qin
6b8b507312 Merge pull request #3176 from yichengq/reject-high-election
etcdmain: reject unreasonably high values of -election-timeout
2015-07-28 10:33:58 -07:00
Yicheng Qin
ec214030d0 etcdmain: reject unreasonably high values of -election-timeout
This helps users to detect setting problem early.
2015-07-28 10:07:57 -07:00
Yicheng Qin
7831a30e46 Merge pull request #3180 from shafreeck/master
Update libraries-and-tools.md
2015-07-27 14:45:31 -07:00
Yicheng Qin
6184e271a4 Merge pull request #3164 from yichengq/pin-endpoint
client: pin itself to an endpoint that given
2015-07-27 14:35:51 -07:00
Yicheng Qin
6fc9dbfe56 Merge pull request #3114 from yichengq/clean-raft-init
etcdserver: clean up start and stop logic of raft
2015-07-27 14:19:25 -07:00
Yicheng Qin
ea2347a40f client: pin itself to an endpoint that given
1. When reset endpoints, client will choose a random endpoint to pin.
2. If the pinned endpoint is healthy, client will keep using it.
3. If the pinned endpoint becomes unhealthy, client will attempt other
endpoints and update its pin.
2015-07-27 13:36:53 -07:00
Yicheng Qin
7696dd3280 etcdserver: clean up start and stop logic of raft
kill TODO and make it more readable.
2015-07-27 13:24:26 -07:00
Yicheng Qin
5e3dc31e6f Merge pull request #3150 from gouyang/master
pkg/mflag: add modified flag package
2015-07-24 15:26:07 -07:00
Xiang Li
a7eef376b7 Merge pull request #3183 from xiang90/txn
*: tnx -> txn
2015-07-25 01:48:06 +08:00
Xiang Li
53a77fa519 *: tnx -> txn 2015-07-24 23:21:09 +08:00
Guohua Ouyang
c9769ee966 etcdmain: Don't print flags when flag parse error
At present it prints the whole usage and flags, which cause the exact
error message is hidden two screens above.

Fixes #3141

Signed-off-by: Guohua Ouyang <gouyang@redhat.com>
2015-07-24 21:29:21 +08:00
Shafreeck Sea
e75446ca27 docs: add cetcd into libraries-and-tools.md 2015-07-24 12:08:39 +00:00
Yicheng Qin
b407f72766 Merge pull request #3166 from yichengq/publish-timeout
etcdserver: rename defaultPublishRetryInterval -> defaultPublishTimeout
2015-07-23 10:30:41 -07:00
Yicheng Qin
b7892b20c1 etcdserver: rename defaultPublishRetryInterval -> defaultPublishTimeout
This makes code more readable and reasonable.
2015-07-23 10:09:28 -07:00
Xiang Li
58bc617dd0 Merge pull request #3175 from xiang90/2.2-ctl-bug
etcdctl: fix exec watch command
2015-07-23 14:37:38 +08:00
Xiang Li
448ca20cdc etcdctl: fix exec watch command
The previous flag parsing has a small issue. It uses
`recursive == true` and `after-index == 0` to determine
if user specifies the sub flags. This is incorrect since
user can specify `after-index = 0`. Then the flag parsing
would be confused.

This commit explicitly find the `--` in the remaining args
and determine the key and cmdArgs accordingly.
2015-07-23 13:13:15 +08:00
Xiang Li
43f4b99d52 Merge pull request #3174 from xiang90/2.2_submit_bug
doc: add reporting bug doc
2015-07-23 13:08:35 +08:00
Xiang Li
1b5e41e3f4 doc: add reporting bug doc 2015-07-23 12:55:38 +08:00
Yicheng Qin
93002caca5 Merge pull request #3165 from yichengq/client-quorum
client: add Quorum option in getOption
2015-07-22 16:54:14 -07:00
Yicheng Qin
b20b87893f client: add Quorum option in getOption 2015-07-22 15:19:34 -07:00
Xiang Li
6be02ff5ec etcdmian: fix initialization confilct
Fix #3142

Ignore flags if etcd is already initialized.
2015-07-21 12:53:21 -07:00
Yicheng Qin
24db661401 etcdmain: warn when listening on HTTP if TLS is set
If the user sets TLS info, this implies that he wants to listen on TLS.
If etcd finds that urls to listen is still HTTP schema, it prints out
warning to notify user about possible wrong setting.
2015-07-21 12:53:21 -07:00
Yicheng Qin
604709cad7 etcdctl: update -peers to default to use schema
Change its default value from `127.0.0.1:4001,127.0.0.1:2379` to
`http://127.0.0.1:4001,http://127.0.0.1:2379`

Adding HTTP schema makes its format consistent with etcd's xxx-urls
flags.
2015-07-21 12:53:21 -07:00
Xiang Li
d9c27138fa discovery: return bad discovery endpoint error 2015-07-21 12:53:21 -07:00
Xiang Li
d2dac0fe59 client: consume json error and return ErrInvaildJSON
The default JSON error is not very readable. We let client
consume the error and return a more understandable error in
the context of etcd.

Fix #3120
2015-07-21 12:53:21 -07:00
Yicheng Qin
6317abf7e4 pkg/transport: fix HTTPS downgrade bug for keepalive listener
If TLS config is empty, etcd downgrades keepalive listener from HTTPS to
HTTP without warning. This results in HTTPS downgrade bug for client urls.
The commit returns error if it cannot listen on TLS.
2015-07-21 12:53:21 -07:00
Mohammad Samman
43437e21f9 etcdctl: added domain discovery flag
provided a domain, will look up SRV records for etcd endpoints

Fixes #2636
2015-07-21 12:53:21 -07:00
Xiang Li
dc3f7f5d90 *: detect duplicate name for discovery bootstrap 2015-07-21 12:53:20 -07:00
Xiang Li
b8279b3591 types: add len func for urlmaps 2015-07-21 12:53:20 -07:00
Xiang Li
ee82ee05b4 etcdctl: support member update command 2015-07-21 12:53:20 -07:00
Xiang Li
6e3769d39e client: add member update 2015-07-21 12:53:20 -07:00
Xiang Li
9f9661f513 etcdctl: print out key and action when watching recursively 2015-07-21 12:53:20 -07:00
Xiang Li
87ef0f0b3e godep: remove go-etcd dependency 2015-07-21 12:53:20 -07:00
Xiang Li
071ad9f72b etcdctl: health use etcd/client 2015-07-21 12:53:20 -07:00
Xiang Li
0b1ddce889 etcdctl: import snap use etcd/client 2015-07-21 12:53:20 -07:00
Xiang Li
adeb101e04 etcdctl: remove old stuff 2015-07-21 12:53:20 -07:00
Xiang Li
759c156e3e etcdctl: exec_watch use etcd/client 2015-07-21 12:53:20 -07:00
Xiang Li
5b01b3877f etcdctl: watch use etcd/client 2015-07-21 12:53:20 -07:00
Xiang Li
b20c06348d etcdctl: ls use etcd/client 2015-07-21 12:53:19 -07:00
Xiang Li
ae1669de26 etcdctl: updatedir use etcd/client 2015-07-21 12:53:19 -07:00
Xiang Li
f12ae45c6a etcdctl: update use etcd/client 2015-07-21 12:53:19 -07:00
Xiang Li
58b19a7c1e etcdctl: rmdir use etcd/client 2015-07-21 12:53:19 -07:00
Xiang Li
9d7a8dd2b0 etcdctl: mk use etcd/client 2015-07-21 12:53:19 -07:00
Xiang Li
61befc7ce6 etcdctl: minor cleanup 2015-07-21 12:53:19 -07:00
Xiang Li
e3fcc450cf etcdctl: make rm use etcd/client 2015-07-21 12:53:19 -07:00
Xiang Li
9d9c3a7180 etcdctl: make setdir/mkdir use etcd/client 2015-07-21 12:53:19 -07:00
Xiang Li
db4b18aee3 etcdctl: make set command use etcd/client 2015-07-21 12:53:19 -07:00
Xiang Li
e9478ba630 etcdctl: make get command use etcd/client 2015-07-21 12:53:19 -07:00
Elan Ruusamäe
77c3613d94 build: use posix shell 2015-05-30 09:34:54 +03:00
210 changed files with 4863 additions and 50005 deletions

View File

@@ -92,21 +92,21 @@ bc1083c870280d44: name=infra2 peerURLs=http://10.0.1.12:2380 clientURLs=http://1
#### Stop the member etcd process
```
$ ssh core@10.0.1.11
$ ssh 10.0.1.11
```
```
$ sudo systemctl stop etcd
$ kill `pgrep etcd`
```
#### Copy the data directory of the now-idle member to the new machine
```
$ tar -cvzf node1.etcd.tar.gz /var/lib/etcd/node1.etcd
$ tar -cvzf infra1.etcd.tar.gz %data_dir%
```
```
$ scp node1.etcd.tar.gz core@10.0.1.13:~/
$ scp infra1.etcd.tar.gz 10.0.1.13:~/
```
#### Update the peer URLs for that member to reflect the new machine
@@ -119,15 +119,15 @@ $ curl http://10.0.1.10:2379/v2/members/b4db3bf5e495e255 -XPUT \
#### Start etcd on the new machine, using the same configuration and the copy of the data directory
```
$ ssh core@10.0.1.13
$ ssh 10.0.1.13
```
```
$ tar -xzvf node1.etcd.tar.gz -C /var/lib/etcd
$ tar -xzvf infra1.etcd.tar.gz -C %data_dir%
```
```
etcd -name node1 \
etcd -name infra1 \
-listen-peer-urls http://10.0.1.13:2380 \
-listen-client-urls http://10.0.1.13:2379,http://127.0.0.1:2379 \
-advertise-client-urls http://10.0.1.13:2379,http://127.0.0.1:2379
@@ -149,8 +149,8 @@ The first step of the recovery is to backup the data directory on a functioning
```sh
etcdctl backup \
--data-dir /var/lib/etcd \
--backup-dir /tmp/etcd_backup
--data-dir %data_dir% \
--backup-dir %backup_data_dir%
```
This command will rewrite some of the metadata contained in the backup (specifically, the node ID and cluster ID), which means that the node will lose its former identity. In order to recreate a cluster from the backup, you will need to start a new, single-node cluster. The metadata is rewritten to prevent the new node from inadvertently being joined onto an existing cluster.
@@ -161,7 +161,7 @@ To restore a backup using the procedure created above, start etcd with the `-for
```sh
etcd \
-data-dir=/tmp/etcd_backup \
-data-dir=%backup_data_dir% \
-force-new-cluster \
...
```
@@ -172,10 +172,10 @@ Once you have verified that etcd has started successfully, shut it down and move
```sh
pkill etcd
rm -fr /var/lib/etcd
mv /tmp/etcd_backup /var/lib/etcd
rm -fr %data_dir%
mv %backup_data_dir% %data_dir%
etcd \
-data-dir=/var/lib/etcd \
-data-dir=%data_dir% \
...
```

View File

@@ -356,6 +356,13 @@ So the first watch after the get should be:
curl 'http://127.0.0.1:2379/v2/keys/foo?wait=true&waitIndex=2008'
```
#### Connection being closed prematurely
The server may close a long polling connection before emitting any events.
This can happend due to a timeout or the server being shutdown.
Since the HTTP header is sent immediately upon accepting the connection, the response will be seen as empty: `200 OK` and empty body.
The clients should be prepared to deal with this scenario and retry the watch.
### Atomically Creating In-Order Keys
Using `POST` on a directory, you can create keys with key names that are created in-order.

View File

@@ -24,49 +24,6 @@ https://github.com/coreos/etcd/blob/master/Documentation/configuration.md.
The default data dir location has changed from {$hostname}.etcd to {name}.etcd.
## Data Directory Migration
The disk format within the data directory changed with etcd 2.0.
If you run etcd 2.0 on an etcd 0.4 data directory it will automatically migrate the data and start.
You will want to coordinate this upgrade by walking through each of your machines in the cluster, stopping etcd 0.4 and then starting etcd 2.0.
If you would rather manually do the migration, to test it out first in another environment, you can use the [migration tool doc][migrationtooldoc].
[migrationtooldoc]: https://github.com/coreos/etcd/blob/master/tools/etcd-migrate/README.md
## Snapshot Migration
If you are only interested in the data in etcd you can migrate a snapshot of your data from a v0.4.9+ cluster into a new etcd 2.0 cluster using a snapshot migration.
The advantage of this method is that you are directly dumping only the etcd data so you can run your old and new cluster side-by-side, snapshot the data, import it and then point your applications at this cluster.
The disadvantage is that the etcd indexes of your data will change which may confuse applications that use etcd.
To get started get the newest data snapshot from the 0.4.9+ cluster:
```
curl http://cluster.example.com:4001/v2/migration/snapshot > backup.snap
```
Now, import the snapshot into your new cluster:
```
etcdctl -C new_cluster.example.com import --snap backup.snap
```
If you have a large amount of data, you can specify more concurrent works to copy data in parallel by using `-c` flag.
If you have hidden keys to copy, you can use `--hidden` flag to specify.
And the data will quickly copy into the new cluster:
```
entering dir: /
entering dir: /foo
entering dir: /foo/bar
copying key: /foo/bar/1 1
entering dir: /
entering dir: /foo2
entering dir: /foo2/bar2
copying key: /foo2/bar2/2 2
```
## Key-Value API
### Read consistency flag

View File

@@ -122,6 +122,8 @@ There two methods that can be used for discovery:
### etcd Discovery
To better understand the design about discovery service protocol, we suggest you read [this](./discovery_protocol.md).
#### Lifetime of a Discovery URL
A discovery URL identifies a unique etcd cluster. Instead of reusing a discovery URL, you should always create discovery URLs for new clusters.
@@ -144,6 +146,8 @@ If you bootstrap an etcd cluster using discovery service with more than the expe
The URL you will use in this case will be `https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83` and the etcd members will use the `https://myetcd.local/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83` directory for registration as they start.
Each member must have a different name flag specified. Or discovery will fail due to duplicated name.
Now we start etcd with those relevant flags for each member:
```
@@ -194,6 +198,8 @@ ETCD_DISCOVERY=https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573d
-discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de
```
Each member must have a different name flag specified. Or discovery will fail due to duplicated name.
Now we start etcd with those relevant flags for each member:
```
@@ -296,6 +302,8 @@ infra2.example.com. 300 IN A 10.0.1.12
etcd cluster members can listen on domain names or IP address, the bootstrap process will resolve DNS A records.
The resolved address in `-initial-advertise-peer-urls` *must match* one of the resolved addresses in the SRV targets. The etcd member reads the resolved address to find out if it belongs to the cluster defined in the SRV records.
```
$ etcd -name infra0 \
-discovery-srv example.com \
@@ -372,6 +380,10 @@ DNS SRV records can also be used to configure the list of peers for an etcd serv
$ etcd --proxy on -discovery-srv example.com
```
#### Error Cases
You might see the an error like `cannot find local etcd $name from SRV records.`. That means the etcd member fails to find itself from the cluster defined in SRV records. The resolved address in `-initial-advertise-peer-urls` *must match* one of the resolved addresses in the SRV targets.
# 0.4 to 2.0+ Migration Guide
In etcd 2.0 we introduced the ability to listen on more than one address and to advertise multiple addresses. This makes using etcd easier when you have complex networking, such as private and public networks on various cloud providers.

View File

@@ -28,7 +28,7 @@ To start etcd automatically using custom settings at startup in Linux, using a [
+ default: "100"
##### -election-timeout
+ Time (in milliseconds) for an election to timeout.
+ Time (in milliseconds) for an election to timeout. See [Documentation/tuning.md](tuning.md#time-parameters) for details.
+ default: "1000"
##### -listen-peer-urls
@@ -193,6 +193,12 @@ Follow the instructions when using these flags.
+ Force to create a new one-member cluster. It commits configuration changes in force to remove all existing members in the cluster and add itself. It needs to be set to [restore a backup][restore].
+ default: false
### Experimental Flags
##### -experimental-v3demo
+ Enable experimental [v3 demo API](rfc/v3api.proto).
+ default: false
### Miscellaneous Flags
##### -version

View File

@@ -0,0 +1,109 @@
# Discovery Service Protocol
Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.
Discovery service protocol is _only_ used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.
The protocol uses a new discovery token to bootstrap one _unique_ etcd cluster. Remember that one discovery token can represent only one etcd cluster. As long as discovery protocol on this token starts, even if fails halfway, it must not be used to bootstrap another etcd cluster.
The rest of this article will walk through the discovery process with examples that correspond to a self-hosted discovery cluster. The public discovery service, discovery.etcd.io, functions the same way, but with a layer of polish to abstract away ugly URLs, generate UUIDs automatically, and provide some protections against excessive requests. At its core, the public discovery service still uses an etcd cluster as the data store as described in this document.
## The Protocol Workflow
The idea of discovery protocol is to use an internal etcd cluster to coordinate bootstrap of a new cluster. First, all new members interact with discovery service and help to generate the expected member list. Then each new member bootstraps its server using this list, which performs the same functionality as -initial-cluster flag.
In the following example workflow, we will list each step of protocol in curl format for ease of understanding.
By convention the etcd discovery protocol uses the key prefix `_etcd/registry`. If `http://example.com` hosts a etcd cluster for discovery service, a full URL to discovery keyspace will be `http://example.com/v2/keys/_etcd/registry`. We will use this as the URL prefix in the example.
### Creating a New Discovery Token
Generate a unique token that will identify the new cluster. This will be used as a unique prefix in discovery keyspace in the following steps. An easy way to do this is to use `uuidgen`:
```
UUID=$(uuidgen)
```
### Specifying the Expected Cluster Size
You need to specify the expected cluster size for this discovery token. The size is used by the discovery service to know when it has found all members that will initially form the cluster.
```
curl -X PUT http://example.com/v2/keys/_etcd/registry/${UUID}/_config/size -d value=${cluster_size}
```
Usually the cluster size is 3, 5 or 7. Check [optimal cluster size](admin_guide.md#optimal-cluster-size) for more details.
### Bringing up etcd Processes
Now that you have your discovery URL, you can use it as `-discovery` flag and bring up etcd processes. Every etcd process will follow this next few steps internally if given a `-discovery` flag.
### Registering itself
The first thing for etcd process is to register itself into the discovery URL as a member. This is done by creating member ID as a key in the discovery URL.
```
curl -X PUT http://example.com/v2/keys/_etcd/registry/${UUID}/${member_id}?prevExist=false -d value="${member_name}=${member_peer_url_1}&${member_name}=${member_peer_url_2}"
```
### Checking the Status
It checks the expected cluster size and registration status in discovery URL, and decides what the next action is.
```
curl -X GET http://example.com/v2/keys/_etcd/registry/${UUID}/_config/size
curl -X GET http://example.com/v2/keys/_etcd/registry/${UUID}
```
If registered members are still not enough, it will wait for left members to appear.
If the number of registered members is bigger than the expected size N, it treats the first N registered members as the member list for the cluster. If the member itself is in the member list, the discovery procedure succeeds and it fetches all peers through the member list. If it is not in the member list, the discovery procedure finishes with the failure that the cluster has been full.
In etcd implementation, the member may check the cluster status even before registering itself. So it could fail quickly if the cluster has been full.
### Waiting for All Members
The wait process is described in details [here](https://github.com/coreos/etcd/blob/master/Documentation/api.md#waiting-for-a-change).
```
curl -X GET http://example.com/v2/keys/_etcd/registry/${UUID}?wait=true&waitIndex=${current_etcd_index}
```
It keeps waiting until finding all members.
## Public Discovery Service
CoreOS Inc. hosts a public discovery service at https://discovery.etcd.io/ , which provides some nice features for ease of use.
### Mask Key Prefix
Public discovery service will redirect `https://discovery.etcd.io/${UUID}` to etcd cluster behind for the key at `/v2/keys/_etcd/registry`. It masks register key prefix for short and readable discovery url.
### Get new token
```
GET /new
Sent query:
size=${cluster_size}
Possible status codes:
200 OK
400 Bad Request
200 Body:
generated discovery url
```
The generation process in the service follows the step from [Creating a New Discovery Token](#creating-a-new-discovery-token) to [Specifying the Expected Cluster Size](#specifying-the-expected-cluster-size).
### Check Discovery Status
```
GET /${UUID}
```
You can check the status for this discovery token, including the machines that have been registered, by requesting the value of the UUID.
### Open-source repository
The repository is located at https://github.com/coreos/discovery.etcd.io. You could use it to build your own public discovery service.

View File

@@ -45,6 +45,7 @@
**C libraries**
- [jdarcy/etcd-api](https://github.com/jdarcy/etcd-api) - Supports v2
- [shafreeck/cetcd](https://github.com/shafreeck/cetcd) - Supports v2
**C++ libraries**
- [edwardcapriolo/etcdcpp](https://github.com/edwardcapriolo/etcdcpp) - Supports v2

View File

@@ -0,0 +1,43 @@
## Reporting Bugs
If you find bugs or documentation mistakes in etcd project, please let us know by [opening an issue](https://github.com/coreos/etcd/issues/new). We treat bugs and mistakes very seriously and believe no issue is too small. Before creating a bug report, please check there that one does not already exist.
To make your bug report accurate and easy to understand, please try to create bug reports that are:
- Specific. Include as much details as possible: which version, what environment, what configuration, etc. You can also attach etcd log (the starting log with etcd configuration is especially important).
- Reproducible. Include the steps to reproduce the problem. We understand some issues might be hard to reproduce, please includes the steps that might lead to the problem. You can also attach the affected etcd data dir and stack strace to the bug report.
- Isolated. Please try to isolate and reproduce the bug with minimum dependencies. It would significantly slow down the speed to fix a bug if too many dependencies are involved in a bug report. Debugging external systems that rely on etcd is out of scope, but we are happy to point you in the right direction or help you interact with etcd in the correct manner.
- Unique. Do not duplicate existing bug report.
- Scoped. One bug per report. Do not follow up with another bug inside one report.
You might also want to read [Elika Etemads article on filing good bug reports](http://fantasai.inkedblade.net/style/talks/filing-good-bugs/) before creating a bug report.
We might ask you for further information to locate a bug. A duplicated bug report will be closed.
## Frequently Asked Questions
### How to get stack trace
``` bash
$ kill -QUIT $PID
```
### How to get etcd version
``` bash
$ etcd --version
```
### How to get etcd configuration and log when it runs as systemd service etcd2.service
``` bash
$ sudo systemctl cat etcd2
$ sudo journalctl -u etcd2
```
Due to an upstream systemd bug, journald may miss the last few log lines when its process exit. If journalctl tells you that etcd stops without fatal or panic message, you could try `sudo journalctl -f -t etcd2` to get full log.

View File

@@ -14,14 +14,14 @@
- more efficient/ low cost keep alive
- a logical group of TTL keys
5. Replace CAS/CAD with multi-object Tnx
5. Replace CAS/CAD with multi-object Txn
- MUCH MORE powerful and flexible
6. Support efficient watching with multiple ranges
7. RPC API supports the completed set of APIs.
- more efficient than JSON/HTTP
- additional tnx/lease support
- additional txn/lease support
8. HTTP API supports a subset of APIs.
- easy for people to try out etcd
@@ -97,9 +97,9 @@ RangeResponse {
}
```
#### Finish a tnx (assume we have foo0=bar0, foo1=bar1)
#### Finish a txn (assume we have foo0=bar0, foo1=bar1)
```
Tnx(TnxRequest {
Txn(TxnRequest {
// mod_index of foo0 is equal to 1, mod_index of foo1 is greater than 1
compare = {
{compareType = equal, key = foo0, mod_index = 1},
@@ -111,7 +111,7 @@ Tnx(TnxRequest {
failure = {PutRequest { key = foo2, value = failure }},
)
TnxResponse {
TxnResponse {
cluster_id = 0x1000,
member_id = 0x1,
index = 3,

View File

@@ -15,10 +15,10 @@ service etcd {
// and generates one event in the event history.
rpc DeleteRange(DeleteRangeRequest) returns (DeleteRangeResponse) {}
// Tnx processes all the requests in one transaction.
// A tnx request increases the index of the store,
// Txn processes all the requests in one transaction.
// A txn request increases the index of the store,
// and generates events with the same index in the event history.
rpc Tnx(TnxRequest) returns (TnxResponse) {}
rpc Txn(TxnRequest) returns (TxnResponse) {}
// Watch watches the events happening or happened in etcd. Both input and output
// are stream. One watch rpc can watch for multiple ranges and get a stream of
@@ -41,10 +41,10 @@ service etcd {
// LeaseAttach attaches keys with a lease.
rpc LeaseAttach(LeaseAttachRequest) returns (LeaseAttachResponse) {}
// LeaseTnx likes Tnx. It has two addition success and failure LeaseAttachRequest list.
// If the Tnx is successful, then the success list will be executed. Or the failure list
// LeaseTxn likes Txn. It has two addition success and failure LeaseAttachRequest list.
// If the Txn is successful, then the success list will be executed. Or the failure list
// will be executed.
rpc LeaseTnx(LeaseTnxRequest) returns (LeaseTnxResponse) {}
rpc LeaseTxn(LeaseTxnRequest) returns (LeaseTxnResponse) {}
// KeepAlive keeps the lease alive.
rpc LeaseKeepAlive(stream LeaseKeepAliveRequest) returns (stream LeaseKeepAliveResponse) {}
@@ -69,7 +69,7 @@ message RangeRequest {
// limit the number of keys returned.
optional int64 limit = 3;
// the response will be consistent with previous request with same token if the token is
// given and is vaild.
// given and is valid.
optional bytes consistent_token = 4;
}
@@ -157,13 +157,13 @@ message Compare {
// if guard evaluates to
// true.
// 3. A list of database operations called f op. Like t op, but executed if guard evaluates to false.
message TnxRequest {
message TxnRequest {
repeated Compare compare = 1;
repeated RequestUnion success = 2;
repeated RequestUnion failure = 3;
}
message TnxResponse {
message TxnResponse {
optional ResponseHeader header = 1;
optional bool succeeded = 2;
repeated ResponseUnion responses = 3;
@@ -240,15 +240,15 @@ message LeaseRevokeResponse {
optional ResponseHeader header = 1;
}
message LeaseTnxRequest {
optional TnxRequest request = 1;
message LeaseTxnRequest {
optional TxnRequest request = 1;
repeated LeaseAttachRequest success = 2;
repeated LeaseAttachRequest failure = 3;
}
message LeaseTnxResponse {
message LeaseTxnResponse {
optional ResponseHeader header = 1;
optional TnxResponse response = 2;
optional TxnResponse response = 2;
repeated LeaseAttachResponse attach_responses = 3;
}

View File

@@ -4,6 +4,8 @@ etcd comes with support for incremental runtime reconfiguration, which allows us
Reconfiguration requests can only be processed when the the majority of the cluster members are functioning. It is **highly recommended** to always have a cluster size greater than two in production. It is unsafe to remove a member from a two member cluster. The majority of a two member cluster is also two. If there is a failure during the removal process, the cluster might not able to make progress and need to [restart from majority failure][majority failure].
To better understand the design behind runtime reconfiguration, we suggest you read [this](runtime-reconf-design.md).
[majority failure]: #restart-cluster-from-majority-failure
## Reconfiguration Use Cases

View File

@@ -0,0 +1,47 @@
### Design of Runtime Reconfiguration
Runtime reconfiguration is one of the hardest and most error prone features in a distributed system, especially in a consensus based system like etcd.
Read on to learn about the design of etcd's runtime reconfiguration commands and how we tackled these problems.
### Two Phase Config Changes Keep you Safe
In etcd, every runtime reconfiguration has to go through [two phases](Documentation/runtime-configuration.md#add-a-new-member) for safety reasons. For example, to add a member you need to first inform cluster of new configuration and then start the new member.
Phase 1 - Inform cluster of new configuration
To add a member into etcd cluster, you need to make an API call to request a new member to be added to the cluster. And this is only way that you can add a new member into an existing cluster. The API call returns when the cluster agrees on the configuration change.
Phase 2 - Start new member
To join the etcd member into the existing cluster, you need to specify the correct `initial-cluster` and set `initial-cluster-state` to `existing`. When the member starts, it will contact the existing cluster first and verify the current cluster configuration matches the expected one specified in `initial-cluster`. When the new member successfully starts, you know your cluster reached the expected configuration.
By splitting the process into two discrete phases users are forced to be explicit regarding cluster membership changes. This actually gives users more flexibility and makes things easier to reason about. For example, if there is an attempt to add a new member with the same ID as an existing member in an etcd cluster, the action will fail immediately during phase one without impacting the running cluster. Similar protection is provided to prevent adding new members by mistake. If a new etcd member attempts to join the cluster before the cluster has accepted the configuration change,, it will not be accepted by the cluster.
Without the explicit workflow around cluster membership etcd would be vulnerable to unexpected cluster membership changes. For example, if etcd is running under an init system such as systemd, etcd would be restarted after being removed via the membership API, and attempt to rejoin the cluster on startup. This cycle would continue every time a member is removed via the API and systemd is set to restart etcd after failing, which is unexpected.
We think runtime reconfiguration should be a low frequent operation. We made the decision to keep it explicit and user-driven to ensure configuration safety and keep your cluster always running smoothly under your control.
### Permanent Loss of Quorum Requires New Cluster
If a cluster permanently loses a majority of its members, a new cluster will need to be started from an old data directory to recover the previous state.
It is entirely possible to force removing the failed members from the existing cluster to recover. However, we decided not to support this method since it bypasses the normal consensus committing phase, which is unsafe. If the member to remove is not actually dead or you force to remove different members through different members in the same cluster, you will end up with diverged cluster with same clusterID. This is very dangerous and hard to debug/fix afterwards.
If you have a correct deployment, the possibility of permanent majority lose is very low. But it is a severe enough problem that worth special care. We strongly suggest you to read the [disaster recovery documentation](admin_guide.md#disaster-recovery) and prepare for permanent majority lose before you put etcd into production.
### Do Not Use Public Discovery Service For Runtime Reconfiguration
The public discovery service should only be used for bootstrapping a cluster. To join member into an existing cluster, you should use runtime reconfiguration API.
Discovery service is designed for bootstrapping an etcd cluster in the cloud environment, when you do not know the IP addresses of all the members beforehand. After you successfully bootstrap a cluster, the IP addresses of all the members are known. Technically, you should not need the discovery service any more.
It seems that using public discovery service is a convenient way to do runtime reconfiguration, after all discovery service already has all the cluster configuration information. However relying on public discovery service brings troubles:
1. it introduces a external dependencies for the entire life-cycle of your cluster, not just bootstrap time. If there is a network issue between your cluster and public discover service, your cluster will suffer from it.
2. public discovery service must reflect correct runtime configuration of your cluster during it life-cycle. It has to provide security mechanism to avoid bad actions, and it is hard.
3. public discovery service has to keep tens of thousands of cluster configurations. Our public discovery service backend is not ready for that workload.
If you want to have a discovery service that supports runtime reconfiguration, the best choice is to build your private one.

View File

@@ -25,6 +25,8 @@ The election timeout should be set based on the heartbeat interval and your netw
Election timeouts should be at least 10 times your ping time so it can account for variance in your network.
For example, if the ping time between your nodes is 10ms then you should have at least a 100ms election timeout.
The upper limit of election timeout is 50000ms, which should only be used when deploying global etcd cluster. First, 5s is the upper limit of average global round-trip time. A reasonable round-trip time for the continental united states is 130ms, and the time between US and japan is around 350-400ms. Because package gets delayed a lot, and network situation may be terrible, 5s is a safe value for it. Then, because election timeout should be an order of magnitude bigger than broadcast time, 50s becomes its maximum.
You should also set your election timeout to at least 5 to 10 times your heartbeat interval to account for variance in leader replication.
For a heartbeat interval of 50ms you should set your election timeout to at least 250ms - 500ms.

26
Godeps/Godeps.json generated
View File

@@ -32,18 +32,28 @@
"Comment": "1.2.0-26-gf7ebb76",
"Rev": "f7ebb761e83e21225d1d8954fde853bf8edd46c4"
},
{
"ImportPath": "github.com/coreos/go-etcd/etcd",
"Comment": "v2.0.0-13-g4cceaf7",
"Rev": "4cceaf7283b76f27c4a732b20730dcdb61053bf5"
},
{
"ImportPath": "github.com/coreos/go-semver/semver",
"Rev": "568e959cd89871e61434c1143528d9162da89ef2"
},
{
"ImportPath": "github.com/coreos/go-systemd/daemon",
"Comment": "v3-6-gcea488b",
"Rev": "cea488b4e6855fee89b6c22a811e3c5baca861b6"
},
{
"ImportPath": "github.com/coreos/go-systemd/journal",
"Comment": "v3-6-gcea488b",
"Rev": "cea488b4e6855fee89b6c22a811e3c5baca861b6"
},
{
"ImportPath": "github.com/coreos/go-systemd/util",
"Comment": "v3-6-gcea488b",
"Rev": "cea488b4e6855fee89b6c22a811e3c5baca861b6"
},
{
"ImportPath": "github.com/coreos/pkg/capnslog",
"Rev": "99f6e6b8f8ea30b0f82769c1411691c44a66d015"
"Rev": "6b7d4a60e79f6aa57a27591288f28e1dcd47a3bc"
},
{
"ImportPath": "github.com/gogo/protobuf/proto",
@@ -98,8 +108,8 @@
"Rev": "9cc77fa25329013ce07362c7742952ff887361f2"
},
{
"ImportPath": "github.com/ugorji/go/codec",
"Rev": "821cda7e48749cacf7cad2c6ed01e96457ca7e9d"
"ImportPath": "github.com/xiang90/probing",
"Rev": "6a0cc1ae81b4cc11db5e491e030e4b98fba79c19"
},
{
"ImportPath": "golang.org/x/crypto/bcrypt",

View File

@@ -1,23 +0,0 @@
package etcd
// Add a new directory with a random etcd-generated key under the given path.
func (c *Client) AddChildDir(key string, ttl uint64) (*Response, error) {
raw, err := c.post(key, "", ttl)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}
// Add a new file with a random etcd-generated key under the given path.
func (c *Client) AddChild(key string, value string, ttl uint64) (*Response, error) {
raw, err := c.post(key, value, ttl)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}

View File

@@ -1,73 +0,0 @@
package etcd
import "testing"
func TestAddChild(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("fooDir", true)
c.Delete("nonexistentDir", true)
}()
c.CreateDir("fooDir", 5)
_, err := c.AddChild("fooDir", "v0", 5)
if err != nil {
t.Fatal(err)
}
_, err = c.AddChild("fooDir", "v1", 5)
if err != nil {
t.Fatal(err)
}
resp, err := c.Get("fooDir", true, false)
// The child with v0 should proceed the child with v1 because it's added
// earlier, so it should have a lower key.
if !(len(resp.Node.Nodes) == 2 && (resp.Node.Nodes[0].Value == "v0" && resp.Node.Nodes[1].Value == "v1")) {
t.Fatalf("AddChild 1 failed. There should be two chlidren whose values are v0 and v1, respectively."+
" The response was: %#v", resp)
}
// Creating a child under a nonexistent directory should succeed.
// The directory should be created.
resp, err = c.AddChild("nonexistentDir", "foo", 5)
if err != nil {
t.Fatal(err)
}
}
func TestAddChildDir(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("fooDir", true)
c.Delete("nonexistentDir", true)
}()
c.CreateDir("fooDir", 5)
_, err := c.AddChildDir("fooDir", 5)
if err != nil {
t.Fatal(err)
}
_, err = c.AddChildDir("fooDir", 5)
if err != nil {
t.Fatal(err)
}
resp, err := c.Get("fooDir", true, false)
// The child with v0 should proceed the child with v1 because it's added
// earlier, so it should have a lower key.
if !(len(resp.Node.Nodes) == 2 && (len(resp.Node.Nodes[0].Nodes) == 0 && len(resp.Node.Nodes[1].Nodes) == 0)) {
t.Fatalf("AddChildDir 1 failed. There should be two chlidren whose values are v0 and v1, respectively."+
" The response was: %#v", resp)
}
// Creating a child under a nonexistent directory should succeed.
// The directory should be created.
resp, err = c.AddChildDir("nonexistentDir", 5)
if err != nil {
t.Fatal(err)
}
}

View File

@@ -1,490 +0,0 @@
package etcd
import (
"crypto/tls"
"crypto/x509"
"encoding/json"
"errors"
"io"
"io/ioutil"
"math/rand"
"net"
"net/http"
"net/url"
"os"
"path"
"strings"
"time"
)
// See SetConsistency for how to use these constants.
const (
// Using strings rather than iota because the consistency level
// could be persisted to disk, so it'd be better to use
// human-readable values.
STRONG_CONSISTENCY = "STRONG"
WEAK_CONSISTENCY = "WEAK"
)
const (
defaultBufferSize = 10
)
func init() {
rand.Seed(int64(time.Now().Nanosecond()))
}
type Config struct {
CertFile string `json:"certFile"`
KeyFile string `json:"keyFile"`
CaCertFile []string `json:"caCertFiles"`
DialTimeout time.Duration `json:"timeout"`
Consistency string `json:"consistency"`
}
type credentials struct {
username string
password string
}
type Client struct {
config Config `json:"config"`
cluster *Cluster `json:"cluster"`
httpClient *http.Client
credentials *credentials
transport *http.Transport
persistence io.Writer
cURLch chan string
// CheckRetry can be used to control the policy for failed requests
// and modify the cluster if needed.
// The client calls it before sending requests again, and
// stops retrying if CheckRetry returns some error. The cases that
// this function needs to handle include no response and unexpected
// http status code of response.
// If CheckRetry is nil, client will call the default one
// `DefaultCheckRetry`.
// Argument cluster is the etcd.Cluster object that these requests have been made on.
// Argument numReqs is the number of http.Requests that have been made so far.
// Argument lastResp is the http.Responses from the last request.
// Argument err is the reason of the failure.
CheckRetry func(cluster *Cluster, numReqs int,
lastResp http.Response, err error) error
}
// NewClient create a basic client that is configured to be used
// with the given machine list.
func NewClient(machines []string) *Client {
config := Config{
// default timeout is one second
DialTimeout: time.Second,
Consistency: WEAK_CONSISTENCY,
}
client := &Client{
cluster: NewCluster(machines),
config: config,
}
client.initHTTPClient()
client.saveConfig()
return client
}
// NewTLSClient create a basic client with TLS configuration
func NewTLSClient(machines []string, cert, key, caCert string) (*Client, error) {
// overwrite the default machine to use https
if len(machines) == 0 {
machines = []string{"https://127.0.0.1:4001"}
}
config := Config{
// default timeout is one second
DialTimeout: time.Second,
Consistency: WEAK_CONSISTENCY,
CertFile: cert,
KeyFile: key,
CaCertFile: make([]string, 0),
}
client := &Client{
cluster: NewCluster(machines),
config: config,
}
err := client.initHTTPSClient(cert, key)
if err != nil {
return nil, err
}
err = client.AddRootCA(caCert)
client.saveConfig()
return client, nil
}
// NewClientFromFile creates a client from a given file path.
// The given file is expected to use the JSON format.
func NewClientFromFile(fpath string) (*Client, error) {
fi, err := os.Open(fpath)
if err != nil {
return nil, err
}
defer func() {
if err := fi.Close(); err != nil {
panic(err)
}
}()
return NewClientFromReader(fi)
}
// NewClientFromReader creates a Client configured from a given reader.
// The configuration is expected to use the JSON format.
func NewClientFromReader(reader io.Reader) (*Client, error) {
c := new(Client)
b, err := ioutil.ReadAll(reader)
if err != nil {
return nil, err
}
err = json.Unmarshal(b, c)
if err != nil {
return nil, err
}
if c.config.CertFile == "" {
c.initHTTPClient()
} else {
err = c.initHTTPSClient(c.config.CertFile, c.config.KeyFile)
}
if err != nil {
return nil, err
}
for _, caCert := range c.config.CaCertFile {
if err := c.AddRootCA(caCert); err != nil {
return nil, err
}
}
return c, nil
}
// Override the Client's HTTP Transport object
func (c *Client) SetTransport(tr *http.Transport) {
c.httpClient.Transport = tr
c.transport = tr
}
func (c *Client) SetCredentials(username, password string) {
c.credentials = &credentials{username, password}
}
func (c *Client) Close() {
c.transport.DisableKeepAlives = true
c.transport.CloseIdleConnections()
}
// initHTTPClient initializes a HTTP client for etcd client
func (c *Client) initHTTPClient() {
c.transport = &http.Transport{
Dial: c.dial,
TLSClientConfig: &tls.Config{
InsecureSkipVerify: true,
},
}
c.httpClient = &http.Client{Transport: c.transport}
}
// initHTTPClient initializes a HTTPS client for etcd client
func (c *Client) initHTTPSClient(cert, key string) error {
if cert == "" || key == "" {
return errors.New("Require both cert and key path")
}
tlsCert, err := tls.LoadX509KeyPair(cert, key)
if err != nil {
return err
}
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{tlsCert},
InsecureSkipVerify: true,
}
tr := &http.Transport{
TLSClientConfig: tlsConfig,
Dial: c.dial,
}
c.httpClient = &http.Client{Transport: tr}
return nil
}
// SetPersistence sets a writer to which the config will be
// written every time it's changed.
func (c *Client) SetPersistence(writer io.Writer) {
c.persistence = writer
}
// SetConsistency changes the consistency level of the client.
//
// When consistency is set to STRONG_CONSISTENCY, all requests,
// including GET, are sent to the leader. This means that, assuming
// the absence of leader failures, GET requests are guaranteed to see
// the changes made by previous requests.
//
// When consistency is set to WEAK_CONSISTENCY, other requests
// are still sent to the leader, but GET requests are sent to a
// random server from the server pool. This reduces the read
// load on the leader, but it's not guaranteed that the GET requests
// will see changes made by previous requests (they might have not
// yet been committed on non-leader servers).
func (c *Client) SetConsistency(consistency string) error {
if !(consistency == STRONG_CONSISTENCY || consistency == WEAK_CONSISTENCY) {
return errors.New("The argument must be either STRONG_CONSISTENCY or WEAK_CONSISTENCY.")
}
c.config.Consistency = consistency
return nil
}
// Sets the DialTimeout value
func (c *Client) SetDialTimeout(d time.Duration) {
c.config.DialTimeout = d
}
// AddRootCA adds a root CA cert for the etcd client
func (c *Client) AddRootCA(caCert string) error {
if c.httpClient == nil {
return errors.New("Client has not been initialized yet!")
}
certBytes, err := ioutil.ReadFile(caCert)
if err != nil {
return err
}
tr, ok := c.httpClient.Transport.(*http.Transport)
if !ok {
panic("AddRootCA(): Transport type assert should not fail")
}
if tr.TLSClientConfig.RootCAs == nil {
caCertPool := x509.NewCertPool()
ok = caCertPool.AppendCertsFromPEM(certBytes)
if ok {
tr.TLSClientConfig.RootCAs = caCertPool
}
tr.TLSClientConfig.InsecureSkipVerify = false
} else {
ok = tr.TLSClientConfig.RootCAs.AppendCertsFromPEM(certBytes)
}
if !ok {
err = errors.New("Unable to load caCert")
}
c.config.CaCertFile = append(c.config.CaCertFile, caCert)
c.saveConfig()
return err
}
// SetCluster updates cluster information using the given machine list.
func (c *Client) SetCluster(machines []string) bool {
success := c.internalSyncCluster(machines)
return success
}
func (c *Client) GetCluster() []string {
return c.cluster.Machines
}
// SyncCluster updates the cluster information using the internal machine list.
// If no members are found, the intenral machine list is left untouched.
func (c *Client) SyncCluster() bool {
return c.internalSyncCluster(c.cluster.Machines)
}
// internalSyncCluster syncs cluster information using the given machine list.
func (c *Client) internalSyncCluster(machines []string) bool {
// comma-separated list of machines in the cluster.
members := ""
for _, machine := range machines {
httpPath := c.createHttpPath(machine, path.Join(version, "members"))
resp, err := c.httpClient.Get(httpPath)
if err != nil {
// try another machine in the cluster
continue
}
if resp.StatusCode != http.StatusOK { // fall-back to old endpoint
httpPath := c.createHttpPath(machine, path.Join(version, "machines"))
resp, err := c.httpClient.Get(httpPath)
if err != nil {
// try another machine in the cluster
continue
}
b, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
// try another machine in the cluster
continue
}
members = string(b)
} else {
b, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
// try another machine in the cluster
continue
}
var mCollection memberCollection
if err := json.Unmarshal(b, &mCollection); err != nil {
// try another machine
continue
}
urls := make([]string, 0)
for _, m := range mCollection {
urls = append(urls, m.ClientURLs...)
}
members = strings.Join(urls, ",")
}
// We should never do an empty cluster update.
if members == "" {
continue
}
// update Machines List
c.cluster.updateFromStr(members)
logger.Debug("sync.machines ", c.cluster.Machines)
c.saveConfig()
return true
}
return false
}
// createHttpPath creates a complete HTTP URL.
// serverName should contain both the host name and a port number, if any.
func (c *Client) createHttpPath(serverName string, _path string) string {
u, err := url.Parse(serverName)
if err != nil {
panic(err)
}
u.Path = path.Join(u.Path, _path)
if u.Scheme == "" {
u.Scheme = "http"
}
return u.String()
}
// dial attempts to open a TCP connection to the provided address, explicitly
// enabling keep-alives with a one-second interval.
func (c *Client) dial(network, addr string) (net.Conn, error) {
conn, err := net.DialTimeout(network, addr, c.config.DialTimeout)
if err != nil {
return nil, err
}
tcpConn, ok := conn.(*net.TCPConn)
if !ok {
return nil, errors.New("Failed type-assertion of net.Conn as *net.TCPConn")
}
// Keep TCP alive to check whether or not the remote machine is down
if err = tcpConn.SetKeepAlive(true); err != nil {
return nil, err
}
if err = tcpConn.SetKeepAlivePeriod(time.Second); err != nil {
return nil, err
}
return tcpConn, nil
}
func (c *Client) OpenCURL() {
c.cURLch = make(chan string, defaultBufferSize)
}
func (c *Client) CloseCURL() {
c.cURLch = nil
}
func (c *Client) sendCURL(command string) {
go func() {
select {
case c.cURLch <- command:
default:
}
}()
}
func (c *Client) RecvCURL() string {
return <-c.cURLch
}
// saveConfig saves the current config using c.persistence.
func (c *Client) saveConfig() error {
if c.persistence != nil {
b, err := json.Marshal(c)
if err != nil {
return err
}
_, err = c.persistence.Write(b)
if err != nil {
return err
}
}
return nil
}
// MarshalJSON implements the Marshaller interface
// as defined by the standard JSON package.
func (c *Client) MarshalJSON() ([]byte, error) {
b, err := json.Marshal(struct {
Config Config `json:"config"`
Cluster *Cluster `json:"cluster"`
}{
Config: c.config,
Cluster: c.cluster,
})
if err != nil {
return nil, err
}
return b, nil
}
// UnmarshalJSON implements the Unmarshaller interface
// as defined by the standard JSON package.
func (c *Client) UnmarshalJSON(b []byte) error {
temp := struct {
Config Config `json:"config"`
Cluster *Cluster `json:"cluster"`
}{}
err := json.Unmarshal(b, &temp)
if err != nil {
return err
}
c.cluster = temp.Cluster
c.config = temp.Config
return nil
}

View File

@@ -1,108 +0,0 @@
package etcd
import (
"encoding/json"
"fmt"
"net"
"net/url"
"os"
"testing"
)
// To pass this test, we need to create a cluster of 3 machines
// The server should be listening on localhost:4001, 4002, 4003
func TestSync(t *testing.T) {
fmt.Println("Make sure there are three nodes at 0.0.0.0:4001-4003")
// Explicit trailing slash to ensure this doesn't reproduce:
// https://github.com/coreos/go-etcd/issues/82
c := NewClient([]string{"http://127.0.0.1:4001/"})
success := c.SyncCluster()
if !success {
t.Fatal("cannot sync machines")
}
for _, m := range c.GetCluster() {
u, err := url.Parse(m)
if err != nil {
t.Fatal(err)
}
if u.Scheme != "http" {
t.Fatal("scheme must be http")
}
host, _, err := net.SplitHostPort(u.Host)
if err != nil {
t.Fatal(err)
}
if host != "localhost" {
t.Fatal("Host must be localhost")
}
}
badMachines := []string{"abc", "edef"}
success = c.SetCluster(badMachines)
if success {
t.Fatal("should not sync on bad machines")
}
goodMachines := []string{"127.0.0.1:4002"}
success = c.SetCluster(goodMachines)
if !success {
t.Fatal("cannot sync machines")
} else {
fmt.Println(c.cluster.Machines)
}
}
func TestPersistence(t *testing.T) {
c := NewClient(nil)
c.SyncCluster()
fo, err := os.Create("config.json")
if err != nil {
t.Fatal(err)
}
defer func() {
if err := fo.Close(); err != nil {
panic(err)
}
}()
c.SetPersistence(fo)
err = c.saveConfig()
if err != nil {
t.Fatal(err)
}
c2, err := NewClientFromFile("config.json")
if err != nil {
t.Fatal(err)
}
// Verify that the two clients have the same config
b1, _ := json.Marshal(c)
b2, _ := json.Marshal(c2)
if string(b1) != string(b2) {
t.Fatalf("The two configs should be equal!")
}
}
func TestClientRetry(t *testing.T) {
c := NewClient([]string{"http://strange", "http://127.0.0.1:4001"})
// use first endpoint as the picked url
c.cluster.picked = 0
if _, err := c.Set("foo", "bar", 5); err != nil {
t.Fatal(err)
}
if _, err := c.Delete("foo", true); err != nil {
t.Fatal(err)
}
}

View File

@@ -1,37 +0,0 @@
package etcd
import (
"math/rand"
"strings"
)
type Cluster struct {
Leader string `json:"leader"`
Machines []string `json:"machines"`
picked int
}
func NewCluster(machines []string) *Cluster {
// if an empty slice was sent in then just assume HTTP 4001 on localhost
if len(machines) == 0 {
machines = []string{"http://127.0.0.1:4001"}
}
// default leader and machines
return &Cluster{
Leader: "",
Machines: machines,
picked: rand.Intn(len(machines)),
}
}
func (cl *Cluster) failure() { cl.picked = rand.Intn(len(cl.Machines)) }
func (cl *Cluster) pick() string { return cl.Machines[cl.picked] }
func (cl *Cluster) updateFromStr(machines string) {
cl.Machines = strings.Split(machines, ",")
for i := range cl.Machines {
cl.Machines[i] = strings.TrimSpace(cl.Machines[i])
}
cl.picked = rand.Intn(len(cl.Machines))
}

View File

@@ -1,34 +0,0 @@
package etcd
import "fmt"
func (c *Client) CompareAndDelete(key string, prevValue string, prevIndex uint64) (*Response, error) {
raw, err := c.RawCompareAndDelete(key, prevValue, prevIndex)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}
func (c *Client) RawCompareAndDelete(key string, prevValue string, prevIndex uint64) (*RawResponse, error) {
if prevValue == "" && prevIndex == 0 {
return nil, fmt.Errorf("You must give either prevValue or prevIndex.")
}
options := Options{}
if prevValue != "" {
options["prevValue"] = prevValue
}
if prevIndex != 0 {
options["prevIndex"] = prevIndex
}
raw, err := c.delete(key, options)
if err != nil {
return nil, err
}
return raw, err
}

View File

@@ -1,46 +0,0 @@
package etcd
import (
"testing"
)
func TestCompareAndDelete(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("foo", true)
}()
c.Set("foo", "bar", 5)
// This should succeed an correct prevValue
resp, err := c.CompareAndDelete("foo", "bar", 0)
if err != nil {
t.Fatal(err)
}
if !(resp.PrevNode.Value == "bar" && resp.PrevNode.Key == "/foo" && resp.PrevNode.TTL == 5) {
t.Fatalf("CompareAndDelete 1 prevNode failed: %#v", resp)
}
resp, _ = c.Set("foo", "bar", 5)
// This should fail because it gives an incorrect prevValue
_, err = c.CompareAndDelete("foo", "xxx", 0)
if err == nil {
t.Fatalf("CompareAndDelete 2 should have failed. The response is: %#v", resp)
}
// This should succeed because it gives an correct prevIndex
resp, err = c.CompareAndDelete("foo", "", resp.Node.ModifiedIndex)
if err != nil {
t.Fatal(err)
}
if !(resp.PrevNode.Value == "bar" && resp.PrevNode.Key == "/foo" && resp.PrevNode.TTL == 5) {
t.Fatalf("CompareAndSwap 3 prevNode failed: %#v", resp)
}
c.Set("foo", "bar", 5)
// This should fail because it gives an incorrect prevIndex
resp, err = c.CompareAndDelete("foo", "", 29817514)
if err == nil {
t.Fatalf("CompareAndDelete 4 should have failed. The response is: %#v", resp)
}
}

View File

@@ -1,36 +0,0 @@
package etcd
import "fmt"
func (c *Client) CompareAndSwap(key string, value string, ttl uint64,
prevValue string, prevIndex uint64) (*Response, error) {
raw, err := c.RawCompareAndSwap(key, value, ttl, prevValue, prevIndex)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}
func (c *Client) RawCompareAndSwap(key string, value string, ttl uint64,
prevValue string, prevIndex uint64) (*RawResponse, error) {
if prevValue == "" && prevIndex == 0 {
return nil, fmt.Errorf("You must give either prevValue or prevIndex.")
}
options := Options{}
if prevValue != "" {
options["prevValue"] = prevValue
}
if prevIndex != 0 {
options["prevIndex"] = prevIndex
}
raw, err := c.put(key, value, ttl, options)
if err != nil {
return nil, err
}
return raw, err
}

View File

@@ -1,57 +0,0 @@
package etcd
import (
"testing"
)
func TestCompareAndSwap(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("foo", true)
}()
c.Set("foo", "bar", 5)
// This should succeed
resp, err := c.CompareAndSwap("foo", "bar2", 5, "bar", 0)
if err != nil {
t.Fatal(err)
}
if !(resp.Node.Value == "bar2" && resp.Node.Key == "/foo" && resp.Node.TTL == 5) {
t.Fatalf("CompareAndSwap 1 failed: %#v", resp)
}
if !(resp.PrevNode.Value == "bar" && resp.PrevNode.Key == "/foo" && resp.PrevNode.TTL == 5) {
t.Fatalf("CompareAndSwap 1 prevNode failed: %#v", resp)
}
// This should fail because it gives an incorrect prevValue
resp, err = c.CompareAndSwap("foo", "bar3", 5, "xxx", 0)
if err == nil {
t.Fatalf("CompareAndSwap 2 should have failed. The response is: %#v", resp)
}
resp, err = c.Set("foo", "bar", 5)
if err != nil {
t.Fatal(err)
}
// This should succeed
resp, err = c.CompareAndSwap("foo", "bar2", 5, "", resp.Node.ModifiedIndex)
if err != nil {
t.Fatal(err)
}
if !(resp.Node.Value == "bar2" && resp.Node.Key == "/foo" && resp.Node.TTL == 5) {
t.Fatalf("CompareAndSwap 3 failed: %#v", resp)
}
if !(resp.PrevNode.Value == "bar" && resp.PrevNode.Key == "/foo" && resp.PrevNode.TTL == 5) {
t.Fatalf("CompareAndSwap 3 prevNode failed: %#v", resp)
}
// This should fail because it gives an incorrect prevIndex
resp, err = c.CompareAndSwap("foo", "bar3", 5, "", 29817514)
if err == nil {
t.Fatalf("CompareAndSwap 4 should have failed. The response is: %#v", resp)
}
}

View File

@@ -1 +0,0 @@
{"config":{"certFile":"","keyFile":"","caCertFiles":null,"timeout":1000000000,"consistency":"STRONG"},"cluster":{"leader":"http://127.0.0.1:4001","machines":["http://127.0.0.1:4001"]}}

View File

@@ -1,55 +0,0 @@
package etcd
import (
"fmt"
"io/ioutil"
"log"
"strings"
)
var logger *etcdLogger
func SetLogger(l *log.Logger) {
logger = &etcdLogger{l}
}
func GetLogger() *log.Logger {
return logger.log
}
type etcdLogger struct {
log *log.Logger
}
func (p *etcdLogger) Debug(args ...interface{}) {
msg := "DEBUG: " + fmt.Sprint(args...)
p.log.Println(msg)
}
func (p *etcdLogger) Debugf(f string, args ...interface{}) {
msg := "DEBUG: " + fmt.Sprintf(f, args...)
// Append newline if necessary
if !strings.HasSuffix(msg, "\n") {
msg = msg + "\n"
}
p.log.Print(msg)
}
func (p *etcdLogger) Warning(args ...interface{}) {
msg := "WARNING: " + fmt.Sprint(args...)
p.log.Println(msg)
}
func (p *etcdLogger) Warningf(f string, args ...interface{}) {
msg := "WARNING: " + fmt.Sprintf(f, args...)
// Append newline if necessary
if !strings.HasSuffix(msg, "\n") {
msg = msg + "\n"
}
p.log.Print(msg)
}
func init() {
// Default logger uses the go default log.
SetLogger(log.New(ioutil.Discard, "go-etcd", log.LstdFlags))
}

View File

@@ -1,28 +0,0 @@
package etcd
import (
"testing"
)
type Foo struct{}
type Bar struct {
one string
two int
}
// Tests that logs don't panic with arbitrary interfaces
func TestDebug(t *testing.T) {
f := &Foo{}
b := &Bar{"asfd", 3}
for _, test := range []interface{}{
1234,
"asdf",
f,
b,
} {
logger.Debug(test)
logger.Debugf("something, %s", test)
logger.Warning(test)
logger.Warningf("something, %s", test)
}
}

View File

@@ -1,40 +0,0 @@
package etcd
// Delete deletes the given key.
//
// When recursive set to false, if the key points to a
// directory the method will fail.
//
// When recursive set to true, if the key points to a file,
// the file will be deleted; if the key points to a directory,
// then everything under the directory (including all child directories)
// will be deleted.
func (c *Client) Delete(key string, recursive bool) (*Response, error) {
raw, err := c.RawDelete(key, recursive, false)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}
// DeleteDir deletes an empty directory or a key value pair
func (c *Client) DeleteDir(key string) (*Response, error) {
raw, err := c.RawDelete(key, false, true)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}
func (c *Client) RawDelete(key string, recursive bool, dir bool) (*RawResponse, error) {
ops := Options{
"recursive": recursive,
"dir": dir,
}
return c.delete(key, ops)
}

View File

@@ -1,81 +0,0 @@
package etcd
import (
"testing"
)
func TestDelete(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("foo", true)
}()
c.Set("foo", "bar", 5)
resp, err := c.Delete("foo", false)
if err != nil {
t.Fatal(err)
}
if !(resp.Node.Value == "") {
t.Fatalf("Delete failed with %s", resp.Node.Value)
}
if !(resp.PrevNode.Value == "bar") {
t.Fatalf("Delete PrevNode failed with %s", resp.Node.Value)
}
resp, err = c.Delete("foo", false)
if err == nil {
t.Fatalf("Delete should have failed because the key foo did not exist. "+
"The response was: %v", resp)
}
}
func TestDeleteAll(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("foo", true)
c.Delete("fooDir", true)
}()
c.SetDir("foo", 5)
// test delete an empty dir
resp, err := c.DeleteDir("foo")
if err != nil {
t.Fatal(err)
}
if !(resp.Node.Value == "") {
t.Fatalf("DeleteAll 1 failed: %#v", resp)
}
if !(resp.PrevNode.Dir == true && resp.PrevNode.Value == "") {
t.Fatalf("DeleteAll 1 PrevNode failed: %#v", resp)
}
c.CreateDir("fooDir", 5)
c.Set("fooDir/foo", "bar", 5)
_, err = c.DeleteDir("fooDir")
if err == nil {
t.Fatal("should not able to delete a non-empty dir with deletedir")
}
resp, err = c.Delete("fooDir", true)
if err != nil {
t.Fatal(err)
}
if !(resp.Node.Value == "") {
t.Fatalf("DeleteAll 2 failed: %#v", resp)
}
if !(resp.PrevNode.Dir == true && resp.PrevNode.Value == "") {
t.Fatalf("DeleteAll 2 PrevNode failed: %#v", resp)
}
resp, err = c.Delete("foo", true)
if err == nil {
t.Fatalf("DeleteAll should have failed because the key foo did not exist. "+
"The response was: %v", resp)
}
}

View File

@@ -1,49 +0,0 @@
package etcd
import (
"encoding/json"
"fmt"
)
const (
ErrCodeEtcdNotReachable = 501
ErrCodeUnhandledHTTPStatus = 502
)
var (
errorMap = map[int]string{
ErrCodeEtcdNotReachable: "All the given peers are not reachable",
}
)
type EtcdError struct {
ErrorCode int `json:"errorCode"`
Message string `json:"message"`
Cause string `json:"cause,omitempty"`
Index uint64 `json:"index"`
}
func (e EtcdError) Error() string {
return fmt.Sprintf("%v: %v (%v) [%v]", e.ErrorCode, e.Message, e.Cause, e.Index)
}
func newError(errorCode int, cause string, index uint64) *EtcdError {
return &EtcdError{
ErrorCode: errorCode,
Message: errorMap[errorCode],
Cause: cause,
Index: index,
}
}
func handleError(b []byte) error {
etcdErr := new(EtcdError)
err := json.Unmarshal(b, etcdErr)
if err != nil {
logger.Warningf("cannot unmarshal etcd error: %v", err)
return err
}
return etcdErr
}

View File

@@ -1,32 +0,0 @@
package etcd
// Get gets the file or directory associated with the given key.
// If the key points to a directory, files and directories under
// it will be returned in sorted or unsorted order, depending on
// the sort flag.
// If recursive is set to false, contents under child directories
// will not be returned.
// If recursive is set to true, all the contents will be returned.
func (c *Client) Get(key string, sort, recursive bool) (*Response, error) {
raw, err := c.RawGet(key, sort, recursive)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}
func (c *Client) RawGet(key string, sort, recursive bool) (*RawResponse, error) {
var q bool
if c.config.Consistency == STRONG_CONSISTENCY {
q = true
}
ops := Options{
"recursive": recursive,
"sorted": sort,
"quorum": q,
}
return c.get(key, ops)
}

View File

@@ -1,131 +0,0 @@
package etcd
import (
"reflect"
"testing"
)
// cleanNode scrubs Expiration, ModifiedIndex and CreatedIndex of a node.
func cleanNode(n *Node) {
n.Expiration = nil
n.ModifiedIndex = 0
n.CreatedIndex = 0
}
// cleanResult scrubs a result object two levels deep of Expiration,
// ModifiedIndex and CreatedIndex.
func cleanResult(result *Response) {
// TODO(philips): make this recursive.
cleanNode(result.Node)
for i, _ := range result.Node.Nodes {
cleanNode(result.Node.Nodes[i])
for j, _ := range result.Node.Nodes[i].Nodes {
cleanNode(result.Node.Nodes[i].Nodes[j])
}
}
}
func TestGet(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("foo", true)
}()
c.Set("foo", "bar", 5)
result, err := c.Get("foo", false, false)
if err != nil {
t.Fatal(err)
}
if result.Node.Key != "/foo" || result.Node.Value != "bar" {
t.Fatalf("Get failed with %s %s %v", result.Node.Key, result.Node.Value, result.Node.TTL)
}
result, err = c.Get("goo", false, false)
if err == nil {
t.Fatalf("should not be able to get non-exist key")
}
}
func TestGetAll(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("fooDir", true)
}()
c.CreateDir("fooDir", 5)
c.Set("fooDir/k0", "v0", 5)
c.Set("fooDir/k1", "v1", 5)
// Return kv-pairs in sorted order
result, err := c.Get("fooDir", true, false)
if err != nil {
t.Fatal(err)
}
expected := Nodes{
&Node{
Key: "/fooDir/k0",
Value: "v0",
TTL: 5,
},
&Node{
Key: "/fooDir/k1",
Value: "v1",
TTL: 5,
},
}
cleanResult(result)
if !reflect.DeepEqual(result.Node.Nodes, expected) {
t.Fatalf("(actual) %v != (expected) %v", result.Node.Nodes, expected)
}
// Test the `recursive` option
c.CreateDir("fooDir/childDir", 5)
c.Set("fooDir/childDir/k2", "v2", 5)
// Return kv-pairs in sorted order
result, err = c.Get("fooDir", true, true)
cleanResult(result)
if err != nil {
t.Fatal(err)
}
expected = Nodes{
&Node{
Key: "/fooDir/childDir",
Dir: true,
Nodes: Nodes{
&Node{
Key: "/fooDir/childDir/k2",
Value: "v2",
TTL: 5,
},
},
TTL: 5,
},
&Node{
Key: "/fooDir/k0",
Value: "v0",
TTL: 5,
},
&Node{
Key: "/fooDir/k1",
Value: "v1",
TTL: 5,
},
}
cleanResult(result)
if !reflect.DeepEqual(result.Node.Nodes, expected) {
t.Fatalf("(actual) %v != (expected) %v", result.Node.Nodes, expected)
}
}

View File

@@ -1,30 +0,0 @@
package etcd
import "encoding/json"
type Member struct {
ID string `json:"id"`
Name string `json:"name"`
PeerURLs []string `json:"peerURLs"`
ClientURLs []string `json:"clientURLs"`
}
type memberCollection []Member
func (c *memberCollection) UnmarshalJSON(data []byte) error {
d := struct {
Members []Member
}{}
if err := json.Unmarshal(data, &d); err != nil {
return err
}
if d.Members == nil {
*c = make([]Member, 0)
return nil
}
*c = d.Members
return nil
}

View File

@@ -1,71 +0,0 @@
package etcd
import (
"encoding/json"
"reflect"
"testing"
)
func TestMemberCollectionUnmarshal(t *testing.T) {
tests := []struct {
body []byte
want memberCollection
}{
{
body: []byte(`{"members":[]}`),
want: memberCollection([]Member{}),
},
{
body: []byte(`{"members":[{"id":"2745e2525fce8fe","peerURLs":["http://127.0.0.1:7003"],"name":"node3","clientURLs":["http://127.0.0.1:4003"]},{"id":"42134f434382925","peerURLs":["http://127.0.0.1:2380","http://127.0.0.1:7001"],"name":"node1","clientURLs":["http://127.0.0.1:2379","http://127.0.0.1:4001"]},{"id":"94088180e21eb87b","peerURLs":["http://127.0.0.1:7002"],"name":"node2","clientURLs":["http://127.0.0.1:4002"]}]}`),
want: memberCollection(
[]Member{
{
ID: "2745e2525fce8fe",
Name: "node3",
PeerURLs: []string{
"http://127.0.0.1:7003",
},
ClientURLs: []string{
"http://127.0.0.1:4003",
},
},
{
ID: "42134f434382925",
Name: "node1",
PeerURLs: []string{
"http://127.0.0.1:2380",
"http://127.0.0.1:7001",
},
ClientURLs: []string{
"http://127.0.0.1:2379",
"http://127.0.0.1:4001",
},
},
{
ID: "94088180e21eb87b",
Name: "node2",
PeerURLs: []string{
"http://127.0.0.1:7002",
},
ClientURLs: []string{
"http://127.0.0.1:4002",
},
},
},
),
},
}
for i, tt := range tests {
var got memberCollection
err := json.Unmarshal(tt.body, &got)
if err != nil {
t.Errorf("#%d: unexpected error: %v", i, err)
continue
}
if !reflect.DeepEqual(tt.want, got) {
t.Errorf("#%d: incorrect output: want=%#v, got=%#v", i, tt.want, got)
}
}
}

View File

@@ -1,72 +0,0 @@
package etcd
import (
"fmt"
"net/url"
"reflect"
)
type Options map[string]interface{}
// An internally-used data structure that represents a mapping
// between valid options and their kinds
type validOptions map[string]reflect.Kind
// Valid options for GET, PUT, POST, DELETE
// Using CAPITALIZED_UNDERSCORE to emphasize that these
// values are meant to be used as constants.
var (
VALID_GET_OPTIONS = validOptions{
"recursive": reflect.Bool,
"quorum": reflect.Bool,
"sorted": reflect.Bool,
"wait": reflect.Bool,
"waitIndex": reflect.Uint64,
}
VALID_PUT_OPTIONS = validOptions{
"prevValue": reflect.String,
"prevIndex": reflect.Uint64,
"prevExist": reflect.Bool,
"dir": reflect.Bool,
}
VALID_POST_OPTIONS = validOptions{}
VALID_DELETE_OPTIONS = validOptions{
"recursive": reflect.Bool,
"dir": reflect.Bool,
"prevValue": reflect.String,
"prevIndex": reflect.Uint64,
}
)
// Convert options to a string of HTML parameters
func (ops Options) toParameters(validOps validOptions) (string, error) {
p := "?"
values := url.Values{}
if ops == nil {
return "", nil
}
for k, v := range ops {
// Check if the given option is valid (that it exists)
kind := validOps[k]
if kind == reflect.Invalid {
return "", fmt.Errorf("Invalid option: %v", k)
}
// Check if the given option is of the valid type
t := reflect.TypeOf(v)
if kind != t.Kind() {
return "", fmt.Errorf("Option %s should be of %v kind, not of %v kind.",
k, kind, t.Kind())
}
values.Set(k, fmt.Sprintf("%v", v))
}
p += values.Encode()
return p, nil
}

View File

@@ -1,403 +0,0 @@
package etcd
import (
"errors"
"fmt"
"io"
"io/ioutil"
"net/http"
"net/url"
"path"
"strings"
"sync"
"time"
)
// Errors introduced by handling requests
var (
ErrRequestCancelled = errors.New("sending request is cancelled")
)
type RawRequest struct {
Method string
RelativePath string
Values url.Values
Cancel <-chan bool
}
// NewRawRequest returns a new RawRequest
func NewRawRequest(method, relativePath string, values url.Values, cancel <-chan bool) *RawRequest {
return &RawRequest{
Method: method,
RelativePath: relativePath,
Values: values,
Cancel: cancel,
}
}
// getCancelable issues a cancelable GET request
func (c *Client) getCancelable(key string, options Options,
cancel <-chan bool) (*RawResponse, error) {
logger.Debugf("get %s [%s]", key, c.cluster.pick())
p := keyToPath(key)
str, err := options.toParameters(VALID_GET_OPTIONS)
if err != nil {
return nil, err
}
p += str
req := NewRawRequest("GET", p, nil, cancel)
resp, err := c.SendRequest(req)
if err != nil {
return nil, err
}
return resp, nil
}
// get issues a GET request
func (c *Client) get(key string, options Options) (*RawResponse, error) {
return c.getCancelable(key, options, nil)
}
// put issues a PUT request
func (c *Client) put(key string, value string, ttl uint64,
options Options) (*RawResponse, error) {
logger.Debugf("put %s, %s, ttl: %d, [%s]", key, value, ttl, c.cluster.pick())
p := keyToPath(key)
str, err := options.toParameters(VALID_PUT_OPTIONS)
if err != nil {
return nil, err
}
p += str
req := NewRawRequest("PUT", p, buildValues(value, ttl), nil)
resp, err := c.SendRequest(req)
if err != nil {
return nil, err
}
return resp, nil
}
// post issues a POST request
func (c *Client) post(key string, value string, ttl uint64) (*RawResponse, error) {
logger.Debugf("post %s, %s, ttl: %d, [%s]", key, value, ttl, c.cluster.pick())
p := keyToPath(key)
req := NewRawRequest("POST", p, buildValues(value, ttl), nil)
resp, err := c.SendRequest(req)
if err != nil {
return nil, err
}
return resp, nil
}
// delete issues a DELETE request
func (c *Client) delete(key string, options Options) (*RawResponse, error) {
logger.Debugf("delete %s [%s]", key, c.cluster.pick())
p := keyToPath(key)
str, err := options.toParameters(VALID_DELETE_OPTIONS)
if err != nil {
return nil, err
}
p += str
req := NewRawRequest("DELETE", p, nil, nil)
resp, err := c.SendRequest(req)
if err != nil {
return nil, err
}
return resp, nil
}
// SendRequest sends a HTTP request and returns a Response as defined by etcd
func (c *Client) SendRequest(rr *RawRequest) (*RawResponse, error) {
var req *http.Request
var resp *http.Response
var httpPath string
var err error
var respBody []byte
var numReqs = 1
checkRetry := c.CheckRetry
if checkRetry == nil {
checkRetry = DefaultCheckRetry
}
cancelled := make(chan bool, 1)
reqLock := new(sync.Mutex)
if rr.Cancel != nil {
cancelRoutine := make(chan bool)
defer close(cancelRoutine)
go func() {
select {
case <-rr.Cancel:
cancelled <- true
logger.Debug("send.request is cancelled")
case <-cancelRoutine:
return
}
// Repeat canceling request until this thread is stopped
// because we have no idea about whether it succeeds.
for {
reqLock.Lock()
c.httpClient.Transport.(*http.Transport).CancelRequest(req)
reqLock.Unlock()
select {
case <-time.After(100 * time.Millisecond):
case <-cancelRoutine:
return
}
}
}()
}
// If we connect to a follower and consistency is required, retry until
// we connect to a leader
sleep := 25 * time.Millisecond
maxSleep := time.Second
for attempt := 0; ; attempt++ {
if attempt > 0 {
select {
case <-cancelled:
return nil, ErrRequestCancelled
case <-time.After(sleep):
sleep = sleep * 2
if sleep > maxSleep {
sleep = maxSleep
}
}
}
logger.Debug("Connecting to etcd: attempt ", attempt+1, " for ", rr.RelativePath)
// get httpPath if not set
if httpPath == "" {
httpPath = c.getHttpPath(rr.RelativePath)
}
// Return a cURL command if curlChan is set
if c.cURLch != nil {
command := fmt.Sprintf("curl -X %s %s", rr.Method, httpPath)
for key, value := range rr.Values {
command += fmt.Sprintf(" -d %s=%s", key, value[0])
}
if c.credentials != nil {
command += fmt.Sprintf(" -u %s", c.credentials.username)
}
c.sendCURL(command)
}
logger.Debug("send.request.to ", httpPath, " | method ", rr.Method)
req, err := func() (*http.Request, error) {
reqLock.Lock()
defer reqLock.Unlock()
if rr.Values == nil {
if req, err = http.NewRequest(rr.Method, httpPath, nil); err != nil {
return nil, err
}
} else {
body := strings.NewReader(rr.Values.Encode())
if req, err = http.NewRequest(rr.Method, httpPath, body); err != nil {
return nil, err
}
req.Header.Set("Content-Type",
"application/x-www-form-urlencoded; param=value")
}
return req, nil
}()
if err != nil {
return nil, err
}
if c.credentials != nil {
req.SetBasicAuth(c.credentials.username, c.credentials.password)
}
resp, err = c.httpClient.Do(req)
// clear previous httpPath
httpPath = ""
defer func() {
if resp != nil {
resp.Body.Close()
}
}()
// If the request was cancelled, return ErrRequestCancelled directly
select {
case <-cancelled:
return nil, ErrRequestCancelled
default:
}
numReqs++
// network error, change a machine!
if err != nil {
logger.Debug("network error: ", err.Error())
lastResp := http.Response{}
if checkErr := checkRetry(c.cluster, numReqs, lastResp, err); checkErr != nil {
return nil, checkErr
}
c.cluster.failure()
continue
}
// if there is no error, it should receive response
logger.Debug("recv.response.from ", httpPath)
if validHttpStatusCode[resp.StatusCode] {
// try to read byte code and break the loop
respBody, err = ioutil.ReadAll(resp.Body)
if err == nil {
logger.Debug("recv.success ", httpPath)
break
}
// ReadAll error may be caused due to cancel request
select {
case <-cancelled:
return nil, ErrRequestCancelled
default:
}
if err == io.ErrUnexpectedEOF {
// underlying connection was closed prematurely, probably by timeout
// TODO: empty body or unexpectedEOF can cause http.Transport to get hosed;
// this allows the client to detect that and take evasive action. Need
// to revisit once code.google.com/p/go/issues/detail?id=8648 gets fixed.
respBody = []byte{}
break
}
}
if resp.StatusCode == http.StatusTemporaryRedirect {
u, err := resp.Location()
if err != nil {
logger.Warning(err)
} else {
// set httpPath for following redirection
httpPath = u.String()
}
resp.Body.Close()
continue
}
if checkErr := checkRetry(c.cluster, numReqs, *resp,
errors.New("Unexpected HTTP status code")); checkErr != nil {
return nil, checkErr
}
resp.Body.Close()
}
r := &RawResponse{
StatusCode: resp.StatusCode,
Body: respBody,
Header: resp.Header,
}
return r, nil
}
// DefaultCheckRetry defines the retrying behaviour for bad HTTP requests
// If we have retried 2 * machine number, stop retrying.
// If status code is InternalServerError, sleep for 200ms.
func DefaultCheckRetry(cluster *Cluster, numReqs int, lastResp http.Response,
err error) error {
if numReqs > 2*len(cluster.Machines) {
errStr := fmt.Sprintf("failed to propose on members %v twice [last error: %v]", cluster.Machines, err)
return newError(ErrCodeEtcdNotReachable, errStr, 0)
}
if isEmptyResponse(lastResp) {
// always retry if it failed to get response from one machine
return nil
}
if !shouldRetry(lastResp) {
body := []byte("nil")
if lastResp.Body != nil {
if b, err := ioutil.ReadAll(lastResp.Body); err == nil {
body = b
}
}
errStr := fmt.Sprintf("unhandled http status [%s] with body [%s]", http.StatusText(lastResp.StatusCode), body)
return newError(ErrCodeUnhandledHTTPStatus, errStr, 0)
}
// sleep some time and expect leader election finish
time.Sleep(time.Millisecond * 200)
logger.Warning("bad response status code", lastResp.StatusCode)
return nil
}
func isEmptyResponse(r http.Response) bool { return r.StatusCode == 0 }
// shouldRetry returns whether the reponse deserves retry.
func shouldRetry(r http.Response) bool {
// TODO: only retry when the cluster is in leader election
// We cannot do it exactly because etcd doesn't support it well.
return r.StatusCode == http.StatusInternalServerError
}
func (c *Client) getHttpPath(s ...string) string {
fullPath := c.cluster.pick() + "/" + version
for _, seg := range s {
fullPath = fullPath + "/" + seg
}
return fullPath
}
// buildValues builds a url.Values map according to the given value and ttl
func buildValues(value string, ttl uint64) url.Values {
v := url.Values{}
if value != "" {
v.Set("value", value)
}
if ttl > 0 {
v.Set("ttl", fmt.Sprintf("%v", ttl))
}
return v
}
// convert key string to http path exclude version, including URL escaping
// for example: key[foo] -> path[keys/foo]
// key[/%z] -> path[keys/%25z]
// key[/] -> path[keys/]
func keyToPath(key string) string {
// URL-escape our key, except for slashes
p := strings.Replace(url.QueryEscape(path.Join("keys", key)), "%2F", "/", -1)
// corner case: if key is "/" or "//" ect
// path join will clear the tailing "/"
// we need to add it back
if p == "keys" {
p = "keys/"
}
return p
}

View File

@@ -1,22 +0,0 @@
package etcd
import "testing"
func TestKeyToPath(t *testing.T) {
tests := []struct {
key string
wpath string
}{
{"", "keys/"},
{"foo", "keys/foo"},
{"foo/bar", "keys/foo/bar"},
{"%z", "keys/%25z"},
{"/", "keys/"},
}
for i, tt := range tests {
path := keyToPath(tt.key)
if path != tt.wpath {
t.Errorf("#%d: path = %s, want %s", i, path, tt.wpath)
}
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -1,93 +0,0 @@
package etcd
//go:generate codecgen -o response.generated.go response.go
import (
"net/http"
"strconv"
"time"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/ugorji/go/codec"
)
const (
rawResponse = iota
normalResponse
)
type responseType int
type RawResponse struct {
StatusCode int
Body []byte
Header http.Header
}
var (
validHttpStatusCode = map[int]bool{
http.StatusCreated: true,
http.StatusOK: true,
http.StatusBadRequest: true,
http.StatusNotFound: true,
http.StatusPreconditionFailed: true,
http.StatusForbidden: true,
http.StatusUnauthorized: true,
}
)
// Unmarshal parses RawResponse and stores the result in Response
func (rr *RawResponse) Unmarshal() (*Response, error) {
if rr.StatusCode != http.StatusOK && rr.StatusCode != http.StatusCreated {
return nil, handleError(rr.Body)
}
resp := new(Response)
err := codec.NewDecoderBytes(rr.Body, new(codec.JsonHandle)).Decode(resp)
if err != nil {
return nil, err
}
// attach index and term to response
resp.EtcdIndex, _ = strconv.ParseUint(rr.Header.Get("X-Etcd-Index"), 10, 64)
resp.RaftIndex, _ = strconv.ParseUint(rr.Header.Get("X-Raft-Index"), 10, 64)
resp.RaftTerm, _ = strconv.ParseUint(rr.Header.Get("X-Raft-Term"), 10, 64)
return resp, nil
}
type Response struct {
Action string `json:"action"`
Node *Node `json:"node"`
PrevNode *Node `json:"prevNode,omitempty"`
EtcdIndex uint64 `json:"etcdIndex"`
RaftIndex uint64 `json:"raftIndex"`
RaftTerm uint64 `json:"raftTerm"`
}
type Node struct {
Key string `json:"key, omitempty"`
Value string `json:"value,omitempty"`
Dir bool `json:"dir,omitempty"`
Expiration *time.Time `json:"expiration,omitempty"`
TTL int64 `json:"ttl,omitempty"`
Nodes Nodes `json:"nodes,omitempty"`
ModifiedIndex uint64 `json:"modifiedIndex,omitempty"`
CreatedIndex uint64 `json:"createdIndex,omitempty"`
}
type Nodes []*Node
// interfaces for sorting
func (ns Nodes) Len() int {
return len(ns)
}
func (ns Nodes) Less(i, j int) bool {
return ns[i].Key < ns[j].Key
}
func (ns Nodes) Swap(i, j int) {
ns[i], ns[j] = ns[j], ns[i]
}

View File

@@ -1,75 +0,0 @@
package etcd
import (
"net/http"
"reflect"
"strings"
"testing"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/ugorji/go/codec"
)
func createTestNode(size int) *Node {
return &Node{
Key: strings.Repeat("a", 30),
Value: strings.Repeat("a", size),
TTL: 123456789,
ModifiedIndex: 123456,
CreatedIndex: 123456,
}
}
func createTestNodeWithChildren(children, size int) *Node {
node := createTestNode(size)
for i := 0; i < children; i++ {
node.Nodes = append(node.Nodes, createTestNode(size))
}
return node
}
func createTestResponse(children, size int) *Response {
return &Response{
Action: "aaaaa",
Node: createTestNodeWithChildren(children, size),
PrevNode: nil,
EtcdIndex: 123456,
RaftIndex: 123456,
RaftTerm: 123456,
}
}
func benchmarkResponseUnmarshalling(b *testing.B, children, size int) {
response := createTestResponse(children, size)
rr := RawResponse{http.StatusOK, make([]byte, 0), http.Header{}}
codec.NewEncoderBytes(&rr.Body, new(codec.JsonHandle)).Encode(response)
b.ResetTimer()
newResponse := new(Response)
var err error
for i := 0; i < b.N; i++ {
if newResponse, err = rr.Unmarshal(); err != nil {
b.Errorf("Error: %v", err)
}
}
if !reflect.DeepEqual(response.Node, newResponse.Node) {
b.Errorf("Unexpected difference in a parsed response: \n%+v\n%+v", response, newResponse)
}
}
func BenchmarkSmallResponseUnmarshal(b *testing.B) {
benchmarkResponseUnmarshalling(b, 30, 20)
}
func BenchmarkManySmallResponseUnmarshal(b *testing.B) {
benchmarkResponseUnmarshalling(b, 3000, 20)
}
func BenchmarkMediumResponseUnmarshal(b *testing.B) {
benchmarkResponseUnmarshalling(b, 300, 200)
}
func BenchmarkLargeResponseUnmarshal(b *testing.B) {
benchmarkResponseUnmarshalling(b, 3000, 2000)
}

View File

@@ -1,42 +0,0 @@
package etcd
import (
"fmt"
"testing"
)
func TestSetCurlChan(t *testing.T) {
c := NewClient(nil)
c.OpenCURL()
defer func() {
c.Delete("foo", true)
}()
_, err := c.Set("foo", "bar", 5)
if err != nil {
t.Fatal(err)
}
expected := fmt.Sprintf("curl -X PUT %s/v2/keys/foo -d value=bar -d ttl=5",
c.cluster.pick())
actual := c.RecvCURL()
if expected != actual {
t.Fatalf(`Command "%s" is not equal to expected value "%s"`,
actual, expected)
}
c.SetConsistency(STRONG_CONSISTENCY)
_, err = c.Get("foo", false, false)
if err != nil {
t.Fatal(err)
}
expected = fmt.Sprintf("curl -X GET %s/v2/keys/foo?quorum=true&recursive=false&sorted=false",
c.cluster.pick())
actual = c.RecvCURL()
if expected != actual {
t.Fatalf(`Command "%s" is not equal to expected value "%s"`,
actual, expected)
}
}

View File

@@ -1,137 +0,0 @@
package etcd
// Set sets the given key to the given value.
// It will create a new key value pair or replace the old one.
// It will not replace a existing directory.
func (c *Client) Set(key string, value string, ttl uint64) (*Response, error) {
raw, err := c.RawSet(key, value, ttl)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}
// SetDir sets the given key to a directory.
// It will create a new directory or replace the old key value pair by a directory.
// It will not replace a existing directory.
func (c *Client) SetDir(key string, ttl uint64) (*Response, error) {
raw, err := c.RawSetDir(key, ttl)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}
// CreateDir creates a directory. It succeeds only if
// the given key does not yet exist.
func (c *Client) CreateDir(key string, ttl uint64) (*Response, error) {
raw, err := c.RawCreateDir(key, ttl)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}
// UpdateDir updates the given directory. It succeeds only if the
// given key already exists.
func (c *Client) UpdateDir(key string, ttl uint64) (*Response, error) {
raw, err := c.RawUpdateDir(key, ttl)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}
// Create creates a file with the given value under the given key. It succeeds
// only if the given key does not yet exist.
func (c *Client) Create(key string, value string, ttl uint64) (*Response, error) {
raw, err := c.RawCreate(key, value, ttl)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}
// CreateInOrder creates a file with a key that's guaranteed to be higher than other
// keys in the given directory. It is useful for creating queues.
func (c *Client) CreateInOrder(dir string, value string, ttl uint64) (*Response, error) {
raw, err := c.RawCreateInOrder(dir, value, ttl)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}
// Update updates the given key to the given value. It succeeds only if the
// given key already exists.
func (c *Client) Update(key string, value string, ttl uint64) (*Response, error) {
raw, err := c.RawUpdate(key, value, ttl)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}
func (c *Client) RawUpdateDir(key string, ttl uint64) (*RawResponse, error) {
ops := Options{
"prevExist": true,
"dir": true,
}
return c.put(key, "", ttl, ops)
}
func (c *Client) RawCreateDir(key string, ttl uint64) (*RawResponse, error) {
ops := Options{
"prevExist": false,
"dir": true,
}
return c.put(key, "", ttl, ops)
}
func (c *Client) RawSet(key string, value string, ttl uint64) (*RawResponse, error) {
return c.put(key, value, ttl, nil)
}
func (c *Client) RawSetDir(key string, ttl uint64) (*RawResponse, error) {
ops := Options{
"dir": true,
}
return c.put(key, "", ttl, ops)
}
func (c *Client) RawUpdate(key string, value string, ttl uint64) (*RawResponse, error) {
ops := Options{
"prevExist": true,
}
return c.put(key, value, ttl, ops)
}
func (c *Client) RawCreate(key string, value string, ttl uint64) (*RawResponse, error) {
ops := Options{
"prevExist": false,
}
return c.put(key, value, ttl, ops)
}
func (c *Client) RawCreateInOrder(dir string, value string, ttl uint64) (*RawResponse, error) {
return c.post(dir, value, ttl)
}

View File

@@ -1,241 +0,0 @@
package etcd
import (
"testing"
)
func TestSet(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("foo", true)
}()
resp, err := c.Set("foo", "bar", 5)
if err != nil {
t.Fatal(err)
}
if resp.Node.Key != "/foo" || resp.Node.Value != "bar" || resp.Node.TTL != 5 {
t.Fatalf("Set 1 failed: %#v", resp)
}
if resp.PrevNode != nil {
t.Fatalf("Set 1 PrevNode failed: %#v", resp)
}
resp, err = c.Set("foo", "bar2", 5)
if err != nil {
t.Fatal(err)
}
if !(resp.Node.Key == "/foo" && resp.Node.Value == "bar2" && resp.Node.TTL == 5) {
t.Fatalf("Set 2 failed: %#v", resp)
}
if resp.PrevNode.Key != "/foo" || resp.PrevNode.Value != "bar" || resp.Node.TTL != 5 {
t.Fatalf("Set 2 PrevNode failed: %#v", resp)
}
}
func TestUpdate(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("foo", true)
c.Delete("nonexistent", true)
}()
resp, err := c.Set("foo", "bar", 5)
if err != nil {
t.Fatal(err)
}
// This should succeed.
resp, err = c.Update("foo", "wakawaka", 5)
if err != nil {
t.Fatal(err)
}
if !(resp.Action == "update" && resp.Node.Key == "/foo" && resp.Node.TTL == 5) {
t.Fatalf("Update 1 failed: %#v", resp)
}
if !(resp.PrevNode.Key == "/foo" && resp.PrevNode.Value == "bar" && resp.Node.TTL == 5) {
t.Fatalf("Update 1 prevValue failed: %#v", resp)
}
// This should fail because the key does not exist.
resp, err = c.Update("nonexistent", "whatever", 5)
if err == nil {
t.Fatalf("The key %v did not exist, so the update should have failed."+
"The response was: %#v", resp.Node.Key, resp)
}
}
func TestCreate(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("newKey", true)
}()
newKey := "/newKey"
newValue := "/newValue"
// This should succeed
resp, err := c.Create(newKey, newValue, 5)
if err != nil {
t.Fatal(err)
}
if !(resp.Action == "create" && resp.Node.Key == newKey &&
resp.Node.Value == newValue && resp.Node.TTL == 5) {
t.Fatalf("Create 1 failed: %#v", resp)
}
if resp.PrevNode != nil {
t.Fatalf("Create 1 PrevNode failed: %#v", resp)
}
// This should fail, because the key is already there
resp, err = c.Create(newKey, newValue, 5)
if err == nil {
t.Fatalf("The key %v did exist, so the creation should have failed."+
"The response was: %#v", resp.Node.Key, resp)
}
}
func TestCreateInOrder(t *testing.T) {
c := NewClient(nil)
dir := "/queue"
defer func() {
c.DeleteDir(dir)
}()
var firstKey, secondKey string
resp, err := c.CreateInOrder(dir, "1", 5)
if err != nil {
t.Fatal(err)
}
if !(resp.Action == "create" && resp.Node.Value == "1" && resp.Node.TTL == 5) {
t.Fatalf("Create 1 failed: %#v", resp)
}
firstKey = resp.Node.Key
resp, err = c.CreateInOrder(dir, "2", 5)
if err != nil {
t.Fatal(err)
}
if !(resp.Action == "create" && resp.Node.Value == "2" && resp.Node.TTL == 5) {
t.Fatalf("Create 2 failed: %#v", resp)
}
secondKey = resp.Node.Key
if firstKey >= secondKey {
t.Fatalf("Expected first key to be greater than second key, but %s is not greater than %s",
firstKey, secondKey)
}
}
func TestSetDir(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("foo", true)
c.Delete("fooDir", true)
}()
resp, err := c.CreateDir("fooDir", 5)
if err != nil {
t.Fatal(err)
}
if !(resp.Node.Key == "/fooDir" && resp.Node.Value == "" && resp.Node.TTL == 5) {
t.Fatalf("SetDir 1 failed: %#v", resp)
}
if resp.PrevNode != nil {
t.Fatalf("SetDir 1 PrevNode failed: %#v", resp)
}
// This should fail because /fooDir already points to a directory
resp, err = c.CreateDir("/fooDir", 5)
if err == nil {
t.Fatalf("fooDir already points to a directory, so SetDir should have failed."+
"The response was: %#v", resp)
}
_, err = c.Set("foo", "bar", 5)
if err != nil {
t.Fatal(err)
}
// This should succeed
// It should replace the key
resp, err = c.SetDir("foo", 5)
if err != nil {
t.Fatal(err)
}
if !(resp.Node.Key == "/foo" && resp.Node.Value == "" && resp.Node.TTL == 5) {
t.Fatalf("SetDir 2 failed: %#v", resp)
}
if !(resp.PrevNode.Key == "/foo" && resp.PrevNode.Value == "bar" && resp.PrevNode.TTL == 5) {
t.Fatalf("SetDir 2 failed: %#v", resp)
}
}
func TestUpdateDir(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("fooDir", true)
}()
resp, err := c.CreateDir("fooDir", 5)
if err != nil {
t.Fatal(err)
}
// This should succeed.
resp, err = c.UpdateDir("fooDir", 5)
if err != nil {
t.Fatal(err)
}
if !(resp.Action == "update" && resp.Node.Key == "/fooDir" &&
resp.Node.Value == "" && resp.Node.TTL == 5) {
t.Fatalf("UpdateDir 1 failed: %#v", resp)
}
if !(resp.PrevNode.Key == "/fooDir" && resp.PrevNode.Dir == true && resp.PrevNode.TTL == 5) {
t.Fatalf("UpdateDir 1 PrevNode failed: %#v", resp)
}
// This should fail because the key does not exist.
resp, err = c.UpdateDir("nonexistentDir", 5)
if err == nil {
t.Fatalf("The key %v did not exist, so the update should have failed."+
"The response was: %#v", resp.Node.Key, resp)
}
}
func TestCreateDir(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("fooDir", true)
}()
// This should succeed
resp, err := c.CreateDir("fooDir", 5)
if err != nil {
t.Fatal(err)
}
if !(resp.Action == "create" && resp.Node.Key == "/fooDir" &&
resp.Node.Value == "" && resp.Node.TTL == 5) {
t.Fatalf("CreateDir 1 failed: %#v", resp)
}
if resp.PrevNode != nil {
t.Fatalf("CreateDir 1 PrevNode failed: %#v", resp)
}
// This should fail, because the key is already there
resp, err = c.CreateDir("fooDir", 5)
if err == nil {
t.Fatalf("The key %v did exist, so the creation should have failed."+
"The response was: %#v", resp.Node.Key, resp)
}
}

View File

@@ -1,6 +0,0 @@
package etcd
const (
version = "v2"
packageVersion = "v2.0.0+git"
)

View File

@@ -1,103 +0,0 @@
package etcd
import (
"errors"
)
// Errors introduced by the Watch command.
var (
ErrWatchStoppedByUser = errors.New("Watch stopped by the user via stop channel")
)
// If recursive is set to true the watch returns the first change under the given
// prefix since the given index.
//
// If recursive is set to false the watch returns the first change to the given key
// since the given index.
//
// To watch for the latest change, set waitIndex = 0.
//
// If a receiver channel is given, it will be a long-term watch. Watch will block at the
//channel. After someone receives the channel, it will go on to watch that
// prefix. If a stop channel is given, the client can close long-term watch using
// the stop channel.
func (c *Client) Watch(prefix string, waitIndex uint64, recursive bool,
receiver chan *Response, stop chan bool) (*Response, error) {
logger.Debugf("watch %s [%s]", prefix, c.cluster.Leader)
if receiver == nil {
raw, err := c.watchOnce(prefix, waitIndex, recursive, stop)
if err != nil {
return nil, err
}
return raw.Unmarshal()
}
defer close(receiver)
for {
raw, err := c.watchOnce(prefix, waitIndex, recursive, stop)
if err != nil {
return nil, err
}
resp, err := raw.Unmarshal()
if err != nil {
return nil, err
}
waitIndex = resp.Node.ModifiedIndex + 1
receiver <- resp
}
}
func (c *Client) RawWatch(prefix string, waitIndex uint64, recursive bool,
receiver chan *RawResponse, stop chan bool) (*RawResponse, error) {
logger.Debugf("rawWatch %s [%s]", prefix, c.cluster.Leader)
if receiver == nil {
return c.watchOnce(prefix, waitIndex, recursive, stop)
}
for {
raw, err := c.watchOnce(prefix, waitIndex, recursive, stop)
if err != nil {
return nil, err
}
resp, err := raw.Unmarshal()
if err != nil {
return nil, err
}
waitIndex = resp.Node.ModifiedIndex + 1
receiver <- raw
}
}
// helper func
// return when there is change under the given prefix
func (c *Client) watchOnce(key string, waitIndex uint64, recursive bool, stop chan bool) (*RawResponse, error) {
options := Options{
"wait": true,
}
if waitIndex > 0 {
options["waitIndex"] = waitIndex
}
if recursive {
options["recursive"] = true
}
resp, err := c.getCancelable(key, options, stop)
if err == ErrRequestCancelled {
return nil, ErrWatchStoppedByUser
}
return resp, err
}

View File

@@ -1,119 +0,0 @@
package etcd
import (
"fmt"
"runtime"
"testing"
"time"
)
func TestWatch(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("watch_foo", true)
}()
go setHelper("watch_foo", "bar", c)
resp, err := c.Watch("watch_foo", 0, false, nil, nil)
if err != nil {
t.Fatal(err)
}
if !(resp.Node.Key == "/watch_foo" && resp.Node.Value == "bar") {
t.Fatalf("Watch 1 failed: %#v", resp)
}
go setHelper("watch_foo", "bar", c)
resp, err = c.Watch("watch_foo", resp.Node.ModifiedIndex+1, false, nil, nil)
if err != nil {
t.Fatal(err)
}
if !(resp.Node.Key == "/watch_foo" && resp.Node.Value == "bar") {
t.Fatalf("Watch 2 failed: %#v", resp)
}
routineNum := runtime.NumGoroutine()
ch := make(chan *Response, 10)
stop := make(chan bool, 1)
go setLoop("watch_foo", "bar", c)
go receiver(ch, stop)
_, err = c.Watch("watch_foo", 0, false, ch, stop)
if err != ErrWatchStoppedByUser {
t.Fatalf("Watch returned a non-user stop error")
}
if newRoutineNum := runtime.NumGoroutine(); newRoutineNum != routineNum {
t.Fatalf("Routine numbers differ after watch stop: %v, %v", routineNum, newRoutineNum)
}
}
func TestWatchAll(t *testing.T) {
c := NewClient(nil)
defer func() {
c.Delete("watch_foo", true)
}()
go setHelper("watch_foo/foo", "bar", c)
resp, err := c.Watch("watch_foo", 0, true, nil, nil)
if err != nil {
t.Fatal(err)
}
if !(resp.Node.Key == "/watch_foo/foo" && resp.Node.Value == "bar") {
t.Fatalf("WatchAll 1 failed: %#v", resp)
}
go setHelper("watch_foo/foo", "bar", c)
resp, err = c.Watch("watch_foo", resp.Node.ModifiedIndex+1, true, nil, nil)
if err != nil {
t.Fatal(err)
}
if !(resp.Node.Key == "/watch_foo/foo" && resp.Node.Value == "bar") {
t.Fatalf("WatchAll 2 failed: %#v", resp)
}
ch := make(chan *Response, 10)
stop := make(chan bool, 1)
routineNum := runtime.NumGoroutine()
go setLoop("watch_foo/foo", "bar", c)
go receiver(ch, stop)
_, err = c.Watch("watch_foo", 0, true, ch, stop)
if err != ErrWatchStoppedByUser {
t.Fatalf("Watch returned a non-user stop error")
}
if newRoutineNum := runtime.NumGoroutine(); newRoutineNum != routineNum {
t.Fatalf("Routine numbers differ after watch stop: %v, %v", routineNum, newRoutineNum)
}
}
func setHelper(key, value string, c *Client) {
time.Sleep(time.Second)
c.Set(key, value, 100)
}
func setLoop(key, value string, c *Client) {
time.Sleep(time.Second)
for i := 0; i < 10; i++ {
newValue := fmt.Sprintf("%s_%v", value, i)
c.Set(key, newValue, 100)
time.Sleep(time.Second / 10)
}
}
func receiver(c chan *Response, stop chan bool) {
for i := 0; i < 10; i++ {
<-c
}
stop <- true
}

View File

@@ -0,0 +1,31 @@
// Code forked from Docker project
package daemon
import (
"errors"
"net"
"os"
)
var SdNotifyNoSocket = errors.New("No socket")
// SdNotify sends a message to the init daemon. It is common to ignore the error.
func SdNotify(state string) error {
socketAddr := &net.UnixAddr{
Name: os.Getenv("NOTIFY_SOCKET"),
Net: "unixgram",
}
if socketAddr.Name == "" {
return SdNotifyNoSocket
}
conn, err := net.DialUnix(socketAddr.Net, nil, socketAddr)
if err != nil {
return err
}
defer conn.Close()
_, err = conn.Write([]byte(state))
return err
}

View File

@@ -0,0 +1,166 @@
// Copyright 2015 CoreOS, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package journal provides write bindings to the systemd journal
package journal
import (
"bytes"
"encoding/binary"
"errors"
"fmt"
"io"
"io/ioutil"
"net"
"os"
"strconv"
"strings"
"syscall"
)
// Priority of a journal message
type Priority int
const (
PriEmerg Priority = iota
PriAlert
PriCrit
PriErr
PriWarning
PriNotice
PriInfo
PriDebug
)
var conn net.Conn
func init() {
var err error
conn, err = net.Dial("unixgram", "/run/systemd/journal/socket")
if err != nil {
conn = nil
}
}
// Enabled returns true iff the systemd journal is available for logging
func Enabled() bool {
return conn != nil
}
// Send a message to the systemd journal. vars is a map of journald fields to
// values. Fields must be composed of uppercase letters, numbers, and
// underscores, but must not start with an underscore. Within these
// restrictions, any arbitrary field name may be used. Some names have special
// significance: see the journalctl documentation
// (http://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html)
// for more details. vars may be nil.
func Send(message string, priority Priority, vars map[string]string) error {
if conn == nil {
return journalError("could not connect to journald socket")
}
data := new(bytes.Buffer)
appendVariable(data, "PRIORITY", strconv.Itoa(int(priority)))
appendVariable(data, "MESSAGE", message)
for k, v := range vars {
appendVariable(data, k, v)
}
_, err := io.Copy(conn, data)
if err != nil && isSocketSpaceError(err) {
file, err := tempFd()
if err != nil {
return journalError(err.Error())
}
_, err = io.Copy(file, data)
if err != nil {
return journalError(err.Error())
}
rights := syscall.UnixRights(int(file.Fd()))
/* this connection should always be a UnixConn, but better safe than sorry */
unixConn, ok := conn.(*net.UnixConn)
if !ok {
return journalError("can't send file through non-Unix connection")
}
unixConn.WriteMsgUnix([]byte{}, rights, nil)
} else if err != nil {
return journalError(err.Error())
}
return nil
}
func appendVariable(w io.Writer, name, value string) {
if !validVarName(name) {
journalError("variable name contains invalid character, ignoring")
}
if strings.ContainsRune(value, '\n') {
/* When the value contains a newline, we write:
* - the variable name, followed by a newline
* - the size (in 64bit little endian format)
* - the data, followed by a newline
*/
fmt.Fprintln(w, name)
binary.Write(w, binary.LittleEndian, uint64(len(value)))
fmt.Fprintln(w, value)
} else {
/* just write the variable and value all on one line */
fmt.Fprintf(w, "%s=%s\n", name, value)
}
}
func validVarName(name string) bool {
/* The variable name must be in uppercase and consist only of characters,
* numbers and underscores, and may not begin with an underscore. (from the docs)
*/
valid := name[0] != '_'
for _, c := range name {
valid = valid && ('A' <= c && c <= 'Z') || ('0' <= c && c <= '9') || c == '_'
}
return valid
}
func isSocketSpaceError(err error) bool {
opErr, ok := err.(*net.OpError)
if !ok {
return false
}
sysErr, ok := opErr.Err.(syscall.Errno)
if !ok {
return false
}
return sysErr == syscall.EMSGSIZE || sysErr == syscall.ENOBUFS
}
func tempFd() (*os.File, error) {
file, err := ioutil.TempFile("/dev/shm/", "journal.XXXXX")
if err != nil {
return nil, err
}
syscall.Unlink(file.Name())
if err != nil {
return nil, err
}
return file, nil
}
func journalError(s string) error {
s = "journal error: " + s
fmt.Fprintln(os.Stderr, s)
return errors.New(s)
}

View File

@@ -0,0 +1,33 @@
// Copyright 2015 CoreOS, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package util contains utility functions related to systemd that applications
// can use to check things like whether systemd is running.
package util
import (
"os"
)
// IsRunningSystemd checks whether the host was booted with systemd as its init
// system. This functions similar to systemd's `sd_booted(3)`: internally, it
// checks whether /run/systemd/system/ exists and is a directory.
// http://www.freedesktop.org/software/systemd/man/sd_booted.html
func IsRunningSystemd() bool {
fi, err := os.Lstat("/run/systemd/system")
if err != nil {
return false
}
return fi.IsDir()
}

View File

@@ -17,7 +17,6 @@ package main
import (
"flag"
oldlog "log"
"os"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/pkg/capnslog"
)
@@ -32,7 +31,6 @@ func init() {
func main() {
rl := capnslog.MustRepoLogger("github.com/coreos/pkg/capnslog/cmd")
capnslog.SetFormatter(capnslog.NewStringFormatter(os.Stderr))
// We can parse the log level configs from the command line
flag.Parse()

View File

@@ -18,6 +18,7 @@ import (
"bufio"
"fmt"
"io"
"runtime"
"strings"
"time"
)
@@ -38,26 +39,68 @@ type StringFormatter struct {
}
func (s *StringFormatter) Format(pkg string, l LogLevel, i int, entries ...interface{}) {
now := time.Now()
y, m, d := now.Date()
h, min, sec := now.Clock()
s.w.WriteString(fmt.Sprintf("%d/%02d/%d %02d:%02d:%02d ", y, m, d, h, min, sec))
s.writeEntries(pkg, l, i, entries...)
now := time.Now().UTC()
s.w.WriteString(now.Format(time.RFC3339))
s.w.WriteByte(' ')
writeEntries(s.w, pkg, l, i, entries...)
s.Flush()
}
func (s *StringFormatter) writeEntries(pkg string, _ LogLevel, _ int, entries ...interface{}) {
func writeEntries(w *bufio.Writer, pkg string, _ LogLevel, _ int, entries ...interface{}) {
if pkg != "" {
s.w.WriteString(pkg + ": ")
w.WriteString(pkg + ": ")
}
str := fmt.Sprint(entries...)
endsInNL := strings.HasSuffix(str, "\n")
s.w.WriteString(str)
w.WriteString(str)
if !endsInNL {
s.w.WriteString("\n")
w.WriteString("\n")
}
s.Flush()
}
func (s *StringFormatter) Flush() {
s.w.Flush()
}
func NewPrettyFormatter(w io.Writer, debug bool) Formatter {
return &PrettyFormatter{
w: bufio.NewWriter(w),
debug: debug,
}
}
type PrettyFormatter struct {
w *bufio.Writer
debug bool
}
func (c *PrettyFormatter) Format(pkg string, l LogLevel, depth int, entries ...interface{}) {
now := time.Now()
ts := now.Format("2006-01-02 15:04:05")
c.w.WriteString(ts)
ms := now.Nanosecond() / 1000
c.w.WriteString(fmt.Sprintf(".%06d", ms))
if c.debug {
_, file, line, ok := runtime.Caller(depth) // It's always the same number of frames to the user's call.
if !ok {
file = "???"
line = 1
} else {
slash := strings.LastIndex(file, "/")
if slash >= 0 {
file = file[slash+1:]
}
}
if line < 0 {
line = 0 // not a real line number
}
c.w.WriteString(fmt.Sprintf(" [%s:%d]", file, line))
}
c.w.WriteString(fmt.Sprint(" ", l.Char(), " | "))
writeEntries(c.w, pkg, l, depth, entries...)
c.Flush()
}
func (c *PrettyFormatter) Flush() {
c.w.Flush()
}

View File

@@ -44,7 +44,7 @@ func (g GlogFormatter) Format(pkg string, level LogLevel, depth int, entries ...
func GlogHeader(level LogLevel, depth int) []byte {
// Lmmdd hh:mm:ss.uuuuuu threadid file:line]
now := time.Now()
now := time.Now().UTC()
_, file, line, ok := runtime.Caller(depth) // It's always the same number of frames to the user's call.
if !ok {
file = "???"
@@ -73,6 +73,7 @@ func GlogHeader(level LogLevel, depth int) []byte {
twoDigits(buf, second)
buf.WriteByte('.')
buf.WriteString(strconv.Itoa(now.Nanosecond() / 1000))
buf.WriteByte('Z')
buf.WriteByte(' ')
buf.WriteString(strconv.Itoa(pid))
buf.WriteByte(' ')

View File

@@ -0,0 +1,47 @@
// Copyright 2015 CoreOS, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package capnslog
import (
"io"
"os"
"syscall"
)
// Here's where the opinionation comes in. We need some sensible defaults,
// especially after taking over the log package. Your project (whatever it may
// be) may see things differently. That's okay; there should be no defaults in
// the main package that cannot be controlled or overridden programatically,
// otherwise it's a bug. Doing so is creating your own init_log.go file much
// like this one.
func init() {
initHijack()
// Go `log` pacakge uses os.Stderr.
SetFormatter(NewDefaultFormatter(os.Stderr))
SetGlobalLogLevel(INFO)
}
func NewDefaultFormatter(out io.Writer) Formatter {
if syscall.Getppid() == 1 {
// We're running under init, which may be systemd.
f, err := NewJournaldFormatter()
if err == nil {
return f
}
}
return NewPrettyFormatter(out, false)
}

View File

@@ -0,0 +1,66 @@
// Copyright 2015 CoreOS, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// +build !windows
package capnslog
import (
"errors"
"fmt"
"os"
"github.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/go-systemd/journal"
)
func NewJournaldFormatter() (Formatter, error) {
if !journal.Enabled() {
return nil, errors.New("No systemd detected")
}
return &journaldFormatter{}, nil
}
type journaldFormatter struct{}
func (j *journaldFormatter) Format(pkg string, l LogLevel, _ int, entries ...interface{}) {
var pri journal.Priority
switch l {
case CRITICAL:
pri = journal.PriCrit
case ERROR:
pri = journal.PriErr
case WARNING:
pri = journal.PriWarning
case NOTICE:
pri = journal.PriNotice
case INFO:
pri = journal.PriInfo
case DEBUG:
pri = journal.PriDebug
case TRACE:
pri = journal.PriDebug
default:
panic("Unhandled loglevel")
}
msg := fmt.Sprint(entries...)
tags := map[string]string{
"PACKAGE": pkg,
}
err := journal.Send(msg, pri, tags)
if err != nil {
fmt.Fprintln(os.Stderr, err)
}
}
func (j *journaldFormatter) Flush() {}

View File

@@ -18,7 +18,7 @@ import (
"log"
)
func init() {
func initHijack() {
pkg := NewPackageLogger("log", "")
w := packageWriter{pkg}
log.SetFlags(0)

View File

@@ -24,7 +24,7 @@ type PackageLogger struct {
level LogLevel
}
const calldepth = 3
const calldepth = 2
func (p *PackageLogger) internalLog(depth int, inLevel LogLevel, entries ...interface{}) {
if inLevel != CRITICAL && p.level < inLevel {

View File

@@ -1,150 +0,0 @@
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
/*
High Performance, Feature-Rich Idiomatic Go codec/encoding library for
binc, msgpack, cbor, json.
Supported Serialization formats are:
- msgpack: https://github.com/msgpack/msgpack
- binc: http://github.com/ugorji/binc
- cbor: http://cbor.io http://tools.ietf.org/html/rfc7049
- json: http://json.org http://tools.ietf.org/html/rfc7159
- simple:
To install:
go get github.com/ugorji/go/codec
This package understands the 'unsafe' tag, to allow using unsafe semantics:
- When decoding into a struct, you need to read the field name as a string
so you can find the struct field it is mapped to.
Using `unsafe` will bypass the allocation and copying overhead of []byte->string conversion.
To install using unsafe, pass the 'unsafe' tag:
go get -tags=unsafe github.com/ugorji/go/codec
For detailed usage information, read the primer at http://ugorji.net/blog/go-codec-primer .
The idiomatic Go support is as seen in other encoding packages in
the standard library (ie json, xml, gob, etc).
Rich Feature Set includes:
- Simple but extremely powerful and feature-rich API
- Very High Performance.
Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X.
- Multiple conversions:
Package coerces types where appropriate
e.g. decode an int in the stream into a float, etc.
- Corner Cases:
Overflows, nil maps/slices, nil values in streams are handled correctly
- Standard field renaming via tags
- Support for omitting empty fields during an encoding
- Encoding from any value and decoding into pointer to any value
(struct, slice, map, primitives, pointers, interface{}, etc)
- Extensions to support efficient encoding/decoding of any named types
- Support encoding.(Binary|Text)(M|Unm)arshaler interfaces
- Decoding without a schema (into a interface{}).
Includes Options to configure what specific map or slice type to use
when decoding an encoded list or map into a nil interface{}
- Encode a struct as an array, and decode struct from an array in the data stream
- Comprehensive support for anonymous fields
- Fast (no-reflection) encoding/decoding of common maps and slices
- Code-generation for faster performance.
- Support binary (e.g. messagepack, cbor) and text (e.g. json) formats
- Support indefinite-length formats to enable true streaming
(for formats which support it e.g. json, cbor)
- Support canonical encoding, where a value is ALWAYS encoded as same sequence of bytes.
This mostly applies to maps, where iteration order is non-deterministic.
- NIL in data stream decoded as zero value
- Never silently skip data when decoding.
User decides whether to return an error or silently skip data when keys or indexes
in the data stream do not map to fields in the struct.
- Encode/Decode from/to chan types (for iterative streaming support)
- Drop-in replacement for encoding/json. `json:` key in struct tag supported.
- Provides a RPC Server and Client Codec for net/rpc communication protocol.
- Handle unique idiosynchracies of codecs e.g.
- For messagepack, configure how ambiguities in handling raw bytes are resolved
- For messagepack, provide rpc server/client codec to support
msgpack-rpc protocol defined at:
https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
Extension Support
Users can register a function to handle the encoding or decoding of
their custom types.
There are no restrictions on what the custom type can be. Some examples:
type BisSet []int
type BitSet64 uint64
type UUID string
type MyStructWithUnexportedFields struct { a int; b bool; c []int; }
type GifImage struct { ... }
As an illustration, MyStructWithUnexportedFields would normally be
encoded as an empty map because it has no exported fields, while UUID
would be encoded as a string. However, with extension support, you can
encode any of these however you like.
RPC
RPC Client and Server Codecs are implemented, so the codecs can be used
with the standard net/rpc package.
Usage
Typical usage model:
// create and configure Handle
var (
bh codec.BincHandle
mh codec.MsgpackHandle
ch codec.CborHandle
)
mh.MapType = reflect.TypeOf(map[string]interface{}(nil))
// configure extensions
// e.g. for msgpack, define functions and enable Time support for tag 1
// mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt)
// create and use decoder/encoder
var (
r io.Reader
w io.Writer
b []byte
h = &bh // or mh to use msgpack
)
dec = codec.NewDecoder(r, h)
dec = codec.NewDecoderBytes(b, h)
err = dec.Decode(&v)
enc = codec.NewEncoder(w, h)
enc = codec.NewEncoderBytes(&b, h)
err = enc.Encode(v)
//RPC Server
go func() {
for {
conn, err := listener.Accept()
rpcCodec := codec.GoRpc.ServerCodec(conn, h)
//OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h)
rpc.ServeCodec(rpcCodec)
}
}()
//RPC Communication (client side)
conn, err = net.Dial("tcp", "localhost:5555")
rpcCodec := codec.GoRpc.ClientCodec(conn, h)
//OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
client := rpc.NewClientWithCodec(rpcCodec)
*/
package codec

View File

@@ -1,148 +0,0 @@
# Codec
High Performance, Feature-Rich Idiomatic Go codec/encoding library for
binc, msgpack, cbor, json.
Supported Serialization formats are:
- msgpack: https://github.com/msgpack/msgpack
- binc: http://github.com/ugorji/binc
- cbor: http://cbor.io http://tools.ietf.org/html/rfc7049
- json: http://json.org http://tools.ietf.org/html/rfc7159
- simple:
To install:
go get github.com/ugorji/go/codec
This package understands the `unsafe` tag, to allow using unsafe semantics:
- When decoding into a struct, you need to read the field name as a string
so you can find the struct field it is mapped to.
Using `unsafe` will bypass the allocation and copying overhead of `[]byte->string` conversion.
To use it, you must pass the `unsafe` tag during install:
```
go install -tags=unsafe github.com/ugorji/go/codec
```
Online documentation: http://godoc.org/github.com/ugorji/go/codec
Detailed Usage/How-to Primer: http://ugorji.net/blog/go-codec-primer
The idiomatic Go support is as seen in other encoding packages in
the standard library (ie json, xml, gob, etc).
Rich Feature Set includes:
- Simple but extremely powerful and feature-rich API
- Very High Performance.
Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X.
- Multiple conversions:
Package coerces types where appropriate
e.g. decode an int in the stream into a float, etc.
- Corner Cases:
Overflows, nil maps/slices, nil values in streams are handled correctly
- Standard field renaming via tags
- Support for omitting empty fields during an encoding
- Encoding from any value and decoding into pointer to any value
(struct, slice, map, primitives, pointers, interface{}, etc)
- Extensions to support efficient encoding/decoding of any named types
- Support encoding.(Binary|Text)(M|Unm)arshaler interfaces
- Decoding without a schema (into a interface{}).
Includes Options to configure what specific map or slice type to use
when decoding an encoded list or map into a nil interface{}
- Encode a struct as an array, and decode struct from an array in the data stream
- Comprehensive support for anonymous fields
- Fast (no-reflection) encoding/decoding of common maps and slices
- Code-generation for faster performance.
- Support binary (e.g. messagepack, cbor) and text (e.g. json) formats
- Support indefinite-length formats to enable true streaming
(for formats which support it e.g. json, cbor)
- Support canonical encoding, where a value is ALWAYS encoded as same sequence of bytes.
This mostly applies to maps, where iteration order is non-deterministic.
- NIL in data stream decoded as zero value
- Never silently skip data when decoding.
User decides whether to return an error or silently skip data when keys or indexes
in the data stream do not map to fields in the struct.
- Encode/Decode from/to chan types (for iterative streaming support)
- Drop-in replacement for encoding/json. `json:` key in struct tag supported.
- Provides a RPC Server and Client Codec for net/rpc communication protocol.
- Handle unique idiosynchracies of codecs e.g.
- For messagepack, configure how ambiguities in handling raw bytes are resolved
- For messagepack, provide rpc server/client codec to support
msgpack-rpc protocol defined at:
https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
## Extension Support
Users can register a function to handle the encoding or decoding of
their custom types.
There are no restrictions on what the custom type can be. Some examples:
type BisSet []int
type BitSet64 uint64
type UUID string
type MyStructWithUnexportedFields struct { a int; b bool; c []int; }
type GifImage struct { ... }
As an illustration, MyStructWithUnexportedFields would normally be
encoded as an empty map because it has no exported fields, while UUID
would be encoded as a string. However, with extension support, you can
encode any of these however you like.
## RPC
RPC Client and Server Codecs are implemented, so the codecs can be used
with the standard net/rpc package.
## Usage
Typical usage model:
// create and configure Handle
var (
bh codec.BincHandle
mh codec.MsgpackHandle
ch codec.CborHandle
)
mh.MapType = reflect.TypeOf(map[string]interface{}(nil))
// configure extensions
// e.g. for msgpack, define functions and enable Time support for tag 1
// mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt)
// create and use decoder/encoder
var (
r io.Reader
w io.Writer
b []byte
h = &bh // or mh to use msgpack
)
dec = codec.NewDecoder(r, h)
dec = codec.NewDecoderBytes(b, h)
err = dec.Decode(&v)
enc = codec.NewEncoder(w, h)
enc = codec.NewEncoderBytes(&b, h)
err = enc.Encode(v)
//RPC Server
go func() {
for {
conn, err := listener.Accept()
rpcCodec := codec.GoRpc.ServerCodec(conn, h)
//OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h)
rpc.ServeCodec(rpcCodec)
}
}()
//RPC Communication (client side)
conn, err = net.Dial("tcp", "localhost:5555")
rpcCodec := codec.GoRpc.ClientCodec(conn, h)
//OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
client := rpc.NewClientWithCodec(rpcCodec)

View File

@@ -1,901 +0,0 @@
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
import (
"math"
"time"
)
const bincDoPrune = true // No longer needed. Needed before as C lib did not support pruning.
// vd as low 4 bits (there are 16 slots)
const (
bincVdSpecial byte = iota
bincVdPosInt
bincVdNegInt
bincVdFloat
bincVdString
bincVdByteArray
bincVdArray
bincVdMap
bincVdTimestamp
bincVdSmallInt
bincVdUnicodeOther
bincVdSymbol
bincVdDecimal
_ // open slot
_ // open slot
bincVdCustomExt = 0x0f
)
const (
bincSpNil byte = iota
bincSpFalse
bincSpTrue
bincSpNan
bincSpPosInf
bincSpNegInf
bincSpZeroFloat
bincSpZero
bincSpNegOne
)
const (
bincFlBin16 byte = iota
bincFlBin32
_ // bincFlBin32e
bincFlBin64
_ // bincFlBin64e
// others not currently supported
)
type bincEncDriver struct {
e *Encoder
w encWriter
m map[string]uint16 // symbols
s uint16 // symbols sequencer
b [scratchByteArrayLen]byte
encNoSeparator
}
func (e *bincEncDriver) IsBuiltinType(rt uintptr) bool {
return rt == timeTypId
}
func (e *bincEncDriver) EncodeBuiltin(rt uintptr, v interface{}) {
if rt == timeTypId {
bs := encodeTime(v.(time.Time))
e.w.writen1(bincVdTimestamp<<4 | uint8(len(bs)))
e.w.writeb(bs)
}
}
func (e *bincEncDriver) EncodeNil() {
e.w.writen1(bincVdSpecial<<4 | bincSpNil)
}
func (e *bincEncDriver) EncodeBool(b bool) {
if b {
e.w.writen1(bincVdSpecial<<4 | bincSpTrue)
} else {
e.w.writen1(bincVdSpecial<<4 | bincSpFalse)
}
}
func (e *bincEncDriver) EncodeFloat32(f float32) {
if f == 0 {
e.w.writen1(bincVdSpecial<<4 | bincSpZeroFloat)
return
}
e.w.writen1(bincVdFloat<<4 | bincFlBin32)
bigenHelper{e.b[:4], e.w}.writeUint32(math.Float32bits(f))
}
func (e *bincEncDriver) EncodeFloat64(f float64) {
if f == 0 {
e.w.writen1(bincVdSpecial<<4 | bincSpZeroFloat)
return
}
bigen.PutUint64(e.b[:8], math.Float64bits(f))
if bincDoPrune {
i := 7
for ; i >= 0 && (e.b[i] == 0); i-- {
}
i++
if i <= 6 {
e.w.writen1(bincVdFloat<<4 | 0x8 | bincFlBin64)
e.w.writen1(byte(i))
e.w.writeb(e.b[:i])
return
}
}
e.w.writen1(bincVdFloat<<4 | bincFlBin64)
e.w.writeb(e.b[:8])
}
func (e *bincEncDriver) encIntegerPrune(bd byte, pos bool, v uint64, lim uint8) {
if lim == 4 {
bigen.PutUint32(e.b[:lim], uint32(v))
} else {
bigen.PutUint64(e.b[:lim], v)
}
if bincDoPrune {
i := pruneSignExt(e.b[:lim], pos)
e.w.writen1(bd | lim - 1 - byte(i))
e.w.writeb(e.b[i:lim])
} else {
e.w.writen1(bd | lim - 1)
e.w.writeb(e.b[:lim])
}
}
func (e *bincEncDriver) EncodeInt(v int64) {
const nbd byte = bincVdNegInt << 4
if v >= 0 {
e.encUint(bincVdPosInt<<4, true, uint64(v))
} else if v == -1 {
e.w.writen1(bincVdSpecial<<4 | bincSpNegOne)
} else {
e.encUint(bincVdNegInt<<4, false, uint64(-v))
}
}
func (e *bincEncDriver) EncodeUint(v uint64) {
e.encUint(bincVdPosInt<<4, true, v)
}
func (e *bincEncDriver) encUint(bd byte, pos bool, v uint64) {
if v == 0 {
e.w.writen1(bincVdSpecial<<4 | bincSpZero)
} else if pos && v >= 1 && v <= 16 {
e.w.writen1(bincVdSmallInt<<4 | byte(v-1))
} else if v <= math.MaxUint8 {
e.w.writen2(bd|0x0, byte(v))
} else if v <= math.MaxUint16 {
e.w.writen1(bd | 0x01)
bigenHelper{e.b[:2], e.w}.writeUint16(uint16(v))
} else if v <= math.MaxUint32 {
e.encIntegerPrune(bd, pos, v, 4)
} else {
e.encIntegerPrune(bd, pos, v, 8)
}
}
func (e *bincEncDriver) EncodeExt(rv interface{}, xtag uint64, ext Ext, _ *Encoder) {
bs := ext.WriteExt(rv)
if bs == nil {
e.EncodeNil()
return
}
e.encodeExtPreamble(uint8(xtag), len(bs))
e.w.writeb(bs)
}
func (e *bincEncDriver) EncodeRawExt(re *RawExt, _ *Encoder) {
e.encodeExtPreamble(uint8(re.Tag), len(re.Data))
e.w.writeb(re.Data)
}
func (e *bincEncDriver) encodeExtPreamble(xtag byte, length int) {
e.encLen(bincVdCustomExt<<4, uint64(length))
e.w.writen1(xtag)
}
func (e *bincEncDriver) EncodeArrayStart(length int) {
e.encLen(bincVdArray<<4, uint64(length))
}
func (e *bincEncDriver) EncodeMapStart(length int) {
e.encLen(bincVdMap<<4, uint64(length))
}
func (e *bincEncDriver) EncodeString(c charEncoding, v string) {
l := uint64(len(v))
e.encBytesLen(c, l)
if l > 0 {
e.w.writestr(v)
}
}
func (e *bincEncDriver) EncodeSymbol(v string) {
// if WriteSymbolsNoRefs {
// e.encodeString(c_UTF8, v)
// return
// }
//symbols only offer benefit when string length > 1.
//This is because strings with length 1 take only 2 bytes to store
//(bd with embedded length, and single byte for string val).
l := len(v)
if l == 0 {
e.encBytesLen(c_UTF8, 0)
return
} else if l == 1 {
e.encBytesLen(c_UTF8, 1)
e.w.writen1(v[0])
return
}
if e.m == nil {
e.m = make(map[string]uint16, 16)
}
ui, ok := e.m[v]
if ok {
if ui <= math.MaxUint8 {
e.w.writen2(bincVdSymbol<<4, byte(ui))
} else {
e.w.writen1(bincVdSymbol<<4 | 0x8)
bigenHelper{e.b[:2], e.w}.writeUint16(ui)
}
} else {
e.s++
ui = e.s
//ui = uint16(atomic.AddUint32(&e.s, 1))
e.m[v] = ui
var lenprec uint8
if l <= math.MaxUint8 {
// lenprec = 0
} else if l <= math.MaxUint16 {
lenprec = 1
} else if int64(l) <= math.MaxUint32 {
lenprec = 2
} else {
lenprec = 3
}
if ui <= math.MaxUint8 {
e.w.writen2(bincVdSymbol<<4|0x0|0x4|lenprec, byte(ui))
} else {
e.w.writen1(bincVdSymbol<<4 | 0x8 | 0x4 | lenprec)
bigenHelper{e.b[:2], e.w}.writeUint16(ui)
}
if lenprec == 0 {
e.w.writen1(byte(l))
} else if lenprec == 1 {
bigenHelper{e.b[:2], e.w}.writeUint16(uint16(l))
} else if lenprec == 2 {
bigenHelper{e.b[:4], e.w}.writeUint32(uint32(l))
} else {
bigenHelper{e.b[:8], e.w}.writeUint64(uint64(l))
}
e.w.writestr(v)
}
}
func (e *bincEncDriver) EncodeStringBytes(c charEncoding, v []byte) {
l := uint64(len(v))
e.encBytesLen(c, l)
if l > 0 {
e.w.writeb(v)
}
}
func (e *bincEncDriver) encBytesLen(c charEncoding, length uint64) {
//TODO: support bincUnicodeOther (for now, just use string or bytearray)
if c == c_RAW {
e.encLen(bincVdByteArray<<4, length)
} else {
e.encLen(bincVdString<<4, length)
}
}
func (e *bincEncDriver) encLen(bd byte, l uint64) {
if l < 12 {
e.w.writen1(bd | uint8(l+4))
} else {
e.encLenNumber(bd, l)
}
}
func (e *bincEncDriver) encLenNumber(bd byte, v uint64) {
if v <= math.MaxUint8 {
e.w.writen2(bd, byte(v))
} else if v <= math.MaxUint16 {
e.w.writen1(bd | 0x01)
bigenHelper{e.b[:2], e.w}.writeUint16(uint16(v))
} else if v <= math.MaxUint32 {
e.w.writen1(bd | 0x02)
bigenHelper{e.b[:4], e.w}.writeUint32(uint32(v))
} else {
e.w.writen1(bd | 0x03)
bigenHelper{e.b[:8], e.w}.writeUint64(uint64(v))
}
}
//------------------------------------
type bincDecSymbol struct {
i uint16
s string
b []byte
}
type bincDecDriver struct {
d *Decoder
h *BincHandle
r decReader
br bool // bytes reader
bdRead bool
bdType valueType
bd byte
vd byte
vs byte
noStreamingCodec
decNoSeparator
b [scratchByteArrayLen]byte
// linear searching on this slice is ok,
// because we typically expect < 32 symbols in each stream.
s []bincDecSymbol
}
func (d *bincDecDriver) readNextBd() {
d.bd = d.r.readn1()
d.vd = d.bd >> 4
d.vs = d.bd & 0x0f
d.bdRead = true
d.bdType = valueTypeUnset
}
func (d *bincDecDriver) IsContainerType(vt valueType) (b bool) {
switch vt {
case valueTypeNil:
return d.vd == bincVdSpecial && d.vs == bincSpNil
case valueTypeBytes:
return d.vd == bincVdByteArray
case valueTypeString:
return d.vd == bincVdString
case valueTypeArray:
return d.vd == bincVdArray
case valueTypeMap:
return d.vd == bincVdMap
}
d.d.errorf("isContainerType: unsupported parameter: %v", vt)
return // "unreachable"
}
func (d *bincDecDriver) TryDecodeAsNil() bool {
if !d.bdRead {
d.readNextBd()
}
if d.bd == bincVdSpecial<<4|bincSpNil {
d.bdRead = false
return true
}
return false
}
func (d *bincDecDriver) IsBuiltinType(rt uintptr) bool {
return rt == timeTypId
}
func (d *bincDecDriver) DecodeBuiltin(rt uintptr, v interface{}) {
if !d.bdRead {
d.readNextBd()
}
if rt == timeTypId {
if d.vd != bincVdTimestamp {
d.d.errorf("Invalid d.vd. Expecting 0x%x. Received: 0x%x", bincVdTimestamp, d.vd)
return
}
tt, err := decodeTime(d.r.readx(int(d.vs)))
if err != nil {
panic(err)
}
var vt *time.Time = v.(*time.Time)
*vt = tt
d.bdRead = false
}
}
func (d *bincDecDriver) decFloatPre(vs, defaultLen byte) {
if vs&0x8 == 0 {
d.r.readb(d.b[0:defaultLen])
} else {
l := d.r.readn1()
if l > 8 {
d.d.errorf("At most 8 bytes used to represent float. Received: %v bytes", l)
return
}
for i := l; i < 8; i++ {
d.b[i] = 0
}
d.r.readb(d.b[0:l])
}
}
func (d *bincDecDriver) decFloat() (f float64) {
//if true { f = math.Float64frombits(bigen.Uint64(d.r.readx(8))); break; }
if x := d.vs & 0x7; x == bincFlBin32 {
d.decFloatPre(d.vs, 4)
f = float64(math.Float32frombits(bigen.Uint32(d.b[0:4])))
} else if x == bincFlBin64 {
d.decFloatPre(d.vs, 8)
f = math.Float64frombits(bigen.Uint64(d.b[0:8]))
} else {
d.d.errorf("only float32 and float64 are supported. d.vd: 0x%x, d.vs: 0x%x", d.vd, d.vs)
return
}
return
}
func (d *bincDecDriver) decUint() (v uint64) {
// need to inline the code (interface conversion and type assertion expensive)
switch d.vs {
case 0:
v = uint64(d.r.readn1())
case 1:
d.r.readb(d.b[6:8])
v = uint64(bigen.Uint16(d.b[6:8]))
case 2:
d.b[4] = 0
d.r.readb(d.b[5:8])
v = uint64(bigen.Uint32(d.b[4:8]))
case 3:
d.r.readb(d.b[4:8])
v = uint64(bigen.Uint32(d.b[4:8]))
case 4, 5, 6:
lim := int(7 - d.vs)
d.r.readb(d.b[lim:8])
for i := 0; i < lim; i++ {
d.b[i] = 0
}
v = uint64(bigen.Uint64(d.b[:8]))
case 7:
d.r.readb(d.b[:8])
v = uint64(bigen.Uint64(d.b[:8]))
default:
d.d.errorf("unsigned integers with greater than 64 bits of precision not supported")
return
}
return
}
func (d *bincDecDriver) decCheckInteger() (ui uint64, neg bool) {
if !d.bdRead {
d.readNextBd()
}
vd, vs := d.vd, d.vs
if vd == bincVdPosInt {
ui = d.decUint()
} else if vd == bincVdNegInt {
ui = d.decUint()
neg = true
} else if vd == bincVdSmallInt {
ui = uint64(d.vs) + 1
} else if vd == bincVdSpecial {
if vs == bincSpZero {
//i = 0
} else if vs == bincSpNegOne {
neg = true
ui = 1
} else {
d.d.errorf("numeric decode fails for special value: d.vs: 0x%x", d.vs)
return
}
} else {
d.d.errorf("number can only be decoded from uint or int values. d.bd: 0x%x, d.vd: 0x%x", d.bd, d.vd)
return
}
return
}
func (d *bincDecDriver) DecodeInt(bitsize uint8) (i int64) {
ui, neg := d.decCheckInteger()
i, overflow := chkOvf.SignedInt(ui)
if overflow {
d.d.errorf("simple: overflow converting %v to signed integer", ui)
return
}
if neg {
i = -i
}
if chkOvf.Int(i, bitsize) {
d.d.errorf("binc: overflow integer: %v", i)
return
}
d.bdRead = false
return
}
func (d *bincDecDriver) DecodeUint(bitsize uint8) (ui uint64) {
ui, neg := d.decCheckInteger()
if neg {
d.d.errorf("Assigning negative signed value to unsigned type")
return
}
if chkOvf.Uint(ui, bitsize) {
d.d.errorf("binc: overflow integer: %v", ui)
return
}
d.bdRead = false
return
}
func (d *bincDecDriver) DecodeFloat(chkOverflow32 bool) (f float64) {
if !d.bdRead {
d.readNextBd()
}
vd, vs := d.vd, d.vs
if vd == bincVdSpecial {
d.bdRead = false
if vs == bincSpNan {
return math.NaN()
} else if vs == bincSpPosInf {
return math.Inf(1)
} else if vs == bincSpZeroFloat || vs == bincSpZero {
return
} else if vs == bincSpNegInf {
return math.Inf(-1)
} else {
d.d.errorf("Invalid d.vs decoding float where d.vd=bincVdSpecial: %v", d.vs)
return
}
} else if vd == bincVdFloat {
f = d.decFloat()
} else {
f = float64(d.DecodeInt(64))
}
if chkOverflow32 && chkOvf.Float32(f) {
d.d.errorf("binc: float32 overflow: %v", f)
return
}
d.bdRead = false
return
}
// bool can be decoded from bool only (single byte).
func (d *bincDecDriver) DecodeBool() (b bool) {
if !d.bdRead {
d.readNextBd()
}
if bd := d.bd; bd == (bincVdSpecial | bincSpFalse) {
// b = false
} else if bd == (bincVdSpecial | bincSpTrue) {
b = true
} else {
d.d.errorf("Invalid single-byte value for bool: %s: %x", msgBadDesc, d.bd)
return
}
d.bdRead = false
return
}
func (d *bincDecDriver) ReadMapStart() (length int) {
if d.vd != bincVdMap {
d.d.errorf("Invalid d.vd for map. Expecting 0x%x. Got: 0x%x", bincVdMap, d.vd)
return
}
length = d.decLen()
d.bdRead = false
return
}
func (d *bincDecDriver) ReadArrayStart() (length int) {
if d.vd != bincVdArray {
d.d.errorf("Invalid d.vd for array. Expecting 0x%x. Got: 0x%x", bincVdArray, d.vd)
return
}
length = d.decLen()
d.bdRead = false
return
}
func (d *bincDecDriver) decLen() int {
if d.vs > 3 {
return int(d.vs - 4)
}
return int(d.decLenNumber())
}
func (d *bincDecDriver) decLenNumber() (v uint64) {
if x := d.vs; x == 0 {
v = uint64(d.r.readn1())
} else if x == 1 {
d.r.readb(d.b[6:8])
v = uint64(bigen.Uint16(d.b[6:8]))
} else if x == 2 {
d.r.readb(d.b[4:8])
v = uint64(bigen.Uint32(d.b[4:8]))
} else {
d.r.readb(d.b[:8])
v = bigen.Uint64(d.b[:8])
}
return
}
func (d *bincDecDriver) decStringAndBytes(bs []byte, withString, zerocopy bool) (bs2 []byte, s string) {
if !d.bdRead {
d.readNextBd()
}
if d.bd == bincVdSpecial<<4|bincSpNil {
d.bdRead = false
return
}
var slen int = -1
// var ok bool
switch d.vd {
case bincVdString, bincVdByteArray:
slen = d.decLen()
if zerocopy {
if d.br {
bs2 = d.r.readx(slen)
} else if len(bs) == 0 {
bs2 = decByteSlice(d.r, slen, d.b[:])
} else {
bs2 = decByteSlice(d.r, slen, bs)
}
} else {
bs2 = decByteSlice(d.r, slen, bs)
}
if withString {
s = string(bs2)
}
case bincVdSymbol:
// zerocopy doesn't apply for symbols,
// as the values must be stored in a table for later use.
//
//from vs: extract numSymbolBytes, containsStringVal, strLenPrecision,
//extract symbol
//if containsStringVal, read it and put in map
//else look in map for string value
var symbol uint16
vs := d.vs
if vs&0x8 == 0 {
symbol = uint16(d.r.readn1())
} else {
symbol = uint16(bigen.Uint16(d.r.readx(2)))
}
if d.s == nil {
d.s = make([]bincDecSymbol, 0, 16)
}
if vs&0x4 == 0 {
for i := range d.s {
j := &d.s[i]
if j.i == symbol {
bs2 = j.b
if withString {
if j.s == "" && bs2 != nil {
j.s = string(bs2)
}
s = j.s
}
break
}
}
} else {
switch vs & 0x3 {
case 0:
slen = int(d.r.readn1())
case 1:
slen = int(bigen.Uint16(d.r.readx(2)))
case 2:
slen = int(bigen.Uint32(d.r.readx(4)))
case 3:
slen = int(bigen.Uint64(d.r.readx(8)))
}
// since using symbols, do not store any part of
// the parameter bs in the map, as it might be a shared buffer.
// bs2 = decByteSlice(d.r, slen, bs)
bs2 = decByteSlice(d.r, slen, nil)
if withString {
s = string(bs2)
}
d.s = append(d.s, bincDecSymbol{symbol, s, bs2})
}
default:
d.d.errorf("Invalid d.vd. Expecting string:0x%x, bytearray:0x%x or symbol: 0x%x. Got: 0x%x",
bincVdString, bincVdByteArray, bincVdSymbol, d.vd)
return
}
d.bdRead = false
return
}
func (d *bincDecDriver) DecodeString() (s string) {
// DecodeBytes does not accomodate symbols, whose impl stores string version in map.
// Use decStringAndBytes directly.
// return string(d.DecodeBytes(d.b[:], true, true))
_, s = d.decStringAndBytes(d.b[:], true, true)
return
}
func (d *bincDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOut []byte) {
if isstring {
bsOut, _ = d.decStringAndBytes(bs, false, zerocopy)
return
}
if !d.bdRead {
d.readNextBd()
}
if d.bd == bincVdSpecial<<4|bincSpNil {
d.bdRead = false
return nil
}
var clen int
if d.vd == bincVdString || d.vd == bincVdByteArray {
clen = d.decLen()
} else {
d.d.errorf("Invalid d.vd for bytes. Expecting string:0x%x or bytearray:0x%x. Got: 0x%x",
bincVdString, bincVdByteArray, d.vd)
return
}
d.bdRead = false
if zerocopy {
if d.br {
return d.r.readx(clen)
} else if len(bs) == 0 {
bs = d.b[:]
}
}
return decByteSlice(d.r, clen, bs)
}
func (d *bincDecDriver) DecodeExt(rv interface{}, xtag uint64, ext Ext) (realxtag uint64) {
if xtag > 0xff {
d.d.errorf("decodeExt: tag must be <= 0xff; got: %v", xtag)
return
}
realxtag1, xbs := d.decodeExtV(ext != nil, uint8(xtag))
realxtag = uint64(realxtag1)
if ext == nil {
re := rv.(*RawExt)
re.Tag = realxtag
re.Data = detachZeroCopyBytes(d.br, re.Data, xbs)
} else {
ext.ReadExt(rv, xbs)
}
return
}
func (d *bincDecDriver) decodeExtV(verifyTag bool, tag byte) (xtag byte, xbs []byte) {
if !d.bdRead {
d.readNextBd()
}
if d.vd == bincVdCustomExt {
l := d.decLen()
xtag = d.r.readn1()
if verifyTag && xtag != tag {
d.d.errorf("Wrong extension tag. Got %b. Expecting: %v", xtag, tag)
return
}
xbs = d.r.readx(l)
} else if d.vd == bincVdByteArray {
xbs = d.DecodeBytes(nil, false, true)
} else {
d.d.errorf("Invalid d.vd for extensions (Expecting extensions or byte array). Got: 0x%x", d.vd)
return
}
d.bdRead = false
return
}
func (d *bincDecDriver) DecodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
if !d.bdRead {
d.readNextBd()
}
switch d.vd {
case bincVdSpecial:
switch d.vs {
case bincSpNil:
vt = valueTypeNil
case bincSpFalse:
vt = valueTypeBool
v = false
case bincSpTrue:
vt = valueTypeBool
v = true
case bincSpNan:
vt = valueTypeFloat
v = math.NaN()
case bincSpPosInf:
vt = valueTypeFloat
v = math.Inf(1)
case bincSpNegInf:
vt = valueTypeFloat
v = math.Inf(-1)
case bincSpZeroFloat:
vt = valueTypeFloat
v = float64(0)
case bincSpZero:
vt = valueTypeUint
v = uint64(0) // int8(0)
case bincSpNegOne:
vt = valueTypeInt
v = int64(-1) // int8(-1)
default:
d.d.errorf("decodeNaked: Unrecognized special value 0x%x", d.vs)
return
}
case bincVdSmallInt:
vt = valueTypeUint
v = uint64(int8(d.vs)) + 1 // int8(d.vs) + 1
case bincVdPosInt:
vt = valueTypeUint
v = d.decUint()
case bincVdNegInt:
vt = valueTypeInt
v = -(int64(d.decUint()))
case bincVdFloat:
vt = valueTypeFloat
v = d.decFloat()
case bincVdSymbol:
vt = valueTypeSymbol
v = d.DecodeString()
case bincVdString:
vt = valueTypeString
v = d.DecodeString()
case bincVdByteArray:
vt = valueTypeBytes
v = d.DecodeBytes(nil, false, false)
case bincVdTimestamp:
vt = valueTypeTimestamp
tt, err := decodeTime(d.r.readx(int(d.vs)))
if err != nil {
panic(err)
}
v = tt
case bincVdCustomExt:
vt = valueTypeExt
l := d.decLen()
var re RawExt
re.Tag = uint64(d.r.readn1())
re.Data = d.r.readx(l)
v = &re
vt = valueTypeExt
case bincVdArray:
vt = valueTypeArray
decodeFurther = true
case bincVdMap:
vt = valueTypeMap
decodeFurther = true
default:
d.d.errorf("decodeNaked: Unrecognized d.vd: 0x%x", d.vd)
return
}
if !decodeFurther {
d.bdRead = false
}
if vt == valueTypeUint && d.h.SignedInteger {
d.bdType = valueTypeInt
v = int64(v.(uint64))
}
return
}
//------------------------------------
//BincHandle is a Handle for the Binc Schema-Free Encoding Format
//defined at https://github.com/ugorji/binc .
//
//BincHandle currently supports all Binc features with the following EXCEPTIONS:
// - only integers up to 64 bits of precision are supported.
// big integers are unsupported.
// - Only IEEE 754 binary32 and binary64 floats are supported (ie Go float32 and float64 types).
// extended precision and decimal IEEE 754 floats are unsupported.
// - Only UTF-8 strings supported.
// Unicode_Other Binc types (UTF16, UTF32) are currently unsupported.
//
//Note that these EXCEPTIONS are temporary and full support is possible and may happen soon.
type BincHandle struct {
BasicHandle
binaryEncodingType
}
func (h *BincHandle) newEncDriver(e *Encoder) encDriver {
return &bincEncDriver{e: e, w: e.w}
}
func (h *BincHandle) newDecDriver(d *Decoder) decDriver {
return &bincDecDriver{d: d, r: d.r, h: h, br: d.bytes}
}
var _ decDriver = (*bincDecDriver)(nil)
var _ encDriver = (*bincEncDriver)(nil)

View File

@@ -1,566 +0,0 @@
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
import "math"
const (
cborMajorUint byte = iota
cborMajorNegInt
cborMajorBytes
cborMajorText
cborMajorArray
cborMajorMap
cborMajorTag
cborMajorOther
)
const (
cborBdFalse byte = 0xf4 + iota
cborBdTrue
cborBdNil
cborBdUndefined
cborBdExt
cborBdFloat16
cborBdFloat32
cborBdFloat64
)
const (
cborBdIndefiniteBytes byte = 0x5f
cborBdIndefiniteString = 0x7f
cborBdIndefiniteArray = 0x9f
cborBdIndefiniteMap = 0xbf
cborBdBreak = 0xff
)
const (
CborStreamBytes byte = 0x5f
CborStreamString = 0x7f
CborStreamArray = 0x9f
CborStreamMap = 0xbf
CborStreamBreak = 0xff
)
const (
cborBaseUint byte = 0x00
cborBaseNegInt = 0x20
cborBaseBytes = 0x40
cborBaseString = 0x60
cborBaseArray = 0x80
cborBaseMap = 0xa0
cborBaseTag = 0xc0
cborBaseSimple = 0xe0
)
// -------------------
type cborEncDriver struct {
e *Encoder
w encWriter
h *CborHandle
noBuiltInTypes
encNoSeparator
x [8]byte
}
func (e *cborEncDriver) EncodeNil() {
e.w.writen1(cborBdNil)
}
func (e *cborEncDriver) EncodeBool(b bool) {
if b {
e.w.writen1(cborBdTrue)
} else {
e.w.writen1(cborBdFalse)
}
}
func (e *cborEncDriver) EncodeFloat32(f float32) {
e.w.writen1(cborBdFloat32)
bigenHelper{e.x[:4], e.w}.writeUint32(math.Float32bits(f))
}
func (e *cborEncDriver) EncodeFloat64(f float64) {
e.w.writen1(cborBdFloat64)
bigenHelper{e.x[:8], e.w}.writeUint64(math.Float64bits(f))
}
func (e *cborEncDriver) encUint(v uint64, bd byte) {
if v <= 0x17 {
e.w.writen1(byte(v) + bd)
} else if v <= math.MaxUint8 {
e.w.writen2(bd+0x18, uint8(v))
} else if v <= math.MaxUint16 {
e.w.writen1(bd + 0x19)
bigenHelper{e.x[:2], e.w}.writeUint16(uint16(v))
} else if v <= math.MaxUint32 {
e.w.writen1(bd + 0x1a)
bigenHelper{e.x[:4], e.w}.writeUint32(uint32(v))
} else if v <= math.MaxUint64 {
e.w.writen1(bd + 0x1b)
bigenHelper{e.x[:8], e.w}.writeUint64(v)
}
}
func (e *cborEncDriver) EncodeInt(v int64) {
if v < 0 {
e.encUint(uint64(-1-v), cborBaseNegInt)
} else {
e.encUint(uint64(v), cborBaseUint)
}
}
func (e *cborEncDriver) EncodeUint(v uint64) {
e.encUint(v, cborBaseUint)
}
func (e *cborEncDriver) encLen(bd byte, length int) {
e.encUint(uint64(length), bd)
}
func (e *cborEncDriver) EncodeExt(rv interface{}, xtag uint64, ext Ext, en *Encoder) {
e.encUint(uint64(xtag), cborBaseTag)
if v := ext.ConvertExt(rv); v == nil {
e.EncodeNil()
} else {
en.encode(v)
}
}
func (e *cborEncDriver) EncodeRawExt(re *RawExt, en *Encoder) {
e.encUint(uint64(re.Tag), cborBaseTag)
if re.Data != nil {
en.encode(re.Data)
} else if re.Value == nil {
e.EncodeNil()
} else {
en.encode(re.Value)
}
}
func (e *cborEncDriver) EncodeArrayStart(length int) {
e.encLen(cborBaseArray, length)
}
func (e *cborEncDriver) EncodeMapStart(length int) {
e.encLen(cborBaseMap, length)
}
func (e *cborEncDriver) EncodeString(c charEncoding, v string) {
e.encLen(cborBaseString, len(v))
e.w.writestr(v)
}
func (e *cborEncDriver) EncodeSymbol(v string) {
e.EncodeString(c_UTF8, v)
}
func (e *cborEncDriver) EncodeStringBytes(c charEncoding, v []byte) {
e.encLen(cborBaseBytes, len(v))
e.w.writeb(v)
}
// ----------------------
type cborDecDriver struct {
d *Decoder
h *CborHandle
r decReader
br bool // bytes reader
bdRead bool
bdType valueType
bd byte
b [scratchByteArrayLen]byte
noBuiltInTypes
decNoSeparator
}
func (d *cborDecDriver) readNextBd() {
d.bd = d.r.readn1()
d.bdRead = true
d.bdType = valueTypeUnset
}
func (d *cborDecDriver) IsContainerType(vt valueType) (bv bool) {
switch vt {
case valueTypeNil:
return d.bd == cborBdNil
case valueTypeBytes:
return d.bd == cborBdIndefiniteBytes || (d.bd >= cborBaseBytes && d.bd < cborBaseString)
case valueTypeString:
return d.bd == cborBdIndefiniteString || (d.bd >= cborBaseString && d.bd < cborBaseArray)
case valueTypeArray:
return d.bd == cborBdIndefiniteArray || (d.bd >= cborBaseArray && d.bd < cborBaseMap)
case valueTypeMap:
return d.bd == cborBdIndefiniteMap || (d.bd >= cborBaseMap && d.bd < cborBaseTag)
}
d.d.errorf("isContainerType: unsupported parameter: %v", vt)
return // "unreachable"
}
func (d *cborDecDriver) TryDecodeAsNil() bool {
if !d.bdRead {
d.readNextBd()
}
// treat Nil and Undefined as nil values
if d.bd == cborBdNil || d.bd == cborBdUndefined {
d.bdRead = false
return true
}
return false
}
func (d *cborDecDriver) CheckBreak() bool {
if !d.bdRead {
d.readNextBd()
}
if d.bd == cborBdBreak {
d.bdRead = false
return true
}
return false
}
func (d *cborDecDriver) decUint() (ui uint64) {
v := d.bd & 0x1f
if v <= 0x17 {
ui = uint64(v)
} else {
if v == 0x18 {
ui = uint64(d.r.readn1())
} else if v == 0x19 {
ui = uint64(bigen.Uint16(d.r.readx(2)))
} else if v == 0x1a {
ui = uint64(bigen.Uint32(d.r.readx(4)))
} else if v == 0x1b {
ui = uint64(bigen.Uint64(d.r.readx(8)))
} else {
d.d.errorf("decUint: Invalid descriptor: %v", d.bd)
return
}
}
return
}
func (d *cborDecDriver) decCheckInteger() (neg bool) {
if !d.bdRead {
d.readNextBd()
}
major := d.bd >> 5
if major == cborMajorUint {
} else if major == cborMajorNegInt {
neg = true
} else {
d.d.errorf("invalid major: %v (bd: %v)", major, d.bd)
return
}
return
}
func (d *cborDecDriver) DecodeInt(bitsize uint8) (i int64) {
neg := d.decCheckInteger()
ui := d.decUint()
// check if this number can be converted to an int without overflow
var overflow bool
if neg {
if i, overflow = chkOvf.SignedInt(ui + 1); overflow {
d.d.errorf("cbor: overflow converting %v to signed integer", ui+1)
return
}
i = -i
} else {
if i, overflow = chkOvf.SignedInt(ui); overflow {
d.d.errorf("cbor: overflow converting %v to signed integer", ui)
return
}
}
if chkOvf.Int(i, bitsize) {
d.d.errorf("cbor: overflow integer: %v", i)
return
}
d.bdRead = false
return
}
func (d *cborDecDriver) DecodeUint(bitsize uint8) (ui uint64) {
if d.decCheckInteger() {
d.d.errorf("Assigning negative signed value to unsigned type")
return
}
ui = d.decUint()
if chkOvf.Uint(ui, bitsize) {
d.d.errorf("cbor: overflow integer: %v", ui)
return
}
d.bdRead = false
return
}
func (d *cborDecDriver) DecodeFloat(chkOverflow32 bool) (f float64) {
if !d.bdRead {
d.readNextBd()
}
if bd := d.bd; bd == cborBdFloat16 {
f = float64(math.Float32frombits(halfFloatToFloatBits(bigen.Uint16(d.r.readx(2)))))
} else if bd == cborBdFloat32 {
f = float64(math.Float32frombits(bigen.Uint32(d.r.readx(4))))
} else if bd == cborBdFloat64 {
f = math.Float64frombits(bigen.Uint64(d.r.readx(8)))
} else if bd >= cborBaseUint && bd < cborBaseBytes {
f = float64(d.DecodeInt(64))
} else {
d.d.errorf("Float only valid from float16/32/64: Invalid descriptor: %v", bd)
return
}
if chkOverflow32 && chkOvf.Float32(f) {
d.d.errorf("cbor: float32 overflow: %v", f)
return
}
d.bdRead = false
return
}
// bool can be decoded from bool only (single byte).
func (d *cborDecDriver) DecodeBool() (b bool) {
if !d.bdRead {
d.readNextBd()
}
if bd := d.bd; bd == cborBdTrue {
b = true
} else if bd == cborBdFalse {
} else {
d.d.errorf("Invalid single-byte value for bool: %s: %x", msgBadDesc, d.bd)
return
}
d.bdRead = false
return
}
func (d *cborDecDriver) ReadMapStart() (length int) {
d.bdRead = false
if d.bd == cborBdIndefiniteMap {
return -1
}
return d.decLen()
}
func (d *cborDecDriver) ReadArrayStart() (length int) {
d.bdRead = false
if d.bd == cborBdIndefiniteArray {
return -1
}
return d.decLen()
}
func (d *cborDecDriver) decLen() int {
return int(d.decUint())
}
func (d *cborDecDriver) decAppendIndefiniteBytes(bs []byte) []byte {
d.bdRead = false
for {
if d.CheckBreak() {
break
}
if major := d.bd >> 5; major != cborMajorBytes && major != cborMajorText {
d.d.errorf("cbor: expect bytes or string major type in indefinite string/bytes; got: %v, byte: %v", major, d.bd)
return nil
}
n := d.decLen()
oldLen := len(bs)
newLen := oldLen + n
if newLen > cap(bs) {
bs2 := make([]byte, newLen, 2*cap(bs)+n)
copy(bs2, bs)
bs = bs2
} else {
bs = bs[:newLen]
}
d.r.readb(bs[oldLen:newLen])
// bs = append(bs, d.r.readn()...)
d.bdRead = false
}
d.bdRead = false
return bs
}
func (d *cborDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOut []byte) {
if !d.bdRead {
d.readNextBd()
}
if d.bd == cborBdNil || d.bd == cborBdUndefined {
d.bdRead = false
return nil
}
if d.bd == cborBdIndefiniteBytes || d.bd == cborBdIndefiniteString {
if bs == nil {
return d.decAppendIndefiniteBytes(nil)
}
return d.decAppendIndefiniteBytes(bs[:0])
}
clen := d.decLen()
d.bdRead = false
if zerocopy {
if d.br {
return d.r.readx(clen)
} else if len(bs) == 0 {
bs = d.b[:]
}
}
return decByteSlice(d.r, clen, bs)
}
func (d *cborDecDriver) DecodeString() (s string) {
return string(d.DecodeBytes(d.b[:], true, true))
}
func (d *cborDecDriver) DecodeExt(rv interface{}, xtag uint64, ext Ext) (realxtag uint64) {
if !d.bdRead {
d.readNextBd()
}
u := d.decUint()
d.bdRead = false
realxtag = u
if ext == nil {
re := rv.(*RawExt)
re.Tag = realxtag
d.d.decode(&re.Value)
} else if xtag != realxtag {
d.d.errorf("Wrong extension tag. Got %b. Expecting: %v", realxtag, xtag)
return
} else {
var v interface{}
d.d.decode(&v)
ext.UpdateExt(rv, v)
}
d.bdRead = false
return
}
func (d *cborDecDriver) DecodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
if !d.bdRead {
d.readNextBd()
}
switch d.bd {
case cborBdNil:
vt = valueTypeNil
case cborBdFalse:
vt = valueTypeBool
v = false
case cborBdTrue:
vt = valueTypeBool
v = true
case cborBdFloat16, cborBdFloat32:
vt = valueTypeFloat
v = d.DecodeFloat(true)
case cborBdFloat64:
vt = valueTypeFloat
v = d.DecodeFloat(false)
case cborBdIndefiniteBytes:
vt = valueTypeBytes
v = d.DecodeBytes(nil, false, false)
case cborBdIndefiniteString:
vt = valueTypeString
v = d.DecodeString()
case cborBdIndefiniteArray:
vt = valueTypeArray
decodeFurther = true
case cborBdIndefiniteMap:
vt = valueTypeMap
decodeFurther = true
default:
switch {
case d.bd >= cborBaseUint && d.bd < cborBaseNegInt:
if d.h.SignedInteger {
vt = valueTypeInt
v = d.DecodeInt(64)
} else {
vt = valueTypeUint
v = d.DecodeUint(64)
}
case d.bd >= cborBaseNegInt && d.bd < cborBaseBytes:
vt = valueTypeInt
v = d.DecodeInt(64)
case d.bd >= cborBaseBytes && d.bd < cborBaseString:
vt = valueTypeBytes
v = d.DecodeBytes(nil, false, false)
case d.bd >= cborBaseString && d.bd < cborBaseArray:
vt = valueTypeString
v = d.DecodeString()
case d.bd >= cborBaseArray && d.bd < cborBaseMap:
vt = valueTypeArray
decodeFurther = true
case d.bd >= cborBaseMap && d.bd < cborBaseTag:
vt = valueTypeMap
decodeFurther = true
case d.bd >= cborBaseTag && d.bd < cborBaseSimple:
vt = valueTypeExt
var re RawExt
ui := d.decUint()
d.bdRead = false
re.Tag = ui
d.d.decode(&re.Value)
v = &re
// decodeFurther = true
default:
d.d.errorf("decodeNaked: Unrecognized d.bd: 0x%x", d.bd)
return
}
}
if !decodeFurther {
d.bdRead = false
}
return
}
// -------------------------
// CborHandle is a Handle for the CBOR encoding format,
// defined at http://tools.ietf.org/html/rfc7049 and documented further at http://cbor.io .
//
// CBOR is comprehensively supported, including support for:
// - indefinite-length arrays/maps/bytes/strings
// - (extension) tags in range 0..0xffff (0 .. 65535)
// - half, single and double-precision floats
// - all numbers (1, 2, 4 and 8-byte signed and unsigned integers)
// - nil, true, false, ...
// - arrays and maps, bytes and text strings
//
// None of the optional extensions (with tags) defined in the spec are supported out-of-the-box.
// Users can implement them as needed (using SetExt), including spec-documented ones:
// - timestamp, BigNum, BigFloat, Decimals, Encoded Text (e.g. URL, regexp, base64, MIME Message), etc.
//
// To encode with indefinite lengths (streaming), users will use
// (Must)Encode methods of *Encoder, along with writing CborStreamXXX constants.
//
// For example, to encode "one-byte" as an indefinite length string:
// var buf bytes.Buffer
// e := NewEncoder(&buf, new(CborHandle))
// buf.WriteByte(CborStreamString)
// e.MustEncode("one-")
// e.MustEncode("byte")
// buf.WriteByte(CborStreamBreak)
// encodedBytes := buf.Bytes()
// var vv interface{}
// NewDecoderBytes(buf.Bytes(), new(CborHandle)).MustDecode(&vv)
// // Now, vv contains the same string "one-byte"
//
type CborHandle struct {
BasicHandle
binaryEncodingType
}
func (h *CborHandle) newEncDriver(e *Encoder) encDriver {
return &cborEncDriver{e: e, w: e.w, h: h}
}
func (h *CborHandle) newDecDriver(d *Decoder) decDriver {
return &cborDecDriver{d: d, r: d.r, h: h, br: d.bytes}
}
var _ decDriver = (*cborDecDriver)(nil)
var _ encDriver = (*cborEncDriver)(nil)

View File

@@ -1,205 +0,0 @@
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
import (
"bufio"
"bytes"
"encoding/hex"
"math"
"os"
"regexp"
"strings"
"testing"
)
func TestCborIndefiniteLength(t *testing.T) {
oldMapType := testCborH.MapType
defer func() {
testCborH.MapType = oldMapType
}()
testCborH.MapType = testMapStrIntfTyp
// var (
// M1 map[string][]byte
// M2 map[uint64]bool
// L1 []interface{}
// S1 []string
// B1 []byte
// )
var v, vv interface{}
// define it (v), encode it using indefinite lengths, decode it (vv), compare v to vv
v = map[string]interface{}{
"one-byte-key": []byte{1, 2, 3, 4, 5, 6},
"two-string-key": "two-value",
"three-list-key": []interface{}{true, false, uint64(1), int64(-1)},
}
var buf bytes.Buffer
// buf.Reset()
e := NewEncoder(&buf, testCborH)
buf.WriteByte(cborBdIndefiniteMap)
//----
buf.WriteByte(cborBdIndefiniteString)
e.MustEncode("one-")
e.MustEncode("byte-")
e.MustEncode("key")
buf.WriteByte(cborBdBreak)
buf.WriteByte(cborBdIndefiniteBytes)
e.MustEncode([]byte{1, 2, 3})
e.MustEncode([]byte{4, 5, 6})
buf.WriteByte(cborBdBreak)
//----
buf.WriteByte(cborBdIndefiniteString)
e.MustEncode("two-")
e.MustEncode("string-")
e.MustEncode("key")
buf.WriteByte(cborBdBreak)
buf.WriteByte(cborBdIndefiniteString)
e.MustEncode([]byte("two-")) // encode as bytes, to check robustness of code
e.MustEncode([]byte("value"))
buf.WriteByte(cborBdBreak)
//----
buf.WriteByte(cborBdIndefiniteString)
e.MustEncode("three-")
e.MustEncode("list-")
e.MustEncode("key")
buf.WriteByte(cborBdBreak)
buf.WriteByte(cborBdIndefiniteArray)
e.MustEncode(true)
e.MustEncode(false)
e.MustEncode(uint64(1))
e.MustEncode(int64(-1))
buf.WriteByte(cborBdBreak)
buf.WriteByte(cborBdBreak) // close map
NewDecoderBytes(buf.Bytes(), testCborH).MustDecode(&vv)
if err := deepEqual(v, vv); err != nil {
logT(t, "-------- Before and After marshal do not match: Error: %v", err)
logT(t, " ....... GOLDEN: (%T) %#v", v, v)
logT(t, " ....... DECODED: (%T) %#v", vv, vv)
failT(t)
}
}
type testCborGolden struct {
Base64 string `codec:"cbor"`
Hex string `codec:"hex"`
Roundtrip bool `codec:"roundtrip"`
Decoded interface{} `codec:"decoded"`
Diagnostic string `codec:"diagnostic"`
Skip bool `codec:"skip"`
}
// Some tests are skipped because they include numbers outside the range of int64/uint64
func doTestCborGoldens(t *testing.T) {
oldMapType := testCborH.MapType
defer func() {
testCborH.MapType = oldMapType
}()
testCborH.MapType = testMapStrIntfTyp
// decode test-cbor-goldens.json into a list of []*testCborGolden
// for each one,
// - decode hex into []byte bs
// - decode bs into interface{} v
// - compare both using deepequal
// - for any miss, record it
var gs []*testCborGolden
f, err := os.Open("test-cbor-goldens.json")
if err != nil {
logT(t, "error opening test-cbor-goldens.json: %v", err)
failT(t)
}
defer f.Close()
jh := new(JsonHandle)
jh.MapType = testMapStrIntfTyp
// d := NewDecoder(f, jh)
d := NewDecoder(bufio.NewReader(f), jh)
// err = d.Decode(&gs)
d.MustDecode(&gs)
if err != nil {
logT(t, "error json decoding test-cbor-goldens.json: %v", err)
failT(t)
}
tagregex := regexp.MustCompile(`[\d]+\(.+?\)`)
hexregex := regexp.MustCompile(`h'([0-9a-fA-F]*)'`)
for i, g := range gs {
// fmt.Printf("%v, skip: %v, isTag: %v, %s\n", i, g.Skip, tagregex.MatchString(g.Diagnostic), g.Diagnostic)
// skip tags or simple or those with prefix, as we can't verify them.
if g.Skip || strings.HasPrefix(g.Diagnostic, "simple(") || tagregex.MatchString(g.Diagnostic) {
// fmt.Printf("%v: skipped\n", i)
logT(t, "[%v] skipping because skip=true OR unsupported simple value or Tag Value", i)
continue
}
// println("++++++++++++", i, "g.Diagnostic", g.Diagnostic)
if hexregex.MatchString(g.Diagnostic) {
// println(i, "g.Diagnostic matched hex")
if s2 := g.Diagnostic[2 : len(g.Diagnostic)-1]; s2 == "" {
g.Decoded = zeroByteSlice
} else if bs2, err2 := hex.DecodeString(s2); err2 == nil {
g.Decoded = bs2
}
// fmt.Printf("%v: hex: %v\n", i, g.Decoded)
}
bs, err := hex.DecodeString(g.Hex)
if err != nil {
logT(t, "[%v] error hex decoding %s [%v]: %v", i, g.Hex, err)
failT(t)
}
var v interface{}
NewDecoderBytes(bs, testCborH).MustDecode(&v)
if _, ok := v.(RawExt); ok {
continue
}
// check the diagnostics to compare
switch g.Diagnostic {
case "Infinity":
b := math.IsInf(v.(float64), 1)
testCborError(t, i, math.Inf(1), v, nil, &b)
case "-Infinity":
b := math.IsInf(v.(float64), -1)
testCborError(t, i, math.Inf(-1), v, nil, &b)
case "NaN":
// println(i, "checking NaN")
b := math.IsNaN(v.(float64))
testCborError(t, i, math.NaN(), v, nil, &b)
case "undefined":
b := v == nil
testCborError(t, i, nil, v, nil, &b)
default:
v0 := g.Decoded
// testCborCoerceJsonNumber(reflect.ValueOf(&v0))
testCborError(t, i, v0, v, deepEqual(v0, v), nil)
}
}
}
func testCborError(t *testing.T, i int, v0, v1 interface{}, err error, equal *bool) {
if err == nil && equal == nil {
// fmt.Printf("%v testCborError passed (err and equal nil)\n", i)
return
}
if err != nil {
logT(t, "[%v] deepEqual error: %v", i, err)
logT(t, " ....... GOLDEN: (%T) %#v", v0, v0)
logT(t, " ....... DECODED: (%T) %#v", v1, v1)
failT(t)
}
if equal != nil && !*equal {
logT(t, "[%v] values not equal", i)
logT(t, " ....... GOLDEN: (%T) %#v", v0, v0)
logT(t, " ....... DECODED: (%T) %#v", v1, v1)
failT(t)
}
// fmt.Printf("%v testCborError passed (checks passed)\n", i)
}
func TestCborGoldens(t *testing.T) {
doTestCborGoldens(t)
}

File diff suppressed because it is too large Load Diff

View File

@@ -1,36 +0,0 @@
# codecgen tool
Generate is given a list of *.go files to parse, and an output file (fout),
codecgen will create an output file __file.go__ which
contains `codec.Selfer` implementations for the named types found
in the files parsed.
Using codecgen is very straightforward.
**Download and install the tool**
`go get -u github.com/ugorji/go/codec/codecgen`
**Run the tool on your files**
The command line format is:
`codecgen [options] (-o outfile) (infile ...)`
```sh
% codecgen -?
Usage of codecgen:
-c="github.com/ugorji/go/codec": codec path
-o="": out file
-r=".*": regex for type name to match
-rt="": tags for go run
-t="": build tag to put in file
-u=false: Use unsafe, e.g. to avoid unnecessary allocation on []byte->string
-x=false: keep temp file
% codecgen -o values_codecgen.go values.go values2.go moretypedefs.go
```
Please see the [blog article](http://ugorji.net/blog/go-codecgen)
for more information on how to use the tool.

View File

@@ -1,271 +0,0 @@
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
// codecgen generates codec.Selfer implementations for a set of types.
package main
import (
"bufio"
"bytes"
"errors"
"flag"
"fmt"
"go/ast"
"go/build"
"go/parser"
"go/token"
"os"
"os/exec"
"path/filepath"
"regexp"
"strconv"
"text/template"
"time"
)
const genFrunMainTmpl = `//+build ignore
package main
{{ if .Types }}import "{{ .ImportPath }}"{{ end }}
func main() {
{{ $.PackageName }}.CodecGenTempWrite{{ .RandString }}()
}
`
// const genFrunPkgTmpl = `//+build codecgen
const genFrunPkgTmpl = `
package {{ $.PackageName }}
import (
{{ if not .CodecPkgFiles }}{{ .CodecPkgName }} "{{ .CodecImportPath }}"{{ end }}
{{/*
{{ if .Types }}"{{ .ImportPath }}"{{ end }}
"io"
*/}}
"os"
"reflect"
"bytes"
"go/format"
)
{{/* This is not used anymore. Remove it.
func write(w io.Writer, s string) {
if _, err := io.WriteString(w, s); err != nil {
panic(err)
}
}
*/}}
func CodecGenTempWrite{{ .RandString }}() {
fout, err := os.Create("{{ .OutFile }}")
if err != nil {
panic(err)
}
defer fout.Close()
var out bytes.Buffer
var typs []reflect.Type
{{ range $index, $element := .Types }}
var t{{ $index }} {{ . }}
typs = append(typs, reflect.TypeOf(t{{ $index }}))
{{ end }}
{{ if not .CodecPkgFiles }}{{ .CodecPkgName }}.{{ end }}Gen(&out, "{{ .BuildTag }}", "{{ .PackageName }}", {{ .UseUnsafe }}, typs...)
bout, err := format.Source(out.Bytes())
if err != nil {
fout.Write(out.Bytes())
panic(err)
}
fout.Write(bout)
}
`
// Generate is given a list of *.go files to parse, and an output file (fout).
//
// It finds all types T in the files, and it creates 2 tmp files (frun).
// - main package file passed to 'go run'
// - package level file which calls *genRunner.Selfer to write Selfer impls for each T.
// We use a package level file so that it can reference unexported types in the package being worked on.
// Tool then executes: "go run __frun__" which creates fout.
// fout contains Codec(En|De)codeSelf implementations for every type T.
//
func Generate(outfile, buildTag, codecPkgPath string, useUnsafe bool, goRunTag string,
regexName *regexp.Regexp, deleteTempFile bool, infiles ...string) (err error) {
// For each file, grab AST, find each type, and write a call to it.
if len(infiles) == 0 {
return
}
if outfile == "" || codecPkgPath == "" {
err = errors.New("outfile and codec package path cannot be blank")
return
}
// We have to parse dir for package, before opening the temp file for writing (else ImportDir fails).
// Also, ImportDir(...) must take an absolute path.
lastdir := filepath.Dir(outfile)
absdir, err := filepath.Abs(lastdir)
if err != nil {
return
}
pkg, err := build.Default.ImportDir(absdir, build.AllowBinary)
if err != nil {
return
}
type tmplT struct {
CodecPkgName string
CodecImportPath string
ImportPath string
OutFile string
PackageName string
RandString string
BuildTag string
Types []string
CodecPkgFiles bool
UseUnsafe bool
}
tv := tmplT{
CodecPkgName: "codec1978",
OutFile: outfile,
CodecImportPath: codecPkgPath,
BuildTag: buildTag,
UseUnsafe: useUnsafe,
RandString: strconv.FormatInt(time.Now().UnixNano(), 10),
}
tv.ImportPath = pkg.ImportPath
if tv.ImportPath == tv.CodecImportPath {
tv.CodecPkgFiles = true
tv.CodecPkgName = "codec"
}
astfiles := make([]*ast.File, len(infiles))
for i, infile := range infiles {
if filepath.Dir(infile) != lastdir {
err = errors.New("in files must all be in same directory as outfile")
return
}
fset := token.NewFileSet()
astfiles[i], err = parser.ParseFile(fset, infile, nil, 0)
if err != nil {
return
}
if i == 0 {
tv.PackageName = astfiles[i].Name.Name
if tv.PackageName == "main" {
// codecgen cannot be run on types in the 'main' package.
// A temporary 'main' package must be created, and should reference the fully built
// package containing the types.
// Also, the temporary main package will conflict with the main package which already has a main method.
err = errors.New("codecgen cannot be run on types in the 'main' package")
return
}
}
}
for _, f := range astfiles {
for _, d := range f.Decls {
if gd, ok := d.(*ast.GenDecl); ok {
for _, dd := range gd.Specs {
if td, ok := dd.(*ast.TypeSpec); ok {
// if len(td.Name.Name) == 0 || td.Name.Name[0] > 'Z' || td.Name.Name[0] < 'A' {
if len(td.Name.Name) == 0 {
continue
}
// only generate for:
// struct: StructType
// primitives (numbers, bool, string): Ident
// map: MapType
// slice, array: ArrayType
// chan: ChanType
// do not generate:
// FuncType, InterfaceType, StarExpr (ptr), etc
switch td.Type.(type) {
case *ast.StructType, *ast.Ident, *ast.MapType, *ast.ArrayType, *ast.ChanType:
if regexName.FindStringIndex(td.Name.Name) != nil {
tv.Types = append(tv.Types, td.Name.Name)
}
}
}
}
}
}
}
if len(tv.Types) == 0 {
return
}
// we cannot use ioutil.TempFile, because we cannot guarantee the file suffix (.go).
// Also, we cannot create file in temp directory,
// because go run will not work (as it needs to see the types here).
// Consequently, create the temp file in the current directory, and remove when done.
// frun, err = ioutil.TempFile("", "codecgen-")
// frunName := filepath.Join(os.TempDir(), "codecgen-"+strconv.FormatInt(time.Now().UnixNano(), 10)+".go")
frunMainName := "codecgen-main-" + tv.RandString + ".generated.go"
frunPkgName := "codecgen-pkg-" + tv.RandString + ".generated.go"
if deleteTempFile {
defer os.Remove(frunMainName)
defer os.Remove(frunPkgName)
}
// var frunMain, frunPkg *os.File
if _, err = gen1(frunMainName, genFrunMainTmpl, &tv); err != nil {
return
}
if _, err = gen1(frunPkgName, genFrunPkgTmpl, &tv); err != nil {
return
}
// remove outfile, so "go run ..." will not think that types in outfile already exist.
os.Remove(outfile)
// execute go run frun
cmd := exec.Command("go", "run", "-tags="+goRunTag, frunMainName) //, frunPkg.Name())
var buf bytes.Buffer
cmd.Stdout = &buf
cmd.Stderr = &buf
if err = cmd.Run(); err != nil {
err = fmt.Errorf("error running 'go run %s': %v, console: %s",
frunMainName, err, buf.Bytes())
return
}
os.Stdout.Write(buf.Bytes())
return
}
func gen1(frunName, tmplStr string, tv interface{}) (frun *os.File, err error) {
os.Remove(frunName)
if frun, err = os.Create(frunName); err != nil {
return
}
defer frun.Close()
t := template.New("")
if t, err = t.Parse(tmplStr); err != nil {
return
}
bw := bufio.NewWriter(frun)
if err = t.Execute(bw, tv); err != nil {
return
}
if err = bw.Flush(); err != nil {
return
}
return
}
func main() {
o := flag.String("o", "", "out file")
c := flag.String("c", genCodecPath, "codec path")
t := flag.String("t", "", "build tag to put in file")
r := flag.String("r", ".*", "regex for type name to match")
rt := flag.String("rt", "", "tags for go run")
x := flag.Bool("x", false, "keep temp file")
u := flag.Bool("u", false, "Use unsafe, e.g. to avoid unnecessary allocation on []byte->string")
flag.Parse()
if err := Generate(*o, *t, *c, *u, *rt,
regexp.MustCompile(*r), !*x, flag.Args()...); err != nil {
fmt.Fprintf(os.Stderr, "codecgen error: %v\n", err)
os.Exit(1)
}
}

View File

@@ -1,3 +0,0 @@
package main
const genCodecPath = "github.com/ugorji/go/codec"

View File

@@ -1,22 +0,0 @@
//+build x,codecgen
package codec
import (
"fmt"
"testing"
)
func TestCodecgenJson1(t *testing.T) {
const callCodecgenDirect bool = true
v := newTestStruc(2, false, !testSkipIntf, false)
var bs []byte
e := NewEncoderBytes(&bs, testJsonH)
if callCodecgenDirect {
v.CodecEncodeSelf(e)
e.w.atEndOfEncode()
} else {
e.MustEncode(v)
}
fmt.Printf("%s\n", bs)
}

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -1,442 +0,0 @@
// //+build ignore
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
// ************************************************************
// DO NOT EDIT.
// THIS FILE IS AUTO-GENERATED from fast-path.go.tmpl
// ************************************************************
package codec
// Fast path functions try to create a fast path encode or decode implementation
// for common maps and slices.
//
// We define the functions and register then in this single file
// so as not to pollute the encode.go and decode.go, and create a dependency in there.
// This file can be omitted without causing a build failure.
//
// The advantage of fast paths is:
// - Many calls bypass reflection altogether
//
// Currently support
// - slice of all builtin types,
// - map of all builtin types to string or interface value
// - symetrical maps of all builtin types (e.g. str-str, uint8-uint8)
// This should provide adequate "typical" implementations.
//
// Note that fast track decode functions must handle values for which an address cannot be obtained.
// For example:
// m2 := map[string]int{}
// p2 := []interface{}{m2}
// // decoding into p2 will bomb if fast track functions do not treat like unaddressable.
//
import (
"reflect"
"sort"
)
const fastpathCheckNilFalse = false // for reflect
const fastpathCheckNilTrue = true // for type switch
type fastpathT struct {}
var fastpathTV fastpathT
type fastpathE struct {
rtid uintptr
rt reflect.Type
encfn func(encFnInfo, reflect.Value)
decfn func(decFnInfo, reflect.Value)
}
type fastpathA [{{ .FastpathLen }}]fastpathE
func (x *fastpathA) index(rtid uintptr) int {
// use binary search to grab the index (adapted from sort/search.go)
h, i, j := 0, 0, {{ .FastpathLen }} // len(x)
for i < j {
h = i + (j-i)/2
if x[h].rtid < rtid {
i = h + 1
} else {
j = h
}
}
if i < {{ .FastpathLen }} && x[i].rtid == rtid {
return i
}
return -1
}
type fastpathAslice []fastpathE
func (x fastpathAslice) Len() int { return len(x) }
func (x fastpathAslice) Less(i, j int) bool { return x[i].rtid < x[j].rtid }
func (x fastpathAslice) Swap(i, j int) { x[i], x[j] = x[j], x[i] }
var fastpathAV fastpathA
// due to possible initialization loop error, make fastpath in an init()
func init() {
if !fastpathEnabled {
return
}
i := 0
fn := func(v interface{}, fe func(encFnInfo, reflect.Value), fd func(decFnInfo, reflect.Value)) (f fastpathE) {
xrt := reflect.TypeOf(v)
xptr := reflect.ValueOf(xrt).Pointer()
fastpathAV[i] = fastpathE{xptr, xrt, fe, fd}
i++
return
}
{{range .Values}}{{if not .Primitive}}{{if .Slice }}
fn([]{{ .Elem }}(nil), (encFnInfo).{{ .MethodNamePfx "fastpathEnc" false }}R, (decFnInfo).{{ .MethodNamePfx "fastpathDec" false }}R){{end}}{{end}}{{end}}
{{range .Values}}{{if not .Primitive}}{{if not .Slice }}
fn(map[{{ .MapKey }}]{{ .Elem }}(nil), (encFnInfo).{{ .MethodNamePfx "fastpathEnc" false }}R, (decFnInfo).{{ .MethodNamePfx "fastpathDec" false }}R){{end}}{{end}}{{end}}
sort.Sort(fastpathAslice(fastpathAV[:]))
}
// -- encode
// -- -- fast path type switch
func fastpathEncodeTypeSwitch(iv interface{}, e *Encoder) bool {
switch v := iv.(type) {
{{range .Values}}{{if not .Primitive}}{{if .Slice }}
case []{{ .Elem }}:{{else}}
case map[{{ .MapKey }}]{{ .Elem }}:{{end}}
fastpathTV.{{ .MethodNamePfx "Enc" false }}V(v, fastpathCheckNilTrue, e){{if .Slice }}
case *[]{{ .Elem }}:{{else}}
case *map[{{ .MapKey }}]{{ .Elem }}:{{end}}
fastpathTV.{{ .MethodNamePfx "Enc" false }}V(*v, fastpathCheckNilTrue, e)
{{end}}{{end}}
default:
return false
}
return true
}
func fastpathEncodeTypeSwitchSlice(iv interface{}, e *Encoder) bool {
switch v := iv.(type) {
{{range .Values}}{{if not .Primitive}}{{if .Slice }}
case []{{ .Elem }}:
fastpathTV.{{ .MethodNamePfx "Enc" false }}V(v, fastpathCheckNilTrue, e)
case *[]{{ .Elem }}:
fastpathTV.{{ .MethodNamePfx "Enc" false }}V(*v, fastpathCheckNilTrue, e)
{{end}}{{end}}{{end}}
default:
return false
}
return true
}
func fastpathEncodeTypeSwitchMap(iv interface{}, e *Encoder) bool {
switch v := iv.(type) {
{{range .Values}}{{if not .Primitive}}{{if not .Slice }}
case map[{{ .MapKey }}]{{ .Elem }}:
fastpathTV.{{ .MethodNamePfx "Enc" false }}V(v, fastpathCheckNilTrue, e)
case *map[{{ .MapKey }}]{{ .Elem }}:
fastpathTV.{{ .MethodNamePfx "Enc" false }}V(*v, fastpathCheckNilTrue, e)
{{end}}{{end}}{{end}}
default:
return false
}
return true
}
// -- -- fast path functions
{{range .Values}}{{if not .Primitive}}{{if .Slice }}
func (f encFnInfo) {{ .MethodNamePfx "fastpathEnc" false }}R(rv reflect.Value) {
fastpathTV.{{ .MethodNamePfx "Enc" false }}V(rv.Interface().([]{{ .Elem }}), fastpathCheckNilFalse, f.e)
}
func (_ fastpathT) {{ .MethodNamePfx "Enc" false }}V(v []{{ .Elem }}, checkNil bool, e *Encoder) {
ee := e.e
if checkNil && v == nil {
ee.EncodeNil()
return
}
ee.EncodeArrayStart(len(v))
if e.be {
for _, v2 := range v {
{{ encmd .Elem "v2"}}
}
} else {
for j, v2 := range v {
if j > 0 {
ee.EncodeArrayEntrySeparator()
}
{{ encmd .Elem "v2"}}
}
ee.EncodeArrayEnd()
}
}
{{end}}{{end}}{{end}}
{{range .Values}}{{if not .Primitive}}{{if not .Slice }}
func (f encFnInfo) {{ .MethodNamePfx "fastpathEnc" false }}R(rv reflect.Value) {
fastpathTV.{{ .MethodNamePfx "Enc" false }}V(rv.Interface().(map[{{ .MapKey }}]{{ .Elem }}), fastpathCheckNilFalse, f.e)
}
func (_ fastpathT) {{ .MethodNamePfx "Enc" false }}V(v map[{{ .MapKey }}]{{ .Elem }}, checkNil bool, e *Encoder) {
ee := e.e
if checkNil && v == nil {
ee.EncodeNil()
return
}
ee.EncodeMapStart(len(v))
{{if eq .MapKey "string"}}asSymbols := e.h.AsSymbols&AsSymbolMapStringKeysFlag != 0{{end}}
if e.be {
for k2, v2 := range v {
{{if eq .MapKey "string"}}if asSymbols {
ee.EncodeSymbol(k2)
} else {
ee.EncodeString(c_UTF8, k2)
}{{else}}{{ encmd .MapKey "k2"}}{{end}}
{{ encmd .Elem "v2"}}
}
} else {
j := 0
for k2, v2 := range v {
if j > 0 {
ee.EncodeMapEntrySeparator()
}
{{if eq .MapKey "string"}}if asSymbols {
ee.EncodeSymbol(k2)
} else {
ee.EncodeString(c_UTF8, k2)
}{{else}}{{ encmd .MapKey "k2"}}{{end}}
ee.EncodeMapKVSeparator()
{{ encmd .Elem "v2"}}
j++
}
ee.EncodeMapEnd()
}
}
{{end}}{{end}}{{end}}
// -- decode
// -- -- fast path type switch
func fastpathDecodeTypeSwitch(iv interface{}, d *Decoder) bool {
switch v := iv.(type) {
{{range .Values}}{{if not .Primitive}}{{if .Slice }}
case []{{ .Elem }}:{{else}}
case map[{{ .MapKey }}]{{ .Elem }}:{{end}}
fastpathTV.{{ .MethodNamePfx "Dec" false }}V(v, fastpathCheckNilFalse, false, d){{if .Slice }}
case *[]{{ .Elem }}:{{else}}
case *map[{{ .MapKey }}]{{ .Elem }}:{{end}}
v2, changed2 := fastpathTV.{{ .MethodNamePfx "Dec" false }}V(*v, fastpathCheckNilFalse, true, d)
if changed2 {
*v = v2
}
{{end}}{{end}}
default:
return false
}
return true
}
// -- -- fast path functions
{{range .Values}}{{if not .Primitive}}{{if .Slice }}
{{/*
Slices can change if they
- did not come from an array
- are addressable (from a ptr)
- are settable (e.g. contained in an interface{})
*/}}
func (f decFnInfo) {{ .MethodNamePfx "fastpathDec" false }}R(rv reflect.Value) {
array := f.seq == seqTypeArray
if !array && rv.CanAddr() { // CanSet => CanAddr + Exported
vp := rv.Addr().Interface().(*[]{{ .Elem }})
v, changed := fastpathTV.{{ .MethodNamePfx "Dec" false }}V(*vp, fastpathCheckNilFalse, !array, f.d)
if changed {
*vp = v
}
} else {
v := rv.Interface().([]{{ .Elem }})
fastpathTV.{{ .MethodNamePfx "Dec" false }}V(v, fastpathCheckNilFalse, false, f.d)
}
}
func (f fastpathT) {{ .MethodNamePfx "Dec" false }}X(vp *[]{{ .Elem }}, checkNil bool, d *Decoder) {
v, changed := f.{{ .MethodNamePfx "Dec" false }}V(*vp, checkNil, true, d)
if changed {
*vp = v
}
}
func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v []{{ .Elem }}, checkNil bool, canChange bool,
d *Decoder) (_ []{{ .Elem }}, changed bool) {
dd := d.d
// if dd.isContainerType(valueTypeNil) { dd.TryDecodeAsNil()
if checkNil && dd.TryDecodeAsNil() {
if v != nil {
changed = true
}
return nil, changed
}
slh, containerLenS := d.decSliceHelperStart()
if canChange && v == nil {
if containerLenS <= 0 {
v = []{{ .Elem }}{}
} else {
v = make([]{{ .Elem }}, containerLenS, containerLenS)
}
changed = true
}
if containerLenS == 0 {
if canChange && len(v) != 0 {
v = v[:0]
changed = true
}{{/*
// slh.End() // dd.ReadArrayEnd()
*/}}
return v, changed
}
// for j := 0; j < containerLenS; j++ {
if containerLenS > 0 {
decLen := containerLenS
if containerLenS > cap(v) {
if canChange {
s := make([]{{ .Elem }}, containerLenS, containerLenS)
// copy(s, v[:cap(v)])
v = s
changed = true
} else {
d.arrayCannotExpand(len(v), containerLenS)
decLen = len(v)
}
} else if containerLenS != len(v) {
v = v[:containerLenS]
changed = true
}
// all checks done. cannot go past len.
j := 0
for ; j < decLen; j++ {
{{ if eq .Elem "interface{}" }}d.decode(&v[j]){{ else }}v[j] = {{ decmd .Elem }}{{ end }}
}
if !canChange {
for ; j < containerLenS; j++ {
d.swallow()
}
}
} else {
j := 0
for ; !dd.CheckBreak(); j++ {
if j >= len(v) {
if canChange {
v = append(v, {{ zerocmd .Elem }})
changed = true
} else {
d.arrayCannotExpand(len(v), j+1)
}
}
if j > 0 {
slh.Sep(j)
}
if j < len(v) { // all checks done. cannot go past len.
{{ if eq .Elem "interface{}" }}d.decode(&v[j])
{{ else }}v[j] = {{ decmd .Elem }}{{ end }}
} else {
d.swallow()
}
}
slh.End()
}
return v, changed
}
{{end}}{{end}}{{end}}
{{range .Values}}{{if not .Primitive}}{{if not .Slice }}
{{/*
Maps can change if they are
- addressable (from a ptr)
- settable (e.g. contained in an interface{})
*/}}
func (f decFnInfo) {{ .MethodNamePfx "fastpathDec" false }}R(rv reflect.Value) {
if rv.CanAddr() {
vp := rv.Addr().Interface().(*map[{{ .MapKey }}]{{ .Elem }})
v, changed := fastpathTV.{{ .MethodNamePfx "Dec" false }}V(*vp, fastpathCheckNilFalse, true, f.d)
if changed {
*vp = v
}
} else {
v := rv.Interface().(map[{{ .MapKey }}]{{ .Elem }})
fastpathTV.{{ .MethodNamePfx "Dec" false }}V(v, fastpathCheckNilFalse, false, f.d)
}
}
func (f fastpathT) {{ .MethodNamePfx "Dec" false }}X(vp *map[{{ .MapKey }}]{{ .Elem }}, checkNil bool, d *Decoder) {
v, changed := f.{{ .MethodNamePfx "Dec" false }}V(*vp, checkNil, true, d)
if changed {
*vp = v
}
}
func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v map[{{ .MapKey }}]{{ .Elem }}, checkNil bool, canChange bool,
d *Decoder) (_ map[{{ .MapKey }}]{{ .Elem }}, changed bool) {
dd := d.d
// if dd.isContainerType(valueTypeNil) {dd.TryDecodeAsNil()
if checkNil && dd.TryDecodeAsNil() {
if v != nil {
changed = true
}
return nil, changed
}
containerLen := dd.ReadMapStart()
if canChange && v == nil {
if containerLen > 0 {
v = make(map[{{ .MapKey }}]{{ .Elem }}, containerLen)
} else {
v = make(map[{{ .MapKey }}]{{ .Elem }}) // supports indefinite-length, etc
}
changed = true
}
if containerLen > 0 {
for j := 0; j < containerLen; j++ {
{{ if eq .MapKey "interface{}" }}var mk interface{}
d.decode(&mk)
if bv, bok := mk.([]byte); bok {
mk = string(bv) // maps cannot have []byte as key. switch to string.
}{{ else }}mk := {{ decmd .MapKey }}{{ end }}
mv := v[mk]
{{ if eq .Elem "interface{}" }}d.decode(&mv)
{{ else }}mv = {{ decmd .Elem }}{{ end }}
if v != nil {
v[mk] = mv
}
}
} else if containerLen < 0 {
for j := 0; !dd.CheckBreak(); j++ {
if j > 0 {
dd.ReadMapEntrySeparator()
}
{{ if eq .MapKey "interface{}" }}var mk interface{}
d.decode(&mk)
if bv, bok := mk.([]byte); bok {
mk = string(bv) // maps cannot have []byte as key. switch to string.
}{{ else }}mk := {{ decmd .MapKey }}{{ end }}
dd.ReadMapKVSeparator()
mv := v[mk]
{{ if eq .Elem "interface{}" }}d.decode(&mv)
{{ else }}mv = {{ decmd .Elem }}{{ end }}
if v != nil {
v[mk] = mv
}
}
dd.ReadMapEnd()
}
return v, changed
}
{{end}}{{end}}{{end}}

View File

@@ -1,77 +0,0 @@
{{var "v"}} := *{{ .Varname }}
{{var "h"}}, {{var "l"}} := z.DecSliceHelperStart()
var {{var "c"}} bool
{{ if not isArray }}if {{var "v"}} == nil {
if {{var "l"}} <= 0 {
{{var "v"}} = make({{ .CTyp }}, 0)
} else {
{{var "v"}} = make({{ .CTyp }}, {{var "l"}})
}
{{var "c"}} = true
}
{{ end }}
if {{var "l"}} == 0 { {{ if isSlice }}
if len({{var "v"}}) != 0 {
{{var "v"}} = {{var "v"}}[:0]
{{var "c"}} = true
} {{ end }}
} else if {{var "l"}} > 0 {
{{ if isChan }}
for {{var "r"}} := 0; {{var "r"}} < {{var "l"}}; {{var "r"}}++ {
var {{var "t"}} {{ .Typ }}
{{ $x := printf "%st%s" .TempVar .Rand }}{{ decLineVar $x }}
{{var "v"}} <- {{var "t"}}
{{ else }}
{{var "n"}} := {{var "l"}}
if {{var "l"}} > cap({{var "v"}}) {
{{ if isArray }}r.ReadArrayCannotExpand(len({{var "v"}}), {{var "l"}})
{{var "n"}} = len({{var "v"}})
{{ else }}{{ if .Immutable }}
{{var "v2"}} := {{var "v"}}
{{var "v"}} = make([]{{ .Typ }}, {{var "l"}}, {{var "l"}})
if len({{var "v"}}) > 0 {
copy({{var "v"}}, {{var "v2"}}[:cap({{var "v2"}})])
}
{{ else }}{{var "v"}} = make([]{{ .Typ }}, {{var "l"}}, {{var "l"}})
{{ end }}{{var "c"}} = true
{{ end }}
} else if {{var "l"}} != len({{var "v"}}) {
{{var "v"}} = {{var "v"}}[:{{var "l"}}]
{{var "c"}} = true
}
{{var "j"}} := 0
for ; {{var "j"}} < {{var "n"}} ; {{var "j"}}++ {
{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
} {{ if isArray }}
for ; {{var "j"}} < {{var "l"}} ; {{var "j"}}++ {
z.DecSwallow()
}{{ end }}
{{ end }}{{/* closing if not chan */}}
} else {
for {{var "j"}} := 0; !r.CheckBreak(); {{var "j"}}++ {
if {{var "j"}} >= len({{var "v"}}) {
{{ if isArray }}r.ReadArrayCannotExpand(len({{var "v"}}), {{var "j"}}+1)
{{ else if isSlice}}{{var "v"}} = append({{var "v"}}, {{zero}})// var {{var "z"}} {{ .Typ }}
{{var "c"}} = true {{ end }}
}
if {{var "j"}} > 0 {
{{var "h"}}.Sep({{var "j"}})
}
{{ if isChan}}
var {{var "t"}} {{ .Typ }}
{{ $x := printf "%st%s" .TempVar .Rand }}{{ decLineVar $x }}
{{var "v"}} <- {{var "t"}}
{{ else }}
if {{var "j"}} < len({{var "v"}}) {
{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
} else {
z.DecSwallow()
}
{{ end }}
}
{{var "h"}}.End()
}
if {{var "c"}} {
*{{ .Varname }} = {{var "v"}}
}

View File

@@ -1,46 +0,0 @@
{{var "v"}} := *{{ .Varname }}
{{var "l"}} := r.ReadMapStart()
if {{var "v"}} == nil {
if {{var "l"}} > 0 {
{{var "v"}} = make(map[{{ .KTyp }}]{{ .Typ }}, {{var "l"}})
} else {
{{var "v"}} = make(map[{{ .KTyp }}]{{ .Typ }}) // supports indefinite-length, etc
}
*{{ .Varname }} = {{var "v"}}
}
if {{var "l"}} > 0 {
for {{var "j"}} := 0; {{var "j"}} < {{var "l"}}; {{var "j"}}++ {
var {{var "mk"}} {{ .KTyp }}
{{ $x := printf "%vmk%v" .TempVar .Rand }}{{ decLineVarK $x }}
{{ if eq .KTyp "interface{}" }}// special case if a byte array.
if {{var "bv"}}, {{var "bok"}} := {{var "mk"}}.([]byte); {{var "bok"}} {
{{var "mk"}} = string({{var "bv"}})
}
{{ end }}
{{var "mv"}} := {{var "v"}}[{{var "mk"}}]
{{ $x := printf "%vmv%v" .TempVar .Rand }}{{ decLineVar $x }}
if {{var "v"}} != nil {
{{var "v"}}[{{var "mk"}}] = {{var "mv"}}
}
}
} else if {{var "l"}} < 0 {
for {{var "j"}} := 0; !r.CheckBreak(); {{var "j"}}++ {
if {{var "j"}} > 0 {
r.ReadMapEntrySeparator()
}
var {{var "mk"}} {{ .KTyp }}
{{ $x := printf "%vmk%v" .TempVar .Rand }}{{ decLineVarK $x }}
{{ if eq .KTyp "interface{}" }}// special case if a byte array.
if {{var "bv"}}, {{var "bok"}} := {{var "mk"}}.([]byte); {{var "bok"}} {
{{var "mk"}} = string({{var "bv"}})
}
{{ end }}
r.ReadMapKVSeparator()
{{var "mv"}} := {{var "v"}}[{{var "mk"}}]
{{ $x := printf "%vmv%v" .TempVar .Rand }}{{ decLineVar $x }}
if {{var "v"}} != nil {
{{var "v"}}[{{var "mk"}}] = {{var "mv"}}
}
}
r.ReadMapEnd()
} // else len==0: TODO: Should we clear map entries?

View File

@@ -1,102 +0,0 @@
// //+build ignore
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
// ************************************************************
// DO NOT EDIT.
// THIS FILE IS AUTO-GENERATED from gen-helper.go.tmpl
// ************************************************************
package codec
// This file is used to generate helper code for codecgen.
// The values here i.e. genHelper(En|De)coder are not to be used directly by
// library users. They WILL change continously and without notice.
//
// To help enforce this, we create an unexported type with exported members.
// The only way to get the type is via the one exported type that we control (somewhat).
//
// When static codecs are created for types, they will use this value
// to perform encoding or decoding of primitives or known slice or map types.
// GenHelperEncoder is exported so that it can be used externally by codecgen.
// Library users: DO NOT USE IT DIRECTLY. IT WILL CHANGE CONTINOUSLY WITHOUT NOTICE.
func GenHelperEncoder(e *Encoder) (genHelperEncoder, encDriver) {
return genHelperEncoder{e: e}, e.e
}
// GenHelperDecoder is exported so that it can be used externally by codecgen.
// Library users: DO NOT USE IT DIRECTLY. IT WILL CHANGE CONTINOUSLY WITHOUT NOTICE.
func GenHelperDecoder(d *Decoder) (genHelperDecoder, decDriver) {
return genHelperDecoder{d: d}, d.d
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
type genHelperEncoder struct {
e *Encoder
F fastpathT
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
type genHelperDecoder struct {
d *Decoder
F fastpathT
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncBasicHandle() *BasicHandle {
return f.e.h
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncBinary() bool {
return f.e.be // f.e.hh.isBinaryEncoding()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncFallback(iv interface{}) {
// println(">>>>>>>>> EncFallback")
f.e.encodeI(iv, false, false)
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecBasicHandle() *BasicHandle {
return f.d.h
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecBinary() bool {
return f.d.be // f.d.hh.isBinaryEncoding()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecSwallow() {
f.d.swallow()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecScratchBuffer() []byte {
return f.d.b[:]
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecFallback(iv interface{}, chkPtr bool) {
// println(">>>>>>>>> DecFallback")
f.d.decodeI(iv, chkPtr, false, false, false)
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecSliceHelperStart() (decSliceHelper, int) {
return f.d.decSliceHelperStart()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecStructFieldNotFound(index int, name string) {
f.d.structFieldNotFound(index, name)
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecArrayCannotExpand(sliceLen, streamLen int) {
f.d.arrayCannotExpand(sliceLen, streamLen)
}

View File

@@ -1,250 +0,0 @@
// //+build ignore
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
// ************************************************************
// DO NOT EDIT.
// THIS FILE IS AUTO-GENERATED from gen-helper.go.tmpl
// ************************************************************
package codec
// This file is used to generate helper code for codecgen.
// The values here i.e. genHelper(En|De)coder are not to be used directly by
// library users. They WILL change continously and without notice.
//
// To help enforce this, we create an unexported type with exported members.
// The only way to get the type is via the one exported type that we control (somewhat).
//
// When static codecs are created for types, they will use this value
// to perform encoding or decoding of primitives or known slice or map types.
// GenHelperEncoder is exported so that it can be used externally by codecgen.
// Library users: DO NOT USE IT DIRECTLY. IT WILL CHANGE CONTINOUSLY WITHOUT NOTICE.
func GenHelperEncoder(e *Encoder) (genHelperEncoder, encDriver) {
return genHelperEncoder{e:e}, e.e
}
// GenHelperDecoder is exported so that it can be used externally by codecgen.
// Library users: DO NOT USE IT DIRECTLY. IT WILL CHANGE CONTINOUSLY WITHOUT NOTICE.
func GenHelperDecoder(d *Decoder) (genHelperDecoder, decDriver) {
return genHelperDecoder{d:d}, d.d
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
type genHelperEncoder struct {
e *Encoder
F fastpathT
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
type genHelperDecoder struct {
d *Decoder
F fastpathT
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncBasicHandle() *BasicHandle {
return f.e.h
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncBinary() bool {
return f.e.be // f.e.hh.isBinaryEncoding()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncFallback(iv interface{}) {
// println(">>>>>>>>> EncFallback")
f.e.encodeI(iv, false, false)
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecBasicHandle() *BasicHandle {
return f.d.h
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecBinary() bool {
return f.d.be // f.d.hh.isBinaryEncoding()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecSwallow() {
f.d.swallow()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecScratchBuffer() []byte {
return f.d.b[:]
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecFallback(iv interface{}, chkPtr bool) {
// println(">>>>>>>>> DecFallback")
f.d.decodeI(iv, chkPtr, false, false, false)
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecSliceHelperStart() (decSliceHelper, int) {
return f.d.decSliceHelperStart()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecStructFieldNotFound(index int, name string) {
f.d.structFieldNotFound(index, name)
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecArrayCannotExpand(sliceLen, streamLen int) {
f.d.arrayCannotExpand(sliceLen, streamLen)
}
{{/*
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncDriver() encDriver {
return f.e.e
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecDriver() decDriver {
return f.d.d
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncNil() {
f.e.e.EncodeNil()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncBytes(v []byte) {
f.e.e.EncodeStringBytes(c_RAW, v)
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncArrayStart(length int) {
f.e.e.EncodeArrayStart(length)
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncArrayEnd() {
f.e.e.EncodeArrayEnd()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncArrayEntrySeparator() {
f.e.e.EncodeArrayEntrySeparator()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncMapStart(length int) {
f.e.e.EncodeMapStart(length)
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncMapEnd() {
f.e.e.EncodeMapEnd()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncMapEntrySeparator() {
f.e.e.EncodeMapEntrySeparator()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) EncMapKVSeparator() {
f.e.e.EncodeMapKVSeparator()
}
// ---------
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecBytes(v *[]byte) {
*v = f.d.d.DecodeBytes(*v)
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecTryNil() bool {
return f.d.d.TryDecodeAsNil()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecContainerIsNil() (b bool) {
return f.d.d.IsContainerType(valueTypeNil)
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecContainerIsMap() (b bool) {
return f.d.d.IsContainerType(valueTypeMap)
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecContainerIsArray() (b bool) {
return f.d.d.IsContainerType(valueTypeArray)
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecCheckBreak() bool {
return f.d.d.CheckBreak()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecMapStart() int {
return f.d.d.ReadMapStart()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecArrayStart() int {
return f.d.d.ReadArrayStart()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecMapEnd() {
f.d.d.ReadMapEnd()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecArrayEnd() {
f.d.d.ReadArrayEnd()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecArrayEntrySeparator() {
f.d.d.ReadArrayEntrySeparator()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecMapEntrySeparator() {
f.d.d.ReadMapEntrySeparator()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) DecMapKVSeparator() {
f.d.d.ReadMapKVSeparator()
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) ReadStringAsBytes(bs []byte) []byte {
return f.d.d.DecodeStringAsBytes(bs)
}
// -- encode calls (primitives)
{{range .Values}}{{if .Primitive }}{{if ne .Primitive "interface{}" }}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) {{ .MethodNamePfx "Enc" true }}(v {{ .Primitive }}) {
ee := f.e.e
{{ encmd .Primitive "v" }}
}
{{ end }}{{ end }}{{ end }}
// -- decode calls (primitives)
{{range .Values}}{{if .Primitive }}{{if ne .Primitive "interface{}" }}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) {{ .MethodNamePfx "Dec" true }}(vp *{{ .Primitive }}) {
dd := f.d.d
*vp = {{ decmd .Primitive }}
}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperDecoder) {{ .MethodNamePfx "Read" true }}() (v {{ .Primitive }}) {
dd := f.d.d
v = {{ decmd .Primitive }}
return
}
{{ end }}{{ end }}{{ end }}
// -- encode calls (slices/maps)
{{range .Values}}{{if not .Primitive }}{{if .Slice }}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) {{ .MethodNamePfx "Enc" false }}(v []{{ .Elem }}) { {{ else }}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
func (f genHelperEncoder) {{ .MethodNamePfx "Enc" false }}(v map[{{ .MapKey }}]{{ .Elem }}) { {{end}}
f.F.{{ .MethodNamePfx "Enc" false }}V(v, false, f.e)
}
{{ end }}{{ end }}
// -- decode calls (slices/maps)
{{range .Values}}{{if not .Primitive }}
// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
{{if .Slice }}func (f genHelperDecoder) {{ .MethodNamePfx "Dec" false }}(vp *[]{{ .Elem }}) {
{{else}}func (f genHelperDecoder) {{ .MethodNamePfx "Dec" false }}(vp *map[{{ .MapKey }}]{{ .Elem }}) { {{end}}
v, changed := f.F.{{ .MethodNamePfx "Dec" false }}V(*vp, false, true, f.d)
if changed {
*vp = v
}
}
{{ end }}{{ end }}
*/}}

View File

@@ -1,136 +0,0 @@
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
// DO NOT EDIT. THIS FILE IS AUTO-GENERATED FROM gen-dec-(map|array).go.tmpl
const genDecMapTmpl = `
{{var "v"}} := *{{ .Varname }}
{{var "l"}} := r.ReadMapStart()
if {{var "v"}} == nil {
if {{var "l"}} > 0 {
{{var "v"}} = make(map[{{ .KTyp }}]{{ .Typ }}, {{var "l"}})
} else {
{{var "v"}} = make(map[{{ .KTyp }}]{{ .Typ }}) // supports indefinite-length, etc
}
*{{ .Varname }} = {{var "v"}}
}
if {{var "l"}} > 0 {
for {{var "j"}} := 0; {{var "j"}} < {{var "l"}}; {{var "j"}}++ {
var {{var "mk"}} {{ .KTyp }}
{{ $x := printf "%vmk%v" .TempVar .Rand }}{{ decLineVarK $x }}
{{ if eq .KTyp "interface{}" }}// special case if a byte array.
if {{var "bv"}}, {{var "bok"}} := {{var "mk"}}.([]byte); {{var "bok"}} {
{{var "mk"}} = string({{var "bv"}})
}
{{ end }}
{{var "mv"}} := {{var "v"}}[{{var "mk"}}]
{{ $x := printf "%vmv%v" .TempVar .Rand }}{{ decLineVar $x }}
if {{var "v"}} != nil {
{{var "v"}}[{{var "mk"}}] = {{var "mv"}}
}
}
} else if {{var "l"}} < 0 {
for {{var "j"}} := 0; !r.CheckBreak(); {{var "j"}}++ {
if {{var "j"}} > 0 {
r.ReadMapEntrySeparator()
}
var {{var "mk"}} {{ .KTyp }}
{{ $x := printf "%vmk%v" .TempVar .Rand }}{{ decLineVarK $x }}
{{ if eq .KTyp "interface{}" }}// special case if a byte array.
if {{var "bv"}}, {{var "bok"}} := {{var "mk"}}.([]byte); {{var "bok"}} {
{{var "mk"}} = string({{var "bv"}})
}
{{ end }}
r.ReadMapKVSeparator()
{{var "mv"}} := {{var "v"}}[{{var "mk"}}]
{{ $x := printf "%vmv%v" .TempVar .Rand }}{{ decLineVar $x }}
if {{var "v"}} != nil {
{{var "v"}}[{{var "mk"}}] = {{var "mv"}}
}
}
r.ReadMapEnd()
} // else len==0: TODO: Should we clear map entries?
`
const genDecListTmpl = `
{{var "v"}} := *{{ .Varname }}
{{var "h"}}, {{var "l"}} := z.DecSliceHelperStart()
var {{var "c"}} bool
{{ if not isArray }}if {{var "v"}} == nil {
if {{var "l"}} <= 0 {
{{var "v"}} = make({{ .CTyp }}, 0)
} else {
{{var "v"}} = make({{ .CTyp }}, {{var "l"}})
}
{{var "c"}} = true
}
{{ end }}
if {{var "l"}} == 0 { {{ if isSlice }}
if len({{var "v"}}) != 0 {
{{var "v"}} = {{var "v"}}[:0]
{{var "c"}} = true
} {{ end }}
} else if {{var "l"}} > 0 {
{{ if isChan }}
for {{var "r"}} := 0; {{var "r"}} < {{var "l"}}; {{var "r"}}++ {
var {{var "t"}} {{ .Typ }}
{{ $x := printf "%st%s" .TempVar .Rand }}{{ decLineVar $x }}
{{var "v"}} <- {{var "t"}}
{{ else }}
{{var "n"}} := {{var "l"}}
if {{var "l"}} > cap({{var "v"}}) {
{{ if isArray }}r.ReadArrayCannotExpand(len({{var "v"}}), {{var "l"}})
{{var "n"}} = len({{var "v"}})
{{ else }}{{ if .Immutable }}
{{var "v2"}} := {{var "v"}}
{{var "v"}} = make([]{{ .Typ }}, {{var "l"}}, {{var "l"}})
if len({{var "v"}}) > 0 {
copy({{var "v"}}, {{var "v2"}}[:cap({{var "v2"}})])
}
{{ else }}{{var "v"}} = make([]{{ .Typ }}, {{var "l"}}, {{var "l"}})
{{ end }}{{var "c"}} = true
{{ end }}
} else if {{var "l"}} != len({{var "v"}}) {
{{var "v"}} = {{var "v"}}[:{{var "l"}}]
{{var "c"}} = true
}
{{var "j"}} := 0
for ; {{var "j"}} < {{var "n"}} ; {{var "j"}}++ {
{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
} {{ if isArray }}
for ; {{var "j"}} < {{var "l"}} ; {{var "j"}}++ {
z.DecSwallow()
}{{ end }}
{{ end }}{{/* closing if not chan */}}
} else {
for {{var "j"}} := 0; !r.CheckBreak(); {{var "j"}}++ {
if {{var "j"}} >= len({{var "v"}}) {
{{ if isArray }}r.ReadArrayCannotExpand(len({{var "v"}}), {{var "j"}}+1)
{{ else if isSlice}}{{var "v"}} = append({{var "v"}}, {{zero}})// var {{var "z"}} {{ .Typ }}
{{var "c"}} = true {{ end }}
}
if {{var "j"}} > 0 {
{{var "h"}}.Sep({{var "j"}})
}
{{ if isChan}}
var {{var "t"}} {{ .Typ }}
{{ $x := printf "%st%s" .TempVar .Rand }}{{ decLineVar $x }}
{{var "v"}} <- {{var "t"}}
{{ else }}
if {{var "j"}} < len({{var "v"}}) {
{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
} else {
z.DecSwallow()
}
{{ end }}
}
{{var "h"}}.End()
}
if {{var "c"}} {
*{{ .Varname }} = {{var "v"}}
}
`

File diff suppressed because it is too large Load Diff

View File

@@ -1,846 +0,0 @@
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
// Contains code shared by both encode and decode.
// Some shared ideas around encoding/decoding
// ------------------------------------------
//
// If an interface{} is passed, we first do a type assertion to see if it is
// a primitive type or a map/slice of primitive types, and use a fastpath to handle it.
//
// If we start with a reflect.Value, we are already in reflect.Value land and
// will try to grab the function for the underlying Type and directly call that function.
// This is more performant than calling reflect.Value.Interface().
//
// This still helps us bypass many layers of reflection, and give best performance.
//
// Containers
// ------------
// Containers in the stream are either associative arrays (key-value pairs) or
// regular arrays (indexed by incrementing integers).
//
// Some streams support indefinite-length containers, and use a breaking
// byte-sequence to denote that the container has come to an end.
//
// Some streams also are text-based, and use explicit separators to denote the
// end/beginning of different values.
//
// During encode, we use a high-level condition to determine how to iterate through
// the container. That decision is based on whether the container is text-based (with
// separators) or binary (without separators). If binary, we do not even call the
// encoding of separators.
//
// During decode, we use a different high-level condition to determine how to iterate
// through the containers. That decision is based on whether the stream contained
// a length prefix, or if it used explicit breaks. If length-prefixed, we assume that
// it has to be binary, and we do not even try to read separators.
//
// The only codec that may suffer (slightly) is cbor, and only when decoding indefinite-length.
// It may suffer because we treat it like a text-based codec, and read separators.
// However, this read is a no-op and the cost is insignificant.
//
// Philosophy
// ------------
// On decode, this codec will update containers appropriately:
// - If struct, update fields from stream into fields of struct.
// If field in stream not found in struct, handle appropriately (based on option).
// If a struct field has no corresponding value in the stream, leave it AS IS.
// If nil in stream, set value to nil/zero value.
// - If map, update map from stream.
// If the stream value is NIL, set the map to nil.
// - if slice, try to update up to length of array in stream.
// if container len is less than stream array length,
// and container cannot be expanded, handled (based on option).
// This means you can decode 4-element stream array into 1-element array.
//
// ------------------------------------
// On encode, user can specify omitEmpty. This means that the value will be omitted
// if the zero value. The problem may occur during decode, where omitted values do not affect
// the value being decoded into. This means that if decoding into a struct with an
// int field with current value=5, and the field is omitted in the stream, then after
// decoding, the value will still be 5 (not 0).
// omitEmpty only works if you guarantee that you always decode into zero-values.
//
// ------------------------------------
// We could have truncated a map to remove keys not available in the stream,
// or set values in the struct which are not in the stream to their zero values.
// We decided against it because there is no efficient way to do it.
// We may introduce it as an option later.
// However, that will require enabling it for both runtime and code generation modes.
//
// To support truncate, we need to do 2 passes over the container:
// map
// - first collect all keys (e.g. in k1)
// - for each key in stream, mark k1 that the key should not be removed
// - after updating map, do second pass and call delete for all keys in k1 which are not marked
// struct:
// - for each field, track the *typeInfo s1
// - iterate through all s1, and for each one not marked, set value to zero
// - this involves checking the possible anonymous fields which are nil ptrs.
// too much work.
//
// ------------------------------------------
// Error Handling is done within the library using panic.
//
// This way, the code doesn't have to keep checking if an error has happened,
// and we don't have to keep sending the error value along with each call
// or storing it in the En|Decoder and checking it constantly along the way.
//
// The disadvantage is that small functions which use panics cannot be inlined.
// The code accounts for that by only using panics behind an interface;
// since interface calls cannot be inlined, this is irrelevant.
//
// We considered storing the error is En|Decoder.
// - once it has its err field set, it cannot be used again.
// - panicing will be optional, controlled by const flag.
// - code should always check error first and return early.
// We eventually decided against it as it makes the code clumsier to always
// check for these error conditions.
import (
"encoding"
"encoding/binary"
"errors"
"fmt"
"math"
"reflect"
"sort"
"strings"
"sync"
"time"
"unicode"
"unicode/utf8"
)
const (
scratchByteArrayLen = 32
// Support encoding.(Binary|Text)(Unm|M)arshaler.
// This constant flag will enable or disable it.
supportMarshalInterfaces = true
// Each Encoder or Decoder uses a cache of functions based on conditionals,
// so that the conditionals are not run every time.
//
// Either a map or a slice is used to keep track of the functions.
// The map is more natural, but has a higher cost than a slice/array.
// This flag (useMapForCodecCache) controls which is used.
//
// From benchmarks, slices with linear search perform better with < 32 entries.
// We have typically seen a high threshold of about 24 entries.
useMapForCodecCache = false
// for debugging, set this to false, to catch panic traces.
// Note that this will always cause rpc tests to fail, since they need io.EOF sent via panic.
recoverPanicToErr = true
// Fast path functions try to create a fast path encode or decode implementation
// for common maps and slices, by by-passing reflection altogether.
fastpathEnabled = true
// if checkStructForEmptyValue, check structs fields to see if an empty value.
// This could be an expensive call, so possibly disable it.
checkStructForEmptyValue = false
// if derefForIsEmptyValue, deref pointers and interfaces when checking isEmptyValue
derefForIsEmptyValue = false
)
var oneByteArr = [1]byte{0}
var zeroByteSlice = oneByteArr[:0:0]
type charEncoding uint8
const (
c_RAW charEncoding = iota
c_UTF8
c_UTF16LE
c_UTF16BE
c_UTF32LE
c_UTF32BE
)
// valueType is the stream type
type valueType uint8
const (
valueTypeUnset valueType = iota
valueTypeNil
valueTypeInt
valueTypeUint
valueTypeFloat
valueTypeBool
valueTypeString
valueTypeSymbol
valueTypeBytes
valueTypeMap
valueTypeArray
valueTypeTimestamp
valueTypeExt
// valueTypeInvalid = 0xff
)
type seqType uint8
const (
_ seqType = iota
seqTypeArray
seqTypeSlice
seqTypeChan
)
var (
bigen = binary.BigEndian
structInfoFieldName = "_struct"
cachedTypeInfo = make(map[uintptr]*typeInfo, 64)
cachedTypeInfoMutex sync.RWMutex
// mapStrIntfTyp = reflect.TypeOf(map[string]interface{}(nil))
intfSliceTyp = reflect.TypeOf([]interface{}(nil))
intfTyp = intfSliceTyp.Elem()
stringTyp = reflect.TypeOf("")
timeTyp = reflect.TypeOf(time.Time{})
rawExtTyp = reflect.TypeOf(RawExt{})
uint8SliceTyp = reflect.TypeOf([]uint8(nil))
mapBySliceTyp = reflect.TypeOf((*MapBySlice)(nil)).Elem()
binaryMarshalerTyp = reflect.TypeOf((*encoding.BinaryMarshaler)(nil)).Elem()
binaryUnmarshalerTyp = reflect.TypeOf((*encoding.BinaryUnmarshaler)(nil)).Elem()
textMarshalerTyp = reflect.TypeOf((*encoding.TextMarshaler)(nil)).Elem()
textUnmarshalerTyp = reflect.TypeOf((*encoding.TextUnmarshaler)(nil)).Elem()
selferTyp = reflect.TypeOf((*Selfer)(nil)).Elem()
uint8SliceTypId = reflect.ValueOf(uint8SliceTyp).Pointer()
rawExtTypId = reflect.ValueOf(rawExtTyp).Pointer()
intfTypId = reflect.ValueOf(intfTyp).Pointer()
timeTypId = reflect.ValueOf(timeTyp).Pointer()
stringTypId = reflect.ValueOf(stringTyp).Pointer()
// mapBySliceTypId = reflect.ValueOf(mapBySliceTyp).Pointer()
intBitsize uint8 = uint8(reflect.TypeOf(int(0)).Bits())
uintBitsize uint8 = uint8(reflect.TypeOf(uint(0)).Bits())
bsAll0x00 = []byte{0, 0, 0, 0, 0, 0, 0, 0}
bsAll0xff = []byte{0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}
chkOvf checkOverflow
noFieldNameToStructFieldInfoErr = errors.New("no field name passed to parseStructFieldInfo")
)
// Selfer defines methods by which a value can encode or decode itself.
//
// Any type which implements Selfer will be able to encode or decode itself.
// Consequently, during (en|de)code, this takes precedence over
// (text|binary)(M|Unm)arshal or extension support.
type Selfer interface {
CodecEncodeSelf(*Encoder)
CodecDecodeSelf(*Decoder)
}
// MapBySlice represents a slice which should be encoded as a map in the stream.
// The slice contains a sequence of key-value pairs.
// This affords storing a map in a specific sequence in the stream.
//
// The support of MapBySlice affords the following:
// - A slice type which implements MapBySlice will be encoded as a map
// - A slice can be decoded from a map in the stream
type MapBySlice interface {
MapBySlice()
}
// WARNING: DO NOT USE DIRECTLY. EXPORTED FOR GODOC BENEFIT. WILL BE REMOVED.
//
// BasicHandle encapsulates the common options and extension functions.
type BasicHandle struct {
extHandle
EncodeOptions
DecodeOptions
}
func (x *BasicHandle) getBasicHandle() *BasicHandle {
return x
}
// Handle is the interface for a specific encoding format.
//
// Typically, a Handle is pre-configured before first time use,
// and not modified while in use. Such a pre-configured Handle
// is safe for concurrent access.
type Handle interface {
getBasicHandle() *BasicHandle
newEncDriver(w *Encoder) encDriver
newDecDriver(r *Decoder) decDriver
isBinary() bool
}
// RawExt represents raw unprocessed extension data.
// Some codecs will decode extension data as a *RawExt if there is no registered extension for the tag.
//
// Only one of Data or Value is nil. If Data is nil, then the content of the RawExt is in the Value.
type RawExt struct {
Tag uint64
// Data is the []byte which represents the raw ext. If Data is nil, ext is exposed in Value.
// Data is used by codecs (e.g. binc, msgpack, simple) which do custom serialization of the types
Data []byte
// Value represents the extension, if Data is nil.
// Value is used by codecs (e.g. cbor) which use the format to do custom serialization of the types.
Value interface{}
}
// Ext handles custom (de)serialization of custom types / extensions.
type Ext interface {
// WriteExt converts a value to a []byte.
// It is used by codecs (e.g. binc, msgpack, simple) which do custom serialization of the types.
WriteExt(v interface{}) []byte
// ReadExt updates a value from a []byte.
// It is used by codecs (e.g. binc, msgpack, simple) which do custom serialization of the types.
ReadExt(dst interface{}, src []byte)
// ConvertExt converts a value into a simpler interface for easy encoding e.g. convert time.Time to int64.
// It is used by codecs (e.g. cbor) which use the format to do custom serialization of the types.
ConvertExt(v interface{}) interface{}
// UpdateExt updates a value from a simpler interface for easy decoding e.g. convert int64 to time.Time.
// It is used by codecs (e.g. cbor) which use the format to do custom serialization of the types.
UpdateExt(dst interface{}, src interface{})
}
// bytesExt is a wrapper implementation to support former AddExt exported method.
type bytesExt struct {
encFn func(reflect.Value) ([]byte, error)
decFn func(reflect.Value, []byte) error
}
func (x bytesExt) WriteExt(v interface{}) []byte {
// fmt.Printf(">>>>>>>>>> WriteExt: %T, %v\n", v, v)
bs, err := x.encFn(reflect.ValueOf(v))
if err != nil {
panic(err)
}
return bs
}
func (x bytesExt) ReadExt(v interface{}, bs []byte) {
// fmt.Printf(">>>>>>>>>> ReadExt: %T, %v\n", v, v)
if err := x.decFn(reflect.ValueOf(v), bs); err != nil {
panic(err)
}
}
func (x bytesExt) ConvertExt(v interface{}) interface{} {
return x.WriteExt(v)
}
func (x bytesExt) UpdateExt(dest interface{}, v interface{}) {
x.ReadExt(dest, v.([]byte))
}
// type errorString string
// func (x errorString) Error() string { return string(x) }
type binaryEncodingType struct{}
func (_ binaryEncodingType) isBinary() bool { return true }
type textEncodingType struct{}
func (_ textEncodingType) isBinary() bool { return false }
// noBuiltInTypes is embedded into many types which do not support builtins
// e.g. msgpack, simple, cbor.
type noBuiltInTypes struct{}
func (_ noBuiltInTypes) IsBuiltinType(rt uintptr) bool { return false }
func (_ noBuiltInTypes) EncodeBuiltin(rt uintptr, v interface{}) {}
func (_ noBuiltInTypes) DecodeBuiltin(rt uintptr, v interface{}) {}
type noStreamingCodec struct{}
func (_ noStreamingCodec) CheckBreak() bool { return false }
// bigenHelper.
// Users must already slice the x completely, because we will not reslice.
type bigenHelper struct {
x []byte // must be correctly sliced to appropriate len. slicing is a cost.
w encWriter
}
func (z bigenHelper) writeUint16(v uint16) {
bigen.PutUint16(z.x, v)
z.w.writeb(z.x)
}
func (z bigenHelper) writeUint32(v uint32) {
bigen.PutUint32(z.x, v)
z.w.writeb(z.x)
}
func (z bigenHelper) writeUint64(v uint64) {
bigen.PutUint64(z.x, v)
z.w.writeb(z.x)
}
type extTypeTagFn struct {
rtid uintptr
rt reflect.Type
tag uint64
ext Ext
}
type extHandle []*extTypeTagFn
// DEPRECATED: AddExt is deprecated in favor of SetExt. It exists for compatibility only.
//
// AddExt registes an encode and decode function for a reflect.Type.
// AddExt internally calls SetExt.
// To deregister an Ext, call AddExt with nil encfn and/or nil decfn.
func (o *extHandle) AddExt(
rt reflect.Type, tag byte,
encfn func(reflect.Value) ([]byte, error), decfn func(reflect.Value, []byte) error,
) (err error) {
if encfn == nil || decfn == nil {
return o.SetExt(rt, uint64(tag), nil)
}
return o.SetExt(rt, uint64(tag), bytesExt{encfn, decfn})
}
// SetExt registers a tag and Ext for a reflect.Type.
//
// Note that the type must be a named type, and specifically not
// a pointer or Interface. An error is returned if that is not honored.
//
// To Deregister an ext, call SetExt with nil Ext
func (o *extHandle) SetExt(rt reflect.Type, tag uint64, ext Ext) (err error) {
// o is a pointer, because we may need to initialize it
if rt.PkgPath() == "" || rt.Kind() == reflect.Interface {
err = fmt.Errorf("codec.Handle.AddExt: Takes named type, especially not a pointer or interface: %T",
reflect.Zero(rt).Interface())
return
}
rtid := reflect.ValueOf(rt).Pointer()
for _, v := range *o {
if v.rtid == rtid {
v.tag, v.ext = tag, ext
return
}
}
*o = append(*o, &extTypeTagFn{rtid, rt, tag, ext})
return
}
func (o extHandle) getExt(rtid uintptr) *extTypeTagFn {
for _, v := range o {
if v.rtid == rtid {
return v
}
}
return nil
}
func (o extHandle) getExtForTag(tag uint64) *extTypeTagFn {
for _, v := range o {
if v.tag == tag {
return v
}
}
return nil
}
type structFieldInfo struct {
encName string // encode name
// only one of 'i' or 'is' can be set. If 'i' is -1, then 'is' has been set.
is []int // (recursive/embedded) field index in struct
i int16 // field index in struct
omitEmpty bool
toArray bool // if field is _struct, is the toArray set?
}
// rv returns the field of the struct.
// If anonymous, it returns an Invalid
func (si *structFieldInfo) field(v reflect.Value, update bool) (rv2 reflect.Value) {
if si.i != -1 {
v = v.Field(int(si.i))
return v
}
// replicate FieldByIndex
for _, x := range si.is {
for v.Kind() == reflect.Ptr {
if v.IsNil() {
if !update {
return
}
v.Set(reflect.New(v.Type().Elem()))
}
v = v.Elem()
}
v = v.Field(x)
}
return v
}
func (si *structFieldInfo) setToZeroValue(v reflect.Value) {
if si.i != -1 {
v = v.Field(int(si.i))
v.Set(reflect.Zero(v.Type()))
// v.Set(reflect.New(v.Type()).Elem())
// v.Set(reflect.New(v.Type()))
} else {
// replicate FieldByIndex
for _, x := range si.is {
for v.Kind() == reflect.Ptr {
if v.IsNil() {
return
}
v = v.Elem()
}
v = v.Field(x)
}
v.Set(reflect.Zero(v.Type()))
}
}
func parseStructFieldInfo(fname string, stag string) *structFieldInfo {
if fname == "" {
panic(noFieldNameToStructFieldInfoErr)
}
si := structFieldInfo{
encName: fname,
}
if stag != "" {
for i, s := range strings.Split(stag, ",") {
if i == 0 {
if s != "" {
si.encName = s
}
} else {
if s == "omitempty" {
si.omitEmpty = true
} else if s == "toarray" {
si.toArray = true
}
}
}
}
// si.encNameBs = []byte(si.encName)
return &si
}
type sfiSortedByEncName []*structFieldInfo
func (p sfiSortedByEncName) Len() int {
return len(p)
}
func (p sfiSortedByEncName) Less(i, j int) bool {
return p[i].encName < p[j].encName
}
func (p sfiSortedByEncName) Swap(i, j int) {
p[i], p[j] = p[j], p[i]
}
// typeInfo keeps information about each type referenced in the encode/decode sequence.
//
// During an encode/decode sequence, we work as below:
// - If base is a built in type, en/decode base value
// - If base is registered as an extension, en/decode base value
// - If type is binary(M/Unm)arshaler, call Binary(M/Unm)arshal method
// - If type is text(M/Unm)arshaler, call Text(M/Unm)arshal method
// - Else decode appropriately based on the reflect.Kind
type typeInfo struct {
sfi []*structFieldInfo // sorted. Used when enc/dec struct to map.
sfip []*structFieldInfo // unsorted. Used when enc/dec struct to array.
rt reflect.Type
rtid uintptr
// baseId gives pointer to the base reflect.Type, after deferencing
// the pointers. E.g. base type of ***time.Time is time.Time.
base reflect.Type
baseId uintptr
baseIndir int8 // number of indirections to get to base
mbs bool // base type (T or *T) is a MapBySlice
bm bool // base type (T or *T) is a binaryMarshaler
bunm bool // base type (T or *T) is a binaryUnmarshaler
bmIndir int8 // number of indirections to get to binaryMarshaler type
bunmIndir int8 // number of indirections to get to binaryUnmarshaler type
tm bool // base type (T or *T) is a textMarshaler
tunm bool // base type (T or *T) is a textUnmarshaler
tmIndir int8 // number of indirections to get to textMarshaler type
tunmIndir int8 // number of indirections to get to textUnmarshaler type
cs bool // base type (T or *T) is a Selfer
csIndir int8 // number of indirections to get to Selfer type
toArray bool // whether this (struct) type should be encoded as an array
}
func (ti *typeInfo) indexForEncName(name string) int {
//tisfi := ti.sfi
const binarySearchThreshold = 16
if sfilen := len(ti.sfi); sfilen < binarySearchThreshold {
// linear search. faster than binary search in my testing up to 16-field structs.
for i, si := range ti.sfi {
if si.encName == name {
return i
}
}
} else {
// binary search. adapted from sort/search.go.
h, i, j := 0, 0, sfilen
for i < j {
h = i + (j-i)/2
if ti.sfi[h].encName < name {
i = h + 1
} else {
j = h
}
}
if i < sfilen && ti.sfi[i].encName == name {
return i
}
}
return -1
}
func getStructTag(t reflect.StructTag) (s string) {
// check for tags: codec, json, in that order.
// this allows seamless support for many configured structs.
s = t.Get("codec")
if s == "" {
s = t.Get("json")
}
return
}
func getTypeInfo(rtid uintptr, rt reflect.Type) (pti *typeInfo) {
var ok bool
cachedTypeInfoMutex.RLock()
pti, ok = cachedTypeInfo[rtid]
cachedTypeInfoMutex.RUnlock()
if ok {
return
}
cachedTypeInfoMutex.Lock()
defer cachedTypeInfoMutex.Unlock()
if pti, ok = cachedTypeInfo[rtid]; ok {
return
}
ti := typeInfo{rt: rt, rtid: rtid}
pti = &ti
var indir int8
if ok, indir = implementsIntf(rt, binaryMarshalerTyp); ok {
ti.bm, ti.bmIndir = true, indir
}
if ok, indir = implementsIntf(rt, binaryUnmarshalerTyp); ok {
ti.bunm, ti.bunmIndir = true, indir
}
if ok, indir = implementsIntf(rt, textMarshalerTyp); ok {
ti.tm, ti.tmIndir = true, indir
}
if ok, indir = implementsIntf(rt, textUnmarshalerTyp); ok {
ti.tunm, ti.tunmIndir = true, indir
}
if ok, indir = implementsIntf(rt, selferTyp); ok {
ti.cs, ti.csIndir = true, indir
}
if ok, _ = implementsIntf(rt, mapBySliceTyp); ok {
ti.mbs = true
}
pt := rt
var ptIndir int8
// for ; pt.Kind() == reflect.Ptr; pt, ptIndir = pt.Elem(), ptIndir+1 { }
for pt.Kind() == reflect.Ptr {
pt = pt.Elem()
ptIndir++
}
if ptIndir == 0 {
ti.base = rt
ti.baseId = rtid
} else {
ti.base = pt
ti.baseId = reflect.ValueOf(pt).Pointer()
ti.baseIndir = ptIndir
}
if rt.Kind() == reflect.Struct {
var siInfo *structFieldInfo
if f, ok := rt.FieldByName(structInfoFieldName); ok {
siInfo = parseStructFieldInfo(structInfoFieldName, getStructTag(f.Tag))
ti.toArray = siInfo.toArray
}
sfip := make([]*structFieldInfo, 0, rt.NumField())
rgetTypeInfo(rt, nil, make(map[string]bool, 16), &sfip, siInfo)
ti.sfip = make([]*structFieldInfo, len(sfip))
ti.sfi = make([]*structFieldInfo, len(sfip))
copy(ti.sfip, sfip)
sort.Sort(sfiSortedByEncName(sfip))
copy(ti.sfi, sfip)
}
// sfi = sfip
cachedTypeInfo[rtid] = pti
return
}
func rgetTypeInfo(rt reflect.Type, indexstack []int, fnameToHastag map[string]bool,
sfi *[]*structFieldInfo, siInfo *structFieldInfo,
) {
for j := 0; j < rt.NumField(); j++ {
f := rt.Field(j)
// func types are skipped.
if tk := f.Type.Kind(); tk == reflect.Func {
continue
}
stag := getStructTag(f.Tag)
if stag == "-" {
continue
}
if r1, _ := utf8.DecodeRuneInString(f.Name); r1 == utf8.RuneError || !unicode.IsUpper(r1) {
continue
}
// if anonymous and there is no struct tag and its a struct (or pointer to struct), inline it.
if f.Anonymous && stag == "" {
ft := f.Type
for ft.Kind() == reflect.Ptr {
ft = ft.Elem()
}
if ft.Kind() == reflect.Struct {
indexstack2 := make([]int, len(indexstack)+1, len(indexstack)+4)
copy(indexstack2, indexstack)
indexstack2[len(indexstack)] = j
// indexstack2 := append(append(make([]int, 0, len(indexstack)+4), indexstack...), j)
rgetTypeInfo(ft, indexstack2, fnameToHastag, sfi, siInfo)
continue
}
}
// do not let fields with same name in embedded structs override field at higher level.
// this must be done after anonymous check, to allow anonymous field
// still include their child fields
if _, ok := fnameToHastag[f.Name]; ok {
continue
}
si := parseStructFieldInfo(f.Name, stag)
// si.ikind = int(f.Type.Kind())
if len(indexstack) == 0 {
si.i = int16(j)
} else {
si.i = -1
si.is = append(append(make([]int, 0, len(indexstack)+4), indexstack...), j)
}
if siInfo != nil {
if siInfo.omitEmpty {
si.omitEmpty = true
}
}
*sfi = append(*sfi, si)
fnameToHastag[f.Name] = stag != ""
}
}
func panicToErr(err *error) {
if recoverPanicToErr {
if x := recover(); x != nil {
//debug.PrintStack()
panicValToErr(x, err)
}
}
}
// func doPanic(tag string, format string, params ...interface{}) {
// params2 := make([]interface{}, len(params)+1)
// params2[0] = tag
// copy(params2[1:], params)
// panic(fmt.Errorf("%s: "+format, params2...))
// }
func isMutableKind(k reflect.Kind) (v bool) {
return k == reflect.Int ||
k == reflect.Int8 ||
k == reflect.Int16 ||
k == reflect.Int32 ||
k == reflect.Int64 ||
k == reflect.Uint ||
k == reflect.Uint8 ||
k == reflect.Uint16 ||
k == reflect.Uint32 ||
k == reflect.Uint64 ||
k == reflect.Float32 ||
k == reflect.Float64 ||
k == reflect.Bool ||
k == reflect.String
}
// these functions must be inlinable, and not call anybody
type checkOverflow struct{}
func (_ checkOverflow) Float32(f float64) (overflow bool) {
if f < 0 {
f = -f
}
return math.MaxFloat32 < f && f <= math.MaxFloat64
}
func (_ checkOverflow) Uint(v uint64, bitsize uint8) (overflow bool) {
if bitsize == 0 || bitsize >= 64 || v == 0 {
return
}
if trunc := (v << (64 - bitsize)) >> (64 - bitsize); v != trunc {
overflow = true
}
return
}
func (_ checkOverflow) Int(v int64, bitsize uint8) (overflow bool) {
if bitsize == 0 || bitsize >= 64 || v == 0 {
return
}
if trunc := (v << (64 - bitsize)) >> (64 - bitsize); v != trunc {
overflow = true
}
return
}
func (_ checkOverflow) SignedInt(v uint64) (i int64, overflow bool) {
//e.g. -127 to 128 for int8
pos := (v >> 63) == 0
ui2 := v & 0x7fffffffffffffff
if pos {
if ui2 > math.MaxInt64 {
overflow = true
return
}
} else {
if ui2 > math.MaxInt64-1 {
overflow = true
return
}
}
i = int64(v)
return
}

View File

@@ -1,151 +0,0 @@
// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
// All non-std package dependencies live in this file,
// so porting to different environment is easy (just update functions).
import (
"errors"
"fmt"
"math"
"reflect"
)
func panicValToErr(panicVal interface{}, err *error) {
if panicVal == nil {
return
}
// case nil
switch xerr := panicVal.(type) {
case error:
*err = xerr
case string:
*err = errors.New(xerr)
default:
*err = fmt.Errorf("%v", panicVal)
}
return
}
func hIsEmptyValue(v reflect.Value, deref, checkStruct bool) bool {
switch v.Kind() {
case reflect.Invalid:
return true
case reflect.Array, reflect.Map, reflect.Slice, reflect.String:
return v.Len() == 0
case reflect.Bool:
return !v.Bool()
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
return v.Int() == 0
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
return v.Uint() == 0
case reflect.Float32, reflect.Float64:
return v.Float() == 0
case reflect.Interface, reflect.Ptr:
if deref {
if v.IsNil() {
return true
}
return hIsEmptyValue(v.Elem(), deref, checkStruct)
} else {
return v.IsNil()
}
case reflect.Struct:
if !checkStruct {
return false
}
// return true if all fields are empty. else return false.
// we cannot use equality check, because some fields may be maps/slices/etc
// and consequently the structs are not comparable.
// return v.Interface() == reflect.Zero(v.Type()).Interface()
for i, n := 0, v.NumField(); i < n; i++ {
if !hIsEmptyValue(v.Field(i), deref, checkStruct) {
return false
}
}
return true
}
return false
}
func isEmptyValue(v reflect.Value) bool {
return hIsEmptyValue(v, derefForIsEmptyValue, checkStructForEmptyValue)
}
func pruneSignExt(v []byte, pos bool) (n int) {
if len(v) < 2 {
} else if pos && v[0] == 0 {
for ; v[n] == 0 && n+1 < len(v) && (v[n+1]&(1<<7) == 0); n++ {
}
} else if !pos && v[0] == 0xff {
for ; v[n] == 0xff && n+1 < len(v) && (v[n+1]&(1<<7) != 0); n++ {
}
}
return
}
func implementsIntf(typ, iTyp reflect.Type) (success bool, indir int8) {
if typ == nil {
return
}
rt := typ
// The type might be a pointer and we need to keep
// dereferencing to the base type until we find an implementation.
for {
if rt.Implements(iTyp) {
return true, indir
}
if p := rt; p.Kind() == reflect.Ptr {
indir++
if indir >= math.MaxInt8 { // insane number of indirections
return false, 0
}
rt = p.Elem()
continue
}
break
}
// No luck yet, but if this is a base type (non-pointer), the pointer might satisfy.
if typ.Kind() != reflect.Ptr {
// Not a pointer, but does the pointer work?
if reflect.PtrTo(typ).Implements(iTyp) {
return true, -1
}
}
return false, 0
}
// validate that this function is correct ...
// culled from OGRE (Object-Oriented Graphics Rendering Engine)
// function: halfToFloatI (http://stderr.org/doc/ogre-doc/api/OgreBitwise_8h-source.html)
func halfFloatToFloatBits(yy uint16) (d uint32) {
y := uint32(yy)
s := (y >> 15) & 0x01
e := (y >> 10) & 0x1f
m := y & 0x03ff
if e == 0 {
if m == 0 { // plu or minus 0
return s << 31
} else { // Denormalized number -- renormalize it
for (m & 0x00000400) == 0 {
m <<= 1
e -= 1
}
e += 1
const zz uint32 = 0x0400
m &= ^zz
}
} else if e == 31 {
if m == 0 { // Inf
return (s << 31) | 0x7f800000
} else { // NaN
return (s << 31) | 0x7f800000 | (m << 13)
}
}
e = e + (127 - 15)
m = m << 13
return (s << 31) | (e << 23) | m
}

View File

@@ -1,20 +0,0 @@
//+build !unsafe
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
// stringView returns a view of the []byte as a string.
// In unsafe mode, it doesn't incur allocation and copying caused by conversion.
// In regular safe mode, it is an allocation and copy.
func stringView(v []byte) string {
return string(v)
}
// bytesView returns a view of the string as a []byte.
// In unsafe mode, it doesn't incur allocation and copying caused by conversion.
// In regular safe mode, it is an allocation and copy.
func bytesView(v string) []byte {
return []byte(v)
}

View File

@@ -1,155 +0,0 @@
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
// All non-std package dependencies related to testing live in this file,
// so porting to different environment is easy (just update functions).
//
// This file sets up the variables used, including testInitFns.
// Each file should add initialization that should be performed
// after flags are parsed.
//
// init is a multi-step process:
// - setup vars (handled by init functions in each file)
// - parse flags
// - setup derived vars (handled by pre-init registered functions - registered in init function)
// - post init (handled by post-init registered functions - registered in init function)
// This way, no one has to manage carefully control the initialization
// using file names, etc.
//
// Tests which require external dependencies need the -tag=x parameter.
// They should be run as:
// go test -tags=x -run=. <other parameters ...>
// Benchmarks should also take this parameter, to include the sereal, xdr, etc.
// To run against codecgen, etc, make sure you pass extra parameters.
// Example usage:
// go test "-tags=x codecgen unsafe" -bench=. <other parameters ...>
//
// To fully test everything:
// go test -tags=x -benchtime=100ms -tv -bg -bi -brw -bu -v -run=. -bench=.
import (
"errors"
"flag"
"fmt"
"reflect"
"sync"
"testing"
)
const (
testLogToT = true
failNowOnFail = true
)
var (
testNoopH = NoopHandle(8)
testMsgpackH = &MsgpackHandle{}
testBincH = &BincHandle{}
testBincHNoSym = &BincHandle{}
testBincHSym = &BincHandle{}
testSimpleH = &SimpleHandle{}
testCborH = &CborHandle{}
testJsonH = &JsonHandle{}
testPreInitFns []func()
testPostInitFns []func()
testOnce sync.Once
)
func init() {
testBincHSym.AsSymbols = AsSymbolAll
testBincHNoSym.AsSymbols = AsSymbolNone
}
func testInitAll() {
flag.Parse()
for _, f := range testPreInitFns {
f()
}
for _, f := range testPostInitFns {
f()
}
}
func logT(x interface{}, format string, args ...interface{}) {
if t, ok := x.(*testing.T); ok && t != nil && testLogToT {
if testVerbose {
t.Logf(format, args...)
}
} else if b, ok := x.(*testing.B); ok && b != nil && testLogToT {
b.Logf(format, args...)
} else {
if len(format) == 0 || format[len(format)-1] != '\n' {
format = format + "\n"
}
fmt.Printf(format, args...)
}
}
func approxDataSize(rv reflect.Value) (sum int) {
switch rk := rv.Kind(); rk {
case reflect.Invalid:
case reflect.Ptr, reflect.Interface:
sum += int(rv.Type().Size())
sum += approxDataSize(rv.Elem())
case reflect.Slice:
sum += int(rv.Type().Size())
for j := 0; j < rv.Len(); j++ {
sum += approxDataSize(rv.Index(j))
}
case reflect.String:
sum += int(rv.Type().Size())
sum += rv.Len()
case reflect.Map:
sum += int(rv.Type().Size())
for _, mk := range rv.MapKeys() {
sum += approxDataSize(mk)
sum += approxDataSize(rv.MapIndex(mk))
}
case reflect.Struct:
//struct size already includes the full data size.
//sum += int(rv.Type().Size())
for j := 0; j < rv.NumField(); j++ {
sum += approxDataSize(rv.Field(j))
}
default:
//pure value types
sum += int(rv.Type().Size())
}
return
}
// ----- functions below are used only by tests (not benchmarks)
func checkErrT(t *testing.T, err error) {
if err != nil {
logT(t, err.Error())
failT(t)
}
}
func checkEqualT(t *testing.T, v1 interface{}, v2 interface{}, desc string) (err error) {
if err = deepEqual(v1, v2); err != nil {
logT(t, "Not Equal: %s: %v. v1: %v, v2: %v", desc, err, v1, v2)
failT(t)
}
return
}
func failT(t *testing.T) {
if failNowOnFail {
t.FailNow()
} else {
t.Fail()
}
}
func deepEqual(v1, v2 interface{}) (err error) {
if !reflect.DeepEqual(v1, v2) {
err = errors.New("Not Match")
}
return
}

View File

@@ -1,39 +0,0 @@
//+build unsafe
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
import (
"unsafe"
)
// This file has unsafe variants of some helper methods.
type unsafeString struct {
Data uintptr
Len int
}
type unsafeBytes struct {
Data uintptr
Len int
Cap int
}
// stringView returns a view of the []byte as a string.
// In unsafe mode, it doesn't incur allocation and copying caused by conversion.
// In regular safe mode, it is an allocation and copy.
func stringView(v []byte) string {
x := unsafeString{uintptr(unsafe.Pointer(&v[0])), len(v)}
return *(*string)(unsafe.Pointer(&x))
}
// bytesView returns a view of the string as a []byte.
// In unsafe mode, it doesn't incur allocation and copying caused by conversion.
// In regular safe mode, it is an allocation and copy.
func bytesView(v string) []byte {
x := unsafeBytes{uintptr(unsafe.Pointer(&v)), len(v), len(v)}
return *(*[]byte)(unsafe.Pointer(&x))
}

View File

@@ -1,924 +0,0 @@
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
// This json support uses base64 encoding for bytes, because you cannot
// store and read any arbitrary string in json (only unicode).
//
// This library specifically supports UTF-8 for encoding and decoding only.
//
// Note that the library will happily encode/decode things which are not valid
// json e.g. a map[int64]string. We do it for consistency. With valid json,
// we will encode and decode appropriately.
// Users can specify their map type if necessary to force it.
//
// Note:
// - we cannot use strconv.Quote and strconv.Unquote because json quotes/unquotes differently.
// We implement it here.
// - Also, strconv.ParseXXX for floats and integers
// - only works on strings resulting in unnecessary allocation and []byte-string conversion.
// - it does a lot of redundant checks, because json numbers are simpler that what it supports.
// - We parse numbers (floats and integers) directly here.
// We only delegate parsing floats if it is a hairy float which could cause a loss of precision.
// In that case, we delegate to strconv.ParseFloat.
//
// Note:
// - encode does not beautify. There is no whitespace when encoding.
// - rpc calls which take single integer arguments or write single numeric arguments will need care.
import (
"bytes"
"encoding/base64"
"fmt"
"strconv"
"unicode/utf16"
"unicode/utf8"
)
//--------------------------------
var jsonLiterals = [...]byte{'t', 'r', 'u', 'e', 'f', 'a', 'l', 's', 'e', 'n', 'u', 'l', 'l'}
var jsonFloat64Pow10 = [...]float64{
1e0, 1e1, 1e2, 1e3, 1e4, 1e5, 1e6, 1e7, 1e8, 1e9,
1e10, 1e11, 1e12, 1e13, 1e14, 1e15, 1e16, 1e17, 1e18, 1e19,
1e20, 1e21, 1e22,
}
var jsonUint64Pow10 = [...]uint64{
1e0, 1e1, 1e2, 1e3, 1e4, 1e5, 1e6, 1e7, 1e8, 1e9,
1e10, 1e11, 1e12, 1e13, 1e14, 1e15, 1e16, 1e17, 1e18, 1e19,
}
const (
// if jsonTrackSkipWhitespace, we track Whitespace and reduce the number of redundant checks.
// Make it a const flag, so that it can be elided during linking if false.
//
// It is not a clear win, because we continually set a flag behind a pointer
// and then check it each time, as opposed to just 4 conditionals on a stack variable.
jsonTrackSkipWhitespace = true
// If !jsonValidateSymbols, decoding will be faster, by skipping some checks:
// - If we see first character of null, false or true,
// do not validate subsequent characters.
// - e.g. if we see a n, assume null and skip next 3 characters,
// and do not validate they are ull.
// P.S. Do not expect a significant decoding boost from this.
jsonValidateSymbols = true
// if jsonTruncateMantissa, truncate mantissa if trailing 0's.
// This is important because it could allow some floats to be decoded without
// deferring to strconv.ParseFloat.
jsonTruncateMantissa = true
// if mantissa >= jsonNumUintCutoff before multiplying by 10, this is an overflow
jsonNumUintCutoff = (1<<64-1)/uint64(10) + 1 // cutoff64(base)
// if mantissa >= jsonNumUintMaxVal, this is an overflow
jsonNumUintMaxVal = 1<<uint64(64) - 1
// jsonNumDigitsUint64Largest = 19
)
type jsonEncDriver struct {
e *Encoder
w encWriter
h *JsonHandle
b [64]byte // scratch
bs []byte // scratch
noBuiltInTypes
}
func (e *jsonEncDriver) EncodeNil() {
e.w.writeb(jsonLiterals[9:13]) // null
}
func (e *jsonEncDriver) EncodeBool(b bool) {
if b {
e.w.writeb(jsonLiterals[0:4]) // true
} else {
e.w.writeb(jsonLiterals[4:9]) // false
}
}
func (e *jsonEncDriver) EncodeFloat32(f float32) {
e.w.writeb(strconv.AppendFloat(e.b[:0], float64(f), 'E', -1, 32))
}
func (e *jsonEncDriver) EncodeFloat64(f float64) {
// e.w.writestr(strconv.FormatFloat(f, 'E', -1, 64))
e.w.writeb(strconv.AppendFloat(e.b[:0], f, 'E', -1, 64))
}
func (e *jsonEncDriver) EncodeInt(v int64) {
e.w.writeb(strconv.AppendInt(e.b[:0], v, 10))
}
func (e *jsonEncDriver) EncodeUint(v uint64) {
e.w.writeb(strconv.AppendUint(e.b[:0], v, 10))
}
func (e *jsonEncDriver) EncodeExt(rv interface{}, xtag uint64, ext Ext, en *Encoder) {
if v := ext.ConvertExt(rv); v == nil {
e.EncodeNil()
} else {
en.encode(v)
}
}
func (e *jsonEncDriver) EncodeRawExt(re *RawExt, en *Encoder) {
// only encodes re.Value (never re.Data)
if re.Value == nil {
e.EncodeNil()
} else {
en.encode(re.Value)
}
}
func (e *jsonEncDriver) EncodeArrayStart(length int) {
e.w.writen1('[')
}
func (e *jsonEncDriver) EncodeArrayEntrySeparator() {
e.w.writen1(',')
}
func (e *jsonEncDriver) EncodeArrayEnd() {
e.w.writen1(']')
}
func (e *jsonEncDriver) EncodeMapStart(length int) {
e.w.writen1('{')
}
func (e *jsonEncDriver) EncodeMapEntrySeparator() {
e.w.writen1(',')
}
func (e *jsonEncDriver) EncodeMapKVSeparator() {
e.w.writen1(':')
}
func (e *jsonEncDriver) EncodeMapEnd() {
e.w.writen1('}')
}
func (e *jsonEncDriver) EncodeString(c charEncoding, v string) {
// e.w.writestr(strconv.Quote(v))
e.quoteStr(v)
}
func (e *jsonEncDriver) EncodeSymbol(v string) {
// e.EncodeString(c_UTF8, v)
e.quoteStr(v)
}
func (e *jsonEncDriver) EncodeStringBytes(c charEncoding, v []byte) {
if c == c_RAW {
slen := base64.StdEncoding.EncodedLen(len(v))
if e.bs == nil {
e.bs = e.b[:]
}
if cap(e.bs) >= slen {
e.bs = e.bs[:slen]
} else {
e.bs = make([]byte, slen)
}
base64.StdEncoding.Encode(e.bs, v)
e.w.writen1('"')
e.w.writeb(e.bs)
e.w.writen1('"')
} else {
// e.EncodeString(c, string(v))
e.quoteStr(stringView(v))
}
}
func (e *jsonEncDriver) quoteStr(s string) {
// adapted from std pkg encoding/json
const hex = "0123456789abcdef"
w := e.w
w.writen1('"')
start := 0
for i := 0; i < len(s); {
if b := s[i]; b < utf8.RuneSelf {
if 0x20 <= b && b != '\\' && b != '"' && b != '<' && b != '>' && b != '&' {
i++
continue
}
if start < i {
w.writestr(s[start:i])
}
switch b {
case '\\', '"':
w.writen2('\\', b)
case '\n':
w.writen2('\\', 'n')
case '\r':
w.writen2('\\', 'r')
case '\b':
w.writen2('\\', 'b')
case '\f':
w.writen2('\\', 'f')
case '\t':
w.writen2('\\', 't')
default:
// encode all bytes < 0x20 (except \r, \n).
// also encode < > & to prevent security holes when served to some browsers.
w.writestr(`\u00`)
w.writen2(hex[b>>4], hex[b&0xF])
}
i++
start = i
continue
}
c, size := utf8.DecodeRuneInString(s[i:])
if c == utf8.RuneError && size == 1 {
if start < i {
w.writestr(s[start:i])
}
w.writestr(`\ufffd`)
i += size
start = i
continue
}
// U+2028 is LINE SEPARATOR. U+2029 is PARAGRAPH SEPARATOR.
// Both technically valid JSON, but bomb on JSONP, so fix here.
if c == '\u2028' || c == '\u2029' {
if start < i {
w.writestr(s[start:i])
}
w.writestr(`\u202`)
w.writen1(hex[c&0xF])
i += size
start = i
continue
}
i += size
}
if start < len(s) {
w.writestr(s[start:])
}
w.writen1('"')
}
//--------------------------------
type jsonNum struct {
bytes []byte // may have [+-.eE0-9]
mantissa uint64 // where mantissa ends, and maybe dot begins.
exponent int16 // exponent value.
manOverflow bool
neg bool // started with -. No initial sign in the bytes above.
dot bool // has dot
explicitExponent bool // explicit exponent
}
func (x *jsonNum) reset() {
x.bytes = x.bytes[:0]
x.manOverflow = false
x.neg = false
x.dot = false
x.explicitExponent = false
x.mantissa = 0
x.exponent = 0
}
// uintExp is called only if exponent > 0.
func (x *jsonNum) uintExp() (n uint64, overflow bool) {
n = x.mantissa
e := x.exponent
if e >= int16(len(jsonUint64Pow10)) {
overflow = true
return
}
n *= jsonUint64Pow10[e]
if n < x.mantissa || n > jsonNumUintMaxVal {
overflow = true
return
}
return
// for i := int16(0); i < e; i++ {
// if n >= jsonNumUintCutoff {
// overflow = true
// return
// }
// n *= 10
// }
// return
}
func (x *jsonNum) floatVal() (f float64) {
// We do not want to lose precision.
// Consequently, we will delegate to strconv.ParseFloat if any of the following happen:
// - There are more digits than in math.MaxUint64: 18446744073709551615 (20 digits)
// We expect up to 99.... (19 digits)
// - The mantissa cannot fit into a 52 bits of uint64
// - The exponent is beyond our scope ie beyong 22.
const uint64MantissaBits = 52
const maxExponent = int16(len(jsonFloat64Pow10)) - 1
parseUsingStrConv := x.manOverflow ||
x.exponent > maxExponent ||
(x.exponent < 0 && -(x.exponent) > maxExponent) ||
x.mantissa>>uint64MantissaBits != 0
if parseUsingStrConv {
var err error
if f, err = strconv.ParseFloat(stringView(x.bytes), 64); err != nil {
panic(fmt.Errorf("parse float: %s, %v", x.bytes, err))
return
}
if x.neg {
f = -f
}
return
}
// all good. so handle parse here.
f = float64(x.mantissa)
// fmt.Printf(".Float: uint64 value: %v, float: %v\n", m, f)
if x.neg {
f = -f
}
if x.exponent > 0 {
f *= jsonFloat64Pow10[x.exponent]
} else if x.exponent < 0 {
f /= jsonFloat64Pow10[-x.exponent]
}
return
}
type jsonDecDriver struct {
d *Decoder
h *JsonHandle
r decReader // *bytesDecReader decReader
ct valueType // container type. one of unset, array or map.
bstr [8]byte // scratch used for string \UXXX parsing
b [64]byte // scratch
wsSkipped bool // whitespace skipped
n jsonNum
noBuiltInTypes
}
// This will skip whitespace characters and return the next byte to read.
// The next byte determines what the value will be one of.
func (d *jsonDecDriver) skipWhitespace(unread bool) (b byte) {
// as initReadNext is not called all the time, we set ct to unSet whenever
// we skipwhitespace, as this is the signal that something new is about to be read.
d.ct = valueTypeUnset
b = d.r.readn1()
if !jsonTrackSkipWhitespace || !d.wsSkipped {
for ; b == ' ' || b == '\t' || b == '\r' || b == '\n'; b = d.r.readn1() {
}
if jsonTrackSkipWhitespace {
d.wsSkipped = true
}
}
if unread {
d.r.unreadn1()
}
return b
}
func (d *jsonDecDriver) CheckBreak() bool {
b := d.skipWhitespace(true)
return b == '}' || b == ']'
}
func (d *jsonDecDriver) readStrIdx(fromIdx, toIdx uint8) {
bs := d.r.readx(int(toIdx - fromIdx))
if jsonValidateSymbols {
if !bytes.Equal(bs, jsonLiterals[fromIdx:toIdx]) {
d.d.errorf("json: expecting %s: got %s", jsonLiterals[fromIdx:toIdx], bs)
return
}
}
if jsonTrackSkipWhitespace {
d.wsSkipped = false
}
}
func (d *jsonDecDriver) TryDecodeAsNil() bool {
b := d.skipWhitespace(true)
if b == 'n' {
d.readStrIdx(9, 13) // null
d.ct = valueTypeNil
return true
}
return false
}
func (d *jsonDecDriver) DecodeBool() bool {
b := d.skipWhitespace(false)
if b == 'f' {
d.readStrIdx(5, 9) // alse
return false
}
if b == 't' {
d.readStrIdx(1, 4) // rue
return true
}
d.d.errorf("json: decode bool: got first char %c", b)
return false // "unreachable"
}
func (d *jsonDecDriver) ReadMapStart() int {
d.expectChar('{')
d.ct = valueTypeMap
return -1
}
func (d *jsonDecDriver) ReadArrayStart() int {
d.expectChar('[')
d.ct = valueTypeArray
return -1
}
func (d *jsonDecDriver) ReadMapEnd() {
d.expectChar('}')
}
func (d *jsonDecDriver) ReadArrayEnd() {
d.expectChar(']')
}
func (d *jsonDecDriver) ReadArrayEntrySeparator() {
d.expectChar(',')
}
func (d *jsonDecDriver) ReadMapEntrySeparator() {
d.expectChar(',')
}
func (d *jsonDecDriver) ReadMapKVSeparator() {
d.expectChar(':')
}
func (d *jsonDecDriver) expectChar(c uint8) {
b := d.skipWhitespace(false)
if b != c {
d.d.errorf("json: expect char %c but got char %c", c, b)
return
}
if jsonTrackSkipWhitespace {
d.wsSkipped = false
}
}
func (d *jsonDecDriver) IsContainerType(vt valueType) bool {
// check container type by checking the first char
if d.ct == valueTypeUnset {
b := d.skipWhitespace(true)
if b == '{' {
d.ct = valueTypeMap
} else if b == '[' {
d.ct = valueTypeArray
} else if b == 'n' {
d.ct = valueTypeNil
} else if b == '"' {
d.ct = valueTypeString
}
}
if vt == valueTypeNil || vt == valueTypeBytes || vt == valueTypeString ||
vt == valueTypeArray || vt == valueTypeMap {
return d.ct == vt
}
// ugorji: made switch into conditionals, so that IsContainerType can be inlined.
// switch vt {
// case valueTypeNil, valueTypeBytes, valueTypeString, valueTypeArray, valueTypeMap:
// return d.ct == vt
// }
d.d.errorf("isContainerType: unsupported parameter: %v", vt)
return false // "unreachable"
}
func (d *jsonDecDriver) decNum(storeBytes bool) {
// storeBytes = true // TODO: remove.
// If it is has a . or an e|E, decode as a float; else decode as an int.
b := d.skipWhitespace(false)
if !(b == '+' || b == '-' || b == '.' || (b >= '0' && b <= '9')) {
d.d.errorf("json: decNum: got first char '%c'", b)
return
}
const cutoff = (1<<64-1)/uint64(10) + 1 // cutoff64(base)
const jsonNumUintMaxVal = 1<<uint64(64) - 1
// var n jsonNum // create stack-copy jsonNum, and set to pointer at end.
// n.bytes = d.n.bytes[:0]
n := &d.n
n.reset()
// The format of a number is as below:
// parsing: sign? digit* dot? digit* e? sign? digit*
// states: 0 1* 2 3* 4 5* 6 7
// We honor this state so we can break correctly.
var state uint8 = 0
var eNeg bool
var e int16
var eof bool
LOOP:
for !eof {
// fmt.Printf("LOOP: b: %q\n", b)
switch b {
case '+':
switch state {
case 0:
state = 2
// do not add sign to the slice ...
b, eof = d.r.readn1eof()
continue
case 6: // typ = jsonNumFloat
state = 7
default:
break LOOP
}
case '-':
switch state {
case 0:
state = 2
n.neg = true
// do not add sign to the slice ...
b, eof = d.r.readn1eof()
continue
case 6: // typ = jsonNumFloat
eNeg = true
state = 7
default:
break LOOP
}
case '.':
switch state {
case 0, 2: // typ = jsonNumFloat
state = 4
n.dot = true
default:
break LOOP
}
case 'e', 'E':
switch state {
case 0, 2, 4: // typ = jsonNumFloat
state = 6
// n.mantissaEndIndex = int16(len(n.bytes))
n.explicitExponent = true
default:
break LOOP
}
case '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
switch state {
case 0:
state = 2
fallthrough
case 2:
fallthrough
case 4:
if n.dot {
n.exponent--
}
if n.mantissa >= jsonNumUintCutoff {
n.manOverflow = true
break
}
v := uint64(b - '0')
n.mantissa *= 10
if v != 0 {
n1 := n.mantissa + v
if n1 < n.mantissa || n1 > jsonNumUintMaxVal {
n.manOverflow = true // n+v overflows
break
}
n.mantissa = n1
}
case 6:
state = 7
fallthrough
case 7:
if !(b == '0' && e == 0) {
e = e*10 + int16(b-'0')
}
default:
break LOOP
}
default:
break LOOP
}
if storeBytes {
n.bytes = append(n.bytes, b)
}
b, eof = d.r.readn1eof()
}
if jsonTruncateMantissa && n.mantissa != 0 {
for n.mantissa%10 == 0 {
n.mantissa /= 10
n.exponent++
}
}
if e != 0 {
if eNeg {
n.exponent -= e
} else {
n.exponent += e
}
}
// d.n = n
if !eof {
d.r.unreadn1()
}
if jsonTrackSkipWhitespace {
d.wsSkipped = false
}
// fmt.Printf("1: n: bytes: %s, neg: %v, dot: %v, exponent: %v, mantissaEndIndex: %v\n",
// n.bytes, n.neg, n.dot, n.exponent, n.mantissaEndIndex)
return
}
func (d *jsonDecDriver) DecodeInt(bitsize uint8) (i int64) {
d.decNum(false)
n := &d.n
if n.manOverflow {
d.d.errorf("json: overflow integer after: %v", n.mantissa)
return
}
var u uint64
if n.exponent == 0 {
u = n.mantissa
} else if n.exponent < 0 {
d.d.errorf("json: fractional integer")
return
} else if n.exponent > 0 {
var overflow bool
if u, overflow = n.uintExp(); overflow {
d.d.errorf("json: overflow integer")
return
}
}
i = int64(u)
if n.neg {
i = -i
}
if chkOvf.Int(i, bitsize) {
d.d.errorf("json: overflow %v bits: %s", bitsize, n.bytes)
return
}
// fmt.Printf("DecodeInt: %v\n", i)
return
}
func (d *jsonDecDriver) DecodeUint(bitsize uint8) (u uint64) {
d.decNum(false)
n := &d.n
if n.neg {
d.d.errorf("json: unsigned integer cannot be negative")
return
}
if n.manOverflow {
d.d.errorf("json: overflow integer after: %v", n.mantissa)
return
}
if n.exponent == 0 {
u = n.mantissa
} else if n.exponent < 0 {
d.d.errorf("json: fractional integer")
return
} else if n.exponent > 0 {
var overflow bool
if u, overflow = n.uintExp(); overflow {
d.d.errorf("json: overflow integer")
return
}
}
if chkOvf.Uint(u, bitsize) {
d.d.errorf("json: overflow %v bits: %s", bitsize, n.bytes)
return
}
// fmt.Printf("DecodeUint: %v\n", u)
return
}
func (d *jsonDecDriver) DecodeFloat(chkOverflow32 bool) (f float64) {
d.decNum(true)
n := &d.n
f = n.floatVal()
if chkOverflow32 && chkOvf.Float32(f) {
d.d.errorf("json: overflow float32: %v, %s", f, n.bytes)
return
}
return
}
func (d *jsonDecDriver) DecodeExt(rv interface{}, xtag uint64, ext Ext) (realxtag uint64) {
if ext == nil {
re := rv.(*RawExt)
re.Tag = xtag
d.d.decode(&re.Value)
} else {
var v interface{}
d.d.decode(&v)
ext.UpdateExt(rv, v)
}
return
}
func (d *jsonDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOut []byte) {
// zerocopy doesn't matter for json, as the bytes must be parsed.
bs0 := d.appendStringAsBytes(d.b[:0])
if isstring {
return bs0
}
slen := base64.StdEncoding.DecodedLen(len(bs0))
if cap(bs) >= slen {
bsOut = bs[:slen]
} else {
bsOut = make([]byte, slen)
}
slen2, err := base64.StdEncoding.Decode(bsOut, bs0)
if err != nil {
d.d.errorf("json: error decoding base64 binary '%s': %v", bs0, err)
return nil
}
if slen != slen2 {
bsOut = bsOut[:slen2]
}
return
}
func (d *jsonDecDriver) DecodeString() (s string) {
return string(d.appendStringAsBytes(d.b[:0]))
}
func (d *jsonDecDriver) appendStringAsBytes(v []byte) []byte {
d.expectChar('"')
for {
c := d.r.readn1()
if c == '"' {
break
} else if c == '\\' {
c = d.r.readn1()
switch c {
case '"', '\\', '/', '\'':
v = append(v, c)
case 'b':
v = append(v, '\b')
case 'f':
v = append(v, '\f')
case 'n':
v = append(v, '\n')
case 'r':
v = append(v, '\r')
case 't':
v = append(v, '\t')
case 'u':
rr := d.jsonU4(false)
// fmt.Printf("$$$$$$$$$: is surrogate: %v\n", utf16.IsSurrogate(rr))
if utf16.IsSurrogate(rr) {
rr = utf16.DecodeRune(rr, d.jsonU4(true))
}
w2 := utf8.EncodeRune(d.bstr[:], rr)
v = append(v, d.bstr[:w2]...)
default:
d.d.errorf("json: unsupported escaped value: %c", c)
return nil
}
} else {
v = append(v, c)
}
}
if jsonTrackSkipWhitespace {
d.wsSkipped = false
}
return v
}
func (d *jsonDecDriver) jsonU4(checkSlashU bool) rune {
if checkSlashU && !(d.r.readn1() == '\\' && d.r.readn1() == 'u') {
d.d.errorf(`json: unquoteStr: invalid unicode sequence. Expecting \u`)
return 0
}
// u, _ := strconv.ParseUint(string(d.bstr[:4]), 16, 64)
var u uint32
for i := 0; i < 4; i++ {
v := d.r.readn1()
if '0' <= v && v <= '9' {
v = v - '0'
} else if 'a' <= v && v <= 'z' {
v = v - 'a' + 10
} else if 'A' <= v && v <= 'Z' {
v = v - 'A' + 10
} else {
d.d.errorf(`json: unquoteStr: invalid hex char in \u unicode sequence: %q`, v)
return 0
}
u = u*16 + uint32(v)
}
return rune(u)
}
func (d *jsonDecDriver) DecodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
n := d.skipWhitespace(true)
switch n {
case 'n':
d.readStrIdx(9, 13) // null
vt = valueTypeNil
case 'f':
d.readStrIdx(4, 9) // false
vt = valueTypeBool
v = false
case 't':
d.readStrIdx(0, 4) // true
vt = valueTypeBool
v = true
case '{':
vt = valueTypeMap
decodeFurther = true
case '[':
vt = valueTypeArray
decodeFurther = true
case '"':
vt = valueTypeString
v = d.DecodeString()
default: // number
d.decNum(true)
n := &d.n
// if the string had a any of [.eE], then decode as float.
switch {
case n.explicitExponent, n.dot, n.exponent < 0, n.manOverflow:
vt = valueTypeFloat
v = n.floatVal()
case n.exponent == 0:
u := n.mantissa
switch {
case n.neg:
vt = valueTypeInt
v = -int64(u)
case d.h.SignedInteger:
vt = valueTypeInt
v = int64(u)
default:
vt = valueTypeUint
v = u
}
default:
u, overflow := n.uintExp()
switch {
case overflow:
vt = valueTypeFloat
v = n.floatVal()
case n.neg:
vt = valueTypeInt
v = -int64(u)
case d.h.SignedInteger:
vt = valueTypeInt
v = int64(u)
default:
vt = valueTypeUint
v = u
}
}
// fmt.Printf("DecodeNaked: Number: %T, %v\n", v, v)
}
return
}
//----------------------
// JsonHandle is a handle for JSON encoding format.
//
// Json is comprehensively supported:
// - decodes numbers into interface{} as int, uint or float64
// - encodes and decodes []byte using base64 Std Encoding
// - UTF-8 support for encoding and decoding
//
// It has better performance than the json library in the standard library,
// by leveraging the performance improvements of the codec library and
// minimizing allocations.
//
// In addition, it doesn't read more bytes than necessary during a decode, which allows
// reading multiple values from a stream containing json and non-json content.
// For example, a user can read a json value, then a cbor value, then a msgpack value,
// all from the same stream in sequence.
type JsonHandle struct {
BasicHandle
textEncodingType
}
func (h *JsonHandle) newEncDriver(e *Encoder) encDriver {
return &jsonEncDriver{e: e, w: e.w, h: h}
}
func (h *JsonHandle) newDecDriver(d *Decoder) decDriver {
// d := jsonDecDriver{r: r.(*bytesDecReader), h: h}
hd := jsonDecDriver{d: d, r: d.r, h: h}
hd.n.bytes = d.b[:]
return &hd
}
var jsonEncodeTerminate = []byte{' '}
func (h *JsonHandle) rpcEncodeTerminate() []byte {
return jsonEncodeTerminate
}
var _ decDriver = (*jsonDecDriver)(nil)
var _ encDriver = (*jsonEncDriver)(nil)

View File

@@ -1,843 +0,0 @@
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
/*
MSGPACK
Msgpack-c implementation powers the c, c++, python, ruby, etc libraries.
We need to maintain compatibility with it and how it encodes integer values
without caring about the type.
For compatibility with behaviour of msgpack-c reference implementation:
- Go intX (>0) and uintX
IS ENCODED AS
msgpack +ve fixnum, unsigned
- Go intX (<0)
IS ENCODED AS
msgpack -ve fixnum, signed
*/
package codec
import (
"fmt"
"io"
"math"
"net/rpc"
)
const (
mpPosFixNumMin byte = 0x00
mpPosFixNumMax = 0x7f
mpFixMapMin = 0x80
mpFixMapMax = 0x8f
mpFixArrayMin = 0x90
mpFixArrayMax = 0x9f
mpFixStrMin = 0xa0
mpFixStrMax = 0xbf
mpNil = 0xc0
_ = 0xc1
mpFalse = 0xc2
mpTrue = 0xc3
mpFloat = 0xca
mpDouble = 0xcb
mpUint8 = 0xcc
mpUint16 = 0xcd
mpUint32 = 0xce
mpUint64 = 0xcf
mpInt8 = 0xd0
mpInt16 = 0xd1
mpInt32 = 0xd2
mpInt64 = 0xd3
// extensions below
mpBin8 = 0xc4
mpBin16 = 0xc5
mpBin32 = 0xc6
mpExt8 = 0xc7
mpExt16 = 0xc8
mpExt32 = 0xc9
mpFixExt1 = 0xd4
mpFixExt2 = 0xd5
mpFixExt4 = 0xd6
mpFixExt8 = 0xd7
mpFixExt16 = 0xd8
mpStr8 = 0xd9 // new
mpStr16 = 0xda
mpStr32 = 0xdb
mpArray16 = 0xdc
mpArray32 = 0xdd
mpMap16 = 0xde
mpMap32 = 0xdf
mpNegFixNumMin = 0xe0
mpNegFixNumMax = 0xff
)
// MsgpackSpecRpcMultiArgs is a special type which signifies to the MsgpackSpecRpcCodec
// that the backend RPC service takes multiple arguments, which have been arranged
// in sequence in the slice.
//
// The Codec then passes it AS-IS to the rpc service (without wrapping it in an
// array of 1 element).
type MsgpackSpecRpcMultiArgs []interface{}
// A MsgpackContainer type specifies the different types of msgpackContainers.
type msgpackContainerType struct {
fixCutoff int
bFixMin, b8, b16, b32 byte
hasFixMin, has8, has8Always bool
}
var (
msgpackContainerStr = msgpackContainerType{32, mpFixStrMin, mpStr8, mpStr16, mpStr32, true, true, false}
msgpackContainerBin = msgpackContainerType{0, 0, mpBin8, mpBin16, mpBin32, false, true, true}
msgpackContainerList = msgpackContainerType{16, mpFixArrayMin, 0, mpArray16, mpArray32, true, false, false}
msgpackContainerMap = msgpackContainerType{16, mpFixMapMin, 0, mpMap16, mpMap32, true, false, false}
)
//---------------------------------------------
type msgpackEncDriver struct {
e *Encoder
w encWriter
h *MsgpackHandle
noBuiltInTypes
encNoSeparator
x [8]byte
}
func (e *msgpackEncDriver) EncodeNil() {
e.w.writen1(mpNil)
}
func (e *msgpackEncDriver) EncodeInt(i int64) {
if i >= 0 {
e.EncodeUint(uint64(i))
} else if i >= -32 {
e.w.writen1(byte(i))
} else if i >= math.MinInt8 {
e.w.writen2(mpInt8, byte(i))
} else if i >= math.MinInt16 {
e.w.writen1(mpInt16)
bigenHelper{e.x[:2], e.w}.writeUint16(uint16(i))
} else if i >= math.MinInt32 {
e.w.writen1(mpInt32)
bigenHelper{e.x[:4], e.w}.writeUint32(uint32(i))
} else {
e.w.writen1(mpInt64)
bigenHelper{e.x[:8], e.w}.writeUint64(uint64(i))
}
}
func (e *msgpackEncDriver) EncodeUint(i uint64) {
if i <= math.MaxInt8 {
e.w.writen1(byte(i))
} else if i <= math.MaxUint8 {
e.w.writen2(mpUint8, byte(i))
} else if i <= math.MaxUint16 {
e.w.writen1(mpUint16)
bigenHelper{e.x[:2], e.w}.writeUint16(uint16(i))
} else if i <= math.MaxUint32 {
e.w.writen1(mpUint32)
bigenHelper{e.x[:4], e.w}.writeUint32(uint32(i))
} else {
e.w.writen1(mpUint64)
bigenHelper{e.x[:8], e.w}.writeUint64(uint64(i))
}
}
func (e *msgpackEncDriver) EncodeBool(b bool) {
if b {
e.w.writen1(mpTrue)
} else {
e.w.writen1(mpFalse)
}
}
func (e *msgpackEncDriver) EncodeFloat32(f float32) {
e.w.writen1(mpFloat)
bigenHelper{e.x[:4], e.w}.writeUint32(math.Float32bits(f))
}
func (e *msgpackEncDriver) EncodeFloat64(f float64) {
e.w.writen1(mpDouble)
bigenHelper{e.x[:8], e.w}.writeUint64(math.Float64bits(f))
}
func (e *msgpackEncDriver) EncodeExt(v interface{}, xtag uint64, ext Ext, _ *Encoder) {
bs := ext.WriteExt(v)
if bs == nil {
e.EncodeNil()
return
}
if e.h.WriteExt {
e.encodeExtPreamble(uint8(xtag), len(bs))
e.w.writeb(bs)
} else {
e.EncodeStringBytes(c_RAW, bs)
}
}
func (e *msgpackEncDriver) EncodeRawExt(re *RawExt, _ *Encoder) {
e.encodeExtPreamble(uint8(re.Tag), len(re.Data))
e.w.writeb(re.Data)
}
func (e *msgpackEncDriver) encodeExtPreamble(xtag byte, l int) {
if l == 1 {
e.w.writen2(mpFixExt1, xtag)
} else if l == 2 {
e.w.writen2(mpFixExt2, xtag)
} else if l == 4 {
e.w.writen2(mpFixExt4, xtag)
} else if l == 8 {
e.w.writen2(mpFixExt8, xtag)
} else if l == 16 {
e.w.writen2(mpFixExt16, xtag)
} else if l < 256 {
e.w.writen2(mpExt8, byte(l))
e.w.writen1(xtag)
} else if l < 65536 {
e.w.writen1(mpExt16)
bigenHelper{e.x[:2], e.w}.writeUint16(uint16(l))
e.w.writen1(xtag)
} else {
e.w.writen1(mpExt32)
bigenHelper{e.x[:4], e.w}.writeUint32(uint32(l))
e.w.writen1(xtag)
}
}
func (e *msgpackEncDriver) EncodeArrayStart(length int) {
e.writeContainerLen(msgpackContainerList, length)
}
func (e *msgpackEncDriver) EncodeMapStart(length int) {
e.writeContainerLen(msgpackContainerMap, length)
}
func (e *msgpackEncDriver) EncodeString(c charEncoding, s string) {
if c == c_RAW && e.h.WriteExt {
e.writeContainerLen(msgpackContainerBin, len(s))
} else {
e.writeContainerLen(msgpackContainerStr, len(s))
}
if len(s) > 0 {
e.w.writestr(s)
}
}
func (e *msgpackEncDriver) EncodeSymbol(v string) {
e.EncodeString(c_UTF8, v)
}
func (e *msgpackEncDriver) EncodeStringBytes(c charEncoding, bs []byte) {
if c == c_RAW && e.h.WriteExt {
e.writeContainerLen(msgpackContainerBin, len(bs))
} else {
e.writeContainerLen(msgpackContainerStr, len(bs))
}
if len(bs) > 0 {
e.w.writeb(bs)
}
}
func (e *msgpackEncDriver) writeContainerLen(ct msgpackContainerType, l int) {
if ct.hasFixMin && l < ct.fixCutoff {
e.w.writen1(ct.bFixMin | byte(l))
} else if ct.has8 && l < 256 && (ct.has8Always || e.h.WriteExt) {
e.w.writen2(ct.b8, uint8(l))
} else if l < 65536 {
e.w.writen1(ct.b16)
bigenHelper{e.x[:2], e.w}.writeUint16(uint16(l))
} else {
e.w.writen1(ct.b32)
bigenHelper{e.x[:4], e.w}.writeUint32(uint32(l))
}
}
//---------------------------------------------
type msgpackDecDriver struct {
d *Decoder
r decReader // *Decoder decReader decReaderT
h *MsgpackHandle
b [scratchByteArrayLen]byte
bd byte
bdRead bool
br bool // bytes reader
bdType valueType
noBuiltInTypes
noStreamingCodec
decNoSeparator
}
// Note: This returns either a primitive (int, bool, etc) for non-containers,
// or a containerType, or a specific type denoting nil or extension.
// It is called when a nil interface{} is passed, leaving it up to the DecDriver
// to introspect the stream and decide how best to decode.
// It deciphers the value by looking at the stream first.
func (d *msgpackDecDriver) DecodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
if !d.bdRead {
d.readNextBd()
}
bd := d.bd
switch bd {
case mpNil:
vt = valueTypeNil
d.bdRead = false
case mpFalse:
vt = valueTypeBool
v = false
case mpTrue:
vt = valueTypeBool
v = true
case mpFloat:
vt = valueTypeFloat
v = float64(math.Float32frombits(bigen.Uint32(d.r.readx(4))))
case mpDouble:
vt = valueTypeFloat
v = math.Float64frombits(bigen.Uint64(d.r.readx(8)))
case mpUint8:
vt = valueTypeUint
v = uint64(d.r.readn1())
case mpUint16:
vt = valueTypeUint
v = uint64(bigen.Uint16(d.r.readx(2)))
case mpUint32:
vt = valueTypeUint
v = uint64(bigen.Uint32(d.r.readx(4)))
case mpUint64:
vt = valueTypeUint
v = uint64(bigen.Uint64(d.r.readx(8)))
case mpInt8:
vt = valueTypeInt
v = int64(int8(d.r.readn1()))
case mpInt16:
vt = valueTypeInt
v = int64(int16(bigen.Uint16(d.r.readx(2))))
case mpInt32:
vt = valueTypeInt
v = int64(int32(bigen.Uint32(d.r.readx(4))))
case mpInt64:
vt = valueTypeInt
v = int64(int64(bigen.Uint64(d.r.readx(8))))
default:
switch {
case bd >= mpPosFixNumMin && bd <= mpPosFixNumMax:
// positive fixnum (always signed)
vt = valueTypeInt
v = int64(int8(bd))
case bd >= mpNegFixNumMin && bd <= mpNegFixNumMax:
// negative fixnum
vt = valueTypeInt
v = int64(int8(bd))
case bd == mpStr8, bd == mpStr16, bd == mpStr32, bd >= mpFixStrMin && bd <= mpFixStrMax:
if d.h.RawToString {
var rvm string
vt = valueTypeString
v = &rvm
} else {
var rvm = zeroByteSlice
vt = valueTypeBytes
v = &rvm
}
decodeFurther = true
case bd == mpBin8, bd == mpBin16, bd == mpBin32:
var rvm = zeroByteSlice
vt = valueTypeBytes
v = &rvm
decodeFurther = true
case bd == mpArray16, bd == mpArray32, bd >= mpFixArrayMin && bd <= mpFixArrayMax:
vt = valueTypeArray
decodeFurther = true
case bd == mpMap16, bd == mpMap32, bd >= mpFixMapMin && bd <= mpFixMapMax:
vt = valueTypeMap
decodeFurther = true
case bd >= mpFixExt1 && bd <= mpFixExt16, bd >= mpExt8 && bd <= mpExt32:
clen := d.readExtLen()
var re RawExt
re.Tag = uint64(d.r.readn1())
re.Data = d.r.readx(clen)
v = &re
vt = valueTypeExt
default:
d.d.errorf("Nil-Deciphered DecodeValue: %s: hex: %x, dec: %d", msgBadDesc, bd, bd)
return
}
}
if !decodeFurther {
d.bdRead = false
}
if vt == valueTypeUint && d.h.SignedInteger {
d.bdType = valueTypeInt
v = int64(v.(uint64))
}
return
}
// int can be decoded from msgpack type: intXXX or uintXXX
func (d *msgpackDecDriver) DecodeInt(bitsize uint8) (i int64) {
if !d.bdRead {
d.readNextBd()
}
switch d.bd {
case mpUint8:
i = int64(uint64(d.r.readn1()))
case mpUint16:
i = int64(uint64(bigen.Uint16(d.r.readx(2))))
case mpUint32:
i = int64(uint64(bigen.Uint32(d.r.readx(4))))
case mpUint64:
i = int64(bigen.Uint64(d.r.readx(8)))
case mpInt8:
i = int64(int8(d.r.readn1()))
case mpInt16:
i = int64(int16(bigen.Uint16(d.r.readx(2))))
case mpInt32:
i = int64(int32(bigen.Uint32(d.r.readx(4))))
case mpInt64:
i = int64(bigen.Uint64(d.r.readx(8)))
default:
switch {
case d.bd >= mpPosFixNumMin && d.bd <= mpPosFixNumMax:
i = int64(int8(d.bd))
case d.bd >= mpNegFixNumMin && d.bd <= mpNegFixNumMax:
i = int64(int8(d.bd))
default:
d.d.errorf("Unhandled single-byte unsigned integer value: %s: %x", msgBadDesc, d.bd)
return
}
}
// check overflow (logic adapted from std pkg reflect/value.go OverflowUint()
if bitsize > 0 {
if trunc := (i << (64 - bitsize)) >> (64 - bitsize); i != trunc {
d.d.errorf("Overflow int value: %v", i)
return
}
}
d.bdRead = false
return
}
// uint can be decoded from msgpack type: intXXX or uintXXX
func (d *msgpackDecDriver) DecodeUint(bitsize uint8) (ui uint64) {
if !d.bdRead {
d.readNextBd()
}
switch d.bd {
case mpUint8:
ui = uint64(d.r.readn1())
case mpUint16:
ui = uint64(bigen.Uint16(d.r.readx(2)))
case mpUint32:
ui = uint64(bigen.Uint32(d.r.readx(4)))
case mpUint64:
ui = bigen.Uint64(d.r.readx(8))
case mpInt8:
if i := int64(int8(d.r.readn1())); i >= 0 {
ui = uint64(i)
} else {
d.d.errorf("Assigning negative signed value: %v, to unsigned type", i)
return
}
case mpInt16:
if i := int64(int16(bigen.Uint16(d.r.readx(2)))); i >= 0 {
ui = uint64(i)
} else {
d.d.errorf("Assigning negative signed value: %v, to unsigned type", i)
return
}
case mpInt32:
if i := int64(int32(bigen.Uint32(d.r.readx(4)))); i >= 0 {
ui = uint64(i)
} else {
d.d.errorf("Assigning negative signed value: %v, to unsigned type", i)
return
}
case mpInt64:
if i := int64(bigen.Uint64(d.r.readx(8))); i >= 0 {
ui = uint64(i)
} else {
d.d.errorf("Assigning negative signed value: %v, to unsigned type", i)
return
}
default:
switch {
case d.bd >= mpPosFixNumMin && d.bd <= mpPosFixNumMax:
ui = uint64(d.bd)
case d.bd >= mpNegFixNumMin && d.bd <= mpNegFixNumMax:
d.d.errorf("Assigning negative signed value: %v, to unsigned type", int(d.bd))
return
default:
d.d.errorf("Unhandled single-byte unsigned integer value: %s: %x", msgBadDesc, d.bd)
return
}
}
// check overflow (logic adapted from std pkg reflect/value.go OverflowUint()
if bitsize > 0 {
if trunc := (ui << (64 - bitsize)) >> (64 - bitsize); ui != trunc {
d.d.errorf("Overflow uint value: %v", ui)
return
}
}
d.bdRead = false
return
}
// float can either be decoded from msgpack type: float, double or intX
func (d *msgpackDecDriver) DecodeFloat(chkOverflow32 bool) (f float64) {
if !d.bdRead {
d.readNextBd()
}
if d.bd == mpFloat {
f = float64(math.Float32frombits(bigen.Uint32(d.r.readx(4))))
} else if d.bd == mpDouble {
f = math.Float64frombits(bigen.Uint64(d.r.readx(8)))
} else {
f = float64(d.DecodeInt(0))
}
if chkOverflow32 && chkOvf.Float32(f) {
d.d.errorf("msgpack: float32 overflow: %v", f)
return
}
d.bdRead = false
return
}
// bool can be decoded from bool, fixnum 0 or 1.
func (d *msgpackDecDriver) DecodeBool() (b bool) {
if !d.bdRead {
d.readNextBd()
}
if d.bd == mpFalse || d.bd == 0 {
// b = false
} else if d.bd == mpTrue || d.bd == 1 {
b = true
} else {
d.d.errorf("Invalid single-byte value for bool: %s: %x", msgBadDesc, d.bd)
return
}
d.bdRead = false
return
}
func (d *msgpackDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOut []byte) {
if !d.bdRead {
d.readNextBd()
}
var clen int
if isstring {
clen = d.readContainerLen(msgpackContainerStr)
} else {
// bytes can be decoded from msgpackContainerStr or msgpackContainerBin
if bd := d.bd; bd == mpBin8 || bd == mpBin16 || bd == mpBin32 {
clen = d.readContainerLen(msgpackContainerBin)
} else {
clen = d.readContainerLen(msgpackContainerStr)
}
}
// println("DecodeBytes: clen: ", clen)
d.bdRead = false
// bytes may be nil, so handle it. if nil, clen=-1.
if clen < 0 {
return nil
}
if zerocopy {
if d.br {
return d.r.readx(clen)
} else if len(bs) == 0 {
bs = d.b[:]
}
}
return decByteSlice(d.r, clen, bs)
}
func (d *msgpackDecDriver) DecodeString() (s string) {
return string(d.DecodeBytes(d.b[:], true, true))
}
func (d *msgpackDecDriver) readNextBd() {
d.bd = d.r.readn1()
d.bdRead = true
d.bdType = valueTypeUnset
}
func (d *msgpackDecDriver) IsContainerType(vt valueType) bool {
bd := d.bd
switch vt {
case valueTypeNil:
return bd == mpNil
case valueTypeBytes:
return bd == mpBin8 || bd == mpBin16 || bd == mpBin32 ||
(!d.h.RawToString &&
(bd == mpStr8 || bd == mpStr16 || bd == mpStr32 || (bd >= mpFixStrMin && bd <= mpFixStrMax)))
case valueTypeString:
return d.h.RawToString &&
(bd == mpStr8 || bd == mpStr16 || bd == mpStr32 || (bd >= mpFixStrMin && bd <= mpFixStrMax))
case valueTypeArray:
return bd == mpArray16 || bd == mpArray32 || (bd >= mpFixArrayMin && bd <= mpFixArrayMax)
case valueTypeMap:
return bd == mpMap16 || bd == mpMap32 || (bd >= mpFixMapMin && bd <= mpFixMapMax)
}
d.d.errorf("isContainerType: unsupported parameter: %v", vt)
return false // "unreachable"
}
func (d *msgpackDecDriver) TryDecodeAsNil() (v bool) {
if !d.bdRead {
d.readNextBd()
}
if d.bd == mpNil {
d.bdRead = false
v = true
}
return
}
func (d *msgpackDecDriver) readContainerLen(ct msgpackContainerType) (clen int) {
bd := d.bd
if bd == mpNil {
clen = -1 // to represent nil
} else if bd == ct.b8 {
clen = int(d.r.readn1())
} else if bd == ct.b16 {
clen = int(bigen.Uint16(d.r.readx(2)))
} else if bd == ct.b32 {
clen = int(bigen.Uint32(d.r.readx(4)))
} else if (ct.bFixMin & bd) == ct.bFixMin {
clen = int(ct.bFixMin ^ bd)
} else {
d.d.errorf("readContainerLen: %s: hex: %x, dec: %d", msgBadDesc, bd, bd)
return
}
d.bdRead = false
return
}
func (d *msgpackDecDriver) ReadMapStart() int {
return d.readContainerLen(msgpackContainerMap)
}
func (d *msgpackDecDriver) ReadArrayStart() int {
return d.readContainerLen(msgpackContainerList)
}
func (d *msgpackDecDriver) readExtLen() (clen int) {
switch d.bd {
case mpNil:
clen = -1 // to represent nil
case mpFixExt1:
clen = 1
case mpFixExt2:
clen = 2
case mpFixExt4:
clen = 4
case mpFixExt8:
clen = 8
case mpFixExt16:
clen = 16
case mpExt8:
clen = int(d.r.readn1())
case mpExt16:
clen = int(bigen.Uint16(d.r.readx(2)))
case mpExt32:
clen = int(bigen.Uint32(d.r.readx(4)))
default:
d.d.errorf("decoding ext bytes: found unexpected byte: %x", d.bd)
return
}
return
}
func (d *msgpackDecDriver) DecodeExt(rv interface{}, xtag uint64, ext Ext) (realxtag uint64) {
if xtag > 0xff {
d.d.errorf("decodeExt: tag must be <= 0xff; got: %v", xtag)
return
}
realxtag1, xbs := d.decodeExtV(ext != nil, uint8(xtag))
realxtag = uint64(realxtag1)
if ext == nil {
re := rv.(*RawExt)
re.Tag = realxtag
re.Data = detachZeroCopyBytes(d.br, re.Data, xbs)
} else {
ext.ReadExt(rv, xbs)
}
return
}
func (d *msgpackDecDriver) decodeExtV(verifyTag bool, tag byte) (xtag byte, xbs []byte) {
if !d.bdRead {
d.readNextBd()
}
xbd := d.bd
if xbd == mpBin8 || xbd == mpBin16 || xbd == mpBin32 {
xbs = d.DecodeBytes(nil, false, true)
} else if xbd == mpStr8 || xbd == mpStr16 || xbd == mpStr32 ||
(xbd >= mpFixStrMin && xbd <= mpFixStrMax) {
xbs = d.DecodeBytes(nil, true, true)
} else {
clen := d.readExtLen()
xtag = d.r.readn1()
if verifyTag && xtag != tag {
d.d.errorf("Wrong extension tag. Got %b. Expecting: %v", xtag, tag)
return
}
xbs = d.r.readx(clen)
}
d.bdRead = false
return
}
//--------------------------------------------------
//MsgpackHandle is a Handle for the Msgpack Schema-Free Encoding Format.
type MsgpackHandle struct {
BasicHandle
binaryEncodingType
// RawToString controls how raw bytes are decoded into a nil interface{}.
RawToString bool
// WriteExt flag supports encoding configured extensions with extension tags.
// It also controls whether other elements of the new spec are encoded (ie Str8).
//
// With WriteExt=false, configured extensions are serialized as raw bytes
// and Str8 is not encoded.
//
// A stream can still be decoded into a typed value, provided an appropriate value
// is provided, but the type cannot be inferred from the stream. If no appropriate
// type is provided (e.g. decoding into a nil interface{}), you get back
// a []byte or string based on the setting of RawToString.
WriteExt bool
}
func (h *MsgpackHandle) newEncDriver(e *Encoder) encDriver {
return &msgpackEncDriver{e: e, w: e.w, h: h}
}
func (h *MsgpackHandle) newDecDriver(d *Decoder) decDriver {
return &msgpackDecDriver{d: d, r: d.r, h: h, br: d.bytes}
}
//--------------------------------------------------
type msgpackSpecRpcCodec struct {
rpcCodec
}
// /////////////// Spec RPC Codec ///////////////////
func (c *msgpackSpecRpcCodec) WriteRequest(r *rpc.Request, body interface{}) error {
// WriteRequest can write to both a Go service, and other services that do
// not abide by the 1 argument rule of a Go service.
// We discriminate based on if the body is a MsgpackSpecRpcMultiArgs
var bodyArr []interface{}
if m, ok := body.(MsgpackSpecRpcMultiArgs); ok {
bodyArr = ([]interface{})(m)
} else {
bodyArr = []interface{}{body}
}
r2 := []interface{}{0, uint32(r.Seq), r.ServiceMethod, bodyArr}
return c.write(r2, nil, false, true)
}
func (c *msgpackSpecRpcCodec) WriteResponse(r *rpc.Response, body interface{}) error {
var moe interface{}
if r.Error != "" {
moe = r.Error
}
if moe != nil && body != nil {
body = nil
}
r2 := []interface{}{1, uint32(r.Seq), moe, body}
return c.write(r2, nil, false, true)
}
func (c *msgpackSpecRpcCodec) ReadResponseHeader(r *rpc.Response) error {
return c.parseCustomHeader(1, &r.Seq, &r.Error)
}
func (c *msgpackSpecRpcCodec) ReadRequestHeader(r *rpc.Request) error {
return c.parseCustomHeader(0, &r.Seq, &r.ServiceMethod)
}
func (c *msgpackSpecRpcCodec) ReadRequestBody(body interface{}) error {
if body == nil { // read and discard
return c.read(nil)
}
bodyArr := []interface{}{body}
return c.read(&bodyArr)
}
func (c *msgpackSpecRpcCodec) parseCustomHeader(expectTypeByte byte, msgid *uint64, methodOrError *string) (err error) {
if c.cls {
return io.EOF
}
// We read the response header by hand
// so that the body can be decoded on its own from the stream at a later time.
const fia byte = 0x94 //four item array descriptor value
// Not sure why the panic of EOF is swallowed above.
// if bs1 := c.dec.r.readn1(); bs1 != fia {
// err = fmt.Errorf("Unexpected value for array descriptor: Expecting %v. Received %v", fia, bs1)
// return
// }
var b byte
b, err = c.br.ReadByte()
if err != nil {
return
}
if b != fia {
err = fmt.Errorf("Unexpected value for array descriptor: Expecting %v. Received %v", fia, b)
return
}
if err = c.read(&b); err != nil {
return
}
if b != expectTypeByte {
err = fmt.Errorf("Unexpected byte descriptor in header. Expecting %v. Received %v", expectTypeByte, b)
return
}
if err = c.read(msgid); err != nil {
return
}
if err = c.read(methodOrError); err != nil {
return
}
return
}
//--------------------------------------------------
// msgpackSpecRpc is the implementation of Rpc that uses custom communication protocol
// as defined in the msgpack spec at https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
type msgpackSpecRpc struct{}
// MsgpackSpecRpc implements Rpc using the communication protocol defined in
// the msgpack spec at https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md .
// Its methods (ServerCodec and ClientCodec) return values that implement RpcCodecBuffered.
var MsgpackSpecRpc msgpackSpecRpc
func (x msgpackSpecRpc) ServerCodec(conn io.ReadWriteCloser, h Handle) rpc.ServerCodec {
return &msgpackSpecRpcCodec{newRPCCodec(conn, h)}
}
func (x msgpackSpecRpc) ClientCodec(conn io.ReadWriteCloser, h Handle) rpc.ClientCodec {
return &msgpackSpecRpcCodec{newRPCCodec(conn, h)}
}
var _ decDriver = (*msgpackDecDriver)(nil)
var _ encDriver = (*msgpackEncDriver)(nil)

View File

@@ -1,164 +0,0 @@
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
import (
"math/rand"
"time"
)
// NoopHandle returns a no-op handle. It basically does nothing.
// It is only useful for benchmarking, as it gives an idea of the
// overhead from the codec framework.
// LIBRARY USERS: *** DO NOT USE ***
func NoopHandle(slen int) *noopHandle {
h := noopHandle{}
h.rand = rand.New(rand.NewSource(time.Now().UnixNano()))
h.B = make([][]byte, slen)
h.S = make([]string, slen)
for i := 0; i < len(h.S); i++ {
b := make([]byte, i+1)
for j := 0; j < len(b); j++ {
b[j] = 'a' + byte(i)
}
h.B[i] = b
h.S[i] = string(b)
}
return &h
}
// noopHandle does nothing.
// It is used to simulate the overhead of the codec framework.
type noopHandle struct {
BasicHandle
binaryEncodingType
noopDrv // noopDrv is unexported here, so we can get a copy of it when needed.
}
type noopDrv struct {
i int
S []string
B [][]byte
mk bool // are we about to read a map key?
ct valueType // last request for IsContainerType.
cb bool // last response for IsContainerType.
rand *rand.Rand
}
func (h *noopDrv) r(v int) int { return h.rand.Intn(v) }
func (h *noopDrv) m(v int) int { h.i++; return h.i % v }
func (h *noopDrv) newEncDriver(_ *Encoder) encDriver { return h }
func (h *noopDrv) newDecDriver(_ *Decoder) decDriver { return h }
// --- encDriver
func (h *noopDrv) EncodeBuiltin(rt uintptr, v interface{}) {}
func (h *noopDrv) EncodeNil() {}
func (h *noopDrv) EncodeInt(i int64) {}
func (h *noopDrv) EncodeUint(i uint64) {}
func (h *noopDrv) EncodeBool(b bool) {}
func (h *noopDrv) EncodeFloat32(f float32) {}
func (h *noopDrv) EncodeFloat64(f float64) {}
func (h *noopDrv) EncodeRawExt(re *RawExt, e *Encoder) {}
func (h *noopDrv) EncodeArrayStart(length int) {}
func (h *noopDrv) EncodeArrayEnd() {}
func (h *noopDrv) EncodeArrayEntrySeparator() {}
func (h *noopDrv) EncodeMapStart(length int) {}
func (h *noopDrv) EncodeMapEnd() {}
func (h *noopDrv) EncodeMapEntrySeparator() {}
func (h *noopDrv) EncodeMapKVSeparator() {}
func (h *noopDrv) EncodeString(c charEncoding, v string) {}
func (h *noopDrv) EncodeSymbol(v string) {}
func (h *noopDrv) EncodeStringBytes(c charEncoding, v []byte) {}
func (h *noopDrv) EncodeExt(rv interface{}, xtag uint64, ext Ext, e *Encoder) {}
// ---- decDriver
func (h *noopDrv) initReadNext() {}
func (h *noopDrv) CheckBreak() bool { return false }
func (h *noopDrv) IsBuiltinType(rt uintptr) bool { return false }
func (h *noopDrv) DecodeBuiltin(rt uintptr, v interface{}) {}
func (h *noopDrv) DecodeInt(bitsize uint8) (i int64) { return int64(h.m(15)) }
func (h *noopDrv) DecodeUint(bitsize uint8) (ui uint64) { return uint64(h.m(35)) }
func (h *noopDrv) DecodeFloat(chkOverflow32 bool) (f float64) { return float64(h.m(95)) }
func (h *noopDrv) DecodeBool() (b bool) { return h.m(2) == 0 }
func (h *noopDrv) DecodeString() (s string) { return h.S[h.m(8)] }
// func (h *noopDrv) DecodeStringAsBytes(bs []byte) []byte { return h.DecodeBytes(bs) }
func (h *noopDrv) DecodeBytes(bs []byte, isstring, zerocopy bool) []byte { return h.B[h.m(len(h.B))] }
func (h *noopDrv) ReadMapEnd() { h.mk = false }
func (h *noopDrv) ReadArrayEnd() {}
func (h *noopDrv) ReadArrayEntrySeparator() {}
func (h *noopDrv) ReadMapEntrySeparator() { h.mk = true }
func (h *noopDrv) ReadMapKVSeparator() { h.mk = false }
// toggle map/slice
func (h *noopDrv) ReadMapStart() int { h.mk = true; return h.m(10) }
func (h *noopDrv) ReadArrayStart() int { return h.m(10) }
func (h *noopDrv) IsContainerType(vt valueType) bool {
// return h.m(2) == 0
// handle kStruct
if h.ct == valueTypeMap && vt == valueTypeArray || h.ct == valueTypeArray && vt == valueTypeMap {
h.cb = !h.cb
h.ct = vt
return h.cb
}
// go in a loop and check it.
h.ct = vt
h.cb = h.m(7) == 0
return h.cb
}
func (h *noopDrv) TryDecodeAsNil() bool {
if h.mk {
return false
} else {
return h.m(8) == 0
}
}
func (h *noopDrv) DecodeExt(rv interface{}, xtag uint64, ext Ext) uint64 {
return 0
}
func (h *noopDrv) DecodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
// use h.r (random) not h.m() because h.m() could cause the same value to be given.
var sk int
if h.mk {
// if mapkey, do not support values of nil OR bytes, array, map or rawext
sk = h.r(7) + 1
} else {
sk = h.r(12)
}
switch sk {
case 0:
vt = valueTypeNil
case 1:
vt, v = valueTypeBool, false
case 2:
vt, v = valueTypeBool, true
case 3:
vt, v = valueTypeInt, h.DecodeInt(64)
case 4:
vt, v = valueTypeUint, h.DecodeUint(64)
case 5:
vt, v = valueTypeFloat, h.DecodeFloat(true)
case 6:
vt, v = valueTypeFloat, h.DecodeFloat(false)
case 7:
vt, v = valueTypeString, h.DecodeString()
case 8:
vt, v = valueTypeBytes, h.B[h.m(len(h.B))]
case 9:
vt, decodeFurther = valueTypeArray, true
case 10:
vt, decodeFurther = valueTypeMap, true
default:
vt, v = valueTypeExt, &RawExt{Tag: h.DecodeUint(64), Data: h.B[h.m(len(h.B))]}
}
h.ct = vt
return
}

View File

@@ -1,3 +0,0 @@
package codec
//go:generate bash prebuild.sh

View File

@@ -1,191 +0,0 @@
#!/bin/bash
# _needgen is a helper function to tell if we need to generate files for msgp, codecgen.
_needgen() {
local a="$1"
zneedgen=0
if [[ ! -e "$a" ]]
then
zneedgen=1
echo 1
return 0
fi
for i in `ls -1 *.go.tmpl gen.go values_test.go`
do
if [[ "$a" -ot "$i" ]]
then
zneedgen=1
echo 1
return 0
fi
done
echo 0
}
# _build generates fast-path.go and gen-helper.go.
#
# It is needed because there is some dependency between the generated code
# and the other classes. Consequently, we have to totally remove the
# generated files and put stubs in place, before calling "go run" again
# to recreate them.
_build() {
if ! [[ "${zforce}" == "1" ||
"1" == $( _needgen "fast-path.generated.go" ) ||
"1" == $( _needgen "gen-helper.generated.go" ) ||
"1" == $( _needgen "gen.generated.go" ) ||
1 == 0 ]]
then
return 0
fi
# echo "Running prebuild"
if [ "${zbak}" == "1" ]
then
# echo "Backing up old generated files"
_zts=`date '+%m%d%Y_%H%M%S'`
_gg=".generated.go"
[ -e "gen-helper${_gg}" ] && mv gen-helper${_gg} gen-helper${_gg}__${_zts}.bak
[ -e "fast-path${_gg}" ] && mv fast-path${_gg} fast-path${_gg}__${_zts}.bak
# [ -e "safe${_gg}" ] && mv safe${_gg} safe${_gg}__${_zts}.bak
# [ -e "unsafe${_gg}" ] && mv unsafe${_gg} unsafe${_gg}__${_zts}.bak
else
rm -f fast-path.generated.go gen.generated.go gen-helper.generated.go *safe.generated.go *_generated_test.go *.generated_ffjson_expose.go
fi
cat > gen.generated.go <<EOF
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
// DO NOT EDIT. THIS FILE IS AUTO-GENERATED FROM gen-dec-(map|array).go.tmpl
const genDecMapTmpl = \`
EOF
cat >> gen.generated.go < gen-dec-map.go.tmpl
cat >> gen.generated.go <<EOF
\`
const genDecListTmpl = \`
EOF
cat >> gen.generated.go < gen-dec-array.go.tmpl
cat >> gen.generated.go <<EOF
\`
EOF
# All functions, variables which must exist are put in this file.
# This way, build works before we generate the right things.
cat > fast-path.generated.go <<EOF
package codec
import "reflect"
// func GenBytesToStringRO(b []byte) string { return string(b) }
func fastpathDecodeTypeSwitch(iv interface{}, d *Decoder) bool { return false }
func fastpathEncodeTypeSwitch(iv interface{}, e *Encoder) bool { return false }
func fastpathEncodeTypeSwitchSlice(iv interface{}, e *Encoder) bool { return false }
func fastpathEncodeTypeSwitchMap(iv interface{}, e *Encoder) bool { return false }
type fastpathE struct {
rtid uintptr
rt reflect.Type
encfn func(encFnInfo, reflect.Value)
decfn func(decFnInfo, reflect.Value)
}
type fastpathA [0]fastpathE
func (x fastpathA) index(rtid uintptr) int { return -1 }
var fastpathAV fastpathA
EOF
cat > gen-from-tmpl.generated.go <<EOF
//+build ignore
package main
//import "flag"
import "ugorji.net/codec"
import "os"
func run(fnameIn, fnameOut string, safe bool) {
fin, err := os.Open(fnameIn)
if err != nil { panic(err) }
defer fin.Close()
fout, err := os.Create(fnameOut)
if err != nil { panic(err) }
defer fout.Close()
err = codec.GenInternalGoFile(fin, fout, safe)
if err != nil { panic(err) }
}
func main() {
// do not make safe/unsafe variants.
// Instead, depend on escape analysis, and place string creation and usage appropriately.
// run("unsafe.go.tmpl", "safe.generated.go", true)
// run("unsafe.go.tmpl", "unsafe.generated.go", false)
run("fast-path.go.tmpl", "fast-path.generated.go", false)
run("gen-helper.go.tmpl", "gen-helper.generated.go", false)
}
EOF
go run gen-from-tmpl.generated.go && \
rm -f gen-from-tmpl.generated.go
}
_codegenerators() {
if [[ $zforce == "1" ||
"1" == $( _needgen "values_msgp${zsfx}" )
|| "1" == $( _needgen "values_codecgen${zsfx}" ) ]]
then
true && \
echo "msgp ... " && \
msgp -tests=false -pkg=codec -o=values_msgp${zsfx} -file=$zfin && \
echo "codecgen - !unsafe ... " && \
codecgen -rt codecgen -t 'x,codecgen,!unsafe' -o values_codecgen${zsfx} $zfin && \
echo "codecgen - unsafe ... " && \
codecgen -u -rt codecgen -t 'x,codecgen,unsafe' -o values_codecgen_unsafe${zsfx} $zfin && \
echo "ffjson ... " && \
ffjson -w values_ffjson${zsfx} $zfin && \
# remove (M|Unm)arshalJSON implementations, so they don't conflict with encoding/json bench \
sed -i 's+ MarshalJSON(+ _MarshalJSON(+g' values_ffjson${zsfx} && \
sed -i 's+ UnmarshalJSON(+ _UnmarshalJSON(+g' values_ffjson${zsfx} && \
echo "generators done!" && \
true
fi
}
# _init reads the arguments and sets up the flags
_init() {
OPTIND=1
while getopts "fb" flag
do
case "x$flag" in
'xf') zforce=1;;
'xb') zbak=1;;
*) echo "prebuild.sh accepts [-fb] only"; return 1;;
esac
done
shift $((OPTIND-1))
OPTIND=1
}
# main script.
# First ensure that this is being run from the basedir (i.e. dirname of script is .)
if [ "." = `dirname $0` ]
then
zmydir=`pwd`
zfin="test_values.generated.go"
zsfx="_generated_test.go"
# rm -f *_generated_test.go
rm -f codecgen-*.go && \
_init "$@" && \
_build && \
cp $zmydir/values_test.go $zmydir/$zfin && \
_codegenerators && \
echo prebuild done successfully
rm -f $zmydir/$zfin
else
echo "Script must be run from the directory it resides in"
fi

View File

@@ -1,29 +0,0 @@
//+build x
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
// These tests are used to verify msgpack and cbor implementations against their python libraries.
// If you have the library installed, you can enable the tests back by removing the //+build ignore.
import (
"testing"
)
func TestMsgpackPythonGenStreams(t *testing.T) {
doTestPythonGenStreams(t, "msgpack", testMsgpackH)
}
func TestCborPythonGenStreams(t *testing.T) {
doTestPythonGenStreams(t, "cbor", testCborH)
}
func TestMsgpackRpcSpecGoClientToPythonSvc(t *testing.T) {
doTestMsgpackRpcSpecGoClientToPythonSvc(t)
}
func TestMsgpackRpcSpecPythonClientToGoSvc(t *testing.T) {
doTestMsgpackRpcSpecPythonClientToGoSvc(t)
}

View File

@@ -1,169 +0,0 @@
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
import (
"bufio"
"io"
"net/rpc"
"sync"
)
// rpcEncodeTerminator allows a handler specify a []byte terminator to send after each Encode.
//
// Some codecs like json need to put a space after each encoded value, to serve as a
// delimiter for things like numbers (else json codec will continue reading till EOF).
type rpcEncodeTerminator interface {
rpcEncodeTerminate() []byte
}
// Rpc provides a rpc Server or Client Codec for rpc communication.
type Rpc interface {
ServerCodec(conn io.ReadWriteCloser, h Handle) rpc.ServerCodec
ClientCodec(conn io.ReadWriteCloser, h Handle) rpc.ClientCodec
}
// RpcCodecBuffered allows access to the underlying bufio.Reader/Writer
// used by the rpc connection. It accomodates use-cases where the connection
// should be used by rpc and non-rpc functions, e.g. streaming a file after
// sending an rpc response.
type RpcCodecBuffered interface {
BufferedReader() *bufio.Reader
BufferedWriter() *bufio.Writer
}
// -------------------------------------
// rpcCodec defines the struct members and common methods.
type rpcCodec struct {
rwc io.ReadWriteCloser
dec *Decoder
enc *Encoder
bw *bufio.Writer
br *bufio.Reader
mu sync.Mutex
cls bool
h Handle
}
func newRPCCodec(conn io.ReadWriteCloser, h Handle) rpcCodec {
bw := bufio.NewWriter(conn)
br := bufio.NewReader(conn)
return rpcCodec{
rwc: conn,
bw: bw,
br: br,
enc: NewEncoder(bw, h),
dec: NewDecoder(br, h),
h: h,
}
}
func (c *rpcCodec) BufferedReader() *bufio.Reader {
return c.br
}
func (c *rpcCodec) BufferedWriter() *bufio.Writer {
return c.bw
}
func (c *rpcCodec) write(obj1, obj2 interface{}, writeObj2, doFlush bool) (err error) {
if c.cls {
return io.EOF
}
if err = c.enc.Encode(obj1); err != nil {
return
}
t, tOk := c.h.(rpcEncodeTerminator)
if tOk {
c.bw.Write(t.rpcEncodeTerminate())
}
if writeObj2 {
if err = c.enc.Encode(obj2); err != nil {
return
}
if tOk {
c.bw.Write(t.rpcEncodeTerminate())
}
}
if doFlush {
return c.bw.Flush()
}
return
}
func (c *rpcCodec) read(obj interface{}) (err error) {
if c.cls {
return io.EOF
}
//If nil is passed in, we should still attempt to read content to nowhere.
if obj == nil {
var obj2 interface{}
return c.dec.Decode(&obj2)
}
return c.dec.Decode(obj)
}
func (c *rpcCodec) Close() error {
if c.cls {
return io.EOF
}
c.cls = true
return c.rwc.Close()
}
func (c *rpcCodec) ReadResponseBody(body interface{}) error {
return c.read(body)
}
// -------------------------------------
type goRpcCodec struct {
rpcCodec
}
func (c *goRpcCodec) WriteRequest(r *rpc.Request, body interface{}) error {
// Must protect for concurrent access as per API
c.mu.Lock()
defer c.mu.Unlock()
return c.write(r, body, true, true)
}
func (c *goRpcCodec) WriteResponse(r *rpc.Response, body interface{}) error {
c.mu.Lock()
defer c.mu.Unlock()
return c.write(r, body, true, true)
}
func (c *goRpcCodec) ReadResponseHeader(r *rpc.Response) error {
return c.read(r)
}
func (c *goRpcCodec) ReadRequestHeader(r *rpc.Request) error {
return c.read(r)
}
func (c *goRpcCodec) ReadRequestBody(body interface{}) error {
return c.read(body)
}
// -------------------------------------
// goRpc is the implementation of Rpc that uses the communication protocol
// as defined in net/rpc package.
type goRpc struct{}
// GoRpc implements Rpc using the communication protocol defined in net/rpc package.
// Its methods (ServerCodec and ClientCodec) return values that implement RpcCodecBuffered.
var GoRpc goRpc
func (x goRpc) ServerCodec(conn io.ReadWriteCloser, h Handle) rpc.ServerCodec {
return &goRpcCodec{newRPCCodec(conn, h)}
}
func (x goRpc) ClientCodec(conn io.ReadWriteCloser, h Handle) rpc.ClientCodec {
return &goRpcCodec{newRPCCodec(conn, h)}
}
var _ RpcCodecBuffered = (*rpcCodec)(nil) // ensure *rpcCodec implements RpcCodecBuffered

View File

@@ -1,505 +0,0 @@
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
import "math"
const (
_ uint8 = iota
simpleVdNil = 1
simpleVdFalse = 2
simpleVdTrue = 3
simpleVdFloat32 = 4
simpleVdFloat64 = 5
// each lasts for 4 (ie n, n+1, n+2, n+3)
simpleVdPosInt = 8
simpleVdNegInt = 12
// containers: each lasts for 4 (ie n, n+1, n+2, ... n+7)
simpleVdString = 216
simpleVdByteArray = 224
simpleVdArray = 232
simpleVdMap = 240
simpleVdExt = 248
)
type simpleEncDriver struct {
e *Encoder
h *SimpleHandle
w encWriter
noBuiltInTypes
b [8]byte
encNoSeparator
}
func (e *simpleEncDriver) EncodeNil() {
e.w.writen1(simpleVdNil)
}
func (e *simpleEncDriver) EncodeBool(b bool) {
if b {
e.w.writen1(simpleVdTrue)
} else {
e.w.writen1(simpleVdFalse)
}
}
func (e *simpleEncDriver) EncodeFloat32(f float32) {
e.w.writen1(simpleVdFloat32)
bigenHelper{e.b[:4], e.w}.writeUint32(math.Float32bits(f))
}
func (e *simpleEncDriver) EncodeFloat64(f float64) {
e.w.writen1(simpleVdFloat64)
bigenHelper{e.b[:8], e.w}.writeUint64(math.Float64bits(f))
}
func (e *simpleEncDriver) EncodeInt(v int64) {
if v < 0 {
e.encUint(uint64(-v), simpleVdNegInt)
} else {
e.encUint(uint64(v), simpleVdPosInt)
}
}
func (e *simpleEncDriver) EncodeUint(v uint64) {
e.encUint(v, simpleVdPosInt)
}
func (e *simpleEncDriver) encUint(v uint64, bd uint8) {
if v <= math.MaxUint8 {
e.w.writen2(bd, uint8(v))
} else if v <= math.MaxUint16 {
e.w.writen1(bd + 1)
bigenHelper{e.b[:2], e.w}.writeUint16(uint16(v))
} else if v <= math.MaxUint32 {
e.w.writen1(bd + 2)
bigenHelper{e.b[:4], e.w}.writeUint32(uint32(v))
} else if v <= math.MaxUint64 {
e.w.writen1(bd + 3)
bigenHelper{e.b[:8], e.w}.writeUint64(v)
}
}
func (e *simpleEncDriver) encLen(bd byte, length int) {
if length == 0 {
e.w.writen1(bd)
} else if length <= math.MaxUint8 {
e.w.writen1(bd + 1)
e.w.writen1(uint8(length))
} else if length <= math.MaxUint16 {
e.w.writen1(bd + 2)
bigenHelper{e.b[:2], e.w}.writeUint16(uint16(length))
} else if int64(length) <= math.MaxUint32 {
e.w.writen1(bd + 3)
bigenHelper{e.b[:4], e.w}.writeUint32(uint32(length))
} else {
e.w.writen1(bd + 4)
bigenHelper{e.b[:8], e.w}.writeUint64(uint64(length))
}
}
func (e *simpleEncDriver) EncodeExt(rv interface{}, xtag uint64, ext Ext, _ *Encoder) {
bs := ext.WriteExt(rv)
if bs == nil {
e.EncodeNil()
return
}
e.encodeExtPreamble(uint8(xtag), len(bs))
e.w.writeb(bs)
}
func (e *simpleEncDriver) EncodeRawExt(re *RawExt, _ *Encoder) {
e.encodeExtPreamble(uint8(re.Tag), len(re.Data))
e.w.writeb(re.Data)
}
func (e *simpleEncDriver) encodeExtPreamble(xtag byte, length int) {
e.encLen(simpleVdExt, length)
e.w.writen1(xtag)
}
func (e *simpleEncDriver) EncodeArrayStart(length int) {
e.encLen(simpleVdArray, length)
}
func (e *simpleEncDriver) EncodeMapStart(length int) {
e.encLen(simpleVdMap, length)
}
func (e *simpleEncDriver) EncodeString(c charEncoding, v string) {
e.encLen(simpleVdString, len(v))
e.w.writestr(v)
}
func (e *simpleEncDriver) EncodeSymbol(v string) {
e.EncodeString(c_UTF8, v)
}
func (e *simpleEncDriver) EncodeStringBytes(c charEncoding, v []byte) {
e.encLen(simpleVdByteArray, len(v))
e.w.writeb(v)
}
//------------------------------------
type simpleDecDriver struct {
d *Decoder
h *SimpleHandle
r decReader
bdRead bool
bdType valueType
bd byte
br bool // bytes reader
noBuiltInTypes
noStreamingCodec
decNoSeparator
b [scratchByteArrayLen]byte
}
func (d *simpleDecDriver) readNextBd() {
d.bd = d.r.readn1()
d.bdRead = true
d.bdType = valueTypeUnset
}
func (d *simpleDecDriver) IsContainerType(vt valueType) bool {
switch vt {
case valueTypeNil:
return d.bd == simpleVdNil
case valueTypeBytes:
const x uint8 = simpleVdByteArray
return d.bd == x || d.bd == x+1 || d.bd == x+2 || d.bd == x+3 || d.bd == x+4
case valueTypeString:
const x uint8 = simpleVdString
return d.bd == x || d.bd == x+1 || d.bd == x+2 || d.bd == x+3 || d.bd == x+4
case valueTypeArray:
const x uint8 = simpleVdArray
return d.bd == x || d.bd == x+1 || d.bd == x+2 || d.bd == x+3 || d.bd == x+4
case valueTypeMap:
const x uint8 = simpleVdMap
return d.bd == x || d.bd == x+1 || d.bd == x+2 || d.bd == x+3 || d.bd == x+4
}
d.d.errorf("isContainerType: unsupported parameter: %v", vt)
return false // "unreachable"
}
func (d *simpleDecDriver) TryDecodeAsNil() bool {
if !d.bdRead {
d.readNextBd()
}
if d.bd == simpleVdNil {
d.bdRead = false
return true
}
return false
}
func (d *simpleDecDriver) decCheckInteger() (ui uint64, neg bool) {
if !d.bdRead {
d.readNextBd()
}
switch d.bd {
case simpleVdPosInt:
ui = uint64(d.r.readn1())
case simpleVdPosInt + 1:
ui = uint64(bigen.Uint16(d.r.readx(2)))
case simpleVdPosInt + 2:
ui = uint64(bigen.Uint32(d.r.readx(4)))
case simpleVdPosInt + 3:
ui = uint64(bigen.Uint64(d.r.readx(8)))
case simpleVdNegInt:
ui = uint64(d.r.readn1())
neg = true
case simpleVdNegInt + 1:
ui = uint64(bigen.Uint16(d.r.readx(2)))
neg = true
case simpleVdNegInt + 2:
ui = uint64(bigen.Uint32(d.r.readx(4)))
neg = true
case simpleVdNegInt + 3:
ui = uint64(bigen.Uint64(d.r.readx(8)))
neg = true
default:
d.d.errorf("decIntAny: Integer only valid from pos/neg integer1..8. Invalid descriptor: %v", d.bd)
return
}
// don't do this check, because callers may only want the unsigned value.
// if ui > math.MaxInt64 {
// d.d.errorf("decIntAny: Integer out of range for signed int64: %v", ui)
// return
// }
return
}
func (d *simpleDecDriver) DecodeInt(bitsize uint8) (i int64) {
ui, neg := d.decCheckInteger()
i, overflow := chkOvf.SignedInt(ui)
if overflow {
d.d.errorf("simple: overflow converting %v to signed integer", ui)
return
}
if neg {
i = -i
}
if chkOvf.Int(i, bitsize) {
d.d.errorf("simple: overflow integer: %v", i)
return
}
d.bdRead = false
return
}
func (d *simpleDecDriver) DecodeUint(bitsize uint8) (ui uint64) {
ui, neg := d.decCheckInteger()
if neg {
d.d.errorf("Assigning negative signed value to unsigned type")
return
}
if chkOvf.Uint(ui, bitsize) {
d.d.errorf("simple: overflow integer: %v", ui)
return
}
d.bdRead = false
return
}
func (d *simpleDecDriver) DecodeFloat(chkOverflow32 bool) (f float64) {
if !d.bdRead {
d.readNextBd()
}
if d.bd == simpleVdFloat32 {
f = float64(math.Float32frombits(bigen.Uint32(d.r.readx(4))))
} else if d.bd == simpleVdFloat64 {
f = math.Float64frombits(bigen.Uint64(d.r.readx(8)))
} else {
if d.bd >= simpleVdPosInt && d.bd <= simpleVdNegInt+3 {
f = float64(d.DecodeInt(64))
} else {
d.d.errorf("Float only valid from float32/64: Invalid descriptor: %v", d.bd)
return
}
}
if chkOverflow32 && chkOvf.Float32(f) {
d.d.errorf("msgpack: float32 overflow: %v", f)
return
}
d.bdRead = false
return
}
// bool can be decoded from bool only (single byte).
func (d *simpleDecDriver) DecodeBool() (b bool) {
if !d.bdRead {
d.readNextBd()
}
if d.bd == simpleVdTrue {
b = true
} else if d.bd == simpleVdFalse {
} else {
d.d.errorf("Invalid single-byte value for bool: %s: %x", msgBadDesc, d.bd)
return
}
d.bdRead = false
return
}
func (d *simpleDecDriver) ReadMapStart() (length int) {
d.bdRead = false
return d.decLen()
}
func (d *simpleDecDriver) ReadArrayStart() (length int) {
d.bdRead = false
return d.decLen()
}
func (d *simpleDecDriver) decLen() int {
switch d.bd % 8 {
case 0:
return 0
case 1:
return int(d.r.readn1())
case 2:
return int(bigen.Uint16(d.r.readx(2)))
case 3:
ui := uint64(bigen.Uint32(d.r.readx(4)))
if chkOvf.Uint(ui, intBitsize) {
d.d.errorf("simple: overflow integer: %v", ui)
return 0
}
return int(ui)
case 4:
ui := bigen.Uint64(d.r.readx(8))
if chkOvf.Uint(ui, intBitsize) {
d.d.errorf("simple: overflow integer: %v", ui)
return 0
}
return int(ui)
}
d.d.errorf("decLen: Cannot read length: bd%8 must be in range 0..4. Got: %d", d.bd%8)
return -1
}
func (d *simpleDecDriver) DecodeString() (s string) {
return string(d.DecodeBytes(d.b[:], true, true))
}
func (d *simpleDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOut []byte) {
if !d.bdRead {
d.readNextBd()
}
if d.bd == simpleVdNil {
d.bdRead = false
return
}
clen := d.decLen()
d.bdRead = false
if zerocopy {
if d.br {
return d.r.readx(clen)
} else if len(bs) == 0 {
bs = d.b[:]
}
}
return decByteSlice(d.r, clen, bs)
}
func (d *simpleDecDriver) DecodeExt(rv interface{}, xtag uint64, ext Ext) (realxtag uint64) {
if xtag > 0xff {
d.d.errorf("decodeExt: tag must be <= 0xff; got: %v", xtag)
return
}
realxtag1, xbs := d.decodeExtV(ext != nil, uint8(xtag))
realxtag = uint64(realxtag1)
if ext == nil {
re := rv.(*RawExt)
re.Tag = realxtag
re.Data = detachZeroCopyBytes(d.br, re.Data, xbs)
} else {
ext.ReadExt(rv, xbs)
}
return
}
func (d *simpleDecDriver) decodeExtV(verifyTag bool, tag byte) (xtag byte, xbs []byte) {
if !d.bdRead {
d.readNextBd()
}
switch d.bd {
case simpleVdExt, simpleVdExt + 1, simpleVdExt + 2, simpleVdExt + 3, simpleVdExt + 4:
l := d.decLen()
xtag = d.r.readn1()
if verifyTag && xtag != tag {
d.d.errorf("Wrong extension tag. Got %b. Expecting: %v", xtag, tag)
return
}
xbs = d.r.readx(l)
case simpleVdByteArray, simpleVdByteArray + 1, simpleVdByteArray + 2, simpleVdByteArray + 3, simpleVdByteArray + 4:
xbs = d.DecodeBytes(nil, false, true)
default:
d.d.errorf("Invalid d.bd for extensions (Expecting extensions or byte array). Got: 0x%x", d.bd)
return
}
d.bdRead = false
return
}
func (d *simpleDecDriver) DecodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
if !d.bdRead {
d.readNextBd()
}
switch d.bd {
case simpleVdNil:
vt = valueTypeNil
case simpleVdFalse:
vt = valueTypeBool
v = false
case simpleVdTrue:
vt = valueTypeBool
v = true
case simpleVdPosInt, simpleVdPosInt + 1, simpleVdPosInt + 2, simpleVdPosInt + 3:
if d.h.SignedInteger {
vt = valueTypeInt
v = d.DecodeInt(64)
} else {
vt = valueTypeUint
v = d.DecodeUint(64)
}
case simpleVdNegInt, simpleVdNegInt + 1, simpleVdNegInt + 2, simpleVdNegInt + 3:
vt = valueTypeInt
v = d.DecodeInt(64)
case simpleVdFloat32:
vt = valueTypeFloat
v = d.DecodeFloat(true)
case simpleVdFloat64:
vt = valueTypeFloat
v = d.DecodeFloat(false)
case simpleVdString, simpleVdString + 1, simpleVdString + 2, simpleVdString + 3, simpleVdString + 4:
vt = valueTypeString
v = d.DecodeString()
case simpleVdByteArray, simpleVdByteArray + 1, simpleVdByteArray + 2, simpleVdByteArray + 3, simpleVdByteArray + 4:
vt = valueTypeBytes
v = d.DecodeBytes(nil, false, false)
case simpleVdExt, simpleVdExt + 1, simpleVdExt + 2, simpleVdExt + 3, simpleVdExt + 4:
vt = valueTypeExt
l := d.decLen()
var re RawExt
re.Tag = uint64(d.r.readn1())
re.Data = d.r.readx(l)
v = &re
case simpleVdArray, simpleVdArray + 1, simpleVdArray + 2, simpleVdArray + 3, simpleVdArray + 4:
vt = valueTypeArray
decodeFurther = true
case simpleVdMap, simpleVdMap + 1, simpleVdMap + 2, simpleVdMap + 3, simpleVdMap + 4:
vt = valueTypeMap
decodeFurther = true
default:
d.d.errorf("decodeNaked: Unrecognized d.bd: 0x%x", d.bd)
return
}
if !decodeFurther {
d.bdRead = false
}
return
}
//------------------------------------
// SimpleHandle is a Handle for a very simple encoding format.
//
// simple is a simplistic codec similar to binc, but not as compact.
// - Encoding of a value is always preceeded by the descriptor byte (bd)
// - True, false, nil are encoded fully in 1 byte (the descriptor)
// - Integers (intXXX, uintXXX) are encoded in 1, 2, 4 or 8 bytes (plus a descriptor byte).
// There are positive (uintXXX and intXXX >= 0) and negative (intXXX < 0) integers.
// - Floats are encoded in 4 or 8 bytes (plus a descriptor byte)
// - Lenght of containers (strings, bytes, array, map, extensions)
// are encoded in 0, 1, 2, 4 or 8 bytes.
// Zero-length containers have no length encoded.
// For others, the number of bytes is given by pow(2, bd%3)
// - maps are encoded as [bd] [length] [[key][value]]...
// - arrays are encoded as [bd] [length] [value]...
// - extensions are encoded as [bd] [length] [tag] [byte]...
// - strings/bytearrays are encoded as [bd] [length] [byte]...
//
// The full spec will be published soon.
type SimpleHandle struct {
BasicHandle
binaryEncodingType
}
func (h *SimpleHandle) newEncDriver(e *Encoder) encDriver {
return &simpleEncDriver{e: e, w: e.w, h: h}
}
func (h *SimpleHandle) newDecDriver(d *Decoder) decDriver {
return &simpleDecDriver{d: d, r: d.r, h: h, br: d.bytes}
}
var _ decDriver = (*simpleDecDriver)(nil)
var _ encDriver = (*simpleEncDriver)(nil)

View File

@@ -1,639 +0,0 @@
[
{
"cbor": "AA==",
"hex": "00",
"roundtrip": true,
"decoded": 0
},
{
"cbor": "AQ==",
"hex": "01",
"roundtrip": true,
"decoded": 1
},
{
"cbor": "Cg==",
"hex": "0a",
"roundtrip": true,
"decoded": 10
},
{
"cbor": "Fw==",
"hex": "17",
"roundtrip": true,
"decoded": 23
},
{
"cbor": "GBg=",
"hex": "1818",
"roundtrip": true,
"decoded": 24
},
{
"cbor": "GBk=",
"hex": "1819",
"roundtrip": true,
"decoded": 25
},
{
"cbor": "GGQ=",
"hex": "1864",
"roundtrip": true,
"decoded": 100
},
{
"cbor": "GQPo",
"hex": "1903e8",
"roundtrip": true,
"decoded": 1000
},
{
"cbor": "GgAPQkA=",
"hex": "1a000f4240",
"roundtrip": true,
"decoded": 1000000
},
{
"cbor": "GwAAAOjUpRAA",
"hex": "1b000000e8d4a51000",
"roundtrip": true,
"decoded": 1000000000000
},
{
"cbor": "G///////////",
"hex": "1bffffffffffffffff",
"roundtrip": true,
"decoded": 18446744073709551615
},
{
"cbor": "wkkBAAAAAAAAAAA=",
"hex": "c249010000000000000000",
"roundtrip": true,
"decoded": 18446744073709551616
},
{
"cbor": "O///////////",
"hex": "3bffffffffffffffff",
"roundtrip": true,
"decoded": -18446744073709551616,
"skip": true
},
{
"cbor": "w0kBAAAAAAAAAAA=",
"hex": "c349010000000000000000",
"roundtrip": true,
"decoded": -18446744073709551617
},
{
"cbor": "IA==",
"hex": "20",
"roundtrip": true,
"decoded": -1
},
{
"cbor": "KQ==",
"hex": "29",
"roundtrip": true,
"decoded": -10
},
{
"cbor": "OGM=",
"hex": "3863",
"roundtrip": true,
"decoded": -100
},
{
"cbor": "OQPn",
"hex": "3903e7",
"roundtrip": true,
"decoded": -1000
},
{
"cbor": "+QAA",
"hex": "f90000",
"roundtrip": true,
"decoded": 0.0
},
{
"cbor": "+YAA",
"hex": "f98000",
"roundtrip": true,
"decoded": -0.0
},
{
"cbor": "+TwA",
"hex": "f93c00",
"roundtrip": true,
"decoded": 1.0
},
{
"cbor": "+z/xmZmZmZma",
"hex": "fb3ff199999999999a",
"roundtrip": true,
"decoded": 1.1
},
{
"cbor": "+T4A",
"hex": "f93e00",
"roundtrip": true,
"decoded": 1.5
},
{
"cbor": "+Xv/",
"hex": "f97bff",
"roundtrip": true,
"decoded": 65504.0
},
{
"cbor": "+kfDUAA=",
"hex": "fa47c35000",
"roundtrip": true,
"decoded": 100000.0
},
{
"cbor": "+n9///8=",
"hex": "fa7f7fffff",
"roundtrip": true,
"decoded": 3.4028234663852886e+38
},
{
"cbor": "+3435DyIAHWc",
"hex": "fb7e37e43c8800759c",
"roundtrip": true,
"decoded": 1.0e+300
},
{
"cbor": "+QAB",
"hex": "f90001",
"roundtrip": true,
"decoded": 5.960464477539063e-08
},
{
"cbor": "+QQA",
"hex": "f90400",
"roundtrip": true,
"decoded": 6.103515625e-05
},
{
"cbor": "+cQA",
"hex": "f9c400",
"roundtrip": true,
"decoded": -4.0
},
{
"cbor": "+8AQZmZmZmZm",
"hex": "fbc010666666666666",
"roundtrip": true,
"decoded": -4.1
},
{
"cbor": "+XwA",
"hex": "f97c00",
"roundtrip": true,
"diagnostic": "Infinity"
},
{
"cbor": "+X4A",
"hex": "f97e00",
"roundtrip": true,
"diagnostic": "NaN"
},
{
"cbor": "+fwA",
"hex": "f9fc00",
"roundtrip": true,
"diagnostic": "-Infinity"
},
{
"cbor": "+n+AAAA=",
"hex": "fa7f800000",
"roundtrip": false,
"diagnostic": "Infinity"
},
{
"cbor": "+n/AAAA=",
"hex": "fa7fc00000",
"roundtrip": false,
"diagnostic": "NaN"
},
{
"cbor": "+v+AAAA=",
"hex": "faff800000",
"roundtrip": false,
"diagnostic": "-Infinity"
},
{
"cbor": "+3/wAAAAAAAA",
"hex": "fb7ff0000000000000",
"roundtrip": false,
"diagnostic": "Infinity"
},
{
"cbor": "+3/4AAAAAAAA",
"hex": "fb7ff8000000000000",
"roundtrip": false,
"diagnostic": "NaN"
},
{
"cbor": "+//wAAAAAAAA",
"hex": "fbfff0000000000000",
"roundtrip": false,
"diagnostic": "-Infinity"
},
{
"cbor": "9A==",
"hex": "f4",
"roundtrip": true,
"decoded": false
},
{
"cbor": "9Q==",
"hex": "f5",
"roundtrip": true,
"decoded": true
},
{
"cbor": "9g==",
"hex": "f6",
"roundtrip": true,
"decoded": null
},
{
"cbor": "9w==",
"hex": "f7",
"roundtrip": true,
"diagnostic": "undefined"
},
{
"cbor": "8A==",
"hex": "f0",
"roundtrip": true,
"diagnostic": "simple(16)"
},
{
"cbor": "+Bg=",
"hex": "f818",
"roundtrip": true,
"diagnostic": "simple(24)"
},
{
"cbor": "+P8=",
"hex": "f8ff",
"roundtrip": true,
"diagnostic": "simple(255)"
},
{
"cbor": "wHQyMDEzLTAzLTIxVDIwOjA0OjAwWg==",
"hex": "c074323031332d30332d32315432303a30343a30305a",
"roundtrip": true,
"diagnostic": "0(\"2013-03-21T20:04:00Z\")"
},
{
"cbor": "wRpRS2ew",
"hex": "c11a514b67b0",
"roundtrip": true,
"diagnostic": "1(1363896240)"
},
{
"cbor": "wftB1FLZ7CAAAA==",
"hex": "c1fb41d452d9ec200000",
"roundtrip": true,
"diagnostic": "1(1363896240.5)"
},
{
"cbor": "10QBAgME",
"hex": "d74401020304",
"roundtrip": true,
"diagnostic": "23(h'01020304')"
},
{
"cbor": "2BhFZElFVEY=",
"hex": "d818456449455446",
"roundtrip": true,
"diagnostic": "24(h'6449455446')"
},
{
"cbor": "2CB2aHR0cDovL3d3dy5leGFtcGxlLmNvbQ==",
"hex": "d82076687474703a2f2f7777772e6578616d706c652e636f6d",
"roundtrip": true,
"diagnostic": "32(\"http://www.example.com\")"
},
{
"cbor": "QA==",
"hex": "40",
"roundtrip": true,
"diagnostic": "h''"
},
{
"cbor": "RAECAwQ=",
"hex": "4401020304",
"roundtrip": true,
"diagnostic": "h'01020304'"
},
{
"cbor": "YA==",
"hex": "60",
"roundtrip": true,
"decoded": ""
},
{
"cbor": "YWE=",
"hex": "6161",
"roundtrip": true,
"decoded": "a"
},
{
"cbor": "ZElFVEY=",
"hex": "6449455446",
"roundtrip": true,
"decoded": "IETF"
},
{
"cbor": "YiJc",
"hex": "62225c",
"roundtrip": true,
"decoded": "\"\\"
},
{
"cbor": "YsO8",
"hex": "62c3bc",
"roundtrip": true,
"decoded": "ü"
},
{
"cbor": "Y+awtA==",
"hex": "63e6b0b4",
"roundtrip": true,
"decoded": "水"
},
{
"cbor": "ZPCQhZE=",
"hex": "64f0908591",
"roundtrip": true,
"decoded": "𐅑"
},
{
"cbor": "gA==",
"hex": "80",
"roundtrip": true,
"decoded": [
]
},
{
"cbor": "gwECAw==",
"hex": "83010203",
"roundtrip": true,
"decoded": [
1,
2,
3
]
},
{
"cbor": "gwGCAgOCBAU=",
"hex": "8301820203820405",
"roundtrip": true,
"decoded": [
1,
[
2,
3
],
[
4,
5
]
]
},
{
"cbor": "mBkBAgMEBQYHCAkKCwwNDg8QERITFBUWFxgYGBk=",
"hex": "98190102030405060708090a0b0c0d0e0f101112131415161718181819",
"roundtrip": true,
"decoded": [
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25
]
},
{
"cbor": "oA==",
"hex": "a0",
"roundtrip": true,
"decoded": {
}
},
{
"cbor": "ogECAwQ=",
"hex": "a201020304",
"roundtrip": true,
"skip": true,
"diagnostic": "{1: 2, 3: 4}"
},
{
"cbor": "omFhAWFiggID",
"hex": "a26161016162820203",
"roundtrip": true,
"decoded": {
"a": 1,
"b": [
2,
3
]
}
},
{
"cbor": "gmFhoWFiYWM=",
"hex": "826161a161626163",
"roundtrip": true,
"decoded": [
"a",
{
"b": "c"
}
]
},
{
"cbor": "pWFhYUFhYmFCYWNhQ2FkYURhZWFF",
"hex": "a56161614161626142616361436164614461656145",
"roundtrip": true,
"decoded": {
"a": "A",
"b": "B",
"c": "C",
"d": "D",
"e": "E"
}
},
{
"cbor": "X0IBAkMDBAX/",
"hex": "5f42010243030405ff",
"roundtrip": false,
"skip": true,
"diagnostic": "(_ h'0102', h'030405')"
},
{
"cbor": "f2VzdHJlYWRtaW5n/w==",
"hex": "7f657374726561646d696e67ff",
"roundtrip": false,
"decoded": "streaming"
},
{
"cbor": "n/8=",
"hex": "9fff",
"roundtrip": false,
"decoded": [
]
},
{
"cbor": "nwGCAgOfBAX//w==",
"hex": "9f018202039f0405ffff",
"roundtrip": false,
"decoded": [
1,
[
2,
3
],
[
4,
5
]
]
},
{
"cbor": "nwGCAgOCBAX/",
"hex": "9f01820203820405ff",
"roundtrip": false,
"decoded": [
1,
[
2,
3
],
[
4,
5
]
]
},
{
"cbor": "gwGCAgOfBAX/",
"hex": "83018202039f0405ff",
"roundtrip": false,
"decoded": [
1,
[
2,
3
],
[
4,
5
]
]
},
{
"cbor": "gwGfAgP/ggQF",
"hex": "83019f0203ff820405",
"roundtrip": false,
"decoded": [
1,
[
2,
3
],
[
4,
5
]
]
},
{
"cbor": "nwECAwQFBgcICQoLDA0ODxAREhMUFRYXGBgYGf8=",
"hex": "9f0102030405060708090a0b0c0d0e0f101112131415161718181819ff",
"roundtrip": false,
"decoded": [
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25
]
},
{
"cbor": "v2FhAWFinwID//8=",
"hex": "bf61610161629f0203ffff",
"roundtrip": false,
"decoded": {
"a": 1,
"b": [
2,
3
]
}
},
{
"cbor": "gmFhv2FiYWP/",
"hex": "826161bf61626163ff",
"roundtrip": false,
"decoded": [
"a",
{
"b": "c"
}
]
},
{
"cbor": "v2NGdW71Y0FtdCH/",
"hex": "bf6346756ef563416d7421ff",
"roundtrip": false,
"decoded": {
"Fun": true,
"Amt": -2
}
}
]

View File

@@ -1,119 +0,0 @@
#!/usr/bin/env python
# This will create golden files in a directory passed to it.
# A Test calls this internally to create the golden files
# So it can process them (so we don't have to checkin the files).
# Ensure msgpack-python and cbor are installed first, using:
# pip install --user msgpack-python
# pip install --user cbor
import cbor, msgpack, msgpackrpc, sys, os, threading
def get_test_data_list():
# get list with all primitive types, and a combo type
l0 = [
-8,
-1616,
-32323232,
-6464646464646464,
192,
1616,
32323232,
6464646464646464,
192,
-3232.0,
-6464646464.0,
3232.0,
6464646464.0,
False,
True,
None,
u"someday",
u"",
u"bytestring",
1328176922000002000,
-2206187877999998000,
270,
-2013855847999995777,
#-6795364578871345152,
]
l1 = [
{ "true": True,
"false": False },
{ "true": "True",
"false": False,
"uint16(1616)": 1616 },
{ "list": [1616, 32323232, True, -3232.0, {"TRUE":True, "FALSE":False}, [True, False] ],
"int32":32323232, "bool": True,
"LONG STRING": "123456789012345678901234567890123456789012345678901234567890",
"SHORT STRING": "1234567890" },
{ True: "true", 8: False, "false": 0 }
]
l = []
l.extend(l0)
l.append(l0)
l.extend(l1)
return l
def build_test_data(destdir):
l = get_test_data_list()
for i in range(len(l)):
# packer = msgpack.Packer()
serialized = msgpack.dumps(l[i])
f = open(os.path.join(destdir, str(i) + '.msgpack.golden'), 'wb')
f.write(serialized)
f.close()
serialized = cbor.dumps(l[i])
f = open(os.path.join(destdir, str(i) + '.cbor.golden'), 'wb')
f.write(serialized)
f.close()
def doRpcServer(port, stopTimeSec):
class EchoHandler(object):
def Echo123(self, msg1, msg2, msg3):
return ("1:%s 2:%s 3:%s" % (msg1, msg2, msg3))
def EchoStruct(self, msg):
return ("%s" % msg)
addr = msgpackrpc.Address('localhost', port)
server = msgpackrpc.Server(EchoHandler())
server.listen(addr)
# run thread to stop it after stopTimeSec seconds if > 0
if stopTimeSec > 0:
def myStopRpcServer():
server.stop()
t = threading.Timer(stopTimeSec, myStopRpcServer)
t.start()
server.start()
def doRpcClientToPythonSvc(port):
address = msgpackrpc.Address('localhost', port)
client = msgpackrpc.Client(address, unpack_encoding='utf-8')
print client.call("Echo123", "A1", "B2", "C3")
print client.call("EchoStruct", {"A" :"Aa", "B":"Bb", "C":"Cc"})
def doRpcClientToGoSvc(port):
# print ">>>> port: ", port, " <<<<<"
address = msgpackrpc.Address('localhost', port)
client = msgpackrpc.Client(address, unpack_encoding='utf-8')
print client.call("TestRpcInt.Echo123", ["A1", "B2", "C3"])
print client.call("TestRpcInt.EchoStruct", {"A" :"Aa", "B":"Bb", "C":"Cc"})
def doMain(args):
if len(args) == 2 and args[0] == "testdata":
build_test_data(args[1])
elif len(args) == 3 and args[0] == "rpc-server":
doRpcServer(int(args[1]), int(args[2]))
elif len(args) == 2 and args[0] == "rpc-client-python-service":
doRpcClientToPythonSvc(int(args[1]))
elif len(args) == 2 and args[0] == "rpc-client-go-service":
doRpcClientToGoSvc(int(args[1]))
else:
print("Usage: test.py " +
"[testdata|rpc-server|rpc-client-python-service|rpc-client-go-service] ...")
if __name__ == "__main__":
doMain(sys.argv[1:])

View File

@@ -1,193 +0,0 @@
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
import (
"time"
)
var (
timeDigits = [...]byte{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9'}
)
// EncodeTime encodes a time.Time as a []byte, including
// information on the instant in time and UTC offset.
//
// Format Description
//
// A timestamp is composed of 3 components:
//
// - secs: signed integer representing seconds since unix epoch
// - nsces: unsigned integer representing fractional seconds as a
// nanosecond offset within secs, in the range 0 <= nsecs < 1e9
// - tz: signed integer representing timezone offset in minutes east of UTC,
// and a dst (daylight savings time) flag
//
// When encoding a timestamp, the first byte is the descriptor, which
// defines which components are encoded and how many bytes are used to
// encode secs and nsecs components. *If secs/nsecs is 0 or tz is UTC, it
// is not encoded in the byte array explicitly*.
//
// Descriptor 8 bits are of the form `A B C DDD EE`:
// A: Is secs component encoded? 1 = true
// B: Is nsecs component encoded? 1 = true
// C: Is tz component encoded? 1 = true
// DDD: Number of extra bytes for secs (range 0-7).
// If A = 1, secs encoded in DDD+1 bytes.
// If A = 0, secs is not encoded, and is assumed to be 0.
// If A = 1, then we need at least 1 byte to encode secs.
// DDD says the number of extra bytes beyond that 1.
// E.g. if DDD=0, then secs is represented in 1 byte.
// if DDD=2, then secs is represented in 3 bytes.
// EE: Number of extra bytes for nsecs (range 0-3).
// If B = 1, nsecs encoded in EE+1 bytes (similar to secs/DDD above)
//
// Following the descriptor bytes, subsequent bytes are:
//
// secs component encoded in `DDD + 1` bytes (if A == 1)
// nsecs component encoded in `EE + 1` bytes (if B == 1)
// tz component encoded in 2 bytes (if C == 1)
//
// secs and nsecs components are integers encoded in a BigEndian
// 2-complement encoding format.
//
// tz component is encoded as 2 bytes (16 bits). Most significant bit 15 to
// Least significant bit 0 are described below:
//
// Timezone offset has a range of -12:00 to +14:00 (ie -720 to +840 minutes).
// Bit 15 = have\_dst: set to 1 if we set the dst flag.
// Bit 14 = dst\_on: set to 1 if dst is in effect at the time, or 0 if not.
// Bits 13..0 = timezone offset in minutes. It is a signed integer in Big Endian format.
//
func encodeTime(t time.Time) []byte {
//t := rv.Interface().(time.Time)
tsecs, tnsecs := t.Unix(), t.Nanosecond()
var (
bd byte
btmp [8]byte
bs [16]byte
i int = 1
)
l := t.Location()
if l == time.UTC {
l = nil
}
if tsecs != 0 {
bd = bd | 0x80
bigen.PutUint64(btmp[:], uint64(tsecs))
f := pruneSignExt(btmp[:], tsecs >= 0)
bd = bd | (byte(7-f) << 2)
copy(bs[i:], btmp[f:])
i = i + (8 - f)
}
if tnsecs != 0 {
bd = bd | 0x40
bigen.PutUint32(btmp[:4], uint32(tnsecs))
f := pruneSignExt(btmp[:4], true)
bd = bd | byte(3-f)
copy(bs[i:], btmp[f:4])
i = i + (4 - f)
}
if l != nil {
bd = bd | 0x20
// Note that Go Libs do not give access to dst flag.
_, zoneOffset := t.Zone()
//zoneName, zoneOffset := t.Zone()
zoneOffset /= 60
z := uint16(zoneOffset)
bigen.PutUint16(btmp[:2], z)
// clear dst flags
bs[i] = btmp[0] & 0x3f
bs[i+1] = btmp[1]
i = i + 2
}
bs[0] = bd
return bs[0:i]
}
// DecodeTime decodes a []byte into a time.Time.
func decodeTime(bs []byte) (tt time.Time, err error) {
bd := bs[0]
var (
tsec int64
tnsec uint32
tz uint16
i byte = 1
i2 byte
n byte
)
if bd&(1<<7) != 0 {
var btmp [8]byte
n = ((bd >> 2) & 0x7) + 1
i2 = i + n
copy(btmp[8-n:], bs[i:i2])
//if first bit of bs[i] is set, then fill btmp[0..8-n] with 0xff (ie sign extend it)
if bs[i]&(1<<7) != 0 {
copy(btmp[0:8-n], bsAll0xff)
//for j,k := byte(0), 8-n; j < k; j++ { btmp[j] = 0xff }
}
i = i2
tsec = int64(bigen.Uint64(btmp[:]))
}
if bd&(1<<6) != 0 {
var btmp [4]byte
n = (bd & 0x3) + 1
i2 = i + n
copy(btmp[4-n:], bs[i:i2])
i = i2
tnsec = bigen.Uint32(btmp[:])
}
if bd&(1<<5) == 0 {
tt = time.Unix(tsec, int64(tnsec)).UTC()
return
}
// In stdlib time.Parse, when a date is parsed without a zone name, it uses "" as zone name.
// However, we need name here, so it can be shown when time is printed.
// Zone name is in form: UTC-08:00.
// Note that Go Libs do not give access to dst flag, so we ignore dst bits
i2 = i + 2
tz = bigen.Uint16(bs[i:i2])
i = i2
// sign extend sign bit into top 2 MSB (which were dst bits):
if tz&(1<<13) == 0 { // positive
tz = tz & 0x3fff //clear 2 MSBs: dst bits
} else { // negative
tz = tz | 0xc000 //set 2 MSBs: dst bits
//tzname[3] = '-' (TODO: verify. this works here)
}
tzint := int16(tz)
if tzint == 0 {
tt = time.Unix(tsec, int64(tnsec)).UTC()
} else {
// For Go Time, do not use a descriptive timezone.
// It's unnecessary, and makes it harder to do a reflect.DeepEqual.
// The Offset already tells what the offset should be, if not on UTC and unknown zone name.
// var zoneName = timeLocUTCName(tzint)
tt = time.Unix(tsec, int64(tnsec)).In(time.FixedZone("", int(tzint)*60))
}
return
}
func timeLocUTCName(tzint int16) string {
if tzint == 0 {
return "UTC"
}
var tzname = []byte("UTC+00:00")
//tzname := fmt.Sprintf("UTC%s%02d:%02d", tzsign, tz/60, tz%60) //perf issue using Sprintf. inline below.
//tzhr, tzmin := tz/60, tz%60 //faster if u convert to int first
var tzhr, tzmin int16
if tzint < 0 {
tzname[3] = '-' // (TODO: verify. this works here)
tzhr, tzmin = -tzint/60, (-tzint)%60
} else {
tzhr, tzmin = tzint/60, tzint%60
}
tzname[4] = timeDigits[tzhr/10]
tzname[5] = timeDigits[tzhr%10]
tzname[7] = timeDigits[tzmin/10]
tzname[8] = timeDigits[tzmin%10]
return string(tzname)
//return time.FixedZone(string(tzname), int(tzint)*60)
}

View File

@@ -1,198 +0,0 @@
// // +build testing
// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
// Use of this source code is governed by a BSD-style license found in the LICENSE file.
package codec
// This file contains values used by tests and benchmarks.
// JSON/BSON do not like maps with keys that are not strings,
// so we only use maps with string keys here.
import (
"math"
"time"
)
var testStrucTime = time.Date(2012, 2, 2, 2, 2, 2, 2000, time.UTC).UTC()
type AnonInTestStruc struct {
AS string
AI64 int64
AI16 int16
AUi64 uint64
ASslice []string
AI64slice []int64
AF64slice []float64
// AMI32U32 map[int32]uint32
// AMU32F64 map[uint32]float64 // json/bson do not like it
AMSU16 map[string]uint16
}
type AnonInTestStrucIntf struct {
Islice []interface{}
Ms map[string]interface{}
Nintf interface{} //don't set this, so we can test for nil
T time.Time
}
type TestStruc struct {
_struct struct{} `codec:",omitempty"` //set omitempty for every field
S string
I64 int64
I16 int16
Ui64 uint64
Ui8 uint8
B bool
By uint8 // byte: msgp doesn't like byte
Sslice []string
I64slice []int64
I16slice []int16
Ui64slice []uint64
Ui8slice []uint8
Bslice []bool
Byslice []byte
Iptrslice []*int64
AnonInTestStruc
//M map[interface{}]interface{} `json:"-",bson:"-"`
Msi64 map[string]int64
// make this a ptr, so that it could be set or not.
// for comparison (e.g. with msgp), give it a struct tag (so it is not inlined),
// make this one omitempty (so it is included if nil).
*AnonInTestStrucIntf `codec:",omitempty"`
Nmap map[string]bool //don't set this, so we can test for nil
Nslice []byte //don't set this, so we can test for nil
Nint64 *int64 //don't set this, so we can test for nil
Mtsptr map[string]*TestStruc
Mts map[string]TestStruc
Its []*TestStruc
Nteststruc *TestStruc
}
// small struct for testing that codecgen works for unexported types
type tLowerFirstLetter struct {
I int
u uint64
S string
b []byte
}
func newTestStruc(depth int, bench bool, useInterface, useStringKeyOnly bool) (ts *TestStruc) {
var i64a, i64b, i64c, i64d int64 = 64, 6464, 646464, 64646464
ts = &TestStruc{
S: "some string",
I64: math.MaxInt64 * 2 / 3, // 64,
I16: 1616,
Ui64: uint64(int64(math.MaxInt64 * 2 / 3)), // 64, //don't use MaxUint64, as bson can't write it
Ui8: 160,
B: true,
By: 5,
Sslice: []string{"one", "two", "three"},
I64slice: []int64{1111, 2222, 3333},
I16slice: []int16{44, 55, 66},
Ui64slice: []uint64{12121212, 34343434, 56565656},
Ui8slice: []uint8{210, 211, 212},
Bslice: []bool{true, false, true, false},
Byslice: []byte{13, 14, 15},
Msi64: map[string]int64{
"one": 1,
"two": 2,
},
AnonInTestStruc: AnonInTestStruc{
// There's more leeway in altering this.
AS: "A-String",
AI64: -64646464,
AI16: 1616,
AUi64: 64646464,
// (U+1D11E)G-clef character may be represented in json as "\uD834\uDD1E".
// single reverse solidus character may be represented in json as "\u005C".
// include these in ASslice below.
ASslice: []string{"Aone", "Atwo", "Athree",
"Afour.reverse_solidus.\u005c", "Afive.Gclef.\U0001d11E"},
AI64slice: []int64{1, -22, 333, -4444, 55555, -666666},
AMSU16: map[string]uint16{"1": 1, "22": 2, "333": 3, "4444": 4},
AF64slice: []float64{11.11e-11, 22.22E+22, 33.33E-33, 44.44e+44, 555.55E-6, 666.66E6},
},
}
if useInterface {
ts.AnonInTestStrucIntf = &AnonInTestStrucIntf{
Islice: []interface{}{"true", true, "no", false, uint64(288), float64(0.4)},
Ms: map[string]interface{}{
"true": "true",
"int64(9)": false,
},
T: testStrucTime,
}
}
//For benchmarks, some things will not work.
if !bench {
//json and bson require string keys in maps
//ts.M = map[interface{}]interface{}{
// true: "true",
// int8(9): false,
//}
//gob cannot encode nil in element in array (encodeArray: nil element)
ts.Iptrslice = []*int64{nil, &i64a, nil, &i64b, nil, &i64c, nil, &i64d, nil}
// ts.Iptrslice = nil
}
if !useStringKeyOnly {
// ts.AnonInTestStruc.AMU32F64 = map[uint32]float64{1: 1, 2: 2, 3: 3} // Json/Bson barf
}
if depth > 0 {
depth--
if ts.Mtsptr == nil {
ts.Mtsptr = make(map[string]*TestStruc)
}
if ts.Mts == nil {
ts.Mts = make(map[string]TestStruc)
}
ts.Mtsptr["0"] = newTestStruc(depth, bench, useInterface, useStringKeyOnly)
ts.Mts["0"] = *(ts.Mtsptr["0"])
ts.Its = append(ts.Its, ts.Mtsptr["0"])
}
return
}
// Some other types
type Sstring string
type Bbool bool
type Sstructsmall struct {
A int
}
type Sstructbig struct {
A int
B bool
c string
// Sval Sstruct
Ssmallptr *Sstructsmall
Ssmall *Sstructsmall
Sptr *Sstructbig
}
type SstructbigMapBySlice struct {
_struct struct{} `codec:",toarray"`
A int
B bool
c string
// Sval Sstruct
Ssmallptr *Sstructsmall
Ssmall *Sstructsmall
Sptr *Sstructbig
}
type Sinterface interface {
Noop()
}

View File

@@ -0,0 +1,24 @@
# Compiled Object files, Static and Dynamic libs (Shared Objects)
*.o
*.a
*.so
# Folders
_obj
_test
# Architecture specific extensions/prefixes
*.[568vq]
[568vq].out
*.cgo1.go
*.cgo2.c
_cgo_defun.c
_cgo_gotypes.go
_cgo_export.*
_testmain.go
*.exe
*.test
*.prof

View File

@@ -0,0 +1,22 @@
The MIT License (MIT)
Copyright (c) 2015 Xiang Li
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

View File

@@ -0,0 +1,39 @@
## Getting Started
### Install the handler
We first need to serve the probing HTTP handler.
```go
http.HandleFunc("/health", probing.NewHandler())
err := http.ListenAndServe(":12345", nil)
if err != nil {
log.Fatal("ListenAndServe: ", err)
}
```
### Start to probe
Now we can start to probe the endpoint.
``` go
id := "example"
probingInterval = 5 * time.Second
url := "http://example.com:12345/health"
p.AddHTTP(id, probingInterval, url)
time.Sleep(13 * time.Second)
status, err := p.Status(id)
fmt.Printf("Total Probing: %d, Total Loss: %d, Estimated RTT: %v, Estimated Clock Difference: %v\n",
status.Total(), status.Loss(), status.SRTT(), status.ClockDiff())
// Total Probing: 2, Total Loss: 0, Estimated RTT: 320.771µs, Estimated Clock Difference: -35.869µs
```
### TODOs:
- TCP probing
- UDP probing
- Gossip based probing
- More accurate RTT estimation
- More accurate Clock difference estimation
- Use a clock interface rather than the real clock

View File

@@ -0,0 +1,134 @@
package probing
import (
"encoding/json"
"errors"
"net/http"
"sync"
"time"
)
var (
ErrNotFound = errors.New("probing: id not found")
ErrExist = errors.New("probing: id exists")
)
type Prober interface {
AddHTTP(id string, probingInterval time.Duration, endpoints []string) error
Remove(id string) error
RemoveAll()
Reset(id string) error
Status(id string) (Status, error)
}
type prober struct {
mu sync.Mutex
targets map[string]*status
tr http.RoundTripper
}
func NewProber(tr http.RoundTripper) Prober {
p := &prober{targets: make(map[string]*status)}
if tr == nil {
p.tr = http.DefaultTransport
} else {
p.tr = tr
}
return p
}
func (p *prober) AddHTTP(id string, probingInterval time.Duration, endpoints []string) error {
p.mu.Lock()
defer p.mu.Unlock()
if _, ok := p.targets[id]; ok {
return ErrExist
}
s := &status{stopC: make(chan struct{})}
p.targets[id] = s
ticker := time.NewTicker(probingInterval)
go func() {
pinned := 0
for {
select {
case <-ticker.C:
start := time.Now()
req, err := http.NewRequest("GET", endpoints[pinned], nil)
if err != nil {
panic(err)
}
resp, err := p.tr.RoundTrip(req)
if err != nil {
s.recordFailure()
pinned = (pinned + 1) % len(endpoints)
continue
}
var hh Health
d := json.NewDecoder(resp.Body)
err = d.Decode(&hh)
resp.Body.Close()
if err != nil || !hh.OK {
s.recordFailure()
pinned = (pinned + 1) % len(endpoints)
continue
}
s.record(time.Since(start), hh.Now)
case <-s.stopC:
ticker.Stop()
return
}
}
}()
return nil
}
func (p *prober) Remove(id string) error {
p.mu.Lock()
defer p.mu.Unlock()
s, ok := p.targets[id]
if !ok {
return ErrNotFound
}
close(s.stopC)
delete(p.targets, id)
return nil
}
func (p *prober) RemoveAll() {
p.mu.Lock()
defer p.mu.Unlock()
for _, s := range p.targets {
close(s.stopC)
}
p.targets = make(map[string]*status)
}
func (p *prober) Reset(id string) error {
p.mu.Lock()
defer p.mu.Unlock()
s, ok := p.targets[id]
if !ok {
return ErrNotFound
}
s.reset()
return nil
}
func (p *prober) Status(id string) (Status, error) {
p.mu.Lock()
defer p.mu.Unlock()
s, ok := p.targets[id]
if !ok {
return nil, ErrNotFound
}
return s, nil
}

View File

@@ -0,0 +1,90 @@
package probing
import (
"net/http/httptest"
"testing"
"time"
)
var (
testID = "testID"
)
func TestProbe(t *testing.T) {
s := httptest.NewServer(NewHandler())
p := NewProber(nil)
p.AddHTTP(testID, time.Millisecond, []string{s.URL})
defer p.Remove(testID)
time.Sleep(100 * time.Millisecond)
status, err := p.Status(testID)
if err != nil {
t.Fatalf("err = %v, want %v", err, nil)
}
if total := status.Total(); total < 50 || total > 150 {
t.Fatalf("total = %v, want around %v", total, 100)
}
if health := status.Health(); health != true {
t.Fatalf("health = %v, want %v", health, true)
}
// become unhealthy
s.Close()
time.Sleep(100 * time.Millisecond)
if total := status.Total(); total < 150 || total > 250 {
t.Fatalf("total = %v, want around %v", total, 200)
}
if loss := status.Loss(); loss < 50 || loss > 150 {
t.Fatalf("loss = %v, want around %v", loss, 200)
}
if health := status.Health(); health != false {
t.Fatalf("health = %v, want %v", health, false)
}
}
func TestProbeReset(t *testing.T) {
s := httptest.NewServer(NewHandler())
defer s.Close()
p := NewProber(nil)
p.AddHTTP(testID, time.Millisecond, []string{s.URL})
defer p.Remove(testID)
time.Sleep(100 * time.Millisecond)
status, err := p.Status(testID)
if err != nil {
t.Fatalf("err = %v, want %v", err, nil)
}
if total := status.Total(); total < 50 || total > 150 {
t.Fatalf("total = %v, want around %v", total, 100)
}
if health := status.Health(); health != true {
t.Fatalf("health = %v, want %v", health, true)
}
p.Reset(testID)
time.Sleep(100 * time.Millisecond)
if total := status.Total(); total < 50 || total > 150 {
t.Fatalf("total = %v, want around %v", total, 100)
}
if health := status.Health(); health != true {
t.Fatalf("health = %v, want %v", health, true)
}
}
func TestProbeRemove(t *testing.T) {
s := httptest.NewServer(NewHandler())
defer s.Close()
p := NewProber(nil)
p.AddHTTP(testID, time.Millisecond, []string{s.URL})
p.Remove(testID)
_, err := p.Status(testID)
if err != ErrNotFound {
t.Fatalf("err = %v, want %v", err, ErrNotFound)
}
}

View File

@@ -0,0 +1,25 @@
package probing
import (
"encoding/json"
"net/http"
"time"
)
func NewHandler() http.Handler {
return &httpHealth{}
}
type httpHealth struct {
}
type Health struct {
OK bool
Now time.Time
}
func (h *httpHealth) ServeHTTP(w http.ResponseWriter, r *http.Request) {
health := Health{OK: true, Now: time.Now()}
e := json.NewEncoder(w)
e.Encode(health)
}

View File

@@ -0,0 +1,96 @@
package probing
import (
"sync"
"time"
)
var (
// weight factor
α = 0.125
)
type Status interface {
Total() int64
Loss() int64
Health() bool
// Estimated smoothed round trip time
SRTT() time.Duration
// Estimated clock difference
ClockDiff() time.Duration
StopNotify() <-chan struct{}
}
type status struct {
mu sync.Mutex
srtt time.Duration
total int64
loss int64
health bool
clockdiff time.Duration
stopC chan struct{}
}
// SRTT = (1-α) * SRTT + α * RTT
func (s *status) SRTT() time.Duration {
s.mu.Lock()
defer s.mu.Unlock()
return s.srtt
}
func (s *status) Total() int64 {
s.mu.Lock()
defer s.mu.Unlock()
return s.total
}
func (s *status) Loss() int64 {
s.mu.Lock()
defer s.mu.Unlock()
return s.loss
}
func (s *status) Health() bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.health
}
func (s *status) ClockDiff() time.Duration {
s.mu.Lock()
defer s.mu.Unlock()
return s.clockdiff
}
func (s *status) StopNotify() <-chan struct{} {
return s.stopC
}
func (s *status) record(rtt time.Duration, when time.Time) {
s.mu.Lock()
defer s.mu.Unlock()
s.total += 1
s.health = true
s.srtt = time.Duration((1-α)*float64(s.srtt) + α*float64(rtt))
s.clockdiff = time.Now().Sub(when) - s.srtt/2
}
func (s *status) recordFailure() {
s.mu.Lock()
defer s.mu.Unlock()
s.total++
s.health = false
s.loss += 1
}
func (s *status) reset() {
s.mu.Lock()
defer s.mu.Unlock()
s.srtt = 0
s.total = 0
s.health = false
s.clockdiff = 0
}

View File

@@ -99,6 +99,10 @@ Now it's time to dig into the full etcd API and other guides.
See [CONTRIBUTING](CONTRIBUTING.md) for details on submitting patches and the contribution workflow.
## Reporting bugs
See [reporting bugs](Documentation/reporting_bugs.md) for details about reporting any issue you may encounter..
## Project Details
### Versioning

Some files were not shown because too many files have changed in this diff Show More