improvement: new project structure

.dockerignore

@@ -1,9 +0,0 @@
-node_modules
-localData/*
-localMetadata/*
-# Keep the .git/HEAD file in order to properly report version
-.git/objects
-.github
-.tox
-coverage
-.DS_Store

.eslintrc

@@ -1,54 +1 @@
-{
+{ "extends": "scality" }
-    "extends": "scality",
-    "plugins": [
-        "mocha"
-    ],
-    "rules": {
-        "import/extensions": "off",
-        "lines-around-directive": "off",
-        "no-underscore-dangle": "off",
-        "indent": "off",
-        "object-curly-newline": "off",
-        "operator-linebreak": "off",
-        "function-paren-newline": "off",
-        "import/newline-after-import": "off",
-        "prefer-destructuring": "off",
-        "implicit-arrow-linebreak": "off",
-        "no-bitwise": "off",
-        "dot-location": "off",
-        "comma-dangle": "off",
-        "no-undef-init": "off",
-        "global-require": "off",
-        "import/no-dynamic-require": "off",
-        "class-methods-use-this": "off",
-        "no-plusplus": "off",
-        "no-else-return": "off",
-        "object-property-newline": "off",
-        "import/order": "off",
-        "no-continue": "off",
-        "no-tabs": "off",
-        "lines-between-class-members": "off",
-        "prefer-spread": "off",
-        "no-lonely-if": "off",
-        "no-useless-escape": "off",
-        "no-restricted-globals": "off",
-        "no-buffer-constructor": "off",
-        "import/no-extraneous-dependencies": "off",
-        "space-unary-ops": "off",
-        "no-useless-return": "off",
-        "no-unexpected-multiline": "off",
-        "no-mixed-operators": "off",
-        "newline-per-chained-call": "off",
-        "operator-assignment": "off",
-        "spaced-comment": "off",
-        "comma-style": "off",
-        "no-restricted-properties": "off",
-        "new-parens": "off",
-        "no-multi-spaces": "off",
-        "quote-props": "off",
-        "mocha/no-exclusive-tests": "error",
-    },
-    "parserOptions": {
-        "ecmaVersion": 2020
-    }
-}

.github/ISSUE_TEMPLATE.md

@@ -1,32 +1,19 @@
-# General support information
+# Issue template
 
-GitHub Issues are **reserved** for actionable bug reports (including
+If you are reporting a new issue, make sure that we do not have any
-documentation inaccuracies), and feature requests.
+duplicates already open. You can ensure this by searching the issue list for
-**All questions** (regarding configuration, use cases, performance, community,
+this repository. If there is a duplicate, please close your issue and add a
-events, setup and usage recommendations, among other things) should be asked on
+comment to the existing issue instead.
-the **[Zenko Forum](http://forum.zenko.io/)**.
 
-> Questions opened as GitHub issues will systematically be closed, and moved to
+## General support information
-> the [Zenko Forum](http://forum.zenko.io/).
 
---------------------------------------------------------------------------------
+GitHub Issues are reserved for actionable bug reports and feature requests.
-
+General questions should be sent to the
-## Avoiding duplicates
+[S3 scality server Forum](http://forum.scality.com/).
-
-When reporting a new issue/requesting a feature, make sure that we do not have
-any duplicates already open:
-
-- search the issue list for this repository (use the search bar, select
-  "Issues" on the left pane after searching);
-- if there is a duplicate, please do not open your issue, and add a comment
-  to the existing issue instead.
-
---------------------------------------------------------------------------------
 
 ## Bug report information
 
-(delete this section (everything between the lines) if you're not reporting a bug
+(delete this section if not applicable)
-but requesting a feature)
 
 ### Description
 
@@ -42,22 +29,13 @@ Describe the results you received
 
 ### Expected result
 
-Describe the results you expected
+Describe the results you expecteds
 
-### Additional information
+### Additional information: (Node.js version, Docker version, etc)
-
-- Node.js version,
-- Docker version,
-- yarn version,
-- distribution/OS,
-- optional: anything else you deem helpful to us.
-
---------------------------------------------------------------------------------
 
 ## Feature Request
 
-(delete this section (everything between the lines) if you're not requesting
+(delete this section if not applicable)
-a feature but reporting a bug)
 
 ### Proposal
 
@@ -74,14 +52,3 @@ What you would like to happen
 ### Use case
 
 Please provide use cases for changing the current behavior
-
-### Additional information
-
-- Is this request for your company? Y/N
-  - If Y: Company name:
-  - Are you using any Scality Enterprise Edition products (RING, Zenko EE)? Y/N
-- Are you willing to contribute this feature yourself?
-- Position/Title:
-- How did you hear about us?
-
---------------------------------------------------------------------------------

.github/actions/setup-ci/action.yaml

@@ -1,43 +0,0 @@
----
-name: "Setup CI environment"
-description: "Setup Cloudserver CI environment"
-
-runs:
-  using: composite
-  steps:
-    - name: Setup etc/hosts
-      shell: bash
-      run: sudo echo "127.0.0.1 bucketwebsitetester.s3-website-us-east-1.amazonaws.com" | sudo tee -a /etc/hosts
-    - name: Setup Credentials
-      shell: bash
-      run: bash .github/scripts/credentials.bash
-    - name: Setup job artifacts directory
-      shell: bash
-      run: |-
-        set -exu;
-        mkdir -p /tmp/artifacts/${JOB_NAME}/;
-    - uses: actions/setup-node@v4
-      with:
-        node-version: '16'
-        cache: 'yarn'
-    - name: install dependencies
-      shell: bash
-      run: yarn install --ignore-engines --frozen-lockfile --network-concurrency 1
-    - uses: actions/cache@v3
-      with:
-        path: ~/.cache/pip
-        key: ${{ runner.os }}-pip
-    - uses: actions/setup-python@v4
-      with:
-        python-version: 3.9
-    - name: Setup python2 test environment
-      shell: bash
-      run: |
-        sudo apt-get install -y libdigest-hmac-perl
-        pip install 's3cmd==2.3.0'
-    - name: fix sproxyd.conf permissions
-      shell: bash
-      run: sudo chown root:root .github/docker/sproxyd/conf/sproxyd0.conf
-    - name: ensure fuse kernel module is loaded (for sproxyd)
-      shell: bash
-      run: sudo modprobe fuse

.github/ceph/Dockerfile

@@ -1,25 +0,0 @@
-FROM ceph/daemon:v3.2.1-stable-3.2-mimic-centos-7
-
-ENV CEPH_DAEMON demo
-ENV CEPH_DEMO_DAEMONS mon,mgr,osd,rgw
-
-ENV CEPH_DEMO_UID zenko
-ENV CEPH_DEMO_ACCESS_KEY accessKey1
-ENV CEPH_DEMO_SECRET_KEY verySecretKey1
-ENV CEPH_DEMO_BUCKET zenkobucket
-
-ENV CEPH_PUBLIC_NETWORK 0.0.0.0/0
-ENV MON_IP 0.0.0.0
-ENV NETWORK_AUTO_DETECT 4
-ENV RGW_CIVETWEB_PORT 8001
-
-RUN rm /etc/yum.repos.d/tcmu-runner.repo
-
-ADD ./entrypoint-wrapper.sh /
-RUN chmod +x /entrypoint-wrapper.sh && \
-    yum install -y python-pip && \
-    yum clean all && \
-    pip install awscli && \
-    rm -rf /root/.cache/pip
-
-ENTRYPOINT [ "/entrypoint-wrapper.sh" ]

.github/ceph/entrypoint-wrapper.sh

@@ -1,37 +0,0 @@
-#!/bin/sh
-
-touch /artifacts/ceph.log
-mkfifo /tmp/entrypoint_output
-# We run this in the background so that we can tail the RGW log after init,
-# because entrypoint.sh never returns
-
-# The next line will be needed when ceph builds 3.2.2 so I'll leave it here
-# bash /opt/ceph-container/bin/entrypoint.sh > /tmp/entrypoint_output &
-
-bash /entrypoint.sh > /tmp/entrypoint_output &
-entrypoint_pid="$!"
-while read -r line; do
-    echo $line
-    # When we find this line server has started
-    if [ -n "$(echo $line | grep 'Creating bucket')" ]; then
-        break
-    fi
-done < /tmp/entrypoint_output
-
-# Make our buckets - CEPH_DEMO_BUCKET is set to force the "Creating bucket" message, but unused
-s3cmd mb s3://cephbucket s3://cephbucket2
-
-mkdir /root/.aws
-cat > /root/.aws/credentials <<EOF
-[default]
-aws_access_key_id = accessKey1
-aws_secret_access_key = verySecretKey1
-EOF
-
-# Enable versioning on them
-for bucket in cephbucket cephbucket2; do
-    echo "Enabling versiong for $bucket"
-    aws --endpoint http://127.0.0.1:8001 s3api  put-bucket-versioning --bucket $bucket --versioning Status=Enabled
-done
-tail -f /var/log/ceph/client.rgw.*.log | tee -a /artifacts/ceph.log
-wait $entrypoint_pid

.github/ceph/wait_for_ceph.sh

@@ -1,11 +0,0 @@
-#!/bin/sh
-
-# This script is needed because RADOS Gateway
-# will open the port before beginning to serve traffic
-# causing wait_for_local_port.bash to exit immediately
-
-echo 'Waiting for ceph'
-while [ -z "$(curl 127.0.0.1:8001 2>/dev/null)" ]; do
-    sleep 1
-    echo -n "."
-done

.github/dependabot.yml

@@ -1,10 +0,0 @@
----
-version: 2
-updates:
-  - package-ecosystem: npm
-    directory: "/"
-    schedule:
-      interval: daily
-      time: "13:00"
-    open-pull-requests-limit: 10
-    target-branch: "development/7.4"

.github/docker/creds.env

@@ -1,36 +0,0 @@
-azurebackend_AZURE_STORAGE_ACCESS_KEY
-azurebackend_AZURE_STORAGE_ACCOUNT_NAME
-azurebackend_AZURE_STORAGE_ENDPOINT
-azurebackend2_AZURE_STORAGE_ACCESS_KEY
-azurebackend2_AZURE_STORAGE_ACCOUNT_NAME
-azurebackend2_AZURE_STORAGE_ENDPOINT
-azurebackendmismatch_AZURE_STORAGE_ACCESS_KEY
-azurebackendmismatch_AZURE_STORAGE_ACCOUNT_NAME
-azurebackendmismatch_AZURE_STORAGE_ENDPOINT
-azurenonexistcontainer_AZURE_STORAGE_ACCESS_KEY
-azurenonexistcontainer_AZURE_STORAGE_ACCOUNT_NAME
-azurenonexistcontainer_AZURE_STORAGE_ENDPOINT
-azuretest_AZURE_BLOB_ENDPOINT
-b2backend_B2_ACCOUNT_ID
-b2backend_B2_STORAGE_ACCESS_KEY
-GOOGLE_SERVICE_EMAIL
-GOOGLE_SERVICE_KEY
-AWS_S3_BACKEND_ACCESS_KEY
-AWS_S3_BACKEND_SECRET_KEY
-AWS_S3_BACKEND_ACCESS_KEY_2
-AWS_S3_BACKEND_SECRET_KEY_2
-AWS_GCP_BACKEND_ACCESS_KEY
-AWS_GCP_BACKEND_SECRET_KEY
-AWS_GCP_BACKEND_ACCESS_KEY_2
-AWS_GCP_BACKEND_SECRET_KEY_2
-b2backend_B2_STORAGE_ENDPOINT
-gcpbackend2_GCP_SERVICE_EMAIL
-gcpbackend2_GCP_SERVICE_KEY
-gcpbackend2_GCP_SERVICE_KEYFILE
-gcpbackend_GCP_SERVICE_EMAIL
-gcpbackend_GCP_SERVICE_KEY
-gcpbackendmismatch_GCP_SERVICE_EMAIL
-gcpbackendmismatch_GCP_SERVICE_KEY
-gcpbackend_GCP_SERVICE_KEYFILE
-gcpbackendmismatch_GCP_SERVICE_KEYFILE
-gcpbackendnoproxy_GCP_SERVICE_KEYFILE

.github/docker/docker-compose.yaml

@@ -1,92 +0,0 @@
-services:
-  cloudserver:
-    image: ${CLOUDSERVER_IMAGE}
-    command: sh -c "yarn start > /artifacts/s3.log"
-    network_mode: "host"
-    volumes:
-      - /tmp/ssl:/ssl
-      - /tmp/ssl-kmip:/ssl-kmip
-      - ${HOME}/.aws/credentials:/root/.aws/credentials
-      - /tmp/artifacts/${JOB_NAME}:/artifacts
-    environment:
-      - CI=true
-      - ENABLE_LOCAL_CACHE=true
-      - REDIS_HOST=0.0.0.0
-      - REDIS_PORT=6379
-      - REPORT_TOKEN=report-token-1
-      - REMOTE_MANAGEMENT_DISABLE=1
-      - HEALTHCHECKS_ALLOWFROM=0.0.0.0/0
-      - DATA_HOST=0.0.0.0
-      - METADATA_HOST=0.0.0.0
-      - S3BACKEND
-      - S3DATA
-      - S3METADATA
-      - MPU_TESTING
-      - S3VAULT
-      - S3_LOCATION_FILE
-      - ENABLE_UTAPI_V2
-      - BUCKET_DENY_FILTER
-      - S3KMS
-      - S3KMIP_PORT
-      - S3KMIP_HOSTS
-      - S3KMIP-COMPOUND_CREATE
-      - S3KMIP_BUCKET_ATTRIBUTE_NAME
-      - S3KMIP_PIPELINE_DEPTH
-      - S3KMIP_KEY
-      - S3KMIP_CERT
-      - S3KMIP_CA
-      - MONGODB_HOSTS=0.0.0.0:27018
-      - MONGODB_RS=rs0
-      - DEFAULT_BUCKET_KEY_FORMAT
-      - METADATA_MAX_CACHED_BUCKETS
-      - ENABLE_NULL_VERSION_COMPAT_MODE
-      - SCUBA_HOST
-      - SCUBA_PORT
-      - SCUBA_HEALTHCHECK_FREQUENCY
-      - S3QUOTA
-      - QUOTA_ENABLE_INFLIGHTS
-    env_file:
-      - creds.env
-    depends_on:
-      - redis
-    extra_hosts:
-      - "bucketwebsitetester.s3-website-us-east-1.amazonaws.com:127.0.0.1"
-      - "pykmip.local:127.0.0.1"
-  redis:
-    image: redis:alpine
-    network_mode: "host"
-  squid:
-    network_mode: "host"
-    profiles: ['ci-proxy']
-    image: scality/ci-squid
-    command: >-
-      sh -c 'mkdir -p /ssl &&
-            openssl req -new -newkey rsa:2048 -sha256 -days 365 -nodes -x509 \
-              -subj "/C=US/ST=Country/L=City/O=Organization/CN=CN=scality-proxy" \
-              -keyout /ssl/myca.pem  -out /ssl/myca.pem &&
-            cp /ssl/myca.pem /ssl/CA.pem &&
-            squid -f /etc/squid/squid.conf -N -z &&
-            squid -f /etc/squid/squid.conf -NYCd 1'
-    volumes:
-      - /tmp/ssl:/ssl
-  pykmip:
-    network_mode: "host"
-    profiles: ['pykmip']
-    image: ${PYKMIP_IMAGE:-ghcr.io/scality/cloudserver/pykmip}
-    volumes:
-      - /tmp/artifacts/${JOB_NAME}:/artifacts
-  mongo:
-    network_mode: "host"
-    profiles: ['mongo', 'ceph']
-    image: ${MONGODB_IMAGE}
-  ceph: 
-    network_mode: "host"
-    profiles: ['ceph']
-    image: ghcr.io/scality/cloudserver/ci-ceph 
-  sproxyd:
-    network_mode: "host"
-    profiles: ['sproxyd']
-    image: sproxyd-standalone
-    build: ./sproxyd
-    user: 0:0
-    privileged: yes

.github/docker/mongodb/Dockerfile

@@ -1,28 +0,0 @@
-FROM mongo:5.0.21
-
-ENV USER=scality \
-    HOME_DIR=/home/scality \
-    CONF_DIR=/conf \
-    DATA_DIR=/data
-
-# Set up directories and permissions
-RUN mkdir -p /data/db /data/configdb && chown -R mongodb:mongodb /data/db /data/configdb; \
-    mkdir /logs; \
-    adduser --uid 1000 --disabled-password --gecos --quiet --shell /bin/bash scality
-
-# Set up environment variables and directories for scality user
-RUN mkdir ${CONF_DIR} && \
-    chown -R ${USER} ${CONF_DIR} && \
-    chown -R ${USER} ${DATA_DIR}
-
-# copy the mongo config file
-COPY /conf/mongod.conf /conf/mongod.conf
-COPY /conf/mongo-run.sh /conf/mongo-run.sh
-COPY /conf/initReplicaSet /conf/initReplicaSet.js
-
-EXPOSE 27017/tcp
-EXPOSE 27018
-
-# Set up CMD
-ENTRYPOINT ["bash", "/conf/mongo-run.sh"]
-CMD ["bash", "/conf/mongo-run.sh"]

.github/docker/mongodb/conf/initReplicaSet

@@ -1,4 +0,0 @@
-rs.initiate({
-    _id: "rs0",
-    members: [{ _id: 0, host: "127.0.0.1:27018" }]
-});

.github/docker/mongodb/conf/mongo-run.sh

@@ -1,10 +0,0 @@
-#!/bin/bash
-set -exo pipefail
-
-init_RS() {
-  sleep 5
-  mongo --port 27018 /conf/initReplicaSet.js
-}
-init_RS &
-
-mongod --bind_ip_all --config=/conf/mongod.conf

.github/docker/mongodb/conf/mongod.conf

@@ -1,15 +0,0 @@
-storage:
-   journal:
-      enabled: true
-   engine: wiredTiger
-   dbPath: "/data/db"
-processManagement:
-   fork: false
-net:
-   port: 27018
-   bindIp: 0.0.0.0
-replication:
-   replSetName: "rs0"
-   enableMajorityReadConcern: true
-security:
-    authorization: disabled

.github/docker/sproxyd/Dockerfile

@@ -1,3 +0,0 @@
-FROM ghcr.io/scality/federation/sproxyd:7.10.6.8
-ADD ./conf/supervisord.conf ./conf/nginx.conf ./conf/fastcgi_params ./conf/sproxyd0.conf /conf/
-RUN chown root:root /conf/sproxyd0.conf

.github/docker/sproxyd/conf/fastcgi_params

@@ -1,26 +0,0 @@
-fastcgi_param  QUERY_STRING       $query_string;
-fastcgi_param  REQUEST_METHOD     $request_method;
-fastcgi_param  CONTENT_TYPE       $content_type;
-fastcgi_param  CONTENT_LENGTH     $content_length;
-
-#fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
-fastcgi_param  SCRIPT_NAME        /var/www;
-fastcgi_param  PATH_INFO          $document_uri;
-
-fastcgi_param  REQUEST_URI        $request_uri;
-fastcgi_param  DOCUMENT_URI       $document_uri;
-fastcgi_param  DOCUMENT_ROOT      $document_root;
-fastcgi_param  SERVER_PROTOCOL    $server_protocol;
-fastcgi_param  HTTPS              $https if_not_empty;
-
-fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
-fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;
-
-fastcgi_param  REMOTE_ADDR        $remote_addr;
-fastcgi_param  REMOTE_PORT        $remote_port;
-fastcgi_param  SERVER_ADDR        $server_addr;
-fastcgi_param  SERVER_PORT        $server_port;
-fastcgi_param  SERVER_NAME        $server_name;
-
-# PHP only, required if PHP was built with --enable-force-cgi-redirect
-fastcgi_param  REDIRECT_STATUS    200;

.github/docker/sproxyd/conf/nginx.conf

@@ -1,88 +0,0 @@
-worker_processes 1;
-error_log  /logs/error.log;
-user root root;
-events {
-    worker_connections  1000;
-    reuse_port on;
-    multi_accept on;
-}
-worker_rlimit_nofile    20000;
-http {
-    root /var/www/;
-    upstream sproxyds {
-        least_conn;
-        keepalive 40;
-        server 127.0.0.1:20000;
-    }
-    server {
-        client_max_body_size 0;
-        client_body_timeout 150;
-        client_header_timeout 150;
-        postpone_output 0;
-        client_body_postpone_size 0;
-        keepalive_requests 1100;
-        keepalive_timeout 300s;
-        server_tokens off;
-        default_type application/octet-stream;
-        gzip off;
-        tcp_nodelay on;
-        tcp_nopush  on;
-        sendfile on;
-        listen 81;
-        server_name localhost;
-        rewrite ^/arc/(.*)$ /dc1/$1 permanent;
-        location ~* ^/proxy/(.*)$ {
-            rewrite ^/proxy/(.*)$ /$1 last;
-        }
-        allow 127.0.0.1;
-
-        deny all;
-        set $usermd '-';
-        set $sentusermd '-';
-    	set $elapsed_ms '-';
-    	set $now '-';
-    	log_by_lua '
-    		if not(ngx.var.http_x_scal_usermd == nil) and  string.len(ngx.var.http_x_scal_usermd) > 2 then
-    			ngx.var.usermd = string.sub(ngx.decode_base64(ngx.var.http_x_scal_usermd),1,-3)
-    		end
-                    if not(ngx.var.sent_http_x_scal_usermd == nil) and string.len(ngx.var.sent_http_x_scal_usermd) > 2 then
-                            ngx.var.sentusermd = string.sub(ngx.decode_base64(ngx.var.sent_http_x_scal_usermd),1,-3)
-                    end
-    		local elapsed_ms = tonumber(ngx.var.request_time)
-    		if not (  elapsed_ms == nil) then
-    			elapsed_ms = elapsed_ms * 1000
-    			ngx.var.elapsed_ms = tostring(elapsed_ms)
-    		end
-    		local time = tonumber(ngx.var.msec) * 1000
-    		ngx.var.now = time
-    	';
-        log_format irm '{ "time":"$now","connection":"$connection","request":"$connection_requests","hrtime":"$msec",'
-                       '"httpMethod":"$request_method","httpURL":"$uri","elapsed_ms":$elapsed_ms,'
-                       '"httpCode":$status,"requestLength":$request_length,"bytesSent":$bytes_sent,'
-		       '"contentLength":"$content_length","sentContentLength":"$sent_http_content_length",'
-		       '"contentType":"$content_type","s3Address":"$remote_addr",'
-		       '"requestUserMd":"$usermd","responseUserMd":"$sentusermd",'
-                       '"ringKeyVersion":"$sent_http_x_scal_version","ringStatus":"$sent_http_x_scal_ring_status",'
-		       '"s3Port":"$remote_port","sproxydStatus":"$upstream_status","req_id":"$http_x_scal_request_uids",'
-		       '"ifMatch":"$http_if_match","ifNoneMatch":"$http_if_none_match",'
-                       '"range":"$http_range","contentRange":"$sent_http_content_range","nginxPID":$PID,'
-                       '"sproxydAddress":"$upstream_addr","sproxydResponseTime_s":"$upstream_response_time" }';
-        access_log /dev/stdout irm;
-    	error_log  /dev/stdout error;
-        location / {
-            proxy_request_buffering off;
-            fastcgi_request_buffering off;
-            fastcgi_no_cache 1;
-            fastcgi_cache_bypass 1;
-            fastcgi_buffering off;
-            fastcgi_ignore_client_abort on;
-            fastcgi_keep_conn on;
-            include fastcgi_params;
-            fastcgi_pass sproxyds;
-            fastcgi_next_upstream error timeout;
-            fastcgi_send_timeout 285s;
-            fastcgi_read_timeout 285s;
-        }
-    }
-}
-

.github/docker/sproxyd/conf/sproxyd0.conf

@@ -1,12 +0,0 @@
-{
-    "general": {
-        "ring": "DATA",
-        "port": 20000,
-        "syslog_facility": "local0"
-    },
-    "ring_driver:0": {
-        "alias": "dc1",
-        "type": "local",
-        "queue_path": "/tmp/ring-objs"
-    },
-}

.github/docker/sproxyd/conf/supervisord.conf

@@ -1,43 +0,0 @@
-[supervisord]
-nodaemon = true
-loglevel = info
-logfile  = %(ENV_LOG_DIR)s/supervisord.log
-pidfile  = %(ENV_SUP_RUN_DIR)s/supervisord.pid
-logfile_maxbytes = 20MB
-logfile_backups = 2
-
-[unix_http_server]
-file = %(ENV_SUP_RUN_DIR)s/supervisor.sock
-
-[rpcinterface:supervisor]
-supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
-
-[supervisorctl]
-serverurl = unix://%(ENV_SUP_RUN_DIR)s/supervisor.sock
-
-[program:nginx]
-directory=%(ENV_SUP_RUN_DIR)s
-command=bash -c "/usr/sbin/nginx -c %(ENV_CONF_DIR)s/nginx.conf -g 'daemon off;'"
-stdout_logfile = %(ENV_LOG_DIR)s/%(program_name)s-%(process_num)s.log
-stderr_logfile = %(ENV_LOG_DIR)s/%(program_name)s-%(process_num)s-stderr.log
-stdout_logfile_maxbytes=100MB
-stdout_logfile_backups=7
-stderr_logfile_maxbytes=100MB
-stderr_logfile_backups=7
-autorestart=true
-autostart=true
-user=root
-
-[program:sproxyd]
-directory=%(ENV_SUP_RUN_DIR)s
-process_name=%(program_name)s-%(process_num)s
-numprocs=1
-numprocs_start=0
-command=/usr/bin/sproxyd -dlw -V127 -c %(ENV_CONF_DIR)s/sproxyd%(process_num)s.conf -P /run%(process_num)s
-stdout_logfile = %(ENV_LOG_DIR)s/%(program_name)s-%(process_num)s.log
-stdout_logfile_maxbytes=100MB
-stdout_logfile_backups=7
-redirect_stderr=true
-autorestart=true
-autostart=true
-user=root

.github/pykmip/certs/kmip-ca.pem

@@ -1,18 +0,0 @@
------BEGIN CERTIFICATE-----
-MIIC6zCCAdOgAwIBAgIUPIpMY95b4HjKAk+FyydZApAEFskwDQYJKoZIhvcNAQEL
-BQAwJDEQMA4GA1UECgwHU2NhbGl0eTEQMA4GA1UEAwwHUm9vdCBDQTAgFw0yMTA0
-MDkwMDI4MTFaGA8yMTIxMDMxNjAwMjgxMVowJDEQMA4GA1UECgwHU2NhbGl0eTEQ
-MA4GA1UEAwwHUm9vdCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
-AKqLFEsWtfRTxnoZrQe63tq+rQnVgninHMahRmXkzyjK/uNhoKnIh8bXdTC/eCZ6
-FBROqBYNL0TJb0HDv1FzcZS1UCUldRqTlvr6wZb0pfrp40fvztsqQgAh1t/Blg5i
-Zv5+ESSlNs5rWbFTxtq+FbMW/ERYTrVfnMkBiLg4Gq0HwID9a5jvJatzrrno2s1m
-OfZCT3HaE3tMZ6vvYuoamvLNdvdH+9KeTmBCursfNejt0rSGjIqfi6DvFJSayydQ
-is5DMSTbCLGdKQmA85VfEQmlQ8v0232WDSd6gVfp2tthDEDHnCbgWkEd1vsTyS85
-ubdt5v4CWGOWV+mu3bf8xM0CAwEAAaMTMBEwDwYDVR0TAQH/BAUwAwEB/zANBgkq
-hkiG9w0BAQsFAAOCAQEARTjc2zV/ol1/LsSzZy6l1R0uFBmR2KumH+Se1Yq2vKpY
-Dv6xmrvmjOUr5RBO77nRhIgdcQA+LyAg8ii2Dfzc8r1RTD+j1bYOxESXctBOBcXM
-Chy6FEBydR6m7S8qQyL+caJWO1WZWp2tapcm6sUG1oRVznWtK1/SHKIzOBwsmJ07
-79KsCJ6wf9tzD05EDTI2QhAObE9/thy+zc8l8cmv9A6p3jKkx9rwXUttSUqTn0CW
-w45bgKg6+DDcrhZ+MATbzuTfhuA4NFUTzK7KeX9sMuOV03Zs8SA3VhAOXmu063M3
-0f9X7P/0RmGTTp7GGCqEINcZdbLh3k7CpFb2Ox998Q==
------END CERTIFICATE-----

.github/pykmip/certs/kmip-cert.pem

@@ -1,18 +0,0 @@
------BEGIN CERTIFICATE-----
-MIIC2zCCAcOgAwIBAgIUIlE8UAkqQ+6mbJDtrt9kkmi8aJYwDQYJKoZIhvcNAQEL
-BQAwJDEQMA4GA1UECgwHU2NhbGl0eTEQMA4GA1UEAwwHUm9vdCBDQTAgFw0yMTA0
-MDkwMDI4MTFaGA8yMTIxMDMxNjAwMjgxMVowKTEQMA4GA1UECgwHU2NhbGl0eTEV
-MBMGA1UEAwwMcHlrbWlwLmxvY2FsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
-CgKCAQEAtxr7pq/lnzVeZz4z52Yc3DeaPqjNfRSyW5cPUlT7ABXFb7+tja7K2C7u
-DYVK+Q+2yJCQwYJY47aKJB++ewam9t2V8Xy0Z8S+0I2ImCwuyeihaD/f6uJZRzms
-ycdECH22BA6tCPlQLnlboRiZzI6rcIvXAbUMvLvFm3nyYIs9qidExRnfyMjISknM
-V+83LT5QW4IcHgKYqzdz2ZmOnk+f4wmMmitcivTdIZCL8Z0cxr7BJlOh5JZ/V5uj
-WUXeNa+ttW0RKKBlg9T+wj0JvwoJBPZTmsMAy3tI9tjLg3DwGYKsflbFeU2tebXI
-gncGFZ/dFxj331GGtq3kz1PzAUYf2wIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQB1
-8HgJ0fu6/pCrDxAm90eESFjmaTFyTN8q00zhq4Cb3zAT9KMWzAygkZ9n4ZFgELPo
-7kBE2H6RcDdoBmjVYd8HnBloDdYzYbncKgt5YBvxRaMSF4/l65BM8wjatyXErqnH
-QLLTRe5AuF0/F0KtPeDQ2JFVu8dZ35W3fyKGPRsEdVOSCTHROmqpGhZCpscyUP4W
-Hb0dBTESQ9mQHw14OCaaahARd0X5WdcA/E+m0fpGqj1rQCXS+PrRcSLe1E1hqPlK
-q/hXSXD5nybwipktELvJCbB7l4HmJr2pIpldeR5+ef68Cs8hqs6DRlsJX9sK2ng+
-TFe5v6SCarqZ9kFvr6Yp
------END CERTIFICATE-----

.github/pykmip/certs/kmip-client-cert.pem

@@ -1,18 +0,0 @@
------BEGIN CERTIFICATE-----
-MIIC8zCCAdugAwIBAgIUBs6nVXQXhrFbClub3aSLg72/DiYwDQYJKoZIhvcNAQEL
-BQAwJDEQMA4GA1UECgwHU2NhbGl0eTEQMA4GA1UEAwwHUm9vdCBDQTAgFw0yMTA0
-MDkwMDI4MTFaGA8yMTIxMDMxNjAwMjgxMVowJTEQMA4GA1UECgwHU2NhbGl0eTER
-MA8GA1UEAwwISm9obiBEb2UwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
-AQC6neSYoBoWh/i2mBpduJnTlXacpJ0iQqLezvcGy8qR0s/48mtfV2IRGTNVsq4L
-jLLRsPGt9KkJlUhHGWhG00cBGEsIiJiBUr+WrEsO04ME/Sk76kX8wk/t9Oljl7jt
-UDnQUwshj+hRFe0iKAyE65JIutu5EiiNtOqMzbVgPNfNniAaGlrgwByJaS9arzsH
-PVju9yZBYzYhwAMyYFcXUGrgvHRCHKmxBi4QmV7DX4TeN4l9TrCyEmqDev4PRFip
-yR2Fh3WGSwWh45HgMT+Jp6Uv6yI4wMXWJAcNkHdx1OhjBoUQrkavvdeVEnCwjQ+p
-SMLm0T4iNxedQWBtDM7ts4EjAgMBAAGjGjAYMBYGA1UdJQEB/wQMMAoGCCsGAQUF
-BwMCMA0GCSqGSIb3DQEBCwUAA4IBAQCMi9HEhZc5jHJMj18Wq00fZy4O9XtjCe0J
-nntW9tzi3rTQcQWKA7i9uVdDoCg+gMFVxWMvV7luFEUc/VYV1v8hFfbIFygzFsZY
-xwv4GQaIwbsgzD+oziia53w0FSuNL0uE0MeKvrt3yzHxCxylHyl+TQd/UdAtAo+k
-RL1sI0mBZx5qo6d1J7ZMCxzAGaT7KjnJvziFr/UbfSNnwDsxsUwGaI1ZeAxJN8DI
-zTrg3f3lrrmHcauEgKnuQwIqaMZR6veG6RkjtcYSlJYID1irkE6njs7+wivOAkzt
-fBt/0PD76FmAI0VArgU/zDB8dGyYzrq39W749LuEfm1TPmlnUtDr
------END CERTIFICATE-----

.github/pykmip/certs/kmip-client-key.pem

@@ -1,28 +0,0 @@
------BEGIN PRIVATE KEY-----
-MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC6neSYoBoWh/i2
-mBpduJnTlXacpJ0iQqLezvcGy8qR0s/48mtfV2IRGTNVsq4LjLLRsPGt9KkJlUhH
-GWhG00cBGEsIiJiBUr+WrEsO04ME/Sk76kX8wk/t9Oljl7jtUDnQUwshj+hRFe0i
-KAyE65JIutu5EiiNtOqMzbVgPNfNniAaGlrgwByJaS9arzsHPVju9yZBYzYhwAMy
-YFcXUGrgvHRCHKmxBi4QmV7DX4TeN4l9TrCyEmqDev4PRFipyR2Fh3WGSwWh45Hg
-MT+Jp6Uv6yI4wMXWJAcNkHdx1OhjBoUQrkavvdeVEnCwjQ+pSMLm0T4iNxedQWBt
-DM7ts4EjAgMBAAECggEANNXdUeUKXdSzcycPV/ea/c+0XFcy8e9B46lfQTpTqQOx
-xD8GbWD1L/gdk6baJgT43+ukEWdSsJbmdtLXti29Ta8OF2VtIDhIbCVtvs3dq3zt
-vrvugsiVDr8nkP306qOrKrNIVIFE+igmEmSaXsu/h/33ladxeeV9/s2DC7NOOjWN
-Mu4KYr5BBbu3qAavdzbrcz7Sch+GzsYqK/pBounCTQu3o9E4TSUcmcsasWmtHN3u
-e6G2UjObdzEW7J0wWvvtJ0wHQUVRueHfqwqKf0dymcZ3xOlx3ZPhKPz5n4F1UGUt
-RQaNazqs5SzZpUgDuPw4k8h/aCHK21Yexw/l4+O9KQKBgQD1WZSRK54zFoExBQgt
-OZSBNZW3Ibti5lSiF0M0g+66yNZSWfPuABEH0tu5CXopdPDXo4kW8NLGEqQStWTX
-RGK0DE9buEL3eebOfjIdS2IZ3t3dX3lMypplVCj4HzAgITlweSH1LLTyAtaaOpwa
-jksqfcn5Zw+XGkyc6GBBVaZetQKBgQDCt6Xf/g26+zjvHscjdzsfBhnYvTOrr6+F
-xqFFxOEOocGr+mL7UTAs+a9m/6lOWhlagk+m+TIZNL8o3IN7KFTYxPYPxTiewgVE
-rIm3JBmPxRiPn01P3HrtjaqfzsXF30j3ele7ix5OxieZq4vsW7ZXP3GZE34a08Ov
-12sE1DlvdwKBgQDzpYQOLhyqazzcqzyVfMrnDYmiFVN7QXTmiudobWRUBUIhAcdl
-oJdJB7K/rJOuO704x+RJ7dnCbZyWH6EGzZifaGIemXuXO21jvpqR0NyZCGOXhUp2
-YfS1j8AntwEZxyS9du2sBjui4gKvomiHTquChOxgSmKHEcznPTTpbN8MyQKBgF5F
-LVCZniolkLXsL7tS8VOez4qoZ0i6wP7CYLf3joJX+/z4N023S9yqcaorItvlMRsp
-tciAIyoi6F2vDRTmPNXJ3dtav4PVKVnLMs1w89MwOCjoljSQ6Q7zpGTEZenbpWbz
-W2BYBS9cLjXu4MpoyInLFINo9YeleLs8TvrCiKAXAoGBANsduqLnlUW/f5zDb5Fe
-SB51+KhBjsVIeYmU+8xtur9Z7IxZXK28wpoEsm7LmX7Va5dERjI+tItBiJ5+Unu1
-Xs2ljDg35ARKHs0dWBJGpbnZg4dbT6xpIL4YMPXm1Zu++PgRpxPIMn646xqd8GlH
-bavm6Km/fXNG58xus+EeLpV5
------END PRIVATE KEY-----

.github/pykmip/certs/kmip-key.pem

@@ -1,28 +0,0 @@
------BEGIN PRIVATE KEY-----
-MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQC3Gvumr+WfNV5n
-PjPnZhzcN5o+qM19FLJblw9SVPsAFcVvv62NrsrYLu4NhUr5D7bIkJDBgljjtook
-H757Bqb23ZXxfLRnxL7QjYiYLC7J6KFoP9/q4llHOazJx0QIfbYEDq0I+VAueVuh
-GJnMjqtwi9cBtQy8u8WbefJgiz2qJ0TFGd/IyMhKScxX7zctPlBbghweApirN3PZ
-mY6eT5/jCYyaK1yK9N0hkIvxnRzGvsEmU6Hkln9Xm6NZRd41r621bREooGWD1P7C
-PQm/CgkE9lOawwDLe0j22MuDcPAZgqx+VsV5Ta15tciCdwYVn90XGPffUYa2reTP
-U/MBRh/bAgMBAAECggEABCvcMcbuDztzBB0Zp5re63Fk1SqZS9Et4wJE+hYvhaf5
-UHtoY8LoohYnnC0+MQBXpKgOdCoZBk8BRKNofnr/UL5pjQ/POFH2GuAujXDsO/NN
-wgc6fapcaE/7DLm6ZgsfG2aOMJclaXmgScI6trtFUpIM+t/6A06vyMP1bpeddwPW
-Fqu7NvpDiEcTRUGd+z1JooYgUhGgC7peYUx5+9zqFrwoDBKxnUOnz3BkDsXBy3qm
-65Vu0BSjuJzf6vVMpNGUHY6JXjopVNWku+JAX0wD+iikOd5sziNVdIj1fnZ+IHIf
-7G5h5owHpvSGzJFQ18/g5VHtJdCm+4WQSnbSJRsCAQKBgQDu4IH8yspyeH44fhoS
-PAp/OtILqSP+Da0zAp2LbhrOgyzyuSTdEAYyptqjqHS6QkB1Bu1H44FS0BYUxRXc
-iu2e9AndiLVCGngsE7TpA/ZVLN1B0LEZEHjM6p4d6zZM6iveKVnPAOkTWTBAgzCt
-b31nj4jL8PdlPKQil1AMrOlRAQKBgQDEOwshzIdr2Iy6B/n4CuBViEtwnbAd5f/c
-atA9bcfF8kCahokJsI4eCCLgBwDZpYKD+v0AwOBlacF6t6TX+vdlJsi5EP7uxZ22
-ILsuWqVm/0H77PACuckc5/qLZoGGC81l0DhnpoeMEb6r/TKOo5xAK1gxdlwNNrq+
-nP1zdZnU2wKBgBAS92xFUR4m0YeHpMV5WNN658t1FEDyNqdqE6PgQtmGpi2nG73s
-aB5cb/X3TfOCpce6MZlWy8sAyZuYL4Jprte1YDySCHBsS43bvZ64b4kHvdPB8UjY
-fOh9GSq2Oy8tysnmSm7NhuGQbNjKeyoQiIXBeNkQW/VqATl6qR5RPFoBAoGACNqV
-JQBCd/Y8W0Ry3eM3vgQ5SyqCQMcY5UwYez0Rz3efvJknY72InAhH8o2+VxOlsOjJ
-M5iAR3MfHLdeg7Q6J2E5m0gOCJ34ALi3WV8TqXMI+iH1rlnNnjVFU7bbTz4HFXnw
-oZSc9w/x53a0KkVtjmOmRg0OGDaI9ILG2MfMmhMCgYB8ZqJtX8qZ2TqKU3XdLZ4z
-T2N7xMFuKohWP420r5jKm3Xw85IC+y1SUTB9XGcL79r2eJzmzmdKQ3A3sf3oyUH3
-RdYWxtKcZ5PAE8hVRtn1ETZqUgxASGOUn/6w0npkYSOXPU5bc0W6RSLkjES0i+c3
-fv3OMNI8qpmQhEjpHHQS1g==
------END PRIVATE KEY-----

.github/workflows/alerts.yaml

@@ -1,35 +0,0 @@
-name: Test alerts
-
-on:
-  push:
-    branches-ignore:
-      - 'development/**'
-      - 'q/*/**'
-
-jobs:
-  run-alert-tests:
-    runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        tests:
-          - name: 1 minute interval tests
-            file: monitoring/alerts.test.yaml
-          
-          - name: 10 seconds interval tests
-            file: monitoring/alerts.10s.test.yaml
-
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-
-      - name: Render and test ${{ matrix.tests.name }}
-        uses: scality/action-prom-render-test@1.0.3
-        with:
-          alert_file_path: monitoring/alerts.yaml
-          test_file_path: ${{ matrix.tests.file }}
-          alert_inputs: |
-            namespace=zenko
-            service=artesca-data-connector-s3api-metrics
-            reportJob=artesca-data-ops-report-handler
-            replicas=3
-          github_token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/codeql.yaml

@@ -1,25 +0,0 @@
----
-name: codeQL
-
-on:
-  push:
-    branches: [w/**, q/*]
-  pull_request:
-    branches: [development/*, stabilization/*, hotfix/*]
-  workflow_dispatch:
-
-jobs:
-  analyze:
-    name: Static analysis with CodeQL
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-
-      - name: Initialize CodeQL
-        uses: github/codeql-action/init@v3
-        with:
-          languages: javascript, python, ruby
-
-      - name: Build and analyze
-        uses: github/codeql-action/analyze@v3

.github/workflows/dependency-review.yaml

@@ -1,16 +0,0 @@
----
-name: dependency review
-
-on:
-  pull_request:
-    branches: [development/*, stabilization/*, hotfix/*]
-
-jobs:
-  dependency-review:
-    runs-on: ubuntu-latest
-    steps:
-      - name: 'Checkout Repository'
-        uses: actions/checkout@v4
-
-      - name: 'Dependency Review'
-        uses: actions/dependency-review-action@v4

.github/workflows/release.yaml

@@ -1,80 +0,0 @@
----
-name: release
-run-name: release ${{ inputs.tag }}
-
-on:
-  workflow_dispatch:
-    inputs:
-      tag:
-        description: 'Tag to be released'
-        required: true
-
-env:
-  PROJECT_NAME: ${{ github.event.repository.name }}
-
-jobs:
-  build-federation-image:
-    runs-on: ubuntu-20.04
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
-      - name: Login to GitHub Registry
-        uses: docker/login-action@v3
-        with:
-          registry: ghcr.io
-          username: ${{ github.repository_owner }}
-          password: ${{ github.token }}
-      - name: Build and push image for federation
-        uses: docker/build-push-action@v5
-        with:
-          push: true
-          context: .
-          file: images/svc-base/Dockerfile
-          tags: |
-            ghcr.io/${{ github.repository }}:${{ github.event.inputs.tag }}-svc-base
-          cache-from: type=gha,scope=federation
-          cache-to: type=gha,mode=max,scope=federation
-
-  release:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-
-      - name: Set up Docker Buildk
-        uses: docker/setup-buildx-action@v3
-
-      - name: Login to Registry
-        uses: docker/login-action@v3
-        with:
-          registry: ghcr.io
-          username: ${{ github.repository_owner }}
-          password: ${{ github.token }}
-
-      - name: Push dashboards into the production namespace
-        run: |
-          oras push ghcr.io/${{ github.repository }}/${{ env.PROJECT_NAME }}-dashboards:${{ github.event.inputs.tag }} \
-            dashboard.json:application/grafana-dashboard+json \
-            alerts.yaml:application/prometheus-alerts+yaml
-        working-directory: monitoring
-
-      - name: Build and push
-        uses: docker/build-push-action@v5
-        with:
-          context: .
-          push: true
-          tags: ghcr.io/${{ github.repository }}:${{ github.event.inputs.tag }}
-          cache-from: type=gha
-          cache-to: type=gha,mode=max
-
-      - name: Create Release
-        uses: softprops/action-gh-release@v2
-        env:
-          GITHUB_TOKEN: ${{ github.token }}
-        with:
-          name: Release ${{ github.event.inputs.tag }}
-          tag_name: ${{ github.event.inputs.tag }}
-          generate_release_notes: true
-          target_commitish: ${{ github.sha }}

.github/workflows/tests.yaml

@@ -1,533 +0,0 @@
----
-name: tests
-
-on:
-  workflow_dispatch:
-
-  push:
-    branches-ignore:
-      - 'development/**'
-      - 'q/*/**'
-
-env:
-  # Secrets
-  azurebackend_AZURE_STORAGE_ACCESS_KEY: >-
-    ${{ secrets.AZURE_STORAGE_ACCESS_KEY }}
-  azurebackend_AZURE_STORAGE_ACCOUNT_NAME: >-
-    ${{ secrets.AZURE_STORAGE_ACCOUNT_NAME }}
-  azurebackend_AZURE_STORAGE_ENDPOINT: >-
-    ${{ secrets.AZURE_STORAGE_ENDPOINT }}
-  azurebackend2_AZURE_STORAGE_ACCESS_KEY: >-
-    ${{ secrets.AZURE_STORAGE_ACCESS_KEY_2 }}
-  azurebackend2_AZURE_STORAGE_ACCOUNT_NAME: >-
-    ${{ secrets.AZURE_STORAGE_ACCOUNT_NAME_2 }}
-  azurebackend2_AZURE_STORAGE_ENDPOINT: >-
-    ${{ secrets.AZURE_STORAGE_ENDPOINT_2 }}
-  azurebackendmismatch_AZURE_STORAGE_ACCESS_KEY: >-
-    ${{ secrets.AZURE_STORAGE_ACCESS_KEY }}
-  azurebackendmismatch_AZURE_STORAGE_ACCOUNT_NAME: >-
-    ${{ secrets.AZURE_STORAGE_ACCOUNT_NAME }}
-  azurebackendmismatch_AZURE_STORAGE_ENDPOINT: >-
-    ${{ secrets.AZURE_STORAGE_ENDPOINT }}
-  azurenonexistcontainer_AZURE_STORAGE_ACCESS_KEY: >-
-    ${{ secrets.AZURE_STORAGE_ACCESS_KEY }}
-  azurenonexistcontainer_AZURE_STORAGE_ACCOUNT_NAME: >-
-    ${{ secrets.AZURE_STORAGE_ACCOUNT_NAME }}
-  azurenonexistcontainer_AZURE_STORAGE_ENDPOINT: >-
-    ${{ secrets.AZURE_STORAGE_ENDPOINT }}
-  azuretest_AZURE_BLOB_ENDPOINT: "${{ secrets.AZURE_STORAGE_ENDPOINT }}"
-  b2backend_B2_ACCOUNT_ID: "${{ secrets.B2BACKEND_B2_ACCOUNT_ID }}"
-  b2backend_B2_STORAGE_ACCESS_KEY: >-
-    ${{ secrets.B2BACKEND_B2_STORAGE_ACCESS_KEY }}
-  GOOGLE_SERVICE_EMAIL: "${{ secrets.GCP_SERVICE_EMAIL }}"
-  GOOGLE_SERVICE_KEY: "${{ secrets.GCP_SERVICE_KEY }}"
-  AWS_S3_BACKEND_ACCESS_KEY: "${{ secrets.AWS_S3_BACKEND_ACCESS_KEY }}"
-  AWS_S3_BACKEND_SECRET_KEY: "${{ secrets.AWS_S3_BACKEND_SECRET_KEY }}"
-  AWS_S3_BACKEND_ACCESS_KEY_2: "${{ secrets.AWS_S3_BACKEND_ACCESS_KEY_2 }}"
-  AWS_S3_BACKEND_SECRET_KEY_2: "${{ secrets.AWS_S3_BACKEND_SECRET_KEY_2 }}"
-  AWS_GCP_BACKEND_ACCESS_KEY: "${{ secrets.AWS_GCP_BACKEND_ACCESS_KEY }}"
-  AWS_GCP_BACKEND_SECRET_KEY: "${{ secrets.AWS_GCP_BACKEND_SECRET_KEY }}"
-  AWS_GCP_BACKEND_ACCESS_KEY_2: "${{ secrets.AWS_GCP_BACKEND_ACCESS_KEY_2 }}"
-  AWS_GCP_BACKEND_SECRET_KEY_2: "${{ secrets.AWS_GCP_BACKEND_SECRET_KEY_2 }}"
-  b2backend_B2_STORAGE_ENDPOINT: "${{ secrets.B2BACKEND_B2_STORAGE_ENDPOINT }}"
-  gcpbackend2_GCP_SERVICE_EMAIL: "${{ secrets.GCP2_SERVICE_EMAIL }}"
-  gcpbackend2_GCP_SERVICE_KEY: "${{ secrets.GCP2_SERVICE_KEY }}"
-  gcpbackend2_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
-  gcpbackend_GCP_SERVICE_EMAIL: "${{ secrets.GCP_SERVICE_EMAIL }}"
-  gcpbackend_GCP_SERVICE_KEY: "${{ secrets.GCP_SERVICE_KEY }}"
-  gcpbackendmismatch_GCP_SERVICE_EMAIL: >-
-    ${{ secrets.GCPBACKENDMISMATCH_GCP_SERVICE_EMAIL }}
-  gcpbackendmismatch_GCP_SERVICE_KEY: >-
-    ${{ secrets.GCPBACKENDMISMATCH_GCP_SERVICE_KEY }}
-  gcpbackend_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
-  gcpbackendmismatch_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
-  gcpbackendnoproxy_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
-  gcpbackendproxy_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
-  # Configs
-  ENABLE_LOCAL_CACHE: "true"
-  REPORT_TOKEN: "report-token-1"
-  REMOTE_MANAGEMENT_DISABLE: "1"
-  # https://github.com/git-lfs/git-lfs/issues/5749
-  GIT_CLONE_PROTECTION_ACTIVE: 'false'
-jobs:
-  linting-coverage:
-    runs-on: ubuntu-latest
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v4
-    - uses: actions/setup-node@v4
-      with:
-        node-version: '16'
-        cache: yarn
-    - name: install dependencies
-      run: yarn install --frozen-lockfile --network-concurrency 1
-    - uses: actions/setup-python@v5
-      with:
-        python-version: '3.9'
-    - uses: actions/cache@v4
-      with:
-        path: ~/.cache/pip
-        key: ${{ runner.os }}-pip
-    - name: Install python deps
-      run: pip install flake8
-    - name: Lint Javascript
-      run: yarn run --silent lint -- --max-warnings 0
-    - name: Lint Markdown
-      run: yarn run --silent lint_md
-    - name: Lint python
-      run: flake8 $(git ls-files "*.py")
-    - name: Lint Yaml
-      run: yamllint -c yamllint.yml $(git ls-files "*.yml")
-    - name: Unit Coverage
-      run: |
-        set -ex
-        mkdir -p $CIRCLE_TEST_REPORTS/unit
-        yarn test
-        yarn run test_legacy_location
-      env:
-        S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
-        CIRCLE_TEST_REPORTS: /tmp
-        CIRCLE_ARTIFACTS: /tmp
-        CI_REPORTS: /tmp
-    - name: Unit Coverage logs
-      run: find /tmp/unit -exec cat {} \;
-    - name: preparing junit files for upload
-      run: |
-        mkdir -p artifacts/junit
-        find . -name "*junit*.xml" -exec cp {} artifacts/junit/ ";"
-      if: always()
-    - name: Upload files to artifacts
-      uses: scality/action-artifacts@v4
-      with:
-        method: upload
-        url: https://artifacts.scality.net
-        user: ${{ secrets.ARTIFACTS_USER }}
-        password: ${{ secrets.ARTIFACTS_PASSWORD }}
-        source: artifacts
-      if: always()
-
-  build:
-    runs-on: ubuntu-20.04
-    permissions:
-      contents: read
-      packages: write
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
-      - name: Login to GitHub Registry
-        uses: docker/login-action@v3
-        with:
-          registry: ghcr.io
-          username: ${{ github.repository_owner }}
-          password: ${{ github.token }}
-      - name: Build and push cloudserver image
-        uses: docker/build-push-action@v5
-        with:
-          push: true
-          context: .
-          provenance: false
-          tags: |
-            ghcr.io/${{ github.repository }}:${{ github.sha }}
-          labels: |
-            git.repository=${{ github.repository }}
-            git.commit-sha=${{ github.sha }}
-          cache-from: type=gha,scope=cloudserver
-          cache-to: type=gha,mode=max,scope=cloudserver
-      - name: Build and push pykmip image
-        uses: docker/build-push-action@v5
-        with:
-          push: true
-          context: .github/pykmip
-          tags: |
-            ghcr.io/${{ github.repository }}/pykmip:${{ github.sha }}
-          labels: |
-            git.repository=${{ github.repository }}
-            git.commit-sha=${{ github.sha }}
-          cache-from: type=gha,scope=pykmip
-          cache-to: type=gha,mode=max,scope=pykmip
-      - name: Build and push MongoDB
-        uses: docker/build-push-action@v5
-        with:
-          push: true
-          context: .github/docker/mongodb
-          tags: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
-          cache-from: type=gha,scope=mongodb
-          cache-to: type=gha,mode=max,scope=mongodb
-
-  multiple-backend:
-    runs-on: ubuntu-latest
-    needs: build
-    env:
-      CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
-      MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
-      S3BACKEND: mem
-      S3_LOCATION_FILE: /usr/src/app/tests/locationConfig/locationConfigTests.json
-      S3DATA: multiple
-      JOB_NAME: ${{ github.job }}
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v4
-    - name: Login to Registry
-      uses: docker/login-action@v3
-      with:
-        registry: ghcr.io
-        username: ${{ github.repository_owner }}
-        password: ${{ github.token }}
-    - name: Setup CI environment
-      uses: ./.github/actions/setup-ci
-    - name: Setup CI services
-      run: docker compose --profile sproxyd up -d
-      working-directory: .github/docker
-    - name: Run multiple backend test
-      run: |-
-        set -o pipefail;
-        bash wait_for_local_port.bash 8000 40
-        bash wait_for_local_port.bash 81 40
-        yarn run multiple_backend_test | tee /tmp/artifacts/${{ github.job }}/tests.log
-      env:
-        S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
-    - name: Upload logs to artifacts
-      uses: scality/action-artifacts@v4
-      with:
-        method: upload
-        url: https://artifacts.scality.net
-        user: ${{ secrets.ARTIFACTS_USER }}
-        password: ${{ secrets.ARTIFACTS_PASSWORD }}
-        source: /tmp/artifacts
-      if: always()
- 
-  mongo-v0-ft-tests:
-    runs-on: ubuntu-latest
-    needs: build
-    env:
-      S3BACKEND: mem
-      MPU_TESTING: "yes" 
-      S3METADATA: mongodb
-      S3KMS: file
-      S3_LOCATION_FILE: /usr/src/app/tests/locationConfig/locationConfigTests.json
-      DEFAULT_BUCKET_KEY_FORMAT: v0
-      MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
-      CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
-      JOB_NAME: ${{ github.job }}
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v4
-    - name: Setup CI environment
-      uses: ./.github/actions/setup-ci
-    - name: Setup CI services
-      run: docker compose --profile mongo up -d
-      working-directory: .github/docker
-    - name: Run functional tests
-      run: |-
-        set -o pipefail;
-        bash wait_for_local_port.bash 8000 40
-        yarn run ft_test | tee /tmp/artifacts/${{ github.job }}/tests.log
-      env:
-        S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
-    - name: Upload logs to artifacts
-      uses: scality/action-artifacts@v4
-      with:
-        method: upload
-        url: https://artifacts.scality.net
-        user: ${{ secrets.ARTIFACTS_USER }}
-        password: ${{ secrets.ARTIFACTS_PASSWORD }}
-        source: /tmp/artifacts
-      if: always()
-
-  mongo-v1-ft-tests:
-    runs-on: ubuntu-latest
-    needs: build
-    env:
-      S3BACKEND: mem
-      MPU_TESTING: "yes" 
-      S3METADATA: mongodb
-      S3KMS: file
-      S3_LOCATION_FILE: /usr/src/app/tests/locationConfig/locationConfigTests.json
-      DEFAULT_BUCKET_KEY_FORMAT: v1
-      METADATA_MAX_CACHED_BUCKETS: 1
-      MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
-      CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
-      JOB_NAME: ${{ github.job }}
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v4
-    - name: Setup CI environment
-      uses: ./.github/actions/setup-ci
-    - name: Setup CI services
-      run: docker compose --profile mongo up -d
-      working-directory: .github/docker
-    - name: Run functional tests
-      run: |-
-        set -o pipefail;        
-        bash wait_for_local_port.bash 8000 40
-        yarn run ft_test | tee /tmp/artifacts/${{ github.job }}/tests.log
-        yarn run ft_mixed_bucket_format_version | tee /tmp/artifacts/${{ github.job }}/mixed-tests.log
-      env:
-        S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
-    - name: Upload logs to artifacts
-      uses: scality/action-artifacts@v4
-      with:
-        method: upload
-        url: https://artifacts.scality.net
-        user: ${{ secrets.ARTIFACTS_USER }}
-        password: ${{ secrets.ARTIFACTS_PASSWORD }}
-        source: /tmp/artifacts
-      if: always()
-
-  file-ft-tests:
-    strategy:
-      matrix:
-        include:
-          - job-name: file-ft-tests
-    name: ${{ matrix.job-name }}
-    runs-on: ubuntu-latest
-    needs: build
-    env:
-      S3BACKEND: file
-      S3VAULT: mem
-      CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
-      MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
-      MPU_TESTING: "yes"
-      JOB_NAME: ${{ matrix.job-name }}
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v4
-    - name: Setup CI environment
-      uses: ./.github/actions/setup-ci
-    - name: Setup matrix job artifacts directory
-      shell: bash
-      run: |
-        set -exu
-        mkdir -p /tmp/artifacts/${{ matrix.job-name }}/
-    - name: Setup CI services
-      run: docker compose up -d
-      working-directory: .github/docker
-    - name: Run file ft tests
-      run: |-
-        set -o pipefail;
-        bash wait_for_local_port.bash 8000 40
-        yarn run ft_test | tee /tmp/artifacts/${{ matrix.job-name }}/tests.log
-    - name: Upload logs to artifacts
-      uses: scality/action-artifacts@v4
-      with:
-        method: upload
-        url: https://artifacts.scality.net
-        user: ${{ secrets.ARTIFACTS_USER }}
-        password: ${{ secrets.ARTIFACTS_PASSWORD }}
-        source: /tmp/artifacts
-      if: always()
-
-  utapi-v2-tests:
-    runs-on: ubuntu-latest
-    needs: build
-    env:
-      ENABLE_UTAPI_V2: t
-      S3BACKEND: mem
-      BUCKET_DENY_FILTER: utapi-event-filter-deny-bucket
-      CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
-      MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
-      JOB_NAME: ${{ github.job }}
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v4
-    - name: Setup CI environment
-      uses: ./.github/actions/setup-ci
-    - name: Setup CI services
-      run: docker compose up -d
-      working-directory: .github/docker
-    - name: Run file utapi v2 tests
-      run: |-
-        set -ex -o pipefail;
-        bash wait_for_local_port.bash 8000 40
-        yarn run test_utapi_v2 | tee /tmp/artifacts/${{ github.job }}/tests.log
-    - name: Upload logs to artifacts
-      uses: scality/action-artifacts@v4
-      with:
-        method: upload
-        url: https://artifacts.scality.net
-        user: ${{ secrets.ARTIFACTS_USER }}
-        password: ${{ secrets.ARTIFACTS_PASSWORD }}
-        source: /tmp/artifacts
-      if: always()
-
-  quota-tests:
-    runs-on: ubuntu-latest
-    needs: build
-    strategy:
-      matrix:
-        inflights:
-          - name: "With Inflights"
-            value: "true"
-          - name: "Without Inflights"
-            value: "false"
-    env:
-      S3METADATA: mongodb
-      S3BACKEND: mem
-      S3QUOTA: scuba
-      QUOTA_ENABLE_INFLIGHTS: ${{ matrix.inflights.value }}
-      SCUBA_HOST: localhost
-      SCUBA_PORT: 8100
-      SCUBA_HEALTHCHECK_FREQUENCY: 100
-      CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
-      MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
-      JOB_NAME: ${{ github.job }}
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v4
-    - name: Setup CI environment
-      uses: ./.github/actions/setup-ci
-    - name: Setup CI services
-      run: docker compose --profile mongo up -d
-      working-directory: .github/docker
-    - name: Run quota tests
-      run: |-
-        set -ex -o pipefail;
-        bash wait_for_local_port.bash 8000 40
-        yarn run test_quota | tee /tmp/artifacts/${{ github.job }}/tests.log
-    - name: Upload logs to artifacts
-      uses: scality/action-artifacts@v4
-      with:
-        method: upload
-        url: https://artifacts.scality.net
-        user: ${{ secrets.ARTIFACTS_USER }}
-        password: ${{ secrets.ARTIFACTS_PASSWORD }}
-        source: /tmp/artifacts
-      if: always()
-
-  kmip-ft-tests:
-    runs-on: ubuntu-latest
-    needs: build
-    env:
-      S3BACKEND: file
-      S3VAULT: mem
-      MPU_TESTING: "yes" 
-      CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
-      PYKMIP_IMAGE: ghcr.io/${{ github.repository }}/pykmip:${{ github.sha }}
-      MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
-      JOB_NAME: ${{ github.job }}
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v4
-    - name: Setup CI environment
-      uses: ./.github/actions/setup-ci
-    - name: Copy KMIP certs
-      run: cp -r ./certs /tmp/ssl-kmip
-      working-directory: .github/pykmip
-    - name: Setup CI services
-      run: docker compose --profile pykmip up -d
-      working-directory: .github/docker
-    - name: Run file KMIP tests
-      run: |-
-        set -ex -o pipefail;
-        bash wait_for_local_port.bash 8000 40
-        bash wait_for_local_port.bash 5696 40
-        yarn run ft_kmip | tee /tmp/artifacts/${{ github.job }}/tests.log
-    - name: Upload logs to artifacts
-      uses: scality/action-artifacts@v4
-      with:
-        method: upload
-        url: https://artifacts.scality.net
-        user: ${{ secrets.ARTIFACTS_USER }}
-        password: ${{ secrets.ARTIFACTS_PASSWORD }}
-        source: /tmp/artifacts
-      if: always()
-
-  ceph-backend-test:
-    runs-on: ubuntu-latest
-    needs: build
-    env:
-      S3BACKEND: mem
-      S3DATA: multiple
-      S3KMS: file
-      CI_CEPH: 'true'
-      MPU_TESTING: "yes" 
-      S3_LOCATION_FILE: /usr/src/app/tests/locationConfig/locationConfigCeph.json
-      MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
-      CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
-      JOB_NAME: ${{ github.job }}
-    steps:
-    - name: Checkout
-      uses: actions/checkout@v4
-    - name: Login to GitHub Registry
-      uses: docker/login-action@v3
-      with:
-        registry: ghcr.io
-        username: ${{ github.repository_owner }}
-        password: ${{ github.token }}
-    - name: Setup CI environment
-      uses: ./.github/actions/setup-ci
-    - uses: ruby/setup-ruby@v1
-      with:
-        ruby-version: '2.5.9'
-    - name: Install Ruby dependencies
-      run: |
-        gem install nokogiri:1.12.5 excon:0.109.0 fog-aws:1.3.0 json mime-types:3.1 rspec:3.5
-    - name: Install Java dependencies
-      run: |
-        sudo apt-get update && sudo apt-get install -y --fix-missing default-jdk maven
-    - name: Setup CI services
-      run: docker compose --profile ceph up -d
-      working-directory: .github/docker
-      env: 
-        S3METADATA: mongodb
-    - name: Run Ceph multiple backend tests
-      run: |-
-        set -ex -o pipefail;
-        bash .github/ceph/wait_for_ceph.sh
-        bash wait_for_local_port.bash 27018 40
-        bash wait_for_local_port.bash 8000 40
-        yarn run multiple_backend_test | tee /tmp/artifacts/${{ github.job }}/multibackend-tests.log
-      env:
-        S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
-        S3METADATA: mem
-    - name: Run Java tests
-      run: |-
-        set -ex -o pipefail;
-        mvn test | tee /tmp/artifacts/${{ github.job }}/java-tests.log
-      working-directory: tests/functional/jaws
-    - name: Run Ruby tests
-      run: |-
-        set -ex -o pipefail;
-        rspec -fd --backtrace tests.rb | tee /tmp/artifacts/${{ github.job }}/ruby-tests.log
-      working-directory: tests/functional/fog
-    - name: Run Javascript AWS SDK tests
-      run: |-
-        set -ex -o pipefail;
-        yarn run ft_awssdk | tee /tmp/artifacts/${{ github.job }}/js-awssdk-tests.log;
-        yarn run ft_s3cmd | tee /tmp/artifacts/${{ github.job }}/js-s3cmd-tests.log;
-      env:
-        S3_LOCATION_FILE: tests/locationConfig/locationConfigCeph.json
-        S3BACKEND: file
-        S3VAULT: mem
-        S3METADATA: mongodb
-    - name: Upload logs to artifacts
-      uses: scality/action-artifacts@v4
-      with:
-        method: upload
-        url: https://artifacts.scality.net
-        user: ${{ secrets.ARTIFACTS_USER }}
-        password: ${{ secrets.ARTIFACTS_PASSWORD }}
-        source: /tmp/artifacts
-      if: always()

.gitignore

@@ -22,14 +22,6 @@ coverage
 # Compiled binary addons (http://nodejs.org/api/addons.html)
 build/Release
 
-# Sphinx build dir
-_build
-
 # Dependency directory
 # https://www.npmjs.org/doc/misc/npm-faq.html#should-i-check-my-node_modules-folder-into-git
 node_modules
-yarn.lock
-.tox
-
-# Junit directory
-junit

Dockerfile

@@ -1,60 +0,0 @@
-ARG NODE_VERSION=16.20-bullseye-slim
-
-FROM node:${NODE_VERSION} as builder
-
-WORKDIR /usr/src/app
-
-RUN apt-get update \
-    && apt-get install -y --no-install-recommends \
-        build-essential \
-        ca-certificates \
-        curl \
-        git \
-        gnupg2 \
-        jq \
-        python3 \
-        ssh \
-        wget \
-        libffi-dev \
-        zlib1g-dev \
-    && apt-get clean \
-    && mkdir -p /root/ssh \
-    && ssh-keyscan -H github.com > /root/ssh/known_hosts
-
-ENV PYTHON=python3
-COPY package.json yarn.lock /usr/src/app/
-RUN npm install typescript -g
-RUN yarn install --production --ignore-optional --frozen-lockfile --ignore-engines --network-concurrency 1
-
-################################################################################
-FROM node:${NODE_VERSION}
-
-RUN apt-get update && \
-    apt-get install -y --no-install-recommends \
-        jq \
-    && rm -rf /var/lib/apt/lists/*
-
-ENV NO_PROXY localhost,127.0.0.1
-ENV no_proxy localhost,127.0.0.1
-
-EXPOSE 8000
-EXPOSE 8002
-
-RUN apt-get update && \
-    apt-get install -y --no-install-recommends \
-        jq \
-        tini \
-    && rm -rf /var/lib/apt/lists/*
-
-WORKDIR /usr/src/app
-
-# Keep the .git directory in order to properly report version
-COPY . /usr/src/app
-COPY --from=builder /usr/src/app/node_modules ./node_modules/
-
-
-VOLUME ["/usr/src/app/localData","/usr/src/app/localMetadata"]
-
-ENTRYPOINT ["tini", "--", "/usr/src/app/docker-entrypoint.sh"]
-
-CMD [ "yarn", "start" ]

README.md

@@ -1,7 +1,12 @@
-# Zenko CloudServer with Vitastor Backend
+# Zenko CloudServer
 
 ![Zenko CloudServer logo](res/scality-cloudserver-logo.png)
 
+[![CircleCI][badgepub]](https://circleci.com/gh/scality/S3)
+[![Scality CI][badgepriv]](http://ci.ironmann.io/gh/scality/S3)
+[![Docker Pulls][badgedocker]](https://hub.docker.com/r/scality/s3server/)
+[![Docker Pulls][badgetwitter]](https://twitter.com/zenko)
+
 ## Overview
 
 CloudServer (formerly S3 Server) is an open-source Amazon S3-compatible
@@ -11,71 +16,125 @@ Scality’s Open Source Multi-Cloud Data Controller.
 CloudServer provides a single AWS S3 API interface to access multiple
 backend data storage both on-premise or public in the cloud.
 
-This repository contains a fork of CloudServer with [Vitastor](https://git.yourcmc.ru/vitalif/vitastor)
+CloudServer is useful for Developers, either to run as part of a
-backend support.
+continous integration test environment to emulate the AWS S3 service locally
+or as an abstraction layer to develop object storage enabled
+application on the go.
 
-## Quick Start with Vitastor
+## Learn more at [www.zenko.io/cloudserver](https://www.zenko.io/cloudserver/)
 
-Vitastor Backend is in experimental status, however you can already try to
+## [May I offer you some lovely documentation?](http://s3-server.readthedocs.io/en/latest/)
-run it and write or read something, or even mount it with [GeeseFS](https://github.com/yandex-cloud/geesefs),
-it works too 😊.
 
-Installation instructions:
+## Docker
 
-### Install Vitastor
+[Run your Zenko CloudServer with Docker](https://hub.docker.com/r/scality/s3server/)
 
-Refer to [Vitastor Quick Start Manual](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/intro/quickstart.en.md).
+## Contributing
 
-### Install Zenko with Vitastor Backend
+In order to contribute, please follow the
+[Contributing Guidelines](
+https://github.com/scality/Guidelines/blob/master/CONTRIBUTING.md).
 
-- Clone this repository: `git clone https://git.yourcmc.ru/vitalif/zenko-cloudserver-vitastor`
+## Installation
-- Install dependencies: `npm install --omit dev` or just `npm install`
-- Clone Vitastor repository: `git clone https://git.yourcmc.ru/vitalif/vitastor`
-- Build Vitastor node.js binding by running `npm install` in `node-binding` subdirectory of Vitastor repository.
-  You need `node-gyp` and `vitastor-client-dev` (Vitastor client library) for it to succeed.
-- Symlink Vitastor module to Zenko: `ln -s /path/to/vitastor/node-binding /path/to/zenko/node_modules/vitastor`
 
-### Install and Configure MongoDB
+### Dependencies
 
-Refer to [MongoDB Manual](https://www.mongodb.com/docs/manual/installation/).
+Building and running the Zenko CloudServer requires node.js 6.9.5 and npm v3
+. Up-to-date versions can be found at
+[Nodesource](https://github.com/nodesource/distributions).
 
-### Setup Zenko
+### Clone source code
 
-- Create a separate pool for S3 object data in your Vitastor cluster: `vitastor-cli create-pool s3-data`
+```shell
-- Retrieve ID of the new pool from `vitastor-cli ls-pools --detail s3-data`
+git clone https://github.com/scality/S3.git
-- In another pool, create an image for storing Vitastor volume metadata: `vitastor-cli create -s 10G s3-volume-meta`
-- Copy `config.json.vitastor` to `config.json`, adjust it to match your domain
-- Copy `authdata.json.example` to `authdata.json` - this is where you set S3 access & secret keys,
-  and also adjust them if you want to. Scality seems to use a separate auth service "Scality Vault" for
-  access keys, but it's not published, so let's use a file for now.
-- Copy `locationConfig.json.vitastor` to `locationConfig.json` - this is where you set Vitastor cluster access data.
-  You should put correct values for `pool_id` (pool ID from the second step) and `metadata_image` (from the third step)
-  in this file.
-
-Note: `locationConfig.json` in this version corresponds to storage classes (like STANDARD, COLD, etc)
-instead of "locations" (zones like us-east-1) as it was in original Zenko CloudServer.
-
-### Start Zenko
-
-Start the S3 server with: `node index.js`
-
-If you use default settings, Zenko CloudServer starts on port 8000.
-The default access key is `accessKey1` with a secret key of `verySecretKey1`.
-
-Now you can access your S3 with `s3cmd` or `geesefs`:
-
-```
-s3cmd --access_key=accessKey1 --secret_key=verySecretKey1 --host=http://localhost:8000 mb s3://testbucket
 ```
 
-```
+### Install js dependencies
-AWS_ACCESS_KEY_ID=accessKey1 \
+
-    AWS_SECRET_ACCESS_KEY=verySecretKey1 \
+Go to the ./S3 folder,
-    geesefs --endpoint http://localhost:8000 testbucket mountdir
+
+```shell
+npm install
 ```
 
-# Author & License
+If you get an error regarding installation of the diskUsage module,
+please install g++.
 
-- [Zenko CloudServer](https://s3-server.readthedocs.io/en/latest/) author is Scality, licensed under [Apache License, version 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+If you get an error regarding level-down bindings, try clearing your npm cache:
-- [Vitastor](https://git.yourcmc.ru/vitalif/vitastor/) and Zenko Vitastor backend author is Vitaliy Filippov, licensed under [VNPL-1.1](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/VNPL-1.1.txt)
+
-  (a "network copyleft" license based on AGPL/SSPL, but worded in a better way)
+```shell
+npm cache clear
+```
+
+## Run it with a file backend
+
+```shell
+npm start
+```
+
+This starts a Zenko CloudServer on port 8000. Two additional ports 9990 and
+9991 are also open locally for internal transfer of metadata and data,
+respectively.
+
+The default access key is accessKey1 with
+a secret key of verySecretKey1.
+
+By default the metadata files will be saved in the
+localMetadata directory and the data files will be saved
+in the localData directory within the ./S3 directory on your
+machine.  These directories have been pre-created within the
+repository.  If you would like to save the data or metadata in
+different locations of your choice, you must specify them with absolute paths.
+So, when starting the server:
+
+```shell
+mkdir -m 700 $(pwd)/myFavoriteDataPath
+mkdir -m 700 $(pwd)/myFavoriteMetadataPath
+export S3DATAPATH="$(pwd)/myFavoriteDataPath"
+export S3METADATAPATH="$(pwd)/myFavoriteMetadataPath"
+npm start
+```
+
+## Run it with multiple data backends
+
+```shell
+export S3DATA='multiple'
+npm start
+```
+
+This starts a Zenko CloudServer on port 8000.
+The default access key is accessKey1 with
+a secret key of verySecretKey1.
+
+With multiple backends, you have the ability to
+choose where each object will be saved by setting
+the following header with a locationConstraint on
+a PUT request:
+
+```shell
+'x-amz-meta-scal-location-constraint':'myLocationConstraint'
+```
+
+If no header is sent with a PUT object request, the
+location constraint of the bucket will determine
+where the data is saved. If the bucket has no location
+constraint, the endpoint of the PUT request will be
+used to determine location.
+
+See the Configuration section below to learn how to set
+location constraints.
+
+## Run it with an in-memory backend
+
+```shell
+npm run mem_backend
+```
+
+This starts a Zenko CloudServer on port 8000.
+The default access key is accessKey1 with
+a secret key of verySecretKey1.
+
+[badgetwitter]: https://img.shields.io/twitter/follow/zenko.svg?style=social&label=Follow
+[badgedocker]: https://img.shields.io/docker/pulls/scality/s3server.svg
+[badgepub]: https://circleci.com/gh/scality/S3.svg?style=svg
+[badgepriv]: http://ci.ironmann.io/gh/scality/S3.svg?style=svg&circle-token=1f105b7518b53853b5b7cf72302a3f75d8c598ae

_config.yml

@@ -1,2 +0,0 @@
----
-theme: jekyll-theme-modernist

authdata.json.example

@@ -1,56 +0,0 @@
-{
-    "accounts": [{
-        "name": "Bart",
-        "email": "sampleaccount1@sampling.com",
-        "arn": "arn:aws:iam::123456789012:root",
-        "canonicalID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be",
-        "shortid": "123456789012",
-        "keys": [{
-            "access": "accessKey1",
-            "secret": "verySecretKey1"
-        }]
-    }, {
-        "name": "Lisa",
-        "email": "sampleaccount2@sampling.com",
-        "arn": "arn:aws:iam::123456789013:root",
-        "canonicalID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2bf",
-        "shortid": "123456789013",
-        "keys": [{
-            "access": "accessKey2",
-            "secret": "verySecretKey2"
-        }]
-    },
-    {
-        "name": "Clueso",
-        "email": "inspector@clueso.info",
-        "arn": "arn:aws:iam::123456789014:root",
-        "canonicalID": "http://acs.zenko.io/accounts/service/clueso",
-        "shortid": "123456789014",
-        "keys": [{
-            "access": "cluesoKey1",
-            "secret": "cluesoSecretKey1"
-        }]
-    },
-    {
-        "name": "Replication",
-        "email": "inspector@replication.info",
-        "arn": "arn:aws:iam::123456789015:root",
-        "canonicalID": "http://acs.zenko.io/accounts/service/replication",
-        "shortid": "123456789015",
-        "keys": [{
-            "access": "replicationKey1",
-            "secret": "replicationSecretKey1"
-        }]
-    },
-    {
-        "name": "Lifecycle",
-        "email": "inspector@lifecycle.info",
-        "arn": "arn:aws:iam::123456789016:root",
-        "canonicalID": "http://acs.zenko.io/accounts/service/lifecycle",
-        "shortid": "123456789016",
-        "keys": [{
-            "access": "lifecycleKey1",
-            "secret": "lifecycleSecretKey1"
-        }]
-    }]
-}

backends/data/external/AwsClient.js

@@ -0,0 +1,554 @@
+const { errors, s3middleware } = require('arsenal');
+const AWS = require('aws-sdk');
+const MD5Sum = s3middleware.MD5Sum;
+const getMetaHeaders = s3middleware.userMetadata.getMetaHeaders;
+const createLogger = require('../multipleBackendLogger');
+const { prepareStream } = require('../../api/apiUtils/object/prepareStream');
+const { logHelper, removeQuotes, trimXMetaPrefix } = require('./utils');
+const { config } = require('../../Config');
+
+const missingVerIdInternalError = errors.InternalError.customizeDescription(
+    'Invalid state. Please ensure versioning is enabled ' +
+    'in AWS for the location constraint and try again.'
+);
+
+class AwsClient {
+    constructor(config) {
+        this.clientType = 'aws_s3';
+        this._s3Params = config.s3Params;
+        this._awsBucketName = config.bucketName;
+        this._bucketMatch = config.bucketMatch;
+        this._dataStoreName = config.dataStoreName;
+        this._serverSideEncryption = config.serverSideEncryption;
+        this._client = new AWS.S3(this._s3Params);
+    }
+
+    _createAwsKey(requestBucketName, requestObjectKey,
+        bucketMatch) {
+        if (bucketMatch) {
+            return requestObjectKey;
+        }
+        return `${requestBucketName}/${requestObjectKey}`;
+    }
+    put(stream, size, keyContext, reqUids, callback) {
+        const awsKey = this._createAwsKey(keyContext.bucketName,
+           keyContext.objectKey, this._bucketMatch);
+        const metaHeaders = trimXMetaPrefix(keyContext.metaHeaders);
+        const log = createLogger(reqUids);
+
+        const putCb = (err, data) => {
+            if (err) {
+                logHelper(log, 'error', 'err from data backend',
+                  err, this._dataStoreName);
+                return callback(errors.ServiceUnavailable
+                  .customizeDescription('Error returned from ' +
+                  `AWS: ${err.message}`)
+                );
+            }
+            if (!data.VersionId) {
+                logHelper(log, 'error', 'missing version id for data ' +
+                    'backend object', missingVerIdInternalError,
+                    this._dataStoreName);
+                return callback(missingVerIdInternalError);
+            }
+            const dataStoreVersionId = data.VersionId;
+            return callback(null, awsKey, dataStoreVersionId);
+        };
+
+        const params = {
+            Bucket: this._awsBucketName,
+            Key: awsKey,
+        };
+        // we call data.put to create a delete marker, but it's actually a
+        // delete request in call to AWS
+        if (keyContext.isDeleteMarker) {
+            return this._client.deleteObject(params, putCb);
+        }
+        const uploadParams = params;
+        uploadParams.Metadata = metaHeaders;
+        uploadParams.ContentLength = size;
+        if (this._serverSideEncryption) {
+            uploadParams.ServerSideEncryption = 'AES256';
+        }
+        if (keyContext.tagging) {
+            uploadParams.Tagging = keyContext.tagging;
+        }
+        if (keyContext.contentType !== undefined) {
+            uploadParams.ContentType = keyContext.contentType;
+        }
+        if (keyContext.cacheControl !== undefined) {
+            uploadParams.CacheControl = keyContext.cacheControl;
+        }
+        if (keyContext.contentDisposition !== undefined) {
+            uploadParams.ContentDisposition = keyContext.contentDisposition;
+        }
+        if (keyContext.contentEncoding !== undefined) {
+            uploadParams.ContentEncoding = keyContext.contentEncoding;
+        }
+        if (!stream) {
+            return this._client.putObject(uploadParams, putCb);
+        }
+
+        uploadParams.Body = stream;
+        return this._client.upload(uploadParams, putCb);
+    }
+    head(objectGetInfo, reqUids, callback) {
+        const log = createLogger(reqUids);
+        const { key, dataStoreVersionId } = objectGetInfo;
+        return this._client.headObject({
+            Bucket: this._awsBucketName,
+            Key: key,
+            VersionId: dataStoreVersionId,
+        }, err => {
+            if (err) {
+                logHelper(log, 'error', 'error heading object ' +
+                'from datastore', err, this._dataStoreName);
+                if (err.code === 'NotFound') {
+                    const error = errors.ServiceUnavailable
+                    .customizeDescription(
+                        'Unexpected error from AWS: "NotFound". Data on AWS ' +
+                        'may have been altered outside of CloudServer.'
+                    );
+                    return callback(error);
+                }
+                return callback(errors.ServiceUnavailable
+                  .customizeDescription('Error returned from ' +
+                  `AWS: ${err.message}`)
+                );
+            }
+            return callback();
+        });
+    }
+    get(objectGetInfo, range, reqUids, callback) {
+        const log = createLogger(reqUids);
+        const { key, dataStoreVersionId } = objectGetInfo;
+        const request = this._client.getObject({
+            Bucket: this._awsBucketName,
+            Key: key,
+            VersionId: dataStoreVersionId,
+            Range: range ? `bytes=${range[0]}-${range[1]}` : null,
+        }).on('success', response => {
+            log.trace('AWS GET request response headers',
+              { responseHeaders: response.httpResponse.headers });
+        });
+        const stream = request.createReadStream().on('error', err => {
+            logHelper(log, 'error', 'error streaming data from AWS',
+              err, this._dataStoreName);
+            return callback(err);
+        });
+        return callback(null, stream);
+    }
+    delete(objectGetInfo, reqUids, callback) {
+        const { key, dataStoreVersionId, deleteVersion } = objectGetInfo;
+        const log = createLogger(reqUids);
+        const params = {
+            Bucket: this._awsBucketName,
+            Key: key,
+        };
+        if (deleteVersion) {
+            params.VersionId = dataStoreVersionId;
+        }
+        return this._client.deleteObject(params, err => {
+            if (err) {
+                logHelper(log, 'error', 'error deleting object from ' +
+                'datastore', err, this._dataStoreName);
+                if (err.code === 'NoSuchVersion') {
+                    // data may have been deleted directly from the AWS backend
+                    // don't want to retry the delete and errors are not
+                    // sent back to client anyway, so no need to return err
+                    return callback();
+                }
+                return callback(errors.ServiceUnavailable
+                  .customizeDescription('Error returned from ' +
+                  `AWS: ${err.message}`)
+                );
+            }
+            return callback();
+        });
+    }
+
+    healthcheck(location, callback) {
+        const awsResp = {};
+        this._client.headBucket({ Bucket: this._awsBucketName },
+        err => {
+            /* eslint-disable no-param-reassign */
+            if (err) {
+                awsResp[location] = { error: err, external: true };
+                return callback(null, awsResp);
+            }
+            return this._client.getBucketVersioning({
+                Bucket: this._awsBucketName },
+            (err, data) => {
+                if (err) {
+                    awsResp[location] = { error: err, external: true };
+                } else if (!data.Status ||
+                    data.Status === 'Suspended') {
+                    awsResp[location] = {
+                        versioningStatus: data.Status,
+                        error: 'Versioning must be enabled',
+                        external: true,
+                    };
+                } else {
+                    awsResp[location] = {
+                        versioningStatus: data.Status,
+                        message: 'Congrats! You own the bucket',
+                    };
+                }
+                return callback(null, awsResp);
+            });
+        });
+    }
+
+    createMPU(key, metaHeaders, bucketName, websiteRedirectHeader, contentType,
+        cacheControl, contentDisposition, contentEncoding, log, callback) {
+        const metaHeadersTrimmed = {};
+        Object.keys(metaHeaders).forEach(header => {
+            if (header.startsWith('x-amz-meta-')) {
+                const headerKey = header.substring(11);
+                metaHeadersTrimmed[headerKey] = metaHeaders[header];
+            }
+        });
+        Object.assign(metaHeaders, metaHeadersTrimmed);
+        const awsBucket = this._awsBucketName;
+        const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
+        const params = {
+            Bucket: awsBucket,
+            Key: awsKey,
+            WebsiteRedirectLocation: websiteRedirectHeader,
+            Metadata: metaHeaders,
+            ContentType: contentType,
+            CacheControl: cacheControl,
+            ContentDisposition: contentDisposition,
+            ContentEncoding: contentEncoding,
+        };
+        return this._client.createMultipartUpload(params, (err, mpuResObj) => {
+            if (err) {
+                logHelper(log, 'error', 'err from data backend',
+                  err, this._dataStoreName);
+                return callback(errors.ServiceUnavailable
+                  .customizeDescription('Error returned from ' +
+                  `AWS: ${err.message}`)
+                );
+            }
+            return callback(null, mpuResObj);
+        });
+    }
+
+    uploadPart(request, streamingV4Params, stream, size, key, uploadId,
+    partNumber, bucketName, log, callback) {
+        let hashedStream = stream;
+        if (request) {
+            const partStream = prepareStream(request, streamingV4Params,
+                log, callback);
+            hashedStream = new MD5Sum();
+            partStream.pipe(hashedStream);
+        }
+
+        const awsBucket = this._awsBucketName;
+        const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
+        const params = { Bucket: awsBucket, Key: awsKey, UploadId: uploadId,
+            Body: hashedStream, ContentLength: size,
+            PartNumber: partNumber };
+        return this._client.uploadPart(params, (err, partResObj) => {
+            if (err) {
+                logHelper(log, 'error', 'err from data backend ' +
+                  'on uploadPart', err, this._dataStoreName);
+                return callback(errors.ServiceUnavailable
+                  .customizeDescription('Error returned from ' +
+                  `AWS: ${err.message}`)
+                );
+            }
+            // Because we manually add quotes to ETag later, remove quotes here
+            const noQuotesETag =
+                partResObj.ETag.substring(1, partResObj.ETag.length - 1);
+            const dataRetrievalInfo = {
+                key: awsKey,
+                dataStoreType: 'aws_s3',
+                dataStoreName: this._dataStoreName,
+                dataStoreETag: noQuotesETag,
+            };
+            return callback(null, dataRetrievalInfo);
+        });
+    }
+
+    listParts(key, uploadId, bucketName, partNumberMarker, maxParts, log,
+    callback) {
+        const awsBucket = this._awsBucketName;
+        const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
+        const params = { Bucket: awsBucket, Key: awsKey, UploadId: uploadId,
+            PartNumberMarker: partNumberMarker, MaxParts: maxParts };
+        return this._client.listParts(params, (err, partList) => {
+            if (err) {
+                logHelper(log, 'error', 'err from data backend on listPart',
+                  err, this._dataStoreName);
+                return callback(errors.ServiceUnavailable
+                  .customizeDescription('Error returned from ' +
+                  `AWS: ${err.message}`)
+                );
+            }
+            // build storedParts object to mimic Scality S3 backend returns
+            const storedParts = {};
+            storedParts.IsTruncated = partList.IsTruncated;
+            storedParts.Contents = [];
+            storedParts.Contents = partList.Parts.map(item => {
+                // We manually add quotes to ETag later, so remove quotes here
+                const noQuotesETag =
+                    item.ETag.substring(1, item.ETag.length - 1);
+                return {
+                    partNumber: item.PartNumber,
+                    value: {
+                        Size: item.Size,
+                        ETag: noQuotesETag,
+                        LastModified: item.LastModified,
+                    },
+                };
+            });
+            return callback(null, storedParts);
+        });
+    }
+
+    /**
+     * completeMPU - complete multipart upload on AWS backend
+     * @param {object} jsonList - user-sent list of parts to include in
+     *                          final mpu object
+     * @param {object} mdInfo - object containing 3 keys: storedParts,
+     *                          mpuOverviewKey, and splitter
+     * @param {string} key - object key
+     * @param {string} uploadId - multipart upload id string
+     * @param {string} bucketName - name of bucket
+     * @param {RequestLogger} log - logger instance
+     * @param {function} callback - callback function
+     * @return {(Error|object)} - return Error if complete MPU fails, otherwise
+     * object containing completed object key, eTag, and contentLength
+     */
+    completeMPU(jsonList, mdInfo, key, uploadId, bucketName, log, callback) {
+        const awsBucket = this._awsBucketName;
+        const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
+        const mpuError = {
+            InvalidPart: true,
+            InvalidPartOrder: true,
+            EntityTooSmall: true,
+        };
+        const partArray = [];
+        const partList = jsonList.Part;
+        partList.forEach(partObj => {
+            const partParams = { PartNumber: partObj.PartNumber[0],
+                ETag: partObj.ETag[0] };
+            partArray.push(partParams);
+        });
+        const mpuParams = {
+            Bucket: awsBucket, Key: awsKey, UploadId: uploadId,
+            MultipartUpload: {
+                Parts: partArray,
+            },
+        };
+        const completeObjData = { key: awsKey };
+        return this._client.completeMultipartUpload(mpuParams,
+        (err, completeMpuRes) => {
+            if (err) {
+                if (mpuError[err.code]) {
+                    logHelper(log, 'trace', 'err from data backend on ' +
+                    'completeMPU', err, this._dataStoreName);
+                    return callback(errors[err.code]);
+                }
+                logHelper(log, 'error', 'err from data backend on ' +
+                'completeMPU', err, this._dataStoreName);
+                return callback(errors.ServiceUnavailable
+                  .customizeDescription('Error returned from ' +
+                  `AWS: ${err.message}`)
+                );
+            }
+            if (!completeMpuRes.VersionId) {
+                logHelper(log, 'error', 'missing version id for data ' +
+                'backend object', missingVerIdInternalError,
+                    this._dataStoreName);
+                return callback(missingVerIdInternalError);
+            }
+            // need to get content length of new object to store
+            // in our metadata
+            return this._client.headObject({ Bucket: awsBucket, Key: awsKey },
+            (err, objHeaders) => {
+                if (err) {
+                    logHelper(log, 'trace', 'err from data backend on ' +
+                    'headObject', err, this._dataStoreName);
+                    return callback(errors.ServiceUnavailable
+                      .customizeDescription('Error returned from ' +
+                      `AWS: ${err.message}`)
+                    );
+                }
+                // remove quotes from eTag because they're added later
+                completeObjData.eTag = completeMpuRes.ETag
+                    .substring(1, completeMpuRes.ETag.length - 1);
+                completeObjData.dataStoreVersionId = completeMpuRes.VersionId;
+                completeObjData.contentLength = objHeaders.ContentLength;
+                return callback(null, completeObjData);
+            });
+        });
+    }
+
+    abortMPU(key, uploadId, bucketName, log, callback) {
+        const awsBucket = this._awsBucketName;
+        const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
+        const abortParams = {
+            Bucket: awsBucket, Key: awsKey, UploadId: uploadId,
+        };
+        return this._client.abortMultipartUpload(abortParams, err => {
+            if (err) {
+                logHelper(log, 'error', 'There was an error aborting ' +
+                'the MPU on AWS S3. You should abort directly on AWS S3 ' +
+                'using the same uploadId.', err, this._dataStoreName);
+                return callback(errors.ServiceUnavailable
+                  .customizeDescription('Error returned from ' +
+                  `AWS: ${err.message}`)
+                );
+            }
+            return callback();
+        });
+    }
+
+    objectPutTagging(key, bucket, objectMD, log, callback) {
+        const awsBucket = this._awsBucketName;
+        const awsKey = this._createAwsKey(bucket, key, this._bucketMatch);
+        const dataStoreVersionId = objectMD.location[0].dataStoreVersionId;
+        const tagParams = {
+            Bucket: awsBucket,
+            Key: awsKey,
+            VersionId: dataStoreVersionId,
+        };
+        const keyArray = Object.keys(objectMD.tags);
+        tagParams.Tagging = {};
+        tagParams.Tagging.TagSet = keyArray.map(key => {
+            const value = objectMD.tags[key];
+            return { Key: key, Value: value };
+        });
+        return this._client.putObjectTagging(tagParams, err => {
+            if (err) {
+                logHelper(log, 'error', 'error from data backend on ' +
+                  'putObjectTagging', err, this._dataStoreName);
+                return callback(errors.ServiceUnavailable
+                  .customizeDescription('Error returned from ' +
+                  `AWS: ${err.message}`)
+                );
+            }
+            return callback();
+        });
+    }
+
+    objectDeleteTagging(key, bucket, objectMD, log, callback) {
+        const awsBucket = this._awsBucketName;
+        const awsKey = this._createAwsKey(bucket, key, this._bucketMatch);
+        const dataStoreVersionId = objectMD.location[0].dataStoreVersionId;
+        const tagParams = {
+            Bucket: awsBucket,
+            Key: awsKey,
+            VersionId: dataStoreVersionId,
+        };
+        return this._client.deleteObjectTagging(tagParams, err => {
+            if (err) {
+                logHelper(log, 'error', 'error from data backend on ' +
+                'deleteObjectTagging', err, this._dataStoreName);
+                return callback(errors.ServiceUnavailable
+                  .customizeDescription('Error returned from ' +
+                  `AWS: ${err.message}`)
+                );
+            }
+            return callback();
+        });
+    }
+    copyObject(request, destLocationConstraintName, sourceKey,
+    sourceLocationConstraintName, storeMetadataParams, log, callback) {
+        const destBucketName = request.bucketName;
+        const destObjectKey = request.objectKey;
+        const destAwsKey = this._createAwsKey(destBucketName, destObjectKey,
+          this._bucketMatch);
+
+        const sourceAwsBucketName =
+          config.getAwsBucketName(sourceLocationConstraintName);
+
+        const metadataDirective = request.headers['x-amz-metadata-directive'];
+        const metaHeaders = trimXMetaPrefix(getMetaHeaders(request.headers));
+        const awsParams = {
+            Bucket: this._awsBucketName,
+            Key: destAwsKey,
+            CopySource: `${sourceAwsBucketName}/${sourceKey}`,
+            Metadata: metaHeaders,
+            MetadataDirective: metadataDirective,
+        };
+        if (destLocationConstraintName &&
+        config.isAWSServerSideEncrytion(destLocationConstraintName)) {
+            awsParams.ServerSideEncryption = 'AES256';
+        }
+        this._client.copyObject(awsParams, (err, copyResult) => {
+            if (err) {
+                if (err.code === 'AccessDenied') {
+                    logHelper(log, 'error', 'Unable to access ' +
+                    `${sourceAwsBucketName} AWS bucket`, err,
+                    this._dataStoreName);
+                    return callback(errors.AccessDenied
+                      .customizeDescription('Error: Unable to access ' +
+                      `${sourceAwsBucketName} AWS bucket`)
+                    );
+                }
+                logHelper(log, 'error', 'error from data backend on ' +
+                'copyObject', err, this._dataStoreName);
+                return callback(errors.ServiceUnavailable
+                  .customizeDescription('Error returned from ' +
+                  `AWS: ${err.message}`)
+                );
+            }
+            if (!copyResult.VersionId) {
+                logHelper(log, 'error', 'missing version id for data ' +
+                'backend object', missingVerIdInternalError,
+                    this._dataStoreName);
+                return callback(missingVerIdInternalError);
+            }
+            return callback(null, destAwsKey, copyResult.VersionId);
+        });
+    }
+    uploadPartCopy(request, awsSourceKey, sourceLocationConstraintName,
+      log, callback) {
+        const destBucketName = request.bucketName;
+        const destObjectKey = request.objectKey;
+        const destAwsKey = this._createAwsKey(destBucketName, destObjectKey,
+        this._bucketMatch);
+
+        const sourceAwsBucketName =
+          config.getAwsBucketName(sourceLocationConstraintName);
+
+        const uploadId = request.query.uploadId;
+        const partNumber = request.query.partNumber;
+        const copySourceRange = request.headers['x-amz-copy-source-range'];
+
+        const params = {
+            Bucket: this._awsBucketName,
+            CopySource: `${sourceAwsBucketName}/${awsSourceKey}`,
+            CopySourceRange: copySourceRange,
+            Key: destAwsKey,
+            PartNumber: partNumber,
+            UploadId: uploadId,
+        };
+        return this._client.uploadPartCopy(params, (err, res) => {
+            if (err) {
+                if (err.code === 'AccessDenied') {
+                    logHelper(log, 'error', 'Unable to access ' +
+                    `${sourceAwsBucketName} AWS bucket`, err,
+                    this._dataStoreName);
+                    return callback(errors.AccessDenied
+                      .customizeDescription('Error: Unable to access ' +
+                      `${sourceAwsBucketName} AWS bucket`)
+                    );
+                }
+                logHelper(log, 'error', 'error from data backend on ' +
+                'uploadPartCopy', err, this._dataStoreName);
+                return callback(errors.ServiceUnavailable
+                  .customizeDescription('Error returned from ' +
+                  `AWS: ${err.message}`)
+                );
+            }
+            const eTag = removeQuotes(res.CopyPartResult.ETag);
+            return callback(null, eTag);
+        });
+    }
+}
+
+module.exports = AwsClient;

backends/data/external/AzureClient.js

@@ -0,0 +1,434 @@
+const url = require('url');
+
+const { errors, s3middleware } = require('arsenal');
+const azure = require('azure-storage');
+const createLogger = require('../multipleBackendLogger');
+const { logHelper, translateAzureMetaHeaders } = require('./utils');
+const { config } = require('../../Config');
+const { validateAndFilterMpuParts } =
+    require('../../api/apiUtils/object/processMpuParts');
+const constants = require('../../../constants');
+const metadata = require('../../metadata/wrapper');
+const azureMpuUtils = s3middleware.azureHelper.mpuUtils;
+
+class AzureClient {
+    constructor(config) {
+        this._azureStorageEndpoint = config.azureStorageEndpoint;
+        this._azureStorageCredentials = config.azureStorageCredentials;
+        this._azureContainerName = config.azureContainerName;
+        this._client = azure.createBlobService(
+            this._azureStorageCredentials.storageAccountName,
+            this._azureStorageCredentials.storageAccessKey,
+            this._azureStorageEndpoint);
+        this._dataStoreName = config.dataStoreName;
+        this._bucketMatch = config.bucketMatch;
+        if (config.proxy) {
+            const parsedUrl = url.parse(config.proxy);
+            if (!parsedUrl.port) {
+                parsedUrl.port = 80;
+            }
+            this._client.setProxy(parsedUrl);
+        }
+    }
+
+    _errorWrapper(s3Method, azureMethod, args, log, cb) {
+        if (log) {
+            log.info(`calling azure ${azureMethod}`);
+        }
+        try {
+            this._client[azureMethod].apply(this._client, args);
+        } catch (err) {
+            const error = errors.ServiceUnavailable;
+            if (log) {
+                log.error('error thrown by Azure Storage Client Library',
+                    { error: err.message, stack: err.stack, s3Method,
+                    azureMethod, dataStoreName: this._dataStoreName });
+            }
+            cb(error.customizeDescription('Error from Azure ' +
+                `method: ${azureMethod} on ${s3Method} S3 call: ` +
+                `${err.message}`));
+        }
+    }
+
+    _createAzureKey(requestBucketName, requestObjectKey,
+        bucketMatch) {
+        if (bucketMatch) {
+            return requestObjectKey;
+        }
+        return `${requestBucketName}/${requestObjectKey}`;
+    }
+
+    _getMetaHeaders(objectMD) {
+        const metaHeaders = {};
+        Object.keys(objectMD).forEach(mdKey => {
+            const isMetaHeader = mdKey.startsWith('x-amz-meta-');
+            if (isMetaHeader) {
+                metaHeaders[mdKey] = objectMD[mdKey];
+            }
+        });
+        return translateAzureMetaHeaders(metaHeaders);
+    }
+
+    // Before putting or deleting object on Azure, check if MPU exists with
+    // same key name. If it does, do not allow put or delete because Azure
+    // will delete all blocks with same key name
+    protectAzureBlocks(bucketName, objectKey, dataStoreName, log, cb) {
+        const mpuBucketName = `${constants.mpuBucketPrefix}${bucketName}`;
+        const splitter = constants.splitter;
+        const listingParams = {
+            prefix: `overview${splitter}${objectKey}`,
+            listingType: 'MPU',
+            splitter,
+            maxKeys: 1,
+        };
+
+        return metadata.listMultipartUploads(mpuBucketName, listingParams,
+        log, (err, mpuList) => {
+            if (err && !err.NoSuchBucket) {
+                log.error('Error listing MPUs for Azure delete',
+                    { error: err, dataStoreName });
+                return cb(errors.ServiceUnavailable);
+            }
+            if (mpuList && mpuList.Uploads && mpuList.Uploads.length > 0) {
+                const error = errors.MPUinProgress;
+                log.error('Error: cannot put/delete object to Azure with ' +
+                    'same key name as ongoing MPU on Azure',
+                    { error, dataStoreName });
+                return cb(error);
+            }
+            // If listMultipartUploads returns a NoSuchBucket error or the
+            // mpu list is empty, there are no conflicting MPUs, so continue
+            return cb();
+        });
+    }
+
+    put(stream, size, keyContext, reqUids, callback) {
+        const log = createLogger(reqUids);
+        // before blob is put, make sure there is no ongoing MPU with same key
+        this.protectAzureBlocks(keyContext.bucketName,
+        keyContext.objectKey, this._dataStoreName, log, err => {
+            // if error returned, there is ongoing MPU, so do not put
+            if (err) {
+                return callback(err.customizeDescription(
+                    `Error putting object to Azure: ${err.message}`));
+            }
+            const azureKey = this._createAzureKey(keyContext.bucketName,
+                keyContext.objectKey, this._bucketMatch);
+            const options = {
+                metadata: translateAzureMetaHeaders(keyContext.metaHeaders,
+                    keyContext.tagging),
+                contentSettings: {
+                    contentType: keyContext.contentType || undefined,
+                    cacheControl: keyContext.cacheControl || undefined,
+                    contentDisposition: keyContext.contentDisposition ||
+                        undefined,
+                    contentEncoding: keyContext.contentEncoding || undefined,
+                },
+            };
+            if (size === 0) {
+                return this._errorWrapper('put', 'createBlockBlobFromText',
+                    [this._azureContainerName, azureKey, '', options,
+                    err => {
+                        if (err) {
+                            logHelper(log, 'error', 'err from Azure PUT data ' +
+                                'backend', err, this._dataStoreName);
+                            return callback(errors.ServiceUnavailable
+                                .customizeDescription('Error returned from ' +
+                                `Azure: ${err.message}`));
+                        }
+                        return callback(null, azureKey);
+                    }], log, callback);
+            }
+            return this._errorWrapper('put', 'createBlockBlobFromStream',
+                [this._azureContainerName, azureKey, stream, size, options,
+                err => {
+                    if (err) {
+                        logHelper(log, 'error', 'err from Azure PUT data ' +
+                            'backend', err, this._dataStoreName);
+                        return callback(errors.ServiceUnavailable
+                            .customizeDescription('Error returned from ' +
+                            `Azure: ${err.message}`));
+                    }
+                    return callback(null, azureKey);
+                }], log, callback);
+        });
+    }
+
+    head(objectGetInfo, reqUids, callback) {
+        const log = createLogger(reqUids);
+        const { key, azureStreamingOptions } = objectGetInfo;
+        return this._errorWrapper('head', 'getBlobProperties',
+            [this._azureContainerName, key, azureStreamingOptions,
+            err => {
+                if (err) {
+                    logHelper(log, 'error', 'err from Azure HEAD data backend',
+                        err, this._dataStoreName);
+                    if (err.code === 'NotFound') {
+                        const error = errors.ServiceUnavailable
+                        .customizeDescription(
+                            'Unexpected error from Azure: "NotFound". Data ' +
+                            'on Azure may have been altered outside of ' +
+                            'CloudServer.');
+                        return callback(error);
+                    }
+                    return callback(errors.ServiceUnavailable
+                        .customizeDescription('Error returned from ' +
+                        `Azure: ${err.message}`));
+                }
+                return callback();
+            }], log, callback);
+    }
+
+    get(objectGetInfo, range, reqUids, callback) {
+        const log = createLogger(reqUids);
+        // for backwards compatibility
+        const { key, response, azureStreamingOptions } = objectGetInfo;
+        let streamingOptions;
+        if (azureStreamingOptions) {
+            // option coming from api.get()
+            streamingOptions = azureStreamingOptions;
+        } else if (range) {
+            // option coming from multipleBackend.upload()
+            const rangeStart = range[0] ? range[0].toString() : undefined;
+            const rangeEnd = range[1] ? range[1].toString() : undefined;
+            streamingOptions = { rangeStart, rangeEnd };
+        }
+        this._errorWrapper('get', 'getBlobToStream',
+            [this._azureContainerName, key, response, streamingOptions,
+            err => {
+                if (err) {
+                    logHelper(log, 'error', 'err from Azure GET data backend',
+                        err, this._dataStoreName);
+                    return callback(errors.ServiceUnavailable);
+                }
+                return callback(null, response);
+            }], log, callback);
+    }
+
+    delete(objectGetInfo, reqUids, callback) {
+        const log = createLogger(reqUids);
+        // for backwards compatibility
+        const key = typeof objectGetInfo === 'string' ? objectGetInfo :
+            objectGetInfo.key;
+        return this._errorWrapper('delete', 'deleteBlobIfExists',
+            [this._azureContainerName, key,
+            err => {
+                if (err) {
+                    const log = createLogger(reqUids);
+                    logHelper(log, 'error', 'error deleting object from ' +
+                        'Azure datastore', err, this._dataStoreName);
+                    return callback(errors.ServiceUnavailable
+                        .customizeDescription('Error returned from ' +
+                        `Azure: ${err.message}`));
+                }
+                return callback();
+            }], log, callback);
+    }
+
+    healthcheck(location, callback, flightCheckOnStartUp) {
+        const azureResp = {};
+        const healthCheckAction = flightCheckOnStartUp ?
+            'createContainerIfNotExists' : 'doesContainerExist';
+        this._errorWrapper('checkAzureHealth', healthCheckAction,
+            [this._azureContainerName, err => {
+                /* eslint-disable no-param-reassign */
+                if (err) {
+                    azureResp[location] = { error: err.message,
+                        external: true };
+                    return callback(null, azureResp);
+                }
+                azureResp[location] = {
+                    message:
+                    'Congrats! You can access the Azure storage account',
+                };
+                return callback(null, azureResp);
+            }], null, callback);
+    }
+
+    uploadPart(request, streamingV4Params, partStream, size, key, uploadId,
+    partNumber, bucket, log, callback) {
+        const azureKey = this._createAzureKey(bucket, key, this._bucketMatch);
+        const params = { bucketName: this._azureContainerName,
+            partNumber, size, objectKey: azureKey, uploadId };
+        const stream = request || partStream;
+
+        if (request && request.headers && request.headers['content-md5']) {
+            params.contentMD5 = request.headers['content-md5'];
+        }
+        const dataRetrievalInfo = {
+            key: azureKey,
+            partNumber,
+            dataStoreName: this._dataStoreName,
+            dataStoreType: 'azure',
+            numberSubParts: azureMpuUtils.getSubPartInfo(size)
+                .expectedNumberSubParts,
+        };
+
+        if (size === 0) {
+            log.debug('0-byte part does not store data',
+                { method: 'uploadPart' });
+            dataRetrievalInfo.dataStoreETag = azureMpuUtils.zeroByteETag;
+            dataRetrievalInfo.numberSubParts = 0;
+            return callback(null, dataRetrievalInfo);
+        }
+        if (size <= azureMpuUtils.maxSubPartSize) {
+            const errorWrapperFn = this._errorWrapper.bind(this);
+            return azureMpuUtils.putSinglePart(errorWrapperFn,
+            stream, params, this._dataStoreName, log, (err, dataStoreETag) => {
+                if (err) {
+                    return callback(err);
+                }
+                dataRetrievalInfo.dataStoreETag = dataStoreETag;
+                return callback(null, dataRetrievalInfo);
+            });
+        }
+        const errorWrapperFn = this._errorWrapper.bind(this);
+        return azureMpuUtils.putSubParts(errorWrapperFn, stream,
+        params, this._dataStoreName, log, (err, dataStoreETag) => {
+            if (err) {
+                return callback(err);
+            }
+            dataRetrievalInfo.dataStoreETag = dataStoreETag;
+            return callback(null, dataRetrievalInfo);
+        });
+    }
+
+    completeMPU(jsonList, mdInfo, key, uploadId, bucket, metaHeaders,
+    contentSettings, log, callback) {
+        const azureKey = this._createAzureKey(bucket, key, this._bucketMatch);
+        const commitList = {
+            UncommittedBlocks: jsonList.uncommittedBlocks || [],
+        };
+        let filteredPartsObj;
+        if (!jsonList.uncommittedBlocks) {
+            const { storedParts, mpuOverviewKey, splitter } = mdInfo;
+            filteredPartsObj = validateAndFilterMpuParts(storedParts, jsonList,
+                mpuOverviewKey, splitter, log);
+
+            filteredPartsObj.partList.forEach(part => {
+                // part.locations is always array of 1, which contains data info
+                const subPartIds =
+                    azureMpuUtils.getSubPartIds(part.locations[0], uploadId);
+                commitList.UncommittedBlocks.push(...subPartIds);
+            });
+        }
+        const options = {
+            contentSettings,
+            metadata: metaHeaders ? translateAzureMetaHeaders(metaHeaders) :
+                undefined,
+        };
+        this._errorWrapper('completeMPU', 'commitBlocks',
+            [this._azureContainerName, azureKey, commitList, options,
+            err => {
+                if (err) {
+                    logHelper(log, 'error', 'err completing MPU on Azure ' +
+                        'datastore', err, this._dataStoreName);
+                    return callback(errors.ServiceUnavailable
+                        .customizeDescription('Error returned from ' +
+                        `Azure: ${err.message}`));
+                }
+                const completeObjData = {
+                    key: azureKey,
+                    filteredPartsObj,
+                };
+                return callback(null, completeObjData);
+            }], log, callback);
+    }
+
+    objectPutTagging(key, bucket, objectMD, log, callback) {
+        const azureKey = this._createAzureKey(bucket, key, this._bucketMatch);
+        const azureMD = this._getMetaHeaders(objectMD);
+        azureMD.tags = JSON.stringify(objectMD.tags);
+        this._errorWrapper('objectPutTagging', 'setBlobMetadata',
+            [this._azureContainerName, azureKey, azureMD,
+            err => {
+                if (err) {
+                    logHelper(log, 'error', 'err putting object tags to ' +
+                        'Azure backend', err, this._dataStoreName);
+                    return callback(errors.ServiceUnavailable);
+                }
+                return callback();
+            }], log, callback);
+    }
+
+    objectDeleteTagging(key, bucket, objectMD, log, callback) {
+        const azureKey = this._createAzureKey(bucket, key, this._bucketMatch);
+        const azureMD = this._getMetaHeaders(objectMD);
+        this._errorWrapper('objectDeleteTagging', 'setBlobMetadata',
+            [this._azureContainerName, azureKey, azureMD,
+            err => {
+                if (err) {
+                    logHelper(log, 'error', 'err putting object tags to ' +
+                        'Azure backend', err, this._dataStoreName);
+                    return callback(errors.ServiceUnavailable);
+                }
+                return callback();
+            }], log, callback);
+    }
+
+    copyObject(request, destLocationConstraintName, sourceKey,
+    sourceLocationConstraintName, storeMetadataParams, log, callback) {
+        const destContainerName = request.bucketName;
+        const destObjectKey = request.objectKey;
+
+        const destAzureKey = this._createAzureKey(destContainerName,
+          destObjectKey, this._bucketMatch);
+
+        const sourceContainerName =
+        config.locationConstraints[sourceLocationConstraintName]
+        .details.azureContainerName;
+
+        let options;
+        if (storeMetadataParams.metaHeaders) {
+            options = { metadata:
+            translateAzureMetaHeaders(storeMetadataParams.metaHeaders) };
+        }
+
+        this._errorWrapper('copyObject', 'startCopyBlob',
+            [`${this._azureStorageEndpoint}` +
+                `${sourceContainerName}/${sourceKey}`,
+                this._azureContainerName, destAzureKey, options,
+            (err, res) => {
+                if (err) {
+                    if (err.code === 'CannotVerifyCopySource') {
+                        logHelper(log, 'error', 'Unable to access ' +
+                        `${sourceContainerName} Azure Container`, err,
+                        this._dataStoreName);
+                        return callback(errors.AccessDenied
+                        .customizeDescription('Error: Unable to access ' +
+                        `${sourceContainerName} Azure Container`)
+                        );
+                    }
+                    logHelper(log, 'error', 'error from data backend on ' +
+                    'copyObject', err, this._dataStoreName);
+                    return callback(errors.ServiceUnavailable
+                    .customizeDescription('Error returned from ' +
+                    `AWS: ${err.message}`)
+                    );
+                }
+                if (res.copy.status === 'pending') {
+                    logHelper(log, 'error', 'Azure copy status is pending',
+                    err, this._dataStoreName);
+                    const copyId = res.copy.id;
+                    this._client.abortCopyBlob(this._azureContainerName,
+                    destAzureKey, copyId, err => {
+                        if (err) {
+                            logHelper(log, 'error', 'error from data backend ' +
+                            'on abortCopyBlob', err, this._dataStoreName);
+                            return callback(errors.ServiceUnavailable
+                            .customizeDescription('Error returned from ' +
+                            `AWS on abortCopyBlob: ${err.message}`)
+                            );
+                        }
+                        return callback(errors.InvalidObjectState
+                        .customizeDescription('Error: Azure copy status was ' +
+                        'pending. It has been aborted successfully')
+                        );
+                    });
+                }
+                return callback(null, destAzureKey);
+            }], log, callback);
+    }
+}
+
+module.exports = AzureClient;

backends/data/external/GCP/GcpService.js

@@ -0,0 +1,246 @@
+const googleAuth = require('google-auto-auth');
+const async = require('async');
+const AWS = require('aws-sdk');
+const { errors } = require('arsenal');
+const Service = AWS.Service;
+
+const GcpSigner = require('./GcpSigner');
+
+function genAuth(authParams, callback) {
+    async.tryEach([
+        function authKeyFile(next) {
+            const authOptions = {
+                scopes: authParams.scopes,
+                keyFilename: authParams.keyFilename,
+            };
+            const auth = googleAuth(authOptions);
+            auth.getToken(err => next(err, auth));
+        },
+        function authCredentials(next) {
+            const authOptions = {
+                scopes: authParams.scopes,
+                credentials: authParams.credentials,
+            };
+            const auth = googleAuth(authOptions);
+            auth.getToken(err => next(err, auth));
+        },
+    ], (err, result) => callback(err, result));
+}
+
+AWS.apiLoader.services.gcp = {};
+const GCP = Service.defineService('gcp', ['2017-11-01'], {
+    _jsonAuth: null,
+    _authParams: null,
+
+    getToken(callback) {
+        if (this._jsonAuth) {
+            return this._jsonAuth.getToken(callback);
+        }
+
+        if (!this._authParams && this.config.authParams &&
+            typeof this.config.authParams === 'object') {
+            this._authParams = this.config.authParams;
+        }
+        return genAuth(this._authParams, (err, auth) => {
+            if (!err) {
+                this._jsonAuth = auth;
+                return this._jsonAuth.getToken(callback);
+            }
+            // should never happen, but if all preconditions fails
+            // can't generate tokens
+            return callback(errors.InternalError.customizeDescription(
+                'Unable to create a google authorizer'));
+        });
+    },
+
+    getSignerClass() {
+        return GcpSigner;
+    },
+
+    validateService() {
+        if (!this.config.region) {
+            this.config.region = 'us-east-1';
+        }
+    },
+
+    upload(params, options, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: upload not implemented'));
+    },
+
+    // Service API
+    listBuckets(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: listBuckets not implemented'));
+    },
+
+    // Bucket APIs
+    getBucketLocation(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: getBucketLocation not implemented'));
+    },
+
+    deleteBucket(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: deleteBucket not implemented'));
+    },
+
+    headBucket(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: headBucket not implemented'));
+    },
+
+    listObjects(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: listObjects not implemented'));
+    },
+
+    listObjectVersions(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: listObjecVersions not implemented'));
+    },
+
+    putBucket(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: putBucket not implemented'));
+    },
+
+    getBucketAcl(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: getBucketAcl not implemented'));
+    },
+
+    putBucketAcl(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: putBucketAcl not implemented'));
+    },
+
+    putBucketWebsite(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: putBucketWebsite not implemented'));
+    },
+
+    getBucketWebsite(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: getBucketWebsite not implemented'));
+    },
+
+    deleteBucketWebsite(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: deleteBucketWebsite not implemented'));
+    },
+
+    putBucketCors(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: putBucketCors not implemented'));
+    },
+
+    getBucketCors(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: getBucketCors not implemented'));
+    },
+
+    deleteBucketCors(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: deleteBucketCors not implemented'));
+    },
+
+    // Object APIs
+    headObject(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: headObject not implemented'));
+    },
+
+    putObject(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: putObject not implemented'));
+    },
+
+    getObject(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: getObject not implemented'));
+    },
+
+    deleteObject(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: deleteObject not implemented'));
+    },
+
+    deleteObjects(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: deleteObjects not implemented'));
+    },
+
+    copyObject(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: copyObject not implemented'));
+    },
+
+    putObjectTagging(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: putObjectTagging not implemented'));
+    },
+
+    deleteObjectTagging(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: deleteObjectTagging not implemented'));
+    },
+
+    putObjectAcl(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: putObjectAcl not implemented'));
+    },
+
+    getObjectAcl(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription('GCP: getObjectAcl not implemented'));
+    },
+
+    // Multipart upload
+    abortMultipartUpload(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription(
+                'GCP: abortMultipartUpload not implemented'));
+    },
+
+    createMultipartUpload(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription(
+                'GCP: createMultipartUpload not implemented'));
+    },
+
+    completeMultipartUpload(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription(
+                'GCP: completeMultipartUpload not implemented'));
+    },
+
+    uploadPart(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription(
+                'GCP: uploadPart not implemented'));
+    },
+
+    uploadPartCopy(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription(
+                'GCP: uploadPartCopy not implemented'));
+    },
+
+    listParts(params, callback) {
+        return callback(errors.NotImplemented
+            .customizeDescription(
+                'GCP: listParts not implemented'));
+    },
+});
+
+Object.defineProperty(AWS.apiLoader.services.gcp, '2017-11-01', {
+    get: function get() {
+        const model = require('./gcp-2017-11-01.api.json');
+        return model;
+    },
+    enumerable: true,
+    configurable: true,
+});
+
+module.exports = GCP;

backends/data/external/GCP/GcpSigner.js

@@ -0,0 +1,48 @@
+const url = require('url');
+const qs = require('querystring');
+const AWS = require('aws-sdk');
+const werelogs = require('werelogs');
+const { constructStringToSignV2 } = require('arsenal').auth.client;
+
+const logger = new werelogs.Logger('GcpSigner');
+
+function genQueryObject(uri) {
+    const queryString = url.parse(uri).query;
+    return qs.parse(queryString);
+}
+
+const GcpSigner = AWS.util.inherit(AWS.Signers.RequestSigner, {
+    constructor: function GcpSigner(request) {
+        AWS.Signers.RequestSigner.call(this, request);
+    },
+
+    addAuthorization: function addAuthorization(credentials, date) {
+        if (!this.request.headers['presigned-expires']) {
+            this.request.headers['x-goog-date'] = AWS.util.date.rfc822(date);
+        }
+
+        const signature =
+            this.sign(credentials.secretAccessKey, this.stringToSign());
+        const auth = `GOOG1 ${credentials.accessKeyId}: ${signature}`;
+
+        this.request.headers.Authorization = auth;
+    },
+
+    stringToSign: function stringToSign() {
+        const requestObject = {
+            url: this.request.path,
+            method: this.request.method,
+            host: this.request.endpoint.host,
+            headers: this.request.headers,
+            query: genQueryObject(this.request.path) || {},
+        };
+        const data = Object.assign({}, this.request.headers);
+        return constructStringToSignV2(requestObject, data, logger, 'GCP');
+    },
+
+    sign: function sign(secret, string) {
+        return AWS.util.crypto.hmac(secret, string, 'base64', 'sha1');
+    },
+});
+
+module.exports = GcpSigner;

backends/data/external/GCP/GcpUtils.js

@@ -0,0 +1 @@
+module.exports = {};

backends/data/external/GCP/gcp-2017-11-01.api.json

@@ -0,0 +1,17 @@
+{
+    "version": "1.0",
+    "metadata": {
+        "apiVersion": "2017-11-01",
+        "checksumFormat": "md5",
+        "endpointPrefix": "s3",
+        "globalEndpoint": "storage.googleapi.com",
+        "protocol": "rest-xml",
+        "serviceAbbreviation": "GCP",
+        "serviceFullName": "Google Cloud Storage",
+        "signatureVersion": "s3",
+        "timestampFormat": "rfc822",
+        "uid": "gcp-2017-11-01"
+    },
+    "operations": {},
+    "shapes": {}
+}

backends/data/external/GCP/index.js

@@ -0,0 +1,5 @@
+module.exports = {
+    GCP: require('./GcpService'),
+    GcpSigner: require('./GcpSigner'),
+    GcpUtils: require('./GcpUtils'),
+};

backends/data/external/GcpClient.js

@@ -0,0 +1,39 @@
+const { GCP } = require('./GCP');
+const AwsClient = require('./AwsClient');
+
+/**
+ * Class representing a Google Cloud Storage backend object
+ * @extends AwsClient
+ */
+class GcpClient extends AwsClient {
+    /**
+     * constructor - creates a Gcp backend client object (inherits )
+     * @param {object} config - configuration object for Gcp Backend up
+     * @param {object} config.s3params - S3 configuration
+     * @param {string} config.bucketName - GCP bucket name
+     * @param {string} config.mpuBucket - GCP mpu bucket name
+     * @param {string} config.overflowBucket - GCP overflow bucket name
+     * @param {boolean} config.bucketMatch - bucket match flag
+     * @param {object} config.authParams - GCP service credentials
+     * @param {string} config.dataStoreName - locationConstraint name
+     * @param {booblean} config.serverSideEncryption - server side encryption
+     * flag
+     * @return {object} - returns a GcpClient object
+     */
+    constructor(config) {
+        super(config);
+        this.clientType = 'gcp';
+        this._gcpBucketName = config.bucketName;
+        this._mpuBucketName = config.mpuBucket;
+        this._overflowBucketname = config.overflowBucket;
+        this._gcpParams = Object.assign(this._s3Params, {
+            mainBucket: this._gcpBucketName,
+            mpuBucket: this._mpuBucketName,
+            overflowBucket: this._overflowBucketname,
+            authParams: config.authParams,
+        });
+        this._client = new GCP(this._gcpParams);
+    }
+}
+
+module.exports = GcpClient;

backends/data/external/utils.js

@@ -0,0 +1,147 @@
+const async = require('async');
+const constants = require('../../../constants');
+const { config } = require('../../../lib/Config');
+
+const awsHealth = {
+    response: undefined,
+    time: 0,
+};
+const azureHealth = {
+    response: undefined,
+    time: 0,
+};
+
+const utils = {
+    logHelper(log, level, description, error, dataStoreName) {
+        log[level](description, { error: error.message,
+          errorName: error.name, dataStoreName });
+    },
+    // take off the 'x-amz-meta-'
+    trimXMetaPrefix(obj) {
+        const newObj = {};
+        Object.keys(obj).forEach(key => {
+            const newKey = key.substring(11);
+            newObj[newKey] = obj[key];
+        });
+        return newObj;
+    },
+    removeQuotes(word) {
+        return word.slice(1, -1);
+    },
+    skipMpuPartProcessing(completeObjData) {
+        const backendType = completeObjData.dataStoreType;
+        if (constants.mpuMDStoredExternallyBackend[backendType]) {
+            return true;
+        }
+        return false;
+    },
+
+    /**
+     * checkAzureBackendMatch - checks that the external backend location for
+     * two data objects is the same and is Azure
+     * @param {array} objectDataOne - data of first object to compare
+     * @param {object} objectDataTwo - data of second object to compare
+     * @return {boolean} - true if both data backends are Azure, false if not
+     */
+    checkAzureBackendMatch(objectDataOne, objectDataTwo) {
+        if (objectDataOne.dataStoreType === 'azure' &&
+        objectDataTwo.dataStoreType === 'azure') {
+            return true;
+        }
+        return false;
+    },
+
+    /**
+     * externalBackendCopy - Server side copy should only be allowed:
+     * 1) if source object and destination object are both on aws or both
+     * on azure.
+     * 2) if azure to azure, must be the same storage account since Azure
+     * copy outside of an account is async
+     * 3) if the source bucket is not an encrypted bucket and the
+     * destination bucket is not an encrypted bucket (unless the copy
+     * is all within the same bucket).
+     * @param {string} locationConstraintSrc - location constraint of the source
+     * @param {string} locationConstraintDest - location constraint of the
+     * destination
+     * @param {object} sourceBucketMD - metadata of the source bucket
+     * @param {object} destBucketMD - metadata of the destination bucket
+     * @return {boolean} - true if copying object from one
+     * externalbackend to the same external backend and for Azure if it is the
+     * same account since Azure copy outside of an account is async
+     */
+    externalBackendCopy(locationConstraintSrc, locationConstraintDest,
+      sourceBucketMD, destBucketMD) {
+        const sourceBucketName = sourceBucketMD.getName();
+        const destBucketName = destBucketMD.getName();
+        const isSameBucket = sourceBucketName === destBucketName;
+        const bucketsNotEncrypted = destBucketMD.getServerSideEncryption()
+        === sourceBucketMD.getServerSideEncryption() &&
+        sourceBucketMD.getServerSideEncryption() === null;
+        const sourceLocationConstraintType =
+        config.getLocationConstraintType(locationConstraintSrc);
+        const locationTypeMatch =
+        config.getLocationConstraintType(locationConstraintSrc) ===
+        config.getLocationConstraintType(locationConstraintDest);
+        return locationTypeMatch && (isSameBucket || bucketsNotEncrypted) &&
+              (sourceLocationConstraintType === 'aws_s3' ||
+              (sourceLocationConstraintType === 'azure' &&
+              config.isSameAzureAccount(locationConstraintSrc,
+              locationConstraintDest)));
+    },
+
+    checkExternalBackend(clients, locations, type, flightCheckOnStartUp,
+      externalBackendHealthCheckInterval, cb) {
+        const checkStatus = type === 'aws_s3' ? awsHealth : azureHealth;
+        if (locations.length === 0) {
+            return process.nextTick(cb, null, []);
+        }
+        if (!flightCheckOnStartUp && checkStatus.response &&
+            Date.now() - checkStatus.time
+            < externalBackendHealthCheckInterval) {
+            return process.nextTick(cb, null, checkStatus.response);
+        }
+        let locationsToCheck;
+        if (flightCheckOnStartUp) {
+            // check all locations if flight check on start up
+            locationsToCheck = locations;
+        } else {
+            const randomLocation = locations[Math.floor(Math.random() *
+                locations.length)];
+            locationsToCheck = [randomLocation];
+        }
+        return async.mapLimit(locationsToCheck, 5, (location, next) => {
+            const client = clients[location];
+            client.healthcheck(location, next, flightCheckOnStartUp);
+        }, (err, results) => {
+            if (err) {
+                return cb(err);
+            }
+            if (!flightCheckOnStartUp) {
+                checkStatus.response = results;
+                checkStatus.time = Date.now();
+            }
+            return cb(null, results);
+        });
+    },
+    translateAzureMetaHeaders(metaHeaders, tags) {
+        const translatedMetaHeaders = {};
+        if (tags) {
+            // tags are passed as string of format 'key1=value1&key2=value2'
+            const tagObj = {};
+            const tagArr = tags.split('&');
+            tagArr.forEach(keypair => {
+                const equalIndex = keypair.indexOf('=');
+                const key = keypair.substring(0, equalIndex);
+                tagObj[key] = keypair.substring(equalIndex + 1);
+            });
+            translatedMetaHeaders.tags = JSON.stringify(tagObj);
+        }
+        Object.keys(metaHeaders).forEach(headerName => {
+            const translated = headerName.substring(11).replace(/-/g, '_');
+            translatedMetaHeaders[translated] = metaHeaders[headerName];
+        });
+        return translatedMetaHeaders;
+    },
+};
+
+module.exports = utils;

backends/data/file/backend.js

@@ -0,0 +1,37 @@
+const arsenal = require('arsenal');
+const { config } = require('../../Config');
+
+class DataFileInterface {
+    constructor() {
+        const { host, port } = config.dataClient;
+
+        this.restClient = new arsenal.network.rest.RESTClient(
+            { host, port });
+    }
+
+    put(stream, size, keyContext, reqUids, callback) {
+        // ignore keyContext
+        this.restClient.put(stream, size, reqUids, callback);
+    }
+
+    get(objectGetInfo, range, reqUids, callback) {
+        const key = objectGetInfo.key ? objectGetInfo.key : objectGetInfo;
+        this.restClient.get(key, range, reqUids, callback);
+    }
+
+    delete(objectGetInfo, reqUids, callback) {
+        const key = objectGetInfo.key ? objectGetInfo.key : objectGetInfo;
+        this.restClient.delete(key, reqUids, callback);
+    }
+
+    getDiskUsage(reqUids, callback) {
+        this.restClient.getAction('diskUsage', reqUids, (err, val) => {
+            if (err) {
+                return callback(err);
+            }
+            return callback(null, JSON.parse(val));
+        });
+    }
+}
+
+module.exports = DataFileInterface;

backends/data/in_memory/backend.js

@@ -0,0 +1,93 @@
+const stream = require('stream');
+const { errors } = require('arsenal');
+const werelogs = require('werelogs');
+
+const logger = new werelogs.Logger('MemDataBackend');
+
+function createLogger(reqUids) {
+    return reqUids ?
+        logger.newRequestLoggerFromSerializedUids(reqUids) :
+        logger.newRequestLogger();
+}
+
+const ds = [];
+let count = 1; // keys are assessed with if (!key)
+
+function resetCount() {
+    count = 1;
+}
+
+const backend = {
+    put: function putMem(request, size, keyContext, reqUids, callback) {
+        const log = createLogger(reqUids);
+        const value = Buffer.alloc(size);
+        let cursor = 0;
+        let exceeded = false;
+        request.on('data', data => {
+            if (cursor + data.length > size) {
+                exceeded = true;
+            }
+            if (!exceeded) {
+                data.copy(value, cursor);
+            }
+            cursor += data.length;
+        })
+        .on('end', () => {
+            if (exceeded) {
+                log.error('data stream exceed announced size',
+                          { size, overflow: cursor });
+                callback(errors.InternalError);
+            } else {
+                ds[count] = { value, keyContext };
+                callback(null, count++);
+            }
+        });
+    },
+
+    get: function getMem(objectGetInfo, range, reqUids, callback) {
+        const key = objectGetInfo.key ? objectGetInfo.key : objectGetInfo;
+        process.nextTick(() => {
+            if (!ds[key]) { return callback(errors.NoSuchKey); }
+            const storedBuffer = ds[key].value;
+            // If a range was sent, use the start from the range.
+            // Otherwise, start at 0
+            let start = range ? range[0] : 0;
+            // If a range was sent, use the end from the range.
+            // End of range should be included so +1
+            // Otherwise, get the full length
+            const end = range ? range[1] + 1 : storedBuffer.length;
+            const chunkSize = 64 * 1024; // 64KB
+            const val = new stream.Readable({
+                read: function read() {
+                    // sets this._read under the hood
+                    // push data onto the read queue, passing null
+                    // will signal the end of the stream (EOF)
+                    while (start < end) {
+                        const finish =
+                            Math.min(start + chunkSize, end);
+                        this.push(storedBuffer.slice(start, finish));
+                        start += chunkSize;
+                    }
+                    if (start >= end) {
+                        this.push(null);
+                    }
+                },
+            });
+            return callback(null, val);
+        });
+    },
+
+    delete: function delMem(objectGetInfo, reqUids, callback) {
+        const key = objectGetInfo.key ? objectGetInfo.key : objectGetInfo;
+        process.nextTick(() => {
+            delete ds[key];
+            return callback(null);
+        });
+    },
+};
+
+module.exports = {
+    backend,
+    ds,
+    resetCount,
+};

backends/data/locationConstraintParser.js

@@ -0,0 +1,126 @@
+const http = require('http');
+const https = require('https');
+const AWS = require('aws-sdk');
+const Sproxy = require('sproxydclient');
+
+const DataFileBackend = require('./file/backend');
+const inMemory = require('./in_memory/backend').backend;
+const AwsClient = require('./external/AwsClient');
+const GcpClient = require('./external/GcpClient');
+const AzureClient = require('./external/AzureClient');
+
+const { config } = require('../Config');
+
+const proxyAddress = 'http://localhost:3128';
+
+function parseLC() {
+    const clients = {};
+
+    Object.keys(config.locationConstraints).forEach(location => {
+        const locationObj = config.locationConstraints[location];
+        if (locationObj.type === 'mem') {
+            clients[location] = inMemory;
+        }
+        if (locationObj.type === 'file') {
+            clients[location] = new DataFileBackend();
+        }
+        if (locationObj.type === 'scality'
+        && locationObj.details.connector.sproxyd) {
+            clients[location] = new Sproxy({
+                bootstrap: locationObj.details.connector
+                    .sproxyd.bootstrap,
+                // Might be undefined which is ok since there is a default
+                // set in sproxydclient if chordCos is undefined
+                chordCos: locationObj.details.connector.sproxyd.chordCos,
+                // Might also be undefined, but there is a default path set
+                // in sproxydclient as well
+                path: locationObj.details.connector.sproxyd.path,
+                // enable immutable optim for all objects
+                immutable: true,
+            });
+            clients[location].clientType = 'scality';
+        }
+        if (locationObj.type === 'aws_s3' || locationObj.type === 'gcp') {
+            if (process.env.CI_PROXY === 'true') {
+                locationObj.details.https = false;
+                locationObj.details.proxy = proxyAddress;
+            }
+            const connectionAgent = locationObj.details.https ?
+                new https.Agent({ keepAlive: true }) :
+                new http.Agent({ keepAlive: true });
+            const protocol = locationObj.details.https ? 'https' : 'http';
+            const httpOptions = locationObj.details.proxy ?
+                { proxy: locationObj.details.proxy, agent: connectionAgent,
+                    timeout: 0 }
+                : { agent: connectionAgent, timeout: 0 };
+            const sslEnabled = locationObj.details.https === true;
+            // TODO: HTTP requests to AWS are not supported with V4 auth for
+            // non-file streams which are used by Backbeat. This option will be
+            // removed once CA certs, proxy setup feature is implemented.
+            const signatureVersion = !sslEnabled ? 'v2' : 'v4';
+            const endpoint = locationObj.type === 'gcp' ?
+                locationObj.details.gcpEndpoint :
+                locationObj.details.awsEndpoint;
+            const s3Params = {
+                endpoint: `${protocol}://${endpoint}`,
+                debug: false,
+                // Not implemented yet for streams in node sdk,
+                // and has no negative impact if stream, so let's
+                // leave it in for future use
+                computeChecksums: true,
+                httpOptions,
+                // needed for encryption
+                signatureVersion,
+                sslEnabled,
+                maxRetries: 0,
+            };
+            // users can either include the desired profile name from their
+            // ~/.aws/credentials file or include the accessKeyId and
+            // secretAccessKey directly in the locationConfig
+            if (locationObj.details.credentialsProfile) {
+                s3Params.credentials = new AWS.SharedIniFileCredentials({
+                    profile: locationObj.details.credentialsProfile });
+            } else {
+                s3Params.accessKeyId =
+                    locationObj.details.credentials.accessKey;
+                s3Params.secretAccessKey =
+                    locationObj.details.credentials.secretKey;
+            }
+            const clientConfig = {
+                s3Params,
+                bucketName: locationObj.details.bucketName,
+                bucketMatch: locationObj.details.bucketMatch,
+                serverSideEncryption: locationObj.details.serverSideEncryption,
+                dataStoreName: location,
+            };
+            if (locationObj.type === 'gcp') {
+                clientConfig.overflowBucket =
+                    locationObj.details.overflowBucketName;
+                clientConfig.mpuBucket = locationObj.details.mpuBucketName;
+                clientConfig.authParams = config.getGcpServiceParams(location);
+            }
+            clients[location] = locationObj.type === 'gcp' ?
+                new GcpClient(clientConfig) : new AwsClient(clientConfig);
+        }
+        if (locationObj.type === 'azure') {
+            if (process.env.CI_PROXY === 'true') {
+                locationObj.details.proxy = proxyAddress;
+            }
+            const azureStorageEndpoint = config.getAzureEndpoint(location);
+            const azureStorageCredentials =
+                config.getAzureStorageCredentials(location);
+            clients[location] = new AzureClient({
+                azureStorageEndpoint,
+                azureStorageCredentials,
+                azureContainerName: locationObj.details.azureContainerName,
+                bucketMatch: locationObj.details.bucketMatch,
+                dataStoreName: location,
+                proxy: locationObj.details.proxy,
+            });
+            clients[location].clientType = 'azure';
+        }
+    });
+    return clients;
+}
+
+module.exports = parseLC;

backends/data/multipleBackendGateway.js

@@ -0,0 +1,289 @@
+const { errors, s3middleware } = require('arsenal');
+const { parseTagFromQuery } = s3middleware.tagging;
+
+const createLogger = require('./multipleBackendLogger');
+const async = require('async');
+
+const { config } = require('../Config');
+const parseLC = require('./locationConstraintParser');
+
+const { checkExternalBackend } = require('./external/utils');
+const { externalBackendHealthCheckInterval } = require('../../constants');
+
+let clients = parseLC(config);
+config.on('location-constraints-update', () => {
+    clients = parseLC(config);
+});
+
+
+const multipleBackendGateway = {
+
+    put: (hashedStream, size, keyContext,
+     backendInfo, reqUids, callback) => {
+        const controllingLocationConstraint =
+            backendInfo.getControllingLocationConstraint();
+        const client = clients[controllingLocationConstraint];
+        if (!client) {
+            const log = createLogger(reqUids);
+            log.error('no data backend matching controlling locationConstraint',
+            { controllingLocationConstraint });
+            return process.nextTick(() => {
+                callback(errors.InternalError);
+            });
+        }
+
+        let writeStream = hashedStream;
+        if (keyContext.cipherBundle && keyContext.cipherBundle.cipher) {
+            writeStream = keyContext.cipherBundle.cipher;
+            hashedStream.pipe(writeStream);
+        }
+
+        if (keyContext.tagging) {
+            const validationTagRes = parseTagFromQuery(keyContext.tagging);
+            if (validationTagRes instanceof Error) {
+                const log = createLogger(reqUids);
+                log.debug('tag validation failed', {
+                    error: validationTagRes,
+                    method: 'multipleBackendGateway put',
+                });
+                return callback(errors.InternalError);
+            }
+        }
+        return client.put(writeStream, size, keyContext,
+            reqUids, (err, key, dataStoreVersionId) => {
+                const log = createLogger(reqUids);
+                log.debug('put to location', { controllingLocationConstraint });
+                if (err) {
+                    log.error('error from datastore',
+                             { error: err, dataStoreType: client.clientType });
+                    return callback(errors.ServiceUnavailable);
+                }
+                const dataRetrievalInfo = {
+                    key,
+                    dataStoreName: controllingLocationConstraint,
+                    dataStoreType: client.clientType,
+                    dataStoreVersionId,
+                };
+                return callback(null, dataRetrievalInfo);
+            });
+    },
+
+    head: (objectGetInfoArr, reqUids, callback) => {
+        if (!objectGetInfoArr || !Array.isArray(objectGetInfoArr)
+            || !objectGetInfoArr[0] || !objectGetInfoArr[0].dataStoreName) {
+            // no-op if no stored data store name
+            return process.nextTick(callback);
+        }
+        const objectGetInfo = objectGetInfoArr[0];
+        const client = clients[objectGetInfo.dataStoreName];
+        if (client.head === undefined) {
+            // no-op if unsupported client method
+            return process.nextTick(callback);
+        }
+        return client.head(objectGetInfo, reqUids, callback);
+    },
+
+    get: (objectGetInfo, range, reqUids, callback) => {
+        let key;
+        let client;
+        // for backwards compatibility
+        if (typeof objectGetInfo === 'string') {
+            key = objectGetInfo;
+            client = clients.sproxyd;
+        } else {
+            key = objectGetInfo.key;
+            client = clients[objectGetInfo.dataStoreName];
+        }
+        if (client.clientType === 'scality') {
+            return client.get(key, range, reqUids, callback);
+        }
+        return client.get(objectGetInfo, range, reqUids, callback);
+    },
+
+    delete: (objectGetInfo, reqUids, callback) => {
+        let key;
+        let client;
+        // for backwards compatibility
+        if (typeof objectGetInfo === 'string') {
+            key = objectGetInfo;
+            client = clients.sproxyd;
+        } else {
+            key = objectGetInfo.key;
+            client = clients[objectGetInfo.dataStoreName];
+        }
+        if (client.clientType === 'scality') {
+            return client.delete(key, reqUids, callback);
+        }
+        return client.delete(objectGetInfo, reqUids, callback);
+    },
+
+    healthcheck: (flightCheckOnStartUp, log, callback) => {
+        const multBackendResp = {};
+        const awsArray = [];
+        const azureArray = [];
+        async.each(Object.keys(clients), (location, cb) => {
+            const client = clients[location];
+            if (client.clientType === 'scality') {
+                return client.healthcheck(log, (err, res) => {
+                    if (err) {
+                        multBackendResp[location] = { error: err };
+                    } else {
+                        multBackendResp[location] = { code: res.statusCode,
+                            message: res.statusMessage };
+                    }
+                    return cb();
+                });
+            } else if (client.clientType === 'aws_s3') {
+                awsArray.push(location);
+                return cb();
+            } else if (client.clientType === 'azure') {
+                azureArray.push(location);
+                return cb();
+            }
+            // if backend type isn't 'scality' or 'aws_s3', it will be
+            //  'mem' or 'file', for which the default response is 200 OK
+            multBackendResp[location] = { code: 200, message: 'OK' };
+            return cb();
+        }, () => {
+            async.parallel([
+                next => checkExternalBackend(
+                  clients, awsArray, 'aws_s3', flightCheckOnStartUp,
+                  externalBackendHealthCheckInterval, next),
+                next => checkExternalBackend(
+                  clients, azureArray, 'azure', flightCheckOnStartUp,
+                  externalBackendHealthCheckInterval, next),
+            ], (errNull, externalResp) => {
+                const externalLocResults = [];
+                externalResp.forEach(resp => externalLocResults.push(...resp));
+                externalLocResults.forEach(locationResult =>
+                  Object.assign(multBackendResp, locationResult));
+                callback(null, multBackendResp);
+            });
+        });
+    },
+
+    createMPU: (key, metaHeaders, bucketName, websiteRedirectHeader,
+    location, contentType, cacheControl, contentDisposition,
+    contentEncoding, log, cb) => {
+        const client = clients[location];
+        if (client.clientType === 'aws_s3') {
+            return client.createMPU(key, metaHeaders, bucketName,
+            websiteRedirectHeader, contentType, cacheControl,
+            contentDisposition, contentEncoding, log, cb);
+        }
+        return cb();
+    },
+
+    uploadPart: (request, streamingV4Params, stream, size, location, key,
+    uploadId, partNumber, bucketName, log, cb) => {
+        const client = clients[location];
+
+        if (client.uploadPart) {
+            return client.uploadPart(request, streamingV4Params, stream, size,
+            key, uploadId, partNumber, bucketName, log, cb);
+        }
+        return cb();
+    },
+
+    listParts: (key, uploadId, location, bucketName, partNumberMarker, maxParts,
+    log, cb) => {
+        const client = clients[location];
+
+        if (client.listParts) {
+            return client.listParts(key, uploadId, bucketName, partNumberMarker,
+                maxParts, log, cb);
+        }
+        return cb();
+    },
+
+    completeMPU: (key, uploadId, location, jsonList, mdInfo, bucketName,
+    userMetadata, contentSettings, log, cb) => {
+        const client = clients[location];
+        if (client.completeMPU) {
+            const args = [jsonList, mdInfo, key, uploadId, bucketName];
+            if (client.clientType === 'azure') {
+                args.push(userMetadata, contentSettings);
+            }
+            return client.completeMPU(...args, log, (err, completeObjData) => {
+                if (err) {
+                    return cb(err);
+                }
+                // eslint-disable-next-line no-param-reassign
+                completeObjData.dataStoreType = client.clientType;
+                return cb(null, completeObjData);
+            });
+        }
+        return cb();
+    },
+
+    abortMPU: (key, uploadId, location, bucketName, log, cb) => {
+        const client = clients[location];
+        if (client.clientType === 'azure') {
+            const skipDataDelete = true;
+            return cb(null, skipDataDelete);
+        }
+        if (client.abortMPU) {
+            return client.abortMPU(key, uploadId, bucketName, log, err => {
+                if (err) {
+                    return cb(err);
+                }
+                return cb();
+            });
+        }
+        return cb();
+    },
+
+    objectTagging: (method, key, bucket, objectMD, log, cb) => {
+        // if legacy, objectMD will not contain dataStoreName, so just return
+        const client = clients[objectMD.dataStoreName];
+        if (client && client[`object${method}Tagging`]) {
+            return client[`object${method}Tagging`](key, bucket, objectMD, log,
+                cb);
+        }
+        return cb();
+    },
+    // NOTE: using copyObject only if copying object from one external
+    // backend to the same external backend
+    copyObject: (request, destLocationConstraintName, externalSourceKey,
+    sourceLocationConstraintName, storeMetadataParams, log, cb) => {
+        const client = clients[destLocationConstraintName];
+        if (client.copyObject) {
+            return client.copyObject(request, destLocationConstraintName,
+            externalSourceKey, sourceLocationConstraintName,
+            storeMetadataParams, log, (err, key, dataStoreVersionId) => {
+                const dataRetrievalInfo = {
+                    key,
+                    dataStoreName: destLocationConstraintName,
+                    dataStoreType: client.clientType,
+                    dataStoreVersionId,
+                };
+                cb(err, dataRetrievalInfo);
+            });
+        }
+        return cb(errors.NotImplemented
+            .customizeDescription('Can not copy object from ' +
+              `${client.clientType} to ${client.clientType}`));
+    },
+    uploadPartCopy: (request, location, awsSourceKey,
+    sourceLocationConstraintName, log, cb) => {
+        const client = clients[location];
+        if (client.uploadPartCopy) {
+            return client.uploadPartCopy(request, awsSourceKey,
+              sourceLocationConstraintName,
+            log, cb);
+        }
+        return cb(errors.NotImplemented.customizeDescription(
+          'Can not copy object from ' +
+          `${client.clientType} to ${client.clientType}`));
+    },
+    protectAzureBlocks: (bucketName, objectKey, location, log, cb) => {
+        const client = clients[location];
+        if (client.protectAzureBlocks) {
+            return client.protectAzureBlocks(bucketName, objectKey, location,
+            log, cb);
+        }
+        return cb();
+    },
+};
+
+module.exports = multipleBackendGateway;

backends/data/multipleBackendLogger.js

@@ -0,0 +1,11 @@
+const werelogs = require('werelogs');
+
+const logger = new werelogs.Logger('MultipleBackendGateway');
+
+function createLogger(reqUids) {
+    return reqUids ?
+        logger.newRequestLoggerFromSerializedUids(reqUids) :
+        logger.newRequestLogger();
+}
+
+module.exports = createLogger;

backends/data/wrapper.js

@@ -0,0 +1,792 @@
+const async = require('async');
+const { errors, s3middleware } = require('arsenal');
+const PassThrough = require('stream').PassThrough;
+
+const DataFileInterface = require('./file/backend');
+const inMemory = require('./in_memory/backend').backend;
+const locationConstraintCheck =
+    require('../api/apiUtils/object/locationConstraintCheck');
+const multipleBackendGateway = require('./multipleBackendGateway');
+const utils = require('./external/utils');
+const { config } = require('../Config');
+const MD5Sum = s3middleware.MD5Sum;
+const NullStream = s3middleware.NullStream;
+const assert = require('assert');
+const kms = require('../kms/wrapper');
+const externalBackends = require('../../constants').externalBackends;
+const constants = require('../../constants');
+const { BackendInfo } = require('../api/apiUtils/object/BackendInfo');
+const RelayMD5Sum = require('../utilities/RelayMD5Sum');
+const skipError = new Error('skip');
+
+let CdmiData;
+try {
+    CdmiData = require('cdmiclient').CdmiData;
+} catch (err) {
+    CdmiData = null;
+}
+
+let client;
+let implName;
+
+if (config.backends.data === 'mem') {
+    client = inMemory;
+    implName = 'mem';
+} else if (config.backends.data === 'file') {
+    client = new DataFileInterface();
+    implName = 'file';
+} else if (config.backends.data === 'multiple') {
+    client = multipleBackendGateway;
+    implName = 'multipleBackends';
+} else if (config.backends.data === 'cdmi') {
+    if (!CdmiData) {
+        throw new Error('Unauthorized backend');
+    }
+
+    client = new CdmiData({
+        path: config.cdmi.path,
+        host: config.cdmi.host,
+        port: config.cdmi.port,
+        readonly: config.cdmi.readonly,
+    });
+    implName = 'cdmi';
+}
+
+/**
+ * _retryDelete - Attempt to delete key again if it failed previously
+ * @param { string | object } objectGetInfo - either string location of object
+ *      to delete or object containing info of object to delete
+ * @param {object} log - Werelogs request logger
+ * @param {number} count - keeps count of number of times function has been run
+ * @param {function} cb - callback
+ * @returns undefined and calls callback
+ */
+const MAX_RETRY = 2;
+
+// This check is done because on a put, complete mpu or copy request to
+// Azure/AWS, if the object already exists on that backend, the existing object
+// should not be deleted, which is the functionality for all other backends
+function _shouldSkipDelete(locations, requestMethod, newObjDataStoreName) {
+    const skipMethods = { PUT: true, POST: true };
+    if (!Array.isArray(locations) || !locations[0] ||
+        !locations[0].dataStoreType) {
+        return false;
+    }
+    const isSkipBackend = externalBackends[locations[0].dataStoreType];
+    const isMatchingBackends =
+        locations[0].dataStoreName === newObjDataStoreName;
+    const isSkipMethod = skipMethods[requestMethod];
+    return (isSkipBackend && isMatchingBackends && isSkipMethod);
+}
+
+function _retryDelete(objectGetInfo, log, count, cb) {
+    if (count > MAX_RETRY) {
+        return cb(errors.InternalError);
+    }
+    return client.delete(objectGetInfo, log.getSerializedUids(), err => {
+        if (err) {
+            log.error('delete error from datastore',
+                      { error: err, implName, moreRetries: 'yes' });
+            return _retryDelete(objectGetInfo, log, count + 1, cb);
+        }
+        return cb();
+    });
+}
+
+function _put(cipherBundle, value, valueSize,
+              keyContext, backendInfo, log, cb) {
+    assert.strictEqual(typeof valueSize, 'number');
+    log.debug('sending put to datastore', { implName, keyContext,
+        method: 'put' });
+    let hashedStream = null;
+    if (value) {
+        hashedStream = new MD5Sum();
+        value.pipe(hashedStream);
+    }
+
+    if (implName === 'multipleBackends') {
+        // Need to send backendInfo to client.put and
+        // client.put will provide dataRetrievalInfo so no
+        // need to construct here
+        /* eslint-disable no-param-reassign */
+        keyContext.cipherBundle = cipherBundle;
+        return client.put(hashedStream,
+               valueSize, keyContext, backendInfo, log.getSerializedUids(),
+               (err, dataRetrievalInfo) => {
+                   if (err) {
+                       log.error('put error from datastore',
+                                 { error: err, implName });
+                       return cb(errors.ServiceUnavailable);
+                   }
+                   return cb(null, dataRetrievalInfo, hashedStream);
+               });
+    }
+    /* eslint-enable no-param-reassign */
+
+    let writeStream = hashedStream;
+    if (cipherBundle && cipherBundle.cipher) {
+        writeStream = cipherBundle.cipher;
+        hashedStream.pipe(writeStream);
+    }
+
+    return client.put(writeStream, valueSize, keyContext,
+                      log.getSerializedUids(), (err, key) => {
+                          if (err) {
+                              log.error('put error from datastore',
+                                        { error: err, implName });
+                              return cb(errors.InternalError);
+                          }
+                          const dataRetrievalInfo = {
+                              key,
+                              dataStoreName: implName,
+                          };
+                          return cb(null, dataRetrievalInfo, hashedStream);
+                      });
+}
+
+const data = {
+    put: (cipherBundle, value, valueSize, keyContext, backendInfo, log, cb) => {
+        _put(cipherBundle, value, valueSize, keyContext, backendInfo, log,
+             (err, dataRetrievalInfo, hashedStream) => {
+                 if (err) {
+                     return cb(err);
+                 }
+                 if (hashedStream) {
+                     if (hashedStream.completedHash) {
+                         return cb(null, dataRetrievalInfo, hashedStream);
+                     }
+                     hashedStream.on('hashed', () => {
+                         hashedStream.removeAllListeners('hashed');
+                         return cb(null, dataRetrievalInfo, hashedStream);
+                     });
+                     return undefined;
+                 }
+                 return cb(null, dataRetrievalInfo);
+             });
+    },
+
+    head: (objectGetInfo, log, cb) => {
+        if (implName !== 'multipleBackends') {
+            // no-op if not multipleBackend implementation;
+            // head is used during get just to check external backend data state
+            return process.nextTick(cb);
+        }
+        return client.head(objectGetInfo, log.getSerializedUids(), cb);
+    },
+
+    get: (objectGetInfo, response, log, cb) => {
+        const isMdModelVersion2 = typeof(objectGetInfo) === 'string';
+        const isRequiredStringKey = constants.clientsRequireStringKey[implName];
+        const key = isMdModelVersion2 ? objectGetInfo : objectGetInfo.key;
+        const clientGetInfo = isRequiredStringKey ? key : objectGetInfo;
+        const range = objectGetInfo.range;
+
+        // If the key is explicitly set to null, the part to
+        // be read doesn't really exist and is only made of zeroes.
+        // This functionality is used by Scality-NFSD.
+        // Otherwise, the key is always defined
+        assert(key === null || key !== undefined);
+        if (key === null) {
+            cb(null, new NullStream(objectGetInfo.size, range));
+            return;
+        }
+        log.debug('sending get to datastore', { implName,
+            key, range, method: 'get' });
+        // We need to use response as a writable stream for AZURE GET
+        if (!isMdModelVersion2 && !isRequiredStringKey && response) {
+            clientGetInfo.response = response;
+        }
+        client.get(clientGetInfo, range, log.getSerializedUids(),
+            (err, stream) => {
+                if (err) {
+                    log.error('get error from datastore',
+                              { error: err, implName });
+                    return cb(errors.ServiceUnavailable);
+                }
+                if (objectGetInfo.cipheredDataKey) {
+                    const serverSideEncryption = {
+                        cryptoScheme: objectGetInfo.cryptoScheme,
+                        masterKeyId: objectGetInfo.masterKeyId,
+                        cipheredDataKey: Buffer.from(
+                            objectGetInfo.cipheredDataKey, 'base64'),
+                    };
+                    const offset = objectGetInfo.range ?
+                        objectGetInfo.range[0] : 0;
+                    return kms.createDecipherBundle(
+                        serverSideEncryption, offset, log,
+                        (err, decipherBundle) => {
+                            if (err) {
+                                log.error('cannot get decipher bundle ' +
+                                    'from kms', {
+                                        method: 'data.wrapper.data.get',
+                                    });
+                                return cb(err);
+                            }
+                            stream.pipe(decipherBundle.decipher);
+                            return cb(null, decipherBundle.decipher);
+                        });
+                }
+                return cb(null, stream);
+            });
+    },
+
+    delete: (objectGetInfo, log, cb) => {
+        const callback = cb || log.end;
+        const isMdModelVersion2 = typeof(objectGetInfo) === 'string';
+        const isRequiredStringKey = constants.clientsRequireStringKey[implName];
+        const key = isMdModelVersion2 ? objectGetInfo : objectGetInfo.key;
+        const clientGetInfo = isRequiredStringKey ? key : objectGetInfo;
+
+        log.trace('sending delete to datastore', {
+            implName, key, method: 'delete' });
+        // If the key is explicitly set to null, the part to
+        // be deleted doesn't really exist.
+        // This functionality is used by Scality-NFSD.
+        // Otherwise, the key is always defined
+        assert(key === null || key !== undefined);
+        if (key === null) {
+            callback(null);
+            return;
+        }
+        _retryDelete(clientGetInfo, log, 0, err => {
+            if (err) {
+                log.error('delete error from datastore',
+                    { error: err, key: objectGetInfo.key, moreRetries: 'no' });
+            }
+            return callback(err);
+        });
+    },
+    // It would be preferable to have an sproxyd batch delete route to
+    // replace this
+    batchDelete: (locations, requestMethod, newObjDataStoreName, log) => {
+        // TODO: The method of persistence of sproxy delete key will
+        // be finalized; refer Issue #312 for the discussion. In the
+        // meantime, we at least log the location of the data we are
+        // about to delete before attempting its deletion.
+        if (_shouldSkipDelete(locations, requestMethod, newObjDataStoreName)) {
+            return;
+        }
+        log.trace('initiating batch delete', {
+            keys: locations,
+            implName,
+            method: 'batchDelete',
+        });
+        async.eachLimit(locations, 5, (loc, next) => {
+            process.nextTick(() => data.delete(loc, log, next));
+        },
+        err => {
+            if (err) {
+                log.error('batch delete failed', { error: err });
+            } else {
+                log.trace('batch delete successfully completed');
+            }
+            log.end();
+        });
+    },
+
+    switch: newClient => {
+        client = newClient;
+        return client;
+    },
+
+    checkHealth: (log, cb, flightCheckOnStartUp) => {
+        if (!client.healthcheck) {
+            const defResp = {};
+            defResp[implName] = { code: 200, message: 'OK' };
+            return cb(null, defResp);
+        }
+        return client.healthcheck(flightCheckOnStartUp, log, (err, result) => {
+            let respBody = {};
+            if (err) {
+                log.error(`error from ${implName}`, { error: err });
+                respBody[implName] = {
+                    error: err,
+                };
+                // error returned as null so async parallel doesn't return
+                // before all backends are checked
+                return cb(null, respBody);
+            }
+            if (implName === 'multipleBackends') {
+                respBody = result;
+                return cb(null, respBody);
+            }
+            respBody[implName] = {
+                code: result.statusCode,
+                message: result.statusMessage,
+            };
+            return cb(null, respBody);
+        });
+    },
+
+    getDiskUsage: (log, cb) => {
+        if (!client.getDiskUsage) {
+            log.debug('returning empty disk usage as fallback', { implName });
+            return cb(null, {});
+        }
+        return client.getDiskUsage(log.getSerializedUids(), cb);
+    },
+
+
+   /**
+    * _putForCopy - put used for copying object
+    * @param {object} cipherBundle - cipher bundle that encrypt the data
+    * @param {object} stream - stream containing the data
+    * @param {object} part - element of dataLocator array
+    * @param {object} dataStoreContext - information of the
+    * destination object
+    * dataStoreContext.bucketName: destination bucket name,
+    * dataStoreContext.owner: owner,
+    * dataStoreContext.namespace: request namespace,
+    * dataStoreContext.objectKey: destination object key name,
+    * @param {BackendInfo} destBackendInfo - Instance of BackendInfo:
+    * Represents the info necessary to evaluate which data backend to use
+    * on a data put call.
+    * @param {object} log - Werelogs request logger
+    * @param {function} cb - callback
+    * @returns {function} cb - callback
+    */
+    _putForCopy: (cipherBundle, stream, part, dataStoreContext,
+    destBackendInfo, log, cb) => data.put(cipherBundle, stream,
+        part.size, dataStoreContext,
+        destBackendInfo, log,
+        (error, partRetrievalInfo) => {
+            if (error) {
+                return cb(error);
+            }
+            const partResult = {
+                key: partRetrievalInfo.key,
+                dataStoreName: partRetrievalInfo
+                    .dataStoreName,
+                dataStoreType: partRetrievalInfo
+                    .dataStoreType,
+                start: part.start,
+                size: part.size,
+            };
+            if (cipherBundle) {
+                partResult.cryptoScheme = cipherBundle.cryptoScheme;
+                partResult.cipheredDataKey = cipherBundle.cipheredDataKey;
+            }
+            if (part.dataStoreETag) {
+                partResult.dataStoreETag = part.dataStoreETag;
+            }
+            if (partRetrievalInfo.dataStoreVersionId) {
+                partResult.dataStoreVersionId =
+                partRetrievalInfo.dataStoreVersionId;
+            }
+            return cb(null, partResult);
+        }),
+
+    /**
+     * _dataCopyPut - put used for copying object with and without
+     * encryption
+     * @param {string} serverSideEncryption - Server side encryption
+     * @param {object} stream - stream containing the data
+     * @param {object} part - element of dataLocator array
+     * @param {object} dataStoreContext - information of the
+     * destination object
+     * dataStoreContext.bucketName: destination bucket name,
+     * dataStoreContext.owner: owner,
+     * dataStoreContext.namespace: request namespace,
+     * dataStoreContext.objectKey: destination object key name,
+     * @param {BackendInfo} destBackendInfo - Instance of BackendInfo:
+     * Represents the info necessary to evaluate which data backend to use
+     * on a data put call.
+     * @param {object} log - Werelogs request logger
+     * @param {function} cb - callback
+     * @returns {function} cb - callback
+     */
+    _dataCopyPut: (serverSideEncryption, stream, part, dataStoreContext,
+    destBackendInfo, log, cb) => {
+        if (serverSideEncryption) {
+            return kms.createCipherBundle(
+            serverSideEncryption,
+            log, (err, cipherBundle) => {
+                if (err) {
+                    log.debug('error getting cipherBundle');
+                    return cb(errors.InternalError);
+                }
+                return data._putForCopy(cipherBundle, stream, part,
+                  dataStoreContext, destBackendInfo, log, cb);
+            });
+        }
+        // Copied object is not encrypted so just put it
+        // without a cipherBundle
+        return data._putForCopy(null, stream, part, dataStoreContext,
+          destBackendInfo, log, cb);
+    },
+
+    /**
+     * copyObject - copy object
+     * @param {object} request - request object
+     * @param {string} sourceLocationConstraintName -
+     * source locationContraint name (awsbackend, azurebackend, ...)
+     * @param {object} storeMetadataParams - metadata information of the
+     * source object
+     * @param {array} dataLocator - source object metadata location(s)
+     * NOTE: for Azure and AWS data backend this array only has one item
+     * @param {object} dataStoreContext - information of the
+     * destination object
+     * dataStoreContext.bucketName: destination bucket name,
+     * dataStoreContext.owner: owner,
+     * dataStoreContext.namespace: request namespace,
+     * dataStoreContext.objectKey: destination object key name,
+     * @param {BackendInfo} destBackendInfo - Instance of BackendInfo:
+     * Represents the info necessary to evaluate which data backend to use
+     * on a data put call.
+     * @param {object} sourceBucketMD - metadata of the source bucket
+     * @param {object} destBucketMD - metadata of the destination bucket
+     * @param {object} log - Werelogs request logger
+     * @param {function} cb - callback
+     * @returns {function} cb - callback
+     */
+    copyObject: (request,
+      sourceLocationConstraintName, storeMetadataParams, dataLocator,
+      dataStoreContext, destBackendInfo, sourceBucketMD, destBucketMD, log,
+      cb) => {
+        const serverSideEncryption = destBucketMD.getServerSideEncryption();
+        if (config.backends.data === 'multiple' &&
+        utils.externalBackendCopy(sourceLocationConstraintName,
+        storeMetadataParams.dataStoreName, sourceBucketMD, destBucketMD)) {
+            const destLocationConstraintName =
+              storeMetadataParams.dataStoreName;
+            const objectGetInfo = dataLocator[0];
+            const externalSourceKey = objectGetInfo.key;
+            return client.copyObject(request, destLocationConstraintName,
+            externalSourceKey, sourceLocationConstraintName,
+            storeMetadataParams, log, (error, objectRetrievalInfo) => {
+                if (error) {
+                    return cb(error);
+                }
+                const putResult = {
+                    key: objectRetrievalInfo.key,
+                    dataStoreName: objectRetrievalInfo.
+                        dataStoreName,
+                    dataStoreType: objectRetrievalInfo.
+                        dataStoreType,
+                    dataStoreVersionId:
+                        objectRetrievalInfo.dataStoreVersionId,
+                    size: storeMetadataParams.size,
+                    dataStoreETag: objectGetInfo.dataStoreETag,
+                    start: objectGetInfo.start,
+                };
+                const putResultArr = [putResult];
+                return cb(null, putResultArr);
+            });
+        }
+
+        // dataLocator is an array.  need to get and put all parts
+        // For now, copy 1 part at a time. Could increase the second
+        // argument here to increase the number of parts
+        // copied at once.
+        return async.mapLimit(dataLocator, 1,
+            // eslint-disable-next-line prefer-arrow-callback
+            function copyPart(part, copyCb) {
+                if (part.dataStoreType === 'azure') {
+                    const passThrough = new PassThrough();
+                    return async.parallel([
+                        parallelCb => data.get(part, passThrough, log, err =>
+                          parallelCb(err)),
+                        parallelCb => data._dataCopyPut(serverSideEncryption,
+                            passThrough,
+                            part, dataStoreContext, destBackendInfo, log,
+                            parallelCb),
+                    ], (err, res) => {
+                        if (err) {
+                            return copyCb(err);
+                        }
+                        return copyCb(null, res[1]);
+                    });
+                }
+                return data.get(part, null, log, (err, stream) => {
+                    if (err) {
+                        return copyCb(err);
+                    }
+                    return data._dataCopyPut(serverSideEncryption, stream,
+                    part, dataStoreContext, destBackendInfo, log, copyCb);
+                });
+            }, (err, results) => {
+                if (err) {
+                    log.debug('error transferring data from source',
+                    { error: err });
+                    return cb(err);
+                }
+                return cb(null, results);
+            });
+    },
+
+
+    _dataCopyPutPart: (request,
+      serverSideEncryption, stream, part,
+      dataStoreContext, destBackendInfo, locations, log, cb) => {
+        const numberPartSize =
+          Number.parseInt(part.size, 10);
+        const partNumber = Number.parseInt(request.query.partNumber, 10);
+        const uploadId = request.query.uploadId;
+        const destObjectKey = request.objectKey;
+        const destBucketName = request.bucketName;
+        const destLocationConstraintName = destBackendInfo
+        .getControllingLocationConstraint();
+        if (externalBackends[config
+            .locationConstraints[destLocationConstraintName]
+            .type]) {
+            return multipleBackendGateway.uploadPart(null, null,
+            stream, numberPartSize,
+            destLocationConstraintName, destObjectKey, uploadId,
+            partNumber, destBucketName, log,
+            (err, partInfo) => {
+                if (err) {
+                    log.error('error putting ' +
+                    'part to AWS', {
+                        error: err,
+                        method:
+                        'objectPutCopyPart::' +
+                        'multipleBackendGateway.' +
+                        'uploadPart',
+                    });
+                    return cb(errors.ServiceUnavailable);
+                }
+                // skip to end of waterfall
+                // because don't need to store
+                // part metadata
+                if (partInfo &&
+                    partInfo.dataStoreType === 'aws_s3') {
+                    // if data backend handles MPU, skip to end
+                    // of waterfall
+                    const partResult = {
+                        dataStoreETag: partInfo.dataStoreETag,
+                    };
+                    locations.push(partResult);
+                    return cb(skipError, partInfo.dataStoreETag);
+                } else if (
+                  partInfo &&
+                  partInfo.dataStoreType === 'azure') {
+                    const partResult = {
+                        key: partInfo.key,
+                        dataStoreName: partInfo.dataStoreName,
+                        dataStoreETag: partInfo.dataStoreETag,
+                        size: numberPartSize,
+                        numberSubParts:
+                          partInfo.numberSubParts,
+                        partNumber: partInfo.partNumber,
+                    };
+                    locations.push(partResult);
+                    return cb();
+                }
+                return cb(skipError);
+            });
+        }
+        if (serverSideEncryption) {
+            return kms.createCipherBundle(
+                serverSideEncryption,
+                log, (err, cipherBundle) => {
+                    if (err) {
+                        log.debug('error getting cipherBundle',
+                        { error: err });
+                        return cb(errors.InternalError);
+                    }
+                    return data.put(cipherBundle, stream,
+                        numberPartSize, dataStoreContext,
+                        destBackendInfo, log,
+                        (error, partRetrievalInfo,
+                        hashedStream) => {
+                            if (error) {
+                                log.debug('error putting ' +
+                                'encrypted part', { error });
+                                return cb(error);
+                            }
+                            const partResult = {
+                                key: partRetrievalInfo.key,
+                                dataStoreName: partRetrievalInfo
+                                    .dataStoreName,
+                                dataStoreETag: hashedStream
+                                    .completedHash,
+                                // Do not include part start
+                                // here since will change in
+                                // final MPU object
+                                size: numberPartSize,
+                                sseCryptoScheme: cipherBundle
+                                    .cryptoScheme,
+                                sseCipheredDataKey: cipherBundle
+                                    .cipheredDataKey,
+                                sseAlgorithm: cipherBundle
+                                    .algorithm,
+                                sseMasterKeyId: cipherBundle
+                                    .masterKeyId,
+                            };
+                            locations.push(partResult);
+                            return cb();
+                        });
+                });
+        }
+        // Copied object is not encrypted so just put it
+        // without a cipherBundle
+        return data.put(null, stream, numberPartSize,
+        dataStoreContext, destBackendInfo,
+        log, (error, partRetrievalInfo, hashedStream) => {
+            if (error) {
+                log.debug('error putting object part',
+                { error });
+                return cb(error);
+            }
+            const partResult = {
+                key: partRetrievalInfo.key,
+                dataStoreName: partRetrievalInfo.dataStoreName,
+                dataStoreETag: hashedStream.completedHash,
+                size: numberPartSize,
+            };
+            locations.push(partResult);
+            return cb();
+        });
+    },
+
+    /**
+     * uploadPartCopy - put copy part
+     * @param {object} request - request object
+     * @param {object} log - Werelogs request logger
+     * @param {object} destBucketMD - destination bucket metadata
+     * @param {string} sourceLocationConstraintName -
+     * source locationContraint name (awsbackend, azurebackend, ...)
+     * @param {string} destLocationConstraintName -
+     * location of the destination MPU object (awsbackend, azurebackend, ...)
+     * @param {array} dataLocator - source object metadata location(s)
+     * NOTE: for Azure and AWS data backend this array
+     * @param {object} dataStoreContext - information of the
+     * destination object
+     * dataStoreContext.bucketName: destination bucket name,
+     * dataStoreContext.owner: owner,
+     * dataStoreContext.namespace: request namespace,
+     * dataStoreContext.objectKey: destination object key name,
+     * dataStoreContext.uploadId: uploadId
+     * dataStoreContext.partNumber: request.query.partNumber
+     * @param {function} callback - callback
+     * @returns {function} cb - callback
+     */
+    uploadPartCopy: (request, log, destBucketMD, sourceLocationConstraintName,
+      destLocationConstraintName, dataLocator, dataStoreContext,
+      callback) => {
+        const serverSideEncryption = destBucketMD.getServerSideEncryption();
+        const lastModified = new Date().toJSON();
+
+        // skip if 0 byte object
+        if (dataLocator.length === 0) {
+            return process.nextTick(() => {
+                callback(null, constants.emptyFileMd5,
+                    lastModified, serverSideEncryption, []);
+            });
+        }
+
+        // if destination mpu was initiated in legacy version
+        if (destLocationConstraintName === undefined) {
+            const backendInfoObj = locationConstraintCheck(request,
+                null, destBucketMD, log);
+            if (backendInfoObj.err) {
+                return process.nextTick(() => {
+                    callback(backendInfoObj.err);
+                });
+            }
+            // eslint-disable-next-line no-param-reassign
+            destLocationConstraintName = backendInfoObj.controllingLC;
+        }
+
+        const locationTypeMatchAWS =
+        config.backends.data === 'multiple' &&
+        config.getLocationConstraintType(sourceLocationConstraintName) ===
+        config.getLocationConstraintType(destLocationConstraintName) &&
+        config.getLocationConstraintType(sourceLocationConstraintName) ===
+        'aws_s3';
+
+        // NOTE: using multipleBackendGateway.uploadPartCopy only if copying
+        // from AWS to AWS
+
+        if (locationTypeMatchAWS && dataLocator.length === 1) {
+            const awsSourceKey = dataLocator[0].key;
+            return multipleBackendGateway.uploadPartCopy(request,
+            destLocationConstraintName, awsSourceKey,
+            sourceLocationConstraintName, log, (error, eTag) => {
+                if (error) {
+                    return callback(error);
+                }
+                return callback(skipError, eTag,
+                    lastModified, serverSideEncryption);
+            });
+        }
+
+        const backendInfo = new BackendInfo(destLocationConstraintName);
+
+        // totalHash will be sent through the RelayMD5Sum transform streams
+        // to collect the md5 from multiple streams
+        let totalHash;
+        const locations = [];
+         // dataLocator is an array.  need to get and put all parts
+         // in order so can get the ETag of full object
+        return async.forEachOfSeries(dataLocator,
+            // eslint-disable-next-line prefer-arrow-callback
+            function copyPart(part, index, cb) {
+                if (part.dataStoreType === 'azure') {
+                    const passThrough = new PassThrough();
+                    return async.parallel([
+                        next => data.get(part, passThrough, log, err => {
+                            if (err) {
+                                log.error('error getting data part ' +
+                                'from Azure', {
+                                    error: err,
+                                    method:
+                                    'objectPutCopyPart::' +
+                                    'multipleBackendGateway.' +
+                                    'copyPart',
+                                });
+                                return next(err);
+                            }
+                            return next();
+                        }),
+                        next => data._dataCopyPutPart(request,
+                          serverSideEncryption, passThrough, part,
+                          dataStoreContext, backendInfo, locations, log, next),
+                    ], err => {
+                        if (err) {
+                            return cb(err);
+                        }
+                        return cb();
+                    });
+                }
+                return data.get(part, null, log, (err, stream) => {
+                    if (err) {
+                        log.debug('error getting object part',
+                        { error: err });
+                        return cb(err);
+                    }
+                    const hashedStream =
+                        new RelayMD5Sum(totalHash, updatedHash => {
+                            totalHash = updatedHash;
+                        });
+                    stream.pipe(hashedStream);
+
+                    // destLocationConstraintName is location of the
+                    // destination MPU object
+                    return data._dataCopyPutPart(request,
+                      serverSideEncryption, hashedStream, part,
+                      dataStoreContext, backendInfo, locations, log, cb);
+                });
+            }, err => {
+                // Digest the final combination of all of the part streams
+                if (err && err !== skipError) {
+                    log.debug('error transferring data from source',
+                    { error: err, method: 'goGetData' });
+                    return callback(err);
+                }
+                if (totalHash) {
+                    totalHash = totalHash.digest('hex');
+                } else {
+                    totalHash = locations[0].dataStoreETag;
+                }
+                if (err && err === skipError) {
+                    return callback(skipError, totalHash,
+                        lastModified, serverSideEncryption);
+                }
+                return callback(null, totalHash,
+                    lastModified, serverSideEncryption, locations);
+            });
+    },
+};
+
+module.exports = data;

backends/metadata/ModelVersion.md

@@ -0,0 +1,82 @@
+# BucketInfo Model Version History
+
+## Model Version 0/1
+
+### Properties
+
+``` javascript
+this._acl = aclInstance;
+this._name = name;
+this._owner = owner;
+this._ownerDisplayName = ownerDisplayName;
+this._creationDate = creationDate;
+```
+
+### Usage
+
+No explicit references in the code since mdBucketModelVersion
+property not added until Model Version 2
+
+## Model Version 2
+
+### Properties Added
+
+``` javascript
+this._mdBucketModelVersion = mdBucketModelVersion || 0
+this._transient = transient || false;
+this._deleted = deleted || false;
+```
+
+### Usage
+
+Used to determine which splitter to use ( < 2 means old splitter)
+
+## Model version 3
+
+### Properties Added
+
+```
+this._serverSideEncryption = serverSideEncryption || null;
+```
+
+### Usage
+
+Used to store the server bucket encryption info
+
+## Model version 4
+
+### Properties Added
+
+```javascript
+this._locationConstraint = LocationConstraint || null;
+```
+
+### Usage
+
+Used to store the location constraint of the bucket
+
+## Model version 5
+
+### Properties Added
+
+```javascript
+this._websiteConfiguration = websiteConfiguration || null;
+this._cors = cors || null;
+```
+
+### Usage
+
+Used to store the bucket website configuration info
+and to store CORS rules to apply to cross-domain requests
+
+## Model version 6
+
+### Properties Added
+
+```javascript
+this._lifecycleConfiguration = lifecycleConfiguration || null;
+```
+
+### Usage
+
+Used to store the bucket lifecycle configuration info

backends/metadata/acl.js

@@ -1,5 +1,6 @@
 const { errors } = require('arsenal');
 
+const getReplicationInfo = require('../api/apiUtils/object/getReplicationInfo');
 const aclUtils = require('../utilities/aclUtils');
 const constants = require('../../constants');
 const metadata = require('../metadata/wrapper');
@@ -12,40 +13,18 @@ const acl = {
         metadata.updateBucket(bucket.getName(), bucket, log, cb);
     },
 
-    /**
-     * returns true if the specified ACL grant is unchanged
-     * @param {string} grant name of the grant
-     * @param {object} oldAcl old acl config
-     * @param {object} newAcl new acl config
-     * @returns {bool} is the grant the same
-     */
-    _aclGrantDidNotChange(grant, oldAcl, newAcl) {
-        if (grant === 'Canned') {
-            return oldAcl.Canned === newAcl.Canned;
-        }
-        /**
-         * An ACL grant is in form of an array of strings
-         * An ACL grant is considered unchanged when both the old and new one
-         * contain the same number of elements, and all elements from one
-         * grant are incuded in the other grant
-         */
-        return oldAcl[grant].length === newAcl[grant].length
-            && oldAcl[grant].every(value => newAcl[grant].includes(value));
-    },
-
     addObjectACL(bucket, objectKey, objectMD, addACLParams, params, log, cb) {
         log.trace('updating object acl in metadata');
-        const isAclUnchanged = Object.keys(objectMD.acl).length === Object.keys(addACLParams).length
+        // eslint-disable-next-line no-param-reassign
-            && Object.keys(objectMD.acl).every(grant => this._aclGrantDidNotChange(grant, objectMD.acl, addACLParams));
-        if (!isAclUnchanged) {
-            /* eslint-disable no-param-reassign */
         objectMD.acl = addACLParams;
-            objectMD.originOp = 's3:ObjectAcl:Put';
+        const replicationInfo = getReplicationInfo(objectKey, bucket, true);
-            /* eslint-disable no-param-reassign */
+        if (replicationInfo) {
-            return metadata.putObjectMD(bucket.getName(), objectKey, objectMD, params, log,
+            // eslint-disable-next-line no-param-reassign
-                cb);
+            objectMD.replicationInfo = Object.assign({},
+                objectMD.replicationInfo, replicationInfo);
         }
-        return cb();
+        metadata.putObjectMD(bucket.getName(), objectKey, objectMD, params, log,
+            cb);
     },
 
     parseAclFromHeaders(params, cb) {
@@ -157,4 +136,3 @@ const acl = {
 };
 
 module.exports = acl;
-

backends/metadata/bucketclient/backend.js

@@ -0,0 +1,213 @@
+const assert = require('assert');
+const bucketclient = require('bucketclient');
+
+const BucketInfo = require('arsenal').models.BucketInfo;
+const logger = require('../../utilities/logger');
+const { config } = require('../../Config');
+
+class BucketClientInterface {
+    constructor() {
+        assert(config.bucketd.bootstrap.length > 0,
+               'bucketd bootstrap list is empty');
+        const { bootstrap, log } = config.bucketd;
+        if (config.https) {
+            const { key, cert, ca } = config.https;
+            logger.info('bucketclient configuration', {
+                bootstrap,
+                log,
+                https: true,
+            });
+            this.client = new bucketclient.RESTClient(bootstrap, log, true,
+                key, cert, ca);
+        } else {
+            logger.info('bucketclient configuration', {
+                bootstrap,
+                log,
+                https: false,
+            });
+            this.client = new bucketclient.RESTClient(bootstrap, log);
+        }
+    }
+
+    createBucket(bucketName, bucketMD, log, cb) {
+        this.client.createBucket(bucketName, log.getSerializedUids(),
+                                 bucketMD.serialize(), cb);
+        return null;
+    }
+
+    getBucketAttributes(bucketName, log, cb) {
+        this.client.getBucketAttributes(bucketName, log.getSerializedUids(),
+            (err, data) => {
+                if (err) {
+                    return cb(err);
+                }
+                return cb(err, BucketInfo.deSerialize(data));
+            });
+        return null;
+    }
+
+    getBucketAndObject(bucketName, objName, params, log, cb) {
+        this.client.getBucketAndObject(bucketName, objName,
+            log.getSerializedUids(), (err, data) => {
+                if (err && (!err.NoSuchKey && !err.ObjNotFound)) {
+                    return cb(err);
+                }
+                return cb(null, JSON.parse(data));
+            }, params);
+        return null;
+    }
+
+    getRaftBuckets(raftId, log, cb) {
+        return this.client.getRaftBuckets(raftId, log.getSerializedUids(),
+        (err, data) => {
+            if (err) {
+                return cb(err);
+            }
+            return cb(null, JSON.parse(data));
+        });
+    }
+
+    putBucketAttributes(bucketName, bucketMD, log, cb) {
+        this.client.putBucketAttributes(bucketName, log.getSerializedUids(),
+                                        bucketMD.serialize(), cb);
+        return null;
+    }
+
+    deleteBucket(bucketName, log, cb) {
+        this.client.deleteBucket(bucketName, log.getSerializedUids(), cb);
+        return null;
+    }
+
+    putObject(bucketName, objName, objVal, params, log, cb) {
+        this.client.putObject(bucketName, objName, JSON.stringify(objVal),
+            log.getSerializedUids(), cb, params);
+        return null;
+    }
+
+    getObject(bucketName, objName, params, log, cb) {
+        this.client.getObject(bucketName, objName, log.getSerializedUids(),
+            (err, data) => {
+                if (err) {
+                    return cb(err);
+                }
+                return cb(err, JSON.parse(data));
+            }, params);
+        return null;
+    }
+
+    deleteObject(bucketName, objName, params, log, cb) {
+        this.client.deleteObject(bucketName, objName, log.getSerializedUids(),
+                                 cb, params);
+        return null;
+    }
+
+    listObject(bucketName, params, log, cb) {
+        this.client.listObject(bucketName, log.getSerializedUids(), params,
+            (err, data) => {
+                if (err) {
+                    return cb(err);
+                }
+                return cb(err, JSON.parse(data));
+            });
+        return null;
+    }
+
+    listMultipartUploads(bucketName, params, log, cb) {
+        this.client.listObject(bucketName, log.getSerializedUids(), params,
+            (err, data) => {
+                if (err) {
+                    return cb(err);
+                }
+                return cb(null, JSON.parse(data));
+            });
+        return null;
+    }
+
+    _analyzeHealthFailure(log, callback) {
+        let doFail = false;
+        const reason = {
+            msg: 'Map is available and failure ratio is acceptable',
+        };
+
+        // The healthCheck exposed by Bucketd is a light one, we need
+        // to inspect all the RaftSession's statuses to make sense of
+        // it:
+        return this.client.getAllRafts(undefined, (error, payload) => {
+            let statuses = null;
+            try {
+                statuses = JSON.parse(payload);
+            } catch (e) {
+                doFail = true;
+                reason.msg = 'could not interpret status: invalid payload';
+                // Can't do anything anymore if we fail here. return.
+                return callback(doFail, reason);
+            }
+
+            const reducer = (acc, payload) => acc + !payload.connectedToLeader;
+            reason.ratio = statuses.reduce(reducer, 0) / statuses.length;
+            /* NOTE FIXME/TODO: acceptableRatio could be configured later on */
+            reason.acceptableRatio = 0.5;
+            /* If the RaftSession 0 (map) does not work, fail anyways */
+            if (!doFail && !statuses[0].connectedToLeader) {
+                doFail = true;
+                reason.msg = 'Bucket map unavailable';
+            }
+            if (!doFail && reason.ratio > reason.acceptableRatio) {
+                doFail = true;
+                reason.msg = 'Ratio of failing Raft Sessions is too high';
+            }
+            return callback(doFail, reason);
+        }, log);
+    }
+
+    /*
+     * Bucketd offers a behavior that diverges from other sub-components on the
+     * healthCheck: If any of the pieces making up the bucket storage fail (ie:
+     * if any Raft Session is down), bucketd returns a 500 for the healthCheck.
+     *
+     * As seen in S3C-1412, this may become an issue for S3, whenever the
+     * system is only partly failing.
+     *
+     * This means that S3 needs to analyze the situation, and interpret this
+     * status depending on the analysis. S3 will then assess the situation as
+     * critical (and consider it a failure), or not (and consider it a success
+     * anyways, thus not diverging from the healthCheck behavior of other
+     * components).
+     */
+    checkHealth(implName, log, cb) {
+        return this.client.healthcheck(log, (err, result) => {
+            const respBody = {};
+            if (err) {
+                return this._analyzeHealthFailure(log, (failure, reason) => {
+                    const message = reason.msg;
+                    // Remove 'msg' from the reason payload.
+                    // eslint-disable-next-line no-param-reassign
+                    reason.msg = undefined;
+                    respBody[implName] = {
+                        code: 200,
+                        message,        // Provide interpreted reason msg
+                        body: reason,   // Provide analysis data
+                    };
+                    if (failure) {
+                        // Setting the `error` field is how the healthCheck
+                        // logic interprets it as an error. Don't forget it !
+                        respBody[implName].error = err;
+                        respBody[implName].code = err.code; // original error
+                    }
+                    // error returned as null so async parallel doesn't return
+                    // before all backends are checked
+                    return cb(null, respBody);
+                }, log);
+            }
+            const parseResult = JSON.parse(result);
+            respBody[implName] = {
+                code: 200,
+                message: 'OK',
+                body: parseResult,
+            };
+            return cb(null, respBody);
+        });
+    }
+}
+
+module.exports = BucketClientInterface;

backends/metadata/bucketfile/backend.js

@@ -0,0 +1,379 @@
+const cluster = require('cluster');
+const arsenal = require('arsenal');
+
+const logger = require('../../utilities/logger');
+const BucketInfo = arsenal.models.BucketInfo;
+const constants = require('../../../constants');
+const { config } = require('../../Config');
+
+const errors = arsenal.errors;
+const MetadataFileClient = arsenal.storage.metadata.MetadataFileClient;
+const versionSep = arsenal.versioning.VersioningConstants.VersionId.Separator;
+
+const METASTORE = '__metastore';
+
+class BucketFileInterface {
+
+    /**
+     * @constructor
+     * @param {object} [params] - constructor params
+     * @param {boolean} [params.noDbOpen=false] - true to skip DB open
+     *   (for unit tests only)
+     */
+    constructor(params) {
+        this.logger = logger;
+        const { host, port } = config.metadataClient;
+        this.mdClient = new MetadataFileClient({ host, port });
+        if (params && params.noDbOpen) {
+            return;
+        }
+        this.mdDB = this.mdClient.openDB(err => {
+            if (err) {
+                throw err;
+            }
+            // the metastore sublevel is used to store bucket attributes
+            this.metastore = this.mdDB.openSub(METASTORE);
+            if (cluster.isMaster) {
+                this.setupMetadataServer();
+            }
+        });
+    }
+
+    setupMetadataServer() {
+        /* Since the bucket creation API is expecting the
+           usersBucket to have attributes, we pre-create the
+           usersBucket attributes here */
+        this.mdClient.logger.debug('setting up metadata server');
+        const usersBucketAttr = new BucketInfo(constants.usersBucket,
+            'admin', 'admin', new Date().toJSON(),
+            BucketInfo.currentModelVersion());
+        this.metastore.put(
+            constants.usersBucket,
+            usersBucketAttr.serialize(), {}, err => {
+                if (err) {
+                    this.logger.fatal('error writing usersBucket ' +
+                                      'attributes to metadata',
+                                      { error: err });
+                    throw (errors.InternalError);
+                }
+            });
+    }
+
+    /**
+     * Load DB if exists
+     * @param {String} bucketName - name of bucket
+     * @param {Object} log - logger
+     * @param {function} cb - callback(err, db, attr)
+     * @return {undefined}
+     */
+    loadDBIfExists(bucketName, log, cb) {
+        this.getBucketAttributes(bucketName, log, (err, attr) => {
+            if (err) {
+                return cb(err);
+            }
+            try {
+                const db = this.mdDB.openSub(bucketName);
+                return cb(null, db, attr);
+            } catch (err) {
+                return cb(errors.InternalError);
+            }
+        });
+        return undefined;
+    }
+
+    createBucket(bucketName, bucketMD, log, cb) {
+        this.getBucketAttributes(bucketName, log, err => {
+            if (err && err !== errors.NoSuchBucket) {
+                return cb(err);
+            }
+            if (err === undefined) {
+                return cb(errors.BucketAlreadyExists);
+            }
+            this.putBucketAttributes(bucketName,
+                                     bucketMD,
+                                     log, cb);
+            return undefined;
+        });
+    }
+
+    getBucketAttributes(bucketName, log, cb) {
+        this.metastore
+            .withRequestLogger(log)
+            .get(bucketName, {}, (err, data) => {
+                if (err) {
+                    if (err.ObjNotFound) {
+                        return cb(errors.NoSuchBucket);
+                    }
+                    const logObj = {
+                        rawError: err,
+                        error: err.message,
+                        errorStack: err.stack,
+                    };
+                    log.error('error getting db attributes', logObj);
+                    return cb(errors.InternalError);
+                }
+                return cb(null, BucketInfo.deSerialize(data));
+            });
+        return undefined;
+    }
+
+    getBucketAndObject(bucketName, objName, params, log, cb) {
+        this.loadDBIfExists(bucketName, log, (err, db, bucketAttr) => {
+            if (err) {
+                return cb(err);
+            }
+            db.withRequestLogger(log)
+                .get(objName, params, (err, objAttr) => {
+                    if (err) {
+                        if (err.ObjNotFound) {
+                            return cb(null, {
+                                bucket: bucketAttr.serialize(),
+                            });
+                        }
+                        const logObj = {
+                            rawError: err,
+                            error: err.message,
+                            errorStack: err.stack,
+                        };
+                        log.error('error getting object', logObj);
+                        return cb(errors.InternalError);
+                    }
+                    return cb(null, {
+                        bucket: bucketAttr.serialize(),
+                        obj: objAttr,
+                    });
+                });
+            return undefined;
+        });
+        return undefined;
+    }
+
+    putBucketAttributes(bucketName, bucketMD, log, cb) {
+        this.metastore
+            .withRequestLogger(log)
+            .put(bucketName, bucketMD.serialize(), {}, err => {
+                if (err) {
+                    const logObj = {
+                        rawError: err,
+                        error: err.message,
+                        errorStack: err.stack,
+                    };
+                    log.error('error putting db attributes', logObj);
+                    return cb(errors.InternalError);
+                }
+                return cb();
+            });
+        return undefined;
+    }
+
+    deleteBucket(bucketName, log, cb) {
+        this.metastore
+            .withRequestLogger(log)
+            .del(bucketName, {}, err => {
+                if (err) {
+                    const logObj = {
+                        rawError: err,
+                        error: err.message,
+                        errorStack: err.stack,
+                    };
+                    log.error('error deleting bucket',
+                              logObj);
+                    return cb(errors.InternalError);
+                }
+                return cb();
+            });
+        return undefined;
+    }
+
+    putObject(bucketName, objName, objVal, params, log, cb) {
+        this.loadDBIfExists(bucketName, log, (err, db) => {
+            if (err) {
+                return cb(err);
+            }
+            db.withRequestLogger(log)
+                .put(objName, JSON.stringify(objVal), params, (err, data) => {
+                    if (err) {
+                        const logObj = {
+                            rawError: err,
+                            error: err.message,
+                            errorStack: err.stack,
+                        };
+                        log.error('error putting object', logObj);
+                        return cb(errors.InternalError);
+                    }
+                    return cb(err, data);
+                });
+            return undefined;
+        });
+    }
+
+    getObject(bucketName, objName, params, log, cb) {
+        this.loadDBIfExists(bucketName, log, (err, db) => {
+            if (err) {
+                return cb(err);
+            }
+            db.withRequestLogger(log).get(objName, params, (err, data) => {
+                if (err) {
+                    if (err.ObjNotFound) {
+                        return cb(errors.NoSuchKey);
+                    }
+                    const logObj = {
+                        rawError: err,
+                        error: err.message,
+                        errorStack: err.stack,
+                    };
+                    log.error('error getting object', logObj);
+                    return cb(errors.InternalError);
+                }
+                return cb(null, JSON.parse(data));
+            });
+            return undefined;
+        });
+    }
+
+    deleteObject(bucketName, objName, params, log, cb) {
+        this.loadDBIfExists(bucketName, log, (err, db) => {
+            if (err) {
+                return cb(err);
+            }
+            db.withRequestLogger(log).del(objName, params, err => {
+                if (err) {
+                    const logObj = {
+                        rawError: err,
+                        error: err.message,
+                        errorStack: err.stack,
+                    };
+                    log.error('error deleting object', logObj);
+                    return cb(errors.InternalError);
+                }
+                return cb();
+            });
+            return undefined;
+        });
+    }
+
+    /**
+     *  This complex function deals with different extensions of bucket listing:
+     *  Delimiter based search or MPU based search.
+     *  @param {String} bucketName - The name of the bucket to list
+     *  @param {Object} params - The params to search
+     *  @param {Object} log - The logger object
+     *  @param {function} cb - Callback when done
+     *  @return {undefined}
+     */
+    internalListObject(bucketName, params, log, cb) {
+        const extName = params.listingType;
+        const extension = new arsenal.algorithms.list[extName](params, log);
+        const requestParams = extension.genMDParams();
+        this.loadDBIfExists(bucketName, log, (err, db) => {
+            if (err) {
+                return cb(err);
+            }
+            let cbDone = false;
+            db.withRequestLogger(log)
+                .createReadStream(requestParams, (err, stream) => {
+                    if (err) {
+                        return cb(err);
+                    }
+                    stream
+                        .on('data', e => {
+                            if (extension.filter(e) < 0) {
+                                stream.emit('end');
+                                stream.destroy();
+                            }
+                        })
+                        .on('error', err => {
+                            if (!cbDone) {
+                                cbDone = true;
+                                const logObj = {
+                                    rawError: err,
+                                    error: err.message,
+                                    errorStack: err.stack,
+                                };
+                                log.error('error listing objects', logObj);
+                                cb(errors.InternalError);
+                            }
+                        })
+                        .on('end', () => {
+                            if (!cbDone) {
+                                cbDone = true;
+                                const data = extension.result();
+                                cb(null, data);
+                            }
+                        });
+                    return undefined;
+                });
+            return undefined;
+        });
+    }
+
+    listObject(bucketName, params, log, cb) {
+        return this.internalListObject(bucketName, params, log, cb);
+    }
+
+    listMultipartUploads(bucketName, params, log, cb) {
+        return this.internalListObject(bucketName, params, log, cb);
+    }
+
+    getUUID(log, cb) {
+        return this.mdDB.getUUID(cb);
+    }
+
+    getDiskUsage(cb) {
+        return this.mdDB.getDiskUsage(cb);
+    }
+
+    countItems(log, cb) {
+        const params = {};
+        const extension = new arsenal.algorithms.list.Basic(params, log);
+        const requestParams = extension.genMDParams();
+
+        const res = {
+            objects: 0,
+            versions: 0,
+            buckets: 0,
+        };
+        let cbDone = false;
+
+        this.mdDB.rawListKeys(requestParams, (err, stream) => {
+            if (err) {
+                return cb(err);
+            }
+            stream
+                .on('data', e => {
+                    if (!e.includes(METASTORE)) {
+                        if (e.includes(constants.usersBucket)) {
+                            res.buckets++;
+                        } else if (e.includes(versionSep)) {
+                            res.versions++;
+                        } else {
+                            res.objects++;
+                        }
+                    }
+                })
+                .on('error', err => {
+                    if (!cbDone) {
+                        cbDone = true;
+                        const logObj = {
+                            error: err,
+                            errorMessage: err.message,
+                            errorStack: err.stack,
+                        };
+                        log.error('error listing objects', logObj);
+                        cb(errors.InternalError);
+                    }
+                })
+                .on('end', () => {
+                    if (!cbDone) {
+                        cbDone = true;
+                        return cb(null, res);
+                    }
+                    return undefined;
+                });
+            return undefined;
+        });
+        return undefined;
+    }
+}
+
+module.exports = BucketFileInterface;

backends/metadata/in_memory/ListMultipartUploadsResult.js

@@ -0,0 +1,34 @@
+const ListResult = require('./ListResult');
+
+class ListMultipartUploadsResult extends ListResult {
+    constructor() {
+        super();
+        this.Uploads = [];
+        this.NextKeyMarker = undefined;
+        this.NextUploadIdMarker = undefined;
+    }
+
+    addUpload(uploadInfo) {
+        this.Uploads.push({
+            key: decodeURIComponent(uploadInfo.key),
+            value: {
+                UploadId: uploadInfo.uploadId,
+                Initiator: {
+                    ID: uploadInfo.initiatorID,
+                    DisplayName: uploadInfo.initiatorDisplayName,
+                },
+                Owner: {
+                    ID: uploadInfo.ownerID,
+                    DisplayName: uploadInfo.ownerDisplayName,
+                },
+                StorageClass: uploadInfo.storageClass,
+                Initiated: uploadInfo.initiated,
+            },
+        });
+        this.MaxKeys += 1;
+    }
+}
+
+module.exports = {
+    ListMultipartUploadsResult,
+};

backends/metadata/in_memory/ListResult.js

@@ -0,0 +1,27 @@
+class ListResult {
+    constructor() {
+        this.IsTruncated = false;
+        this.NextMarker = undefined;
+        this.CommonPrefixes = [];
+        /*
+        Note:  this.MaxKeys will get incremented as
+        keys are added so that when response is returned,
+        this.MaxKeys will equal total keys in response
+        (with each CommonPrefix counting as 1 key)
+        */
+        this.MaxKeys = 0;
+    }
+
+    addCommonPrefix(prefix) {
+        if (!this.hasCommonPrefix(prefix)) {
+            this.CommonPrefixes.push(prefix);
+            this.MaxKeys += 1;
+        }
+    }
+
+    hasCommonPrefix(prefix) {
+        return (this.CommonPrefixes.indexOf(prefix) !== -1);
+    }
+}
+
+module.exports = ListResult;

backends/metadata/in_memory/backend.js

@@ -0,0 +1,331 @@
+const { errors, algorithms, versioning } = require('arsenal');
+
+const getMultipartUploadListing = require('./getMultipartUploadListing');
+const { metadata } = require('./metadata');
+const { config } = require('../../Config');
+
+const genVID = versioning.VersionID.generateVersionId;
+
+const defaultMaxKeys = 1000;
+let uidCounter = 0;
+
+function generateVersionId() {
+    return genVID(uidCounter++, config.replicationGroupId);
+}
+
+function formatVersionKey(key, versionId) {
+    return `${key}\0${versionId}`;
+}
+
+function inc(str) {
+    return str ? (str.slice(0, str.length - 1) +
+            String.fromCharCode(str.charCodeAt(str.length - 1) + 1)) : str;
+}
+
+const metastore = {
+    createBucket: (bucketName, bucketMD, log, cb) => {
+        process.nextTick(() => {
+            metastore.getBucketAttributes(bucketName, log, (err, bucket) => {
+                // TODO Check whether user already owns the bucket,
+                // if so return "BucketAlreadyOwnedByYou"
+                // If not owned by user, return "BucketAlreadyExists"
+                if (bucket) {
+                    return cb(errors.BucketAlreadyExists);
+                }
+                metadata.buckets.set(bucketName, bucketMD);
+                metadata.keyMaps.set(bucketName, new Map);
+                return cb();
+            });
+        });
+    },
+
+    putBucketAttributes: (bucketName, bucketMD, log, cb) => {
+        process.nextTick(() => {
+            metastore.getBucketAttributes(bucketName, log, err => {
+                if (err) {
+                    return cb(err);
+                }
+                metadata.buckets.set(bucketName, bucketMD);
+                return cb();
+            });
+        });
+    },
+
+    getBucketAttributes: (bucketName, log, cb) => {
+        process.nextTick(() => {
+            if (!metadata.buckets.has(bucketName)) {
+                return cb(errors.NoSuchBucket);
+            }
+            return cb(null, metadata.buckets.get(bucketName));
+        });
+    },
+
+    deleteBucket: (bucketName, log, cb) => {
+        process.nextTick(() => {
+            metastore.getBucketAttributes(bucketName, log, err => {
+                if (err) {
+                    return cb(err);
+                }
+                if (metadata.keyMaps.has(bucketName)
+                    && metadata.keyMaps.get(bucketName).length > 0) {
+                    return cb(errors.BucketNotEmpty);
+                }
+                metadata.buckets.delete(bucketName);
+                metadata.keyMaps.delete(bucketName);
+                return cb(null);
+            });
+        });
+    },
+
+    putObject: (bucketName, objName, objVal, params, log, cb) => {
+        process.nextTick(() => {
+            metastore.getBucketAttributes(bucketName, log, err => {
+                if (err) {
+                    return cb(err);
+                }
+                /*
+                    valid combinations of versioning options:
+                    - !versioning && !versionId: normal non-versioning put
+                    - versioning && !versionId: create a new version
+                    - versionId: update (PUT/DELETE) an existing version,
+                        and also update master version in case the put
+                        version is newer or same version than master.
+                        if versionId === '' update master version
+                */
+
+                if (params && params.versionId) {
+                    objVal.versionId = params.versionId; // eslint-disable-line
+                    const mst = metadata.keyMaps.get(bucketName).get(objName);
+                    if (mst && mst.versionId === params.versionId || !mst) {
+                        metadata.keyMaps.get(bucketName).set(objName, objVal);
+                    }
+                    // eslint-disable-next-line
+                    objName = formatVersionKey(objName, params.versionId);
+                    metadata.keyMaps.get(bucketName).set(objName, objVal);
+                    return cb(null, `{"versionId":"${objVal.versionId}"}`);
+                }
+                if (params && params.versioning) {
+                    const versionId = generateVersionId();
+                    objVal.versionId = versionId; // eslint-disable-line
+                    metadata.keyMaps.get(bucketName).set(objName, objVal);
+                    // eslint-disable-next-line
+                    objName = formatVersionKey(objName, versionId);
+                    metadata.keyMaps.get(bucketName).set(objName, objVal);
+                    return cb(null, `{"versionId":"${versionId}"}`);
+                }
+                if (params && params.versionId === '') {
+                    const versionId = generateVersionId();
+                    objVal.versionId = versionId; // eslint-disable-line
+                    metadata.keyMaps.get(bucketName).set(objName, objVal);
+                    return cb(null, `{"versionId":"${objVal.versionId}"}`);
+                }
+                metadata.keyMaps.get(bucketName).set(objName, objVal);
+                return cb(null);
+            });
+        });
+    },
+
+    getBucketAndObject: (bucketName, objName, params, log, cb) => {
+        process.nextTick(() => {
+            metastore.getBucketAttributes(bucketName, log, (err, bucket) => {
+                if (err) {
+                    return cb(err, { bucket });
+                }
+                if (params && params.versionId) {
+                    // eslint-disable-next-line
+                    objName = formatVersionKey(objName, params.versionId);
+                }
+                if (!metadata.keyMaps.has(bucketName)
+                    || !metadata.keyMaps.get(bucketName).has(objName)) {
+                    return cb(null, { bucket: bucket.serialize() });
+                }
+                return cb(null, {
+                    bucket: bucket.serialize(),
+                    obj: JSON.stringify(
+                        metadata.keyMaps.get(bucketName).get(objName)
+                    ),
+                });
+            });
+        });
+    },
+
+    getObject: (bucketName, objName, params, log, cb) => {
+        process.nextTick(() => {
+            metastore.getBucketAttributes(bucketName, log, err => {
+                if (err) {
+                    return cb(err);
+                }
+                if (params && params.versionId) {
+                    // eslint-disable-next-line
+                    objName = formatVersionKey(objName, params.versionId);
+                }
+                if (!metadata.keyMaps.has(bucketName)
+                    || !metadata.keyMaps.get(bucketName).has(objName)) {
+                    return cb(errors.NoSuchKey);
+                }
+                return cb(null, metadata.keyMaps.get(bucketName).get(objName));
+            });
+        });
+    },
+
+    deleteObject: (bucketName, objName, params, log, cb) => {
+        process.nextTick(() => {
+            metastore.getBucketAttributes(bucketName, log, err => {
+                if (err) {
+                    return cb(err);
+                }
+                if (!metadata.keyMaps.get(bucketName).has(objName)) {
+                    return cb(errors.NoSuchKey);
+                }
+                if (params && params.versionId) {
+                    const baseKey = inc(formatVersionKey(objName, ''));
+                    const vobjName = formatVersionKey(objName,
+                            params.versionId);
+                    metadata.keyMaps.get(bucketName).delete(vobjName);
+                    const mst = metadata.keyMaps.get(bucketName).get(objName);
+                    if (mst.versionId === params.versionId) {
+                        const keys = [];
+                        metadata.keyMaps.get(bucketName).forEach((val, key) => {
+                            if (key < baseKey && key > vobjName) {
+                                keys.push(key);
+                            }
+                        });
+                        if (keys.length === 0) {
+                            metadata.keyMaps.get(bucketName).delete(objName);
+                            return cb();
+                        }
+                        const key = keys.sort()[0];
+                        const value = metadata.keyMaps.get(bucketName).get(key);
+                        metadata.keyMaps.get(bucketName).set(objName, value);
+                    }
+                    return cb();
+                }
+                metadata.keyMaps.get(bucketName).delete(objName);
+                return cb();
+            });
+        });
+    },
+
+    _hasDeleteMarker(key, keyMap) {
+        const objectMD = keyMap.get(key);
+        if (objectMD['x-amz-delete-marker'] !== undefined) {
+            return (objectMD['x-amz-delete-marker'] === true);
+        }
+        return false;
+    },
+
+    listObject(bucketName, params, log, cb) {
+        process.nextTick(() => {
+            const {
+                prefix,
+                marker,
+                delimiter,
+                maxKeys,
+                continuationToken,
+                startAfter,
+            } = params;
+            if (prefix && typeof prefix !== 'string') {
+                return cb(errors.InvalidArgument);
+            }
+
+            if (marker && typeof marker !== 'string') {
+                return cb(errors.InvalidArgument);
+            }
+
+            if (delimiter && typeof delimiter !== 'string') {
+                return cb(errors.InvalidArgument);
+            }
+
+            if (maxKeys && typeof maxKeys !== 'number') {
+                return cb(errors.InvalidArgument);
+            }
+
+            if (continuationToken && typeof continuationToken !== 'string') {
+                return cb(errors.InvalidArgument);
+            }
+
+            if (startAfter && typeof startAfter !== 'string') {
+                return cb(errors.InvalidArgument);
+            }
+
+            // If paramMaxKeys is undefined, the default parameter will set it.
+            // However, if it is null, the default parameter will not set it.
+            let numKeys = maxKeys;
+            if (numKeys === null) {
+                numKeys = defaultMaxKeys;
+            }
+
+            if (!metadata.keyMaps.has(bucketName)) {
+                return cb(errors.NoSuchBucket);
+            }
+
+            // If marker specified, edit the keys array so it
+            // only contains keys that occur alphabetically after the marker
+            const listingType = params.listingType;
+            const extension = new algorithms.list[listingType](params, log);
+            const listingParams = extension.genMDParams();
+
+            const keys = [];
+            metadata.keyMaps.get(bucketName).forEach((val, key) => {
+                if (listingParams.gt && listingParams.gt >= key) {
+                    return null;
+                }
+                if (listingParams.gte && listingParams.gte > key) {
+                    return null;
+                }
+                if (listingParams.lt && key >= listingParams.lt) {
+                    return null;
+                }
+                if (listingParams.lte && key > listingParams.lte) {
+                    return null;
+                }
+                return keys.push(key);
+            });
+            keys.sort();
+
+            // Iterate through keys array and filter keys containing
+            // delimiter into response.CommonPrefixes and filter remaining
+            // keys into response.Contents
+            for (let i = 0; i < keys.length; ++i) {
+                const currentKey = keys[i];
+                // Do not list object with delete markers
+                if (this._hasDeleteMarker(currentKey,
+                    metadata.keyMaps.get(bucketName))) {
+                    continue;
+                }
+                const objMD = metadata.keyMaps.get(bucketName).get(currentKey);
+                const value = JSON.stringify(objMD);
+                const obj = {
+                    key: currentKey,
+                    value,
+                };
+                // calling Ext.filter(obj) adds the obj to the Ext result if
+                // not filtered.
+                // Also, Ext.filter returns false when hit max keys.
+                // What a nifty function!
+                if (extension.filter(obj) < 0) {
+                    break;
+                }
+            }
+            return cb(null, extension.result());
+        });
+    },
+
+    listMultipartUploads(bucketName, listingParams, log, cb) {
+        process.nextTick(() => {
+            metastore.getBucketAttributes(bucketName, log, (err, bucket) => {
+                if (bucket === undefined) {
+                    // no on going multipart uploads, return empty listing
+                    return cb(null, {
+                        IsTruncated: false,
+                        NextMarker: undefined,
+                        MaxKeys: 0,
+                    });
+                }
+                return getMultipartUploadListing(bucket, listingParams, cb);
+            });
+        });
+    },
+};
+
+module.exports = metastore;

backends/metadata/in_memory/bucket_mem.js.Design.md

@@ -0,0 +1,62 @@
+# bucket_mem design
+
+## RATIONALE
+
+The bucket API will be used for managing buckets behind the S3 interface.
+
+We plan to have only 2 backends using this interface:
+
+* One production backend
+* One debug backend purely in memory
+
+One important remark here is that we don't want an abstraction but a
+duck-typing style interface (different classes MemoryBucket and Bucket having
+the same methods putObjectMD(), getObjectMD(), etc).
+
+Notes about the memory backend: The backend is currently a simple key/value
+store in memory. The functions actually use nextTick() to emulate the future
+asynchronous behavior of the production backend.
+
+## BUCKET API
+
+The bucket API is a very simple API with 5 functions:
+
+- putObjectMD(): put metadata for an object in the bucket
+- getObjectMD(): get metadata from the bucket
+- deleteObjectMD(): delete metadata for an object from the bucket
+- deleteBucketMD(): delete a bucket
+- getBucketListObjects(): perform the complex bucket listing AWS search
+  function with various flavors. This function returns a response in a
+  ListBucketResult object.
+
+getBucketListObjects(prefix, marker, delimiter, maxKeys, callback) behavior is
+the following:
+
+prefix (not required): Limits the response to keys that begin with the
+specified prefix. You can use prefixes to separate a bucket into different
+groupings of keys. (You can think of using prefix to make groups in the same
+way you'd use a folder in a file system.)
+
+marker (not required): Specifies the key to start with when listing objects in
+a bucket. Amazon S3 returns object keys in alphabetical order, starting with
+key after the marker in order.
+
+delimiter (not required): A delimiter is a character you use to group keys.
+All keys that contain the same string between the prefix, if specified, and the
+first occurrence of the delimiter after the prefix are grouped under a single
+result element, CommonPrefixes. If you don't specify the prefix parameter, then
+the substring starts at the beginning of the key. The keys that are grouped
+under CommonPrefixes are not returned elsewhere in the response.
+
+maxKeys: Sets the maximum number of keys returned in the response body. You can
+add this to your request if you want to retrieve fewer than the default 1000
+keys.  The response might contain fewer keys but will never contain more. If
+there are additional keys that satisfy the search criteria but were not
+returned because maxKeys was exceeded, the response contains an attribute of
+IsTruncated set to true and a NextMarker. To return the additional keys, call
+the function again using NextMarker as your marker argument in the function.
+
+Any key that does not contain the delimiter will be returned individually in
+Contents rather than in CommonPrefixes.
+
+If there is an error, the error subfield is returned in the response.

backends/metadata/in_memory/bucket_utilities.js

@@ -0,0 +1,51 @@
+function markerFilterMPU(allMarkers, array) {
+    const { keyMarker, uploadIdMarker } = allMarkers;
+    for (let i = 0; i < array.length; i++) {
+        // If the keyMarker is the same as the key,
+        // check the uploadIdMarker.  If uploadIdMarker is the same
+        // as or alphabetically after the uploadId of the item,
+        // eliminate the item.
+        if (uploadIdMarker && keyMarker === array[i].key) {
+            const laterId =
+                [uploadIdMarker, array[i].uploadId].sort()[1];
+            if (array[i].uploadId === laterId) {
+                break;
+            } else {
+                array.shift();
+                i--;
+            }
+        } else {
+        // If the keyMarker is alphabetically after the key
+        // of the item in the array, eliminate the item from the array.
+            const laterItem =
+                [keyMarker, array[i].key].sort()[1];
+            if (keyMarker === array[i].key || keyMarker === laterItem) {
+                array.shift();
+                i--;
+            } else {
+                break;
+            }
+        }
+    }
+    return array;
+}
+
+function prefixFilter(prefix, array) {
+    for (let i = 0; i < array.length; i++) {
+        if (array[i].indexOf(prefix) !== 0) {
+            array.splice(i, 1);
+            i--;
+        }
+    }
+    return array;
+}
+
+function isKeyInContents(responseObject, key) {
+    return responseObject.Contents.some(val => val.key === key);
+}
+
+module.exports = {
+    markerFilterMPU,
+    prefixFilter,
+    isKeyInContents,
+};

backends/metadata/in_memory/getMultipartUploadListing.js

@@ -0,0 +1,148 @@
+const { errors } = require('arsenal');
+
+const { markerFilterMPU, prefixFilter } = require('./bucket_utilities');
+const { ListMultipartUploadsResult } = require('./ListMultipartUploadsResult');
+const { metadata } = require('./metadata');
+
+const defaultMaxKeys = 1000;
+function getMultipartUploadListing(bucket, params, callback) {
+    const { delimiter, keyMarker,
+        uploadIdMarker, prefix, queryPrefixLength, splitter } = params;
+    const splitterLen = splitter.length;
+    const maxKeys = params.maxKeys !== undefined ?
+        Number.parseInt(params.maxKeys, 10) : defaultMaxKeys;
+    const response = new ListMultipartUploadsResult();
+    const keyMap = metadata.keyMaps.get(bucket.getName());
+    if (prefix) {
+        response.Prefix = prefix;
+        if (typeof prefix !== 'string') {
+            return callback(errors.InvalidArgument);
+        }
+    }
+
+    if (keyMarker) {
+        response.KeyMarker = keyMarker;
+        if (typeof keyMarker !== 'string') {
+            return callback(errors.InvalidArgument);
+        }
+    }
+
+    if (uploadIdMarker) {
+        response.UploadIdMarker = uploadIdMarker;
+        if (typeof uploadIdMarker !== 'string') {
+            return callback(errors.InvalidArgument);
+        }
+    }
+
+    if (delimiter) {
+        response.Delimiter = delimiter;
+        if (typeof delimiter !== 'string') {
+            return callback(errors.InvalidArgument);
+        }
+    }
+
+    if (maxKeys && typeof maxKeys !== 'number') {
+        return callback(errors.InvalidArgument);
+    }
+
+    // Sort uploads alphatebetically by objectKey and if same objectKey,
+    // then sort in ascending order by time initiated
+    let uploads = [];
+    keyMap.forEach((val, key) => {
+        uploads.push(key);
+    });
+    uploads.sort((a, b) => {
+        const aIndex = a.indexOf(splitter);
+        const bIndex = b.indexOf(splitter);
+        const aObjectKey = a.substring(aIndex + splitterLen);
+        const bObjectKey = b.substring(bIndex + splitterLen);
+        const aInitiated = keyMap.get(a).initiated;
+        const bInitiated = keyMap.get(b).initiated;
+        if (aObjectKey === bObjectKey) {
+            if (Date.parse(aInitiated) >= Date.parse(bInitiated)) {
+                return 1;
+            }
+            if (Date.parse(aInitiated) < Date.parse(bInitiated)) {
+                return -1;
+            }
+        }
+        return (aObjectKey < bObjectKey) ? -1 : 1;
+    });
+    // Edit the uploads array so it only
+    // contains keys that contain the prefix
+    uploads = prefixFilter(prefix, uploads);
+    uploads = uploads.map(stringKey => {
+        const index = stringKey.indexOf(splitter);
+        const index2 = stringKey.indexOf(splitter, index + splitterLen);
+        const storedMD = keyMap.get(stringKey);
+        return {
+            key: stringKey.substring(index + splitterLen, index2),
+            uploadId: stringKey.substring(index2 + splitterLen),
+            bucket: storedMD.eventualStorageBucket,
+            initiatorID: storedMD.initiator.ID,
+            initiatorDisplayName: storedMD.initiator.DisplayName,
+            ownerID: storedMD['owner-id'],
+            ownerDisplayName: storedMD['owner-display-name'],
+            storageClass: storedMD['x-amz-storage-class'],
+            initiated: storedMD.initiated,
+        };
+    });
+    // If keyMarker specified, edit the uploads array so it
+    // only contains keys that occur alphabetically after the marker.
+    // If there is also an uploadIdMarker specified, filter to eliminate
+    // any uploads that share the keyMarker and have an uploadId before
+    //  the uploadIdMarker.
+    if (keyMarker) {
+        const allMarkers = {
+            keyMarker,
+            uploadIdMarker,
+        };
+        uploads = markerFilterMPU(allMarkers, uploads);
+    }
+
+    // Iterate through uploads and filter uploads
+    // with keys containing delimiter
+    // into response.CommonPrefixes and filter remaining uploads
+    // into response.Uploads
+    for (let i = 0; i < uploads.length; i++) {
+        const currentUpload = uploads[i];
+        // If hit maxKeys, stop adding keys to response
+        if (response.MaxKeys >= maxKeys) {
+            response.IsTruncated = true;
+            break;
+        }
+        // If a delimiter is specified, find its
+        // index in the current key AFTER THE OCCURRENCE OF THE PREFIX
+        // THAT WAS SENT IN THE QUERY (not the prefix including the splitter
+        // and other elements)
+        let delimiterIndexAfterPrefix = -1;
+        const currentKeyWithoutPrefix =
+            currentUpload.key.slice(queryPrefixLength);
+        let sliceEnd;
+        if (delimiter) {
+            delimiterIndexAfterPrefix = currentKeyWithoutPrefix
+                .indexOf(delimiter);
+            sliceEnd = delimiterIndexAfterPrefix + queryPrefixLength;
+        }
+        // If delimiter occurs in current key, add key to
+        // response.CommonPrefixes.
+        // Otherwise add upload to response.Uploads
+        if (delimiterIndexAfterPrefix > -1) {
+            const keySubstring = currentUpload.key.slice(0, sliceEnd + 1);
+            response.addCommonPrefix(keySubstring);
+        } else {
+            response.NextKeyMarker = currentUpload.key;
+            response.NextUploadIdMarker = currentUpload.uploadId;
+            response.addUpload(currentUpload);
+        }
+    }
+    // `response.MaxKeys` should be the value from the original `MaxUploads`
+    // parameter specified by the user (or else the default 1000). Redefine it
+    // here, so it does not equal the value of `uploads.length`.
+    response.MaxKeys = maxKeys;
+    // If `response.MaxKeys` is 0, `response.IsTruncated` should be `false`.
+    response.IsTruncated = maxKeys === 0 ? false : response.IsTruncated;
+    return callback(null, response);
+}
+
+module.exports = getMultipartUploadListing;

backends/metadata/in_memory/metadata.js

@@ -0,0 +1,8 @@
+const metadata = {
+    buckets: new Map,
+    keyMaps: new Map,
+};
+
+module.exports = {
+    metadata,
+};

backends/metadata/metadataUtils.js

@@ -0,0 +1,274 @@
+const async = require('async');
+const { errors } = require('arsenal');
+
+const metadata = require('./wrapper');
+const BucketInfo = require('arsenal').models.BucketInfo;
+const { isBucketAuthorized, isObjAuthorized } =
+    require('../api/apiUtils/authorization/aclChecks');
+const bucketShield = require('../api/apiUtils/bucket/bucketShield');
+
+/** _parseListEntries - parse the values returned in a listing by metadata
+ * @param {object[]} entries - Version or Content entries in a metadata listing
+ * @param {string} entries[].key - metadata key
+ * @param {string} entries[].value - stringified object metadata
+ * @return {object} - mapped array with parsed value or JSON parsing err
+ */
+function _parseListEntries(entries) {
+    return entries.map(entry => {
+        if (typeof entry.value === 'string') {
+            const tmp = JSON.parse(entry.value);
+            return {
+                key: entry.key,
+                value: {
+                    Size: tmp['content-length'],
+                    ETag: tmp['content-md5'],
+                    VersionId: tmp.versionId,
+                    IsNull: tmp.isNull,
+                    IsDeleteMarker: tmp.isDeleteMarker,
+                    LastModified: tmp['last-modified'],
+                    Owner: {
+                        DisplayName: tmp['owner-display-name'],
+                        ID: tmp['owner-id'],
+                    },
+                    StorageClass: tmp['x-amz-storage-class'],
+                    // MPU listing properties
+                    Initiated: tmp.initiated,
+                    Initiator: tmp.initiator,
+                    EventualStorageBucket: tmp.eventualStorageBucket,
+                    partLocations: tmp.partLocations,
+                    creationDate: tmp.creationDate,
+                },
+            };
+        }
+        return entry;
+    });
+}
+
+/** parseListEntries - parse the values returned in a listing by metadata
+ * @param {object[]} entries - Version or Content entries in a metadata listing
+ * @param {string} entries[].key - metadata key
+ * @param {string} entries[].value - stringified object metadata
+ * @return {(object|Error)} - mapped array with parsed value or JSON parsing err
+ */
+function parseListEntries(entries) {
+    // wrap private function in a try/catch clause
+    // just in case JSON parsing throws an exception
+    try {
+        return _parseListEntries(entries);
+    } catch (e) {
+        return e;
+    }
+}
+
+/** getNullVersion - return metadata of null version if it exists
+ * @param {object} objMD - metadata of master version
+ * @param {string} bucketName - name of bucket
+ * @param {string} objectKey - name of object key
+ * @param {RequestLogger} log - request logger
+ * @param {function} cb - callback
+ * @return {undefined} - and call callback with params err, objMD of null ver
+ */
+function getNullVersion(objMD, bucketName, objectKey, log, cb) {
+    const options = {};
+    if (objMD.isNull || !objMD.versionId) {
+        // null version is current version
+        log.debug('found null version');
+        return process.nextTick(() => cb(null, objMD));
+    }
+    if (objMD.nullVersionId) {
+        // the latest version is not the null version, but null version exists
+        log.debug('null version exists, get the null version');
+        options.versionId = objMD.nullVersionId;
+        return metadata.getObjectMD(bucketName, objectKey, options, log, cb);
+    }
+    log.debug('could not find a null version');
+    return process.nextTick(() => cb());
+}
+
+/** metadataGetBucketAndObject - retrieves bucket and specified version
+ * NOTE: If the value of `versionId` param is 'null', this function returns the
+ * master version objMD. The null version object md must be retrieved in a
+ * separate step using the master object md: see getNullVersion().
+ * @param {string} requestType - type of request
+ * @param {string} bucketName - name of bucket
+ * @param {string} objectKey - name of object key
+ * @param {string} [versionId] - version of object to retrieve
+ * @param {RequestLogger} log - request logger
+ * @param {function} cb - callback
+ * @return {undefined} - and call callback with err, bucket md and object md
+ */
+function metadataGetBucketAndObject(requestType, bucketName, objectKey,
+    versionId, log, cb) {
+    const options = {
+        // if attempting to get 'null' version, must retrieve null version id
+        // from most current object md (versionId = undefined)
+        versionId: versionId === 'null' ? undefined : versionId,
+    };
+    return metadata.getBucketAndObjectMD(bucketName, objectKey, options, log,
+        (err, data) => {
+            if (err) {
+                log.debug('metadata get failed', { error: err });
+                return cb(err);
+            }
+            const bucket = data.bucket ? BucketInfo.deSerialize(data.bucket) :
+                undefined;
+            const obj = data.obj ? JSON.parse(data.obj) : undefined;
+            if (!bucket) {
+                log.debug('bucketAttrs is undefined', {
+                    bucket: bucketName,
+                    method: 'metadataGetBucketAndObject',
+                });
+                return cb(errors.NoSuchBucket);
+            }
+            if (bucketShield(bucket, requestType)) {
+                log.debug('bucket is shielded from request', {
+                    requestType,
+                    method: 'metadataGetBucketAndObject',
+                });
+                return cb(errors.NoSuchBucket);
+            }
+            log.trace('found bucket in metadata');
+            return cb(null, bucket, obj);
+        });
+}
+
+/** metadataGetObject - retrieves specified object or version from metadata
+ * @param {string} bucketName - name of bucket
+ * @param {string} objectKey - name of object key
+ * @param {string} [versionId] - version of object to retrieve
+ * @param {RequestLogger} log - request logger
+ * @param {function} cb - callback
+ * @return {undefined} - and call callback with err, bucket md and object md
+ */
+function metadataGetObject(bucketName, objectKey, versionId, log, cb) {
+    const options = {
+        // if attempting to get 'null' version, must first retrieve null version
+        // id from most current object md (by setting versionId as undefined
+        // we retrieve the most current object md)
+        versionId: versionId === 'null' ? undefined : versionId,
+    };
+    return metadata.getObjectMD(bucketName, objectKey, options, log,
+        (err, objMD) => {
+            if (err) {
+                log.debug('err getting object MD from metadata',
+                { error: err });
+                return cb(err);
+            }
+            if (versionId === 'null') {
+                return getNullVersion(objMD, bucketName, objectKey, log, cb);
+            }
+            return cb(null, objMD);
+        });
+}
+
+/** metadataValidateBucketAndObj - retrieve bucket and object md from metadata
+ * and check if user is authorized to access them.
+ * @param {object} params - function parameters
+ * @param {AuthInfo} params.authInfo - AuthInfo class instance, requester's info
+ * @param {string} params.bucketName - name of bucket
+ * @param {string} params.objectKey - name of object
+ * @param {string} [params.versionId] - version id if getting specific version
+ * @param {string} params.requestType - type of request
+ * @param {RequestLogger} log - request logger
+ * @param {function} callback - callback
+ * @return {undefined} - and call callback with params err, bucket md
+ */
+function metadataValidateBucketAndObj(params, log, callback) {
+    const { authInfo, bucketName, objectKey, versionId, requestType } = params;
+    const canonicalID = authInfo.getCanonicalID();
+    async.waterfall([
+        function getBucketAndObjectMD(next) {
+            return metadataGetBucketAndObject(requestType, bucketName,
+                objectKey, versionId, log, next);
+        },
+        function checkBucketAuth(bucket, objMD, next) {
+            if (!isBucketAuthorized(bucket, requestType, canonicalID)) {
+                log.debug('access denied for user on bucket', { requestType });
+                return next(errors.AccessDenied, bucket);
+            }
+            return next(null, bucket, objMD);
+        },
+        function handleNullVersionGet(bucket, objMD, next) {
+            if (objMD && versionId === 'null') {
+                return getNullVersion(objMD, bucketName, objectKey, log,
+                    (err, nullVer) => next(err, bucket, nullVer));
+            }
+            return next(null, bucket, objMD);
+        },
+        function checkObjectAuth(bucket, objMD, next) {
+            if (!objMD) {
+                return next(null, bucket);
+            }
+            if (!isObjAuthorized(bucket, objMD, requestType, canonicalID)) {
+                log.debug('access denied for user on object', { requestType });
+                return next(errors.AccessDenied, bucket);
+            }
+            return next(null, bucket, objMD);
+        },
+    ], (err, bucket, objMD) => {
+        if (err) {
+            // still return bucket for cors headers
+            return callback(err, bucket);
+        }
+        return callback(null, bucket, objMD);
+    });
+}
+
+/** metadataGetBucket - retrieves bucket from metadata, returning error if
+ * bucket is shielded
+ * @param {string} requestType - type of request
+ * @param {string} bucketName - name of bucket
+ * @param {RequestLogger} log - request logger
+ * @param {function} cb - callback
+ * @return {undefined} - and call callback with err, bucket md
+ */
+function metadataGetBucket(requestType, bucketName, log, cb) {
+    return metadata.getBucket(bucketName, log, (err, bucket) => {
+        if (err) {
+            log.debug('metadata getbucket failed', { error: err });
+            return cb(err);
+        }
+        if (bucketShield(bucket, requestType)) {
+            log.debug('bucket is shielded from request', {
+                requestType,
+                method: 'metadataGetBucketAndObject',
+            });
+            return cb(errors.NoSuchBucket);
+        }
+        log.trace('found bucket in metadata');
+        return cb(null, bucket);
+    });
+}
+
+/** metadataValidateBucket - retrieve bucket from metadata and check if user
+ * is authorized to access it
+ * @param {object} params - function parameters
+ * @param {AuthInfo} params.authInfo - AuthInfo class instance, requester's info
+ * @param {string} params.bucketName - name of bucket
+ * @param {string} params.requestType - type of request
+ * @param {RequestLogger} log - request logger
+ * @param {function} callback - callback
+ * @return {undefined} - and call callback with params err, bucket md
+ */
+function metadataValidateBucket(params, log, callback) {
+    const { authInfo, bucketName, requestType } = params;
+    const canonicalID = authInfo.getCanonicalID();
+    return metadataGetBucket(requestType, bucketName, log, (err, bucket) => {
+        if (err) {
+            return callback(err);
+        }
+        // still return bucket for cors headers
+        if (!isBucketAuthorized(bucket, requestType, canonicalID)) {
+            log.debug('access denied for user on bucket', { requestType });
+            return callback(errors.AccessDenied, bucket);
+        }
+        return callback(null, bucket);
+    });
+}
+
+module.exports = {
+    parseListEntries,
+    metadataGetObject,
+    metadataValidateBucketAndObj,
+    metadataValidateBucket,
+};

backends/metadata/mongoclient/Mongoclient.md

@@ -0,0 +1,173 @@
+# Mongoclient
+
+We introduce a new metadata backend called *mongoclient* for
+[MongoDB](https://www.mongodb.com). This backend takes advantage of
+MongoDB being a document store to store the metadata (bucket and
+object attributes) as JSON objects.
+
+## Overall Design
+
+The mongoclient backend strictly follows the metadata interface that
+stores bucket and object attributes, which consists of the methods
+createBucket(), getBucketAttributes(), getBucketAndObject()
+(attributes), putBucketAttributes(), deleteBucket(), putObject(),
+getObject(), deleteObject(), listObject(), listMultipartUploads() and
+the management methods getUUID(), getDiskUsage() and countItems(). The
+mongoclient backend also knows how to deal with versioning, it is also
+compatible with the various listing algorithms implemented in Arsenal.
+
+FIXME: There should be a document describing the metadata (currently
+duck-typing) interface.
+
+### Why Using MongoDB for Storing Bucket and Object Attributes
+
+We chose MongoDB for various reasons:
+
+- MongoDB supports replication, especially through the Raft protocol.
+
+- MongoDB supports a basic replication scheme called 'Replica Set' and
+  more advanced sharding schemes if required.
+
+- MongoDB is open source and an enterprise standard.
+
+- MongoDB is a document store (natively supports JSON) and supports a
+  very flexible search interface.
+
+### Choice of Mongo Client Library
+
+We chose to use the official MongoDB driver for NodeJS:
+[https://github.com/mongodb/node-mongodb-native](https://github.com/mongodb/node-mongodb-native)
+
+### Granularity for Buckets
+
+We chose to have one collection for one bucket mapping. First because
+in a simple mode of replication called 'replica set' it works from the
+get-go, but if one or many buckets grow to big it is possible to use
+more advanced schemes such as sharding. MongoDB supports a mix of
+sharded and non-sharded collections.
+
+### Storing Database Information
+
+We need a special collection called the *Infostore* (stored under the
+name __infostore which is impossible to create through the S3 bucket
+naming scheme) to store specific database properties such as the
+unique *uuid* for Orbit.
+
+### Storing Bucket Attributes
+
+We need to use a special collection called the *Metastore* (stored
+under the name __metastore which is impossible to create through the
+S3 bucket naming scheme).
+
+### Versioning Format
+
+We chose to keep the same versioning format that we use in some other
+Scality products in order to facilitate the compatibility between the
+different products.
+
+FIXME: Document the versioning internals in the upper layers and
+document the versioning format
+
+### Dealing with Concurrency
+
+We chose not to use transactions (aka
+[https://docs.mongodb.com/manual/tutorial/perform-two-phase-commits/)
+because it is a known fact there is an overhead of using them, and we
+thought there was no real need for them since we could leverage Mongo
+ordered operations guarantees and atomic writes.
+
+Example of corner cases:
+
+#### CreateBucket()
+
+Since it is not possible to create a collection AND at the same time
+register the bucket in the Metastore we chose to only update the
+Metastore. A non-existing collection (NamespaceNotFound error in
+Mongo) is one possible normal state for an empty bucket.
+
+#### DeleteBucket()
+
+In this case the bucket is *locked* by the upper layers (use of a
+transient delete flag) so we don't have to worry about that and by the
+fact the bucket is empty neither (which is also checked by the upper
+layers).
+
+We first drop() the collection and then we asynchronously delete the
+bucket name entry from the metastore (the removal from the metastore
+is atomic which is not absolutely necessary in this case but more
+robust in term of design).
+
+If we fail in between we still have an entry in the metastore which is
+good because we need to manage the delete flag. For the upper layers
+the operation has not completed until this flag is removed. The upper
+layers will restart the deleteBucket() which is fine because we manage
+the case where the collection does not exist.
+
+#### PutObject() with a Version
+
+We need to store the versioned object then update the master object
+(the latest version). For this we use the
+[BulkWrite](http://mongodb.github.io/node-mongodb-native/3.0/api/Collection.html#bulkWrite)
+method. This is not a transaction but guarantees that the 2 operations
+will happen sequentially in the MongoDB oplog. Indeed if the
+BulkWrite() fails in between we would end up creating an orphan (which
+is not critical) but if the operation succeeds then we are sure that
+the master is always pointing to the right object. If there is a
+concurrency between 2 clients then we are sure that the 2 groups of
+operations will be clearly decided in the oplog (the last writer will
+win).
+
+#### DeleteObject()
+
+This is probably the most complex case to manage because it involves a
+lot of different cases:
+
+##### Deleting an Object when Versioning is not Enabled
+
+This case is a straightforward atomic delete. Atomicity is not really
+required because we assume version IDs are random enough but it is
+more robust to do so.
+
+##### Deleting an Object when Versioning is Enabled
+
+This case is more complex since we have to deal with the 2 cases:
+
+Case 1: The caller asks for a deletion of a version which is not a master:
+This case is a straight-forward atomic delete.
+
+Case 2: The caller asks for a deletion of a version which is the master: In
+this case we need to create a special flag called PHD (as PlaceHolDer)
+that indicates the master is no longer valid (with a new unique
+virtual version ID). We force the ordering of operations in a
+bulkWrite() to first replace the master with the PHD flag and then
+physically delete the version. If the call fail in between we will be
+left with a master with a PHD flag. If the call succeeds we try to
+find if the master with the PHD flag is left alone in such case we
+delete it otherwise we trigger an asynchronous repair that will spawn
+after AYNC_REPAIR_TIMEOUT=15s that will reassign the master to the
+latest version.
+
+In all cases the physical deletion or the repair of the master are
+checked against the PHD flag AND the actual unique virtual version
+ID. We do this to check against potential concurrent deletions,
+repairs or updates. Only the last writer/deleter has the right to
+physically perform the operation, otherwise it is superseded by other
+operations.
+
+##### Getting an object with a PHD flag
+
+If the caller is asking for the latest version of an object and the
+PHD flag is set we perform a search on the bucket to find the latest
+version and we return it.
+
+#### Listing Objects
+
+The mongoclient backend implements a readable key/value stream called
+*MongoReadStream* that follows the LevelDB duck typing interface used
+in Arsenal/lib/algos listing algorithms. Note it does not require any
+LevelDB package.
+
+#### Generating the UUID
+
+To avoid race conditions we always (try to) generate a new UUID and we
+condition the insertion to the non-existence of the document.

backends/metadata/mongoclient/backend.js

@@ -0,0 +1,766 @@
+/*
+ * we assume good default setting of write concern is good for all
+ * bulk writes. Note that bulk writes are not transactions but ordered
+ * writes. They may fail in between. To some extend those situations
+ * may generate orphans but not alter the proper conduct of operations
+ * (what he user wants and what we acknowledge to the user).
+ *
+ * Orphan situations may be recovered by the Lifecycle.
+ *
+ * We use proper atomic operations when needed.
+ */
+const async = require('async');
+const arsenal = require('arsenal');
+
+const logger = require('../../utilities/logger');
+
+const constants = require('../../../constants');
+const { config } = require('../../Config');
+
+const errors = arsenal.errors;
+const versioning = arsenal.versioning;
+const BucketInfo = arsenal.models.BucketInfo;
+
+const MongoClient = require('mongodb').MongoClient;
+const Uuid = require('uuid');
+const diskusage = require('diskusage');
+
+const genVID = versioning.VersionID.generateVersionId;
+
+const MongoReadStream = require('./readStream');
+const MongoUtils = require('./utils');
+
+const USERSBUCKET = '__usersbucket';
+const METASTORE = '__metastore';
+const INFOSTORE = '__infostore';
+const __UUID = 'uuid';
+const ASYNC_REPAIR_TIMEOUT = 15000;
+
+let uidCounter = 0;
+
+const VID_SEP = versioning.VersioningConstants.VersionId.Separator;
+
+function generateVersionId() {
+    // generate a unique number for each member of the nodejs cluster
+    return genVID(`${process.pid}.${uidCounter++}`,
+                  config.replicationGroupId);
+}
+
+function formatVersionKey(key, versionId) {
+    return `${key}${VID_SEP}${versionId}`;
+}
+
+function inc(str) {
+    return str ? (str.slice(0, str.length - 1) +
+            String.fromCharCode(str.charCodeAt(str.length - 1) + 1)) : str;
+}
+
+const VID_SEPPLUS = inc(VID_SEP);
+
+function generatePHDVersion(versionId) {
+    return {
+        isPHD: true,
+        versionId,
+    };
+}
+
+class MongoClientInterface {
+    constructor() {
+        const mongoUrl =
+              `mongodb://${config.mongodb.host}:${config.mongodb.port}`;
+        this.logger = logger;
+        this.client = null;
+        this.db = null;
+        this.logger.debug(`connecting to ${mongoUrl}`);
+        // FIXME: constructors shall not have side effect so there
+        // should be an async_init(cb) method in the wrapper to
+        // initialize this backend
+        MongoClient.connect(mongoUrl, (err, client) => {
+            if (err) {
+                throw (errors.InternalError);
+            }
+            this.logger.debug('connected to mongodb');
+            this.client = client;
+            this.db = client.db(config.mongodb.database, {
+                ignoreUndefined: true,
+            });
+            this.usersBucketHack();
+        });
+    }
+
+    usersBucketHack() {
+        /* FIXME: Since the bucket creation API is expecting the
+           usersBucket to have attributes, we pre-create the
+           usersBucket attributes here (see bucketCreation.js line
+           36)*/
+        const usersBucketAttr = new BucketInfo(constants.usersBucket,
+            'admin', 'admin', new Date().toJSON(),
+            BucketInfo.currentModelVersion());
+        this.createBucket(
+            constants.usersBucket,
+            usersBucketAttr, {}, err => {
+                if (err) {
+                    this.logger.fatal('error writing usersBucket ' +
+                                      'attributes to metastore',
+                                      { error: err });
+                    throw (errors.InternalError);
+                }
+            });
+    }
+
+    getCollection(name) {
+        /* mongo has a problem with .. in collection names */
+        const newName = (name === constants.usersBucket) ?
+              USERSBUCKET : name;
+        return this.db.collection(newName);
+    }
+
+    createBucket(bucketName, bucketMD, log, cb) {
+        // FIXME: there should be a version of BucketInfo.serialize()
+        // that does not JSON.stringify()
+        const bucketInfo = BucketInfo.fromObj(bucketMD);
+        const bucketMDStr = bucketInfo.serialize();
+        const newBucketMD = JSON.parse(bucketMDStr);
+        const m = this.getCollection(METASTORE);
+        // we don't have to test bucket existence here as it is done
+        // on the upper layers
+        m.update({
+            _id: bucketName,
+        }, {
+            _id: bucketName,
+            value: newBucketMD,
+        }, {
+            upsert: true,
+        }, () => cb());
+    }
+
+    getBucketAttributes(bucketName, log, cb) {
+        const m = this.getCollection(METASTORE);
+        m.findOne({
+            _id: bucketName,
+        }, {}, (err, doc) => {
+            if (err) {
+                return cb(errors.InternalError);
+            }
+            if (!doc) {
+                return cb(errors.NoSuchBucket);
+            }
+            // FIXME: there should be a version of BucketInfo.deserialize()
+            // that properly inits w/o JSON.parse()
+            const bucketMDStr = JSON.stringify(doc.value);
+            const bucketMD = BucketInfo.deSerialize(bucketMDStr);
+            return cb(null, bucketMD);
+        });
+    }
+
+    getBucketAndObject(bucketName, objName, params, log, cb) {
+        this.getBucketAttributes(bucketName, log, (err, bucket) => {
+            if (err) {
+                return cb(err);
+            }
+            this.getObject(bucketName, objName, params, log, (err, obj) => {
+                if (err) {
+                    if (err === errors.NoSuchKey) {
+                        return cb(null,
+                                  { bucket:
+                                    BucketInfo.fromObj(bucket).serialize(),
+                                  });
+                    }
+                    return cb(err);
+                }
+                return cb(null, {
+                    bucket: BucketInfo.fromObj(bucket).serialize(),
+                    obj: JSON.stringify(obj),
+                });
+            });
+            return undefined;
+        });
+    }
+
+    putBucketAttributes(bucketName, bucketMD, log, cb) {
+        // FIXME: there should be a version of BucketInfo.serialize()
+        // that does not JSON.stringify()
+        const bucketInfo = BucketInfo.fromObj(bucketMD);
+        const bucketMDStr = bucketInfo.serialize();
+        const newBucketMD = JSON.parse(bucketMDStr);
+        const m = this.getCollection(METASTORE);
+        m.update({
+            _id: bucketName,
+        }, {
+            _id: bucketName,
+            value: newBucketMD,
+        }, {
+            upsert: true,
+        }, () => cb());
+    }
+
+    /*
+     * Delete bucket from metastore
+     */
+    deleteBucketStep2(bucketName, log, cb) {
+        const m = this.getCollection(METASTORE);
+        m.findOneAndDelete({
+            _id: bucketName,
+        }, {}, (err, result) => {
+            if (err) {
+                return cb(errors.InternalError);
+            }
+            if (result.ok !== 1) {
+                return cb(errors.InternalError);
+            }
+            return cb(null);
+        });
+    }
+
+    /*
+     * Drop the bucket then process to step 2. Checking
+     * the count is already done by the upper layer. We don't need to be
+     * atomic because the call is protected by a delete_pending flag
+     * in the upper layer.
+     * 2 cases here:
+     * 1) the collection may not yet exist (or being already dropped
+     * by a previous call)
+     * 2) the collection may exist.
+     */
+    deleteBucket(bucketName, log, cb) {
+        const c = this.getCollection(bucketName);
+        c.drop({}, err => {
+            if (err) {
+                if (err.codeName === 'NamespaceNotFound') {
+                    return this.deleteBucketStep2(bucketName, log, cb);
+                }
+                return cb(errors.InternalError);
+            }
+            return this.deleteBucketStep2(bucketName, log, cb);
+        });
+    }
+
+    /*
+     * In this case we generate a versionId and
+     * sequentially create the object THEN update the master
+     */
+    putObjectVerCase1(c, bucketName, objName, objVal, params, log, cb) {
+        const versionId = generateVersionId();
+        // eslint-disable-next-line
+        objVal.versionId = versionId;
+        const vObjName = formatVersionKey(objName, versionId);
+        c.bulkWrite([{
+            updateOne: {
+                filter: {
+                    _id: vObjName,
+                },
+                update: {
+                    _id: vObjName, value: objVal,
+                },
+                upsert: true,
+            },
+        }, {
+            updateOne: {
+                filter: {
+                    _id: objName,
+                },
+                update: {
+                    _id: objName, value: objVal,
+                },
+                upsert: true,
+            },
+        }], {
+            ordered: 1,
+        }, () => cb(null, `{"versionId": "${versionId}"}`));
+    }
+
+    /*
+     * Case used when versioning has been disabled after objects
+     * have been created with versions
+     */
+    putObjectVerCase2(c, bucketName, objName, objVal, params, log, cb) {
+        const versionId = generateVersionId();
+        // eslint-disable-next-line
+        objVal.versionId = versionId;
+        c.update({
+            _id: objName,
+        }, {
+            _id: objName,
+            value: objVal,
+        }, {
+            upsert: true,
+        }, () => cb(null, `{"versionId": "${objVal.versionId}"}`));
+    }
+
+    /*
+     * In this case the aller provides a versionId. This function will
+     * sequentially update the object with given versionId THEN the
+     * master iff the provided versionId matches the one of the master
+     */
+    putObjectVerCase3(c, bucketName, objName, objVal, params, log, cb) {
+        // eslint-disable-next-line
+        objVal.versionId = params.versionId;
+        const vObjName = formatVersionKey(objName, params.versionId);
+        c.bulkWrite([{
+            updateOne: {
+                filter: {
+                    _id: vObjName,
+                },
+                update: {
+                    _id: vObjName, value: objVal,
+                },
+                upsert: true,
+            },
+        }, {
+            updateOne: {
+                // eslint-disable-next-line
+                filter: {
+                    _id: objName,
+                    'value.versionId': params.versionId,
+                },
+                update: {
+                    _id: objName, value: objVal,
+                },
+                upsert: true,
+            },
+        }], {
+            ordered: 1,
+        }, () => cb(null, `{"versionId": "${objVal.versionId}"}`));
+    }
+
+    /*
+     * Put object when versioning is not enabled
+     */
+    putObjectNoVer(c, bucketName, objName, objVal, params, log, cb) {
+        c.update({
+            _id: objName,
+        }, {
+            _id: objName,
+            value: objVal,
+        }, {
+            upsert: true,
+        }, () => cb());
+    }
+
+    putObject(bucketName, objName, objVal, params, log, cb) {
+        MongoUtils.serialize(objVal);
+        const c = this.getCollection(bucketName);
+        if (params && params.versioning && !params.versionId) {
+            return this.putObjectVerCase1(c, bucketName, objName, objVal,
+                                          params, log, cb);
+        } else if (params && params.versionId === '') {
+            return this.putObjectVerCase2(c, bucketName, objName, objVal,
+                                          params, log, cb);
+        } else if (params && params.versionId) {
+            return this.putObjectVerCase3(c, bucketName, objName, objVal,
+                                          params, log, cb);
+        }
+        return this.putObjectNoVer(c, bucketName, objName, objVal,
+                                   params, log, cb);
+    }
+
+    getObject(bucketName, objName, params, log, cb) {
+        const c = this.getCollection(bucketName);
+        if (params && params.versionId) {
+            // eslint-disable-next-line
+            objName = formatVersionKey(objName, params.versionId);
+        }
+        c.findOne({
+            _id: objName,
+        }, {}, (err, doc) => {
+            if (err) {
+                return cb(errors.InternalError);
+            }
+            if (!doc) {
+                return cb(errors.NoSuchKey);
+            }
+            if (doc.value.isPHD) {
+                this.getLatestVersion(c, objName, log, (err, value) => {
+                    if (err) {
+                        log.error('getting latest version', err);
+                        return cb(err);
+                    }
+                    return cb(null, value);
+                });
+                return undefined;
+            }
+            MongoUtils.unserialize(doc.value);
+            return cb(null, doc.value);
+        });
+    }
+
+    /*
+     * This function return the latest version
+     */
+    getLatestVersion(c, objName, log, cb) {
+        c.find({
+            _id: {
+                $gt: objName,
+                $lt: `${objName}${VID_SEPPLUS}`,
+            },
+        }, {}).
+            sort({
+                _id: 1,
+            }).
+            limit(1).
+            toArray(
+                (err, keys) => {
+                    if (err) {
+                        return cb(errors.InternalError);
+                    }
+                    if (keys.length === 0) {
+                        return cb(errors.NoSuchKey);
+                    }
+                    MongoUtils.unserialize(keys[0].value);
+                    return cb(null, keys[0].value);
+                });
+    }
+
+    /*
+     * repair the master with a new value. There can be
+     * race-conditions or legit updates so place an atomic condition
+     * on PHD flag and mst version.
+     */
+    repair(c, objName, objVal, mst, log, cb) {
+        MongoUtils.serialize(objVal);
+        // eslint-disable-next-line
+        c.findOneAndReplace({
+            _id: objName,
+            'value.isPHD': true,
+            'value.versionId': mst.versionId,
+        }, {
+            _id: objName,
+            value: objVal,
+        }, {
+            upsert: true,
+        }, (err, result) => {
+            if (err) {
+                return cb(errors.InternalError);
+            }
+            if (result.ok !== 1) {
+                return cb(errors.InternalError);
+            }
+            return cb(null);
+        });
+    }
+
+    /*
+     * Get the latest version and repair. The process is safe because
+     * we never replace a non-PHD master
+     */
+    asyncRepair(c, objName, mst, log) {
+        this.getLatestVersion(c, objName, log, (err, value) => {
+            if (err) {
+                log.error('async-repair: getting latest version', err);
+                return undefined;
+            }
+            this.repair(c, objName, value, mst, log, err => {
+                if (err) {
+                    log.error('async-repair failed', err);
+                    return undefined;
+                }
+                log.debug('async-repair success');
+                return undefined;
+            });
+            return undefined;
+        });
+    }
+
+    /*
+     * the master is a PHD so we try to see if it is the latest of its
+     * kind to get rid of it, otherwise we asynchronously repair it
+     */
+    deleteOrRepairPHD(c, bucketName, objName, mst, log, cb) {
+        // Check if there are other versions available
+        this.getLatestVersion(c, objName, log, err => {
+            if (err) {
+                if (err === errors.NoSuchKey) {
+                    // We try to delete the master. A race condition
+                    // is possible here: another process may recreate
+                    // a master or re-delete it in between so place an
+                    // atomic condition on the PHD flag and the mst
+                    // version:
+                    // eslint-disable-next-line
+                    c.findOneAndDelete({
+                        _id: objName,
+                        'value.isPHD': true,
+                        'value.versionId': mst.versionId,
+                    }, {}, err => {
+                        if (err) {
+                            return cb(errors.InternalError);
+                        }
+                        // do not test result.ok === 1 because
+                        // both cases are expected
+                        return cb(null);
+                    });
+                    return undefined;
+                }
+                return cb(err);
+            }
+            // We have other versions available so repair:
+            setTimeout(() => {
+                this.asyncRepair(c, objName, mst, log);
+            }, ASYNC_REPAIR_TIMEOUT);
+            return cb(null);
+        });
+    }
+
+    /*
+     * Delete object when versioning is enabled and the version is
+     * master. In this case we sequentially update the master with a
+     * PHD flag (placeholder) and a unique non-existing version THEN
+     * we delete the specified versioned object. THEN we try to delete
+     * or repair the PHD we just created
+     */
+    deleteObjectVerMaster(c, bucketName, objName, params, log, cb) {
+        const vObjName = formatVersionKey(objName, params.versionId);
+        const _vid = generateVersionId();
+        const mst = generatePHDVersion(_vid);
+        c.bulkWrite([{
+            updateOne: {
+                filter: {
+                    _id: objName,
+                },
+                update: {
+                    _id: objName, value: mst,
+                },
+                upsert: true,
+            },
+        }, {
+            deleteOne: {
+                filter: {
+                    _id: vObjName,
+                },
+            },
+        }], {
+            ordered: 1,
+        }, () => this.deleteOrRepairPHD(c, bucketName, objName, mst, log, cb));
+    }
+
+    /*
+     * Delete object when versioning is enabled and the version is
+     * not master. It is a straight-forward atomic delete
+     */
+    deleteObjectVerNotMaster(c, bucketName, objName, params, log, cb) {
+        const vObjName = formatVersionKey(objName, params.versionId);
+        c.findOneAndDelete({
+            _id: vObjName,
+        }, {}, (err, result) => {
+            if (err) {
+                return cb(errors.InternalError);
+            }
+            if (result.ok !== 1) {
+                return cb(errors.InternalError);
+            }
+            return cb(null);
+        });
+    }
+
+    /*
+     * Delete object when versioning is enabled. We first find the
+     * master, if it is already a PHD we have a special processing,
+     * then we check if it matches the master versionId in such case
+     * we will create a PHD, otherwise we delete it
+     */
+    deleteObjectVer(c, bucketName, objName, params, log, cb) {
+        c.findOne({
+            _id: objName,
+        }, {}, (err, mst) => {
+            if (err) {
+                return cb(errors.InternalError);
+            }
+            if (!mst) {
+                return cb(errors.NoSuchKey);
+            }
+            if (mst.value.isPHD ||
+                mst.value.versionId === params.versionId) {
+                return this.deleteObjectVerMaster(c, bucketName, objName,
+                                                  params, log, cb);
+            }
+            return this.deleteObjectVerNotMaster(c, bucketName, objName,
+                                                 params, log, cb);
+        });
+    }
+
+    /*
+     * Atomically delete an object when versioning is not enabled
+     */
+    deleteObjectNoVer(c, bucketName, objName, params, log, cb) {
+        c.findOneAndDelete({
+            _id: objName,
+        }, {}, (err, result) => {
+            if (err) {
+                return cb(errors.InternalError);
+            }
+            if (result.ok !== 1) {
+                return cb(errors.InternalError);
+            }
+            return cb(null);
+        });
+    }
+
+    deleteObject(bucketName, objName, params, log, cb) {
+        const c = this.getCollection(bucketName);
+        if (params && params.versionId) {
+            return this.deleteObjectVer(c, bucketName, objName,
+                                        params, log, cb);
+        }
+        return this.deleteObjectNoVer(c, bucketName, objName,
+                                      params, log, cb);
+    }
+
+    internalListObject(bucketName, params, log, cb) {
+        const extName = params.listingType;
+        const extension = new arsenal.algorithms.list[extName](params, log);
+        const requestParams = extension.genMDParams();
+        const c = this.getCollection(bucketName);
+        let cbDone = false;
+        const stream = new MongoReadStream(c, requestParams);
+        stream
+            .on('data', e => {
+                if (extension.filter(e) < 0) {
+                    stream.emit('end');
+                    stream.destroy();
+                }
+            })
+            .on('error', err => {
+                if (!cbDone) {
+                    cbDone = true;
+                    const logObj = {
+                        rawError: err,
+                        error: err.message,
+                        errorStack: err.stack,
+                    };
+                    log.error('error listing objects', logObj);
+                    cb(errors.InternalError);
+                }
+            })
+            .on('end', () => {
+                if (!cbDone) {
+                    cbDone = true;
+                    const data = extension.result();
+                    cb(null, data);
+                }
+            });
+        return undefined;
+    }
+
+    listObject(bucketName, params, log, cb) {
+        return this.internalListObject(bucketName, params, log, cb);
+    }
+
+    listMultipartUploads(bucketName, params, log, cb) {
+        return this.internalListObject(bucketName, params, log, cb);
+    }
+
+    readUUID(log, cb) {
+        const i = this.getCollection(INFOSTORE);
+        i.findOne({
+            _id: __UUID,
+        }, {}, (err, doc) => {
+            if (err) {
+                return cb(errors.InternalError);
+            }
+            if (!doc) {
+                return cb(errors.NoSuchKey);
+            }
+            return cb(null, doc.value);
+        });
+    }
+
+    writeUUIDIfNotExists(uuid, log, cb) {
+        const i = this.getCollection(INFOSTORE);
+        i.insert({
+            _id: __UUID,
+            value: uuid,
+        }, {}, err => {
+            if (err) {
+                if (err.code === 11000) {
+                    // duplicate key error
+                    // FIXME: define a KeyAlreadyExists error in Arsenal
+                    return cb(errors.EntityAlreadyExists);
+                }
+                return cb(errors.InternalError);
+            }
+            // FIXME: shoud we check for result.ok === 1 ?
+            return cb(null);
+        });
+    }
+
+    /*
+     * we always try to generate a new UUID in order to be atomic in
+     * case of concurrency. The write will fail if it already exists.
+     */
+    getUUID(log, cb) {
+        const uuid = Uuid.v4();
+        this.writeUUIDIfNotExists(uuid, log, err => {
+            if (err) {
+                if (err === errors.InternalError) {
+                    return cb(err);
+                }
+                return this.readUUID(log, cb);
+            }
+            return cb(null, uuid);
+        });
+    }
+
+    getDiskUsage(cb) {
+        // FIXME: for basic one server deployment the infrastructure
+        // configurator shall set a path to the actual MongoDB volume.
+        // For Kub/cluster deployments there should be a more sophisticated
+        // way for guessing free space.
+        diskusage.check(config.mongodb.path !== undefined ?
+                        config.mongodb.path : '/', cb);
+    }
+
+    countItems(log, cb) {
+        const res = {
+            objects: 0,
+            versions: 0,
+            buckets: 0,
+        };
+        this.db.listCollections().toArray((err, collInfos) => {
+            async.eachLimit(collInfos, 10, (value, next) => {
+                if (value.name === METASTORE ||
+                    value.name === INFOSTORE ||
+                    value.name === USERSBUCKET) {
+                    // skip
+                    return next();
+                }
+                res.buckets++;
+                const c = this.getCollection(value.name);
+                // FIXME: there is currently no way of distinguishing
+                // master from versions and searching for VID_SEP
+                // does not work because there cannot be null bytes
+                // in $regex
+                c.count({
+                    // eslint-disable-next-line
+                    'value.versionId': {
+                        '$exists': false,
+                    },
+                }, {}, (err, result) => {
+                    if (err) {
+                        return next(errors.InternalError);
+                    }
+                    res.objects += result;
+                    c.count({
+                        // eslint-disable-next-line
+                        'value.versionId': {
+                            '$exists': true,
+                        },
+                    }, {}, (err, result) => {
+                        if (err) {
+                            return next(errors.InternalError);
+                        }
+                        res.versions += result;
+                        return next();
+                    });
+                    return undefined;
+                });
+                return undefined;
+            }, err => {
+                if (err) {
+                    return cb(err);
+                }
+                return cb(null, res);
+            });
+        });
+    }
+}
+
+module.exports = MongoClientInterface;

backends/metadata/mongoclient/readStream.js

@@ -0,0 +1,133 @@
+const Readable = require('stream').Readable;
+const MongoUtils = require('./utils');
+
+class MongoReadStream extends Readable {
+    constructor(c, options) {
+        super({
+            objectMode: true,
+            highWaterMark: 0,
+        });
+
+        if (options.limit === 0) {
+            return;
+        }
+
+        const query = {
+            _id: {},
+        };
+        if (options.reverse) {
+            if (options.start) {
+                query._id.$lte = options.start;
+            }
+            if (options.end) {
+                query._id.$gte = options.end;
+            }
+            if (options.gt) {
+                query._id.$lt = options.gt;
+            }
+            if (options.gte) {
+                query._id.$lte = options.gte;
+            }
+            if (options.lt) {
+                query._id.$gt = options.lt;
+            }
+            if (options.lte) {
+                query._id.$gte = options.lte;
+            }
+        } else {
+            if (options.start) {
+                query._id.$gte = options.start;
+            }
+            if (options.end) {
+                query._id.$lte = options.end;
+            }
+            if (options.gt) {
+                query._id.$gt = options.gt;
+            }
+            if (options.gte) {
+                query._id.$gte = options.gte;
+            }
+            if (options.lt) {
+                query._id.$lt = options.lt;
+            }
+            if (options.lte) {
+                query._id.$lte = options.lte;
+            }
+        }
+
+        if (!Object.keys(query._id).length) {
+            delete query._id;
+        }
+
+        this._cursor = c.find(query).sort({
+            _id: options.reverse ? -1 : 1,
+        });
+        if (options.limit && options.limit !== -1) {
+            this._cursor = this._cursor.limit(options.limit);
+        }
+        this._options = options;
+        this._destroyed = false;
+        this.on('end', this._cleanup.bind(this));
+    }
+
+    _read() {
+        if (this._destroyed) {
+            return;
+        }
+
+        this._cursor.next((err, doc) => {
+            if (this._destroyed) {
+                return;
+            }
+            if (err) {
+                this.emit('error', err);
+                return;
+            }
+
+            let key = undefined;
+            let value = undefined;
+
+            if (doc) {
+                key = doc._id;
+                MongoUtils.unserialize(doc.value);
+                value = JSON.stringify(doc.value);
+            }
+
+            if (key === undefined && value === undefined) {
+                this.push(null);
+            } else if (this._options.keys !== false &&
+                       this._options.values === false) {
+                this.push(key);
+            } else if (this._options.keys === false &&
+                       this._options.values !== false) {
+                this.push(value);
+            } else {
+                this.push({
+                    key,
+                    value,
+                });
+            }
+        });
+    }
+
+    _cleanup() {
+        if (this._destroyed) {
+            return;
+        }
+        this._destroyed = true;
+
+        this._cursor.close(err => {
+            if (err) {
+                this.emit('error', err);
+                return;
+            }
+            this.emit('close');
+        });
+    }
+
+    destroy() {
+        return this._cleanup();
+    }
+}
+
+module.exports = MongoReadStream;

backends/metadata/mongoclient/utils.js

@@ -0,0 +1,30 @@
+
+function escape(obj) {
+    return JSON.parse(JSON.stringify(obj).
+                      replace(/\$/g, '\uFF04').
+                      replace(/\./g, '\uFF0E'));
+}
+
+function unescape(obj) {
+    return JSON.parse(JSON.stringify(obj).
+                      replace(/\uFF04/g, '$').
+                      replace(/\uFF0E/g, '.'));
+}
+
+function serialize(objMD) {
+    // Tags require special handling since dot and dollar are accepted
+    if (objMD.tags) {
+        // eslint-disable-next-line
+        objMD.tags = escape(objMD.tags);
+    }
+}
+
+function unserialize(objMD) {
+    // Tags require special handling
+    if (objMD.tags) {
+        // eslint-disable-next-line
+        objMD.tags = unescape(objMD.tags);
+    }
+}
+
+module.exports = { escape, unescape, serialize, unserialize };

backends/metadata/wrapper.js

@@ -0,0 +1,256 @@
+const errors = require('arsenal').errors;
+
+const BucketClientInterface = require('./bucketclient/backend');
+const BucketFileInterface = require('./bucketfile/backend');
+const MongoClientInterface = require('./mongoclient/backend');
+const BucketInfo = require('arsenal').models.BucketInfo;
+const inMemory = require('./in_memory/backend');
+const { config } = require('../Config');
+
+let CdmiMetadata;
+try {
+    CdmiMetadata = require('cdmiclient').CdmiMetadata;
+} catch (err) {
+    CdmiMetadata = null;
+}
+
+let client;
+let implName;
+
+if (config.backends.metadata === 'mem') {
+    client = inMemory;
+    implName = 'memorybucket';
+} else if (config.backends.metadata === 'file') {
+    client = new BucketFileInterface();
+    implName = 'bucketfile';
+} else if (config.backends.metadata === 'scality') {
+    client = new BucketClientInterface();
+    implName = 'bucketclient';
+} else if (config.backends.metadata === 'mongodb') {
+    client = new MongoClientInterface();
+    implName = 'mongoclient';
+} else if (config.backends.metadata === 'cdmi') {
+    if (!CdmiMetadata) {
+        throw new Error('Unauthorized backend');
+    }
+
+    client = new CdmiMetadata({
+        path: config.cdmi.path,
+        host: config.cdmi.host,
+        port: config.cdmi.port,
+        readonly: config.cdmi.readonly,
+    });
+    implName = 'cdmi';
+}
+
+const metadata = {
+    createBucket: (bucketName, bucketMD, log, cb) => {
+        log.debug('creating bucket in metadata');
+        client.createBucket(bucketName, bucketMD, log, err => {
+            if (err) {
+                log.debug('error from metadata', { implName, error: err });
+                return cb(err);
+            }
+            log.trace('bucket created in metadata');
+            return cb(err);
+        });
+    },
+
+    updateBucket: (bucketName, bucketMD, log, cb) => {
+        log.debug('updating bucket in metadata');
+        client.putBucketAttributes(bucketName, bucketMD, log, err => {
+            if (err) {
+                log.debug('error from metadata', { implName, error: err });
+                return cb(err);
+            }
+            log.trace('bucket updated in metadata');
+            return cb(err);
+        });
+    },
+
+    getBucket: (bucketName, log, cb) => {
+        log.debug('getting bucket from metadata');
+        client.getBucketAttributes(bucketName, log, (err, data) => {
+            if (err) {
+                log.debug('error from metadata', { implName, error: err });
+                return cb(err);
+            }
+            log.trace('bucket retrieved from metadata');
+            return cb(err, BucketInfo.fromObj(data));
+        });
+    },
+
+    deleteBucket: (bucketName, log, cb) => {
+        log.debug('deleting bucket from metadata');
+        client.deleteBucket(bucketName, log, err => {
+            if (err) {
+                log.debug('error from metadata', { implName, error: err });
+                return cb(err);
+            }
+            log.debug('Deleted bucket from Metadata');
+            return cb(err);
+        });
+    },
+
+    putObjectMD: (bucketName, objName, objVal, params, log, cb) => {
+        log.debug('putting object in metadata');
+        const value = typeof objVal.getValue === 'function' ?
+            objVal.getValue() : objVal;
+        client.putObject(bucketName, objName, value, params, log,
+        (err, data) => {
+            if (err) {
+                log.debug('error from metadata', { implName, error: err });
+                return cb(err);
+            }
+            if (data) {
+                log.debug('object version successfully put in metadata',
+                { version: data });
+            } else {
+                log.debug('object successfully put in metadata');
+            }
+            return cb(err, data);
+        });
+    },
+
+    getBucketAndObjectMD: (bucketName, objName, params, log, cb) => {
+        log.debug('getting bucket and object from metadata',
+                  { database: bucketName, object: objName });
+        client.getBucketAndObject(bucketName, objName, params, log,
+            (err, data) => {
+                if (err) {
+                    log.debug('error from metadata', { implName, err });
+                    return cb(err);
+                }
+                log.debug('bucket and object retrieved from metadata',
+                { database: bucketName, object: objName });
+                return cb(err, data);
+            });
+    },
+
+    getObjectMD: (bucketName, objName, params, log, cb) => {
+        log.debug('getting object from metadata');
+        client.getObject(bucketName, objName, params, log, (err, data) => {
+            if (err) {
+                log.debug('error from metadata', { implName, err });
+                return cb(err);
+            }
+            log.debug('object retrieved from metadata');
+            return cb(err, data);
+        });
+    },
+
+    deleteObjectMD: (bucketName, objName, params, log, cb) => {
+        log.debug('deleting object from metadata');
+        client.deleteObject(bucketName, objName, params, log, err => {
+            if (err) {
+                log.debug('error from metadata', { implName, err });
+                return cb(err);
+            }
+            log.debug('object deleted from metadata');
+            return cb(err);
+        });
+    },
+
+    listObject: (bucketName, listingParams, log, cb) => {
+        const metadataUtils = require('./metadataUtils');
+        if (listingParams.listingType === undefined) {
+            // eslint-disable-next-line
+            listingParams.listingType = 'Delimiter';
+        }
+        client.listObject(bucketName, listingParams, log, (err, data) => {
+            log.debug('getting object listing from metadata');
+            if (err) {
+                log.debug('error from metadata', { implName, err });
+                return cb(err);
+            }
+            log.debug('object listing retrieved from metadata');
+            if (listingParams.listingType === 'DelimiterVersions') {
+                // eslint-disable-next-line
+                data.Versions = metadataUtils.parseListEntries(data.Versions);
+                if (data.Versions instanceof Error) {
+                    log.error('error parsing metadata listing', {
+                        error: data.Versions,
+                        listingType: listingParams.listingType,
+                        method: 'listObject',
+                    });
+                    return cb(errors.InternalError);
+                }
+                return cb(null, data);
+            }
+            // eslint-disable-next-line
+            data.Contents = metadataUtils.parseListEntries(data.Contents);
+            if (data.Contents instanceof Error) {
+                log.error('error parsing metadata listing', {
+                    error: data.Contents,
+                    listingType: listingParams.listingType,
+                    method: 'listObject',
+                });
+                return cb(errors.InternalError);
+            }
+            return cb(null, data);
+        });
+    },
+
+    listMultipartUploads: (bucketName, listingParams, log, cb) => {
+        client.listMultipartUploads(bucketName, listingParams, log,
+            (err, data) => {
+                log.debug('getting mpu listing from metadata');
+                if (err) {
+                    log.debug('error from metadata', { implName, err });
+                    return cb(err);
+                }
+                log.debug('mpu listing retrieved from metadata');
+                return cb(err, data);
+            });
+    },
+
+    switch: newClient => {
+        client = newClient;
+    },
+
+    getRaftBuckets: (raftId, log, cb) => {
+        if (!client.getRaftBuckets) {
+            return cb();
+        }
+        return client.getRaftBuckets(raftId, log, cb);
+    },
+
+    checkHealth: (log, cb) => {
+        if (!client.checkHealth) {
+            const defResp = {};
+            defResp[implName] = { code: 200, message: 'OK' };
+            return cb(null, defResp);
+        }
+        return client.checkHealth(implName, log, cb);
+    },
+
+    getUUID: (log, cb) => {
+        if (!client.getUUID) {
+            log.debug('returning empty uuid as fallback', { implName });
+            return cb(null, '');
+        }
+        return client.getUUID(log, cb);
+    },
+
+    getDiskUsage: (log, cb) => {
+        if (!client.getDiskUsage) {
+            log.debug('returning empty disk usage as fallback', { implName });
+            return cb(null, {});
+        }
+        return client.getDiskUsage(cb);
+    },
+
+    countItems: (log, cb) => {
+        if (!client.countItems) {
+            log.debug('returning zero item counts as fallback', { implName });
+            return cb(null, {
+                buckets: 0,
+                objects: 0,
+                versions: 0,
+            });
+        }
+        return client.countItems(log, cb);
+    },
+};
+
+module.exports = metadata;

bin/create_bucket_with_NFS_enabled.js

@@ -1,4 +0,0 @@
-#!/usr/bin/env node
-'use strict'; // eslint-disable-line strict
-
-require('../lib/nfs/utilities.js').createBucketWithNFSEnabled();

bin/search_bucket.js

@@ -1,108 +0,0 @@
-#!/bin/sh
-// 2>/dev/null ; exec "$(which nodejs 2>/dev/null || which node)" "$0" "$@"
-'use strict'; // eslint-disable-line strict
-
-const { auth } = require('arsenal');
-const commander = require('commander');
-
-const http = require('http');
-const https = require('https');
-const logger = require('../lib/utilities/logger');
-
-function _performSearch(host,
-                        port,
-                        bucketName,
-                        query,
-                        listVersions,
-                        accessKey,
-                        secretKey,
-                        sessionToken,
-                        verbose, ssl) {
-    const escapedSearch = encodeURIComponent(query);
-    const options = {
-        host,
-        port,
-        method: 'GET',
-        path: `/${bucketName}/?search=${escapedSearch}${listVersions ? '&&versions' : ''}`,
-        headers: {
-            'Content-Length': 0,
-        },
-        rejectUnauthorized: false,
-        versions: '',
-    };
-    if (sessionToken) {
-        options.headers['x-amz-security-token'] = sessionToken;
-    }
-    const transport = ssl ? https : http;
-    const request = transport.request(options, response => {
-        if (verbose) {
-            logger.info('response status code', {
-                statusCode: response.statusCode,
-            });
-            logger.info('response headers', { headers: response.headers });
-        }
-        const body = [];
-        response.setEncoding('utf8');
-        response.on('data', chunk => body.push(chunk));
-        response.on('end', () => {
-            if (response.statusCode >= 200 && response.statusCode < 300) {
-                logger.info('Success');
-                process.stdout.write(body.join(''));
-                process.exit(0);
-            } else {
-                logger.error('request failed with HTTP Status ', {
-                    statusCode: response.statusCode,
-                    body: body.join(''),
-                });
-                process.exit(1);
-            }
-        });
-    });
-    // generateV4Headers exepects request object with path that does not
-    // include query
-    request.path = `/${bucketName}`;
-    const requestData = listVersions ? { search: query, versions: '' } : { search: query };
-    auth.client.generateV4Headers(request, requestData, accessKey, secretKey, 's3');
-    request.path = `/${bucketName}?search=${escapedSearch}${listVersions ? '&&versions' : ''}`;
-    if (verbose) {
-        logger.info('request headers', { headers: request._headers });
-    }
-    request.end();
-}
-
-/**
- * This function is used as a binary to send a request to S3 to perform a
- * search on the objects in a bucket
- *
- * @return {undefined}
- */
-function searchBucket() {
-    // TODO: Include other bucket listing possible query params?
-    commander
-        .version('0.0.1')
-        .option('-a, --access-key <accessKey>', 'Access key id')
-        .option('-k, --secret-key <secretKey>', 'Secret access key')
-        .option('-t, --session-token <sessionToken>', 'Session token')
-        .option('-b, --bucket <bucket>', 'Name of the bucket')
-        .option('-q, --query <query>', 'Search query')
-        .option('-h, --host <host>', 'Host of the server')
-        .option('-p, --port <port>', 'Port of the server')
-        .option('-s', '--ssl', 'Enable ssl')
-        .option('-l, --list-versions', 'List all versions of the objects that meet the search query, ' +
-            'otherwise only list the latest version')
-        .option('-v, --verbose')
-        .parse(process.argv);
-    const { host, port, accessKey, secretKey, sessionToken, bucket, query, listVersions, verbose, ssl } =
-        commander;
-
-    if (!host || !port || !accessKey || !secretKey || !bucket || !query) {
-        logger.error('missing parameter');
-        commander.outputHelp();
-        process.exit(1);
-    }
-
-    _performSearch(host, port, bucket, query, listVersions, accessKey, secretKey, sessionToken, verbose,
-        ssl);
-}
-
-searchBucket();

circle.yml

@@ -0,0 +1,45 @@
+---
+general:
+  branches:
+    ignore:
+      - /^ultron\/.*/   # Ignore ultron/* branches
+  artifacts:
+    - coverage/
+
+machine:
+  node:
+    version: 6.13.1
+  services:
+    - redis
+    - docker
+  ruby:
+    version: "2.4.1"
+  environment:
+    CXX: g++-4.9
+    ENABLE_LOCAL_CACHE: true
+    REPORT_TOKEN: report-token-1
+  hosts:
+    bucketwebsitetester.s3-website-us-east-1.amazonaws.com: 127.0.0.1
+
+dependencies:
+  override:
+    - rm -rf node_modules
+    - npm install
+  post:
+    - sudo pip install flake8 yamllint
+    - sudo pip install s3cmd==1.6.1
+    # fog and ruby testing dependencies
+    - gem install fog-aws -v 1.3.0
+    - gem install mime-types -v 3.1
+    - gem install rspec -v 3.5
+    - gem install json
+    # java sdk dependencies
+    - sudo apt-get install -y -q default-jdk
+
+
+test:
+  override:
+    - docker run --name squid-proxy -d --net=host
+      --publish 3128:3128 sameersbn/squid:3.3.8-23
+    - bash tests.bash:
+        parallel: true

config.json.example

@@ -1,143 +0,0 @@
-{
-    "port": 8000,
-    "listenOn": [],
-    "metricsPort": 8002,
-    "metricsListenOn": [],
-    "replicationGroupId": "RG001",
-    "workers": 4,
-    "restEndpoints": {
-        "localhost": "us-east-1",
-        "127.0.0.1": "us-east-1",
-        "cloudserver-front": "us-east-1",
-        "s3.docker.test": "us-east-1",
-        "127.0.0.2": "us-east-1",
-        "s3.amazonaws.com": "us-east-1",
-        "zenko-cloudserver-replicator": "us-east-1",
-        "lb": "us-east-1"
-    },
-    "websiteEndpoints": ["s3-website-us-east-1.amazonaws.com",
-                        "s3-website.us-east-2.amazonaws.com",
-                        "s3-website-us-west-1.amazonaws.com",
-                        "s3-website-us-west-2.amazonaws.com",
-                        "s3-website.ap-south-1.amazonaws.com",
-                        "s3-website.ap-northeast-2.amazonaws.com",
-                        "s3-website-ap-southeast-1.amazonaws.com",
-                        "s3-website-ap-southeast-2.amazonaws.com",
-                        "s3-website-ap-northeast-1.amazonaws.com",
-                        "s3-website.eu-central-1.amazonaws.com",
-                        "s3-website-eu-west-1.amazonaws.com",
-                        "s3-website-sa-east-1.amazonaws.com",
-                        "s3-website.localhost",
-                        "s3-website.scality.test",
-                        "zenkoazuretest.blob.core.windows.net"],
-    "replicationEndpoints": [{
-        "site": "zenko",
-        "servers": ["127.0.0.1:8000"],
-        "default": true
-    }, {
-        "site": "us-east-2",
-        "type": "aws_s3"
-    }],
-    "backbeat": {
-        "host": "localhost",
-        "port": 8900
-    },
-    "workflowEngineOperator": {
-        "host": "localhost",
-        "port": 3001
-    },
-    "cdmi": {
-        "host": "localhost",
-        "port": 81,
-        "path": "/dewpoint",
-        "readonly": true
-    },
-    "bucketd": {
-        "bootstrap": ["localhost:9000"]
-    },
-    "vaultd": {
-        "host": "localhost",
-        "port": 8500
-    },
-    "clusters": 1,
-    "log": {
-        "logLevel": "info",
-        "dumpLevel": "error"
-    },
-    "healthChecks": {
-        "allowFrom": ["127.0.0.1/8", "::1"]
-    },
-    "metadataClient": {
-        "host": "localhost",
-        "port": 9990
-    },
-    "dataClient": {
-        "host": "localhost",
-        "port": 9991
-    },
-    "pfsClient": {
-        "host": "localhost",
-        "port": 9992
-    },
-    "metadataDaemon": {
-        "bindAddress": "localhost",
-        "port": 9990
-    },
-    "dataDaemon": {
-        "bindAddress": "localhost",
-        "port": 9991
-    },
-    "pfsDaemon": {
-        "bindAddress": "localhost",
-        "port": 9992
-    },
-    "recordLog": {
-        "enabled": true,
-        "recordLogName": "s3-recordlog"
-    },
-    "mongodb": {
-        "replicaSetHosts": "localhost:27018,localhost:27019,localhost:27020",
-        "writeConcern": "majority",
-        "replicaSet": "rs0",
-        "readPreference": "primary",
-        "database": "metadata"
-    },
-    "authdata": "authdata.json",
-    "backends": {
-        "auth": "file",
-        "data": "file",
-        "metadata": "mongodb",
-        "kms": "file",
-        "quota": "none"
-    },
-    "externalBackends": {
-        "aws_s3": {
-            "httpAgent": {
-                "keepAlive": false,
-                "keepAliveMsecs": 1000,
-                "maxFreeSockets": 256,
-                "maxSockets": null
-            }
-        },
-        "gcp": {
-            "httpAgent": {
-                "keepAlive": true,
-                "keepAliveMsecs": 1000,
-                "maxFreeSockets": 256,
-                "maxSockets": null
-            }
-        }
-    },
-    "requests": {
-        "viaProxy": false,
-        "trustedProxyCIDRs": [],
-        "extractClientIPFromHeader": ""
-    },
-    "bucketNotificationDestinations": [
-        {
-            "resource": "target1",
-            "type": "dummy",
-            "host": "localhost:6000"
-        }
-    ]
-}

config.json.vitastor

@@ -1,71 +0,0 @@
-{
-    "port": 8000,
-    "listenOn": [],
-    "metricsPort": 8002,
-    "metricsListenOn": [],
-    "replicationGroupId": "RG001",
-    "restEndpoints": {
-        "localhost": "STANDARD",
-        "127.0.0.1": "STANDARD",
-        "yourhostname.ru": "STANDARD"
-    },
-    "websiteEndpoints": [
-        "static.yourhostname.ru"
-    ],
-    "replicationEndpoints": [ {
-        "site": "zenko",
-        "servers": ["127.0.0.1:8000"],
-        "default": true
-    } ],
-    "log": {
-        "logLevel": "info",
-        "dumpLevel": "error"
-    },
-    "healthChecks": {
-        "allowFrom": ["127.0.0.1/8", "::1"]
-    },
-    "backends": {
-        "metadata": "mongodb"
-    },
-    "mongodb": {
-        "replicaSetHosts": "127.0.0.1:27017",
-        "writeConcern": "majority",
-        "replicaSet": "rs0",
-        "readPreference": "primary",
-        "database": "s3",
-        "authCredentials": {
-            "username": "s3",
-            "password": ""
-        }
-    },
-    "externalBackends": {
-        "aws_s3": {
-            "httpAgent": {
-                "keepAlive": false,
-                "keepAliveMsecs": 1000,
-                "maxFreeSockets": 256,
-                "maxSockets": null
-            }
-        },
-        "gcp": {
-            "httpAgent": {
-                "keepAlive": true,
-                "keepAliveMsecs": 1000,
-                "maxFreeSockets": 256,
-                "maxSockets": null
-            }
-        }
-    },
-    "requests": {
-        "viaProxy": false,
-        "trustedProxyCIDRs": [],
-        "extractClientIPFromHeader": ""
-    },
-    "bucketNotificationDestinations": [
-        {
-            "resource": "target1",
-            "type": "dummy",
-            "host": "localhost:6000"
-        }
-    ]
-}

config/authdata.json

@@ -0,0 +1,23 @@
+{
+    "accounts": [{
+        "name": "Bart",
+        "email": "sampleaccount1@sampling.com",
+        "arn": "arn:aws:iam::123456789012:root",
+        "canonicalID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be",
+        "shortid": "123456789012",
+        "keys": [{
+            "access": "accessKey1",
+            "secret": "verySecretKey1"
+        }]
+    }, {
+        "name": "Lisa",
+        "email": "sampleaccount2@sampling.com",
+        "arn": "arn:aws:iam::123456789013:root",
+        "canonicalID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2bf",
+        "shortid": "123456789013",
+        "keys": [{
+            "access": "accessKey2",
+            "secret": "verySecretKey2"
+        }]
+    }]
+}

config/config.json

@@ -23,8 +23,7 @@
                         "s3-website-eu-west-1.amazonaws.com",
                         "s3-website-sa-east-1.amazonaws.com",
                         "s3-website.localhost",
-                        "s3-website.scality.test",
+                        "s3-website.scality.test"],
-                        "zenkoazuretest.blob.core.windows.net"],
     "replicationEndpoints": [{
         "site": "zenko",
         "servers": ["127.0.0.1:8000"],
@@ -40,7 +39,7 @@
         "readonly": true
     },
     "bucketd": {
-        "bootstrap": ["localhost"]
+        "bootstrap": ["localhost:9000"]
     },
     "vaultd": {
         "host": "localhost",
@@ -75,52 +74,8 @@
         "recordLogName": "s3-recordlog"
     },
     "mongodb": {
-       "replicaSetHosts": "localhost:27017,localhost:27018,localhost:27019",
+	"host": "localhost",
-       "writeConcern": "majority",
+	"port": 27018,
-       "replicaSet": "rs0",
-       "readPreference": "primary",
 	"database": "metadata"
-    },
-    "certFilePaths": {
-        "key": "tests/unit/testConfigs/allOptsConfig/key.txt",
-        "cert": "tests/unit/testConfigs/allOptsConfig/cert.txt",
-        "ca": "tests/unit/testConfigs/allOptsConfig/caBundle.txt"
-    },
-    "outboundProxy": {
-        "url": "http://test:8001",
-        "caBundle": "tests/unit/testConfigs/allOptsConfig/caBundle.txt",
-        "key": "tests/unit/testConfigs/allOptsConfig/key.txt",
-        "cert": "tests/unit/testConfigs/allOptsConfig/cert.txt"
-    },
-    "localCache": {
-        "name": "zenko",
-        "sentinels": "localhost:6379"
-    },
-    "redis": {
-        "name": "zenko",
-        "sentinels": "localhost:6379"
-    },
-    "scuba": {
-        "host": "localhost",
-        "port": 8100
-    },
-    "quota": {
-        "maxStalenessMS": 86400000
-    },
-    "utapi": {
-        "redis": {
-            "host": "localhost",
-            "port": 6379,
-            "retry": {
-                "connectBackoff": {
-                    "min": 10,
-                    "max": 1000,
-                    "factor": 1.5,
-                    "jitter": 0.1,
-                    "deadline": 10000
     }
 }
-        }
-    },
-    "overlayVersion": 4
-}

config/locationConfig.json

@@ -1,112 +1,82 @@
 {
     "us-east-1": {
         "type": "file",
-        "objectId": "us-east-1",
         "legacyAwsBehavior": true,
         "details": {}
     },
     "us-east-2": {
         "type": "file",
-        "objectId": "us-east-2",
         "legacyAwsBehavior": false,
         "details": {}
     },
     "us-west-1": {
         "type": "file",
-        "objectId": "us-west-1",
         "legacyAwsBehavior": false,
         "details": {}
     },
     "us-west-2": {
         "type": "file",
-        "objectId": "us-west-2",
         "legacyAwsBehavior": false,
         "details": {}
     },
     "ca-central-1": {
         "type": "file",
-        "objectId": "ca-central-1",
         "legacyAwsBehavior": false,
         "details": {}
     },
     "cn-north-1": {
         "type": "file",
-        "objectId": "cn-north-1",
         "legacyAwsBehavior": false,
         "details": {}
     },
     "ap-south-1": {
         "type": "file",
-        "objectId": "ap-south-1",
         "legacyAwsBehavior": false,
         "details": {}
     },
     "ap-northeast-1": {
         "type": "file",
-        "objectId": "ap-northeast-1",
         "legacyAwsBehavior": false,
         "details": {}
     },
     "ap-northeast-2": {
         "type": "file",
-        "objectId": "ap-northeast-2",
         "legacyAwsBehavior": false,
         "details": {}
     },
     "ap-southeast-1": {
         "type": "file",
-        "objectId": "ap-southeast-1",
         "legacyAwsBehavior": false,
         "details": {}
     },
     "ap-southeast-2": {
         "type": "file",
-        "objectId": "ap-southeast-2",
         "legacyAwsBehavior": false,
         "details": {}
     },
     "eu-central-1": {
         "type": "file",
-        "objectId": "eu-central-1",
         "legacyAwsBehavior": false,
         "details": {}
     },
     "eu-west-1": {
         "type": "file",
-        "objectId": "eu-west-1",
         "legacyAwsBehavior": false,
         "details": {}
     },
     "eu-west-2": {
         "type": "file",
-        "objectId": "eu-west-2",
         "legacyAwsBehavior": false,
         "details": {}
     },
     "EU": {
         "type": "file",
-        "objectId": "EU",
         "legacyAwsBehavior": false,
         "details": {}
     },
     "sa-east-1": {
         "type": "file",
-        "objectId": "sa-east-1",
         "legacyAwsBehavior": false,
         "details": {}
-    },
-    "location-dmf-v1": {
-        "type": "dmf",
-        "objectId": "location-dmf-v1",
-        "legacyAwsBehavior": false,
-        "isCold": true,
-        "details": {}
-    },
-    "location-azure-archive-v1": {
-        "type": "azure_archive",
-        "objectId": "location-azure-archive-v1",
-        "legacyAwsBehavior": false,
-        "isCold": true,
-        "details": {}
     }
 }

constants.js

@@ -39,8 +39,6 @@ const constants = {
     // once the multipart upload is complete.
     mpuBucketPrefix: 'mpuShadowBucket',
     blacklistedPrefixes: { bucket: [], object: [] },
-    // GCP Object Tagging Prefix
-    gcpTaggingPrefix: 'aws-tag-',
     // PublicId is used as the canonicalID for a request that contains
     // no authentication information.  Requestor can access
     // only public resources
@@ -66,71 +64,48 @@ const constants = {
     // http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
     minimumAllowedPartSize: 5242880,
 
-    // AWS sets a maximum total parts limit
-    // https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html
-    maximumAllowedPartCount: 10000,
-
-    gcpMaximumAllowedPartCount: 1024,
-
     // Max size on put part or copy part is 5GB. For functional
     // testing use 110 MB as max
     maximumAllowedPartSize: process.env.MPU_TESTING === 'yes' ? 110100480 :
         5368709120,
 
-    // Max size allowed in a single put object request is 5GB
-    // https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html
-    maximumAllowedUploadSize: 5368709120,
-
     // AWS states max size for user-defined metadata (x-amz-meta- headers) is
     // 2 KB: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html
     // In testing, AWS seems to allow up to 88 more bytes, so we do the same.
     maximumMetaHeadersSize: 2136,
 
-    // Maximum HTTP headers size allowed
-    maxHttpHeadersSize: 14122,
-
     // hex digest of sha256 hash of empty string:
     emptyStringHash: crypto.createHash('sha256')
         .update('', 'binary').digest('hex'),
 
     // Queries supported by AWS that we do not currently support.
-    // Non-bucket queries
     unsupportedQueries: [
         'accelerate',
         'analytics',
         'inventory',
         'logging',
         'metrics',
-        'policyStatus',
+        'notification',
-        'publicAccessBlock',
+        'policy',
         'requestPayment',
+        'restore',
         'torrent',
     ],
-
     // Headers supported by AWS that we do not currently support.
     unsupportedHeaders: [
+        'x-amz-server-side-encryption',
         'x-amz-server-side-encryption-customer-algorithm',
+        'x-amz-server-side-encryption-aws-kms-key-id',
         'x-amz-server-side-encryption-context',
         'x-amz-server-side-encryption-customer-key',
         'x-amz-server-side-encryption-customer-key-md5',
     ],
 
     // user metadata header to set object locationConstraint
-    objectLocationConstraintHeader: 'x-amz-storage-class',
+    objectLocationConstraintHeader: 'x-amz-meta-scal-location-constraint',
-    lastModifiedHeader: 'x-amz-meta-x-scal-last-modified',
     legacyLocations: ['sproxyd', 'legacy'],
-    // declare here all existing service accounts and their properties
-    // (if any, otherwise an empty object)
-    serviceAccountProperties: {
-        replication: {},
-        lifecycle: {},
-        gc: {},
-        'md-ingestion': {
-            canReplicate: true,
-        },
-    },
     /* eslint-disable camelcase */
-    externalBackends: { aws_s3: true, azure: true, gcp: true, pfs: true, dmf: true, azure_archive: true },
+    externalBackends: { aws_s3: true, azure: true, gcp: true },
     // some of the available data backends  (if called directly rather
     // than through the multiple backend gateway) need a key provided
     // as a string as first parameter of the get/delete methods.
@@ -139,110 +114,15 @@ const constants = {
     // for external backends, don't call unless at least 1 minute
     // (60,000 milliseconds) since last call
     externalBackendHealthCheckInterval: 60000,
-    versioningNotImplBackends: { azure: true, gcp: true },
+    versioningNotImplBackends: { azure: true },
-    mpuMDStoredExternallyBackend: { aws_s3: true, gcp: true },
+    mpuMDStoredExternallyBackend: { aws_s3: true },
-    skipBatchDeleteBackends: { azure: true, gcp: true },
-    s3HandledBackends: { azure: true, gcp: true },
-    hasCopyPartBackends: { aws_s3: true, gcp: true },
     /* eslint-enable camelcase */
     mpuMDStoredOnS3Backend: { azure: true },
     azureAccountNameRegex: /^[a-z0-9]{3,24}$/,
     base64Regex: new RegExp('^(?:[A-Za-z0-9+/]{4})*' +
         '(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$'),
-    productName: 'APN/1.0 Scality/1.0 Scality CloudServer for Zenko',
-    // location constraint delimiter
-    zenkoSeparator: ':',
     // user metadata applied on zenko objects
     zenkoIDHeader: 'x-amz-meta-zenko-instance-id',
-    bucketOwnerActions: [
-        'bucketDeleteCors',
-        'bucketDeleteLifecycle',
-        'bucketDeletePolicy',
-        'bucketDeleteReplication',
-        'bucketDeleteWebsite',
-        'bucketGetCors',
-        'bucketGetLifecycle',
-        'bucketGetLocation',
-        'bucketGetPolicy',
-        'bucketGetReplication',
-        'bucketGetVersioning',
-        'bucketGetWebsite',
-        'bucketPutCors',
-        'bucketPutLifecycle',
-        'bucketPutPolicy',
-        'bucketPutReplication',
-        'bucketPutVersioning',
-        'bucketPutWebsite',
-        'objectDeleteTagging',
-        'objectGetTagging',
-        'objectPutTagging',
-        'objectPutLegalHold',
-        'objectPutRetention',
-    ],
-    // response header to be sent when there are invalid
-    // user metadata in the object's metadata
-    invalidObjectUserMetadataHeader: 'x-amz-missing-meta',
-    // Bucket specific queries supported by AWS that we do not currently support
-    // these queries may or may not be supported at object level
-    unsupportedBucketQueries: [
-    ],
-    suppressedUtapiEventFields: [
-        'object',
-        'location',
-        'versionId',
-    ],
-    allowedUtapiEventFilterFields: [
-        'operationId',
-        'location',
-        'account',
-        'user',
-        'bucket',
-    ],
-    arrayOfAllowed: [
-        'objectPutTagging',
-        'objectPutLegalHold',
-        'objectPutRetention',
-    ],
-    allowedUtapiEventFilterStates: ['allow', 'deny'],
-    allowedRestoreObjectRequestTierValues: ['Standard'],
-    lifecycleListing: {
-        CURRENT_TYPE: 'current',
-        NON_CURRENT_TYPE: 'noncurrent',
-        ORPHAN_DM_TYPE: 'orphan',
-    },
-    multiObjectDeleteConcurrency: 50,
-    maxScannedLifecycleListingEntries: 10000,
-    overheadField: [
-        'content-length',
-        'owner-id',
-        'versionId',
-        'isNull',
-        'isDeleteMarker',
-    ],
-    unsupportedSignatureChecksums: new Set([
-        'STREAMING-UNSIGNED-PAYLOAD-TRAILER',
-        'STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER',
-        'STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD',
-        'STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER',
-    ]),
-    supportedSignatureChecksums: new Set([
-        'UNSIGNED-PAYLOAD',
-        'STREAMING-AWS4-HMAC-SHA256-PAYLOAD',
-    ]),
-    ipv4Regex: /^(\d{1,3}\.){3}\d{1,3}(\/(3[0-2]|[12]?\d))?$/,
-    ipv6Regex: /^([\da-f]{1,4}:){7}[\da-f]{1,4}$/i,
-    // The AWS assumed Role resource type
-    assumedRoleArnResourceType: 'assumed-role',
-    // Session name of the backbeat lifecycle assumed role session.
-    backbeatLifecycleSessionName: 'backbeat-lifecycle',
-    actionsToConsiderAsObjectPut: [
-        'initiateMultipartUpload',
-        'objectPutPart',
-        'completeMultipartUpload',
-    ],
-    // if requester is not bucket owner, bucket policy actions should be denied with
-    // MethodNotAllowed error
-    onlyOwnerAllowed: ['bucketDeletePolicy', 'bucketGetPolicy', 'bucketPutPolicy'],
 };
 
 module.exports = constants;

core/kms/utilities.js

@@ -79,7 +79,7 @@ function createEncryptedBucket() {
         .option('-b, --bucket <bucket>', 'Name of the bucket')
         .option('-h, --host <host>', 'Host of the server')
         .option('-p, --port <port>', 'Port of the server')
-        .option('-s, --ssl', 'Enable ssl')
+        .option('-s', '--ssl', 'Enable ssl')
         .option('-v, --verbose')
         .option('-l, --location-constraint <locationConstraint>',
         'location Constraint')

core/kms/wrapper.js

@@ -8,6 +8,19 @@ const inMemory = require('./in_memory/backend').backend;
 const file = require('./file/backend');
 const KMIPClient = require('arsenal').network.kmipClient;
 const Common = require('./common');
+let scalityKMS;
+let scalityKMSImpl;
+try {
+     // eslint-disable-next-line import/no-unresolved
+    const ScalityKMS = require('scality-kms');
+    scalityKMS = new ScalityKMS(config.kms);
+    scalityKMSImpl = 'scalityKms';
+} catch (err) {
+    logger.warn('scality kms unavailable. ' +
+      'Using file kms backend unless mem specified.');
+    scalityKMS = file;
+    scalityKMSImpl = 'fileKms';
+}
 
 let client;
 let implName;
@@ -19,9 +32,8 @@ if (config.backends.kms === 'mem') {
     client = file;
     implName = 'fileKms';
 } else if (config.backends.kms === 'scality') {
-    const ScalityKMS = require('scality-kms');
+    client = scalityKMS;
-    client = new ScalityKMS(config.kms);
+    implName = scalityKMSImpl;
-    implName = 'scalityKms';
 } else if (config.backends.kms === 'kmip') {
     const kmipConfig = { kmip: config.kmip };
     if (!kmipConfig.kmip) {
@@ -57,13 +69,16 @@ class KMS {
      /**
       *
       * @param {string} bucketName - bucket name
-      * @param {object} sseConfig - SSE configuration
+      * @param {object} headers - request headers
       * @param {object} log - logger object
       * @param {function} cb - callback
       * @returns {undefined}
       * @callback called with (err, serverSideEncryptionInfo: object)
       */
-    static bucketLevelEncryption(bucketName, sseConfig, log, cb) {
+    static bucketLevelEncryption(bucketName, headers, log, cb) {
+        const sseAlgorithm = headers['x-amz-scal-server-side-encryption'];
+        const sseMasterKeyId =
+                  headers['x-amz-scal-server-side-encryption-aws-kms-key-id'];
         /*
         The purpose of bucket level encryption is so that the client does not
         have to send appropriate headers to trigger encryption on each object
@@ -71,31 +86,35 @@ class KMS {
         feasible in this system because we do not want to store this key
         in the bucket metadata.
          */
-        const { algorithm, configuredMasterKeyId, mandatory } = sseConfig;
+        if (sseAlgorithm === 'AES256' ||
-        const _mandatory = mandatory === true;
+            (sseAlgorithm === 'aws:kms' && sseMasterKeyId === undefined)) {
-        if (algorithm === 'AES256' || algorithm === 'aws:kms') {
+            this.createBucketKey(bucketName, log, (err, masterKeyId) => {
-            return this.createBucketKey(bucketName, log, (err, masterKeyId) => {
                 if (err) {
-                    return cb(err);
+                    cb(err);
+                    return;
                 }
-
                 const serverSideEncryptionInfo = {
                     cryptoScheme: 1,
-                    algorithm,
+                    algorithm: sseAlgorithm,
                     masterKeyId,
-                    mandatory: _mandatory,
+                    mandatory: true,
                 };
-
+                cb(null, serverSideEncryptionInfo);
-                if (algorithm === 'aws:kms' && configuredMasterKeyId) {
-                    serverSideEncryptionInfo.configuredMasterKeyId = configuredMasterKeyId;
-                }
-                return cb(null, serverSideEncryptionInfo);
             });
-        }
+        } else if (sseAlgorithm === 'aws:kms') {
+            const serverSideEncryptionInfo = {
+                cryptoScheme: 1,
+                algorithm: sseAlgorithm,
+                masterKeyId: sseMasterKeyId,
+                mandatory: true,
+            };
+            cb(null, serverSideEncryptionInfo);
+        } else {
             /*
              * no encryption
              */
-        return cb(null, null);
+            cb(null, null);
+        }
     }
 
     /**
@@ -150,18 +169,9 @@ class KMS {
     static createCipherBundle(serverSideEncryptionInfo,
                               log, cb) {
         const dataKey = this.createDataKey(log);
-
-        const { algorithm, configuredMasterKeyId, masterKeyId: bucketMasterKeyId } = serverSideEncryptionInfo;
-
-        let masterKeyId = bucketMasterKeyId;
-        if (configuredMasterKeyId) {
-            log.debug('using user configured kms master key id');
-            masterKeyId = configuredMasterKeyId;
-        }
-
         const cipherBundle = {
-            algorithm,
+            algorithm: serverSideEncryptionInfo.algorithm,
-            masterKeyId,
+            masterKeyId: serverSideEncryptionInfo.masterKeyId,
             cryptoScheme: 1,
             cipheredDataKey: null,
             cipher: null,
@@ -171,7 +181,7 @@ class KMS {
             function cipherDataKey(next) {
                 log.debug('ciphering a data key');
                 return client.cipherDataKey(cipherBundle.cryptoScheme,
-                    cipherBundle.masterKeyId,
+                    serverSideEncryptionInfo.masterKeyId,
                     dataKey, log, (err, cipheredDataKey) => {
                         if (err) {
                             log.debug('error from kms',
@@ -286,29 +296,6 @@ class KMS {
             return cb(err, decipherBundle);
         });
     }
-
-    static checkHealth(log, cb) {
-        if (!client.healthcheck) {
-            return cb(null, {
-                [implName]: { code: 200, message: 'OK' },
-            });
-        }
-        return client.healthcheck(log, err => {
-            const respBody = {};
-            if (err) {
-                respBody[implName] = {
-                    error: err.description,
-                    code: err.code,
-                };
-            } else {
-                respBody[implName] = {
-                    code: 200,
-                    message: 'OK',
-                };
-            }
-            return cb(null, respBody);
-        });
-    }
 }
 
 module.exports = KMS;

core/utapi/utapi.js

@@ -0,0 +1,18 @@
+const utapiServer = require('utapi').UtapiServer;
+const _config = require('../Config').config;
+
+// start utapi server
+if (_config.utapi) {
+    const fullConfig = Object.assign({}, _config.utapi);
+    if (_config.vaultd) {
+        Object.assign(fullConfig, { vaultd: _config.vaultd });
+    }
+    if (_config.https) {
+        Object.assign(fullConfig, { https: _config.https });
+    }
+    // copy healthcheck IPs
+    if (_config.healthChecks) {
+        Object.assign(fullConfig, { healthChecks: _config.healthChecks });
+    }
+    utapiServer(fullConfig);
+}

core/utapi/utapiReplay.js

@@ -0,0 +1,6 @@
+const UtapiReplay = require('utapi').UtapiReplay;
+const _config = require('../Config').config;
+
+// start utapi server
+const replay = new UtapiReplay(_config.utapi);
+replay.start();

core/utapi/utilities.js

@@ -2,55 +2,11 @@ const http = require('http');
 const https = require('https');
 const commander = require('commander');
 const { auth } = require('arsenal');
-const { UtapiClient, utapiVersion } = require('utapi');
+const { UtapiClient } = require('utapi');
 const logger = require('../utilities/logger');
 const _config = require('../Config').config;
-const { suppressedUtapiEventFields: suppressedEventFields } = require('../../constants');
 // setup utapi client
-let utapiConfig;
+const utapi = new UtapiClient(_config.utapi);
-
-if (utapiVersion === 1 && _config.utapi) {
-    if (_config.utapi.redis === undefined) {
-        utapiConfig = Object.assign({}, _config.utapi, { redis: _config.redis });
-    } else {
-        utapiConfig = Object.assign({}, _config.utapi);
-    }
-} else if (utapiVersion === 2) {
-    utapiConfig = Object.assign({
-        tls: _config.https,
-        suppressedEventFields,
-    }, _config.utapi || {});
-}
-
-const utapi = new UtapiClient(utapiConfig);
-
-const bucketOwnerMetrics = [
-    'completeMultipartUpload',
-    'multiObjectDelete',
-    'abortMultipartUpload',
-    'copyObject',
-    'deleteObject',
-    'putObject',
-    'uploadPartCopy',
-    'uploadPart',
-];
-
-function evalAuthInfo(authInfo, canonicalID, action) {
-    let accountId = authInfo.getCanonicalID();
-    let userId = authInfo.isRequesterAnIAMUser() ?
-        authInfo.getShortid() : undefined;
-    // If action impacts 'numberOfObjectsStored' or 'storageUtilized' metric
-    // only the bucket owner account's metrics should be updated
-    const canonicalIdMatch = authInfo.getCanonicalID() === canonicalID;
-    if (bucketOwnerMetrics.includes(action) && !canonicalIdMatch) {
-        accountId = canonicalID;
-        userId = undefined;
-    }
-    return {
-        accountId,
-        userId,
-    };
-}
 
 function _listMetrics(host,
                       port,
@@ -63,11 +19,6 @@ function _listMetrics(host,
                       recent,
                       ssl) {
     const listAction = recent ? 'ListRecentMetrics' : 'ListMetrics';
-    // If recent listing, we do not provide `timeRange` in the request
-    const requestObj = recent
-        ? { [metric]: metricType }
-        : { timeRange, [metric]: metricType };
-    const requestBody = JSON.stringify(requestObj);
     const options = {
         host,
         port,
@@ -76,7 +27,6 @@ function _listMetrics(host,
         headers: {
             'content-type': 'application/json',
             'cache-control': 'no-cache',
-            'content-length': Buffer.byteLength(requestBody),
         },
         rejectUnauthorized: false,
     };
@@ -114,8 +64,10 @@ function _listMetrics(host,
     if (verbose) {
         logger.info('request headers', { headers: request._headers });
     }
-
+    // If recent listing, we do not provide `timeRange` in the request
-    request.write(requestBody);
+    const requestObj = recent ? {} : { timeRange };
+    requestObj[metric] = metricType;
+    request.write(JSON.stringify(requestObj));
     request.end();
 }
 
@@ -193,19 +145,6 @@ function listMetrics(metricType) {
         }
     });
 
-    // The string `commander[metric]` is a comma-separated list of resources
-    // given by the user.
-    const resources = commander[metric].split(',');
-
-    // Validate passed accounts to remove any canonicalIDs
-    if (metric === 'accounts') {
-        const invalid = resources.filter(r => !/^\d{12}$/.test(r));
-        if (invalid.length > 0) {
-            logger.error(`Invalid account ID: ${invalid.join(', ')}`);
-            process.exit(1);
-        }
-    }
-
     const timeRange = [];
     // If recent listing, we disregard any start or end option given
     if (!recent) {
@@ -228,7 +167,9 @@ function listMetrics(metricType) {
             timeRange.push(numEnd);
         }
     }
-
+    // The string `commander[metric]` is a comma-separated list of resources
+    // given by the user.
+    const resources = commander[metric].split(',');
     _listMetrics(host, port, metric, resources, timeRange, accessKey, secretKey,
         verbose, recent, ssl);
 }
@@ -249,84 +190,13 @@ function listMetrics(metricType) {
  * @param {number} [metricObj.newByteLength] - (optional) new object size
  * @param {number|null} [metricObj.oldByteLength] - (optional) old object size
  * (obj. overwrites)
- * @param {number} [metricObj.numberOfObjects] - (optional) number of objects
+ * @param {number} [metricObj.numberOfObjects] - (optional) number of obects
  * added/deleted
- * @param {boolean} [metricObject.isDelete] - (optional) Indicates whether this
+ * @return {function} - `utapi.pushMetric`
- * is a delete operation
- * @return {function | undefined} - `utapi.pushMetric` or undefined if the action is
- * filtered out and not pushed to utapi.
  */
 function pushMetric(action, log, metricObj) {
-    const {
+    const { bucket, keys, byteLength, newByteLength,
-        bucket,
+            oldByteLength, numberOfObjects, authInfo, canonicalID } = metricObj;
-        keys,
-        versionId,
-        byteLength,
-        newByteLength,
-        oldByteLength,
-        numberOfObjects,
-        authInfo,
-        canonicalID,
-        location,
-        isDelete,
-        removedDeleteMarkers,
-    } = metricObj;
-
-    if (utapiVersion === 2) {
-        const incomingBytes = action === 'getObject' ? 0 : newByteLength;
-        let sizeDelta = incomingBytes;
-        if (Number.isInteger(oldByteLength) && Number.isInteger(newByteLength)) {
-            sizeDelta = newByteLength - oldByteLength;
-            // Include oldByteLength in conditional so we don't end up with `-0`
-        } else if (action === 'completeMultipartUpload' && !versionId && oldByteLength) {
-            // If this is a non-versioned bucket we need to decrement
-            // the sizeDelta added by uploadPart when completeMPU is called.
-            sizeDelta = -oldByteLength;
-        } else if (action === 'abortMultipartUpload' && byteLength) {
-            sizeDelta = -byteLength;
-        } else if (action === 'putDeleteMarkerObject' && byteLength) {
-            sizeDelta = -byteLength;
-        }
-
-        let objectDelta = isDelete ? -numberOfObjects : numberOfObjects;
-        // putDeleteMarkerObject does not pass numberOfObjects
-        if ((action === 'putDeleteMarkerObject' && byteLength === null)
-            || action === 'replicateDelete'
-            || action === 'replicateObject') {
-            objectDelta = 1;
-        } else if (action === 'multiObjectDelete') {
-            objectDelta = -(numberOfObjects + removedDeleteMarkers);
-        }
-
-        const utapiObj = {
-            operationId: action,
-            bucket,
-            location,
-            objectDelta,
-            sizeDelta: isDelete ? -byteLength : sizeDelta,
-            incomingBytes,
-            outgoingBytes: action === 'getObject' ? newByteLength : 0,
-        };
-
-        // Any operation from lifecycle that does not change object count or size is dropped
-        const isLifecycle = _config.lifecycleRoleName
-            && authInfo && authInfo.arn.endsWith(`:assumed-role/${_config.lifecycleRoleName}/backbeat-lifecycle`);
-        if (isLifecycle && !objectDelta && !sizeDelta) {
-            log.trace('ignoring pushMetric from lifecycle service user', { action, bucket, keys });
-            return undefined;
-        }
-
-        if (keys && keys.length === 1) {
-            [utapiObj.object] = keys;
-            if (versionId) {
-                utapiObj.versionId = versionId;
-            }
-        }
-
-        utapiObj.account = authInfo ? evalAuthInfo(authInfo, canonicalID, action).accountId : canonicalID;
-        utapiObj.user = authInfo ? evalAuthInfo(authInfo, canonicalID, action).userId : undefined;
-        return utapi.pushMetric(utapiObj);
-    }
     const utapiObj = {
         bucket,
         keys,
@@ -339,56 +209,16 @@ function pushMetric(action, log, metricObj) {
     // account-level metrics and the shortId for user-level metrics. Otherwise
     // check if the canonical ID is already provided for account-level metrics.
     if (authInfo) {
-        const { accountId, userId } = evalAuthInfo(authInfo, canonicalID, action);
+        utapiObj.accountId = authInfo.getCanonicalID();
-        utapiObj.accountId = accountId;
+        utapiObj.userId = authInfo.isRequesterAnIAMUser() ?
-        utapiObj.userId = userId;
+            authInfo.getShortid() : undefined;
     } else if (canonicalID) {
         utapiObj.accountId = canonicalID;
     }
     return utapi.pushMetric(action, log.getSerializedUids(), utapiObj);
 }
 
-/**
- * internal: get the unique location ID from the location name
- *
- * @param {string} location - location name
- * @return {string} - location unique ID
- */
-function _getLocationId(location) {
-    return _config.locationConstraints[location].objectId;
-}
-
-/**
- * Call the Utapi Client 'getLocationMetric' method with the
- * associated parameters
- * @param {string} location - name of data backend to list metric for
- * @param {object} log - werelogs logger
- * @param {function} cb - callback to call
- * @return {function} - `utapi.getLocationMetric`
- */
-function getLocationMetric(location, log, cb) {
-    const locationId = _getLocationId(location);
-    return utapi.getLocationMetric(locationId, log.getSerializedUids(), cb);
-}
-
-/**
- * Call the Utapi Client 'pushLocationMetric' method with the
- * associated parameters
- * @param {string} location - name of data backend
- * @param {number} byteLength - number of bytes
- * @param {object} log - werelogs logger
- * @param {function} cb - callback to call
- * @return {function} - `utapi.pushLocationMetric`
- */
-function pushLocationMetric(location, byteLength, log, cb) {
-    const locationId = _getLocationId(location);
-    return utapi.pushLocationMetric(locationId, byteLength,
-        log.getSerializedUids(), cb);
-}
-
 module.exports = {
     listMetrics,
     pushMetric,
-    getLocationMetric,
-    pushLocationMetric,
 };

docker-entrypoint.sh

@@ -1,220 +0,0 @@
-#!/bin/bash
-
-# set -e stops the execution of a script if a command or pipeline has an error
-set -e
-
-# modifying config.json
-JQ_FILTERS_CONFIG="."
-
-# ENDPOINT var can accept comma separated values
-# for multiple endpoint locations
-if [[ "$ENDPOINT" ]]; then
-    IFS="," read -ra HOST_NAMES <<< "$ENDPOINT"
-    for host in "${HOST_NAMES[@]}"; do
-        JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .restEndpoints[\"$host\"]=\"us-east-1\""
-    done
-    echo "Host name has been modified to ${HOST_NAMES[@]}"
-    echo "Note: In your /etc/hosts file on Linux, OS X, or Unix with root permissions, make sure to associate 127.0.0.1 with ${HOST_NAMES[@]}"
-fi
-
-if [[ "$LOG_LEVEL" ]]; then
-    if [[ "$LOG_LEVEL" == "info" || "$LOG_LEVEL" == "debug" || "$LOG_LEVEL" == "trace" ]]; then
-        JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .log.logLevel=\"$LOG_LEVEL\""
-        echo "Log level has been modified to $LOG_LEVEL"
-    else
-        echo "The log level you provided is incorrect (info/debug/trace)"
-    fi
-fi
-
-if [[ "$SSL" && "$HOST_NAMES" ]]; then
-# This condition makes sure that the certificates are not generated twice. (for docker restart)
-if [ ! -f ./ca.key ] || [ ! -f ./ca.crt ] || [ ! -f ./server.key ] || [ ! -f ./server.crt ] ; then
-# Compute config for utapi tests
-cat >>req.cfg <<EOF
-[req]
-distinguished_name = req_distinguished_name
-prompt = no
-req_extensions = s3_req
-
-[req_distinguished_name]
-CN = ${HOST_NAMES[0]}
-
-[s3_req]
-subjectAltName = @alt_names
-extendedKeyUsage = serverAuth, clientAuth
-
-[alt_names]
-DNS.1 = *.${HOST_NAMES[0]}
-DNS.2 = ${HOST_NAMES[0]}
-
-EOF
-
-## Generate SSL key and certificates
-# Generate a private key for your CSR
-openssl genrsa -out ca.key 2048
-# Generate a self signed certificate for your local Certificate Authority
-openssl req -new -x509 -extensions v3_ca -key ca.key -out ca.crt -days 99999  -subj "/C=US/ST=Country/L=City/O=Organization/CN=S3 CA Server"
-# Generate a key for S3 Server
-openssl genrsa -out server.key 2048
-# Generate a Certificate Signing Request for S3 Server
-openssl req -new -key server.key -out server.csr -config req.cfg
-# Generate a local-CA-signed certificate for S3 Server
-openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 99999 -sha256 -extfile req.cfg -extensions s3_req
-fi
-## Update S3Server config.json
-# This condition makes sure that certFilePaths section is not added twice. (for docker restart)
-if ! grep -q "certFilePaths" ./config.json; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .certFilePaths= { \"key\": \".\/server.key\", \"cert\": \".\/server.crt\", \"ca\": \".\/ca.crt\" }"
-fi
-fi
-
-if [[ "$LISTEN_ADDR" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .metadataDaemon.bindAddress=\"$LISTEN_ADDR\""
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .dataDaemon.bindAddress=\"$LISTEN_ADDR\""
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .pfsDaemon.bindAddress=\"$LISTEN_ADDR\""
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .listenOn=[\"$LISTEN_ADDR:8000\"]"
-fi
-
-if [[ "$REPLICATION_GROUP_ID" ]] ; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .replicationGroupId=\"$REPLICATION_GROUP_ID\""
-fi
-
-if [[ "$DATA_HOST" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .dataClient.host=\"$DATA_HOST\""
-fi
-
-if [[ "$METADATA_HOST" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .metadataClient.host=\"$METADATA_HOST\""
-fi
-
-if [[ "$PFSD_HOST" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .pfsClient.host=\"$PFSD_HOST\""
-fi
-
-if [[ "$MONGODB_HOSTS" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .mongodb.replicaSetHosts=\"$MONGODB_HOSTS\""
-fi
-
-if [[ "$MONGODB_RS" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .mongodb.replicaSet=\"$MONGODB_RS\""
-fi
-
-if [[ "$MONGODB_DATABASE" ]]; then
-   JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .mongodb.database=\"$MONGODB_DATABASE\""
-fi
-
-if [ -z "$REDIS_HA_NAME" ]; then
-    REDIS_HA_NAME='mymaster'
-fi
-
-if [[ "$REDIS_SENTINELS" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.name=\"$REDIS_HA_NAME\""
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.sentinels=\"$REDIS_SENTINELS\""
-elif [[ "$REDIS_HOST" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.host=\"$REDIS_HOST\""
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.port=6379"
-fi
-
-if [[ "$REDIS_PORT" ]] && [[ ! "$REDIS_SENTINELS" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.port=$REDIS_PORT"
-fi
-
-if [[ "$REDIS_SENTINELS" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.name=\"$REDIS_HA_NAME\""
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.sentinels=\"$REDIS_SENTINELS\""
-elif [[ "$REDIS_HA_HOST" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.host=\"$REDIS_HA_HOST\""
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.port=6379"
-fi
-
-if [[ "$REDIS_HA_PORT" ]] && [[ ! "$REDIS_SENTINELS" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.port=$REDIS_HA_PORT"
-fi
-
-if [[ "$RECORDLOG_ENABLED" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .recordLog.enabled=true"
-fi
-
-if [[ "$STORAGE_LIMIT_ENABLED" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .utapi.metrics[.utapi.metrics | length]=\"location\""
-fi
-
-if [[ "$CRR_METRICS_HOST" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .backbeat.host=\"$CRR_METRICS_HOST\""
-fi
-
-if [[ "$CRR_METRICS_PORT" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .backbeat.port=$CRR_METRICS_PORT"
-fi
-
-if [[ "$WE_OPERATOR_HOST" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .workflowEngineOperator.host=\"$WE_OPERATOR_HOST\""
-fi
-
-if [[ "$WE_OPERATOR_PORT" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .workflowEngineOperator.port=$WE_OPERATOR_PORT"
-fi
-
-if [[ "$HEALTHCHECKS_ALLOWFROM" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .healthChecks.allowFrom=[\"$HEALTHCHECKS_ALLOWFROM\"]"
-fi
-
-# external backends http(s) agent config
-
-# AWS
-if [[ "$AWS_S3_HTTPAGENT_KEEPALIVE" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.aws_s3.httpAgent.keepAlive=$AWS_S3_HTTPAGENT_KEEPALIVE"
-fi
-
-if [[ "$AWS_S3_HTTPAGENT_KEEPALIVE_MS" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.aws_s3.httpAgent.keepAliveMsecs=$AWS_S3_HTTPAGENT_KEEPALIVE_MS"
-fi
-
-if [[ "$AWS_S3_HTTPAGENT_KEEPALIVE_MAX_SOCKETS" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.aws_s3.httpAgent.maxSockets=$AWS_S3_HTTPAGENT_KEEPALIVE_MAX_SOCKETS"
-fi
-
-if [[ "$AWS_S3_HTTPAGENT_KEEPALIVE_MAX_FREE_SOCKETS" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.aws_s3.httpAgent.maxFreeSockets=$AWS_S3_HTTPAGENT_KEEPALIVE_MAX_FREE_SOCKETS"
-fi
-
-#GCP
-if [[ "$GCP_HTTPAGENT_KEEPALIVE" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.gcp.httpAgent.keepAlive=$GCP_HTTPAGENT_KEEPALIVE"
-fi
-
-if [[ "$GCP_HTTPAGENT_KEEPALIVE_MS" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.gcp.httpAgent.keepAliveMsecs=$GCP_HTTPAGENT_KEEPALIVE_MS"
-fi
-
-if [[ "$GCP_HTTPAGENT_KEEPALIVE_MAX_SOCKETS" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.gcp.httpAgent.maxSockets=$GCP_HTTPAGENT_KEEPALIVE_MAX_SOCKETS"
-fi
-
-if [[ "$GCP_HTTPAGENT_KEEPALIVE_MAX_FREE_SOCKETS" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.gcp.httpAgent.maxFreeSockets=$GCP_HTTPAGENT_KEEPALIVE_MAX_FREE_SOCKETS"
-fi
-
-if [[ -n "$BUCKET_DENY_FILTER" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .utapi.filter.deny.bucket=[\"$BUCKET_DENY_FILTER\"]"
-fi
-
-if [[ "$TESTING_MODE" ]]; then
-    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .testingMode=true"
-fi
-
-if [[ $JQ_FILTERS_CONFIG != "." ]]; then
-    jq "$JQ_FILTERS_CONFIG" config.json > config.json.tmp
-    mv config.json.tmp config.json
-fi
-
-if test -v INITIAL_INSTANCE_ID && test -v S3METADATAPATH && ! test -f ${S3METADATAPATH}/uuid ; then
-    echo -n ${INITIAL_INSTANCE_ID} > ${S3METADATAPATH}/uuid
-fi
-
-# s3 secret credentials for Zenko
-if [ -r /run/secrets/s3-credentials ] ; then
-    . /run/secrets/s3-credentials
-fi
-
-exec "$@"

docker/Dockerfile

@@ -0,0 +1,25 @@
+FROM node:6-slim
+MAINTAINER Giorgio Regni <gr@scality.com>
+
+WORKDIR /usr/src/app
+
+# Keep the .git directory in order to properly report version
+COPY ./package.json .
+
+RUN apt-get update \
+    && apt-get install -y jq python git build-essential --no-install-recommends \
+    && npm install --production \
+    && apt-get autoremove --purge -y python git build-essential \
+    && rm -rf /var/lib/apt/lists/* \
+    && npm cache clear \
+    && rm -rf ~/.node-gyp \
+    && rm -rf /tmp/npm-*
+
+COPY ./ ./
+
+VOLUME ["/usr/src/app/localData","/usr/src/app/localMetadata"]
+
+ENTRYPOINT ["/usr/src/app/docker-entrypoint.sh"]
+CMD [ "npm", "start" ]
+
+EXPOSE 8000

docker/DockerfileMem

@@ -7,16 +7,16 @@ COPY . /usr/src/app
 
 RUN apt-get update \
     && apt-get install -y jq python git build-essential --no-install-recommends \
-    && yarn install --production \
+    && npm install --production \
     && apt-get autoremove --purge -y python git build-essential \
     && rm -rf /var/lib/apt/lists/* \
-    && yarn cache clean \
+    && npm cache clear \
     && rm -rf ~/.node-gyp \
-    && rm -rf /tmp/yarn-*
+    && rm -rf /tmp/npm-*
 
 ENV S3BACKEND mem
 
 ENTRYPOINT ["/usr/src/app/docker-entrypoint.sh"]
-CMD [ "yarn", "start" ]
+CMD [ "npm", "start" ]
 
 EXPOSE 8000

docker/docker-entrypoint.sh

@@ -0,0 +1,107 @@
+#!/bin/bash
+
+# set -e stops the execution of a script if a command or pipeline has an error
+set -e
+
+# modifying config.json
+JQ_FILTERS_CONFIG="."
+
+if [[ "$ENDPOINT" ]]; then
+    HOST_NAME="$ENDPOINT"
+fi
+
+if [[ "$HOST_NAME" ]]; then
+    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .restEndpoints[\"$HOST_NAME\"]=\"us-east-1\""
+    echo "Host name has been modified to $HOST_NAME"
+    echo "Note: In your /etc/hosts file on Linux, OS X, or Unix with root permissions, make sure to associate 127.0.0.1 with $HOST_NAME"
+fi
+
+if [[ "$LOG_LEVEL" ]]; then
+    if [[ "$LOG_LEVEL" == "info" || "$LOG_LEVEL" == "debug" || "$LOG_LEVEL" == "trace" ]]; then
+        JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .log.logLevel=\"$LOG_LEVEL\""
+        echo "Log level has been modified to $LOG_LEVEL"
+    else
+        echo "The log level you provided is incorrect (info/debug/trace)"
+    fi
+fi
+
+if [[ "$SSL" && "$HOST_NAME" ]]; then
+# This condition makes sure that the certificates are not generated twice. (for docker restart)
+if [ ! -f ./ca.key ] || [ ! -f ./ca.crt ] || [ ! -f ./server.key ] || [ ! -f ./server.crt ] ; then
+# Compute config for utapi tests
+cat >>req.cfg <<EOF
+[req]
+distinguished_name = req_distinguished_name
+prompt = no
+req_extensions = s3_req
+
+[req_distinguished_name]
+CN = ${HOST_NAME}
+
+[s3_req]
+subjectAltName = @alt_names
+extendedKeyUsage = serverAuth, clientAuth
+
+[alt_names]
+DNS.1 = *.${HOST_NAME}
+DNS.2 = ${HOST_NAME}
+
+EOF
+
+## Generate SSL key and certificates
+# Generate a private key for your CSR
+openssl genrsa -out ca.key 2048
+# Generate a self signed certificate for your local Certificate Authority
+openssl req -new -x509 -extensions v3_ca -key ca.key -out ca.crt -days 99999  -subj "/C=US/ST=Country/L=City/O=Organization/CN=S3 CA Server"
+# Generate a key for S3 Server
+openssl genrsa -out server.key 2048
+# Generate a Certificate Signing Request for S3 Server
+openssl req -new -key server.key -out server.csr -config req.cfg
+# Generate a local-CA-signed certificate for S3 Server
+openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 99999 -sha256 -extfile req.cfg -extensions s3_req
+fi
+## Update S3Server config.json
+# This condition makes sure that certFilePaths section is not added twice. (for docker restart)
+if ! grep -q "certFilePaths" ./config.json; then
+    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .certFilePaths= { \"key\": \".\/server.key\", \"cert\": \".\/server.crt\", \"ca\": \".\/ca.crt\" }"
+fi
+fi
+
+if [[ "$LISTEN_ADDR" ]]; then
+    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .metadataDaemon.bindAddress=\"$LISTEN_ADDR\""
+    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .dataDaemon.bindAddress=\"$LISTEN_ADDR\""
+    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .listenOn=[\"$LISTEN_ADDR:8000\"]"
+fi
+
+if [[ "$DATA_HOST" ]]; then
+    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .dataClient.host=\"$DATA_HOST\""
+fi
+
+if [[ "$METADATA_HOST" ]]; then
+    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .metadataClient.host=\"$METADATA_HOST\""
+fi
+
+if [[ "$REDIS_HOST" ]]; then
+    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.host=\"$REDIS_HOST\""
+    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.port=6379"
+fi
+
+if [[ "$REDIS_PORT" ]]; then
+    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.port=$REDIS_PORT"
+fi
+
+if [[ "$RECORDLOG_ENABLED" ]]; then
+    JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .recordLog.enabled=true"
+fi
+
+if [[ $JQ_FILTERS_CONFIG != "." ]]; then
+    jq "$JQ_FILTERS_CONFIG" config.json > config.json.tmp
+    mv config.json.tmp config.json
+fi
+
+# s3 secret credentials for Zenko
+if [ -r /run/secrets/s3-credentials ] ; then
+    . /run/secrets/s3-credentials
+fi
+
+exec "$@"

docs/ARCHITECTURE.rst

@@ -27,7 +27,7 @@ including null versions and delete markers, described in the above
 links.
 
 Implementation of Bucket Versioning in Zenko CloudServer
---------------------------------------------------------
+-----------------------------------------
 
 Overview of Metadata and API Component Roles
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -66,7 +66,7 @@ The second section, `"Implementation of Bucket Versioning in
 API" <#implementation-of-bucket-versioning-in-api>`__, describes the way
 the metadata options are used in the API within S3 actions to create new
 versions, update their metadata, and delete them. The management of null
-versions and creation of delete markers is also described in this
+versions and creation of delete markers are also described in this
 section.
 
 Implementation of Bucket Versioning in Metadata
@@ -179,13 +179,12 @@ PUT
    the master version with this version.
 -  ``versionId: <versionId>`` create or update a specific version (for updating
    version's ACL or tags, or remote updates in geo-replication)
-
+   -  if the version identified by ``versionId`` happens to be the latest
-   *  if the version identified by ``versionId`` happens to be the latest
       version, the master version will be updated as well
-   *  if the master version is not as recent as the version identified by
+   -  if the master version is not as recent as the version identified by
       ``versionId``, as may happen with cross-region replication, the master
       will be updated as well
-   *  note that with ``versionId`` set to an empty string ``''``, it will
+   -  note that with ``versionId`` set to an empty string ``''``, it will
       overwrite the master version only (same as no options, but the master
       version will have a ``versionId`` property set in its metadata like
       any other version). The ``versionId`` will never be exposed to an
@@ -209,13 +208,10 @@ A deletion targeting the latest version of an object has to:
 -  delete the specified version identified by ``versionId``
 -  replace the master version with a version that is a placeholder for
    deletion
-
     -  this version contains a special keyword, 'isPHD', to indicate the
          master version was deleted and needs to be updated
-
 -  initiate a repair operation to update the value of the master
    version:
-
     -  involves listing the versions of the object and get the latest
        version to replace the placeholder delete version
     -  if no more versions exist, metadata deletes the master version,
@@ -746,7 +742,7 @@ Operation
 Startup
 ~~~~~~~
 
-The simplest deployment is still to launch with yarn start, this will
+The simplest deployment is still to launch with npm start, this will
 start one instance of the Zenko CloudServer connector and will listen on the
 locally bound dmd ports 9990 and 9991 (by default, see below).
 
@@ -755,7 +751,7 @@ command in the Zenko CloudServer directory:
 
 ::
 
-   yarn run start_dmd
+   npm run start_dmd
 
 This will open two ports:
 
@@ -768,9 +764,9 @@ This will open two ports:
 Then, one or more instances of Zenko CloudServer without the dmd can be started
 elsewhere with:
 
-.. code:: sh
+::
 
-   yarn run start_s3server
+   npm run start_s3server
 
 Configuration
 ~~~~~~~~~~~~~

docs/BUCKET_POLICIES.md

@@ -1,146 +0,0 @@
-# Bucket Policy Documentation
-
-## Description
-
-Bucket policy is a method of controlling access to a user's account at the
-resource level.
-There are three associated APIs:
-
-- PUT Bucket policy (see https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTpolicy.html)
-- GET Bucket policy (see https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETpolicy.html)
-- DELETE Bucket policy (see https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketDELETEpolicy.html)
-
-More information on bucket policies in general can be found at
-https://docs.aws.amazon.com/AmazonS3/latest/dev/using-iam-policies.html.
-
-## Requirements
-
-To prevent loss of access to a bucket, the root owner of a bucket will always
-be able to perform any of the three bucket policy-related operations, even
-if permission is explicitly denied.
-All other users must have permission to perform the desired operation.
-
-## Design
-
-On a PUTBucketPolicy request, the user provides a policy in JSON format.
-The policy is evaluated against our policy schema in Arsenal and, once
-validated, is stored as part of the bucket's metadata.
-On a GETBucketPolicy request, the policy is retrieved from the bucket's
-metadata.
-On a DELETEBucketPolicy request, the policy is deleted from the bucket's
-metadata.
-
-All other APIs are updated to check if a bucket policy is attached to the bucket
-the request is made on. If there is a policy, user authorization to perform
-the requested action is checked.
-
-### Differences Between Bucket and IAM Policies
-
-IAM policies are attached to an IAM identity and define what actions that
-identity is allowed to or denied from doing on what resource.
-Bucket policies attach only to buckets and define what actions are allowed or
-denied for which principles on that bucket. Permissions specified in a bucket
-policy apply to all objects in that bucket unless otherwise specified.
-
-Besides their attachment origins, the main structural difference between
-IAM policy and bucket policy is the requirement of a "Principal" element in
-bucket policies. This field is redundant in IAM policies.
-
-### Policy Validation
-
-For general guidelines for bucket policy structure, see examples here:
-https://docs.aws.amazon.com/AmazonS3/latest/dev//example-bucket-policies.html.
-
-Each bucket policy statement object requires at least four keys:
-"Effect", "Principle", "Resource", and "Action".
-
-"Effect" defines the effect of the policy and can have a string value of either
-"Allow" or "Deny".
-"Resource" defines to which bucket or list of buckets a policy is attached.
-An object within the bucket is also a valid resource. The element value can be
-either a single bucket or object ARN string or an array of ARNs.
-"Action" lists which action(s) the policy controls. Its value can also be either
-a string or array of S3 APIs. Each action is the API name prepended by "s3:".
-"Principle" specifies which user(s) are granted or denied access to the bucket
-resource. Its value can be a string or an object containing an array of users.
-Valid users can be identified with an account ARN, account id, or user ARN.
-
-There are also two optional bucket policy statement keys: Sid and Condition.
-
-"Sid" stands for "statement id". If this key is not included, one will be
-generated for the statement.
-"Condition" lists the condition under which a statement will take affect.
-The possibilities are as follows:
-
-- ArnEquals
-- ArnEqualsIfExists
-- ArnLike
-- ArnLikeIfExists
-- ArnNotEquals
-- ArnNotEqualsIfExists
-- ArnNotLike
-- ArnNotLikeIfExists
-- BinaryEquals
-- BinaryEqualsIfExists
-- BinaryNotEquals
-- BinaryNotEqualsIfExists
-- Bool
-- BoolIfExists
-- DateEquals
-- DateEqualsIfExists
-- DateGreaterThan
-- DateGreaterThanEquals
-- DateGreaterThanEqualsIfExists
-- DateGreaterThanIfExists
-- DateLessThan
-- DateLessThanEquals
-- DateLessThanEqualsIfExists
-- DateLessThanIfExists
-- DateNotEquals
-- DateNotEqualsIfExists
-- IpAddress
-- IpAddressIfExists
-- NotIpAddress
-- NotIpAddressIfExists
-- Null
-- NumericEquals
-- NumericEqualsIfExists
-- NumericGreaterThan
-- NumericGreaterThanEquals
-- NumericGreaterThanEqualsIfExists
-- NumericGreaterThanIfExists
-- NumericLessThan
-- NumericLessThanEquals
-- NumericLessThanEqualsIfExists
-- NumericLessThanIfExists
-- NumericNotEquals
-- NumericNotEqualsIfExists
-- StringEquals
-- StringEqualsIfExists
-- StringEqualsIgnoreCase
-- StringEqualsIgnoreCaseIfExists
-- StringLike
-- StringLikeIfExists
-- StringNotEquals
-- StringNotEqualsIfExists
-- StringNotEqualsIgnoreCase
-- StringNotEqualsIgnoreCaseIfExists
-- StringNotLike
-- StringNotLikeIfExists
-
-The value of the Condition key will be an object containing the desired
-condition name as that key. The value of inner object can be a string, boolean,
-number, or object, depending on the condition.
-
-## Authorization with Multiple Access Control Mechanisms
-
-In the case where multiple access control mechanisms (such as IAM policies,
-bucket policies, and ACLs) refer to the same resource, the principle of
-least-privilege is applied. Unless an action is explicitly allowed, access will
-by default be denied. An explicit DENY in any policy will trump another
-policy's ALLOW for an action. The request will only be allowed if at least one
-policy specifies an ALLOW, and there is no overriding DENY.
-
-The following diagram illustrates this logic:
-
-![Access_Control_Authorization_Chart](./images/access_control_authorization.png)

docs/CLIENTS.rst

@@ -178,7 +178,7 @@ Ruby
 ~~~~
 
 `AWS SDK for Ruby - Version 2 <http://docs.aws.amazon.com/sdkforruby/api/>`__
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. code:: ruby
 
@@ -239,7 +239,6 @@ Python
 Client integration
 
 .. code:: python
-
     import boto3
 
     client = boto3.client(
@@ -254,7 +253,6 @@ Client integration
 Full integration (with object mapping)
 
 .. code:: python
-
     import os
 
     from botocore.utils import fix_s3_host
@@ -295,51 +293,3 @@ Should force path-style requests even though v3 advertises it does by default.
     $client->createBucket(array(
         'Bucket' => 'bucketphp',
     ));
-
-Go
-~~
-
-`AWS Go SDK <https://github.com/aws/aws-sdk-go>`__
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. code:: go
-
-   package main
-
-   import (
-      "context"
-      "fmt"
-      "log"
-      "os"
-      "time"
-
-      "github.com/aws/aws-sdk-go/aws"
-      "github.com/aws/aws-sdk-go/aws/endpoints"
-      "github.com/aws/aws-sdk-go/aws/session"
-      "github.com/aws/aws-sdk-go/service/s3"
-   )
-
-   func main() {
-      os.Setenv("AWS_ACCESS_KEY_ID", "accessKey1")
-      os.Setenv("AWS_SECRET_ACCESS_KEY", "verySecretKey1")
-      endpoint := "http://localhost:8000"
-      timeout := time.Duration(10) * time.Second
-      sess := session.Must(session.NewSession())
-
-      // Create a context with a timeout that will abort the upload if it takes
-      // more than the passed in timeout.
-      ctx, cancel := context.WithTimeout(context.Background(), timeout)
-      defer cancel()
-
-      svc := s3.New(sess, &aws.Config{
-         Region:   aws.String(endpoints.UsEast1RegionID),
-         Endpoint: &endpoint,
-      })
-
-      out, err := svc.ListBucketsWithContext(ctx, &s3.ListBucketsInput{})
-      if err != nil {
-         log.Fatal(err)
-      } else {
-         fmt.Println(out)
-      }
-   }

docs/CONTRIBUTING.rst

@@ -14,7 +14,7 @@ Got an idea? Get started!
 In order to contribute, please follow the `Contributing
 Guidelines <https://github.com/scality/Guidelines/blob/master/CONTRIBUTING.md>`__.
 If anything is unclear to you, reach out to us on
-`forum <https://forum.zenko.io/>`__ or via a GitHub issue.
+`slack <https://zenko-io.slack.com/>`__ or via a GitHub issue.
 
 Don't write code? There are other ways to help!
 -----------------------------------------------

docs/DOCKER.rst

@@ -1,7 +1,11 @@
 Docker
 ======
 
-.. _environment-variables:
+-  `Environment Variables <#environment-variables>`__
+-  `Tunables and setup tips <#tunables-and-setup-tips>`__
+-  `Examples for continuous integration with
+   Docker <#continuous-integration-with-docker-hosted CloudServer>`__
+-  `Examples for going in production with Docker <#in-production-with-docker-hosted CloudServer>`__
 
 Environment Variables
 ---------------------
@@ -11,23 +15,21 @@ S3DATA
 
 S3DATA=multiple
 ^^^^^^^^^^^^^^^
-
+Allows you to run Scality Zenko CloudServer with multiple data backends, defined
-This variable enables running CloudServer with multiple data backends, defined
 as regions.
+When using multiple data backends, a custom ``locationConfig.json`` file is
+mandatory. It will allow you to set custom regions. You will then need to
+provide associated rest_endpoints for each custom region in your
+``config.json`` file.
+`Learn more about multiple backends configuration <../GETTING_STARTED/#location-configuration>`__
 
-For multiple data backends, a custom locationConfig.json file is required.
+If you are using Scality RING endpoints, please refer to your customer
-This file enables you to set custom regions. You must provide associated 
+documentation.
-rest_endpoints for each custom region in config.json.
 
-`Learn more about multiple-backend configurations <GETTING_STARTED.html#location-configuration>`__
+Running it with an AWS S3 hosted backend
-
+""""""""""""""""""""""""""""""""""""""""
-If you are using Scality RING endpoints, refer to your customer documentation.
+To run CloudServer with an S3 AWS backend, you will have to add a new section
-
+to your ``locationConfig.json`` file with the ``aws_s3`` location type:
-Running CloudServer with an AWS S3-Hosted Backend
-"""""""""""""""""""""""""""""""""""""""""""""""""
-
-To run CloudServer with an S3 AWS backend, add a new section to the 
-``locationConfig.json`` file with the ``aws_s3`` location type:
 
 .. code:: json
 
@@ -43,9 +45,10 @@ To run CloudServer with an S3 AWS backend, add a new section to the
     }
 (...)
 
-Edit your AWS credentials file to enable your preferred command-line tool.
+You will also have to edit your AWS credentials file to be able to use your
-This file must mention credentials for all backends in use. You can use 
+command line tool of choice. This file should mention credentials for all the
-several profiles if multiple profiles are configured.
+backends you're using. You can use several profiles when using multiple
+profiles.
 
 .. code:: json
 
@@ -56,124 +59,110 @@ several profiles if multiple profiles are configured.
 aws_access_key_id={{YOUR_ACCESS_KEY}}
 aws_secret_access_key={{YOUR_SECRET_KEY}}
 
-As with locationConfig.json, the AWS credentials file must be mounted at 
+Just as you need to mount your locationConfig.json, you will need to mount your
-run time: ``-v ~/.aws/credentials:/root/.aws/credentials`` on Unix-like 
+AWS credentials file at run time:
-systems (Linux, OS X, etc.), or 
+``-v ~/.aws/credentials:/root/.aws/credentials`` on Linux, OS X, or Unix or
 ``-v C:\Users\USERNAME\.aws\credential:/root/.aws/credentials`` on Windows
 
-.. note:: One account cannot copy to another account with a source and
+NOTE: One account can't copy to another account with a source and
 destination on real AWS unless the account associated with the
-   accessKey/secretKey pairs used for the destination bucket has source 
+access Key/secret Key pairs used for the destination bucket has rights
-   bucket access privileges. To enable this, update ACLs directly on AWS.
+to get in the source bucket. ACL's would have to be updated
+on AWS directly to enable this.
 
 S3BACKEND
-~~~~~~~~~
+~~~~~~
 
 S3BACKEND=file
-^^^^^^^^^^^^^^
+^^^^^^^^^^^
-
+When storing file data, for it to be persistent you must mount docker volumes
-For stored file data to persist, you must mount Docker volumes
+for both data and metadata. See `this section <#using-docker-volumes-in-production>`__
-for both data and metadata. See :ref:`In Production with a Docker-Hosted CloudServer <in-production-w-a-Docker-hosted-cloudserver>`
 
 S3BACKEND=mem
-^^^^^^^^^^^^^
+^^^^^^^^^^
-
+This is ideal for testing - no data will remain after container is shutdown.
-This is ideal for testing: no data remains after the container is shut down.
 
 ENDPOINT
 ~~~~~~~~
 
-This variable specifies the endpoint. To direct CloudServer requests to 
+This variable specifies your endpoint. If you have a domain such as
-new.host.com, for example, specify the endpoint with:
+new.host.com, by specifying that here, you and your users can direct s3
+server requests to new.host.com.
 
-.. code-block:: shell
+.. code:: shell
 
-    $ docker run -d --name cloudserver -p 8000:8000 -e ENDPOINT=new.host.com zenko/cloudserver
+    docker run -d --name s3server -p 8000:8000 -e ENDPOINT=new.host.com scality/s3server
 
-.. note:: On Unix-like systems (Linux, OS X, etc.) edit /etc/hosts
+Note: In your ``/etc/hosts`` file on Linux, OS X, or Unix with root
-   to associate 127.0.0.1 with new.host.com.
+permissions, make sure to associate 127.0.0.1 with ``new.host.com``
-
-REMOTE_MANAGEMENT_DISABLE
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-CloudServer is a part of `Zenko <https://www.zenko.io/>`__. When you run CloudServer standalone it will still try to connect to Orbit by default (browser-based graphical user interface for Zenko).
-
-Setting this variable to true(1) will default to accessKey1 and verySecretKey1 for credentials and disable the automatic Orbit management:
-
-.. code-block:: shell
-
-    $ docker run -d --name cloudserver -p 8000:8000 -e REMOTE_MANAGEMENT_DISABLE=1 zenko/cloudserver
 
 SCALITY\_ACCESS\_KEY\_ID and SCALITY\_SECRET\_ACCESS\_KEY
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 These variables specify authentication credentials for an account named
-“CustomAccount”.
+"CustomAccount".
 
-Set account credentials for multiple accounts by editing conf/authdata.json
+You can set credentials for many accounts by editing
-(see below for further details). To specify one set for personal use, set these 
+``conf/authdata.json`` (see below for further info), but if you just
-environment variables:
+want to specify one set of your own, you can use these environment
+variables.
 
-.. code-block:: shell
+.. code:: shell
 
-   $ docker run -d --name cloudserver -p 8000:8000 -e SCALITY_ACCESS_KEY_ID=newAccessKey \
+    docker run -d --name s3server -p 8000:8000 -e SCALITY_ACCESS_KEY_ID=newAccessKey
-   -e SCALITY_SECRET_ACCESS_KEY=newSecretKey zenko/cloudserver
+    -e SCALITY_SECRET_ACCESS_KEY=newSecretKey scality/s3server
 
-.. note:: This takes precedence over the contents of the authdata.json 
+Note: Anything in the ``authdata.json`` file will be ignored. Note: The
-	  file. The authdata.json file is ignored. 
+old ``ACCESS_KEY`` and ``SECRET_KEY`` environment variables are now
-
+deprecated
-.. note:: The ACCESS_KEY and SECRET_KEY environment variables are 
-	  deprecated.
 
 LOG\_LEVEL
 ~~~~~~~~~~
 
-This variable changes the log level. There are three levels: info, debug, 
+This variable allows you to change the log level: info, debug or trace.
-and trace. The default is info. Debug provides more detailed logs, and trace
+The default is info. Debug will give you more detailed logs and trace
-provides the most detailed logs.
+will give you the most detailed.
 
-.. code-block:: shell
+.. code:: shell
 
-    $ docker run -d --name cloudserver -p 8000:8000 -e LOG_LEVEL=trace zenko/cloudserver
+    docker run -d --name s3server -p 8000:8000 -e LOG_LEVEL=trace scality/s3server
 
 SSL
 ~~~
 
-Set true, this variable runs CloudServer with SSL.
+This variable set to true allows you to run S3 with SSL:
 
-If SSL is set true: 
+**Note1**: You also need to specify the ENDPOINT environment variable.
+**Note2**: In your ``/etc/hosts`` file on Linux, OS X, or Unix with root
+permissions, make sure to associate 127.0.0.1 with ``<YOUR_ENDPOINT>``
 
-* The ENDPOINT environment variable must also be specified.
+**Warning**: These certs, being self-signed (and the CA being generated
-
+inside the container) will be untrusted by any clients, and could
-* On Unix-like systems (Linux, OS X, etc.), 127.0.0.1 must be associated with
+disappear on a container upgrade. That's ok as long as it's for quick
-  <YOUR_ENDPOINT> in /etc/hosts.
+testing. Also, best security practice for non-testing would be to use an
-
+extra container to do SSL/TLS termination such as haproxy/nginx/stunnel
-   .. Warning:: Self-signed certs with a CA generated within the container are 
+to limit what an exploit on either component could expose, as well as
-      suitable for testing purposes only. Clients cannot trust them, and they may
+certificates in a mounted volume
-      disappear altogether on a container upgrade. The best security practice for 
-      production environments is to use an extra container, such as 
-      haproxy/nginx/stunnel, for SSL/TLS termination and to pull certificates
-      from a mounted volume, limiting what an exploit on either component
-      can expose. 
 
 .. code:: shell
 
-     $ docker run -d --name cloudserver -p 8000:8000 -e SSL=TRUE -e ENDPOINT=<YOUR_ENDPOINT> \
+    docker run -d --name s3server -p 8000:8000 -e SSL=TRUE -e ENDPOINT=<YOUR_ENDPOINT>
-     zenko/cloudserver
+    scality/s3server
 
-  For more information about using ClousdServer with SSL, see `Using SSL <GETTING_STARTED.html#Using SSL>`__
+More information about how to use S3 server with SSL
+`here <https://s3.scality.com/v1.0/page/scality-with-ssl>`__
 
 LISTEN\_ADDR
 ~~~~~~~~~~~~
 
-This variable causes CloudServer and its data and metadata components to 
+This variable instructs the Zenko CloudServer, and its data and metadata
-listen on the specified address. This allows starting the data or metadata 
+components to listen on the specified address. This allows starting the data
-servers as standalone services, for example.
+or metadata servers as standalone services, for example.
 
 .. code:: shell
 
     docker run -d --name s3server-data -p 9991:9991 -e LISTEN_ADDR=0.0.0.0
-    scality/s3server yarn run start_dataserver
+    scality/s3server npm run start_dataserver
 
 
 DATA\_HOST and METADATA\_HOST
@@ -185,8 +174,8 @@ Zenko CloudServer.
 
 .. code:: shell
 
-    $ docker run -d --name cloudserver -e DATA_HOST=cloudserver-data \
+    docker run -d --name s3server -e DATA_HOST=s3server-data
-    -e METADATA_HOST=cloudserver-metadata zenko/cloudserver yarn run start_s3server
+    -e METADATA_HOST=s3server-metadata scality/s3server npm run start_s3server
 
 REDIS\_HOST
 ~~~~~~~~~~~
@@ -196,21 +185,19 @@ localhost.
 
 .. code:: shell
 
-    $ docker run -d --name cloudserver -p 8000:8000 \
+    docker run -d --name s3server -p 8000:8000
-    -e REDIS_HOST=my-redis-server.example.com zenko/cloudserver
+    -e REDIS_HOST=my-redis-server.example.com scality/s3server
 
 REDIS\_PORT
 ~~~~~~~~~~~
 
-Use this variable to connect to the Redis cache server on a port other 
+Use this variable to connect to the redis cache server on another port than
-than the default 6379.
+the default 6379.
 
 .. code:: shell
 
-    $ docker run -d --name cloudserver -p 8000:8000 \
+    docker run -d --name s3server -p 8000:8000
-    -e REDIS_PORT=6379 zenko/cloudserver
+    -e REDIS_PORT=6379 scality/s3server
-
-.. _tunables-and-setup-tips:
 
 Tunables and Setup Tips
 -----------------------
@@ -218,57 +205,60 @@ Tunables and Setup Tips
 Using Docker Volumes
 ~~~~~~~~~~~~~~~~~~~~
 
-CloudServer runs with a file backend by default, meaning that data is 
+Zenko CloudServer runs with a file backend by default.
-stored inside the CloudServer’s Docker container.
 
-For data and metadata to persist, data and metadata must be hosted in Docker 
+So, by default, the data is stored inside your Zenko CloudServer Docker
-volumes outside the CloudServer’s Docker container. Otherwise, the data
+container.
-and metadata are destroyed when the container is erased.
 
-.. code-block:: shell
+However, if you want your data and metadata to persist, you **MUST** use
+Docker volumes to host your data and metadata outside your Zenko CloudServer
+Docker container. Otherwise, the data and metadata will be destroyed
+when you erase the container.
 
-    $ docker run -­v $(pwd)/data:/usr/src/app/localData -­v $(pwd)/metadata:/usr/src/app/localMetadata \
+.. code:: shell
-    -p 8000:8000 ­-d zenko/cloudserver
 
-This command mounts the ./data host directory to the container
+    docker run -­v $(pwd)/data:/usr/src/app/localData -­v $(pwd)/metadata:/usr/src/app/localMetadata
-at /usr/src/app/localData and the ./metadata host directory to
+    -p 8000:8000 ­-d scality/s3server
-the container at /usr/src/app/localMetaData. 
 
-.. tip:: These host directories can be mounted to any accessible mount 
+This command mounts the host directory, ``./data``, into the container
-   point, such as /mnt/data and /mnt/metadata, for example.
+at ``/usr/src/app/localData`` and the host directory, ``./metadata``, into
+the container at ``/usr/src/app/localMetaData``. It can also be any host
+mount point, like ``/mnt/data`` and ``/mnt/metadata``.
 
-Adding, Modifying, or Deleting Accounts or Credentials
+Adding modifying or deleting accounts or users credentials
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-1. Create a customized authdata.json file locally based on /conf/authdata.json.
+1. Create locally a customized ``authdata.json`` based on our ``/conf/authdata.json``.
-
-2. Use `Docker volumes <https://docs.docker.com/storage/volumes/>`__
-   to override the default ``authdata.json`` through a Docker file mapping.
 
+2. Use `Docker
+   Volume <https://docs.docker.com/engine/tutorials/dockervolumes/>`__
+   to override the default ``authdata.json`` through a docker file mapping.
 For example:
 
-.. code-block:: shell
+.. code:: shell
 
-    $ docker run -v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json -p 8000:8000 -d \
+    docker run -v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json -p 8000:8000 -d
-    zenko/cloudserver
+    scality/s3server
 
-Specifying a Host Name
+Specifying your own host name
-~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-To specify a host name (for example, s3.domain.name), provide your own
+To specify a host name (e.g. s3.domain.name), you can provide your own
-`config.json <https://github.com/scality/cloudserver/blob/master/config.json>`__
+`config.json <https://github.com/scality/S3/blob/master/config.json>`__
-file using `Docker volumes <https://docs.docker.com/storage/volumes/>`__.
+using `Docker
+Volume <https://docs.docker.com/engine/tutorials/dockervolumes/>`__.
 
-First, add a new key-value pair to the restEndpoints section of your
+First add a new key-value pair in the restEndpoints section of your
-config.json. Make the key the host name you want, and the value the default 
+config.json. The key in the key-value pair should be the host name you
-location\_constraint for this endpoint.
+would like to add and the value is the default location\_constraint for
+this endpoint.
 
 For example, ``s3.example.com`` is mapped to ``us-east-1`` which is one
 of the ``location_constraints`` listed in your locationConfig.json file
 `here <https://github.com/scality/S3/blob/master/locationConfig.json>`__.
 
-For more information about location configuration, see:
+More information about location configuration
-`GETTING STARTED <GETTING_STARTED.html#location-configuration>`__
+`here <https://github.com/scality/S3/blob/master/README.md#location-configuration>`__
 
 .. code:: json
 
@@ -276,33 +266,33 @@ For more information about location configuration, see:
         "localhost": "file",
         "127.0.0.1": "file",
         ...
-        "cloudserver.example.com": "us-east-1"
+        "s3.example.com": "us-east-1"
     },
 
-Next, run CloudServer using a `Docker volume 
+Then, run your Scality S3 Server using `Docker
-<https://docs.docker.com/engine/tutorials/dockervolumes/>`__:
+Volume <https://docs.docker.com/engine/tutorials/dockervolumes/>`__:
 
-.. code-block:: shell
+.. code:: shell
 
-    $ docker run -v $(pwd)/config.json:/usr/src/app/config.json -p 8000:8000 -d zenko/cloudserver
+    docker run -v $(pwd)/config.json:/usr/src/app/config.json -p 8000:8000 -d scality/s3server
 
-The local ``config.json`` file overrides the default one through a Docker 
+Your local ``config.json`` file will override the default one through a
-file mapping.
+docker file mapping.
 
-Running as an Unprivileged User
+Running as an unprivileged user
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-CloudServer runs as root by default.
+Zenko CloudServer runs as root by default.
 
-To change this, modify the dockerfile and specify a user before the 
+You can change that by modifing the dockerfile and specifying a user
-entry point.
+before the entrypoint.
 
-The user must exist within the container, and must own the 
+The user needs to exist within the container, and own the folder
-/usr/src/app directory for CloudServer to run.
+**/usr/src/app** for Scality Zenko CloudServer to run properly.
 
-For example, the following dockerfile lines can be modified:
+For instance, you can modify these lines in the dockerfile:
 
-.. code-block:: shell
+.. code:: shell
 
     ...
     && groupadd -r -g 1001 scality \
@@ -314,58 +304,54 @@ For example, the following dockerfile lines can be modified:
     USER scality
     ENTRYPOINT ["/usr/src/app/docker-entrypoint.sh"]
 
-.. _continuous-integration-with-docker-hosted-cloudserver:
+Continuous integration with Docker hosted CloudServer
+-----------------------------------------------------
 
-Continuous Integration with a Docker-Hosted CloudServer
+When you start the Docker Scality Zenko CloudServer image, you can adjust the
--------------------------------------------------------
+configuration of the Scality Zenko CloudServer instance by passing one or more
+environment variables on the docker run command line.
 
-When you start the Docker CloudServer image, you can adjust the
+Sample ways to run it for CI are:
-configuration of the CloudServer instance by passing one or more
-environment variables on the ``docker run`` command line.
 
+- With custom locations (one in-memory, one hosted on AWS), and custom
+  credentials mounted:
 
-To run CloudServer for CI with custom locations (one in-memory, 
+.. code:: shell
-one hosted on AWS), and custom credentials mounted:
 
-.. code-block:: shell
+    docker run --name CloudServer -p 8000:8000
+    -v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json
+    -v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json
+    -v ~/.aws/credentials:/root/.aws/credentials
+    -e S3DATA=multiple -e S3BACKEND=mem scality/s3server
 
-   $ docker run --name CloudServer -p 8000:8000 \
+- With custom locations, (one in-memory, one hosted on AWS, one file),
-   -v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json \
+  and custom credentials set as environment variables
-   -v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json \
+  (see `this section <#scality-access-key-id-and-scality-secret-access-key>`__):
-   -v ~/.aws/credentials:/root/.aws/credentials \
-   -e S3DATA=multiple -e S3BACKEND=mem zenko/cloudserver
 
-To run CloudServer for CI with custom locations, (one in-memory, one
+.. code:: shell
-hosted on AWS, and one file), and custom credentials `set as environment 
-variables <GETTING_STARTED.html#scality-access-key-id-and-scality-secret-access-key>`__):
 
-.. code-block:: shell
+    docker run --name CloudServer -p 8000:8000
+    -v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json
+    -v ~/.aws/credentials:/root/.aws/credentials
+    -v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata
+    -e SCALITY_ACCESS_KEY_ID=accessKey1
+    -e SCALITY_SECRET_ACCESS_KEY=verySecretKey1
+    -e S3DATA=multiple -e S3BACKEND=mem scality/s3server
 
-   $ docker run --name CloudServer -p 8000:8000 \
+In production with Docker hosted CloudServer
-   -v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json \
+--------------------------------------------
-   -v ~/.aws/credentials:/root/.aws/credentials \
-   -v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata \
-   -e SCALITY_ACCESS_KEY_ID=accessKey1 \
-   -e SCALITY_SECRET_ACCESS_KEY=verySecretKey1 \
-   -e S3DATA=multiple -e S3BACKEND=mem zenko/cloudserver
 
-.. _in-production-w-a-Docker-hosted-cloudserver:
+In production, we expect that data will be persistent, that you will use the
+multiple backends capabilities of Zenko CloudServer, and that you will have a
+custom endpoint for your local storage, and custom credentials for your local
+storage:
 
-In Production with a Docker-Hosted CloudServer
+.. code:: shell
-----------------------------------------------
 
-Because data must persist in production settings, CloudServer offers
+    docker run -d --name CloudServer
-multiple-backend capabilities. This requires a custom endpoint 
+    -v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata
-and custom credentials for local storage.
+    -v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json
-
+    -v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json
-Customize these with:
+    -v ~/.aws/credentials:/root/.aws/credentials -e S3DATA=multiple
-
+    -e ENDPOINT=custom.endpoint.com
-.. code-block:: shell
+    -p 8000:8000 ­-d scality/s3server
-
-   $ docker run -d --name CloudServer \
-   -v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata \
-   -v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json \
-   -v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json \
-   -v ~/.aws/credentials:/root/.aws/credentials -e S3DATA=multiple \
-   -e ENDPOINT=custom.endpoint.com \
-   -p 8000:8000 ­-d zenko/cloudserver \

docs/GETTING_STARTED.rst

@@ -1,201 +1,193 @@
 Getting Started
-===============
+=================
 
 .. figure:: ../res/scality-cloudserver-logo.png
    :alt: Zenko CloudServer logo
 
-
+|CircleCI| |Scality CI|
-Dependencies
-------------
-
-Building and running the Scality Zenko CloudServer requires node.js 10.x and
-yarn v1.17.x. Up-to-date versions can be found at
-`Nodesource <https://github.com/nodesource/distributions>`__.
 
 Installation
 ------------
 
-1. Clone the source code
+Dependencies
+~~~~~~~~~~~~
 
-   .. code-block:: shell
+Building and running the Scality Zenko CloudServer requires node.js 6.9.5 and
+npm v3 . Up-to-date versions can be found at
+`Nodesource <https://github.com/nodesource/distributions>`__.
 
-      $ git clone https://github.com/scality/cloudserver.git
+Clone source code
-
+~~~~~~~~~~~~~~~~~
-2. Go to the cloudserver directory and use yarn to install the js dependencies.
-
-   .. code-block:: shell
-
-      $ cd cloudserver
-      $ yarn install
-
-Running CloudServer with a File Backend
----------------------------------------
-
-.. code-block:: shell
-
-   $ yarn start
-
-This starts a Zenko CloudServer on port 8000. Two additional ports, 9990 
-and 9991, are also open locally for internal transfer of metadata and 
-data, respectively.
-
-The default access key is accessKey1. The secret key is verySecretKey1.
-
-By default, metadata files are saved in the localMetadata directory and 
-data files are saved in the localData directory in the local ./cloudserver 
-directory. These directories are pre-created within the repository. To 
-save data or metadata in different locations, you must specify them using 
-absolute paths. Thus, when starting the server:
-
-.. code-block:: shell
-
-   $ mkdir -m 700 $(pwd)/myFavoriteDataPath
-   $ mkdir -m 700 $(pwd)/myFavoriteMetadataPath
-   $ export S3DATAPATH="$(pwd)/myFavoriteDataPath"
-   $ export S3METADATAPATH="$(pwd)/myFavoriteMetadataPath"
-   $ yarn start
-
-Running CloudServer with Multiple Data Backends
------------------------------------------------
-
-.. code-block:: shell
-
-   $ export S3DATA='multiple'
-   $ yarn start
-
-This starts a Zenko CloudServer on port 8000. 
-
-The default access key is accessKey1. The secret key is verySecretKey1.
-
-With multiple backends, you can choose where each object is saved by setting
-the following header with a location constraint in a PUT request:
-
-.. code-block:: shell
-
-    'x-amz-meta-scal-location-constraint':'myLocationConstraint'
-
-If no header is sent with a PUT object request, the bucket’s location
-constraint determines where the data is saved. If the bucket has no
-location constraint, the endpoint of the PUT request determines location.
-
-See the Configuration_ section to set location constraints.
-
-Run CloudServer with an In-Memory Backend
------------------------------------------
-
-.. code-block:: shell
-
-   $ yarn run mem_backend
-
-This starts a Zenko CloudServer on port 8000. 
-
-The default access key is accessKey1. The secret key is verySecretKey1.
-
-Run CloudServer with Vault User Management
-------------------------------------------
 
 .. code:: shell
 
-   export S3VAULT=vault
+    git clone https://github.com/scality/S3.git
-   yarn start
 
-Note: Vault is proprietary and must be accessed separately.
+Install js dependencies
-This starts a Zenko CloudServer using Vault for user management.
+~~~~~~~~~~~~~~~~~~~~~~~
 
-Run CloudServer for Continuous Integration Testing or in Production with Docker
+Go to the ./S3 folder,
--------------------------------------------------------------------------------
 
-Run Cloudserver with `DOCKER <DOCKER.html>`__
+.. code:: shell
 
-Testing
+    npm install
-~~~~~~~
 
-Run unit tests with the command:
+Run it with a file backend
+--------------------------
 
-.. code-block:: shell
+.. code:: shell
 
-   $ yarn test
+    npm start
 
-Run multiple-backend unit tests with:
+This starts an Zenko CloudServer on port 8000. Two additional ports 9990 and
+9991 are also open locally for internal transfer of metadata and data,
+respectively.
 
-.. code-block:: shell
+The default access key is accessKey1 with a secret key of
+verySecretKey1.
 
-   $ CI=true S3DATA=multiple yarn start
+By default the metadata files will be saved in the localMetadata
-   $ yarn run multiple_backend_test
+directory and the data files will be saved in the localData directory
+within the ./S3 directory on your machine. These directories have been
+pre-created within the repository. If you would like to save the data or
+metadata in different locations of your choice, you must specify them
+with absolute paths. So, when starting the server:
 
-Run the linter with:
+.. code:: shell
 
-.. code-block:: shell
+    mkdir -m 700 $(pwd)/myFavoriteDataPath
+    mkdir -m 700 $(pwd)/myFavoriteMetadataPath
+    export S3DATAPATH="$(pwd)/myFavoriteDataPath"
+    export S3METADATAPATH="$(pwd)/myFavoriteMetadataPath"
+    npm start
 
-   $ yarn run lint
+Run it with multiple data backends
+----------------------------------
 
-Running Functional Tests Locally
+.. code:: shell
+
+    export S3DATA='multiple'
+    npm start
+
+This starts an Zenko CloudServer on port 8000. The default access key is
+accessKey1 with a secret key of verySecretKey1.
+
+With multiple backends, you have the ability to choose where each object
+will be saved by setting the following header with a locationConstraint
+on a PUT request:
+
+.. code:: shell
+
+    'x-amz-meta-scal-location-constraint':'myLocationConstraint'
+
+If no header is sent with a PUT object request, the location constraint
+of the bucket will determine where the data is saved. If the bucket has
+no location constraint, the endpoint of the PUT request will be used to
+determine location.
+
+See the Configuration section below to learn how to set location
+constraints.
+
+Run it with an in-memory backend
 --------------------------------
 
-To pass AWS and Azure backend tests locally, modify 
+.. code:: shell
-tests/locationConfig/locationConfigTests.json so that ``awsbackend`` 
+
-specifies the bucketname of a bucket you have access to based on your
+    npm run mem_backend
-credentials, and modify ``azurebackend`` with details for your Azure account.
+
+This starts an Zenko CloudServer on port 8000. The default access key is
+accessKey1 with a secret key of verySecretKey1.
+
+Run it for continuous integration testing or in production with Docker
+----------------------------------------------------------------------
+
+`DOCKER <../DOCKER/>`__
+
+Testing
+-------
+
+You can run the unit tests with the following command:
+
+.. code:: shell
+
+    npm test
+
+You can run the multiple backend unit tests with:
+
+.. code:: shell
+    CI=true S3DATA=multiple npm start
+    npm run multiple_backend_test
+
+You can run the linter with:
+
+.. code:: shell
+
+    npm run lint
+
+Running functional tests locally:
+
+For the AWS backend and Azure backend tests to pass locally,
+you must modify tests/locationConfigTests.json so that awsbackend
+specifies a bucketname of a bucket you have access to based on
+your credentials profile and modify "azurebackend" with details
+for your Azure account.
 
 The test suite requires additional tools, **s3cmd** and **Redis**
 installed in the environment the tests are running in.
 
-1. Install `s3cmd <http://s3tools.org/download>`__
+-  Install `s3cmd <http://s3tools.org/download>`__
+-  Install `redis <https://redis.io/download>`__ and start Redis.
+-  Add localCache section to your ``config.json``:
 
-2. Install `redis <https://redis.io/download>`__ and start Redis.
+::
-
-3. Add localCache section to ``config.json``:
-
-   .. code:: json
 
     "localCache": {
         "host": REDIS_HOST,
         "port": REDIS_PORT
     }
 
-   where ``REDIS_HOST`` is the Redis instance IP address (``"127.0.0.1"``
+where ``REDIS_HOST`` is your Redis instance IP address (``"127.0.0.1"``
-   if Redis is running locally) and ``REDIS_PORT`` is the Redis instance
+if your Redis is running locally) and ``REDIS_PORT`` is your Redis
-   port (``6379`` by default)
+instance port (``6379`` by default)
 
-4. Add the following to the local etc/hosts file:
+-  Add the following to the etc/hosts file on your machine:
 
-   .. code-block:: shell
+.. code:: shell
 
     127.0.0.1 bucketwebsitetester.s3-website-us-east-1.amazonaws.com
 
-5. Start Zenko CloudServer in memory and run the functional tests:
+-  Start the Zenko CloudServer in memory and run the functional tests:
 
-   .. code-block:: shell
+.. code:: shell
 
-      $ CI=true yarn run mem_backend
+    CI=true npm run mem_backend
-      $ CI=true yarn run ft_test
+    CI=true npm run ft_test
-
-.. _Configuration:
 
 Configuration
 -------------
 
-There are three configuration files for Zenko CloudServer:
+There are three configuration files for your Scality Zenko CloudServer:
 
-* ``conf/authdata.json``, for authentication.
+1. ``conf/authdata.json``, described above for authentication
 
-* ``locationConfig.json``, to configure where data is saved.
+2. ``locationConfig.json``, to set up configuration options for
 
-* ``config.json``, for general configuration options.
+   where data will be saved
 
-.. _location-configuration:
+3. ``config.json``, for general configuration options
 
 Location Configuration
 ~~~~~~~~~~~~~~~~~~~~~~
 
-You must specify at least one locationConstraint in locationConfig.json
+You must specify at least one locationConstraint in your
-(or leave it as pre-configured).
+locationConfig.json (or leave as pre-configured).
 
-You must also specify 'us-east-1' as a locationConstraint. If you put a 
+You must also specify 'us-east-1' as a locationConstraint so if you only
-bucket to an unknown endpoint and do not specify a locationConstraint in
+define one locationConstraint, that would be it. If you put a bucket to
-the PUT bucket call, us-east-1 is used.
+an unknown endpoint and do not specify a locationConstraint in the put
+bucket call, us-east-1 will be used.
 
-For instance, the following locationConstraint saves data sent to
+For instance, the following locationConstraint will save data sent to
 ``myLocationConstraint`` to the file backend:
 
 .. code:: json
@@ -206,16 +198,17 @@ For instance, the following locationConstraint saves data sent to
         "details": {}
     },
 
-Each locationConstraint must include the ``type``, ``legacyAwsBehavior``,
+Each locationConstraint must include the ``type``,
-and ``details`` keys. ``type`` indicates which backend is used for that
+``legacyAwsBehavior``, and ``details`` keys. ``type`` indicates which
-region. Supported backends are mem, file, and scality.``legacyAwsBehavior``
+backend will be used for that region. Currently, mem, file, and scality
-indicates whether the region behaves the same as the AWS S3 'us-east-1' 
+are the supported backends. ``legacyAwsBehavior`` indicates whether the
-region. If the locationConstraint type is ``scality``, ``details`` must 
+region will have the same behavior as the AWS S3 'us-east-1' region. If
-contain connector information for sproxyd. If the locationConstraint type
+the locationConstraint type is scality, ``details`` should contain
-is ``mem`` or ``file``, ``details`` must be empty.
+connector information for sproxyd. If the locationConstraint type is mem
+or file, ``details`` should be empty.
 
-Once locationConstraints is set in locationConfig.json, specify a default
+Once you have your locationConstraints in your locationConfig.json, you
-locationConstraint for each endpoint.
+can specify a default locationConstraint for each of your endpoints.
 
 For instance, the following sets the ``localhost`` endpoint to the
 ``myLocationConstraint`` data backend defined above:
@@ -226,24 +219,26 @@ For instance, the following sets the ``localhost`` endpoint to the
          "localhost": "myLocationConstraint"
     },
 
-To use an endpoint other than localhost for Zenko CloudServer, the endpoint
+If you would like to use an endpoint other than localhost for your
-must be listed in ``restEndpoints``. Otherwise, if the server is running
+Scality Zenko CloudServer, that endpoint MUST be listed in your
-with a:
+``restEndpoints``. Otherwise if your server is running with a:
 
-*  **file backend**: The default location constraint is ``file``
+-  **file backend**: your default location constraint will be ``file``
-*  **memory backend**: The default location constraint is ``mem``
+
+-  **memory backend**: your default location constraint will be ``mem``
 
 Endpoints
 ~~~~~~~~~
 
-The Zenko CloudServer supports endpoints that are rendered in either:
+Note that our Zenko CloudServer supports both:
 
-* path style: http://myhostname.com/mybucket or
+-  path-style: http://myhostname.com/mybucket
-* hosted style: http://mybucket.myhostname.com
+-  hosted-style: http://mybucket.myhostname.com
 
-However, if an IP address is specified for the host, hosted-style requests
+However, hosted-style requests will not hit the server if you are using
-cannot reach the server. Use path-style requests in that case. For example,
+an ip address for your host. So, make sure you are using path-style
-if you are using the AWS SDK for JavaScript, instantiate your client like this:
+requests in that case. For instance, if you are using the AWS SDK for
+JavaScript, you would instantiate your client like this:
 
 .. code:: js
 
@@ -252,97 +247,85 @@ if you are using the AWS SDK for JavaScript, instantiate your client like this:
        s3ForcePathStyle: true,
     });
 
-Setting Your Own Access and Secret Key Pairs
+Setting your own access key and secret key pairs
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Credentials can be set for many accounts by editing ``conf/authdata.json``, 
+You can set credentials for many accounts by editing
-but use the ``SCALITY_ACCESS_KEY_ID`` and ``SCALITY_SECRET_ACCESS_KEY`` 
+``conf/authdata.json`` but if you want to specify one set of your own
-environment variables to specify your own credentials.
+credentials, you can use ``SCALITY_ACCESS_KEY_ID`` and
-
+``SCALITY_SECRET_ACCESS_KEY`` environment variables.
-_`scality-access-key-id-and-scality-secret-access-key`
 
 SCALITY\_ACCESS\_KEY\_ID and SCALITY\_SECRET\_ACCESS\_KEY
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 These variables specify authentication credentials for an account named
-“CustomAccount”.
+"CustomAccount".
 
-.. note:: Anything in the ``authdata.json`` file is ignored.
+Note: Anything in the ``authdata.json`` file will be ignored.
 
-.. code-block:: shell
+.. code:: shell
 
-   $ SCALITY_ACCESS_KEY_ID=newAccessKey SCALITY_SECRET_ACCESS_KEY=newSecretKey yarn start
+    SCALITY_ACCESS_KEY_ID=newAccessKey SCALITY_SECRET_ACCESS_KEY=newSecretKey npm start
 
-.. _Using_SSL:
 
-Using SSL
+Scality with SSL
-~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~
 
-To use https with your local CloudServer, you must set up
+If you wish to use https with your local Zenko CloudServer, you need to set up
-SSL certificates. 
+SSL certificates. Here is a simple guide of how to do it.
 
-1. Deploy CloudServer using `our DockerHub page
+Deploying Zenko CloudServer
-   <https://hub.docker.com/r/zenko/cloudserver/>`__ (run it with a file
+^^^^^^^^^^^^^^^^^^^
-   backend).
 
-   .. Note:: If Docker is not installed locally, follow the
+First, you need to deploy **Zenko CloudServer**. This can be done very easily
-      `instructions to install it for your distribution 
+via `our **DockerHub**
-      <https://docs.docker.com/engine/installation/>`__
+page <https://hub.docker.com/r/scality/s3server/>`__ (you want to run it
+with a file backend).
 
-2. Update the CloudServer container’s config 
+    *Note:* *- If you don't have docker installed on your machine, here
+    are the `instructions to install it for your
+    distribution <https://docs.docker.com/engine/installation/>`__*
 
-   Add your certificates to your container. To do this, 
+Updating your Zenko CloudServer container's config
-   #. exec inside the CloudServer container. 
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-   #. Run ``$> docker ps`` to find the container’s ID (the corresponding 
+You're going to add your certificates to your container. In order to do
-      image name is ``scality/cloudserver``. 
+so, you need to exec inside your Zenko CloudServer container. Run a
+``$> docker ps`` and find your container's id (the corresponding image
+name should be ``scality/s3server``. Copy the corresponding container id
+(here we'll use ``894aee038c5e``, and run:
 
-   #. Copy the corresponding container ID (``894aee038c5e`` in the present
+.. code:: sh
-      example), and run:
-
-      .. code-block:: shell
 
     $> docker exec -it 894aee038c5e bash
 
-      This puts you inside your container, using an interactive terminal.
+You're now inside your container, using an interactive terminal :)
 
-3. Generate the SSL key and certificates. The paths where the different
+Generate SSL key and certificates
-   files are stored are defined after the ``-out`` option in each of the 
+**********************************
-   following commands.
 
-    #. Generate a private key for your certificate signing request (CSR):
+There are 5 steps to this generation. The paths where the different
+files are stored are defined after the ``-out`` option in each command
 
-       .. code-block:: shell
+.. code:: sh
 
+    # Generate a private key for your CSR
     $> openssl genrsa -out ca.key 2048
-
+    # Generate a self signed certificate for your local Certificate Authority
-    #. Generate a self-signed certificate for your local certificate 
-       authority (CA):
-
-       .. code:: shell
-
     $> openssl req -new -x509 -extensions v3_ca -key ca.key -out ca.crt -days 99999  -subj "/C=US/ST=Country/L=City/O=Organization/CN=scality.test"
 
-    #. Generate a key for the CloudServer:
+    # Generate a key for Zenko CloudServer
-
-       .. code:: shell
-
     $> openssl genrsa -out test.key 2048
-
+    # Generate a Certificate Signing Request for S3 Server
-    #. Generate a CSR for CloudServer:
-
-       .. code:: shell
-
     $> openssl req -new -key test.key -out test.csr -subj "/C=US/ST=Country/L=City/O=Organization/CN=*.scality.test"
-
+    # Generate a local-CA-signed certificate for S3 Server
-    #. Generate a certificate for CloudServer signed by the local CA:
-
-       .. code:: shell
-
     $> openssl x509 -req -in test.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out test.crt -days 99999 -sha256
 
-4. Update Zenko CloudServer ``config.json``. Add a ``certFilePaths`` 
+Update Zenko CloudServer ``config.json``
-   section to ``./config.json`` with appropriate paths:
+**********************************
+
+Add a ``certFilePaths`` section to ``./config.json`` with the
+appropriate paths:
 
 .. code:: json
 
@@ -352,36 +335,42 @@ SSL certificates.
             "ca": "./ca.crt"
         }
 
-5. Run your container with the new config. 
+Run your container with the new config
+****************************************
 
-   #. Exit the container by running ``$> exit``. 
+First, you need to exit your container. Simply run ``$> exit``. Then,
+you need to restart your container. Normally, a simple
+``$> docker restart s3server`` should do the trick.
 
-   #. Restart the container with ``$> docker restart cloudserver``.
+Update your host config
+^^^^^^^^^^^^^^^^^^^^^^^
 
-6. Update the host configuration by adding s3.scality.test 
+Associates local IP addresses with hostname
-   to /etc/hosts:
+*******************************************
 
-   .. code:: bash
+In your ``/etc/hosts`` file on Linux, OS X, or Unix (with root
+permissions), edit the line of localhost so it looks like this:
+
+::
 
     127.0.0.1      localhost s3.scality.test
 
-7. Copy the local certificate authority (ca.crt in step 4) from your 
+Copy the local certificate authority from your container
-   container. Choose the path to save this file to (in the present 
+*********************************************************
-   example, ``/root/ca.crt``), and run:
 
-   .. code:: shell
+In the above commands, it's the file named ``ca.crt``. Choose the path
+you want to save this file at (here we chose ``/root/ca.crt``), and run
+something like:
+
+.. code:: sh
 
     $> docker cp 894aee038c5e:/usr/src/app/ca.crt /root/ca.crt
 
-   .. note:: Your container ID will be different, and your path to 
+Test your config
-	     ca.crt may be different.
+^^^^^^^^^^^^^^^^^
 
-Test the Config
+If you do not have aws-sdk installed, run ``$> npm install aws-sdk``. In
-^^^^^^^^^^^^^^^
+a ``test.js`` file, paste the following script:
-
-If aws-sdk is not installed, run ``$> yarn install aws-sdk``. 
-
-Paste the following script into a file named "test.js":
 
 .. code:: js
 
@@ -421,13 +410,8 @@ Paste the following script into a file named "test.js":
         });
     });
 
-Now run this script with:
+Now run that script with ``$> nodejs test.js``. If all goes well, it
-
+should output ``SSL is cool!``. Enjoy that added security!
-.. code::
-
-   $> nodejs test.js
- 
-On success, the script outputs ``SSL is cool!``.
 
 
 .. |CircleCI| image:: https://circleci.com/gh/scality/S3.svg?style=svg

docs/Healthchecks.md

@@ -1,7 +1,6 @@
 # S3 Healthcheck
 
-Scality S3 exposes a healthcheck route `/live` on the port used
+Scality S3 exposes a healthcheck route `/_/healthcheck` which returns a
-for the metrics (defaults to port 8002) which returns a
 response with HTTP code
 
 - 200 OK

docs/INTEGRATIONS.rst

@@ -4,32 +4,38 @@ Integrations
 High Availability
 =================
 
-`Docker Swarm <https://docs.docker.com/engine/swarm/>`__ is a clustering tool
+`Docker swarm <https://docs.docker.com/engine/swarm/>`__ is a
-developed by Docker for use with its containers. It can be used to start
+clustering tool developped by Docker and ready to use with its
-services, which we define to ensure CloudServer's continuous availability to
+containers. It allows to start a service, which we define and use as a
-end users. A swarm defines a manager and *n* workers among *n* + 1 servers.
+means to ensure Zenko CloudServer's continuous availability to the end user.
-
+Indeed, a swarm defines a manager and n workers among n+1 servers. We
-This tutorial shows how to perform a basic setup with three servers, which
+will do a basic setup in this tutorial, with just 3 servers, which
-provides strong service resiliency, while remaining easy to use and
+already provides a strong service resiliency, whilst remaining easy to
-maintain. We will use NFS through Docker to share data and
+do as an individual. We will use NFS through docker to share data and
 metadata between the different servers.
 
-Sections are labeled **On Server**, **On Clients**, or
+You will see that the steps of this tutorial are defined as **On
-**On All Machines**, referring respectively to NFS server, NFS clients, or
+Server**, **On Clients**, **On All Machines**. This refers respectively
-NFS server and clients. In the present example, the server’s IP address is
+to NFS Server, NFS Clients, or NFS Server and Clients. In our example,
-**10.200.15.113** and the client IP addresses are **10.200.15.96** and
+the IP of the Server will be **10.200.15.113**, while the IPs of the
-**10.200.15.97**
+Clients will be **10.200.15.96 and 10.200.15.97**
 
-1. Install Docker (on All Machines)
+Installing docker
+-----------------
 
-   Docker 17.03.0-ce is used for this tutorial. Docker 1.12.6 and later will
+Any version from docker 1.12.6 onwards should work; we used Docker
-   likely work, but is not tested.
+17.03.0-ce for this tutorial.
 
-   * On Ubuntu 14.04
+On All Machines
-     Install Docker CE for Ubuntu as `documented at Docker
+~~~~~~~~~~~~~~~
-     <https://docs.docker.com/install/linux/docker-ce/ubuntu/>`__.
+
-     Install the aufs dependency as recommended by Docker. The required
+On Ubuntu 14.04
-     commands are:
+^^^^^^^^^^^^^^^
+
+The docker website has `solid
+documentation <https://docs.docker.com/engine/installation/linux/ubuntu/>`__.
+We have chosen to install the aufs dependency, as recommended by Docker.
+Here are the required commands:
 
 .. code:: sh
 
@@ -41,10 +47,12 @@ NFS server and clients. In the present example, the server’s IP address is
     $> sudo apt-get update
     $> sudo apt-get install docker-ce
 
-    * On CentOS 7
+On CentOS 7
-      Install Docker CE as `documented at Docker
+^^^^^^^^^^^
-      <https://docs.docker.com/install/linux/docker-ce/centos/>`__.
+
-      The required commands are:
+The docker website has `solid
+documentation <https://docs.docker.com/engine/installation/linux/centos/>`__.
+Here are the required commands:
 
 .. code:: sh
 
@@ -54,22 +62,29 @@ NFS server and clients. In the present example, the server’s IP address is
     $> sudo yum install docker-ce
     $> sudo systemctl start docker
 
-2. Install NFS on Client(s)
+Configure NFS
+-------------
 
-   NFS clients mount Docker volumes over the NFS server’s shared folders.
+On Clients
-   If the NFS commons are installed, manual mounts are no longer needed.
+~~~~~~~~~~
 
-   * On Ubuntu 14.04
+Your NFS Clients will mount Docker volumes over your NFS Server's shared
+folders. Hence, you don't have to mount anything manually, you just have
+to install the NFS commons:
 
-     Install the NFS commons with apt-get:
+On Ubuntu 14.04
+^^^^^^^^^^^^^^^
+
+Simply install the NFS commons:
 
 .. code:: sh
 
     $> sudo apt-get install nfs-common
 
-   * On CentOS 7
+On CentOS 7
+^^^^^^^^^^^
 
-     Install the NFS utils; then start required services:
+Install the NFS utils, and then start the required services:
 
 .. code:: sh
 
@@ -83,22 +98,27 @@ NFS server and clients. In the present example, the server’s IP address is
     $> sudo systemctl start nfs-lock
     $> sudo systemctl start nfs-idmap
 
-3. Install NFS (on Server)
+On Server
+~~~~~~~~~
 
-   The NFS server hosts the data and metadata. The package(s) to install on it
+Your NFS Server will be the machine to physically host the data and
-   differs from the package installed on the clients.
+metadata. The package(s) we will install on it is slightly different
+from the one we installed on the clients.
 
-   * On Ubuntu 14.04
+On Ubuntu 14.04
+^^^^^^^^^^^^^^^
 
-     Install the NFS server-specific package and the NFS commons:
+Install the NFS server specific package and the NFS commons:
 
 .. code:: sh
 
     $> sudo apt-get install nfs-kernel-server nfs-common
 
-   * On CentOS 7
+On CentOS 7
+^^^^^^^^^^^
 
-     Install the NFS utils and start the required services:
+Same steps as with the client: install the NFS utils and start the
+required services:
 
 .. code:: sh
 
@@ -112,55 +132,74 @@ NFS server and clients. In the present example, the server’s IP address is
     $> sudo systemctl start nfs-lock
     $> sudo systemctl start nfs-idmap
 
-   For both distributions:
+On Ubuntu 14.04 and CentOS 7
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-   #. Choose where shared data and metadata from the local
+Choose where your shared data and metadata from your local `Zenko CloudServer
-      `CloudServer <http://www.zenko.io/cloudserver/>`__ shall be stored (The
+<http://www.zenko.io/cloudserver/>`__ will be stored.
-      present example uses /var/nfs/data and /var/nfs/metadata). Set permissions
+We chose to go with /var/nfs/data and /var/nfs/metadata. You also need
-      for these folders for
+to set proper sharing permissions for these folders as they'll be shared
-      sharing over NFS:
+over NFS:
 
 .. code:: sh
 
     $> mkdir -p /var/nfs/data /var/nfs/metadata
     $> chmod -R 777 /var/nfs/
 
-   #. The /etc/exports file configures network permissions and r-w-x permissions
+Now you need to update your **/etc/exports** file. This is the file that
-      for NFS access. Edit /etc/exports, adding the following lines:
+configures network permissions and rwx permissions for NFS access. By
+default, Ubuntu applies the no\_subtree\_check option, so we declared
+both folders with the same permissions, even though they're in the same
+tree:
+
+.. code:: sh
+
+    $> sudo vim /etc/exports
+
+In this file, add the following lines:
 
 .. code:: sh
 
     /var/nfs/data        10.200.15.96(rw,sync,no_root_squash) 10.200.15.97(rw,sync,no_root_squash)
     /var/nfs/metadata    10.200.15.96(rw,sync,no_root_squash) 10.200.15.97(rw,sync,no_root_squash)
 
-      Ubuntu applies the no\_subtree\_check option by default, so both
+Export this new NFS table:
-      folders are declared with the same permissions, even though they’re in
-      the same tree.
-
-   #. Export this new NFS table:
 
 .. code:: sh
 
     $> sudo exportfs -a
 
-   #. Edit the ``MountFlags`` option in the Docker config in
+Eventually, you need to allow for NFS mount from Docker volumes on other
-      /lib/systemd/system/docker.service to enable NFS mount from Docker volumes
+machines. You need to change the Docker config in
-      on other machines:
+**/lib/systemd/system/docker.service**:
+
+.. code:: sh
+
+    $> sudo vim /lib/systemd/system/docker.service
+
+In this file, change the **MountFlags** option:
 
 .. code:: sh
 
     MountFlags=shared
 
-   #. Restart the NFS server and Docker daemons to apply these changes.
+Now you just need to restart the NFS server and docker daemons so your
+changes apply.
 
-      * On Ubuntu 14.04
+On Ubuntu 14.04
+^^^^^^^^^^^^^^^
+
+Restart your NFS Server and docker services:
 
 .. code:: sh
 
     $> sudo service nfs-kernel-server restart
     $> sudo service docker restart
 
-      * On CentOS 7
+On CentOS 7
+^^^^^^^^^^^
+
+Restart your NFS Server and docker daemons:
 
 .. code:: sh
 
@@ -168,30 +207,34 @@ NFS server and clients. In the present example, the server’s IP address is
     $> sudo systemctl daemon-reload
     $> sudo systemctl restart docker
 
+Set up your Docker Swarm service
+--------------------------------
 
-4. Set Up a Docker Swarm
+On All Machines
+~~~~~~~~~~~~~~~
 
-  * On all machines and distributions:
+On Ubuntu 14.04 and CentOS 7
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-    Set up the Docker volumes to be mounted to the NFS server for CloudServer’s
+We will now set up the Docker volumes that will be mounted to the NFS
-    data and metadata storage. The following commands must be replicated on all
+Server and serve as data and metadata storage for Zenko CloudServer. These two
-    machines:
+commands have to be replicated on all machines:
 
 .. code:: sh
 
     $> docker volume create --driver local --opt type=nfs --opt o=addr=10.200.15.113,rw --opt device=:/var/nfs/data --name data
     $> docker volume create --driver local --opt type=nfs --opt o=addr=10.200.15.113,rw --opt device=:/var/nfs/metadata --name metadata
 
-    There is no need to ``docker exec`` these volumes to mount them: the
+There is no need to ""docker exec" these volumes to mount them: the
-    Docker Swarm manager does this when the Docker service is started.
+Docker Swarm manager will do it when the Docker service will be started.
 
-  * On a server:
+On Server
+^^^^^^^^^
 
-    To start a Docker service on a Docker Swarm cluster, initialize the cluster
+To start a Docker service on a Docker Swarm cluster, you first have to
-    (that is, define a manager), prompt workers/nodes to join in, and then start
+initialize that cluster (i.e.: define a manager), then have the
-    the service.
+workers/nodes join in, and then start the service. Initialize the swarm
-
+cluster, and look at the response:
-    Initialize the swarm cluster, and review its response:
 
 .. code:: sh
 
@@ -207,10 +250,11 @@ NFS server and clients. In the present example, the server’s IP address is
 
     To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
 
-  * On clients:
+On Clients
+^^^^^^^^^^
 
-    Copy and paste the command provided by your Docker Swarm init. A successful
+Simply copy/paste the command provided by your docker swarm init. When
-    request/response will resemble:
+all goes well, you'll get something like this:
 
 .. code:: sh
 
@@ -218,159 +262,182 @@ NFS server and clients. In the present example, the server’s IP address is
 
     This node joined a swarm as a worker.
 
-Set Up Docker Swarm on Clients on a Server
+On Server
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^
 
-Start the service on the Swarm cluster.
+Start the service on your swarm cluster!
 
 .. code:: sh
 
-  $> docker service create --name s3 --replicas 1 --mount type=volume,source=data,target=/usr/src/app/localData --mount type=volume,source=metadata,target=/usr/src/app/localMetadata -p 8000:8000 scality/cloudserver
+    $> docker service create --name s3 --replicas 1 --mount type=volume,source=data,target=/usr/src/app/localData --mount type=volume,source=metadata,target=/usr/src/app/localMetadata -p 8000:8000 scality/s3server
 
-On a successful installation, ``docker service ls`` returns the following
+If you run a docker service ls, you should have the following output:
-output:
 
 .. code:: sh
 
     $> docker service ls
     ID            NAME  MODE        REPLICAS  IMAGE
-    ocmggza412ft  s3    replicated  1/1       scality/cloudserver:latest
+    ocmggza412ft  s3    replicated  1/1       scality/s3server:latest
 
-If the service does not start, consider disabling apparmor/SELinux.
+If your service won't start, consider disabling apparmor/SELinux.
 
-Testing the High-Availability CloudServer
+Testing your High Availability S3Server
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+---------------------------------------
 
-On all machines (client/server) and distributions (Ubuntu and CentOS),
+On All Machines
-determine where CloudServer is running using ``docker ps``. CloudServer can
-operate on any node of the Swarm cluster, manager or worker. When you find
-it, you can kill it with ``docker stop <container id>``. It will respawn
-on a different node. Now, if one server falls, or if Docker stops
-unexpectedly, the end user will still be able to access your the local CloudServer.
-
-Troubleshooting
 ~~~~~~~~~~~~~~~
 
-To troubleshoot the service, run:
+On Ubuntu 14.04 and CentOS 7
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Try to find out where your Scality Zenko CloudServer is actually running using
+the **docker ps** command. It can be on any node of the swarm cluster,
+manager or worker. When you find it, you can kill it, with **docker stop
+<container id>** and you'll see it respawn on a different node of the
+swarm cluster. Now you see, if one of your servers falls, or if docker
+stops unexpectedly, your end user will still be able to access your
+local Zenko CloudServer.
+
+Troubleshooting
+---------------
+
+To troubleshoot the service you can run:
 
 .. code:: sh
 
     $> docker service ps s3docker service ps s3
     ID                         NAME      IMAGE             NODE                               DESIRED STATE  CURRENT STATE       ERROR
-    0ar81cw4lvv8chafm8pw48wbc  s3.1      scality/cloudserver  localhost.localdomain.localdomain  Running        Running 7 days ago
+    0ar81cw4lvv8chafm8pw48wbc  s3.1      scality/s3server  localhost.localdomain.localdomain  Running        Running 7 days ago
-    cvmf3j3bz8w6r4h0lf3pxo6eu   \_ s3.1  scality/cloudserver  localhost.localdomain.localdomain  Shutdown       Failed 7 days ago   "task: non-zero exit (137)"
+    cvmf3j3bz8w6r4h0lf3pxo6eu   \_ s3.1  scality/s3server  localhost.localdomain.localdomain  Shutdown       Failed 7 days ago   "task: non-zero exit (137)"
 
-If the error is truncated, view the error in detail by inspecting the
+If the error is truncated it is possible to have a more detailed view of
-Docker task ID:
+the error by inspecting the docker task ID:
 
 .. code:: sh
 
     $> docker inspect cvmf3j3bz8w6r4h0lf3pxo6eu
 
 Off you go!
-~~~~~~~~~~~
+-----------
+
+Let us know what you use this functionality for, and if you'd like any
+specific developments around it. Or, even better: come and contribute to
+our `Github repository <https://github.com/scality/s3/>`__! We look
+forward to meeting you!
 
-Let us know how you use this and if you'd like any specific developments 
-around it. Even better: come and contribute to our `Github repository 
-<https://github.com/scality/s3/>`__! We look forward to meeting you!
 
 S3FS
 ====
-
+Export your buckets as a filesystem with s3fs on top of Zenko CloudServer
-You can export buckets as a filesystem with s3fs on CloudServer.
 
 `s3fs <https://github.com/s3fs-fuse/s3fs-fuse>`__ is an open source
-tool, available both on Debian and RedHat distributions, that enables
+tool that allows you to mount an S3 bucket on a filesystem-like backend.
-you to mount an S3 bucket on a filesystem-like backend. This tutorial uses
+It is available both on Debian and RedHat distributions. For this
-an Ubuntu 14.04 host to deploy and use s3fs over CloudServer.
+tutorial, we used an Ubuntu 14.04 host to deploy and use s3fs over
+Scality's Zenko CloudServer.
 
 Deploying Zenko CloudServer with SSL
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+----------------------------
 
-First, deploy CloudServer with a file backend using `our DockerHub page
+First, you need to deploy **Zenko CloudServer**. This can be done very easily
-<https://hub.docker.com/r/zenko/cloudserver>`__.
+via `our DockerHub
+page <https://hub.docker.com/r/scality/s3server/>`__ (you want to run it
+with a file backend).
 
-.. note::
+    *Note:* *- If you don't have docker installed on your machine, here
+    are the `instructions to install it for your
+    distribution <https://docs.docker.com/engine/installation/>`__*
 
-  If Docker is not installed on your machine, follow
+You also necessarily have to set up SSL with Zenko CloudServer to use s3fs. We
-  `these instructions <https://docs.docker.com/engine/installation/>`__
+have a nice
-  to install it for your distribution.
+`tutorial <https://s3.scality.com/v1.0/page/scality-with-ssl>`__ to help
+you do it.
 
-You must also set up SSL with CloudServer to use s3fs. See `Using SSL
+s3fs setup
-<./GETTING_STARTED#Using_SSL>`__ for instructions.
+----------
-
-s3fs Setup
-~~~~~~~~~~
 
 Installing s3fs
----------------
+~~~~~~~~~~~~~~~
 
-Follow the instructions in the s3fs `README
+s3fs has quite a few dependencies. As explained in their
-<https://github.com/s3fs-fuse/s3fs-fuse/blob/master/README.md#installation-from-pre-built-packages>`__,
+`README <https://github.com/s3fs-fuse/s3fs-fuse/blob/master/README.md#installation>`__,
+the following commands should install everything for Ubuntu 14.04:
 
-Check that s3fs is properly installed. A version check should return
+.. code:: sh
-a response resembling:
+
+    $> sudo apt-get install automake autotools-dev g++ git libcurl4-gnutls-dev
+    $> sudo apt-get install libfuse-dev libssl-dev libxml2-dev make pkg-config
+
+Now you want to install s3fs per se:
+
+.. code:: sh
+
+    $> git clone https://github.com/s3fs-fuse/s3fs-fuse.git
+    $> cd s3fs-fuse
+    $> ./autogen.sh
+    $> ./configure
+    $> make
+    $> sudo make install
+
+Check that s3fs is properly installed by checking its version. it should
+answer as below:
 
 .. code:: sh
 
      $> s3fs --version
 
     Amazon Simple Storage Service File System V1.80(commit:d40da2c) with OpenSSL
-    Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
-    License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>
-    This is free software: you are free to change and redistribute it.
-    There is NO WARRANTY, to the extent permitted by law.
 
 Configuring s3fs
-----------------
+~~~~~~~~~~~~~~~~
 
 s3fs expects you to provide it with a password file. Our file is
 ``/etc/passwd-s3fs``. The structure for this file is
-``ACCESSKEYID:SECRETKEYID``, so, for CloudServer, you can run:
+``ACCESSKEYID:SECRETKEYID``, so, for S3Server, you can run:
 
 .. code:: sh
 
     $> echo 'accessKey1:verySecretKey1' > /etc/passwd-s3fs
     $> chmod 600 /etc/passwd-s3fs
 
-Using CloudServer with s3fs
+Using Zenko CloudServer with s3fs
----------------------------
+------------------------
 
-1. Use /mnt/tests3fs as a mount point.
+First, you're going to need a mountpoint; we chose ``/mnt/tests3fs``:
 
 .. code:: sh
 
     $> mkdir /mnt/tests3fs
 
-2. Create a bucket on your local CloudServer. In the present example it is
+Then, you want to create a bucket on your local Zenko CloudServer; we named it
-   named “tests3fs”.
+``tests3fs``:
 
 .. code:: sh
 
     $> s3cmd mb s3://tests3fs
 
-3. Mount the bucket to your mount point with s3fs:
+    *Note:* *- If you've never used s3cmd with our Zenko CloudServer, our README
+    provides you with a `recommended
+    config <https://github.com/scality/S3/blob/master/README.md#s3cmd>`__*
+
+Now you can mount your bucket to your mountpoint with s3fs:
 
 .. code:: sh
 
     $> s3fs tests3fs /mnt/tests3fs -o passwd_file=/etc/passwd-s3fs -o url="https://s3.scality.test:8000/" -o use_path_request_style
 
-   The structure of this command is:
+    *If you're curious, the structure of this command is*
-   ``s3fs BUCKET_NAME PATH/TO/MOUNTPOINT -o OPTIONS``. Of these mandatory
+    ``s3fs BUCKET_NAME PATH/TO/MOUNTPOINT -o OPTIONS``\ *, and the
-   options:
+    options are mandatory and serve the following purposes:
+    * ``passwd_file``\ *: specifiy path to password file;
+    * ``url``\ *: specify the hostname used by your SSL provider;
+    * ``use_path_request_style``\ *: force path style (by default, s3fs
+    uses subdomains (DNS style)).*
 
-   * ``passwd_file`` specifies the path to the password file.
+| From now on, you can either add files to your mountpoint, or add
-   * ``url`` specifies the host name used by your SSL provider.
+  objects to your bucket, and they'll show in the other.
-   * ``use_path_request_style`` forces the path style (by default,
+| For example, let's' create two files, and then a directory with a file
-       s3fs uses DNS-style subdomains).
+  in our mountpoint:
-
-Once the bucket is mounted, files added to the mount point or
-objects added to the bucket will appear in both locations.
-
-Example
--------
-
-   Create two files, and then a directory with a file in our mount point:
 
 .. code:: sh
 
@@ -378,7 +445,7 @@ Example
     $> mkdir /mnt/tests3fs/dir1
     $> touch /mnt/tests3fs/dir1/file3
 
-   Now, use s3cmd to show what is in CloudServer:
+Now, I can use s3cmd to show me what is actually in S3Server:
 
 .. code:: sh
 
@@ -389,30 +456,27 @@ Example
     2017-02-28 17:28         0   s3://tests3fs/file1
     2017-02-28 17:28         0   s3://tests3fs/file2
 
-   Now you can enjoy a filesystem view on your local CloudServer.
+Now you can enjoy a filesystem view on your local Zenko CloudServer!
 
 
 Duplicity
 =========
 
-How to back up your files with CloudServer.
+How to backup your files with Zenko CloudServer.
 
-Installing Duplicity and its Dependencies
+Installing
+-----------
+
+Installing Duplicity and its dependencies
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-To install `Duplicity <http://duplicity.nongnu.org/>`__,
+Second, you want to install
-go to `this site <https://code.launchpad.net/duplicity/0.7-series>`__.
+`Duplicity <http://duplicity.nongnu.org/index.html>`__. You have to
-Download the latest tarball. Decompress it and follow the instructions
+download `this
-in the README.
+tarball <https://code.launchpad.net/duplicity/0.7-series/0.7.11/+download/duplicity-0.7.11.tar.gz>`__,
-
+decompress it, and then checkout the README inside, which will give you
-.. code:: sh
+a list of dependencies to install. If you're using Ubuntu 14.04, this is
-
+your lucky day: here is a lazy step by step install.
-   $> tar zxvf duplicity-0.7.11.tar.gz
-   $> cd duplicity-0.7.11
-   $> python setup.py install
-
-You may receive error messages indicating the need to install some or all
-of the following dependencies:
 
 .. code:: sh
 
@@ -420,20 +484,30 @@ of the following dependencies:
     $> apt-get install python-dev python-pip python-lockfile
     $> pip install -U boto
 
-Testing the Installation
+Then you want to actually install Duplicity:
-------------------------
 
-1. Check that CloudServer is running. Run ``$> docker ps``. You should
+.. code:: sh
-   see one container named ``scality/cloudserver``. If you do not, run
-   ``$> docker start cloudserver`` and check again.
 
+    $> tar zxvf duplicity-0.7.11.tar.gz
+    $> cd duplicity-0.7.11
+    $> python setup.py install
 
-2. Duplicity uses a module called “Boto” to send requests to S3. Boto
+Using
-   requires a configuration file located in ``/etc/boto.cfg`` to store
+------
-   your credentials and preferences. A minimal configuration
+
-   you can fine tune `following these instructions
+Testing your installation
-   <http://boto.cloudhackers.com/en/latest/getting_started.html>`__ is
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
-   shown here:
+
+First, we're just going to quickly check that Zenko CloudServer is actually
+running. To do so, simply run ``$> docker ps`` . You should see one
+container named ``scality/s3server``. If that is not the case, try
+``$> docker start s3server``, and check again.
+
+Secondly, as you probably know, Duplicity uses a module called **Boto**
+to send requests to S3. Boto requires a configuration file located in
+**``/etc/boto.cfg``** to have your credentials and preferences. Here is
+a minimalistic config `that you can finetune following these
+instructions <http://boto.cloudhackers.com/en/latest/getting_started.html>`__.
 
 ::
 
@@ -447,25 +521,26 @@ Testing the Installation
     # If using SSL, unmute and provide absolute path to local CA certificate
     # ca_certificates_file = /absolute/path/to/ca.crt
 
- .. note:: To set up SSL with CloudServer, check out our `Using SSL
+    *Note:* *If you want to set up SSL with Zenko CloudServer, check out our
-	   <./GETTING_STARTED#Using_SSL>`__ in GETTING STARTED.
+    `tutorial <http://link/to/SSL/tutorial>`__*
 
-3. At this point all requirements to run CloudServer as a backend to Duplicity
+At this point, we've met all the requirements to start running Zenko CloudServer
-   have been met. A local folder/file should back up to the local S3.
+as a backend to Duplicity. So we should be able to back up a local
-   Try it with the decompressed Duplicity folder:
+folder/file to local S3. Let's try with the duplicity decompressed
+folder:
 
 .. code:: sh
 
     $> duplicity duplicity-0.7.11 "s3://127.0.0.1:8000/testbucket/"
 
-.. note:: Duplicity will prompt for a symmetric encryption passphrase.
+    *Note:* *Duplicity will prompt you for a symmetric encryption
-	  Save it carefully, as you will need it to recover your data.
+    passphrase. Save it somewhere as you will need it to recover your
-	  Alternatively, you can add the ``--no-encryption`` flag
+    data. Alternatively, you can also add the ``--no-encryption`` flag
-	  and the data will be stored plain.
+    and the data will be stored plain.*
 
-   If this command is successful, you will receive an output resembling:
+If this command is succesful, you will get an output looking like this:
 
-   .. code:: sh
+::
 
     --------------[ Backup Statistics ]--------------
     StartTime 1486486547.13 (Tue Feb  7 16:55:47 2017)
@@ -485,13 +560,15 @@ Testing the Installation
     Errors 0
     -------------------------------------------------
 
-Congratulations! You can now back up to your local S3 through Duplicity.
+Congratulations! You can now backup to your local S3 through duplicity
+:)
 
-Automating Backups
+Automating backups
-------------------
+~~~~~~~~~~~~~~~~~~~
 
-The easiest way to back up files periodically is to write a bash script
+Now you probably want to back up your files periodically. The easiest
-and add it to your crontab. A suggested script follows.
+way to do this is to write a bash script and add it to your crontab.
+Here is my suggestion for such a file:
 
 .. code:: sh
 
@@ -500,33 +577,33 @@ and add it to your crontab. A suggested script follows.
     # Export your passphrase so you don't have to type anything
     export PASSPHRASE="mypassphrase"
 
-    # To use a GPG key, put it here and uncomment the line below
+    # If you want to use a GPG Key, put it here and unmute the line below
     #GPG_KEY=
 
     # Define your backup bucket, with localhost specified
-    DEST="s3://127.0.0.1:8000/testbucketcloudserver/"
+    DEST="s3://127.0.0.1:8000/testbuckets3server/"
 
-    # Define the absolute path to the folder to back up
+    # Define the absolute path to the folder you want to backup
     SOURCE=/root/testfolder
 
     # Set to "full" for full backups, and "incremental" for incremental backups
-    # Warning: you must perform one full backup befor you can perform
+    # Warning: you have to perform one full backup befor you can perform
     # incremental ones on top of it
     FULL=incremental
 
-    # How long to keep backups. If you don't want to delete old backups, keep
+    # How long to keep backups for; if you don't want to delete old
-    # this value empty; otherwise, the syntax is "1Y" for one year, "1M" for
+    # backups, keep empty; otherwise, syntax is "1Y" for one year, "1M"
-    # one month, "1D" for one day.
+    # for one month, "1D" for one day
     OLDER_THAN="1Y"
 
-    # is_running checks whether Duplicity is currently completing a task
+    # is_running checks whether duplicity is currently completing a task
     is_running=$(ps -ef | grep duplicity  | grep python | wc -l)
 
-    # If Duplicity is already completing a task, this will not run
+    # If duplicity is already completing a task, this will simply not run
     if [ $is_running -eq 0 ]; then
         echo "Backup for ${SOURCE} started"
 
-        # To delete backups older than a certain time, do it here
+        # If you want to delete backups older than a certain time, we do it here
         if [ "$OLDER_THAN" != "" ]; then
             echo "Removing backups older than ${OLDER_THAN}"
             duplicity remove-older-than ${OLDER_THAN} ${DEST}
@@ -549,17 +626,17 @@ and add it to your crontab. A suggested script follows.
     # Forget the passphrase...
     unset PASSPHRASE
 
-Put this file in ``/usr/local/sbin/backup.sh``. Run ``crontab -e`` and
+So let's say you put this file in ``/usr/local/sbin/backup.sh.`` Next
-paste your configuration into the file that opens. If you're unfamiliar
+you want to run ``crontab -e`` and paste your configuration in the file
-with Cron, here is a good `HowTo
+that opens. If you're unfamiliar with Cron, here is a good `How
-<https://help.ubuntu.com/community/CronHowto>`__. If the folder being
+To <https://help.ubuntu.com/community/CronHowto>`__. The folder I'm
-backed up is a folder to be modified permanently during the work day,
+backing up is a folder I modify permanently during my workday, so I want
-we can set incremental backups every 5 minutes from 8 AM to 9 PM Monday
+incremental backups every 5mn from 8AM to 9PM monday to friday. Here is
-through Friday by pasting the following line into crontab:
+the line I will paste in my crontab:
 
-.. code:: sh
+.. code:: cron
 
     */5 8-20 * * 1-5 /usr/local/sbin/backup.sh
 
-Adding or removing files from the folder being backed up will result in
+Now I can try and add / remove files from the folder I'm backing up, and
-incremental backups in the bucket.
+I will see incremental backups in my bucket.

docs/MD_SEARCH.rst

@@ -1,263 +0,0 @@
-Metadata Search Documentation
-=============================
-
-Description
------------
-
-This feature enables metadata search to be performed on the metadata of objects
-stored in Zenko.
-
-Requirements
-------------
-
-* MongoDB
-
-Design
-------
-
-The Metadata Search feature expands on the existing :code:`GET Bucket` S3 API by
-enabling users to conduct metadata searches by adding the custom Zenko query
-string parameter, :code:`search`. The :code:`search` parameter is structured as a pseudo
-SQL WHERE clause, and supports basic SQL operators. For example:
-:code:`"A=1 AND B=2 OR C=3"` (complex queries can be built using nesting
-operators, :code:`(` and :code:`)`).
-
-The search process is as follows:
-
-* Zenko receives a :code:`GET` request.
-
-  .. code::
-
-    # regular getBucket request
-    GET /bucketname HTTP/1.1
-    Host: 127.0.0.1:8000
-    Date: Wed, 18 Oct 2018 17:50:00 GMT
-    Authorization: authorization string
-
-    # getBucket versions request
-    GET /bucketname?versions HTTP/1.1
-    Host: 127.0.0.1:8000
-    Date: Wed, 18 Oct 2018 17:50:00 GMT
-    Authorization: authorization string
-
-    # search getBucket request
-    GET /bucketname?search=key%3Dsearch-item HTTP/1.1
-    Host: 127.0.0.1:8000
-    Date: Wed, 18 Oct 2018 17:50:00 GMT
-    Authorization: authorization string
-
-* If the request does *not* contain the :code:`search` query parameter, Zenko performs
-  a normal bucket listing and returns an XML result containing the list of
-  objects.
-* If the request *does* contain the :code:`search` query parameter, Zenko parses and
-  validates the search string.
-
-  - If the search string is invalid, Zenko returns an :code:`InvalidArgument` error.
-
-    .. code::
-
-      <?xml version=\"1.0\" encoding=\"UTF-8\"?>
-      <Error>
-        <Code>InvalidArgument</Code>
-        <Message>Invalid sql where clause sent as search query</Message>
-        <Resource></Resource>
-        <RequestId>d1d6afc64345a8e1198e</RequestId>
-      </Error>
-
-  - If the search string is valid, Zenko parses it and generates an abstract
-    syntax tree (AST). The AST is then passed to the MongoDB backend to be
-    used as the query filter for retrieving objects from a bucket that
-    satisfies the requested search conditions. Zenko parses the filtered
-    results and returns them as the response.
-
-Metadata search results have the same structure as a :code:`GET Bucket` response:
-
-.. code:: xml
-
-  <?xml version="1.0" encoding="UTF-8"?>
-  <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
-      <Name>bucketname</Name>
-      <Prefix/>
-      <Marker/>
-      <MaxKeys>1000</MaxKeys>
-      <IsTruncated>false</IsTruncated>
-      <Contents>
-          <Key>objectKey</Key>
-          <LastModified>2018-04-19T18:31:49.426Z</LastModified>
-          <ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag>
-          <Size>0</Size>
-          <Owner>
-              <ID>79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be</ID>
-              <DisplayName>Bart</DisplayName>
-          </Owner>
-          <StorageClass>STANDARD</StorageClass>
-      </Contents>
-      <Contents>
-          ...
-      </Contents>
-  </ListBucketResult>
-
-Performing Metadata Searches with Zenko
----------------------------------------
-
-You can perform metadata searches by:
-
-+ Using the :code:`search_bucket` tool in the
-  `Scality/S3 <https://github.com/scality/S3>`_ GitHub repository.
-+ Creating a signed HTTP request to Zenko in your preferred programming
-  language.
-
-Using the S3 Tool
-+++++++++++++++++
-
-After cloning the `Scality/S3 <https://github.com/scality/S3>`_ GitHub repository
-and installing the necessary dependencies, run the following command in the S3
-project’s root directory to access the search tool:
-
-.. code::
-
-  node bin/search_bucket
-
-This generates the following output:
-
-.. code::
-
-    Usage: search_bucket [options]
-
-    Options:
-
-      -V, --version                 output the version number
-      -a, --access-key <accessKey>  Access key id
-      -k, --secret-key <secretKey>  Secret access key
-      -b, --bucket <bucket>         Name of the bucket
-      -q, --query <query>           Search query
-      -h, --host <host>             Host of the server
-      -p, --port <port>             Port of the server
-      -s                            --ssl
-      -v, --verbose
-      -h, --help                    output usage information
-
-In the following examples, Zenko Server is accessible on endpoint
-:code:`http://127.0.0.1:8000` and contains the bucket :code:`zenkobucket`.
-
-.. code::
-
-    # search for objects with metadata "blue"
-    node bin/search_bucket -a accessKey1 -k verySecretKey1 -b zenkobucket \
-        -q "x-amz-meta-color=blue" -h 127.0.0.1 -p 8000
-
-    # search for objects tagged with "type=color"
-    node bin/search_bucket -a accessKey1 -k verySecretKey1 -b zenkobucket \
-        -q "tags.type=color" -h 127.0.0.1 -p 8000
-
-Coding Examples
-+++++++++++++++
-
-Search requests can be also performed by making HTTP requests authenticated
-with one of the AWS Signature schemes: version 2 or version 4. \
-For more about authentication scheme, see:
-
-* https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
-* http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
-* http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
-
-You can also view examples for making requests with Auth V4 in various
-languages `here <../../../examples>`__.
-
-Specifying Metadata Fields
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-To search system metadata headers:
-
-.. code::
-
-    {system-metadata-key}{supported SQL op}{search value}
-    # example
-    key = blueObject
-    size > 0
-    key LIKE "blue.*"
-
-To search custom user metadata:
-
-.. code::
-
-    # metadata must be prefixed with "x-amz-meta-"
-    x-amz-meta-{user-metadata-key}{supported SQL op}{search value}
-
-    # example
-    x-amz-meta-color = blue
-    x-amz-meta-color != red
-    x-amz-meta-color LIKE "b.*"
-
-To search tags:
-
-.. code::
-
-    # tag searches must be prefixed with "tags."
-    tags.{tag-key}{supported SQL op}{search value}
-    # example
-    tags.type = color
-
-Examples queries:
-
-.. code::
-
-    # searching for objects with custom metadata "color"=blue" and are tagged
-    # "type"="color"
-
-    tags.type="color" AND  x-amz-meta-color="blue"
-
-    # searching for objects with the object key containing the substring "blue"
-    # or (custom metadata "color"=blue" and are tagged "type"="color")
-
-    key LIKE '.*blue.*' OR (x-amz-meta-color="blue" AND tags.type="color")
-
-Differences from SQL
-++++++++++++++++++++
-
-Zenko metadata search queries are similar to SQL-query :code:`WHERE` clauses, but
-differ in that:
-
-* They follow the :code:`PCRE` format
-* They do not require values with hyphens to be enclosed in
-  backticks, :code:``(`)``
-
-  .. code::
-
-        # SQL query
-        `x-amz-meta-search-item` = `ice-cream-cone`
-
-        # MD Search query
-        x-amz-meta-search-item = ice-cream-cone
-
-* Search queries do not support all SQL operators.
-
-  .. code::
-
-    # Supported SQL operators:
-    =, <, >, <=, >=, !=, AND, OR, LIKE, <>
-
-    # Unsupported SQL operators:
-    NOT, BETWEEN, IN, IS, +, -, %, ^, /, *, !
-
-Using Regular Expressions in Metadata Search
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Regular expressions in Zenko metadata search differ from SQL in the following
-ways:
-
-+ Wildcards are represented with :code:`.*` instead of :code:`%`.
-+ Regex patterns must be wrapped in quotes. Failure to do this can lead to
-  misinterpretation of patterns.
-+ As with :code:`PCRE`, regular expressions can be entered in either the
-  :code:`/pattern/` syntax or as the pattern itself if regex options are
-  not required.
-
-Example regular expressions:
-
-.. code::
-
-    # search for strings containing word substring "helloworld"
-        ".*helloworld.*"
-        "/.*helloworld.*/"
-        "/.*helloworld.*/i"