Compare commits
1 Commits
developmen
...
improvemen
Author | SHA1 | Date |
---|---|---|
Rahul Padigela | b04f88b222 |
|
@ -1,9 +0,0 @@
|
||||||
node_modules
|
|
||||||
localData/*
|
|
||||||
localMetadata/*
|
|
||||||
# Keep the .git/HEAD file in order to properly report version
|
|
||||||
.git/objects
|
|
||||||
.github
|
|
||||||
.tox
|
|
||||||
coverage
|
|
||||||
.DS_Store
|
|
55
.eslintrc
55
.eslintrc
|
@ -1,54 +1 @@
|
||||||
{
|
{ "extends": "scality" }
|
||||||
"extends": "scality",
|
|
||||||
"plugins": [
|
|
||||||
"mocha"
|
|
||||||
],
|
|
||||||
"rules": {
|
|
||||||
"import/extensions": "off",
|
|
||||||
"lines-around-directive": "off",
|
|
||||||
"no-underscore-dangle": "off",
|
|
||||||
"indent": "off",
|
|
||||||
"object-curly-newline": "off",
|
|
||||||
"operator-linebreak": "off",
|
|
||||||
"function-paren-newline": "off",
|
|
||||||
"import/newline-after-import": "off",
|
|
||||||
"prefer-destructuring": "off",
|
|
||||||
"implicit-arrow-linebreak": "off",
|
|
||||||
"no-bitwise": "off",
|
|
||||||
"dot-location": "off",
|
|
||||||
"comma-dangle": "off",
|
|
||||||
"no-undef-init": "off",
|
|
||||||
"global-require": "off",
|
|
||||||
"import/no-dynamic-require": "off",
|
|
||||||
"class-methods-use-this": "off",
|
|
||||||
"no-plusplus": "off",
|
|
||||||
"no-else-return": "off",
|
|
||||||
"object-property-newline": "off",
|
|
||||||
"import/order": "off",
|
|
||||||
"no-continue": "off",
|
|
||||||
"no-tabs": "off",
|
|
||||||
"lines-between-class-members": "off",
|
|
||||||
"prefer-spread": "off",
|
|
||||||
"no-lonely-if": "off",
|
|
||||||
"no-useless-escape": "off",
|
|
||||||
"no-restricted-globals": "off",
|
|
||||||
"no-buffer-constructor": "off",
|
|
||||||
"import/no-extraneous-dependencies": "off",
|
|
||||||
"space-unary-ops": "off",
|
|
||||||
"no-useless-return": "off",
|
|
||||||
"no-unexpected-multiline": "off",
|
|
||||||
"no-mixed-operators": "off",
|
|
||||||
"newline-per-chained-call": "off",
|
|
||||||
"operator-assignment": "off",
|
|
||||||
"spaced-comment": "off",
|
|
||||||
"comma-style": "off",
|
|
||||||
"no-restricted-properties": "off",
|
|
||||||
"new-parens": "off",
|
|
||||||
"no-multi-spaces": "off",
|
|
||||||
"quote-props": "off",
|
|
||||||
"mocha/no-exclusive-tests": "error",
|
|
||||||
},
|
|
||||||
"parserOptions": {
|
|
||||||
"ecmaVersion": 2020
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
|
@ -1,32 +1,19 @@
|
||||||
# General support information
|
# Issue template
|
||||||
|
|
||||||
GitHub Issues are **reserved** for actionable bug reports (including
|
If you are reporting a new issue, make sure that we do not have any
|
||||||
documentation inaccuracies), and feature requests.
|
duplicates already open. You can ensure this by searching the issue list for
|
||||||
**All questions** (regarding configuration, use cases, performance, community,
|
this repository. If there is a duplicate, please close your issue and add a
|
||||||
events, setup and usage recommendations, among other things) should be asked on
|
comment to the existing issue instead.
|
||||||
the **[Zenko Forum](http://forum.zenko.io/)**.
|
|
||||||
|
|
||||||
> Questions opened as GitHub issues will systematically be closed, and moved to
|
## General support information
|
||||||
> the [Zenko Forum](http://forum.zenko.io/).
|
|
||||||
|
|
||||||
--------------------------------------------------------------------------------
|
GitHub Issues are reserved for actionable bug reports and feature requests.
|
||||||
|
General questions should be sent to the
|
||||||
## Avoiding duplicates
|
[S3 scality server Forum](http://forum.scality.com/).
|
||||||
|
|
||||||
When reporting a new issue/requesting a feature, make sure that we do not have
|
|
||||||
any duplicates already open:
|
|
||||||
|
|
||||||
- search the issue list for this repository (use the search bar, select
|
|
||||||
"Issues" on the left pane after searching);
|
|
||||||
- if there is a duplicate, please do not open your issue, and add a comment
|
|
||||||
to the existing issue instead.
|
|
||||||
|
|
||||||
--------------------------------------------------------------------------------
|
|
||||||
|
|
||||||
## Bug report information
|
## Bug report information
|
||||||
|
|
||||||
(delete this section (everything between the lines) if you're not reporting a bug
|
(delete this section if not applicable)
|
||||||
but requesting a feature)
|
|
||||||
|
|
||||||
### Description
|
### Description
|
||||||
|
|
||||||
|
@ -42,22 +29,13 @@ Describe the results you received
|
||||||
|
|
||||||
### Expected result
|
### Expected result
|
||||||
|
|
||||||
Describe the results you expected
|
Describe the results you expecteds
|
||||||
|
|
||||||
### Additional information
|
### Additional information: (Node.js version, Docker version, etc)
|
||||||
|
|
||||||
- Node.js version,
|
|
||||||
- Docker version,
|
|
||||||
- yarn version,
|
|
||||||
- distribution/OS,
|
|
||||||
- optional: anything else you deem helpful to us.
|
|
||||||
|
|
||||||
--------------------------------------------------------------------------------
|
|
||||||
|
|
||||||
## Feature Request
|
## Feature Request
|
||||||
|
|
||||||
(delete this section (everything between the lines) if you're not requesting
|
(delete this section if not applicable)
|
||||||
a feature but reporting a bug)
|
|
||||||
|
|
||||||
### Proposal
|
### Proposal
|
||||||
|
|
||||||
|
@ -74,14 +52,3 @@ What you would like to happen
|
||||||
### Use case
|
### Use case
|
||||||
|
|
||||||
Please provide use cases for changing the current behavior
|
Please provide use cases for changing the current behavior
|
||||||
|
|
||||||
### Additional information
|
|
||||||
|
|
||||||
- Is this request for your company? Y/N
|
|
||||||
- If Y: Company name:
|
|
||||||
- Are you using any Scality Enterprise Edition products (RING, Zenko EE)? Y/N
|
|
||||||
- Are you willing to contribute this feature yourself?
|
|
||||||
- Position/Title:
|
|
||||||
- How did you hear about us?
|
|
||||||
|
|
||||||
--------------------------------------------------------------------------------
|
|
||||||
|
|
|
@ -1,43 +0,0 @@
|
||||||
---
|
|
||||||
name: "Setup CI environment"
|
|
||||||
description: "Setup Cloudserver CI environment"
|
|
||||||
|
|
||||||
runs:
|
|
||||||
using: composite
|
|
||||||
steps:
|
|
||||||
- name: Setup etc/hosts
|
|
||||||
shell: bash
|
|
||||||
run: sudo echo "127.0.0.1 bucketwebsitetester.s3-website-us-east-1.amazonaws.com" | sudo tee -a /etc/hosts
|
|
||||||
- name: Setup Credentials
|
|
||||||
shell: bash
|
|
||||||
run: bash .github/scripts/credentials.bash
|
|
||||||
- name: Setup job artifacts directory
|
|
||||||
shell: bash
|
|
||||||
run: |-
|
|
||||||
set -exu;
|
|
||||||
mkdir -p /tmp/artifacts/${JOB_NAME}/;
|
|
||||||
- uses: actions/setup-node@v4
|
|
||||||
with:
|
|
||||||
node-version: '16'
|
|
||||||
cache: 'yarn'
|
|
||||||
- name: install dependencies
|
|
||||||
shell: bash
|
|
||||||
run: yarn install --ignore-engines --frozen-lockfile --network-concurrency 1
|
|
||||||
- uses: actions/cache@v3
|
|
||||||
with:
|
|
||||||
path: ~/.cache/pip
|
|
||||||
key: ${{ runner.os }}-pip
|
|
||||||
- uses: actions/setup-python@v4
|
|
||||||
with:
|
|
||||||
python-version: 3.9
|
|
||||||
- name: Setup python2 test environment
|
|
||||||
shell: bash
|
|
||||||
run: |
|
|
||||||
sudo apt-get install -y libdigest-hmac-perl
|
|
||||||
pip install 's3cmd==2.3.0'
|
|
||||||
- name: fix sproxyd.conf permissions
|
|
||||||
shell: bash
|
|
||||||
run: sudo chown root:root .github/docker/sproxyd/conf/sproxyd0.conf
|
|
||||||
- name: ensure fuse kernel module is loaded (for sproxyd)
|
|
||||||
shell: bash
|
|
||||||
run: sudo modprobe fuse
|
|
|
@ -1,25 +0,0 @@
|
||||||
FROM ceph/daemon:v3.2.1-stable-3.2-mimic-centos-7
|
|
||||||
|
|
||||||
ENV CEPH_DAEMON demo
|
|
||||||
ENV CEPH_DEMO_DAEMONS mon,mgr,osd,rgw
|
|
||||||
|
|
||||||
ENV CEPH_DEMO_UID zenko
|
|
||||||
ENV CEPH_DEMO_ACCESS_KEY accessKey1
|
|
||||||
ENV CEPH_DEMO_SECRET_KEY verySecretKey1
|
|
||||||
ENV CEPH_DEMO_BUCKET zenkobucket
|
|
||||||
|
|
||||||
ENV CEPH_PUBLIC_NETWORK 0.0.0.0/0
|
|
||||||
ENV MON_IP 0.0.0.0
|
|
||||||
ENV NETWORK_AUTO_DETECT 4
|
|
||||||
ENV RGW_CIVETWEB_PORT 8001
|
|
||||||
|
|
||||||
RUN rm /etc/yum.repos.d/tcmu-runner.repo
|
|
||||||
|
|
||||||
ADD ./entrypoint-wrapper.sh /
|
|
||||||
RUN chmod +x /entrypoint-wrapper.sh && \
|
|
||||||
yum install -y python-pip && \
|
|
||||||
yum clean all && \
|
|
||||||
pip install awscli && \
|
|
||||||
rm -rf /root/.cache/pip
|
|
||||||
|
|
||||||
ENTRYPOINT [ "/entrypoint-wrapper.sh" ]
|
|
|
@ -1,37 +0,0 @@
|
||||||
#!/bin/sh
|
|
||||||
|
|
||||||
touch /artifacts/ceph.log
|
|
||||||
mkfifo /tmp/entrypoint_output
|
|
||||||
# We run this in the background so that we can tail the RGW log after init,
|
|
||||||
# because entrypoint.sh never returns
|
|
||||||
|
|
||||||
# The next line will be needed when ceph builds 3.2.2 so I'll leave it here
|
|
||||||
# bash /opt/ceph-container/bin/entrypoint.sh > /tmp/entrypoint_output &
|
|
||||||
|
|
||||||
bash /entrypoint.sh > /tmp/entrypoint_output &
|
|
||||||
entrypoint_pid="$!"
|
|
||||||
while read -r line; do
|
|
||||||
echo $line
|
|
||||||
# When we find this line server has started
|
|
||||||
if [ -n "$(echo $line | grep 'Creating bucket')" ]; then
|
|
||||||
break
|
|
||||||
fi
|
|
||||||
done < /tmp/entrypoint_output
|
|
||||||
|
|
||||||
# Make our buckets - CEPH_DEMO_BUCKET is set to force the "Creating bucket" message, but unused
|
|
||||||
s3cmd mb s3://cephbucket s3://cephbucket2
|
|
||||||
|
|
||||||
mkdir /root/.aws
|
|
||||||
cat > /root/.aws/credentials <<EOF
|
|
||||||
[default]
|
|
||||||
aws_access_key_id = accessKey1
|
|
||||||
aws_secret_access_key = verySecretKey1
|
|
||||||
EOF
|
|
||||||
|
|
||||||
# Enable versioning on them
|
|
||||||
for bucket in cephbucket cephbucket2; do
|
|
||||||
echo "Enabling versiong for $bucket"
|
|
||||||
aws --endpoint http://127.0.0.1:8001 s3api put-bucket-versioning --bucket $bucket --versioning Status=Enabled
|
|
||||||
done
|
|
||||||
tail -f /var/log/ceph/client.rgw.*.log | tee -a /artifacts/ceph.log
|
|
||||||
wait $entrypoint_pid
|
|
|
@ -1,11 +0,0 @@
|
||||||
#!/bin/sh
|
|
||||||
|
|
||||||
# This script is needed because RADOS Gateway
|
|
||||||
# will open the port before beginning to serve traffic
|
|
||||||
# causing wait_for_local_port.bash to exit immediately
|
|
||||||
|
|
||||||
echo 'Waiting for ceph'
|
|
||||||
while [ -z "$(curl 127.0.0.1:8001 2>/dev/null)" ]; do
|
|
||||||
sleep 1
|
|
||||||
echo -n "."
|
|
||||||
done
|
|
|
@ -1,10 +0,0 @@
|
||||||
---
|
|
||||||
version: 2
|
|
||||||
updates:
|
|
||||||
- package-ecosystem: npm
|
|
||||||
directory: "/"
|
|
||||||
schedule:
|
|
||||||
interval: daily
|
|
||||||
time: "13:00"
|
|
||||||
open-pull-requests-limit: 10
|
|
||||||
target-branch: "development/7.4"
|
|
|
@ -1,36 +0,0 @@
|
||||||
azurebackend_AZURE_STORAGE_ACCESS_KEY
|
|
||||||
azurebackend_AZURE_STORAGE_ACCOUNT_NAME
|
|
||||||
azurebackend_AZURE_STORAGE_ENDPOINT
|
|
||||||
azurebackend2_AZURE_STORAGE_ACCESS_KEY
|
|
||||||
azurebackend2_AZURE_STORAGE_ACCOUNT_NAME
|
|
||||||
azurebackend2_AZURE_STORAGE_ENDPOINT
|
|
||||||
azurebackendmismatch_AZURE_STORAGE_ACCESS_KEY
|
|
||||||
azurebackendmismatch_AZURE_STORAGE_ACCOUNT_NAME
|
|
||||||
azurebackendmismatch_AZURE_STORAGE_ENDPOINT
|
|
||||||
azurenonexistcontainer_AZURE_STORAGE_ACCESS_KEY
|
|
||||||
azurenonexistcontainer_AZURE_STORAGE_ACCOUNT_NAME
|
|
||||||
azurenonexistcontainer_AZURE_STORAGE_ENDPOINT
|
|
||||||
azuretest_AZURE_BLOB_ENDPOINT
|
|
||||||
b2backend_B2_ACCOUNT_ID
|
|
||||||
b2backend_B2_STORAGE_ACCESS_KEY
|
|
||||||
GOOGLE_SERVICE_EMAIL
|
|
||||||
GOOGLE_SERVICE_KEY
|
|
||||||
AWS_S3_BACKEND_ACCESS_KEY
|
|
||||||
AWS_S3_BACKEND_SECRET_KEY
|
|
||||||
AWS_S3_BACKEND_ACCESS_KEY_2
|
|
||||||
AWS_S3_BACKEND_SECRET_KEY_2
|
|
||||||
AWS_GCP_BACKEND_ACCESS_KEY
|
|
||||||
AWS_GCP_BACKEND_SECRET_KEY
|
|
||||||
AWS_GCP_BACKEND_ACCESS_KEY_2
|
|
||||||
AWS_GCP_BACKEND_SECRET_KEY_2
|
|
||||||
b2backend_B2_STORAGE_ENDPOINT
|
|
||||||
gcpbackend2_GCP_SERVICE_EMAIL
|
|
||||||
gcpbackend2_GCP_SERVICE_KEY
|
|
||||||
gcpbackend2_GCP_SERVICE_KEYFILE
|
|
||||||
gcpbackend_GCP_SERVICE_EMAIL
|
|
||||||
gcpbackend_GCP_SERVICE_KEY
|
|
||||||
gcpbackendmismatch_GCP_SERVICE_EMAIL
|
|
||||||
gcpbackendmismatch_GCP_SERVICE_KEY
|
|
||||||
gcpbackend_GCP_SERVICE_KEYFILE
|
|
||||||
gcpbackendmismatch_GCP_SERVICE_KEYFILE
|
|
||||||
gcpbackendnoproxy_GCP_SERVICE_KEYFILE
|
|
|
@ -1,92 +0,0 @@
|
||||||
services:
|
|
||||||
cloudserver:
|
|
||||||
image: ${CLOUDSERVER_IMAGE}
|
|
||||||
command: sh -c "yarn start > /artifacts/s3.log"
|
|
||||||
network_mode: "host"
|
|
||||||
volumes:
|
|
||||||
- /tmp/ssl:/ssl
|
|
||||||
- /tmp/ssl-kmip:/ssl-kmip
|
|
||||||
- ${HOME}/.aws/credentials:/root/.aws/credentials
|
|
||||||
- /tmp/artifacts/${JOB_NAME}:/artifacts
|
|
||||||
environment:
|
|
||||||
- CI=true
|
|
||||||
- ENABLE_LOCAL_CACHE=true
|
|
||||||
- REDIS_HOST=0.0.0.0
|
|
||||||
- REDIS_PORT=6379
|
|
||||||
- REPORT_TOKEN=report-token-1
|
|
||||||
- REMOTE_MANAGEMENT_DISABLE=1
|
|
||||||
- HEALTHCHECKS_ALLOWFROM=0.0.0.0/0
|
|
||||||
- DATA_HOST=0.0.0.0
|
|
||||||
- METADATA_HOST=0.0.0.0
|
|
||||||
- S3BACKEND
|
|
||||||
- S3DATA
|
|
||||||
- S3METADATA
|
|
||||||
- MPU_TESTING
|
|
||||||
- S3VAULT
|
|
||||||
- S3_LOCATION_FILE
|
|
||||||
- ENABLE_UTAPI_V2
|
|
||||||
- BUCKET_DENY_FILTER
|
|
||||||
- S3KMS
|
|
||||||
- S3KMIP_PORT
|
|
||||||
- S3KMIP_HOSTS
|
|
||||||
- S3KMIP-COMPOUND_CREATE
|
|
||||||
- S3KMIP_BUCKET_ATTRIBUTE_NAME
|
|
||||||
- S3KMIP_PIPELINE_DEPTH
|
|
||||||
- S3KMIP_KEY
|
|
||||||
- S3KMIP_CERT
|
|
||||||
- S3KMIP_CA
|
|
||||||
- MONGODB_HOSTS=0.0.0.0:27018
|
|
||||||
- MONGODB_RS=rs0
|
|
||||||
- DEFAULT_BUCKET_KEY_FORMAT
|
|
||||||
- METADATA_MAX_CACHED_BUCKETS
|
|
||||||
- ENABLE_NULL_VERSION_COMPAT_MODE
|
|
||||||
- SCUBA_HOST
|
|
||||||
- SCUBA_PORT
|
|
||||||
- SCUBA_HEALTHCHECK_FREQUENCY
|
|
||||||
- S3QUOTA
|
|
||||||
- QUOTA_ENABLE_INFLIGHTS
|
|
||||||
env_file:
|
|
||||||
- creds.env
|
|
||||||
depends_on:
|
|
||||||
- redis
|
|
||||||
extra_hosts:
|
|
||||||
- "bucketwebsitetester.s3-website-us-east-1.amazonaws.com:127.0.0.1"
|
|
||||||
- "pykmip.local:127.0.0.1"
|
|
||||||
redis:
|
|
||||||
image: redis:alpine
|
|
||||||
network_mode: "host"
|
|
||||||
squid:
|
|
||||||
network_mode: "host"
|
|
||||||
profiles: ['ci-proxy']
|
|
||||||
image: scality/ci-squid
|
|
||||||
command: >-
|
|
||||||
sh -c 'mkdir -p /ssl &&
|
|
||||||
openssl req -new -newkey rsa:2048 -sha256 -days 365 -nodes -x509 \
|
|
||||||
-subj "/C=US/ST=Country/L=City/O=Organization/CN=CN=scality-proxy" \
|
|
||||||
-keyout /ssl/myca.pem -out /ssl/myca.pem &&
|
|
||||||
cp /ssl/myca.pem /ssl/CA.pem &&
|
|
||||||
squid -f /etc/squid/squid.conf -N -z &&
|
|
||||||
squid -f /etc/squid/squid.conf -NYCd 1'
|
|
||||||
volumes:
|
|
||||||
- /tmp/ssl:/ssl
|
|
||||||
pykmip:
|
|
||||||
network_mode: "host"
|
|
||||||
profiles: ['pykmip']
|
|
||||||
image: ${PYKMIP_IMAGE:-ghcr.io/scality/cloudserver/pykmip}
|
|
||||||
volumes:
|
|
||||||
- /tmp/artifacts/${JOB_NAME}:/artifacts
|
|
||||||
mongo:
|
|
||||||
network_mode: "host"
|
|
||||||
profiles: ['mongo', 'ceph']
|
|
||||||
image: ${MONGODB_IMAGE}
|
|
||||||
ceph:
|
|
||||||
network_mode: "host"
|
|
||||||
profiles: ['ceph']
|
|
||||||
image: ghcr.io/scality/cloudserver/ci-ceph
|
|
||||||
sproxyd:
|
|
||||||
network_mode: "host"
|
|
||||||
profiles: ['sproxyd']
|
|
||||||
image: sproxyd-standalone
|
|
||||||
build: ./sproxyd
|
|
||||||
user: 0:0
|
|
||||||
privileged: yes
|
|
|
@ -1,28 +0,0 @@
|
||||||
FROM mongo:5.0.21
|
|
||||||
|
|
||||||
ENV USER=scality \
|
|
||||||
HOME_DIR=/home/scality \
|
|
||||||
CONF_DIR=/conf \
|
|
||||||
DATA_DIR=/data
|
|
||||||
|
|
||||||
# Set up directories and permissions
|
|
||||||
RUN mkdir -p /data/db /data/configdb && chown -R mongodb:mongodb /data/db /data/configdb; \
|
|
||||||
mkdir /logs; \
|
|
||||||
adduser --uid 1000 --disabled-password --gecos --quiet --shell /bin/bash scality
|
|
||||||
|
|
||||||
# Set up environment variables and directories for scality user
|
|
||||||
RUN mkdir ${CONF_DIR} && \
|
|
||||||
chown -R ${USER} ${CONF_DIR} && \
|
|
||||||
chown -R ${USER} ${DATA_DIR}
|
|
||||||
|
|
||||||
# copy the mongo config file
|
|
||||||
COPY /conf/mongod.conf /conf/mongod.conf
|
|
||||||
COPY /conf/mongo-run.sh /conf/mongo-run.sh
|
|
||||||
COPY /conf/initReplicaSet /conf/initReplicaSet.js
|
|
||||||
|
|
||||||
EXPOSE 27017/tcp
|
|
||||||
EXPOSE 27018
|
|
||||||
|
|
||||||
# Set up CMD
|
|
||||||
ENTRYPOINT ["bash", "/conf/mongo-run.sh"]
|
|
||||||
CMD ["bash", "/conf/mongo-run.sh"]
|
|
|
@ -1,4 +0,0 @@
|
||||||
rs.initiate({
|
|
||||||
_id: "rs0",
|
|
||||||
members: [{ _id: 0, host: "127.0.0.1:27018" }]
|
|
||||||
});
|
|
|
@ -1,10 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
set -exo pipefail
|
|
||||||
|
|
||||||
init_RS() {
|
|
||||||
sleep 5
|
|
||||||
mongo --port 27018 /conf/initReplicaSet.js
|
|
||||||
}
|
|
||||||
init_RS &
|
|
||||||
|
|
||||||
mongod --bind_ip_all --config=/conf/mongod.conf
|
|
|
@ -1,15 +0,0 @@
|
||||||
storage:
|
|
||||||
journal:
|
|
||||||
enabled: true
|
|
||||||
engine: wiredTiger
|
|
||||||
dbPath: "/data/db"
|
|
||||||
processManagement:
|
|
||||||
fork: false
|
|
||||||
net:
|
|
||||||
port: 27018
|
|
||||||
bindIp: 0.0.0.0
|
|
||||||
replication:
|
|
||||||
replSetName: "rs0"
|
|
||||||
enableMajorityReadConcern: true
|
|
||||||
security:
|
|
||||||
authorization: disabled
|
|
|
@ -1,3 +0,0 @@
|
||||||
FROM ghcr.io/scality/federation/sproxyd:7.10.6.8
|
|
||||||
ADD ./conf/supervisord.conf ./conf/nginx.conf ./conf/fastcgi_params ./conf/sproxyd0.conf /conf/
|
|
||||||
RUN chown root:root /conf/sproxyd0.conf
|
|
|
@ -1,26 +0,0 @@
|
||||||
fastcgi_param QUERY_STRING $query_string;
|
|
||||||
fastcgi_param REQUEST_METHOD $request_method;
|
|
||||||
fastcgi_param CONTENT_TYPE $content_type;
|
|
||||||
fastcgi_param CONTENT_LENGTH $content_length;
|
|
||||||
|
|
||||||
#fastcgi_param SCRIPT_NAME $fastcgi_script_name;
|
|
||||||
fastcgi_param SCRIPT_NAME /var/www;
|
|
||||||
fastcgi_param PATH_INFO $document_uri;
|
|
||||||
|
|
||||||
fastcgi_param REQUEST_URI $request_uri;
|
|
||||||
fastcgi_param DOCUMENT_URI $document_uri;
|
|
||||||
fastcgi_param DOCUMENT_ROOT $document_root;
|
|
||||||
fastcgi_param SERVER_PROTOCOL $server_protocol;
|
|
||||||
fastcgi_param HTTPS $https if_not_empty;
|
|
||||||
|
|
||||||
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
|
|
||||||
fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;
|
|
||||||
|
|
||||||
fastcgi_param REMOTE_ADDR $remote_addr;
|
|
||||||
fastcgi_param REMOTE_PORT $remote_port;
|
|
||||||
fastcgi_param SERVER_ADDR $server_addr;
|
|
||||||
fastcgi_param SERVER_PORT $server_port;
|
|
||||||
fastcgi_param SERVER_NAME $server_name;
|
|
||||||
|
|
||||||
# PHP only, required if PHP was built with --enable-force-cgi-redirect
|
|
||||||
fastcgi_param REDIRECT_STATUS 200;
|
|
|
@ -1,88 +0,0 @@
|
||||||
worker_processes 1;
|
|
||||||
error_log /logs/error.log;
|
|
||||||
user root root;
|
|
||||||
events {
|
|
||||||
worker_connections 1000;
|
|
||||||
reuse_port on;
|
|
||||||
multi_accept on;
|
|
||||||
}
|
|
||||||
worker_rlimit_nofile 20000;
|
|
||||||
http {
|
|
||||||
root /var/www/;
|
|
||||||
upstream sproxyds {
|
|
||||||
least_conn;
|
|
||||||
keepalive 40;
|
|
||||||
server 127.0.0.1:20000;
|
|
||||||
}
|
|
||||||
server {
|
|
||||||
client_max_body_size 0;
|
|
||||||
client_body_timeout 150;
|
|
||||||
client_header_timeout 150;
|
|
||||||
postpone_output 0;
|
|
||||||
client_body_postpone_size 0;
|
|
||||||
keepalive_requests 1100;
|
|
||||||
keepalive_timeout 300s;
|
|
||||||
server_tokens off;
|
|
||||||
default_type application/octet-stream;
|
|
||||||
gzip off;
|
|
||||||
tcp_nodelay on;
|
|
||||||
tcp_nopush on;
|
|
||||||
sendfile on;
|
|
||||||
listen 81;
|
|
||||||
server_name localhost;
|
|
||||||
rewrite ^/arc/(.*)$ /dc1/$1 permanent;
|
|
||||||
location ~* ^/proxy/(.*)$ {
|
|
||||||
rewrite ^/proxy/(.*)$ /$1 last;
|
|
||||||
}
|
|
||||||
allow 127.0.0.1;
|
|
||||||
|
|
||||||
deny all;
|
|
||||||
set $usermd '-';
|
|
||||||
set $sentusermd '-';
|
|
||||||
set $elapsed_ms '-';
|
|
||||||
set $now '-';
|
|
||||||
log_by_lua '
|
|
||||||
if not(ngx.var.http_x_scal_usermd == nil) and string.len(ngx.var.http_x_scal_usermd) > 2 then
|
|
||||||
ngx.var.usermd = string.sub(ngx.decode_base64(ngx.var.http_x_scal_usermd),1,-3)
|
|
||||||
end
|
|
||||||
if not(ngx.var.sent_http_x_scal_usermd == nil) and string.len(ngx.var.sent_http_x_scal_usermd) > 2 then
|
|
||||||
ngx.var.sentusermd = string.sub(ngx.decode_base64(ngx.var.sent_http_x_scal_usermd),1,-3)
|
|
||||||
end
|
|
||||||
local elapsed_ms = tonumber(ngx.var.request_time)
|
|
||||||
if not ( elapsed_ms == nil) then
|
|
||||||
elapsed_ms = elapsed_ms * 1000
|
|
||||||
ngx.var.elapsed_ms = tostring(elapsed_ms)
|
|
||||||
end
|
|
||||||
local time = tonumber(ngx.var.msec) * 1000
|
|
||||||
ngx.var.now = time
|
|
||||||
';
|
|
||||||
log_format irm '{ "time":"$now","connection":"$connection","request":"$connection_requests","hrtime":"$msec",'
|
|
||||||
'"httpMethod":"$request_method","httpURL":"$uri","elapsed_ms":$elapsed_ms,'
|
|
||||||
'"httpCode":$status,"requestLength":$request_length,"bytesSent":$bytes_sent,'
|
|
||||||
'"contentLength":"$content_length","sentContentLength":"$sent_http_content_length",'
|
|
||||||
'"contentType":"$content_type","s3Address":"$remote_addr",'
|
|
||||||
'"requestUserMd":"$usermd","responseUserMd":"$sentusermd",'
|
|
||||||
'"ringKeyVersion":"$sent_http_x_scal_version","ringStatus":"$sent_http_x_scal_ring_status",'
|
|
||||||
'"s3Port":"$remote_port","sproxydStatus":"$upstream_status","req_id":"$http_x_scal_request_uids",'
|
|
||||||
'"ifMatch":"$http_if_match","ifNoneMatch":"$http_if_none_match",'
|
|
||||||
'"range":"$http_range","contentRange":"$sent_http_content_range","nginxPID":$PID,'
|
|
||||||
'"sproxydAddress":"$upstream_addr","sproxydResponseTime_s":"$upstream_response_time" }';
|
|
||||||
access_log /dev/stdout irm;
|
|
||||||
error_log /dev/stdout error;
|
|
||||||
location / {
|
|
||||||
proxy_request_buffering off;
|
|
||||||
fastcgi_request_buffering off;
|
|
||||||
fastcgi_no_cache 1;
|
|
||||||
fastcgi_cache_bypass 1;
|
|
||||||
fastcgi_buffering off;
|
|
||||||
fastcgi_ignore_client_abort on;
|
|
||||||
fastcgi_keep_conn on;
|
|
||||||
include fastcgi_params;
|
|
||||||
fastcgi_pass sproxyds;
|
|
||||||
fastcgi_next_upstream error timeout;
|
|
||||||
fastcgi_send_timeout 285s;
|
|
||||||
fastcgi_read_timeout 285s;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
|
@ -1,12 +0,0 @@
|
||||||
{
|
|
||||||
"general": {
|
|
||||||
"ring": "DATA",
|
|
||||||
"port": 20000,
|
|
||||||
"syslog_facility": "local0"
|
|
||||||
},
|
|
||||||
"ring_driver:0": {
|
|
||||||
"alias": "dc1",
|
|
||||||
"type": "local",
|
|
||||||
"queue_path": "/tmp/ring-objs"
|
|
||||||
},
|
|
||||||
}
|
|
|
@ -1,43 +0,0 @@
|
||||||
[supervisord]
|
|
||||||
nodaemon = true
|
|
||||||
loglevel = info
|
|
||||||
logfile = %(ENV_LOG_DIR)s/supervisord.log
|
|
||||||
pidfile = %(ENV_SUP_RUN_DIR)s/supervisord.pid
|
|
||||||
logfile_maxbytes = 20MB
|
|
||||||
logfile_backups = 2
|
|
||||||
|
|
||||||
[unix_http_server]
|
|
||||||
file = %(ENV_SUP_RUN_DIR)s/supervisor.sock
|
|
||||||
|
|
||||||
[rpcinterface:supervisor]
|
|
||||||
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
|
|
||||||
|
|
||||||
[supervisorctl]
|
|
||||||
serverurl = unix://%(ENV_SUP_RUN_DIR)s/supervisor.sock
|
|
||||||
|
|
||||||
[program:nginx]
|
|
||||||
directory=%(ENV_SUP_RUN_DIR)s
|
|
||||||
command=bash -c "/usr/sbin/nginx -c %(ENV_CONF_DIR)s/nginx.conf -g 'daemon off;'"
|
|
||||||
stdout_logfile = %(ENV_LOG_DIR)s/%(program_name)s-%(process_num)s.log
|
|
||||||
stderr_logfile = %(ENV_LOG_DIR)s/%(program_name)s-%(process_num)s-stderr.log
|
|
||||||
stdout_logfile_maxbytes=100MB
|
|
||||||
stdout_logfile_backups=7
|
|
||||||
stderr_logfile_maxbytes=100MB
|
|
||||||
stderr_logfile_backups=7
|
|
||||||
autorestart=true
|
|
||||||
autostart=true
|
|
||||||
user=root
|
|
||||||
|
|
||||||
[program:sproxyd]
|
|
||||||
directory=%(ENV_SUP_RUN_DIR)s
|
|
||||||
process_name=%(program_name)s-%(process_num)s
|
|
||||||
numprocs=1
|
|
||||||
numprocs_start=0
|
|
||||||
command=/usr/bin/sproxyd -dlw -V127 -c %(ENV_CONF_DIR)s/sproxyd%(process_num)s.conf -P /run%(process_num)s
|
|
||||||
stdout_logfile = %(ENV_LOG_DIR)s/%(program_name)s-%(process_num)s.log
|
|
||||||
stdout_logfile_maxbytes=100MB
|
|
||||||
stdout_logfile_backups=7
|
|
||||||
redirect_stderr=true
|
|
||||||
autorestart=true
|
|
||||||
autostart=true
|
|
||||||
user=root
|
|
|
@ -1,18 +0,0 @@
|
||||||
-----BEGIN CERTIFICATE-----
|
|
||||||
MIIC6zCCAdOgAwIBAgIUPIpMY95b4HjKAk+FyydZApAEFskwDQYJKoZIhvcNAQEL
|
|
||||||
BQAwJDEQMA4GA1UECgwHU2NhbGl0eTEQMA4GA1UEAwwHUm9vdCBDQTAgFw0yMTA0
|
|
||||||
MDkwMDI4MTFaGA8yMTIxMDMxNjAwMjgxMVowJDEQMA4GA1UECgwHU2NhbGl0eTEQ
|
|
||||||
MA4GA1UEAwwHUm9vdCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
|
|
||||||
AKqLFEsWtfRTxnoZrQe63tq+rQnVgninHMahRmXkzyjK/uNhoKnIh8bXdTC/eCZ6
|
|
||||||
FBROqBYNL0TJb0HDv1FzcZS1UCUldRqTlvr6wZb0pfrp40fvztsqQgAh1t/Blg5i
|
|
||||||
Zv5+ESSlNs5rWbFTxtq+FbMW/ERYTrVfnMkBiLg4Gq0HwID9a5jvJatzrrno2s1m
|
|
||||||
OfZCT3HaE3tMZ6vvYuoamvLNdvdH+9KeTmBCursfNejt0rSGjIqfi6DvFJSayydQ
|
|
||||||
is5DMSTbCLGdKQmA85VfEQmlQ8v0232WDSd6gVfp2tthDEDHnCbgWkEd1vsTyS85
|
|
||||||
ubdt5v4CWGOWV+mu3bf8xM0CAwEAAaMTMBEwDwYDVR0TAQH/BAUwAwEB/zANBgkq
|
|
||||||
hkiG9w0BAQsFAAOCAQEARTjc2zV/ol1/LsSzZy6l1R0uFBmR2KumH+Se1Yq2vKpY
|
|
||||||
Dv6xmrvmjOUr5RBO77nRhIgdcQA+LyAg8ii2Dfzc8r1RTD+j1bYOxESXctBOBcXM
|
|
||||||
Chy6FEBydR6m7S8qQyL+caJWO1WZWp2tapcm6sUG1oRVznWtK1/SHKIzOBwsmJ07
|
|
||||||
79KsCJ6wf9tzD05EDTI2QhAObE9/thy+zc8l8cmv9A6p3jKkx9rwXUttSUqTn0CW
|
|
||||||
w45bgKg6+DDcrhZ+MATbzuTfhuA4NFUTzK7KeX9sMuOV03Zs8SA3VhAOXmu063M3
|
|
||||||
0f9X7P/0RmGTTp7GGCqEINcZdbLh3k7CpFb2Ox998Q==
|
|
||||||
-----END CERTIFICATE-----
|
|
|
@ -1,18 +0,0 @@
|
||||||
-----BEGIN CERTIFICATE-----
|
|
||||||
MIIC2zCCAcOgAwIBAgIUIlE8UAkqQ+6mbJDtrt9kkmi8aJYwDQYJKoZIhvcNAQEL
|
|
||||||
BQAwJDEQMA4GA1UECgwHU2NhbGl0eTEQMA4GA1UEAwwHUm9vdCBDQTAgFw0yMTA0
|
|
||||||
MDkwMDI4MTFaGA8yMTIxMDMxNjAwMjgxMVowKTEQMA4GA1UECgwHU2NhbGl0eTEV
|
|
||||||
MBMGA1UEAwwMcHlrbWlwLmxvY2FsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
|
|
||||||
CgKCAQEAtxr7pq/lnzVeZz4z52Yc3DeaPqjNfRSyW5cPUlT7ABXFb7+tja7K2C7u
|
|
||||||
DYVK+Q+2yJCQwYJY47aKJB++ewam9t2V8Xy0Z8S+0I2ImCwuyeihaD/f6uJZRzms
|
|
||||||
ycdECH22BA6tCPlQLnlboRiZzI6rcIvXAbUMvLvFm3nyYIs9qidExRnfyMjISknM
|
|
||||||
V+83LT5QW4IcHgKYqzdz2ZmOnk+f4wmMmitcivTdIZCL8Z0cxr7BJlOh5JZ/V5uj
|
|
||||||
WUXeNa+ttW0RKKBlg9T+wj0JvwoJBPZTmsMAy3tI9tjLg3DwGYKsflbFeU2tebXI
|
|
||||||
gncGFZ/dFxj331GGtq3kz1PzAUYf2wIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQB1
|
|
||||||
8HgJ0fu6/pCrDxAm90eESFjmaTFyTN8q00zhq4Cb3zAT9KMWzAygkZ9n4ZFgELPo
|
|
||||||
7kBE2H6RcDdoBmjVYd8HnBloDdYzYbncKgt5YBvxRaMSF4/l65BM8wjatyXErqnH
|
|
||||||
QLLTRe5AuF0/F0KtPeDQ2JFVu8dZ35W3fyKGPRsEdVOSCTHROmqpGhZCpscyUP4W
|
|
||||||
Hb0dBTESQ9mQHw14OCaaahARd0X5WdcA/E+m0fpGqj1rQCXS+PrRcSLe1E1hqPlK
|
|
||||||
q/hXSXD5nybwipktELvJCbB7l4HmJr2pIpldeR5+ef68Cs8hqs6DRlsJX9sK2ng+
|
|
||||||
TFe5v6SCarqZ9kFvr6Yp
|
|
||||||
-----END CERTIFICATE-----
|
|
|
@ -1,18 +0,0 @@
|
||||||
-----BEGIN CERTIFICATE-----
|
|
||||||
MIIC8zCCAdugAwIBAgIUBs6nVXQXhrFbClub3aSLg72/DiYwDQYJKoZIhvcNAQEL
|
|
||||||
BQAwJDEQMA4GA1UECgwHU2NhbGl0eTEQMA4GA1UEAwwHUm9vdCBDQTAgFw0yMTA0
|
|
||||||
MDkwMDI4MTFaGA8yMTIxMDMxNjAwMjgxMVowJTEQMA4GA1UECgwHU2NhbGl0eTER
|
|
||||||
MA8GA1UEAwwISm9obiBEb2UwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
|
|
||||||
AQC6neSYoBoWh/i2mBpduJnTlXacpJ0iQqLezvcGy8qR0s/48mtfV2IRGTNVsq4L
|
|
||||||
jLLRsPGt9KkJlUhHGWhG00cBGEsIiJiBUr+WrEsO04ME/Sk76kX8wk/t9Oljl7jt
|
|
||||||
UDnQUwshj+hRFe0iKAyE65JIutu5EiiNtOqMzbVgPNfNniAaGlrgwByJaS9arzsH
|
|
||||||
PVju9yZBYzYhwAMyYFcXUGrgvHRCHKmxBi4QmV7DX4TeN4l9TrCyEmqDev4PRFip
|
|
||||||
yR2Fh3WGSwWh45HgMT+Jp6Uv6yI4wMXWJAcNkHdx1OhjBoUQrkavvdeVEnCwjQ+p
|
|
||||||
SMLm0T4iNxedQWBtDM7ts4EjAgMBAAGjGjAYMBYGA1UdJQEB/wQMMAoGCCsGAQUF
|
|
||||||
BwMCMA0GCSqGSIb3DQEBCwUAA4IBAQCMi9HEhZc5jHJMj18Wq00fZy4O9XtjCe0J
|
|
||||||
nntW9tzi3rTQcQWKA7i9uVdDoCg+gMFVxWMvV7luFEUc/VYV1v8hFfbIFygzFsZY
|
|
||||||
xwv4GQaIwbsgzD+oziia53w0FSuNL0uE0MeKvrt3yzHxCxylHyl+TQd/UdAtAo+k
|
|
||||||
RL1sI0mBZx5qo6d1J7ZMCxzAGaT7KjnJvziFr/UbfSNnwDsxsUwGaI1ZeAxJN8DI
|
|
||||||
zTrg3f3lrrmHcauEgKnuQwIqaMZR6veG6RkjtcYSlJYID1irkE6njs7+wivOAkzt
|
|
||||||
fBt/0PD76FmAI0VArgU/zDB8dGyYzrq39W749LuEfm1TPmlnUtDr
|
|
||||||
-----END CERTIFICATE-----
|
|
|
@ -1,28 +0,0 @@
|
||||||
-----BEGIN PRIVATE KEY-----
|
|
||||||
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC6neSYoBoWh/i2
|
|
||||||
mBpduJnTlXacpJ0iQqLezvcGy8qR0s/48mtfV2IRGTNVsq4LjLLRsPGt9KkJlUhH
|
|
||||||
GWhG00cBGEsIiJiBUr+WrEsO04ME/Sk76kX8wk/t9Oljl7jtUDnQUwshj+hRFe0i
|
|
||||||
KAyE65JIutu5EiiNtOqMzbVgPNfNniAaGlrgwByJaS9arzsHPVju9yZBYzYhwAMy
|
|
||||||
YFcXUGrgvHRCHKmxBi4QmV7DX4TeN4l9TrCyEmqDev4PRFipyR2Fh3WGSwWh45Hg
|
|
||||||
MT+Jp6Uv6yI4wMXWJAcNkHdx1OhjBoUQrkavvdeVEnCwjQ+pSMLm0T4iNxedQWBt
|
|
||||||
DM7ts4EjAgMBAAECggEANNXdUeUKXdSzcycPV/ea/c+0XFcy8e9B46lfQTpTqQOx
|
|
||||||
xD8GbWD1L/gdk6baJgT43+ukEWdSsJbmdtLXti29Ta8OF2VtIDhIbCVtvs3dq3zt
|
|
||||||
vrvugsiVDr8nkP306qOrKrNIVIFE+igmEmSaXsu/h/33ladxeeV9/s2DC7NOOjWN
|
|
||||||
Mu4KYr5BBbu3qAavdzbrcz7Sch+GzsYqK/pBounCTQu3o9E4TSUcmcsasWmtHN3u
|
|
||||||
e6G2UjObdzEW7J0wWvvtJ0wHQUVRueHfqwqKf0dymcZ3xOlx3ZPhKPz5n4F1UGUt
|
|
||||||
RQaNazqs5SzZpUgDuPw4k8h/aCHK21Yexw/l4+O9KQKBgQD1WZSRK54zFoExBQgt
|
|
||||||
OZSBNZW3Ibti5lSiF0M0g+66yNZSWfPuABEH0tu5CXopdPDXo4kW8NLGEqQStWTX
|
|
||||||
RGK0DE9buEL3eebOfjIdS2IZ3t3dX3lMypplVCj4HzAgITlweSH1LLTyAtaaOpwa
|
|
||||||
jksqfcn5Zw+XGkyc6GBBVaZetQKBgQDCt6Xf/g26+zjvHscjdzsfBhnYvTOrr6+F
|
|
||||||
xqFFxOEOocGr+mL7UTAs+a9m/6lOWhlagk+m+TIZNL8o3IN7KFTYxPYPxTiewgVE
|
|
||||||
rIm3JBmPxRiPn01P3HrtjaqfzsXF30j3ele7ix5OxieZq4vsW7ZXP3GZE34a08Ov
|
|
||||||
12sE1DlvdwKBgQDzpYQOLhyqazzcqzyVfMrnDYmiFVN7QXTmiudobWRUBUIhAcdl
|
|
||||||
oJdJB7K/rJOuO704x+RJ7dnCbZyWH6EGzZifaGIemXuXO21jvpqR0NyZCGOXhUp2
|
|
||||||
YfS1j8AntwEZxyS9du2sBjui4gKvomiHTquChOxgSmKHEcznPTTpbN8MyQKBgF5F
|
|
||||||
LVCZniolkLXsL7tS8VOez4qoZ0i6wP7CYLf3joJX+/z4N023S9yqcaorItvlMRsp
|
|
||||||
tciAIyoi6F2vDRTmPNXJ3dtav4PVKVnLMs1w89MwOCjoljSQ6Q7zpGTEZenbpWbz
|
|
||||||
W2BYBS9cLjXu4MpoyInLFINo9YeleLs8TvrCiKAXAoGBANsduqLnlUW/f5zDb5Fe
|
|
||||||
SB51+KhBjsVIeYmU+8xtur9Z7IxZXK28wpoEsm7LmX7Va5dERjI+tItBiJ5+Unu1
|
|
||||||
Xs2ljDg35ARKHs0dWBJGpbnZg4dbT6xpIL4YMPXm1Zu++PgRpxPIMn646xqd8GlH
|
|
||||||
bavm6Km/fXNG58xus+EeLpV5
|
|
||||||
-----END PRIVATE KEY-----
|
|
|
@ -1,28 +0,0 @@
|
||||||
-----BEGIN PRIVATE KEY-----
|
|
||||||
MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQC3Gvumr+WfNV5n
|
|
||||||
PjPnZhzcN5o+qM19FLJblw9SVPsAFcVvv62NrsrYLu4NhUr5D7bIkJDBgljjtook
|
|
||||||
H757Bqb23ZXxfLRnxL7QjYiYLC7J6KFoP9/q4llHOazJx0QIfbYEDq0I+VAueVuh
|
|
||||||
GJnMjqtwi9cBtQy8u8WbefJgiz2qJ0TFGd/IyMhKScxX7zctPlBbghweApirN3PZ
|
|
||||||
mY6eT5/jCYyaK1yK9N0hkIvxnRzGvsEmU6Hkln9Xm6NZRd41r621bREooGWD1P7C
|
|
||||||
PQm/CgkE9lOawwDLe0j22MuDcPAZgqx+VsV5Ta15tciCdwYVn90XGPffUYa2reTP
|
|
||||||
U/MBRh/bAgMBAAECggEABCvcMcbuDztzBB0Zp5re63Fk1SqZS9Et4wJE+hYvhaf5
|
|
||||||
UHtoY8LoohYnnC0+MQBXpKgOdCoZBk8BRKNofnr/UL5pjQ/POFH2GuAujXDsO/NN
|
|
||||||
wgc6fapcaE/7DLm6ZgsfG2aOMJclaXmgScI6trtFUpIM+t/6A06vyMP1bpeddwPW
|
|
||||||
Fqu7NvpDiEcTRUGd+z1JooYgUhGgC7peYUx5+9zqFrwoDBKxnUOnz3BkDsXBy3qm
|
|
||||||
65Vu0BSjuJzf6vVMpNGUHY6JXjopVNWku+JAX0wD+iikOd5sziNVdIj1fnZ+IHIf
|
|
||||||
7G5h5owHpvSGzJFQ18/g5VHtJdCm+4WQSnbSJRsCAQKBgQDu4IH8yspyeH44fhoS
|
|
||||||
PAp/OtILqSP+Da0zAp2LbhrOgyzyuSTdEAYyptqjqHS6QkB1Bu1H44FS0BYUxRXc
|
|
||||||
iu2e9AndiLVCGngsE7TpA/ZVLN1B0LEZEHjM6p4d6zZM6iveKVnPAOkTWTBAgzCt
|
|
||||||
b31nj4jL8PdlPKQil1AMrOlRAQKBgQDEOwshzIdr2Iy6B/n4CuBViEtwnbAd5f/c
|
|
||||||
atA9bcfF8kCahokJsI4eCCLgBwDZpYKD+v0AwOBlacF6t6TX+vdlJsi5EP7uxZ22
|
|
||||||
ILsuWqVm/0H77PACuckc5/qLZoGGC81l0DhnpoeMEb6r/TKOo5xAK1gxdlwNNrq+
|
|
||||||
nP1zdZnU2wKBgBAS92xFUR4m0YeHpMV5WNN658t1FEDyNqdqE6PgQtmGpi2nG73s
|
|
||||||
aB5cb/X3TfOCpce6MZlWy8sAyZuYL4Jprte1YDySCHBsS43bvZ64b4kHvdPB8UjY
|
|
||||||
fOh9GSq2Oy8tysnmSm7NhuGQbNjKeyoQiIXBeNkQW/VqATl6qR5RPFoBAoGACNqV
|
|
||||||
JQBCd/Y8W0Ry3eM3vgQ5SyqCQMcY5UwYez0Rz3efvJknY72InAhH8o2+VxOlsOjJ
|
|
||||||
M5iAR3MfHLdeg7Q6J2E5m0gOCJ34ALi3WV8TqXMI+iH1rlnNnjVFU7bbTz4HFXnw
|
|
||||||
oZSc9w/x53a0KkVtjmOmRg0OGDaI9ILG2MfMmhMCgYB8ZqJtX8qZ2TqKU3XdLZ4z
|
|
||||||
T2N7xMFuKohWP420r5jKm3Xw85IC+y1SUTB9XGcL79r2eJzmzmdKQ3A3sf3oyUH3
|
|
||||||
RdYWxtKcZ5PAE8hVRtn1ETZqUgxASGOUn/6w0npkYSOXPU5bc0W6RSLkjES0i+c3
|
|
||||||
fv3OMNI8qpmQhEjpHHQS1g==
|
|
||||||
-----END PRIVATE KEY-----
|
|
|
@ -1,35 +0,0 @@
|
||||||
name: Test alerts
|
|
||||||
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
branches-ignore:
|
|
||||||
- 'development/**'
|
|
||||||
- 'q/*/**'
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
run-alert-tests:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
strategy:
|
|
||||||
matrix:
|
|
||||||
tests:
|
|
||||||
- name: 1 minute interval tests
|
|
||||||
file: monitoring/alerts.test.yaml
|
|
||||||
|
|
||||||
- name: 10 seconds interval tests
|
|
||||||
file: monitoring/alerts.10s.test.yaml
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
|
|
||||||
- name: Render and test ${{ matrix.tests.name }}
|
|
||||||
uses: scality/action-prom-render-test@1.0.3
|
|
||||||
with:
|
|
||||||
alert_file_path: monitoring/alerts.yaml
|
|
||||||
test_file_path: ${{ matrix.tests.file }}
|
|
||||||
alert_inputs: |
|
|
||||||
namespace=zenko
|
|
||||||
service=artesca-data-connector-s3api-metrics
|
|
||||||
reportJob=artesca-data-ops-report-handler
|
|
||||||
replicas=3
|
|
||||||
github_token: ${{ secrets.GITHUB_TOKEN }}
|
|
|
@ -1,25 +0,0 @@
|
||||||
---
|
|
||||||
name: codeQL
|
|
||||||
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
branches: [w/**, q/*]
|
|
||||||
pull_request:
|
|
||||||
branches: [development/*, stabilization/*, hotfix/*]
|
|
||||||
workflow_dispatch:
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
analyze:
|
|
||||||
name: Static analysis with CodeQL
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
steps:
|
|
||||||
- name: Checkout code
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
|
|
||||||
- name: Initialize CodeQL
|
|
||||||
uses: github/codeql-action/init@v3
|
|
||||||
with:
|
|
||||||
languages: javascript, python, ruby
|
|
||||||
|
|
||||||
- name: Build and analyze
|
|
||||||
uses: github/codeql-action/analyze@v3
|
|
|
@ -1,16 +0,0 @@
|
||||||
---
|
|
||||||
name: dependency review
|
|
||||||
|
|
||||||
on:
|
|
||||||
pull_request:
|
|
||||||
branches: [development/*, stabilization/*, hotfix/*]
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
dependency-review:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
steps:
|
|
||||||
- name: 'Checkout Repository'
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
|
|
||||||
- name: 'Dependency Review'
|
|
||||||
uses: actions/dependency-review-action@v4
|
|
|
@ -1,80 +0,0 @@
|
||||||
---
|
|
||||||
name: release
|
|
||||||
run-name: release ${{ inputs.tag }}
|
|
||||||
|
|
||||||
on:
|
|
||||||
workflow_dispatch:
|
|
||||||
inputs:
|
|
||||||
tag:
|
|
||||||
description: 'Tag to be released'
|
|
||||||
required: true
|
|
||||||
|
|
||||||
env:
|
|
||||||
PROJECT_NAME: ${{ github.event.repository.name }}
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
build-federation-image:
|
|
||||||
runs-on: ubuntu-20.04
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
- name: Set up Docker Buildx
|
|
||||||
uses: docker/setup-buildx-action@v3
|
|
||||||
- name: Login to GitHub Registry
|
|
||||||
uses: docker/login-action@v3
|
|
||||||
with:
|
|
||||||
registry: ghcr.io
|
|
||||||
username: ${{ github.repository_owner }}
|
|
||||||
password: ${{ github.token }}
|
|
||||||
- name: Build and push image for federation
|
|
||||||
uses: docker/build-push-action@v5
|
|
||||||
with:
|
|
||||||
push: true
|
|
||||||
context: .
|
|
||||||
file: images/svc-base/Dockerfile
|
|
||||||
tags: |
|
|
||||||
ghcr.io/${{ github.repository }}:${{ github.event.inputs.tag }}-svc-base
|
|
||||||
cache-from: type=gha,scope=federation
|
|
||||||
cache-to: type=gha,mode=max,scope=federation
|
|
||||||
|
|
||||||
release:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
|
|
||||||
- name: Set up Docker Buildk
|
|
||||||
uses: docker/setup-buildx-action@v3
|
|
||||||
|
|
||||||
- name: Login to Registry
|
|
||||||
uses: docker/login-action@v3
|
|
||||||
with:
|
|
||||||
registry: ghcr.io
|
|
||||||
username: ${{ github.repository_owner }}
|
|
||||||
password: ${{ github.token }}
|
|
||||||
|
|
||||||
- name: Push dashboards into the production namespace
|
|
||||||
run: |
|
|
||||||
oras push ghcr.io/${{ github.repository }}/${{ env.PROJECT_NAME }}-dashboards:${{ github.event.inputs.tag }} \
|
|
||||||
dashboard.json:application/grafana-dashboard+json \
|
|
||||||
alerts.yaml:application/prometheus-alerts+yaml
|
|
||||||
working-directory: monitoring
|
|
||||||
|
|
||||||
- name: Build and push
|
|
||||||
uses: docker/build-push-action@v5
|
|
||||||
with:
|
|
||||||
context: .
|
|
||||||
push: true
|
|
||||||
tags: ghcr.io/${{ github.repository }}:${{ github.event.inputs.tag }}
|
|
||||||
cache-from: type=gha
|
|
||||||
cache-to: type=gha,mode=max
|
|
||||||
|
|
||||||
- name: Create Release
|
|
||||||
uses: softprops/action-gh-release@v2
|
|
||||||
env:
|
|
||||||
GITHUB_TOKEN: ${{ github.token }}
|
|
||||||
with:
|
|
||||||
name: Release ${{ github.event.inputs.tag }}
|
|
||||||
tag_name: ${{ github.event.inputs.tag }}
|
|
||||||
generate_release_notes: true
|
|
||||||
target_commitish: ${{ github.sha }}
|
|
|
@ -1,533 +0,0 @@
|
||||||
---
|
|
||||||
name: tests
|
|
||||||
|
|
||||||
on:
|
|
||||||
workflow_dispatch:
|
|
||||||
|
|
||||||
push:
|
|
||||||
branches-ignore:
|
|
||||||
- 'development/**'
|
|
||||||
- 'q/*/**'
|
|
||||||
|
|
||||||
env:
|
|
||||||
# Secrets
|
|
||||||
azurebackend_AZURE_STORAGE_ACCESS_KEY: >-
|
|
||||||
${{ secrets.AZURE_STORAGE_ACCESS_KEY }}
|
|
||||||
azurebackend_AZURE_STORAGE_ACCOUNT_NAME: >-
|
|
||||||
${{ secrets.AZURE_STORAGE_ACCOUNT_NAME }}
|
|
||||||
azurebackend_AZURE_STORAGE_ENDPOINT: >-
|
|
||||||
${{ secrets.AZURE_STORAGE_ENDPOINT }}
|
|
||||||
azurebackend2_AZURE_STORAGE_ACCESS_KEY: >-
|
|
||||||
${{ secrets.AZURE_STORAGE_ACCESS_KEY_2 }}
|
|
||||||
azurebackend2_AZURE_STORAGE_ACCOUNT_NAME: >-
|
|
||||||
${{ secrets.AZURE_STORAGE_ACCOUNT_NAME_2 }}
|
|
||||||
azurebackend2_AZURE_STORAGE_ENDPOINT: >-
|
|
||||||
${{ secrets.AZURE_STORAGE_ENDPOINT_2 }}
|
|
||||||
azurebackendmismatch_AZURE_STORAGE_ACCESS_KEY: >-
|
|
||||||
${{ secrets.AZURE_STORAGE_ACCESS_KEY }}
|
|
||||||
azurebackendmismatch_AZURE_STORAGE_ACCOUNT_NAME: >-
|
|
||||||
${{ secrets.AZURE_STORAGE_ACCOUNT_NAME }}
|
|
||||||
azurebackendmismatch_AZURE_STORAGE_ENDPOINT: >-
|
|
||||||
${{ secrets.AZURE_STORAGE_ENDPOINT }}
|
|
||||||
azurenonexistcontainer_AZURE_STORAGE_ACCESS_KEY: >-
|
|
||||||
${{ secrets.AZURE_STORAGE_ACCESS_KEY }}
|
|
||||||
azurenonexistcontainer_AZURE_STORAGE_ACCOUNT_NAME: >-
|
|
||||||
${{ secrets.AZURE_STORAGE_ACCOUNT_NAME }}
|
|
||||||
azurenonexistcontainer_AZURE_STORAGE_ENDPOINT: >-
|
|
||||||
${{ secrets.AZURE_STORAGE_ENDPOINT }}
|
|
||||||
azuretest_AZURE_BLOB_ENDPOINT: "${{ secrets.AZURE_STORAGE_ENDPOINT }}"
|
|
||||||
b2backend_B2_ACCOUNT_ID: "${{ secrets.B2BACKEND_B2_ACCOUNT_ID }}"
|
|
||||||
b2backend_B2_STORAGE_ACCESS_KEY: >-
|
|
||||||
${{ secrets.B2BACKEND_B2_STORAGE_ACCESS_KEY }}
|
|
||||||
GOOGLE_SERVICE_EMAIL: "${{ secrets.GCP_SERVICE_EMAIL }}"
|
|
||||||
GOOGLE_SERVICE_KEY: "${{ secrets.GCP_SERVICE_KEY }}"
|
|
||||||
AWS_S3_BACKEND_ACCESS_KEY: "${{ secrets.AWS_S3_BACKEND_ACCESS_KEY }}"
|
|
||||||
AWS_S3_BACKEND_SECRET_KEY: "${{ secrets.AWS_S3_BACKEND_SECRET_KEY }}"
|
|
||||||
AWS_S3_BACKEND_ACCESS_KEY_2: "${{ secrets.AWS_S3_BACKEND_ACCESS_KEY_2 }}"
|
|
||||||
AWS_S3_BACKEND_SECRET_KEY_2: "${{ secrets.AWS_S3_BACKEND_SECRET_KEY_2 }}"
|
|
||||||
AWS_GCP_BACKEND_ACCESS_KEY: "${{ secrets.AWS_GCP_BACKEND_ACCESS_KEY }}"
|
|
||||||
AWS_GCP_BACKEND_SECRET_KEY: "${{ secrets.AWS_GCP_BACKEND_SECRET_KEY }}"
|
|
||||||
AWS_GCP_BACKEND_ACCESS_KEY_2: "${{ secrets.AWS_GCP_BACKEND_ACCESS_KEY_2 }}"
|
|
||||||
AWS_GCP_BACKEND_SECRET_KEY_2: "${{ secrets.AWS_GCP_BACKEND_SECRET_KEY_2 }}"
|
|
||||||
b2backend_B2_STORAGE_ENDPOINT: "${{ secrets.B2BACKEND_B2_STORAGE_ENDPOINT }}"
|
|
||||||
gcpbackend2_GCP_SERVICE_EMAIL: "${{ secrets.GCP2_SERVICE_EMAIL }}"
|
|
||||||
gcpbackend2_GCP_SERVICE_KEY: "${{ secrets.GCP2_SERVICE_KEY }}"
|
|
||||||
gcpbackend2_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
|
|
||||||
gcpbackend_GCP_SERVICE_EMAIL: "${{ secrets.GCP_SERVICE_EMAIL }}"
|
|
||||||
gcpbackend_GCP_SERVICE_KEY: "${{ secrets.GCP_SERVICE_KEY }}"
|
|
||||||
gcpbackendmismatch_GCP_SERVICE_EMAIL: >-
|
|
||||||
${{ secrets.GCPBACKENDMISMATCH_GCP_SERVICE_EMAIL }}
|
|
||||||
gcpbackendmismatch_GCP_SERVICE_KEY: >-
|
|
||||||
${{ secrets.GCPBACKENDMISMATCH_GCP_SERVICE_KEY }}
|
|
||||||
gcpbackend_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
|
|
||||||
gcpbackendmismatch_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
|
|
||||||
gcpbackendnoproxy_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
|
|
||||||
gcpbackendproxy_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
|
|
||||||
# Configs
|
|
||||||
ENABLE_LOCAL_CACHE: "true"
|
|
||||||
REPORT_TOKEN: "report-token-1"
|
|
||||||
REMOTE_MANAGEMENT_DISABLE: "1"
|
|
||||||
# https://github.com/git-lfs/git-lfs/issues/5749
|
|
||||||
GIT_CLONE_PROTECTION_ACTIVE: 'false'
|
|
||||||
jobs:
|
|
||||||
linting-coverage:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
- uses: actions/setup-node@v4
|
|
||||||
with:
|
|
||||||
node-version: '16'
|
|
||||||
cache: yarn
|
|
||||||
- name: install dependencies
|
|
||||||
run: yarn install --frozen-lockfile --network-concurrency 1
|
|
||||||
- uses: actions/setup-python@v5
|
|
||||||
with:
|
|
||||||
python-version: '3.9'
|
|
||||||
- uses: actions/cache@v4
|
|
||||||
with:
|
|
||||||
path: ~/.cache/pip
|
|
||||||
key: ${{ runner.os }}-pip
|
|
||||||
- name: Install python deps
|
|
||||||
run: pip install flake8
|
|
||||||
- name: Lint Javascript
|
|
||||||
run: yarn run --silent lint -- --max-warnings 0
|
|
||||||
- name: Lint Markdown
|
|
||||||
run: yarn run --silent lint_md
|
|
||||||
- name: Lint python
|
|
||||||
run: flake8 $(git ls-files "*.py")
|
|
||||||
- name: Lint Yaml
|
|
||||||
run: yamllint -c yamllint.yml $(git ls-files "*.yml")
|
|
||||||
- name: Unit Coverage
|
|
||||||
run: |
|
|
||||||
set -ex
|
|
||||||
mkdir -p $CIRCLE_TEST_REPORTS/unit
|
|
||||||
yarn test
|
|
||||||
yarn run test_legacy_location
|
|
||||||
env:
|
|
||||||
S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
|
|
||||||
CIRCLE_TEST_REPORTS: /tmp
|
|
||||||
CIRCLE_ARTIFACTS: /tmp
|
|
||||||
CI_REPORTS: /tmp
|
|
||||||
- name: Unit Coverage logs
|
|
||||||
run: find /tmp/unit -exec cat {} \;
|
|
||||||
- name: preparing junit files for upload
|
|
||||||
run: |
|
|
||||||
mkdir -p artifacts/junit
|
|
||||||
find . -name "*junit*.xml" -exec cp {} artifacts/junit/ ";"
|
|
||||||
if: always()
|
|
||||||
- name: Upload files to artifacts
|
|
||||||
uses: scality/action-artifacts@v4
|
|
||||||
with:
|
|
||||||
method: upload
|
|
||||||
url: https://artifacts.scality.net
|
|
||||||
user: ${{ secrets.ARTIFACTS_USER }}
|
|
||||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
|
||||||
source: artifacts
|
|
||||||
if: always()
|
|
||||||
|
|
||||||
build:
|
|
||||||
runs-on: ubuntu-20.04
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
packages: write
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
- name: Set up Docker Buildx
|
|
||||||
uses: docker/setup-buildx-action@v3
|
|
||||||
- name: Login to GitHub Registry
|
|
||||||
uses: docker/login-action@v3
|
|
||||||
with:
|
|
||||||
registry: ghcr.io
|
|
||||||
username: ${{ github.repository_owner }}
|
|
||||||
password: ${{ github.token }}
|
|
||||||
- name: Build and push cloudserver image
|
|
||||||
uses: docker/build-push-action@v5
|
|
||||||
with:
|
|
||||||
push: true
|
|
||||||
context: .
|
|
||||||
provenance: false
|
|
||||||
tags: |
|
|
||||||
ghcr.io/${{ github.repository }}:${{ github.sha }}
|
|
||||||
labels: |
|
|
||||||
git.repository=${{ github.repository }}
|
|
||||||
git.commit-sha=${{ github.sha }}
|
|
||||||
cache-from: type=gha,scope=cloudserver
|
|
||||||
cache-to: type=gha,mode=max,scope=cloudserver
|
|
||||||
- name: Build and push pykmip image
|
|
||||||
uses: docker/build-push-action@v5
|
|
||||||
with:
|
|
||||||
push: true
|
|
||||||
context: .github/pykmip
|
|
||||||
tags: |
|
|
||||||
ghcr.io/${{ github.repository }}/pykmip:${{ github.sha }}
|
|
||||||
labels: |
|
|
||||||
git.repository=${{ github.repository }}
|
|
||||||
git.commit-sha=${{ github.sha }}
|
|
||||||
cache-from: type=gha,scope=pykmip
|
|
||||||
cache-to: type=gha,mode=max,scope=pykmip
|
|
||||||
- name: Build and push MongoDB
|
|
||||||
uses: docker/build-push-action@v5
|
|
||||||
with:
|
|
||||||
push: true
|
|
||||||
context: .github/docker/mongodb
|
|
||||||
tags: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
|
||||||
cache-from: type=gha,scope=mongodb
|
|
||||||
cache-to: type=gha,mode=max,scope=mongodb
|
|
||||||
|
|
||||||
multiple-backend:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
needs: build
|
|
||||||
env:
|
|
||||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
|
||||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
|
||||||
S3BACKEND: mem
|
|
||||||
S3_LOCATION_FILE: /usr/src/app/tests/locationConfig/locationConfigTests.json
|
|
||||||
S3DATA: multiple
|
|
||||||
JOB_NAME: ${{ github.job }}
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
- name: Login to Registry
|
|
||||||
uses: docker/login-action@v3
|
|
||||||
with:
|
|
||||||
registry: ghcr.io
|
|
||||||
username: ${{ github.repository_owner }}
|
|
||||||
password: ${{ github.token }}
|
|
||||||
- name: Setup CI environment
|
|
||||||
uses: ./.github/actions/setup-ci
|
|
||||||
- name: Setup CI services
|
|
||||||
run: docker compose --profile sproxyd up -d
|
|
||||||
working-directory: .github/docker
|
|
||||||
- name: Run multiple backend test
|
|
||||||
run: |-
|
|
||||||
set -o pipefail;
|
|
||||||
bash wait_for_local_port.bash 8000 40
|
|
||||||
bash wait_for_local_port.bash 81 40
|
|
||||||
yarn run multiple_backend_test | tee /tmp/artifacts/${{ github.job }}/tests.log
|
|
||||||
env:
|
|
||||||
S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
|
|
||||||
- name: Upload logs to artifacts
|
|
||||||
uses: scality/action-artifacts@v4
|
|
||||||
with:
|
|
||||||
method: upload
|
|
||||||
url: https://artifacts.scality.net
|
|
||||||
user: ${{ secrets.ARTIFACTS_USER }}
|
|
||||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
|
||||||
source: /tmp/artifacts
|
|
||||||
if: always()
|
|
||||||
|
|
||||||
mongo-v0-ft-tests:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
needs: build
|
|
||||||
env:
|
|
||||||
S3BACKEND: mem
|
|
||||||
MPU_TESTING: "yes"
|
|
||||||
S3METADATA: mongodb
|
|
||||||
S3KMS: file
|
|
||||||
S3_LOCATION_FILE: /usr/src/app/tests/locationConfig/locationConfigTests.json
|
|
||||||
DEFAULT_BUCKET_KEY_FORMAT: v0
|
|
||||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
|
||||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
|
||||||
JOB_NAME: ${{ github.job }}
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
- name: Setup CI environment
|
|
||||||
uses: ./.github/actions/setup-ci
|
|
||||||
- name: Setup CI services
|
|
||||||
run: docker compose --profile mongo up -d
|
|
||||||
working-directory: .github/docker
|
|
||||||
- name: Run functional tests
|
|
||||||
run: |-
|
|
||||||
set -o pipefail;
|
|
||||||
bash wait_for_local_port.bash 8000 40
|
|
||||||
yarn run ft_test | tee /tmp/artifacts/${{ github.job }}/tests.log
|
|
||||||
env:
|
|
||||||
S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
|
|
||||||
- name: Upload logs to artifacts
|
|
||||||
uses: scality/action-artifacts@v4
|
|
||||||
with:
|
|
||||||
method: upload
|
|
||||||
url: https://artifacts.scality.net
|
|
||||||
user: ${{ secrets.ARTIFACTS_USER }}
|
|
||||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
|
||||||
source: /tmp/artifacts
|
|
||||||
if: always()
|
|
||||||
|
|
||||||
mongo-v1-ft-tests:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
needs: build
|
|
||||||
env:
|
|
||||||
S3BACKEND: mem
|
|
||||||
MPU_TESTING: "yes"
|
|
||||||
S3METADATA: mongodb
|
|
||||||
S3KMS: file
|
|
||||||
S3_LOCATION_FILE: /usr/src/app/tests/locationConfig/locationConfigTests.json
|
|
||||||
DEFAULT_BUCKET_KEY_FORMAT: v1
|
|
||||||
METADATA_MAX_CACHED_BUCKETS: 1
|
|
||||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
|
||||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
|
||||||
JOB_NAME: ${{ github.job }}
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
- name: Setup CI environment
|
|
||||||
uses: ./.github/actions/setup-ci
|
|
||||||
- name: Setup CI services
|
|
||||||
run: docker compose --profile mongo up -d
|
|
||||||
working-directory: .github/docker
|
|
||||||
- name: Run functional tests
|
|
||||||
run: |-
|
|
||||||
set -o pipefail;
|
|
||||||
bash wait_for_local_port.bash 8000 40
|
|
||||||
yarn run ft_test | tee /tmp/artifacts/${{ github.job }}/tests.log
|
|
||||||
yarn run ft_mixed_bucket_format_version | tee /tmp/artifacts/${{ github.job }}/mixed-tests.log
|
|
||||||
env:
|
|
||||||
S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
|
|
||||||
- name: Upload logs to artifacts
|
|
||||||
uses: scality/action-artifacts@v4
|
|
||||||
with:
|
|
||||||
method: upload
|
|
||||||
url: https://artifacts.scality.net
|
|
||||||
user: ${{ secrets.ARTIFACTS_USER }}
|
|
||||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
|
||||||
source: /tmp/artifacts
|
|
||||||
if: always()
|
|
||||||
|
|
||||||
file-ft-tests:
|
|
||||||
strategy:
|
|
||||||
matrix:
|
|
||||||
include:
|
|
||||||
- job-name: file-ft-tests
|
|
||||||
name: ${{ matrix.job-name }}
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
needs: build
|
|
||||||
env:
|
|
||||||
S3BACKEND: file
|
|
||||||
S3VAULT: mem
|
|
||||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
|
||||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
|
||||||
MPU_TESTING: "yes"
|
|
||||||
JOB_NAME: ${{ matrix.job-name }}
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
- name: Setup CI environment
|
|
||||||
uses: ./.github/actions/setup-ci
|
|
||||||
- name: Setup matrix job artifacts directory
|
|
||||||
shell: bash
|
|
||||||
run: |
|
|
||||||
set -exu
|
|
||||||
mkdir -p /tmp/artifacts/${{ matrix.job-name }}/
|
|
||||||
- name: Setup CI services
|
|
||||||
run: docker compose up -d
|
|
||||||
working-directory: .github/docker
|
|
||||||
- name: Run file ft tests
|
|
||||||
run: |-
|
|
||||||
set -o pipefail;
|
|
||||||
bash wait_for_local_port.bash 8000 40
|
|
||||||
yarn run ft_test | tee /tmp/artifacts/${{ matrix.job-name }}/tests.log
|
|
||||||
- name: Upload logs to artifacts
|
|
||||||
uses: scality/action-artifacts@v4
|
|
||||||
with:
|
|
||||||
method: upload
|
|
||||||
url: https://artifacts.scality.net
|
|
||||||
user: ${{ secrets.ARTIFACTS_USER }}
|
|
||||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
|
||||||
source: /tmp/artifacts
|
|
||||||
if: always()
|
|
||||||
|
|
||||||
utapi-v2-tests:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
needs: build
|
|
||||||
env:
|
|
||||||
ENABLE_UTAPI_V2: t
|
|
||||||
S3BACKEND: mem
|
|
||||||
BUCKET_DENY_FILTER: utapi-event-filter-deny-bucket
|
|
||||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
|
||||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
|
||||||
JOB_NAME: ${{ github.job }}
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
- name: Setup CI environment
|
|
||||||
uses: ./.github/actions/setup-ci
|
|
||||||
- name: Setup CI services
|
|
||||||
run: docker compose up -d
|
|
||||||
working-directory: .github/docker
|
|
||||||
- name: Run file utapi v2 tests
|
|
||||||
run: |-
|
|
||||||
set -ex -o pipefail;
|
|
||||||
bash wait_for_local_port.bash 8000 40
|
|
||||||
yarn run test_utapi_v2 | tee /tmp/artifacts/${{ github.job }}/tests.log
|
|
||||||
- name: Upload logs to artifacts
|
|
||||||
uses: scality/action-artifacts@v4
|
|
||||||
with:
|
|
||||||
method: upload
|
|
||||||
url: https://artifacts.scality.net
|
|
||||||
user: ${{ secrets.ARTIFACTS_USER }}
|
|
||||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
|
||||||
source: /tmp/artifacts
|
|
||||||
if: always()
|
|
||||||
|
|
||||||
quota-tests:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
needs: build
|
|
||||||
strategy:
|
|
||||||
matrix:
|
|
||||||
inflights:
|
|
||||||
- name: "With Inflights"
|
|
||||||
value: "true"
|
|
||||||
- name: "Without Inflights"
|
|
||||||
value: "false"
|
|
||||||
env:
|
|
||||||
S3METADATA: mongodb
|
|
||||||
S3BACKEND: mem
|
|
||||||
S3QUOTA: scuba
|
|
||||||
QUOTA_ENABLE_INFLIGHTS: ${{ matrix.inflights.value }}
|
|
||||||
SCUBA_HOST: localhost
|
|
||||||
SCUBA_PORT: 8100
|
|
||||||
SCUBA_HEALTHCHECK_FREQUENCY: 100
|
|
||||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
|
||||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
|
||||||
JOB_NAME: ${{ github.job }}
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
- name: Setup CI environment
|
|
||||||
uses: ./.github/actions/setup-ci
|
|
||||||
- name: Setup CI services
|
|
||||||
run: docker compose --profile mongo up -d
|
|
||||||
working-directory: .github/docker
|
|
||||||
- name: Run quota tests
|
|
||||||
run: |-
|
|
||||||
set -ex -o pipefail;
|
|
||||||
bash wait_for_local_port.bash 8000 40
|
|
||||||
yarn run test_quota | tee /tmp/artifacts/${{ github.job }}/tests.log
|
|
||||||
- name: Upload logs to artifacts
|
|
||||||
uses: scality/action-artifacts@v4
|
|
||||||
with:
|
|
||||||
method: upload
|
|
||||||
url: https://artifacts.scality.net
|
|
||||||
user: ${{ secrets.ARTIFACTS_USER }}
|
|
||||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
|
||||||
source: /tmp/artifacts
|
|
||||||
if: always()
|
|
||||||
|
|
||||||
kmip-ft-tests:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
needs: build
|
|
||||||
env:
|
|
||||||
S3BACKEND: file
|
|
||||||
S3VAULT: mem
|
|
||||||
MPU_TESTING: "yes"
|
|
||||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
|
||||||
PYKMIP_IMAGE: ghcr.io/${{ github.repository }}/pykmip:${{ github.sha }}
|
|
||||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
|
||||||
JOB_NAME: ${{ github.job }}
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
- name: Setup CI environment
|
|
||||||
uses: ./.github/actions/setup-ci
|
|
||||||
- name: Copy KMIP certs
|
|
||||||
run: cp -r ./certs /tmp/ssl-kmip
|
|
||||||
working-directory: .github/pykmip
|
|
||||||
- name: Setup CI services
|
|
||||||
run: docker compose --profile pykmip up -d
|
|
||||||
working-directory: .github/docker
|
|
||||||
- name: Run file KMIP tests
|
|
||||||
run: |-
|
|
||||||
set -ex -o pipefail;
|
|
||||||
bash wait_for_local_port.bash 8000 40
|
|
||||||
bash wait_for_local_port.bash 5696 40
|
|
||||||
yarn run ft_kmip | tee /tmp/artifacts/${{ github.job }}/tests.log
|
|
||||||
- name: Upload logs to artifacts
|
|
||||||
uses: scality/action-artifacts@v4
|
|
||||||
with:
|
|
||||||
method: upload
|
|
||||||
url: https://artifacts.scality.net
|
|
||||||
user: ${{ secrets.ARTIFACTS_USER }}
|
|
||||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
|
||||||
source: /tmp/artifacts
|
|
||||||
if: always()
|
|
||||||
|
|
||||||
ceph-backend-test:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
needs: build
|
|
||||||
env:
|
|
||||||
S3BACKEND: mem
|
|
||||||
S3DATA: multiple
|
|
||||||
S3KMS: file
|
|
||||||
CI_CEPH: 'true'
|
|
||||||
MPU_TESTING: "yes"
|
|
||||||
S3_LOCATION_FILE: /usr/src/app/tests/locationConfig/locationConfigCeph.json
|
|
||||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
|
||||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
|
||||||
JOB_NAME: ${{ github.job }}
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
- name: Login to GitHub Registry
|
|
||||||
uses: docker/login-action@v3
|
|
||||||
with:
|
|
||||||
registry: ghcr.io
|
|
||||||
username: ${{ github.repository_owner }}
|
|
||||||
password: ${{ github.token }}
|
|
||||||
- name: Setup CI environment
|
|
||||||
uses: ./.github/actions/setup-ci
|
|
||||||
- uses: ruby/setup-ruby@v1
|
|
||||||
with:
|
|
||||||
ruby-version: '2.5.9'
|
|
||||||
- name: Install Ruby dependencies
|
|
||||||
run: |
|
|
||||||
gem install nokogiri:1.12.5 excon:0.109.0 fog-aws:1.3.0 json mime-types:3.1 rspec:3.5
|
|
||||||
- name: Install Java dependencies
|
|
||||||
run: |
|
|
||||||
sudo apt-get update && sudo apt-get install -y --fix-missing default-jdk maven
|
|
||||||
- name: Setup CI services
|
|
||||||
run: docker compose --profile ceph up -d
|
|
||||||
working-directory: .github/docker
|
|
||||||
env:
|
|
||||||
S3METADATA: mongodb
|
|
||||||
- name: Run Ceph multiple backend tests
|
|
||||||
run: |-
|
|
||||||
set -ex -o pipefail;
|
|
||||||
bash .github/ceph/wait_for_ceph.sh
|
|
||||||
bash wait_for_local_port.bash 27018 40
|
|
||||||
bash wait_for_local_port.bash 8000 40
|
|
||||||
yarn run multiple_backend_test | tee /tmp/artifacts/${{ github.job }}/multibackend-tests.log
|
|
||||||
env:
|
|
||||||
S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
|
|
||||||
S3METADATA: mem
|
|
||||||
- name: Run Java tests
|
|
||||||
run: |-
|
|
||||||
set -ex -o pipefail;
|
|
||||||
mvn test | tee /tmp/artifacts/${{ github.job }}/java-tests.log
|
|
||||||
working-directory: tests/functional/jaws
|
|
||||||
- name: Run Ruby tests
|
|
||||||
run: |-
|
|
||||||
set -ex -o pipefail;
|
|
||||||
rspec -fd --backtrace tests.rb | tee /tmp/artifacts/${{ github.job }}/ruby-tests.log
|
|
||||||
working-directory: tests/functional/fog
|
|
||||||
- name: Run Javascript AWS SDK tests
|
|
||||||
run: |-
|
|
||||||
set -ex -o pipefail;
|
|
||||||
yarn run ft_awssdk | tee /tmp/artifacts/${{ github.job }}/js-awssdk-tests.log;
|
|
||||||
yarn run ft_s3cmd | tee /tmp/artifacts/${{ github.job }}/js-s3cmd-tests.log;
|
|
||||||
env:
|
|
||||||
S3_LOCATION_FILE: tests/locationConfig/locationConfigCeph.json
|
|
||||||
S3BACKEND: file
|
|
||||||
S3VAULT: mem
|
|
||||||
S3METADATA: mongodb
|
|
||||||
- name: Upload logs to artifacts
|
|
||||||
uses: scality/action-artifacts@v4
|
|
||||||
with:
|
|
||||||
method: upload
|
|
||||||
url: https://artifacts.scality.net
|
|
||||||
user: ${{ secrets.ARTIFACTS_USER }}
|
|
||||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
|
||||||
source: /tmp/artifacts
|
|
||||||
if: always()
|
|
|
@ -22,14 +22,6 @@ coverage
|
||||||
# Compiled binary addons (http://nodejs.org/api/addons.html)
|
# Compiled binary addons (http://nodejs.org/api/addons.html)
|
||||||
build/Release
|
build/Release
|
||||||
|
|
||||||
# Sphinx build dir
|
|
||||||
_build
|
|
||||||
|
|
||||||
# Dependency directory
|
# Dependency directory
|
||||||
# https://www.npmjs.org/doc/misc/npm-faq.html#should-i-check-my-node_modules-folder-into-git
|
# https://www.npmjs.org/doc/misc/npm-faq.html#should-i-check-my-node_modules-folder-into-git
|
||||||
node_modules
|
node_modules
|
||||||
yarn.lock
|
|
||||||
.tox
|
|
||||||
|
|
||||||
# Junit directory
|
|
||||||
junit
|
|
||||||
|
|
60
Dockerfile
60
Dockerfile
|
@ -1,60 +0,0 @@
|
||||||
ARG NODE_VERSION=16.20-bullseye-slim
|
|
||||||
|
|
||||||
FROM node:${NODE_VERSION} as builder
|
|
||||||
|
|
||||||
WORKDIR /usr/src/app
|
|
||||||
|
|
||||||
RUN apt-get update \
|
|
||||||
&& apt-get install -y --no-install-recommends \
|
|
||||||
build-essential \
|
|
||||||
ca-certificates \
|
|
||||||
curl \
|
|
||||||
git \
|
|
||||||
gnupg2 \
|
|
||||||
jq \
|
|
||||||
python3 \
|
|
||||||
ssh \
|
|
||||||
wget \
|
|
||||||
libffi-dev \
|
|
||||||
zlib1g-dev \
|
|
||||||
&& apt-get clean \
|
|
||||||
&& mkdir -p /root/ssh \
|
|
||||||
&& ssh-keyscan -H github.com > /root/ssh/known_hosts
|
|
||||||
|
|
||||||
ENV PYTHON=python3
|
|
||||||
COPY package.json yarn.lock /usr/src/app/
|
|
||||||
RUN npm install typescript -g
|
|
||||||
RUN yarn install --production --ignore-optional --frozen-lockfile --ignore-engines --network-concurrency 1
|
|
||||||
|
|
||||||
################################################################################
|
|
||||||
FROM node:${NODE_VERSION}
|
|
||||||
|
|
||||||
RUN apt-get update && \
|
|
||||||
apt-get install -y --no-install-recommends \
|
|
||||||
jq \
|
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
|
||||||
|
|
||||||
ENV NO_PROXY localhost,127.0.0.1
|
|
||||||
ENV no_proxy localhost,127.0.0.1
|
|
||||||
|
|
||||||
EXPOSE 8000
|
|
||||||
EXPOSE 8002
|
|
||||||
|
|
||||||
RUN apt-get update && \
|
|
||||||
apt-get install -y --no-install-recommends \
|
|
||||||
jq \
|
|
||||||
tini \
|
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
|
||||||
|
|
||||||
WORKDIR /usr/src/app
|
|
||||||
|
|
||||||
# Keep the .git directory in order to properly report version
|
|
||||||
COPY . /usr/src/app
|
|
||||||
COPY --from=builder /usr/src/app/node_modules ./node_modules/
|
|
||||||
|
|
||||||
|
|
||||||
VOLUME ["/usr/src/app/localData","/usr/src/app/localMetadata"]
|
|
||||||
|
|
||||||
ENTRYPOINT ["tini", "--", "/usr/src/app/docker-entrypoint.sh"]
|
|
||||||
|
|
||||||
CMD [ "yarn", "start" ]
|
|
165
README.md
165
README.md
|
@ -1,7 +1,12 @@
|
||||||
# Zenko CloudServer with Vitastor Backend
|
# Zenko CloudServer
|
||||||
|
|
||||||
![Zenko CloudServer logo](res/scality-cloudserver-logo.png)
|
![Zenko CloudServer logo](res/scality-cloudserver-logo.png)
|
||||||
|
|
||||||
|
[![CircleCI][badgepub]](https://circleci.com/gh/scality/S3)
|
||||||
|
[![Scality CI][badgepriv]](http://ci.ironmann.io/gh/scality/S3)
|
||||||
|
[![Docker Pulls][badgedocker]](https://hub.docker.com/r/scality/s3server/)
|
||||||
|
[![Docker Pulls][badgetwitter]](https://twitter.com/zenko)
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
CloudServer (formerly S3 Server) is an open-source Amazon S3-compatible
|
CloudServer (formerly S3 Server) is an open-source Amazon S3-compatible
|
||||||
|
@ -11,71 +16,125 @@ Scality’s Open Source Multi-Cloud Data Controller.
|
||||||
CloudServer provides a single AWS S3 API interface to access multiple
|
CloudServer provides a single AWS S3 API interface to access multiple
|
||||||
backend data storage both on-premise or public in the cloud.
|
backend data storage both on-premise or public in the cloud.
|
||||||
|
|
||||||
This repository contains a fork of CloudServer with [Vitastor](https://git.yourcmc.ru/vitalif/vitastor)
|
CloudServer is useful for Developers, either to run as part of a
|
||||||
backend support.
|
continous integration test environment to emulate the AWS S3 service locally
|
||||||
|
or as an abstraction layer to develop object storage enabled
|
||||||
|
application on the go.
|
||||||
|
|
||||||
## Quick Start with Vitastor
|
## Learn more at [www.zenko.io/cloudserver](https://www.zenko.io/cloudserver/)
|
||||||
|
|
||||||
Vitastor Backend is in experimental status, however you can already try to
|
## [May I offer you some lovely documentation?](http://s3-server.readthedocs.io/en/latest/)
|
||||||
run it and write or read something, or even mount it with [GeeseFS](https://github.com/yandex-cloud/geesefs),
|
|
||||||
it works too 😊.
|
|
||||||
|
|
||||||
Installation instructions:
|
## Docker
|
||||||
|
|
||||||
### Install Vitastor
|
[Run your Zenko CloudServer with Docker](https://hub.docker.com/r/scality/s3server/)
|
||||||
|
|
||||||
Refer to [Vitastor Quick Start Manual](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/intro/quickstart.en.md).
|
## Contributing
|
||||||
|
|
||||||
### Install Zenko with Vitastor Backend
|
In order to contribute, please follow the
|
||||||
|
[Contributing Guidelines](
|
||||||
|
https://github.com/scality/Guidelines/blob/master/CONTRIBUTING.md).
|
||||||
|
|
||||||
- Clone this repository: `git clone https://git.yourcmc.ru/vitalif/zenko-cloudserver-vitastor`
|
## Installation
|
||||||
- Install dependencies: `npm install --omit dev` or just `npm install`
|
|
||||||
- Clone Vitastor repository: `git clone https://git.yourcmc.ru/vitalif/vitastor`
|
|
||||||
- Build Vitastor node.js binding by running `npm install` in `node-binding` subdirectory of Vitastor repository.
|
|
||||||
You need `node-gyp` and `vitastor-client-dev` (Vitastor client library) for it to succeed.
|
|
||||||
- Symlink Vitastor module to Zenko: `ln -s /path/to/vitastor/node-binding /path/to/zenko/node_modules/vitastor`
|
|
||||||
|
|
||||||
### Install and Configure MongoDB
|
### Dependencies
|
||||||
|
|
||||||
Refer to [MongoDB Manual](https://www.mongodb.com/docs/manual/installation/).
|
Building and running the Zenko CloudServer requires node.js 6.9.5 and npm v3
|
||||||
|
. Up-to-date versions can be found at
|
||||||
|
[Nodesource](https://github.com/nodesource/distributions).
|
||||||
|
|
||||||
### Setup Zenko
|
### Clone source code
|
||||||
|
|
||||||
- Create a separate pool for S3 object data in your Vitastor cluster: `vitastor-cli create-pool s3-data`
|
```shell
|
||||||
- Retrieve ID of the new pool from `vitastor-cli ls-pools --detail s3-data`
|
git clone https://github.com/scality/S3.git
|
||||||
- In another pool, create an image for storing Vitastor volume metadata: `vitastor-cli create -s 10G s3-volume-meta`
|
|
||||||
- Copy `config.json.vitastor` to `config.json`, adjust it to match your domain
|
|
||||||
- Copy `authdata.json.example` to `authdata.json` - this is where you set S3 access & secret keys,
|
|
||||||
and also adjust them if you want to. Scality seems to use a separate auth service "Scality Vault" for
|
|
||||||
access keys, but it's not published, so let's use a file for now.
|
|
||||||
- Copy `locationConfig.json.vitastor` to `locationConfig.json` - this is where you set Vitastor cluster access data.
|
|
||||||
You should put correct values for `pool_id` (pool ID from the second step) and `metadata_image` (from the third step)
|
|
||||||
in this file.
|
|
||||||
|
|
||||||
Note: `locationConfig.json` in this version corresponds to storage classes (like STANDARD, COLD, etc)
|
|
||||||
instead of "locations" (zones like us-east-1) as it was in original Zenko CloudServer.
|
|
||||||
|
|
||||||
### Start Zenko
|
|
||||||
|
|
||||||
Start the S3 server with: `node index.js`
|
|
||||||
|
|
||||||
If you use default settings, Zenko CloudServer starts on port 8000.
|
|
||||||
The default access key is `accessKey1` with a secret key of `verySecretKey1`.
|
|
||||||
|
|
||||||
Now you can access your S3 with `s3cmd` or `geesefs`:
|
|
||||||
|
|
||||||
```
|
|
||||||
s3cmd --access_key=accessKey1 --secret_key=verySecretKey1 --host=http://localhost:8000 mb s3://testbucket
|
|
||||||
```
|
```
|
||||||
|
|
||||||
```
|
### Install js dependencies
|
||||||
AWS_ACCESS_KEY_ID=accessKey1 \
|
|
||||||
AWS_SECRET_ACCESS_KEY=verySecretKey1 \
|
Go to the ./S3 folder,
|
||||||
geesefs --endpoint http://localhost:8000 testbucket mountdir
|
|
||||||
|
```shell
|
||||||
|
npm install
|
||||||
```
|
```
|
||||||
|
|
||||||
# Author & License
|
If you get an error regarding installation of the diskUsage module,
|
||||||
|
please install g++.
|
||||||
|
|
||||||
- [Zenko CloudServer](https://s3-server.readthedocs.io/en/latest/) author is Scality, licensed under [Apache License, version 2.0](https://www.apache.org/licenses/LICENSE-2.0)
|
If you get an error regarding level-down bindings, try clearing your npm cache:
|
||||||
- [Vitastor](https://git.yourcmc.ru/vitalif/vitastor/) and Zenko Vitastor backend author is Vitaliy Filippov, licensed under [VNPL-1.1](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/VNPL-1.1.txt)
|
|
||||||
(a "network copyleft" license based on AGPL/SSPL, but worded in a better way)
|
```shell
|
||||||
|
npm cache clear
|
||||||
|
```
|
||||||
|
|
||||||
|
## Run it with a file backend
|
||||||
|
|
||||||
|
```shell
|
||||||
|
npm start
|
||||||
|
```
|
||||||
|
|
||||||
|
This starts a Zenko CloudServer on port 8000. Two additional ports 9990 and
|
||||||
|
9991 are also open locally for internal transfer of metadata and data,
|
||||||
|
respectively.
|
||||||
|
|
||||||
|
The default access key is accessKey1 with
|
||||||
|
a secret key of verySecretKey1.
|
||||||
|
|
||||||
|
By default the metadata files will be saved in the
|
||||||
|
localMetadata directory and the data files will be saved
|
||||||
|
in the localData directory within the ./S3 directory on your
|
||||||
|
machine. These directories have been pre-created within the
|
||||||
|
repository. If you would like to save the data or metadata in
|
||||||
|
different locations of your choice, you must specify them with absolute paths.
|
||||||
|
So, when starting the server:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
mkdir -m 700 $(pwd)/myFavoriteDataPath
|
||||||
|
mkdir -m 700 $(pwd)/myFavoriteMetadataPath
|
||||||
|
export S3DATAPATH="$(pwd)/myFavoriteDataPath"
|
||||||
|
export S3METADATAPATH="$(pwd)/myFavoriteMetadataPath"
|
||||||
|
npm start
|
||||||
|
```
|
||||||
|
|
||||||
|
## Run it with multiple data backends
|
||||||
|
|
||||||
|
```shell
|
||||||
|
export S3DATA='multiple'
|
||||||
|
npm start
|
||||||
|
```
|
||||||
|
|
||||||
|
This starts a Zenko CloudServer on port 8000.
|
||||||
|
The default access key is accessKey1 with
|
||||||
|
a secret key of verySecretKey1.
|
||||||
|
|
||||||
|
With multiple backends, you have the ability to
|
||||||
|
choose where each object will be saved by setting
|
||||||
|
the following header with a locationConstraint on
|
||||||
|
a PUT request:
|
||||||
|
|
||||||
|
```shell
|
||||||
|
'x-amz-meta-scal-location-constraint':'myLocationConstraint'
|
||||||
|
```
|
||||||
|
|
||||||
|
If no header is sent with a PUT object request, the
|
||||||
|
location constraint of the bucket will determine
|
||||||
|
where the data is saved. If the bucket has no location
|
||||||
|
constraint, the endpoint of the PUT request will be
|
||||||
|
used to determine location.
|
||||||
|
|
||||||
|
See the Configuration section below to learn how to set
|
||||||
|
location constraints.
|
||||||
|
|
||||||
|
## Run it with an in-memory backend
|
||||||
|
|
||||||
|
```shell
|
||||||
|
npm run mem_backend
|
||||||
|
```
|
||||||
|
|
||||||
|
This starts a Zenko CloudServer on port 8000.
|
||||||
|
The default access key is accessKey1 with
|
||||||
|
a secret key of verySecretKey1.
|
||||||
|
|
||||||
|
[badgetwitter]: https://img.shields.io/twitter/follow/zenko.svg?style=social&label=Follow
|
||||||
|
[badgedocker]: https://img.shields.io/docker/pulls/scality/s3server.svg
|
||||||
|
[badgepub]: https://circleci.com/gh/scality/S3.svg?style=svg
|
||||||
|
[badgepriv]: http://ci.ironmann.io/gh/scality/S3.svg?style=svg&circle-token=1f105b7518b53853b5b7cf72302a3f75d8c598ae
|
||||||
|
|
|
@ -1,2 +0,0 @@
|
||||||
---
|
|
||||||
theme: jekyll-theme-modernist
|
|
|
@ -1,56 +0,0 @@
|
||||||
{
|
|
||||||
"accounts": [{
|
|
||||||
"name": "Bart",
|
|
||||||
"email": "sampleaccount1@sampling.com",
|
|
||||||
"arn": "arn:aws:iam::123456789012:root",
|
|
||||||
"canonicalID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be",
|
|
||||||
"shortid": "123456789012",
|
|
||||||
"keys": [{
|
|
||||||
"access": "accessKey1",
|
|
||||||
"secret": "verySecretKey1"
|
|
||||||
}]
|
|
||||||
}, {
|
|
||||||
"name": "Lisa",
|
|
||||||
"email": "sampleaccount2@sampling.com",
|
|
||||||
"arn": "arn:aws:iam::123456789013:root",
|
|
||||||
"canonicalID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2bf",
|
|
||||||
"shortid": "123456789013",
|
|
||||||
"keys": [{
|
|
||||||
"access": "accessKey2",
|
|
||||||
"secret": "verySecretKey2"
|
|
||||||
}]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "Clueso",
|
|
||||||
"email": "inspector@clueso.info",
|
|
||||||
"arn": "arn:aws:iam::123456789014:root",
|
|
||||||
"canonicalID": "http://acs.zenko.io/accounts/service/clueso",
|
|
||||||
"shortid": "123456789014",
|
|
||||||
"keys": [{
|
|
||||||
"access": "cluesoKey1",
|
|
||||||
"secret": "cluesoSecretKey1"
|
|
||||||
}]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "Replication",
|
|
||||||
"email": "inspector@replication.info",
|
|
||||||
"arn": "arn:aws:iam::123456789015:root",
|
|
||||||
"canonicalID": "http://acs.zenko.io/accounts/service/replication",
|
|
||||||
"shortid": "123456789015",
|
|
||||||
"keys": [{
|
|
||||||
"access": "replicationKey1",
|
|
||||||
"secret": "replicationSecretKey1"
|
|
||||||
}]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "Lifecycle",
|
|
||||||
"email": "inspector@lifecycle.info",
|
|
||||||
"arn": "arn:aws:iam::123456789016:root",
|
|
||||||
"canonicalID": "http://acs.zenko.io/accounts/service/lifecycle",
|
|
||||||
"shortid": "123456789016",
|
|
||||||
"keys": [{
|
|
||||||
"access": "lifecycleKey1",
|
|
||||||
"secret": "lifecycleSecretKey1"
|
|
||||||
}]
|
|
||||||
}]
|
|
||||||
}
|
|
|
@ -0,0 +1,554 @@
|
||||||
|
const { errors, s3middleware } = require('arsenal');
|
||||||
|
const AWS = require('aws-sdk');
|
||||||
|
const MD5Sum = s3middleware.MD5Sum;
|
||||||
|
const getMetaHeaders = s3middleware.userMetadata.getMetaHeaders;
|
||||||
|
const createLogger = require('../multipleBackendLogger');
|
||||||
|
const { prepareStream } = require('../../api/apiUtils/object/prepareStream');
|
||||||
|
const { logHelper, removeQuotes, trimXMetaPrefix } = require('./utils');
|
||||||
|
const { config } = require('../../Config');
|
||||||
|
|
||||||
|
const missingVerIdInternalError = errors.InternalError.customizeDescription(
|
||||||
|
'Invalid state. Please ensure versioning is enabled ' +
|
||||||
|
'in AWS for the location constraint and try again.'
|
||||||
|
);
|
||||||
|
|
||||||
|
class AwsClient {
|
||||||
|
constructor(config) {
|
||||||
|
this.clientType = 'aws_s3';
|
||||||
|
this._s3Params = config.s3Params;
|
||||||
|
this._awsBucketName = config.bucketName;
|
||||||
|
this._bucketMatch = config.bucketMatch;
|
||||||
|
this._dataStoreName = config.dataStoreName;
|
||||||
|
this._serverSideEncryption = config.serverSideEncryption;
|
||||||
|
this._client = new AWS.S3(this._s3Params);
|
||||||
|
}
|
||||||
|
|
||||||
|
_createAwsKey(requestBucketName, requestObjectKey,
|
||||||
|
bucketMatch) {
|
||||||
|
if (bucketMatch) {
|
||||||
|
return requestObjectKey;
|
||||||
|
}
|
||||||
|
return `${requestBucketName}/${requestObjectKey}`;
|
||||||
|
}
|
||||||
|
put(stream, size, keyContext, reqUids, callback) {
|
||||||
|
const awsKey = this._createAwsKey(keyContext.bucketName,
|
||||||
|
keyContext.objectKey, this._bucketMatch);
|
||||||
|
const metaHeaders = trimXMetaPrefix(keyContext.metaHeaders);
|
||||||
|
const log = createLogger(reqUids);
|
||||||
|
|
||||||
|
const putCb = (err, data) => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'err from data backend',
|
||||||
|
err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (!data.VersionId) {
|
||||||
|
logHelper(log, 'error', 'missing version id for data ' +
|
||||||
|
'backend object', missingVerIdInternalError,
|
||||||
|
this._dataStoreName);
|
||||||
|
return callback(missingVerIdInternalError);
|
||||||
|
}
|
||||||
|
const dataStoreVersionId = data.VersionId;
|
||||||
|
return callback(null, awsKey, dataStoreVersionId);
|
||||||
|
};
|
||||||
|
|
||||||
|
const params = {
|
||||||
|
Bucket: this._awsBucketName,
|
||||||
|
Key: awsKey,
|
||||||
|
};
|
||||||
|
// we call data.put to create a delete marker, but it's actually a
|
||||||
|
// delete request in call to AWS
|
||||||
|
if (keyContext.isDeleteMarker) {
|
||||||
|
return this._client.deleteObject(params, putCb);
|
||||||
|
}
|
||||||
|
const uploadParams = params;
|
||||||
|
uploadParams.Metadata = metaHeaders;
|
||||||
|
uploadParams.ContentLength = size;
|
||||||
|
if (this._serverSideEncryption) {
|
||||||
|
uploadParams.ServerSideEncryption = 'AES256';
|
||||||
|
}
|
||||||
|
if (keyContext.tagging) {
|
||||||
|
uploadParams.Tagging = keyContext.tagging;
|
||||||
|
}
|
||||||
|
if (keyContext.contentType !== undefined) {
|
||||||
|
uploadParams.ContentType = keyContext.contentType;
|
||||||
|
}
|
||||||
|
if (keyContext.cacheControl !== undefined) {
|
||||||
|
uploadParams.CacheControl = keyContext.cacheControl;
|
||||||
|
}
|
||||||
|
if (keyContext.contentDisposition !== undefined) {
|
||||||
|
uploadParams.ContentDisposition = keyContext.contentDisposition;
|
||||||
|
}
|
||||||
|
if (keyContext.contentEncoding !== undefined) {
|
||||||
|
uploadParams.ContentEncoding = keyContext.contentEncoding;
|
||||||
|
}
|
||||||
|
if (!stream) {
|
||||||
|
return this._client.putObject(uploadParams, putCb);
|
||||||
|
}
|
||||||
|
|
||||||
|
uploadParams.Body = stream;
|
||||||
|
return this._client.upload(uploadParams, putCb);
|
||||||
|
}
|
||||||
|
head(objectGetInfo, reqUids, callback) {
|
||||||
|
const log = createLogger(reqUids);
|
||||||
|
const { key, dataStoreVersionId } = objectGetInfo;
|
||||||
|
return this._client.headObject({
|
||||||
|
Bucket: this._awsBucketName,
|
||||||
|
Key: key,
|
||||||
|
VersionId: dataStoreVersionId,
|
||||||
|
}, err => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'error heading object ' +
|
||||||
|
'from datastore', err, this._dataStoreName);
|
||||||
|
if (err.code === 'NotFound') {
|
||||||
|
const error = errors.ServiceUnavailable
|
||||||
|
.customizeDescription(
|
||||||
|
'Unexpected error from AWS: "NotFound". Data on AWS ' +
|
||||||
|
'may have been altered outside of CloudServer.'
|
||||||
|
);
|
||||||
|
return callback(error);
|
||||||
|
}
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return callback();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
get(objectGetInfo, range, reqUids, callback) {
|
||||||
|
const log = createLogger(reqUids);
|
||||||
|
const { key, dataStoreVersionId } = objectGetInfo;
|
||||||
|
const request = this._client.getObject({
|
||||||
|
Bucket: this._awsBucketName,
|
||||||
|
Key: key,
|
||||||
|
VersionId: dataStoreVersionId,
|
||||||
|
Range: range ? `bytes=${range[0]}-${range[1]}` : null,
|
||||||
|
}).on('success', response => {
|
||||||
|
log.trace('AWS GET request response headers',
|
||||||
|
{ responseHeaders: response.httpResponse.headers });
|
||||||
|
});
|
||||||
|
const stream = request.createReadStream().on('error', err => {
|
||||||
|
logHelper(log, 'error', 'error streaming data from AWS',
|
||||||
|
err, this._dataStoreName);
|
||||||
|
return callback(err);
|
||||||
|
});
|
||||||
|
return callback(null, stream);
|
||||||
|
}
|
||||||
|
delete(objectGetInfo, reqUids, callback) {
|
||||||
|
const { key, dataStoreVersionId, deleteVersion } = objectGetInfo;
|
||||||
|
const log = createLogger(reqUids);
|
||||||
|
const params = {
|
||||||
|
Bucket: this._awsBucketName,
|
||||||
|
Key: key,
|
||||||
|
};
|
||||||
|
if (deleteVersion) {
|
||||||
|
params.VersionId = dataStoreVersionId;
|
||||||
|
}
|
||||||
|
return this._client.deleteObject(params, err => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'error deleting object from ' +
|
||||||
|
'datastore', err, this._dataStoreName);
|
||||||
|
if (err.code === 'NoSuchVersion') {
|
||||||
|
// data may have been deleted directly from the AWS backend
|
||||||
|
// don't want to retry the delete and errors are not
|
||||||
|
// sent back to client anyway, so no need to return err
|
||||||
|
return callback();
|
||||||
|
}
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return callback();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
healthcheck(location, callback) {
|
||||||
|
const awsResp = {};
|
||||||
|
this._client.headBucket({ Bucket: this._awsBucketName },
|
||||||
|
err => {
|
||||||
|
/* eslint-disable no-param-reassign */
|
||||||
|
if (err) {
|
||||||
|
awsResp[location] = { error: err, external: true };
|
||||||
|
return callback(null, awsResp);
|
||||||
|
}
|
||||||
|
return this._client.getBucketVersioning({
|
||||||
|
Bucket: this._awsBucketName },
|
||||||
|
(err, data) => {
|
||||||
|
if (err) {
|
||||||
|
awsResp[location] = { error: err, external: true };
|
||||||
|
} else if (!data.Status ||
|
||||||
|
data.Status === 'Suspended') {
|
||||||
|
awsResp[location] = {
|
||||||
|
versioningStatus: data.Status,
|
||||||
|
error: 'Versioning must be enabled',
|
||||||
|
external: true,
|
||||||
|
};
|
||||||
|
} else {
|
||||||
|
awsResp[location] = {
|
||||||
|
versioningStatus: data.Status,
|
||||||
|
message: 'Congrats! You own the bucket',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return callback(null, awsResp);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
createMPU(key, metaHeaders, bucketName, websiteRedirectHeader, contentType,
|
||||||
|
cacheControl, contentDisposition, contentEncoding, log, callback) {
|
||||||
|
const metaHeadersTrimmed = {};
|
||||||
|
Object.keys(metaHeaders).forEach(header => {
|
||||||
|
if (header.startsWith('x-amz-meta-')) {
|
||||||
|
const headerKey = header.substring(11);
|
||||||
|
metaHeadersTrimmed[headerKey] = metaHeaders[header];
|
||||||
|
}
|
||||||
|
});
|
||||||
|
Object.assign(metaHeaders, metaHeadersTrimmed);
|
||||||
|
const awsBucket = this._awsBucketName;
|
||||||
|
const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
|
||||||
|
const params = {
|
||||||
|
Bucket: awsBucket,
|
||||||
|
Key: awsKey,
|
||||||
|
WebsiteRedirectLocation: websiteRedirectHeader,
|
||||||
|
Metadata: metaHeaders,
|
||||||
|
ContentType: contentType,
|
||||||
|
CacheControl: cacheControl,
|
||||||
|
ContentDisposition: contentDisposition,
|
||||||
|
ContentEncoding: contentEncoding,
|
||||||
|
};
|
||||||
|
return this._client.createMultipartUpload(params, (err, mpuResObj) => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'err from data backend',
|
||||||
|
err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return callback(null, mpuResObj);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
uploadPart(request, streamingV4Params, stream, size, key, uploadId,
|
||||||
|
partNumber, bucketName, log, callback) {
|
||||||
|
let hashedStream = stream;
|
||||||
|
if (request) {
|
||||||
|
const partStream = prepareStream(request, streamingV4Params,
|
||||||
|
log, callback);
|
||||||
|
hashedStream = new MD5Sum();
|
||||||
|
partStream.pipe(hashedStream);
|
||||||
|
}
|
||||||
|
|
||||||
|
const awsBucket = this._awsBucketName;
|
||||||
|
const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
|
||||||
|
const params = { Bucket: awsBucket, Key: awsKey, UploadId: uploadId,
|
||||||
|
Body: hashedStream, ContentLength: size,
|
||||||
|
PartNumber: partNumber };
|
||||||
|
return this._client.uploadPart(params, (err, partResObj) => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'err from data backend ' +
|
||||||
|
'on uploadPart', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// Because we manually add quotes to ETag later, remove quotes here
|
||||||
|
const noQuotesETag =
|
||||||
|
partResObj.ETag.substring(1, partResObj.ETag.length - 1);
|
||||||
|
const dataRetrievalInfo = {
|
||||||
|
key: awsKey,
|
||||||
|
dataStoreType: 'aws_s3',
|
||||||
|
dataStoreName: this._dataStoreName,
|
||||||
|
dataStoreETag: noQuotesETag,
|
||||||
|
};
|
||||||
|
return callback(null, dataRetrievalInfo);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
listParts(key, uploadId, bucketName, partNumberMarker, maxParts, log,
|
||||||
|
callback) {
|
||||||
|
const awsBucket = this._awsBucketName;
|
||||||
|
const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
|
||||||
|
const params = { Bucket: awsBucket, Key: awsKey, UploadId: uploadId,
|
||||||
|
PartNumberMarker: partNumberMarker, MaxParts: maxParts };
|
||||||
|
return this._client.listParts(params, (err, partList) => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'err from data backend on listPart',
|
||||||
|
err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// build storedParts object to mimic Scality S3 backend returns
|
||||||
|
const storedParts = {};
|
||||||
|
storedParts.IsTruncated = partList.IsTruncated;
|
||||||
|
storedParts.Contents = [];
|
||||||
|
storedParts.Contents = partList.Parts.map(item => {
|
||||||
|
// We manually add quotes to ETag later, so remove quotes here
|
||||||
|
const noQuotesETag =
|
||||||
|
item.ETag.substring(1, item.ETag.length - 1);
|
||||||
|
return {
|
||||||
|
partNumber: item.PartNumber,
|
||||||
|
value: {
|
||||||
|
Size: item.Size,
|
||||||
|
ETag: noQuotesETag,
|
||||||
|
LastModified: item.LastModified,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
});
|
||||||
|
return callback(null, storedParts);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* completeMPU - complete multipart upload on AWS backend
|
||||||
|
* @param {object} jsonList - user-sent list of parts to include in
|
||||||
|
* final mpu object
|
||||||
|
* @param {object} mdInfo - object containing 3 keys: storedParts,
|
||||||
|
* mpuOverviewKey, and splitter
|
||||||
|
* @param {string} key - object key
|
||||||
|
* @param {string} uploadId - multipart upload id string
|
||||||
|
* @param {string} bucketName - name of bucket
|
||||||
|
* @param {RequestLogger} log - logger instance
|
||||||
|
* @param {function} callback - callback function
|
||||||
|
* @return {(Error|object)} - return Error if complete MPU fails, otherwise
|
||||||
|
* object containing completed object key, eTag, and contentLength
|
||||||
|
*/
|
||||||
|
completeMPU(jsonList, mdInfo, key, uploadId, bucketName, log, callback) {
|
||||||
|
const awsBucket = this._awsBucketName;
|
||||||
|
const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
|
||||||
|
const mpuError = {
|
||||||
|
InvalidPart: true,
|
||||||
|
InvalidPartOrder: true,
|
||||||
|
EntityTooSmall: true,
|
||||||
|
};
|
||||||
|
const partArray = [];
|
||||||
|
const partList = jsonList.Part;
|
||||||
|
partList.forEach(partObj => {
|
||||||
|
const partParams = { PartNumber: partObj.PartNumber[0],
|
||||||
|
ETag: partObj.ETag[0] };
|
||||||
|
partArray.push(partParams);
|
||||||
|
});
|
||||||
|
const mpuParams = {
|
||||||
|
Bucket: awsBucket, Key: awsKey, UploadId: uploadId,
|
||||||
|
MultipartUpload: {
|
||||||
|
Parts: partArray,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const completeObjData = { key: awsKey };
|
||||||
|
return this._client.completeMultipartUpload(mpuParams,
|
||||||
|
(err, completeMpuRes) => {
|
||||||
|
if (err) {
|
||||||
|
if (mpuError[err.code]) {
|
||||||
|
logHelper(log, 'trace', 'err from data backend on ' +
|
||||||
|
'completeMPU', err, this._dataStoreName);
|
||||||
|
return callback(errors[err.code]);
|
||||||
|
}
|
||||||
|
logHelper(log, 'error', 'err from data backend on ' +
|
||||||
|
'completeMPU', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (!completeMpuRes.VersionId) {
|
||||||
|
logHelper(log, 'error', 'missing version id for data ' +
|
||||||
|
'backend object', missingVerIdInternalError,
|
||||||
|
this._dataStoreName);
|
||||||
|
return callback(missingVerIdInternalError);
|
||||||
|
}
|
||||||
|
// need to get content length of new object to store
|
||||||
|
// in our metadata
|
||||||
|
return this._client.headObject({ Bucket: awsBucket, Key: awsKey },
|
||||||
|
(err, objHeaders) => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'trace', 'err from data backend on ' +
|
||||||
|
'headObject', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// remove quotes from eTag because they're added later
|
||||||
|
completeObjData.eTag = completeMpuRes.ETag
|
||||||
|
.substring(1, completeMpuRes.ETag.length - 1);
|
||||||
|
completeObjData.dataStoreVersionId = completeMpuRes.VersionId;
|
||||||
|
completeObjData.contentLength = objHeaders.ContentLength;
|
||||||
|
return callback(null, completeObjData);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
abortMPU(key, uploadId, bucketName, log, callback) {
|
||||||
|
const awsBucket = this._awsBucketName;
|
||||||
|
const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
|
||||||
|
const abortParams = {
|
||||||
|
Bucket: awsBucket, Key: awsKey, UploadId: uploadId,
|
||||||
|
};
|
||||||
|
return this._client.abortMultipartUpload(abortParams, err => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'There was an error aborting ' +
|
||||||
|
'the MPU on AWS S3. You should abort directly on AWS S3 ' +
|
||||||
|
'using the same uploadId.', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return callback();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
objectPutTagging(key, bucket, objectMD, log, callback) {
|
||||||
|
const awsBucket = this._awsBucketName;
|
||||||
|
const awsKey = this._createAwsKey(bucket, key, this._bucketMatch);
|
||||||
|
const dataStoreVersionId = objectMD.location[0].dataStoreVersionId;
|
||||||
|
const tagParams = {
|
||||||
|
Bucket: awsBucket,
|
||||||
|
Key: awsKey,
|
||||||
|
VersionId: dataStoreVersionId,
|
||||||
|
};
|
||||||
|
const keyArray = Object.keys(objectMD.tags);
|
||||||
|
tagParams.Tagging = {};
|
||||||
|
tagParams.Tagging.TagSet = keyArray.map(key => {
|
||||||
|
const value = objectMD.tags[key];
|
||||||
|
return { Key: key, Value: value };
|
||||||
|
});
|
||||||
|
return this._client.putObjectTagging(tagParams, err => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'error from data backend on ' +
|
||||||
|
'putObjectTagging', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return callback();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
objectDeleteTagging(key, bucket, objectMD, log, callback) {
|
||||||
|
const awsBucket = this._awsBucketName;
|
||||||
|
const awsKey = this._createAwsKey(bucket, key, this._bucketMatch);
|
||||||
|
const dataStoreVersionId = objectMD.location[0].dataStoreVersionId;
|
||||||
|
const tagParams = {
|
||||||
|
Bucket: awsBucket,
|
||||||
|
Key: awsKey,
|
||||||
|
VersionId: dataStoreVersionId,
|
||||||
|
};
|
||||||
|
return this._client.deleteObjectTagging(tagParams, err => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'error from data backend on ' +
|
||||||
|
'deleteObjectTagging', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return callback();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
copyObject(request, destLocationConstraintName, sourceKey,
|
||||||
|
sourceLocationConstraintName, storeMetadataParams, log, callback) {
|
||||||
|
const destBucketName = request.bucketName;
|
||||||
|
const destObjectKey = request.objectKey;
|
||||||
|
const destAwsKey = this._createAwsKey(destBucketName, destObjectKey,
|
||||||
|
this._bucketMatch);
|
||||||
|
|
||||||
|
const sourceAwsBucketName =
|
||||||
|
config.getAwsBucketName(sourceLocationConstraintName);
|
||||||
|
|
||||||
|
const metadataDirective = request.headers['x-amz-metadata-directive'];
|
||||||
|
const metaHeaders = trimXMetaPrefix(getMetaHeaders(request.headers));
|
||||||
|
const awsParams = {
|
||||||
|
Bucket: this._awsBucketName,
|
||||||
|
Key: destAwsKey,
|
||||||
|
CopySource: `${sourceAwsBucketName}/${sourceKey}`,
|
||||||
|
Metadata: metaHeaders,
|
||||||
|
MetadataDirective: metadataDirective,
|
||||||
|
};
|
||||||
|
if (destLocationConstraintName &&
|
||||||
|
config.isAWSServerSideEncrytion(destLocationConstraintName)) {
|
||||||
|
awsParams.ServerSideEncryption = 'AES256';
|
||||||
|
}
|
||||||
|
this._client.copyObject(awsParams, (err, copyResult) => {
|
||||||
|
if (err) {
|
||||||
|
if (err.code === 'AccessDenied') {
|
||||||
|
logHelper(log, 'error', 'Unable to access ' +
|
||||||
|
`${sourceAwsBucketName} AWS bucket`, err,
|
||||||
|
this._dataStoreName);
|
||||||
|
return callback(errors.AccessDenied
|
||||||
|
.customizeDescription('Error: Unable to access ' +
|
||||||
|
`${sourceAwsBucketName} AWS bucket`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
logHelper(log, 'error', 'error from data backend on ' +
|
||||||
|
'copyObject', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (!copyResult.VersionId) {
|
||||||
|
logHelper(log, 'error', 'missing version id for data ' +
|
||||||
|
'backend object', missingVerIdInternalError,
|
||||||
|
this._dataStoreName);
|
||||||
|
return callback(missingVerIdInternalError);
|
||||||
|
}
|
||||||
|
return callback(null, destAwsKey, copyResult.VersionId);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
uploadPartCopy(request, awsSourceKey, sourceLocationConstraintName,
|
||||||
|
log, callback) {
|
||||||
|
const destBucketName = request.bucketName;
|
||||||
|
const destObjectKey = request.objectKey;
|
||||||
|
const destAwsKey = this._createAwsKey(destBucketName, destObjectKey,
|
||||||
|
this._bucketMatch);
|
||||||
|
|
||||||
|
const sourceAwsBucketName =
|
||||||
|
config.getAwsBucketName(sourceLocationConstraintName);
|
||||||
|
|
||||||
|
const uploadId = request.query.uploadId;
|
||||||
|
const partNumber = request.query.partNumber;
|
||||||
|
const copySourceRange = request.headers['x-amz-copy-source-range'];
|
||||||
|
|
||||||
|
const params = {
|
||||||
|
Bucket: this._awsBucketName,
|
||||||
|
CopySource: `${sourceAwsBucketName}/${awsSourceKey}`,
|
||||||
|
CopySourceRange: copySourceRange,
|
||||||
|
Key: destAwsKey,
|
||||||
|
PartNumber: partNumber,
|
||||||
|
UploadId: uploadId,
|
||||||
|
};
|
||||||
|
return this._client.uploadPartCopy(params, (err, res) => {
|
||||||
|
if (err) {
|
||||||
|
if (err.code === 'AccessDenied') {
|
||||||
|
logHelper(log, 'error', 'Unable to access ' +
|
||||||
|
`${sourceAwsBucketName} AWS bucket`, err,
|
||||||
|
this._dataStoreName);
|
||||||
|
return callback(errors.AccessDenied
|
||||||
|
.customizeDescription('Error: Unable to access ' +
|
||||||
|
`${sourceAwsBucketName} AWS bucket`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
logHelper(log, 'error', 'error from data backend on ' +
|
||||||
|
'uploadPartCopy', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const eTag = removeQuotes(res.CopyPartResult.ETag);
|
||||||
|
return callback(null, eTag);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = AwsClient;
|
|
@ -0,0 +1,434 @@
|
||||||
|
const url = require('url');
|
||||||
|
|
||||||
|
const { errors, s3middleware } = require('arsenal');
|
||||||
|
const azure = require('azure-storage');
|
||||||
|
const createLogger = require('../multipleBackendLogger');
|
||||||
|
const { logHelper, translateAzureMetaHeaders } = require('./utils');
|
||||||
|
const { config } = require('../../Config');
|
||||||
|
const { validateAndFilterMpuParts } =
|
||||||
|
require('../../api/apiUtils/object/processMpuParts');
|
||||||
|
const constants = require('../../../constants');
|
||||||
|
const metadata = require('../../metadata/wrapper');
|
||||||
|
const azureMpuUtils = s3middleware.azureHelper.mpuUtils;
|
||||||
|
|
||||||
|
class AzureClient {
|
||||||
|
constructor(config) {
|
||||||
|
this._azureStorageEndpoint = config.azureStorageEndpoint;
|
||||||
|
this._azureStorageCredentials = config.azureStorageCredentials;
|
||||||
|
this._azureContainerName = config.azureContainerName;
|
||||||
|
this._client = azure.createBlobService(
|
||||||
|
this._azureStorageCredentials.storageAccountName,
|
||||||
|
this._azureStorageCredentials.storageAccessKey,
|
||||||
|
this._azureStorageEndpoint);
|
||||||
|
this._dataStoreName = config.dataStoreName;
|
||||||
|
this._bucketMatch = config.bucketMatch;
|
||||||
|
if (config.proxy) {
|
||||||
|
const parsedUrl = url.parse(config.proxy);
|
||||||
|
if (!parsedUrl.port) {
|
||||||
|
parsedUrl.port = 80;
|
||||||
|
}
|
||||||
|
this._client.setProxy(parsedUrl);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
_errorWrapper(s3Method, azureMethod, args, log, cb) {
|
||||||
|
if (log) {
|
||||||
|
log.info(`calling azure ${azureMethod}`);
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
this._client[azureMethod].apply(this._client, args);
|
||||||
|
} catch (err) {
|
||||||
|
const error = errors.ServiceUnavailable;
|
||||||
|
if (log) {
|
||||||
|
log.error('error thrown by Azure Storage Client Library',
|
||||||
|
{ error: err.message, stack: err.stack, s3Method,
|
||||||
|
azureMethod, dataStoreName: this._dataStoreName });
|
||||||
|
}
|
||||||
|
cb(error.customizeDescription('Error from Azure ' +
|
||||||
|
`method: ${azureMethod} on ${s3Method} S3 call: ` +
|
||||||
|
`${err.message}`));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
_createAzureKey(requestBucketName, requestObjectKey,
|
||||||
|
bucketMatch) {
|
||||||
|
if (bucketMatch) {
|
||||||
|
return requestObjectKey;
|
||||||
|
}
|
||||||
|
return `${requestBucketName}/${requestObjectKey}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
_getMetaHeaders(objectMD) {
|
||||||
|
const metaHeaders = {};
|
||||||
|
Object.keys(objectMD).forEach(mdKey => {
|
||||||
|
const isMetaHeader = mdKey.startsWith('x-amz-meta-');
|
||||||
|
if (isMetaHeader) {
|
||||||
|
metaHeaders[mdKey] = objectMD[mdKey];
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return translateAzureMetaHeaders(metaHeaders);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Before putting or deleting object on Azure, check if MPU exists with
|
||||||
|
// same key name. If it does, do not allow put or delete because Azure
|
||||||
|
// will delete all blocks with same key name
|
||||||
|
protectAzureBlocks(bucketName, objectKey, dataStoreName, log, cb) {
|
||||||
|
const mpuBucketName = `${constants.mpuBucketPrefix}${bucketName}`;
|
||||||
|
const splitter = constants.splitter;
|
||||||
|
const listingParams = {
|
||||||
|
prefix: `overview${splitter}${objectKey}`,
|
||||||
|
listingType: 'MPU',
|
||||||
|
splitter,
|
||||||
|
maxKeys: 1,
|
||||||
|
};
|
||||||
|
|
||||||
|
return metadata.listMultipartUploads(mpuBucketName, listingParams,
|
||||||
|
log, (err, mpuList) => {
|
||||||
|
if (err && !err.NoSuchBucket) {
|
||||||
|
log.error('Error listing MPUs for Azure delete',
|
||||||
|
{ error: err, dataStoreName });
|
||||||
|
return cb(errors.ServiceUnavailable);
|
||||||
|
}
|
||||||
|
if (mpuList && mpuList.Uploads && mpuList.Uploads.length > 0) {
|
||||||
|
const error = errors.MPUinProgress;
|
||||||
|
log.error('Error: cannot put/delete object to Azure with ' +
|
||||||
|
'same key name as ongoing MPU on Azure',
|
||||||
|
{ error, dataStoreName });
|
||||||
|
return cb(error);
|
||||||
|
}
|
||||||
|
// If listMultipartUploads returns a NoSuchBucket error or the
|
||||||
|
// mpu list is empty, there are no conflicting MPUs, so continue
|
||||||
|
return cb();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
put(stream, size, keyContext, reqUids, callback) {
|
||||||
|
const log = createLogger(reqUids);
|
||||||
|
// before blob is put, make sure there is no ongoing MPU with same key
|
||||||
|
this.protectAzureBlocks(keyContext.bucketName,
|
||||||
|
keyContext.objectKey, this._dataStoreName, log, err => {
|
||||||
|
// if error returned, there is ongoing MPU, so do not put
|
||||||
|
if (err) {
|
||||||
|
return callback(err.customizeDescription(
|
||||||
|
`Error putting object to Azure: ${err.message}`));
|
||||||
|
}
|
||||||
|
const azureKey = this._createAzureKey(keyContext.bucketName,
|
||||||
|
keyContext.objectKey, this._bucketMatch);
|
||||||
|
const options = {
|
||||||
|
metadata: translateAzureMetaHeaders(keyContext.metaHeaders,
|
||||||
|
keyContext.tagging),
|
||||||
|
contentSettings: {
|
||||||
|
contentType: keyContext.contentType || undefined,
|
||||||
|
cacheControl: keyContext.cacheControl || undefined,
|
||||||
|
contentDisposition: keyContext.contentDisposition ||
|
||||||
|
undefined,
|
||||||
|
contentEncoding: keyContext.contentEncoding || undefined,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
if (size === 0) {
|
||||||
|
return this._errorWrapper('put', 'createBlockBlobFromText',
|
||||||
|
[this._azureContainerName, azureKey, '', options,
|
||||||
|
err => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'err from Azure PUT data ' +
|
||||||
|
'backend', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`Azure: ${err.message}`));
|
||||||
|
}
|
||||||
|
return callback(null, azureKey);
|
||||||
|
}], log, callback);
|
||||||
|
}
|
||||||
|
return this._errorWrapper('put', 'createBlockBlobFromStream',
|
||||||
|
[this._azureContainerName, azureKey, stream, size, options,
|
||||||
|
err => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'err from Azure PUT data ' +
|
||||||
|
'backend', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`Azure: ${err.message}`));
|
||||||
|
}
|
||||||
|
return callback(null, azureKey);
|
||||||
|
}], log, callback);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
head(objectGetInfo, reqUids, callback) {
|
||||||
|
const log = createLogger(reqUids);
|
||||||
|
const { key, azureStreamingOptions } = objectGetInfo;
|
||||||
|
return this._errorWrapper('head', 'getBlobProperties',
|
||||||
|
[this._azureContainerName, key, azureStreamingOptions,
|
||||||
|
err => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'err from Azure HEAD data backend',
|
||||||
|
err, this._dataStoreName);
|
||||||
|
if (err.code === 'NotFound') {
|
||||||
|
const error = errors.ServiceUnavailable
|
||||||
|
.customizeDescription(
|
||||||
|
'Unexpected error from Azure: "NotFound". Data ' +
|
||||||
|
'on Azure may have been altered outside of ' +
|
||||||
|
'CloudServer.');
|
||||||
|
return callback(error);
|
||||||
|
}
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`Azure: ${err.message}`));
|
||||||
|
}
|
||||||
|
return callback();
|
||||||
|
}], log, callback);
|
||||||
|
}
|
||||||
|
|
||||||
|
get(objectGetInfo, range, reqUids, callback) {
|
||||||
|
const log = createLogger(reqUids);
|
||||||
|
// for backwards compatibility
|
||||||
|
const { key, response, azureStreamingOptions } = objectGetInfo;
|
||||||
|
let streamingOptions;
|
||||||
|
if (azureStreamingOptions) {
|
||||||
|
// option coming from api.get()
|
||||||
|
streamingOptions = azureStreamingOptions;
|
||||||
|
} else if (range) {
|
||||||
|
// option coming from multipleBackend.upload()
|
||||||
|
const rangeStart = range[0] ? range[0].toString() : undefined;
|
||||||
|
const rangeEnd = range[1] ? range[1].toString() : undefined;
|
||||||
|
streamingOptions = { rangeStart, rangeEnd };
|
||||||
|
}
|
||||||
|
this._errorWrapper('get', 'getBlobToStream',
|
||||||
|
[this._azureContainerName, key, response, streamingOptions,
|
||||||
|
err => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'err from Azure GET data backend',
|
||||||
|
err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable);
|
||||||
|
}
|
||||||
|
return callback(null, response);
|
||||||
|
}], log, callback);
|
||||||
|
}
|
||||||
|
|
||||||
|
delete(objectGetInfo, reqUids, callback) {
|
||||||
|
const log = createLogger(reqUids);
|
||||||
|
// for backwards compatibility
|
||||||
|
const key = typeof objectGetInfo === 'string' ? objectGetInfo :
|
||||||
|
objectGetInfo.key;
|
||||||
|
return this._errorWrapper('delete', 'deleteBlobIfExists',
|
||||||
|
[this._azureContainerName, key,
|
||||||
|
err => {
|
||||||
|
if (err) {
|
||||||
|
const log = createLogger(reqUids);
|
||||||
|
logHelper(log, 'error', 'error deleting object from ' +
|
||||||
|
'Azure datastore', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`Azure: ${err.message}`));
|
||||||
|
}
|
||||||
|
return callback();
|
||||||
|
}], log, callback);
|
||||||
|
}
|
||||||
|
|
||||||
|
healthcheck(location, callback, flightCheckOnStartUp) {
|
||||||
|
const azureResp = {};
|
||||||
|
const healthCheckAction = flightCheckOnStartUp ?
|
||||||
|
'createContainerIfNotExists' : 'doesContainerExist';
|
||||||
|
this._errorWrapper('checkAzureHealth', healthCheckAction,
|
||||||
|
[this._azureContainerName, err => {
|
||||||
|
/* eslint-disable no-param-reassign */
|
||||||
|
if (err) {
|
||||||
|
azureResp[location] = { error: err.message,
|
||||||
|
external: true };
|
||||||
|
return callback(null, azureResp);
|
||||||
|
}
|
||||||
|
azureResp[location] = {
|
||||||
|
message:
|
||||||
|
'Congrats! You can access the Azure storage account',
|
||||||
|
};
|
||||||
|
return callback(null, azureResp);
|
||||||
|
}], null, callback);
|
||||||
|
}
|
||||||
|
|
||||||
|
uploadPart(request, streamingV4Params, partStream, size, key, uploadId,
|
||||||
|
partNumber, bucket, log, callback) {
|
||||||
|
const azureKey = this._createAzureKey(bucket, key, this._bucketMatch);
|
||||||
|
const params = { bucketName: this._azureContainerName,
|
||||||
|
partNumber, size, objectKey: azureKey, uploadId };
|
||||||
|
const stream = request || partStream;
|
||||||
|
|
||||||
|
if (request && request.headers && request.headers['content-md5']) {
|
||||||
|
params.contentMD5 = request.headers['content-md5'];
|
||||||
|
}
|
||||||
|
const dataRetrievalInfo = {
|
||||||
|
key: azureKey,
|
||||||
|
partNumber,
|
||||||
|
dataStoreName: this._dataStoreName,
|
||||||
|
dataStoreType: 'azure',
|
||||||
|
numberSubParts: azureMpuUtils.getSubPartInfo(size)
|
||||||
|
.expectedNumberSubParts,
|
||||||
|
};
|
||||||
|
|
||||||
|
if (size === 0) {
|
||||||
|
log.debug('0-byte part does not store data',
|
||||||
|
{ method: 'uploadPart' });
|
||||||
|
dataRetrievalInfo.dataStoreETag = azureMpuUtils.zeroByteETag;
|
||||||
|
dataRetrievalInfo.numberSubParts = 0;
|
||||||
|
return callback(null, dataRetrievalInfo);
|
||||||
|
}
|
||||||
|
if (size <= azureMpuUtils.maxSubPartSize) {
|
||||||
|
const errorWrapperFn = this._errorWrapper.bind(this);
|
||||||
|
return azureMpuUtils.putSinglePart(errorWrapperFn,
|
||||||
|
stream, params, this._dataStoreName, log, (err, dataStoreETag) => {
|
||||||
|
if (err) {
|
||||||
|
return callback(err);
|
||||||
|
}
|
||||||
|
dataRetrievalInfo.dataStoreETag = dataStoreETag;
|
||||||
|
return callback(null, dataRetrievalInfo);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
const errorWrapperFn = this._errorWrapper.bind(this);
|
||||||
|
return azureMpuUtils.putSubParts(errorWrapperFn, stream,
|
||||||
|
params, this._dataStoreName, log, (err, dataStoreETag) => {
|
||||||
|
if (err) {
|
||||||
|
return callback(err);
|
||||||
|
}
|
||||||
|
dataRetrievalInfo.dataStoreETag = dataStoreETag;
|
||||||
|
return callback(null, dataRetrievalInfo);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
completeMPU(jsonList, mdInfo, key, uploadId, bucket, metaHeaders,
|
||||||
|
contentSettings, log, callback) {
|
||||||
|
const azureKey = this._createAzureKey(bucket, key, this._bucketMatch);
|
||||||
|
const commitList = {
|
||||||
|
UncommittedBlocks: jsonList.uncommittedBlocks || [],
|
||||||
|
};
|
||||||
|
let filteredPartsObj;
|
||||||
|
if (!jsonList.uncommittedBlocks) {
|
||||||
|
const { storedParts, mpuOverviewKey, splitter } = mdInfo;
|
||||||
|
filteredPartsObj = validateAndFilterMpuParts(storedParts, jsonList,
|
||||||
|
mpuOverviewKey, splitter, log);
|
||||||
|
|
||||||
|
filteredPartsObj.partList.forEach(part => {
|
||||||
|
// part.locations is always array of 1, which contains data info
|
||||||
|
const subPartIds =
|
||||||
|
azureMpuUtils.getSubPartIds(part.locations[0], uploadId);
|
||||||
|
commitList.UncommittedBlocks.push(...subPartIds);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
const options = {
|
||||||
|
contentSettings,
|
||||||
|
metadata: metaHeaders ? translateAzureMetaHeaders(metaHeaders) :
|
||||||
|
undefined,
|
||||||
|
};
|
||||||
|
this._errorWrapper('completeMPU', 'commitBlocks',
|
||||||
|
[this._azureContainerName, azureKey, commitList, options,
|
||||||
|
err => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'err completing MPU on Azure ' +
|
||||||
|
'datastore', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`Azure: ${err.message}`));
|
||||||
|
}
|
||||||
|
const completeObjData = {
|
||||||
|
key: azureKey,
|
||||||
|
filteredPartsObj,
|
||||||
|
};
|
||||||
|
return callback(null, completeObjData);
|
||||||
|
}], log, callback);
|
||||||
|
}
|
||||||
|
|
||||||
|
objectPutTagging(key, bucket, objectMD, log, callback) {
|
||||||
|
const azureKey = this._createAzureKey(bucket, key, this._bucketMatch);
|
||||||
|
const azureMD = this._getMetaHeaders(objectMD);
|
||||||
|
azureMD.tags = JSON.stringify(objectMD.tags);
|
||||||
|
this._errorWrapper('objectPutTagging', 'setBlobMetadata',
|
||||||
|
[this._azureContainerName, azureKey, azureMD,
|
||||||
|
err => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'err putting object tags to ' +
|
||||||
|
'Azure backend', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable);
|
||||||
|
}
|
||||||
|
return callback();
|
||||||
|
}], log, callback);
|
||||||
|
}
|
||||||
|
|
||||||
|
objectDeleteTagging(key, bucket, objectMD, log, callback) {
|
||||||
|
const azureKey = this._createAzureKey(bucket, key, this._bucketMatch);
|
||||||
|
const azureMD = this._getMetaHeaders(objectMD);
|
||||||
|
this._errorWrapper('objectDeleteTagging', 'setBlobMetadata',
|
||||||
|
[this._azureContainerName, azureKey, azureMD,
|
||||||
|
err => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'err putting object tags to ' +
|
||||||
|
'Azure backend', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable);
|
||||||
|
}
|
||||||
|
return callback();
|
||||||
|
}], log, callback);
|
||||||
|
}
|
||||||
|
|
||||||
|
copyObject(request, destLocationConstraintName, sourceKey,
|
||||||
|
sourceLocationConstraintName, storeMetadataParams, log, callback) {
|
||||||
|
const destContainerName = request.bucketName;
|
||||||
|
const destObjectKey = request.objectKey;
|
||||||
|
|
||||||
|
const destAzureKey = this._createAzureKey(destContainerName,
|
||||||
|
destObjectKey, this._bucketMatch);
|
||||||
|
|
||||||
|
const sourceContainerName =
|
||||||
|
config.locationConstraints[sourceLocationConstraintName]
|
||||||
|
.details.azureContainerName;
|
||||||
|
|
||||||
|
let options;
|
||||||
|
if (storeMetadataParams.metaHeaders) {
|
||||||
|
options = { metadata:
|
||||||
|
translateAzureMetaHeaders(storeMetadataParams.metaHeaders) };
|
||||||
|
}
|
||||||
|
|
||||||
|
this._errorWrapper('copyObject', 'startCopyBlob',
|
||||||
|
[`${this._azureStorageEndpoint}` +
|
||||||
|
`${sourceContainerName}/${sourceKey}`,
|
||||||
|
this._azureContainerName, destAzureKey, options,
|
||||||
|
(err, res) => {
|
||||||
|
if (err) {
|
||||||
|
if (err.code === 'CannotVerifyCopySource') {
|
||||||
|
logHelper(log, 'error', 'Unable to access ' +
|
||||||
|
`${sourceContainerName} Azure Container`, err,
|
||||||
|
this._dataStoreName);
|
||||||
|
return callback(errors.AccessDenied
|
||||||
|
.customizeDescription('Error: Unable to access ' +
|
||||||
|
`${sourceContainerName} Azure Container`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
logHelper(log, 'error', 'error from data backend on ' +
|
||||||
|
'copyObject', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (res.copy.status === 'pending') {
|
||||||
|
logHelper(log, 'error', 'Azure copy status is pending',
|
||||||
|
err, this._dataStoreName);
|
||||||
|
const copyId = res.copy.id;
|
||||||
|
this._client.abortCopyBlob(this._azureContainerName,
|
||||||
|
destAzureKey, copyId, err => {
|
||||||
|
if (err) {
|
||||||
|
logHelper(log, 'error', 'error from data backend ' +
|
||||||
|
'on abortCopyBlob', err, this._dataStoreName);
|
||||||
|
return callback(errors.ServiceUnavailable
|
||||||
|
.customizeDescription('Error returned from ' +
|
||||||
|
`AWS on abortCopyBlob: ${err.message}`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return callback(errors.InvalidObjectState
|
||||||
|
.customizeDescription('Error: Azure copy status was ' +
|
||||||
|
'pending. It has been aborted successfully')
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return callback(null, destAzureKey);
|
||||||
|
}], log, callback);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = AzureClient;
|
|
@ -0,0 +1,246 @@
|
||||||
|
const googleAuth = require('google-auto-auth');
|
||||||
|
const async = require('async');
|
||||||
|
const AWS = require('aws-sdk');
|
||||||
|
const { errors } = require('arsenal');
|
||||||
|
const Service = AWS.Service;
|
||||||
|
|
||||||
|
const GcpSigner = require('./GcpSigner');
|
||||||
|
|
||||||
|
function genAuth(authParams, callback) {
|
||||||
|
async.tryEach([
|
||||||
|
function authKeyFile(next) {
|
||||||
|
const authOptions = {
|
||||||
|
scopes: authParams.scopes,
|
||||||
|
keyFilename: authParams.keyFilename,
|
||||||
|
};
|
||||||
|
const auth = googleAuth(authOptions);
|
||||||
|
auth.getToken(err => next(err, auth));
|
||||||
|
},
|
||||||
|
function authCredentials(next) {
|
||||||
|
const authOptions = {
|
||||||
|
scopes: authParams.scopes,
|
||||||
|
credentials: authParams.credentials,
|
||||||
|
};
|
||||||
|
const auth = googleAuth(authOptions);
|
||||||
|
auth.getToken(err => next(err, auth));
|
||||||
|
},
|
||||||
|
], (err, result) => callback(err, result));
|
||||||
|
}
|
||||||
|
|
||||||
|
AWS.apiLoader.services.gcp = {};
|
||||||
|
const GCP = Service.defineService('gcp', ['2017-11-01'], {
|
||||||
|
_jsonAuth: null,
|
||||||
|
_authParams: null,
|
||||||
|
|
||||||
|
getToken(callback) {
|
||||||
|
if (this._jsonAuth) {
|
||||||
|
return this._jsonAuth.getToken(callback);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!this._authParams && this.config.authParams &&
|
||||||
|
typeof this.config.authParams === 'object') {
|
||||||
|
this._authParams = this.config.authParams;
|
||||||
|
}
|
||||||
|
return genAuth(this._authParams, (err, auth) => {
|
||||||
|
if (!err) {
|
||||||
|
this._jsonAuth = auth;
|
||||||
|
return this._jsonAuth.getToken(callback);
|
||||||
|
}
|
||||||
|
// should never happen, but if all preconditions fails
|
||||||
|
// can't generate tokens
|
||||||
|
return callback(errors.InternalError.customizeDescription(
|
||||||
|
'Unable to create a google authorizer'));
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
getSignerClass() {
|
||||||
|
return GcpSigner;
|
||||||
|
},
|
||||||
|
|
||||||
|
validateService() {
|
||||||
|
if (!this.config.region) {
|
||||||
|
this.config.region = 'us-east-1';
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
|
upload(params, options, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: upload not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
// Service API
|
||||||
|
listBuckets(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: listBuckets not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
// Bucket APIs
|
||||||
|
getBucketLocation(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: getBucketLocation not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
deleteBucket(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: deleteBucket not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
headBucket(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: headBucket not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
listObjects(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: listObjects not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
listObjectVersions(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: listObjecVersions not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
putBucket(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: putBucket not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
getBucketAcl(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: getBucketAcl not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
putBucketAcl(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: putBucketAcl not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
putBucketWebsite(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: putBucketWebsite not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
getBucketWebsite(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: getBucketWebsite not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
deleteBucketWebsite(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: deleteBucketWebsite not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
putBucketCors(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: putBucketCors not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
getBucketCors(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: getBucketCors not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
deleteBucketCors(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: deleteBucketCors not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
// Object APIs
|
||||||
|
headObject(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: headObject not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
putObject(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: putObject not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
getObject(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: getObject not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
deleteObject(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: deleteObject not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
deleteObjects(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: deleteObjects not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
copyObject(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: copyObject not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
putObjectTagging(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: putObjectTagging not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
deleteObjectTagging(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: deleteObjectTagging not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
putObjectAcl(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: putObjectAcl not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
getObjectAcl(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription('GCP: getObjectAcl not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
// Multipart upload
|
||||||
|
abortMultipartUpload(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription(
|
||||||
|
'GCP: abortMultipartUpload not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
createMultipartUpload(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription(
|
||||||
|
'GCP: createMultipartUpload not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
completeMultipartUpload(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription(
|
||||||
|
'GCP: completeMultipartUpload not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
uploadPart(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription(
|
||||||
|
'GCP: uploadPart not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
uploadPartCopy(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription(
|
||||||
|
'GCP: uploadPartCopy not implemented'));
|
||||||
|
},
|
||||||
|
|
||||||
|
listParts(params, callback) {
|
||||||
|
return callback(errors.NotImplemented
|
||||||
|
.customizeDescription(
|
||||||
|
'GCP: listParts not implemented'));
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
Object.defineProperty(AWS.apiLoader.services.gcp, '2017-11-01', {
|
||||||
|
get: function get() {
|
||||||
|
const model = require('./gcp-2017-11-01.api.json');
|
||||||
|
return model;
|
||||||
|
},
|
||||||
|
enumerable: true,
|
||||||
|
configurable: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
module.exports = GCP;
|
|
@ -0,0 +1,48 @@
|
||||||
|
const url = require('url');
|
||||||
|
const qs = require('querystring');
|
||||||
|
const AWS = require('aws-sdk');
|
||||||
|
const werelogs = require('werelogs');
|
||||||
|
const { constructStringToSignV2 } = require('arsenal').auth.client;
|
||||||
|
|
||||||
|
const logger = new werelogs.Logger('GcpSigner');
|
||||||
|
|
||||||
|
function genQueryObject(uri) {
|
||||||
|
const queryString = url.parse(uri).query;
|
||||||
|
return qs.parse(queryString);
|
||||||
|
}
|
||||||
|
|
||||||
|
const GcpSigner = AWS.util.inherit(AWS.Signers.RequestSigner, {
|
||||||
|
constructor: function GcpSigner(request) {
|
||||||
|
AWS.Signers.RequestSigner.call(this, request);
|
||||||
|
},
|
||||||
|
|
||||||
|
addAuthorization: function addAuthorization(credentials, date) {
|
||||||
|
if (!this.request.headers['presigned-expires']) {
|
||||||
|
this.request.headers['x-goog-date'] = AWS.util.date.rfc822(date);
|
||||||
|
}
|
||||||
|
|
||||||
|
const signature =
|
||||||
|
this.sign(credentials.secretAccessKey, this.stringToSign());
|
||||||
|
const auth = `GOOG1 ${credentials.accessKeyId}: ${signature}`;
|
||||||
|
|
||||||
|
this.request.headers.Authorization = auth;
|
||||||
|
},
|
||||||
|
|
||||||
|
stringToSign: function stringToSign() {
|
||||||
|
const requestObject = {
|
||||||
|
url: this.request.path,
|
||||||
|
method: this.request.method,
|
||||||
|
host: this.request.endpoint.host,
|
||||||
|
headers: this.request.headers,
|
||||||
|
query: genQueryObject(this.request.path) || {},
|
||||||
|
};
|
||||||
|
const data = Object.assign({}, this.request.headers);
|
||||||
|
return constructStringToSignV2(requestObject, data, logger, 'GCP');
|
||||||
|
},
|
||||||
|
|
||||||
|
sign: function sign(secret, string) {
|
||||||
|
return AWS.util.crypto.hmac(secret, string, 'base64', 'sha1');
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
module.exports = GcpSigner;
|
|
@ -0,0 +1 @@
|
||||||
|
module.exports = {};
|
|
@ -0,0 +1,17 @@
|
||||||
|
{
|
||||||
|
"version": "1.0",
|
||||||
|
"metadata": {
|
||||||
|
"apiVersion": "2017-11-01",
|
||||||
|
"checksumFormat": "md5",
|
||||||
|
"endpointPrefix": "s3",
|
||||||
|
"globalEndpoint": "storage.googleapi.com",
|
||||||
|
"protocol": "rest-xml",
|
||||||
|
"serviceAbbreviation": "GCP",
|
||||||
|
"serviceFullName": "Google Cloud Storage",
|
||||||
|
"signatureVersion": "s3",
|
||||||
|
"timestampFormat": "rfc822",
|
||||||
|
"uid": "gcp-2017-11-01"
|
||||||
|
},
|
||||||
|
"operations": {},
|
||||||
|
"shapes": {}
|
||||||
|
}
|
|
@ -0,0 +1,5 @@
|
||||||
|
module.exports = {
|
||||||
|
GCP: require('./GcpService'),
|
||||||
|
GcpSigner: require('./GcpSigner'),
|
||||||
|
GcpUtils: require('./GcpUtils'),
|
||||||
|
};
|
|
@ -0,0 +1,39 @@
|
||||||
|
const { GCP } = require('./GCP');
|
||||||
|
const AwsClient = require('./AwsClient');
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Class representing a Google Cloud Storage backend object
|
||||||
|
* @extends AwsClient
|
||||||
|
*/
|
||||||
|
class GcpClient extends AwsClient {
|
||||||
|
/**
|
||||||
|
* constructor - creates a Gcp backend client object (inherits )
|
||||||
|
* @param {object} config - configuration object for Gcp Backend up
|
||||||
|
* @param {object} config.s3params - S3 configuration
|
||||||
|
* @param {string} config.bucketName - GCP bucket name
|
||||||
|
* @param {string} config.mpuBucket - GCP mpu bucket name
|
||||||
|
* @param {string} config.overflowBucket - GCP overflow bucket name
|
||||||
|
* @param {boolean} config.bucketMatch - bucket match flag
|
||||||
|
* @param {object} config.authParams - GCP service credentials
|
||||||
|
* @param {string} config.dataStoreName - locationConstraint name
|
||||||
|
* @param {booblean} config.serverSideEncryption - server side encryption
|
||||||
|
* flag
|
||||||
|
* @return {object} - returns a GcpClient object
|
||||||
|
*/
|
||||||
|
constructor(config) {
|
||||||
|
super(config);
|
||||||
|
this.clientType = 'gcp';
|
||||||
|
this._gcpBucketName = config.bucketName;
|
||||||
|
this._mpuBucketName = config.mpuBucket;
|
||||||
|
this._overflowBucketname = config.overflowBucket;
|
||||||
|
this._gcpParams = Object.assign(this._s3Params, {
|
||||||
|
mainBucket: this._gcpBucketName,
|
||||||
|
mpuBucket: this._mpuBucketName,
|
||||||
|
overflowBucket: this._overflowBucketname,
|
||||||
|
authParams: config.authParams,
|
||||||
|
});
|
||||||
|
this._client = new GCP(this._gcpParams);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = GcpClient;
|
|
@ -0,0 +1,147 @@
|
||||||
|
const async = require('async');
|
||||||
|
const constants = require('../../../constants');
|
||||||
|
const { config } = require('../../../lib/Config');
|
||||||
|
|
||||||
|
const awsHealth = {
|
||||||
|
response: undefined,
|
||||||
|
time: 0,
|
||||||
|
};
|
||||||
|
const azureHealth = {
|
||||||
|
response: undefined,
|
||||||
|
time: 0,
|
||||||
|
};
|
||||||
|
|
||||||
|
const utils = {
|
||||||
|
logHelper(log, level, description, error, dataStoreName) {
|
||||||
|
log[level](description, { error: error.message,
|
||||||
|
errorName: error.name, dataStoreName });
|
||||||
|
},
|
||||||
|
// take off the 'x-amz-meta-'
|
||||||
|
trimXMetaPrefix(obj) {
|
||||||
|
const newObj = {};
|
||||||
|
Object.keys(obj).forEach(key => {
|
||||||
|
const newKey = key.substring(11);
|
||||||
|
newObj[newKey] = obj[key];
|
||||||
|
});
|
||||||
|
return newObj;
|
||||||
|
},
|
||||||
|
removeQuotes(word) {
|
||||||
|
return word.slice(1, -1);
|
||||||
|
},
|
||||||
|
skipMpuPartProcessing(completeObjData) {
|
||||||
|
const backendType = completeObjData.dataStoreType;
|
||||||
|
if (constants.mpuMDStoredExternallyBackend[backendType]) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
},
|
||||||
|
|
||||||
|
/**
|
||||||
|
* checkAzureBackendMatch - checks that the external backend location for
|
||||||
|
* two data objects is the same and is Azure
|
||||||
|
* @param {array} objectDataOne - data of first object to compare
|
||||||
|
* @param {object} objectDataTwo - data of second object to compare
|
||||||
|
* @return {boolean} - true if both data backends are Azure, false if not
|
||||||
|
*/
|
||||||
|
checkAzureBackendMatch(objectDataOne, objectDataTwo) {
|
||||||
|
if (objectDataOne.dataStoreType === 'azure' &&
|
||||||
|
objectDataTwo.dataStoreType === 'azure') {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
},
|
||||||
|
|
||||||
|
/**
|
||||||
|
* externalBackendCopy - Server side copy should only be allowed:
|
||||||
|
* 1) if source object and destination object are both on aws or both
|
||||||
|
* on azure.
|
||||||
|
* 2) if azure to azure, must be the same storage account since Azure
|
||||||
|
* copy outside of an account is async
|
||||||
|
* 3) if the source bucket is not an encrypted bucket and the
|
||||||
|
* destination bucket is not an encrypted bucket (unless the copy
|
||||||
|
* is all within the same bucket).
|
||||||
|
* @param {string} locationConstraintSrc - location constraint of the source
|
||||||
|
* @param {string} locationConstraintDest - location constraint of the
|
||||||
|
* destination
|
||||||
|
* @param {object} sourceBucketMD - metadata of the source bucket
|
||||||
|
* @param {object} destBucketMD - metadata of the destination bucket
|
||||||
|
* @return {boolean} - true if copying object from one
|
||||||
|
* externalbackend to the same external backend and for Azure if it is the
|
||||||
|
* same account since Azure copy outside of an account is async
|
||||||
|
*/
|
||||||
|
externalBackendCopy(locationConstraintSrc, locationConstraintDest,
|
||||||
|
sourceBucketMD, destBucketMD) {
|
||||||
|
const sourceBucketName = sourceBucketMD.getName();
|
||||||
|
const destBucketName = destBucketMD.getName();
|
||||||
|
const isSameBucket = sourceBucketName === destBucketName;
|
||||||
|
const bucketsNotEncrypted = destBucketMD.getServerSideEncryption()
|
||||||
|
=== sourceBucketMD.getServerSideEncryption() &&
|
||||||
|
sourceBucketMD.getServerSideEncryption() === null;
|
||||||
|
const sourceLocationConstraintType =
|
||||||
|
config.getLocationConstraintType(locationConstraintSrc);
|
||||||
|
const locationTypeMatch =
|
||||||
|
config.getLocationConstraintType(locationConstraintSrc) ===
|
||||||
|
config.getLocationConstraintType(locationConstraintDest);
|
||||||
|
return locationTypeMatch && (isSameBucket || bucketsNotEncrypted) &&
|
||||||
|
(sourceLocationConstraintType === 'aws_s3' ||
|
||||||
|
(sourceLocationConstraintType === 'azure' &&
|
||||||
|
config.isSameAzureAccount(locationConstraintSrc,
|
||||||
|
locationConstraintDest)));
|
||||||
|
},
|
||||||
|
|
||||||
|
checkExternalBackend(clients, locations, type, flightCheckOnStartUp,
|
||||||
|
externalBackendHealthCheckInterval, cb) {
|
||||||
|
const checkStatus = type === 'aws_s3' ? awsHealth : azureHealth;
|
||||||
|
if (locations.length === 0) {
|
||||||
|
return process.nextTick(cb, null, []);
|
||||||
|
}
|
||||||
|
if (!flightCheckOnStartUp && checkStatus.response &&
|
||||||
|
Date.now() - checkStatus.time
|
||||||
|
< externalBackendHealthCheckInterval) {
|
||||||
|
return process.nextTick(cb, null, checkStatus.response);
|
||||||
|
}
|
||||||
|
let locationsToCheck;
|
||||||
|
if (flightCheckOnStartUp) {
|
||||||
|
// check all locations if flight check on start up
|
||||||
|
locationsToCheck = locations;
|
||||||
|
} else {
|
||||||
|
const randomLocation = locations[Math.floor(Math.random() *
|
||||||
|
locations.length)];
|
||||||
|
locationsToCheck = [randomLocation];
|
||||||
|
}
|
||||||
|
return async.mapLimit(locationsToCheck, 5, (location, next) => {
|
||||||
|
const client = clients[location];
|
||||||
|
client.healthcheck(location, next, flightCheckOnStartUp);
|
||||||
|
}, (err, results) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
if (!flightCheckOnStartUp) {
|
||||||
|
checkStatus.response = results;
|
||||||
|
checkStatus.time = Date.now();
|
||||||
|
}
|
||||||
|
return cb(null, results);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
translateAzureMetaHeaders(metaHeaders, tags) {
|
||||||
|
const translatedMetaHeaders = {};
|
||||||
|
if (tags) {
|
||||||
|
// tags are passed as string of format 'key1=value1&key2=value2'
|
||||||
|
const tagObj = {};
|
||||||
|
const tagArr = tags.split('&');
|
||||||
|
tagArr.forEach(keypair => {
|
||||||
|
const equalIndex = keypair.indexOf('=');
|
||||||
|
const key = keypair.substring(0, equalIndex);
|
||||||
|
tagObj[key] = keypair.substring(equalIndex + 1);
|
||||||
|
});
|
||||||
|
translatedMetaHeaders.tags = JSON.stringify(tagObj);
|
||||||
|
}
|
||||||
|
Object.keys(metaHeaders).forEach(headerName => {
|
||||||
|
const translated = headerName.substring(11).replace(/-/g, '_');
|
||||||
|
translatedMetaHeaders[translated] = metaHeaders[headerName];
|
||||||
|
});
|
||||||
|
return translatedMetaHeaders;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
module.exports = utils;
|
|
@ -0,0 +1,37 @@
|
||||||
|
const arsenal = require('arsenal');
|
||||||
|
const { config } = require('../../Config');
|
||||||
|
|
||||||
|
class DataFileInterface {
|
||||||
|
constructor() {
|
||||||
|
const { host, port } = config.dataClient;
|
||||||
|
|
||||||
|
this.restClient = new arsenal.network.rest.RESTClient(
|
||||||
|
{ host, port });
|
||||||
|
}
|
||||||
|
|
||||||
|
put(stream, size, keyContext, reqUids, callback) {
|
||||||
|
// ignore keyContext
|
||||||
|
this.restClient.put(stream, size, reqUids, callback);
|
||||||
|
}
|
||||||
|
|
||||||
|
get(objectGetInfo, range, reqUids, callback) {
|
||||||
|
const key = objectGetInfo.key ? objectGetInfo.key : objectGetInfo;
|
||||||
|
this.restClient.get(key, range, reqUids, callback);
|
||||||
|
}
|
||||||
|
|
||||||
|
delete(objectGetInfo, reqUids, callback) {
|
||||||
|
const key = objectGetInfo.key ? objectGetInfo.key : objectGetInfo;
|
||||||
|
this.restClient.delete(key, reqUids, callback);
|
||||||
|
}
|
||||||
|
|
||||||
|
getDiskUsage(reqUids, callback) {
|
||||||
|
this.restClient.getAction('diskUsage', reqUids, (err, val) => {
|
||||||
|
if (err) {
|
||||||
|
return callback(err);
|
||||||
|
}
|
||||||
|
return callback(null, JSON.parse(val));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = DataFileInterface;
|
|
@ -0,0 +1,93 @@
|
||||||
|
const stream = require('stream');
|
||||||
|
const { errors } = require('arsenal');
|
||||||
|
const werelogs = require('werelogs');
|
||||||
|
|
||||||
|
const logger = new werelogs.Logger('MemDataBackend');
|
||||||
|
|
||||||
|
function createLogger(reqUids) {
|
||||||
|
return reqUids ?
|
||||||
|
logger.newRequestLoggerFromSerializedUids(reqUids) :
|
||||||
|
logger.newRequestLogger();
|
||||||
|
}
|
||||||
|
|
||||||
|
const ds = [];
|
||||||
|
let count = 1; // keys are assessed with if (!key)
|
||||||
|
|
||||||
|
function resetCount() {
|
||||||
|
count = 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const backend = {
|
||||||
|
put: function putMem(request, size, keyContext, reqUids, callback) {
|
||||||
|
const log = createLogger(reqUids);
|
||||||
|
const value = Buffer.alloc(size);
|
||||||
|
let cursor = 0;
|
||||||
|
let exceeded = false;
|
||||||
|
request.on('data', data => {
|
||||||
|
if (cursor + data.length > size) {
|
||||||
|
exceeded = true;
|
||||||
|
}
|
||||||
|
if (!exceeded) {
|
||||||
|
data.copy(value, cursor);
|
||||||
|
}
|
||||||
|
cursor += data.length;
|
||||||
|
})
|
||||||
|
.on('end', () => {
|
||||||
|
if (exceeded) {
|
||||||
|
log.error('data stream exceed announced size',
|
||||||
|
{ size, overflow: cursor });
|
||||||
|
callback(errors.InternalError);
|
||||||
|
} else {
|
||||||
|
ds[count] = { value, keyContext };
|
||||||
|
callback(null, count++);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
get: function getMem(objectGetInfo, range, reqUids, callback) {
|
||||||
|
const key = objectGetInfo.key ? objectGetInfo.key : objectGetInfo;
|
||||||
|
process.nextTick(() => {
|
||||||
|
if (!ds[key]) { return callback(errors.NoSuchKey); }
|
||||||
|
const storedBuffer = ds[key].value;
|
||||||
|
// If a range was sent, use the start from the range.
|
||||||
|
// Otherwise, start at 0
|
||||||
|
let start = range ? range[0] : 0;
|
||||||
|
// If a range was sent, use the end from the range.
|
||||||
|
// End of range should be included so +1
|
||||||
|
// Otherwise, get the full length
|
||||||
|
const end = range ? range[1] + 1 : storedBuffer.length;
|
||||||
|
const chunkSize = 64 * 1024; // 64KB
|
||||||
|
const val = new stream.Readable({
|
||||||
|
read: function read() {
|
||||||
|
// sets this._read under the hood
|
||||||
|
// push data onto the read queue, passing null
|
||||||
|
// will signal the end of the stream (EOF)
|
||||||
|
while (start < end) {
|
||||||
|
const finish =
|
||||||
|
Math.min(start + chunkSize, end);
|
||||||
|
this.push(storedBuffer.slice(start, finish));
|
||||||
|
start += chunkSize;
|
||||||
|
}
|
||||||
|
if (start >= end) {
|
||||||
|
this.push(null);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
});
|
||||||
|
return callback(null, val);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
delete: function delMem(objectGetInfo, reqUids, callback) {
|
||||||
|
const key = objectGetInfo.key ? objectGetInfo.key : objectGetInfo;
|
||||||
|
process.nextTick(() => {
|
||||||
|
delete ds[key];
|
||||||
|
return callback(null);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
backend,
|
||||||
|
ds,
|
||||||
|
resetCount,
|
||||||
|
};
|
|
@ -0,0 +1,126 @@
|
||||||
|
const http = require('http');
|
||||||
|
const https = require('https');
|
||||||
|
const AWS = require('aws-sdk');
|
||||||
|
const Sproxy = require('sproxydclient');
|
||||||
|
|
||||||
|
const DataFileBackend = require('./file/backend');
|
||||||
|
const inMemory = require('./in_memory/backend').backend;
|
||||||
|
const AwsClient = require('./external/AwsClient');
|
||||||
|
const GcpClient = require('./external/GcpClient');
|
||||||
|
const AzureClient = require('./external/AzureClient');
|
||||||
|
|
||||||
|
const { config } = require('../Config');
|
||||||
|
|
||||||
|
const proxyAddress = 'http://localhost:3128';
|
||||||
|
|
||||||
|
function parseLC() {
|
||||||
|
const clients = {};
|
||||||
|
|
||||||
|
Object.keys(config.locationConstraints).forEach(location => {
|
||||||
|
const locationObj = config.locationConstraints[location];
|
||||||
|
if (locationObj.type === 'mem') {
|
||||||
|
clients[location] = inMemory;
|
||||||
|
}
|
||||||
|
if (locationObj.type === 'file') {
|
||||||
|
clients[location] = new DataFileBackend();
|
||||||
|
}
|
||||||
|
if (locationObj.type === 'scality'
|
||||||
|
&& locationObj.details.connector.sproxyd) {
|
||||||
|
clients[location] = new Sproxy({
|
||||||
|
bootstrap: locationObj.details.connector
|
||||||
|
.sproxyd.bootstrap,
|
||||||
|
// Might be undefined which is ok since there is a default
|
||||||
|
// set in sproxydclient if chordCos is undefined
|
||||||
|
chordCos: locationObj.details.connector.sproxyd.chordCos,
|
||||||
|
// Might also be undefined, but there is a default path set
|
||||||
|
// in sproxydclient as well
|
||||||
|
path: locationObj.details.connector.sproxyd.path,
|
||||||
|
// enable immutable optim for all objects
|
||||||
|
immutable: true,
|
||||||
|
});
|
||||||
|
clients[location].clientType = 'scality';
|
||||||
|
}
|
||||||
|
if (locationObj.type === 'aws_s3' || locationObj.type === 'gcp') {
|
||||||
|
if (process.env.CI_PROXY === 'true') {
|
||||||
|
locationObj.details.https = false;
|
||||||
|
locationObj.details.proxy = proxyAddress;
|
||||||
|
}
|
||||||
|
const connectionAgent = locationObj.details.https ?
|
||||||
|
new https.Agent({ keepAlive: true }) :
|
||||||
|
new http.Agent({ keepAlive: true });
|
||||||
|
const protocol = locationObj.details.https ? 'https' : 'http';
|
||||||
|
const httpOptions = locationObj.details.proxy ?
|
||||||
|
{ proxy: locationObj.details.proxy, agent: connectionAgent,
|
||||||
|
timeout: 0 }
|
||||||
|
: { agent: connectionAgent, timeout: 0 };
|
||||||
|
const sslEnabled = locationObj.details.https === true;
|
||||||
|
// TODO: HTTP requests to AWS are not supported with V4 auth for
|
||||||
|
// non-file streams which are used by Backbeat. This option will be
|
||||||
|
// removed once CA certs, proxy setup feature is implemented.
|
||||||
|
const signatureVersion = !sslEnabled ? 'v2' : 'v4';
|
||||||
|
const endpoint = locationObj.type === 'gcp' ?
|
||||||
|
locationObj.details.gcpEndpoint :
|
||||||
|
locationObj.details.awsEndpoint;
|
||||||
|
const s3Params = {
|
||||||
|
endpoint: `${protocol}://${endpoint}`,
|
||||||
|
debug: false,
|
||||||
|
// Not implemented yet for streams in node sdk,
|
||||||
|
// and has no negative impact if stream, so let's
|
||||||
|
// leave it in for future use
|
||||||
|
computeChecksums: true,
|
||||||
|
httpOptions,
|
||||||
|
// needed for encryption
|
||||||
|
signatureVersion,
|
||||||
|
sslEnabled,
|
||||||
|
maxRetries: 0,
|
||||||
|
};
|
||||||
|
// users can either include the desired profile name from their
|
||||||
|
// ~/.aws/credentials file or include the accessKeyId and
|
||||||
|
// secretAccessKey directly in the locationConfig
|
||||||
|
if (locationObj.details.credentialsProfile) {
|
||||||
|
s3Params.credentials = new AWS.SharedIniFileCredentials({
|
||||||
|
profile: locationObj.details.credentialsProfile });
|
||||||
|
} else {
|
||||||
|
s3Params.accessKeyId =
|
||||||
|
locationObj.details.credentials.accessKey;
|
||||||
|
s3Params.secretAccessKey =
|
||||||
|
locationObj.details.credentials.secretKey;
|
||||||
|
}
|
||||||
|
const clientConfig = {
|
||||||
|
s3Params,
|
||||||
|
bucketName: locationObj.details.bucketName,
|
||||||
|
bucketMatch: locationObj.details.bucketMatch,
|
||||||
|
serverSideEncryption: locationObj.details.serverSideEncryption,
|
||||||
|
dataStoreName: location,
|
||||||
|
};
|
||||||
|
if (locationObj.type === 'gcp') {
|
||||||
|
clientConfig.overflowBucket =
|
||||||
|
locationObj.details.overflowBucketName;
|
||||||
|
clientConfig.mpuBucket = locationObj.details.mpuBucketName;
|
||||||
|
clientConfig.authParams = config.getGcpServiceParams(location);
|
||||||
|
}
|
||||||
|
clients[location] = locationObj.type === 'gcp' ?
|
||||||
|
new GcpClient(clientConfig) : new AwsClient(clientConfig);
|
||||||
|
}
|
||||||
|
if (locationObj.type === 'azure') {
|
||||||
|
if (process.env.CI_PROXY === 'true') {
|
||||||
|
locationObj.details.proxy = proxyAddress;
|
||||||
|
}
|
||||||
|
const azureStorageEndpoint = config.getAzureEndpoint(location);
|
||||||
|
const azureStorageCredentials =
|
||||||
|
config.getAzureStorageCredentials(location);
|
||||||
|
clients[location] = new AzureClient({
|
||||||
|
azureStorageEndpoint,
|
||||||
|
azureStorageCredentials,
|
||||||
|
azureContainerName: locationObj.details.azureContainerName,
|
||||||
|
bucketMatch: locationObj.details.bucketMatch,
|
||||||
|
dataStoreName: location,
|
||||||
|
proxy: locationObj.details.proxy,
|
||||||
|
});
|
||||||
|
clients[location].clientType = 'azure';
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return clients;
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = parseLC;
|
|
@ -0,0 +1,289 @@
|
||||||
|
const { errors, s3middleware } = require('arsenal');
|
||||||
|
const { parseTagFromQuery } = s3middleware.tagging;
|
||||||
|
|
||||||
|
const createLogger = require('./multipleBackendLogger');
|
||||||
|
const async = require('async');
|
||||||
|
|
||||||
|
const { config } = require('../Config');
|
||||||
|
const parseLC = require('./locationConstraintParser');
|
||||||
|
|
||||||
|
const { checkExternalBackend } = require('./external/utils');
|
||||||
|
const { externalBackendHealthCheckInterval } = require('../../constants');
|
||||||
|
|
||||||
|
let clients = parseLC(config);
|
||||||
|
config.on('location-constraints-update', () => {
|
||||||
|
clients = parseLC(config);
|
||||||
|
});
|
||||||
|
|
||||||
|
|
||||||
|
const multipleBackendGateway = {
|
||||||
|
|
||||||
|
put: (hashedStream, size, keyContext,
|
||||||
|
backendInfo, reqUids, callback) => {
|
||||||
|
const controllingLocationConstraint =
|
||||||
|
backendInfo.getControllingLocationConstraint();
|
||||||
|
const client = clients[controllingLocationConstraint];
|
||||||
|
if (!client) {
|
||||||
|
const log = createLogger(reqUids);
|
||||||
|
log.error('no data backend matching controlling locationConstraint',
|
||||||
|
{ controllingLocationConstraint });
|
||||||
|
return process.nextTick(() => {
|
||||||
|
callback(errors.InternalError);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
let writeStream = hashedStream;
|
||||||
|
if (keyContext.cipherBundle && keyContext.cipherBundle.cipher) {
|
||||||
|
writeStream = keyContext.cipherBundle.cipher;
|
||||||
|
hashedStream.pipe(writeStream);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (keyContext.tagging) {
|
||||||
|
const validationTagRes = parseTagFromQuery(keyContext.tagging);
|
||||||
|
if (validationTagRes instanceof Error) {
|
||||||
|
const log = createLogger(reqUids);
|
||||||
|
log.debug('tag validation failed', {
|
||||||
|
error: validationTagRes,
|
||||||
|
method: 'multipleBackendGateway put',
|
||||||
|
});
|
||||||
|
return callback(errors.InternalError);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return client.put(writeStream, size, keyContext,
|
||||||
|
reqUids, (err, key, dataStoreVersionId) => {
|
||||||
|
const log = createLogger(reqUids);
|
||||||
|
log.debug('put to location', { controllingLocationConstraint });
|
||||||
|
if (err) {
|
||||||
|
log.error('error from datastore',
|
||||||
|
{ error: err, dataStoreType: client.clientType });
|
||||||
|
return callback(errors.ServiceUnavailable);
|
||||||
|
}
|
||||||
|
const dataRetrievalInfo = {
|
||||||
|
key,
|
||||||
|
dataStoreName: controllingLocationConstraint,
|
||||||
|
dataStoreType: client.clientType,
|
||||||
|
dataStoreVersionId,
|
||||||
|
};
|
||||||
|
return callback(null, dataRetrievalInfo);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
head: (objectGetInfoArr, reqUids, callback) => {
|
||||||
|
if (!objectGetInfoArr || !Array.isArray(objectGetInfoArr)
|
||||||
|
|| !objectGetInfoArr[0] || !objectGetInfoArr[0].dataStoreName) {
|
||||||
|
// no-op if no stored data store name
|
||||||
|
return process.nextTick(callback);
|
||||||
|
}
|
||||||
|
const objectGetInfo = objectGetInfoArr[0];
|
||||||
|
const client = clients[objectGetInfo.dataStoreName];
|
||||||
|
if (client.head === undefined) {
|
||||||
|
// no-op if unsupported client method
|
||||||
|
return process.nextTick(callback);
|
||||||
|
}
|
||||||
|
return client.head(objectGetInfo, reqUids, callback);
|
||||||
|
},
|
||||||
|
|
||||||
|
get: (objectGetInfo, range, reqUids, callback) => {
|
||||||
|
let key;
|
||||||
|
let client;
|
||||||
|
// for backwards compatibility
|
||||||
|
if (typeof objectGetInfo === 'string') {
|
||||||
|
key = objectGetInfo;
|
||||||
|
client = clients.sproxyd;
|
||||||
|
} else {
|
||||||
|
key = objectGetInfo.key;
|
||||||
|
client = clients[objectGetInfo.dataStoreName];
|
||||||
|
}
|
||||||
|
if (client.clientType === 'scality') {
|
||||||
|
return client.get(key, range, reqUids, callback);
|
||||||
|
}
|
||||||
|
return client.get(objectGetInfo, range, reqUids, callback);
|
||||||
|
},
|
||||||
|
|
||||||
|
delete: (objectGetInfo, reqUids, callback) => {
|
||||||
|
let key;
|
||||||
|
let client;
|
||||||
|
// for backwards compatibility
|
||||||
|
if (typeof objectGetInfo === 'string') {
|
||||||
|
key = objectGetInfo;
|
||||||
|
client = clients.sproxyd;
|
||||||
|
} else {
|
||||||
|
key = objectGetInfo.key;
|
||||||
|
client = clients[objectGetInfo.dataStoreName];
|
||||||
|
}
|
||||||
|
if (client.clientType === 'scality') {
|
||||||
|
return client.delete(key, reqUids, callback);
|
||||||
|
}
|
||||||
|
return client.delete(objectGetInfo, reqUids, callback);
|
||||||
|
},
|
||||||
|
|
||||||
|
healthcheck: (flightCheckOnStartUp, log, callback) => {
|
||||||
|
const multBackendResp = {};
|
||||||
|
const awsArray = [];
|
||||||
|
const azureArray = [];
|
||||||
|
async.each(Object.keys(clients), (location, cb) => {
|
||||||
|
const client = clients[location];
|
||||||
|
if (client.clientType === 'scality') {
|
||||||
|
return client.healthcheck(log, (err, res) => {
|
||||||
|
if (err) {
|
||||||
|
multBackendResp[location] = { error: err };
|
||||||
|
} else {
|
||||||
|
multBackendResp[location] = { code: res.statusCode,
|
||||||
|
message: res.statusMessage };
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
});
|
||||||
|
} else if (client.clientType === 'aws_s3') {
|
||||||
|
awsArray.push(location);
|
||||||
|
return cb();
|
||||||
|
} else if (client.clientType === 'azure') {
|
||||||
|
azureArray.push(location);
|
||||||
|
return cb();
|
||||||
|
}
|
||||||
|
// if backend type isn't 'scality' or 'aws_s3', it will be
|
||||||
|
// 'mem' or 'file', for which the default response is 200 OK
|
||||||
|
multBackendResp[location] = { code: 200, message: 'OK' };
|
||||||
|
return cb();
|
||||||
|
}, () => {
|
||||||
|
async.parallel([
|
||||||
|
next => checkExternalBackend(
|
||||||
|
clients, awsArray, 'aws_s3', flightCheckOnStartUp,
|
||||||
|
externalBackendHealthCheckInterval, next),
|
||||||
|
next => checkExternalBackend(
|
||||||
|
clients, azureArray, 'azure', flightCheckOnStartUp,
|
||||||
|
externalBackendHealthCheckInterval, next),
|
||||||
|
], (errNull, externalResp) => {
|
||||||
|
const externalLocResults = [];
|
||||||
|
externalResp.forEach(resp => externalLocResults.push(...resp));
|
||||||
|
externalLocResults.forEach(locationResult =>
|
||||||
|
Object.assign(multBackendResp, locationResult));
|
||||||
|
callback(null, multBackendResp);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
createMPU: (key, metaHeaders, bucketName, websiteRedirectHeader,
|
||||||
|
location, contentType, cacheControl, contentDisposition,
|
||||||
|
contentEncoding, log, cb) => {
|
||||||
|
const client = clients[location];
|
||||||
|
if (client.clientType === 'aws_s3') {
|
||||||
|
return client.createMPU(key, metaHeaders, bucketName,
|
||||||
|
websiteRedirectHeader, contentType, cacheControl,
|
||||||
|
contentDisposition, contentEncoding, log, cb);
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
},
|
||||||
|
|
||||||
|
uploadPart: (request, streamingV4Params, stream, size, location, key,
|
||||||
|
uploadId, partNumber, bucketName, log, cb) => {
|
||||||
|
const client = clients[location];
|
||||||
|
|
||||||
|
if (client.uploadPart) {
|
||||||
|
return client.uploadPart(request, streamingV4Params, stream, size,
|
||||||
|
key, uploadId, partNumber, bucketName, log, cb);
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
},
|
||||||
|
|
||||||
|
listParts: (key, uploadId, location, bucketName, partNumberMarker, maxParts,
|
||||||
|
log, cb) => {
|
||||||
|
const client = clients[location];
|
||||||
|
|
||||||
|
if (client.listParts) {
|
||||||
|
return client.listParts(key, uploadId, bucketName, partNumberMarker,
|
||||||
|
maxParts, log, cb);
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
},
|
||||||
|
|
||||||
|
completeMPU: (key, uploadId, location, jsonList, mdInfo, bucketName,
|
||||||
|
userMetadata, contentSettings, log, cb) => {
|
||||||
|
const client = clients[location];
|
||||||
|
if (client.completeMPU) {
|
||||||
|
const args = [jsonList, mdInfo, key, uploadId, bucketName];
|
||||||
|
if (client.clientType === 'azure') {
|
||||||
|
args.push(userMetadata, contentSettings);
|
||||||
|
}
|
||||||
|
return client.completeMPU(...args, log, (err, completeObjData) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
// eslint-disable-next-line no-param-reassign
|
||||||
|
completeObjData.dataStoreType = client.clientType;
|
||||||
|
return cb(null, completeObjData);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
},
|
||||||
|
|
||||||
|
abortMPU: (key, uploadId, location, bucketName, log, cb) => {
|
||||||
|
const client = clients[location];
|
||||||
|
if (client.clientType === 'azure') {
|
||||||
|
const skipDataDelete = true;
|
||||||
|
return cb(null, skipDataDelete);
|
||||||
|
}
|
||||||
|
if (client.abortMPU) {
|
||||||
|
return client.abortMPU(key, uploadId, bucketName, log, err => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
},
|
||||||
|
|
||||||
|
objectTagging: (method, key, bucket, objectMD, log, cb) => {
|
||||||
|
// if legacy, objectMD will not contain dataStoreName, so just return
|
||||||
|
const client = clients[objectMD.dataStoreName];
|
||||||
|
if (client && client[`object${method}Tagging`]) {
|
||||||
|
return client[`object${method}Tagging`](key, bucket, objectMD, log,
|
||||||
|
cb);
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
},
|
||||||
|
// NOTE: using copyObject only if copying object from one external
|
||||||
|
// backend to the same external backend
|
||||||
|
copyObject: (request, destLocationConstraintName, externalSourceKey,
|
||||||
|
sourceLocationConstraintName, storeMetadataParams, log, cb) => {
|
||||||
|
const client = clients[destLocationConstraintName];
|
||||||
|
if (client.copyObject) {
|
||||||
|
return client.copyObject(request, destLocationConstraintName,
|
||||||
|
externalSourceKey, sourceLocationConstraintName,
|
||||||
|
storeMetadataParams, log, (err, key, dataStoreVersionId) => {
|
||||||
|
const dataRetrievalInfo = {
|
||||||
|
key,
|
||||||
|
dataStoreName: destLocationConstraintName,
|
||||||
|
dataStoreType: client.clientType,
|
||||||
|
dataStoreVersionId,
|
||||||
|
};
|
||||||
|
cb(err, dataRetrievalInfo);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return cb(errors.NotImplemented
|
||||||
|
.customizeDescription('Can not copy object from ' +
|
||||||
|
`${client.clientType} to ${client.clientType}`));
|
||||||
|
},
|
||||||
|
uploadPartCopy: (request, location, awsSourceKey,
|
||||||
|
sourceLocationConstraintName, log, cb) => {
|
||||||
|
const client = clients[location];
|
||||||
|
if (client.uploadPartCopy) {
|
||||||
|
return client.uploadPartCopy(request, awsSourceKey,
|
||||||
|
sourceLocationConstraintName,
|
||||||
|
log, cb);
|
||||||
|
}
|
||||||
|
return cb(errors.NotImplemented.customizeDescription(
|
||||||
|
'Can not copy object from ' +
|
||||||
|
`${client.clientType} to ${client.clientType}`));
|
||||||
|
},
|
||||||
|
protectAzureBlocks: (bucketName, objectKey, location, log, cb) => {
|
||||||
|
const client = clients[location];
|
||||||
|
if (client.protectAzureBlocks) {
|
||||||
|
return client.protectAzureBlocks(bucketName, objectKey, location,
|
||||||
|
log, cb);
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
module.exports = multipleBackendGateway;
|
|
@ -0,0 +1,11 @@
|
||||||
|
const werelogs = require('werelogs');
|
||||||
|
|
||||||
|
const logger = new werelogs.Logger('MultipleBackendGateway');
|
||||||
|
|
||||||
|
function createLogger(reqUids) {
|
||||||
|
return reqUids ?
|
||||||
|
logger.newRequestLoggerFromSerializedUids(reqUids) :
|
||||||
|
logger.newRequestLogger();
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = createLogger;
|
|
@ -0,0 +1,792 @@
|
||||||
|
const async = require('async');
|
||||||
|
const { errors, s3middleware } = require('arsenal');
|
||||||
|
const PassThrough = require('stream').PassThrough;
|
||||||
|
|
||||||
|
const DataFileInterface = require('./file/backend');
|
||||||
|
const inMemory = require('./in_memory/backend').backend;
|
||||||
|
const locationConstraintCheck =
|
||||||
|
require('../api/apiUtils/object/locationConstraintCheck');
|
||||||
|
const multipleBackendGateway = require('./multipleBackendGateway');
|
||||||
|
const utils = require('./external/utils');
|
||||||
|
const { config } = require('../Config');
|
||||||
|
const MD5Sum = s3middleware.MD5Sum;
|
||||||
|
const NullStream = s3middleware.NullStream;
|
||||||
|
const assert = require('assert');
|
||||||
|
const kms = require('../kms/wrapper');
|
||||||
|
const externalBackends = require('../../constants').externalBackends;
|
||||||
|
const constants = require('../../constants');
|
||||||
|
const { BackendInfo } = require('../api/apiUtils/object/BackendInfo');
|
||||||
|
const RelayMD5Sum = require('../utilities/RelayMD5Sum');
|
||||||
|
const skipError = new Error('skip');
|
||||||
|
|
||||||
|
let CdmiData;
|
||||||
|
try {
|
||||||
|
CdmiData = require('cdmiclient').CdmiData;
|
||||||
|
} catch (err) {
|
||||||
|
CdmiData = null;
|
||||||
|
}
|
||||||
|
|
||||||
|
let client;
|
||||||
|
let implName;
|
||||||
|
|
||||||
|
if (config.backends.data === 'mem') {
|
||||||
|
client = inMemory;
|
||||||
|
implName = 'mem';
|
||||||
|
} else if (config.backends.data === 'file') {
|
||||||
|
client = new DataFileInterface();
|
||||||
|
implName = 'file';
|
||||||
|
} else if (config.backends.data === 'multiple') {
|
||||||
|
client = multipleBackendGateway;
|
||||||
|
implName = 'multipleBackends';
|
||||||
|
} else if (config.backends.data === 'cdmi') {
|
||||||
|
if (!CdmiData) {
|
||||||
|
throw new Error('Unauthorized backend');
|
||||||
|
}
|
||||||
|
|
||||||
|
client = new CdmiData({
|
||||||
|
path: config.cdmi.path,
|
||||||
|
host: config.cdmi.host,
|
||||||
|
port: config.cdmi.port,
|
||||||
|
readonly: config.cdmi.readonly,
|
||||||
|
});
|
||||||
|
implName = 'cdmi';
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* _retryDelete - Attempt to delete key again if it failed previously
|
||||||
|
* @param { string | object } objectGetInfo - either string location of object
|
||||||
|
* to delete or object containing info of object to delete
|
||||||
|
* @param {object} log - Werelogs request logger
|
||||||
|
* @param {number} count - keeps count of number of times function has been run
|
||||||
|
* @param {function} cb - callback
|
||||||
|
* @returns undefined and calls callback
|
||||||
|
*/
|
||||||
|
const MAX_RETRY = 2;
|
||||||
|
|
||||||
|
// This check is done because on a put, complete mpu or copy request to
|
||||||
|
// Azure/AWS, if the object already exists on that backend, the existing object
|
||||||
|
// should not be deleted, which is the functionality for all other backends
|
||||||
|
function _shouldSkipDelete(locations, requestMethod, newObjDataStoreName) {
|
||||||
|
const skipMethods = { PUT: true, POST: true };
|
||||||
|
if (!Array.isArray(locations) || !locations[0] ||
|
||||||
|
!locations[0].dataStoreType) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
const isSkipBackend = externalBackends[locations[0].dataStoreType];
|
||||||
|
const isMatchingBackends =
|
||||||
|
locations[0].dataStoreName === newObjDataStoreName;
|
||||||
|
const isSkipMethod = skipMethods[requestMethod];
|
||||||
|
return (isSkipBackend && isMatchingBackends && isSkipMethod);
|
||||||
|
}
|
||||||
|
|
||||||
|
function _retryDelete(objectGetInfo, log, count, cb) {
|
||||||
|
if (count > MAX_RETRY) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return client.delete(objectGetInfo, log.getSerializedUids(), err => {
|
||||||
|
if (err) {
|
||||||
|
log.error('delete error from datastore',
|
||||||
|
{ error: err, implName, moreRetries: 'yes' });
|
||||||
|
return _retryDelete(objectGetInfo, log, count + 1, cb);
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function _put(cipherBundle, value, valueSize,
|
||||||
|
keyContext, backendInfo, log, cb) {
|
||||||
|
assert.strictEqual(typeof valueSize, 'number');
|
||||||
|
log.debug('sending put to datastore', { implName, keyContext,
|
||||||
|
method: 'put' });
|
||||||
|
let hashedStream = null;
|
||||||
|
if (value) {
|
||||||
|
hashedStream = new MD5Sum();
|
||||||
|
value.pipe(hashedStream);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (implName === 'multipleBackends') {
|
||||||
|
// Need to send backendInfo to client.put and
|
||||||
|
// client.put will provide dataRetrievalInfo so no
|
||||||
|
// need to construct here
|
||||||
|
/* eslint-disable no-param-reassign */
|
||||||
|
keyContext.cipherBundle = cipherBundle;
|
||||||
|
return client.put(hashedStream,
|
||||||
|
valueSize, keyContext, backendInfo, log.getSerializedUids(),
|
||||||
|
(err, dataRetrievalInfo) => {
|
||||||
|
if (err) {
|
||||||
|
log.error('put error from datastore',
|
||||||
|
{ error: err, implName });
|
||||||
|
return cb(errors.ServiceUnavailable);
|
||||||
|
}
|
||||||
|
return cb(null, dataRetrievalInfo, hashedStream);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
/* eslint-enable no-param-reassign */
|
||||||
|
|
||||||
|
let writeStream = hashedStream;
|
||||||
|
if (cipherBundle && cipherBundle.cipher) {
|
||||||
|
writeStream = cipherBundle.cipher;
|
||||||
|
hashedStream.pipe(writeStream);
|
||||||
|
}
|
||||||
|
|
||||||
|
return client.put(writeStream, valueSize, keyContext,
|
||||||
|
log.getSerializedUids(), (err, key) => {
|
||||||
|
if (err) {
|
||||||
|
log.error('put error from datastore',
|
||||||
|
{ error: err, implName });
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
const dataRetrievalInfo = {
|
||||||
|
key,
|
||||||
|
dataStoreName: implName,
|
||||||
|
};
|
||||||
|
return cb(null, dataRetrievalInfo, hashedStream);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
const data = {
|
||||||
|
put: (cipherBundle, value, valueSize, keyContext, backendInfo, log, cb) => {
|
||||||
|
_put(cipherBundle, value, valueSize, keyContext, backendInfo, log,
|
||||||
|
(err, dataRetrievalInfo, hashedStream) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
if (hashedStream) {
|
||||||
|
if (hashedStream.completedHash) {
|
||||||
|
return cb(null, dataRetrievalInfo, hashedStream);
|
||||||
|
}
|
||||||
|
hashedStream.on('hashed', () => {
|
||||||
|
hashedStream.removeAllListeners('hashed');
|
||||||
|
return cb(null, dataRetrievalInfo, hashedStream);
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
return cb(null, dataRetrievalInfo);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
head: (objectGetInfo, log, cb) => {
|
||||||
|
if (implName !== 'multipleBackends') {
|
||||||
|
// no-op if not multipleBackend implementation;
|
||||||
|
// head is used during get just to check external backend data state
|
||||||
|
return process.nextTick(cb);
|
||||||
|
}
|
||||||
|
return client.head(objectGetInfo, log.getSerializedUids(), cb);
|
||||||
|
},
|
||||||
|
|
||||||
|
get: (objectGetInfo, response, log, cb) => {
|
||||||
|
const isMdModelVersion2 = typeof(objectGetInfo) === 'string';
|
||||||
|
const isRequiredStringKey = constants.clientsRequireStringKey[implName];
|
||||||
|
const key = isMdModelVersion2 ? objectGetInfo : objectGetInfo.key;
|
||||||
|
const clientGetInfo = isRequiredStringKey ? key : objectGetInfo;
|
||||||
|
const range = objectGetInfo.range;
|
||||||
|
|
||||||
|
// If the key is explicitly set to null, the part to
|
||||||
|
// be read doesn't really exist and is only made of zeroes.
|
||||||
|
// This functionality is used by Scality-NFSD.
|
||||||
|
// Otherwise, the key is always defined
|
||||||
|
assert(key === null || key !== undefined);
|
||||||
|
if (key === null) {
|
||||||
|
cb(null, new NullStream(objectGetInfo.size, range));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
log.debug('sending get to datastore', { implName,
|
||||||
|
key, range, method: 'get' });
|
||||||
|
// We need to use response as a writable stream for AZURE GET
|
||||||
|
if (!isMdModelVersion2 && !isRequiredStringKey && response) {
|
||||||
|
clientGetInfo.response = response;
|
||||||
|
}
|
||||||
|
client.get(clientGetInfo, range, log.getSerializedUids(),
|
||||||
|
(err, stream) => {
|
||||||
|
if (err) {
|
||||||
|
log.error('get error from datastore',
|
||||||
|
{ error: err, implName });
|
||||||
|
return cb(errors.ServiceUnavailable);
|
||||||
|
}
|
||||||
|
if (objectGetInfo.cipheredDataKey) {
|
||||||
|
const serverSideEncryption = {
|
||||||
|
cryptoScheme: objectGetInfo.cryptoScheme,
|
||||||
|
masterKeyId: objectGetInfo.masterKeyId,
|
||||||
|
cipheredDataKey: Buffer.from(
|
||||||
|
objectGetInfo.cipheredDataKey, 'base64'),
|
||||||
|
};
|
||||||
|
const offset = objectGetInfo.range ?
|
||||||
|
objectGetInfo.range[0] : 0;
|
||||||
|
return kms.createDecipherBundle(
|
||||||
|
serverSideEncryption, offset, log,
|
||||||
|
(err, decipherBundle) => {
|
||||||
|
if (err) {
|
||||||
|
log.error('cannot get decipher bundle ' +
|
||||||
|
'from kms', {
|
||||||
|
method: 'data.wrapper.data.get',
|
||||||
|
});
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
stream.pipe(decipherBundle.decipher);
|
||||||
|
return cb(null, decipherBundle.decipher);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return cb(null, stream);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
delete: (objectGetInfo, log, cb) => {
|
||||||
|
const callback = cb || log.end;
|
||||||
|
const isMdModelVersion2 = typeof(objectGetInfo) === 'string';
|
||||||
|
const isRequiredStringKey = constants.clientsRequireStringKey[implName];
|
||||||
|
const key = isMdModelVersion2 ? objectGetInfo : objectGetInfo.key;
|
||||||
|
const clientGetInfo = isRequiredStringKey ? key : objectGetInfo;
|
||||||
|
|
||||||
|
log.trace('sending delete to datastore', {
|
||||||
|
implName, key, method: 'delete' });
|
||||||
|
// If the key is explicitly set to null, the part to
|
||||||
|
// be deleted doesn't really exist.
|
||||||
|
// This functionality is used by Scality-NFSD.
|
||||||
|
// Otherwise, the key is always defined
|
||||||
|
assert(key === null || key !== undefined);
|
||||||
|
if (key === null) {
|
||||||
|
callback(null);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
_retryDelete(clientGetInfo, log, 0, err => {
|
||||||
|
if (err) {
|
||||||
|
log.error('delete error from datastore',
|
||||||
|
{ error: err, key: objectGetInfo.key, moreRetries: 'no' });
|
||||||
|
}
|
||||||
|
return callback(err);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
// It would be preferable to have an sproxyd batch delete route to
|
||||||
|
// replace this
|
||||||
|
batchDelete: (locations, requestMethod, newObjDataStoreName, log) => {
|
||||||
|
// TODO: The method of persistence of sproxy delete key will
|
||||||
|
// be finalized; refer Issue #312 for the discussion. In the
|
||||||
|
// meantime, we at least log the location of the data we are
|
||||||
|
// about to delete before attempting its deletion.
|
||||||
|
if (_shouldSkipDelete(locations, requestMethod, newObjDataStoreName)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
log.trace('initiating batch delete', {
|
||||||
|
keys: locations,
|
||||||
|
implName,
|
||||||
|
method: 'batchDelete',
|
||||||
|
});
|
||||||
|
async.eachLimit(locations, 5, (loc, next) => {
|
||||||
|
process.nextTick(() => data.delete(loc, log, next));
|
||||||
|
},
|
||||||
|
err => {
|
||||||
|
if (err) {
|
||||||
|
log.error('batch delete failed', { error: err });
|
||||||
|
} else {
|
||||||
|
log.trace('batch delete successfully completed');
|
||||||
|
}
|
||||||
|
log.end();
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
switch: newClient => {
|
||||||
|
client = newClient;
|
||||||
|
return client;
|
||||||
|
},
|
||||||
|
|
||||||
|
checkHealth: (log, cb, flightCheckOnStartUp) => {
|
||||||
|
if (!client.healthcheck) {
|
||||||
|
const defResp = {};
|
||||||
|
defResp[implName] = { code: 200, message: 'OK' };
|
||||||
|
return cb(null, defResp);
|
||||||
|
}
|
||||||
|
return client.healthcheck(flightCheckOnStartUp, log, (err, result) => {
|
||||||
|
let respBody = {};
|
||||||
|
if (err) {
|
||||||
|
log.error(`error from ${implName}`, { error: err });
|
||||||
|
respBody[implName] = {
|
||||||
|
error: err,
|
||||||
|
};
|
||||||
|
// error returned as null so async parallel doesn't return
|
||||||
|
// before all backends are checked
|
||||||
|
return cb(null, respBody);
|
||||||
|
}
|
||||||
|
if (implName === 'multipleBackends') {
|
||||||
|
respBody = result;
|
||||||
|
return cb(null, respBody);
|
||||||
|
}
|
||||||
|
respBody[implName] = {
|
||||||
|
code: result.statusCode,
|
||||||
|
message: result.statusMessage,
|
||||||
|
};
|
||||||
|
return cb(null, respBody);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
getDiskUsage: (log, cb) => {
|
||||||
|
if (!client.getDiskUsage) {
|
||||||
|
log.debug('returning empty disk usage as fallback', { implName });
|
||||||
|
return cb(null, {});
|
||||||
|
}
|
||||||
|
return client.getDiskUsage(log.getSerializedUids(), cb);
|
||||||
|
},
|
||||||
|
|
||||||
|
|
||||||
|
/**
|
||||||
|
* _putForCopy - put used for copying object
|
||||||
|
* @param {object} cipherBundle - cipher bundle that encrypt the data
|
||||||
|
* @param {object} stream - stream containing the data
|
||||||
|
* @param {object} part - element of dataLocator array
|
||||||
|
* @param {object} dataStoreContext - information of the
|
||||||
|
* destination object
|
||||||
|
* dataStoreContext.bucketName: destination bucket name,
|
||||||
|
* dataStoreContext.owner: owner,
|
||||||
|
* dataStoreContext.namespace: request namespace,
|
||||||
|
* dataStoreContext.objectKey: destination object key name,
|
||||||
|
* @param {BackendInfo} destBackendInfo - Instance of BackendInfo:
|
||||||
|
* Represents the info necessary to evaluate which data backend to use
|
||||||
|
* on a data put call.
|
||||||
|
* @param {object} log - Werelogs request logger
|
||||||
|
* @param {function} cb - callback
|
||||||
|
* @returns {function} cb - callback
|
||||||
|
*/
|
||||||
|
_putForCopy: (cipherBundle, stream, part, dataStoreContext,
|
||||||
|
destBackendInfo, log, cb) => data.put(cipherBundle, stream,
|
||||||
|
part.size, dataStoreContext,
|
||||||
|
destBackendInfo, log,
|
||||||
|
(error, partRetrievalInfo) => {
|
||||||
|
if (error) {
|
||||||
|
return cb(error);
|
||||||
|
}
|
||||||
|
const partResult = {
|
||||||
|
key: partRetrievalInfo.key,
|
||||||
|
dataStoreName: partRetrievalInfo
|
||||||
|
.dataStoreName,
|
||||||
|
dataStoreType: partRetrievalInfo
|
||||||
|
.dataStoreType,
|
||||||
|
start: part.start,
|
||||||
|
size: part.size,
|
||||||
|
};
|
||||||
|
if (cipherBundle) {
|
||||||
|
partResult.cryptoScheme = cipherBundle.cryptoScheme;
|
||||||
|
partResult.cipheredDataKey = cipherBundle.cipheredDataKey;
|
||||||
|
}
|
||||||
|
if (part.dataStoreETag) {
|
||||||
|
partResult.dataStoreETag = part.dataStoreETag;
|
||||||
|
}
|
||||||
|
if (partRetrievalInfo.dataStoreVersionId) {
|
||||||
|
partResult.dataStoreVersionId =
|
||||||
|
partRetrievalInfo.dataStoreVersionId;
|
||||||
|
}
|
||||||
|
return cb(null, partResult);
|
||||||
|
}),
|
||||||
|
|
||||||
|
/**
|
||||||
|
* _dataCopyPut - put used for copying object with and without
|
||||||
|
* encryption
|
||||||
|
* @param {string} serverSideEncryption - Server side encryption
|
||||||
|
* @param {object} stream - stream containing the data
|
||||||
|
* @param {object} part - element of dataLocator array
|
||||||
|
* @param {object} dataStoreContext - information of the
|
||||||
|
* destination object
|
||||||
|
* dataStoreContext.bucketName: destination bucket name,
|
||||||
|
* dataStoreContext.owner: owner,
|
||||||
|
* dataStoreContext.namespace: request namespace,
|
||||||
|
* dataStoreContext.objectKey: destination object key name,
|
||||||
|
* @param {BackendInfo} destBackendInfo - Instance of BackendInfo:
|
||||||
|
* Represents the info necessary to evaluate which data backend to use
|
||||||
|
* on a data put call.
|
||||||
|
* @param {object} log - Werelogs request logger
|
||||||
|
* @param {function} cb - callback
|
||||||
|
* @returns {function} cb - callback
|
||||||
|
*/
|
||||||
|
_dataCopyPut: (serverSideEncryption, stream, part, dataStoreContext,
|
||||||
|
destBackendInfo, log, cb) => {
|
||||||
|
if (serverSideEncryption) {
|
||||||
|
return kms.createCipherBundle(
|
||||||
|
serverSideEncryption,
|
||||||
|
log, (err, cipherBundle) => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('error getting cipherBundle');
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return data._putForCopy(cipherBundle, stream, part,
|
||||||
|
dataStoreContext, destBackendInfo, log, cb);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
// Copied object is not encrypted so just put it
|
||||||
|
// without a cipherBundle
|
||||||
|
return data._putForCopy(null, stream, part, dataStoreContext,
|
||||||
|
destBackendInfo, log, cb);
|
||||||
|
},
|
||||||
|
|
||||||
|
/**
|
||||||
|
* copyObject - copy object
|
||||||
|
* @param {object} request - request object
|
||||||
|
* @param {string} sourceLocationConstraintName -
|
||||||
|
* source locationContraint name (awsbackend, azurebackend, ...)
|
||||||
|
* @param {object} storeMetadataParams - metadata information of the
|
||||||
|
* source object
|
||||||
|
* @param {array} dataLocator - source object metadata location(s)
|
||||||
|
* NOTE: for Azure and AWS data backend this array only has one item
|
||||||
|
* @param {object} dataStoreContext - information of the
|
||||||
|
* destination object
|
||||||
|
* dataStoreContext.bucketName: destination bucket name,
|
||||||
|
* dataStoreContext.owner: owner,
|
||||||
|
* dataStoreContext.namespace: request namespace,
|
||||||
|
* dataStoreContext.objectKey: destination object key name,
|
||||||
|
* @param {BackendInfo} destBackendInfo - Instance of BackendInfo:
|
||||||
|
* Represents the info necessary to evaluate which data backend to use
|
||||||
|
* on a data put call.
|
||||||
|
* @param {object} sourceBucketMD - metadata of the source bucket
|
||||||
|
* @param {object} destBucketMD - metadata of the destination bucket
|
||||||
|
* @param {object} log - Werelogs request logger
|
||||||
|
* @param {function} cb - callback
|
||||||
|
* @returns {function} cb - callback
|
||||||
|
*/
|
||||||
|
copyObject: (request,
|
||||||
|
sourceLocationConstraintName, storeMetadataParams, dataLocator,
|
||||||
|
dataStoreContext, destBackendInfo, sourceBucketMD, destBucketMD, log,
|
||||||
|
cb) => {
|
||||||
|
const serverSideEncryption = destBucketMD.getServerSideEncryption();
|
||||||
|
if (config.backends.data === 'multiple' &&
|
||||||
|
utils.externalBackendCopy(sourceLocationConstraintName,
|
||||||
|
storeMetadataParams.dataStoreName, sourceBucketMD, destBucketMD)) {
|
||||||
|
const destLocationConstraintName =
|
||||||
|
storeMetadataParams.dataStoreName;
|
||||||
|
const objectGetInfo = dataLocator[0];
|
||||||
|
const externalSourceKey = objectGetInfo.key;
|
||||||
|
return client.copyObject(request, destLocationConstraintName,
|
||||||
|
externalSourceKey, sourceLocationConstraintName,
|
||||||
|
storeMetadataParams, log, (error, objectRetrievalInfo) => {
|
||||||
|
if (error) {
|
||||||
|
return cb(error);
|
||||||
|
}
|
||||||
|
const putResult = {
|
||||||
|
key: objectRetrievalInfo.key,
|
||||||
|
dataStoreName: objectRetrievalInfo.
|
||||||
|
dataStoreName,
|
||||||
|
dataStoreType: objectRetrievalInfo.
|
||||||
|
dataStoreType,
|
||||||
|
dataStoreVersionId:
|
||||||
|
objectRetrievalInfo.dataStoreVersionId,
|
||||||
|
size: storeMetadataParams.size,
|
||||||
|
dataStoreETag: objectGetInfo.dataStoreETag,
|
||||||
|
start: objectGetInfo.start,
|
||||||
|
};
|
||||||
|
const putResultArr = [putResult];
|
||||||
|
return cb(null, putResultArr);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// dataLocator is an array. need to get and put all parts
|
||||||
|
// For now, copy 1 part at a time. Could increase the second
|
||||||
|
// argument here to increase the number of parts
|
||||||
|
// copied at once.
|
||||||
|
return async.mapLimit(dataLocator, 1,
|
||||||
|
// eslint-disable-next-line prefer-arrow-callback
|
||||||
|
function copyPart(part, copyCb) {
|
||||||
|
if (part.dataStoreType === 'azure') {
|
||||||
|
const passThrough = new PassThrough();
|
||||||
|
return async.parallel([
|
||||||
|
parallelCb => data.get(part, passThrough, log, err =>
|
||||||
|
parallelCb(err)),
|
||||||
|
parallelCb => data._dataCopyPut(serverSideEncryption,
|
||||||
|
passThrough,
|
||||||
|
part, dataStoreContext, destBackendInfo, log,
|
||||||
|
parallelCb),
|
||||||
|
], (err, res) => {
|
||||||
|
if (err) {
|
||||||
|
return copyCb(err);
|
||||||
|
}
|
||||||
|
return copyCb(null, res[1]);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return data.get(part, null, log, (err, stream) => {
|
||||||
|
if (err) {
|
||||||
|
return copyCb(err);
|
||||||
|
}
|
||||||
|
return data._dataCopyPut(serverSideEncryption, stream,
|
||||||
|
part, dataStoreContext, destBackendInfo, log, copyCb);
|
||||||
|
});
|
||||||
|
}, (err, results) => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('error transferring data from source',
|
||||||
|
{ error: err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
return cb(null, results);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
|
||||||
|
_dataCopyPutPart: (request,
|
||||||
|
serverSideEncryption, stream, part,
|
||||||
|
dataStoreContext, destBackendInfo, locations, log, cb) => {
|
||||||
|
const numberPartSize =
|
||||||
|
Number.parseInt(part.size, 10);
|
||||||
|
const partNumber = Number.parseInt(request.query.partNumber, 10);
|
||||||
|
const uploadId = request.query.uploadId;
|
||||||
|
const destObjectKey = request.objectKey;
|
||||||
|
const destBucketName = request.bucketName;
|
||||||
|
const destLocationConstraintName = destBackendInfo
|
||||||
|
.getControllingLocationConstraint();
|
||||||
|
if (externalBackends[config
|
||||||
|
.locationConstraints[destLocationConstraintName]
|
||||||
|
.type]) {
|
||||||
|
return multipleBackendGateway.uploadPart(null, null,
|
||||||
|
stream, numberPartSize,
|
||||||
|
destLocationConstraintName, destObjectKey, uploadId,
|
||||||
|
partNumber, destBucketName, log,
|
||||||
|
(err, partInfo) => {
|
||||||
|
if (err) {
|
||||||
|
log.error('error putting ' +
|
||||||
|
'part to AWS', {
|
||||||
|
error: err,
|
||||||
|
method:
|
||||||
|
'objectPutCopyPart::' +
|
||||||
|
'multipleBackendGateway.' +
|
||||||
|
'uploadPart',
|
||||||
|
});
|
||||||
|
return cb(errors.ServiceUnavailable);
|
||||||
|
}
|
||||||
|
// skip to end of waterfall
|
||||||
|
// because don't need to store
|
||||||
|
// part metadata
|
||||||
|
if (partInfo &&
|
||||||
|
partInfo.dataStoreType === 'aws_s3') {
|
||||||
|
// if data backend handles MPU, skip to end
|
||||||
|
// of waterfall
|
||||||
|
const partResult = {
|
||||||
|
dataStoreETag: partInfo.dataStoreETag,
|
||||||
|
};
|
||||||
|
locations.push(partResult);
|
||||||
|
return cb(skipError, partInfo.dataStoreETag);
|
||||||
|
} else if (
|
||||||
|
partInfo &&
|
||||||
|
partInfo.dataStoreType === 'azure') {
|
||||||
|
const partResult = {
|
||||||
|
key: partInfo.key,
|
||||||
|
dataStoreName: partInfo.dataStoreName,
|
||||||
|
dataStoreETag: partInfo.dataStoreETag,
|
||||||
|
size: numberPartSize,
|
||||||
|
numberSubParts:
|
||||||
|
partInfo.numberSubParts,
|
||||||
|
partNumber: partInfo.partNumber,
|
||||||
|
};
|
||||||
|
locations.push(partResult);
|
||||||
|
return cb();
|
||||||
|
}
|
||||||
|
return cb(skipError);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if (serverSideEncryption) {
|
||||||
|
return kms.createCipherBundle(
|
||||||
|
serverSideEncryption,
|
||||||
|
log, (err, cipherBundle) => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('error getting cipherBundle',
|
||||||
|
{ error: err });
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return data.put(cipherBundle, stream,
|
||||||
|
numberPartSize, dataStoreContext,
|
||||||
|
destBackendInfo, log,
|
||||||
|
(error, partRetrievalInfo,
|
||||||
|
hashedStream) => {
|
||||||
|
if (error) {
|
||||||
|
log.debug('error putting ' +
|
||||||
|
'encrypted part', { error });
|
||||||
|
return cb(error);
|
||||||
|
}
|
||||||
|
const partResult = {
|
||||||
|
key: partRetrievalInfo.key,
|
||||||
|
dataStoreName: partRetrievalInfo
|
||||||
|
.dataStoreName,
|
||||||
|
dataStoreETag: hashedStream
|
||||||
|
.completedHash,
|
||||||
|
// Do not include part start
|
||||||
|
// here since will change in
|
||||||
|
// final MPU object
|
||||||
|
size: numberPartSize,
|
||||||
|
sseCryptoScheme: cipherBundle
|
||||||
|
.cryptoScheme,
|
||||||
|
sseCipheredDataKey: cipherBundle
|
||||||
|
.cipheredDataKey,
|
||||||
|
sseAlgorithm: cipherBundle
|
||||||
|
.algorithm,
|
||||||
|
sseMasterKeyId: cipherBundle
|
||||||
|
.masterKeyId,
|
||||||
|
};
|
||||||
|
locations.push(partResult);
|
||||||
|
return cb();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
// Copied object is not encrypted so just put it
|
||||||
|
// without a cipherBundle
|
||||||
|
return data.put(null, stream, numberPartSize,
|
||||||
|
dataStoreContext, destBackendInfo,
|
||||||
|
log, (error, partRetrievalInfo, hashedStream) => {
|
||||||
|
if (error) {
|
||||||
|
log.debug('error putting object part',
|
||||||
|
{ error });
|
||||||
|
return cb(error);
|
||||||
|
}
|
||||||
|
const partResult = {
|
||||||
|
key: partRetrievalInfo.key,
|
||||||
|
dataStoreName: partRetrievalInfo.dataStoreName,
|
||||||
|
dataStoreETag: hashedStream.completedHash,
|
||||||
|
size: numberPartSize,
|
||||||
|
};
|
||||||
|
locations.push(partResult);
|
||||||
|
return cb();
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
/**
|
||||||
|
* uploadPartCopy - put copy part
|
||||||
|
* @param {object} request - request object
|
||||||
|
* @param {object} log - Werelogs request logger
|
||||||
|
* @param {object} destBucketMD - destination bucket metadata
|
||||||
|
* @param {string} sourceLocationConstraintName -
|
||||||
|
* source locationContraint name (awsbackend, azurebackend, ...)
|
||||||
|
* @param {string} destLocationConstraintName -
|
||||||
|
* location of the destination MPU object (awsbackend, azurebackend, ...)
|
||||||
|
* @param {array} dataLocator - source object metadata location(s)
|
||||||
|
* NOTE: for Azure and AWS data backend this array
|
||||||
|
* @param {object} dataStoreContext - information of the
|
||||||
|
* destination object
|
||||||
|
* dataStoreContext.bucketName: destination bucket name,
|
||||||
|
* dataStoreContext.owner: owner,
|
||||||
|
* dataStoreContext.namespace: request namespace,
|
||||||
|
* dataStoreContext.objectKey: destination object key name,
|
||||||
|
* dataStoreContext.uploadId: uploadId
|
||||||
|
* dataStoreContext.partNumber: request.query.partNumber
|
||||||
|
* @param {function} callback - callback
|
||||||
|
* @returns {function} cb - callback
|
||||||
|
*/
|
||||||
|
uploadPartCopy: (request, log, destBucketMD, sourceLocationConstraintName,
|
||||||
|
destLocationConstraintName, dataLocator, dataStoreContext,
|
||||||
|
callback) => {
|
||||||
|
const serverSideEncryption = destBucketMD.getServerSideEncryption();
|
||||||
|
const lastModified = new Date().toJSON();
|
||||||
|
|
||||||
|
// skip if 0 byte object
|
||||||
|
if (dataLocator.length === 0) {
|
||||||
|
return process.nextTick(() => {
|
||||||
|
callback(null, constants.emptyFileMd5,
|
||||||
|
lastModified, serverSideEncryption, []);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// if destination mpu was initiated in legacy version
|
||||||
|
if (destLocationConstraintName === undefined) {
|
||||||
|
const backendInfoObj = locationConstraintCheck(request,
|
||||||
|
null, destBucketMD, log);
|
||||||
|
if (backendInfoObj.err) {
|
||||||
|
return process.nextTick(() => {
|
||||||
|
callback(backendInfoObj.err);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
// eslint-disable-next-line no-param-reassign
|
||||||
|
destLocationConstraintName = backendInfoObj.controllingLC;
|
||||||
|
}
|
||||||
|
|
||||||
|
const locationTypeMatchAWS =
|
||||||
|
config.backends.data === 'multiple' &&
|
||||||
|
config.getLocationConstraintType(sourceLocationConstraintName) ===
|
||||||
|
config.getLocationConstraintType(destLocationConstraintName) &&
|
||||||
|
config.getLocationConstraintType(sourceLocationConstraintName) ===
|
||||||
|
'aws_s3';
|
||||||
|
|
||||||
|
// NOTE: using multipleBackendGateway.uploadPartCopy only if copying
|
||||||
|
// from AWS to AWS
|
||||||
|
|
||||||
|
if (locationTypeMatchAWS && dataLocator.length === 1) {
|
||||||
|
const awsSourceKey = dataLocator[0].key;
|
||||||
|
return multipleBackendGateway.uploadPartCopy(request,
|
||||||
|
destLocationConstraintName, awsSourceKey,
|
||||||
|
sourceLocationConstraintName, log, (error, eTag) => {
|
||||||
|
if (error) {
|
||||||
|
return callback(error);
|
||||||
|
}
|
||||||
|
return callback(skipError, eTag,
|
||||||
|
lastModified, serverSideEncryption);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
const backendInfo = new BackendInfo(destLocationConstraintName);
|
||||||
|
|
||||||
|
// totalHash will be sent through the RelayMD5Sum transform streams
|
||||||
|
// to collect the md5 from multiple streams
|
||||||
|
let totalHash;
|
||||||
|
const locations = [];
|
||||||
|
// dataLocator is an array. need to get and put all parts
|
||||||
|
// in order so can get the ETag of full object
|
||||||
|
return async.forEachOfSeries(dataLocator,
|
||||||
|
// eslint-disable-next-line prefer-arrow-callback
|
||||||
|
function copyPart(part, index, cb) {
|
||||||
|
if (part.dataStoreType === 'azure') {
|
||||||
|
const passThrough = new PassThrough();
|
||||||
|
return async.parallel([
|
||||||
|
next => data.get(part, passThrough, log, err => {
|
||||||
|
if (err) {
|
||||||
|
log.error('error getting data part ' +
|
||||||
|
'from Azure', {
|
||||||
|
error: err,
|
||||||
|
method:
|
||||||
|
'objectPutCopyPart::' +
|
||||||
|
'multipleBackendGateway.' +
|
||||||
|
'copyPart',
|
||||||
|
});
|
||||||
|
return next(err);
|
||||||
|
}
|
||||||
|
return next();
|
||||||
|
}),
|
||||||
|
next => data._dataCopyPutPart(request,
|
||||||
|
serverSideEncryption, passThrough, part,
|
||||||
|
dataStoreContext, backendInfo, locations, log, next),
|
||||||
|
], err => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return data.get(part, null, log, (err, stream) => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('error getting object part',
|
||||||
|
{ error: err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
const hashedStream =
|
||||||
|
new RelayMD5Sum(totalHash, updatedHash => {
|
||||||
|
totalHash = updatedHash;
|
||||||
|
});
|
||||||
|
stream.pipe(hashedStream);
|
||||||
|
|
||||||
|
// destLocationConstraintName is location of the
|
||||||
|
// destination MPU object
|
||||||
|
return data._dataCopyPutPart(request,
|
||||||
|
serverSideEncryption, hashedStream, part,
|
||||||
|
dataStoreContext, backendInfo, locations, log, cb);
|
||||||
|
});
|
||||||
|
}, err => {
|
||||||
|
// Digest the final combination of all of the part streams
|
||||||
|
if (err && err !== skipError) {
|
||||||
|
log.debug('error transferring data from source',
|
||||||
|
{ error: err, method: 'goGetData' });
|
||||||
|
return callback(err);
|
||||||
|
}
|
||||||
|
if (totalHash) {
|
||||||
|
totalHash = totalHash.digest('hex');
|
||||||
|
} else {
|
||||||
|
totalHash = locations[0].dataStoreETag;
|
||||||
|
}
|
||||||
|
if (err && err === skipError) {
|
||||||
|
return callback(skipError, totalHash,
|
||||||
|
lastModified, serverSideEncryption);
|
||||||
|
}
|
||||||
|
return callback(null, totalHash,
|
||||||
|
lastModified, serverSideEncryption, locations);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
module.exports = data;
|
|
@ -0,0 +1,82 @@
|
||||||
|
# BucketInfo Model Version History
|
||||||
|
|
||||||
|
## Model Version 0/1
|
||||||
|
|
||||||
|
### Properties
|
||||||
|
|
||||||
|
``` javascript
|
||||||
|
this._acl = aclInstance;
|
||||||
|
this._name = name;
|
||||||
|
this._owner = owner;
|
||||||
|
this._ownerDisplayName = ownerDisplayName;
|
||||||
|
this._creationDate = creationDate;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
|
||||||
|
No explicit references in the code since mdBucketModelVersion
|
||||||
|
property not added until Model Version 2
|
||||||
|
|
||||||
|
## Model Version 2
|
||||||
|
|
||||||
|
### Properties Added
|
||||||
|
|
||||||
|
``` javascript
|
||||||
|
this._mdBucketModelVersion = mdBucketModelVersion || 0
|
||||||
|
this._transient = transient || false;
|
||||||
|
this._deleted = deleted || false;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
|
||||||
|
Used to determine which splitter to use ( < 2 means old splitter)
|
||||||
|
|
||||||
|
## Model version 3
|
||||||
|
|
||||||
|
### Properties Added
|
||||||
|
|
||||||
|
```
|
||||||
|
this._serverSideEncryption = serverSideEncryption || null;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
|
||||||
|
Used to store the server bucket encryption info
|
||||||
|
|
||||||
|
## Model version 4
|
||||||
|
|
||||||
|
### Properties Added
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
this._locationConstraint = LocationConstraint || null;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
|
||||||
|
Used to store the location constraint of the bucket
|
||||||
|
|
||||||
|
## Model version 5
|
||||||
|
|
||||||
|
### Properties Added
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
this._websiteConfiguration = websiteConfiguration || null;
|
||||||
|
this._cors = cors || null;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
|
||||||
|
Used to store the bucket website configuration info
|
||||||
|
and to store CORS rules to apply to cross-domain requests
|
||||||
|
|
||||||
|
## Model version 6
|
||||||
|
|
||||||
|
### Properties Added
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
this._lifecycleConfiguration = lifecycleConfiguration || null;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
|
||||||
|
Used to store the bucket lifecycle configuration info
|
|
@ -1,5 +1,6 @@
|
||||||
const { errors } = require('arsenal');
|
const { errors } = require('arsenal');
|
||||||
|
|
||||||
|
const getReplicationInfo = require('../api/apiUtils/object/getReplicationInfo');
|
||||||
const aclUtils = require('../utilities/aclUtils');
|
const aclUtils = require('../utilities/aclUtils');
|
||||||
const constants = require('../../constants');
|
const constants = require('../../constants');
|
||||||
const metadata = require('../metadata/wrapper');
|
const metadata = require('../metadata/wrapper');
|
||||||
|
@ -12,40 +13,18 @@ const acl = {
|
||||||
metadata.updateBucket(bucket.getName(), bucket, log, cb);
|
metadata.updateBucket(bucket.getName(), bucket, log, cb);
|
||||||
},
|
},
|
||||||
|
|
||||||
/**
|
|
||||||
* returns true if the specified ACL grant is unchanged
|
|
||||||
* @param {string} grant name of the grant
|
|
||||||
* @param {object} oldAcl old acl config
|
|
||||||
* @param {object} newAcl new acl config
|
|
||||||
* @returns {bool} is the grant the same
|
|
||||||
*/
|
|
||||||
_aclGrantDidNotChange(grant, oldAcl, newAcl) {
|
|
||||||
if (grant === 'Canned') {
|
|
||||||
return oldAcl.Canned === newAcl.Canned;
|
|
||||||
}
|
|
||||||
/**
|
|
||||||
* An ACL grant is in form of an array of strings
|
|
||||||
* An ACL grant is considered unchanged when both the old and new one
|
|
||||||
* contain the same number of elements, and all elements from one
|
|
||||||
* grant are incuded in the other grant
|
|
||||||
*/
|
|
||||||
return oldAcl[grant].length === newAcl[grant].length
|
|
||||||
&& oldAcl[grant].every(value => newAcl[grant].includes(value));
|
|
||||||
},
|
|
||||||
|
|
||||||
addObjectACL(bucket, objectKey, objectMD, addACLParams, params, log, cb) {
|
addObjectACL(bucket, objectKey, objectMD, addACLParams, params, log, cb) {
|
||||||
log.trace('updating object acl in metadata');
|
log.trace('updating object acl in metadata');
|
||||||
const isAclUnchanged = Object.keys(objectMD.acl).length === Object.keys(addACLParams).length
|
// eslint-disable-next-line no-param-reassign
|
||||||
&& Object.keys(objectMD.acl).every(grant => this._aclGrantDidNotChange(grant, objectMD.acl, addACLParams));
|
objectMD.acl = addACLParams;
|
||||||
if (!isAclUnchanged) {
|
const replicationInfo = getReplicationInfo(objectKey, bucket, true);
|
||||||
/* eslint-disable no-param-reassign */
|
if (replicationInfo) {
|
||||||
objectMD.acl = addACLParams;
|
// eslint-disable-next-line no-param-reassign
|
||||||
objectMD.originOp = 's3:ObjectAcl:Put';
|
objectMD.replicationInfo = Object.assign({},
|
||||||
/* eslint-disable no-param-reassign */
|
objectMD.replicationInfo, replicationInfo);
|
||||||
return metadata.putObjectMD(bucket.getName(), objectKey, objectMD, params, log,
|
|
||||||
cb);
|
|
||||||
}
|
}
|
||||||
return cb();
|
metadata.putObjectMD(bucket.getName(), objectKey, objectMD, params, log,
|
||||||
|
cb);
|
||||||
},
|
},
|
||||||
|
|
||||||
parseAclFromHeaders(params, cb) {
|
parseAclFromHeaders(params, cb) {
|
||||||
|
@ -157,4 +136,3 @@ const acl = {
|
||||||
};
|
};
|
||||||
|
|
||||||
module.exports = acl;
|
module.exports = acl;
|
||||||
|
|
|
@ -0,0 +1,213 @@
|
||||||
|
const assert = require('assert');
|
||||||
|
const bucketclient = require('bucketclient');
|
||||||
|
|
||||||
|
const BucketInfo = require('arsenal').models.BucketInfo;
|
||||||
|
const logger = require('../../utilities/logger');
|
||||||
|
const { config } = require('../../Config');
|
||||||
|
|
||||||
|
class BucketClientInterface {
|
||||||
|
constructor() {
|
||||||
|
assert(config.bucketd.bootstrap.length > 0,
|
||||||
|
'bucketd bootstrap list is empty');
|
||||||
|
const { bootstrap, log } = config.bucketd;
|
||||||
|
if (config.https) {
|
||||||
|
const { key, cert, ca } = config.https;
|
||||||
|
logger.info('bucketclient configuration', {
|
||||||
|
bootstrap,
|
||||||
|
log,
|
||||||
|
https: true,
|
||||||
|
});
|
||||||
|
this.client = new bucketclient.RESTClient(bootstrap, log, true,
|
||||||
|
key, cert, ca);
|
||||||
|
} else {
|
||||||
|
logger.info('bucketclient configuration', {
|
||||||
|
bootstrap,
|
||||||
|
log,
|
||||||
|
https: false,
|
||||||
|
});
|
||||||
|
this.client = new bucketclient.RESTClient(bootstrap, log);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
createBucket(bucketName, bucketMD, log, cb) {
|
||||||
|
this.client.createBucket(bucketName, log.getSerializedUids(),
|
||||||
|
bucketMD.serialize(), cb);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
getBucketAttributes(bucketName, log, cb) {
|
||||||
|
this.client.getBucketAttributes(bucketName, log.getSerializedUids(),
|
||||||
|
(err, data) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
return cb(err, BucketInfo.deSerialize(data));
|
||||||
|
});
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
getBucketAndObject(bucketName, objName, params, log, cb) {
|
||||||
|
this.client.getBucketAndObject(bucketName, objName,
|
||||||
|
log.getSerializedUids(), (err, data) => {
|
||||||
|
if (err && (!err.NoSuchKey && !err.ObjNotFound)) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
return cb(null, JSON.parse(data));
|
||||||
|
}, params);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
getRaftBuckets(raftId, log, cb) {
|
||||||
|
return this.client.getRaftBuckets(raftId, log.getSerializedUids(),
|
||||||
|
(err, data) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
return cb(null, JSON.parse(data));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
putBucketAttributes(bucketName, bucketMD, log, cb) {
|
||||||
|
this.client.putBucketAttributes(bucketName, log.getSerializedUids(),
|
||||||
|
bucketMD.serialize(), cb);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
deleteBucket(bucketName, log, cb) {
|
||||||
|
this.client.deleteBucket(bucketName, log.getSerializedUids(), cb);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
putObject(bucketName, objName, objVal, params, log, cb) {
|
||||||
|
this.client.putObject(bucketName, objName, JSON.stringify(objVal),
|
||||||
|
log.getSerializedUids(), cb, params);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
getObject(bucketName, objName, params, log, cb) {
|
||||||
|
this.client.getObject(bucketName, objName, log.getSerializedUids(),
|
||||||
|
(err, data) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
return cb(err, JSON.parse(data));
|
||||||
|
}, params);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
deleteObject(bucketName, objName, params, log, cb) {
|
||||||
|
this.client.deleteObject(bucketName, objName, log.getSerializedUids(),
|
||||||
|
cb, params);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
listObject(bucketName, params, log, cb) {
|
||||||
|
this.client.listObject(bucketName, log.getSerializedUids(), params,
|
||||||
|
(err, data) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
return cb(err, JSON.parse(data));
|
||||||
|
});
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
listMultipartUploads(bucketName, params, log, cb) {
|
||||||
|
this.client.listObject(bucketName, log.getSerializedUids(), params,
|
||||||
|
(err, data) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
return cb(null, JSON.parse(data));
|
||||||
|
});
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
_analyzeHealthFailure(log, callback) {
|
||||||
|
let doFail = false;
|
||||||
|
const reason = {
|
||||||
|
msg: 'Map is available and failure ratio is acceptable',
|
||||||
|
};
|
||||||
|
|
||||||
|
// The healthCheck exposed by Bucketd is a light one, we need
|
||||||
|
// to inspect all the RaftSession's statuses to make sense of
|
||||||
|
// it:
|
||||||
|
return this.client.getAllRafts(undefined, (error, payload) => {
|
||||||
|
let statuses = null;
|
||||||
|
try {
|
||||||
|
statuses = JSON.parse(payload);
|
||||||
|
} catch (e) {
|
||||||
|
doFail = true;
|
||||||
|
reason.msg = 'could not interpret status: invalid payload';
|
||||||
|
// Can't do anything anymore if we fail here. return.
|
||||||
|
return callback(doFail, reason);
|
||||||
|
}
|
||||||
|
|
||||||
|
const reducer = (acc, payload) => acc + !payload.connectedToLeader;
|
||||||
|
reason.ratio = statuses.reduce(reducer, 0) / statuses.length;
|
||||||
|
/* NOTE FIXME/TODO: acceptableRatio could be configured later on */
|
||||||
|
reason.acceptableRatio = 0.5;
|
||||||
|
/* If the RaftSession 0 (map) does not work, fail anyways */
|
||||||
|
if (!doFail && !statuses[0].connectedToLeader) {
|
||||||
|
doFail = true;
|
||||||
|
reason.msg = 'Bucket map unavailable';
|
||||||
|
}
|
||||||
|
if (!doFail && reason.ratio > reason.acceptableRatio) {
|
||||||
|
doFail = true;
|
||||||
|
reason.msg = 'Ratio of failing Raft Sessions is too high';
|
||||||
|
}
|
||||||
|
return callback(doFail, reason);
|
||||||
|
}, log);
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Bucketd offers a behavior that diverges from other sub-components on the
|
||||||
|
* healthCheck: If any of the pieces making up the bucket storage fail (ie:
|
||||||
|
* if any Raft Session is down), bucketd returns a 500 for the healthCheck.
|
||||||
|
*
|
||||||
|
* As seen in S3C-1412, this may become an issue for S3, whenever the
|
||||||
|
* system is only partly failing.
|
||||||
|
*
|
||||||
|
* This means that S3 needs to analyze the situation, and interpret this
|
||||||
|
* status depending on the analysis. S3 will then assess the situation as
|
||||||
|
* critical (and consider it a failure), or not (and consider it a success
|
||||||
|
* anyways, thus not diverging from the healthCheck behavior of other
|
||||||
|
* components).
|
||||||
|
*/
|
||||||
|
checkHealth(implName, log, cb) {
|
||||||
|
return this.client.healthcheck(log, (err, result) => {
|
||||||
|
const respBody = {};
|
||||||
|
if (err) {
|
||||||
|
return this._analyzeHealthFailure(log, (failure, reason) => {
|
||||||
|
const message = reason.msg;
|
||||||
|
// Remove 'msg' from the reason payload.
|
||||||
|
// eslint-disable-next-line no-param-reassign
|
||||||
|
reason.msg = undefined;
|
||||||
|
respBody[implName] = {
|
||||||
|
code: 200,
|
||||||
|
message, // Provide interpreted reason msg
|
||||||
|
body: reason, // Provide analysis data
|
||||||
|
};
|
||||||
|
if (failure) {
|
||||||
|
// Setting the `error` field is how the healthCheck
|
||||||
|
// logic interprets it as an error. Don't forget it !
|
||||||
|
respBody[implName].error = err;
|
||||||
|
respBody[implName].code = err.code; // original error
|
||||||
|
}
|
||||||
|
// error returned as null so async parallel doesn't return
|
||||||
|
// before all backends are checked
|
||||||
|
return cb(null, respBody);
|
||||||
|
}, log);
|
||||||
|
}
|
||||||
|
const parseResult = JSON.parse(result);
|
||||||
|
respBody[implName] = {
|
||||||
|
code: 200,
|
||||||
|
message: 'OK',
|
||||||
|
body: parseResult,
|
||||||
|
};
|
||||||
|
return cb(null, respBody);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = BucketClientInterface;
|
|
@ -0,0 +1,379 @@
|
||||||
|
const cluster = require('cluster');
|
||||||
|
const arsenal = require('arsenal');
|
||||||
|
|
||||||
|
const logger = require('../../utilities/logger');
|
||||||
|
const BucketInfo = arsenal.models.BucketInfo;
|
||||||
|
const constants = require('../../../constants');
|
||||||
|
const { config } = require('../../Config');
|
||||||
|
|
||||||
|
const errors = arsenal.errors;
|
||||||
|
const MetadataFileClient = arsenal.storage.metadata.MetadataFileClient;
|
||||||
|
const versionSep = arsenal.versioning.VersioningConstants.VersionId.Separator;
|
||||||
|
|
||||||
|
const METASTORE = '__metastore';
|
||||||
|
|
||||||
|
class BucketFileInterface {
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @constructor
|
||||||
|
* @param {object} [params] - constructor params
|
||||||
|
* @param {boolean} [params.noDbOpen=false] - true to skip DB open
|
||||||
|
* (for unit tests only)
|
||||||
|
*/
|
||||||
|
constructor(params) {
|
||||||
|
this.logger = logger;
|
||||||
|
const { host, port } = config.metadataClient;
|
||||||
|
this.mdClient = new MetadataFileClient({ host, port });
|
||||||
|
if (params && params.noDbOpen) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.mdDB = this.mdClient.openDB(err => {
|
||||||
|
if (err) {
|
||||||
|
throw err;
|
||||||
|
}
|
||||||
|
// the metastore sublevel is used to store bucket attributes
|
||||||
|
this.metastore = this.mdDB.openSub(METASTORE);
|
||||||
|
if (cluster.isMaster) {
|
||||||
|
this.setupMetadataServer();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
setupMetadataServer() {
|
||||||
|
/* Since the bucket creation API is expecting the
|
||||||
|
usersBucket to have attributes, we pre-create the
|
||||||
|
usersBucket attributes here */
|
||||||
|
this.mdClient.logger.debug('setting up metadata server');
|
||||||
|
const usersBucketAttr = new BucketInfo(constants.usersBucket,
|
||||||
|
'admin', 'admin', new Date().toJSON(),
|
||||||
|
BucketInfo.currentModelVersion());
|
||||||
|
this.metastore.put(
|
||||||
|
constants.usersBucket,
|
||||||
|
usersBucketAttr.serialize(), {}, err => {
|
||||||
|
if (err) {
|
||||||
|
this.logger.fatal('error writing usersBucket ' +
|
||||||
|
'attributes to metadata',
|
||||||
|
{ error: err });
|
||||||
|
throw (errors.InternalError);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Load DB if exists
|
||||||
|
* @param {String} bucketName - name of bucket
|
||||||
|
* @param {Object} log - logger
|
||||||
|
* @param {function} cb - callback(err, db, attr)
|
||||||
|
* @return {undefined}
|
||||||
|
*/
|
||||||
|
loadDBIfExists(bucketName, log, cb) {
|
||||||
|
this.getBucketAttributes(bucketName, log, (err, attr) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const db = this.mdDB.openSub(bucketName);
|
||||||
|
return cb(null, db, attr);
|
||||||
|
} catch (err) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
createBucket(bucketName, bucketMD, log, cb) {
|
||||||
|
this.getBucketAttributes(bucketName, log, err => {
|
||||||
|
if (err && err !== errors.NoSuchBucket) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
if (err === undefined) {
|
||||||
|
return cb(errors.BucketAlreadyExists);
|
||||||
|
}
|
||||||
|
this.putBucketAttributes(bucketName,
|
||||||
|
bucketMD,
|
||||||
|
log, cb);
|
||||||
|
return undefined;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
getBucketAttributes(bucketName, log, cb) {
|
||||||
|
this.metastore
|
||||||
|
.withRequestLogger(log)
|
||||||
|
.get(bucketName, {}, (err, data) => {
|
||||||
|
if (err) {
|
||||||
|
if (err.ObjNotFound) {
|
||||||
|
return cb(errors.NoSuchBucket);
|
||||||
|
}
|
||||||
|
const logObj = {
|
||||||
|
rawError: err,
|
||||||
|
error: err.message,
|
||||||
|
errorStack: err.stack,
|
||||||
|
};
|
||||||
|
log.error('error getting db attributes', logObj);
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return cb(null, BucketInfo.deSerialize(data));
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
getBucketAndObject(bucketName, objName, params, log, cb) {
|
||||||
|
this.loadDBIfExists(bucketName, log, (err, db, bucketAttr) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
db.withRequestLogger(log)
|
||||||
|
.get(objName, params, (err, objAttr) => {
|
||||||
|
if (err) {
|
||||||
|
if (err.ObjNotFound) {
|
||||||
|
return cb(null, {
|
||||||
|
bucket: bucketAttr.serialize(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
const logObj = {
|
||||||
|
rawError: err,
|
||||||
|
error: err.message,
|
||||||
|
errorStack: err.stack,
|
||||||
|
};
|
||||||
|
log.error('error getting object', logObj);
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return cb(null, {
|
||||||
|
bucket: bucketAttr.serialize(),
|
||||||
|
obj: objAttr,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
putBucketAttributes(bucketName, bucketMD, log, cb) {
|
||||||
|
this.metastore
|
||||||
|
.withRequestLogger(log)
|
||||||
|
.put(bucketName, bucketMD.serialize(), {}, err => {
|
||||||
|
if (err) {
|
||||||
|
const logObj = {
|
||||||
|
rawError: err,
|
||||||
|
error: err.message,
|
||||||
|
errorStack: err.stack,
|
||||||
|
};
|
||||||
|
log.error('error putting db attributes', logObj);
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
deleteBucket(bucketName, log, cb) {
|
||||||
|
this.metastore
|
||||||
|
.withRequestLogger(log)
|
||||||
|
.del(bucketName, {}, err => {
|
||||||
|
if (err) {
|
||||||
|
const logObj = {
|
||||||
|
rawError: err,
|
||||||
|
error: err.message,
|
||||||
|
errorStack: err.stack,
|
||||||
|
};
|
||||||
|
log.error('error deleting bucket',
|
||||||
|
logObj);
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
putObject(bucketName, objName, objVal, params, log, cb) {
|
||||||
|
this.loadDBIfExists(bucketName, log, (err, db) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
db.withRequestLogger(log)
|
||||||
|
.put(objName, JSON.stringify(objVal), params, (err, data) => {
|
||||||
|
if (err) {
|
||||||
|
const logObj = {
|
||||||
|
rawError: err,
|
||||||
|
error: err.message,
|
||||||
|
errorStack: err.stack,
|
||||||
|
};
|
||||||
|
log.error('error putting object', logObj);
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return cb(err, data);
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
getObject(bucketName, objName, params, log, cb) {
|
||||||
|
this.loadDBIfExists(bucketName, log, (err, db) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
db.withRequestLogger(log).get(objName, params, (err, data) => {
|
||||||
|
if (err) {
|
||||||
|
if (err.ObjNotFound) {
|
||||||
|
return cb(errors.NoSuchKey);
|
||||||
|
}
|
||||||
|
const logObj = {
|
||||||
|
rawError: err,
|
||||||
|
error: err.message,
|
||||||
|
errorStack: err.stack,
|
||||||
|
};
|
||||||
|
log.error('error getting object', logObj);
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return cb(null, JSON.parse(data));
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
deleteObject(bucketName, objName, params, log, cb) {
|
||||||
|
this.loadDBIfExists(bucketName, log, (err, db) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
db.withRequestLogger(log).del(objName, params, err => {
|
||||||
|
if (err) {
|
||||||
|
const logObj = {
|
||||||
|
rawError: err,
|
||||||
|
error: err.message,
|
||||||
|
errorStack: err.stack,
|
||||||
|
};
|
||||||
|
log.error('error deleting object', logObj);
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* This complex function deals with different extensions of bucket listing:
|
||||||
|
* Delimiter based search or MPU based search.
|
||||||
|
* @param {String} bucketName - The name of the bucket to list
|
||||||
|
* @param {Object} params - The params to search
|
||||||
|
* @param {Object} log - The logger object
|
||||||
|
* @param {function} cb - Callback when done
|
||||||
|
* @return {undefined}
|
||||||
|
*/
|
||||||
|
internalListObject(bucketName, params, log, cb) {
|
||||||
|
const extName = params.listingType;
|
||||||
|
const extension = new arsenal.algorithms.list[extName](params, log);
|
||||||
|
const requestParams = extension.genMDParams();
|
||||||
|
this.loadDBIfExists(bucketName, log, (err, db) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
let cbDone = false;
|
||||||
|
db.withRequestLogger(log)
|
||||||
|
.createReadStream(requestParams, (err, stream) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
stream
|
||||||
|
.on('data', e => {
|
||||||
|
if (extension.filter(e) < 0) {
|
||||||
|
stream.emit('end');
|
||||||
|
stream.destroy();
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.on('error', err => {
|
||||||
|
if (!cbDone) {
|
||||||
|
cbDone = true;
|
||||||
|
const logObj = {
|
||||||
|
rawError: err,
|
||||||
|
error: err.message,
|
||||||
|
errorStack: err.stack,
|
||||||
|
};
|
||||||
|
log.error('error listing objects', logObj);
|
||||||
|
cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.on('end', () => {
|
||||||
|
if (!cbDone) {
|
||||||
|
cbDone = true;
|
||||||
|
const data = extension.result();
|
||||||
|
cb(null, data);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
listObject(bucketName, params, log, cb) {
|
||||||
|
return this.internalListObject(bucketName, params, log, cb);
|
||||||
|
}
|
||||||
|
|
||||||
|
listMultipartUploads(bucketName, params, log, cb) {
|
||||||
|
return this.internalListObject(bucketName, params, log, cb);
|
||||||
|
}
|
||||||
|
|
||||||
|
getUUID(log, cb) {
|
||||||
|
return this.mdDB.getUUID(cb);
|
||||||
|
}
|
||||||
|
|
||||||
|
getDiskUsage(cb) {
|
||||||
|
return this.mdDB.getDiskUsage(cb);
|
||||||
|
}
|
||||||
|
|
||||||
|
countItems(log, cb) {
|
||||||
|
const params = {};
|
||||||
|
const extension = new arsenal.algorithms.list.Basic(params, log);
|
||||||
|
const requestParams = extension.genMDParams();
|
||||||
|
|
||||||
|
const res = {
|
||||||
|
objects: 0,
|
||||||
|
versions: 0,
|
||||||
|
buckets: 0,
|
||||||
|
};
|
||||||
|
let cbDone = false;
|
||||||
|
|
||||||
|
this.mdDB.rawListKeys(requestParams, (err, stream) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
stream
|
||||||
|
.on('data', e => {
|
||||||
|
if (!e.includes(METASTORE)) {
|
||||||
|
if (e.includes(constants.usersBucket)) {
|
||||||
|
res.buckets++;
|
||||||
|
} else if (e.includes(versionSep)) {
|
||||||
|
res.versions++;
|
||||||
|
} else {
|
||||||
|
res.objects++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.on('error', err => {
|
||||||
|
if (!cbDone) {
|
||||||
|
cbDone = true;
|
||||||
|
const logObj = {
|
||||||
|
error: err,
|
||||||
|
errorMessage: err.message,
|
||||||
|
errorStack: err.stack,
|
||||||
|
};
|
||||||
|
log.error('error listing objects', logObj);
|
||||||
|
cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.on('end', () => {
|
||||||
|
if (!cbDone) {
|
||||||
|
cbDone = true;
|
||||||
|
return cb(null, res);
|
||||||
|
}
|
||||||
|
return undefined;
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = BucketFileInterface;
|
|
@ -0,0 +1,34 @@
|
||||||
|
const ListResult = require('./ListResult');
|
||||||
|
|
||||||
|
class ListMultipartUploadsResult extends ListResult {
|
||||||
|
constructor() {
|
||||||
|
super();
|
||||||
|
this.Uploads = [];
|
||||||
|
this.NextKeyMarker = undefined;
|
||||||
|
this.NextUploadIdMarker = undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
addUpload(uploadInfo) {
|
||||||
|
this.Uploads.push({
|
||||||
|
key: decodeURIComponent(uploadInfo.key),
|
||||||
|
value: {
|
||||||
|
UploadId: uploadInfo.uploadId,
|
||||||
|
Initiator: {
|
||||||
|
ID: uploadInfo.initiatorID,
|
||||||
|
DisplayName: uploadInfo.initiatorDisplayName,
|
||||||
|
},
|
||||||
|
Owner: {
|
||||||
|
ID: uploadInfo.ownerID,
|
||||||
|
DisplayName: uploadInfo.ownerDisplayName,
|
||||||
|
},
|
||||||
|
StorageClass: uploadInfo.storageClass,
|
||||||
|
Initiated: uploadInfo.initiated,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
this.MaxKeys += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
ListMultipartUploadsResult,
|
||||||
|
};
|
|
@ -0,0 +1,27 @@
|
||||||
|
class ListResult {
|
||||||
|
constructor() {
|
||||||
|
this.IsTruncated = false;
|
||||||
|
this.NextMarker = undefined;
|
||||||
|
this.CommonPrefixes = [];
|
||||||
|
/*
|
||||||
|
Note: this.MaxKeys will get incremented as
|
||||||
|
keys are added so that when response is returned,
|
||||||
|
this.MaxKeys will equal total keys in response
|
||||||
|
(with each CommonPrefix counting as 1 key)
|
||||||
|
*/
|
||||||
|
this.MaxKeys = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
addCommonPrefix(prefix) {
|
||||||
|
if (!this.hasCommonPrefix(prefix)) {
|
||||||
|
this.CommonPrefixes.push(prefix);
|
||||||
|
this.MaxKeys += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
hasCommonPrefix(prefix) {
|
||||||
|
return (this.CommonPrefixes.indexOf(prefix) !== -1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = ListResult;
|
|
@ -0,0 +1,331 @@
|
||||||
|
const { errors, algorithms, versioning } = require('arsenal');
|
||||||
|
|
||||||
|
const getMultipartUploadListing = require('./getMultipartUploadListing');
|
||||||
|
const { metadata } = require('./metadata');
|
||||||
|
const { config } = require('../../Config');
|
||||||
|
|
||||||
|
const genVID = versioning.VersionID.generateVersionId;
|
||||||
|
|
||||||
|
const defaultMaxKeys = 1000;
|
||||||
|
let uidCounter = 0;
|
||||||
|
|
||||||
|
function generateVersionId() {
|
||||||
|
return genVID(uidCounter++, config.replicationGroupId);
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatVersionKey(key, versionId) {
|
||||||
|
return `${key}\0${versionId}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function inc(str) {
|
||||||
|
return str ? (str.slice(0, str.length - 1) +
|
||||||
|
String.fromCharCode(str.charCodeAt(str.length - 1) + 1)) : str;
|
||||||
|
}
|
||||||
|
|
||||||
|
const metastore = {
|
||||||
|
createBucket: (bucketName, bucketMD, log, cb) => {
|
||||||
|
process.nextTick(() => {
|
||||||
|
metastore.getBucketAttributes(bucketName, log, (err, bucket) => {
|
||||||
|
// TODO Check whether user already owns the bucket,
|
||||||
|
// if so return "BucketAlreadyOwnedByYou"
|
||||||
|
// If not owned by user, return "BucketAlreadyExists"
|
||||||
|
if (bucket) {
|
||||||
|
return cb(errors.BucketAlreadyExists);
|
||||||
|
}
|
||||||
|
metadata.buckets.set(bucketName, bucketMD);
|
||||||
|
metadata.keyMaps.set(bucketName, new Map);
|
||||||
|
return cb();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
putBucketAttributes: (bucketName, bucketMD, log, cb) => {
|
||||||
|
process.nextTick(() => {
|
||||||
|
metastore.getBucketAttributes(bucketName, log, err => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
metadata.buckets.set(bucketName, bucketMD);
|
||||||
|
return cb();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
getBucketAttributes: (bucketName, log, cb) => {
|
||||||
|
process.nextTick(() => {
|
||||||
|
if (!metadata.buckets.has(bucketName)) {
|
||||||
|
return cb(errors.NoSuchBucket);
|
||||||
|
}
|
||||||
|
return cb(null, metadata.buckets.get(bucketName));
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
deleteBucket: (bucketName, log, cb) => {
|
||||||
|
process.nextTick(() => {
|
||||||
|
metastore.getBucketAttributes(bucketName, log, err => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
if (metadata.keyMaps.has(bucketName)
|
||||||
|
&& metadata.keyMaps.get(bucketName).length > 0) {
|
||||||
|
return cb(errors.BucketNotEmpty);
|
||||||
|
}
|
||||||
|
metadata.buckets.delete(bucketName);
|
||||||
|
metadata.keyMaps.delete(bucketName);
|
||||||
|
return cb(null);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
putObject: (bucketName, objName, objVal, params, log, cb) => {
|
||||||
|
process.nextTick(() => {
|
||||||
|
metastore.getBucketAttributes(bucketName, log, err => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
/*
|
||||||
|
valid combinations of versioning options:
|
||||||
|
- !versioning && !versionId: normal non-versioning put
|
||||||
|
- versioning && !versionId: create a new version
|
||||||
|
- versionId: update (PUT/DELETE) an existing version,
|
||||||
|
and also update master version in case the put
|
||||||
|
version is newer or same version than master.
|
||||||
|
if versionId === '' update master version
|
||||||
|
*/
|
||||||
|
|
||||||
|
if (params && params.versionId) {
|
||||||
|
objVal.versionId = params.versionId; // eslint-disable-line
|
||||||
|
const mst = metadata.keyMaps.get(bucketName).get(objName);
|
||||||
|
if (mst && mst.versionId === params.versionId || !mst) {
|
||||||
|
metadata.keyMaps.get(bucketName).set(objName, objVal);
|
||||||
|
}
|
||||||
|
// eslint-disable-next-line
|
||||||
|
objName = formatVersionKey(objName, params.versionId);
|
||||||
|
metadata.keyMaps.get(bucketName).set(objName, objVal);
|
||||||
|
return cb(null, `{"versionId":"${objVal.versionId}"}`);
|
||||||
|
}
|
||||||
|
if (params && params.versioning) {
|
||||||
|
const versionId = generateVersionId();
|
||||||
|
objVal.versionId = versionId; // eslint-disable-line
|
||||||
|
metadata.keyMaps.get(bucketName).set(objName, objVal);
|
||||||
|
// eslint-disable-next-line
|
||||||
|
objName = formatVersionKey(objName, versionId);
|
||||||
|
metadata.keyMaps.get(bucketName).set(objName, objVal);
|
||||||
|
return cb(null, `{"versionId":"${versionId}"}`);
|
||||||
|
}
|
||||||
|
if (params && params.versionId === '') {
|
||||||
|
const versionId = generateVersionId();
|
||||||
|
objVal.versionId = versionId; // eslint-disable-line
|
||||||
|
metadata.keyMaps.get(bucketName).set(objName, objVal);
|
||||||
|
return cb(null, `{"versionId":"${objVal.versionId}"}`);
|
||||||
|
}
|
||||||
|
metadata.keyMaps.get(bucketName).set(objName, objVal);
|
||||||
|
return cb(null);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
getBucketAndObject: (bucketName, objName, params, log, cb) => {
|
||||||
|
process.nextTick(() => {
|
||||||
|
metastore.getBucketAttributes(bucketName, log, (err, bucket) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err, { bucket });
|
||||||
|
}
|
||||||
|
if (params && params.versionId) {
|
||||||
|
// eslint-disable-next-line
|
||||||
|
objName = formatVersionKey(objName, params.versionId);
|
||||||
|
}
|
||||||
|
if (!metadata.keyMaps.has(bucketName)
|
||||||
|
|| !metadata.keyMaps.get(bucketName).has(objName)) {
|
||||||
|
return cb(null, { bucket: bucket.serialize() });
|
||||||
|
}
|
||||||
|
return cb(null, {
|
||||||
|
bucket: bucket.serialize(),
|
||||||
|
obj: JSON.stringify(
|
||||||
|
metadata.keyMaps.get(bucketName).get(objName)
|
||||||
|
),
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
getObject: (bucketName, objName, params, log, cb) => {
|
||||||
|
process.nextTick(() => {
|
||||||
|
metastore.getBucketAttributes(bucketName, log, err => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
if (params && params.versionId) {
|
||||||
|
// eslint-disable-next-line
|
||||||
|
objName = formatVersionKey(objName, params.versionId);
|
||||||
|
}
|
||||||
|
if (!metadata.keyMaps.has(bucketName)
|
||||||
|
|| !metadata.keyMaps.get(bucketName).has(objName)) {
|
||||||
|
return cb(errors.NoSuchKey);
|
||||||
|
}
|
||||||
|
return cb(null, metadata.keyMaps.get(bucketName).get(objName));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
deleteObject: (bucketName, objName, params, log, cb) => {
|
||||||
|
process.nextTick(() => {
|
||||||
|
metastore.getBucketAttributes(bucketName, log, err => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
if (!metadata.keyMaps.get(bucketName).has(objName)) {
|
||||||
|
return cb(errors.NoSuchKey);
|
||||||
|
}
|
||||||
|
if (params && params.versionId) {
|
||||||
|
const baseKey = inc(formatVersionKey(objName, ''));
|
||||||
|
const vobjName = formatVersionKey(objName,
|
||||||
|
params.versionId);
|
||||||
|
metadata.keyMaps.get(bucketName).delete(vobjName);
|
||||||
|
const mst = metadata.keyMaps.get(bucketName).get(objName);
|
||||||
|
if (mst.versionId === params.versionId) {
|
||||||
|
const keys = [];
|
||||||
|
metadata.keyMaps.get(bucketName).forEach((val, key) => {
|
||||||
|
if (key < baseKey && key > vobjName) {
|
||||||
|
keys.push(key);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
if (keys.length === 0) {
|
||||||
|
metadata.keyMaps.get(bucketName).delete(objName);
|
||||||
|
return cb();
|
||||||
|
}
|
||||||
|
const key = keys.sort()[0];
|
||||||
|
const value = metadata.keyMaps.get(bucketName).get(key);
|
||||||
|
metadata.keyMaps.get(bucketName).set(objName, value);
|
||||||
|
}
|
||||||
|
return cb();
|
||||||
|
}
|
||||||
|
metadata.keyMaps.get(bucketName).delete(objName);
|
||||||
|
return cb();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
_hasDeleteMarker(key, keyMap) {
|
||||||
|
const objectMD = keyMap.get(key);
|
||||||
|
if (objectMD['x-amz-delete-marker'] !== undefined) {
|
||||||
|
return (objectMD['x-amz-delete-marker'] === true);
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
},
|
||||||
|
|
||||||
|
listObject(bucketName, params, log, cb) {
|
||||||
|
process.nextTick(() => {
|
||||||
|
const {
|
||||||
|
prefix,
|
||||||
|
marker,
|
||||||
|
delimiter,
|
||||||
|
maxKeys,
|
||||||
|
continuationToken,
|
||||||
|
startAfter,
|
||||||
|
} = params;
|
||||||
|
if (prefix && typeof prefix !== 'string') {
|
||||||
|
return cb(errors.InvalidArgument);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (marker && typeof marker !== 'string') {
|
||||||
|
return cb(errors.InvalidArgument);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (delimiter && typeof delimiter !== 'string') {
|
||||||
|
return cb(errors.InvalidArgument);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (maxKeys && typeof maxKeys !== 'number') {
|
||||||
|
return cb(errors.InvalidArgument);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (continuationToken && typeof continuationToken !== 'string') {
|
||||||
|
return cb(errors.InvalidArgument);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (startAfter && typeof startAfter !== 'string') {
|
||||||
|
return cb(errors.InvalidArgument);
|
||||||
|
}
|
||||||
|
|
||||||
|
// If paramMaxKeys is undefined, the default parameter will set it.
|
||||||
|
// However, if it is null, the default parameter will not set it.
|
||||||
|
let numKeys = maxKeys;
|
||||||
|
if (numKeys === null) {
|
||||||
|
numKeys = defaultMaxKeys;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!metadata.keyMaps.has(bucketName)) {
|
||||||
|
return cb(errors.NoSuchBucket);
|
||||||
|
}
|
||||||
|
|
||||||
|
// If marker specified, edit the keys array so it
|
||||||
|
// only contains keys that occur alphabetically after the marker
|
||||||
|
const listingType = params.listingType;
|
||||||
|
const extension = new algorithms.list[listingType](params, log);
|
||||||
|
const listingParams = extension.genMDParams();
|
||||||
|
|
||||||
|
const keys = [];
|
||||||
|
metadata.keyMaps.get(bucketName).forEach((val, key) => {
|
||||||
|
if (listingParams.gt && listingParams.gt >= key) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
if (listingParams.gte && listingParams.gte > key) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
if (listingParams.lt && key >= listingParams.lt) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
if (listingParams.lte && key > listingParams.lte) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
return keys.push(key);
|
||||||
|
});
|
||||||
|
keys.sort();
|
||||||
|
|
||||||
|
// Iterate through keys array and filter keys containing
|
||||||
|
// delimiter into response.CommonPrefixes and filter remaining
|
||||||
|
// keys into response.Contents
|
||||||
|
for (let i = 0; i < keys.length; ++i) {
|
||||||
|
const currentKey = keys[i];
|
||||||
|
// Do not list object with delete markers
|
||||||
|
if (this._hasDeleteMarker(currentKey,
|
||||||
|
metadata.keyMaps.get(bucketName))) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const objMD = metadata.keyMaps.get(bucketName).get(currentKey);
|
||||||
|
const value = JSON.stringify(objMD);
|
||||||
|
const obj = {
|
||||||
|
key: currentKey,
|
||||||
|
value,
|
||||||
|
};
|
||||||
|
// calling Ext.filter(obj) adds the obj to the Ext result if
|
||||||
|
// not filtered.
|
||||||
|
// Also, Ext.filter returns false when hit max keys.
|
||||||
|
// What a nifty function!
|
||||||
|
if (extension.filter(obj) < 0) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return cb(null, extension.result());
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
listMultipartUploads(bucketName, listingParams, log, cb) {
|
||||||
|
process.nextTick(() => {
|
||||||
|
metastore.getBucketAttributes(bucketName, log, (err, bucket) => {
|
||||||
|
if (bucket === undefined) {
|
||||||
|
// no on going multipart uploads, return empty listing
|
||||||
|
return cb(null, {
|
||||||
|
IsTruncated: false,
|
||||||
|
NextMarker: undefined,
|
||||||
|
MaxKeys: 0,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return getMultipartUploadListing(bucket, listingParams, cb);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
module.exports = metastore;
|
|
@ -0,0 +1,62 @@
|
||||||
|
# bucket_mem design
|
||||||
|
|
||||||
|
## RATIONALE
|
||||||
|
|
||||||
|
The bucket API will be used for managing buckets behind the S3 interface.
|
||||||
|
|
||||||
|
We plan to have only 2 backends using this interface:
|
||||||
|
|
||||||
|
* One production backend
|
||||||
|
* One debug backend purely in memory
|
||||||
|
|
||||||
|
One important remark here is that we don't want an abstraction but a
|
||||||
|
duck-typing style interface (different classes MemoryBucket and Bucket having
|
||||||
|
the same methods putObjectMD(), getObjectMD(), etc).
|
||||||
|
|
||||||
|
Notes about the memory backend: The backend is currently a simple key/value
|
||||||
|
store in memory. The functions actually use nextTick() to emulate the future
|
||||||
|
asynchronous behavior of the production backend.
|
||||||
|
|
||||||
|
## BUCKET API
|
||||||
|
|
||||||
|
The bucket API is a very simple API with 5 functions:
|
||||||
|
|
||||||
|
- putObjectMD(): put metadata for an object in the bucket
|
||||||
|
- getObjectMD(): get metadata from the bucket
|
||||||
|
- deleteObjectMD(): delete metadata for an object from the bucket
|
||||||
|
- deleteBucketMD(): delete a bucket
|
||||||
|
- getBucketListObjects(): perform the complex bucket listing AWS search
|
||||||
|
function with various flavors. This function returns a response in a
|
||||||
|
ListBucketResult object.
|
||||||
|
|
||||||
|
getBucketListObjects(prefix, marker, delimiter, maxKeys, callback) behavior is
|
||||||
|
the following:
|
||||||
|
|
||||||
|
prefix (not required): Limits the response to keys that begin with the
|
||||||
|
specified prefix. You can use prefixes to separate a bucket into different
|
||||||
|
groupings of keys. (You can think of using prefix to make groups in the same
|
||||||
|
way you'd use a folder in a file system.)
|
||||||
|
|
||||||
|
marker (not required): Specifies the key to start with when listing objects in
|
||||||
|
a bucket. Amazon S3 returns object keys in alphabetical order, starting with
|
||||||
|
key after the marker in order.
|
||||||
|
|
||||||
|
delimiter (not required): A delimiter is a character you use to group keys.
|
||||||
|
All keys that contain the same string between the prefix, if specified, and the
|
||||||
|
first occurrence of the delimiter after the prefix are grouped under a single
|
||||||
|
result element, CommonPrefixes. If you don't specify the prefix parameter, then
|
||||||
|
the substring starts at the beginning of the key. The keys that are grouped
|
||||||
|
under CommonPrefixes are not returned elsewhere in the response.
|
||||||
|
|
||||||
|
maxKeys: Sets the maximum number of keys returned in the response body. You can
|
||||||
|
add this to your request if you want to retrieve fewer than the default 1000
|
||||||
|
keys. The response might contain fewer keys but will never contain more. If
|
||||||
|
there are additional keys that satisfy the search criteria but were not
|
||||||
|
returned because maxKeys was exceeded, the response contains an attribute of
|
||||||
|
IsTruncated set to true and a NextMarker. To return the additional keys, call
|
||||||
|
the function again using NextMarker as your marker argument in the function.
|
||||||
|
|
||||||
|
Any key that does not contain the delimiter will be returned individually in
|
||||||
|
Contents rather than in CommonPrefixes.
|
||||||
|
|
||||||
|
If there is an error, the error subfield is returned in the response.
|
|
@ -0,0 +1,51 @@
|
||||||
|
function markerFilterMPU(allMarkers, array) {
|
||||||
|
const { keyMarker, uploadIdMarker } = allMarkers;
|
||||||
|
for (let i = 0; i < array.length; i++) {
|
||||||
|
// If the keyMarker is the same as the key,
|
||||||
|
// check the uploadIdMarker. If uploadIdMarker is the same
|
||||||
|
// as or alphabetically after the uploadId of the item,
|
||||||
|
// eliminate the item.
|
||||||
|
if (uploadIdMarker && keyMarker === array[i].key) {
|
||||||
|
const laterId =
|
||||||
|
[uploadIdMarker, array[i].uploadId].sort()[1];
|
||||||
|
if (array[i].uploadId === laterId) {
|
||||||
|
break;
|
||||||
|
} else {
|
||||||
|
array.shift();
|
||||||
|
i--;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// If the keyMarker is alphabetically after the key
|
||||||
|
// of the item in the array, eliminate the item from the array.
|
||||||
|
const laterItem =
|
||||||
|
[keyMarker, array[i].key].sort()[1];
|
||||||
|
if (keyMarker === array[i].key || keyMarker === laterItem) {
|
||||||
|
array.shift();
|
||||||
|
i--;
|
||||||
|
} else {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return array;
|
||||||
|
}
|
||||||
|
|
||||||
|
function prefixFilter(prefix, array) {
|
||||||
|
for (let i = 0; i < array.length; i++) {
|
||||||
|
if (array[i].indexOf(prefix) !== 0) {
|
||||||
|
array.splice(i, 1);
|
||||||
|
i--;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return array;
|
||||||
|
}
|
||||||
|
|
||||||
|
function isKeyInContents(responseObject, key) {
|
||||||
|
return responseObject.Contents.some(val => val.key === key);
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
markerFilterMPU,
|
||||||
|
prefixFilter,
|
||||||
|
isKeyInContents,
|
||||||
|
};
|
|
@ -0,0 +1,148 @@
|
||||||
|
const { errors } = require('arsenal');
|
||||||
|
|
||||||
|
const { markerFilterMPU, prefixFilter } = require('./bucket_utilities');
|
||||||
|
const { ListMultipartUploadsResult } = require('./ListMultipartUploadsResult');
|
||||||
|
const { metadata } = require('./metadata');
|
||||||
|
|
||||||
|
const defaultMaxKeys = 1000;
|
||||||
|
function getMultipartUploadListing(bucket, params, callback) {
|
||||||
|
const { delimiter, keyMarker,
|
||||||
|
uploadIdMarker, prefix, queryPrefixLength, splitter } = params;
|
||||||
|
const splitterLen = splitter.length;
|
||||||
|
const maxKeys = params.maxKeys !== undefined ?
|
||||||
|
Number.parseInt(params.maxKeys, 10) : defaultMaxKeys;
|
||||||
|
const response = new ListMultipartUploadsResult();
|
||||||
|
const keyMap = metadata.keyMaps.get(bucket.getName());
|
||||||
|
if (prefix) {
|
||||||
|
response.Prefix = prefix;
|
||||||
|
if (typeof prefix !== 'string') {
|
||||||
|
return callback(errors.InvalidArgument);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (keyMarker) {
|
||||||
|
response.KeyMarker = keyMarker;
|
||||||
|
if (typeof keyMarker !== 'string') {
|
||||||
|
return callback(errors.InvalidArgument);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (uploadIdMarker) {
|
||||||
|
response.UploadIdMarker = uploadIdMarker;
|
||||||
|
if (typeof uploadIdMarker !== 'string') {
|
||||||
|
return callback(errors.InvalidArgument);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (delimiter) {
|
||||||
|
response.Delimiter = delimiter;
|
||||||
|
if (typeof delimiter !== 'string') {
|
||||||
|
return callback(errors.InvalidArgument);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (maxKeys && typeof maxKeys !== 'number') {
|
||||||
|
return callback(errors.InvalidArgument);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sort uploads alphatebetically by objectKey and if same objectKey,
|
||||||
|
// then sort in ascending order by time initiated
|
||||||
|
let uploads = [];
|
||||||
|
keyMap.forEach((val, key) => {
|
||||||
|
uploads.push(key);
|
||||||
|
});
|
||||||
|
uploads.sort((a, b) => {
|
||||||
|
const aIndex = a.indexOf(splitter);
|
||||||
|
const bIndex = b.indexOf(splitter);
|
||||||
|
const aObjectKey = a.substring(aIndex + splitterLen);
|
||||||
|
const bObjectKey = b.substring(bIndex + splitterLen);
|
||||||
|
const aInitiated = keyMap.get(a).initiated;
|
||||||
|
const bInitiated = keyMap.get(b).initiated;
|
||||||
|
if (aObjectKey === bObjectKey) {
|
||||||
|
if (Date.parse(aInitiated) >= Date.parse(bInitiated)) {
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
if (Date.parse(aInitiated) < Date.parse(bInitiated)) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return (aObjectKey < bObjectKey) ? -1 : 1;
|
||||||
|
});
|
||||||
|
// Edit the uploads array so it only
|
||||||
|
// contains keys that contain the prefix
|
||||||
|
uploads = prefixFilter(prefix, uploads);
|
||||||
|
uploads = uploads.map(stringKey => {
|
||||||
|
const index = stringKey.indexOf(splitter);
|
||||||
|
const index2 = stringKey.indexOf(splitter, index + splitterLen);
|
||||||
|
const storedMD = keyMap.get(stringKey);
|
||||||
|
return {
|
||||||
|
key: stringKey.substring(index + splitterLen, index2),
|
||||||
|
uploadId: stringKey.substring(index2 + splitterLen),
|
||||||
|
bucket: storedMD.eventualStorageBucket,
|
||||||
|
initiatorID: storedMD.initiator.ID,
|
||||||
|
initiatorDisplayName: storedMD.initiator.DisplayName,
|
||||||
|
ownerID: storedMD['owner-id'],
|
||||||
|
ownerDisplayName: storedMD['owner-display-name'],
|
||||||
|
storageClass: storedMD['x-amz-storage-class'],
|
||||||
|
initiated: storedMD.initiated,
|
||||||
|
};
|
||||||
|
});
|
||||||
|
// If keyMarker specified, edit the uploads array so it
|
||||||
|
// only contains keys that occur alphabetically after the marker.
|
||||||
|
// If there is also an uploadIdMarker specified, filter to eliminate
|
||||||
|
// any uploads that share the keyMarker and have an uploadId before
|
||||||
|
// the uploadIdMarker.
|
||||||
|
if (keyMarker) {
|
||||||
|
const allMarkers = {
|
||||||
|
keyMarker,
|
||||||
|
uploadIdMarker,
|
||||||
|
};
|
||||||
|
uploads = markerFilterMPU(allMarkers, uploads);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Iterate through uploads and filter uploads
|
||||||
|
// with keys containing delimiter
|
||||||
|
// into response.CommonPrefixes and filter remaining uploads
|
||||||
|
// into response.Uploads
|
||||||
|
for (let i = 0; i < uploads.length; i++) {
|
||||||
|
const currentUpload = uploads[i];
|
||||||
|
// If hit maxKeys, stop adding keys to response
|
||||||
|
if (response.MaxKeys >= maxKeys) {
|
||||||
|
response.IsTruncated = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
// If a delimiter is specified, find its
|
||||||
|
// index in the current key AFTER THE OCCURRENCE OF THE PREFIX
|
||||||
|
// THAT WAS SENT IN THE QUERY (not the prefix including the splitter
|
||||||
|
// and other elements)
|
||||||
|
let delimiterIndexAfterPrefix = -1;
|
||||||
|
const currentKeyWithoutPrefix =
|
||||||
|
currentUpload.key.slice(queryPrefixLength);
|
||||||
|
let sliceEnd;
|
||||||
|
if (delimiter) {
|
||||||
|
delimiterIndexAfterPrefix = currentKeyWithoutPrefix
|
||||||
|
.indexOf(delimiter);
|
||||||
|
sliceEnd = delimiterIndexAfterPrefix + queryPrefixLength;
|
||||||
|
}
|
||||||
|
// If delimiter occurs in current key, add key to
|
||||||
|
// response.CommonPrefixes.
|
||||||
|
// Otherwise add upload to response.Uploads
|
||||||
|
if (delimiterIndexAfterPrefix > -1) {
|
||||||
|
const keySubstring = currentUpload.key.slice(0, sliceEnd + 1);
|
||||||
|
response.addCommonPrefix(keySubstring);
|
||||||
|
} else {
|
||||||
|
response.NextKeyMarker = currentUpload.key;
|
||||||
|
response.NextUploadIdMarker = currentUpload.uploadId;
|
||||||
|
response.addUpload(currentUpload);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// `response.MaxKeys` should be the value from the original `MaxUploads`
|
||||||
|
// parameter specified by the user (or else the default 1000). Redefine it
|
||||||
|
// here, so it does not equal the value of `uploads.length`.
|
||||||
|
response.MaxKeys = maxKeys;
|
||||||
|
// If `response.MaxKeys` is 0, `response.IsTruncated` should be `false`.
|
||||||
|
response.IsTruncated = maxKeys === 0 ? false : response.IsTruncated;
|
||||||
|
return callback(null, response);
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = getMultipartUploadListing;
|
|
@ -0,0 +1,8 @@
|
||||||
|
const metadata = {
|
||||||
|
buckets: new Map,
|
||||||
|
keyMaps: new Map,
|
||||||
|
};
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
metadata,
|
||||||
|
};
|
|
@ -0,0 +1,274 @@
|
||||||
|
const async = require('async');
|
||||||
|
const { errors } = require('arsenal');
|
||||||
|
|
||||||
|
const metadata = require('./wrapper');
|
||||||
|
const BucketInfo = require('arsenal').models.BucketInfo;
|
||||||
|
const { isBucketAuthorized, isObjAuthorized } =
|
||||||
|
require('../api/apiUtils/authorization/aclChecks');
|
||||||
|
const bucketShield = require('../api/apiUtils/bucket/bucketShield');
|
||||||
|
|
||||||
|
/** _parseListEntries - parse the values returned in a listing by metadata
|
||||||
|
* @param {object[]} entries - Version or Content entries in a metadata listing
|
||||||
|
* @param {string} entries[].key - metadata key
|
||||||
|
* @param {string} entries[].value - stringified object metadata
|
||||||
|
* @return {object} - mapped array with parsed value or JSON parsing err
|
||||||
|
*/
|
||||||
|
function _parseListEntries(entries) {
|
||||||
|
return entries.map(entry => {
|
||||||
|
if (typeof entry.value === 'string') {
|
||||||
|
const tmp = JSON.parse(entry.value);
|
||||||
|
return {
|
||||||
|
key: entry.key,
|
||||||
|
value: {
|
||||||
|
Size: tmp['content-length'],
|
||||||
|
ETag: tmp['content-md5'],
|
||||||
|
VersionId: tmp.versionId,
|
||||||
|
IsNull: tmp.isNull,
|
||||||
|
IsDeleteMarker: tmp.isDeleteMarker,
|
||||||
|
LastModified: tmp['last-modified'],
|
||||||
|
Owner: {
|
||||||
|
DisplayName: tmp['owner-display-name'],
|
||||||
|
ID: tmp['owner-id'],
|
||||||
|
},
|
||||||
|
StorageClass: tmp['x-amz-storage-class'],
|
||||||
|
// MPU listing properties
|
||||||
|
Initiated: tmp.initiated,
|
||||||
|
Initiator: tmp.initiator,
|
||||||
|
EventualStorageBucket: tmp.eventualStorageBucket,
|
||||||
|
partLocations: tmp.partLocations,
|
||||||
|
creationDate: tmp.creationDate,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return entry;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/** parseListEntries - parse the values returned in a listing by metadata
|
||||||
|
* @param {object[]} entries - Version or Content entries in a metadata listing
|
||||||
|
* @param {string} entries[].key - metadata key
|
||||||
|
* @param {string} entries[].value - stringified object metadata
|
||||||
|
* @return {(object|Error)} - mapped array with parsed value or JSON parsing err
|
||||||
|
*/
|
||||||
|
function parseListEntries(entries) {
|
||||||
|
// wrap private function in a try/catch clause
|
||||||
|
// just in case JSON parsing throws an exception
|
||||||
|
try {
|
||||||
|
return _parseListEntries(entries);
|
||||||
|
} catch (e) {
|
||||||
|
return e;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** getNullVersion - return metadata of null version if it exists
|
||||||
|
* @param {object} objMD - metadata of master version
|
||||||
|
* @param {string} bucketName - name of bucket
|
||||||
|
* @param {string} objectKey - name of object key
|
||||||
|
* @param {RequestLogger} log - request logger
|
||||||
|
* @param {function} cb - callback
|
||||||
|
* @return {undefined} - and call callback with params err, objMD of null ver
|
||||||
|
*/
|
||||||
|
function getNullVersion(objMD, bucketName, objectKey, log, cb) {
|
||||||
|
const options = {};
|
||||||
|
if (objMD.isNull || !objMD.versionId) {
|
||||||
|
// null version is current version
|
||||||
|
log.debug('found null version');
|
||||||
|
return process.nextTick(() => cb(null, objMD));
|
||||||
|
}
|
||||||
|
if (objMD.nullVersionId) {
|
||||||
|
// the latest version is not the null version, but null version exists
|
||||||
|
log.debug('null version exists, get the null version');
|
||||||
|
options.versionId = objMD.nullVersionId;
|
||||||
|
return metadata.getObjectMD(bucketName, objectKey, options, log, cb);
|
||||||
|
}
|
||||||
|
log.debug('could not find a null version');
|
||||||
|
return process.nextTick(() => cb());
|
||||||
|
}
|
||||||
|
|
||||||
|
/** metadataGetBucketAndObject - retrieves bucket and specified version
|
||||||
|
* NOTE: If the value of `versionId` param is 'null', this function returns the
|
||||||
|
* master version objMD. The null version object md must be retrieved in a
|
||||||
|
* separate step using the master object md: see getNullVersion().
|
||||||
|
* @param {string} requestType - type of request
|
||||||
|
* @param {string} bucketName - name of bucket
|
||||||
|
* @param {string} objectKey - name of object key
|
||||||
|
* @param {string} [versionId] - version of object to retrieve
|
||||||
|
* @param {RequestLogger} log - request logger
|
||||||
|
* @param {function} cb - callback
|
||||||
|
* @return {undefined} - and call callback with err, bucket md and object md
|
||||||
|
*/
|
||||||
|
function metadataGetBucketAndObject(requestType, bucketName, objectKey,
|
||||||
|
versionId, log, cb) {
|
||||||
|
const options = {
|
||||||
|
// if attempting to get 'null' version, must retrieve null version id
|
||||||
|
// from most current object md (versionId = undefined)
|
||||||
|
versionId: versionId === 'null' ? undefined : versionId,
|
||||||
|
};
|
||||||
|
return metadata.getBucketAndObjectMD(bucketName, objectKey, options, log,
|
||||||
|
(err, data) => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('metadata get failed', { error: err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
const bucket = data.bucket ? BucketInfo.deSerialize(data.bucket) :
|
||||||
|
undefined;
|
||||||
|
const obj = data.obj ? JSON.parse(data.obj) : undefined;
|
||||||
|
if (!bucket) {
|
||||||
|
log.debug('bucketAttrs is undefined', {
|
||||||
|
bucket: bucketName,
|
||||||
|
method: 'metadataGetBucketAndObject',
|
||||||
|
});
|
||||||
|
return cb(errors.NoSuchBucket);
|
||||||
|
}
|
||||||
|
if (bucketShield(bucket, requestType)) {
|
||||||
|
log.debug('bucket is shielded from request', {
|
||||||
|
requestType,
|
||||||
|
method: 'metadataGetBucketAndObject',
|
||||||
|
});
|
||||||
|
return cb(errors.NoSuchBucket);
|
||||||
|
}
|
||||||
|
log.trace('found bucket in metadata');
|
||||||
|
return cb(null, bucket, obj);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/** metadataGetObject - retrieves specified object or version from metadata
|
||||||
|
* @param {string} bucketName - name of bucket
|
||||||
|
* @param {string} objectKey - name of object key
|
||||||
|
* @param {string} [versionId] - version of object to retrieve
|
||||||
|
* @param {RequestLogger} log - request logger
|
||||||
|
* @param {function} cb - callback
|
||||||
|
* @return {undefined} - and call callback with err, bucket md and object md
|
||||||
|
*/
|
||||||
|
function metadataGetObject(bucketName, objectKey, versionId, log, cb) {
|
||||||
|
const options = {
|
||||||
|
// if attempting to get 'null' version, must first retrieve null version
|
||||||
|
// id from most current object md (by setting versionId as undefined
|
||||||
|
// we retrieve the most current object md)
|
||||||
|
versionId: versionId === 'null' ? undefined : versionId,
|
||||||
|
};
|
||||||
|
return metadata.getObjectMD(bucketName, objectKey, options, log,
|
||||||
|
(err, objMD) => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('err getting object MD from metadata',
|
||||||
|
{ error: err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
if (versionId === 'null') {
|
||||||
|
return getNullVersion(objMD, bucketName, objectKey, log, cb);
|
||||||
|
}
|
||||||
|
return cb(null, objMD);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/** metadataValidateBucketAndObj - retrieve bucket and object md from metadata
|
||||||
|
* and check if user is authorized to access them.
|
||||||
|
* @param {object} params - function parameters
|
||||||
|
* @param {AuthInfo} params.authInfo - AuthInfo class instance, requester's info
|
||||||
|
* @param {string} params.bucketName - name of bucket
|
||||||
|
* @param {string} params.objectKey - name of object
|
||||||
|
* @param {string} [params.versionId] - version id if getting specific version
|
||||||
|
* @param {string} params.requestType - type of request
|
||||||
|
* @param {RequestLogger} log - request logger
|
||||||
|
* @param {function} callback - callback
|
||||||
|
* @return {undefined} - and call callback with params err, bucket md
|
||||||
|
*/
|
||||||
|
function metadataValidateBucketAndObj(params, log, callback) {
|
||||||
|
const { authInfo, bucketName, objectKey, versionId, requestType } = params;
|
||||||
|
const canonicalID = authInfo.getCanonicalID();
|
||||||
|
async.waterfall([
|
||||||
|
function getBucketAndObjectMD(next) {
|
||||||
|
return metadataGetBucketAndObject(requestType, bucketName,
|
||||||
|
objectKey, versionId, log, next);
|
||||||
|
},
|
||||||
|
function checkBucketAuth(bucket, objMD, next) {
|
||||||
|
if (!isBucketAuthorized(bucket, requestType, canonicalID)) {
|
||||||
|
log.debug('access denied for user on bucket', { requestType });
|
||||||
|
return next(errors.AccessDenied, bucket);
|
||||||
|
}
|
||||||
|
return next(null, bucket, objMD);
|
||||||
|
},
|
||||||
|
function handleNullVersionGet(bucket, objMD, next) {
|
||||||
|
if (objMD && versionId === 'null') {
|
||||||
|
return getNullVersion(objMD, bucketName, objectKey, log,
|
||||||
|
(err, nullVer) => next(err, bucket, nullVer));
|
||||||
|
}
|
||||||
|
return next(null, bucket, objMD);
|
||||||
|
},
|
||||||
|
function checkObjectAuth(bucket, objMD, next) {
|
||||||
|
if (!objMD) {
|
||||||
|
return next(null, bucket);
|
||||||
|
}
|
||||||
|
if (!isObjAuthorized(bucket, objMD, requestType, canonicalID)) {
|
||||||
|
log.debug('access denied for user on object', { requestType });
|
||||||
|
return next(errors.AccessDenied, bucket);
|
||||||
|
}
|
||||||
|
return next(null, bucket, objMD);
|
||||||
|
},
|
||||||
|
], (err, bucket, objMD) => {
|
||||||
|
if (err) {
|
||||||
|
// still return bucket for cors headers
|
||||||
|
return callback(err, bucket);
|
||||||
|
}
|
||||||
|
return callback(null, bucket, objMD);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/** metadataGetBucket - retrieves bucket from metadata, returning error if
|
||||||
|
* bucket is shielded
|
||||||
|
* @param {string} requestType - type of request
|
||||||
|
* @param {string} bucketName - name of bucket
|
||||||
|
* @param {RequestLogger} log - request logger
|
||||||
|
* @param {function} cb - callback
|
||||||
|
* @return {undefined} - and call callback with err, bucket md
|
||||||
|
*/
|
||||||
|
function metadataGetBucket(requestType, bucketName, log, cb) {
|
||||||
|
return metadata.getBucket(bucketName, log, (err, bucket) => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('metadata getbucket failed', { error: err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
if (bucketShield(bucket, requestType)) {
|
||||||
|
log.debug('bucket is shielded from request', {
|
||||||
|
requestType,
|
||||||
|
method: 'metadataGetBucketAndObject',
|
||||||
|
});
|
||||||
|
return cb(errors.NoSuchBucket);
|
||||||
|
}
|
||||||
|
log.trace('found bucket in metadata');
|
||||||
|
return cb(null, bucket);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/** metadataValidateBucket - retrieve bucket from metadata and check if user
|
||||||
|
* is authorized to access it
|
||||||
|
* @param {object} params - function parameters
|
||||||
|
* @param {AuthInfo} params.authInfo - AuthInfo class instance, requester's info
|
||||||
|
* @param {string} params.bucketName - name of bucket
|
||||||
|
* @param {string} params.requestType - type of request
|
||||||
|
* @param {RequestLogger} log - request logger
|
||||||
|
* @param {function} callback - callback
|
||||||
|
* @return {undefined} - and call callback with params err, bucket md
|
||||||
|
*/
|
||||||
|
function metadataValidateBucket(params, log, callback) {
|
||||||
|
const { authInfo, bucketName, requestType } = params;
|
||||||
|
const canonicalID = authInfo.getCanonicalID();
|
||||||
|
return metadataGetBucket(requestType, bucketName, log, (err, bucket) => {
|
||||||
|
if (err) {
|
||||||
|
return callback(err);
|
||||||
|
}
|
||||||
|
// still return bucket for cors headers
|
||||||
|
if (!isBucketAuthorized(bucket, requestType, canonicalID)) {
|
||||||
|
log.debug('access denied for user on bucket', { requestType });
|
||||||
|
return callback(errors.AccessDenied, bucket);
|
||||||
|
}
|
||||||
|
return callback(null, bucket);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
parseListEntries,
|
||||||
|
metadataGetObject,
|
||||||
|
metadataValidateBucketAndObj,
|
||||||
|
metadataValidateBucket,
|
||||||
|
};
|
|
@ -0,0 +1,173 @@
|
||||||
|
# Mongoclient
|
||||||
|
|
||||||
|
We introduce a new metadata backend called *mongoclient* for
|
||||||
|
[MongoDB](https://www.mongodb.com). This backend takes advantage of
|
||||||
|
MongoDB being a document store to store the metadata (bucket and
|
||||||
|
object attributes) as JSON objects.
|
||||||
|
|
||||||
|
## Overall Design
|
||||||
|
|
||||||
|
The mongoclient backend strictly follows the metadata interface that
|
||||||
|
stores bucket and object attributes, which consists of the methods
|
||||||
|
createBucket(), getBucketAttributes(), getBucketAndObject()
|
||||||
|
(attributes), putBucketAttributes(), deleteBucket(), putObject(),
|
||||||
|
getObject(), deleteObject(), listObject(), listMultipartUploads() and
|
||||||
|
the management methods getUUID(), getDiskUsage() and countItems(). The
|
||||||
|
mongoclient backend also knows how to deal with versioning, it is also
|
||||||
|
compatible with the various listing algorithms implemented in Arsenal.
|
||||||
|
|
||||||
|
FIXME: There should be a document describing the metadata (currently
|
||||||
|
duck-typing) interface.
|
||||||
|
|
||||||
|
### Why Using MongoDB for Storing Bucket and Object Attributes
|
||||||
|
|
||||||
|
We chose MongoDB for various reasons:
|
||||||
|
|
||||||
|
- MongoDB supports replication, especially through the Raft protocol.
|
||||||
|
|
||||||
|
- MongoDB supports a basic replication scheme called 'Replica Set' and
|
||||||
|
more advanced sharding schemes if required.
|
||||||
|
|
||||||
|
- MongoDB is open source and an enterprise standard.
|
||||||
|
|
||||||
|
- MongoDB is a document store (natively supports JSON) and supports a
|
||||||
|
very flexible search interface.
|
||||||
|
|
||||||
|
### Choice of Mongo Client Library
|
||||||
|
|
||||||
|
We chose to use the official MongoDB driver for NodeJS:
|
||||||
|
[https://github.com/mongodb/node-mongodb-native](https://github.com/mongodb/node-mongodb-native)
|
||||||
|
|
||||||
|
### Granularity for Buckets
|
||||||
|
|
||||||
|
We chose to have one collection for one bucket mapping. First because
|
||||||
|
in a simple mode of replication called 'replica set' it works from the
|
||||||
|
get-go, but if one or many buckets grow to big it is possible to use
|
||||||
|
more advanced schemes such as sharding. MongoDB supports a mix of
|
||||||
|
sharded and non-sharded collections.
|
||||||
|
|
||||||
|
### Storing Database Information
|
||||||
|
|
||||||
|
We need a special collection called the *Infostore* (stored under the
|
||||||
|
name __infostore which is impossible to create through the S3 bucket
|
||||||
|
naming scheme) to store specific database properties such as the
|
||||||
|
unique *uuid* for Orbit.
|
||||||
|
|
||||||
|
### Storing Bucket Attributes
|
||||||
|
|
||||||
|
We need to use a special collection called the *Metastore* (stored
|
||||||
|
under the name __metastore which is impossible to create through the
|
||||||
|
S3 bucket naming scheme).
|
||||||
|
|
||||||
|
### Versioning Format
|
||||||
|
|
||||||
|
We chose to keep the same versioning format that we use in some other
|
||||||
|
Scality products in order to facilitate the compatibility between the
|
||||||
|
different products.
|
||||||
|
|
||||||
|
FIXME: Document the versioning internals in the upper layers and
|
||||||
|
document the versioning format
|
||||||
|
|
||||||
|
### Dealing with Concurrency
|
||||||
|
|
||||||
|
We chose not to use transactions (aka
|
||||||
|
[https://docs.mongodb.com/manual/tutorial/perform-two-phase-commits/)
|
||||||
|
because it is a known fact there is an overhead of using them, and we
|
||||||
|
thought there was no real need for them since we could leverage Mongo
|
||||||
|
ordered operations guarantees and atomic writes.
|
||||||
|
|
||||||
|
Example of corner cases:
|
||||||
|
|
||||||
|
#### CreateBucket()
|
||||||
|
|
||||||
|
Since it is not possible to create a collection AND at the same time
|
||||||
|
register the bucket in the Metastore we chose to only update the
|
||||||
|
Metastore. A non-existing collection (NamespaceNotFound error in
|
||||||
|
Mongo) is one possible normal state for an empty bucket.
|
||||||
|
|
||||||
|
#### DeleteBucket()
|
||||||
|
|
||||||
|
In this case the bucket is *locked* by the upper layers (use of a
|
||||||
|
transient delete flag) so we don't have to worry about that and by the
|
||||||
|
fact the bucket is empty neither (which is also checked by the upper
|
||||||
|
layers).
|
||||||
|
|
||||||
|
We first drop() the collection and then we asynchronously delete the
|
||||||
|
bucket name entry from the metastore (the removal from the metastore
|
||||||
|
is atomic which is not absolutely necessary in this case but more
|
||||||
|
robust in term of design).
|
||||||
|
|
||||||
|
If we fail in between we still have an entry in the metastore which is
|
||||||
|
good because we need to manage the delete flag. For the upper layers
|
||||||
|
the operation has not completed until this flag is removed. The upper
|
||||||
|
layers will restart the deleteBucket() which is fine because we manage
|
||||||
|
the case where the collection does not exist.
|
||||||
|
|
||||||
|
#### PutObject() with a Version
|
||||||
|
|
||||||
|
We need to store the versioned object then update the master object
|
||||||
|
(the latest version). For this we use the
|
||||||
|
[BulkWrite](http://mongodb.github.io/node-mongodb-native/3.0/api/Collection.html#bulkWrite)
|
||||||
|
method. This is not a transaction but guarantees that the 2 operations
|
||||||
|
will happen sequentially in the MongoDB oplog. Indeed if the
|
||||||
|
BulkWrite() fails in between we would end up creating an orphan (which
|
||||||
|
is not critical) but if the operation succeeds then we are sure that
|
||||||
|
the master is always pointing to the right object. If there is a
|
||||||
|
concurrency between 2 clients then we are sure that the 2 groups of
|
||||||
|
operations will be clearly decided in the oplog (the last writer will
|
||||||
|
win).
|
||||||
|
|
||||||
|
#### DeleteObject()
|
||||||
|
|
||||||
|
This is probably the most complex case to manage because it involves a
|
||||||
|
lot of different cases:
|
||||||
|
|
||||||
|
##### Deleting an Object when Versioning is not Enabled
|
||||||
|
|
||||||
|
This case is a straightforward atomic delete. Atomicity is not really
|
||||||
|
required because we assume version IDs are random enough but it is
|
||||||
|
more robust to do so.
|
||||||
|
|
||||||
|
##### Deleting an Object when Versioning is Enabled
|
||||||
|
|
||||||
|
This case is more complex since we have to deal with the 2 cases:
|
||||||
|
|
||||||
|
Case 1: The caller asks for a deletion of a version which is not a master:
|
||||||
|
This case is a straight-forward atomic delete.
|
||||||
|
|
||||||
|
Case 2: The caller asks for a deletion of a version which is the master: In
|
||||||
|
this case we need to create a special flag called PHD (as PlaceHolDer)
|
||||||
|
that indicates the master is no longer valid (with a new unique
|
||||||
|
virtual version ID). We force the ordering of operations in a
|
||||||
|
bulkWrite() to first replace the master with the PHD flag and then
|
||||||
|
physically delete the version. If the call fail in between we will be
|
||||||
|
left with a master with a PHD flag. If the call succeeds we try to
|
||||||
|
find if the master with the PHD flag is left alone in such case we
|
||||||
|
delete it otherwise we trigger an asynchronous repair that will spawn
|
||||||
|
after AYNC_REPAIR_TIMEOUT=15s that will reassign the master to the
|
||||||
|
latest version.
|
||||||
|
|
||||||
|
In all cases the physical deletion or the repair of the master are
|
||||||
|
checked against the PHD flag AND the actual unique virtual version
|
||||||
|
ID. We do this to check against potential concurrent deletions,
|
||||||
|
repairs or updates. Only the last writer/deleter has the right to
|
||||||
|
physically perform the operation, otherwise it is superseded by other
|
||||||
|
operations.
|
||||||
|
|
||||||
|
##### Getting an object with a PHD flag
|
||||||
|
|
||||||
|
If the caller is asking for the latest version of an object and the
|
||||||
|
PHD flag is set we perform a search on the bucket to find the latest
|
||||||
|
version and we return it.
|
||||||
|
|
||||||
|
#### Listing Objects
|
||||||
|
|
||||||
|
The mongoclient backend implements a readable key/value stream called
|
||||||
|
*MongoReadStream* that follows the LevelDB duck typing interface used
|
||||||
|
in Arsenal/lib/algos listing algorithms. Note it does not require any
|
||||||
|
LevelDB package.
|
||||||
|
|
||||||
|
#### Generating the UUID
|
||||||
|
|
||||||
|
To avoid race conditions we always (try to) generate a new UUID and we
|
||||||
|
condition the insertion to the non-existence of the document.
|
|
@ -0,0 +1,766 @@
|
||||||
|
/*
|
||||||
|
* we assume good default setting of write concern is good for all
|
||||||
|
* bulk writes. Note that bulk writes are not transactions but ordered
|
||||||
|
* writes. They may fail in between. To some extend those situations
|
||||||
|
* may generate orphans but not alter the proper conduct of operations
|
||||||
|
* (what he user wants and what we acknowledge to the user).
|
||||||
|
*
|
||||||
|
* Orphan situations may be recovered by the Lifecycle.
|
||||||
|
*
|
||||||
|
* We use proper atomic operations when needed.
|
||||||
|
*/
|
||||||
|
const async = require('async');
|
||||||
|
const arsenal = require('arsenal');
|
||||||
|
|
||||||
|
const logger = require('../../utilities/logger');
|
||||||
|
|
||||||
|
const constants = require('../../../constants');
|
||||||
|
const { config } = require('../../Config');
|
||||||
|
|
||||||
|
const errors = arsenal.errors;
|
||||||
|
const versioning = arsenal.versioning;
|
||||||
|
const BucketInfo = arsenal.models.BucketInfo;
|
||||||
|
|
||||||
|
const MongoClient = require('mongodb').MongoClient;
|
||||||
|
const Uuid = require('uuid');
|
||||||
|
const diskusage = require('diskusage');
|
||||||
|
|
||||||
|
const genVID = versioning.VersionID.generateVersionId;
|
||||||
|
|
||||||
|
const MongoReadStream = require('./readStream');
|
||||||
|
const MongoUtils = require('./utils');
|
||||||
|
|
||||||
|
const USERSBUCKET = '__usersbucket';
|
||||||
|
const METASTORE = '__metastore';
|
||||||
|
const INFOSTORE = '__infostore';
|
||||||
|
const __UUID = 'uuid';
|
||||||
|
const ASYNC_REPAIR_TIMEOUT = 15000;
|
||||||
|
|
||||||
|
let uidCounter = 0;
|
||||||
|
|
||||||
|
const VID_SEP = versioning.VersioningConstants.VersionId.Separator;
|
||||||
|
|
||||||
|
function generateVersionId() {
|
||||||
|
// generate a unique number for each member of the nodejs cluster
|
||||||
|
return genVID(`${process.pid}.${uidCounter++}`,
|
||||||
|
config.replicationGroupId);
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatVersionKey(key, versionId) {
|
||||||
|
return `${key}${VID_SEP}${versionId}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function inc(str) {
|
||||||
|
return str ? (str.slice(0, str.length - 1) +
|
||||||
|
String.fromCharCode(str.charCodeAt(str.length - 1) + 1)) : str;
|
||||||
|
}
|
||||||
|
|
||||||
|
const VID_SEPPLUS = inc(VID_SEP);
|
||||||
|
|
||||||
|
function generatePHDVersion(versionId) {
|
||||||
|
return {
|
||||||
|
isPHD: true,
|
||||||
|
versionId,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
class MongoClientInterface {
|
||||||
|
constructor() {
|
||||||
|
const mongoUrl =
|
||||||
|
`mongodb://${config.mongodb.host}:${config.mongodb.port}`;
|
||||||
|
this.logger = logger;
|
||||||
|
this.client = null;
|
||||||
|
this.db = null;
|
||||||
|
this.logger.debug(`connecting to ${mongoUrl}`);
|
||||||
|
// FIXME: constructors shall not have side effect so there
|
||||||
|
// should be an async_init(cb) method in the wrapper to
|
||||||
|
// initialize this backend
|
||||||
|
MongoClient.connect(mongoUrl, (err, client) => {
|
||||||
|
if (err) {
|
||||||
|
throw (errors.InternalError);
|
||||||
|
}
|
||||||
|
this.logger.debug('connected to mongodb');
|
||||||
|
this.client = client;
|
||||||
|
this.db = client.db(config.mongodb.database, {
|
||||||
|
ignoreUndefined: true,
|
||||||
|
});
|
||||||
|
this.usersBucketHack();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
usersBucketHack() {
|
||||||
|
/* FIXME: Since the bucket creation API is expecting the
|
||||||
|
usersBucket to have attributes, we pre-create the
|
||||||
|
usersBucket attributes here (see bucketCreation.js line
|
||||||
|
36)*/
|
||||||
|
const usersBucketAttr = new BucketInfo(constants.usersBucket,
|
||||||
|
'admin', 'admin', new Date().toJSON(),
|
||||||
|
BucketInfo.currentModelVersion());
|
||||||
|
this.createBucket(
|
||||||
|
constants.usersBucket,
|
||||||
|
usersBucketAttr, {}, err => {
|
||||||
|
if (err) {
|
||||||
|
this.logger.fatal('error writing usersBucket ' +
|
||||||
|
'attributes to metastore',
|
||||||
|
{ error: err });
|
||||||
|
throw (errors.InternalError);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
getCollection(name) {
|
||||||
|
/* mongo has a problem with .. in collection names */
|
||||||
|
const newName = (name === constants.usersBucket) ?
|
||||||
|
USERSBUCKET : name;
|
||||||
|
return this.db.collection(newName);
|
||||||
|
}
|
||||||
|
|
||||||
|
createBucket(bucketName, bucketMD, log, cb) {
|
||||||
|
// FIXME: there should be a version of BucketInfo.serialize()
|
||||||
|
// that does not JSON.stringify()
|
||||||
|
const bucketInfo = BucketInfo.fromObj(bucketMD);
|
||||||
|
const bucketMDStr = bucketInfo.serialize();
|
||||||
|
const newBucketMD = JSON.parse(bucketMDStr);
|
||||||
|
const m = this.getCollection(METASTORE);
|
||||||
|
// we don't have to test bucket existence here as it is done
|
||||||
|
// on the upper layers
|
||||||
|
m.update({
|
||||||
|
_id: bucketName,
|
||||||
|
}, {
|
||||||
|
_id: bucketName,
|
||||||
|
value: newBucketMD,
|
||||||
|
}, {
|
||||||
|
upsert: true,
|
||||||
|
}, () => cb());
|
||||||
|
}
|
||||||
|
|
||||||
|
getBucketAttributes(bucketName, log, cb) {
|
||||||
|
const m = this.getCollection(METASTORE);
|
||||||
|
m.findOne({
|
||||||
|
_id: bucketName,
|
||||||
|
}, {}, (err, doc) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
if (!doc) {
|
||||||
|
return cb(errors.NoSuchBucket);
|
||||||
|
}
|
||||||
|
// FIXME: there should be a version of BucketInfo.deserialize()
|
||||||
|
// that properly inits w/o JSON.parse()
|
||||||
|
const bucketMDStr = JSON.stringify(doc.value);
|
||||||
|
const bucketMD = BucketInfo.deSerialize(bucketMDStr);
|
||||||
|
return cb(null, bucketMD);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
getBucketAndObject(bucketName, objName, params, log, cb) {
|
||||||
|
this.getBucketAttributes(bucketName, log, (err, bucket) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
this.getObject(bucketName, objName, params, log, (err, obj) => {
|
||||||
|
if (err) {
|
||||||
|
if (err === errors.NoSuchKey) {
|
||||||
|
return cb(null,
|
||||||
|
{ bucket:
|
||||||
|
BucketInfo.fromObj(bucket).serialize(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
return cb(null, {
|
||||||
|
bucket: BucketInfo.fromObj(bucket).serialize(),
|
||||||
|
obj: JSON.stringify(obj),
|
||||||
|
});
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
putBucketAttributes(bucketName, bucketMD, log, cb) {
|
||||||
|
// FIXME: there should be a version of BucketInfo.serialize()
|
||||||
|
// that does not JSON.stringify()
|
||||||
|
const bucketInfo = BucketInfo.fromObj(bucketMD);
|
||||||
|
const bucketMDStr = bucketInfo.serialize();
|
||||||
|
const newBucketMD = JSON.parse(bucketMDStr);
|
||||||
|
const m = this.getCollection(METASTORE);
|
||||||
|
m.update({
|
||||||
|
_id: bucketName,
|
||||||
|
}, {
|
||||||
|
_id: bucketName,
|
||||||
|
value: newBucketMD,
|
||||||
|
}, {
|
||||||
|
upsert: true,
|
||||||
|
}, () => cb());
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Delete bucket from metastore
|
||||||
|
*/
|
||||||
|
deleteBucketStep2(bucketName, log, cb) {
|
||||||
|
const m = this.getCollection(METASTORE);
|
||||||
|
m.findOneAndDelete({
|
||||||
|
_id: bucketName,
|
||||||
|
}, {}, (err, result) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
if (result.ok !== 1) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return cb(null);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Drop the bucket then process to step 2. Checking
|
||||||
|
* the count is already done by the upper layer. We don't need to be
|
||||||
|
* atomic because the call is protected by a delete_pending flag
|
||||||
|
* in the upper layer.
|
||||||
|
* 2 cases here:
|
||||||
|
* 1) the collection may not yet exist (or being already dropped
|
||||||
|
* by a previous call)
|
||||||
|
* 2) the collection may exist.
|
||||||
|
*/
|
||||||
|
deleteBucket(bucketName, log, cb) {
|
||||||
|
const c = this.getCollection(bucketName);
|
||||||
|
c.drop({}, err => {
|
||||||
|
if (err) {
|
||||||
|
if (err.codeName === 'NamespaceNotFound') {
|
||||||
|
return this.deleteBucketStep2(bucketName, log, cb);
|
||||||
|
}
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return this.deleteBucketStep2(bucketName, log, cb);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* In this case we generate a versionId and
|
||||||
|
* sequentially create the object THEN update the master
|
||||||
|
*/
|
||||||
|
putObjectVerCase1(c, bucketName, objName, objVal, params, log, cb) {
|
||||||
|
const versionId = generateVersionId();
|
||||||
|
// eslint-disable-next-line
|
||||||
|
objVal.versionId = versionId;
|
||||||
|
const vObjName = formatVersionKey(objName, versionId);
|
||||||
|
c.bulkWrite([{
|
||||||
|
updateOne: {
|
||||||
|
filter: {
|
||||||
|
_id: vObjName,
|
||||||
|
},
|
||||||
|
update: {
|
||||||
|
_id: vObjName, value: objVal,
|
||||||
|
},
|
||||||
|
upsert: true,
|
||||||
|
},
|
||||||
|
}, {
|
||||||
|
updateOne: {
|
||||||
|
filter: {
|
||||||
|
_id: objName,
|
||||||
|
},
|
||||||
|
update: {
|
||||||
|
_id: objName, value: objVal,
|
||||||
|
},
|
||||||
|
upsert: true,
|
||||||
|
},
|
||||||
|
}], {
|
||||||
|
ordered: 1,
|
||||||
|
}, () => cb(null, `{"versionId": "${versionId}"}`));
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Case used when versioning has been disabled after objects
|
||||||
|
* have been created with versions
|
||||||
|
*/
|
||||||
|
putObjectVerCase2(c, bucketName, objName, objVal, params, log, cb) {
|
||||||
|
const versionId = generateVersionId();
|
||||||
|
// eslint-disable-next-line
|
||||||
|
objVal.versionId = versionId;
|
||||||
|
c.update({
|
||||||
|
_id: objName,
|
||||||
|
}, {
|
||||||
|
_id: objName,
|
||||||
|
value: objVal,
|
||||||
|
}, {
|
||||||
|
upsert: true,
|
||||||
|
}, () => cb(null, `{"versionId": "${objVal.versionId}"}`));
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* In this case the aller provides a versionId. This function will
|
||||||
|
* sequentially update the object with given versionId THEN the
|
||||||
|
* master iff the provided versionId matches the one of the master
|
||||||
|
*/
|
||||||
|
putObjectVerCase3(c, bucketName, objName, objVal, params, log, cb) {
|
||||||
|
// eslint-disable-next-line
|
||||||
|
objVal.versionId = params.versionId;
|
||||||
|
const vObjName = formatVersionKey(objName, params.versionId);
|
||||||
|
c.bulkWrite([{
|
||||||
|
updateOne: {
|
||||||
|
filter: {
|
||||||
|
_id: vObjName,
|
||||||
|
},
|
||||||
|
update: {
|
||||||
|
_id: vObjName, value: objVal,
|
||||||
|
},
|
||||||
|
upsert: true,
|
||||||
|
},
|
||||||
|
}, {
|
||||||
|
updateOne: {
|
||||||
|
// eslint-disable-next-line
|
||||||
|
filter: {
|
||||||
|
_id: objName,
|
||||||
|
'value.versionId': params.versionId,
|
||||||
|
},
|
||||||
|
update: {
|
||||||
|
_id: objName, value: objVal,
|
||||||
|
},
|
||||||
|
upsert: true,
|
||||||
|
},
|
||||||
|
}], {
|
||||||
|
ordered: 1,
|
||||||
|
}, () => cb(null, `{"versionId": "${objVal.versionId}"}`));
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Put object when versioning is not enabled
|
||||||
|
*/
|
||||||
|
putObjectNoVer(c, bucketName, objName, objVal, params, log, cb) {
|
||||||
|
c.update({
|
||||||
|
_id: objName,
|
||||||
|
}, {
|
||||||
|
_id: objName,
|
||||||
|
value: objVal,
|
||||||
|
}, {
|
||||||
|
upsert: true,
|
||||||
|
}, () => cb());
|
||||||
|
}
|
||||||
|
|
||||||
|
putObject(bucketName, objName, objVal, params, log, cb) {
|
||||||
|
MongoUtils.serialize(objVal);
|
||||||
|
const c = this.getCollection(bucketName);
|
||||||
|
if (params && params.versioning && !params.versionId) {
|
||||||
|
return this.putObjectVerCase1(c, bucketName, objName, objVal,
|
||||||
|
params, log, cb);
|
||||||
|
} else if (params && params.versionId === '') {
|
||||||
|
return this.putObjectVerCase2(c, bucketName, objName, objVal,
|
||||||
|
params, log, cb);
|
||||||
|
} else if (params && params.versionId) {
|
||||||
|
return this.putObjectVerCase3(c, bucketName, objName, objVal,
|
||||||
|
params, log, cb);
|
||||||
|
}
|
||||||
|
return this.putObjectNoVer(c, bucketName, objName, objVal,
|
||||||
|
params, log, cb);
|
||||||
|
}
|
||||||
|
|
||||||
|
getObject(bucketName, objName, params, log, cb) {
|
||||||
|
const c = this.getCollection(bucketName);
|
||||||
|
if (params && params.versionId) {
|
||||||
|
// eslint-disable-next-line
|
||||||
|
objName = formatVersionKey(objName, params.versionId);
|
||||||
|
}
|
||||||
|
c.findOne({
|
||||||
|
_id: objName,
|
||||||
|
}, {}, (err, doc) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
if (!doc) {
|
||||||
|
return cb(errors.NoSuchKey);
|
||||||
|
}
|
||||||
|
if (doc.value.isPHD) {
|
||||||
|
this.getLatestVersion(c, objName, log, (err, value) => {
|
||||||
|
if (err) {
|
||||||
|
log.error('getting latest version', err);
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
return cb(null, value);
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
MongoUtils.unserialize(doc.value);
|
||||||
|
return cb(null, doc.value);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* This function return the latest version
|
||||||
|
*/
|
||||||
|
getLatestVersion(c, objName, log, cb) {
|
||||||
|
c.find({
|
||||||
|
_id: {
|
||||||
|
$gt: objName,
|
||||||
|
$lt: `${objName}${VID_SEPPLUS}`,
|
||||||
|
},
|
||||||
|
}, {}).
|
||||||
|
sort({
|
||||||
|
_id: 1,
|
||||||
|
}).
|
||||||
|
limit(1).
|
||||||
|
toArray(
|
||||||
|
(err, keys) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
if (keys.length === 0) {
|
||||||
|
return cb(errors.NoSuchKey);
|
||||||
|
}
|
||||||
|
MongoUtils.unserialize(keys[0].value);
|
||||||
|
return cb(null, keys[0].value);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* repair the master with a new value. There can be
|
||||||
|
* race-conditions or legit updates so place an atomic condition
|
||||||
|
* on PHD flag and mst version.
|
||||||
|
*/
|
||||||
|
repair(c, objName, objVal, mst, log, cb) {
|
||||||
|
MongoUtils.serialize(objVal);
|
||||||
|
// eslint-disable-next-line
|
||||||
|
c.findOneAndReplace({
|
||||||
|
_id: objName,
|
||||||
|
'value.isPHD': true,
|
||||||
|
'value.versionId': mst.versionId,
|
||||||
|
}, {
|
||||||
|
_id: objName,
|
||||||
|
value: objVal,
|
||||||
|
}, {
|
||||||
|
upsert: true,
|
||||||
|
}, (err, result) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
if (result.ok !== 1) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return cb(null);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Get the latest version and repair. The process is safe because
|
||||||
|
* we never replace a non-PHD master
|
||||||
|
*/
|
||||||
|
asyncRepair(c, objName, mst, log) {
|
||||||
|
this.getLatestVersion(c, objName, log, (err, value) => {
|
||||||
|
if (err) {
|
||||||
|
log.error('async-repair: getting latest version', err);
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
this.repair(c, objName, value, mst, log, err => {
|
||||||
|
if (err) {
|
||||||
|
log.error('async-repair failed', err);
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
log.debug('async-repair success');
|
||||||
|
return undefined;
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* the master is a PHD so we try to see if it is the latest of its
|
||||||
|
* kind to get rid of it, otherwise we asynchronously repair it
|
||||||
|
*/
|
||||||
|
deleteOrRepairPHD(c, bucketName, objName, mst, log, cb) {
|
||||||
|
// Check if there are other versions available
|
||||||
|
this.getLatestVersion(c, objName, log, err => {
|
||||||
|
if (err) {
|
||||||
|
if (err === errors.NoSuchKey) {
|
||||||
|
// We try to delete the master. A race condition
|
||||||
|
// is possible here: another process may recreate
|
||||||
|
// a master or re-delete it in between so place an
|
||||||
|
// atomic condition on the PHD flag and the mst
|
||||||
|
// version:
|
||||||
|
// eslint-disable-next-line
|
||||||
|
c.findOneAndDelete({
|
||||||
|
_id: objName,
|
||||||
|
'value.isPHD': true,
|
||||||
|
'value.versionId': mst.versionId,
|
||||||
|
}, {}, err => {
|
||||||
|
if (err) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
// do not test result.ok === 1 because
|
||||||
|
// both cases are expected
|
||||||
|
return cb(null);
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
// We have other versions available so repair:
|
||||||
|
setTimeout(() => {
|
||||||
|
this.asyncRepair(c, objName, mst, log);
|
||||||
|
}, ASYNC_REPAIR_TIMEOUT);
|
||||||
|
return cb(null);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Delete object when versioning is enabled and the version is
|
||||||
|
* master. In this case we sequentially update the master with a
|
||||||
|
* PHD flag (placeholder) and a unique non-existing version THEN
|
||||||
|
* we delete the specified versioned object. THEN we try to delete
|
||||||
|
* or repair the PHD we just created
|
||||||
|
*/
|
||||||
|
deleteObjectVerMaster(c, bucketName, objName, params, log, cb) {
|
||||||
|
const vObjName = formatVersionKey(objName, params.versionId);
|
||||||
|
const _vid = generateVersionId();
|
||||||
|
const mst = generatePHDVersion(_vid);
|
||||||
|
c.bulkWrite([{
|
||||||
|
updateOne: {
|
||||||
|
filter: {
|
||||||
|
_id: objName,
|
||||||
|
},
|
||||||
|
update: {
|
||||||
|
_id: objName, value: mst,
|
||||||
|
},
|
||||||
|
upsert: true,
|
||||||
|
},
|
||||||
|
}, {
|
||||||
|
deleteOne: {
|
||||||
|
filter: {
|
||||||
|
_id: vObjName,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}], {
|
||||||
|
ordered: 1,
|
||||||
|
}, () => this.deleteOrRepairPHD(c, bucketName, objName, mst, log, cb));
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Delete object when versioning is enabled and the version is
|
||||||
|
* not master. It is a straight-forward atomic delete
|
||||||
|
*/
|
||||||
|
deleteObjectVerNotMaster(c, bucketName, objName, params, log, cb) {
|
||||||
|
const vObjName = formatVersionKey(objName, params.versionId);
|
||||||
|
c.findOneAndDelete({
|
||||||
|
_id: vObjName,
|
||||||
|
}, {}, (err, result) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
if (result.ok !== 1) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return cb(null);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Delete object when versioning is enabled. We first find the
|
||||||
|
* master, if it is already a PHD we have a special processing,
|
||||||
|
* then we check if it matches the master versionId in such case
|
||||||
|
* we will create a PHD, otherwise we delete it
|
||||||
|
*/
|
||||||
|
deleteObjectVer(c, bucketName, objName, params, log, cb) {
|
||||||
|
c.findOne({
|
||||||
|
_id: objName,
|
||||||
|
}, {}, (err, mst) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
if (!mst) {
|
||||||
|
return cb(errors.NoSuchKey);
|
||||||
|
}
|
||||||
|
if (mst.value.isPHD ||
|
||||||
|
mst.value.versionId === params.versionId) {
|
||||||
|
return this.deleteObjectVerMaster(c, bucketName, objName,
|
||||||
|
params, log, cb);
|
||||||
|
}
|
||||||
|
return this.deleteObjectVerNotMaster(c, bucketName, objName,
|
||||||
|
params, log, cb);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* Atomically delete an object when versioning is not enabled
|
||||||
|
*/
|
||||||
|
deleteObjectNoVer(c, bucketName, objName, params, log, cb) {
|
||||||
|
c.findOneAndDelete({
|
||||||
|
_id: objName,
|
||||||
|
}, {}, (err, result) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
if (result.ok !== 1) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return cb(null);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
deleteObject(bucketName, objName, params, log, cb) {
|
||||||
|
const c = this.getCollection(bucketName);
|
||||||
|
if (params && params.versionId) {
|
||||||
|
return this.deleteObjectVer(c, bucketName, objName,
|
||||||
|
params, log, cb);
|
||||||
|
}
|
||||||
|
return this.deleteObjectNoVer(c, bucketName, objName,
|
||||||
|
params, log, cb);
|
||||||
|
}
|
||||||
|
|
||||||
|
internalListObject(bucketName, params, log, cb) {
|
||||||
|
const extName = params.listingType;
|
||||||
|
const extension = new arsenal.algorithms.list[extName](params, log);
|
||||||
|
const requestParams = extension.genMDParams();
|
||||||
|
const c = this.getCollection(bucketName);
|
||||||
|
let cbDone = false;
|
||||||
|
const stream = new MongoReadStream(c, requestParams);
|
||||||
|
stream
|
||||||
|
.on('data', e => {
|
||||||
|
if (extension.filter(e) < 0) {
|
||||||
|
stream.emit('end');
|
||||||
|
stream.destroy();
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.on('error', err => {
|
||||||
|
if (!cbDone) {
|
||||||
|
cbDone = true;
|
||||||
|
const logObj = {
|
||||||
|
rawError: err,
|
||||||
|
error: err.message,
|
||||||
|
errorStack: err.stack,
|
||||||
|
};
|
||||||
|
log.error('error listing objects', logObj);
|
||||||
|
cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.on('end', () => {
|
||||||
|
if (!cbDone) {
|
||||||
|
cbDone = true;
|
||||||
|
const data = extension.result();
|
||||||
|
cb(null, data);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
listObject(bucketName, params, log, cb) {
|
||||||
|
return this.internalListObject(bucketName, params, log, cb);
|
||||||
|
}
|
||||||
|
|
||||||
|
listMultipartUploads(bucketName, params, log, cb) {
|
||||||
|
return this.internalListObject(bucketName, params, log, cb);
|
||||||
|
}
|
||||||
|
|
||||||
|
readUUID(log, cb) {
|
||||||
|
const i = this.getCollection(INFOSTORE);
|
||||||
|
i.findOne({
|
||||||
|
_id: __UUID,
|
||||||
|
}, {}, (err, doc) => {
|
||||||
|
if (err) {
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
if (!doc) {
|
||||||
|
return cb(errors.NoSuchKey);
|
||||||
|
}
|
||||||
|
return cb(null, doc.value);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
writeUUIDIfNotExists(uuid, log, cb) {
|
||||||
|
const i = this.getCollection(INFOSTORE);
|
||||||
|
i.insert({
|
||||||
|
_id: __UUID,
|
||||||
|
value: uuid,
|
||||||
|
}, {}, err => {
|
||||||
|
if (err) {
|
||||||
|
if (err.code === 11000) {
|
||||||
|
// duplicate key error
|
||||||
|
// FIXME: define a KeyAlreadyExists error in Arsenal
|
||||||
|
return cb(errors.EntityAlreadyExists);
|
||||||
|
}
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
// FIXME: shoud we check for result.ok === 1 ?
|
||||||
|
return cb(null);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/*
|
||||||
|
* we always try to generate a new UUID in order to be atomic in
|
||||||
|
* case of concurrency. The write will fail if it already exists.
|
||||||
|
*/
|
||||||
|
getUUID(log, cb) {
|
||||||
|
const uuid = Uuid.v4();
|
||||||
|
this.writeUUIDIfNotExists(uuid, log, err => {
|
||||||
|
if (err) {
|
||||||
|
if (err === errors.InternalError) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
return this.readUUID(log, cb);
|
||||||
|
}
|
||||||
|
return cb(null, uuid);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
getDiskUsage(cb) {
|
||||||
|
// FIXME: for basic one server deployment the infrastructure
|
||||||
|
// configurator shall set a path to the actual MongoDB volume.
|
||||||
|
// For Kub/cluster deployments there should be a more sophisticated
|
||||||
|
// way for guessing free space.
|
||||||
|
diskusage.check(config.mongodb.path !== undefined ?
|
||||||
|
config.mongodb.path : '/', cb);
|
||||||
|
}
|
||||||
|
|
||||||
|
countItems(log, cb) {
|
||||||
|
const res = {
|
||||||
|
objects: 0,
|
||||||
|
versions: 0,
|
||||||
|
buckets: 0,
|
||||||
|
};
|
||||||
|
this.db.listCollections().toArray((err, collInfos) => {
|
||||||
|
async.eachLimit(collInfos, 10, (value, next) => {
|
||||||
|
if (value.name === METASTORE ||
|
||||||
|
value.name === INFOSTORE ||
|
||||||
|
value.name === USERSBUCKET) {
|
||||||
|
// skip
|
||||||
|
return next();
|
||||||
|
}
|
||||||
|
res.buckets++;
|
||||||
|
const c = this.getCollection(value.name);
|
||||||
|
// FIXME: there is currently no way of distinguishing
|
||||||
|
// master from versions and searching for VID_SEP
|
||||||
|
// does not work because there cannot be null bytes
|
||||||
|
// in $regex
|
||||||
|
c.count({
|
||||||
|
// eslint-disable-next-line
|
||||||
|
'value.versionId': {
|
||||||
|
'$exists': false,
|
||||||
|
},
|
||||||
|
}, {}, (err, result) => {
|
||||||
|
if (err) {
|
||||||
|
return next(errors.InternalError);
|
||||||
|
}
|
||||||
|
res.objects += result;
|
||||||
|
c.count({
|
||||||
|
// eslint-disable-next-line
|
||||||
|
'value.versionId': {
|
||||||
|
'$exists': true,
|
||||||
|
},
|
||||||
|
}, {}, (err, result) => {
|
||||||
|
if (err) {
|
||||||
|
return next(errors.InternalError);
|
||||||
|
}
|
||||||
|
res.versions += result;
|
||||||
|
return next();
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
});
|
||||||
|
return undefined;
|
||||||
|
}, err => {
|
||||||
|
if (err) {
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
return cb(null, res);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = MongoClientInterface;
|
|
@ -0,0 +1,133 @@
|
||||||
|
const Readable = require('stream').Readable;
|
||||||
|
const MongoUtils = require('./utils');
|
||||||
|
|
||||||
|
class MongoReadStream extends Readable {
|
||||||
|
constructor(c, options) {
|
||||||
|
super({
|
||||||
|
objectMode: true,
|
||||||
|
highWaterMark: 0,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (options.limit === 0) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const query = {
|
||||||
|
_id: {},
|
||||||
|
};
|
||||||
|
if (options.reverse) {
|
||||||
|
if (options.start) {
|
||||||
|
query._id.$lte = options.start;
|
||||||
|
}
|
||||||
|
if (options.end) {
|
||||||
|
query._id.$gte = options.end;
|
||||||
|
}
|
||||||
|
if (options.gt) {
|
||||||
|
query._id.$lt = options.gt;
|
||||||
|
}
|
||||||
|
if (options.gte) {
|
||||||
|
query._id.$lte = options.gte;
|
||||||
|
}
|
||||||
|
if (options.lt) {
|
||||||
|
query._id.$gt = options.lt;
|
||||||
|
}
|
||||||
|
if (options.lte) {
|
||||||
|
query._id.$gte = options.lte;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if (options.start) {
|
||||||
|
query._id.$gte = options.start;
|
||||||
|
}
|
||||||
|
if (options.end) {
|
||||||
|
query._id.$lte = options.end;
|
||||||
|
}
|
||||||
|
if (options.gt) {
|
||||||
|
query._id.$gt = options.gt;
|
||||||
|
}
|
||||||
|
if (options.gte) {
|
||||||
|
query._id.$gte = options.gte;
|
||||||
|
}
|
||||||
|
if (options.lt) {
|
||||||
|
query._id.$lt = options.lt;
|
||||||
|
}
|
||||||
|
if (options.lte) {
|
||||||
|
query._id.$lte = options.lte;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!Object.keys(query._id).length) {
|
||||||
|
delete query._id;
|
||||||
|
}
|
||||||
|
|
||||||
|
this._cursor = c.find(query).sort({
|
||||||
|
_id: options.reverse ? -1 : 1,
|
||||||
|
});
|
||||||
|
if (options.limit && options.limit !== -1) {
|
||||||
|
this._cursor = this._cursor.limit(options.limit);
|
||||||
|
}
|
||||||
|
this._options = options;
|
||||||
|
this._destroyed = false;
|
||||||
|
this.on('end', this._cleanup.bind(this));
|
||||||
|
}
|
||||||
|
|
||||||
|
_read() {
|
||||||
|
if (this._destroyed) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
this._cursor.next((err, doc) => {
|
||||||
|
if (this._destroyed) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (err) {
|
||||||
|
this.emit('error', err);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
let key = undefined;
|
||||||
|
let value = undefined;
|
||||||
|
|
||||||
|
if (doc) {
|
||||||
|
key = doc._id;
|
||||||
|
MongoUtils.unserialize(doc.value);
|
||||||
|
value = JSON.stringify(doc.value);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (key === undefined && value === undefined) {
|
||||||
|
this.push(null);
|
||||||
|
} else if (this._options.keys !== false &&
|
||||||
|
this._options.values === false) {
|
||||||
|
this.push(key);
|
||||||
|
} else if (this._options.keys === false &&
|
||||||
|
this._options.values !== false) {
|
||||||
|
this.push(value);
|
||||||
|
} else {
|
||||||
|
this.push({
|
||||||
|
key,
|
||||||
|
value,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
_cleanup() {
|
||||||
|
if (this._destroyed) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this._destroyed = true;
|
||||||
|
|
||||||
|
this._cursor.close(err => {
|
||||||
|
if (err) {
|
||||||
|
this.emit('error', err);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.emit('close');
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
destroy() {
|
||||||
|
return this._cleanup();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = MongoReadStream;
|
|
@ -0,0 +1,30 @@
|
||||||
|
|
||||||
|
function escape(obj) {
|
||||||
|
return JSON.parse(JSON.stringify(obj).
|
||||||
|
replace(/\$/g, '\uFF04').
|
||||||
|
replace(/\./g, '\uFF0E'));
|
||||||
|
}
|
||||||
|
|
||||||
|
function unescape(obj) {
|
||||||
|
return JSON.parse(JSON.stringify(obj).
|
||||||
|
replace(/\uFF04/g, '$').
|
||||||
|
replace(/\uFF0E/g, '.'));
|
||||||
|
}
|
||||||
|
|
||||||
|
function serialize(objMD) {
|
||||||
|
// Tags require special handling since dot and dollar are accepted
|
||||||
|
if (objMD.tags) {
|
||||||
|
// eslint-disable-next-line
|
||||||
|
objMD.tags = escape(objMD.tags);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function unserialize(objMD) {
|
||||||
|
// Tags require special handling
|
||||||
|
if (objMD.tags) {
|
||||||
|
// eslint-disable-next-line
|
||||||
|
objMD.tags = unescape(objMD.tags);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { escape, unescape, serialize, unserialize };
|
|
@ -0,0 +1,256 @@
|
||||||
|
const errors = require('arsenal').errors;
|
||||||
|
|
||||||
|
const BucketClientInterface = require('./bucketclient/backend');
|
||||||
|
const BucketFileInterface = require('./bucketfile/backend');
|
||||||
|
const MongoClientInterface = require('./mongoclient/backend');
|
||||||
|
const BucketInfo = require('arsenal').models.BucketInfo;
|
||||||
|
const inMemory = require('./in_memory/backend');
|
||||||
|
const { config } = require('../Config');
|
||||||
|
|
||||||
|
let CdmiMetadata;
|
||||||
|
try {
|
||||||
|
CdmiMetadata = require('cdmiclient').CdmiMetadata;
|
||||||
|
} catch (err) {
|
||||||
|
CdmiMetadata = null;
|
||||||
|
}
|
||||||
|
|
||||||
|
let client;
|
||||||
|
let implName;
|
||||||
|
|
||||||
|
if (config.backends.metadata === 'mem') {
|
||||||
|
client = inMemory;
|
||||||
|
implName = 'memorybucket';
|
||||||
|
} else if (config.backends.metadata === 'file') {
|
||||||
|
client = new BucketFileInterface();
|
||||||
|
implName = 'bucketfile';
|
||||||
|
} else if (config.backends.metadata === 'scality') {
|
||||||
|
client = new BucketClientInterface();
|
||||||
|
implName = 'bucketclient';
|
||||||
|
} else if (config.backends.metadata === 'mongodb') {
|
||||||
|
client = new MongoClientInterface();
|
||||||
|
implName = 'mongoclient';
|
||||||
|
} else if (config.backends.metadata === 'cdmi') {
|
||||||
|
if (!CdmiMetadata) {
|
||||||
|
throw new Error('Unauthorized backend');
|
||||||
|
}
|
||||||
|
|
||||||
|
client = new CdmiMetadata({
|
||||||
|
path: config.cdmi.path,
|
||||||
|
host: config.cdmi.host,
|
||||||
|
port: config.cdmi.port,
|
||||||
|
readonly: config.cdmi.readonly,
|
||||||
|
});
|
||||||
|
implName = 'cdmi';
|
||||||
|
}
|
||||||
|
|
||||||
|
const metadata = {
|
||||||
|
createBucket: (bucketName, bucketMD, log, cb) => {
|
||||||
|
log.debug('creating bucket in metadata');
|
||||||
|
client.createBucket(bucketName, bucketMD, log, err => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('error from metadata', { implName, error: err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
log.trace('bucket created in metadata');
|
||||||
|
return cb(err);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
updateBucket: (bucketName, bucketMD, log, cb) => {
|
||||||
|
log.debug('updating bucket in metadata');
|
||||||
|
client.putBucketAttributes(bucketName, bucketMD, log, err => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('error from metadata', { implName, error: err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
log.trace('bucket updated in metadata');
|
||||||
|
return cb(err);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
getBucket: (bucketName, log, cb) => {
|
||||||
|
log.debug('getting bucket from metadata');
|
||||||
|
client.getBucketAttributes(bucketName, log, (err, data) => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('error from metadata', { implName, error: err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
log.trace('bucket retrieved from metadata');
|
||||||
|
return cb(err, BucketInfo.fromObj(data));
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
deleteBucket: (bucketName, log, cb) => {
|
||||||
|
log.debug('deleting bucket from metadata');
|
||||||
|
client.deleteBucket(bucketName, log, err => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('error from metadata', { implName, error: err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
log.debug('Deleted bucket from Metadata');
|
||||||
|
return cb(err);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
putObjectMD: (bucketName, objName, objVal, params, log, cb) => {
|
||||||
|
log.debug('putting object in metadata');
|
||||||
|
const value = typeof objVal.getValue === 'function' ?
|
||||||
|
objVal.getValue() : objVal;
|
||||||
|
client.putObject(bucketName, objName, value, params, log,
|
||||||
|
(err, data) => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('error from metadata', { implName, error: err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
if (data) {
|
||||||
|
log.debug('object version successfully put in metadata',
|
||||||
|
{ version: data });
|
||||||
|
} else {
|
||||||
|
log.debug('object successfully put in metadata');
|
||||||
|
}
|
||||||
|
return cb(err, data);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
getBucketAndObjectMD: (bucketName, objName, params, log, cb) => {
|
||||||
|
log.debug('getting bucket and object from metadata',
|
||||||
|
{ database: bucketName, object: objName });
|
||||||
|
client.getBucketAndObject(bucketName, objName, params, log,
|
||||||
|
(err, data) => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('error from metadata', { implName, err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
log.debug('bucket and object retrieved from metadata',
|
||||||
|
{ database: bucketName, object: objName });
|
||||||
|
return cb(err, data);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
getObjectMD: (bucketName, objName, params, log, cb) => {
|
||||||
|
log.debug('getting object from metadata');
|
||||||
|
client.getObject(bucketName, objName, params, log, (err, data) => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('error from metadata', { implName, err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
log.debug('object retrieved from metadata');
|
||||||
|
return cb(err, data);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
deleteObjectMD: (bucketName, objName, params, log, cb) => {
|
||||||
|
log.debug('deleting object from metadata');
|
||||||
|
client.deleteObject(bucketName, objName, params, log, err => {
|
||||||
|
if (err) {
|
||||||
|
log.debug('error from metadata', { implName, err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
log.debug('object deleted from metadata');
|
||||||
|
return cb(err);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
listObject: (bucketName, listingParams, log, cb) => {
|
||||||
|
const metadataUtils = require('./metadataUtils');
|
||||||
|
if (listingParams.listingType === undefined) {
|
||||||
|
// eslint-disable-next-line
|
||||||
|
listingParams.listingType = 'Delimiter';
|
||||||
|
}
|
||||||
|
client.listObject(bucketName, listingParams, log, (err, data) => {
|
||||||
|
log.debug('getting object listing from metadata');
|
||||||
|
if (err) {
|
||||||
|
log.debug('error from metadata', { implName, err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
log.debug('object listing retrieved from metadata');
|
||||||
|
if (listingParams.listingType === 'DelimiterVersions') {
|
||||||
|
// eslint-disable-next-line
|
||||||
|
data.Versions = metadataUtils.parseListEntries(data.Versions);
|
||||||
|
if (data.Versions instanceof Error) {
|
||||||
|
log.error('error parsing metadata listing', {
|
||||||
|
error: data.Versions,
|
||||||
|
listingType: listingParams.listingType,
|
||||||
|
method: 'listObject',
|
||||||
|
});
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return cb(null, data);
|
||||||
|
}
|
||||||
|
// eslint-disable-next-line
|
||||||
|
data.Contents = metadataUtils.parseListEntries(data.Contents);
|
||||||
|
if (data.Contents instanceof Error) {
|
||||||
|
log.error('error parsing metadata listing', {
|
||||||
|
error: data.Contents,
|
||||||
|
listingType: listingParams.listingType,
|
||||||
|
method: 'listObject',
|
||||||
|
});
|
||||||
|
return cb(errors.InternalError);
|
||||||
|
}
|
||||||
|
return cb(null, data);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
listMultipartUploads: (bucketName, listingParams, log, cb) => {
|
||||||
|
client.listMultipartUploads(bucketName, listingParams, log,
|
||||||
|
(err, data) => {
|
||||||
|
log.debug('getting mpu listing from metadata');
|
||||||
|
if (err) {
|
||||||
|
log.debug('error from metadata', { implName, err });
|
||||||
|
return cb(err);
|
||||||
|
}
|
||||||
|
log.debug('mpu listing retrieved from metadata');
|
||||||
|
return cb(err, data);
|
||||||
|
});
|
||||||
|
},
|
||||||
|
|
||||||
|
switch: newClient => {
|
||||||
|
client = newClient;
|
||||||
|
},
|
||||||
|
|
||||||
|
getRaftBuckets: (raftId, log, cb) => {
|
||||||
|
if (!client.getRaftBuckets) {
|
||||||
|
return cb();
|
||||||
|
}
|
||||||
|
return client.getRaftBuckets(raftId, log, cb);
|
||||||
|
},
|
||||||
|
|
||||||
|
checkHealth: (log, cb) => {
|
||||||
|
if (!client.checkHealth) {
|
||||||
|
const defResp = {};
|
||||||
|
defResp[implName] = { code: 200, message: 'OK' };
|
||||||
|
return cb(null, defResp);
|
||||||
|
}
|
||||||
|
return client.checkHealth(implName, log, cb);
|
||||||
|
},
|
||||||
|
|
||||||
|
getUUID: (log, cb) => {
|
||||||
|
if (!client.getUUID) {
|
||||||
|
log.debug('returning empty uuid as fallback', { implName });
|
||||||
|
return cb(null, '');
|
||||||
|
}
|
||||||
|
return client.getUUID(log, cb);
|
||||||
|
},
|
||||||
|
|
||||||
|
getDiskUsage: (log, cb) => {
|
||||||
|
if (!client.getDiskUsage) {
|
||||||
|
log.debug('returning empty disk usage as fallback', { implName });
|
||||||
|
return cb(null, {});
|
||||||
|
}
|
||||||
|
return client.getDiskUsage(cb);
|
||||||
|
},
|
||||||
|
|
||||||
|
countItems: (log, cb) => {
|
||||||
|
if (!client.countItems) {
|
||||||
|
log.debug('returning zero item counts as fallback', { implName });
|
||||||
|
return cb(null, {
|
||||||
|
buckets: 0,
|
||||||
|
objects: 0,
|
||||||
|
versions: 0,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return client.countItems(log, cb);
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
module.exports = metadata;
|
|
@ -1,4 +0,0 @@
|
||||||
#!/usr/bin/env node
|
|
||||||
'use strict'; // eslint-disable-line strict
|
|
||||||
|
|
||||||
require('../lib/nfs/utilities.js').createBucketWithNFSEnabled();
|
|
|
@ -1,108 +0,0 @@
|
||||||
#!/bin/sh
|
|
||||||
// 2>/dev/null ; exec "$(which nodejs 2>/dev/null || which node)" "$0" "$@"
|
|
||||||
'use strict'; // eslint-disable-line strict
|
|
||||||
|
|
||||||
const { auth } = require('arsenal');
|
|
||||||
const commander = require('commander');
|
|
||||||
|
|
||||||
const http = require('http');
|
|
||||||
const https = require('https');
|
|
||||||
const logger = require('../lib/utilities/logger');
|
|
||||||
|
|
||||||
function _performSearch(host,
|
|
||||||
port,
|
|
||||||
bucketName,
|
|
||||||
query,
|
|
||||||
listVersions,
|
|
||||||
accessKey,
|
|
||||||
secretKey,
|
|
||||||
sessionToken,
|
|
||||||
verbose, ssl) {
|
|
||||||
const escapedSearch = encodeURIComponent(query);
|
|
||||||
const options = {
|
|
||||||
host,
|
|
||||||
port,
|
|
||||||
method: 'GET',
|
|
||||||
path: `/${bucketName}/?search=${escapedSearch}${listVersions ? '&&versions' : ''}`,
|
|
||||||
headers: {
|
|
||||||
'Content-Length': 0,
|
|
||||||
},
|
|
||||||
rejectUnauthorized: false,
|
|
||||||
versions: '',
|
|
||||||
};
|
|
||||||
if (sessionToken) {
|
|
||||||
options.headers['x-amz-security-token'] = sessionToken;
|
|
||||||
}
|
|
||||||
const transport = ssl ? https : http;
|
|
||||||
const request = transport.request(options, response => {
|
|
||||||
if (verbose) {
|
|
||||||
logger.info('response status code', {
|
|
||||||
statusCode: response.statusCode,
|
|
||||||
});
|
|
||||||
logger.info('response headers', { headers: response.headers });
|
|
||||||
}
|
|
||||||
const body = [];
|
|
||||||
response.setEncoding('utf8');
|
|
||||||
response.on('data', chunk => body.push(chunk));
|
|
||||||
response.on('end', () => {
|
|
||||||
if (response.statusCode >= 200 && response.statusCode < 300) {
|
|
||||||
logger.info('Success');
|
|
||||||
process.stdout.write(body.join(''));
|
|
||||||
process.exit(0);
|
|
||||||
} else {
|
|
||||||
logger.error('request failed with HTTP Status ', {
|
|
||||||
statusCode: response.statusCode,
|
|
||||||
body: body.join(''),
|
|
||||||
});
|
|
||||||
process.exit(1);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
});
|
|
||||||
// generateV4Headers exepects request object with path that does not
|
|
||||||
// include query
|
|
||||||
request.path = `/${bucketName}`;
|
|
||||||
const requestData = listVersions ? { search: query, versions: '' } : { search: query };
|
|
||||||
auth.client.generateV4Headers(request, requestData, accessKey, secretKey, 's3');
|
|
||||||
request.path = `/${bucketName}?search=${escapedSearch}${listVersions ? '&&versions' : ''}`;
|
|
||||||
if (verbose) {
|
|
||||||
logger.info('request headers', { headers: request._headers });
|
|
||||||
}
|
|
||||||
request.end();
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* This function is used as a binary to send a request to S3 to perform a
|
|
||||||
* search on the objects in a bucket
|
|
||||||
*
|
|
||||||
* @return {undefined}
|
|
||||||
*/
|
|
||||||
function searchBucket() {
|
|
||||||
// TODO: Include other bucket listing possible query params?
|
|
||||||
commander
|
|
||||||
.version('0.0.1')
|
|
||||||
.option('-a, --access-key <accessKey>', 'Access key id')
|
|
||||||
.option('-k, --secret-key <secretKey>', 'Secret access key')
|
|
||||||
.option('-t, --session-token <sessionToken>', 'Session token')
|
|
||||||
.option('-b, --bucket <bucket>', 'Name of the bucket')
|
|
||||||
.option('-q, --query <query>', 'Search query')
|
|
||||||
.option('-h, --host <host>', 'Host of the server')
|
|
||||||
.option('-p, --port <port>', 'Port of the server')
|
|
||||||
.option('-s', '--ssl', 'Enable ssl')
|
|
||||||
.option('-l, --list-versions', 'List all versions of the objects that meet the search query, ' +
|
|
||||||
'otherwise only list the latest version')
|
|
||||||
.option('-v, --verbose')
|
|
||||||
.parse(process.argv);
|
|
||||||
const { host, port, accessKey, secretKey, sessionToken, bucket, query, listVersions, verbose, ssl } =
|
|
||||||
commander;
|
|
||||||
|
|
||||||
if (!host || !port || !accessKey || !secretKey || !bucket || !query) {
|
|
||||||
logger.error('missing parameter');
|
|
||||||
commander.outputHelp();
|
|
||||||
process.exit(1);
|
|
||||||
}
|
|
||||||
|
|
||||||
_performSearch(host, port, bucket, query, listVersions, accessKey, secretKey, sessionToken, verbose,
|
|
||||||
ssl);
|
|
||||||
}
|
|
||||||
|
|
||||||
searchBucket();
|
|
|
@ -0,0 +1,45 @@
|
||||||
|
---
|
||||||
|
general:
|
||||||
|
branches:
|
||||||
|
ignore:
|
||||||
|
- /^ultron\/.*/ # Ignore ultron/* branches
|
||||||
|
artifacts:
|
||||||
|
- coverage/
|
||||||
|
|
||||||
|
machine:
|
||||||
|
node:
|
||||||
|
version: 6.13.1
|
||||||
|
services:
|
||||||
|
- redis
|
||||||
|
- docker
|
||||||
|
ruby:
|
||||||
|
version: "2.4.1"
|
||||||
|
environment:
|
||||||
|
CXX: g++-4.9
|
||||||
|
ENABLE_LOCAL_CACHE: true
|
||||||
|
REPORT_TOKEN: report-token-1
|
||||||
|
hosts:
|
||||||
|
bucketwebsitetester.s3-website-us-east-1.amazonaws.com: 127.0.0.1
|
||||||
|
|
||||||
|
dependencies:
|
||||||
|
override:
|
||||||
|
- rm -rf node_modules
|
||||||
|
- npm install
|
||||||
|
post:
|
||||||
|
- sudo pip install flake8 yamllint
|
||||||
|
- sudo pip install s3cmd==1.6.1
|
||||||
|
# fog and ruby testing dependencies
|
||||||
|
- gem install fog-aws -v 1.3.0
|
||||||
|
- gem install mime-types -v 3.1
|
||||||
|
- gem install rspec -v 3.5
|
||||||
|
- gem install json
|
||||||
|
# java sdk dependencies
|
||||||
|
- sudo apt-get install -y -q default-jdk
|
||||||
|
|
||||||
|
|
||||||
|
test:
|
||||||
|
override:
|
||||||
|
- docker run --name squid-proxy -d --net=host
|
||||||
|
--publish 3128:3128 sameersbn/squid:3.3.8-23
|
||||||
|
- bash tests.bash:
|
||||||
|
parallel: true
|
|
@ -1,143 +0,0 @@
|
||||||
{
|
|
||||||
"port": 8000,
|
|
||||||
"listenOn": [],
|
|
||||||
"metricsPort": 8002,
|
|
||||||
"metricsListenOn": [],
|
|
||||||
"replicationGroupId": "RG001",
|
|
||||||
"workers": 4,
|
|
||||||
"restEndpoints": {
|
|
||||||
"localhost": "us-east-1",
|
|
||||||
"127.0.0.1": "us-east-1",
|
|
||||||
"cloudserver-front": "us-east-1",
|
|
||||||
"s3.docker.test": "us-east-1",
|
|
||||||
"127.0.0.2": "us-east-1",
|
|
||||||
"s3.amazonaws.com": "us-east-1",
|
|
||||||
"zenko-cloudserver-replicator": "us-east-1",
|
|
||||||
"lb": "us-east-1"
|
|
||||||
},
|
|
||||||
"websiteEndpoints": ["s3-website-us-east-1.amazonaws.com",
|
|
||||||
"s3-website.us-east-2.amazonaws.com",
|
|
||||||
"s3-website-us-west-1.amazonaws.com",
|
|
||||||
"s3-website-us-west-2.amazonaws.com",
|
|
||||||
"s3-website.ap-south-1.amazonaws.com",
|
|
||||||
"s3-website.ap-northeast-2.amazonaws.com",
|
|
||||||
"s3-website-ap-southeast-1.amazonaws.com",
|
|
||||||
"s3-website-ap-southeast-2.amazonaws.com",
|
|
||||||
"s3-website-ap-northeast-1.amazonaws.com",
|
|
||||||
"s3-website.eu-central-1.amazonaws.com",
|
|
||||||
"s3-website-eu-west-1.amazonaws.com",
|
|
||||||
"s3-website-sa-east-1.amazonaws.com",
|
|
||||||
"s3-website.localhost",
|
|
||||||
"s3-website.scality.test",
|
|
||||||
"zenkoazuretest.blob.core.windows.net"],
|
|
||||||
"replicationEndpoints": [{
|
|
||||||
"site": "zenko",
|
|
||||||
"servers": ["127.0.0.1:8000"],
|
|
||||||
"default": true
|
|
||||||
}, {
|
|
||||||
"site": "us-east-2",
|
|
||||||
"type": "aws_s3"
|
|
||||||
}],
|
|
||||||
"backbeat": {
|
|
||||||
"host": "localhost",
|
|
||||||
"port": 8900
|
|
||||||
},
|
|
||||||
"workflowEngineOperator": {
|
|
||||||
"host": "localhost",
|
|
||||||
"port": 3001
|
|
||||||
},
|
|
||||||
"cdmi": {
|
|
||||||
"host": "localhost",
|
|
||||||
"port": 81,
|
|
||||||
"path": "/dewpoint",
|
|
||||||
"readonly": true
|
|
||||||
},
|
|
||||||
"bucketd": {
|
|
||||||
"bootstrap": ["localhost:9000"]
|
|
||||||
},
|
|
||||||
"vaultd": {
|
|
||||||
"host": "localhost",
|
|
||||||
"port": 8500
|
|
||||||
},
|
|
||||||
"clusters": 1,
|
|
||||||
"log": {
|
|
||||||
"logLevel": "info",
|
|
||||||
"dumpLevel": "error"
|
|
||||||
},
|
|
||||||
"healthChecks": {
|
|
||||||
"allowFrom": ["127.0.0.1/8", "::1"]
|
|
||||||
},
|
|
||||||
"metadataClient": {
|
|
||||||
"host": "localhost",
|
|
||||||
"port": 9990
|
|
||||||
},
|
|
||||||
"dataClient": {
|
|
||||||
"host": "localhost",
|
|
||||||
"port": 9991
|
|
||||||
},
|
|
||||||
"pfsClient": {
|
|
||||||
"host": "localhost",
|
|
||||||
"port": 9992
|
|
||||||
},
|
|
||||||
"metadataDaemon": {
|
|
||||||
"bindAddress": "localhost",
|
|
||||||
"port": 9990
|
|
||||||
},
|
|
||||||
"dataDaemon": {
|
|
||||||
"bindAddress": "localhost",
|
|
||||||
"port": 9991
|
|
||||||
},
|
|
||||||
"pfsDaemon": {
|
|
||||||
"bindAddress": "localhost",
|
|
||||||
"port": 9992
|
|
||||||
},
|
|
||||||
"recordLog": {
|
|
||||||
"enabled": true,
|
|
||||||
"recordLogName": "s3-recordlog"
|
|
||||||
},
|
|
||||||
"mongodb": {
|
|
||||||
"replicaSetHosts": "localhost:27018,localhost:27019,localhost:27020",
|
|
||||||
"writeConcern": "majority",
|
|
||||||
"replicaSet": "rs0",
|
|
||||||
"readPreference": "primary",
|
|
||||||
"database": "metadata"
|
|
||||||
},
|
|
||||||
"authdata": "authdata.json",
|
|
||||||
"backends": {
|
|
||||||
"auth": "file",
|
|
||||||
"data": "file",
|
|
||||||
"metadata": "mongodb",
|
|
||||||
"kms": "file",
|
|
||||||
"quota": "none"
|
|
||||||
},
|
|
||||||
"externalBackends": {
|
|
||||||
"aws_s3": {
|
|
||||||
"httpAgent": {
|
|
||||||
"keepAlive": false,
|
|
||||||
"keepAliveMsecs": 1000,
|
|
||||||
"maxFreeSockets": 256,
|
|
||||||
"maxSockets": null
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"gcp": {
|
|
||||||
"httpAgent": {
|
|
||||||
"keepAlive": true,
|
|
||||||
"keepAliveMsecs": 1000,
|
|
||||||
"maxFreeSockets": 256,
|
|
||||||
"maxSockets": null
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"requests": {
|
|
||||||
"viaProxy": false,
|
|
||||||
"trustedProxyCIDRs": [],
|
|
||||||
"extractClientIPFromHeader": ""
|
|
||||||
},
|
|
||||||
"bucketNotificationDestinations": [
|
|
||||||
{
|
|
||||||
"resource": "target1",
|
|
||||||
"type": "dummy",
|
|
||||||
"host": "localhost:6000"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
|
@ -1,71 +0,0 @@
|
||||||
{
|
|
||||||
"port": 8000,
|
|
||||||
"listenOn": [],
|
|
||||||
"metricsPort": 8002,
|
|
||||||
"metricsListenOn": [],
|
|
||||||
"replicationGroupId": "RG001",
|
|
||||||
"restEndpoints": {
|
|
||||||
"localhost": "STANDARD",
|
|
||||||
"127.0.0.1": "STANDARD",
|
|
||||||
"yourhostname.ru": "STANDARD"
|
|
||||||
},
|
|
||||||
"websiteEndpoints": [
|
|
||||||
"static.yourhostname.ru"
|
|
||||||
],
|
|
||||||
"replicationEndpoints": [ {
|
|
||||||
"site": "zenko",
|
|
||||||
"servers": ["127.0.0.1:8000"],
|
|
||||||
"default": true
|
|
||||||
} ],
|
|
||||||
"log": {
|
|
||||||
"logLevel": "info",
|
|
||||||
"dumpLevel": "error"
|
|
||||||
},
|
|
||||||
"healthChecks": {
|
|
||||||
"allowFrom": ["127.0.0.1/8", "::1"]
|
|
||||||
},
|
|
||||||
"backends": {
|
|
||||||
"metadata": "mongodb"
|
|
||||||
},
|
|
||||||
"mongodb": {
|
|
||||||
"replicaSetHosts": "127.0.0.1:27017",
|
|
||||||
"writeConcern": "majority",
|
|
||||||
"replicaSet": "rs0",
|
|
||||||
"readPreference": "primary",
|
|
||||||
"database": "s3",
|
|
||||||
"authCredentials": {
|
|
||||||
"username": "s3",
|
|
||||||
"password": ""
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"externalBackends": {
|
|
||||||
"aws_s3": {
|
|
||||||
"httpAgent": {
|
|
||||||
"keepAlive": false,
|
|
||||||
"keepAliveMsecs": 1000,
|
|
||||||
"maxFreeSockets": 256,
|
|
||||||
"maxSockets": null
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"gcp": {
|
|
||||||
"httpAgent": {
|
|
||||||
"keepAlive": true,
|
|
||||||
"keepAliveMsecs": 1000,
|
|
||||||
"maxFreeSockets": 256,
|
|
||||||
"maxSockets": null
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"requests": {
|
|
||||||
"viaProxy": false,
|
|
||||||
"trustedProxyCIDRs": [],
|
|
||||||
"extractClientIPFromHeader": ""
|
|
||||||
},
|
|
||||||
"bucketNotificationDestinations": [
|
|
||||||
{
|
|
||||||
"resource": "target1",
|
|
||||||
"type": "dummy",
|
|
||||||
"host": "localhost:6000"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,23 @@
|
||||||
|
{
|
||||||
|
"accounts": [{
|
||||||
|
"name": "Bart",
|
||||||
|
"email": "sampleaccount1@sampling.com",
|
||||||
|
"arn": "arn:aws:iam::123456789012:root",
|
||||||
|
"canonicalID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be",
|
||||||
|
"shortid": "123456789012",
|
||||||
|
"keys": [{
|
||||||
|
"access": "accessKey1",
|
||||||
|
"secret": "verySecretKey1"
|
||||||
|
}]
|
||||||
|
}, {
|
||||||
|
"name": "Lisa",
|
||||||
|
"email": "sampleaccount2@sampling.com",
|
||||||
|
"arn": "arn:aws:iam::123456789013:root",
|
||||||
|
"canonicalID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2bf",
|
||||||
|
"shortid": "123456789013",
|
||||||
|
"keys": [{
|
||||||
|
"access": "accessKey2",
|
||||||
|
"secret": "verySecretKey2"
|
||||||
|
}]
|
||||||
|
}]
|
||||||
|
}
|
|
@ -23,8 +23,7 @@
|
||||||
"s3-website-eu-west-1.amazonaws.com",
|
"s3-website-eu-west-1.amazonaws.com",
|
||||||
"s3-website-sa-east-1.amazonaws.com",
|
"s3-website-sa-east-1.amazonaws.com",
|
||||||
"s3-website.localhost",
|
"s3-website.localhost",
|
||||||
"s3-website.scality.test",
|
"s3-website.scality.test"],
|
||||||
"zenkoazuretest.blob.core.windows.net"],
|
|
||||||
"replicationEndpoints": [{
|
"replicationEndpoints": [{
|
||||||
"site": "zenko",
|
"site": "zenko",
|
||||||
"servers": ["127.0.0.1:8000"],
|
"servers": ["127.0.0.1:8000"],
|
||||||
|
@ -40,7 +39,7 @@
|
||||||
"readonly": true
|
"readonly": true
|
||||||
},
|
},
|
||||||
"bucketd": {
|
"bucketd": {
|
||||||
"bootstrap": ["localhost"]
|
"bootstrap": ["localhost:9000"]
|
||||||
},
|
},
|
||||||
"vaultd": {
|
"vaultd": {
|
||||||
"host": "localhost",
|
"host": "localhost",
|
||||||
|
@ -75,52 +74,8 @@
|
||||||
"recordLogName": "s3-recordlog"
|
"recordLogName": "s3-recordlog"
|
||||||
},
|
},
|
||||||
"mongodb": {
|
"mongodb": {
|
||||||
"replicaSetHosts": "localhost:27017,localhost:27018,localhost:27019",
|
"host": "localhost",
|
||||||
"writeConcern": "majority",
|
"port": 27018,
|
||||||
"replicaSet": "rs0",
|
"database": "metadata"
|
||||||
"readPreference": "primary",
|
}
|
||||||
"database": "metadata"
|
|
||||||
},
|
|
||||||
"certFilePaths": {
|
|
||||||
"key": "tests/unit/testConfigs/allOptsConfig/key.txt",
|
|
||||||
"cert": "tests/unit/testConfigs/allOptsConfig/cert.txt",
|
|
||||||
"ca": "tests/unit/testConfigs/allOptsConfig/caBundle.txt"
|
|
||||||
},
|
|
||||||
"outboundProxy": {
|
|
||||||
"url": "http://test:8001",
|
|
||||||
"caBundle": "tests/unit/testConfigs/allOptsConfig/caBundle.txt",
|
|
||||||
"key": "tests/unit/testConfigs/allOptsConfig/key.txt",
|
|
||||||
"cert": "tests/unit/testConfigs/allOptsConfig/cert.txt"
|
|
||||||
},
|
|
||||||
"localCache": {
|
|
||||||
"name": "zenko",
|
|
||||||
"sentinels": "localhost:6379"
|
|
||||||
},
|
|
||||||
"redis": {
|
|
||||||
"name": "zenko",
|
|
||||||
"sentinels": "localhost:6379"
|
|
||||||
},
|
|
||||||
"scuba": {
|
|
||||||
"host": "localhost",
|
|
||||||
"port": 8100
|
|
||||||
},
|
|
||||||
"quota": {
|
|
||||||
"maxStalenessMS": 86400000
|
|
||||||
},
|
|
||||||
"utapi": {
|
|
||||||
"redis": {
|
|
||||||
"host": "localhost",
|
|
||||||
"port": 6379,
|
|
||||||
"retry": {
|
|
||||||
"connectBackoff": {
|
|
||||||
"min": 10,
|
|
||||||
"max": 1000,
|
|
||||||
"factor": 1.5,
|
|
||||||
"jitter": 0.1,
|
|
||||||
"deadline": 10000
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overlayVersion": 4
|
|
||||||
}
|
}
|
|
@ -1,112 +1,82 @@
|
||||||
{
|
{
|
||||||
"us-east-1": {
|
"us-east-1": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "us-east-1",
|
|
||||||
"legacyAwsBehavior": true,
|
"legacyAwsBehavior": true,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"us-east-2": {
|
"us-east-2": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "us-east-2",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"us-west-1": {
|
"us-west-1": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "us-west-1",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"us-west-2": {
|
"us-west-2": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "us-west-2",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"ca-central-1": {
|
"ca-central-1": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "ca-central-1",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"cn-north-1": {
|
"cn-north-1": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "cn-north-1",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"ap-south-1": {
|
"ap-south-1": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "ap-south-1",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"ap-northeast-1": {
|
"ap-northeast-1": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "ap-northeast-1",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"ap-northeast-2": {
|
"ap-northeast-2": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "ap-northeast-2",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"ap-southeast-1": {
|
"ap-southeast-1": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "ap-southeast-1",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"ap-southeast-2": {
|
"ap-southeast-2": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "ap-southeast-2",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"eu-central-1": {
|
"eu-central-1": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "eu-central-1",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"eu-west-1": {
|
"eu-west-1": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "eu-west-1",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"eu-west-2": {
|
"eu-west-2": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "eu-west-2",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"EU": {
|
"EU": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "EU",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
"sa-east-1": {
|
"sa-east-1": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"objectId": "sa-east-1",
|
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
|
||||||
"location-dmf-v1": {
|
|
||||||
"type": "dmf",
|
|
||||||
"objectId": "location-dmf-v1",
|
|
||||||
"legacyAwsBehavior": false,
|
|
||||||
"isCold": true,
|
|
||||||
"details": {}
|
|
||||||
},
|
|
||||||
"location-azure-archive-v1": {
|
|
||||||
"type": "azure_archive",
|
|
||||||
"objectId": "location-azure-archive-v1",
|
|
||||||
"legacyAwsBehavior": false,
|
|
||||||
"isCold": true,
|
|
||||||
"details": {}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
138
constants.js
138
constants.js
|
@ -39,8 +39,6 @@ const constants = {
|
||||||
// once the multipart upload is complete.
|
// once the multipart upload is complete.
|
||||||
mpuBucketPrefix: 'mpuShadowBucket',
|
mpuBucketPrefix: 'mpuShadowBucket',
|
||||||
blacklistedPrefixes: { bucket: [], object: [] },
|
blacklistedPrefixes: { bucket: [], object: [] },
|
||||||
// GCP Object Tagging Prefix
|
|
||||||
gcpTaggingPrefix: 'aws-tag-',
|
|
||||||
// PublicId is used as the canonicalID for a request that contains
|
// PublicId is used as the canonicalID for a request that contains
|
||||||
// no authentication information. Requestor can access
|
// no authentication information. Requestor can access
|
||||||
// only public resources
|
// only public resources
|
||||||
|
@ -66,71 +64,48 @@ const constants = {
|
||||||
// http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
|
// http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
|
||||||
minimumAllowedPartSize: 5242880,
|
minimumAllowedPartSize: 5242880,
|
||||||
|
|
||||||
// AWS sets a maximum total parts limit
|
|
||||||
// https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html
|
|
||||||
maximumAllowedPartCount: 10000,
|
|
||||||
|
|
||||||
gcpMaximumAllowedPartCount: 1024,
|
|
||||||
|
|
||||||
// Max size on put part or copy part is 5GB. For functional
|
// Max size on put part or copy part is 5GB. For functional
|
||||||
// testing use 110 MB as max
|
// testing use 110 MB as max
|
||||||
maximumAllowedPartSize: process.env.MPU_TESTING === 'yes' ? 110100480 :
|
maximumAllowedPartSize: process.env.MPU_TESTING === 'yes' ? 110100480 :
|
||||||
5368709120,
|
5368709120,
|
||||||
|
|
||||||
// Max size allowed in a single put object request is 5GB
|
|
||||||
// https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html
|
|
||||||
maximumAllowedUploadSize: 5368709120,
|
|
||||||
|
|
||||||
// AWS states max size for user-defined metadata (x-amz-meta- headers) is
|
// AWS states max size for user-defined metadata (x-amz-meta- headers) is
|
||||||
// 2 KB: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html
|
// 2 KB: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html
|
||||||
// In testing, AWS seems to allow up to 88 more bytes, so we do the same.
|
// In testing, AWS seems to allow up to 88 more bytes, so we do the same.
|
||||||
maximumMetaHeadersSize: 2136,
|
maximumMetaHeadersSize: 2136,
|
||||||
|
|
||||||
// Maximum HTTP headers size allowed
|
|
||||||
maxHttpHeadersSize: 14122,
|
|
||||||
|
|
||||||
// hex digest of sha256 hash of empty string:
|
// hex digest of sha256 hash of empty string:
|
||||||
emptyStringHash: crypto.createHash('sha256')
|
emptyStringHash: crypto.createHash('sha256')
|
||||||
.update('', 'binary').digest('hex'),
|
.update('', 'binary').digest('hex'),
|
||||||
|
|
||||||
// Queries supported by AWS that we do not currently support.
|
// Queries supported by AWS that we do not currently support.
|
||||||
// Non-bucket queries
|
|
||||||
unsupportedQueries: [
|
unsupportedQueries: [
|
||||||
'accelerate',
|
'accelerate',
|
||||||
'analytics',
|
'analytics',
|
||||||
'inventory',
|
'inventory',
|
||||||
'logging',
|
'logging',
|
||||||
'metrics',
|
'metrics',
|
||||||
'policyStatus',
|
'notification',
|
||||||
'publicAccessBlock',
|
'policy',
|
||||||
'requestPayment',
|
'requestPayment',
|
||||||
|
'restore',
|
||||||
'torrent',
|
'torrent',
|
||||||
],
|
],
|
||||||
|
|
||||||
// Headers supported by AWS that we do not currently support.
|
// Headers supported by AWS that we do not currently support.
|
||||||
unsupportedHeaders: [
|
unsupportedHeaders: [
|
||||||
|
'x-amz-server-side-encryption',
|
||||||
'x-amz-server-side-encryption-customer-algorithm',
|
'x-amz-server-side-encryption-customer-algorithm',
|
||||||
|
'x-amz-server-side-encryption-aws-kms-key-id',
|
||||||
'x-amz-server-side-encryption-context',
|
'x-amz-server-side-encryption-context',
|
||||||
'x-amz-server-side-encryption-customer-key',
|
'x-amz-server-side-encryption-customer-key',
|
||||||
'x-amz-server-side-encryption-customer-key-md5',
|
'x-amz-server-side-encryption-customer-key-md5',
|
||||||
],
|
],
|
||||||
|
|
||||||
// user metadata header to set object locationConstraint
|
// user metadata header to set object locationConstraint
|
||||||
objectLocationConstraintHeader: 'x-amz-storage-class',
|
objectLocationConstraintHeader: 'x-amz-meta-scal-location-constraint',
|
||||||
lastModifiedHeader: 'x-amz-meta-x-scal-last-modified',
|
|
||||||
legacyLocations: ['sproxyd', 'legacy'],
|
legacyLocations: ['sproxyd', 'legacy'],
|
||||||
// declare here all existing service accounts and their properties
|
|
||||||
// (if any, otherwise an empty object)
|
|
||||||
serviceAccountProperties: {
|
|
||||||
replication: {},
|
|
||||||
lifecycle: {},
|
|
||||||
gc: {},
|
|
||||||
'md-ingestion': {
|
|
||||||
canReplicate: true,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
/* eslint-disable camelcase */
|
/* eslint-disable camelcase */
|
||||||
externalBackends: { aws_s3: true, azure: true, gcp: true, pfs: true, dmf: true, azure_archive: true },
|
externalBackends: { aws_s3: true, azure: true, gcp: true },
|
||||||
// some of the available data backends (if called directly rather
|
// some of the available data backends (if called directly rather
|
||||||
// than through the multiple backend gateway) need a key provided
|
// than through the multiple backend gateway) need a key provided
|
||||||
// as a string as first parameter of the get/delete methods.
|
// as a string as first parameter of the get/delete methods.
|
||||||
|
@ -139,110 +114,15 @@ const constants = {
|
||||||
// for external backends, don't call unless at least 1 minute
|
// for external backends, don't call unless at least 1 minute
|
||||||
// (60,000 milliseconds) since last call
|
// (60,000 milliseconds) since last call
|
||||||
externalBackendHealthCheckInterval: 60000,
|
externalBackendHealthCheckInterval: 60000,
|
||||||
versioningNotImplBackends: { azure: true, gcp: true },
|
versioningNotImplBackends: { azure: true },
|
||||||
mpuMDStoredExternallyBackend: { aws_s3: true, gcp: true },
|
mpuMDStoredExternallyBackend: { aws_s3: true },
|
||||||
skipBatchDeleteBackends: { azure: true, gcp: true },
|
|
||||||
s3HandledBackends: { azure: true, gcp: true },
|
|
||||||
hasCopyPartBackends: { aws_s3: true, gcp: true },
|
|
||||||
/* eslint-enable camelcase */
|
/* eslint-enable camelcase */
|
||||||
mpuMDStoredOnS3Backend: { azure: true },
|
mpuMDStoredOnS3Backend: { azure: true },
|
||||||
azureAccountNameRegex: /^[a-z0-9]{3,24}$/,
|
azureAccountNameRegex: /^[a-z0-9]{3,24}$/,
|
||||||
base64Regex: new RegExp('^(?:[A-Za-z0-9+/]{4})*' +
|
base64Regex: new RegExp('^(?:[A-Za-z0-9+/]{4})*' +
|
||||||
'(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$'),
|
'(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$'),
|
||||||
productName: 'APN/1.0 Scality/1.0 Scality CloudServer for Zenko',
|
|
||||||
// location constraint delimiter
|
|
||||||
zenkoSeparator: ':',
|
|
||||||
// user metadata applied on zenko objects
|
// user metadata applied on zenko objects
|
||||||
zenkoIDHeader: 'x-amz-meta-zenko-instance-id',
|
zenkoIDHeader: 'x-amz-meta-zenko-instance-id',
|
||||||
bucketOwnerActions: [
|
|
||||||
'bucketDeleteCors',
|
|
||||||
'bucketDeleteLifecycle',
|
|
||||||
'bucketDeletePolicy',
|
|
||||||
'bucketDeleteReplication',
|
|
||||||
'bucketDeleteWebsite',
|
|
||||||
'bucketGetCors',
|
|
||||||
'bucketGetLifecycle',
|
|
||||||
'bucketGetLocation',
|
|
||||||
'bucketGetPolicy',
|
|
||||||
'bucketGetReplication',
|
|
||||||
'bucketGetVersioning',
|
|
||||||
'bucketGetWebsite',
|
|
||||||
'bucketPutCors',
|
|
||||||
'bucketPutLifecycle',
|
|
||||||
'bucketPutPolicy',
|
|
||||||
'bucketPutReplication',
|
|
||||||
'bucketPutVersioning',
|
|
||||||
'bucketPutWebsite',
|
|
||||||
'objectDeleteTagging',
|
|
||||||
'objectGetTagging',
|
|
||||||
'objectPutTagging',
|
|
||||||
'objectPutLegalHold',
|
|
||||||
'objectPutRetention',
|
|
||||||
],
|
|
||||||
// response header to be sent when there are invalid
|
|
||||||
// user metadata in the object's metadata
|
|
||||||
invalidObjectUserMetadataHeader: 'x-amz-missing-meta',
|
|
||||||
// Bucket specific queries supported by AWS that we do not currently support
|
|
||||||
// these queries may or may not be supported at object level
|
|
||||||
unsupportedBucketQueries: [
|
|
||||||
],
|
|
||||||
suppressedUtapiEventFields: [
|
|
||||||
'object',
|
|
||||||
'location',
|
|
||||||
'versionId',
|
|
||||||
],
|
|
||||||
allowedUtapiEventFilterFields: [
|
|
||||||
'operationId',
|
|
||||||
'location',
|
|
||||||
'account',
|
|
||||||
'user',
|
|
||||||
'bucket',
|
|
||||||
],
|
|
||||||
arrayOfAllowed: [
|
|
||||||
'objectPutTagging',
|
|
||||||
'objectPutLegalHold',
|
|
||||||
'objectPutRetention',
|
|
||||||
],
|
|
||||||
allowedUtapiEventFilterStates: ['allow', 'deny'],
|
|
||||||
allowedRestoreObjectRequestTierValues: ['Standard'],
|
|
||||||
lifecycleListing: {
|
|
||||||
CURRENT_TYPE: 'current',
|
|
||||||
NON_CURRENT_TYPE: 'noncurrent',
|
|
||||||
ORPHAN_DM_TYPE: 'orphan',
|
|
||||||
},
|
|
||||||
multiObjectDeleteConcurrency: 50,
|
|
||||||
maxScannedLifecycleListingEntries: 10000,
|
|
||||||
overheadField: [
|
|
||||||
'content-length',
|
|
||||||
'owner-id',
|
|
||||||
'versionId',
|
|
||||||
'isNull',
|
|
||||||
'isDeleteMarker',
|
|
||||||
],
|
|
||||||
unsupportedSignatureChecksums: new Set([
|
|
||||||
'STREAMING-UNSIGNED-PAYLOAD-TRAILER',
|
|
||||||
'STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER',
|
|
||||||
'STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD',
|
|
||||||
'STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER',
|
|
||||||
]),
|
|
||||||
supportedSignatureChecksums: new Set([
|
|
||||||
'UNSIGNED-PAYLOAD',
|
|
||||||
'STREAMING-AWS4-HMAC-SHA256-PAYLOAD',
|
|
||||||
]),
|
|
||||||
ipv4Regex: /^(\d{1,3}\.){3}\d{1,3}(\/(3[0-2]|[12]?\d))?$/,
|
|
||||||
ipv6Regex: /^([\da-f]{1,4}:){7}[\da-f]{1,4}$/i,
|
|
||||||
// The AWS assumed Role resource type
|
|
||||||
assumedRoleArnResourceType: 'assumed-role',
|
|
||||||
// Session name of the backbeat lifecycle assumed role session.
|
|
||||||
backbeatLifecycleSessionName: 'backbeat-lifecycle',
|
|
||||||
actionsToConsiderAsObjectPut: [
|
|
||||||
'initiateMultipartUpload',
|
|
||||||
'objectPutPart',
|
|
||||||
'completeMultipartUpload',
|
|
||||||
],
|
|
||||||
// if requester is not bucket owner, bucket policy actions should be denied with
|
|
||||||
// MethodNotAllowed error
|
|
||||||
onlyOwnerAllowed: ['bucketDeletePolicy', 'bucketGetPolicy', 'bucketPutPolicy'],
|
|
||||||
};
|
};
|
||||||
|
|
||||||
module.exports = constants;
|
module.exports = constants;
|
||||||
|
|
|
@ -79,7 +79,7 @@ function createEncryptedBucket() {
|
||||||
.option('-b, --bucket <bucket>', 'Name of the bucket')
|
.option('-b, --bucket <bucket>', 'Name of the bucket')
|
||||||
.option('-h, --host <host>', 'Host of the server')
|
.option('-h, --host <host>', 'Host of the server')
|
||||||
.option('-p, --port <port>', 'Port of the server')
|
.option('-p, --port <port>', 'Port of the server')
|
||||||
.option('-s, --ssl', 'Enable ssl')
|
.option('-s', '--ssl', 'Enable ssl')
|
||||||
.option('-v, --verbose')
|
.option('-v, --verbose')
|
||||||
.option('-l, --location-constraint <locationConstraint>',
|
.option('-l, --location-constraint <locationConstraint>',
|
||||||
'location Constraint')
|
'location Constraint')
|
|
@ -8,6 +8,19 @@ const inMemory = require('./in_memory/backend').backend;
|
||||||
const file = require('./file/backend');
|
const file = require('./file/backend');
|
||||||
const KMIPClient = require('arsenal').network.kmipClient;
|
const KMIPClient = require('arsenal').network.kmipClient;
|
||||||
const Common = require('./common');
|
const Common = require('./common');
|
||||||
|
let scalityKMS;
|
||||||
|
let scalityKMSImpl;
|
||||||
|
try {
|
||||||
|
// eslint-disable-next-line import/no-unresolved
|
||||||
|
const ScalityKMS = require('scality-kms');
|
||||||
|
scalityKMS = new ScalityKMS(config.kms);
|
||||||
|
scalityKMSImpl = 'scalityKms';
|
||||||
|
} catch (err) {
|
||||||
|
logger.warn('scality kms unavailable. ' +
|
||||||
|
'Using file kms backend unless mem specified.');
|
||||||
|
scalityKMS = file;
|
||||||
|
scalityKMSImpl = 'fileKms';
|
||||||
|
}
|
||||||
|
|
||||||
let client;
|
let client;
|
||||||
let implName;
|
let implName;
|
||||||
|
@ -19,9 +32,8 @@ if (config.backends.kms === 'mem') {
|
||||||
client = file;
|
client = file;
|
||||||
implName = 'fileKms';
|
implName = 'fileKms';
|
||||||
} else if (config.backends.kms === 'scality') {
|
} else if (config.backends.kms === 'scality') {
|
||||||
const ScalityKMS = require('scality-kms');
|
client = scalityKMS;
|
||||||
client = new ScalityKMS(config.kms);
|
implName = scalityKMSImpl;
|
||||||
implName = 'scalityKms';
|
|
||||||
} else if (config.backends.kms === 'kmip') {
|
} else if (config.backends.kms === 'kmip') {
|
||||||
const kmipConfig = { kmip: config.kmip };
|
const kmipConfig = { kmip: config.kmip };
|
||||||
if (!kmipConfig.kmip) {
|
if (!kmipConfig.kmip) {
|
||||||
|
@ -57,45 +69,52 @@ class KMS {
|
||||||
/**
|
/**
|
||||||
*
|
*
|
||||||
* @param {string} bucketName - bucket name
|
* @param {string} bucketName - bucket name
|
||||||
* @param {object} sseConfig - SSE configuration
|
* @param {object} headers - request headers
|
||||||
* @param {object} log - logger object
|
* @param {object} log - logger object
|
||||||
* @param {function} cb - callback
|
* @param {function} cb - callback
|
||||||
* @returns {undefined}
|
* @returns {undefined}
|
||||||
* @callback called with (err, serverSideEncryptionInfo: object)
|
* @callback called with (err, serverSideEncryptionInfo: object)
|
||||||
*/
|
*/
|
||||||
static bucketLevelEncryption(bucketName, sseConfig, log, cb) {
|
static bucketLevelEncryption(bucketName, headers, log, cb) {
|
||||||
|
const sseAlgorithm = headers['x-amz-scal-server-side-encryption'];
|
||||||
|
const sseMasterKeyId =
|
||||||
|
headers['x-amz-scal-server-side-encryption-aws-kms-key-id'];
|
||||||
/*
|
/*
|
||||||
The purpose of bucket level encryption is so that the client does not
|
The purpose of bucket level encryption is so that the client does not
|
||||||
have to send appropriate headers to trigger encryption on each object
|
have to send appropriate headers to trigger encryption on each object
|
||||||
put in an "encrypted bucket". Customer provided keys are not
|
put in an "encrypted bucket". Customer provided keys are not
|
||||||
feasible in this system because we do not want to store this key
|
feasible in this system because we do not want to store this key
|
||||||
in the bucket metadata.
|
in the bucket metadata.
|
||||||
*/
|
*/
|
||||||
const { algorithm, configuredMasterKeyId, mandatory } = sseConfig;
|
if (sseAlgorithm === 'AES256' ||
|
||||||
const _mandatory = mandatory === true;
|
(sseAlgorithm === 'aws:kms' && sseMasterKeyId === undefined)) {
|
||||||
if (algorithm === 'AES256' || algorithm === 'aws:kms') {
|
this.createBucketKey(bucketName, log, (err, masterKeyId) => {
|
||||||
return this.createBucketKey(bucketName, log, (err, masterKeyId) => {
|
|
||||||
if (err) {
|
if (err) {
|
||||||
return cb(err);
|
cb(err);
|
||||||
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
const serverSideEncryptionInfo = {
|
const serverSideEncryptionInfo = {
|
||||||
cryptoScheme: 1,
|
cryptoScheme: 1,
|
||||||
algorithm,
|
algorithm: sseAlgorithm,
|
||||||
masterKeyId,
|
masterKeyId,
|
||||||
mandatory: _mandatory,
|
mandatory: true,
|
||||||
};
|
};
|
||||||
|
cb(null, serverSideEncryptionInfo);
|
||||||
if (algorithm === 'aws:kms' && configuredMasterKeyId) {
|
|
||||||
serverSideEncryptionInfo.configuredMasterKeyId = configuredMasterKeyId;
|
|
||||||
}
|
|
||||||
return cb(null, serverSideEncryptionInfo);
|
|
||||||
});
|
});
|
||||||
|
} else if (sseAlgorithm === 'aws:kms') {
|
||||||
|
const serverSideEncryptionInfo = {
|
||||||
|
cryptoScheme: 1,
|
||||||
|
algorithm: sseAlgorithm,
|
||||||
|
masterKeyId: sseMasterKeyId,
|
||||||
|
mandatory: true,
|
||||||
|
};
|
||||||
|
cb(null, serverSideEncryptionInfo);
|
||||||
|
} else {
|
||||||
|
/*
|
||||||
|
* no encryption
|
||||||
|
*/
|
||||||
|
cb(null, null);
|
||||||
}
|
}
|
||||||
/*
|
|
||||||
* no encryption
|
|
||||||
*/
|
|
||||||
return cb(null, null);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
@ -150,18 +169,9 @@ class KMS {
|
||||||
static createCipherBundle(serverSideEncryptionInfo,
|
static createCipherBundle(serverSideEncryptionInfo,
|
||||||
log, cb) {
|
log, cb) {
|
||||||
const dataKey = this.createDataKey(log);
|
const dataKey = this.createDataKey(log);
|
||||||
|
|
||||||
const { algorithm, configuredMasterKeyId, masterKeyId: bucketMasterKeyId } = serverSideEncryptionInfo;
|
|
||||||
|
|
||||||
let masterKeyId = bucketMasterKeyId;
|
|
||||||
if (configuredMasterKeyId) {
|
|
||||||
log.debug('using user configured kms master key id');
|
|
||||||
masterKeyId = configuredMasterKeyId;
|
|
||||||
}
|
|
||||||
|
|
||||||
const cipherBundle = {
|
const cipherBundle = {
|
||||||
algorithm,
|
algorithm: serverSideEncryptionInfo.algorithm,
|
||||||
masterKeyId,
|
masterKeyId: serverSideEncryptionInfo.masterKeyId,
|
||||||
cryptoScheme: 1,
|
cryptoScheme: 1,
|
||||||
cipheredDataKey: null,
|
cipheredDataKey: null,
|
||||||
cipher: null,
|
cipher: null,
|
||||||
|
@ -171,7 +181,7 @@ class KMS {
|
||||||
function cipherDataKey(next) {
|
function cipherDataKey(next) {
|
||||||
log.debug('ciphering a data key');
|
log.debug('ciphering a data key');
|
||||||
return client.cipherDataKey(cipherBundle.cryptoScheme,
|
return client.cipherDataKey(cipherBundle.cryptoScheme,
|
||||||
cipherBundle.masterKeyId,
|
serverSideEncryptionInfo.masterKeyId,
|
||||||
dataKey, log, (err, cipheredDataKey) => {
|
dataKey, log, (err, cipheredDataKey) => {
|
||||||
if (err) {
|
if (err) {
|
||||||
log.debug('error from kms',
|
log.debug('error from kms',
|
||||||
|
@ -286,29 +296,6 @@ class KMS {
|
||||||
return cb(err, decipherBundle);
|
return cb(err, decipherBundle);
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
static checkHealth(log, cb) {
|
|
||||||
if (!client.healthcheck) {
|
|
||||||
return cb(null, {
|
|
||||||
[implName]: { code: 200, message: 'OK' },
|
|
||||||
});
|
|
||||||
}
|
|
||||||
return client.healthcheck(log, err => {
|
|
||||||
const respBody = {};
|
|
||||||
if (err) {
|
|
||||||
respBody[implName] = {
|
|
||||||
error: err.description,
|
|
||||||
code: err.code,
|
|
||||||
};
|
|
||||||
} else {
|
|
||||||
respBody[implName] = {
|
|
||||||
code: 200,
|
|
||||||
message: 'OK',
|
|
||||||
};
|
|
||||||
}
|
|
||||||
return cb(null, respBody);
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
module.exports = KMS;
|
module.exports = KMS;
|
|
@ -0,0 +1,18 @@
|
||||||
|
const utapiServer = require('utapi').UtapiServer;
|
||||||
|
const _config = require('../Config').config;
|
||||||
|
|
||||||
|
// start utapi server
|
||||||
|
if (_config.utapi) {
|
||||||
|
const fullConfig = Object.assign({}, _config.utapi);
|
||||||
|
if (_config.vaultd) {
|
||||||
|
Object.assign(fullConfig, { vaultd: _config.vaultd });
|
||||||
|
}
|
||||||
|
if (_config.https) {
|
||||||
|
Object.assign(fullConfig, { https: _config.https });
|
||||||
|
}
|
||||||
|
// copy healthcheck IPs
|
||||||
|
if (_config.healthChecks) {
|
||||||
|
Object.assign(fullConfig, { healthChecks: _config.healthChecks });
|
||||||
|
}
|
||||||
|
utapiServer(fullConfig);
|
||||||
|
}
|
|
@ -0,0 +1,6 @@
|
||||||
|
const UtapiReplay = require('utapi').UtapiReplay;
|
||||||
|
const _config = require('../Config').config;
|
||||||
|
|
||||||
|
// start utapi server
|
||||||
|
const replay = new UtapiReplay(_config.utapi);
|
||||||
|
replay.start();
|
|
@ -2,72 +2,23 @@ const http = require('http');
|
||||||
const https = require('https');
|
const https = require('https');
|
||||||
const commander = require('commander');
|
const commander = require('commander');
|
||||||
const { auth } = require('arsenal');
|
const { auth } = require('arsenal');
|
||||||
const { UtapiClient, utapiVersion } = require('utapi');
|
const { UtapiClient } = require('utapi');
|
||||||
const logger = require('../utilities/logger');
|
const logger = require('../utilities/logger');
|
||||||
const _config = require('../Config').config;
|
const _config = require('../Config').config;
|
||||||
const { suppressedUtapiEventFields: suppressedEventFields } = require('../../constants');
|
|
||||||
// setup utapi client
|
// setup utapi client
|
||||||
let utapiConfig;
|
const utapi = new UtapiClient(_config.utapi);
|
||||||
|
|
||||||
if (utapiVersion === 1 && _config.utapi) {
|
|
||||||
if (_config.utapi.redis === undefined) {
|
|
||||||
utapiConfig = Object.assign({}, _config.utapi, { redis: _config.redis });
|
|
||||||
} else {
|
|
||||||
utapiConfig = Object.assign({}, _config.utapi);
|
|
||||||
}
|
|
||||||
} else if (utapiVersion === 2) {
|
|
||||||
utapiConfig = Object.assign({
|
|
||||||
tls: _config.https,
|
|
||||||
suppressedEventFields,
|
|
||||||
}, _config.utapi || {});
|
|
||||||
}
|
|
||||||
|
|
||||||
const utapi = new UtapiClient(utapiConfig);
|
|
||||||
|
|
||||||
const bucketOwnerMetrics = [
|
|
||||||
'completeMultipartUpload',
|
|
||||||
'multiObjectDelete',
|
|
||||||
'abortMultipartUpload',
|
|
||||||
'copyObject',
|
|
||||||
'deleteObject',
|
|
||||||
'putObject',
|
|
||||||
'uploadPartCopy',
|
|
||||||
'uploadPart',
|
|
||||||
];
|
|
||||||
|
|
||||||
function evalAuthInfo(authInfo, canonicalID, action) {
|
|
||||||
let accountId = authInfo.getCanonicalID();
|
|
||||||
let userId = authInfo.isRequesterAnIAMUser() ?
|
|
||||||
authInfo.getShortid() : undefined;
|
|
||||||
// If action impacts 'numberOfObjectsStored' or 'storageUtilized' metric
|
|
||||||
// only the bucket owner account's metrics should be updated
|
|
||||||
const canonicalIdMatch = authInfo.getCanonicalID() === canonicalID;
|
|
||||||
if (bucketOwnerMetrics.includes(action) && !canonicalIdMatch) {
|
|
||||||
accountId = canonicalID;
|
|
||||||
userId = undefined;
|
|
||||||
}
|
|
||||||
return {
|
|
||||||
accountId,
|
|
||||||
userId,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
function _listMetrics(host,
|
function _listMetrics(host,
|
||||||
port,
|
port,
|
||||||
metric,
|
metric,
|
||||||
metricType,
|
metricType,
|
||||||
timeRange,
|
timeRange,
|
||||||
accessKey,
|
accessKey,
|
||||||
secretKey,
|
secretKey,
|
||||||
verbose,
|
verbose,
|
||||||
recent,
|
recent,
|
||||||
ssl) {
|
ssl) {
|
||||||
const listAction = recent ? 'ListRecentMetrics' : 'ListMetrics';
|
const listAction = recent ? 'ListRecentMetrics' : 'ListMetrics';
|
||||||
// If recent listing, we do not provide `timeRange` in the request
|
|
||||||
const requestObj = recent
|
|
||||||
? { [metric]: metricType }
|
|
||||||
: { timeRange, [metric]: metricType };
|
|
||||||
const requestBody = JSON.stringify(requestObj);
|
|
||||||
const options = {
|
const options = {
|
||||||
host,
|
host,
|
||||||
port,
|
port,
|
||||||
|
@ -76,7 +27,6 @@ function _listMetrics(host,
|
||||||
headers: {
|
headers: {
|
||||||
'content-type': 'application/json',
|
'content-type': 'application/json',
|
||||||
'cache-control': 'no-cache',
|
'cache-control': 'no-cache',
|
||||||
'content-length': Buffer.byteLength(requestBody),
|
|
||||||
},
|
},
|
||||||
rejectUnauthorized: false,
|
rejectUnauthorized: false,
|
||||||
};
|
};
|
||||||
|
@ -114,8 +64,10 @@ function _listMetrics(host,
|
||||||
if (verbose) {
|
if (verbose) {
|
||||||
logger.info('request headers', { headers: request._headers });
|
logger.info('request headers', { headers: request._headers });
|
||||||
}
|
}
|
||||||
|
// If recent listing, we do not provide `timeRange` in the request
|
||||||
request.write(requestBody);
|
const requestObj = recent ? {} : { timeRange };
|
||||||
|
requestObj[metric] = metricType;
|
||||||
|
request.write(JSON.stringify(requestObj));
|
||||||
request.end();
|
request.end();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -138,7 +90,7 @@ function listMetrics(metricType) {
|
||||||
if (metricType === 'buckets') {
|
if (metricType === 'buckets') {
|
||||||
commander
|
commander
|
||||||
.option('-b, --buckets <buckets>', 'Name of bucket(s) with ' +
|
.option('-b, --buckets <buckets>', 'Name of bucket(s) with ' +
|
||||||
'a comma separator if more than one');
|
'a comma separator if more than one');
|
||||||
} else {
|
} else {
|
||||||
commander
|
commander
|
||||||
.option('-m, --metric <metric>', 'Metric type')
|
.option('-m, --metric <metric>', 'Metric type')
|
||||||
|
@ -193,19 +145,6 @@ function listMetrics(metricType) {
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
// The string `commander[metric]` is a comma-separated list of resources
|
|
||||||
// given by the user.
|
|
||||||
const resources = commander[metric].split(',');
|
|
||||||
|
|
||||||
// Validate passed accounts to remove any canonicalIDs
|
|
||||||
if (metric === 'accounts') {
|
|
||||||
const invalid = resources.filter(r => !/^\d{12}$/.test(r));
|
|
||||||
if (invalid.length > 0) {
|
|
||||||
logger.error(`Invalid account ID: ${invalid.join(', ')}`);
|
|
||||||
process.exit(1);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const timeRange = [];
|
const timeRange = [];
|
||||||
// If recent listing, we disregard any start or end option given
|
// If recent listing, we disregard any start or end option given
|
||||||
if (!recent) {
|
if (!recent) {
|
||||||
|
@ -228,7 +167,9 @@ function listMetrics(metricType) {
|
||||||
timeRange.push(numEnd);
|
timeRange.push(numEnd);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// The string `commander[metric]` is a comma-separated list of resources
|
||||||
|
// given by the user.
|
||||||
|
const resources = commander[metric].split(',');
|
||||||
_listMetrics(host, port, metric, resources, timeRange, accessKey, secretKey,
|
_listMetrics(host, port, metric, resources, timeRange, accessKey, secretKey,
|
||||||
verbose, recent, ssl);
|
verbose, recent, ssl);
|
||||||
}
|
}
|
||||||
|
@ -249,84 +190,13 @@ function listMetrics(metricType) {
|
||||||
* @param {number} [metricObj.newByteLength] - (optional) new object size
|
* @param {number} [metricObj.newByteLength] - (optional) new object size
|
||||||
* @param {number|null} [metricObj.oldByteLength] - (optional) old object size
|
* @param {number|null} [metricObj.oldByteLength] - (optional) old object size
|
||||||
* (obj. overwrites)
|
* (obj. overwrites)
|
||||||
* @param {number} [metricObj.numberOfObjects] - (optional) number of objects
|
* @param {number} [metricObj.numberOfObjects] - (optional) number of obects
|
||||||
* added/deleted
|
* added/deleted
|
||||||
* @param {boolean} [metricObject.isDelete] - (optional) Indicates whether this
|
* @return {function} - `utapi.pushMetric`
|
||||||
* is a delete operation
|
|
||||||
* @return {function | undefined} - `utapi.pushMetric` or undefined if the action is
|
|
||||||
* filtered out and not pushed to utapi.
|
|
||||||
*/
|
*/
|
||||||
function pushMetric(action, log, metricObj) {
|
function pushMetric(action, log, metricObj) {
|
||||||
const {
|
const { bucket, keys, byteLength, newByteLength,
|
||||||
bucket,
|
oldByteLength, numberOfObjects, authInfo, canonicalID } = metricObj;
|
||||||
keys,
|
|
||||||
versionId,
|
|
||||||
byteLength,
|
|
||||||
newByteLength,
|
|
||||||
oldByteLength,
|
|
||||||
numberOfObjects,
|
|
||||||
authInfo,
|
|
||||||
canonicalID,
|
|
||||||
location,
|
|
||||||
isDelete,
|
|
||||||
removedDeleteMarkers,
|
|
||||||
} = metricObj;
|
|
||||||
|
|
||||||
if (utapiVersion === 2) {
|
|
||||||
const incomingBytes = action === 'getObject' ? 0 : newByteLength;
|
|
||||||
let sizeDelta = incomingBytes;
|
|
||||||
if (Number.isInteger(oldByteLength) && Number.isInteger(newByteLength)) {
|
|
||||||
sizeDelta = newByteLength - oldByteLength;
|
|
||||||
// Include oldByteLength in conditional so we don't end up with `-0`
|
|
||||||
} else if (action === 'completeMultipartUpload' && !versionId && oldByteLength) {
|
|
||||||
// If this is a non-versioned bucket we need to decrement
|
|
||||||
// the sizeDelta added by uploadPart when completeMPU is called.
|
|
||||||
sizeDelta = -oldByteLength;
|
|
||||||
} else if (action === 'abortMultipartUpload' && byteLength) {
|
|
||||||
sizeDelta = -byteLength;
|
|
||||||
} else if (action === 'putDeleteMarkerObject' && byteLength) {
|
|
||||||
sizeDelta = -byteLength;
|
|
||||||
}
|
|
||||||
|
|
||||||
let objectDelta = isDelete ? -numberOfObjects : numberOfObjects;
|
|
||||||
// putDeleteMarkerObject does not pass numberOfObjects
|
|
||||||
if ((action === 'putDeleteMarkerObject' && byteLength === null)
|
|
||||||
|| action === 'replicateDelete'
|
|
||||||
|| action === 'replicateObject') {
|
|
||||||
objectDelta = 1;
|
|
||||||
} else if (action === 'multiObjectDelete') {
|
|
||||||
objectDelta = -(numberOfObjects + removedDeleteMarkers);
|
|
||||||
}
|
|
||||||
|
|
||||||
const utapiObj = {
|
|
||||||
operationId: action,
|
|
||||||
bucket,
|
|
||||||
location,
|
|
||||||
objectDelta,
|
|
||||||
sizeDelta: isDelete ? -byteLength : sizeDelta,
|
|
||||||
incomingBytes,
|
|
||||||
outgoingBytes: action === 'getObject' ? newByteLength : 0,
|
|
||||||
};
|
|
||||||
|
|
||||||
// Any operation from lifecycle that does not change object count or size is dropped
|
|
||||||
const isLifecycle = _config.lifecycleRoleName
|
|
||||||
&& authInfo && authInfo.arn.endsWith(`:assumed-role/${_config.lifecycleRoleName}/backbeat-lifecycle`);
|
|
||||||
if (isLifecycle && !objectDelta && !sizeDelta) {
|
|
||||||
log.trace('ignoring pushMetric from lifecycle service user', { action, bucket, keys });
|
|
||||||
return undefined;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (keys && keys.length === 1) {
|
|
||||||
[utapiObj.object] = keys;
|
|
||||||
if (versionId) {
|
|
||||||
utapiObj.versionId = versionId;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
utapiObj.account = authInfo ? evalAuthInfo(authInfo, canonicalID, action).accountId : canonicalID;
|
|
||||||
utapiObj.user = authInfo ? evalAuthInfo(authInfo, canonicalID, action).userId : undefined;
|
|
||||||
return utapi.pushMetric(utapiObj);
|
|
||||||
}
|
|
||||||
const utapiObj = {
|
const utapiObj = {
|
||||||
bucket,
|
bucket,
|
||||||
keys,
|
keys,
|
||||||
|
@ -339,56 +209,16 @@ function pushMetric(action, log, metricObj) {
|
||||||
// account-level metrics and the shortId for user-level metrics. Otherwise
|
// account-level metrics and the shortId for user-level metrics. Otherwise
|
||||||
// check if the canonical ID is already provided for account-level metrics.
|
// check if the canonical ID is already provided for account-level metrics.
|
||||||
if (authInfo) {
|
if (authInfo) {
|
||||||
const { accountId, userId } = evalAuthInfo(authInfo, canonicalID, action);
|
utapiObj.accountId = authInfo.getCanonicalID();
|
||||||
utapiObj.accountId = accountId;
|
utapiObj.userId = authInfo.isRequesterAnIAMUser() ?
|
||||||
utapiObj.userId = userId;
|
authInfo.getShortid() : undefined;
|
||||||
} else if (canonicalID) {
|
} else if (canonicalID) {
|
||||||
utapiObj.accountId = canonicalID;
|
utapiObj.accountId = canonicalID;
|
||||||
}
|
}
|
||||||
return utapi.pushMetric(action, log.getSerializedUids(), utapiObj);
|
return utapi.pushMetric(action, log.getSerializedUids(), utapiObj);
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* internal: get the unique location ID from the location name
|
|
||||||
*
|
|
||||||
* @param {string} location - location name
|
|
||||||
* @return {string} - location unique ID
|
|
||||||
*/
|
|
||||||
function _getLocationId(location) {
|
|
||||||
return _config.locationConstraints[location].objectId;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Call the Utapi Client 'getLocationMetric' method with the
|
|
||||||
* associated parameters
|
|
||||||
* @param {string} location - name of data backend to list metric for
|
|
||||||
* @param {object} log - werelogs logger
|
|
||||||
* @param {function} cb - callback to call
|
|
||||||
* @return {function} - `utapi.getLocationMetric`
|
|
||||||
*/
|
|
||||||
function getLocationMetric(location, log, cb) {
|
|
||||||
const locationId = _getLocationId(location);
|
|
||||||
return utapi.getLocationMetric(locationId, log.getSerializedUids(), cb);
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Call the Utapi Client 'pushLocationMetric' method with the
|
|
||||||
* associated parameters
|
|
||||||
* @param {string} location - name of data backend
|
|
||||||
* @param {number} byteLength - number of bytes
|
|
||||||
* @param {object} log - werelogs logger
|
|
||||||
* @param {function} cb - callback to call
|
|
||||||
* @return {function} - `utapi.pushLocationMetric`
|
|
||||||
*/
|
|
||||||
function pushLocationMetric(location, byteLength, log, cb) {
|
|
||||||
const locationId = _getLocationId(location);
|
|
||||||
return utapi.pushLocationMetric(locationId, byteLength,
|
|
||||||
log.getSerializedUids(), cb);
|
|
||||||
}
|
|
||||||
|
|
||||||
module.exports = {
|
module.exports = {
|
||||||
listMetrics,
|
listMetrics,
|
||||||
pushMetric,
|
pushMetric,
|
||||||
getLocationMetric,
|
|
||||||
pushLocationMetric,
|
|
||||||
};
|
};
|
|
@ -1,220 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# set -e stops the execution of a script if a command or pipeline has an error
|
|
||||||
set -e
|
|
||||||
|
|
||||||
# modifying config.json
|
|
||||||
JQ_FILTERS_CONFIG="."
|
|
||||||
|
|
||||||
# ENDPOINT var can accept comma separated values
|
|
||||||
# for multiple endpoint locations
|
|
||||||
if [[ "$ENDPOINT" ]]; then
|
|
||||||
IFS="," read -ra HOST_NAMES <<< "$ENDPOINT"
|
|
||||||
for host in "${HOST_NAMES[@]}"; do
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .restEndpoints[\"$host\"]=\"us-east-1\""
|
|
||||||
done
|
|
||||||
echo "Host name has been modified to ${HOST_NAMES[@]}"
|
|
||||||
echo "Note: In your /etc/hosts file on Linux, OS X, or Unix with root permissions, make sure to associate 127.0.0.1 with ${HOST_NAMES[@]}"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$LOG_LEVEL" ]]; then
|
|
||||||
if [[ "$LOG_LEVEL" == "info" || "$LOG_LEVEL" == "debug" || "$LOG_LEVEL" == "trace" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .log.logLevel=\"$LOG_LEVEL\""
|
|
||||||
echo "Log level has been modified to $LOG_LEVEL"
|
|
||||||
else
|
|
||||||
echo "The log level you provided is incorrect (info/debug/trace)"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$SSL" && "$HOST_NAMES" ]]; then
|
|
||||||
# This condition makes sure that the certificates are not generated twice. (for docker restart)
|
|
||||||
if [ ! -f ./ca.key ] || [ ! -f ./ca.crt ] || [ ! -f ./server.key ] || [ ! -f ./server.crt ] ; then
|
|
||||||
# Compute config for utapi tests
|
|
||||||
cat >>req.cfg <<EOF
|
|
||||||
[req]
|
|
||||||
distinguished_name = req_distinguished_name
|
|
||||||
prompt = no
|
|
||||||
req_extensions = s3_req
|
|
||||||
|
|
||||||
[req_distinguished_name]
|
|
||||||
CN = ${HOST_NAMES[0]}
|
|
||||||
|
|
||||||
[s3_req]
|
|
||||||
subjectAltName = @alt_names
|
|
||||||
extendedKeyUsage = serverAuth, clientAuth
|
|
||||||
|
|
||||||
[alt_names]
|
|
||||||
DNS.1 = *.${HOST_NAMES[0]}
|
|
||||||
DNS.2 = ${HOST_NAMES[0]}
|
|
||||||
|
|
||||||
EOF
|
|
||||||
|
|
||||||
## Generate SSL key and certificates
|
|
||||||
# Generate a private key for your CSR
|
|
||||||
openssl genrsa -out ca.key 2048
|
|
||||||
# Generate a self signed certificate for your local Certificate Authority
|
|
||||||
openssl req -new -x509 -extensions v3_ca -key ca.key -out ca.crt -days 99999 -subj "/C=US/ST=Country/L=City/O=Organization/CN=S3 CA Server"
|
|
||||||
# Generate a key for S3 Server
|
|
||||||
openssl genrsa -out server.key 2048
|
|
||||||
# Generate a Certificate Signing Request for S3 Server
|
|
||||||
openssl req -new -key server.key -out server.csr -config req.cfg
|
|
||||||
# Generate a local-CA-signed certificate for S3 Server
|
|
||||||
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 99999 -sha256 -extfile req.cfg -extensions s3_req
|
|
||||||
fi
|
|
||||||
## Update S3Server config.json
|
|
||||||
# This condition makes sure that certFilePaths section is not added twice. (for docker restart)
|
|
||||||
if ! grep -q "certFilePaths" ./config.json; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .certFilePaths= { \"key\": \".\/server.key\", \"cert\": \".\/server.crt\", \"ca\": \".\/ca.crt\" }"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$LISTEN_ADDR" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .metadataDaemon.bindAddress=\"$LISTEN_ADDR\""
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .dataDaemon.bindAddress=\"$LISTEN_ADDR\""
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .pfsDaemon.bindAddress=\"$LISTEN_ADDR\""
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .listenOn=[\"$LISTEN_ADDR:8000\"]"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$REPLICATION_GROUP_ID" ]] ; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .replicationGroupId=\"$REPLICATION_GROUP_ID\""
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$DATA_HOST" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .dataClient.host=\"$DATA_HOST\""
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$METADATA_HOST" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .metadataClient.host=\"$METADATA_HOST\""
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$PFSD_HOST" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .pfsClient.host=\"$PFSD_HOST\""
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$MONGODB_HOSTS" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .mongodb.replicaSetHosts=\"$MONGODB_HOSTS\""
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$MONGODB_RS" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .mongodb.replicaSet=\"$MONGODB_RS\""
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$MONGODB_DATABASE" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .mongodb.database=\"$MONGODB_DATABASE\""
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ -z "$REDIS_HA_NAME" ]; then
|
|
||||||
REDIS_HA_NAME='mymaster'
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$REDIS_SENTINELS" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.name=\"$REDIS_HA_NAME\""
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.sentinels=\"$REDIS_SENTINELS\""
|
|
||||||
elif [[ "$REDIS_HOST" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.host=\"$REDIS_HOST\""
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.port=6379"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$REDIS_PORT" ]] && [[ ! "$REDIS_SENTINELS" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.port=$REDIS_PORT"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$REDIS_SENTINELS" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.name=\"$REDIS_HA_NAME\""
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.sentinels=\"$REDIS_SENTINELS\""
|
|
||||||
elif [[ "$REDIS_HA_HOST" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.host=\"$REDIS_HA_HOST\""
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.port=6379"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$REDIS_HA_PORT" ]] && [[ ! "$REDIS_SENTINELS" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.port=$REDIS_HA_PORT"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$RECORDLOG_ENABLED" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .recordLog.enabled=true"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$STORAGE_LIMIT_ENABLED" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .utapi.metrics[.utapi.metrics | length]=\"location\""
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$CRR_METRICS_HOST" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .backbeat.host=\"$CRR_METRICS_HOST\""
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$CRR_METRICS_PORT" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .backbeat.port=$CRR_METRICS_PORT"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$WE_OPERATOR_HOST" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .workflowEngineOperator.host=\"$WE_OPERATOR_HOST\""
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$WE_OPERATOR_PORT" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .workflowEngineOperator.port=$WE_OPERATOR_PORT"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$HEALTHCHECKS_ALLOWFROM" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .healthChecks.allowFrom=[\"$HEALTHCHECKS_ALLOWFROM\"]"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# external backends http(s) agent config
|
|
||||||
|
|
||||||
# AWS
|
|
||||||
if [[ "$AWS_S3_HTTPAGENT_KEEPALIVE" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.aws_s3.httpAgent.keepAlive=$AWS_S3_HTTPAGENT_KEEPALIVE"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$AWS_S3_HTTPAGENT_KEEPALIVE_MS" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.aws_s3.httpAgent.keepAliveMsecs=$AWS_S3_HTTPAGENT_KEEPALIVE_MS"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$AWS_S3_HTTPAGENT_KEEPALIVE_MAX_SOCKETS" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.aws_s3.httpAgent.maxSockets=$AWS_S3_HTTPAGENT_KEEPALIVE_MAX_SOCKETS"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$AWS_S3_HTTPAGENT_KEEPALIVE_MAX_FREE_SOCKETS" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.aws_s3.httpAgent.maxFreeSockets=$AWS_S3_HTTPAGENT_KEEPALIVE_MAX_FREE_SOCKETS"
|
|
||||||
fi
|
|
||||||
|
|
||||||
#GCP
|
|
||||||
if [[ "$GCP_HTTPAGENT_KEEPALIVE" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.gcp.httpAgent.keepAlive=$GCP_HTTPAGENT_KEEPALIVE"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$GCP_HTTPAGENT_KEEPALIVE_MS" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.gcp.httpAgent.keepAliveMsecs=$GCP_HTTPAGENT_KEEPALIVE_MS"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$GCP_HTTPAGENT_KEEPALIVE_MAX_SOCKETS" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.gcp.httpAgent.maxSockets=$GCP_HTTPAGENT_KEEPALIVE_MAX_SOCKETS"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$GCP_HTTPAGENT_KEEPALIVE_MAX_FREE_SOCKETS" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.gcp.httpAgent.maxFreeSockets=$GCP_HTTPAGENT_KEEPALIVE_MAX_FREE_SOCKETS"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -n "$BUCKET_DENY_FILTER" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .utapi.filter.deny.bucket=[\"$BUCKET_DENY_FILTER\"]"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$TESTING_MODE" ]]; then
|
|
||||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .testingMode=true"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ $JQ_FILTERS_CONFIG != "." ]]; then
|
|
||||||
jq "$JQ_FILTERS_CONFIG" config.json > config.json.tmp
|
|
||||||
mv config.json.tmp config.json
|
|
||||||
fi
|
|
||||||
|
|
||||||
if test -v INITIAL_INSTANCE_ID && test -v S3METADATAPATH && ! test -f ${S3METADATAPATH}/uuid ; then
|
|
||||||
echo -n ${INITIAL_INSTANCE_ID} > ${S3METADATAPATH}/uuid
|
|
||||||
fi
|
|
||||||
|
|
||||||
# s3 secret credentials for Zenko
|
|
||||||
if [ -r /run/secrets/s3-credentials ] ; then
|
|
||||||
. /run/secrets/s3-credentials
|
|
||||||
fi
|
|
||||||
|
|
||||||
exec "$@"
|
|
|
@ -0,0 +1,25 @@
|
||||||
|
FROM node:6-slim
|
||||||
|
MAINTAINER Giorgio Regni <gr@scality.com>
|
||||||
|
|
||||||
|
WORKDIR /usr/src/app
|
||||||
|
|
||||||
|
# Keep the .git directory in order to properly report version
|
||||||
|
COPY ./package.json .
|
||||||
|
|
||||||
|
RUN apt-get update \
|
||||||
|
&& apt-get install -y jq python git build-essential --no-install-recommends \
|
||||||
|
&& npm install --production \
|
||||||
|
&& apt-get autoremove --purge -y python git build-essential \
|
||||||
|
&& rm -rf /var/lib/apt/lists/* \
|
||||||
|
&& npm cache clear \
|
||||||
|
&& rm -rf ~/.node-gyp \
|
||||||
|
&& rm -rf /tmp/npm-*
|
||||||
|
|
||||||
|
COPY ./ ./
|
||||||
|
|
||||||
|
VOLUME ["/usr/src/app/localData","/usr/src/app/localMetadata"]
|
||||||
|
|
||||||
|
ENTRYPOINT ["/usr/src/app/docker-entrypoint.sh"]
|
||||||
|
CMD [ "npm", "start" ]
|
||||||
|
|
||||||
|
EXPOSE 8000
|
|
@ -7,16 +7,16 @@ COPY . /usr/src/app
|
||||||
|
|
||||||
RUN apt-get update \
|
RUN apt-get update \
|
||||||
&& apt-get install -y jq python git build-essential --no-install-recommends \
|
&& apt-get install -y jq python git build-essential --no-install-recommends \
|
||||||
&& yarn install --production \
|
&& npm install --production \
|
||||||
&& apt-get autoremove --purge -y python git build-essential \
|
&& apt-get autoremove --purge -y python git build-essential \
|
||||||
&& rm -rf /var/lib/apt/lists/* \
|
&& rm -rf /var/lib/apt/lists/* \
|
||||||
&& yarn cache clean \
|
&& npm cache clear \
|
||||||
&& rm -rf ~/.node-gyp \
|
&& rm -rf ~/.node-gyp \
|
||||||
&& rm -rf /tmp/yarn-*
|
&& rm -rf /tmp/npm-*
|
||||||
|
|
||||||
ENV S3BACKEND mem
|
ENV S3BACKEND mem
|
||||||
|
|
||||||
ENTRYPOINT ["/usr/src/app/docker-entrypoint.sh"]
|
ENTRYPOINT ["/usr/src/app/docker-entrypoint.sh"]
|
||||||
CMD [ "yarn", "start" ]
|
CMD [ "npm", "start" ]
|
||||||
|
|
||||||
EXPOSE 8000
|
EXPOSE 8000
|
|
@ -0,0 +1,107 @@
|
||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# set -e stops the execution of a script if a command or pipeline has an error
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# modifying config.json
|
||||||
|
JQ_FILTERS_CONFIG="."
|
||||||
|
|
||||||
|
if [[ "$ENDPOINT" ]]; then
|
||||||
|
HOST_NAME="$ENDPOINT"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$HOST_NAME" ]]; then
|
||||||
|
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .restEndpoints[\"$HOST_NAME\"]=\"us-east-1\""
|
||||||
|
echo "Host name has been modified to $HOST_NAME"
|
||||||
|
echo "Note: In your /etc/hosts file on Linux, OS X, or Unix with root permissions, make sure to associate 127.0.0.1 with $HOST_NAME"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$LOG_LEVEL" ]]; then
|
||||||
|
if [[ "$LOG_LEVEL" == "info" || "$LOG_LEVEL" == "debug" || "$LOG_LEVEL" == "trace" ]]; then
|
||||||
|
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .log.logLevel=\"$LOG_LEVEL\""
|
||||||
|
echo "Log level has been modified to $LOG_LEVEL"
|
||||||
|
else
|
||||||
|
echo "The log level you provided is incorrect (info/debug/trace)"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$SSL" && "$HOST_NAME" ]]; then
|
||||||
|
# This condition makes sure that the certificates are not generated twice. (for docker restart)
|
||||||
|
if [ ! -f ./ca.key ] || [ ! -f ./ca.crt ] || [ ! -f ./server.key ] || [ ! -f ./server.crt ] ; then
|
||||||
|
# Compute config for utapi tests
|
||||||
|
cat >>req.cfg <<EOF
|
||||||
|
[req]
|
||||||
|
distinguished_name = req_distinguished_name
|
||||||
|
prompt = no
|
||||||
|
req_extensions = s3_req
|
||||||
|
|
||||||
|
[req_distinguished_name]
|
||||||
|
CN = ${HOST_NAME}
|
||||||
|
|
||||||
|
[s3_req]
|
||||||
|
subjectAltName = @alt_names
|
||||||
|
extendedKeyUsage = serverAuth, clientAuth
|
||||||
|
|
||||||
|
[alt_names]
|
||||||
|
DNS.1 = *.${HOST_NAME}
|
||||||
|
DNS.2 = ${HOST_NAME}
|
||||||
|
|
||||||
|
EOF
|
||||||
|
|
||||||
|
## Generate SSL key and certificates
|
||||||
|
# Generate a private key for your CSR
|
||||||
|
openssl genrsa -out ca.key 2048
|
||||||
|
# Generate a self signed certificate for your local Certificate Authority
|
||||||
|
openssl req -new -x509 -extensions v3_ca -key ca.key -out ca.crt -days 99999 -subj "/C=US/ST=Country/L=City/O=Organization/CN=S3 CA Server"
|
||||||
|
# Generate a key for S3 Server
|
||||||
|
openssl genrsa -out server.key 2048
|
||||||
|
# Generate a Certificate Signing Request for S3 Server
|
||||||
|
openssl req -new -key server.key -out server.csr -config req.cfg
|
||||||
|
# Generate a local-CA-signed certificate for S3 Server
|
||||||
|
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 99999 -sha256 -extfile req.cfg -extensions s3_req
|
||||||
|
fi
|
||||||
|
## Update S3Server config.json
|
||||||
|
# This condition makes sure that certFilePaths section is not added twice. (for docker restart)
|
||||||
|
if ! grep -q "certFilePaths" ./config.json; then
|
||||||
|
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .certFilePaths= { \"key\": \".\/server.key\", \"cert\": \".\/server.crt\", \"ca\": \".\/ca.crt\" }"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$LISTEN_ADDR" ]]; then
|
||||||
|
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .metadataDaemon.bindAddress=\"$LISTEN_ADDR\""
|
||||||
|
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .dataDaemon.bindAddress=\"$LISTEN_ADDR\""
|
||||||
|
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .listenOn=[\"$LISTEN_ADDR:8000\"]"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$DATA_HOST" ]]; then
|
||||||
|
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .dataClient.host=\"$DATA_HOST\""
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$METADATA_HOST" ]]; then
|
||||||
|
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .metadataClient.host=\"$METADATA_HOST\""
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$REDIS_HOST" ]]; then
|
||||||
|
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.host=\"$REDIS_HOST\""
|
||||||
|
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.port=6379"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$REDIS_PORT" ]]; then
|
||||||
|
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.port=$REDIS_PORT"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$RECORDLOG_ENABLED" ]]; then
|
||||||
|
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .recordLog.enabled=true"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ $JQ_FILTERS_CONFIG != "." ]]; then
|
||||||
|
jq "$JQ_FILTERS_CONFIG" config.json > config.json.tmp
|
||||||
|
mv config.json.tmp config.json
|
||||||
|
fi
|
||||||
|
|
||||||
|
# s3 secret credentials for Zenko
|
||||||
|
if [ -r /run/secrets/s3-credentials ] ; then
|
||||||
|
. /run/secrets/s3-credentials
|
||||||
|
fi
|
||||||
|
|
||||||
|
exec "$@"
|
|
@ -27,7 +27,7 @@ including null versions and delete markers, described in the above
|
||||||
links.
|
links.
|
||||||
|
|
||||||
Implementation of Bucket Versioning in Zenko CloudServer
|
Implementation of Bucket Versioning in Zenko CloudServer
|
||||||
--------------------------------------------------------
|
-----------------------------------------
|
||||||
|
|
||||||
Overview of Metadata and API Component Roles
|
Overview of Metadata and API Component Roles
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
@ -66,7 +66,7 @@ The second section, `"Implementation of Bucket Versioning in
|
||||||
API" <#implementation-of-bucket-versioning-in-api>`__, describes the way
|
API" <#implementation-of-bucket-versioning-in-api>`__, describes the way
|
||||||
the metadata options are used in the API within S3 actions to create new
|
the metadata options are used in the API within S3 actions to create new
|
||||||
versions, update their metadata, and delete them. The management of null
|
versions, update their metadata, and delete them. The management of null
|
||||||
versions and creation of delete markers is also described in this
|
versions and creation of delete markers are also described in this
|
||||||
section.
|
section.
|
||||||
|
|
||||||
Implementation of Bucket Versioning in Metadata
|
Implementation of Bucket Versioning in Metadata
|
||||||
|
@ -179,13 +179,12 @@ PUT
|
||||||
the master version with this version.
|
the master version with this version.
|
||||||
- ``versionId: <versionId>`` create or update a specific version (for updating
|
- ``versionId: <versionId>`` create or update a specific version (for updating
|
||||||
version's ACL or tags, or remote updates in geo-replication)
|
version's ACL or tags, or remote updates in geo-replication)
|
||||||
|
- if the version identified by ``versionId`` happens to be the latest
|
||||||
* if the version identified by ``versionId`` happens to be the latest
|
|
||||||
version, the master version will be updated as well
|
version, the master version will be updated as well
|
||||||
* if the master version is not as recent as the version identified by
|
- if the master version is not as recent as the version identified by
|
||||||
``versionId``, as may happen with cross-region replication, the master
|
``versionId``, as may happen with cross-region replication, the master
|
||||||
will be updated as well
|
will be updated as well
|
||||||
* note that with ``versionId`` set to an empty string ``''``, it will
|
- note that with ``versionId`` set to an empty string ``''``, it will
|
||||||
overwrite the master version only (same as no options, but the master
|
overwrite the master version only (same as no options, but the master
|
||||||
version will have a ``versionId`` property set in its metadata like
|
version will have a ``versionId`` property set in its metadata like
|
||||||
any other version). The ``versionId`` will never be exposed to an
|
any other version). The ``versionId`` will never be exposed to an
|
||||||
|
@ -209,13 +208,10 @@ A deletion targeting the latest version of an object has to:
|
||||||
- delete the specified version identified by ``versionId``
|
- delete the specified version identified by ``versionId``
|
||||||
- replace the master version with a version that is a placeholder for
|
- replace the master version with a version that is a placeholder for
|
||||||
deletion
|
deletion
|
||||||
|
|
||||||
- this version contains a special keyword, 'isPHD', to indicate the
|
- this version contains a special keyword, 'isPHD', to indicate the
|
||||||
master version was deleted and needs to be updated
|
master version was deleted and needs to be updated
|
||||||
|
|
||||||
- initiate a repair operation to update the value of the master
|
- initiate a repair operation to update the value of the master
|
||||||
version:
|
version:
|
||||||
|
|
||||||
- involves listing the versions of the object and get the latest
|
- involves listing the versions of the object and get the latest
|
||||||
version to replace the placeholder delete version
|
version to replace the placeholder delete version
|
||||||
- if no more versions exist, metadata deletes the master version,
|
- if no more versions exist, metadata deletes the master version,
|
||||||
|
@ -746,7 +742,7 @@ Operation
|
||||||
Startup
|
Startup
|
||||||
~~~~~~~
|
~~~~~~~
|
||||||
|
|
||||||
The simplest deployment is still to launch with yarn start, this will
|
The simplest deployment is still to launch with npm start, this will
|
||||||
start one instance of the Zenko CloudServer connector and will listen on the
|
start one instance of the Zenko CloudServer connector and will listen on the
|
||||||
locally bound dmd ports 9990 and 9991 (by default, see below).
|
locally bound dmd ports 9990 and 9991 (by default, see below).
|
||||||
|
|
||||||
|
@ -755,22 +751,22 @@ command in the Zenko CloudServer directory:
|
||||||
|
|
||||||
::
|
::
|
||||||
|
|
||||||
yarn run start_dmd
|
npm run start_dmd
|
||||||
|
|
||||||
This will open two ports:
|
This will open two ports:
|
||||||
|
|
||||||
- one is based on socket.io and is used for metadata transfers (9990 by
|
- one is based on socket.io and is used for metadata transfers (9990 by
|
||||||
default)
|
default)
|
||||||
|
|
||||||
- the other is a REST interface used for data transfers (9991 by
|
- the other is a REST interface used for data transfers (9991 by
|
||||||
default)
|
default)
|
||||||
|
|
||||||
Then, one or more instances of Zenko CloudServer without the dmd can be started
|
Then, one or more instances of Zenko CloudServer without the dmd can be started
|
||||||
elsewhere with:
|
elsewhere with:
|
||||||
|
|
||||||
.. code:: sh
|
::
|
||||||
|
|
||||||
yarn run start_s3server
|
npm run start_s3server
|
||||||
|
|
||||||
Configuration
|
Configuration
|
||||||
~~~~~~~~~~~~~
|
~~~~~~~~~~~~~
|
||||||
|
@ -796,10 +792,10 @@ access:
|
||||||
|
|
||||||
To run a remote dmd, you have to do the following:
|
To run a remote dmd, you have to do the following:
|
||||||
|
|
||||||
- change both ``"host"`` attributes to the IP or host name where the
|
- change both ``"host"`` attributes to the IP or host name where the
|
||||||
dmd is run.
|
dmd is run.
|
||||||
|
|
||||||
- Modify the ``"bindAddress"`` attributes in ``"metadataDaemon"`` and
|
- Modify the ``"bindAddress"`` attributes in ``"metadataDaemon"`` and
|
||||||
``"dataDaemon"`` sections where the dmd is run to accept remote
|
``"dataDaemon"`` sections where the dmd is run to accept remote
|
||||||
connections (e.g. ``"::"``)
|
connections (e.g. ``"::"``)
|
||||||
|
|
||||||
|
@ -835,13 +831,13 @@ and ``createReadStream``. They more or less map the parameters accepted
|
||||||
by the corresponding calls in the LevelUp implementation of LevelDB.
|
by the corresponding calls in the LevelUp implementation of LevelDB.
|
||||||
They differ in the following:
|
They differ in the following:
|
||||||
|
|
||||||
- The ``sync`` option is ignored (under the hood, puts are gathered
|
- The ``sync`` option is ignored (under the hood, puts are gathered
|
||||||
into batches which have their ``sync`` property enforced when they
|
into batches which have their ``sync`` property enforced when they
|
||||||
are committed to the storage)
|
are committed to the storage)
|
||||||
|
|
||||||
- Some additional versioning-specific options are supported
|
- Some additional versioning-specific options are supported
|
||||||
|
|
||||||
- ``createReadStream`` becomes asynchronous, takes an additional
|
- ``createReadStream`` becomes asynchronous, takes an additional
|
||||||
callback argument and returns the stream in the second callback
|
callback argument and returns the stream in the second callback
|
||||||
parameter
|
parameter
|
||||||
|
|
||||||
|
@ -851,10 +847,10 @@ with ``DEBUG='socket.io*'`` environment variable set.
|
||||||
One parameter controls the timeout value after which RPC commands sent
|
One parameter controls the timeout value after which RPC commands sent
|
||||||
end with a timeout error, it can be changed either:
|
end with a timeout error, it can be changed either:
|
||||||
|
|
||||||
- via the ``DEFAULT_CALL_TIMEOUT_MS`` option in
|
- via the ``DEFAULT_CALL_TIMEOUT_MS`` option in
|
||||||
``lib/network/rpc/rpc.js``
|
``lib/network/rpc/rpc.js``
|
||||||
|
|
||||||
- or in the constructor call of the ``MetadataFileClient`` object (in
|
- or in the constructor call of the ``MetadataFileClient`` object (in
|
||||||
``lib/metadata/bucketfile/backend.js`` as ``callTimeoutMs``.
|
``lib/metadata/bucketfile/backend.js`` as ``callTimeoutMs``.
|
||||||
|
|
||||||
Default value is 30000.
|
Default value is 30000.
|
||||||
|
@ -868,10 +864,10 @@ can tune the behavior (for better throughput or getting it more robust
|
||||||
on weak networks), they have to be set in ``mdserver.js`` file directly,
|
on weak networks), they have to be set in ``mdserver.js`` file directly,
|
||||||
as there is no support in ``config.json`` for now for those options:
|
as there is no support in ``config.json`` for now for those options:
|
||||||
|
|
||||||
- ``streamMaxPendingAck``: max number of pending ack events not yet
|
- ``streamMaxPendingAck``: max number of pending ack events not yet
|
||||||
received (default is 5)
|
received (default is 5)
|
||||||
|
|
||||||
- ``streamAckTimeoutMs``: timeout for receiving an ack after an output
|
- ``streamAckTimeoutMs``: timeout for receiving an ack after an output
|
||||||
stream packet is sent to the client (default is 5000)
|
stream packet is sent to the client (default is 5000)
|
||||||
|
|
||||||
Data exchange through the REST data port
|
Data exchange through the REST data port
|
||||||
|
@ -922,17 +918,17 @@ Listing Types
|
||||||
We use three different types of metadata listing for various operations.
|
We use three different types of metadata listing for various operations.
|
||||||
Here are the scenarios we use each for:
|
Here are the scenarios we use each for:
|
||||||
|
|
||||||
- 'Delimiter' - when no versions are possible in the bucket since it is
|
- 'Delimiter' - when no versions are possible in the bucket since it is
|
||||||
an internally-used only bucket which is not exposed to a user.
|
an internally-used only bucket which is not exposed to a user.
|
||||||
Namely,
|
Namely,
|
||||||
|
|
||||||
1. to list objects in the "user's bucket" to respond to a GET SERVICE
|
1. to list objects in the "user's bucket" to respond to a GET SERVICE
|
||||||
request and
|
request and
|
||||||
2. to do internal listings on an MPU shadow bucket to complete multipart
|
2. to do internal listings on an MPU shadow bucket to complete multipart
|
||||||
upload operations.
|
upload operations.
|
||||||
|
|
||||||
- 'DelimiterVersion' - to list all versions in a bucket
|
- 'DelimiterVersion' - to list all versions in a bucket
|
||||||
- 'DelimiterMaster' - to list just the master versions of objects in a
|
- 'DelimiterMaster' - to list just the master versions of objects in a
|
||||||
bucket
|
bucket
|
||||||
|
|
||||||
Algorithms
|
Algorithms
|
||||||
|
|
|
@ -1,146 +0,0 @@
|
||||||
# Bucket Policy Documentation
|
|
||||||
|
|
||||||
## Description
|
|
||||||
|
|
||||||
Bucket policy is a method of controlling access to a user's account at the
|
|
||||||
resource level.
|
|
||||||
There are three associated APIs:
|
|
||||||
|
|
||||||
- PUT Bucket policy (see https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTpolicy.html)
|
|
||||||
- GET Bucket policy (see https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETpolicy.html)
|
|
||||||
- DELETE Bucket policy (see https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketDELETEpolicy.html)
|
|
||||||
|
|
||||||
More information on bucket policies in general can be found at
|
|
||||||
https://docs.aws.amazon.com/AmazonS3/latest/dev/using-iam-policies.html.
|
|
||||||
|
|
||||||
## Requirements
|
|
||||||
|
|
||||||
To prevent loss of access to a bucket, the root owner of a bucket will always
|
|
||||||
be able to perform any of the three bucket policy-related operations, even
|
|
||||||
if permission is explicitly denied.
|
|
||||||
All other users must have permission to perform the desired operation.
|
|
||||||
|
|
||||||
## Design
|
|
||||||
|
|
||||||
On a PUTBucketPolicy request, the user provides a policy in JSON format.
|
|
||||||
The policy is evaluated against our policy schema in Arsenal and, once
|
|
||||||
validated, is stored as part of the bucket's metadata.
|
|
||||||
On a GETBucketPolicy request, the policy is retrieved from the bucket's
|
|
||||||
metadata.
|
|
||||||
On a DELETEBucketPolicy request, the policy is deleted from the bucket's
|
|
||||||
metadata.
|
|
||||||
|
|
||||||
All other APIs are updated to check if a bucket policy is attached to the bucket
|
|
||||||
the request is made on. If there is a policy, user authorization to perform
|
|
||||||
the requested action is checked.
|
|
||||||
|
|
||||||
### Differences Between Bucket and IAM Policies
|
|
||||||
|
|
||||||
IAM policies are attached to an IAM identity and define what actions that
|
|
||||||
identity is allowed to or denied from doing on what resource.
|
|
||||||
Bucket policies attach only to buckets and define what actions are allowed or
|
|
||||||
denied for which principles on that bucket. Permissions specified in a bucket
|
|
||||||
policy apply to all objects in that bucket unless otherwise specified.
|
|
||||||
|
|
||||||
Besides their attachment origins, the main structural difference between
|
|
||||||
IAM policy and bucket policy is the requirement of a "Principal" element in
|
|
||||||
bucket policies. This field is redundant in IAM policies.
|
|
||||||
|
|
||||||
### Policy Validation
|
|
||||||
|
|
||||||
For general guidelines for bucket policy structure, see examples here:
|
|
||||||
https://docs.aws.amazon.com/AmazonS3/latest/dev//example-bucket-policies.html.
|
|
||||||
|
|
||||||
Each bucket policy statement object requires at least four keys:
|
|
||||||
"Effect", "Principle", "Resource", and "Action".
|
|
||||||
|
|
||||||
"Effect" defines the effect of the policy and can have a string value of either
|
|
||||||
"Allow" or "Deny".
|
|
||||||
"Resource" defines to which bucket or list of buckets a policy is attached.
|
|
||||||
An object within the bucket is also a valid resource. The element value can be
|
|
||||||
either a single bucket or object ARN string or an array of ARNs.
|
|
||||||
"Action" lists which action(s) the policy controls. Its value can also be either
|
|
||||||
a string or array of S3 APIs. Each action is the API name prepended by "s3:".
|
|
||||||
"Principle" specifies which user(s) are granted or denied access to the bucket
|
|
||||||
resource. Its value can be a string or an object containing an array of users.
|
|
||||||
Valid users can be identified with an account ARN, account id, or user ARN.
|
|
||||||
|
|
||||||
There are also two optional bucket policy statement keys: Sid and Condition.
|
|
||||||
|
|
||||||
"Sid" stands for "statement id". If this key is not included, one will be
|
|
||||||
generated for the statement.
|
|
||||||
"Condition" lists the condition under which a statement will take affect.
|
|
||||||
The possibilities are as follows:
|
|
||||||
|
|
||||||
- ArnEquals
|
|
||||||
- ArnEqualsIfExists
|
|
||||||
- ArnLike
|
|
||||||
- ArnLikeIfExists
|
|
||||||
- ArnNotEquals
|
|
||||||
- ArnNotEqualsIfExists
|
|
||||||
- ArnNotLike
|
|
||||||
- ArnNotLikeIfExists
|
|
||||||
- BinaryEquals
|
|
||||||
- BinaryEqualsIfExists
|
|
||||||
- BinaryNotEquals
|
|
||||||
- BinaryNotEqualsIfExists
|
|
||||||
- Bool
|
|
||||||
- BoolIfExists
|
|
||||||
- DateEquals
|
|
||||||
- DateEqualsIfExists
|
|
||||||
- DateGreaterThan
|
|
||||||
- DateGreaterThanEquals
|
|
||||||
- DateGreaterThanEqualsIfExists
|
|
||||||
- DateGreaterThanIfExists
|
|
||||||
- DateLessThan
|
|
||||||
- DateLessThanEquals
|
|
||||||
- DateLessThanEqualsIfExists
|
|
||||||
- DateLessThanIfExists
|
|
||||||
- DateNotEquals
|
|
||||||
- DateNotEqualsIfExists
|
|
||||||
- IpAddress
|
|
||||||
- IpAddressIfExists
|
|
||||||
- NotIpAddress
|
|
||||||
- NotIpAddressIfExists
|
|
||||||
- Null
|
|
||||||
- NumericEquals
|
|
||||||
- NumericEqualsIfExists
|
|
||||||
- NumericGreaterThan
|
|
||||||
- NumericGreaterThanEquals
|
|
||||||
- NumericGreaterThanEqualsIfExists
|
|
||||||
- NumericGreaterThanIfExists
|
|
||||||
- NumericLessThan
|
|
||||||
- NumericLessThanEquals
|
|
||||||
- NumericLessThanEqualsIfExists
|
|
||||||
- NumericLessThanIfExists
|
|
||||||
- NumericNotEquals
|
|
||||||
- NumericNotEqualsIfExists
|
|
||||||
- StringEquals
|
|
||||||
- StringEqualsIfExists
|
|
||||||
- StringEqualsIgnoreCase
|
|
||||||
- StringEqualsIgnoreCaseIfExists
|
|
||||||
- StringLike
|
|
||||||
- StringLikeIfExists
|
|
||||||
- StringNotEquals
|
|
||||||
- StringNotEqualsIfExists
|
|
||||||
- StringNotEqualsIgnoreCase
|
|
||||||
- StringNotEqualsIgnoreCaseIfExists
|
|
||||||
- StringNotLike
|
|
||||||
- StringNotLikeIfExists
|
|
||||||
|
|
||||||
The value of the Condition key will be an object containing the desired
|
|
||||||
condition name as that key. The value of inner object can be a string, boolean,
|
|
||||||
number, or object, depending on the condition.
|
|
||||||
|
|
||||||
## Authorization with Multiple Access Control Mechanisms
|
|
||||||
|
|
||||||
In the case where multiple access control mechanisms (such as IAM policies,
|
|
||||||
bucket policies, and ACLs) refer to the same resource, the principle of
|
|
||||||
least-privilege is applied. Unless an action is explicitly allowed, access will
|
|
||||||
by default be denied. An explicit DENY in any policy will trump another
|
|
||||||
policy's ALLOW for an action. The request will only be allowed if at least one
|
|
||||||
policy specifies an ALLOW, and there is no overriding DENY.
|
|
||||||
|
|
||||||
The following diagram illustrates this logic:
|
|
||||||
|
|
||||||
![Access_Control_Authorization_Chart](./images/access_control_authorization.png)
|
|
|
@ -178,7 +178,7 @@ Ruby
|
||||||
~~~~
|
~~~~
|
||||||
|
|
||||||
`AWS SDK for Ruby - Version 2 <http://docs.aws.amazon.com/sdkforruby/api/>`__
|
`AWS SDK for Ruby - Version 2 <http://docs.aws.amazon.com/sdkforruby/api/>`__
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
.. code:: ruby
|
.. code:: ruby
|
||||||
|
|
||||||
|
@ -239,7 +239,6 @@ Python
|
||||||
Client integration
|
Client integration
|
||||||
|
|
||||||
.. code:: python
|
.. code:: python
|
||||||
|
|
||||||
import boto3
|
import boto3
|
||||||
|
|
||||||
client = boto3.client(
|
client = boto3.client(
|
||||||
|
@ -254,7 +253,6 @@ Client integration
|
||||||
Full integration (with object mapping)
|
Full integration (with object mapping)
|
||||||
|
|
||||||
.. code:: python
|
.. code:: python
|
||||||
|
|
||||||
import os
|
import os
|
||||||
|
|
||||||
from botocore.utils import fix_s3_host
|
from botocore.utils import fix_s3_host
|
||||||
|
@ -295,51 +293,3 @@ Should force path-style requests even though v3 advertises it does by default.
|
||||||
$client->createBucket(array(
|
$client->createBucket(array(
|
||||||
'Bucket' => 'bucketphp',
|
'Bucket' => 'bucketphp',
|
||||||
));
|
));
|
||||||
|
|
||||||
Go
|
|
||||||
~~
|
|
||||||
|
|
||||||
`AWS Go SDK <https://github.com/aws/aws-sdk-go>`__
|
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
||||||
|
|
||||||
.. code:: go
|
|
||||||
|
|
||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"fmt"
|
|
||||||
"log"
|
|
||||||
"os"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/aws/aws-sdk-go/aws"
|
|
||||||
"github.com/aws/aws-sdk-go/aws/endpoints"
|
|
||||||
"github.com/aws/aws-sdk-go/aws/session"
|
|
||||||
"github.com/aws/aws-sdk-go/service/s3"
|
|
||||||
)
|
|
||||||
|
|
||||||
func main() {
|
|
||||||
os.Setenv("AWS_ACCESS_KEY_ID", "accessKey1")
|
|
||||||
os.Setenv("AWS_SECRET_ACCESS_KEY", "verySecretKey1")
|
|
||||||
endpoint := "http://localhost:8000"
|
|
||||||
timeout := time.Duration(10) * time.Second
|
|
||||||
sess := session.Must(session.NewSession())
|
|
||||||
|
|
||||||
// Create a context with a timeout that will abort the upload if it takes
|
|
||||||
// more than the passed in timeout.
|
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), timeout)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
svc := s3.New(sess, &aws.Config{
|
|
||||||
Region: aws.String(endpoints.UsEast1RegionID),
|
|
||||||
Endpoint: &endpoint,
|
|
||||||
})
|
|
||||||
|
|
||||||
out, err := svc.ListBucketsWithContext(ctx, &s3.ListBucketsInput{})
|
|
||||||
if err != nil {
|
|
||||||
log.Fatal(err)
|
|
||||||
} else {
|
|
||||||
fmt.Println(out)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
|
@ -14,7 +14,7 @@ Got an idea? Get started!
|
||||||
In order to contribute, please follow the `Contributing
|
In order to contribute, please follow the `Contributing
|
||||||
Guidelines <https://github.com/scality/Guidelines/blob/master/CONTRIBUTING.md>`__.
|
Guidelines <https://github.com/scality/Guidelines/blob/master/CONTRIBUTING.md>`__.
|
||||||
If anything is unclear to you, reach out to us on
|
If anything is unclear to you, reach out to us on
|
||||||
`forum <https://forum.zenko.io/>`__ or via a GitHub issue.
|
`slack <https://zenko-io.slack.com/>`__ or via a GitHub issue.
|
||||||
|
|
||||||
Don't write code? There are other ways to help!
|
Don't write code? There are other ways to help!
|
||||||
-----------------------------------------------
|
-----------------------------------------------
|
||||||
|
|
378
docs/DOCKER.rst
378
docs/DOCKER.rst
|
@ -1,7 +1,11 @@
|
||||||
Docker
|
Docker
|
||||||
======
|
======
|
||||||
|
|
||||||
.. _environment-variables:
|
- `Environment Variables <#environment-variables>`__
|
||||||
|
- `Tunables and setup tips <#tunables-and-setup-tips>`__
|
||||||
|
- `Examples for continuous integration with
|
||||||
|
Docker <#continuous-integration-with-docker-hosted CloudServer>`__
|
||||||
|
- `Examples for going in production with Docker <#in-production-with-docker-hosted CloudServer>`__
|
||||||
|
|
||||||
Environment Variables
|
Environment Variables
|
||||||
---------------------
|
---------------------
|
||||||
|
@ -11,27 +15,25 @@ S3DATA
|
||||||
|
|
||||||
S3DATA=multiple
|
S3DATA=multiple
|
||||||
^^^^^^^^^^^^^^^
|
^^^^^^^^^^^^^^^
|
||||||
|
Allows you to run Scality Zenko CloudServer with multiple data backends, defined
|
||||||
This variable enables running CloudServer with multiple data backends, defined
|
|
||||||
as regions.
|
as regions.
|
||||||
|
When using multiple data backends, a custom ``locationConfig.json`` file is
|
||||||
|
mandatory. It will allow you to set custom regions. You will then need to
|
||||||
|
provide associated rest_endpoints for each custom region in your
|
||||||
|
``config.json`` file.
|
||||||
|
`Learn more about multiple backends configuration <../GETTING_STARTED/#location-configuration>`__
|
||||||
|
|
||||||
For multiple data backends, a custom locationConfig.json file is required.
|
If you are using Scality RING endpoints, please refer to your customer
|
||||||
This file enables you to set custom regions. You must provide associated
|
documentation.
|
||||||
rest_endpoints for each custom region in config.json.
|
|
||||||
|
|
||||||
`Learn more about multiple-backend configurations <GETTING_STARTED.html#location-configuration>`__
|
Running it with an AWS S3 hosted backend
|
||||||
|
""""""""""""""""""""""""""""""""""""""""
|
||||||
If you are using Scality RING endpoints, refer to your customer documentation.
|
To run CloudServer with an S3 AWS backend, you will have to add a new section
|
||||||
|
to your ``locationConfig.json`` file with the ``aws_s3`` location type:
|
||||||
Running CloudServer with an AWS S3-Hosted Backend
|
|
||||||
"""""""""""""""""""""""""""""""""""""""""""""""""
|
|
||||||
|
|
||||||
To run CloudServer with an S3 AWS backend, add a new section to the
|
|
||||||
``locationConfig.json`` file with the ``aws_s3`` location type:
|
|
||||||
|
|
||||||
.. code:: json
|
.. code:: json
|
||||||
|
|
||||||
(...)
|
(...)
|
||||||
"awsbackend": {
|
"awsbackend": {
|
||||||
"type": "aws_s3",
|
"type": "aws_s3",
|
||||||
"details": {
|
"details": {
|
||||||
|
@ -41,139 +43,126 @@ To run CloudServer with an S3 AWS backend, add a new section to the
|
||||||
"credentialsProfile": "aws_hosted_profile"
|
"credentialsProfile": "aws_hosted_profile"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
(...)
|
(...)
|
||||||
|
|
||||||
Edit your AWS credentials file to enable your preferred command-line tool.
|
You will also have to edit your AWS credentials file to be able to use your
|
||||||
This file must mention credentials for all backends in use. You can use
|
command line tool of choice. This file should mention credentials for all the
|
||||||
several profiles if multiple profiles are configured.
|
backends you're using. You can use several profiles when using multiple
|
||||||
|
profiles.
|
||||||
|
|
||||||
.. code:: json
|
.. code:: json
|
||||||
|
|
||||||
[default]
|
[default]
|
||||||
aws_access_key_id=accessKey1
|
aws_access_key_id=accessKey1
|
||||||
aws_secret_access_key=verySecretKey1
|
aws_secret_access_key=verySecretKey1
|
||||||
[aws_hosted_profile]
|
[aws_hosted_profile]
|
||||||
aws_access_key_id={{YOUR_ACCESS_KEY}}
|
aws_access_key_id={{YOUR_ACCESS_KEY}}
|
||||||
aws_secret_access_key={{YOUR_SECRET_KEY}}
|
aws_secret_access_key={{YOUR_SECRET_KEY}}
|
||||||
|
|
||||||
As with locationConfig.json, the AWS credentials file must be mounted at
|
Just as you need to mount your locationConfig.json, you will need to mount your
|
||||||
run time: ``-v ~/.aws/credentials:/root/.aws/credentials`` on Unix-like
|
AWS credentials file at run time:
|
||||||
systems (Linux, OS X, etc.), or
|
``-v ~/.aws/credentials:/root/.aws/credentials`` on Linux, OS X, or Unix or
|
||||||
``-v C:\Users\USERNAME\.aws\credential:/root/.aws/credentials`` on Windows
|
``-v C:\Users\USERNAME\.aws\credential:/root/.aws/credentials`` on Windows
|
||||||
|
|
||||||
.. note:: One account cannot copy to another account with a source and
|
NOTE: One account can't copy to another account with a source and
|
||||||
destination on real AWS unless the account associated with the
|
destination on real AWS unless the account associated with the
|
||||||
accessKey/secretKey pairs used for the destination bucket has source
|
access Key/secret Key pairs used for the destination bucket has rights
|
||||||
bucket access privileges. To enable this, update ACLs directly on AWS.
|
to get in the source bucket. ACL's would have to be updated
|
||||||
|
on AWS directly to enable this.
|
||||||
|
|
||||||
S3BACKEND
|
S3BACKEND
|
||||||
~~~~~~~~~
|
~~~~~~
|
||||||
|
|
||||||
S3BACKEND=file
|
S3BACKEND=file
|
||||||
^^^^^^^^^^^^^^
|
^^^^^^^^^^^
|
||||||
|
When storing file data, for it to be persistent you must mount docker volumes
|
||||||
For stored file data to persist, you must mount Docker volumes
|
for both data and metadata. See `this section <#using-docker-volumes-in-production>`__
|
||||||
for both data and metadata. See :ref:`In Production with a Docker-Hosted CloudServer <in-production-w-a-Docker-hosted-cloudserver>`
|
|
||||||
|
|
||||||
S3BACKEND=mem
|
S3BACKEND=mem
|
||||||
^^^^^^^^^^^^^
|
^^^^^^^^^^
|
||||||
|
This is ideal for testing - no data will remain after container is shutdown.
|
||||||
This is ideal for testing: no data remains after the container is shut down.
|
|
||||||
|
|
||||||
ENDPOINT
|
ENDPOINT
|
||||||
~~~~~~~~
|
~~~~~~~~
|
||||||
|
|
||||||
This variable specifies the endpoint. To direct CloudServer requests to
|
This variable specifies your endpoint. If you have a domain such as
|
||||||
new.host.com, for example, specify the endpoint with:
|
new.host.com, by specifying that here, you and your users can direct s3
|
||||||
|
server requests to new.host.com.
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code:: shell
|
||||||
|
|
||||||
$ docker run -d --name cloudserver -p 8000:8000 -e ENDPOINT=new.host.com zenko/cloudserver
|
docker run -d --name s3server -p 8000:8000 -e ENDPOINT=new.host.com scality/s3server
|
||||||
|
|
||||||
.. note:: On Unix-like systems (Linux, OS X, etc.) edit /etc/hosts
|
Note: In your ``/etc/hosts`` file on Linux, OS X, or Unix with root
|
||||||
to associate 127.0.0.1 with new.host.com.
|
permissions, make sure to associate 127.0.0.1 with ``new.host.com``
|
||||||
|
|
||||||
REMOTE_MANAGEMENT_DISABLE
|
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
||||||
|
|
||||||
CloudServer is a part of `Zenko <https://www.zenko.io/>`__. When you run CloudServer standalone it will still try to connect to Orbit by default (browser-based graphical user interface for Zenko).
|
|
||||||
|
|
||||||
Setting this variable to true(1) will default to accessKey1 and verySecretKey1 for credentials and disable the automatic Orbit management:
|
|
||||||
|
|
||||||
.. code-block:: shell
|
|
||||||
|
|
||||||
$ docker run -d --name cloudserver -p 8000:8000 -e REMOTE_MANAGEMENT_DISABLE=1 zenko/cloudserver
|
|
||||||
|
|
||||||
SCALITY\_ACCESS\_KEY\_ID and SCALITY\_SECRET\_ACCESS\_KEY
|
SCALITY\_ACCESS\_KEY\_ID and SCALITY\_SECRET\_ACCESS\_KEY
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
These variables specify authentication credentials for an account named
|
These variables specify authentication credentials for an account named
|
||||||
“CustomAccount”.
|
"CustomAccount".
|
||||||
|
|
||||||
Set account credentials for multiple accounts by editing conf/authdata.json
|
You can set credentials for many accounts by editing
|
||||||
(see below for further details). To specify one set for personal use, set these
|
``conf/authdata.json`` (see below for further info), but if you just
|
||||||
environment variables:
|
want to specify one set of your own, you can use these environment
|
||||||
|
variables.
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code:: shell
|
||||||
|
|
||||||
$ docker run -d --name cloudserver -p 8000:8000 -e SCALITY_ACCESS_KEY_ID=newAccessKey \
|
docker run -d --name s3server -p 8000:8000 -e SCALITY_ACCESS_KEY_ID=newAccessKey
|
||||||
-e SCALITY_SECRET_ACCESS_KEY=newSecretKey zenko/cloudserver
|
-e SCALITY_SECRET_ACCESS_KEY=newSecretKey scality/s3server
|
||||||
|
|
||||||
.. note:: This takes precedence over the contents of the authdata.json
|
Note: Anything in the ``authdata.json`` file will be ignored. Note: The
|
||||||
file. The authdata.json file is ignored.
|
old ``ACCESS_KEY`` and ``SECRET_KEY`` environment variables are now
|
||||||
|
deprecated
|
||||||
.. note:: The ACCESS_KEY and SECRET_KEY environment variables are
|
|
||||||
deprecated.
|
|
||||||
|
|
||||||
LOG\_LEVEL
|
LOG\_LEVEL
|
||||||
~~~~~~~~~~
|
~~~~~~~~~~
|
||||||
|
|
||||||
This variable changes the log level. There are three levels: info, debug,
|
This variable allows you to change the log level: info, debug or trace.
|
||||||
and trace. The default is info. Debug provides more detailed logs, and trace
|
The default is info. Debug will give you more detailed logs and trace
|
||||||
provides the most detailed logs.
|
will give you the most detailed.
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code:: shell
|
||||||
|
|
||||||
$ docker run -d --name cloudserver -p 8000:8000 -e LOG_LEVEL=trace zenko/cloudserver
|
docker run -d --name s3server -p 8000:8000 -e LOG_LEVEL=trace scality/s3server
|
||||||
|
|
||||||
SSL
|
SSL
|
||||||
~~~
|
~~~
|
||||||
|
|
||||||
Set true, this variable runs CloudServer with SSL.
|
This variable set to true allows you to run S3 with SSL:
|
||||||
|
|
||||||
If SSL is set true:
|
**Note1**: You also need to specify the ENDPOINT environment variable.
|
||||||
|
**Note2**: In your ``/etc/hosts`` file on Linux, OS X, or Unix with root
|
||||||
|
permissions, make sure to associate 127.0.0.1 with ``<YOUR_ENDPOINT>``
|
||||||
|
|
||||||
* The ENDPOINT environment variable must also be specified.
|
**Warning**: These certs, being self-signed (and the CA being generated
|
||||||
|
inside the container) will be untrusted by any clients, and could
|
||||||
* On Unix-like systems (Linux, OS X, etc.), 127.0.0.1 must be associated with
|
disappear on a container upgrade. That's ok as long as it's for quick
|
||||||
<YOUR_ENDPOINT> in /etc/hosts.
|
testing. Also, best security practice for non-testing would be to use an
|
||||||
|
extra container to do SSL/TLS termination such as haproxy/nginx/stunnel
|
||||||
.. Warning:: Self-signed certs with a CA generated within the container are
|
to limit what an exploit on either component could expose, as well as
|
||||||
suitable for testing purposes only. Clients cannot trust them, and they may
|
certificates in a mounted volume
|
||||||
disappear altogether on a container upgrade. The best security practice for
|
|
||||||
production environments is to use an extra container, such as
|
|
||||||
haproxy/nginx/stunnel, for SSL/TLS termination and to pull certificates
|
|
||||||
from a mounted volume, limiting what an exploit on either component
|
|
||||||
can expose.
|
|
||||||
|
|
||||||
.. code:: shell
|
.. code:: shell
|
||||||
|
|
||||||
$ docker run -d --name cloudserver -p 8000:8000 -e SSL=TRUE -e ENDPOINT=<YOUR_ENDPOINT> \
|
docker run -d --name s3server -p 8000:8000 -e SSL=TRUE -e ENDPOINT=<YOUR_ENDPOINT>
|
||||||
zenko/cloudserver
|
scality/s3server
|
||||||
|
|
||||||
For more information about using ClousdServer with SSL, see `Using SSL <GETTING_STARTED.html#Using SSL>`__
|
More information about how to use S3 server with SSL
|
||||||
|
`here <https://s3.scality.com/v1.0/page/scality-with-ssl>`__
|
||||||
|
|
||||||
LISTEN\_ADDR
|
LISTEN\_ADDR
|
||||||
~~~~~~~~~~~~
|
~~~~~~~~~~~~
|
||||||
|
|
||||||
This variable causes CloudServer and its data and metadata components to
|
This variable instructs the Zenko CloudServer, and its data and metadata
|
||||||
listen on the specified address. This allows starting the data or metadata
|
components to listen on the specified address. This allows starting the data
|
||||||
servers as standalone services, for example.
|
or metadata servers as standalone services, for example.
|
||||||
|
|
||||||
.. code:: shell
|
.. code:: shell
|
||||||
|
|
||||||
docker run -d --name s3server-data -p 9991:9991 -e LISTEN_ADDR=0.0.0.0
|
docker run -d --name s3server-data -p 9991:9991 -e LISTEN_ADDR=0.0.0.0
|
||||||
scality/s3server yarn run start_dataserver
|
scality/s3server npm run start_dataserver
|
||||||
|
|
||||||
|
|
||||||
DATA\_HOST and METADATA\_HOST
|
DATA\_HOST and METADATA\_HOST
|
||||||
|
@ -185,8 +174,8 @@ Zenko CloudServer.
|
||||||
|
|
||||||
.. code:: shell
|
.. code:: shell
|
||||||
|
|
||||||
$ docker run -d --name cloudserver -e DATA_HOST=cloudserver-data \
|
docker run -d --name s3server -e DATA_HOST=s3server-data
|
||||||
-e METADATA_HOST=cloudserver-metadata zenko/cloudserver yarn run start_s3server
|
-e METADATA_HOST=s3server-metadata scality/s3server npm run start_s3server
|
||||||
|
|
||||||
REDIS\_HOST
|
REDIS\_HOST
|
||||||
~~~~~~~~~~~
|
~~~~~~~~~~~
|
||||||
|
@ -196,21 +185,19 @@ localhost.
|
||||||
|
|
||||||
.. code:: shell
|
.. code:: shell
|
||||||
|
|
||||||
$ docker run -d --name cloudserver -p 8000:8000 \
|
docker run -d --name s3server -p 8000:8000
|
||||||
-e REDIS_HOST=my-redis-server.example.com zenko/cloudserver
|
-e REDIS_HOST=my-redis-server.example.com scality/s3server
|
||||||
|
|
||||||
REDIS\_PORT
|
REDIS\_PORT
|
||||||
~~~~~~~~~~~
|
~~~~~~~~~~~
|
||||||
|
|
||||||
Use this variable to connect to the Redis cache server on a port other
|
Use this variable to connect to the redis cache server on another port than
|
||||||
than the default 6379.
|
the default 6379.
|
||||||
|
|
||||||
.. code:: shell
|
.. code:: shell
|
||||||
|
|
||||||
$ docker run -d --name cloudserver -p 8000:8000 \
|
docker run -d --name s3server -p 8000:8000
|
||||||
-e REDIS_PORT=6379 zenko/cloudserver
|
-e REDIS_PORT=6379 scality/s3server
|
||||||
|
|
||||||
.. _tunables-and-setup-tips:
|
|
||||||
|
|
||||||
Tunables and Setup Tips
|
Tunables and Setup Tips
|
||||||
-----------------------
|
-----------------------
|
||||||
|
@ -218,57 +205,60 @@ Tunables and Setup Tips
|
||||||
Using Docker Volumes
|
Using Docker Volumes
|
||||||
~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
CloudServer runs with a file backend by default, meaning that data is
|
Zenko CloudServer runs with a file backend by default.
|
||||||
stored inside the CloudServer’s Docker container.
|
|
||||||
|
|
||||||
For data and metadata to persist, data and metadata must be hosted in Docker
|
So, by default, the data is stored inside your Zenko CloudServer Docker
|
||||||
volumes outside the CloudServer’s Docker container. Otherwise, the data
|
container.
|
||||||
and metadata are destroyed when the container is erased.
|
|
||||||
|
|
||||||
.. code-block:: shell
|
However, if you want your data and metadata to persist, you **MUST** use
|
||||||
|
Docker volumes to host your data and metadata outside your Zenko CloudServer
|
||||||
|
Docker container. Otherwise, the data and metadata will be destroyed
|
||||||
|
when you erase the container.
|
||||||
|
|
||||||
$ docker run -v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata \
|
.. code:: shell
|
||||||
-p 8000:8000 -d zenko/cloudserver
|
|
||||||
|
|
||||||
This command mounts the ./data host directory to the container
|
docker run -v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata
|
||||||
at /usr/src/app/localData and the ./metadata host directory to
|
-p 8000:8000 -d scality/s3server
|
||||||
the container at /usr/src/app/localMetaData.
|
|
||||||
|
|
||||||
.. tip:: These host directories can be mounted to any accessible mount
|
This command mounts the host directory, ``./data``, into the container
|
||||||
point, such as /mnt/data and /mnt/metadata, for example.
|
at ``/usr/src/app/localData`` and the host directory, ``./metadata``, into
|
||||||
|
the container at ``/usr/src/app/localMetaData``. It can also be any host
|
||||||
|
mount point, like ``/mnt/data`` and ``/mnt/metadata``.
|
||||||
|
|
||||||
Adding, Modifying, or Deleting Accounts or Credentials
|
Adding modifying or deleting accounts or users credentials
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
1. Create a customized authdata.json file locally based on /conf/authdata.json.
|
1. Create locally a customized ``authdata.json`` based on our ``/conf/authdata.json``.
|
||||||
|
|
||||||
2. Use `Docker volumes <https://docs.docker.com/storage/volumes/>`__
|
|
||||||
to override the default ``authdata.json`` through a Docker file mapping.
|
|
||||||
|
|
||||||
|
2. Use `Docker
|
||||||
|
Volume <https://docs.docker.com/engine/tutorials/dockervolumes/>`__
|
||||||
|
to override the default ``authdata.json`` through a docker file mapping.
|
||||||
For example:
|
For example:
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code:: shell
|
||||||
|
|
||||||
$ docker run -v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json -p 8000:8000 -d \
|
docker run -v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json -p 8000:8000 -d
|
||||||
zenko/cloudserver
|
scality/s3server
|
||||||
|
|
||||||
Specifying a Host Name
|
Specifying your own host name
|
||||||
~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
To specify a host name (for example, s3.domain.name), provide your own
|
To specify a host name (e.g. s3.domain.name), you can provide your own
|
||||||
`config.json <https://github.com/scality/cloudserver/blob/master/config.json>`__
|
`config.json <https://github.com/scality/S3/blob/master/config.json>`__
|
||||||
file using `Docker volumes <https://docs.docker.com/storage/volumes/>`__.
|
using `Docker
|
||||||
|
Volume <https://docs.docker.com/engine/tutorials/dockervolumes/>`__.
|
||||||
|
|
||||||
First, add a new key-value pair to the restEndpoints section of your
|
First add a new key-value pair in the restEndpoints section of your
|
||||||
config.json. Make the key the host name you want, and the value the default
|
config.json. The key in the key-value pair should be the host name you
|
||||||
location\_constraint for this endpoint.
|
would like to add and the value is the default location\_constraint for
|
||||||
|
this endpoint.
|
||||||
|
|
||||||
For example, ``s3.example.com`` is mapped to ``us-east-1`` which is one
|
For example, ``s3.example.com`` is mapped to ``us-east-1`` which is one
|
||||||
of the ``location_constraints`` listed in your locationConfig.json file
|
of the ``location_constraints`` listed in your locationConfig.json file
|
||||||
`here <https://github.com/scality/S3/blob/master/locationConfig.json>`__.
|
`here <https://github.com/scality/S3/blob/master/locationConfig.json>`__.
|
||||||
|
|
||||||
For more information about location configuration, see:
|
More information about location configuration
|
||||||
`GETTING STARTED <GETTING_STARTED.html#location-configuration>`__
|
`here <https://github.com/scality/S3/blob/master/README.md#location-configuration>`__
|
||||||
|
|
||||||
.. code:: json
|
.. code:: json
|
||||||
|
|
||||||
|
@ -276,33 +266,33 @@ For more information about location configuration, see:
|
||||||
"localhost": "file",
|
"localhost": "file",
|
||||||
"127.0.0.1": "file",
|
"127.0.0.1": "file",
|
||||||
...
|
...
|
||||||
"cloudserver.example.com": "us-east-1"
|
"s3.example.com": "us-east-1"
|
||||||
},
|
},
|
||||||
|
|
||||||
Next, run CloudServer using a `Docker volume
|
Then, run your Scality S3 Server using `Docker
|
||||||
<https://docs.docker.com/engine/tutorials/dockervolumes/>`__:
|
Volume <https://docs.docker.com/engine/tutorials/dockervolumes/>`__:
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code:: shell
|
||||||
|
|
||||||
$ docker run -v $(pwd)/config.json:/usr/src/app/config.json -p 8000:8000 -d zenko/cloudserver
|
docker run -v $(pwd)/config.json:/usr/src/app/config.json -p 8000:8000 -d scality/s3server
|
||||||
|
|
||||||
The local ``config.json`` file overrides the default one through a Docker
|
Your local ``config.json`` file will override the default one through a
|
||||||
file mapping.
|
docker file mapping.
|
||||||
|
|
||||||
Running as an Unprivileged User
|
Running as an unprivileged user
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
CloudServer runs as root by default.
|
Zenko CloudServer runs as root by default.
|
||||||
|
|
||||||
To change this, modify the dockerfile and specify a user before the
|
You can change that by modifing the dockerfile and specifying a user
|
||||||
entry point.
|
before the entrypoint.
|
||||||
|
|
||||||
The user must exist within the container, and must own the
|
The user needs to exist within the container, and own the folder
|
||||||
/usr/src/app directory for CloudServer to run.
|
**/usr/src/app** for Scality Zenko CloudServer to run properly.
|
||||||
|
|
||||||
For example, the following dockerfile lines can be modified:
|
For instance, you can modify these lines in the dockerfile:
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code:: shell
|
||||||
|
|
||||||
...
|
...
|
||||||
&& groupadd -r -g 1001 scality \
|
&& groupadd -r -g 1001 scality \
|
||||||
|
@ -314,58 +304,54 @@ For example, the following dockerfile lines can be modified:
|
||||||
USER scality
|
USER scality
|
||||||
ENTRYPOINT ["/usr/src/app/docker-entrypoint.sh"]
|
ENTRYPOINT ["/usr/src/app/docker-entrypoint.sh"]
|
||||||
|
|
||||||
.. _continuous-integration-with-docker-hosted-cloudserver:
|
Continuous integration with Docker hosted CloudServer
|
||||||
|
-----------------------------------------------------
|
||||||
|
|
||||||
Continuous Integration with a Docker-Hosted CloudServer
|
When you start the Docker Scality Zenko CloudServer image, you can adjust the
|
||||||
-------------------------------------------------------
|
configuration of the Scality Zenko CloudServer instance by passing one or more
|
||||||
|
environment variables on the docker run command line.
|
||||||
|
|
||||||
When you start the Docker CloudServer image, you can adjust the
|
Sample ways to run it for CI are:
|
||||||
configuration of the CloudServer instance by passing one or more
|
|
||||||
environment variables on the ``docker run`` command line.
|
|
||||||
|
|
||||||
|
- With custom locations (one in-memory, one hosted on AWS), and custom
|
||||||
|
credentials mounted:
|
||||||
|
|
||||||
To run CloudServer for CI with custom locations (one in-memory,
|
.. code:: shell
|
||||||
one hosted on AWS), and custom credentials mounted:
|
|
||||||
|
|
||||||
.. code-block:: shell
|
docker run --name CloudServer -p 8000:8000
|
||||||
|
-v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json
|
||||||
|
-v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json
|
||||||
|
-v ~/.aws/credentials:/root/.aws/credentials
|
||||||
|
-e S3DATA=multiple -e S3BACKEND=mem scality/s3server
|
||||||
|
|
||||||
$ docker run --name CloudServer -p 8000:8000 \
|
- With custom locations, (one in-memory, one hosted on AWS, one file),
|
||||||
-v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json \
|
and custom credentials set as environment variables
|
||||||
-v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json \
|
(see `this section <#scality-access-key-id-and-scality-secret-access-key>`__):
|
||||||
-v ~/.aws/credentials:/root/.aws/credentials \
|
|
||||||
-e S3DATA=multiple -e S3BACKEND=mem zenko/cloudserver
|
|
||||||
|
|
||||||
To run CloudServer for CI with custom locations, (one in-memory, one
|
.. code:: shell
|
||||||
hosted on AWS, and one file), and custom credentials `set as environment
|
|
||||||
variables <GETTING_STARTED.html#scality-access-key-id-and-scality-secret-access-key>`__):
|
|
||||||
|
|
||||||
.. code-block:: shell
|
docker run --name CloudServer -p 8000:8000
|
||||||
|
-v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json
|
||||||
|
-v ~/.aws/credentials:/root/.aws/credentials
|
||||||
|
-v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata
|
||||||
|
-e SCALITY_ACCESS_KEY_ID=accessKey1
|
||||||
|
-e SCALITY_SECRET_ACCESS_KEY=verySecretKey1
|
||||||
|
-e S3DATA=multiple -e S3BACKEND=mem scality/s3server
|
||||||
|
|
||||||
$ docker run --name CloudServer -p 8000:8000 \
|
In production with Docker hosted CloudServer
|
||||||
-v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json \
|
--------------------------------------------
|
||||||
-v ~/.aws/credentials:/root/.aws/credentials \
|
|
||||||
-v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata \
|
|
||||||
-e SCALITY_ACCESS_KEY_ID=accessKey1 \
|
|
||||||
-e SCALITY_SECRET_ACCESS_KEY=verySecretKey1 \
|
|
||||||
-e S3DATA=multiple -e S3BACKEND=mem zenko/cloudserver
|
|
||||||
|
|
||||||
.. _in-production-w-a-Docker-hosted-cloudserver:
|
In production, we expect that data will be persistent, that you will use the
|
||||||
|
multiple backends capabilities of Zenko CloudServer, and that you will have a
|
||||||
|
custom endpoint for your local storage, and custom credentials for your local
|
||||||
|
storage:
|
||||||
|
|
||||||
In Production with a Docker-Hosted CloudServer
|
.. code:: shell
|
||||||
----------------------------------------------
|
|
||||||
|
|
||||||
Because data must persist in production settings, CloudServer offers
|
docker run -d --name CloudServer
|
||||||
multiple-backend capabilities. This requires a custom endpoint
|
-v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata
|
||||||
and custom credentials for local storage.
|
-v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json
|
||||||
|
-v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json
|
||||||
Customize these with:
|
-v ~/.aws/credentials:/root/.aws/credentials -e S3DATA=multiple
|
||||||
|
-e ENDPOINT=custom.endpoint.com
|
||||||
.. code-block:: shell
|
-p 8000:8000 -d scality/s3server
|
||||||
|
|
||||||
$ docker run -d --name CloudServer \
|
|
||||||
-v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata \
|
|
||||||
-v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json \
|
|
||||||
-v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json \
|
|
||||||
-v ~/.aws/credentials:/root/.aws/credentials -e S3DATA=multiple \
|
|
||||||
-e ENDPOINT=custom.endpoint.com \
|
|
||||||
-p 8000:8000 -d zenko/cloudserver \
|
|
||||||
|
|
|
@ -1,221 +1,214 @@
|
||||||
Getting Started
|
Getting Started
|
||||||
===============
|
=================
|
||||||
|
|
||||||
.. figure:: ../res/scality-cloudserver-logo.png
|
.. figure:: ../res/scality-cloudserver-logo.png
|
||||||
:alt: Zenko CloudServer logo
|
:alt: Zenko CloudServer logo
|
||||||
|
|
||||||
|
|CircleCI| |Scality CI|
|
||||||
Dependencies
|
|
||||||
------------
|
|
||||||
|
|
||||||
Building and running the Scality Zenko CloudServer requires node.js 10.x and
|
|
||||||
yarn v1.17.x. Up-to-date versions can be found at
|
|
||||||
`Nodesource <https://github.com/nodesource/distributions>`__.
|
|
||||||
|
|
||||||
Installation
|
Installation
|
||||||
------------
|
------------
|
||||||
|
|
||||||
1. Clone the source code
|
Dependencies
|
||||||
|
~~~~~~~~~~~~
|
||||||
|
|
||||||
.. code-block:: shell
|
Building and running the Scality Zenko CloudServer requires node.js 6.9.5 and
|
||||||
|
npm v3 . Up-to-date versions can be found at
|
||||||
|
`Nodesource <https://github.com/nodesource/distributions>`__.
|
||||||
|
|
||||||
$ git clone https://github.com/scality/cloudserver.git
|
Clone source code
|
||||||
|
~~~~~~~~~~~~~~~~~
|
||||||
2. Go to the cloudserver directory and use yarn to install the js dependencies.
|
|
||||||
|
|
||||||
.. code-block:: shell
|
|
||||||
|
|
||||||
$ cd cloudserver
|
|
||||||
$ yarn install
|
|
||||||
|
|
||||||
Running CloudServer with a File Backend
|
|
||||||
---------------------------------------
|
|
||||||
|
|
||||||
.. code-block:: shell
|
|
||||||
|
|
||||||
$ yarn start
|
|
||||||
|
|
||||||
This starts a Zenko CloudServer on port 8000. Two additional ports, 9990
|
|
||||||
and 9991, are also open locally for internal transfer of metadata and
|
|
||||||
data, respectively.
|
|
||||||
|
|
||||||
The default access key is accessKey1. The secret key is verySecretKey1.
|
|
||||||
|
|
||||||
By default, metadata files are saved in the localMetadata directory and
|
|
||||||
data files are saved in the localData directory in the local ./cloudserver
|
|
||||||
directory. These directories are pre-created within the repository. To
|
|
||||||
save data or metadata in different locations, you must specify them using
|
|
||||||
absolute paths. Thus, when starting the server:
|
|
||||||
|
|
||||||
.. code-block:: shell
|
|
||||||
|
|
||||||
$ mkdir -m 700 $(pwd)/myFavoriteDataPath
|
|
||||||
$ mkdir -m 700 $(pwd)/myFavoriteMetadataPath
|
|
||||||
$ export S3DATAPATH="$(pwd)/myFavoriteDataPath"
|
|
||||||
$ export S3METADATAPATH="$(pwd)/myFavoriteMetadataPath"
|
|
||||||
$ yarn start
|
|
||||||
|
|
||||||
Running CloudServer with Multiple Data Backends
|
|
||||||
-----------------------------------------------
|
|
||||||
|
|
||||||
.. code-block:: shell
|
|
||||||
|
|
||||||
$ export S3DATA='multiple'
|
|
||||||
$ yarn start
|
|
||||||
|
|
||||||
This starts a Zenko CloudServer on port 8000.
|
|
||||||
|
|
||||||
The default access key is accessKey1. The secret key is verySecretKey1.
|
|
||||||
|
|
||||||
With multiple backends, you can choose where each object is saved by setting
|
|
||||||
the following header with a location constraint in a PUT request:
|
|
||||||
|
|
||||||
.. code-block:: shell
|
|
||||||
|
|
||||||
'x-amz-meta-scal-location-constraint':'myLocationConstraint'
|
|
||||||
|
|
||||||
If no header is sent with a PUT object request, the bucket’s location
|
|
||||||
constraint determines where the data is saved. If the bucket has no
|
|
||||||
location constraint, the endpoint of the PUT request determines location.
|
|
||||||
|
|
||||||
See the Configuration_ section to set location constraints.
|
|
||||||
|
|
||||||
Run CloudServer with an In-Memory Backend
|
|
||||||
-----------------------------------------
|
|
||||||
|
|
||||||
.. code-block:: shell
|
|
||||||
|
|
||||||
$ yarn run mem_backend
|
|
||||||
|
|
||||||
This starts a Zenko CloudServer on port 8000.
|
|
||||||
|
|
||||||
The default access key is accessKey1. The secret key is verySecretKey1.
|
|
||||||
|
|
||||||
Run CloudServer with Vault User Management
|
|
||||||
------------------------------------------
|
|
||||||
|
|
||||||
.. code:: shell
|
.. code:: shell
|
||||||
|
|
||||||
export S3VAULT=vault
|
git clone https://github.com/scality/S3.git
|
||||||
yarn start
|
|
||||||
|
|
||||||
Note: Vault is proprietary and must be accessed separately.
|
Install js dependencies
|
||||||
This starts a Zenko CloudServer using Vault for user management.
|
~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
Run CloudServer for Continuous Integration Testing or in Production with Docker
|
Go to the ./S3 folder,
|
||||||
-------------------------------------------------------------------------------
|
|
||||||
|
|
||||||
Run Cloudserver with `DOCKER <DOCKER.html>`__
|
.. code:: shell
|
||||||
|
|
||||||
Testing
|
npm install
|
||||||
~~~~~~~
|
|
||||||
|
|
||||||
Run unit tests with the command:
|
Run it with a file backend
|
||||||
|
--------------------------
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code:: shell
|
||||||
|
|
||||||
$ yarn test
|
npm start
|
||||||
|
|
||||||
Run multiple-backend unit tests with:
|
This starts an Zenko CloudServer on port 8000. Two additional ports 9990 and
|
||||||
|
9991 are also open locally for internal transfer of metadata and data,
|
||||||
|
respectively.
|
||||||
|
|
||||||
.. code-block:: shell
|
The default access key is accessKey1 with a secret key of
|
||||||
|
verySecretKey1.
|
||||||
|
|
||||||
$ CI=true S3DATA=multiple yarn start
|
By default the metadata files will be saved in the localMetadata
|
||||||
$ yarn run multiple_backend_test
|
directory and the data files will be saved in the localData directory
|
||||||
|
within the ./S3 directory on your machine. These directories have been
|
||||||
|
pre-created within the repository. If you would like to save the data or
|
||||||
|
metadata in different locations of your choice, you must specify them
|
||||||
|
with absolute paths. So, when starting the server:
|
||||||
|
|
||||||
Run the linter with:
|
.. code:: shell
|
||||||
|
|
||||||
.. code-block:: shell
|
mkdir -m 700 $(pwd)/myFavoriteDataPath
|
||||||
|
mkdir -m 700 $(pwd)/myFavoriteMetadataPath
|
||||||
|
export S3DATAPATH="$(pwd)/myFavoriteDataPath"
|
||||||
|
export S3METADATAPATH="$(pwd)/myFavoriteMetadataPath"
|
||||||
|
npm start
|
||||||
|
|
||||||
$ yarn run lint
|
Run it with multiple data backends
|
||||||
|
----------------------------------
|
||||||
|
|
||||||
Running Functional Tests Locally
|
.. code:: shell
|
||||||
|
|
||||||
|
export S3DATA='multiple'
|
||||||
|
npm start
|
||||||
|
|
||||||
|
This starts an Zenko CloudServer on port 8000. The default access key is
|
||||||
|
accessKey1 with a secret key of verySecretKey1.
|
||||||
|
|
||||||
|
With multiple backends, you have the ability to choose where each object
|
||||||
|
will be saved by setting the following header with a locationConstraint
|
||||||
|
on a PUT request:
|
||||||
|
|
||||||
|
.. code:: shell
|
||||||
|
|
||||||
|
'x-amz-meta-scal-location-constraint':'myLocationConstraint'
|
||||||
|
|
||||||
|
If no header is sent with a PUT object request, the location constraint
|
||||||
|
of the bucket will determine where the data is saved. If the bucket has
|
||||||
|
no location constraint, the endpoint of the PUT request will be used to
|
||||||
|
determine location.
|
||||||
|
|
||||||
|
See the Configuration section below to learn how to set location
|
||||||
|
constraints.
|
||||||
|
|
||||||
|
Run it with an in-memory backend
|
||||||
--------------------------------
|
--------------------------------
|
||||||
|
|
||||||
To pass AWS and Azure backend tests locally, modify
|
.. code:: shell
|
||||||
tests/locationConfig/locationConfigTests.json so that ``awsbackend``
|
|
||||||
specifies the bucketname of a bucket you have access to based on your
|
npm run mem_backend
|
||||||
credentials, and modify ``azurebackend`` with details for your Azure account.
|
|
||||||
|
This starts an Zenko CloudServer on port 8000. The default access key is
|
||||||
|
accessKey1 with a secret key of verySecretKey1.
|
||||||
|
|
||||||
|
Run it for continuous integration testing or in production with Docker
|
||||||
|
----------------------------------------------------------------------
|
||||||
|
|
||||||
|
`DOCKER <../DOCKER/>`__
|
||||||
|
|
||||||
|
Testing
|
||||||
|
-------
|
||||||
|
|
||||||
|
You can run the unit tests with the following command:
|
||||||
|
|
||||||
|
.. code:: shell
|
||||||
|
|
||||||
|
npm test
|
||||||
|
|
||||||
|
You can run the multiple backend unit tests with:
|
||||||
|
|
||||||
|
.. code:: shell
|
||||||
|
CI=true S3DATA=multiple npm start
|
||||||
|
npm run multiple_backend_test
|
||||||
|
|
||||||
|
You can run the linter with:
|
||||||
|
|
||||||
|
.. code:: shell
|
||||||
|
|
||||||
|
npm run lint
|
||||||
|
|
||||||
|
Running functional tests locally:
|
||||||
|
|
||||||
|
For the AWS backend and Azure backend tests to pass locally,
|
||||||
|
you must modify tests/locationConfigTests.json so that awsbackend
|
||||||
|
specifies a bucketname of a bucket you have access to based on
|
||||||
|
your credentials profile and modify "azurebackend" with details
|
||||||
|
for your Azure account.
|
||||||
|
|
||||||
The test suite requires additional tools, **s3cmd** and **Redis**
|
The test suite requires additional tools, **s3cmd** and **Redis**
|
||||||
installed in the environment the tests are running in.
|
installed in the environment the tests are running in.
|
||||||
|
|
||||||
1. Install `s3cmd <http://s3tools.org/download>`__
|
- Install `s3cmd <http://s3tools.org/download>`__
|
||||||
|
- Install `redis <https://redis.io/download>`__ and start Redis.
|
||||||
|
- Add localCache section to your ``config.json``:
|
||||||
|
|
||||||
2. Install `redis <https://redis.io/download>`__ and start Redis.
|
::
|
||||||
|
|
||||||
3. Add localCache section to ``config.json``:
|
"localCache": {
|
||||||
|
|
||||||
.. code:: json
|
|
||||||
|
|
||||||
"localCache": {
|
|
||||||
"host": REDIS_HOST,
|
"host": REDIS_HOST,
|
||||||
"port": REDIS_PORT
|
"port": REDIS_PORT
|
||||||
}
|
}
|
||||||
|
|
||||||
where ``REDIS_HOST`` is the Redis instance IP address (``"127.0.0.1"``
|
where ``REDIS_HOST`` is your Redis instance IP address (``"127.0.0.1"``
|
||||||
if Redis is running locally) and ``REDIS_PORT`` is the Redis instance
|
if your Redis is running locally) and ``REDIS_PORT`` is your Redis
|
||||||
port (``6379`` by default)
|
instance port (``6379`` by default)
|
||||||
|
|
||||||
4. Add the following to the local etc/hosts file:
|
- Add the following to the etc/hosts file on your machine:
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code:: shell
|
||||||
|
|
||||||
127.0.0.1 bucketwebsitetester.s3-website-us-east-1.amazonaws.com
|
127.0.0.1 bucketwebsitetester.s3-website-us-east-1.amazonaws.com
|
||||||
|
|
||||||
5. Start Zenko CloudServer in memory and run the functional tests:
|
- Start the Zenko CloudServer in memory and run the functional tests:
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code:: shell
|
||||||
|
|
||||||
$ CI=true yarn run mem_backend
|
CI=true npm run mem_backend
|
||||||
$ CI=true yarn run ft_test
|
CI=true npm run ft_test
|
||||||
|
|
||||||
.. _Configuration:
|
|
||||||
|
|
||||||
Configuration
|
Configuration
|
||||||
-------------
|
-------------
|
||||||
|
|
||||||
There are three configuration files for Zenko CloudServer:
|
There are three configuration files for your Scality Zenko CloudServer:
|
||||||
|
|
||||||
* ``conf/authdata.json``, for authentication.
|
1. ``conf/authdata.json``, described above for authentication
|
||||||
|
|
||||||
* ``locationConfig.json``, to configure where data is saved.
|
2. ``locationConfig.json``, to set up configuration options for
|
||||||
|
|
||||||
* ``config.json``, for general configuration options.
|
where data will be saved
|
||||||
|
|
||||||
.. _location-configuration:
|
3. ``config.json``, for general configuration options
|
||||||
|
|
||||||
Location Configuration
|
Location Configuration
|
||||||
~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
You must specify at least one locationConstraint in locationConfig.json
|
You must specify at least one locationConstraint in your
|
||||||
(or leave it as pre-configured).
|
locationConfig.json (or leave as pre-configured).
|
||||||
|
|
||||||
You must also specify 'us-east-1' as a locationConstraint. If you put a
|
You must also specify 'us-east-1' as a locationConstraint so if you only
|
||||||
bucket to an unknown endpoint and do not specify a locationConstraint in
|
define one locationConstraint, that would be it. If you put a bucket to
|
||||||
the PUT bucket call, us-east-1 is used.
|
an unknown endpoint and do not specify a locationConstraint in the put
|
||||||
|
bucket call, us-east-1 will be used.
|
||||||
|
|
||||||
For instance, the following locationConstraint saves data sent to
|
For instance, the following locationConstraint will save data sent to
|
||||||
``myLocationConstraint`` to the file backend:
|
``myLocationConstraint`` to the file backend:
|
||||||
|
|
||||||
.. code:: json
|
.. code:: json
|
||||||
|
|
||||||
"myLocationConstraint": {
|
"myLocationConstraint": {
|
||||||
"type": "file",
|
"type": "file",
|
||||||
"legacyAwsBehavior": false,
|
"legacyAwsBehavior": false,
|
||||||
"details": {}
|
"details": {}
|
||||||
},
|
},
|
||||||
|
|
||||||
Each locationConstraint must include the ``type``, ``legacyAwsBehavior``,
|
Each locationConstraint must include the ``type``,
|
||||||
and ``details`` keys. ``type`` indicates which backend is used for that
|
``legacyAwsBehavior``, and ``details`` keys. ``type`` indicates which
|
||||||
region. Supported backends are mem, file, and scality.``legacyAwsBehavior``
|
backend will be used for that region. Currently, mem, file, and scality
|
||||||
indicates whether the region behaves the same as the AWS S3 'us-east-1'
|
are the supported backends. ``legacyAwsBehavior`` indicates whether the
|
||||||
region. If the locationConstraint type is ``scality``, ``details`` must
|
region will have the same behavior as the AWS S3 'us-east-1' region. If
|
||||||
contain connector information for sproxyd. If the locationConstraint type
|
the locationConstraint type is scality, ``details`` should contain
|
||||||
is ``mem`` or ``file``, ``details`` must be empty.
|
connector information for sproxyd. If the locationConstraint type is mem
|
||||||
|
or file, ``details`` should be empty.
|
||||||
|
|
||||||
Once locationConstraints is set in locationConfig.json, specify a default
|
Once you have your locationConstraints in your locationConfig.json, you
|
||||||
locationConstraint for each endpoint.
|
can specify a default locationConstraint for each of your endpoints.
|
||||||
|
|
||||||
For instance, the following sets the ``localhost`` endpoint to the
|
For instance, the following sets the ``localhost`` endpoint to the
|
||||||
``myLocationConstraint`` data backend defined above:
|
``myLocationConstraint`` data backend defined above:
|
||||||
|
@ -226,24 +219,26 @@ For instance, the following sets the ``localhost`` endpoint to the
|
||||||
"localhost": "myLocationConstraint"
|
"localhost": "myLocationConstraint"
|
||||||
},
|
},
|
||||||
|
|
||||||
To use an endpoint other than localhost for Zenko CloudServer, the endpoint
|
If you would like to use an endpoint other than localhost for your
|
||||||
must be listed in ``restEndpoints``. Otherwise, if the server is running
|
Scality Zenko CloudServer, that endpoint MUST be listed in your
|
||||||
with a:
|
``restEndpoints``. Otherwise if your server is running with a:
|
||||||
|
|
||||||
* **file backend**: The default location constraint is ``file``
|
- **file backend**: your default location constraint will be ``file``
|
||||||
* **memory backend**: The default location constraint is ``mem``
|
|
||||||
|
- **memory backend**: your default location constraint will be ``mem``
|
||||||
|
|
||||||
Endpoints
|
Endpoints
|
||||||
~~~~~~~~~
|
~~~~~~~~~
|
||||||
|
|
||||||
The Zenko CloudServer supports endpoints that are rendered in either:
|
Note that our Zenko CloudServer supports both:
|
||||||
|
|
||||||
* path style: http://myhostname.com/mybucket or
|
- path-style: http://myhostname.com/mybucket
|
||||||
* hosted style: http://mybucket.myhostname.com
|
- hosted-style: http://mybucket.myhostname.com
|
||||||
|
|
||||||
However, if an IP address is specified for the host, hosted-style requests
|
However, hosted-style requests will not hit the server if you are using
|
||||||
cannot reach the server. Use path-style requests in that case. For example,
|
an ip address for your host. So, make sure you are using path-style
|
||||||
if you are using the AWS SDK for JavaScript, instantiate your client like this:
|
requests in that case. For instance, if you are using the AWS SDK for
|
||||||
|
JavaScript, you would instantiate your client like this:
|
||||||
|
|
||||||
.. code:: js
|
.. code:: js
|
||||||
|
|
||||||
|
@ -252,99 +247,87 @@ if you are using the AWS SDK for JavaScript, instantiate your client like this:
|
||||||
s3ForcePathStyle: true,
|
s3ForcePathStyle: true,
|
||||||
});
|
});
|
||||||
|
|
||||||
Setting Your Own Access and Secret Key Pairs
|
Setting your own access key and secret key pairs
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
Credentials can be set for many accounts by editing ``conf/authdata.json``,
|
You can set credentials for many accounts by editing
|
||||||
but use the ``SCALITY_ACCESS_KEY_ID`` and ``SCALITY_SECRET_ACCESS_KEY``
|
``conf/authdata.json`` but if you want to specify one set of your own
|
||||||
environment variables to specify your own credentials.
|
credentials, you can use ``SCALITY_ACCESS_KEY_ID`` and
|
||||||
|
``SCALITY_SECRET_ACCESS_KEY`` environment variables.
|
||||||
_`scality-access-key-id-and-scality-secret-access-key`
|
|
||||||
|
|
||||||
SCALITY\_ACCESS\_KEY\_ID and SCALITY\_SECRET\_ACCESS\_KEY
|
SCALITY\_ACCESS\_KEY\_ID and SCALITY\_SECRET\_ACCESS\_KEY
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
These variables specify authentication credentials for an account named
|
These variables specify authentication credentials for an account named
|
||||||
“CustomAccount”.
|
"CustomAccount".
|
||||||
|
|
||||||
.. note:: Anything in the ``authdata.json`` file is ignored.
|
Note: Anything in the ``authdata.json`` file will be ignored.
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code:: shell
|
||||||
|
|
||||||
$ SCALITY_ACCESS_KEY_ID=newAccessKey SCALITY_SECRET_ACCESS_KEY=newSecretKey yarn start
|
SCALITY_ACCESS_KEY_ID=newAccessKey SCALITY_SECRET_ACCESS_KEY=newSecretKey npm start
|
||||||
|
|
||||||
.. _Using_SSL:
|
|
||||||
|
|
||||||
Using SSL
|
Scality with SSL
|
||||||
~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
To use https with your local CloudServer, you must set up
|
If you wish to use https with your local Zenko CloudServer, you need to set up
|
||||||
SSL certificates.
|
SSL certificates. Here is a simple guide of how to do it.
|
||||||
|
|
||||||
1. Deploy CloudServer using `our DockerHub page
|
Deploying Zenko CloudServer
|
||||||
<https://hub.docker.com/r/zenko/cloudserver/>`__ (run it with a file
|
^^^^^^^^^^^^^^^^^^^
|
||||||
backend).
|
|
||||||
|
|
||||||
.. Note:: If Docker is not installed locally, follow the
|
First, you need to deploy **Zenko CloudServer**. This can be done very easily
|
||||||
`instructions to install it for your distribution
|
via `our **DockerHub**
|
||||||
<https://docs.docker.com/engine/installation/>`__
|
page <https://hub.docker.com/r/scality/s3server/>`__ (you want to run it
|
||||||
|
with a file backend).
|
||||||
|
|
||||||
2. Update the CloudServer container’s config
|
*Note:* *- If you don't have docker installed on your machine, here
|
||||||
|
are the `instructions to install it for your
|
||||||
|
distribution <https://docs.docker.com/engine/installation/>`__*
|
||||||
|
|
||||||
Add your certificates to your container. To do this,
|
Updating your Zenko CloudServer container's config
|
||||||
#. exec inside the CloudServer container.
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
#. Run ``$> docker ps`` to find the container’s ID (the corresponding
|
You're going to add your certificates to your container. In order to do
|
||||||
image name is ``scality/cloudserver``.
|
so, you need to exec inside your Zenko CloudServer container. Run a
|
||||||
|
``$> docker ps`` and find your container's id (the corresponding image
|
||||||
#. Copy the corresponding container ID (``894aee038c5e`` in the present
|
name should be ``scality/s3server``. Copy the corresponding container id
|
||||||
example), and run:
|
(here we'll use ``894aee038c5e``, and run:
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code:: sh
|
||||||
|
|
||||||
$> docker exec -it 894aee038c5e bash
|
$> docker exec -it 894aee038c5e bash
|
||||||
|
|
||||||
This puts you inside your container, using an interactive terminal.
|
You're now inside your container, using an interactive terminal :)
|
||||||
|
|
||||||
3. Generate the SSL key and certificates. The paths where the different
|
Generate SSL key and certificates
|
||||||
files are stored are defined after the ``-out`` option in each of the
|
**********************************
|
||||||
following commands.
|
|
||||||
|
|
||||||
#. Generate a private key for your certificate signing request (CSR):
|
There are 5 steps to this generation. The paths where the different
|
||||||
|
files are stored are defined after the ``-out`` option in each command
|
||||||
|
|
||||||
.. code-block:: shell
|
.. code:: sh
|
||||||
|
|
||||||
$> openssl genrsa -out ca.key 2048
|
# Generate a private key for your CSR
|
||||||
|
$> openssl genrsa -out ca.key 2048
|
||||||
|
# Generate a self signed certificate for your local Certificate Authority
|
||||||
|
$> openssl req -new -x509 -extensions v3_ca -key ca.key -out ca.crt -days 99999 -subj "/C=US/ST=Country/L=City/O=Organization/CN=scality.test"
|
||||||
|
|
||||||
#. Generate a self-signed certificate for your local certificate
|
# Generate a key for Zenko CloudServer
|
||||||
authority (CA):
|
$> openssl genrsa -out test.key 2048
|
||||||
|
# Generate a Certificate Signing Request for S3 Server
|
||||||
|
$> openssl req -new -key test.key -out test.csr -subj "/C=US/ST=Country/L=City/O=Organization/CN=*.scality.test"
|
||||||
|
# Generate a local-CA-signed certificate for S3 Server
|
||||||
|
$> openssl x509 -req -in test.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out test.crt -days 99999 -sha256
|
||||||
|
|
||||||
.. code:: shell
|
Update Zenko CloudServer ``config.json``
|
||||||
|
**********************************
|
||||||
|
|
||||||
$> openssl req -new -x509 -extensions v3_ca -key ca.key -out ca.crt -days 99999 -subj "/C=US/ST=Country/L=City/O=Organization/CN=scality.test"
|
Add a ``certFilePaths`` section to ``./config.json`` with the
|
||||||
|
appropriate paths:
|
||||||
|
|
||||||
#. Generate a key for the CloudServer:
|
.. code:: json
|
||||||
|
|
||||||
.. code:: shell
|
|
||||||
|
|
||||||
$> openssl genrsa -out test.key 2048
|
|
||||||
|
|
||||||
#. Generate a CSR for CloudServer:
|
|
||||||
|
|
||||||
.. code:: shell
|
|
||||||
|
|
||||||
$> openssl req -new -key test.key -out test.csr -subj "/C=US/ST=Country/L=City/O=Organization/CN=*.scality.test"
|
|
||||||
|
|
||||||
#. Generate a certificate for CloudServer signed by the local CA:
|
|
||||||
|
|
||||||
.. code:: shell
|
|
||||||
|
|
||||||
$> openssl x509 -req -in test.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out test.crt -days 99999 -sha256
|
|
||||||
|
|
||||||
4. Update Zenko CloudServer ``config.json``. Add a ``certFilePaths``
|
|
||||||
section to ``./config.json`` with appropriate paths:
|
|
||||||
|
|
||||||
.. code:: json
|
|
||||||
|
|
||||||
"certFilePaths": {
|
"certFilePaths": {
|
||||||
"key": "./test.key",
|
"key": "./test.key",
|
||||||
|
@ -352,36 +335,42 @@ SSL certificates.
|
||||||
"ca": "./ca.crt"
|
"ca": "./ca.crt"
|
||||||
}
|
}
|
||||||
|
|
||||||
5. Run your container with the new config.
|
Run your container with the new config
|
||||||
|
****************************************
|
||||||
|
|
||||||
#. Exit the container by running ``$> exit``.
|
First, you need to exit your container. Simply run ``$> exit``. Then,
|
||||||
|
you need to restart your container. Normally, a simple
|
||||||
|
``$> docker restart s3server`` should do the trick.
|
||||||
|
|
||||||
#. Restart the container with ``$> docker restart cloudserver``.
|
Update your host config
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
6. Update the host configuration by adding s3.scality.test
|
Associates local IP addresses with hostname
|
||||||
to /etc/hosts:
|
*******************************************
|
||||||
|
|
||||||
.. code:: bash
|
In your ``/etc/hosts`` file on Linux, OS X, or Unix (with root
|
||||||
|
permissions), edit the line of localhost so it looks like this:
|
||||||
|
|
||||||
127.0.0.1 localhost s3.scality.test
|
::
|
||||||
|
|
||||||
7. Copy the local certificate authority (ca.crt in step 4) from your
|
127.0.0.1 localhost s3.scality.test
|
||||||
container. Choose the path to save this file to (in the present
|
|
||||||
example, ``/root/ca.crt``), and run:
|
|
||||||
|
|
||||||
.. code:: shell
|
Copy the local certificate authority from your container
|
||||||
|
*********************************************************
|
||||||
|
|
||||||
$> docker cp 894aee038c5e:/usr/src/app/ca.crt /root/ca.crt
|
In the above commands, it's the file named ``ca.crt``. Choose the path
|
||||||
|
you want to save this file at (here we chose ``/root/ca.crt``), and run
|
||||||
|
something like:
|
||||||
|
|
||||||
.. note:: Your container ID will be different, and your path to
|
.. code:: sh
|
||||||
ca.crt may be different.
|
|
||||||
|
|
||||||
Test the Config
|
$> docker cp 894aee038c5e:/usr/src/app/ca.crt /root/ca.crt
|
||||||
^^^^^^^^^^^^^^^
|
|
||||||
|
|
||||||
If aws-sdk is not installed, run ``$> yarn install aws-sdk``.
|
Test your config
|
||||||
|
^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
Paste the following script into a file named "test.js":
|
If you do not have aws-sdk installed, run ``$> npm install aws-sdk``. In
|
||||||
|
a ``test.js`` file, paste the following script:
|
||||||
|
|
||||||
.. code:: js
|
.. code:: js
|
||||||
|
|
||||||
|
@ -421,13 +410,8 @@ Paste the following script into a file named "test.js":
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
Now run this script with:
|
Now run that script with ``$> nodejs test.js``. If all goes well, it
|
||||||
|
should output ``SSL is cool!``. Enjoy that added security!
|
||||||
.. code::
|
|
||||||
|
|
||||||
$> nodejs test.js
|
|
||||||
|
|
||||||
On success, the script outputs ``SSL is cool!``.
|
|
||||||
|
|
||||||
|
|
||||||
.. |CircleCI| image:: https://circleci.com/gh/scality/S3.svg?style=svg
|
.. |CircleCI| image:: https://circleci.com/gh/scality/S3.svg?style=svg
|
||||||
|
|
|
@ -1,7 +1,6 @@
|
||||||
# S3 Healthcheck
|
# S3 Healthcheck
|
||||||
|
|
||||||
Scality S3 exposes a healthcheck route `/live` on the port used
|
Scality S3 exposes a healthcheck route `/_/healthcheck` which returns a
|
||||||
for the metrics (defaults to port 8002) which returns a
|
|
||||||
response with HTTP code
|
response with HTTP code
|
||||||
|
|
||||||
- 200 OK
|
- 200 OK
|
|
@ -4,415 +4,479 @@ Integrations
|
||||||
High Availability
|
High Availability
|
||||||
=================
|
=================
|
||||||
|
|
||||||
`Docker Swarm <https://docs.docker.com/engine/swarm/>`__ is a clustering tool
|
`Docker swarm <https://docs.docker.com/engine/swarm/>`__ is a
|
||||||
developed by Docker for use with its containers. It can be used to start
|
clustering tool developped by Docker and ready to use with its
|
||||||
services, which we define to ensure CloudServer's continuous availability to
|
containers. It allows to start a service, which we define and use as a
|
||||||
end users. A swarm defines a manager and *n* workers among *n* + 1 servers.
|
means to ensure Zenko CloudServer's continuous availability to the end user.
|
||||||
|
Indeed, a swarm defines a manager and n workers among n+1 servers. We
|
||||||
This tutorial shows how to perform a basic setup with three servers, which
|
will do a basic setup in this tutorial, with just 3 servers, which
|
||||||
provides strong service resiliency, while remaining easy to use and
|
already provides a strong service resiliency, whilst remaining easy to
|
||||||
maintain. We will use NFS through Docker to share data and
|
do as an individual. We will use NFS through docker to share data and
|
||||||
metadata between the different servers.
|
metadata between the different servers.
|
||||||
|
|
||||||
Sections are labeled **On Server**, **On Clients**, or
|
You will see that the steps of this tutorial are defined as **On
|
||||||
**On All Machines**, referring respectively to NFS server, NFS clients, or
|
Server**, **On Clients**, **On All Machines**. This refers respectively
|
||||||
NFS server and clients. In the present example, the server’s IP address is
|
to NFS Server, NFS Clients, or NFS Server and Clients. In our example,
|
||||||
**10.200.15.113** and the client IP addresses are **10.200.15.96** and
|
the IP of the Server will be **10.200.15.113**, while the IPs of the
|
||||||
**10.200.15.97**
|
Clients will be **10.200.15.96 and 10.200.15.97**
|
||||||
|
|
||||||
1. Install Docker (on All Machines)
|
Installing docker
|
||||||
|
-----------------
|
||||||
|
|
||||||
Docker 17.03.0-ce is used for this tutorial. Docker 1.12.6 and later will
|
Any version from docker 1.12.6 onwards should work; we used Docker
|
||||||
likely work, but is not tested.
|
17.03.0-ce for this tutorial.
|
||||||
|
|
||||||
* On Ubuntu 14.04
|
On All Machines
|
||||||
Install Docker CE for Ubuntu as `documented at Docker
|
~~~~~~~~~~~~~~~
|
||||||
<https://docs.docker.com/install/linux/docker-ce/ubuntu/>`__.
|
|
||||||
Install the aufs dependency as recommended by Docker. The required
|
|
||||||
commands are:
|
|
||||||
|
|
||||||
.. code:: sh
|
On Ubuntu 14.04
|
||||||
|
^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
$> sudo apt-get update
|
The docker website has `solid
|
||||||
$> sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual
|
documentation <https://docs.docker.com/engine/installation/linux/ubuntu/>`__.
|
||||||
$> sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
|
We have chosen to install the aufs dependency, as recommended by Docker.
|
||||||
$> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
|
Here are the required commands:
|
||||||
$> sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
|
|
||||||
$> sudo apt-get update
|
|
||||||
$> sudo apt-get install docker-ce
|
|
||||||
|
|
||||||
* On CentOS 7
|
|
||||||
Install Docker CE as `documented at Docker
|
|
||||||
<https://docs.docker.com/install/linux/docker-ce/centos/>`__.
|
|
||||||
The required commands are:
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
$> sudo yum install -y yum-utils
|
|
||||||
$> sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
|
|
||||||
$> sudo yum makecache fast
|
|
||||||
$> sudo yum install docker-ce
|
|
||||||
$> sudo systemctl start docker
|
|
||||||
|
|
||||||
2. Install NFS on Client(s)
|
|
||||||
|
|
||||||
NFS clients mount Docker volumes over the NFS server’s shared folders.
|
|
||||||
If the NFS commons are installed, manual mounts are no longer needed.
|
|
||||||
|
|
||||||
* On Ubuntu 14.04
|
|
||||||
|
|
||||||
Install the NFS commons with apt-get:
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
$> sudo apt-get install nfs-common
|
|
||||||
|
|
||||||
* On CentOS 7
|
|
||||||
|
|
||||||
Install the NFS utils; then start required services:
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
$> yum install nfs-utils
|
|
||||||
$> sudo systemctl enable rpcbind
|
|
||||||
$> sudo systemctl enable nfs-server
|
|
||||||
$> sudo systemctl enable nfs-lock
|
|
||||||
$> sudo systemctl enable nfs-idmap
|
|
||||||
$> sudo systemctl start rpcbind
|
|
||||||
$> sudo systemctl start nfs-server
|
|
||||||
$> sudo systemctl start nfs-lock
|
|
||||||
$> sudo systemctl start nfs-idmap
|
|
||||||
|
|
||||||
3. Install NFS (on Server)
|
|
||||||
|
|
||||||
The NFS server hosts the data and metadata. The package(s) to install on it
|
|
||||||
differs from the package installed on the clients.
|
|
||||||
|
|
||||||
* On Ubuntu 14.04
|
|
||||||
|
|
||||||
Install the NFS server-specific package and the NFS commons:
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
$> sudo apt-get install nfs-kernel-server nfs-common
|
|
||||||
|
|
||||||
* On CentOS 7
|
|
||||||
|
|
||||||
Install the NFS utils and start the required services:
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
$> yum install nfs-utils
|
|
||||||
$> sudo systemctl enable rpcbind
|
|
||||||
$> sudo systemctl enable nfs-server
|
|
||||||
$> sudo systemctl enable nfs-lock
|
|
||||||
$> sudo systemctl enable nfs-idmap
|
|
||||||
$> sudo systemctl start rpcbind
|
|
||||||
$> sudo systemctl start nfs-server
|
|
||||||
$> sudo systemctl start nfs-lock
|
|
||||||
$> sudo systemctl start nfs-idmap
|
|
||||||
|
|
||||||
For both distributions:
|
|
||||||
|
|
||||||
#. Choose where shared data and metadata from the local
|
|
||||||
`CloudServer <http://www.zenko.io/cloudserver/>`__ shall be stored (The
|
|
||||||
present example uses /var/nfs/data and /var/nfs/metadata). Set permissions
|
|
||||||
for these folders for
|
|
||||||
sharing over NFS:
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
$> mkdir -p /var/nfs/data /var/nfs/metadata
|
|
||||||
$> chmod -R 777 /var/nfs/
|
|
||||||
|
|
||||||
#. The /etc/exports file configures network permissions and r-w-x permissions
|
|
||||||
for NFS access. Edit /etc/exports, adding the following lines:
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
/var/nfs/data 10.200.15.96(rw,sync,no_root_squash) 10.200.15.97(rw,sync,no_root_squash)
|
|
||||||
/var/nfs/metadata 10.200.15.96(rw,sync,no_root_squash) 10.200.15.97(rw,sync,no_root_squash)
|
|
||||||
|
|
||||||
Ubuntu applies the no\_subtree\_check option by default, so both
|
|
||||||
folders are declared with the same permissions, even though they’re in
|
|
||||||
the same tree.
|
|
||||||
|
|
||||||
#. Export this new NFS table:
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
$> sudo exportfs -a
|
|
||||||
|
|
||||||
#. Edit the ``MountFlags`` option in the Docker config in
|
|
||||||
/lib/systemd/system/docker.service to enable NFS mount from Docker volumes
|
|
||||||
on other machines:
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
MountFlags=shared
|
|
||||||
|
|
||||||
#. Restart the NFS server and Docker daemons to apply these changes.
|
|
||||||
|
|
||||||
* On Ubuntu 14.04
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
$> sudo service nfs-kernel-server restart
|
|
||||||
$> sudo service docker restart
|
|
||||||
|
|
||||||
* On CentOS 7
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
$> sudo systemctl restart nfs-server
|
|
||||||
$> sudo systemctl daemon-reload
|
|
||||||
$> sudo systemctl restart docker
|
|
||||||
|
|
||||||
|
|
||||||
4. Set Up a Docker Swarm
|
|
||||||
|
|
||||||
* On all machines and distributions:
|
|
||||||
|
|
||||||
Set up the Docker volumes to be mounted to the NFS server for CloudServer’s
|
|
||||||
data and metadata storage. The following commands must be replicated on all
|
|
||||||
machines:
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
$> docker volume create --driver local --opt type=nfs --opt o=addr=10.200.15.113,rw --opt device=:/var/nfs/data --name data
|
|
||||||
$> docker volume create --driver local --opt type=nfs --opt o=addr=10.200.15.113,rw --opt device=:/var/nfs/metadata --name metadata
|
|
||||||
|
|
||||||
There is no need to ``docker exec`` these volumes to mount them: the
|
|
||||||
Docker Swarm manager does this when the Docker service is started.
|
|
||||||
|
|
||||||
* On a server:
|
|
||||||
|
|
||||||
To start a Docker service on a Docker Swarm cluster, initialize the cluster
|
|
||||||
(that is, define a manager), prompt workers/nodes to join in, and then start
|
|
||||||
the service.
|
|
||||||
|
|
||||||
Initialize the swarm cluster, and review its response:
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
$> docker swarm init --advertise-addr 10.200.15.113
|
|
||||||
|
|
||||||
Swarm initialized: current node (db2aqfu3bzfzzs9b1kfeaglmq) is now a manager.
|
|
||||||
|
|
||||||
To add a worker to this swarm, run the following command:
|
|
||||||
|
|
||||||
docker swarm join \
|
|
||||||
--token SWMTKN-1-5yxxencrdoelr7mpltljn325uz4v6fe1gojl14lzceij3nujzu-2vfs9u6ipgcq35r90xws3stka \
|
|
||||||
10.200.15.113:2377
|
|
||||||
|
|
||||||
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
|
|
||||||
|
|
||||||
* On clients:
|
|
||||||
|
|
||||||
Copy and paste the command provided by your Docker Swarm init. A successful
|
|
||||||
request/response will resemble:
|
|
||||||
|
|
||||||
.. code:: sh
|
|
||||||
|
|
||||||
$> docker swarm join --token SWMTKN-1-5yxxencrdoelr7mpltljn325uz4v6fe1gojl14lzceij3nujzu-2vfs9u6ipgcq35r90xws3stka 10.200.15.113:2377
|
|
||||||
|
|
||||||
This node joined a swarm as a worker.
|
|
||||||
|
|
||||||
Set Up Docker Swarm on Clients on a Server
|
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
||||||
|
|
||||||
Start the service on the Swarm cluster.
|
|
||||||
|
|
||||||
.. code:: sh
|
.. code:: sh
|
||||||
|
|
||||||
$> docker service create --name s3 --replicas 1 --mount type=volume,source=data,target=/usr/src/app/localData --mount type=volume,source=metadata,target=/usr/src/app/localMetadata -p 8000:8000 scality/cloudserver
|
$> sudo apt-get update
|
||||||
|
$> sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual
|
||||||
|
$> sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
|
||||||
|
$> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
|
||||||
|
$> sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
|
||||||
|
$> sudo apt-get update
|
||||||
|
$> sudo apt-get install docker-ce
|
||||||
|
|
||||||
On a successful installation, ``docker service ls`` returns the following
|
On CentOS 7
|
||||||
output:
|
^^^^^^^^^^^
|
||||||
|
|
||||||
|
The docker website has `solid
|
||||||
|
documentation <https://docs.docker.com/engine/installation/linux/centos/>`__.
|
||||||
|
Here are the required commands:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> sudo yum install -y yum-utils
|
||||||
|
$> sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
|
||||||
|
$> sudo yum makecache fast
|
||||||
|
$> sudo yum install docker-ce
|
||||||
|
$> sudo systemctl start docker
|
||||||
|
|
||||||
|
Configure NFS
|
||||||
|
-------------
|
||||||
|
|
||||||
|
On Clients
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Your NFS Clients will mount Docker volumes over your NFS Server's shared
|
||||||
|
folders. Hence, you don't have to mount anything manually, you just have
|
||||||
|
to install the NFS commons:
|
||||||
|
|
||||||
|
On Ubuntu 14.04
|
||||||
|
^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
Simply install the NFS commons:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> sudo apt-get install nfs-common
|
||||||
|
|
||||||
|
On CentOS 7
|
||||||
|
^^^^^^^^^^^
|
||||||
|
|
||||||
|
Install the NFS utils, and then start the required services:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> yum install nfs-utils
|
||||||
|
$> sudo systemctl enable rpcbind
|
||||||
|
$> sudo systemctl enable nfs-server
|
||||||
|
$> sudo systemctl enable nfs-lock
|
||||||
|
$> sudo systemctl enable nfs-idmap
|
||||||
|
$> sudo systemctl start rpcbind
|
||||||
|
$> sudo systemctl start nfs-server
|
||||||
|
$> sudo systemctl start nfs-lock
|
||||||
|
$> sudo systemctl start nfs-idmap
|
||||||
|
|
||||||
|
On Server
|
||||||
|
~~~~~~~~~
|
||||||
|
|
||||||
|
Your NFS Server will be the machine to physically host the data and
|
||||||
|
metadata. The package(s) we will install on it is slightly different
|
||||||
|
from the one we installed on the clients.
|
||||||
|
|
||||||
|
On Ubuntu 14.04
|
||||||
|
^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
Install the NFS server specific package and the NFS commons:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> sudo apt-get install nfs-kernel-server nfs-common
|
||||||
|
|
||||||
|
On CentOS 7
|
||||||
|
^^^^^^^^^^^
|
||||||
|
|
||||||
|
Same steps as with the client: install the NFS utils and start the
|
||||||
|
required services:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> yum install nfs-utils
|
||||||
|
$> sudo systemctl enable rpcbind
|
||||||
|
$> sudo systemctl enable nfs-server
|
||||||
|
$> sudo systemctl enable nfs-lock
|
||||||
|
$> sudo systemctl enable nfs-idmap
|
||||||
|
$> sudo systemctl start rpcbind
|
||||||
|
$> sudo systemctl start nfs-server
|
||||||
|
$> sudo systemctl start nfs-lock
|
||||||
|
$> sudo systemctl start nfs-idmap
|
||||||
|
|
||||||
|
On Ubuntu 14.04 and CentOS 7
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
Choose where your shared data and metadata from your local `Zenko CloudServer
|
||||||
|
<http://www.zenko.io/cloudserver/>`__ will be stored.
|
||||||
|
We chose to go with /var/nfs/data and /var/nfs/metadata. You also need
|
||||||
|
to set proper sharing permissions for these folders as they'll be shared
|
||||||
|
over NFS:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> mkdir -p /var/nfs/data /var/nfs/metadata
|
||||||
|
$> chmod -R 777 /var/nfs/
|
||||||
|
|
||||||
|
Now you need to update your **/etc/exports** file. This is the file that
|
||||||
|
configures network permissions and rwx permissions for NFS access. By
|
||||||
|
default, Ubuntu applies the no\_subtree\_check option, so we declared
|
||||||
|
both folders with the same permissions, even though they're in the same
|
||||||
|
tree:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> sudo vim /etc/exports
|
||||||
|
|
||||||
|
In this file, add the following lines:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
/var/nfs/data 10.200.15.96(rw,sync,no_root_squash) 10.200.15.97(rw,sync,no_root_squash)
|
||||||
|
/var/nfs/metadata 10.200.15.96(rw,sync,no_root_squash) 10.200.15.97(rw,sync,no_root_squash)
|
||||||
|
|
||||||
|
Export this new NFS table:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> sudo exportfs -a
|
||||||
|
|
||||||
|
Eventually, you need to allow for NFS mount from Docker volumes on other
|
||||||
|
machines. You need to change the Docker config in
|
||||||
|
**/lib/systemd/system/docker.service**:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> sudo vim /lib/systemd/system/docker.service
|
||||||
|
|
||||||
|
In this file, change the **MountFlags** option:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
MountFlags=shared
|
||||||
|
|
||||||
|
Now you just need to restart the NFS server and docker daemons so your
|
||||||
|
changes apply.
|
||||||
|
|
||||||
|
On Ubuntu 14.04
|
||||||
|
^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
Restart your NFS Server and docker services:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> sudo service nfs-kernel-server restart
|
||||||
|
$> sudo service docker restart
|
||||||
|
|
||||||
|
On CentOS 7
|
||||||
|
^^^^^^^^^^^
|
||||||
|
|
||||||
|
Restart your NFS Server and docker daemons:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> sudo systemctl restart nfs-server
|
||||||
|
$> sudo systemctl daemon-reload
|
||||||
|
$> sudo systemctl restart docker
|
||||||
|
|
||||||
|
Set up your Docker Swarm service
|
||||||
|
--------------------------------
|
||||||
|
|
||||||
|
On All Machines
|
||||||
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
On Ubuntu 14.04 and CentOS 7
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
We will now set up the Docker volumes that will be mounted to the NFS
|
||||||
|
Server and serve as data and metadata storage for Zenko CloudServer. These two
|
||||||
|
commands have to be replicated on all machines:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> docker volume create --driver local --opt type=nfs --opt o=addr=10.200.15.113,rw --opt device=:/var/nfs/data --name data
|
||||||
|
$> docker volume create --driver local --opt type=nfs --opt o=addr=10.200.15.113,rw --opt device=:/var/nfs/metadata --name metadata
|
||||||
|
|
||||||
|
There is no need to ""docker exec" these volumes to mount them: the
|
||||||
|
Docker Swarm manager will do it when the Docker service will be started.
|
||||||
|
|
||||||
|
On Server
|
||||||
|
^^^^^^^^^
|
||||||
|
|
||||||
|
To start a Docker service on a Docker Swarm cluster, you first have to
|
||||||
|
initialize that cluster (i.e.: define a manager), then have the
|
||||||
|
workers/nodes join in, and then start the service. Initialize the swarm
|
||||||
|
cluster, and look at the response:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> docker swarm init --advertise-addr 10.200.15.113
|
||||||
|
|
||||||
|
Swarm initialized: current node (db2aqfu3bzfzzs9b1kfeaglmq) is now a manager.
|
||||||
|
|
||||||
|
To add a worker to this swarm, run the following command:
|
||||||
|
|
||||||
|
docker swarm join \
|
||||||
|
--token SWMTKN-1-5yxxencrdoelr7mpltljn325uz4v6fe1gojl14lzceij3nujzu-2vfs9u6ipgcq35r90xws3stka \
|
||||||
|
10.200.15.113:2377
|
||||||
|
|
||||||
|
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
|
||||||
|
|
||||||
|
On Clients
|
||||||
|
^^^^^^^^^^
|
||||||
|
|
||||||
|
Simply copy/paste the command provided by your docker swarm init. When
|
||||||
|
all goes well, you'll get something like this:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> docker swarm join --token SWMTKN-1-5yxxencrdoelr7mpltljn325uz4v6fe1gojl14lzceij3nujzu-2vfs9u6ipgcq35r90xws3stka 10.200.15.113:2377
|
||||||
|
|
||||||
|
This node joined a swarm as a worker.
|
||||||
|
|
||||||
|
On Server
|
||||||
|
^^^^^^^^^
|
||||||
|
|
||||||
|
Start the service on your swarm cluster!
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> docker service create --name s3 --replicas 1 --mount type=volume,source=data,target=/usr/src/app/localData --mount type=volume,source=metadata,target=/usr/src/app/localMetadata -p 8000:8000 scality/s3server
|
||||||
|
|
||||||
|
If you run a docker service ls, you should have the following output:
|
||||||
|
|
||||||
.. code:: sh
|
.. code:: sh
|
||||||
|
|
||||||
$> docker service ls
|
$> docker service ls
|
||||||
ID NAME MODE REPLICAS IMAGE
|
ID NAME MODE REPLICAS IMAGE
|
||||||
ocmggza412ft s3 replicated 1/1 scality/cloudserver:latest
|
ocmggza412ft s3 replicated 1/1 scality/s3server:latest
|
||||||
|
|
||||||
If the service does not start, consider disabling apparmor/SELinux.
|
If your service won't start, consider disabling apparmor/SELinux.
|
||||||
|
|
||||||
Testing the High-Availability CloudServer
|
Testing your High Availability S3Server
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
---------------------------------------
|
||||||
|
|
||||||
On all machines (client/server) and distributions (Ubuntu and CentOS),
|
On All Machines
|
||||||
determine where CloudServer is running using ``docker ps``. CloudServer can
|
|
||||||
operate on any node of the Swarm cluster, manager or worker. When you find
|
|
||||||
it, you can kill it with ``docker stop <container id>``. It will respawn
|
|
||||||
on a different node. Now, if one server falls, or if Docker stops
|
|
||||||
unexpectedly, the end user will still be able to access your the local CloudServer.
|
|
||||||
|
|
||||||
Troubleshooting
|
|
||||||
~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
To troubleshoot the service, run:
|
On Ubuntu 14.04 and CentOS 7
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
Try to find out where your Scality Zenko CloudServer is actually running using
|
||||||
|
the **docker ps** command. It can be on any node of the swarm cluster,
|
||||||
|
manager or worker. When you find it, you can kill it, with **docker stop
|
||||||
|
<container id>** and you'll see it respawn on a different node of the
|
||||||
|
swarm cluster. Now you see, if one of your servers falls, or if docker
|
||||||
|
stops unexpectedly, your end user will still be able to access your
|
||||||
|
local Zenko CloudServer.
|
||||||
|
|
||||||
|
Troubleshooting
|
||||||
|
---------------
|
||||||
|
|
||||||
|
To troubleshoot the service you can run:
|
||||||
|
|
||||||
.. code:: sh
|
.. code:: sh
|
||||||
|
|
||||||
$> docker service ps s3docker service ps s3
|
$> docker service ps s3docker service ps s3
|
||||||
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
|
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
|
||||||
0ar81cw4lvv8chafm8pw48wbc s3.1 scality/cloudserver localhost.localdomain.localdomain Running Running 7 days ago
|
0ar81cw4lvv8chafm8pw48wbc s3.1 scality/s3server localhost.localdomain.localdomain Running Running 7 days ago
|
||||||
cvmf3j3bz8w6r4h0lf3pxo6eu \_ s3.1 scality/cloudserver localhost.localdomain.localdomain Shutdown Failed 7 days ago "task: non-zero exit (137)"
|
cvmf3j3bz8w6r4h0lf3pxo6eu \_ s3.1 scality/s3server localhost.localdomain.localdomain Shutdown Failed 7 days ago "task: non-zero exit (137)"
|
||||||
|
|
||||||
If the error is truncated, view the error in detail by inspecting the
|
If the error is truncated it is possible to have a more detailed view of
|
||||||
Docker task ID:
|
the error by inspecting the docker task ID:
|
||||||
|
|
||||||
.. code:: sh
|
.. code:: sh
|
||||||
|
|
||||||
$> docker inspect cvmf3j3bz8w6r4h0lf3pxo6eu
|
$> docker inspect cvmf3j3bz8w6r4h0lf3pxo6eu
|
||||||
|
|
||||||
Off you go!
|
Off you go!
|
||||||
~~~~~~~~~~~
|
-----------
|
||||||
|
|
||||||
|
Let us know what you use this functionality for, and if you'd like any
|
||||||
|
specific developments around it. Or, even better: come and contribute to
|
||||||
|
our `Github repository <https://github.com/scality/s3/>`__! We look
|
||||||
|
forward to meeting you!
|
||||||
|
|
||||||
Let us know how you use this and if you'd like any specific developments
|
|
||||||
around it. Even better: come and contribute to our `Github repository
|
|
||||||
<https://github.com/scality/s3/>`__! We look forward to meeting you!
|
|
||||||
|
|
||||||
S3FS
|
S3FS
|
||||||
====
|
====
|
||||||
|
Export your buckets as a filesystem with s3fs on top of Zenko CloudServer
|
||||||
You can export buckets as a filesystem with s3fs on CloudServer.
|
|
||||||
|
|
||||||
`s3fs <https://github.com/s3fs-fuse/s3fs-fuse>`__ is an open source
|
`s3fs <https://github.com/s3fs-fuse/s3fs-fuse>`__ is an open source
|
||||||
tool, available both on Debian and RedHat distributions, that enables
|
tool that allows you to mount an S3 bucket on a filesystem-like backend.
|
||||||
you to mount an S3 bucket on a filesystem-like backend. This tutorial uses
|
It is available both on Debian and RedHat distributions. For this
|
||||||
an Ubuntu 14.04 host to deploy and use s3fs over CloudServer.
|
tutorial, we used an Ubuntu 14.04 host to deploy and use s3fs over
|
||||||
|
Scality's Zenko CloudServer.
|
||||||
|
|
||||||
Deploying Zenko CloudServer with SSL
|
Deploying Zenko CloudServer with SSL
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
----------------------------
|
||||||
|
|
||||||
First, deploy CloudServer with a file backend using `our DockerHub page
|
First, you need to deploy **Zenko CloudServer**. This can be done very easily
|
||||||
<https://hub.docker.com/r/zenko/cloudserver>`__.
|
via `our DockerHub
|
||||||
|
page <https://hub.docker.com/r/scality/s3server/>`__ (you want to run it
|
||||||
|
with a file backend).
|
||||||
|
|
||||||
.. note::
|
*Note:* *- If you don't have docker installed on your machine, here
|
||||||
|
are the `instructions to install it for your
|
||||||
|
distribution <https://docs.docker.com/engine/installation/>`__*
|
||||||
|
|
||||||
If Docker is not installed on your machine, follow
|
You also necessarily have to set up SSL with Zenko CloudServer to use s3fs. We
|
||||||
`these instructions <https://docs.docker.com/engine/installation/>`__
|
have a nice
|
||||||
to install it for your distribution.
|
`tutorial <https://s3.scality.com/v1.0/page/scality-with-ssl>`__ to help
|
||||||
|
you do it.
|
||||||
|
|
||||||
You must also set up SSL with CloudServer to use s3fs. See `Using SSL
|
s3fs setup
|
||||||
<./GETTING_STARTED#Using_SSL>`__ for instructions.
|
----------
|
||||||
|
|
||||||
s3fs Setup
|
|
||||||
~~~~~~~~~~
|
|
||||||
|
|
||||||
Installing s3fs
|
Installing s3fs
|
||||||
---------------
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
Follow the instructions in the s3fs `README
|
s3fs has quite a few dependencies. As explained in their
|
||||||
<https://github.com/s3fs-fuse/s3fs-fuse/blob/master/README.md#installation-from-pre-built-packages>`__,
|
`README <https://github.com/s3fs-fuse/s3fs-fuse/blob/master/README.md#installation>`__,
|
||||||
|
the following commands should install everything for Ubuntu 14.04:
|
||||||
Check that s3fs is properly installed. A version check should return
|
|
||||||
a response resembling:
|
|
||||||
|
|
||||||
.. code:: sh
|
.. code:: sh
|
||||||
|
|
||||||
$> s3fs --version
|
$> sudo apt-get install automake autotools-dev g++ git libcurl4-gnutls-dev
|
||||||
|
$> sudo apt-get install libfuse-dev libssl-dev libxml2-dev make pkg-config
|
||||||
|
|
||||||
|
Now you want to install s3fs per se:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> git clone https://github.com/s3fs-fuse/s3fs-fuse.git
|
||||||
|
$> cd s3fs-fuse
|
||||||
|
$> ./autogen.sh
|
||||||
|
$> ./configure
|
||||||
|
$> make
|
||||||
|
$> sudo make install
|
||||||
|
|
||||||
|
Check that s3fs is properly installed by checking its version. it should
|
||||||
|
answer as below:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
|
$> s3fs --version
|
||||||
|
|
||||||
Amazon Simple Storage Service File System V1.80(commit:d40da2c) with OpenSSL
|
Amazon Simple Storage Service File System V1.80(commit:d40da2c) with OpenSSL
|
||||||
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
|
|
||||||
License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>
|
|
||||||
This is free software: you are free to change and redistribute it.
|
|
||||||
There is NO WARRANTY, to the extent permitted by law.
|
|
||||||
|
|
||||||
Configuring s3fs
|
Configuring s3fs
|
||||||
----------------
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
s3fs expects you to provide it with a password file. Our file is
|
s3fs expects you to provide it with a password file. Our file is
|
||||||
``/etc/passwd-s3fs``. The structure for this file is
|
``/etc/passwd-s3fs``. The structure for this file is
|
||||||
``ACCESSKEYID:SECRETKEYID``, so, for CloudServer, you can run:
|
``ACCESSKEYID:SECRETKEYID``, so, for S3Server, you can run:
|
||||||
|
|
||||||
.. code:: sh
|
.. code:: sh
|
||||||
|
|
||||||
$> echo 'accessKey1:verySecretKey1' > /etc/passwd-s3fs
|
$> echo 'accessKey1:verySecretKey1' > /etc/passwd-s3fs
|
||||||
$> chmod 600 /etc/passwd-s3fs
|
$> chmod 600 /etc/passwd-s3fs
|
||||||
|
|
||||||
Using CloudServer with s3fs
|
Using Zenko CloudServer with s3fs
|
||||||
---------------------------
|
------------------------
|
||||||
|
|
||||||
1. Use /mnt/tests3fs as a mount point.
|
First, you're going to need a mountpoint; we chose ``/mnt/tests3fs``:
|
||||||
|
|
||||||
.. code:: sh
|
.. code:: sh
|
||||||
|
|
||||||
$> mkdir /mnt/tests3fs
|
$> mkdir /mnt/tests3fs
|
||||||
|
|
||||||
2. Create a bucket on your local CloudServer. In the present example it is
|
Then, you want to create a bucket on your local Zenko CloudServer; we named it
|
||||||
named “tests3fs”.
|
``tests3fs``:
|
||||||
|
|
||||||
.. code:: sh
|
.. code:: sh
|
||||||
|
|
||||||
$> s3cmd mb s3://tests3fs
|
$> s3cmd mb s3://tests3fs
|
||||||
|
|
||||||
3. Mount the bucket to your mount point with s3fs:
|
*Note:* *- If you've never used s3cmd with our Zenko CloudServer, our README
|
||||||
|
provides you with a `recommended
|
||||||
|
config <https://github.com/scality/S3/blob/master/README.md#s3cmd>`__*
|
||||||
|
|
||||||
.. code:: sh
|
Now you can mount your bucket to your mountpoint with s3fs:
|
||||||
|
|
||||||
|
.. code:: sh
|
||||||
|
|
||||||
$> s3fs tests3fs /mnt/tests3fs -o passwd_file=/etc/passwd-s3fs -o url="https://s3.scality.test:8000/" -o use_path_request_style
|
$> s3fs tests3fs /mnt/tests3fs -o passwd_file=/etc/passwd-s3fs -o url="https://s3.scality.test:8000/" -o use_path_request_style
|
||||||
|
|
||||||
The structure of this command is:
|
*If you're curious, the structure of this command is*
|
||||||
``s3fs BUCKET_NAME PATH/TO/MOUNTPOINT -o OPTIONS``. Of these mandatory
|
``s3fs BUCKET_NAME PATH/TO/MOUNTPOINT -o OPTIONS``\ *, and the
|
||||||
options:
|
options are mandatory and serve the following purposes:
|
||||||
|
* ``passwd_file``\ *: specifiy path to password file;
|
||||||
|
* ``url``\ *: specify the hostname used by your SSL provider;
|
||||||
|
* ``use_path_request_style``\ *: force path style (by default, s3fs
|
||||||
|
uses subdomains (DNS style)).*
|
||||||
|
|
||||||
* ``passwd_file`` specifies the path to the password file.
|
| From now on, you can either add files to your mountpoint, or add
|
||||||
* ``url`` specifies the host name used by your SSL provider.
|
objects to your bucket, and they'll show in the other.
|
||||||
* ``use_path_request_style`` forces the path style (by default,
|
| For example, let's' create two files, and then a directory with a file
|
||||||
s3fs uses DNS-style subdomains).
|
in our mountpoint:
|
||||||
|
|
||||||
Once the bucket is mounted, files added to the mount point or
|
.. code:: sh
|
||||||
objects added to the bucket will appear in both locations.
|
|
||||||
|
|
||||||
Example
|
$> touch /mnt/tests3fs/file1 /mnt/tests3fs/file2
|
||||||
-------
|
$> mkdir /mnt/tests3fs/dir1
|
||||||
|
$> touch /mnt/tests3fs/dir1/file3
|
||||||
|
|
||||||
Create two files, and then a directory with a file in our mount point:
|
Now, I can use s3cmd to show me what is actually in S3Server:
|
||||||
|
|
||||||
.. code:: sh
|
.. code:: sh
|
||||||
|
|
||||||
$> touch /mnt/tests3fs/file1 /mnt/tests3fs/file2
|
$> s3cmd ls -r s3://tests3fs
|
||||||
$> mkdir /mnt/tests3fs/dir1
|
|
||||||
$> touch /mnt/tests3fs/dir1/file3
|
|
||||||
|
|
||||||
Now, use s3cmd to show what is in CloudServer:
|
2017-02-28 17:28 0 s3://tests3fs/dir1/
|
||||||
|
2017-02-28 17:29 0 s3://tests3fs/dir1/file3
|
||||||
|
2017-02-28 17:28 0 s3://tests3fs/file1
|
||||||
|
2017-02-28 17:28 0 s3://tests3fs/file2
|
||||||
|
|
||||||
.. code:: sh
|
Now you can enjoy a filesystem view on your local Zenko CloudServer!
|
||||||
|
|
||||||
$> s3cmd ls -r s3://tests3fs
|
|
||||||
|
|
||||||
2017-02-28 17:28 0 s3://tests3fs/dir1/
|
|
||||||
2017-02-28 17:29 0 s3://tests3fs/dir1/file3
|
|
||||||
2017-02-28 17:28 0 s3://tests3fs/file1
|
|
||||||
2017-02-28 17:28 0 s3://tests3fs/file2
|
|
||||||
|
|
||||||
Now you can enjoy a filesystem view on your local CloudServer.
|
|
||||||
|
|
||||||
|
|
||||||
Duplicity
|
Duplicity
|
||||||
=========
|
=========
|
||||||
|
|
||||||
How to back up your files with CloudServer.
|
How to backup your files with Zenko CloudServer.
|
||||||
|
|
||||||
Installing Duplicity and its Dependencies
|
Installing
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Installing Duplicity and its dependencies
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
To install `Duplicity <http://duplicity.nongnu.org/>`__,
|
Second, you want to install
|
||||||
go to `this site <https://code.launchpad.net/duplicity/0.7-series>`__.
|
`Duplicity <http://duplicity.nongnu.org/index.html>`__. You have to
|
||||||
Download the latest tarball. Decompress it and follow the instructions
|
download `this
|
||||||
in the README.
|
tarball <https://code.launchpad.net/duplicity/0.7-series/0.7.11/+download/duplicity-0.7.11.tar.gz>`__,
|
||||||
|
decompress it, and then checkout the README inside, which will give you
|
||||||
.. code:: sh
|
a list of dependencies to install. If you're using Ubuntu 14.04, this is
|
||||||
|
your lucky day: here is a lazy step by step install.
|
||||||
$> tar zxvf duplicity-0.7.11.tar.gz
|
|
||||||
$> cd duplicity-0.7.11
|
|
||||||
$> python setup.py install
|
|
||||||
|
|
||||||
You may receive error messages indicating the need to install some or all
|
|
||||||
of the following dependencies:
|
|
||||||
|
|
||||||
.. code:: sh
|
.. code:: sh
|
||||||
|
|
||||||
|
@ -420,20 +484,30 @@ of the following dependencies:
|
||||||
$> apt-get install python-dev python-pip python-lockfile
|
$> apt-get install python-dev python-pip python-lockfile
|
||||||
$> pip install -U boto
|
$> pip install -U boto
|
||||||
|
|
||||||
Testing the Installation
|
Then you want to actually install Duplicity:
|
||||||
------------------------
|
|
||||||
|
|
||||||
1. Check that CloudServer is running. Run ``$> docker ps``. You should
|
.. code:: sh
|
||||||
see one container named ``scality/cloudserver``. If you do not, run
|
|
||||||
``$> docker start cloudserver`` and check again.
|
|
||||||
|
|
||||||
|
$> tar zxvf duplicity-0.7.11.tar.gz
|
||||||
|
$> cd duplicity-0.7.11
|
||||||
|
$> python setup.py install
|
||||||
|
|
||||||
2. Duplicity uses a module called “Boto” to send requests to S3. Boto
|
Using
|
||||||
requires a configuration file located in ``/etc/boto.cfg`` to store
|
------
|
||||||
your credentials and preferences. A minimal configuration
|
|
||||||
you can fine tune `following these instructions
|
Testing your installation
|
||||||
<http://boto.cloudhackers.com/en/latest/getting_started.html>`__ is
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
shown here:
|
|
||||||
|
First, we're just going to quickly check that Zenko CloudServer is actually
|
||||||
|
running. To do so, simply run ``$> docker ps`` . You should see one
|
||||||
|
container named ``scality/s3server``. If that is not the case, try
|
||||||
|
``$> docker start s3server``, and check again.
|
||||||
|
|
||||||
|
Secondly, as you probably know, Duplicity uses a module called **Boto**
|
||||||
|
to send requests to S3. Boto requires a configuration file located in
|
||||||
|
**``/etc/boto.cfg``** to have your credentials and preferences. Here is
|
||||||
|
a minimalistic config `that you can finetune following these
|
||||||
|
instructions <http://boto.cloudhackers.com/en/latest/getting_started.html>`__.
|
||||||
|
|
||||||
::
|
::
|
||||||
|
|
||||||
|
@ -447,51 +521,54 @@ Testing the Installation
|
||||||
# If using SSL, unmute and provide absolute path to local CA certificate
|
# If using SSL, unmute and provide absolute path to local CA certificate
|
||||||
# ca_certificates_file = /absolute/path/to/ca.crt
|
# ca_certificates_file = /absolute/path/to/ca.crt
|
||||||
|
|
||||||
.. note:: To set up SSL with CloudServer, check out our `Using SSL
|
*Note:* *If you want to set up SSL with Zenko CloudServer, check out our
|
||||||
<./GETTING_STARTED#Using_SSL>`__ in GETTING STARTED.
|
`tutorial <http://link/to/SSL/tutorial>`__*
|
||||||
|
|
||||||
3. At this point all requirements to run CloudServer as a backend to Duplicity
|
At this point, we've met all the requirements to start running Zenko CloudServer
|
||||||
have been met. A local folder/file should back up to the local S3.
|
as a backend to Duplicity. So we should be able to back up a local
|
||||||
Try it with the decompressed Duplicity folder:
|
folder/file to local S3. Let's try with the duplicity decompressed
|
||||||
|
folder:
|
||||||
|
|
||||||
.. code:: sh
|
.. code:: sh
|
||||||
|
|
||||||
$> duplicity duplicity-0.7.11 "s3://127.0.0.1:8000/testbucket/"
|
$> duplicity duplicity-0.7.11 "s3://127.0.0.1:8000/testbucket/"
|
||||||
|
|
||||||
.. note:: Duplicity will prompt for a symmetric encryption passphrase.
|
*Note:* *Duplicity will prompt you for a symmetric encryption
|
||||||
Save it carefully, as you will need it to recover your data.
|
passphrase. Save it somewhere as you will need it to recover your
|
||||||
Alternatively, you can add the ``--no-encryption`` flag
|
data. Alternatively, you can also add the ``--no-encryption`` flag
|
||||||
and the data will be stored plain.
|
and the data will be stored plain.*
|
||||||
|
|
||||||
If this command is successful, you will receive an output resembling:
|
If this command is succesful, you will get an output looking like this:
|
||||||
|
|
||||||
.. code:: sh
|
::
|
||||||
|
|
||||||
--------------[ Backup Statistics ]--------------
|
--------------[ Backup Statistics ]--------------
|
||||||
StartTime 1486486547.13 (Tue Feb 7 16:55:47 2017)
|
StartTime 1486486547.13 (Tue Feb 7 16:55:47 2017)
|
||||||
EndTime 1486486547.40 (Tue Feb 7 16:55:47 2017)
|
EndTime 1486486547.40 (Tue Feb 7 16:55:47 2017)
|
||||||
ElapsedTime 0.27 (0.27 seconds)
|
ElapsedTime 0.27 (0.27 seconds)
|
||||||
SourceFiles 388
|
SourceFiles 388
|
||||||
SourceFileSize 6634529 (6.33 MB)
|
SourceFileSize 6634529 (6.33 MB)
|
||||||
NewFiles 388
|
NewFiles 388
|
||||||
NewFileSize 6634529 (6.33 MB)
|
NewFileSize 6634529 (6.33 MB)
|
||||||
DeletedFiles 0
|
DeletedFiles 0
|
||||||
ChangedFiles 0
|
ChangedFiles 0
|
||||||
ChangedFileSize 0 (0 bytes)
|
ChangedFileSize 0 (0 bytes)
|
||||||
ChangedDeltaSize 0 (0 bytes)
|
ChangedDeltaSize 0 (0 bytes)
|
||||||
DeltaEntries 388
|
DeltaEntries 388
|
||||||
RawDeltaSize 6392865 (6.10 MB)
|
RawDeltaSize 6392865 (6.10 MB)
|
||||||
TotalDestinationSizeChange 2003677 (1.91 MB)
|
TotalDestinationSizeChange 2003677 (1.91 MB)
|
||||||
Errors 0
|
Errors 0
|
||||||
-------------------------------------------------
|
-------------------------------------------------
|
||||||
|
|
||||||
Congratulations! You can now back up to your local S3 through Duplicity.
|
Congratulations! You can now backup to your local S3 through duplicity
|
||||||
|
:)
|
||||||
|
|
||||||
Automating Backups
|
Automating backups
|
||||||
------------------
|
~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
The easiest way to back up files periodically is to write a bash script
|
Now you probably want to back up your files periodically. The easiest
|
||||||
and add it to your crontab. A suggested script follows.
|
way to do this is to write a bash script and add it to your crontab.
|
||||||
|
Here is my suggestion for such a file:
|
||||||
|
|
||||||
.. code:: sh
|
.. code:: sh
|
||||||
|
|
||||||
|
@ -500,33 +577,33 @@ and add it to your crontab. A suggested script follows.
|
||||||
# Export your passphrase so you don't have to type anything
|
# Export your passphrase so you don't have to type anything
|
||||||
export PASSPHRASE="mypassphrase"
|
export PASSPHRASE="mypassphrase"
|
||||||
|
|
||||||
# To use a GPG key, put it here and uncomment the line below
|
# If you want to use a GPG Key, put it here and unmute the line below
|
||||||
#GPG_KEY=
|
#GPG_KEY=
|
||||||
|
|
||||||
# Define your backup bucket, with localhost specified
|
# Define your backup bucket, with localhost specified
|
||||||
DEST="s3://127.0.0.1:8000/testbucketcloudserver/"
|
DEST="s3://127.0.0.1:8000/testbuckets3server/"
|
||||||
|
|
||||||
# Define the absolute path to the folder to back up
|
# Define the absolute path to the folder you want to backup
|
||||||
SOURCE=/root/testfolder
|
SOURCE=/root/testfolder
|
||||||
|
|
||||||
# Set to "full" for full backups, and "incremental" for incremental backups
|
# Set to "full" for full backups, and "incremental" for incremental backups
|
||||||
# Warning: you must perform one full backup befor you can perform
|
# Warning: you have to perform one full backup befor you can perform
|
||||||
# incremental ones on top of it
|
# incremental ones on top of it
|
||||||
FULL=incremental
|
FULL=incremental
|
||||||
|
|
||||||
# How long to keep backups. If you don't want to delete old backups, keep
|
# How long to keep backups for; if you don't want to delete old
|
||||||
# this value empty; otherwise, the syntax is "1Y" for one year, "1M" for
|
# backups, keep empty; otherwise, syntax is "1Y" for one year, "1M"
|
||||||
# one month, "1D" for one day.
|
# for one month, "1D" for one day
|
||||||
OLDER_THAN="1Y"
|
OLDER_THAN="1Y"
|
||||||
|
|
||||||
# is_running checks whether Duplicity is currently completing a task
|
# is_running checks whether duplicity is currently completing a task
|
||||||
is_running=$(ps -ef | grep duplicity | grep python | wc -l)
|
is_running=$(ps -ef | grep duplicity | grep python | wc -l)
|
||||||
|
|
||||||
# If Duplicity is already completing a task, this will not run
|
# If duplicity is already completing a task, this will simply not run
|
||||||
if [ $is_running -eq 0 ]; then
|
if [ $is_running -eq 0 ]; then
|
||||||
echo "Backup for ${SOURCE} started"
|
echo "Backup for ${SOURCE} started"
|
||||||
|
|
||||||
# To delete backups older than a certain time, do it here
|
# If you want to delete backups older than a certain time, we do it here
|
||||||
if [ "$OLDER_THAN" != "" ]; then
|
if [ "$OLDER_THAN" != "" ]; then
|
||||||
echo "Removing backups older than ${OLDER_THAN}"
|
echo "Removing backups older than ${OLDER_THAN}"
|
||||||
duplicity remove-older-than ${OLDER_THAN} ${DEST}
|
duplicity remove-older-than ${OLDER_THAN} ${DEST}
|
||||||
|
@ -549,17 +626,17 @@ and add it to your crontab. A suggested script follows.
|
||||||
# Forget the passphrase...
|
# Forget the passphrase...
|
||||||
unset PASSPHRASE
|
unset PASSPHRASE
|
||||||
|
|
||||||
Put this file in ``/usr/local/sbin/backup.sh``. Run ``crontab -e`` and
|
So let's say you put this file in ``/usr/local/sbin/backup.sh.`` Next
|
||||||
paste your configuration into the file that opens. If you're unfamiliar
|
you want to run ``crontab -e`` and paste your configuration in the file
|
||||||
with Cron, here is a good `HowTo
|
that opens. If you're unfamiliar with Cron, here is a good `How
|
||||||
<https://help.ubuntu.com/community/CronHowto>`__. If the folder being
|
To <https://help.ubuntu.com/community/CronHowto>`__. The folder I'm
|
||||||
backed up is a folder to be modified permanently during the work day,
|
backing up is a folder I modify permanently during my workday, so I want
|
||||||
we can set incremental backups every 5 minutes from 8 AM to 9 PM Monday
|
incremental backups every 5mn from 8AM to 9PM monday to friday. Here is
|
||||||
through Friday by pasting the following line into crontab:
|
the line I will paste in my crontab:
|
||||||
|
|
||||||
.. code:: sh
|
.. code:: cron
|
||||||
|
|
||||||
*/5 8-20 * * 1-5 /usr/local/sbin/backup.sh
|
*/5 8-20 * * 1-5 /usr/local/sbin/backup.sh
|
||||||
|
|
||||||
Adding or removing files from the folder being backed up will result in
|
Now I can try and add / remove files from the folder I'm backing up, and
|
||||||
incremental backups in the bucket.
|
I will see incremental backups in my bucket.
|
||||||
|
|
|
@ -1,263 +0,0 @@
|
||||||
Metadata Search Documentation
|
|
||||||
=============================
|
|
||||||
|
|
||||||
Description
|
|
||||||
-----------
|
|
||||||
|
|
||||||
This feature enables metadata search to be performed on the metadata of objects
|
|
||||||
stored in Zenko.
|
|
||||||
|
|
||||||
Requirements
|
|
||||||
------------
|
|
||||||
|
|
||||||
* MongoDB
|
|
||||||
|
|
||||||
Design
|
|
||||||
------
|
|
||||||
|
|
||||||
The Metadata Search feature expands on the existing :code:`GET Bucket` S3 API by
|
|
||||||
enabling users to conduct metadata searches by adding the custom Zenko query
|
|
||||||
string parameter, :code:`search`. The :code:`search` parameter is structured as a pseudo
|
|
||||||
SQL WHERE clause, and supports basic SQL operators. For example:
|
|
||||||
:code:`"A=1 AND B=2 OR C=3"` (complex queries can be built using nesting
|
|
||||||
operators, :code:`(` and :code:`)`).
|
|
||||||
|
|
||||||
The search process is as follows:
|
|
||||||
|
|
||||||
* Zenko receives a :code:`GET` request.
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
# regular getBucket request
|
|
||||||
GET /bucketname HTTP/1.1
|
|
||||||
Host: 127.0.0.1:8000
|
|
||||||
Date: Wed, 18 Oct 2018 17:50:00 GMT
|
|
||||||
Authorization: authorization string
|
|
||||||
|
|
||||||
# getBucket versions request
|
|
||||||
GET /bucketname?versions HTTP/1.1
|
|
||||||
Host: 127.0.0.1:8000
|
|
||||||
Date: Wed, 18 Oct 2018 17:50:00 GMT
|
|
||||||
Authorization: authorization string
|
|
||||||
|
|
||||||
# search getBucket request
|
|
||||||
GET /bucketname?search=key%3Dsearch-item HTTP/1.1
|
|
||||||
Host: 127.0.0.1:8000
|
|
||||||
Date: Wed, 18 Oct 2018 17:50:00 GMT
|
|
||||||
Authorization: authorization string
|
|
||||||
|
|
||||||
* If the request does *not* contain the :code:`search` query parameter, Zenko performs
|
|
||||||
a normal bucket listing and returns an XML result containing the list of
|
|
||||||
objects.
|
|
||||||
* If the request *does* contain the :code:`search` query parameter, Zenko parses and
|
|
||||||
validates the search string.
|
|
||||||
|
|
||||||
- If the search string is invalid, Zenko returns an :code:`InvalidArgument` error.
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
<?xml version=\"1.0\" encoding=\"UTF-8\"?>
|
|
||||||
<Error>
|
|
||||||
<Code>InvalidArgument</Code>
|
|
||||||
<Message>Invalid sql where clause sent as search query</Message>
|
|
||||||
<Resource></Resource>
|
|
||||||
<RequestId>d1d6afc64345a8e1198e</RequestId>
|
|
||||||
</Error>
|
|
||||||
|
|
||||||
- If the search string is valid, Zenko parses it and generates an abstract
|
|
||||||
syntax tree (AST). The AST is then passed to the MongoDB backend to be
|
|
||||||
used as the query filter for retrieving objects from a bucket that
|
|
||||||
satisfies the requested search conditions. Zenko parses the filtered
|
|
||||||
results and returns them as the response.
|
|
||||||
|
|
||||||
Metadata search results have the same structure as a :code:`GET Bucket` response:
|
|
||||||
|
|
||||||
.. code:: xml
|
|
||||||
|
|
||||||
<?xml version="1.0" encoding="UTF-8"?>
|
|
||||||
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
|
||||||
<Name>bucketname</Name>
|
|
||||||
<Prefix/>
|
|
||||||
<Marker/>
|
|
||||||
<MaxKeys>1000</MaxKeys>
|
|
||||||
<IsTruncated>false</IsTruncated>
|
|
||||||
<Contents>
|
|
||||||
<Key>objectKey</Key>
|
|
||||||
<LastModified>2018-04-19T18:31:49.426Z</LastModified>
|
|
||||||
<ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag>
|
|
||||||
<Size>0</Size>
|
|
||||||
<Owner>
|
|
||||||
<ID>79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be</ID>
|
|
||||||
<DisplayName>Bart</DisplayName>
|
|
||||||
</Owner>
|
|
||||||
<StorageClass>STANDARD</StorageClass>
|
|
||||||
</Contents>
|
|
||||||
<Contents>
|
|
||||||
...
|
|
||||||
</Contents>
|
|
||||||
</ListBucketResult>
|
|
||||||
|
|
||||||
Performing Metadata Searches with Zenko
|
|
||||||
---------------------------------------
|
|
||||||
|
|
||||||
You can perform metadata searches by:
|
|
||||||
|
|
||||||
+ Using the :code:`search_bucket` tool in the
|
|
||||||
`Scality/S3 <https://github.com/scality/S3>`_ GitHub repository.
|
|
||||||
+ Creating a signed HTTP request to Zenko in your preferred programming
|
|
||||||
language.
|
|
||||||
|
|
||||||
Using the S3 Tool
|
|
||||||
+++++++++++++++++
|
|
||||||
|
|
||||||
After cloning the `Scality/S3 <https://github.com/scality/S3>`_ GitHub repository
|
|
||||||
and installing the necessary dependencies, run the following command in the S3
|
|
||||||
project’s root directory to access the search tool:
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
node bin/search_bucket
|
|
||||||
|
|
||||||
This generates the following output:
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
Usage: search_bucket [options]
|
|
||||||
|
|
||||||
Options:
|
|
||||||
|
|
||||||
-V, --version output the version number
|
|
||||||
-a, --access-key <accessKey> Access key id
|
|
||||||
-k, --secret-key <secretKey> Secret access key
|
|
||||||
-b, --bucket <bucket> Name of the bucket
|
|
||||||
-q, --query <query> Search query
|
|
||||||
-h, --host <host> Host of the server
|
|
||||||
-p, --port <port> Port of the server
|
|
||||||
-s --ssl
|
|
||||||
-v, --verbose
|
|
||||||
-h, --help output usage information
|
|
||||||
|
|
||||||
In the following examples, Zenko Server is accessible on endpoint
|
|
||||||
:code:`http://127.0.0.1:8000` and contains the bucket :code:`zenkobucket`.
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
# search for objects with metadata "blue"
|
|
||||||
node bin/search_bucket -a accessKey1 -k verySecretKey1 -b zenkobucket \
|
|
||||||
-q "x-amz-meta-color=blue" -h 127.0.0.1 -p 8000
|
|
||||||
|
|
||||||
# search for objects tagged with "type=color"
|
|
||||||
node bin/search_bucket -a accessKey1 -k verySecretKey1 -b zenkobucket \
|
|
||||||
-q "tags.type=color" -h 127.0.0.1 -p 8000
|
|
||||||
|
|
||||||
Coding Examples
|
|
||||||
+++++++++++++++
|
|
||||||
|
|
||||||
Search requests can be also performed by making HTTP requests authenticated
|
|
||||||
with one of the AWS Signature schemes: version 2 or version 4. \
|
|
||||||
For more about authentication scheme, see:
|
|
||||||
|
|
||||||
* https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
|
|
||||||
* http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
|
|
||||||
* http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
|
|
||||||
|
|
||||||
You can also view examples for making requests with Auth V4 in various
|
|
||||||
languages `here <../../../examples>`__.
|
|
||||||
|
|
||||||
Specifying Metadata Fields
|
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
||||||
|
|
||||||
To search system metadata headers:
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
{system-metadata-key}{supported SQL op}{search value}
|
|
||||||
# example
|
|
||||||
key = blueObject
|
|
||||||
size > 0
|
|
||||||
key LIKE "blue.*"
|
|
||||||
|
|
||||||
To search custom user metadata:
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
# metadata must be prefixed with "x-amz-meta-"
|
|
||||||
x-amz-meta-{user-metadata-key}{supported SQL op}{search value}
|
|
||||||
|
|
||||||
# example
|
|
||||||
x-amz-meta-color = blue
|
|
||||||
x-amz-meta-color != red
|
|
||||||
x-amz-meta-color LIKE "b.*"
|
|
||||||
|
|
||||||
To search tags:
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
# tag searches must be prefixed with "tags."
|
|
||||||
tags.{tag-key}{supported SQL op}{search value}
|
|
||||||
# example
|
|
||||||
tags.type = color
|
|
||||||
|
|
||||||
Examples queries:
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
# searching for objects with custom metadata "color"=blue" and are tagged
|
|
||||||
# "type"="color"
|
|
||||||
|
|
||||||
tags.type="color" AND x-amz-meta-color="blue"
|
|
||||||
|
|
||||||
# searching for objects with the object key containing the substring "blue"
|
|
||||||
# or (custom metadata "color"=blue" and are tagged "type"="color")
|
|
||||||
|
|
||||||
key LIKE '.*blue.*' OR (x-amz-meta-color="blue" AND tags.type="color")
|
|
||||||
|
|
||||||
Differences from SQL
|
|
||||||
++++++++++++++++++++
|
|
||||||
|
|
||||||
Zenko metadata search queries are similar to SQL-query :code:`WHERE` clauses, but
|
|
||||||
differ in that:
|
|
||||||
|
|
||||||
* They follow the :code:`PCRE` format
|
|
||||||
* They do not require values with hyphens to be enclosed in
|
|
||||||
backticks, :code:``(`)``
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
# SQL query
|
|
||||||
`x-amz-meta-search-item` = `ice-cream-cone`
|
|
||||||
|
|
||||||
# MD Search query
|
|
||||||
x-amz-meta-search-item = ice-cream-cone
|
|
||||||
|
|
||||||
* Search queries do not support all SQL operators.
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
# Supported SQL operators:
|
|
||||||
=, <, >, <=, >=, !=, AND, OR, LIKE, <>
|
|
||||||
|
|
||||||
# Unsupported SQL operators:
|
|
||||||
NOT, BETWEEN, IN, IS, +, -, %, ^, /, *, !
|
|
||||||
|
|
||||||
Using Regular Expressions in Metadata Search
|
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
||||||
|
|
||||||
Regular expressions in Zenko metadata search differ from SQL in the following
|
|
||||||
ways:
|
|
||||||
|
|
||||||
+ Wildcards are represented with :code:`.*` instead of :code:`%`.
|
|
||||||
+ Regex patterns must be wrapped in quotes. Failure to do this can lead to
|
|
||||||
misinterpretation of patterns.
|
|
||||||
+ As with :code:`PCRE`, regular expressions can be entered in either the
|
|
||||||
:code:`/pattern/` syntax or as the pattern itself if regex options are
|
|
||||||
not required.
|
|
||||||
|
|
||||||
Example regular expressions:
|
|
||||||
|
|
||||||
.. code::
|
|
||||||
|
|
||||||
# search for strings containing word substring "helloworld"
|
|
||||||
".*helloworld.*"
|
|
||||||
"/.*helloworld.*/"
|
|
||||||
"/.*helloworld.*/i"
|
|
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue