Compare commits
1 Commits
developmen
...
improvemen
Author | SHA1 | Date |
---|---|---|
Rahul Padigela | b04f88b222 |
|
@ -1,9 +0,0 @@
|
|||
node_modules
|
||||
localData/*
|
||||
localMetadata/*
|
||||
# Keep the .git/HEAD file in order to properly report version
|
||||
.git/objects
|
||||
.github
|
||||
.tox
|
||||
coverage
|
||||
.DS_Store
|
55
.eslintrc
55
.eslintrc
|
@ -1,54 +1 @@
|
|||
{
|
||||
"extends": "scality",
|
||||
"plugins": [
|
||||
"mocha"
|
||||
],
|
||||
"rules": {
|
||||
"import/extensions": "off",
|
||||
"lines-around-directive": "off",
|
||||
"no-underscore-dangle": "off",
|
||||
"indent": "off",
|
||||
"object-curly-newline": "off",
|
||||
"operator-linebreak": "off",
|
||||
"function-paren-newline": "off",
|
||||
"import/newline-after-import": "off",
|
||||
"prefer-destructuring": "off",
|
||||
"implicit-arrow-linebreak": "off",
|
||||
"no-bitwise": "off",
|
||||
"dot-location": "off",
|
||||
"comma-dangle": "off",
|
||||
"no-undef-init": "off",
|
||||
"global-require": "off",
|
||||
"import/no-dynamic-require": "off",
|
||||
"class-methods-use-this": "off",
|
||||
"no-plusplus": "off",
|
||||
"no-else-return": "off",
|
||||
"object-property-newline": "off",
|
||||
"import/order": "off",
|
||||
"no-continue": "off",
|
||||
"no-tabs": "off",
|
||||
"lines-between-class-members": "off",
|
||||
"prefer-spread": "off",
|
||||
"no-lonely-if": "off",
|
||||
"no-useless-escape": "off",
|
||||
"no-restricted-globals": "off",
|
||||
"no-buffer-constructor": "off",
|
||||
"import/no-extraneous-dependencies": "off",
|
||||
"space-unary-ops": "off",
|
||||
"no-useless-return": "off",
|
||||
"no-unexpected-multiline": "off",
|
||||
"no-mixed-operators": "off",
|
||||
"newline-per-chained-call": "off",
|
||||
"operator-assignment": "off",
|
||||
"spaced-comment": "off",
|
||||
"comma-style": "off",
|
||||
"no-restricted-properties": "off",
|
||||
"new-parens": "off",
|
||||
"no-multi-spaces": "off",
|
||||
"quote-props": "off",
|
||||
"mocha/no-exclusive-tests": "error",
|
||||
},
|
||||
"parserOptions": {
|
||||
"ecmaVersion": 2020
|
||||
}
|
||||
}
|
||||
{ "extends": "scality" }
|
||||
|
|
|
@ -1,32 +1,19 @@
|
|||
# General support information
|
||||
# Issue template
|
||||
|
||||
GitHub Issues are **reserved** for actionable bug reports (including
|
||||
documentation inaccuracies), and feature requests.
|
||||
**All questions** (regarding configuration, use cases, performance, community,
|
||||
events, setup and usage recommendations, among other things) should be asked on
|
||||
the **[Zenko Forum](http://forum.zenko.io/)**.
|
||||
If you are reporting a new issue, make sure that we do not have any
|
||||
duplicates already open. You can ensure this by searching the issue list for
|
||||
this repository. If there is a duplicate, please close your issue and add a
|
||||
comment to the existing issue instead.
|
||||
|
||||
> Questions opened as GitHub issues will systematically be closed, and moved to
|
||||
> the [Zenko Forum](http://forum.zenko.io/).
|
||||
## General support information
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
## Avoiding duplicates
|
||||
|
||||
When reporting a new issue/requesting a feature, make sure that we do not have
|
||||
any duplicates already open:
|
||||
|
||||
- search the issue list for this repository (use the search bar, select
|
||||
"Issues" on the left pane after searching);
|
||||
- if there is a duplicate, please do not open your issue, and add a comment
|
||||
to the existing issue instead.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
GitHub Issues are reserved for actionable bug reports and feature requests.
|
||||
General questions should be sent to the
|
||||
[S3 scality server Forum](http://forum.scality.com/).
|
||||
|
||||
## Bug report information
|
||||
|
||||
(delete this section (everything between the lines) if you're not reporting a bug
|
||||
but requesting a feature)
|
||||
(delete this section if not applicable)
|
||||
|
||||
### Description
|
||||
|
||||
|
@ -42,22 +29,13 @@ Describe the results you received
|
|||
|
||||
### Expected result
|
||||
|
||||
Describe the results you expected
|
||||
Describe the results you expecteds
|
||||
|
||||
### Additional information
|
||||
|
||||
- Node.js version,
|
||||
- Docker version,
|
||||
- yarn version,
|
||||
- distribution/OS,
|
||||
- optional: anything else you deem helpful to us.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
### Additional information: (Node.js version, Docker version, etc)
|
||||
|
||||
## Feature Request
|
||||
|
||||
(delete this section (everything between the lines) if you're not requesting
|
||||
a feature but reporting a bug)
|
||||
(delete this section if not applicable)
|
||||
|
||||
### Proposal
|
||||
|
||||
|
@ -74,14 +52,3 @@ What you would like to happen
|
|||
### Use case
|
||||
|
||||
Please provide use cases for changing the current behavior
|
||||
|
||||
### Additional information
|
||||
|
||||
- Is this request for your company? Y/N
|
||||
- If Y: Company name:
|
||||
- Are you using any Scality Enterprise Edition products (RING, Zenko EE)? Y/N
|
||||
- Are you willing to contribute this feature yourself?
|
||||
- Position/Title:
|
||||
- How did you hear about us?
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
|
|
@ -1,43 +0,0 @@
|
|||
---
|
||||
name: "Setup CI environment"
|
||||
description: "Setup Cloudserver CI environment"
|
||||
|
||||
runs:
|
||||
using: composite
|
||||
steps:
|
||||
- name: Setup etc/hosts
|
||||
shell: bash
|
||||
run: sudo echo "127.0.0.1 bucketwebsitetester.s3-website-us-east-1.amazonaws.com" | sudo tee -a /etc/hosts
|
||||
- name: Setup Credentials
|
||||
shell: bash
|
||||
run: bash .github/scripts/credentials.bash
|
||||
- name: Setup job artifacts directory
|
||||
shell: bash
|
||||
run: |-
|
||||
set -exu;
|
||||
mkdir -p /tmp/artifacts/${JOB_NAME}/;
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '16'
|
||||
cache: 'yarn'
|
||||
- name: install dependencies
|
||||
shell: bash
|
||||
run: yarn install --ignore-engines --frozen-lockfile --network-concurrency 1
|
||||
- uses: actions/cache@v3
|
||||
with:
|
||||
path: ~/.cache/pip
|
||||
key: ${{ runner.os }}-pip
|
||||
- uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: 3.9
|
||||
- name: Setup python2 test environment
|
||||
shell: bash
|
||||
run: |
|
||||
sudo apt-get install -y libdigest-hmac-perl
|
||||
pip install 's3cmd==2.3.0'
|
||||
- name: fix sproxyd.conf permissions
|
||||
shell: bash
|
||||
run: sudo chown root:root .github/docker/sproxyd/conf/sproxyd0.conf
|
||||
- name: ensure fuse kernel module is loaded (for sproxyd)
|
||||
shell: bash
|
||||
run: sudo modprobe fuse
|
|
@ -1,25 +0,0 @@
|
|||
FROM ceph/daemon:v3.2.1-stable-3.2-mimic-centos-7
|
||||
|
||||
ENV CEPH_DAEMON demo
|
||||
ENV CEPH_DEMO_DAEMONS mon,mgr,osd,rgw
|
||||
|
||||
ENV CEPH_DEMO_UID zenko
|
||||
ENV CEPH_DEMO_ACCESS_KEY accessKey1
|
||||
ENV CEPH_DEMO_SECRET_KEY verySecretKey1
|
||||
ENV CEPH_DEMO_BUCKET zenkobucket
|
||||
|
||||
ENV CEPH_PUBLIC_NETWORK 0.0.0.0/0
|
||||
ENV MON_IP 0.0.0.0
|
||||
ENV NETWORK_AUTO_DETECT 4
|
||||
ENV RGW_CIVETWEB_PORT 8001
|
||||
|
||||
RUN rm /etc/yum.repos.d/tcmu-runner.repo
|
||||
|
||||
ADD ./entrypoint-wrapper.sh /
|
||||
RUN chmod +x /entrypoint-wrapper.sh && \
|
||||
yum install -y python-pip && \
|
||||
yum clean all && \
|
||||
pip install awscli && \
|
||||
rm -rf /root/.cache/pip
|
||||
|
||||
ENTRYPOINT [ "/entrypoint-wrapper.sh" ]
|
|
@ -1,37 +0,0 @@
|
|||
#!/bin/sh
|
||||
|
||||
touch /artifacts/ceph.log
|
||||
mkfifo /tmp/entrypoint_output
|
||||
# We run this in the background so that we can tail the RGW log after init,
|
||||
# because entrypoint.sh never returns
|
||||
|
||||
# The next line will be needed when ceph builds 3.2.2 so I'll leave it here
|
||||
# bash /opt/ceph-container/bin/entrypoint.sh > /tmp/entrypoint_output &
|
||||
|
||||
bash /entrypoint.sh > /tmp/entrypoint_output &
|
||||
entrypoint_pid="$!"
|
||||
while read -r line; do
|
||||
echo $line
|
||||
# When we find this line server has started
|
||||
if [ -n "$(echo $line | grep 'Creating bucket')" ]; then
|
||||
break
|
||||
fi
|
||||
done < /tmp/entrypoint_output
|
||||
|
||||
# Make our buckets - CEPH_DEMO_BUCKET is set to force the "Creating bucket" message, but unused
|
||||
s3cmd mb s3://cephbucket s3://cephbucket2
|
||||
|
||||
mkdir /root/.aws
|
||||
cat > /root/.aws/credentials <<EOF
|
||||
[default]
|
||||
aws_access_key_id = accessKey1
|
||||
aws_secret_access_key = verySecretKey1
|
||||
EOF
|
||||
|
||||
# Enable versioning on them
|
||||
for bucket in cephbucket cephbucket2; do
|
||||
echo "Enabling versiong for $bucket"
|
||||
aws --endpoint http://127.0.0.1:8001 s3api put-bucket-versioning --bucket $bucket --versioning Status=Enabled
|
||||
done
|
||||
tail -f /var/log/ceph/client.rgw.*.log | tee -a /artifacts/ceph.log
|
||||
wait $entrypoint_pid
|
|
@ -1,11 +0,0 @@
|
|||
#!/bin/sh
|
||||
|
||||
# This script is needed because RADOS Gateway
|
||||
# will open the port before beginning to serve traffic
|
||||
# causing wait_for_local_port.bash to exit immediately
|
||||
|
||||
echo 'Waiting for ceph'
|
||||
while [ -z "$(curl 127.0.0.1:8001 2>/dev/null)" ]; do
|
||||
sleep 1
|
||||
echo -n "."
|
||||
done
|
|
@ -1,10 +0,0 @@
|
|||
---
|
||||
version: 2
|
||||
updates:
|
||||
- package-ecosystem: npm
|
||||
directory: "/"
|
||||
schedule:
|
||||
interval: daily
|
||||
time: "13:00"
|
||||
open-pull-requests-limit: 10
|
||||
target-branch: "development/7.4"
|
|
@ -1,36 +0,0 @@
|
|||
azurebackend_AZURE_STORAGE_ACCESS_KEY
|
||||
azurebackend_AZURE_STORAGE_ACCOUNT_NAME
|
||||
azurebackend_AZURE_STORAGE_ENDPOINT
|
||||
azurebackend2_AZURE_STORAGE_ACCESS_KEY
|
||||
azurebackend2_AZURE_STORAGE_ACCOUNT_NAME
|
||||
azurebackend2_AZURE_STORAGE_ENDPOINT
|
||||
azurebackendmismatch_AZURE_STORAGE_ACCESS_KEY
|
||||
azurebackendmismatch_AZURE_STORAGE_ACCOUNT_NAME
|
||||
azurebackendmismatch_AZURE_STORAGE_ENDPOINT
|
||||
azurenonexistcontainer_AZURE_STORAGE_ACCESS_KEY
|
||||
azurenonexistcontainer_AZURE_STORAGE_ACCOUNT_NAME
|
||||
azurenonexistcontainer_AZURE_STORAGE_ENDPOINT
|
||||
azuretest_AZURE_BLOB_ENDPOINT
|
||||
b2backend_B2_ACCOUNT_ID
|
||||
b2backend_B2_STORAGE_ACCESS_KEY
|
||||
GOOGLE_SERVICE_EMAIL
|
||||
GOOGLE_SERVICE_KEY
|
||||
AWS_S3_BACKEND_ACCESS_KEY
|
||||
AWS_S3_BACKEND_SECRET_KEY
|
||||
AWS_S3_BACKEND_ACCESS_KEY_2
|
||||
AWS_S3_BACKEND_SECRET_KEY_2
|
||||
AWS_GCP_BACKEND_ACCESS_KEY
|
||||
AWS_GCP_BACKEND_SECRET_KEY
|
||||
AWS_GCP_BACKEND_ACCESS_KEY_2
|
||||
AWS_GCP_BACKEND_SECRET_KEY_2
|
||||
b2backend_B2_STORAGE_ENDPOINT
|
||||
gcpbackend2_GCP_SERVICE_EMAIL
|
||||
gcpbackend2_GCP_SERVICE_KEY
|
||||
gcpbackend2_GCP_SERVICE_KEYFILE
|
||||
gcpbackend_GCP_SERVICE_EMAIL
|
||||
gcpbackend_GCP_SERVICE_KEY
|
||||
gcpbackendmismatch_GCP_SERVICE_EMAIL
|
||||
gcpbackendmismatch_GCP_SERVICE_KEY
|
||||
gcpbackend_GCP_SERVICE_KEYFILE
|
||||
gcpbackendmismatch_GCP_SERVICE_KEYFILE
|
||||
gcpbackendnoproxy_GCP_SERVICE_KEYFILE
|
|
@ -1,92 +0,0 @@
|
|||
services:
|
||||
cloudserver:
|
||||
image: ${CLOUDSERVER_IMAGE}
|
||||
command: sh -c "yarn start > /artifacts/s3.log"
|
||||
network_mode: "host"
|
||||
volumes:
|
||||
- /tmp/ssl:/ssl
|
||||
- /tmp/ssl-kmip:/ssl-kmip
|
||||
- ${HOME}/.aws/credentials:/root/.aws/credentials
|
||||
- /tmp/artifacts/${JOB_NAME}:/artifacts
|
||||
environment:
|
||||
- CI=true
|
||||
- ENABLE_LOCAL_CACHE=true
|
||||
- REDIS_HOST=0.0.0.0
|
||||
- REDIS_PORT=6379
|
||||
- REPORT_TOKEN=report-token-1
|
||||
- REMOTE_MANAGEMENT_DISABLE=1
|
||||
- HEALTHCHECKS_ALLOWFROM=0.0.0.0/0
|
||||
- DATA_HOST=0.0.0.0
|
||||
- METADATA_HOST=0.0.0.0
|
||||
- S3BACKEND
|
||||
- S3DATA
|
||||
- S3METADATA
|
||||
- MPU_TESTING
|
||||
- S3VAULT
|
||||
- S3_LOCATION_FILE
|
||||
- ENABLE_UTAPI_V2
|
||||
- BUCKET_DENY_FILTER
|
||||
- S3KMS
|
||||
- S3KMIP_PORT
|
||||
- S3KMIP_HOSTS
|
||||
- S3KMIP-COMPOUND_CREATE
|
||||
- S3KMIP_BUCKET_ATTRIBUTE_NAME
|
||||
- S3KMIP_PIPELINE_DEPTH
|
||||
- S3KMIP_KEY
|
||||
- S3KMIP_CERT
|
||||
- S3KMIP_CA
|
||||
- MONGODB_HOSTS=0.0.0.0:27018
|
||||
- MONGODB_RS=rs0
|
||||
- DEFAULT_BUCKET_KEY_FORMAT
|
||||
- METADATA_MAX_CACHED_BUCKETS
|
||||
- ENABLE_NULL_VERSION_COMPAT_MODE
|
||||
- SCUBA_HOST
|
||||
- SCUBA_PORT
|
||||
- SCUBA_HEALTHCHECK_FREQUENCY
|
||||
- S3QUOTA
|
||||
- QUOTA_ENABLE_INFLIGHTS
|
||||
env_file:
|
||||
- creds.env
|
||||
depends_on:
|
||||
- redis
|
||||
extra_hosts:
|
||||
- "bucketwebsitetester.s3-website-us-east-1.amazonaws.com:127.0.0.1"
|
||||
- "pykmip.local:127.0.0.1"
|
||||
redis:
|
||||
image: redis:alpine
|
||||
network_mode: "host"
|
||||
squid:
|
||||
network_mode: "host"
|
||||
profiles: ['ci-proxy']
|
||||
image: scality/ci-squid
|
||||
command: >-
|
||||
sh -c 'mkdir -p /ssl &&
|
||||
openssl req -new -newkey rsa:2048 -sha256 -days 365 -nodes -x509 \
|
||||
-subj "/C=US/ST=Country/L=City/O=Organization/CN=CN=scality-proxy" \
|
||||
-keyout /ssl/myca.pem -out /ssl/myca.pem &&
|
||||
cp /ssl/myca.pem /ssl/CA.pem &&
|
||||
squid -f /etc/squid/squid.conf -N -z &&
|
||||
squid -f /etc/squid/squid.conf -NYCd 1'
|
||||
volumes:
|
||||
- /tmp/ssl:/ssl
|
||||
pykmip:
|
||||
network_mode: "host"
|
||||
profiles: ['pykmip']
|
||||
image: ${PYKMIP_IMAGE:-ghcr.io/scality/cloudserver/pykmip}
|
||||
volumes:
|
||||
- /tmp/artifacts/${JOB_NAME}:/artifacts
|
||||
mongo:
|
||||
network_mode: "host"
|
||||
profiles: ['mongo', 'ceph']
|
||||
image: ${MONGODB_IMAGE}
|
||||
ceph:
|
||||
network_mode: "host"
|
||||
profiles: ['ceph']
|
||||
image: ghcr.io/scality/cloudserver/ci-ceph
|
||||
sproxyd:
|
||||
network_mode: "host"
|
||||
profiles: ['sproxyd']
|
||||
image: sproxyd-standalone
|
||||
build: ./sproxyd
|
||||
user: 0:0
|
||||
privileged: yes
|
|
@ -1,28 +0,0 @@
|
|||
FROM mongo:5.0.21
|
||||
|
||||
ENV USER=scality \
|
||||
HOME_DIR=/home/scality \
|
||||
CONF_DIR=/conf \
|
||||
DATA_DIR=/data
|
||||
|
||||
# Set up directories and permissions
|
||||
RUN mkdir -p /data/db /data/configdb && chown -R mongodb:mongodb /data/db /data/configdb; \
|
||||
mkdir /logs; \
|
||||
adduser --uid 1000 --disabled-password --gecos --quiet --shell /bin/bash scality
|
||||
|
||||
# Set up environment variables and directories for scality user
|
||||
RUN mkdir ${CONF_DIR} && \
|
||||
chown -R ${USER} ${CONF_DIR} && \
|
||||
chown -R ${USER} ${DATA_DIR}
|
||||
|
||||
# copy the mongo config file
|
||||
COPY /conf/mongod.conf /conf/mongod.conf
|
||||
COPY /conf/mongo-run.sh /conf/mongo-run.sh
|
||||
COPY /conf/initReplicaSet /conf/initReplicaSet.js
|
||||
|
||||
EXPOSE 27017/tcp
|
||||
EXPOSE 27018
|
||||
|
||||
# Set up CMD
|
||||
ENTRYPOINT ["bash", "/conf/mongo-run.sh"]
|
||||
CMD ["bash", "/conf/mongo-run.sh"]
|
|
@ -1,4 +0,0 @@
|
|||
rs.initiate({
|
||||
_id: "rs0",
|
||||
members: [{ _id: 0, host: "127.0.0.1:27018" }]
|
||||
});
|
|
@ -1,10 +0,0 @@
|
|||
#!/bin/bash
|
||||
set -exo pipefail
|
||||
|
||||
init_RS() {
|
||||
sleep 5
|
||||
mongo --port 27018 /conf/initReplicaSet.js
|
||||
}
|
||||
init_RS &
|
||||
|
||||
mongod --bind_ip_all --config=/conf/mongod.conf
|
|
@ -1,15 +0,0 @@
|
|||
storage:
|
||||
journal:
|
||||
enabled: true
|
||||
engine: wiredTiger
|
||||
dbPath: "/data/db"
|
||||
processManagement:
|
||||
fork: false
|
||||
net:
|
||||
port: 27018
|
||||
bindIp: 0.0.0.0
|
||||
replication:
|
||||
replSetName: "rs0"
|
||||
enableMajorityReadConcern: true
|
||||
security:
|
||||
authorization: disabled
|
|
@ -1,3 +0,0 @@
|
|||
FROM ghcr.io/scality/federation/sproxyd:7.10.6.8
|
||||
ADD ./conf/supervisord.conf ./conf/nginx.conf ./conf/fastcgi_params ./conf/sproxyd0.conf /conf/
|
||||
RUN chown root:root /conf/sproxyd0.conf
|
|
@ -1,26 +0,0 @@
|
|||
fastcgi_param QUERY_STRING $query_string;
|
||||
fastcgi_param REQUEST_METHOD $request_method;
|
||||
fastcgi_param CONTENT_TYPE $content_type;
|
||||
fastcgi_param CONTENT_LENGTH $content_length;
|
||||
|
||||
#fastcgi_param SCRIPT_NAME $fastcgi_script_name;
|
||||
fastcgi_param SCRIPT_NAME /var/www;
|
||||
fastcgi_param PATH_INFO $document_uri;
|
||||
|
||||
fastcgi_param REQUEST_URI $request_uri;
|
||||
fastcgi_param DOCUMENT_URI $document_uri;
|
||||
fastcgi_param DOCUMENT_ROOT $document_root;
|
||||
fastcgi_param SERVER_PROTOCOL $server_protocol;
|
||||
fastcgi_param HTTPS $https if_not_empty;
|
||||
|
||||
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
|
||||
fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;
|
||||
|
||||
fastcgi_param REMOTE_ADDR $remote_addr;
|
||||
fastcgi_param REMOTE_PORT $remote_port;
|
||||
fastcgi_param SERVER_ADDR $server_addr;
|
||||
fastcgi_param SERVER_PORT $server_port;
|
||||
fastcgi_param SERVER_NAME $server_name;
|
||||
|
||||
# PHP only, required if PHP was built with --enable-force-cgi-redirect
|
||||
fastcgi_param REDIRECT_STATUS 200;
|
|
@ -1,88 +0,0 @@
|
|||
worker_processes 1;
|
||||
error_log /logs/error.log;
|
||||
user root root;
|
||||
events {
|
||||
worker_connections 1000;
|
||||
reuse_port on;
|
||||
multi_accept on;
|
||||
}
|
||||
worker_rlimit_nofile 20000;
|
||||
http {
|
||||
root /var/www/;
|
||||
upstream sproxyds {
|
||||
least_conn;
|
||||
keepalive 40;
|
||||
server 127.0.0.1:20000;
|
||||
}
|
||||
server {
|
||||
client_max_body_size 0;
|
||||
client_body_timeout 150;
|
||||
client_header_timeout 150;
|
||||
postpone_output 0;
|
||||
client_body_postpone_size 0;
|
||||
keepalive_requests 1100;
|
||||
keepalive_timeout 300s;
|
||||
server_tokens off;
|
||||
default_type application/octet-stream;
|
||||
gzip off;
|
||||
tcp_nodelay on;
|
||||
tcp_nopush on;
|
||||
sendfile on;
|
||||
listen 81;
|
||||
server_name localhost;
|
||||
rewrite ^/arc/(.*)$ /dc1/$1 permanent;
|
||||
location ~* ^/proxy/(.*)$ {
|
||||
rewrite ^/proxy/(.*)$ /$1 last;
|
||||
}
|
||||
allow 127.0.0.1;
|
||||
|
||||
deny all;
|
||||
set $usermd '-';
|
||||
set $sentusermd '-';
|
||||
set $elapsed_ms '-';
|
||||
set $now '-';
|
||||
log_by_lua '
|
||||
if not(ngx.var.http_x_scal_usermd == nil) and string.len(ngx.var.http_x_scal_usermd) > 2 then
|
||||
ngx.var.usermd = string.sub(ngx.decode_base64(ngx.var.http_x_scal_usermd),1,-3)
|
||||
end
|
||||
if not(ngx.var.sent_http_x_scal_usermd == nil) and string.len(ngx.var.sent_http_x_scal_usermd) > 2 then
|
||||
ngx.var.sentusermd = string.sub(ngx.decode_base64(ngx.var.sent_http_x_scal_usermd),1,-3)
|
||||
end
|
||||
local elapsed_ms = tonumber(ngx.var.request_time)
|
||||
if not ( elapsed_ms == nil) then
|
||||
elapsed_ms = elapsed_ms * 1000
|
||||
ngx.var.elapsed_ms = tostring(elapsed_ms)
|
||||
end
|
||||
local time = tonumber(ngx.var.msec) * 1000
|
||||
ngx.var.now = time
|
||||
';
|
||||
log_format irm '{ "time":"$now","connection":"$connection","request":"$connection_requests","hrtime":"$msec",'
|
||||
'"httpMethod":"$request_method","httpURL":"$uri","elapsed_ms":$elapsed_ms,'
|
||||
'"httpCode":$status,"requestLength":$request_length,"bytesSent":$bytes_sent,'
|
||||
'"contentLength":"$content_length","sentContentLength":"$sent_http_content_length",'
|
||||
'"contentType":"$content_type","s3Address":"$remote_addr",'
|
||||
'"requestUserMd":"$usermd","responseUserMd":"$sentusermd",'
|
||||
'"ringKeyVersion":"$sent_http_x_scal_version","ringStatus":"$sent_http_x_scal_ring_status",'
|
||||
'"s3Port":"$remote_port","sproxydStatus":"$upstream_status","req_id":"$http_x_scal_request_uids",'
|
||||
'"ifMatch":"$http_if_match","ifNoneMatch":"$http_if_none_match",'
|
||||
'"range":"$http_range","contentRange":"$sent_http_content_range","nginxPID":$PID,'
|
||||
'"sproxydAddress":"$upstream_addr","sproxydResponseTime_s":"$upstream_response_time" }';
|
||||
access_log /dev/stdout irm;
|
||||
error_log /dev/stdout error;
|
||||
location / {
|
||||
proxy_request_buffering off;
|
||||
fastcgi_request_buffering off;
|
||||
fastcgi_no_cache 1;
|
||||
fastcgi_cache_bypass 1;
|
||||
fastcgi_buffering off;
|
||||
fastcgi_ignore_client_abort on;
|
||||
fastcgi_keep_conn on;
|
||||
include fastcgi_params;
|
||||
fastcgi_pass sproxyds;
|
||||
fastcgi_next_upstream error timeout;
|
||||
fastcgi_send_timeout 285s;
|
||||
fastcgi_read_timeout 285s;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
{
|
||||
"general": {
|
||||
"ring": "DATA",
|
||||
"port": 20000,
|
||||
"syslog_facility": "local0"
|
||||
},
|
||||
"ring_driver:0": {
|
||||
"alias": "dc1",
|
||||
"type": "local",
|
||||
"queue_path": "/tmp/ring-objs"
|
||||
},
|
||||
}
|
|
@ -1,43 +0,0 @@
|
|||
[supervisord]
|
||||
nodaemon = true
|
||||
loglevel = info
|
||||
logfile = %(ENV_LOG_DIR)s/supervisord.log
|
||||
pidfile = %(ENV_SUP_RUN_DIR)s/supervisord.pid
|
||||
logfile_maxbytes = 20MB
|
||||
logfile_backups = 2
|
||||
|
||||
[unix_http_server]
|
||||
file = %(ENV_SUP_RUN_DIR)s/supervisor.sock
|
||||
|
||||
[rpcinterface:supervisor]
|
||||
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
|
||||
|
||||
[supervisorctl]
|
||||
serverurl = unix://%(ENV_SUP_RUN_DIR)s/supervisor.sock
|
||||
|
||||
[program:nginx]
|
||||
directory=%(ENV_SUP_RUN_DIR)s
|
||||
command=bash -c "/usr/sbin/nginx -c %(ENV_CONF_DIR)s/nginx.conf -g 'daemon off;'"
|
||||
stdout_logfile = %(ENV_LOG_DIR)s/%(program_name)s-%(process_num)s.log
|
||||
stderr_logfile = %(ENV_LOG_DIR)s/%(program_name)s-%(process_num)s-stderr.log
|
||||
stdout_logfile_maxbytes=100MB
|
||||
stdout_logfile_backups=7
|
||||
stderr_logfile_maxbytes=100MB
|
||||
stderr_logfile_backups=7
|
||||
autorestart=true
|
||||
autostart=true
|
||||
user=root
|
||||
|
||||
[program:sproxyd]
|
||||
directory=%(ENV_SUP_RUN_DIR)s
|
||||
process_name=%(program_name)s-%(process_num)s
|
||||
numprocs=1
|
||||
numprocs_start=0
|
||||
command=/usr/bin/sproxyd -dlw -V127 -c %(ENV_CONF_DIR)s/sproxyd%(process_num)s.conf -P /run%(process_num)s
|
||||
stdout_logfile = %(ENV_LOG_DIR)s/%(program_name)s-%(process_num)s.log
|
||||
stdout_logfile_maxbytes=100MB
|
||||
stdout_logfile_backups=7
|
||||
redirect_stderr=true
|
||||
autorestart=true
|
||||
autostart=true
|
||||
user=root
|
|
@ -1,18 +0,0 @@
|
|||
-----BEGIN CERTIFICATE-----
|
||||
MIIC6zCCAdOgAwIBAgIUPIpMY95b4HjKAk+FyydZApAEFskwDQYJKoZIhvcNAQEL
|
||||
BQAwJDEQMA4GA1UECgwHU2NhbGl0eTEQMA4GA1UEAwwHUm9vdCBDQTAgFw0yMTA0
|
||||
MDkwMDI4MTFaGA8yMTIxMDMxNjAwMjgxMVowJDEQMA4GA1UECgwHU2NhbGl0eTEQ
|
||||
MA4GA1UEAwwHUm9vdCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
|
||||
AKqLFEsWtfRTxnoZrQe63tq+rQnVgninHMahRmXkzyjK/uNhoKnIh8bXdTC/eCZ6
|
||||
FBROqBYNL0TJb0HDv1FzcZS1UCUldRqTlvr6wZb0pfrp40fvztsqQgAh1t/Blg5i
|
||||
Zv5+ESSlNs5rWbFTxtq+FbMW/ERYTrVfnMkBiLg4Gq0HwID9a5jvJatzrrno2s1m
|
||||
OfZCT3HaE3tMZ6vvYuoamvLNdvdH+9KeTmBCursfNejt0rSGjIqfi6DvFJSayydQ
|
||||
is5DMSTbCLGdKQmA85VfEQmlQ8v0232WDSd6gVfp2tthDEDHnCbgWkEd1vsTyS85
|
||||
ubdt5v4CWGOWV+mu3bf8xM0CAwEAAaMTMBEwDwYDVR0TAQH/BAUwAwEB/zANBgkq
|
||||
hkiG9w0BAQsFAAOCAQEARTjc2zV/ol1/LsSzZy6l1R0uFBmR2KumH+Se1Yq2vKpY
|
||||
Dv6xmrvmjOUr5RBO77nRhIgdcQA+LyAg8ii2Dfzc8r1RTD+j1bYOxESXctBOBcXM
|
||||
Chy6FEBydR6m7S8qQyL+caJWO1WZWp2tapcm6sUG1oRVznWtK1/SHKIzOBwsmJ07
|
||||
79KsCJ6wf9tzD05EDTI2QhAObE9/thy+zc8l8cmv9A6p3jKkx9rwXUttSUqTn0CW
|
||||
w45bgKg6+DDcrhZ+MATbzuTfhuA4NFUTzK7KeX9sMuOV03Zs8SA3VhAOXmu063M3
|
||||
0f9X7P/0RmGTTp7GGCqEINcZdbLh3k7CpFb2Ox998Q==
|
||||
-----END CERTIFICATE-----
|
|
@ -1,18 +0,0 @@
|
|||
-----BEGIN CERTIFICATE-----
|
||||
MIIC2zCCAcOgAwIBAgIUIlE8UAkqQ+6mbJDtrt9kkmi8aJYwDQYJKoZIhvcNAQEL
|
||||
BQAwJDEQMA4GA1UECgwHU2NhbGl0eTEQMA4GA1UEAwwHUm9vdCBDQTAgFw0yMTA0
|
||||
MDkwMDI4MTFaGA8yMTIxMDMxNjAwMjgxMVowKTEQMA4GA1UECgwHU2NhbGl0eTEV
|
||||
MBMGA1UEAwwMcHlrbWlwLmxvY2FsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
|
||||
CgKCAQEAtxr7pq/lnzVeZz4z52Yc3DeaPqjNfRSyW5cPUlT7ABXFb7+tja7K2C7u
|
||||
DYVK+Q+2yJCQwYJY47aKJB++ewam9t2V8Xy0Z8S+0I2ImCwuyeihaD/f6uJZRzms
|
||||
ycdECH22BA6tCPlQLnlboRiZzI6rcIvXAbUMvLvFm3nyYIs9qidExRnfyMjISknM
|
||||
V+83LT5QW4IcHgKYqzdz2ZmOnk+f4wmMmitcivTdIZCL8Z0cxr7BJlOh5JZ/V5uj
|
||||
WUXeNa+ttW0RKKBlg9T+wj0JvwoJBPZTmsMAy3tI9tjLg3DwGYKsflbFeU2tebXI
|
||||
gncGFZ/dFxj331GGtq3kz1PzAUYf2wIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQB1
|
||||
8HgJ0fu6/pCrDxAm90eESFjmaTFyTN8q00zhq4Cb3zAT9KMWzAygkZ9n4ZFgELPo
|
||||
7kBE2H6RcDdoBmjVYd8HnBloDdYzYbncKgt5YBvxRaMSF4/l65BM8wjatyXErqnH
|
||||
QLLTRe5AuF0/F0KtPeDQ2JFVu8dZ35W3fyKGPRsEdVOSCTHROmqpGhZCpscyUP4W
|
||||
Hb0dBTESQ9mQHw14OCaaahARd0X5WdcA/E+m0fpGqj1rQCXS+PrRcSLe1E1hqPlK
|
||||
q/hXSXD5nybwipktELvJCbB7l4HmJr2pIpldeR5+ef68Cs8hqs6DRlsJX9sK2ng+
|
||||
TFe5v6SCarqZ9kFvr6Yp
|
||||
-----END CERTIFICATE-----
|
|
@ -1,18 +0,0 @@
|
|||
-----BEGIN CERTIFICATE-----
|
||||
MIIC8zCCAdugAwIBAgIUBs6nVXQXhrFbClub3aSLg72/DiYwDQYJKoZIhvcNAQEL
|
||||
BQAwJDEQMA4GA1UECgwHU2NhbGl0eTEQMA4GA1UEAwwHUm9vdCBDQTAgFw0yMTA0
|
||||
MDkwMDI4MTFaGA8yMTIxMDMxNjAwMjgxMVowJTEQMA4GA1UECgwHU2NhbGl0eTER
|
||||
MA8GA1UEAwwISm9obiBEb2UwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
|
||||
AQC6neSYoBoWh/i2mBpduJnTlXacpJ0iQqLezvcGy8qR0s/48mtfV2IRGTNVsq4L
|
||||
jLLRsPGt9KkJlUhHGWhG00cBGEsIiJiBUr+WrEsO04ME/Sk76kX8wk/t9Oljl7jt
|
||||
UDnQUwshj+hRFe0iKAyE65JIutu5EiiNtOqMzbVgPNfNniAaGlrgwByJaS9arzsH
|
||||
PVju9yZBYzYhwAMyYFcXUGrgvHRCHKmxBi4QmV7DX4TeN4l9TrCyEmqDev4PRFip
|
||||
yR2Fh3WGSwWh45HgMT+Jp6Uv6yI4wMXWJAcNkHdx1OhjBoUQrkavvdeVEnCwjQ+p
|
||||
SMLm0T4iNxedQWBtDM7ts4EjAgMBAAGjGjAYMBYGA1UdJQEB/wQMMAoGCCsGAQUF
|
||||
BwMCMA0GCSqGSIb3DQEBCwUAA4IBAQCMi9HEhZc5jHJMj18Wq00fZy4O9XtjCe0J
|
||||
nntW9tzi3rTQcQWKA7i9uVdDoCg+gMFVxWMvV7luFEUc/VYV1v8hFfbIFygzFsZY
|
||||
xwv4GQaIwbsgzD+oziia53w0FSuNL0uE0MeKvrt3yzHxCxylHyl+TQd/UdAtAo+k
|
||||
RL1sI0mBZx5qo6d1J7ZMCxzAGaT7KjnJvziFr/UbfSNnwDsxsUwGaI1ZeAxJN8DI
|
||||
zTrg3f3lrrmHcauEgKnuQwIqaMZR6veG6RkjtcYSlJYID1irkE6njs7+wivOAkzt
|
||||
fBt/0PD76FmAI0VArgU/zDB8dGyYzrq39W749LuEfm1TPmlnUtDr
|
||||
-----END CERTIFICATE-----
|
|
@ -1,28 +0,0 @@
|
|||
-----BEGIN PRIVATE KEY-----
|
||||
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC6neSYoBoWh/i2
|
||||
mBpduJnTlXacpJ0iQqLezvcGy8qR0s/48mtfV2IRGTNVsq4LjLLRsPGt9KkJlUhH
|
||||
GWhG00cBGEsIiJiBUr+WrEsO04ME/Sk76kX8wk/t9Oljl7jtUDnQUwshj+hRFe0i
|
||||
KAyE65JIutu5EiiNtOqMzbVgPNfNniAaGlrgwByJaS9arzsHPVju9yZBYzYhwAMy
|
||||
YFcXUGrgvHRCHKmxBi4QmV7DX4TeN4l9TrCyEmqDev4PRFipyR2Fh3WGSwWh45Hg
|
||||
MT+Jp6Uv6yI4wMXWJAcNkHdx1OhjBoUQrkavvdeVEnCwjQ+pSMLm0T4iNxedQWBt
|
||||
DM7ts4EjAgMBAAECggEANNXdUeUKXdSzcycPV/ea/c+0XFcy8e9B46lfQTpTqQOx
|
||||
xD8GbWD1L/gdk6baJgT43+ukEWdSsJbmdtLXti29Ta8OF2VtIDhIbCVtvs3dq3zt
|
||||
vrvugsiVDr8nkP306qOrKrNIVIFE+igmEmSaXsu/h/33ladxeeV9/s2DC7NOOjWN
|
||||
Mu4KYr5BBbu3qAavdzbrcz7Sch+GzsYqK/pBounCTQu3o9E4TSUcmcsasWmtHN3u
|
||||
e6G2UjObdzEW7J0wWvvtJ0wHQUVRueHfqwqKf0dymcZ3xOlx3ZPhKPz5n4F1UGUt
|
||||
RQaNazqs5SzZpUgDuPw4k8h/aCHK21Yexw/l4+O9KQKBgQD1WZSRK54zFoExBQgt
|
||||
OZSBNZW3Ibti5lSiF0M0g+66yNZSWfPuABEH0tu5CXopdPDXo4kW8NLGEqQStWTX
|
||||
RGK0DE9buEL3eebOfjIdS2IZ3t3dX3lMypplVCj4HzAgITlweSH1LLTyAtaaOpwa
|
||||
jksqfcn5Zw+XGkyc6GBBVaZetQKBgQDCt6Xf/g26+zjvHscjdzsfBhnYvTOrr6+F
|
||||
xqFFxOEOocGr+mL7UTAs+a9m/6lOWhlagk+m+TIZNL8o3IN7KFTYxPYPxTiewgVE
|
||||
rIm3JBmPxRiPn01P3HrtjaqfzsXF30j3ele7ix5OxieZq4vsW7ZXP3GZE34a08Ov
|
||||
12sE1DlvdwKBgQDzpYQOLhyqazzcqzyVfMrnDYmiFVN7QXTmiudobWRUBUIhAcdl
|
||||
oJdJB7K/rJOuO704x+RJ7dnCbZyWH6EGzZifaGIemXuXO21jvpqR0NyZCGOXhUp2
|
||||
YfS1j8AntwEZxyS9du2sBjui4gKvomiHTquChOxgSmKHEcznPTTpbN8MyQKBgF5F
|
||||
LVCZniolkLXsL7tS8VOez4qoZ0i6wP7CYLf3joJX+/z4N023S9yqcaorItvlMRsp
|
||||
tciAIyoi6F2vDRTmPNXJ3dtav4PVKVnLMs1w89MwOCjoljSQ6Q7zpGTEZenbpWbz
|
||||
W2BYBS9cLjXu4MpoyInLFINo9YeleLs8TvrCiKAXAoGBANsduqLnlUW/f5zDb5Fe
|
||||
SB51+KhBjsVIeYmU+8xtur9Z7IxZXK28wpoEsm7LmX7Va5dERjI+tItBiJ5+Unu1
|
||||
Xs2ljDg35ARKHs0dWBJGpbnZg4dbT6xpIL4YMPXm1Zu++PgRpxPIMn646xqd8GlH
|
||||
bavm6Km/fXNG58xus+EeLpV5
|
||||
-----END PRIVATE KEY-----
|
|
@ -1,28 +0,0 @@
|
|||
-----BEGIN PRIVATE KEY-----
|
||||
MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQC3Gvumr+WfNV5n
|
||||
PjPnZhzcN5o+qM19FLJblw9SVPsAFcVvv62NrsrYLu4NhUr5D7bIkJDBgljjtook
|
||||
H757Bqb23ZXxfLRnxL7QjYiYLC7J6KFoP9/q4llHOazJx0QIfbYEDq0I+VAueVuh
|
||||
GJnMjqtwi9cBtQy8u8WbefJgiz2qJ0TFGd/IyMhKScxX7zctPlBbghweApirN3PZ
|
||||
mY6eT5/jCYyaK1yK9N0hkIvxnRzGvsEmU6Hkln9Xm6NZRd41r621bREooGWD1P7C
|
||||
PQm/CgkE9lOawwDLe0j22MuDcPAZgqx+VsV5Ta15tciCdwYVn90XGPffUYa2reTP
|
||||
U/MBRh/bAgMBAAECggEABCvcMcbuDztzBB0Zp5re63Fk1SqZS9Et4wJE+hYvhaf5
|
||||
UHtoY8LoohYnnC0+MQBXpKgOdCoZBk8BRKNofnr/UL5pjQ/POFH2GuAujXDsO/NN
|
||||
wgc6fapcaE/7DLm6ZgsfG2aOMJclaXmgScI6trtFUpIM+t/6A06vyMP1bpeddwPW
|
||||
Fqu7NvpDiEcTRUGd+z1JooYgUhGgC7peYUx5+9zqFrwoDBKxnUOnz3BkDsXBy3qm
|
||||
65Vu0BSjuJzf6vVMpNGUHY6JXjopVNWku+JAX0wD+iikOd5sziNVdIj1fnZ+IHIf
|
||||
7G5h5owHpvSGzJFQ18/g5VHtJdCm+4WQSnbSJRsCAQKBgQDu4IH8yspyeH44fhoS
|
||||
PAp/OtILqSP+Da0zAp2LbhrOgyzyuSTdEAYyptqjqHS6QkB1Bu1H44FS0BYUxRXc
|
||||
iu2e9AndiLVCGngsE7TpA/ZVLN1B0LEZEHjM6p4d6zZM6iveKVnPAOkTWTBAgzCt
|
||||
b31nj4jL8PdlPKQil1AMrOlRAQKBgQDEOwshzIdr2Iy6B/n4CuBViEtwnbAd5f/c
|
||||
atA9bcfF8kCahokJsI4eCCLgBwDZpYKD+v0AwOBlacF6t6TX+vdlJsi5EP7uxZ22
|
||||
ILsuWqVm/0H77PACuckc5/qLZoGGC81l0DhnpoeMEb6r/TKOo5xAK1gxdlwNNrq+
|
||||
nP1zdZnU2wKBgBAS92xFUR4m0YeHpMV5WNN658t1FEDyNqdqE6PgQtmGpi2nG73s
|
||||
aB5cb/X3TfOCpce6MZlWy8sAyZuYL4Jprte1YDySCHBsS43bvZ64b4kHvdPB8UjY
|
||||
fOh9GSq2Oy8tysnmSm7NhuGQbNjKeyoQiIXBeNkQW/VqATl6qR5RPFoBAoGACNqV
|
||||
JQBCd/Y8W0Ry3eM3vgQ5SyqCQMcY5UwYez0Rz3efvJknY72InAhH8o2+VxOlsOjJ
|
||||
M5iAR3MfHLdeg7Q6J2E5m0gOCJ34ALi3WV8TqXMI+iH1rlnNnjVFU7bbTz4HFXnw
|
||||
oZSc9w/x53a0KkVtjmOmRg0OGDaI9ILG2MfMmhMCgYB8ZqJtX8qZ2TqKU3XdLZ4z
|
||||
T2N7xMFuKohWP420r5jKm3Xw85IC+y1SUTB9XGcL79r2eJzmzmdKQ3A3sf3oyUH3
|
||||
RdYWxtKcZ5PAE8hVRtn1ETZqUgxASGOUn/6w0npkYSOXPU5bc0W6RSLkjES0i+c3
|
||||
fv3OMNI8qpmQhEjpHHQS1g==
|
||||
-----END PRIVATE KEY-----
|
|
@ -1,35 +0,0 @@
|
|||
name: Test alerts
|
||||
|
||||
on:
|
||||
push:
|
||||
branches-ignore:
|
||||
- 'development/**'
|
||||
- 'q/*/**'
|
||||
|
||||
jobs:
|
||||
run-alert-tests:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
tests:
|
||||
- name: 1 minute interval tests
|
||||
file: monitoring/alerts.test.yaml
|
||||
|
||||
- name: 10 seconds interval tests
|
||||
file: monitoring/alerts.10s.test.yaml
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Render and test ${{ matrix.tests.name }}
|
||||
uses: scality/action-prom-render-test@1.0.3
|
||||
with:
|
||||
alert_file_path: monitoring/alerts.yaml
|
||||
test_file_path: ${{ matrix.tests.file }}
|
||||
alert_inputs: |
|
||||
namespace=zenko
|
||||
service=artesca-data-connector-s3api-metrics
|
||||
reportJob=artesca-data-ops-report-handler
|
||||
replicas=3
|
||||
github_token: ${{ secrets.GITHUB_TOKEN }}
|
|
@ -1,25 +0,0 @@
|
|||
---
|
||||
name: codeQL
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [w/**, q/*]
|
||||
pull_request:
|
||||
branches: [development/*, stabilization/*, hotfix/*]
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
analyze:
|
||||
name: Static analysis with CodeQL
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Initialize CodeQL
|
||||
uses: github/codeql-action/init@v3
|
||||
with:
|
||||
languages: javascript, python, ruby
|
||||
|
||||
- name: Build and analyze
|
||||
uses: github/codeql-action/analyze@v3
|
|
@ -1,16 +0,0 @@
|
|||
---
|
||||
name: dependency review
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [development/*, stabilization/*, hotfix/*]
|
||||
|
||||
jobs:
|
||||
dependency-review:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: 'Checkout Repository'
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: 'Dependency Review'
|
||||
uses: actions/dependency-review-action@v4
|
|
@ -1,80 +0,0 @@
|
|||
---
|
||||
name: release
|
||||
run-name: release ${{ inputs.tag }}
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
tag:
|
||||
description: 'Tag to be released'
|
||||
required: true
|
||||
|
||||
env:
|
||||
PROJECT_NAME: ${{ github.event.repository.name }}
|
||||
|
||||
jobs:
|
||||
build-federation-image:
|
||||
runs-on: ubuntu-20.04
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
- name: Login to GitHub Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.repository_owner }}
|
||||
password: ${{ github.token }}
|
||||
- name: Build and push image for federation
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
push: true
|
||||
context: .
|
||||
file: images/svc-base/Dockerfile
|
||||
tags: |
|
||||
ghcr.io/${{ github.repository }}:${{ github.event.inputs.tag }}-svc-base
|
||||
cache-from: type=gha,scope=federation
|
||||
cache-to: type=gha,mode=max,scope=federation
|
||||
|
||||
release:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Docker Buildk
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Login to Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.repository_owner }}
|
||||
password: ${{ github.token }}
|
||||
|
||||
- name: Push dashboards into the production namespace
|
||||
run: |
|
||||
oras push ghcr.io/${{ github.repository }}/${{ env.PROJECT_NAME }}-dashboards:${{ github.event.inputs.tag }} \
|
||||
dashboard.json:application/grafana-dashboard+json \
|
||||
alerts.yaml:application/prometheus-alerts+yaml
|
||||
working-directory: monitoring
|
||||
|
||||
- name: Build and push
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: .
|
||||
push: true
|
||||
tags: ghcr.io/${{ github.repository }}:${{ github.event.inputs.tag }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Create Release
|
||||
uses: softprops/action-gh-release@v2
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
with:
|
||||
name: Release ${{ github.event.inputs.tag }}
|
||||
tag_name: ${{ github.event.inputs.tag }}
|
||||
generate_release_notes: true
|
||||
target_commitish: ${{ github.sha }}
|
|
@ -1,533 +0,0 @@
|
|||
---
|
||||
name: tests
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
push:
|
||||
branches-ignore:
|
||||
- 'development/**'
|
||||
- 'q/*/**'
|
||||
|
||||
env:
|
||||
# Secrets
|
||||
azurebackend_AZURE_STORAGE_ACCESS_KEY: >-
|
||||
${{ secrets.AZURE_STORAGE_ACCESS_KEY }}
|
||||
azurebackend_AZURE_STORAGE_ACCOUNT_NAME: >-
|
||||
${{ secrets.AZURE_STORAGE_ACCOUNT_NAME }}
|
||||
azurebackend_AZURE_STORAGE_ENDPOINT: >-
|
||||
${{ secrets.AZURE_STORAGE_ENDPOINT }}
|
||||
azurebackend2_AZURE_STORAGE_ACCESS_KEY: >-
|
||||
${{ secrets.AZURE_STORAGE_ACCESS_KEY_2 }}
|
||||
azurebackend2_AZURE_STORAGE_ACCOUNT_NAME: >-
|
||||
${{ secrets.AZURE_STORAGE_ACCOUNT_NAME_2 }}
|
||||
azurebackend2_AZURE_STORAGE_ENDPOINT: >-
|
||||
${{ secrets.AZURE_STORAGE_ENDPOINT_2 }}
|
||||
azurebackendmismatch_AZURE_STORAGE_ACCESS_KEY: >-
|
||||
${{ secrets.AZURE_STORAGE_ACCESS_KEY }}
|
||||
azurebackendmismatch_AZURE_STORAGE_ACCOUNT_NAME: >-
|
||||
${{ secrets.AZURE_STORAGE_ACCOUNT_NAME }}
|
||||
azurebackendmismatch_AZURE_STORAGE_ENDPOINT: >-
|
||||
${{ secrets.AZURE_STORAGE_ENDPOINT }}
|
||||
azurenonexistcontainer_AZURE_STORAGE_ACCESS_KEY: >-
|
||||
${{ secrets.AZURE_STORAGE_ACCESS_KEY }}
|
||||
azurenonexistcontainer_AZURE_STORAGE_ACCOUNT_NAME: >-
|
||||
${{ secrets.AZURE_STORAGE_ACCOUNT_NAME }}
|
||||
azurenonexistcontainer_AZURE_STORAGE_ENDPOINT: >-
|
||||
${{ secrets.AZURE_STORAGE_ENDPOINT }}
|
||||
azuretest_AZURE_BLOB_ENDPOINT: "${{ secrets.AZURE_STORAGE_ENDPOINT }}"
|
||||
b2backend_B2_ACCOUNT_ID: "${{ secrets.B2BACKEND_B2_ACCOUNT_ID }}"
|
||||
b2backend_B2_STORAGE_ACCESS_KEY: >-
|
||||
${{ secrets.B2BACKEND_B2_STORAGE_ACCESS_KEY }}
|
||||
GOOGLE_SERVICE_EMAIL: "${{ secrets.GCP_SERVICE_EMAIL }}"
|
||||
GOOGLE_SERVICE_KEY: "${{ secrets.GCP_SERVICE_KEY }}"
|
||||
AWS_S3_BACKEND_ACCESS_KEY: "${{ secrets.AWS_S3_BACKEND_ACCESS_KEY }}"
|
||||
AWS_S3_BACKEND_SECRET_KEY: "${{ secrets.AWS_S3_BACKEND_SECRET_KEY }}"
|
||||
AWS_S3_BACKEND_ACCESS_KEY_2: "${{ secrets.AWS_S3_BACKEND_ACCESS_KEY_2 }}"
|
||||
AWS_S3_BACKEND_SECRET_KEY_2: "${{ secrets.AWS_S3_BACKEND_SECRET_KEY_2 }}"
|
||||
AWS_GCP_BACKEND_ACCESS_KEY: "${{ secrets.AWS_GCP_BACKEND_ACCESS_KEY }}"
|
||||
AWS_GCP_BACKEND_SECRET_KEY: "${{ secrets.AWS_GCP_BACKEND_SECRET_KEY }}"
|
||||
AWS_GCP_BACKEND_ACCESS_KEY_2: "${{ secrets.AWS_GCP_BACKEND_ACCESS_KEY_2 }}"
|
||||
AWS_GCP_BACKEND_SECRET_KEY_2: "${{ secrets.AWS_GCP_BACKEND_SECRET_KEY_2 }}"
|
||||
b2backend_B2_STORAGE_ENDPOINT: "${{ secrets.B2BACKEND_B2_STORAGE_ENDPOINT }}"
|
||||
gcpbackend2_GCP_SERVICE_EMAIL: "${{ secrets.GCP2_SERVICE_EMAIL }}"
|
||||
gcpbackend2_GCP_SERVICE_KEY: "${{ secrets.GCP2_SERVICE_KEY }}"
|
||||
gcpbackend2_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
|
||||
gcpbackend_GCP_SERVICE_EMAIL: "${{ secrets.GCP_SERVICE_EMAIL }}"
|
||||
gcpbackend_GCP_SERVICE_KEY: "${{ secrets.GCP_SERVICE_KEY }}"
|
||||
gcpbackendmismatch_GCP_SERVICE_EMAIL: >-
|
||||
${{ secrets.GCPBACKENDMISMATCH_GCP_SERVICE_EMAIL }}
|
||||
gcpbackendmismatch_GCP_SERVICE_KEY: >-
|
||||
${{ secrets.GCPBACKENDMISMATCH_GCP_SERVICE_KEY }}
|
||||
gcpbackend_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
|
||||
gcpbackendmismatch_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
|
||||
gcpbackendnoproxy_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
|
||||
gcpbackendproxy_GCP_SERVICE_KEYFILE: /root/.gcp/servicekey
|
||||
# Configs
|
||||
ENABLE_LOCAL_CACHE: "true"
|
||||
REPORT_TOKEN: "report-token-1"
|
||||
REMOTE_MANAGEMENT_DISABLE: "1"
|
||||
# https://github.com/git-lfs/git-lfs/issues/5749
|
||||
GIT_CLONE_PROTECTION_ACTIVE: 'false'
|
||||
jobs:
|
||||
linting-coverage:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '16'
|
||||
cache: yarn
|
||||
- name: install dependencies
|
||||
run: yarn install --frozen-lockfile --network-concurrency 1
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.9'
|
||||
- uses: actions/cache@v4
|
||||
with:
|
||||
path: ~/.cache/pip
|
||||
key: ${{ runner.os }}-pip
|
||||
- name: Install python deps
|
||||
run: pip install flake8
|
||||
- name: Lint Javascript
|
||||
run: yarn run --silent lint -- --max-warnings 0
|
||||
- name: Lint Markdown
|
||||
run: yarn run --silent lint_md
|
||||
- name: Lint python
|
||||
run: flake8 $(git ls-files "*.py")
|
||||
- name: Lint Yaml
|
||||
run: yamllint -c yamllint.yml $(git ls-files "*.yml")
|
||||
- name: Unit Coverage
|
||||
run: |
|
||||
set -ex
|
||||
mkdir -p $CIRCLE_TEST_REPORTS/unit
|
||||
yarn test
|
||||
yarn run test_legacy_location
|
||||
env:
|
||||
S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
|
||||
CIRCLE_TEST_REPORTS: /tmp
|
||||
CIRCLE_ARTIFACTS: /tmp
|
||||
CI_REPORTS: /tmp
|
||||
- name: Unit Coverage logs
|
||||
run: find /tmp/unit -exec cat {} \;
|
||||
- name: preparing junit files for upload
|
||||
run: |
|
||||
mkdir -p artifacts/junit
|
||||
find . -name "*junit*.xml" -exec cp {} artifacts/junit/ ";"
|
||||
if: always()
|
||||
- name: Upload files to artifacts
|
||||
uses: scality/action-artifacts@v4
|
||||
with:
|
||||
method: upload
|
||||
url: https://artifacts.scality.net
|
||||
user: ${{ secrets.ARTIFACTS_USER }}
|
||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
||||
source: artifacts
|
||||
if: always()
|
||||
|
||||
build:
|
||||
runs-on: ubuntu-20.04
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
- name: Login to GitHub Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.repository_owner }}
|
||||
password: ${{ github.token }}
|
||||
- name: Build and push cloudserver image
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
push: true
|
||||
context: .
|
||||
provenance: false
|
||||
tags: |
|
||||
ghcr.io/${{ github.repository }}:${{ github.sha }}
|
||||
labels: |
|
||||
git.repository=${{ github.repository }}
|
||||
git.commit-sha=${{ github.sha }}
|
||||
cache-from: type=gha,scope=cloudserver
|
||||
cache-to: type=gha,mode=max,scope=cloudserver
|
||||
- name: Build and push pykmip image
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
push: true
|
||||
context: .github/pykmip
|
||||
tags: |
|
||||
ghcr.io/${{ github.repository }}/pykmip:${{ github.sha }}
|
||||
labels: |
|
||||
git.repository=${{ github.repository }}
|
||||
git.commit-sha=${{ github.sha }}
|
||||
cache-from: type=gha,scope=pykmip
|
||||
cache-to: type=gha,mode=max,scope=pykmip
|
||||
- name: Build and push MongoDB
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
push: true
|
||||
context: .github/docker/mongodb
|
||||
tags: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
||||
cache-from: type=gha,scope=mongodb
|
||||
cache-to: type=gha,mode=max,scope=mongodb
|
||||
|
||||
multiple-backend:
|
||||
runs-on: ubuntu-latest
|
||||
needs: build
|
||||
env:
|
||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
||||
S3BACKEND: mem
|
||||
S3_LOCATION_FILE: /usr/src/app/tests/locationConfig/locationConfigTests.json
|
||||
S3DATA: multiple
|
||||
JOB_NAME: ${{ github.job }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
- name: Login to Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.repository_owner }}
|
||||
password: ${{ github.token }}
|
||||
- name: Setup CI environment
|
||||
uses: ./.github/actions/setup-ci
|
||||
- name: Setup CI services
|
||||
run: docker compose --profile sproxyd up -d
|
||||
working-directory: .github/docker
|
||||
- name: Run multiple backend test
|
||||
run: |-
|
||||
set -o pipefail;
|
||||
bash wait_for_local_port.bash 8000 40
|
||||
bash wait_for_local_port.bash 81 40
|
||||
yarn run multiple_backend_test | tee /tmp/artifacts/${{ github.job }}/tests.log
|
||||
env:
|
||||
S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
|
||||
- name: Upload logs to artifacts
|
||||
uses: scality/action-artifacts@v4
|
||||
with:
|
||||
method: upload
|
||||
url: https://artifacts.scality.net
|
||||
user: ${{ secrets.ARTIFACTS_USER }}
|
||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
||||
source: /tmp/artifacts
|
||||
if: always()
|
||||
|
||||
mongo-v0-ft-tests:
|
||||
runs-on: ubuntu-latest
|
||||
needs: build
|
||||
env:
|
||||
S3BACKEND: mem
|
||||
MPU_TESTING: "yes"
|
||||
S3METADATA: mongodb
|
||||
S3KMS: file
|
||||
S3_LOCATION_FILE: /usr/src/app/tests/locationConfig/locationConfigTests.json
|
||||
DEFAULT_BUCKET_KEY_FORMAT: v0
|
||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
||||
JOB_NAME: ${{ github.job }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
- name: Setup CI environment
|
||||
uses: ./.github/actions/setup-ci
|
||||
- name: Setup CI services
|
||||
run: docker compose --profile mongo up -d
|
||||
working-directory: .github/docker
|
||||
- name: Run functional tests
|
||||
run: |-
|
||||
set -o pipefail;
|
||||
bash wait_for_local_port.bash 8000 40
|
||||
yarn run ft_test | tee /tmp/artifacts/${{ github.job }}/tests.log
|
||||
env:
|
||||
S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
|
||||
- name: Upload logs to artifacts
|
||||
uses: scality/action-artifacts@v4
|
||||
with:
|
||||
method: upload
|
||||
url: https://artifacts.scality.net
|
||||
user: ${{ secrets.ARTIFACTS_USER }}
|
||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
||||
source: /tmp/artifacts
|
||||
if: always()
|
||||
|
||||
mongo-v1-ft-tests:
|
||||
runs-on: ubuntu-latest
|
||||
needs: build
|
||||
env:
|
||||
S3BACKEND: mem
|
||||
MPU_TESTING: "yes"
|
||||
S3METADATA: mongodb
|
||||
S3KMS: file
|
||||
S3_LOCATION_FILE: /usr/src/app/tests/locationConfig/locationConfigTests.json
|
||||
DEFAULT_BUCKET_KEY_FORMAT: v1
|
||||
METADATA_MAX_CACHED_BUCKETS: 1
|
||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
||||
JOB_NAME: ${{ github.job }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
- name: Setup CI environment
|
||||
uses: ./.github/actions/setup-ci
|
||||
- name: Setup CI services
|
||||
run: docker compose --profile mongo up -d
|
||||
working-directory: .github/docker
|
||||
- name: Run functional tests
|
||||
run: |-
|
||||
set -o pipefail;
|
||||
bash wait_for_local_port.bash 8000 40
|
||||
yarn run ft_test | tee /tmp/artifacts/${{ github.job }}/tests.log
|
||||
yarn run ft_mixed_bucket_format_version | tee /tmp/artifacts/${{ github.job }}/mixed-tests.log
|
||||
env:
|
||||
S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
|
||||
- name: Upload logs to artifacts
|
||||
uses: scality/action-artifacts@v4
|
||||
with:
|
||||
method: upload
|
||||
url: https://artifacts.scality.net
|
||||
user: ${{ secrets.ARTIFACTS_USER }}
|
||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
||||
source: /tmp/artifacts
|
||||
if: always()
|
||||
|
||||
file-ft-tests:
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- job-name: file-ft-tests
|
||||
name: ${{ matrix.job-name }}
|
||||
runs-on: ubuntu-latest
|
||||
needs: build
|
||||
env:
|
||||
S3BACKEND: file
|
||||
S3VAULT: mem
|
||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
||||
MPU_TESTING: "yes"
|
||||
JOB_NAME: ${{ matrix.job-name }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
- name: Setup CI environment
|
||||
uses: ./.github/actions/setup-ci
|
||||
- name: Setup matrix job artifacts directory
|
||||
shell: bash
|
||||
run: |
|
||||
set -exu
|
||||
mkdir -p /tmp/artifacts/${{ matrix.job-name }}/
|
||||
- name: Setup CI services
|
||||
run: docker compose up -d
|
||||
working-directory: .github/docker
|
||||
- name: Run file ft tests
|
||||
run: |-
|
||||
set -o pipefail;
|
||||
bash wait_for_local_port.bash 8000 40
|
||||
yarn run ft_test | tee /tmp/artifacts/${{ matrix.job-name }}/tests.log
|
||||
- name: Upload logs to artifacts
|
||||
uses: scality/action-artifacts@v4
|
||||
with:
|
||||
method: upload
|
||||
url: https://artifacts.scality.net
|
||||
user: ${{ secrets.ARTIFACTS_USER }}
|
||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
||||
source: /tmp/artifacts
|
||||
if: always()
|
||||
|
||||
utapi-v2-tests:
|
||||
runs-on: ubuntu-latest
|
||||
needs: build
|
||||
env:
|
||||
ENABLE_UTAPI_V2: t
|
||||
S3BACKEND: mem
|
||||
BUCKET_DENY_FILTER: utapi-event-filter-deny-bucket
|
||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
||||
JOB_NAME: ${{ github.job }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
- name: Setup CI environment
|
||||
uses: ./.github/actions/setup-ci
|
||||
- name: Setup CI services
|
||||
run: docker compose up -d
|
||||
working-directory: .github/docker
|
||||
- name: Run file utapi v2 tests
|
||||
run: |-
|
||||
set -ex -o pipefail;
|
||||
bash wait_for_local_port.bash 8000 40
|
||||
yarn run test_utapi_v2 | tee /tmp/artifacts/${{ github.job }}/tests.log
|
||||
- name: Upload logs to artifacts
|
||||
uses: scality/action-artifacts@v4
|
||||
with:
|
||||
method: upload
|
||||
url: https://artifacts.scality.net
|
||||
user: ${{ secrets.ARTIFACTS_USER }}
|
||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
||||
source: /tmp/artifacts
|
||||
if: always()
|
||||
|
||||
quota-tests:
|
||||
runs-on: ubuntu-latest
|
||||
needs: build
|
||||
strategy:
|
||||
matrix:
|
||||
inflights:
|
||||
- name: "With Inflights"
|
||||
value: "true"
|
||||
- name: "Without Inflights"
|
||||
value: "false"
|
||||
env:
|
||||
S3METADATA: mongodb
|
||||
S3BACKEND: mem
|
||||
S3QUOTA: scuba
|
||||
QUOTA_ENABLE_INFLIGHTS: ${{ matrix.inflights.value }}
|
||||
SCUBA_HOST: localhost
|
||||
SCUBA_PORT: 8100
|
||||
SCUBA_HEALTHCHECK_FREQUENCY: 100
|
||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
||||
JOB_NAME: ${{ github.job }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
- name: Setup CI environment
|
||||
uses: ./.github/actions/setup-ci
|
||||
- name: Setup CI services
|
||||
run: docker compose --profile mongo up -d
|
||||
working-directory: .github/docker
|
||||
- name: Run quota tests
|
||||
run: |-
|
||||
set -ex -o pipefail;
|
||||
bash wait_for_local_port.bash 8000 40
|
||||
yarn run test_quota | tee /tmp/artifacts/${{ github.job }}/tests.log
|
||||
- name: Upload logs to artifacts
|
||||
uses: scality/action-artifacts@v4
|
||||
with:
|
||||
method: upload
|
||||
url: https://artifacts.scality.net
|
||||
user: ${{ secrets.ARTIFACTS_USER }}
|
||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
||||
source: /tmp/artifacts
|
||||
if: always()
|
||||
|
||||
kmip-ft-tests:
|
||||
runs-on: ubuntu-latest
|
||||
needs: build
|
||||
env:
|
||||
S3BACKEND: file
|
||||
S3VAULT: mem
|
||||
MPU_TESTING: "yes"
|
||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
||||
PYKMIP_IMAGE: ghcr.io/${{ github.repository }}/pykmip:${{ github.sha }}
|
||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
||||
JOB_NAME: ${{ github.job }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
- name: Setup CI environment
|
||||
uses: ./.github/actions/setup-ci
|
||||
- name: Copy KMIP certs
|
||||
run: cp -r ./certs /tmp/ssl-kmip
|
||||
working-directory: .github/pykmip
|
||||
- name: Setup CI services
|
||||
run: docker compose --profile pykmip up -d
|
||||
working-directory: .github/docker
|
||||
- name: Run file KMIP tests
|
||||
run: |-
|
||||
set -ex -o pipefail;
|
||||
bash wait_for_local_port.bash 8000 40
|
||||
bash wait_for_local_port.bash 5696 40
|
||||
yarn run ft_kmip | tee /tmp/artifacts/${{ github.job }}/tests.log
|
||||
- name: Upload logs to artifacts
|
||||
uses: scality/action-artifacts@v4
|
||||
with:
|
||||
method: upload
|
||||
url: https://artifacts.scality.net
|
||||
user: ${{ secrets.ARTIFACTS_USER }}
|
||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
||||
source: /tmp/artifacts
|
||||
if: always()
|
||||
|
||||
ceph-backend-test:
|
||||
runs-on: ubuntu-latest
|
||||
needs: build
|
||||
env:
|
||||
S3BACKEND: mem
|
||||
S3DATA: multiple
|
||||
S3KMS: file
|
||||
CI_CEPH: 'true'
|
||||
MPU_TESTING: "yes"
|
||||
S3_LOCATION_FILE: /usr/src/app/tests/locationConfig/locationConfigCeph.json
|
||||
MONGODB_IMAGE: ghcr.io/${{ github.repository }}/ci-mongodb:${{ github.sha }}
|
||||
CLOUDSERVER_IMAGE: ghcr.io/${{ github.repository }}:${{ github.sha }}
|
||||
JOB_NAME: ${{ github.job }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
- name: Login to GitHub Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.repository_owner }}
|
||||
password: ${{ github.token }}
|
||||
- name: Setup CI environment
|
||||
uses: ./.github/actions/setup-ci
|
||||
- uses: ruby/setup-ruby@v1
|
||||
with:
|
||||
ruby-version: '2.5.9'
|
||||
- name: Install Ruby dependencies
|
||||
run: |
|
||||
gem install nokogiri:1.12.5 excon:0.109.0 fog-aws:1.3.0 json mime-types:3.1 rspec:3.5
|
||||
- name: Install Java dependencies
|
||||
run: |
|
||||
sudo apt-get update && sudo apt-get install -y --fix-missing default-jdk maven
|
||||
- name: Setup CI services
|
||||
run: docker compose --profile ceph up -d
|
||||
working-directory: .github/docker
|
||||
env:
|
||||
S3METADATA: mongodb
|
||||
- name: Run Ceph multiple backend tests
|
||||
run: |-
|
||||
set -ex -o pipefail;
|
||||
bash .github/ceph/wait_for_ceph.sh
|
||||
bash wait_for_local_port.bash 27018 40
|
||||
bash wait_for_local_port.bash 8000 40
|
||||
yarn run multiple_backend_test | tee /tmp/artifacts/${{ github.job }}/multibackend-tests.log
|
||||
env:
|
||||
S3_LOCATION_FILE: tests/locationConfig/locationConfigTests.json
|
||||
S3METADATA: mem
|
||||
- name: Run Java tests
|
||||
run: |-
|
||||
set -ex -o pipefail;
|
||||
mvn test | tee /tmp/artifacts/${{ github.job }}/java-tests.log
|
||||
working-directory: tests/functional/jaws
|
||||
- name: Run Ruby tests
|
||||
run: |-
|
||||
set -ex -o pipefail;
|
||||
rspec -fd --backtrace tests.rb | tee /tmp/artifacts/${{ github.job }}/ruby-tests.log
|
||||
working-directory: tests/functional/fog
|
||||
- name: Run Javascript AWS SDK tests
|
||||
run: |-
|
||||
set -ex -o pipefail;
|
||||
yarn run ft_awssdk | tee /tmp/artifacts/${{ github.job }}/js-awssdk-tests.log;
|
||||
yarn run ft_s3cmd | tee /tmp/artifacts/${{ github.job }}/js-s3cmd-tests.log;
|
||||
env:
|
||||
S3_LOCATION_FILE: tests/locationConfig/locationConfigCeph.json
|
||||
S3BACKEND: file
|
||||
S3VAULT: mem
|
||||
S3METADATA: mongodb
|
||||
- name: Upload logs to artifacts
|
||||
uses: scality/action-artifacts@v4
|
||||
with:
|
||||
method: upload
|
||||
url: https://artifacts.scality.net
|
||||
user: ${{ secrets.ARTIFACTS_USER }}
|
||||
password: ${{ secrets.ARTIFACTS_PASSWORD }}
|
||||
source: /tmp/artifacts
|
||||
if: always()
|
|
@ -22,14 +22,6 @@ coverage
|
|||
# Compiled binary addons (http://nodejs.org/api/addons.html)
|
||||
build/Release
|
||||
|
||||
# Sphinx build dir
|
||||
_build
|
||||
|
||||
# Dependency directory
|
||||
# https://www.npmjs.org/doc/misc/npm-faq.html#should-i-check-my-node_modules-folder-into-git
|
||||
node_modules
|
||||
yarn.lock
|
||||
.tox
|
||||
|
||||
# Junit directory
|
||||
junit
|
||||
|
|
60
Dockerfile
60
Dockerfile
|
@ -1,60 +0,0 @@
|
|||
ARG NODE_VERSION=16.20-bullseye-slim
|
||||
|
||||
FROM node:${NODE_VERSION} as builder
|
||||
|
||||
WORKDIR /usr/src/app
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
ca-certificates \
|
||||
curl \
|
||||
git \
|
||||
gnupg2 \
|
||||
jq \
|
||||
python3 \
|
||||
ssh \
|
||||
wget \
|
||||
libffi-dev \
|
||||
zlib1g-dev \
|
||||
&& apt-get clean \
|
||||
&& mkdir -p /root/ssh \
|
||||
&& ssh-keyscan -H github.com > /root/ssh/known_hosts
|
||||
|
||||
ENV PYTHON=python3
|
||||
COPY package.json yarn.lock /usr/src/app/
|
||||
RUN npm install typescript -g
|
||||
RUN yarn install --production --ignore-optional --frozen-lockfile --ignore-engines --network-concurrency 1
|
||||
|
||||
################################################################################
|
||||
FROM node:${NODE_VERSION}
|
||||
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
jq \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
ENV NO_PROXY localhost,127.0.0.1
|
||||
ENV no_proxy localhost,127.0.0.1
|
||||
|
||||
EXPOSE 8000
|
||||
EXPOSE 8002
|
||||
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
jq \
|
||||
tini \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /usr/src/app
|
||||
|
||||
# Keep the .git directory in order to properly report version
|
||||
COPY . /usr/src/app
|
||||
COPY --from=builder /usr/src/app/node_modules ./node_modules/
|
||||
|
||||
|
||||
VOLUME ["/usr/src/app/localData","/usr/src/app/localMetadata"]
|
||||
|
||||
ENTRYPOINT ["tini", "--", "/usr/src/app/docker-entrypoint.sh"]
|
||||
|
||||
CMD [ "yarn", "start" ]
|
165
README.md
165
README.md
|
@ -1,7 +1,12 @@
|
|||
# Zenko CloudServer with Vitastor Backend
|
||||
# Zenko CloudServer
|
||||
|
||||
![Zenko CloudServer logo](res/scality-cloudserver-logo.png)
|
||||
|
||||
[![CircleCI][badgepub]](https://circleci.com/gh/scality/S3)
|
||||
[![Scality CI][badgepriv]](http://ci.ironmann.io/gh/scality/S3)
|
||||
[![Docker Pulls][badgedocker]](https://hub.docker.com/r/scality/s3server/)
|
||||
[![Docker Pulls][badgetwitter]](https://twitter.com/zenko)
|
||||
|
||||
## Overview
|
||||
|
||||
CloudServer (formerly S3 Server) is an open-source Amazon S3-compatible
|
||||
|
@ -11,71 +16,125 @@ Scality’s Open Source Multi-Cloud Data Controller.
|
|||
CloudServer provides a single AWS S3 API interface to access multiple
|
||||
backend data storage both on-premise or public in the cloud.
|
||||
|
||||
This repository contains a fork of CloudServer with [Vitastor](https://git.yourcmc.ru/vitalif/vitastor)
|
||||
backend support.
|
||||
CloudServer is useful for Developers, either to run as part of a
|
||||
continous integration test environment to emulate the AWS S3 service locally
|
||||
or as an abstraction layer to develop object storage enabled
|
||||
application on the go.
|
||||
|
||||
## Quick Start with Vitastor
|
||||
## Learn more at [www.zenko.io/cloudserver](https://www.zenko.io/cloudserver/)
|
||||
|
||||
Vitastor Backend is in experimental status, however you can already try to
|
||||
run it and write or read something, or even mount it with [GeeseFS](https://github.com/yandex-cloud/geesefs),
|
||||
it works too 😊.
|
||||
## [May I offer you some lovely documentation?](http://s3-server.readthedocs.io/en/latest/)
|
||||
|
||||
Installation instructions:
|
||||
## Docker
|
||||
|
||||
### Install Vitastor
|
||||
[Run your Zenko CloudServer with Docker](https://hub.docker.com/r/scality/s3server/)
|
||||
|
||||
Refer to [Vitastor Quick Start Manual](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/docs/intro/quickstart.en.md).
|
||||
## Contributing
|
||||
|
||||
### Install Zenko with Vitastor Backend
|
||||
In order to contribute, please follow the
|
||||
[Contributing Guidelines](
|
||||
https://github.com/scality/Guidelines/blob/master/CONTRIBUTING.md).
|
||||
|
||||
- Clone this repository: `git clone https://git.yourcmc.ru/vitalif/zenko-cloudserver-vitastor`
|
||||
- Install dependencies: `npm install --omit dev` or just `npm install`
|
||||
- Clone Vitastor repository: `git clone https://git.yourcmc.ru/vitalif/vitastor`
|
||||
- Build Vitastor node.js binding by running `npm install` in `node-binding` subdirectory of Vitastor repository.
|
||||
You need `node-gyp` and `vitastor-client-dev` (Vitastor client library) for it to succeed.
|
||||
- Symlink Vitastor module to Zenko: `ln -s /path/to/vitastor/node-binding /path/to/zenko/node_modules/vitastor`
|
||||
## Installation
|
||||
|
||||
### Install and Configure MongoDB
|
||||
### Dependencies
|
||||
|
||||
Refer to [MongoDB Manual](https://www.mongodb.com/docs/manual/installation/).
|
||||
Building and running the Zenko CloudServer requires node.js 6.9.5 and npm v3
|
||||
. Up-to-date versions can be found at
|
||||
[Nodesource](https://github.com/nodesource/distributions).
|
||||
|
||||
### Setup Zenko
|
||||
### Clone source code
|
||||
|
||||
- Create a separate pool for S3 object data in your Vitastor cluster: `vitastor-cli create-pool s3-data`
|
||||
- Retrieve ID of the new pool from `vitastor-cli ls-pools --detail s3-data`
|
||||
- In another pool, create an image for storing Vitastor volume metadata: `vitastor-cli create -s 10G s3-volume-meta`
|
||||
- Copy `config.json.vitastor` to `config.json`, adjust it to match your domain
|
||||
- Copy `authdata.json.example` to `authdata.json` - this is where you set S3 access & secret keys,
|
||||
and also adjust them if you want to. Scality seems to use a separate auth service "Scality Vault" for
|
||||
access keys, but it's not published, so let's use a file for now.
|
||||
- Copy `locationConfig.json.vitastor` to `locationConfig.json` - this is where you set Vitastor cluster access data.
|
||||
You should put correct values for `pool_id` (pool ID from the second step) and `metadata_image` (from the third step)
|
||||
in this file.
|
||||
|
||||
Note: `locationConfig.json` in this version corresponds to storage classes (like STANDARD, COLD, etc)
|
||||
instead of "locations" (zones like us-east-1) as it was in original Zenko CloudServer.
|
||||
|
||||
### Start Zenko
|
||||
|
||||
Start the S3 server with: `node index.js`
|
||||
|
||||
If you use default settings, Zenko CloudServer starts on port 8000.
|
||||
The default access key is `accessKey1` with a secret key of `verySecretKey1`.
|
||||
|
||||
Now you can access your S3 with `s3cmd` or `geesefs`:
|
||||
|
||||
```
|
||||
s3cmd --access_key=accessKey1 --secret_key=verySecretKey1 --host=http://localhost:8000 mb s3://testbucket
|
||||
```shell
|
||||
git clone https://github.com/scality/S3.git
|
||||
```
|
||||
|
||||
```
|
||||
AWS_ACCESS_KEY_ID=accessKey1 \
|
||||
AWS_SECRET_ACCESS_KEY=verySecretKey1 \
|
||||
geesefs --endpoint http://localhost:8000 testbucket mountdir
|
||||
### Install js dependencies
|
||||
|
||||
Go to the ./S3 folder,
|
||||
|
||||
```shell
|
||||
npm install
|
||||
```
|
||||
|
||||
# Author & License
|
||||
If you get an error regarding installation of the diskUsage module,
|
||||
please install g++.
|
||||
|
||||
- [Zenko CloudServer](https://s3-server.readthedocs.io/en/latest/) author is Scality, licensed under [Apache License, version 2.0](https://www.apache.org/licenses/LICENSE-2.0)
|
||||
- [Vitastor](https://git.yourcmc.ru/vitalif/vitastor/) and Zenko Vitastor backend author is Vitaliy Filippov, licensed under [VNPL-1.1](https://git.yourcmc.ru/vitalif/vitastor/src/branch/master/VNPL-1.1.txt)
|
||||
(a "network copyleft" license based on AGPL/SSPL, but worded in a better way)
|
||||
If you get an error regarding level-down bindings, try clearing your npm cache:
|
||||
|
||||
```shell
|
||||
npm cache clear
|
||||
```
|
||||
|
||||
## Run it with a file backend
|
||||
|
||||
```shell
|
||||
npm start
|
||||
```
|
||||
|
||||
This starts a Zenko CloudServer on port 8000. Two additional ports 9990 and
|
||||
9991 are also open locally for internal transfer of metadata and data,
|
||||
respectively.
|
||||
|
||||
The default access key is accessKey1 with
|
||||
a secret key of verySecretKey1.
|
||||
|
||||
By default the metadata files will be saved in the
|
||||
localMetadata directory and the data files will be saved
|
||||
in the localData directory within the ./S3 directory on your
|
||||
machine. These directories have been pre-created within the
|
||||
repository. If you would like to save the data or metadata in
|
||||
different locations of your choice, you must specify them with absolute paths.
|
||||
So, when starting the server:
|
||||
|
||||
```shell
|
||||
mkdir -m 700 $(pwd)/myFavoriteDataPath
|
||||
mkdir -m 700 $(pwd)/myFavoriteMetadataPath
|
||||
export S3DATAPATH="$(pwd)/myFavoriteDataPath"
|
||||
export S3METADATAPATH="$(pwd)/myFavoriteMetadataPath"
|
||||
npm start
|
||||
```
|
||||
|
||||
## Run it with multiple data backends
|
||||
|
||||
```shell
|
||||
export S3DATA='multiple'
|
||||
npm start
|
||||
```
|
||||
|
||||
This starts a Zenko CloudServer on port 8000.
|
||||
The default access key is accessKey1 with
|
||||
a secret key of verySecretKey1.
|
||||
|
||||
With multiple backends, you have the ability to
|
||||
choose where each object will be saved by setting
|
||||
the following header with a locationConstraint on
|
||||
a PUT request:
|
||||
|
||||
```shell
|
||||
'x-amz-meta-scal-location-constraint':'myLocationConstraint'
|
||||
```
|
||||
|
||||
If no header is sent with a PUT object request, the
|
||||
location constraint of the bucket will determine
|
||||
where the data is saved. If the bucket has no location
|
||||
constraint, the endpoint of the PUT request will be
|
||||
used to determine location.
|
||||
|
||||
See the Configuration section below to learn how to set
|
||||
location constraints.
|
||||
|
||||
## Run it with an in-memory backend
|
||||
|
||||
```shell
|
||||
npm run mem_backend
|
||||
```
|
||||
|
||||
This starts a Zenko CloudServer on port 8000.
|
||||
The default access key is accessKey1 with
|
||||
a secret key of verySecretKey1.
|
||||
|
||||
[badgetwitter]: https://img.shields.io/twitter/follow/zenko.svg?style=social&label=Follow
|
||||
[badgedocker]: https://img.shields.io/docker/pulls/scality/s3server.svg
|
||||
[badgepub]: https://circleci.com/gh/scality/S3.svg?style=svg
|
||||
[badgepriv]: http://ci.ironmann.io/gh/scality/S3.svg?style=svg&circle-token=1f105b7518b53853b5b7cf72302a3f75d8c598ae
|
||||
|
|
|
@ -1,2 +0,0 @@
|
|||
---
|
||||
theme: jekyll-theme-modernist
|
|
@ -1,56 +0,0 @@
|
|||
{
|
||||
"accounts": [{
|
||||
"name": "Bart",
|
||||
"email": "sampleaccount1@sampling.com",
|
||||
"arn": "arn:aws:iam::123456789012:root",
|
||||
"canonicalID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be",
|
||||
"shortid": "123456789012",
|
||||
"keys": [{
|
||||
"access": "accessKey1",
|
||||
"secret": "verySecretKey1"
|
||||
}]
|
||||
}, {
|
||||
"name": "Lisa",
|
||||
"email": "sampleaccount2@sampling.com",
|
||||
"arn": "arn:aws:iam::123456789013:root",
|
||||
"canonicalID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2bf",
|
||||
"shortid": "123456789013",
|
||||
"keys": [{
|
||||
"access": "accessKey2",
|
||||
"secret": "verySecretKey2"
|
||||
}]
|
||||
},
|
||||
{
|
||||
"name": "Clueso",
|
||||
"email": "inspector@clueso.info",
|
||||
"arn": "arn:aws:iam::123456789014:root",
|
||||
"canonicalID": "http://acs.zenko.io/accounts/service/clueso",
|
||||
"shortid": "123456789014",
|
||||
"keys": [{
|
||||
"access": "cluesoKey1",
|
||||
"secret": "cluesoSecretKey1"
|
||||
}]
|
||||
},
|
||||
{
|
||||
"name": "Replication",
|
||||
"email": "inspector@replication.info",
|
||||
"arn": "arn:aws:iam::123456789015:root",
|
||||
"canonicalID": "http://acs.zenko.io/accounts/service/replication",
|
||||
"shortid": "123456789015",
|
||||
"keys": [{
|
||||
"access": "replicationKey1",
|
||||
"secret": "replicationSecretKey1"
|
||||
}]
|
||||
},
|
||||
{
|
||||
"name": "Lifecycle",
|
||||
"email": "inspector@lifecycle.info",
|
||||
"arn": "arn:aws:iam::123456789016:root",
|
||||
"canonicalID": "http://acs.zenko.io/accounts/service/lifecycle",
|
||||
"shortid": "123456789016",
|
||||
"keys": [{
|
||||
"access": "lifecycleKey1",
|
||||
"secret": "lifecycleSecretKey1"
|
||||
}]
|
||||
}]
|
||||
}
|
|
@ -0,0 +1,554 @@
|
|||
const { errors, s3middleware } = require('arsenal');
|
||||
const AWS = require('aws-sdk');
|
||||
const MD5Sum = s3middleware.MD5Sum;
|
||||
const getMetaHeaders = s3middleware.userMetadata.getMetaHeaders;
|
||||
const createLogger = require('../multipleBackendLogger');
|
||||
const { prepareStream } = require('../../api/apiUtils/object/prepareStream');
|
||||
const { logHelper, removeQuotes, trimXMetaPrefix } = require('./utils');
|
||||
const { config } = require('../../Config');
|
||||
|
||||
const missingVerIdInternalError = errors.InternalError.customizeDescription(
|
||||
'Invalid state. Please ensure versioning is enabled ' +
|
||||
'in AWS for the location constraint and try again.'
|
||||
);
|
||||
|
||||
class AwsClient {
|
||||
constructor(config) {
|
||||
this.clientType = 'aws_s3';
|
||||
this._s3Params = config.s3Params;
|
||||
this._awsBucketName = config.bucketName;
|
||||
this._bucketMatch = config.bucketMatch;
|
||||
this._dataStoreName = config.dataStoreName;
|
||||
this._serverSideEncryption = config.serverSideEncryption;
|
||||
this._client = new AWS.S3(this._s3Params);
|
||||
}
|
||||
|
||||
_createAwsKey(requestBucketName, requestObjectKey,
|
||||
bucketMatch) {
|
||||
if (bucketMatch) {
|
||||
return requestObjectKey;
|
||||
}
|
||||
return `${requestBucketName}/${requestObjectKey}`;
|
||||
}
|
||||
put(stream, size, keyContext, reqUids, callback) {
|
||||
const awsKey = this._createAwsKey(keyContext.bucketName,
|
||||
keyContext.objectKey, this._bucketMatch);
|
||||
const metaHeaders = trimXMetaPrefix(keyContext.metaHeaders);
|
||||
const log = createLogger(reqUids);
|
||||
|
||||
const putCb = (err, data) => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'err from data backend',
|
||||
err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS: ${err.message}`)
|
||||
);
|
||||
}
|
||||
if (!data.VersionId) {
|
||||
logHelper(log, 'error', 'missing version id for data ' +
|
||||
'backend object', missingVerIdInternalError,
|
||||
this._dataStoreName);
|
||||
return callback(missingVerIdInternalError);
|
||||
}
|
||||
const dataStoreVersionId = data.VersionId;
|
||||
return callback(null, awsKey, dataStoreVersionId);
|
||||
};
|
||||
|
||||
const params = {
|
||||
Bucket: this._awsBucketName,
|
||||
Key: awsKey,
|
||||
};
|
||||
// we call data.put to create a delete marker, but it's actually a
|
||||
// delete request in call to AWS
|
||||
if (keyContext.isDeleteMarker) {
|
||||
return this._client.deleteObject(params, putCb);
|
||||
}
|
||||
const uploadParams = params;
|
||||
uploadParams.Metadata = metaHeaders;
|
||||
uploadParams.ContentLength = size;
|
||||
if (this._serverSideEncryption) {
|
||||
uploadParams.ServerSideEncryption = 'AES256';
|
||||
}
|
||||
if (keyContext.tagging) {
|
||||
uploadParams.Tagging = keyContext.tagging;
|
||||
}
|
||||
if (keyContext.contentType !== undefined) {
|
||||
uploadParams.ContentType = keyContext.contentType;
|
||||
}
|
||||
if (keyContext.cacheControl !== undefined) {
|
||||
uploadParams.CacheControl = keyContext.cacheControl;
|
||||
}
|
||||
if (keyContext.contentDisposition !== undefined) {
|
||||
uploadParams.ContentDisposition = keyContext.contentDisposition;
|
||||
}
|
||||
if (keyContext.contentEncoding !== undefined) {
|
||||
uploadParams.ContentEncoding = keyContext.contentEncoding;
|
||||
}
|
||||
if (!stream) {
|
||||
return this._client.putObject(uploadParams, putCb);
|
||||
}
|
||||
|
||||
uploadParams.Body = stream;
|
||||
return this._client.upload(uploadParams, putCb);
|
||||
}
|
||||
head(objectGetInfo, reqUids, callback) {
|
||||
const log = createLogger(reqUids);
|
||||
const { key, dataStoreVersionId } = objectGetInfo;
|
||||
return this._client.headObject({
|
||||
Bucket: this._awsBucketName,
|
||||
Key: key,
|
||||
VersionId: dataStoreVersionId,
|
||||
}, err => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'error heading object ' +
|
||||
'from datastore', err, this._dataStoreName);
|
||||
if (err.code === 'NotFound') {
|
||||
const error = errors.ServiceUnavailable
|
||||
.customizeDescription(
|
||||
'Unexpected error from AWS: "NotFound". Data on AWS ' +
|
||||
'may have been altered outside of CloudServer.'
|
||||
);
|
||||
return callback(error);
|
||||
}
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS: ${err.message}`)
|
||||
);
|
||||
}
|
||||
return callback();
|
||||
});
|
||||
}
|
||||
get(objectGetInfo, range, reqUids, callback) {
|
||||
const log = createLogger(reqUids);
|
||||
const { key, dataStoreVersionId } = objectGetInfo;
|
||||
const request = this._client.getObject({
|
||||
Bucket: this._awsBucketName,
|
||||
Key: key,
|
||||
VersionId: dataStoreVersionId,
|
||||
Range: range ? `bytes=${range[0]}-${range[1]}` : null,
|
||||
}).on('success', response => {
|
||||
log.trace('AWS GET request response headers',
|
||||
{ responseHeaders: response.httpResponse.headers });
|
||||
});
|
||||
const stream = request.createReadStream().on('error', err => {
|
||||
logHelper(log, 'error', 'error streaming data from AWS',
|
||||
err, this._dataStoreName);
|
||||
return callback(err);
|
||||
});
|
||||
return callback(null, stream);
|
||||
}
|
||||
delete(objectGetInfo, reqUids, callback) {
|
||||
const { key, dataStoreVersionId, deleteVersion } = objectGetInfo;
|
||||
const log = createLogger(reqUids);
|
||||
const params = {
|
||||
Bucket: this._awsBucketName,
|
||||
Key: key,
|
||||
};
|
||||
if (deleteVersion) {
|
||||
params.VersionId = dataStoreVersionId;
|
||||
}
|
||||
return this._client.deleteObject(params, err => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'error deleting object from ' +
|
||||
'datastore', err, this._dataStoreName);
|
||||
if (err.code === 'NoSuchVersion') {
|
||||
// data may have been deleted directly from the AWS backend
|
||||
// don't want to retry the delete and errors are not
|
||||
// sent back to client anyway, so no need to return err
|
||||
return callback();
|
||||
}
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS: ${err.message}`)
|
||||
);
|
||||
}
|
||||
return callback();
|
||||
});
|
||||
}
|
||||
|
||||
healthcheck(location, callback) {
|
||||
const awsResp = {};
|
||||
this._client.headBucket({ Bucket: this._awsBucketName },
|
||||
err => {
|
||||
/* eslint-disable no-param-reassign */
|
||||
if (err) {
|
||||
awsResp[location] = { error: err, external: true };
|
||||
return callback(null, awsResp);
|
||||
}
|
||||
return this._client.getBucketVersioning({
|
||||
Bucket: this._awsBucketName },
|
||||
(err, data) => {
|
||||
if (err) {
|
||||
awsResp[location] = { error: err, external: true };
|
||||
} else if (!data.Status ||
|
||||
data.Status === 'Suspended') {
|
||||
awsResp[location] = {
|
||||
versioningStatus: data.Status,
|
||||
error: 'Versioning must be enabled',
|
||||
external: true,
|
||||
};
|
||||
} else {
|
||||
awsResp[location] = {
|
||||
versioningStatus: data.Status,
|
||||
message: 'Congrats! You own the bucket',
|
||||
};
|
||||
}
|
||||
return callback(null, awsResp);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
createMPU(key, metaHeaders, bucketName, websiteRedirectHeader, contentType,
|
||||
cacheControl, contentDisposition, contentEncoding, log, callback) {
|
||||
const metaHeadersTrimmed = {};
|
||||
Object.keys(metaHeaders).forEach(header => {
|
||||
if (header.startsWith('x-amz-meta-')) {
|
||||
const headerKey = header.substring(11);
|
||||
metaHeadersTrimmed[headerKey] = metaHeaders[header];
|
||||
}
|
||||
});
|
||||
Object.assign(metaHeaders, metaHeadersTrimmed);
|
||||
const awsBucket = this._awsBucketName;
|
||||
const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
|
||||
const params = {
|
||||
Bucket: awsBucket,
|
||||
Key: awsKey,
|
||||
WebsiteRedirectLocation: websiteRedirectHeader,
|
||||
Metadata: metaHeaders,
|
||||
ContentType: contentType,
|
||||
CacheControl: cacheControl,
|
||||
ContentDisposition: contentDisposition,
|
||||
ContentEncoding: contentEncoding,
|
||||
};
|
||||
return this._client.createMultipartUpload(params, (err, mpuResObj) => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'err from data backend',
|
||||
err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS: ${err.message}`)
|
||||
);
|
||||
}
|
||||
return callback(null, mpuResObj);
|
||||
});
|
||||
}
|
||||
|
||||
uploadPart(request, streamingV4Params, stream, size, key, uploadId,
|
||||
partNumber, bucketName, log, callback) {
|
||||
let hashedStream = stream;
|
||||
if (request) {
|
||||
const partStream = prepareStream(request, streamingV4Params,
|
||||
log, callback);
|
||||
hashedStream = new MD5Sum();
|
||||
partStream.pipe(hashedStream);
|
||||
}
|
||||
|
||||
const awsBucket = this._awsBucketName;
|
||||
const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
|
||||
const params = { Bucket: awsBucket, Key: awsKey, UploadId: uploadId,
|
||||
Body: hashedStream, ContentLength: size,
|
||||
PartNumber: partNumber };
|
||||
return this._client.uploadPart(params, (err, partResObj) => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'err from data backend ' +
|
||||
'on uploadPart', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS: ${err.message}`)
|
||||
);
|
||||
}
|
||||
// Because we manually add quotes to ETag later, remove quotes here
|
||||
const noQuotesETag =
|
||||
partResObj.ETag.substring(1, partResObj.ETag.length - 1);
|
||||
const dataRetrievalInfo = {
|
||||
key: awsKey,
|
||||
dataStoreType: 'aws_s3',
|
||||
dataStoreName: this._dataStoreName,
|
||||
dataStoreETag: noQuotesETag,
|
||||
};
|
||||
return callback(null, dataRetrievalInfo);
|
||||
});
|
||||
}
|
||||
|
||||
listParts(key, uploadId, bucketName, partNumberMarker, maxParts, log,
|
||||
callback) {
|
||||
const awsBucket = this._awsBucketName;
|
||||
const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
|
||||
const params = { Bucket: awsBucket, Key: awsKey, UploadId: uploadId,
|
||||
PartNumberMarker: partNumberMarker, MaxParts: maxParts };
|
||||
return this._client.listParts(params, (err, partList) => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'err from data backend on listPart',
|
||||
err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS: ${err.message}`)
|
||||
);
|
||||
}
|
||||
// build storedParts object to mimic Scality S3 backend returns
|
||||
const storedParts = {};
|
||||
storedParts.IsTruncated = partList.IsTruncated;
|
||||
storedParts.Contents = [];
|
||||
storedParts.Contents = partList.Parts.map(item => {
|
||||
// We manually add quotes to ETag later, so remove quotes here
|
||||
const noQuotesETag =
|
||||
item.ETag.substring(1, item.ETag.length - 1);
|
||||
return {
|
||||
partNumber: item.PartNumber,
|
||||
value: {
|
||||
Size: item.Size,
|
||||
ETag: noQuotesETag,
|
||||
LastModified: item.LastModified,
|
||||
},
|
||||
};
|
||||
});
|
||||
return callback(null, storedParts);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* completeMPU - complete multipart upload on AWS backend
|
||||
* @param {object} jsonList - user-sent list of parts to include in
|
||||
* final mpu object
|
||||
* @param {object} mdInfo - object containing 3 keys: storedParts,
|
||||
* mpuOverviewKey, and splitter
|
||||
* @param {string} key - object key
|
||||
* @param {string} uploadId - multipart upload id string
|
||||
* @param {string} bucketName - name of bucket
|
||||
* @param {RequestLogger} log - logger instance
|
||||
* @param {function} callback - callback function
|
||||
* @return {(Error|object)} - return Error if complete MPU fails, otherwise
|
||||
* object containing completed object key, eTag, and contentLength
|
||||
*/
|
||||
completeMPU(jsonList, mdInfo, key, uploadId, bucketName, log, callback) {
|
||||
const awsBucket = this._awsBucketName;
|
||||
const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
|
||||
const mpuError = {
|
||||
InvalidPart: true,
|
||||
InvalidPartOrder: true,
|
||||
EntityTooSmall: true,
|
||||
};
|
||||
const partArray = [];
|
||||
const partList = jsonList.Part;
|
||||
partList.forEach(partObj => {
|
||||
const partParams = { PartNumber: partObj.PartNumber[0],
|
||||
ETag: partObj.ETag[0] };
|
||||
partArray.push(partParams);
|
||||
});
|
||||
const mpuParams = {
|
||||
Bucket: awsBucket, Key: awsKey, UploadId: uploadId,
|
||||
MultipartUpload: {
|
||||
Parts: partArray,
|
||||
},
|
||||
};
|
||||
const completeObjData = { key: awsKey };
|
||||
return this._client.completeMultipartUpload(mpuParams,
|
||||
(err, completeMpuRes) => {
|
||||
if (err) {
|
||||
if (mpuError[err.code]) {
|
||||
logHelper(log, 'trace', 'err from data backend on ' +
|
||||
'completeMPU', err, this._dataStoreName);
|
||||
return callback(errors[err.code]);
|
||||
}
|
||||
logHelper(log, 'error', 'err from data backend on ' +
|
||||
'completeMPU', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS: ${err.message}`)
|
||||
);
|
||||
}
|
||||
if (!completeMpuRes.VersionId) {
|
||||
logHelper(log, 'error', 'missing version id for data ' +
|
||||
'backend object', missingVerIdInternalError,
|
||||
this._dataStoreName);
|
||||
return callback(missingVerIdInternalError);
|
||||
}
|
||||
// need to get content length of new object to store
|
||||
// in our metadata
|
||||
return this._client.headObject({ Bucket: awsBucket, Key: awsKey },
|
||||
(err, objHeaders) => {
|
||||
if (err) {
|
||||
logHelper(log, 'trace', 'err from data backend on ' +
|
||||
'headObject', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS: ${err.message}`)
|
||||
);
|
||||
}
|
||||
// remove quotes from eTag because they're added later
|
||||
completeObjData.eTag = completeMpuRes.ETag
|
||||
.substring(1, completeMpuRes.ETag.length - 1);
|
||||
completeObjData.dataStoreVersionId = completeMpuRes.VersionId;
|
||||
completeObjData.contentLength = objHeaders.ContentLength;
|
||||
return callback(null, completeObjData);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
abortMPU(key, uploadId, bucketName, log, callback) {
|
||||
const awsBucket = this._awsBucketName;
|
||||
const awsKey = this._createAwsKey(bucketName, key, this._bucketMatch);
|
||||
const abortParams = {
|
||||
Bucket: awsBucket, Key: awsKey, UploadId: uploadId,
|
||||
};
|
||||
return this._client.abortMultipartUpload(abortParams, err => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'There was an error aborting ' +
|
||||
'the MPU on AWS S3. You should abort directly on AWS S3 ' +
|
||||
'using the same uploadId.', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS: ${err.message}`)
|
||||
);
|
||||
}
|
||||
return callback();
|
||||
});
|
||||
}
|
||||
|
||||
objectPutTagging(key, bucket, objectMD, log, callback) {
|
||||
const awsBucket = this._awsBucketName;
|
||||
const awsKey = this._createAwsKey(bucket, key, this._bucketMatch);
|
||||
const dataStoreVersionId = objectMD.location[0].dataStoreVersionId;
|
||||
const tagParams = {
|
||||
Bucket: awsBucket,
|
||||
Key: awsKey,
|
||||
VersionId: dataStoreVersionId,
|
||||
};
|
||||
const keyArray = Object.keys(objectMD.tags);
|
||||
tagParams.Tagging = {};
|
||||
tagParams.Tagging.TagSet = keyArray.map(key => {
|
||||
const value = objectMD.tags[key];
|
||||
return { Key: key, Value: value };
|
||||
});
|
||||
return this._client.putObjectTagging(tagParams, err => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'error from data backend on ' +
|
||||
'putObjectTagging', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS: ${err.message}`)
|
||||
);
|
||||
}
|
||||
return callback();
|
||||
});
|
||||
}
|
||||
|
||||
objectDeleteTagging(key, bucket, objectMD, log, callback) {
|
||||
const awsBucket = this._awsBucketName;
|
||||
const awsKey = this._createAwsKey(bucket, key, this._bucketMatch);
|
||||
const dataStoreVersionId = objectMD.location[0].dataStoreVersionId;
|
||||
const tagParams = {
|
||||
Bucket: awsBucket,
|
||||
Key: awsKey,
|
||||
VersionId: dataStoreVersionId,
|
||||
};
|
||||
return this._client.deleteObjectTagging(tagParams, err => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'error from data backend on ' +
|
||||
'deleteObjectTagging', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS: ${err.message}`)
|
||||
);
|
||||
}
|
||||
return callback();
|
||||
});
|
||||
}
|
||||
copyObject(request, destLocationConstraintName, sourceKey,
|
||||
sourceLocationConstraintName, storeMetadataParams, log, callback) {
|
||||
const destBucketName = request.bucketName;
|
||||
const destObjectKey = request.objectKey;
|
||||
const destAwsKey = this._createAwsKey(destBucketName, destObjectKey,
|
||||
this._bucketMatch);
|
||||
|
||||
const sourceAwsBucketName =
|
||||
config.getAwsBucketName(sourceLocationConstraintName);
|
||||
|
||||
const metadataDirective = request.headers['x-amz-metadata-directive'];
|
||||
const metaHeaders = trimXMetaPrefix(getMetaHeaders(request.headers));
|
||||
const awsParams = {
|
||||
Bucket: this._awsBucketName,
|
||||
Key: destAwsKey,
|
||||
CopySource: `${sourceAwsBucketName}/${sourceKey}`,
|
||||
Metadata: metaHeaders,
|
||||
MetadataDirective: metadataDirective,
|
||||
};
|
||||
if (destLocationConstraintName &&
|
||||
config.isAWSServerSideEncrytion(destLocationConstraintName)) {
|
||||
awsParams.ServerSideEncryption = 'AES256';
|
||||
}
|
||||
this._client.copyObject(awsParams, (err, copyResult) => {
|
||||
if (err) {
|
||||
if (err.code === 'AccessDenied') {
|
||||
logHelper(log, 'error', 'Unable to access ' +
|
||||
`${sourceAwsBucketName} AWS bucket`, err,
|
||||
this._dataStoreName);
|
||||
return callback(errors.AccessDenied
|
||||
.customizeDescription('Error: Unable to access ' +
|
||||
`${sourceAwsBucketName} AWS bucket`)
|
||||
);
|
||||
}
|
||||
logHelper(log, 'error', 'error from data backend on ' +
|
||||
'copyObject', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS: ${err.message}`)
|
||||
);
|
||||
}
|
||||
if (!copyResult.VersionId) {
|
||||
logHelper(log, 'error', 'missing version id for data ' +
|
||||
'backend object', missingVerIdInternalError,
|
||||
this._dataStoreName);
|
||||
return callback(missingVerIdInternalError);
|
||||
}
|
||||
return callback(null, destAwsKey, copyResult.VersionId);
|
||||
});
|
||||
}
|
||||
uploadPartCopy(request, awsSourceKey, sourceLocationConstraintName,
|
||||
log, callback) {
|
||||
const destBucketName = request.bucketName;
|
||||
const destObjectKey = request.objectKey;
|
||||
const destAwsKey = this._createAwsKey(destBucketName, destObjectKey,
|
||||
this._bucketMatch);
|
||||
|
||||
const sourceAwsBucketName =
|
||||
config.getAwsBucketName(sourceLocationConstraintName);
|
||||
|
||||
const uploadId = request.query.uploadId;
|
||||
const partNumber = request.query.partNumber;
|
||||
const copySourceRange = request.headers['x-amz-copy-source-range'];
|
||||
|
||||
const params = {
|
||||
Bucket: this._awsBucketName,
|
||||
CopySource: `${sourceAwsBucketName}/${awsSourceKey}`,
|
||||
CopySourceRange: copySourceRange,
|
||||
Key: destAwsKey,
|
||||
PartNumber: partNumber,
|
||||
UploadId: uploadId,
|
||||
};
|
||||
return this._client.uploadPartCopy(params, (err, res) => {
|
||||
if (err) {
|
||||
if (err.code === 'AccessDenied') {
|
||||
logHelper(log, 'error', 'Unable to access ' +
|
||||
`${sourceAwsBucketName} AWS bucket`, err,
|
||||
this._dataStoreName);
|
||||
return callback(errors.AccessDenied
|
||||
.customizeDescription('Error: Unable to access ' +
|
||||
`${sourceAwsBucketName} AWS bucket`)
|
||||
);
|
||||
}
|
||||
logHelper(log, 'error', 'error from data backend on ' +
|
||||
'uploadPartCopy', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS: ${err.message}`)
|
||||
);
|
||||
}
|
||||
const eTag = removeQuotes(res.CopyPartResult.ETag);
|
||||
return callback(null, eTag);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = AwsClient;
|
|
@ -0,0 +1,434 @@
|
|||
const url = require('url');
|
||||
|
||||
const { errors, s3middleware } = require('arsenal');
|
||||
const azure = require('azure-storage');
|
||||
const createLogger = require('../multipleBackendLogger');
|
||||
const { logHelper, translateAzureMetaHeaders } = require('./utils');
|
||||
const { config } = require('../../Config');
|
||||
const { validateAndFilterMpuParts } =
|
||||
require('../../api/apiUtils/object/processMpuParts');
|
||||
const constants = require('../../../constants');
|
||||
const metadata = require('../../metadata/wrapper');
|
||||
const azureMpuUtils = s3middleware.azureHelper.mpuUtils;
|
||||
|
||||
class AzureClient {
|
||||
constructor(config) {
|
||||
this._azureStorageEndpoint = config.azureStorageEndpoint;
|
||||
this._azureStorageCredentials = config.azureStorageCredentials;
|
||||
this._azureContainerName = config.azureContainerName;
|
||||
this._client = azure.createBlobService(
|
||||
this._azureStorageCredentials.storageAccountName,
|
||||
this._azureStorageCredentials.storageAccessKey,
|
||||
this._azureStorageEndpoint);
|
||||
this._dataStoreName = config.dataStoreName;
|
||||
this._bucketMatch = config.bucketMatch;
|
||||
if (config.proxy) {
|
||||
const parsedUrl = url.parse(config.proxy);
|
||||
if (!parsedUrl.port) {
|
||||
parsedUrl.port = 80;
|
||||
}
|
||||
this._client.setProxy(parsedUrl);
|
||||
}
|
||||
}
|
||||
|
||||
_errorWrapper(s3Method, azureMethod, args, log, cb) {
|
||||
if (log) {
|
||||
log.info(`calling azure ${azureMethod}`);
|
||||
}
|
||||
try {
|
||||
this._client[azureMethod].apply(this._client, args);
|
||||
} catch (err) {
|
||||
const error = errors.ServiceUnavailable;
|
||||
if (log) {
|
||||
log.error('error thrown by Azure Storage Client Library',
|
||||
{ error: err.message, stack: err.stack, s3Method,
|
||||
azureMethod, dataStoreName: this._dataStoreName });
|
||||
}
|
||||
cb(error.customizeDescription('Error from Azure ' +
|
||||
`method: ${azureMethod} on ${s3Method} S3 call: ` +
|
||||
`${err.message}`));
|
||||
}
|
||||
}
|
||||
|
||||
_createAzureKey(requestBucketName, requestObjectKey,
|
||||
bucketMatch) {
|
||||
if (bucketMatch) {
|
||||
return requestObjectKey;
|
||||
}
|
||||
return `${requestBucketName}/${requestObjectKey}`;
|
||||
}
|
||||
|
||||
_getMetaHeaders(objectMD) {
|
||||
const metaHeaders = {};
|
||||
Object.keys(objectMD).forEach(mdKey => {
|
||||
const isMetaHeader = mdKey.startsWith('x-amz-meta-');
|
||||
if (isMetaHeader) {
|
||||
metaHeaders[mdKey] = objectMD[mdKey];
|
||||
}
|
||||
});
|
||||
return translateAzureMetaHeaders(metaHeaders);
|
||||
}
|
||||
|
||||
// Before putting or deleting object on Azure, check if MPU exists with
|
||||
// same key name. If it does, do not allow put or delete because Azure
|
||||
// will delete all blocks with same key name
|
||||
protectAzureBlocks(bucketName, objectKey, dataStoreName, log, cb) {
|
||||
const mpuBucketName = `${constants.mpuBucketPrefix}${bucketName}`;
|
||||
const splitter = constants.splitter;
|
||||
const listingParams = {
|
||||
prefix: `overview${splitter}${objectKey}`,
|
||||
listingType: 'MPU',
|
||||
splitter,
|
||||
maxKeys: 1,
|
||||
};
|
||||
|
||||
return metadata.listMultipartUploads(mpuBucketName, listingParams,
|
||||
log, (err, mpuList) => {
|
||||
if (err && !err.NoSuchBucket) {
|
||||
log.error('Error listing MPUs for Azure delete',
|
||||
{ error: err, dataStoreName });
|
||||
return cb(errors.ServiceUnavailable);
|
||||
}
|
||||
if (mpuList && mpuList.Uploads && mpuList.Uploads.length > 0) {
|
||||
const error = errors.MPUinProgress;
|
||||
log.error('Error: cannot put/delete object to Azure with ' +
|
||||
'same key name as ongoing MPU on Azure',
|
||||
{ error, dataStoreName });
|
||||
return cb(error);
|
||||
}
|
||||
// If listMultipartUploads returns a NoSuchBucket error or the
|
||||
// mpu list is empty, there are no conflicting MPUs, so continue
|
||||
return cb();
|
||||
});
|
||||
}
|
||||
|
||||
put(stream, size, keyContext, reqUids, callback) {
|
||||
const log = createLogger(reqUids);
|
||||
// before blob is put, make sure there is no ongoing MPU with same key
|
||||
this.protectAzureBlocks(keyContext.bucketName,
|
||||
keyContext.objectKey, this._dataStoreName, log, err => {
|
||||
// if error returned, there is ongoing MPU, so do not put
|
||||
if (err) {
|
||||
return callback(err.customizeDescription(
|
||||
`Error putting object to Azure: ${err.message}`));
|
||||
}
|
||||
const azureKey = this._createAzureKey(keyContext.bucketName,
|
||||
keyContext.objectKey, this._bucketMatch);
|
||||
const options = {
|
||||
metadata: translateAzureMetaHeaders(keyContext.metaHeaders,
|
||||
keyContext.tagging),
|
||||
contentSettings: {
|
||||
contentType: keyContext.contentType || undefined,
|
||||
cacheControl: keyContext.cacheControl || undefined,
|
||||
contentDisposition: keyContext.contentDisposition ||
|
||||
undefined,
|
||||
contentEncoding: keyContext.contentEncoding || undefined,
|
||||
},
|
||||
};
|
||||
if (size === 0) {
|
||||
return this._errorWrapper('put', 'createBlockBlobFromText',
|
||||
[this._azureContainerName, azureKey, '', options,
|
||||
err => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'err from Azure PUT data ' +
|
||||
'backend', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`Azure: ${err.message}`));
|
||||
}
|
||||
return callback(null, azureKey);
|
||||
}], log, callback);
|
||||
}
|
||||
return this._errorWrapper('put', 'createBlockBlobFromStream',
|
||||
[this._azureContainerName, azureKey, stream, size, options,
|
||||
err => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'err from Azure PUT data ' +
|
||||
'backend', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`Azure: ${err.message}`));
|
||||
}
|
||||
return callback(null, azureKey);
|
||||
}], log, callback);
|
||||
});
|
||||
}
|
||||
|
||||
head(objectGetInfo, reqUids, callback) {
|
||||
const log = createLogger(reqUids);
|
||||
const { key, azureStreamingOptions } = objectGetInfo;
|
||||
return this._errorWrapper('head', 'getBlobProperties',
|
||||
[this._azureContainerName, key, azureStreamingOptions,
|
||||
err => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'err from Azure HEAD data backend',
|
||||
err, this._dataStoreName);
|
||||
if (err.code === 'NotFound') {
|
||||
const error = errors.ServiceUnavailable
|
||||
.customizeDescription(
|
||||
'Unexpected error from Azure: "NotFound". Data ' +
|
||||
'on Azure may have been altered outside of ' +
|
||||
'CloudServer.');
|
||||
return callback(error);
|
||||
}
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`Azure: ${err.message}`));
|
||||
}
|
||||
return callback();
|
||||
}], log, callback);
|
||||
}
|
||||
|
||||
get(objectGetInfo, range, reqUids, callback) {
|
||||
const log = createLogger(reqUids);
|
||||
// for backwards compatibility
|
||||
const { key, response, azureStreamingOptions } = objectGetInfo;
|
||||
let streamingOptions;
|
||||
if (azureStreamingOptions) {
|
||||
// option coming from api.get()
|
||||
streamingOptions = azureStreamingOptions;
|
||||
} else if (range) {
|
||||
// option coming from multipleBackend.upload()
|
||||
const rangeStart = range[0] ? range[0].toString() : undefined;
|
||||
const rangeEnd = range[1] ? range[1].toString() : undefined;
|
||||
streamingOptions = { rangeStart, rangeEnd };
|
||||
}
|
||||
this._errorWrapper('get', 'getBlobToStream',
|
||||
[this._azureContainerName, key, response, streamingOptions,
|
||||
err => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'err from Azure GET data backend',
|
||||
err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable);
|
||||
}
|
||||
return callback(null, response);
|
||||
}], log, callback);
|
||||
}
|
||||
|
||||
delete(objectGetInfo, reqUids, callback) {
|
||||
const log = createLogger(reqUids);
|
||||
// for backwards compatibility
|
||||
const key = typeof objectGetInfo === 'string' ? objectGetInfo :
|
||||
objectGetInfo.key;
|
||||
return this._errorWrapper('delete', 'deleteBlobIfExists',
|
||||
[this._azureContainerName, key,
|
||||
err => {
|
||||
if (err) {
|
||||
const log = createLogger(reqUids);
|
||||
logHelper(log, 'error', 'error deleting object from ' +
|
||||
'Azure datastore', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`Azure: ${err.message}`));
|
||||
}
|
||||
return callback();
|
||||
}], log, callback);
|
||||
}
|
||||
|
||||
healthcheck(location, callback, flightCheckOnStartUp) {
|
||||
const azureResp = {};
|
||||
const healthCheckAction = flightCheckOnStartUp ?
|
||||
'createContainerIfNotExists' : 'doesContainerExist';
|
||||
this._errorWrapper('checkAzureHealth', healthCheckAction,
|
||||
[this._azureContainerName, err => {
|
||||
/* eslint-disable no-param-reassign */
|
||||
if (err) {
|
||||
azureResp[location] = { error: err.message,
|
||||
external: true };
|
||||
return callback(null, azureResp);
|
||||
}
|
||||
azureResp[location] = {
|
||||
message:
|
||||
'Congrats! You can access the Azure storage account',
|
||||
};
|
||||
return callback(null, azureResp);
|
||||
}], null, callback);
|
||||
}
|
||||
|
||||
uploadPart(request, streamingV4Params, partStream, size, key, uploadId,
|
||||
partNumber, bucket, log, callback) {
|
||||
const azureKey = this._createAzureKey(bucket, key, this._bucketMatch);
|
||||
const params = { bucketName: this._azureContainerName,
|
||||
partNumber, size, objectKey: azureKey, uploadId };
|
||||
const stream = request || partStream;
|
||||
|
||||
if (request && request.headers && request.headers['content-md5']) {
|
||||
params.contentMD5 = request.headers['content-md5'];
|
||||
}
|
||||
const dataRetrievalInfo = {
|
||||
key: azureKey,
|
||||
partNumber,
|
||||
dataStoreName: this._dataStoreName,
|
||||
dataStoreType: 'azure',
|
||||
numberSubParts: azureMpuUtils.getSubPartInfo(size)
|
||||
.expectedNumberSubParts,
|
||||
};
|
||||
|
||||
if (size === 0) {
|
||||
log.debug('0-byte part does not store data',
|
||||
{ method: 'uploadPart' });
|
||||
dataRetrievalInfo.dataStoreETag = azureMpuUtils.zeroByteETag;
|
||||
dataRetrievalInfo.numberSubParts = 0;
|
||||
return callback(null, dataRetrievalInfo);
|
||||
}
|
||||
if (size <= azureMpuUtils.maxSubPartSize) {
|
||||
const errorWrapperFn = this._errorWrapper.bind(this);
|
||||
return azureMpuUtils.putSinglePart(errorWrapperFn,
|
||||
stream, params, this._dataStoreName, log, (err, dataStoreETag) => {
|
||||
if (err) {
|
||||
return callback(err);
|
||||
}
|
||||
dataRetrievalInfo.dataStoreETag = dataStoreETag;
|
||||
return callback(null, dataRetrievalInfo);
|
||||
});
|
||||
}
|
||||
const errorWrapperFn = this._errorWrapper.bind(this);
|
||||
return azureMpuUtils.putSubParts(errorWrapperFn, stream,
|
||||
params, this._dataStoreName, log, (err, dataStoreETag) => {
|
||||
if (err) {
|
||||
return callback(err);
|
||||
}
|
||||
dataRetrievalInfo.dataStoreETag = dataStoreETag;
|
||||
return callback(null, dataRetrievalInfo);
|
||||
});
|
||||
}
|
||||
|
||||
completeMPU(jsonList, mdInfo, key, uploadId, bucket, metaHeaders,
|
||||
contentSettings, log, callback) {
|
||||
const azureKey = this._createAzureKey(bucket, key, this._bucketMatch);
|
||||
const commitList = {
|
||||
UncommittedBlocks: jsonList.uncommittedBlocks || [],
|
||||
};
|
||||
let filteredPartsObj;
|
||||
if (!jsonList.uncommittedBlocks) {
|
||||
const { storedParts, mpuOverviewKey, splitter } = mdInfo;
|
||||
filteredPartsObj = validateAndFilterMpuParts(storedParts, jsonList,
|
||||
mpuOverviewKey, splitter, log);
|
||||
|
||||
filteredPartsObj.partList.forEach(part => {
|
||||
// part.locations is always array of 1, which contains data info
|
||||
const subPartIds =
|
||||
azureMpuUtils.getSubPartIds(part.locations[0], uploadId);
|
||||
commitList.UncommittedBlocks.push(...subPartIds);
|
||||
});
|
||||
}
|
||||
const options = {
|
||||
contentSettings,
|
||||
metadata: metaHeaders ? translateAzureMetaHeaders(metaHeaders) :
|
||||
undefined,
|
||||
};
|
||||
this._errorWrapper('completeMPU', 'commitBlocks',
|
||||
[this._azureContainerName, azureKey, commitList, options,
|
||||
err => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'err completing MPU on Azure ' +
|
||||
'datastore', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`Azure: ${err.message}`));
|
||||
}
|
||||
const completeObjData = {
|
||||
key: azureKey,
|
||||
filteredPartsObj,
|
||||
};
|
||||
return callback(null, completeObjData);
|
||||
}], log, callback);
|
||||
}
|
||||
|
||||
objectPutTagging(key, bucket, objectMD, log, callback) {
|
||||
const azureKey = this._createAzureKey(bucket, key, this._bucketMatch);
|
||||
const azureMD = this._getMetaHeaders(objectMD);
|
||||
azureMD.tags = JSON.stringify(objectMD.tags);
|
||||
this._errorWrapper('objectPutTagging', 'setBlobMetadata',
|
||||
[this._azureContainerName, azureKey, azureMD,
|
||||
err => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'err putting object tags to ' +
|
||||
'Azure backend', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable);
|
||||
}
|
||||
return callback();
|
||||
}], log, callback);
|
||||
}
|
||||
|
||||
objectDeleteTagging(key, bucket, objectMD, log, callback) {
|
||||
const azureKey = this._createAzureKey(bucket, key, this._bucketMatch);
|
||||
const azureMD = this._getMetaHeaders(objectMD);
|
||||
this._errorWrapper('objectDeleteTagging', 'setBlobMetadata',
|
||||
[this._azureContainerName, azureKey, azureMD,
|
||||
err => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'err putting object tags to ' +
|
||||
'Azure backend', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable);
|
||||
}
|
||||
return callback();
|
||||
}], log, callback);
|
||||
}
|
||||
|
||||
copyObject(request, destLocationConstraintName, sourceKey,
|
||||
sourceLocationConstraintName, storeMetadataParams, log, callback) {
|
||||
const destContainerName = request.bucketName;
|
||||
const destObjectKey = request.objectKey;
|
||||
|
||||
const destAzureKey = this._createAzureKey(destContainerName,
|
||||
destObjectKey, this._bucketMatch);
|
||||
|
||||
const sourceContainerName =
|
||||
config.locationConstraints[sourceLocationConstraintName]
|
||||
.details.azureContainerName;
|
||||
|
||||
let options;
|
||||
if (storeMetadataParams.metaHeaders) {
|
||||
options = { metadata:
|
||||
translateAzureMetaHeaders(storeMetadataParams.metaHeaders) };
|
||||
}
|
||||
|
||||
this._errorWrapper('copyObject', 'startCopyBlob',
|
||||
[`${this._azureStorageEndpoint}` +
|
||||
`${sourceContainerName}/${sourceKey}`,
|
||||
this._azureContainerName, destAzureKey, options,
|
||||
(err, res) => {
|
||||
if (err) {
|
||||
if (err.code === 'CannotVerifyCopySource') {
|
||||
logHelper(log, 'error', 'Unable to access ' +
|
||||
`${sourceContainerName} Azure Container`, err,
|
||||
this._dataStoreName);
|
||||
return callback(errors.AccessDenied
|
||||
.customizeDescription('Error: Unable to access ' +
|
||||
`${sourceContainerName} Azure Container`)
|
||||
);
|
||||
}
|
||||
logHelper(log, 'error', 'error from data backend on ' +
|
||||
'copyObject', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS: ${err.message}`)
|
||||
);
|
||||
}
|
||||
if (res.copy.status === 'pending') {
|
||||
logHelper(log, 'error', 'Azure copy status is pending',
|
||||
err, this._dataStoreName);
|
||||
const copyId = res.copy.id;
|
||||
this._client.abortCopyBlob(this._azureContainerName,
|
||||
destAzureKey, copyId, err => {
|
||||
if (err) {
|
||||
logHelper(log, 'error', 'error from data backend ' +
|
||||
'on abortCopyBlob', err, this._dataStoreName);
|
||||
return callback(errors.ServiceUnavailable
|
||||
.customizeDescription('Error returned from ' +
|
||||
`AWS on abortCopyBlob: ${err.message}`)
|
||||
);
|
||||
}
|
||||
return callback(errors.InvalidObjectState
|
||||
.customizeDescription('Error: Azure copy status was ' +
|
||||
'pending. It has been aborted successfully')
|
||||
);
|
||||
});
|
||||
}
|
||||
return callback(null, destAzureKey);
|
||||
}], log, callback);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = AzureClient;
|
|
@ -0,0 +1,246 @@
|
|||
const googleAuth = require('google-auto-auth');
|
||||
const async = require('async');
|
||||
const AWS = require('aws-sdk');
|
||||
const { errors } = require('arsenal');
|
||||
const Service = AWS.Service;
|
||||
|
||||
const GcpSigner = require('./GcpSigner');
|
||||
|
||||
function genAuth(authParams, callback) {
|
||||
async.tryEach([
|
||||
function authKeyFile(next) {
|
||||
const authOptions = {
|
||||
scopes: authParams.scopes,
|
||||
keyFilename: authParams.keyFilename,
|
||||
};
|
||||
const auth = googleAuth(authOptions);
|
||||
auth.getToken(err => next(err, auth));
|
||||
},
|
||||
function authCredentials(next) {
|
||||
const authOptions = {
|
||||
scopes: authParams.scopes,
|
||||
credentials: authParams.credentials,
|
||||
};
|
||||
const auth = googleAuth(authOptions);
|
||||
auth.getToken(err => next(err, auth));
|
||||
},
|
||||
], (err, result) => callback(err, result));
|
||||
}
|
||||
|
||||
AWS.apiLoader.services.gcp = {};
|
||||
const GCP = Service.defineService('gcp', ['2017-11-01'], {
|
||||
_jsonAuth: null,
|
||||
_authParams: null,
|
||||
|
||||
getToken(callback) {
|
||||
if (this._jsonAuth) {
|
||||
return this._jsonAuth.getToken(callback);
|
||||
}
|
||||
|
||||
if (!this._authParams && this.config.authParams &&
|
||||
typeof this.config.authParams === 'object') {
|
||||
this._authParams = this.config.authParams;
|
||||
}
|
||||
return genAuth(this._authParams, (err, auth) => {
|
||||
if (!err) {
|
||||
this._jsonAuth = auth;
|
||||
return this._jsonAuth.getToken(callback);
|
||||
}
|
||||
// should never happen, but if all preconditions fails
|
||||
// can't generate tokens
|
||||
return callback(errors.InternalError.customizeDescription(
|
||||
'Unable to create a google authorizer'));
|
||||
});
|
||||
},
|
||||
|
||||
getSignerClass() {
|
||||
return GcpSigner;
|
||||
},
|
||||
|
||||
validateService() {
|
||||
if (!this.config.region) {
|
||||
this.config.region = 'us-east-1';
|
||||
}
|
||||
},
|
||||
|
||||
upload(params, options, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: upload not implemented'));
|
||||
},
|
||||
|
||||
// Service API
|
||||
listBuckets(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: listBuckets not implemented'));
|
||||
},
|
||||
|
||||
// Bucket APIs
|
||||
getBucketLocation(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: getBucketLocation not implemented'));
|
||||
},
|
||||
|
||||
deleteBucket(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: deleteBucket not implemented'));
|
||||
},
|
||||
|
||||
headBucket(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: headBucket not implemented'));
|
||||
},
|
||||
|
||||
listObjects(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: listObjects not implemented'));
|
||||
},
|
||||
|
||||
listObjectVersions(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: listObjecVersions not implemented'));
|
||||
},
|
||||
|
||||
putBucket(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: putBucket not implemented'));
|
||||
},
|
||||
|
||||
getBucketAcl(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: getBucketAcl not implemented'));
|
||||
},
|
||||
|
||||
putBucketAcl(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: putBucketAcl not implemented'));
|
||||
},
|
||||
|
||||
putBucketWebsite(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: putBucketWebsite not implemented'));
|
||||
},
|
||||
|
||||
getBucketWebsite(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: getBucketWebsite not implemented'));
|
||||
},
|
||||
|
||||
deleteBucketWebsite(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: deleteBucketWebsite not implemented'));
|
||||
},
|
||||
|
||||
putBucketCors(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: putBucketCors not implemented'));
|
||||
},
|
||||
|
||||
getBucketCors(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: getBucketCors not implemented'));
|
||||
},
|
||||
|
||||
deleteBucketCors(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: deleteBucketCors not implemented'));
|
||||
},
|
||||
|
||||
// Object APIs
|
||||
headObject(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: headObject not implemented'));
|
||||
},
|
||||
|
||||
putObject(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: putObject not implemented'));
|
||||
},
|
||||
|
||||
getObject(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: getObject not implemented'));
|
||||
},
|
||||
|
||||
deleteObject(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: deleteObject not implemented'));
|
||||
},
|
||||
|
||||
deleteObjects(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: deleteObjects not implemented'));
|
||||
},
|
||||
|
||||
copyObject(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: copyObject not implemented'));
|
||||
},
|
||||
|
||||
putObjectTagging(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: putObjectTagging not implemented'));
|
||||
},
|
||||
|
||||
deleteObjectTagging(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: deleteObjectTagging not implemented'));
|
||||
},
|
||||
|
||||
putObjectAcl(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: putObjectAcl not implemented'));
|
||||
},
|
||||
|
||||
getObjectAcl(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription('GCP: getObjectAcl not implemented'));
|
||||
},
|
||||
|
||||
// Multipart upload
|
||||
abortMultipartUpload(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription(
|
||||
'GCP: abortMultipartUpload not implemented'));
|
||||
},
|
||||
|
||||
createMultipartUpload(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription(
|
||||
'GCP: createMultipartUpload not implemented'));
|
||||
},
|
||||
|
||||
completeMultipartUpload(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription(
|
||||
'GCP: completeMultipartUpload not implemented'));
|
||||
},
|
||||
|
||||
uploadPart(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription(
|
||||
'GCP: uploadPart not implemented'));
|
||||
},
|
||||
|
||||
uploadPartCopy(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription(
|
||||
'GCP: uploadPartCopy not implemented'));
|
||||
},
|
||||
|
||||
listParts(params, callback) {
|
||||
return callback(errors.NotImplemented
|
||||
.customizeDescription(
|
||||
'GCP: listParts not implemented'));
|
||||
},
|
||||
});
|
||||
|
||||
Object.defineProperty(AWS.apiLoader.services.gcp, '2017-11-01', {
|
||||
get: function get() {
|
||||
const model = require('./gcp-2017-11-01.api.json');
|
||||
return model;
|
||||
},
|
||||
enumerable: true,
|
||||
configurable: true,
|
||||
});
|
||||
|
||||
module.exports = GCP;
|
|
@ -0,0 +1,48 @@
|
|||
const url = require('url');
|
||||
const qs = require('querystring');
|
||||
const AWS = require('aws-sdk');
|
||||
const werelogs = require('werelogs');
|
||||
const { constructStringToSignV2 } = require('arsenal').auth.client;
|
||||
|
||||
const logger = new werelogs.Logger('GcpSigner');
|
||||
|
||||
function genQueryObject(uri) {
|
||||
const queryString = url.parse(uri).query;
|
||||
return qs.parse(queryString);
|
||||
}
|
||||
|
||||
const GcpSigner = AWS.util.inherit(AWS.Signers.RequestSigner, {
|
||||
constructor: function GcpSigner(request) {
|
||||
AWS.Signers.RequestSigner.call(this, request);
|
||||
},
|
||||
|
||||
addAuthorization: function addAuthorization(credentials, date) {
|
||||
if (!this.request.headers['presigned-expires']) {
|
||||
this.request.headers['x-goog-date'] = AWS.util.date.rfc822(date);
|
||||
}
|
||||
|
||||
const signature =
|
||||
this.sign(credentials.secretAccessKey, this.stringToSign());
|
||||
const auth = `GOOG1 ${credentials.accessKeyId}: ${signature}`;
|
||||
|
||||
this.request.headers.Authorization = auth;
|
||||
},
|
||||
|
||||
stringToSign: function stringToSign() {
|
||||
const requestObject = {
|
||||
url: this.request.path,
|
||||
method: this.request.method,
|
||||
host: this.request.endpoint.host,
|
||||
headers: this.request.headers,
|
||||
query: genQueryObject(this.request.path) || {},
|
||||
};
|
||||
const data = Object.assign({}, this.request.headers);
|
||||
return constructStringToSignV2(requestObject, data, logger, 'GCP');
|
||||
},
|
||||
|
||||
sign: function sign(secret, string) {
|
||||
return AWS.util.crypto.hmac(secret, string, 'base64', 'sha1');
|
||||
},
|
||||
});
|
||||
|
||||
module.exports = GcpSigner;
|
|
@ -0,0 +1 @@
|
|||
module.exports = {};
|
|
@ -0,0 +1,17 @@
|
|||
{
|
||||
"version": "1.0",
|
||||
"metadata": {
|
||||
"apiVersion": "2017-11-01",
|
||||
"checksumFormat": "md5",
|
||||
"endpointPrefix": "s3",
|
||||
"globalEndpoint": "storage.googleapi.com",
|
||||
"protocol": "rest-xml",
|
||||
"serviceAbbreviation": "GCP",
|
||||
"serviceFullName": "Google Cloud Storage",
|
||||
"signatureVersion": "s3",
|
||||
"timestampFormat": "rfc822",
|
||||
"uid": "gcp-2017-11-01"
|
||||
},
|
||||
"operations": {},
|
||||
"shapes": {}
|
||||
}
|
|
@ -0,0 +1,5 @@
|
|||
module.exports = {
|
||||
GCP: require('./GcpService'),
|
||||
GcpSigner: require('./GcpSigner'),
|
||||
GcpUtils: require('./GcpUtils'),
|
||||
};
|
|
@ -0,0 +1,39 @@
|
|||
const { GCP } = require('./GCP');
|
||||
const AwsClient = require('./AwsClient');
|
||||
|
||||
/**
|
||||
* Class representing a Google Cloud Storage backend object
|
||||
* @extends AwsClient
|
||||
*/
|
||||
class GcpClient extends AwsClient {
|
||||
/**
|
||||
* constructor - creates a Gcp backend client object (inherits )
|
||||
* @param {object} config - configuration object for Gcp Backend up
|
||||
* @param {object} config.s3params - S3 configuration
|
||||
* @param {string} config.bucketName - GCP bucket name
|
||||
* @param {string} config.mpuBucket - GCP mpu bucket name
|
||||
* @param {string} config.overflowBucket - GCP overflow bucket name
|
||||
* @param {boolean} config.bucketMatch - bucket match flag
|
||||
* @param {object} config.authParams - GCP service credentials
|
||||
* @param {string} config.dataStoreName - locationConstraint name
|
||||
* @param {booblean} config.serverSideEncryption - server side encryption
|
||||
* flag
|
||||
* @return {object} - returns a GcpClient object
|
||||
*/
|
||||
constructor(config) {
|
||||
super(config);
|
||||
this.clientType = 'gcp';
|
||||
this._gcpBucketName = config.bucketName;
|
||||
this._mpuBucketName = config.mpuBucket;
|
||||
this._overflowBucketname = config.overflowBucket;
|
||||
this._gcpParams = Object.assign(this._s3Params, {
|
||||
mainBucket: this._gcpBucketName,
|
||||
mpuBucket: this._mpuBucketName,
|
||||
overflowBucket: this._overflowBucketname,
|
||||
authParams: config.authParams,
|
||||
});
|
||||
this._client = new GCP(this._gcpParams);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = GcpClient;
|
|
@ -0,0 +1,147 @@
|
|||
const async = require('async');
|
||||
const constants = require('../../../constants');
|
||||
const { config } = require('../../../lib/Config');
|
||||
|
||||
const awsHealth = {
|
||||
response: undefined,
|
||||
time: 0,
|
||||
};
|
||||
const azureHealth = {
|
||||
response: undefined,
|
||||
time: 0,
|
||||
};
|
||||
|
||||
const utils = {
|
||||
logHelper(log, level, description, error, dataStoreName) {
|
||||
log[level](description, { error: error.message,
|
||||
errorName: error.name, dataStoreName });
|
||||
},
|
||||
// take off the 'x-amz-meta-'
|
||||
trimXMetaPrefix(obj) {
|
||||
const newObj = {};
|
||||
Object.keys(obj).forEach(key => {
|
||||
const newKey = key.substring(11);
|
||||
newObj[newKey] = obj[key];
|
||||
});
|
||||
return newObj;
|
||||
},
|
||||
removeQuotes(word) {
|
||||
return word.slice(1, -1);
|
||||
},
|
||||
skipMpuPartProcessing(completeObjData) {
|
||||
const backendType = completeObjData.dataStoreType;
|
||||
if (constants.mpuMDStoredExternallyBackend[backendType]) {
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
},
|
||||
|
||||
/**
|
||||
* checkAzureBackendMatch - checks that the external backend location for
|
||||
* two data objects is the same and is Azure
|
||||
* @param {array} objectDataOne - data of first object to compare
|
||||
* @param {object} objectDataTwo - data of second object to compare
|
||||
* @return {boolean} - true if both data backends are Azure, false if not
|
||||
*/
|
||||
checkAzureBackendMatch(objectDataOne, objectDataTwo) {
|
||||
if (objectDataOne.dataStoreType === 'azure' &&
|
||||
objectDataTwo.dataStoreType === 'azure') {
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
},
|
||||
|
||||
/**
|
||||
* externalBackendCopy - Server side copy should only be allowed:
|
||||
* 1) if source object and destination object are both on aws or both
|
||||
* on azure.
|
||||
* 2) if azure to azure, must be the same storage account since Azure
|
||||
* copy outside of an account is async
|
||||
* 3) if the source bucket is not an encrypted bucket and the
|
||||
* destination bucket is not an encrypted bucket (unless the copy
|
||||
* is all within the same bucket).
|
||||
* @param {string} locationConstraintSrc - location constraint of the source
|
||||
* @param {string} locationConstraintDest - location constraint of the
|
||||
* destination
|
||||
* @param {object} sourceBucketMD - metadata of the source bucket
|
||||
* @param {object} destBucketMD - metadata of the destination bucket
|
||||
* @return {boolean} - true if copying object from one
|
||||
* externalbackend to the same external backend and for Azure if it is the
|
||||
* same account since Azure copy outside of an account is async
|
||||
*/
|
||||
externalBackendCopy(locationConstraintSrc, locationConstraintDest,
|
||||
sourceBucketMD, destBucketMD) {
|
||||
const sourceBucketName = sourceBucketMD.getName();
|
||||
const destBucketName = destBucketMD.getName();
|
||||
const isSameBucket = sourceBucketName === destBucketName;
|
||||
const bucketsNotEncrypted = destBucketMD.getServerSideEncryption()
|
||||
=== sourceBucketMD.getServerSideEncryption() &&
|
||||
sourceBucketMD.getServerSideEncryption() === null;
|
||||
const sourceLocationConstraintType =
|
||||
config.getLocationConstraintType(locationConstraintSrc);
|
||||
const locationTypeMatch =
|
||||
config.getLocationConstraintType(locationConstraintSrc) ===
|
||||
config.getLocationConstraintType(locationConstraintDest);
|
||||
return locationTypeMatch && (isSameBucket || bucketsNotEncrypted) &&
|
||||
(sourceLocationConstraintType === 'aws_s3' ||
|
||||
(sourceLocationConstraintType === 'azure' &&
|
||||
config.isSameAzureAccount(locationConstraintSrc,
|
||||
locationConstraintDest)));
|
||||
},
|
||||
|
||||
checkExternalBackend(clients, locations, type, flightCheckOnStartUp,
|
||||
externalBackendHealthCheckInterval, cb) {
|
||||
const checkStatus = type === 'aws_s3' ? awsHealth : azureHealth;
|
||||
if (locations.length === 0) {
|
||||
return process.nextTick(cb, null, []);
|
||||
}
|
||||
if (!flightCheckOnStartUp && checkStatus.response &&
|
||||
Date.now() - checkStatus.time
|
||||
< externalBackendHealthCheckInterval) {
|
||||
return process.nextTick(cb, null, checkStatus.response);
|
||||
}
|
||||
let locationsToCheck;
|
||||
if (flightCheckOnStartUp) {
|
||||
// check all locations if flight check on start up
|
||||
locationsToCheck = locations;
|
||||
} else {
|
||||
const randomLocation = locations[Math.floor(Math.random() *
|
||||
locations.length)];
|
||||
locationsToCheck = [randomLocation];
|
||||
}
|
||||
return async.mapLimit(locationsToCheck, 5, (location, next) => {
|
||||
const client = clients[location];
|
||||
client.healthcheck(location, next, flightCheckOnStartUp);
|
||||
}, (err, results) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
if (!flightCheckOnStartUp) {
|
||||
checkStatus.response = results;
|
||||
checkStatus.time = Date.now();
|
||||
}
|
||||
return cb(null, results);
|
||||
});
|
||||
},
|
||||
translateAzureMetaHeaders(metaHeaders, tags) {
|
||||
const translatedMetaHeaders = {};
|
||||
if (tags) {
|
||||
// tags are passed as string of format 'key1=value1&key2=value2'
|
||||
const tagObj = {};
|
||||
const tagArr = tags.split('&');
|
||||
tagArr.forEach(keypair => {
|
||||
const equalIndex = keypair.indexOf('=');
|
||||
const key = keypair.substring(0, equalIndex);
|
||||
tagObj[key] = keypair.substring(equalIndex + 1);
|
||||
});
|
||||
translatedMetaHeaders.tags = JSON.stringify(tagObj);
|
||||
}
|
||||
Object.keys(metaHeaders).forEach(headerName => {
|
||||
const translated = headerName.substring(11).replace(/-/g, '_');
|
||||
translatedMetaHeaders[translated] = metaHeaders[headerName];
|
||||
});
|
||||
return translatedMetaHeaders;
|
||||
},
|
||||
};
|
||||
|
||||
module.exports = utils;
|
|
@ -0,0 +1,37 @@
|
|||
const arsenal = require('arsenal');
|
||||
const { config } = require('../../Config');
|
||||
|
||||
class DataFileInterface {
|
||||
constructor() {
|
||||
const { host, port } = config.dataClient;
|
||||
|
||||
this.restClient = new arsenal.network.rest.RESTClient(
|
||||
{ host, port });
|
||||
}
|
||||
|
||||
put(stream, size, keyContext, reqUids, callback) {
|
||||
// ignore keyContext
|
||||
this.restClient.put(stream, size, reqUids, callback);
|
||||
}
|
||||
|
||||
get(objectGetInfo, range, reqUids, callback) {
|
||||
const key = objectGetInfo.key ? objectGetInfo.key : objectGetInfo;
|
||||
this.restClient.get(key, range, reqUids, callback);
|
||||
}
|
||||
|
||||
delete(objectGetInfo, reqUids, callback) {
|
||||
const key = objectGetInfo.key ? objectGetInfo.key : objectGetInfo;
|
||||
this.restClient.delete(key, reqUids, callback);
|
||||
}
|
||||
|
||||
getDiskUsage(reqUids, callback) {
|
||||
this.restClient.getAction('diskUsage', reqUids, (err, val) => {
|
||||
if (err) {
|
||||
return callback(err);
|
||||
}
|
||||
return callback(null, JSON.parse(val));
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = DataFileInterface;
|
|
@ -0,0 +1,93 @@
|
|||
const stream = require('stream');
|
||||
const { errors } = require('arsenal');
|
||||
const werelogs = require('werelogs');
|
||||
|
||||
const logger = new werelogs.Logger('MemDataBackend');
|
||||
|
||||
function createLogger(reqUids) {
|
||||
return reqUids ?
|
||||
logger.newRequestLoggerFromSerializedUids(reqUids) :
|
||||
logger.newRequestLogger();
|
||||
}
|
||||
|
||||
const ds = [];
|
||||
let count = 1; // keys are assessed with if (!key)
|
||||
|
||||
function resetCount() {
|
||||
count = 1;
|
||||
}
|
||||
|
||||
const backend = {
|
||||
put: function putMem(request, size, keyContext, reqUids, callback) {
|
||||
const log = createLogger(reqUids);
|
||||
const value = Buffer.alloc(size);
|
||||
let cursor = 0;
|
||||
let exceeded = false;
|
||||
request.on('data', data => {
|
||||
if (cursor + data.length > size) {
|
||||
exceeded = true;
|
||||
}
|
||||
if (!exceeded) {
|
||||
data.copy(value, cursor);
|
||||
}
|
||||
cursor += data.length;
|
||||
})
|
||||
.on('end', () => {
|
||||
if (exceeded) {
|
||||
log.error('data stream exceed announced size',
|
||||
{ size, overflow: cursor });
|
||||
callback(errors.InternalError);
|
||||
} else {
|
||||
ds[count] = { value, keyContext };
|
||||
callback(null, count++);
|
||||
}
|
||||
});
|
||||
},
|
||||
|
||||
get: function getMem(objectGetInfo, range, reqUids, callback) {
|
||||
const key = objectGetInfo.key ? objectGetInfo.key : objectGetInfo;
|
||||
process.nextTick(() => {
|
||||
if (!ds[key]) { return callback(errors.NoSuchKey); }
|
||||
const storedBuffer = ds[key].value;
|
||||
// If a range was sent, use the start from the range.
|
||||
// Otherwise, start at 0
|
||||
let start = range ? range[0] : 0;
|
||||
// If a range was sent, use the end from the range.
|
||||
// End of range should be included so +1
|
||||
// Otherwise, get the full length
|
||||
const end = range ? range[1] + 1 : storedBuffer.length;
|
||||
const chunkSize = 64 * 1024; // 64KB
|
||||
const val = new stream.Readable({
|
||||
read: function read() {
|
||||
// sets this._read under the hood
|
||||
// push data onto the read queue, passing null
|
||||
// will signal the end of the stream (EOF)
|
||||
while (start < end) {
|
||||
const finish =
|
||||
Math.min(start + chunkSize, end);
|
||||
this.push(storedBuffer.slice(start, finish));
|
||||
start += chunkSize;
|
||||
}
|
||||
if (start >= end) {
|
||||
this.push(null);
|
||||
}
|
||||
},
|
||||
});
|
||||
return callback(null, val);
|
||||
});
|
||||
},
|
||||
|
||||
delete: function delMem(objectGetInfo, reqUids, callback) {
|
||||
const key = objectGetInfo.key ? objectGetInfo.key : objectGetInfo;
|
||||
process.nextTick(() => {
|
||||
delete ds[key];
|
||||
return callback(null);
|
||||
});
|
||||
},
|
||||
};
|
||||
|
||||
module.exports = {
|
||||
backend,
|
||||
ds,
|
||||
resetCount,
|
||||
};
|
|
@ -0,0 +1,126 @@
|
|||
const http = require('http');
|
||||
const https = require('https');
|
||||
const AWS = require('aws-sdk');
|
||||
const Sproxy = require('sproxydclient');
|
||||
|
||||
const DataFileBackend = require('./file/backend');
|
||||
const inMemory = require('./in_memory/backend').backend;
|
||||
const AwsClient = require('./external/AwsClient');
|
||||
const GcpClient = require('./external/GcpClient');
|
||||
const AzureClient = require('./external/AzureClient');
|
||||
|
||||
const { config } = require('../Config');
|
||||
|
||||
const proxyAddress = 'http://localhost:3128';
|
||||
|
||||
function parseLC() {
|
||||
const clients = {};
|
||||
|
||||
Object.keys(config.locationConstraints).forEach(location => {
|
||||
const locationObj = config.locationConstraints[location];
|
||||
if (locationObj.type === 'mem') {
|
||||
clients[location] = inMemory;
|
||||
}
|
||||
if (locationObj.type === 'file') {
|
||||
clients[location] = new DataFileBackend();
|
||||
}
|
||||
if (locationObj.type === 'scality'
|
||||
&& locationObj.details.connector.sproxyd) {
|
||||
clients[location] = new Sproxy({
|
||||
bootstrap: locationObj.details.connector
|
||||
.sproxyd.bootstrap,
|
||||
// Might be undefined which is ok since there is a default
|
||||
// set in sproxydclient if chordCos is undefined
|
||||
chordCos: locationObj.details.connector.sproxyd.chordCos,
|
||||
// Might also be undefined, but there is a default path set
|
||||
// in sproxydclient as well
|
||||
path: locationObj.details.connector.sproxyd.path,
|
||||
// enable immutable optim for all objects
|
||||
immutable: true,
|
||||
});
|
||||
clients[location].clientType = 'scality';
|
||||
}
|
||||
if (locationObj.type === 'aws_s3' || locationObj.type === 'gcp') {
|
||||
if (process.env.CI_PROXY === 'true') {
|
||||
locationObj.details.https = false;
|
||||
locationObj.details.proxy = proxyAddress;
|
||||
}
|
||||
const connectionAgent = locationObj.details.https ?
|
||||
new https.Agent({ keepAlive: true }) :
|
||||
new http.Agent({ keepAlive: true });
|
||||
const protocol = locationObj.details.https ? 'https' : 'http';
|
||||
const httpOptions = locationObj.details.proxy ?
|
||||
{ proxy: locationObj.details.proxy, agent: connectionAgent,
|
||||
timeout: 0 }
|
||||
: { agent: connectionAgent, timeout: 0 };
|
||||
const sslEnabled = locationObj.details.https === true;
|
||||
// TODO: HTTP requests to AWS are not supported with V4 auth for
|
||||
// non-file streams which are used by Backbeat. This option will be
|
||||
// removed once CA certs, proxy setup feature is implemented.
|
||||
const signatureVersion = !sslEnabled ? 'v2' : 'v4';
|
||||
const endpoint = locationObj.type === 'gcp' ?
|
||||
locationObj.details.gcpEndpoint :
|
||||
locationObj.details.awsEndpoint;
|
||||
const s3Params = {
|
||||
endpoint: `${protocol}://${endpoint}`,
|
||||
debug: false,
|
||||
// Not implemented yet for streams in node sdk,
|
||||
// and has no negative impact if stream, so let's
|
||||
// leave it in for future use
|
||||
computeChecksums: true,
|
||||
httpOptions,
|
||||
// needed for encryption
|
||||
signatureVersion,
|
||||
sslEnabled,
|
||||
maxRetries: 0,
|
||||
};
|
||||
// users can either include the desired profile name from their
|
||||
// ~/.aws/credentials file or include the accessKeyId and
|
||||
// secretAccessKey directly in the locationConfig
|
||||
if (locationObj.details.credentialsProfile) {
|
||||
s3Params.credentials = new AWS.SharedIniFileCredentials({
|
||||
profile: locationObj.details.credentialsProfile });
|
||||
} else {
|
||||
s3Params.accessKeyId =
|
||||
locationObj.details.credentials.accessKey;
|
||||
s3Params.secretAccessKey =
|
||||
locationObj.details.credentials.secretKey;
|
||||
}
|
||||
const clientConfig = {
|
||||
s3Params,
|
||||
bucketName: locationObj.details.bucketName,
|
||||
bucketMatch: locationObj.details.bucketMatch,
|
||||
serverSideEncryption: locationObj.details.serverSideEncryption,
|
||||
dataStoreName: location,
|
||||
};
|
||||
if (locationObj.type === 'gcp') {
|
||||
clientConfig.overflowBucket =
|
||||
locationObj.details.overflowBucketName;
|
||||
clientConfig.mpuBucket = locationObj.details.mpuBucketName;
|
||||
clientConfig.authParams = config.getGcpServiceParams(location);
|
||||
}
|
||||
clients[location] = locationObj.type === 'gcp' ?
|
||||
new GcpClient(clientConfig) : new AwsClient(clientConfig);
|
||||
}
|
||||
if (locationObj.type === 'azure') {
|
||||
if (process.env.CI_PROXY === 'true') {
|
||||
locationObj.details.proxy = proxyAddress;
|
||||
}
|
||||
const azureStorageEndpoint = config.getAzureEndpoint(location);
|
||||
const azureStorageCredentials =
|
||||
config.getAzureStorageCredentials(location);
|
||||
clients[location] = new AzureClient({
|
||||
azureStorageEndpoint,
|
||||
azureStorageCredentials,
|
||||
azureContainerName: locationObj.details.azureContainerName,
|
||||
bucketMatch: locationObj.details.bucketMatch,
|
||||
dataStoreName: location,
|
||||
proxy: locationObj.details.proxy,
|
||||
});
|
||||
clients[location].clientType = 'azure';
|
||||
}
|
||||
});
|
||||
return clients;
|
||||
}
|
||||
|
||||
module.exports = parseLC;
|
|
@ -0,0 +1,289 @@
|
|||
const { errors, s3middleware } = require('arsenal');
|
||||
const { parseTagFromQuery } = s3middleware.tagging;
|
||||
|
||||
const createLogger = require('./multipleBackendLogger');
|
||||
const async = require('async');
|
||||
|
||||
const { config } = require('../Config');
|
||||
const parseLC = require('./locationConstraintParser');
|
||||
|
||||
const { checkExternalBackend } = require('./external/utils');
|
||||
const { externalBackendHealthCheckInterval } = require('../../constants');
|
||||
|
||||
let clients = parseLC(config);
|
||||
config.on('location-constraints-update', () => {
|
||||
clients = parseLC(config);
|
||||
});
|
||||
|
||||
|
||||
const multipleBackendGateway = {
|
||||
|
||||
put: (hashedStream, size, keyContext,
|
||||
backendInfo, reqUids, callback) => {
|
||||
const controllingLocationConstraint =
|
||||
backendInfo.getControllingLocationConstraint();
|
||||
const client = clients[controllingLocationConstraint];
|
||||
if (!client) {
|
||||
const log = createLogger(reqUids);
|
||||
log.error('no data backend matching controlling locationConstraint',
|
||||
{ controllingLocationConstraint });
|
||||
return process.nextTick(() => {
|
||||
callback(errors.InternalError);
|
||||
});
|
||||
}
|
||||
|
||||
let writeStream = hashedStream;
|
||||
if (keyContext.cipherBundle && keyContext.cipherBundle.cipher) {
|
||||
writeStream = keyContext.cipherBundle.cipher;
|
||||
hashedStream.pipe(writeStream);
|
||||
}
|
||||
|
||||
if (keyContext.tagging) {
|
||||
const validationTagRes = parseTagFromQuery(keyContext.tagging);
|
||||
if (validationTagRes instanceof Error) {
|
||||
const log = createLogger(reqUids);
|
||||
log.debug('tag validation failed', {
|
||||
error: validationTagRes,
|
||||
method: 'multipleBackendGateway put',
|
||||
});
|
||||
return callback(errors.InternalError);
|
||||
}
|
||||
}
|
||||
return client.put(writeStream, size, keyContext,
|
||||
reqUids, (err, key, dataStoreVersionId) => {
|
||||
const log = createLogger(reqUids);
|
||||
log.debug('put to location', { controllingLocationConstraint });
|
||||
if (err) {
|
||||
log.error('error from datastore',
|
||||
{ error: err, dataStoreType: client.clientType });
|
||||
return callback(errors.ServiceUnavailable);
|
||||
}
|
||||
const dataRetrievalInfo = {
|
||||
key,
|
||||
dataStoreName: controllingLocationConstraint,
|
||||
dataStoreType: client.clientType,
|
||||
dataStoreVersionId,
|
||||
};
|
||||
return callback(null, dataRetrievalInfo);
|
||||
});
|
||||
},
|
||||
|
||||
head: (objectGetInfoArr, reqUids, callback) => {
|
||||
if (!objectGetInfoArr || !Array.isArray(objectGetInfoArr)
|
||||
|| !objectGetInfoArr[0] || !objectGetInfoArr[0].dataStoreName) {
|
||||
// no-op if no stored data store name
|
||||
return process.nextTick(callback);
|
||||
}
|
||||
const objectGetInfo = objectGetInfoArr[0];
|
||||
const client = clients[objectGetInfo.dataStoreName];
|
||||
if (client.head === undefined) {
|
||||
// no-op if unsupported client method
|
||||
return process.nextTick(callback);
|
||||
}
|
||||
return client.head(objectGetInfo, reqUids, callback);
|
||||
},
|
||||
|
||||
get: (objectGetInfo, range, reqUids, callback) => {
|
||||
let key;
|
||||
let client;
|
||||
// for backwards compatibility
|
||||
if (typeof objectGetInfo === 'string') {
|
||||
key = objectGetInfo;
|
||||
client = clients.sproxyd;
|
||||
} else {
|
||||
key = objectGetInfo.key;
|
||||
client = clients[objectGetInfo.dataStoreName];
|
||||
}
|
||||
if (client.clientType === 'scality') {
|
||||
return client.get(key, range, reqUids, callback);
|
||||
}
|
||||
return client.get(objectGetInfo, range, reqUids, callback);
|
||||
},
|
||||
|
||||
delete: (objectGetInfo, reqUids, callback) => {
|
||||
let key;
|
||||
let client;
|
||||
// for backwards compatibility
|
||||
if (typeof objectGetInfo === 'string') {
|
||||
key = objectGetInfo;
|
||||
client = clients.sproxyd;
|
||||
} else {
|
||||
key = objectGetInfo.key;
|
||||
client = clients[objectGetInfo.dataStoreName];
|
||||
}
|
||||
if (client.clientType === 'scality') {
|
||||
return client.delete(key, reqUids, callback);
|
||||
}
|
||||
return client.delete(objectGetInfo, reqUids, callback);
|
||||
},
|
||||
|
||||
healthcheck: (flightCheckOnStartUp, log, callback) => {
|
||||
const multBackendResp = {};
|
||||
const awsArray = [];
|
||||
const azureArray = [];
|
||||
async.each(Object.keys(clients), (location, cb) => {
|
||||
const client = clients[location];
|
||||
if (client.clientType === 'scality') {
|
||||
return client.healthcheck(log, (err, res) => {
|
||||
if (err) {
|
||||
multBackendResp[location] = { error: err };
|
||||
} else {
|
||||
multBackendResp[location] = { code: res.statusCode,
|
||||
message: res.statusMessage };
|
||||
}
|
||||
return cb();
|
||||
});
|
||||
} else if (client.clientType === 'aws_s3') {
|
||||
awsArray.push(location);
|
||||
return cb();
|
||||
} else if (client.clientType === 'azure') {
|
||||
azureArray.push(location);
|
||||
return cb();
|
||||
}
|
||||
// if backend type isn't 'scality' or 'aws_s3', it will be
|
||||
// 'mem' or 'file', for which the default response is 200 OK
|
||||
multBackendResp[location] = { code: 200, message: 'OK' };
|
||||
return cb();
|
||||
}, () => {
|
||||
async.parallel([
|
||||
next => checkExternalBackend(
|
||||
clients, awsArray, 'aws_s3', flightCheckOnStartUp,
|
||||
externalBackendHealthCheckInterval, next),
|
||||
next => checkExternalBackend(
|
||||
clients, azureArray, 'azure', flightCheckOnStartUp,
|
||||
externalBackendHealthCheckInterval, next),
|
||||
], (errNull, externalResp) => {
|
||||
const externalLocResults = [];
|
||||
externalResp.forEach(resp => externalLocResults.push(...resp));
|
||||
externalLocResults.forEach(locationResult =>
|
||||
Object.assign(multBackendResp, locationResult));
|
||||
callback(null, multBackendResp);
|
||||
});
|
||||
});
|
||||
},
|
||||
|
||||
createMPU: (key, metaHeaders, bucketName, websiteRedirectHeader,
|
||||
location, contentType, cacheControl, contentDisposition,
|
||||
contentEncoding, log, cb) => {
|
||||
const client = clients[location];
|
||||
if (client.clientType === 'aws_s3') {
|
||||
return client.createMPU(key, metaHeaders, bucketName,
|
||||
websiteRedirectHeader, contentType, cacheControl,
|
||||
contentDisposition, contentEncoding, log, cb);
|
||||
}
|
||||
return cb();
|
||||
},
|
||||
|
||||
uploadPart: (request, streamingV4Params, stream, size, location, key,
|
||||
uploadId, partNumber, bucketName, log, cb) => {
|
||||
const client = clients[location];
|
||||
|
||||
if (client.uploadPart) {
|
||||
return client.uploadPart(request, streamingV4Params, stream, size,
|
||||
key, uploadId, partNumber, bucketName, log, cb);
|
||||
}
|
||||
return cb();
|
||||
},
|
||||
|
||||
listParts: (key, uploadId, location, bucketName, partNumberMarker, maxParts,
|
||||
log, cb) => {
|
||||
const client = clients[location];
|
||||
|
||||
if (client.listParts) {
|
||||
return client.listParts(key, uploadId, bucketName, partNumberMarker,
|
||||
maxParts, log, cb);
|
||||
}
|
||||
return cb();
|
||||
},
|
||||
|
||||
completeMPU: (key, uploadId, location, jsonList, mdInfo, bucketName,
|
||||
userMetadata, contentSettings, log, cb) => {
|
||||
const client = clients[location];
|
||||
if (client.completeMPU) {
|
||||
const args = [jsonList, mdInfo, key, uploadId, bucketName];
|
||||
if (client.clientType === 'azure') {
|
||||
args.push(userMetadata, contentSettings);
|
||||
}
|
||||
return client.completeMPU(...args, log, (err, completeObjData) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
// eslint-disable-next-line no-param-reassign
|
||||
completeObjData.dataStoreType = client.clientType;
|
||||
return cb(null, completeObjData);
|
||||
});
|
||||
}
|
||||
return cb();
|
||||
},
|
||||
|
||||
abortMPU: (key, uploadId, location, bucketName, log, cb) => {
|
||||
const client = clients[location];
|
||||
if (client.clientType === 'azure') {
|
||||
const skipDataDelete = true;
|
||||
return cb(null, skipDataDelete);
|
||||
}
|
||||
if (client.abortMPU) {
|
||||
return client.abortMPU(key, uploadId, bucketName, log, err => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
return cb();
|
||||
});
|
||||
}
|
||||
return cb();
|
||||
},
|
||||
|
||||
objectTagging: (method, key, bucket, objectMD, log, cb) => {
|
||||
// if legacy, objectMD will not contain dataStoreName, so just return
|
||||
const client = clients[objectMD.dataStoreName];
|
||||
if (client && client[`object${method}Tagging`]) {
|
||||
return client[`object${method}Tagging`](key, bucket, objectMD, log,
|
||||
cb);
|
||||
}
|
||||
return cb();
|
||||
},
|
||||
// NOTE: using copyObject only if copying object from one external
|
||||
// backend to the same external backend
|
||||
copyObject: (request, destLocationConstraintName, externalSourceKey,
|
||||
sourceLocationConstraintName, storeMetadataParams, log, cb) => {
|
||||
const client = clients[destLocationConstraintName];
|
||||
if (client.copyObject) {
|
||||
return client.copyObject(request, destLocationConstraintName,
|
||||
externalSourceKey, sourceLocationConstraintName,
|
||||
storeMetadataParams, log, (err, key, dataStoreVersionId) => {
|
||||
const dataRetrievalInfo = {
|
||||
key,
|
||||
dataStoreName: destLocationConstraintName,
|
||||
dataStoreType: client.clientType,
|
||||
dataStoreVersionId,
|
||||
};
|
||||
cb(err, dataRetrievalInfo);
|
||||
});
|
||||
}
|
||||
return cb(errors.NotImplemented
|
||||
.customizeDescription('Can not copy object from ' +
|
||||
`${client.clientType} to ${client.clientType}`));
|
||||
},
|
||||
uploadPartCopy: (request, location, awsSourceKey,
|
||||
sourceLocationConstraintName, log, cb) => {
|
||||
const client = clients[location];
|
||||
if (client.uploadPartCopy) {
|
||||
return client.uploadPartCopy(request, awsSourceKey,
|
||||
sourceLocationConstraintName,
|
||||
log, cb);
|
||||
}
|
||||
return cb(errors.NotImplemented.customizeDescription(
|
||||
'Can not copy object from ' +
|
||||
`${client.clientType} to ${client.clientType}`));
|
||||
},
|
||||
protectAzureBlocks: (bucketName, objectKey, location, log, cb) => {
|
||||
const client = clients[location];
|
||||
if (client.protectAzureBlocks) {
|
||||
return client.protectAzureBlocks(bucketName, objectKey, location,
|
||||
log, cb);
|
||||
}
|
||||
return cb();
|
||||
},
|
||||
};
|
||||
|
||||
module.exports = multipleBackendGateway;
|
|
@ -0,0 +1,11 @@
|
|||
const werelogs = require('werelogs');
|
||||
|
||||
const logger = new werelogs.Logger('MultipleBackendGateway');
|
||||
|
||||
function createLogger(reqUids) {
|
||||
return reqUids ?
|
||||
logger.newRequestLoggerFromSerializedUids(reqUids) :
|
||||
logger.newRequestLogger();
|
||||
}
|
||||
|
||||
module.exports = createLogger;
|
|
@ -0,0 +1,792 @@
|
|||
const async = require('async');
|
||||
const { errors, s3middleware } = require('arsenal');
|
||||
const PassThrough = require('stream').PassThrough;
|
||||
|
||||
const DataFileInterface = require('./file/backend');
|
||||
const inMemory = require('./in_memory/backend').backend;
|
||||
const locationConstraintCheck =
|
||||
require('../api/apiUtils/object/locationConstraintCheck');
|
||||
const multipleBackendGateway = require('./multipleBackendGateway');
|
||||
const utils = require('./external/utils');
|
||||
const { config } = require('../Config');
|
||||
const MD5Sum = s3middleware.MD5Sum;
|
||||
const NullStream = s3middleware.NullStream;
|
||||
const assert = require('assert');
|
||||
const kms = require('../kms/wrapper');
|
||||
const externalBackends = require('../../constants').externalBackends;
|
||||
const constants = require('../../constants');
|
||||
const { BackendInfo } = require('../api/apiUtils/object/BackendInfo');
|
||||
const RelayMD5Sum = require('../utilities/RelayMD5Sum');
|
||||
const skipError = new Error('skip');
|
||||
|
||||
let CdmiData;
|
||||
try {
|
||||
CdmiData = require('cdmiclient').CdmiData;
|
||||
} catch (err) {
|
||||
CdmiData = null;
|
||||
}
|
||||
|
||||
let client;
|
||||
let implName;
|
||||
|
||||
if (config.backends.data === 'mem') {
|
||||
client = inMemory;
|
||||
implName = 'mem';
|
||||
} else if (config.backends.data === 'file') {
|
||||
client = new DataFileInterface();
|
||||
implName = 'file';
|
||||
} else if (config.backends.data === 'multiple') {
|
||||
client = multipleBackendGateway;
|
||||
implName = 'multipleBackends';
|
||||
} else if (config.backends.data === 'cdmi') {
|
||||
if (!CdmiData) {
|
||||
throw new Error('Unauthorized backend');
|
||||
}
|
||||
|
||||
client = new CdmiData({
|
||||
path: config.cdmi.path,
|
||||
host: config.cdmi.host,
|
||||
port: config.cdmi.port,
|
||||
readonly: config.cdmi.readonly,
|
||||
});
|
||||
implName = 'cdmi';
|
||||
}
|
||||
|
||||
/**
|
||||
* _retryDelete - Attempt to delete key again if it failed previously
|
||||
* @param { string | object } objectGetInfo - either string location of object
|
||||
* to delete or object containing info of object to delete
|
||||
* @param {object} log - Werelogs request logger
|
||||
* @param {number} count - keeps count of number of times function has been run
|
||||
* @param {function} cb - callback
|
||||
* @returns undefined and calls callback
|
||||
*/
|
||||
const MAX_RETRY = 2;
|
||||
|
||||
// This check is done because on a put, complete mpu or copy request to
|
||||
// Azure/AWS, if the object already exists on that backend, the existing object
|
||||
// should not be deleted, which is the functionality for all other backends
|
||||
function _shouldSkipDelete(locations, requestMethod, newObjDataStoreName) {
|
||||
const skipMethods = { PUT: true, POST: true };
|
||||
if (!Array.isArray(locations) || !locations[0] ||
|
||||
!locations[0].dataStoreType) {
|
||||
return false;
|
||||
}
|
||||
const isSkipBackend = externalBackends[locations[0].dataStoreType];
|
||||
const isMatchingBackends =
|
||||
locations[0].dataStoreName === newObjDataStoreName;
|
||||
const isSkipMethod = skipMethods[requestMethod];
|
||||
return (isSkipBackend && isMatchingBackends && isSkipMethod);
|
||||
}
|
||||
|
||||
function _retryDelete(objectGetInfo, log, count, cb) {
|
||||
if (count > MAX_RETRY) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return client.delete(objectGetInfo, log.getSerializedUids(), err => {
|
||||
if (err) {
|
||||
log.error('delete error from datastore',
|
||||
{ error: err, implName, moreRetries: 'yes' });
|
||||
return _retryDelete(objectGetInfo, log, count + 1, cb);
|
||||
}
|
||||
return cb();
|
||||
});
|
||||
}
|
||||
|
||||
function _put(cipherBundle, value, valueSize,
|
||||
keyContext, backendInfo, log, cb) {
|
||||
assert.strictEqual(typeof valueSize, 'number');
|
||||
log.debug('sending put to datastore', { implName, keyContext,
|
||||
method: 'put' });
|
||||
let hashedStream = null;
|
||||
if (value) {
|
||||
hashedStream = new MD5Sum();
|
||||
value.pipe(hashedStream);
|
||||
}
|
||||
|
||||
if (implName === 'multipleBackends') {
|
||||
// Need to send backendInfo to client.put and
|
||||
// client.put will provide dataRetrievalInfo so no
|
||||
// need to construct here
|
||||
/* eslint-disable no-param-reassign */
|
||||
keyContext.cipherBundle = cipherBundle;
|
||||
return client.put(hashedStream,
|
||||
valueSize, keyContext, backendInfo, log.getSerializedUids(),
|
||||
(err, dataRetrievalInfo) => {
|
||||
if (err) {
|
||||
log.error('put error from datastore',
|
||||
{ error: err, implName });
|
||||
return cb(errors.ServiceUnavailable);
|
||||
}
|
||||
return cb(null, dataRetrievalInfo, hashedStream);
|
||||
});
|
||||
}
|
||||
/* eslint-enable no-param-reassign */
|
||||
|
||||
let writeStream = hashedStream;
|
||||
if (cipherBundle && cipherBundle.cipher) {
|
||||
writeStream = cipherBundle.cipher;
|
||||
hashedStream.pipe(writeStream);
|
||||
}
|
||||
|
||||
return client.put(writeStream, valueSize, keyContext,
|
||||
log.getSerializedUids(), (err, key) => {
|
||||
if (err) {
|
||||
log.error('put error from datastore',
|
||||
{ error: err, implName });
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
const dataRetrievalInfo = {
|
||||
key,
|
||||
dataStoreName: implName,
|
||||
};
|
||||
return cb(null, dataRetrievalInfo, hashedStream);
|
||||
});
|
||||
}
|
||||
|
||||
const data = {
|
||||
put: (cipherBundle, value, valueSize, keyContext, backendInfo, log, cb) => {
|
||||
_put(cipherBundle, value, valueSize, keyContext, backendInfo, log,
|
||||
(err, dataRetrievalInfo, hashedStream) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
if (hashedStream) {
|
||||
if (hashedStream.completedHash) {
|
||||
return cb(null, dataRetrievalInfo, hashedStream);
|
||||
}
|
||||
hashedStream.on('hashed', () => {
|
||||
hashedStream.removeAllListeners('hashed');
|
||||
return cb(null, dataRetrievalInfo, hashedStream);
|
||||
});
|
||||
return undefined;
|
||||
}
|
||||
return cb(null, dataRetrievalInfo);
|
||||
});
|
||||
},
|
||||
|
||||
head: (objectGetInfo, log, cb) => {
|
||||
if (implName !== 'multipleBackends') {
|
||||
// no-op if not multipleBackend implementation;
|
||||
// head is used during get just to check external backend data state
|
||||
return process.nextTick(cb);
|
||||
}
|
||||
return client.head(objectGetInfo, log.getSerializedUids(), cb);
|
||||
},
|
||||
|
||||
get: (objectGetInfo, response, log, cb) => {
|
||||
const isMdModelVersion2 = typeof(objectGetInfo) === 'string';
|
||||
const isRequiredStringKey = constants.clientsRequireStringKey[implName];
|
||||
const key = isMdModelVersion2 ? objectGetInfo : objectGetInfo.key;
|
||||
const clientGetInfo = isRequiredStringKey ? key : objectGetInfo;
|
||||
const range = objectGetInfo.range;
|
||||
|
||||
// If the key is explicitly set to null, the part to
|
||||
// be read doesn't really exist and is only made of zeroes.
|
||||
// This functionality is used by Scality-NFSD.
|
||||
// Otherwise, the key is always defined
|
||||
assert(key === null || key !== undefined);
|
||||
if (key === null) {
|
||||
cb(null, new NullStream(objectGetInfo.size, range));
|
||||
return;
|
||||
}
|
||||
log.debug('sending get to datastore', { implName,
|
||||
key, range, method: 'get' });
|
||||
// We need to use response as a writable stream for AZURE GET
|
||||
if (!isMdModelVersion2 && !isRequiredStringKey && response) {
|
||||
clientGetInfo.response = response;
|
||||
}
|
||||
client.get(clientGetInfo, range, log.getSerializedUids(),
|
||||
(err, stream) => {
|
||||
if (err) {
|
||||
log.error('get error from datastore',
|
||||
{ error: err, implName });
|
||||
return cb(errors.ServiceUnavailable);
|
||||
}
|
||||
if (objectGetInfo.cipheredDataKey) {
|
||||
const serverSideEncryption = {
|
||||
cryptoScheme: objectGetInfo.cryptoScheme,
|
||||
masterKeyId: objectGetInfo.masterKeyId,
|
||||
cipheredDataKey: Buffer.from(
|
||||
objectGetInfo.cipheredDataKey, 'base64'),
|
||||
};
|
||||
const offset = objectGetInfo.range ?
|
||||
objectGetInfo.range[0] : 0;
|
||||
return kms.createDecipherBundle(
|
||||
serverSideEncryption, offset, log,
|
||||
(err, decipherBundle) => {
|
||||
if (err) {
|
||||
log.error('cannot get decipher bundle ' +
|
||||
'from kms', {
|
||||
method: 'data.wrapper.data.get',
|
||||
});
|
||||
return cb(err);
|
||||
}
|
||||
stream.pipe(decipherBundle.decipher);
|
||||
return cb(null, decipherBundle.decipher);
|
||||
});
|
||||
}
|
||||
return cb(null, stream);
|
||||
});
|
||||
},
|
||||
|
||||
delete: (objectGetInfo, log, cb) => {
|
||||
const callback = cb || log.end;
|
||||
const isMdModelVersion2 = typeof(objectGetInfo) === 'string';
|
||||
const isRequiredStringKey = constants.clientsRequireStringKey[implName];
|
||||
const key = isMdModelVersion2 ? objectGetInfo : objectGetInfo.key;
|
||||
const clientGetInfo = isRequiredStringKey ? key : objectGetInfo;
|
||||
|
||||
log.trace('sending delete to datastore', {
|
||||
implName, key, method: 'delete' });
|
||||
// If the key is explicitly set to null, the part to
|
||||
// be deleted doesn't really exist.
|
||||
// This functionality is used by Scality-NFSD.
|
||||
// Otherwise, the key is always defined
|
||||
assert(key === null || key !== undefined);
|
||||
if (key === null) {
|
||||
callback(null);
|
||||
return;
|
||||
}
|
||||
_retryDelete(clientGetInfo, log, 0, err => {
|
||||
if (err) {
|
||||
log.error('delete error from datastore',
|
||||
{ error: err, key: objectGetInfo.key, moreRetries: 'no' });
|
||||
}
|
||||
return callback(err);
|
||||
});
|
||||
},
|
||||
// It would be preferable to have an sproxyd batch delete route to
|
||||
// replace this
|
||||
batchDelete: (locations, requestMethod, newObjDataStoreName, log) => {
|
||||
// TODO: The method of persistence of sproxy delete key will
|
||||
// be finalized; refer Issue #312 for the discussion. In the
|
||||
// meantime, we at least log the location of the data we are
|
||||
// about to delete before attempting its deletion.
|
||||
if (_shouldSkipDelete(locations, requestMethod, newObjDataStoreName)) {
|
||||
return;
|
||||
}
|
||||
log.trace('initiating batch delete', {
|
||||
keys: locations,
|
||||
implName,
|
||||
method: 'batchDelete',
|
||||
});
|
||||
async.eachLimit(locations, 5, (loc, next) => {
|
||||
process.nextTick(() => data.delete(loc, log, next));
|
||||
},
|
||||
err => {
|
||||
if (err) {
|
||||
log.error('batch delete failed', { error: err });
|
||||
} else {
|
||||
log.trace('batch delete successfully completed');
|
||||
}
|
||||
log.end();
|
||||
});
|
||||
},
|
||||
|
||||
switch: newClient => {
|
||||
client = newClient;
|
||||
return client;
|
||||
},
|
||||
|
||||
checkHealth: (log, cb, flightCheckOnStartUp) => {
|
||||
if (!client.healthcheck) {
|
||||
const defResp = {};
|
||||
defResp[implName] = { code: 200, message: 'OK' };
|
||||
return cb(null, defResp);
|
||||
}
|
||||
return client.healthcheck(flightCheckOnStartUp, log, (err, result) => {
|
||||
let respBody = {};
|
||||
if (err) {
|
||||
log.error(`error from ${implName}`, { error: err });
|
||||
respBody[implName] = {
|
||||
error: err,
|
||||
};
|
||||
// error returned as null so async parallel doesn't return
|
||||
// before all backends are checked
|
||||
return cb(null, respBody);
|
||||
}
|
||||
if (implName === 'multipleBackends') {
|
||||
respBody = result;
|
||||
return cb(null, respBody);
|
||||
}
|
||||
respBody[implName] = {
|
||||
code: result.statusCode,
|
||||
message: result.statusMessage,
|
||||
};
|
||||
return cb(null, respBody);
|
||||
});
|
||||
},
|
||||
|
||||
getDiskUsage: (log, cb) => {
|
||||
if (!client.getDiskUsage) {
|
||||
log.debug('returning empty disk usage as fallback', { implName });
|
||||
return cb(null, {});
|
||||
}
|
||||
return client.getDiskUsage(log.getSerializedUids(), cb);
|
||||
},
|
||||
|
||||
|
||||
/**
|
||||
* _putForCopy - put used for copying object
|
||||
* @param {object} cipherBundle - cipher bundle that encrypt the data
|
||||
* @param {object} stream - stream containing the data
|
||||
* @param {object} part - element of dataLocator array
|
||||
* @param {object} dataStoreContext - information of the
|
||||
* destination object
|
||||
* dataStoreContext.bucketName: destination bucket name,
|
||||
* dataStoreContext.owner: owner,
|
||||
* dataStoreContext.namespace: request namespace,
|
||||
* dataStoreContext.objectKey: destination object key name,
|
||||
* @param {BackendInfo} destBackendInfo - Instance of BackendInfo:
|
||||
* Represents the info necessary to evaluate which data backend to use
|
||||
* on a data put call.
|
||||
* @param {object} log - Werelogs request logger
|
||||
* @param {function} cb - callback
|
||||
* @returns {function} cb - callback
|
||||
*/
|
||||
_putForCopy: (cipherBundle, stream, part, dataStoreContext,
|
||||
destBackendInfo, log, cb) => data.put(cipherBundle, stream,
|
||||
part.size, dataStoreContext,
|
||||
destBackendInfo, log,
|
||||
(error, partRetrievalInfo) => {
|
||||
if (error) {
|
||||
return cb(error);
|
||||
}
|
||||
const partResult = {
|
||||
key: partRetrievalInfo.key,
|
||||
dataStoreName: partRetrievalInfo
|
||||
.dataStoreName,
|
||||
dataStoreType: partRetrievalInfo
|
||||
.dataStoreType,
|
||||
start: part.start,
|
||||
size: part.size,
|
||||
};
|
||||
if (cipherBundle) {
|
||||
partResult.cryptoScheme = cipherBundle.cryptoScheme;
|
||||
partResult.cipheredDataKey = cipherBundle.cipheredDataKey;
|
||||
}
|
||||
if (part.dataStoreETag) {
|
||||
partResult.dataStoreETag = part.dataStoreETag;
|
||||
}
|
||||
if (partRetrievalInfo.dataStoreVersionId) {
|
||||
partResult.dataStoreVersionId =
|
||||
partRetrievalInfo.dataStoreVersionId;
|
||||
}
|
||||
return cb(null, partResult);
|
||||
}),
|
||||
|
||||
/**
|
||||
* _dataCopyPut - put used for copying object with and without
|
||||
* encryption
|
||||
* @param {string} serverSideEncryption - Server side encryption
|
||||
* @param {object} stream - stream containing the data
|
||||
* @param {object} part - element of dataLocator array
|
||||
* @param {object} dataStoreContext - information of the
|
||||
* destination object
|
||||
* dataStoreContext.bucketName: destination bucket name,
|
||||
* dataStoreContext.owner: owner,
|
||||
* dataStoreContext.namespace: request namespace,
|
||||
* dataStoreContext.objectKey: destination object key name,
|
||||
* @param {BackendInfo} destBackendInfo - Instance of BackendInfo:
|
||||
* Represents the info necessary to evaluate which data backend to use
|
||||
* on a data put call.
|
||||
* @param {object} log - Werelogs request logger
|
||||
* @param {function} cb - callback
|
||||
* @returns {function} cb - callback
|
||||
*/
|
||||
_dataCopyPut: (serverSideEncryption, stream, part, dataStoreContext,
|
||||
destBackendInfo, log, cb) => {
|
||||
if (serverSideEncryption) {
|
||||
return kms.createCipherBundle(
|
||||
serverSideEncryption,
|
||||
log, (err, cipherBundle) => {
|
||||
if (err) {
|
||||
log.debug('error getting cipherBundle');
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return data._putForCopy(cipherBundle, stream, part,
|
||||
dataStoreContext, destBackendInfo, log, cb);
|
||||
});
|
||||
}
|
||||
// Copied object is not encrypted so just put it
|
||||
// without a cipherBundle
|
||||
return data._putForCopy(null, stream, part, dataStoreContext,
|
||||
destBackendInfo, log, cb);
|
||||
},
|
||||
|
||||
/**
|
||||
* copyObject - copy object
|
||||
* @param {object} request - request object
|
||||
* @param {string} sourceLocationConstraintName -
|
||||
* source locationContraint name (awsbackend, azurebackend, ...)
|
||||
* @param {object} storeMetadataParams - metadata information of the
|
||||
* source object
|
||||
* @param {array} dataLocator - source object metadata location(s)
|
||||
* NOTE: for Azure and AWS data backend this array only has one item
|
||||
* @param {object} dataStoreContext - information of the
|
||||
* destination object
|
||||
* dataStoreContext.bucketName: destination bucket name,
|
||||
* dataStoreContext.owner: owner,
|
||||
* dataStoreContext.namespace: request namespace,
|
||||
* dataStoreContext.objectKey: destination object key name,
|
||||
* @param {BackendInfo} destBackendInfo - Instance of BackendInfo:
|
||||
* Represents the info necessary to evaluate which data backend to use
|
||||
* on a data put call.
|
||||
* @param {object} sourceBucketMD - metadata of the source bucket
|
||||
* @param {object} destBucketMD - metadata of the destination bucket
|
||||
* @param {object} log - Werelogs request logger
|
||||
* @param {function} cb - callback
|
||||
* @returns {function} cb - callback
|
||||
*/
|
||||
copyObject: (request,
|
||||
sourceLocationConstraintName, storeMetadataParams, dataLocator,
|
||||
dataStoreContext, destBackendInfo, sourceBucketMD, destBucketMD, log,
|
||||
cb) => {
|
||||
const serverSideEncryption = destBucketMD.getServerSideEncryption();
|
||||
if (config.backends.data === 'multiple' &&
|
||||
utils.externalBackendCopy(sourceLocationConstraintName,
|
||||
storeMetadataParams.dataStoreName, sourceBucketMD, destBucketMD)) {
|
||||
const destLocationConstraintName =
|
||||
storeMetadataParams.dataStoreName;
|
||||
const objectGetInfo = dataLocator[0];
|
||||
const externalSourceKey = objectGetInfo.key;
|
||||
return client.copyObject(request, destLocationConstraintName,
|
||||
externalSourceKey, sourceLocationConstraintName,
|
||||
storeMetadataParams, log, (error, objectRetrievalInfo) => {
|
||||
if (error) {
|
||||
return cb(error);
|
||||
}
|
||||
const putResult = {
|
||||
key: objectRetrievalInfo.key,
|
||||
dataStoreName: objectRetrievalInfo.
|
||||
dataStoreName,
|
||||
dataStoreType: objectRetrievalInfo.
|
||||
dataStoreType,
|
||||
dataStoreVersionId:
|
||||
objectRetrievalInfo.dataStoreVersionId,
|
||||
size: storeMetadataParams.size,
|
||||
dataStoreETag: objectGetInfo.dataStoreETag,
|
||||
start: objectGetInfo.start,
|
||||
};
|
||||
const putResultArr = [putResult];
|
||||
return cb(null, putResultArr);
|
||||
});
|
||||
}
|
||||
|
||||
// dataLocator is an array. need to get and put all parts
|
||||
// For now, copy 1 part at a time. Could increase the second
|
||||
// argument here to increase the number of parts
|
||||
// copied at once.
|
||||
return async.mapLimit(dataLocator, 1,
|
||||
// eslint-disable-next-line prefer-arrow-callback
|
||||
function copyPart(part, copyCb) {
|
||||
if (part.dataStoreType === 'azure') {
|
||||
const passThrough = new PassThrough();
|
||||
return async.parallel([
|
||||
parallelCb => data.get(part, passThrough, log, err =>
|
||||
parallelCb(err)),
|
||||
parallelCb => data._dataCopyPut(serverSideEncryption,
|
||||
passThrough,
|
||||
part, dataStoreContext, destBackendInfo, log,
|
||||
parallelCb),
|
||||
], (err, res) => {
|
||||
if (err) {
|
||||
return copyCb(err);
|
||||
}
|
||||
return copyCb(null, res[1]);
|
||||
});
|
||||
}
|
||||
return data.get(part, null, log, (err, stream) => {
|
||||
if (err) {
|
||||
return copyCb(err);
|
||||
}
|
||||
return data._dataCopyPut(serverSideEncryption, stream,
|
||||
part, dataStoreContext, destBackendInfo, log, copyCb);
|
||||
});
|
||||
}, (err, results) => {
|
||||
if (err) {
|
||||
log.debug('error transferring data from source',
|
||||
{ error: err });
|
||||
return cb(err);
|
||||
}
|
||||
return cb(null, results);
|
||||
});
|
||||
},
|
||||
|
||||
|
||||
_dataCopyPutPart: (request,
|
||||
serverSideEncryption, stream, part,
|
||||
dataStoreContext, destBackendInfo, locations, log, cb) => {
|
||||
const numberPartSize =
|
||||
Number.parseInt(part.size, 10);
|
||||
const partNumber = Number.parseInt(request.query.partNumber, 10);
|
||||
const uploadId = request.query.uploadId;
|
||||
const destObjectKey = request.objectKey;
|
||||
const destBucketName = request.bucketName;
|
||||
const destLocationConstraintName = destBackendInfo
|
||||
.getControllingLocationConstraint();
|
||||
if (externalBackends[config
|
||||
.locationConstraints[destLocationConstraintName]
|
||||
.type]) {
|
||||
return multipleBackendGateway.uploadPart(null, null,
|
||||
stream, numberPartSize,
|
||||
destLocationConstraintName, destObjectKey, uploadId,
|
||||
partNumber, destBucketName, log,
|
||||
(err, partInfo) => {
|
||||
if (err) {
|
||||
log.error('error putting ' +
|
||||
'part to AWS', {
|
||||
error: err,
|
||||
method:
|
||||
'objectPutCopyPart::' +
|
||||
'multipleBackendGateway.' +
|
||||
'uploadPart',
|
||||
});
|
||||
return cb(errors.ServiceUnavailable);
|
||||
}
|
||||
// skip to end of waterfall
|
||||
// because don't need to store
|
||||
// part metadata
|
||||
if (partInfo &&
|
||||
partInfo.dataStoreType === 'aws_s3') {
|
||||
// if data backend handles MPU, skip to end
|
||||
// of waterfall
|
||||
const partResult = {
|
||||
dataStoreETag: partInfo.dataStoreETag,
|
||||
};
|
||||
locations.push(partResult);
|
||||
return cb(skipError, partInfo.dataStoreETag);
|
||||
} else if (
|
||||
partInfo &&
|
||||
partInfo.dataStoreType === 'azure') {
|
||||
const partResult = {
|
||||
key: partInfo.key,
|
||||
dataStoreName: partInfo.dataStoreName,
|
||||
dataStoreETag: partInfo.dataStoreETag,
|
||||
size: numberPartSize,
|
||||
numberSubParts:
|
||||
partInfo.numberSubParts,
|
||||
partNumber: partInfo.partNumber,
|
||||
};
|
||||
locations.push(partResult);
|
||||
return cb();
|
||||
}
|
||||
return cb(skipError);
|
||||
});
|
||||
}
|
||||
if (serverSideEncryption) {
|
||||
return kms.createCipherBundle(
|
||||
serverSideEncryption,
|
||||
log, (err, cipherBundle) => {
|
||||
if (err) {
|
||||
log.debug('error getting cipherBundle',
|
||||
{ error: err });
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return data.put(cipherBundle, stream,
|
||||
numberPartSize, dataStoreContext,
|
||||
destBackendInfo, log,
|
||||
(error, partRetrievalInfo,
|
||||
hashedStream) => {
|
||||
if (error) {
|
||||
log.debug('error putting ' +
|
||||
'encrypted part', { error });
|
||||
return cb(error);
|
||||
}
|
||||
const partResult = {
|
||||
key: partRetrievalInfo.key,
|
||||
dataStoreName: partRetrievalInfo
|
||||
.dataStoreName,
|
||||
dataStoreETag: hashedStream
|
||||
.completedHash,
|
||||
// Do not include part start
|
||||
// here since will change in
|
||||
// final MPU object
|
||||
size: numberPartSize,
|
||||
sseCryptoScheme: cipherBundle
|
||||
.cryptoScheme,
|
||||
sseCipheredDataKey: cipherBundle
|
||||
.cipheredDataKey,
|
||||
sseAlgorithm: cipherBundle
|
||||
.algorithm,
|
||||
sseMasterKeyId: cipherBundle
|
||||
.masterKeyId,
|
||||
};
|
||||
locations.push(partResult);
|
||||
return cb();
|
||||
});
|
||||
});
|
||||
}
|
||||
// Copied object is not encrypted so just put it
|
||||
// without a cipherBundle
|
||||
return data.put(null, stream, numberPartSize,
|
||||
dataStoreContext, destBackendInfo,
|
||||
log, (error, partRetrievalInfo, hashedStream) => {
|
||||
if (error) {
|
||||
log.debug('error putting object part',
|
||||
{ error });
|
||||
return cb(error);
|
||||
}
|
||||
const partResult = {
|
||||
key: partRetrievalInfo.key,
|
||||
dataStoreName: partRetrievalInfo.dataStoreName,
|
||||
dataStoreETag: hashedStream.completedHash,
|
||||
size: numberPartSize,
|
||||
};
|
||||
locations.push(partResult);
|
||||
return cb();
|
||||
});
|
||||
},
|
||||
|
||||
/**
|
||||
* uploadPartCopy - put copy part
|
||||
* @param {object} request - request object
|
||||
* @param {object} log - Werelogs request logger
|
||||
* @param {object} destBucketMD - destination bucket metadata
|
||||
* @param {string} sourceLocationConstraintName -
|
||||
* source locationContraint name (awsbackend, azurebackend, ...)
|
||||
* @param {string} destLocationConstraintName -
|
||||
* location of the destination MPU object (awsbackend, azurebackend, ...)
|
||||
* @param {array} dataLocator - source object metadata location(s)
|
||||
* NOTE: for Azure and AWS data backend this array
|
||||
* @param {object} dataStoreContext - information of the
|
||||
* destination object
|
||||
* dataStoreContext.bucketName: destination bucket name,
|
||||
* dataStoreContext.owner: owner,
|
||||
* dataStoreContext.namespace: request namespace,
|
||||
* dataStoreContext.objectKey: destination object key name,
|
||||
* dataStoreContext.uploadId: uploadId
|
||||
* dataStoreContext.partNumber: request.query.partNumber
|
||||
* @param {function} callback - callback
|
||||
* @returns {function} cb - callback
|
||||
*/
|
||||
uploadPartCopy: (request, log, destBucketMD, sourceLocationConstraintName,
|
||||
destLocationConstraintName, dataLocator, dataStoreContext,
|
||||
callback) => {
|
||||
const serverSideEncryption = destBucketMD.getServerSideEncryption();
|
||||
const lastModified = new Date().toJSON();
|
||||
|
||||
// skip if 0 byte object
|
||||
if (dataLocator.length === 0) {
|
||||
return process.nextTick(() => {
|
||||
callback(null, constants.emptyFileMd5,
|
||||
lastModified, serverSideEncryption, []);
|
||||
});
|
||||
}
|
||||
|
||||
// if destination mpu was initiated in legacy version
|
||||
if (destLocationConstraintName === undefined) {
|
||||
const backendInfoObj = locationConstraintCheck(request,
|
||||
null, destBucketMD, log);
|
||||
if (backendInfoObj.err) {
|
||||
return process.nextTick(() => {
|
||||
callback(backendInfoObj.err);
|
||||
});
|
||||
}
|
||||
// eslint-disable-next-line no-param-reassign
|
||||
destLocationConstraintName = backendInfoObj.controllingLC;
|
||||
}
|
||||
|
||||
const locationTypeMatchAWS =
|
||||
config.backends.data === 'multiple' &&
|
||||
config.getLocationConstraintType(sourceLocationConstraintName) ===
|
||||
config.getLocationConstraintType(destLocationConstraintName) &&
|
||||
config.getLocationConstraintType(sourceLocationConstraintName) ===
|
||||
'aws_s3';
|
||||
|
||||
// NOTE: using multipleBackendGateway.uploadPartCopy only if copying
|
||||
// from AWS to AWS
|
||||
|
||||
if (locationTypeMatchAWS && dataLocator.length === 1) {
|
||||
const awsSourceKey = dataLocator[0].key;
|
||||
return multipleBackendGateway.uploadPartCopy(request,
|
||||
destLocationConstraintName, awsSourceKey,
|
||||
sourceLocationConstraintName, log, (error, eTag) => {
|
||||
if (error) {
|
||||
return callback(error);
|
||||
}
|
||||
return callback(skipError, eTag,
|
||||
lastModified, serverSideEncryption);
|
||||
});
|
||||
}
|
||||
|
||||
const backendInfo = new BackendInfo(destLocationConstraintName);
|
||||
|
||||
// totalHash will be sent through the RelayMD5Sum transform streams
|
||||
// to collect the md5 from multiple streams
|
||||
let totalHash;
|
||||
const locations = [];
|
||||
// dataLocator is an array. need to get and put all parts
|
||||
// in order so can get the ETag of full object
|
||||
return async.forEachOfSeries(dataLocator,
|
||||
// eslint-disable-next-line prefer-arrow-callback
|
||||
function copyPart(part, index, cb) {
|
||||
if (part.dataStoreType === 'azure') {
|
||||
const passThrough = new PassThrough();
|
||||
return async.parallel([
|
||||
next => data.get(part, passThrough, log, err => {
|
||||
if (err) {
|
||||
log.error('error getting data part ' +
|
||||
'from Azure', {
|
||||
error: err,
|
||||
method:
|
||||
'objectPutCopyPart::' +
|
||||
'multipleBackendGateway.' +
|
||||
'copyPart',
|
||||
});
|
||||
return next(err);
|
||||
}
|
||||
return next();
|
||||
}),
|
||||
next => data._dataCopyPutPart(request,
|
||||
serverSideEncryption, passThrough, part,
|
||||
dataStoreContext, backendInfo, locations, log, next),
|
||||
], err => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
return cb();
|
||||
});
|
||||
}
|
||||
return data.get(part, null, log, (err, stream) => {
|
||||
if (err) {
|
||||
log.debug('error getting object part',
|
||||
{ error: err });
|
||||
return cb(err);
|
||||
}
|
||||
const hashedStream =
|
||||
new RelayMD5Sum(totalHash, updatedHash => {
|
||||
totalHash = updatedHash;
|
||||
});
|
||||
stream.pipe(hashedStream);
|
||||
|
||||
// destLocationConstraintName is location of the
|
||||
// destination MPU object
|
||||
return data._dataCopyPutPart(request,
|
||||
serverSideEncryption, hashedStream, part,
|
||||
dataStoreContext, backendInfo, locations, log, cb);
|
||||
});
|
||||
}, err => {
|
||||
// Digest the final combination of all of the part streams
|
||||
if (err && err !== skipError) {
|
||||
log.debug('error transferring data from source',
|
||||
{ error: err, method: 'goGetData' });
|
||||
return callback(err);
|
||||
}
|
||||
if (totalHash) {
|
||||
totalHash = totalHash.digest('hex');
|
||||
} else {
|
||||
totalHash = locations[0].dataStoreETag;
|
||||
}
|
||||
if (err && err === skipError) {
|
||||
return callback(skipError, totalHash,
|
||||
lastModified, serverSideEncryption);
|
||||
}
|
||||
return callback(null, totalHash,
|
||||
lastModified, serverSideEncryption, locations);
|
||||
});
|
||||
},
|
||||
};
|
||||
|
||||
module.exports = data;
|
|
@ -0,0 +1,82 @@
|
|||
# BucketInfo Model Version History
|
||||
|
||||
## Model Version 0/1
|
||||
|
||||
### Properties
|
||||
|
||||
``` javascript
|
||||
this._acl = aclInstance;
|
||||
this._name = name;
|
||||
this._owner = owner;
|
||||
this._ownerDisplayName = ownerDisplayName;
|
||||
this._creationDate = creationDate;
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
No explicit references in the code since mdBucketModelVersion
|
||||
property not added until Model Version 2
|
||||
|
||||
## Model Version 2
|
||||
|
||||
### Properties Added
|
||||
|
||||
``` javascript
|
||||
this._mdBucketModelVersion = mdBucketModelVersion || 0
|
||||
this._transient = transient || false;
|
||||
this._deleted = deleted || false;
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
Used to determine which splitter to use ( < 2 means old splitter)
|
||||
|
||||
## Model version 3
|
||||
|
||||
### Properties Added
|
||||
|
||||
```
|
||||
this._serverSideEncryption = serverSideEncryption || null;
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
Used to store the server bucket encryption info
|
||||
|
||||
## Model version 4
|
||||
|
||||
### Properties Added
|
||||
|
||||
```javascript
|
||||
this._locationConstraint = LocationConstraint || null;
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
Used to store the location constraint of the bucket
|
||||
|
||||
## Model version 5
|
||||
|
||||
### Properties Added
|
||||
|
||||
```javascript
|
||||
this._websiteConfiguration = websiteConfiguration || null;
|
||||
this._cors = cors || null;
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
Used to store the bucket website configuration info
|
||||
and to store CORS rules to apply to cross-domain requests
|
||||
|
||||
## Model version 6
|
||||
|
||||
### Properties Added
|
||||
|
||||
```javascript
|
||||
this._lifecycleConfiguration = lifecycleConfiguration || null;
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
Used to store the bucket lifecycle configuration info
|
|
@ -1,5 +1,6 @@
|
|||
const { errors } = require('arsenal');
|
||||
|
||||
const getReplicationInfo = require('../api/apiUtils/object/getReplicationInfo');
|
||||
const aclUtils = require('../utilities/aclUtils');
|
||||
const constants = require('../../constants');
|
||||
const metadata = require('../metadata/wrapper');
|
||||
|
@ -12,40 +13,18 @@ const acl = {
|
|||
metadata.updateBucket(bucket.getName(), bucket, log, cb);
|
||||
},
|
||||
|
||||
/**
|
||||
* returns true if the specified ACL grant is unchanged
|
||||
* @param {string} grant name of the grant
|
||||
* @param {object} oldAcl old acl config
|
||||
* @param {object} newAcl new acl config
|
||||
* @returns {bool} is the grant the same
|
||||
*/
|
||||
_aclGrantDidNotChange(grant, oldAcl, newAcl) {
|
||||
if (grant === 'Canned') {
|
||||
return oldAcl.Canned === newAcl.Canned;
|
||||
}
|
||||
/**
|
||||
* An ACL grant is in form of an array of strings
|
||||
* An ACL grant is considered unchanged when both the old and new one
|
||||
* contain the same number of elements, and all elements from one
|
||||
* grant are incuded in the other grant
|
||||
*/
|
||||
return oldAcl[grant].length === newAcl[grant].length
|
||||
&& oldAcl[grant].every(value => newAcl[grant].includes(value));
|
||||
},
|
||||
|
||||
addObjectACL(bucket, objectKey, objectMD, addACLParams, params, log, cb) {
|
||||
log.trace('updating object acl in metadata');
|
||||
const isAclUnchanged = Object.keys(objectMD.acl).length === Object.keys(addACLParams).length
|
||||
&& Object.keys(objectMD.acl).every(grant => this._aclGrantDidNotChange(grant, objectMD.acl, addACLParams));
|
||||
if (!isAclUnchanged) {
|
||||
/* eslint-disable no-param-reassign */
|
||||
objectMD.acl = addACLParams;
|
||||
objectMD.originOp = 's3:ObjectAcl:Put';
|
||||
/* eslint-disable no-param-reassign */
|
||||
return metadata.putObjectMD(bucket.getName(), objectKey, objectMD, params, log,
|
||||
cb);
|
||||
// eslint-disable-next-line no-param-reassign
|
||||
objectMD.acl = addACLParams;
|
||||
const replicationInfo = getReplicationInfo(objectKey, bucket, true);
|
||||
if (replicationInfo) {
|
||||
// eslint-disable-next-line no-param-reassign
|
||||
objectMD.replicationInfo = Object.assign({},
|
||||
objectMD.replicationInfo, replicationInfo);
|
||||
}
|
||||
return cb();
|
||||
metadata.putObjectMD(bucket.getName(), objectKey, objectMD, params, log,
|
||||
cb);
|
||||
},
|
||||
|
||||
parseAclFromHeaders(params, cb) {
|
||||
|
@ -157,4 +136,3 @@ const acl = {
|
|||
};
|
||||
|
||||
module.exports = acl;
|
||||
|
|
@ -0,0 +1,213 @@
|
|||
const assert = require('assert');
|
||||
const bucketclient = require('bucketclient');
|
||||
|
||||
const BucketInfo = require('arsenal').models.BucketInfo;
|
||||
const logger = require('../../utilities/logger');
|
||||
const { config } = require('../../Config');
|
||||
|
||||
class BucketClientInterface {
|
||||
constructor() {
|
||||
assert(config.bucketd.bootstrap.length > 0,
|
||||
'bucketd bootstrap list is empty');
|
||||
const { bootstrap, log } = config.bucketd;
|
||||
if (config.https) {
|
||||
const { key, cert, ca } = config.https;
|
||||
logger.info('bucketclient configuration', {
|
||||
bootstrap,
|
||||
log,
|
||||
https: true,
|
||||
});
|
||||
this.client = new bucketclient.RESTClient(bootstrap, log, true,
|
||||
key, cert, ca);
|
||||
} else {
|
||||
logger.info('bucketclient configuration', {
|
||||
bootstrap,
|
||||
log,
|
||||
https: false,
|
||||
});
|
||||
this.client = new bucketclient.RESTClient(bootstrap, log);
|
||||
}
|
||||
}
|
||||
|
||||
createBucket(bucketName, bucketMD, log, cb) {
|
||||
this.client.createBucket(bucketName, log.getSerializedUids(),
|
||||
bucketMD.serialize(), cb);
|
||||
return null;
|
||||
}
|
||||
|
||||
getBucketAttributes(bucketName, log, cb) {
|
||||
this.client.getBucketAttributes(bucketName, log.getSerializedUids(),
|
||||
(err, data) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
return cb(err, BucketInfo.deSerialize(data));
|
||||
});
|
||||
return null;
|
||||
}
|
||||
|
||||
getBucketAndObject(bucketName, objName, params, log, cb) {
|
||||
this.client.getBucketAndObject(bucketName, objName,
|
||||
log.getSerializedUids(), (err, data) => {
|
||||
if (err && (!err.NoSuchKey && !err.ObjNotFound)) {
|
||||
return cb(err);
|
||||
}
|
||||
return cb(null, JSON.parse(data));
|
||||
}, params);
|
||||
return null;
|
||||
}
|
||||
|
||||
getRaftBuckets(raftId, log, cb) {
|
||||
return this.client.getRaftBuckets(raftId, log.getSerializedUids(),
|
||||
(err, data) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
return cb(null, JSON.parse(data));
|
||||
});
|
||||
}
|
||||
|
||||
putBucketAttributes(bucketName, bucketMD, log, cb) {
|
||||
this.client.putBucketAttributes(bucketName, log.getSerializedUids(),
|
||||
bucketMD.serialize(), cb);
|
||||
return null;
|
||||
}
|
||||
|
||||
deleteBucket(bucketName, log, cb) {
|
||||
this.client.deleteBucket(bucketName, log.getSerializedUids(), cb);
|
||||
return null;
|
||||
}
|
||||
|
||||
putObject(bucketName, objName, objVal, params, log, cb) {
|
||||
this.client.putObject(bucketName, objName, JSON.stringify(objVal),
|
||||
log.getSerializedUids(), cb, params);
|
||||
return null;
|
||||
}
|
||||
|
||||
getObject(bucketName, objName, params, log, cb) {
|
||||
this.client.getObject(bucketName, objName, log.getSerializedUids(),
|
||||
(err, data) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
return cb(err, JSON.parse(data));
|
||||
}, params);
|
||||
return null;
|
||||
}
|
||||
|
||||
deleteObject(bucketName, objName, params, log, cb) {
|
||||
this.client.deleteObject(bucketName, objName, log.getSerializedUids(),
|
||||
cb, params);
|
||||
return null;
|
||||
}
|
||||
|
||||
listObject(bucketName, params, log, cb) {
|
||||
this.client.listObject(bucketName, log.getSerializedUids(), params,
|
||||
(err, data) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
return cb(err, JSON.parse(data));
|
||||
});
|
||||
return null;
|
||||
}
|
||||
|
||||
listMultipartUploads(bucketName, params, log, cb) {
|
||||
this.client.listObject(bucketName, log.getSerializedUids(), params,
|
||||
(err, data) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
return cb(null, JSON.parse(data));
|
||||
});
|
||||
return null;
|
||||
}
|
||||
|
||||
_analyzeHealthFailure(log, callback) {
|
||||
let doFail = false;
|
||||
const reason = {
|
||||
msg: 'Map is available and failure ratio is acceptable',
|
||||
};
|
||||
|
||||
// The healthCheck exposed by Bucketd is a light one, we need
|
||||
// to inspect all the RaftSession's statuses to make sense of
|
||||
// it:
|
||||
return this.client.getAllRafts(undefined, (error, payload) => {
|
||||
let statuses = null;
|
||||
try {
|
||||
statuses = JSON.parse(payload);
|
||||
} catch (e) {
|
||||
doFail = true;
|
||||
reason.msg = 'could not interpret status: invalid payload';
|
||||
// Can't do anything anymore if we fail here. return.
|
||||
return callback(doFail, reason);
|
||||
}
|
||||
|
||||
const reducer = (acc, payload) => acc + !payload.connectedToLeader;
|
||||
reason.ratio = statuses.reduce(reducer, 0) / statuses.length;
|
||||
/* NOTE FIXME/TODO: acceptableRatio could be configured later on */
|
||||
reason.acceptableRatio = 0.5;
|
||||
/* If the RaftSession 0 (map) does not work, fail anyways */
|
||||
if (!doFail && !statuses[0].connectedToLeader) {
|
||||
doFail = true;
|
||||
reason.msg = 'Bucket map unavailable';
|
||||
}
|
||||
if (!doFail && reason.ratio > reason.acceptableRatio) {
|
||||
doFail = true;
|
||||
reason.msg = 'Ratio of failing Raft Sessions is too high';
|
||||
}
|
||||
return callback(doFail, reason);
|
||||
}, log);
|
||||
}
|
||||
|
||||
/*
|
||||
* Bucketd offers a behavior that diverges from other sub-components on the
|
||||
* healthCheck: If any of the pieces making up the bucket storage fail (ie:
|
||||
* if any Raft Session is down), bucketd returns a 500 for the healthCheck.
|
||||
*
|
||||
* As seen in S3C-1412, this may become an issue for S3, whenever the
|
||||
* system is only partly failing.
|
||||
*
|
||||
* This means that S3 needs to analyze the situation, and interpret this
|
||||
* status depending on the analysis. S3 will then assess the situation as
|
||||
* critical (and consider it a failure), or not (and consider it a success
|
||||
* anyways, thus not diverging from the healthCheck behavior of other
|
||||
* components).
|
||||
*/
|
||||
checkHealth(implName, log, cb) {
|
||||
return this.client.healthcheck(log, (err, result) => {
|
||||
const respBody = {};
|
||||
if (err) {
|
||||
return this._analyzeHealthFailure(log, (failure, reason) => {
|
||||
const message = reason.msg;
|
||||
// Remove 'msg' from the reason payload.
|
||||
// eslint-disable-next-line no-param-reassign
|
||||
reason.msg = undefined;
|
||||
respBody[implName] = {
|
||||
code: 200,
|
||||
message, // Provide interpreted reason msg
|
||||
body: reason, // Provide analysis data
|
||||
};
|
||||
if (failure) {
|
||||
// Setting the `error` field is how the healthCheck
|
||||
// logic interprets it as an error. Don't forget it !
|
||||
respBody[implName].error = err;
|
||||
respBody[implName].code = err.code; // original error
|
||||
}
|
||||
// error returned as null so async parallel doesn't return
|
||||
// before all backends are checked
|
||||
return cb(null, respBody);
|
||||
}, log);
|
||||
}
|
||||
const parseResult = JSON.parse(result);
|
||||
respBody[implName] = {
|
||||
code: 200,
|
||||
message: 'OK',
|
||||
body: parseResult,
|
||||
};
|
||||
return cb(null, respBody);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = BucketClientInterface;
|
|
@ -0,0 +1,379 @@
|
|||
const cluster = require('cluster');
|
||||
const arsenal = require('arsenal');
|
||||
|
||||
const logger = require('../../utilities/logger');
|
||||
const BucketInfo = arsenal.models.BucketInfo;
|
||||
const constants = require('../../../constants');
|
||||
const { config } = require('../../Config');
|
||||
|
||||
const errors = arsenal.errors;
|
||||
const MetadataFileClient = arsenal.storage.metadata.MetadataFileClient;
|
||||
const versionSep = arsenal.versioning.VersioningConstants.VersionId.Separator;
|
||||
|
||||
const METASTORE = '__metastore';
|
||||
|
||||
class BucketFileInterface {
|
||||
|
||||
/**
|
||||
* @constructor
|
||||
* @param {object} [params] - constructor params
|
||||
* @param {boolean} [params.noDbOpen=false] - true to skip DB open
|
||||
* (for unit tests only)
|
||||
*/
|
||||
constructor(params) {
|
||||
this.logger = logger;
|
||||
const { host, port } = config.metadataClient;
|
||||
this.mdClient = new MetadataFileClient({ host, port });
|
||||
if (params && params.noDbOpen) {
|
||||
return;
|
||||
}
|
||||
this.mdDB = this.mdClient.openDB(err => {
|
||||
if (err) {
|
||||
throw err;
|
||||
}
|
||||
// the metastore sublevel is used to store bucket attributes
|
||||
this.metastore = this.mdDB.openSub(METASTORE);
|
||||
if (cluster.isMaster) {
|
||||
this.setupMetadataServer();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
setupMetadataServer() {
|
||||
/* Since the bucket creation API is expecting the
|
||||
usersBucket to have attributes, we pre-create the
|
||||
usersBucket attributes here */
|
||||
this.mdClient.logger.debug('setting up metadata server');
|
||||
const usersBucketAttr = new BucketInfo(constants.usersBucket,
|
||||
'admin', 'admin', new Date().toJSON(),
|
||||
BucketInfo.currentModelVersion());
|
||||
this.metastore.put(
|
||||
constants.usersBucket,
|
||||
usersBucketAttr.serialize(), {}, err => {
|
||||
if (err) {
|
||||
this.logger.fatal('error writing usersBucket ' +
|
||||
'attributes to metadata',
|
||||
{ error: err });
|
||||
throw (errors.InternalError);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Load DB if exists
|
||||
* @param {String} bucketName - name of bucket
|
||||
* @param {Object} log - logger
|
||||
* @param {function} cb - callback(err, db, attr)
|
||||
* @return {undefined}
|
||||
*/
|
||||
loadDBIfExists(bucketName, log, cb) {
|
||||
this.getBucketAttributes(bucketName, log, (err, attr) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
try {
|
||||
const db = this.mdDB.openSub(bucketName);
|
||||
return cb(null, db, attr);
|
||||
} catch (err) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
});
|
||||
return undefined;
|
||||
}
|
||||
|
||||
createBucket(bucketName, bucketMD, log, cb) {
|
||||
this.getBucketAttributes(bucketName, log, err => {
|
||||
if (err && err !== errors.NoSuchBucket) {
|
||||
return cb(err);
|
||||
}
|
||||
if (err === undefined) {
|
||||
return cb(errors.BucketAlreadyExists);
|
||||
}
|
||||
this.putBucketAttributes(bucketName,
|
||||
bucketMD,
|
||||
log, cb);
|
||||
return undefined;
|
||||
});
|
||||
}
|
||||
|
||||
getBucketAttributes(bucketName, log, cb) {
|
||||
this.metastore
|
||||
.withRequestLogger(log)
|
||||
.get(bucketName, {}, (err, data) => {
|
||||
if (err) {
|
||||
if (err.ObjNotFound) {
|
||||
return cb(errors.NoSuchBucket);
|
||||
}
|
||||
const logObj = {
|
||||
rawError: err,
|
||||
error: err.message,
|
||||
errorStack: err.stack,
|
||||
};
|
||||
log.error('error getting db attributes', logObj);
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return cb(null, BucketInfo.deSerialize(data));
|
||||
});
|
||||
return undefined;
|
||||
}
|
||||
|
||||
getBucketAndObject(bucketName, objName, params, log, cb) {
|
||||
this.loadDBIfExists(bucketName, log, (err, db, bucketAttr) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
db.withRequestLogger(log)
|
||||
.get(objName, params, (err, objAttr) => {
|
||||
if (err) {
|
||||
if (err.ObjNotFound) {
|
||||
return cb(null, {
|
||||
bucket: bucketAttr.serialize(),
|
||||
});
|
||||
}
|
||||
const logObj = {
|
||||
rawError: err,
|
||||
error: err.message,
|
||||
errorStack: err.stack,
|
||||
};
|
||||
log.error('error getting object', logObj);
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return cb(null, {
|
||||
bucket: bucketAttr.serialize(),
|
||||
obj: objAttr,
|
||||
});
|
||||
});
|
||||
return undefined;
|
||||
});
|
||||
return undefined;
|
||||
}
|
||||
|
||||
putBucketAttributes(bucketName, bucketMD, log, cb) {
|
||||
this.metastore
|
||||
.withRequestLogger(log)
|
||||
.put(bucketName, bucketMD.serialize(), {}, err => {
|
||||
if (err) {
|
||||
const logObj = {
|
||||
rawError: err,
|
||||
error: err.message,
|
||||
errorStack: err.stack,
|
||||
};
|
||||
log.error('error putting db attributes', logObj);
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return cb();
|
||||
});
|
||||
return undefined;
|
||||
}
|
||||
|
||||
deleteBucket(bucketName, log, cb) {
|
||||
this.metastore
|
||||
.withRequestLogger(log)
|
||||
.del(bucketName, {}, err => {
|
||||
if (err) {
|
||||
const logObj = {
|
||||
rawError: err,
|
||||
error: err.message,
|
||||
errorStack: err.stack,
|
||||
};
|
||||
log.error('error deleting bucket',
|
||||
logObj);
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return cb();
|
||||
});
|
||||
return undefined;
|
||||
}
|
||||
|
||||
putObject(bucketName, objName, objVal, params, log, cb) {
|
||||
this.loadDBIfExists(bucketName, log, (err, db) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
db.withRequestLogger(log)
|
||||
.put(objName, JSON.stringify(objVal), params, (err, data) => {
|
||||
if (err) {
|
||||
const logObj = {
|
||||
rawError: err,
|
||||
error: err.message,
|
||||
errorStack: err.stack,
|
||||
};
|
||||
log.error('error putting object', logObj);
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return cb(err, data);
|
||||
});
|
||||
return undefined;
|
||||
});
|
||||
}
|
||||
|
||||
getObject(bucketName, objName, params, log, cb) {
|
||||
this.loadDBIfExists(bucketName, log, (err, db) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
db.withRequestLogger(log).get(objName, params, (err, data) => {
|
||||
if (err) {
|
||||
if (err.ObjNotFound) {
|
||||
return cb(errors.NoSuchKey);
|
||||
}
|
||||
const logObj = {
|
||||
rawError: err,
|
||||
error: err.message,
|
||||
errorStack: err.stack,
|
||||
};
|
||||
log.error('error getting object', logObj);
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return cb(null, JSON.parse(data));
|
||||
});
|
||||
return undefined;
|
||||
});
|
||||
}
|
||||
|
||||
deleteObject(bucketName, objName, params, log, cb) {
|
||||
this.loadDBIfExists(bucketName, log, (err, db) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
db.withRequestLogger(log).del(objName, params, err => {
|
||||
if (err) {
|
||||
const logObj = {
|
||||
rawError: err,
|
||||
error: err.message,
|
||||
errorStack: err.stack,
|
||||
};
|
||||
log.error('error deleting object', logObj);
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return cb();
|
||||
});
|
||||
return undefined;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* This complex function deals with different extensions of bucket listing:
|
||||
* Delimiter based search or MPU based search.
|
||||
* @param {String} bucketName - The name of the bucket to list
|
||||
* @param {Object} params - The params to search
|
||||
* @param {Object} log - The logger object
|
||||
* @param {function} cb - Callback when done
|
||||
* @return {undefined}
|
||||
*/
|
||||
internalListObject(bucketName, params, log, cb) {
|
||||
const extName = params.listingType;
|
||||
const extension = new arsenal.algorithms.list[extName](params, log);
|
||||
const requestParams = extension.genMDParams();
|
||||
this.loadDBIfExists(bucketName, log, (err, db) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
let cbDone = false;
|
||||
db.withRequestLogger(log)
|
||||
.createReadStream(requestParams, (err, stream) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
stream
|
||||
.on('data', e => {
|
||||
if (extension.filter(e) < 0) {
|
||||
stream.emit('end');
|
||||
stream.destroy();
|
||||
}
|
||||
})
|
||||
.on('error', err => {
|
||||
if (!cbDone) {
|
||||
cbDone = true;
|
||||
const logObj = {
|
||||
rawError: err,
|
||||
error: err.message,
|
||||
errorStack: err.stack,
|
||||
};
|
||||
log.error('error listing objects', logObj);
|
||||
cb(errors.InternalError);
|
||||
}
|
||||
})
|
||||
.on('end', () => {
|
||||
if (!cbDone) {
|
||||
cbDone = true;
|
||||
const data = extension.result();
|
||||
cb(null, data);
|
||||
}
|
||||
});
|
||||
return undefined;
|
||||
});
|
||||
return undefined;
|
||||
});
|
||||
}
|
||||
|
||||
listObject(bucketName, params, log, cb) {
|
||||
return this.internalListObject(bucketName, params, log, cb);
|
||||
}
|
||||
|
||||
listMultipartUploads(bucketName, params, log, cb) {
|
||||
return this.internalListObject(bucketName, params, log, cb);
|
||||
}
|
||||
|
||||
getUUID(log, cb) {
|
||||
return this.mdDB.getUUID(cb);
|
||||
}
|
||||
|
||||
getDiskUsage(cb) {
|
||||
return this.mdDB.getDiskUsage(cb);
|
||||
}
|
||||
|
||||
countItems(log, cb) {
|
||||
const params = {};
|
||||
const extension = new arsenal.algorithms.list.Basic(params, log);
|
||||
const requestParams = extension.genMDParams();
|
||||
|
||||
const res = {
|
||||
objects: 0,
|
||||
versions: 0,
|
||||
buckets: 0,
|
||||
};
|
||||
let cbDone = false;
|
||||
|
||||
this.mdDB.rawListKeys(requestParams, (err, stream) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
stream
|
||||
.on('data', e => {
|
||||
if (!e.includes(METASTORE)) {
|
||||
if (e.includes(constants.usersBucket)) {
|
||||
res.buckets++;
|
||||
} else if (e.includes(versionSep)) {
|
||||
res.versions++;
|
||||
} else {
|
||||
res.objects++;
|
||||
}
|
||||
}
|
||||
})
|
||||
.on('error', err => {
|
||||
if (!cbDone) {
|
||||
cbDone = true;
|
||||
const logObj = {
|
||||
error: err,
|
||||
errorMessage: err.message,
|
||||
errorStack: err.stack,
|
||||
};
|
||||
log.error('error listing objects', logObj);
|
||||
cb(errors.InternalError);
|
||||
}
|
||||
})
|
||||
.on('end', () => {
|
||||
if (!cbDone) {
|
||||
cbDone = true;
|
||||
return cb(null, res);
|
||||
}
|
||||
return undefined;
|
||||
});
|
||||
return undefined;
|
||||
});
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = BucketFileInterface;
|
|
@ -0,0 +1,34 @@
|
|||
const ListResult = require('./ListResult');
|
||||
|
||||
class ListMultipartUploadsResult extends ListResult {
|
||||
constructor() {
|
||||
super();
|
||||
this.Uploads = [];
|
||||
this.NextKeyMarker = undefined;
|
||||
this.NextUploadIdMarker = undefined;
|
||||
}
|
||||
|
||||
addUpload(uploadInfo) {
|
||||
this.Uploads.push({
|
||||
key: decodeURIComponent(uploadInfo.key),
|
||||
value: {
|
||||
UploadId: uploadInfo.uploadId,
|
||||
Initiator: {
|
||||
ID: uploadInfo.initiatorID,
|
||||
DisplayName: uploadInfo.initiatorDisplayName,
|
||||
},
|
||||
Owner: {
|
||||
ID: uploadInfo.ownerID,
|
||||
DisplayName: uploadInfo.ownerDisplayName,
|
||||
},
|
||||
StorageClass: uploadInfo.storageClass,
|
||||
Initiated: uploadInfo.initiated,
|
||||
},
|
||||
});
|
||||
this.MaxKeys += 1;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
ListMultipartUploadsResult,
|
||||
};
|
|
@ -0,0 +1,27 @@
|
|||
class ListResult {
|
||||
constructor() {
|
||||
this.IsTruncated = false;
|
||||
this.NextMarker = undefined;
|
||||
this.CommonPrefixes = [];
|
||||
/*
|
||||
Note: this.MaxKeys will get incremented as
|
||||
keys are added so that when response is returned,
|
||||
this.MaxKeys will equal total keys in response
|
||||
(with each CommonPrefix counting as 1 key)
|
||||
*/
|
||||
this.MaxKeys = 0;
|
||||
}
|
||||
|
||||
addCommonPrefix(prefix) {
|
||||
if (!this.hasCommonPrefix(prefix)) {
|
||||
this.CommonPrefixes.push(prefix);
|
||||
this.MaxKeys += 1;
|
||||
}
|
||||
}
|
||||
|
||||
hasCommonPrefix(prefix) {
|
||||
return (this.CommonPrefixes.indexOf(prefix) !== -1);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = ListResult;
|
|
@ -0,0 +1,331 @@
|
|||
const { errors, algorithms, versioning } = require('arsenal');
|
||||
|
||||
const getMultipartUploadListing = require('./getMultipartUploadListing');
|
||||
const { metadata } = require('./metadata');
|
||||
const { config } = require('../../Config');
|
||||
|
||||
const genVID = versioning.VersionID.generateVersionId;
|
||||
|
||||
const defaultMaxKeys = 1000;
|
||||
let uidCounter = 0;
|
||||
|
||||
function generateVersionId() {
|
||||
return genVID(uidCounter++, config.replicationGroupId);
|
||||
}
|
||||
|
||||
function formatVersionKey(key, versionId) {
|
||||
return `${key}\0${versionId}`;
|
||||
}
|
||||
|
||||
function inc(str) {
|
||||
return str ? (str.slice(0, str.length - 1) +
|
||||
String.fromCharCode(str.charCodeAt(str.length - 1) + 1)) : str;
|
||||
}
|
||||
|
||||
const metastore = {
|
||||
createBucket: (bucketName, bucketMD, log, cb) => {
|
||||
process.nextTick(() => {
|
||||
metastore.getBucketAttributes(bucketName, log, (err, bucket) => {
|
||||
// TODO Check whether user already owns the bucket,
|
||||
// if so return "BucketAlreadyOwnedByYou"
|
||||
// If not owned by user, return "BucketAlreadyExists"
|
||||
if (bucket) {
|
||||
return cb(errors.BucketAlreadyExists);
|
||||
}
|
||||
metadata.buckets.set(bucketName, bucketMD);
|
||||
metadata.keyMaps.set(bucketName, new Map);
|
||||
return cb();
|
||||
});
|
||||
});
|
||||
},
|
||||
|
||||
putBucketAttributes: (bucketName, bucketMD, log, cb) => {
|
||||
process.nextTick(() => {
|
||||
metastore.getBucketAttributes(bucketName, log, err => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
metadata.buckets.set(bucketName, bucketMD);
|
||||
return cb();
|
||||
});
|
||||
});
|
||||
},
|
||||
|
||||
getBucketAttributes: (bucketName, log, cb) => {
|
||||
process.nextTick(() => {
|
||||
if (!metadata.buckets.has(bucketName)) {
|
||||
return cb(errors.NoSuchBucket);
|
||||
}
|
||||
return cb(null, metadata.buckets.get(bucketName));
|
||||
});
|
||||
},
|
||||
|
||||
deleteBucket: (bucketName, log, cb) => {
|
||||
process.nextTick(() => {
|
||||
metastore.getBucketAttributes(bucketName, log, err => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
if (metadata.keyMaps.has(bucketName)
|
||||
&& metadata.keyMaps.get(bucketName).length > 0) {
|
||||
return cb(errors.BucketNotEmpty);
|
||||
}
|
||||
metadata.buckets.delete(bucketName);
|
||||
metadata.keyMaps.delete(bucketName);
|
||||
return cb(null);
|
||||
});
|
||||
});
|
||||
},
|
||||
|
||||
putObject: (bucketName, objName, objVal, params, log, cb) => {
|
||||
process.nextTick(() => {
|
||||
metastore.getBucketAttributes(bucketName, log, err => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
/*
|
||||
valid combinations of versioning options:
|
||||
- !versioning && !versionId: normal non-versioning put
|
||||
- versioning && !versionId: create a new version
|
||||
- versionId: update (PUT/DELETE) an existing version,
|
||||
and also update master version in case the put
|
||||
version is newer or same version than master.
|
||||
if versionId === '' update master version
|
||||
*/
|
||||
|
||||
if (params && params.versionId) {
|
||||
objVal.versionId = params.versionId; // eslint-disable-line
|
||||
const mst = metadata.keyMaps.get(bucketName).get(objName);
|
||||
if (mst && mst.versionId === params.versionId || !mst) {
|
||||
metadata.keyMaps.get(bucketName).set(objName, objVal);
|
||||
}
|
||||
// eslint-disable-next-line
|
||||
objName = formatVersionKey(objName, params.versionId);
|
||||
metadata.keyMaps.get(bucketName).set(objName, objVal);
|
||||
return cb(null, `{"versionId":"${objVal.versionId}"}`);
|
||||
}
|
||||
if (params && params.versioning) {
|
||||
const versionId = generateVersionId();
|
||||
objVal.versionId = versionId; // eslint-disable-line
|
||||
metadata.keyMaps.get(bucketName).set(objName, objVal);
|
||||
// eslint-disable-next-line
|
||||
objName = formatVersionKey(objName, versionId);
|
||||
metadata.keyMaps.get(bucketName).set(objName, objVal);
|
||||
return cb(null, `{"versionId":"${versionId}"}`);
|
||||
}
|
||||
if (params && params.versionId === '') {
|
||||
const versionId = generateVersionId();
|
||||
objVal.versionId = versionId; // eslint-disable-line
|
||||
metadata.keyMaps.get(bucketName).set(objName, objVal);
|
||||
return cb(null, `{"versionId":"${objVal.versionId}"}`);
|
||||
}
|
||||
metadata.keyMaps.get(bucketName).set(objName, objVal);
|
||||
return cb(null);
|
||||
});
|
||||
});
|
||||
},
|
||||
|
||||
getBucketAndObject: (bucketName, objName, params, log, cb) => {
|
||||
process.nextTick(() => {
|
||||
metastore.getBucketAttributes(bucketName, log, (err, bucket) => {
|
||||
if (err) {
|
||||
return cb(err, { bucket });
|
||||
}
|
||||
if (params && params.versionId) {
|
||||
// eslint-disable-next-line
|
||||
objName = formatVersionKey(objName, params.versionId);
|
||||
}
|
||||
if (!metadata.keyMaps.has(bucketName)
|
||||
|| !metadata.keyMaps.get(bucketName).has(objName)) {
|
||||
return cb(null, { bucket: bucket.serialize() });
|
||||
}
|
||||
return cb(null, {
|
||||
bucket: bucket.serialize(),
|
||||
obj: JSON.stringify(
|
||||
metadata.keyMaps.get(bucketName).get(objName)
|
||||
),
|
||||
});
|
||||
});
|
||||
});
|
||||
},
|
||||
|
||||
getObject: (bucketName, objName, params, log, cb) => {
|
||||
process.nextTick(() => {
|
||||
metastore.getBucketAttributes(bucketName, log, err => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
if (params && params.versionId) {
|
||||
// eslint-disable-next-line
|
||||
objName = formatVersionKey(objName, params.versionId);
|
||||
}
|
||||
if (!metadata.keyMaps.has(bucketName)
|
||||
|| !metadata.keyMaps.get(bucketName).has(objName)) {
|
||||
return cb(errors.NoSuchKey);
|
||||
}
|
||||
return cb(null, metadata.keyMaps.get(bucketName).get(objName));
|
||||
});
|
||||
});
|
||||
},
|
||||
|
||||
deleteObject: (bucketName, objName, params, log, cb) => {
|
||||
process.nextTick(() => {
|
||||
metastore.getBucketAttributes(bucketName, log, err => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
if (!metadata.keyMaps.get(bucketName).has(objName)) {
|
||||
return cb(errors.NoSuchKey);
|
||||
}
|
||||
if (params && params.versionId) {
|
||||
const baseKey = inc(formatVersionKey(objName, ''));
|
||||
const vobjName = formatVersionKey(objName,
|
||||
params.versionId);
|
||||
metadata.keyMaps.get(bucketName).delete(vobjName);
|
||||
const mst = metadata.keyMaps.get(bucketName).get(objName);
|
||||
if (mst.versionId === params.versionId) {
|
||||
const keys = [];
|
||||
metadata.keyMaps.get(bucketName).forEach((val, key) => {
|
||||
if (key < baseKey && key > vobjName) {
|
||||
keys.push(key);
|
||||
}
|
||||
});
|
||||
if (keys.length === 0) {
|
||||
metadata.keyMaps.get(bucketName).delete(objName);
|
||||
return cb();
|
||||
}
|
||||
const key = keys.sort()[0];
|
||||
const value = metadata.keyMaps.get(bucketName).get(key);
|
||||
metadata.keyMaps.get(bucketName).set(objName, value);
|
||||
}
|
||||
return cb();
|
||||
}
|
||||
metadata.keyMaps.get(bucketName).delete(objName);
|
||||
return cb();
|
||||
});
|
||||
});
|
||||
},
|
||||
|
||||
_hasDeleteMarker(key, keyMap) {
|
||||
const objectMD = keyMap.get(key);
|
||||
if (objectMD['x-amz-delete-marker'] !== undefined) {
|
||||
return (objectMD['x-amz-delete-marker'] === true);
|
||||
}
|
||||
return false;
|
||||
},
|
||||
|
||||
listObject(bucketName, params, log, cb) {
|
||||
process.nextTick(() => {
|
||||
const {
|
||||
prefix,
|
||||
marker,
|
||||
delimiter,
|
||||
maxKeys,
|
||||
continuationToken,
|
||||
startAfter,
|
||||
} = params;
|
||||
if (prefix && typeof prefix !== 'string') {
|
||||
return cb(errors.InvalidArgument);
|
||||
}
|
||||
|
||||
if (marker && typeof marker !== 'string') {
|
||||
return cb(errors.InvalidArgument);
|
||||
}
|
||||
|
||||
if (delimiter && typeof delimiter !== 'string') {
|
||||
return cb(errors.InvalidArgument);
|
||||
}
|
||||
|
||||
if (maxKeys && typeof maxKeys !== 'number') {
|
||||
return cb(errors.InvalidArgument);
|
||||
}
|
||||
|
||||
if (continuationToken && typeof continuationToken !== 'string') {
|
||||
return cb(errors.InvalidArgument);
|
||||
}
|
||||
|
||||
if (startAfter && typeof startAfter !== 'string') {
|
||||
return cb(errors.InvalidArgument);
|
||||
}
|
||||
|
||||
// If paramMaxKeys is undefined, the default parameter will set it.
|
||||
// However, if it is null, the default parameter will not set it.
|
||||
let numKeys = maxKeys;
|
||||
if (numKeys === null) {
|
||||
numKeys = defaultMaxKeys;
|
||||
}
|
||||
|
||||
if (!metadata.keyMaps.has(bucketName)) {
|
||||
return cb(errors.NoSuchBucket);
|
||||
}
|
||||
|
||||
// If marker specified, edit the keys array so it
|
||||
// only contains keys that occur alphabetically after the marker
|
||||
const listingType = params.listingType;
|
||||
const extension = new algorithms.list[listingType](params, log);
|
||||
const listingParams = extension.genMDParams();
|
||||
|
||||
const keys = [];
|
||||
metadata.keyMaps.get(bucketName).forEach((val, key) => {
|
||||
if (listingParams.gt && listingParams.gt >= key) {
|
||||
return null;
|
||||
}
|
||||
if (listingParams.gte && listingParams.gte > key) {
|
||||
return null;
|
||||
}
|
||||
if (listingParams.lt && key >= listingParams.lt) {
|
||||
return null;
|
||||
}
|
||||
if (listingParams.lte && key > listingParams.lte) {
|
||||
return null;
|
||||
}
|
||||
return keys.push(key);
|
||||
});
|
||||
keys.sort();
|
||||
|
||||
// Iterate through keys array and filter keys containing
|
||||
// delimiter into response.CommonPrefixes and filter remaining
|
||||
// keys into response.Contents
|
||||
for (let i = 0; i < keys.length; ++i) {
|
||||
const currentKey = keys[i];
|
||||
// Do not list object with delete markers
|
||||
if (this._hasDeleteMarker(currentKey,
|
||||
metadata.keyMaps.get(bucketName))) {
|
||||
continue;
|
||||
}
|
||||
const objMD = metadata.keyMaps.get(bucketName).get(currentKey);
|
||||
const value = JSON.stringify(objMD);
|
||||
const obj = {
|
||||
key: currentKey,
|
||||
value,
|
||||
};
|
||||
// calling Ext.filter(obj) adds the obj to the Ext result if
|
||||
// not filtered.
|
||||
// Also, Ext.filter returns false when hit max keys.
|
||||
// What a nifty function!
|
||||
if (extension.filter(obj) < 0) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
return cb(null, extension.result());
|
||||
});
|
||||
},
|
||||
|
||||
listMultipartUploads(bucketName, listingParams, log, cb) {
|
||||
process.nextTick(() => {
|
||||
metastore.getBucketAttributes(bucketName, log, (err, bucket) => {
|
||||
if (bucket === undefined) {
|
||||
// no on going multipart uploads, return empty listing
|
||||
return cb(null, {
|
||||
IsTruncated: false,
|
||||
NextMarker: undefined,
|
||||
MaxKeys: 0,
|
||||
});
|
||||
}
|
||||
return getMultipartUploadListing(bucket, listingParams, cb);
|
||||
});
|
||||
});
|
||||
},
|
||||
};
|
||||
|
||||
module.exports = metastore;
|
|
@ -0,0 +1,62 @@
|
|||
# bucket_mem design
|
||||
|
||||
## RATIONALE
|
||||
|
||||
The bucket API will be used for managing buckets behind the S3 interface.
|
||||
|
||||
We plan to have only 2 backends using this interface:
|
||||
|
||||
* One production backend
|
||||
* One debug backend purely in memory
|
||||
|
||||
One important remark here is that we don't want an abstraction but a
|
||||
duck-typing style interface (different classes MemoryBucket and Bucket having
|
||||
the same methods putObjectMD(), getObjectMD(), etc).
|
||||
|
||||
Notes about the memory backend: The backend is currently a simple key/value
|
||||
store in memory. The functions actually use nextTick() to emulate the future
|
||||
asynchronous behavior of the production backend.
|
||||
|
||||
## BUCKET API
|
||||
|
||||
The bucket API is a very simple API with 5 functions:
|
||||
|
||||
- putObjectMD(): put metadata for an object in the bucket
|
||||
- getObjectMD(): get metadata from the bucket
|
||||
- deleteObjectMD(): delete metadata for an object from the bucket
|
||||
- deleteBucketMD(): delete a bucket
|
||||
- getBucketListObjects(): perform the complex bucket listing AWS search
|
||||
function with various flavors. This function returns a response in a
|
||||
ListBucketResult object.
|
||||
|
||||
getBucketListObjects(prefix, marker, delimiter, maxKeys, callback) behavior is
|
||||
the following:
|
||||
|
||||
prefix (not required): Limits the response to keys that begin with the
|
||||
specified prefix. You can use prefixes to separate a bucket into different
|
||||
groupings of keys. (You can think of using prefix to make groups in the same
|
||||
way you'd use a folder in a file system.)
|
||||
|
||||
marker (not required): Specifies the key to start with when listing objects in
|
||||
a bucket. Amazon S3 returns object keys in alphabetical order, starting with
|
||||
key after the marker in order.
|
||||
|
||||
delimiter (not required): A delimiter is a character you use to group keys.
|
||||
All keys that contain the same string between the prefix, if specified, and the
|
||||
first occurrence of the delimiter after the prefix are grouped under a single
|
||||
result element, CommonPrefixes. If you don't specify the prefix parameter, then
|
||||
the substring starts at the beginning of the key. The keys that are grouped
|
||||
under CommonPrefixes are not returned elsewhere in the response.
|
||||
|
||||
maxKeys: Sets the maximum number of keys returned in the response body. You can
|
||||
add this to your request if you want to retrieve fewer than the default 1000
|
||||
keys. The response might contain fewer keys but will never contain more. If
|
||||
there are additional keys that satisfy the search criteria but were not
|
||||
returned because maxKeys was exceeded, the response contains an attribute of
|
||||
IsTruncated set to true and a NextMarker. To return the additional keys, call
|
||||
the function again using NextMarker as your marker argument in the function.
|
||||
|
||||
Any key that does not contain the delimiter will be returned individually in
|
||||
Contents rather than in CommonPrefixes.
|
||||
|
||||
If there is an error, the error subfield is returned in the response.
|
|
@ -0,0 +1,51 @@
|
|||
function markerFilterMPU(allMarkers, array) {
|
||||
const { keyMarker, uploadIdMarker } = allMarkers;
|
||||
for (let i = 0; i < array.length; i++) {
|
||||
// If the keyMarker is the same as the key,
|
||||
// check the uploadIdMarker. If uploadIdMarker is the same
|
||||
// as or alphabetically after the uploadId of the item,
|
||||
// eliminate the item.
|
||||
if (uploadIdMarker && keyMarker === array[i].key) {
|
||||
const laterId =
|
||||
[uploadIdMarker, array[i].uploadId].sort()[1];
|
||||
if (array[i].uploadId === laterId) {
|
||||
break;
|
||||
} else {
|
||||
array.shift();
|
||||
i--;
|
||||
}
|
||||
} else {
|
||||
// If the keyMarker is alphabetically after the key
|
||||
// of the item in the array, eliminate the item from the array.
|
||||
const laterItem =
|
||||
[keyMarker, array[i].key].sort()[1];
|
||||
if (keyMarker === array[i].key || keyMarker === laterItem) {
|
||||
array.shift();
|
||||
i--;
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
return array;
|
||||
}
|
||||
|
||||
function prefixFilter(prefix, array) {
|
||||
for (let i = 0; i < array.length; i++) {
|
||||
if (array[i].indexOf(prefix) !== 0) {
|
||||
array.splice(i, 1);
|
||||
i--;
|
||||
}
|
||||
}
|
||||
return array;
|
||||
}
|
||||
|
||||
function isKeyInContents(responseObject, key) {
|
||||
return responseObject.Contents.some(val => val.key === key);
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
markerFilterMPU,
|
||||
prefixFilter,
|
||||
isKeyInContents,
|
||||
};
|
|
@ -0,0 +1,148 @@
|
|||
const { errors } = require('arsenal');
|
||||
|
||||
const { markerFilterMPU, prefixFilter } = require('./bucket_utilities');
|
||||
const { ListMultipartUploadsResult } = require('./ListMultipartUploadsResult');
|
||||
const { metadata } = require('./metadata');
|
||||
|
||||
const defaultMaxKeys = 1000;
|
||||
function getMultipartUploadListing(bucket, params, callback) {
|
||||
const { delimiter, keyMarker,
|
||||
uploadIdMarker, prefix, queryPrefixLength, splitter } = params;
|
||||
const splitterLen = splitter.length;
|
||||
const maxKeys = params.maxKeys !== undefined ?
|
||||
Number.parseInt(params.maxKeys, 10) : defaultMaxKeys;
|
||||
const response = new ListMultipartUploadsResult();
|
||||
const keyMap = metadata.keyMaps.get(bucket.getName());
|
||||
if (prefix) {
|
||||
response.Prefix = prefix;
|
||||
if (typeof prefix !== 'string') {
|
||||
return callback(errors.InvalidArgument);
|
||||
}
|
||||
}
|
||||
|
||||
if (keyMarker) {
|
||||
response.KeyMarker = keyMarker;
|
||||
if (typeof keyMarker !== 'string') {
|
||||
return callback(errors.InvalidArgument);
|
||||
}
|
||||
}
|
||||
|
||||
if (uploadIdMarker) {
|
||||
response.UploadIdMarker = uploadIdMarker;
|
||||
if (typeof uploadIdMarker !== 'string') {
|
||||
return callback(errors.InvalidArgument);
|
||||
}
|
||||
}
|
||||
|
||||
if (delimiter) {
|
||||
response.Delimiter = delimiter;
|
||||
if (typeof delimiter !== 'string') {
|
||||
return callback(errors.InvalidArgument);
|
||||
}
|
||||
}
|
||||
|
||||
if (maxKeys && typeof maxKeys !== 'number') {
|
||||
return callback(errors.InvalidArgument);
|
||||
}
|
||||
|
||||
// Sort uploads alphatebetically by objectKey and if same objectKey,
|
||||
// then sort in ascending order by time initiated
|
||||
let uploads = [];
|
||||
keyMap.forEach((val, key) => {
|
||||
uploads.push(key);
|
||||
});
|
||||
uploads.sort((a, b) => {
|
||||
const aIndex = a.indexOf(splitter);
|
||||
const bIndex = b.indexOf(splitter);
|
||||
const aObjectKey = a.substring(aIndex + splitterLen);
|
||||
const bObjectKey = b.substring(bIndex + splitterLen);
|
||||
const aInitiated = keyMap.get(a).initiated;
|
||||
const bInitiated = keyMap.get(b).initiated;
|
||||
if (aObjectKey === bObjectKey) {
|
||||
if (Date.parse(aInitiated) >= Date.parse(bInitiated)) {
|
||||
return 1;
|
||||
}
|
||||
if (Date.parse(aInitiated) < Date.parse(bInitiated)) {
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
return (aObjectKey < bObjectKey) ? -1 : 1;
|
||||
});
|
||||
// Edit the uploads array so it only
|
||||
// contains keys that contain the prefix
|
||||
uploads = prefixFilter(prefix, uploads);
|
||||
uploads = uploads.map(stringKey => {
|
||||
const index = stringKey.indexOf(splitter);
|
||||
const index2 = stringKey.indexOf(splitter, index + splitterLen);
|
||||
const storedMD = keyMap.get(stringKey);
|
||||
return {
|
||||
key: stringKey.substring(index + splitterLen, index2),
|
||||
uploadId: stringKey.substring(index2 + splitterLen),
|
||||
bucket: storedMD.eventualStorageBucket,
|
||||
initiatorID: storedMD.initiator.ID,
|
||||
initiatorDisplayName: storedMD.initiator.DisplayName,
|
||||
ownerID: storedMD['owner-id'],
|
||||
ownerDisplayName: storedMD['owner-display-name'],
|
||||
storageClass: storedMD['x-amz-storage-class'],
|
||||
initiated: storedMD.initiated,
|
||||
};
|
||||
});
|
||||
// If keyMarker specified, edit the uploads array so it
|
||||
// only contains keys that occur alphabetically after the marker.
|
||||
// If there is also an uploadIdMarker specified, filter to eliminate
|
||||
// any uploads that share the keyMarker and have an uploadId before
|
||||
// the uploadIdMarker.
|
||||
if (keyMarker) {
|
||||
const allMarkers = {
|
||||
keyMarker,
|
||||
uploadIdMarker,
|
||||
};
|
||||
uploads = markerFilterMPU(allMarkers, uploads);
|
||||
}
|
||||
|
||||
// Iterate through uploads and filter uploads
|
||||
// with keys containing delimiter
|
||||
// into response.CommonPrefixes and filter remaining uploads
|
||||
// into response.Uploads
|
||||
for (let i = 0; i < uploads.length; i++) {
|
||||
const currentUpload = uploads[i];
|
||||
// If hit maxKeys, stop adding keys to response
|
||||
if (response.MaxKeys >= maxKeys) {
|
||||
response.IsTruncated = true;
|
||||
break;
|
||||
}
|
||||
// If a delimiter is specified, find its
|
||||
// index in the current key AFTER THE OCCURRENCE OF THE PREFIX
|
||||
// THAT WAS SENT IN THE QUERY (not the prefix including the splitter
|
||||
// and other elements)
|
||||
let delimiterIndexAfterPrefix = -1;
|
||||
const currentKeyWithoutPrefix =
|
||||
currentUpload.key.slice(queryPrefixLength);
|
||||
let sliceEnd;
|
||||
if (delimiter) {
|
||||
delimiterIndexAfterPrefix = currentKeyWithoutPrefix
|
||||
.indexOf(delimiter);
|
||||
sliceEnd = delimiterIndexAfterPrefix + queryPrefixLength;
|
||||
}
|
||||
// If delimiter occurs in current key, add key to
|
||||
// response.CommonPrefixes.
|
||||
// Otherwise add upload to response.Uploads
|
||||
if (delimiterIndexAfterPrefix > -1) {
|
||||
const keySubstring = currentUpload.key.slice(0, sliceEnd + 1);
|
||||
response.addCommonPrefix(keySubstring);
|
||||
} else {
|
||||
response.NextKeyMarker = currentUpload.key;
|
||||
response.NextUploadIdMarker = currentUpload.uploadId;
|
||||
response.addUpload(currentUpload);
|
||||
}
|
||||
}
|
||||
// `response.MaxKeys` should be the value from the original `MaxUploads`
|
||||
// parameter specified by the user (or else the default 1000). Redefine it
|
||||
// here, so it does not equal the value of `uploads.length`.
|
||||
response.MaxKeys = maxKeys;
|
||||
// If `response.MaxKeys` is 0, `response.IsTruncated` should be `false`.
|
||||
response.IsTruncated = maxKeys === 0 ? false : response.IsTruncated;
|
||||
return callback(null, response);
|
||||
}
|
||||
|
||||
module.exports = getMultipartUploadListing;
|
|
@ -0,0 +1,8 @@
|
|||
const metadata = {
|
||||
buckets: new Map,
|
||||
keyMaps: new Map,
|
||||
};
|
||||
|
||||
module.exports = {
|
||||
metadata,
|
||||
};
|
|
@ -0,0 +1,274 @@
|
|||
const async = require('async');
|
||||
const { errors } = require('arsenal');
|
||||
|
||||
const metadata = require('./wrapper');
|
||||
const BucketInfo = require('arsenal').models.BucketInfo;
|
||||
const { isBucketAuthorized, isObjAuthorized } =
|
||||
require('../api/apiUtils/authorization/aclChecks');
|
||||
const bucketShield = require('../api/apiUtils/bucket/bucketShield');
|
||||
|
||||
/** _parseListEntries - parse the values returned in a listing by metadata
|
||||
* @param {object[]} entries - Version or Content entries in a metadata listing
|
||||
* @param {string} entries[].key - metadata key
|
||||
* @param {string} entries[].value - stringified object metadata
|
||||
* @return {object} - mapped array with parsed value or JSON parsing err
|
||||
*/
|
||||
function _parseListEntries(entries) {
|
||||
return entries.map(entry => {
|
||||
if (typeof entry.value === 'string') {
|
||||
const tmp = JSON.parse(entry.value);
|
||||
return {
|
||||
key: entry.key,
|
||||
value: {
|
||||
Size: tmp['content-length'],
|
||||
ETag: tmp['content-md5'],
|
||||
VersionId: tmp.versionId,
|
||||
IsNull: tmp.isNull,
|
||||
IsDeleteMarker: tmp.isDeleteMarker,
|
||||
LastModified: tmp['last-modified'],
|
||||
Owner: {
|
||||
DisplayName: tmp['owner-display-name'],
|
||||
ID: tmp['owner-id'],
|
||||
},
|
||||
StorageClass: tmp['x-amz-storage-class'],
|
||||
// MPU listing properties
|
||||
Initiated: tmp.initiated,
|
||||
Initiator: tmp.initiator,
|
||||
EventualStorageBucket: tmp.eventualStorageBucket,
|
||||
partLocations: tmp.partLocations,
|
||||
creationDate: tmp.creationDate,
|
||||
},
|
||||
};
|
||||
}
|
||||
return entry;
|
||||
});
|
||||
}
|
||||
|
||||
/** parseListEntries - parse the values returned in a listing by metadata
|
||||
* @param {object[]} entries - Version or Content entries in a metadata listing
|
||||
* @param {string} entries[].key - metadata key
|
||||
* @param {string} entries[].value - stringified object metadata
|
||||
* @return {(object|Error)} - mapped array with parsed value or JSON parsing err
|
||||
*/
|
||||
function parseListEntries(entries) {
|
||||
// wrap private function in a try/catch clause
|
||||
// just in case JSON parsing throws an exception
|
||||
try {
|
||||
return _parseListEntries(entries);
|
||||
} catch (e) {
|
||||
return e;
|
||||
}
|
||||
}
|
||||
|
||||
/** getNullVersion - return metadata of null version if it exists
|
||||
* @param {object} objMD - metadata of master version
|
||||
* @param {string} bucketName - name of bucket
|
||||
* @param {string} objectKey - name of object key
|
||||
* @param {RequestLogger} log - request logger
|
||||
* @param {function} cb - callback
|
||||
* @return {undefined} - and call callback with params err, objMD of null ver
|
||||
*/
|
||||
function getNullVersion(objMD, bucketName, objectKey, log, cb) {
|
||||
const options = {};
|
||||
if (objMD.isNull || !objMD.versionId) {
|
||||
// null version is current version
|
||||
log.debug('found null version');
|
||||
return process.nextTick(() => cb(null, objMD));
|
||||
}
|
||||
if (objMD.nullVersionId) {
|
||||
// the latest version is not the null version, but null version exists
|
||||
log.debug('null version exists, get the null version');
|
||||
options.versionId = objMD.nullVersionId;
|
||||
return metadata.getObjectMD(bucketName, objectKey, options, log, cb);
|
||||
}
|
||||
log.debug('could not find a null version');
|
||||
return process.nextTick(() => cb());
|
||||
}
|
||||
|
||||
/** metadataGetBucketAndObject - retrieves bucket and specified version
|
||||
* NOTE: If the value of `versionId` param is 'null', this function returns the
|
||||
* master version objMD. The null version object md must be retrieved in a
|
||||
* separate step using the master object md: see getNullVersion().
|
||||
* @param {string} requestType - type of request
|
||||
* @param {string} bucketName - name of bucket
|
||||
* @param {string} objectKey - name of object key
|
||||
* @param {string} [versionId] - version of object to retrieve
|
||||
* @param {RequestLogger} log - request logger
|
||||
* @param {function} cb - callback
|
||||
* @return {undefined} - and call callback with err, bucket md and object md
|
||||
*/
|
||||
function metadataGetBucketAndObject(requestType, bucketName, objectKey,
|
||||
versionId, log, cb) {
|
||||
const options = {
|
||||
// if attempting to get 'null' version, must retrieve null version id
|
||||
// from most current object md (versionId = undefined)
|
||||
versionId: versionId === 'null' ? undefined : versionId,
|
||||
};
|
||||
return metadata.getBucketAndObjectMD(bucketName, objectKey, options, log,
|
||||
(err, data) => {
|
||||
if (err) {
|
||||
log.debug('metadata get failed', { error: err });
|
||||
return cb(err);
|
||||
}
|
||||
const bucket = data.bucket ? BucketInfo.deSerialize(data.bucket) :
|
||||
undefined;
|
||||
const obj = data.obj ? JSON.parse(data.obj) : undefined;
|
||||
if (!bucket) {
|
||||
log.debug('bucketAttrs is undefined', {
|
||||
bucket: bucketName,
|
||||
method: 'metadataGetBucketAndObject',
|
||||
});
|
||||
return cb(errors.NoSuchBucket);
|
||||
}
|
||||
if (bucketShield(bucket, requestType)) {
|
||||
log.debug('bucket is shielded from request', {
|
||||
requestType,
|
||||
method: 'metadataGetBucketAndObject',
|
||||
});
|
||||
return cb(errors.NoSuchBucket);
|
||||
}
|
||||
log.trace('found bucket in metadata');
|
||||
return cb(null, bucket, obj);
|
||||
});
|
||||
}
|
||||
|
||||
/** metadataGetObject - retrieves specified object or version from metadata
|
||||
* @param {string} bucketName - name of bucket
|
||||
* @param {string} objectKey - name of object key
|
||||
* @param {string} [versionId] - version of object to retrieve
|
||||
* @param {RequestLogger} log - request logger
|
||||
* @param {function} cb - callback
|
||||
* @return {undefined} - and call callback with err, bucket md and object md
|
||||
*/
|
||||
function metadataGetObject(bucketName, objectKey, versionId, log, cb) {
|
||||
const options = {
|
||||
// if attempting to get 'null' version, must first retrieve null version
|
||||
// id from most current object md (by setting versionId as undefined
|
||||
// we retrieve the most current object md)
|
||||
versionId: versionId === 'null' ? undefined : versionId,
|
||||
};
|
||||
return metadata.getObjectMD(bucketName, objectKey, options, log,
|
||||
(err, objMD) => {
|
||||
if (err) {
|
||||
log.debug('err getting object MD from metadata',
|
||||
{ error: err });
|
||||
return cb(err);
|
||||
}
|
||||
if (versionId === 'null') {
|
||||
return getNullVersion(objMD, bucketName, objectKey, log, cb);
|
||||
}
|
||||
return cb(null, objMD);
|
||||
});
|
||||
}
|
||||
|
||||
/** metadataValidateBucketAndObj - retrieve bucket and object md from metadata
|
||||
* and check if user is authorized to access them.
|
||||
* @param {object} params - function parameters
|
||||
* @param {AuthInfo} params.authInfo - AuthInfo class instance, requester's info
|
||||
* @param {string} params.bucketName - name of bucket
|
||||
* @param {string} params.objectKey - name of object
|
||||
* @param {string} [params.versionId] - version id if getting specific version
|
||||
* @param {string} params.requestType - type of request
|
||||
* @param {RequestLogger} log - request logger
|
||||
* @param {function} callback - callback
|
||||
* @return {undefined} - and call callback with params err, bucket md
|
||||
*/
|
||||
function metadataValidateBucketAndObj(params, log, callback) {
|
||||
const { authInfo, bucketName, objectKey, versionId, requestType } = params;
|
||||
const canonicalID = authInfo.getCanonicalID();
|
||||
async.waterfall([
|
||||
function getBucketAndObjectMD(next) {
|
||||
return metadataGetBucketAndObject(requestType, bucketName,
|
||||
objectKey, versionId, log, next);
|
||||
},
|
||||
function checkBucketAuth(bucket, objMD, next) {
|
||||
if (!isBucketAuthorized(bucket, requestType, canonicalID)) {
|
||||
log.debug('access denied for user on bucket', { requestType });
|
||||
return next(errors.AccessDenied, bucket);
|
||||
}
|
||||
return next(null, bucket, objMD);
|
||||
},
|
||||
function handleNullVersionGet(bucket, objMD, next) {
|
||||
if (objMD && versionId === 'null') {
|
||||
return getNullVersion(objMD, bucketName, objectKey, log,
|
||||
(err, nullVer) => next(err, bucket, nullVer));
|
||||
}
|
||||
return next(null, bucket, objMD);
|
||||
},
|
||||
function checkObjectAuth(bucket, objMD, next) {
|
||||
if (!objMD) {
|
||||
return next(null, bucket);
|
||||
}
|
||||
if (!isObjAuthorized(bucket, objMD, requestType, canonicalID)) {
|
||||
log.debug('access denied for user on object', { requestType });
|
||||
return next(errors.AccessDenied, bucket);
|
||||
}
|
||||
return next(null, bucket, objMD);
|
||||
},
|
||||
], (err, bucket, objMD) => {
|
||||
if (err) {
|
||||
// still return bucket for cors headers
|
||||
return callback(err, bucket);
|
||||
}
|
||||
return callback(null, bucket, objMD);
|
||||
});
|
||||
}
|
||||
|
||||
/** metadataGetBucket - retrieves bucket from metadata, returning error if
|
||||
* bucket is shielded
|
||||
* @param {string} requestType - type of request
|
||||
* @param {string} bucketName - name of bucket
|
||||
* @param {RequestLogger} log - request logger
|
||||
* @param {function} cb - callback
|
||||
* @return {undefined} - and call callback with err, bucket md
|
||||
*/
|
||||
function metadataGetBucket(requestType, bucketName, log, cb) {
|
||||
return metadata.getBucket(bucketName, log, (err, bucket) => {
|
||||
if (err) {
|
||||
log.debug('metadata getbucket failed', { error: err });
|
||||
return cb(err);
|
||||
}
|
||||
if (bucketShield(bucket, requestType)) {
|
||||
log.debug('bucket is shielded from request', {
|
||||
requestType,
|
||||
method: 'metadataGetBucketAndObject',
|
||||
});
|
||||
return cb(errors.NoSuchBucket);
|
||||
}
|
||||
log.trace('found bucket in metadata');
|
||||
return cb(null, bucket);
|
||||
});
|
||||
}
|
||||
|
||||
/** metadataValidateBucket - retrieve bucket from metadata and check if user
|
||||
* is authorized to access it
|
||||
* @param {object} params - function parameters
|
||||
* @param {AuthInfo} params.authInfo - AuthInfo class instance, requester's info
|
||||
* @param {string} params.bucketName - name of bucket
|
||||
* @param {string} params.requestType - type of request
|
||||
* @param {RequestLogger} log - request logger
|
||||
* @param {function} callback - callback
|
||||
* @return {undefined} - and call callback with params err, bucket md
|
||||
*/
|
||||
function metadataValidateBucket(params, log, callback) {
|
||||
const { authInfo, bucketName, requestType } = params;
|
||||
const canonicalID = authInfo.getCanonicalID();
|
||||
return metadataGetBucket(requestType, bucketName, log, (err, bucket) => {
|
||||
if (err) {
|
||||
return callback(err);
|
||||
}
|
||||
// still return bucket for cors headers
|
||||
if (!isBucketAuthorized(bucket, requestType, canonicalID)) {
|
||||
log.debug('access denied for user on bucket', { requestType });
|
||||
return callback(errors.AccessDenied, bucket);
|
||||
}
|
||||
return callback(null, bucket);
|
||||
});
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
parseListEntries,
|
||||
metadataGetObject,
|
||||
metadataValidateBucketAndObj,
|
||||
metadataValidateBucket,
|
||||
};
|
|
@ -0,0 +1,173 @@
|
|||
# Mongoclient
|
||||
|
||||
We introduce a new metadata backend called *mongoclient* for
|
||||
[MongoDB](https://www.mongodb.com). This backend takes advantage of
|
||||
MongoDB being a document store to store the metadata (bucket and
|
||||
object attributes) as JSON objects.
|
||||
|
||||
## Overall Design
|
||||
|
||||
The mongoclient backend strictly follows the metadata interface that
|
||||
stores bucket and object attributes, which consists of the methods
|
||||
createBucket(), getBucketAttributes(), getBucketAndObject()
|
||||
(attributes), putBucketAttributes(), deleteBucket(), putObject(),
|
||||
getObject(), deleteObject(), listObject(), listMultipartUploads() and
|
||||
the management methods getUUID(), getDiskUsage() and countItems(). The
|
||||
mongoclient backend also knows how to deal with versioning, it is also
|
||||
compatible with the various listing algorithms implemented in Arsenal.
|
||||
|
||||
FIXME: There should be a document describing the metadata (currently
|
||||
duck-typing) interface.
|
||||
|
||||
### Why Using MongoDB for Storing Bucket and Object Attributes
|
||||
|
||||
We chose MongoDB for various reasons:
|
||||
|
||||
- MongoDB supports replication, especially through the Raft protocol.
|
||||
|
||||
- MongoDB supports a basic replication scheme called 'Replica Set' and
|
||||
more advanced sharding schemes if required.
|
||||
|
||||
- MongoDB is open source and an enterprise standard.
|
||||
|
||||
- MongoDB is a document store (natively supports JSON) and supports a
|
||||
very flexible search interface.
|
||||
|
||||
### Choice of Mongo Client Library
|
||||
|
||||
We chose to use the official MongoDB driver for NodeJS:
|
||||
[https://github.com/mongodb/node-mongodb-native](https://github.com/mongodb/node-mongodb-native)
|
||||
|
||||
### Granularity for Buckets
|
||||
|
||||
We chose to have one collection for one bucket mapping. First because
|
||||
in a simple mode of replication called 'replica set' it works from the
|
||||
get-go, but if one or many buckets grow to big it is possible to use
|
||||
more advanced schemes such as sharding. MongoDB supports a mix of
|
||||
sharded and non-sharded collections.
|
||||
|
||||
### Storing Database Information
|
||||
|
||||
We need a special collection called the *Infostore* (stored under the
|
||||
name __infostore which is impossible to create through the S3 bucket
|
||||
naming scheme) to store specific database properties such as the
|
||||
unique *uuid* for Orbit.
|
||||
|
||||
### Storing Bucket Attributes
|
||||
|
||||
We need to use a special collection called the *Metastore* (stored
|
||||
under the name __metastore which is impossible to create through the
|
||||
S3 bucket naming scheme).
|
||||
|
||||
### Versioning Format
|
||||
|
||||
We chose to keep the same versioning format that we use in some other
|
||||
Scality products in order to facilitate the compatibility between the
|
||||
different products.
|
||||
|
||||
FIXME: Document the versioning internals in the upper layers and
|
||||
document the versioning format
|
||||
|
||||
### Dealing with Concurrency
|
||||
|
||||
We chose not to use transactions (aka
|
||||
[https://docs.mongodb.com/manual/tutorial/perform-two-phase-commits/)
|
||||
because it is a known fact there is an overhead of using them, and we
|
||||
thought there was no real need for them since we could leverage Mongo
|
||||
ordered operations guarantees and atomic writes.
|
||||
|
||||
Example of corner cases:
|
||||
|
||||
#### CreateBucket()
|
||||
|
||||
Since it is not possible to create a collection AND at the same time
|
||||
register the bucket in the Metastore we chose to only update the
|
||||
Metastore. A non-existing collection (NamespaceNotFound error in
|
||||
Mongo) is one possible normal state for an empty bucket.
|
||||
|
||||
#### DeleteBucket()
|
||||
|
||||
In this case the bucket is *locked* by the upper layers (use of a
|
||||
transient delete flag) so we don't have to worry about that and by the
|
||||
fact the bucket is empty neither (which is also checked by the upper
|
||||
layers).
|
||||
|
||||
We first drop() the collection and then we asynchronously delete the
|
||||
bucket name entry from the metastore (the removal from the metastore
|
||||
is atomic which is not absolutely necessary in this case but more
|
||||
robust in term of design).
|
||||
|
||||
If we fail in between we still have an entry in the metastore which is
|
||||
good because we need to manage the delete flag. For the upper layers
|
||||
the operation has not completed until this flag is removed. The upper
|
||||
layers will restart the deleteBucket() which is fine because we manage
|
||||
the case where the collection does not exist.
|
||||
|
||||
#### PutObject() with a Version
|
||||
|
||||
We need to store the versioned object then update the master object
|
||||
(the latest version). For this we use the
|
||||
[BulkWrite](http://mongodb.github.io/node-mongodb-native/3.0/api/Collection.html#bulkWrite)
|
||||
method. This is not a transaction but guarantees that the 2 operations
|
||||
will happen sequentially in the MongoDB oplog. Indeed if the
|
||||
BulkWrite() fails in between we would end up creating an orphan (which
|
||||
is not critical) but if the operation succeeds then we are sure that
|
||||
the master is always pointing to the right object. If there is a
|
||||
concurrency between 2 clients then we are sure that the 2 groups of
|
||||
operations will be clearly decided in the oplog (the last writer will
|
||||
win).
|
||||
|
||||
#### DeleteObject()
|
||||
|
||||
This is probably the most complex case to manage because it involves a
|
||||
lot of different cases:
|
||||
|
||||
##### Deleting an Object when Versioning is not Enabled
|
||||
|
||||
This case is a straightforward atomic delete. Atomicity is not really
|
||||
required because we assume version IDs are random enough but it is
|
||||
more robust to do so.
|
||||
|
||||
##### Deleting an Object when Versioning is Enabled
|
||||
|
||||
This case is more complex since we have to deal with the 2 cases:
|
||||
|
||||
Case 1: The caller asks for a deletion of a version which is not a master:
|
||||
This case is a straight-forward atomic delete.
|
||||
|
||||
Case 2: The caller asks for a deletion of a version which is the master: In
|
||||
this case we need to create a special flag called PHD (as PlaceHolDer)
|
||||
that indicates the master is no longer valid (with a new unique
|
||||
virtual version ID). We force the ordering of operations in a
|
||||
bulkWrite() to first replace the master with the PHD flag and then
|
||||
physically delete the version. If the call fail in between we will be
|
||||
left with a master with a PHD flag. If the call succeeds we try to
|
||||
find if the master with the PHD flag is left alone in such case we
|
||||
delete it otherwise we trigger an asynchronous repair that will spawn
|
||||
after AYNC_REPAIR_TIMEOUT=15s that will reassign the master to the
|
||||
latest version.
|
||||
|
||||
In all cases the physical deletion or the repair of the master are
|
||||
checked against the PHD flag AND the actual unique virtual version
|
||||
ID. We do this to check against potential concurrent deletions,
|
||||
repairs or updates. Only the last writer/deleter has the right to
|
||||
physically perform the operation, otherwise it is superseded by other
|
||||
operations.
|
||||
|
||||
##### Getting an object with a PHD flag
|
||||
|
||||
If the caller is asking for the latest version of an object and the
|
||||
PHD flag is set we perform a search on the bucket to find the latest
|
||||
version and we return it.
|
||||
|
||||
#### Listing Objects
|
||||
|
||||
The mongoclient backend implements a readable key/value stream called
|
||||
*MongoReadStream* that follows the LevelDB duck typing interface used
|
||||
in Arsenal/lib/algos listing algorithms. Note it does not require any
|
||||
LevelDB package.
|
||||
|
||||
#### Generating the UUID
|
||||
|
||||
To avoid race conditions we always (try to) generate a new UUID and we
|
||||
condition the insertion to the non-existence of the document.
|
|
@ -0,0 +1,766 @@
|
|||
/*
|
||||
* we assume good default setting of write concern is good for all
|
||||
* bulk writes. Note that bulk writes are not transactions but ordered
|
||||
* writes. They may fail in between. To some extend those situations
|
||||
* may generate orphans but not alter the proper conduct of operations
|
||||
* (what he user wants and what we acknowledge to the user).
|
||||
*
|
||||
* Orphan situations may be recovered by the Lifecycle.
|
||||
*
|
||||
* We use proper atomic operations when needed.
|
||||
*/
|
||||
const async = require('async');
|
||||
const arsenal = require('arsenal');
|
||||
|
||||
const logger = require('../../utilities/logger');
|
||||
|
||||
const constants = require('../../../constants');
|
||||
const { config } = require('../../Config');
|
||||
|
||||
const errors = arsenal.errors;
|
||||
const versioning = arsenal.versioning;
|
||||
const BucketInfo = arsenal.models.BucketInfo;
|
||||
|
||||
const MongoClient = require('mongodb').MongoClient;
|
||||
const Uuid = require('uuid');
|
||||
const diskusage = require('diskusage');
|
||||
|
||||
const genVID = versioning.VersionID.generateVersionId;
|
||||
|
||||
const MongoReadStream = require('./readStream');
|
||||
const MongoUtils = require('./utils');
|
||||
|
||||
const USERSBUCKET = '__usersbucket';
|
||||
const METASTORE = '__metastore';
|
||||
const INFOSTORE = '__infostore';
|
||||
const __UUID = 'uuid';
|
||||
const ASYNC_REPAIR_TIMEOUT = 15000;
|
||||
|
||||
let uidCounter = 0;
|
||||
|
||||
const VID_SEP = versioning.VersioningConstants.VersionId.Separator;
|
||||
|
||||
function generateVersionId() {
|
||||
// generate a unique number for each member of the nodejs cluster
|
||||
return genVID(`${process.pid}.${uidCounter++}`,
|
||||
config.replicationGroupId);
|
||||
}
|
||||
|
||||
function formatVersionKey(key, versionId) {
|
||||
return `${key}${VID_SEP}${versionId}`;
|
||||
}
|
||||
|
||||
function inc(str) {
|
||||
return str ? (str.slice(0, str.length - 1) +
|
||||
String.fromCharCode(str.charCodeAt(str.length - 1) + 1)) : str;
|
||||
}
|
||||
|
||||
const VID_SEPPLUS = inc(VID_SEP);
|
||||
|
||||
function generatePHDVersion(versionId) {
|
||||
return {
|
||||
isPHD: true,
|
||||
versionId,
|
||||
};
|
||||
}
|
||||
|
||||
class MongoClientInterface {
|
||||
constructor() {
|
||||
const mongoUrl =
|
||||
`mongodb://${config.mongodb.host}:${config.mongodb.port}`;
|
||||
this.logger = logger;
|
||||
this.client = null;
|
||||
this.db = null;
|
||||
this.logger.debug(`connecting to ${mongoUrl}`);
|
||||
// FIXME: constructors shall not have side effect so there
|
||||
// should be an async_init(cb) method in the wrapper to
|
||||
// initialize this backend
|
||||
MongoClient.connect(mongoUrl, (err, client) => {
|
||||
if (err) {
|
||||
throw (errors.InternalError);
|
||||
}
|
||||
this.logger.debug('connected to mongodb');
|
||||
this.client = client;
|
||||
this.db = client.db(config.mongodb.database, {
|
||||
ignoreUndefined: true,
|
||||
});
|
||||
this.usersBucketHack();
|
||||
});
|
||||
}
|
||||
|
||||
usersBucketHack() {
|
||||
/* FIXME: Since the bucket creation API is expecting the
|
||||
usersBucket to have attributes, we pre-create the
|
||||
usersBucket attributes here (see bucketCreation.js line
|
||||
36)*/
|
||||
const usersBucketAttr = new BucketInfo(constants.usersBucket,
|
||||
'admin', 'admin', new Date().toJSON(),
|
||||
BucketInfo.currentModelVersion());
|
||||
this.createBucket(
|
||||
constants.usersBucket,
|
||||
usersBucketAttr, {}, err => {
|
||||
if (err) {
|
||||
this.logger.fatal('error writing usersBucket ' +
|
||||
'attributes to metastore',
|
||||
{ error: err });
|
||||
throw (errors.InternalError);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
getCollection(name) {
|
||||
/* mongo has a problem with .. in collection names */
|
||||
const newName = (name === constants.usersBucket) ?
|
||||
USERSBUCKET : name;
|
||||
return this.db.collection(newName);
|
||||
}
|
||||
|
||||
createBucket(bucketName, bucketMD, log, cb) {
|
||||
// FIXME: there should be a version of BucketInfo.serialize()
|
||||
// that does not JSON.stringify()
|
||||
const bucketInfo = BucketInfo.fromObj(bucketMD);
|
||||
const bucketMDStr = bucketInfo.serialize();
|
||||
const newBucketMD = JSON.parse(bucketMDStr);
|
||||
const m = this.getCollection(METASTORE);
|
||||
// we don't have to test bucket existence here as it is done
|
||||
// on the upper layers
|
||||
m.update({
|
||||
_id: bucketName,
|
||||
}, {
|
||||
_id: bucketName,
|
||||
value: newBucketMD,
|
||||
}, {
|
||||
upsert: true,
|
||||
}, () => cb());
|
||||
}
|
||||
|
||||
getBucketAttributes(bucketName, log, cb) {
|
||||
const m = this.getCollection(METASTORE);
|
||||
m.findOne({
|
||||
_id: bucketName,
|
||||
}, {}, (err, doc) => {
|
||||
if (err) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
if (!doc) {
|
||||
return cb(errors.NoSuchBucket);
|
||||
}
|
||||
// FIXME: there should be a version of BucketInfo.deserialize()
|
||||
// that properly inits w/o JSON.parse()
|
||||
const bucketMDStr = JSON.stringify(doc.value);
|
||||
const bucketMD = BucketInfo.deSerialize(bucketMDStr);
|
||||
return cb(null, bucketMD);
|
||||
});
|
||||
}
|
||||
|
||||
getBucketAndObject(bucketName, objName, params, log, cb) {
|
||||
this.getBucketAttributes(bucketName, log, (err, bucket) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
this.getObject(bucketName, objName, params, log, (err, obj) => {
|
||||
if (err) {
|
||||
if (err === errors.NoSuchKey) {
|
||||
return cb(null,
|
||||
{ bucket:
|
||||
BucketInfo.fromObj(bucket).serialize(),
|
||||
});
|
||||
}
|
||||
return cb(err);
|
||||
}
|
||||
return cb(null, {
|
||||
bucket: BucketInfo.fromObj(bucket).serialize(),
|
||||
obj: JSON.stringify(obj),
|
||||
});
|
||||
});
|
||||
return undefined;
|
||||
});
|
||||
}
|
||||
|
||||
putBucketAttributes(bucketName, bucketMD, log, cb) {
|
||||
// FIXME: there should be a version of BucketInfo.serialize()
|
||||
// that does not JSON.stringify()
|
||||
const bucketInfo = BucketInfo.fromObj(bucketMD);
|
||||
const bucketMDStr = bucketInfo.serialize();
|
||||
const newBucketMD = JSON.parse(bucketMDStr);
|
||||
const m = this.getCollection(METASTORE);
|
||||
m.update({
|
||||
_id: bucketName,
|
||||
}, {
|
||||
_id: bucketName,
|
||||
value: newBucketMD,
|
||||
}, {
|
||||
upsert: true,
|
||||
}, () => cb());
|
||||
}
|
||||
|
||||
/*
|
||||
* Delete bucket from metastore
|
||||
*/
|
||||
deleteBucketStep2(bucketName, log, cb) {
|
||||
const m = this.getCollection(METASTORE);
|
||||
m.findOneAndDelete({
|
||||
_id: bucketName,
|
||||
}, {}, (err, result) => {
|
||||
if (err) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
if (result.ok !== 1) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return cb(null);
|
||||
});
|
||||
}
|
||||
|
||||
/*
|
||||
* Drop the bucket then process to step 2. Checking
|
||||
* the count is already done by the upper layer. We don't need to be
|
||||
* atomic because the call is protected by a delete_pending flag
|
||||
* in the upper layer.
|
||||
* 2 cases here:
|
||||
* 1) the collection may not yet exist (or being already dropped
|
||||
* by a previous call)
|
||||
* 2) the collection may exist.
|
||||
*/
|
||||
deleteBucket(bucketName, log, cb) {
|
||||
const c = this.getCollection(bucketName);
|
||||
c.drop({}, err => {
|
||||
if (err) {
|
||||
if (err.codeName === 'NamespaceNotFound') {
|
||||
return this.deleteBucketStep2(bucketName, log, cb);
|
||||
}
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return this.deleteBucketStep2(bucketName, log, cb);
|
||||
});
|
||||
}
|
||||
|
||||
/*
|
||||
* In this case we generate a versionId and
|
||||
* sequentially create the object THEN update the master
|
||||
*/
|
||||
putObjectVerCase1(c, bucketName, objName, objVal, params, log, cb) {
|
||||
const versionId = generateVersionId();
|
||||
// eslint-disable-next-line
|
||||
objVal.versionId = versionId;
|
||||
const vObjName = formatVersionKey(objName, versionId);
|
||||
c.bulkWrite([{
|
||||
updateOne: {
|
||||
filter: {
|
||||
_id: vObjName,
|
||||
},
|
||||
update: {
|
||||
_id: vObjName, value: objVal,
|
||||
},
|
||||
upsert: true,
|
||||
},
|
||||
}, {
|
||||
updateOne: {
|
||||
filter: {
|
||||
_id: objName,
|
||||
},
|
||||
update: {
|
||||
_id: objName, value: objVal,
|
||||
},
|
||||
upsert: true,
|
||||
},
|
||||
}], {
|
||||
ordered: 1,
|
||||
}, () => cb(null, `{"versionId": "${versionId}"}`));
|
||||
}
|
||||
|
||||
/*
|
||||
* Case used when versioning has been disabled after objects
|
||||
* have been created with versions
|
||||
*/
|
||||
putObjectVerCase2(c, bucketName, objName, objVal, params, log, cb) {
|
||||
const versionId = generateVersionId();
|
||||
// eslint-disable-next-line
|
||||
objVal.versionId = versionId;
|
||||
c.update({
|
||||
_id: objName,
|
||||
}, {
|
||||
_id: objName,
|
||||
value: objVal,
|
||||
}, {
|
||||
upsert: true,
|
||||
}, () => cb(null, `{"versionId": "${objVal.versionId}"}`));
|
||||
}
|
||||
|
||||
/*
|
||||
* In this case the aller provides a versionId. This function will
|
||||
* sequentially update the object with given versionId THEN the
|
||||
* master iff the provided versionId matches the one of the master
|
||||
*/
|
||||
putObjectVerCase3(c, bucketName, objName, objVal, params, log, cb) {
|
||||
// eslint-disable-next-line
|
||||
objVal.versionId = params.versionId;
|
||||
const vObjName = formatVersionKey(objName, params.versionId);
|
||||
c.bulkWrite([{
|
||||
updateOne: {
|
||||
filter: {
|
||||
_id: vObjName,
|
||||
},
|
||||
update: {
|
||||
_id: vObjName, value: objVal,
|
||||
},
|
||||
upsert: true,
|
||||
},
|
||||
}, {
|
||||
updateOne: {
|
||||
// eslint-disable-next-line
|
||||
filter: {
|
||||
_id: objName,
|
||||
'value.versionId': params.versionId,
|
||||
},
|
||||
update: {
|
||||
_id: objName, value: objVal,
|
||||
},
|
||||
upsert: true,
|
||||
},
|
||||
}], {
|
||||
ordered: 1,
|
||||
}, () => cb(null, `{"versionId": "${objVal.versionId}"}`));
|
||||
}
|
||||
|
||||
/*
|
||||
* Put object when versioning is not enabled
|
||||
*/
|
||||
putObjectNoVer(c, bucketName, objName, objVal, params, log, cb) {
|
||||
c.update({
|
||||
_id: objName,
|
||||
}, {
|
||||
_id: objName,
|
||||
value: objVal,
|
||||
}, {
|
||||
upsert: true,
|
||||
}, () => cb());
|
||||
}
|
||||
|
||||
putObject(bucketName, objName, objVal, params, log, cb) {
|
||||
MongoUtils.serialize(objVal);
|
||||
const c = this.getCollection(bucketName);
|
||||
if (params && params.versioning && !params.versionId) {
|
||||
return this.putObjectVerCase1(c, bucketName, objName, objVal,
|
||||
params, log, cb);
|
||||
} else if (params && params.versionId === '') {
|
||||
return this.putObjectVerCase2(c, bucketName, objName, objVal,
|
||||
params, log, cb);
|
||||
} else if (params && params.versionId) {
|
||||
return this.putObjectVerCase3(c, bucketName, objName, objVal,
|
||||
params, log, cb);
|
||||
}
|
||||
return this.putObjectNoVer(c, bucketName, objName, objVal,
|
||||
params, log, cb);
|
||||
}
|
||||
|
||||
getObject(bucketName, objName, params, log, cb) {
|
||||
const c = this.getCollection(bucketName);
|
||||
if (params && params.versionId) {
|
||||
// eslint-disable-next-line
|
||||
objName = formatVersionKey(objName, params.versionId);
|
||||
}
|
||||
c.findOne({
|
||||
_id: objName,
|
||||
}, {}, (err, doc) => {
|
||||
if (err) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
if (!doc) {
|
||||
return cb(errors.NoSuchKey);
|
||||
}
|
||||
if (doc.value.isPHD) {
|
||||
this.getLatestVersion(c, objName, log, (err, value) => {
|
||||
if (err) {
|
||||
log.error('getting latest version', err);
|
||||
return cb(err);
|
||||
}
|
||||
return cb(null, value);
|
||||
});
|
||||
return undefined;
|
||||
}
|
||||
MongoUtils.unserialize(doc.value);
|
||||
return cb(null, doc.value);
|
||||
});
|
||||
}
|
||||
|
||||
/*
|
||||
* This function return the latest version
|
||||
*/
|
||||
getLatestVersion(c, objName, log, cb) {
|
||||
c.find({
|
||||
_id: {
|
||||
$gt: objName,
|
||||
$lt: `${objName}${VID_SEPPLUS}`,
|
||||
},
|
||||
}, {}).
|
||||
sort({
|
||||
_id: 1,
|
||||
}).
|
||||
limit(1).
|
||||
toArray(
|
||||
(err, keys) => {
|
||||
if (err) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
if (keys.length === 0) {
|
||||
return cb(errors.NoSuchKey);
|
||||
}
|
||||
MongoUtils.unserialize(keys[0].value);
|
||||
return cb(null, keys[0].value);
|
||||
});
|
||||
}
|
||||
|
||||
/*
|
||||
* repair the master with a new value. There can be
|
||||
* race-conditions or legit updates so place an atomic condition
|
||||
* on PHD flag and mst version.
|
||||
*/
|
||||
repair(c, objName, objVal, mst, log, cb) {
|
||||
MongoUtils.serialize(objVal);
|
||||
// eslint-disable-next-line
|
||||
c.findOneAndReplace({
|
||||
_id: objName,
|
||||
'value.isPHD': true,
|
||||
'value.versionId': mst.versionId,
|
||||
}, {
|
||||
_id: objName,
|
||||
value: objVal,
|
||||
}, {
|
||||
upsert: true,
|
||||
}, (err, result) => {
|
||||
if (err) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
if (result.ok !== 1) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return cb(null);
|
||||
});
|
||||
}
|
||||
|
||||
/*
|
||||
* Get the latest version and repair. The process is safe because
|
||||
* we never replace a non-PHD master
|
||||
*/
|
||||
asyncRepair(c, objName, mst, log) {
|
||||
this.getLatestVersion(c, objName, log, (err, value) => {
|
||||
if (err) {
|
||||
log.error('async-repair: getting latest version', err);
|
||||
return undefined;
|
||||
}
|
||||
this.repair(c, objName, value, mst, log, err => {
|
||||
if (err) {
|
||||
log.error('async-repair failed', err);
|
||||
return undefined;
|
||||
}
|
||||
log.debug('async-repair success');
|
||||
return undefined;
|
||||
});
|
||||
return undefined;
|
||||
});
|
||||
}
|
||||
|
||||
/*
|
||||
* the master is a PHD so we try to see if it is the latest of its
|
||||
* kind to get rid of it, otherwise we asynchronously repair it
|
||||
*/
|
||||
deleteOrRepairPHD(c, bucketName, objName, mst, log, cb) {
|
||||
// Check if there are other versions available
|
||||
this.getLatestVersion(c, objName, log, err => {
|
||||
if (err) {
|
||||
if (err === errors.NoSuchKey) {
|
||||
// We try to delete the master. A race condition
|
||||
// is possible here: another process may recreate
|
||||
// a master or re-delete it in between so place an
|
||||
// atomic condition on the PHD flag and the mst
|
||||
// version:
|
||||
// eslint-disable-next-line
|
||||
c.findOneAndDelete({
|
||||
_id: objName,
|
||||
'value.isPHD': true,
|
||||
'value.versionId': mst.versionId,
|
||||
}, {}, err => {
|
||||
if (err) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
// do not test result.ok === 1 because
|
||||
// both cases are expected
|
||||
return cb(null);
|
||||
});
|
||||
return undefined;
|
||||
}
|
||||
return cb(err);
|
||||
}
|
||||
// We have other versions available so repair:
|
||||
setTimeout(() => {
|
||||
this.asyncRepair(c, objName, mst, log);
|
||||
}, ASYNC_REPAIR_TIMEOUT);
|
||||
return cb(null);
|
||||
});
|
||||
}
|
||||
|
||||
/*
|
||||
* Delete object when versioning is enabled and the version is
|
||||
* master. In this case we sequentially update the master with a
|
||||
* PHD flag (placeholder) and a unique non-existing version THEN
|
||||
* we delete the specified versioned object. THEN we try to delete
|
||||
* or repair the PHD we just created
|
||||
*/
|
||||
deleteObjectVerMaster(c, bucketName, objName, params, log, cb) {
|
||||
const vObjName = formatVersionKey(objName, params.versionId);
|
||||
const _vid = generateVersionId();
|
||||
const mst = generatePHDVersion(_vid);
|
||||
c.bulkWrite([{
|
||||
updateOne: {
|
||||
filter: {
|
||||
_id: objName,
|
||||
},
|
||||
update: {
|
||||
_id: objName, value: mst,
|
||||
},
|
||||
upsert: true,
|
||||
},
|
||||
}, {
|
||||
deleteOne: {
|
||||
filter: {
|
||||
_id: vObjName,
|
||||
},
|
||||
},
|
||||
}], {
|
||||
ordered: 1,
|
||||
}, () => this.deleteOrRepairPHD(c, bucketName, objName, mst, log, cb));
|
||||
}
|
||||
|
||||
/*
|
||||
* Delete object when versioning is enabled and the version is
|
||||
* not master. It is a straight-forward atomic delete
|
||||
*/
|
||||
deleteObjectVerNotMaster(c, bucketName, objName, params, log, cb) {
|
||||
const vObjName = formatVersionKey(objName, params.versionId);
|
||||
c.findOneAndDelete({
|
||||
_id: vObjName,
|
||||
}, {}, (err, result) => {
|
||||
if (err) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
if (result.ok !== 1) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return cb(null);
|
||||
});
|
||||
}
|
||||
|
||||
/*
|
||||
* Delete object when versioning is enabled. We first find the
|
||||
* master, if it is already a PHD we have a special processing,
|
||||
* then we check if it matches the master versionId in such case
|
||||
* we will create a PHD, otherwise we delete it
|
||||
*/
|
||||
deleteObjectVer(c, bucketName, objName, params, log, cb) {
|
||||
c.findOne({
|
||||
_id: objName,
|
||||
}, {}, (err, mst) => {
|
||||
if (err) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
if (!mst) {
|
||||
return cb(errors.NoSuchKey);
|
||||
}
|
||||
if (mst.value.isPHD ||
|
||||
mst.value.versionId === params.versionId) {
|
||||
return this.deleteObjectVerMaster(c, bucketName, objName,
|
||||
params, log, cb);
|
||||
}
|
||||
return this.deleteObjectVerNotMaster(c, bucketName, objName,
|
||||
params, log, cb);
|
||||
});
|
||||
}
|
||||
|
||||
/*
|
||||
* Atomically delete an object when versioning is not enabled
|
||||
*/
|
||||
deleteObjectNoVer(c, bucketName, objName, params, log, cb) {
|
||||
c.findOneAndDelete({
|
||||
_id: objName,
|
||||
}, {}, (err, result) => {
|
||||
if (err) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
if (result.ok !== 1) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return cb(null);
|
||||
});
|
||||
}
|
||||
|
||||
deleteObject(bucketName, objName, params, log, cb) {
|
||||
const c = this.getCollection(bucketName);
|
||||
if (params && params.versionId) {
|
||||
return this.deleteObjectVer(c, bucketName, objName,
|
||||
params, log, cb);
|
||||
}
|
||||
return this.deleteObjectNoVer(c, bucketName, objName,
|
||||
params, log, cb);
|
||||
}
|
||||
|
||||
internalListObject(bucketName, params, log, cb) {
|
||||
const extName = params.listingType;
|
||||
const extension = new arsenal.algorithms.list[extName](params, log);
|
||||
const requestParams = extension.genMDParams();
|
||||
const c = this.getCollection(bucketName);
|
||||
let cbDone = false;
|
||||
const stream = new MongoReadStream(c, requestParams);
|
||||
stream
|
||||
.on('data', e => {
|
||||
if (extension.filter(e) < 0) {
|
||||
stream.emit('end');
|
||||
stream.destroy();
|
||||
}
|
||||
})
|
||||
.on('error', err => {
|
||||
if (!cbDone) {
|
||||
cbDone = true;
|
||||
const logObj = {
|
||||
rawError: err,
|
||||
error: err.message,
|
||||
errorStack: err.stack,
|
||||
};
|
||||
log.error('error listing objects', logObj);
|
||||
cb(errors.InternalError);
|
||||
}
|
||||
})
|
||||
.on('end', () => {
|
||||
if (!cbDone) {
|
||||
cbDone = true;
|
||||
const data = extension.result();
|
||||
cb(null, data);
|
||||
}
|
||||
});
|
||||
return undefined;
|
||||
}
|
||||
|
||||
listObject(bucketName, params, log, cb) {
|
||||
return this.internalListObject(bucketName, params, log, cb);
|
||||
}
|
||||
|
||||
listMultipartUploads(bucketName, params, log, cb) {
|
||||
return this.internalListObject(bucketName, params, log, cb);
|
||||
}
|
||||
|
||||
readUUID(log, cb) {
|
||||
const i = this.getCollection(INFOSTORE);
|
||||
i.findOne({
|
||||
_id: __UUID,
|
||||
}, {}, (err, doc) => {
|
||||
if (err) {
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
if (!doc) {
|
||||
return cb(errors.NoSuchKey);
|
||||
}
|
||||
return cb(null, doc.value);
|
||||
});
|
||||
}
|
||||
|
||||
writeUUIDIfNotExists(uuid, log, cb) {
|
||||
const i = this.getCollection(INFOSTORE);
|
||||
i.insert({
|
||||
_id: __UUID,
|
||||
value: uuid,
|
||||
}, {}, err => {
|
||||
if (err) {
|
||||
if (err.code === 11000) {
|
||||
// duplicate key error
|
||||
// FIXME: define a KeyAlreadyExists error in Arsenal
|
||||
return cb(errors.EntityAlreadyExists);
|
||||
}
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
// FIXME: shoud we check for result.ok === 1 ?
|
||||
return cb(null);
|
||||
});
|
||||
}
|
||||
|
||||
/*
|
||||
* we always try to generate a new UUID in order to be atomic in
|
||||
* case of concurrency. The write will fail if it already exists.
|
||||
*/
|
||||
getUUID(log, cb) {
|
||||
const uuid = Uuid.v4();
|
||||
this.writeUUIDIfNotExists(uuid, log, err => {
|
||||
if (err) {
|
||||
if (err === errors.InternalError) {
|
||||
return cb(err);
|
||||
}
|
||||
return this.readUUID(log, cb);
|
||||
}
|
||||
return cb(null, uuid);
|
||||
});
|
||||
}
|
||||
|
||||
getDiskUsage(cb) {
|
||||
// FIXME: for basic one server deployment the infrastructure
|
||||
// configurator shall set a path to the actual MongoDB volume.
|
||||
// For Kub/cluster deployments there should be a more sophisticated
|
||||
// way for guessing free space.
|
||||
diskusage.check(config.mongodb.path !== undefined ?
|
||||
config.mongodb.path : '/', cb);
|
||||
}
|
||||
|
||||
countItems(log, cb) {
|
||||
const res = {
|
||||
objects: 0,
|
||||
versions: 0,
|
||||
buckets: 0,
|
||||
};
|
||||
this.db.listCollections().toArray((err, collInfos) => {
|
||||
async.eachLimit(collInfos, 10, (value, next) => {
|
||||
if (value.name === METASTORE ||
|
||||
value.name === INFOSTORE ||
|
||||
value.name === USERSBUCKET) {
|
||||
// skip
|
||||
return next();
|
||||
}
|
||||
res.buckets++;
|
||||
const c = this.getCollection(value.name);
|
||||
// FIXME: there is currently no way of distinguishing
|
||||
// master from versions and searching for VID_SEP
|
||||
// does not work because there cannot be null bytes
|
||||
// in $regex
|
||||
c.count({
|
||||
// eslint-disable-next-line
|
||||
'value.versionId': {
|
||||
'$exists': false,
|
||||
},
|
||||
}, {}, (err, result) => {
|
||||
if (err) {
|
||||
return next(errors.InternalError);
|
||||
}
|
||||
res.objects += result;
|
||||
c.count({
|
||||
// eslint-disable-next-line
|
||||
'value.versionId': {
|
||||
'$exists': true,
|
||||
},
|
||||
}, {}, (err, result) => {
|
||||
if (err) {
|
||||
return next(errors.InternalError);
|
||||
}
|
||||
res.versions += result;
|
||||
return next();
|
||||
});
|
||||
return undefined;
|
||||
});
|
||||
return undefined;
|
||||
}, err => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
}
|
||||
return cb(null, res);
|
||||
});
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = MongoClientInterface;
|
|
@ -0,0 +1,133 @@
|
|||
const Readable = require('stream').Readable;
|
||||
const MongoUtils = require('./utils');
|
||||
|
||||
class MongoReadStream extends Readable {
|
||||
constructor(c, options) {
|
||||
super({
|
||||
objectMode: true,
|
||||
highWaterMark: 0,
|
||||
});
|
||||
|
||||
if (options.limit === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
const query = {
|
||||
_id: {},
|
||||
};
|
||||
if (options.reverse) {
|
||||
if (options.start) {
|
||||
query._id.$lte = options.start;
|
||||
}
|
||||
if (options.end) {
|
||||
query._id.$gte = options.end;
|
||||
}
|
||||
if (options.gt) {
|
||||
query._id.$lt = options.gt;
|
||||
}
|
||||
if (options.gte) {
|
||||
query._id.$lte = options.gte;
|
||||
}
|
||||
if (options.lt) {
|
||||
query._id.$gt = options.lt;
|
||||
}
|
||||
if (options.lte) {
|
||||
query._id.$gte = options.lte;
|
||||
}
|
||||
} else {
|
||||
if (options.start) {
|
||||
query._id.$gte = options.start;
|
||||
}
|
||||
if (options.end) {
|
||||
query._id.$lte = options.end;
|
||||
}
|
||||
if (options.gt) {
|
||||
query._id.$gt = options.gt;
|
||||
}
|
||||
if (options.gte) {
|
||||
query._id.$gte = options.gte;
|
||||
}
|
||||
if (options.lt) {
|
||||
query._id.$lt = options.lt;
|
||||
}
|
||||
if (options.lte) {
|
||||
query._id.$lte = options.lte;
|
||||
}
|
||||
}
|
||||
|
||||
if (!Object.keys(query._id).length) {
|
||||
delete query._id;
|
||||
}
|
||||
|
||||
this._cursor = c.find(query).sort({
|
||||
_id: options.reverse ? -1 : 1,
|
||||
});
|
||||
if (options.limit && options.limit !== -1) {
|
||||
this._cursor = this._cursor.limit(options.limit);
|
||||
}
|
||||
this._options = options;
|
||||
this._destroyed = false;
|
||||
this.on('end', this._cleanup.bind(this));
|
||||
}
|
||||
|
||||
_read() {
|
||||
if (this._destroyed) {
|
||||
return;
|
||||
}
|
||||
|
||||
this._cursor.next((err, doc) => {
|
||||
if (this._destroyed) {
|
||||
return;
|
||||
}
|
||||
if (err) {
|
||||
this.emit('error', err);
|
||||
return;
|
||||
}
|
||||
|
||||
let key = undefined;
|
||||
let value = undefined;
|
||||
|
||||
if (doc) {
|
||||
key = doc._id;
|
||||
MongoUtils.unserialize(doc.value);
|
||||
value = JSON.stringify(doc.value);
|
||||
}
|
||||
|
||||
if (key === undefined && value === undefined) {
|
||||
this.push(null);
|
||||
} else if (this._options.keys !== false &&
|
||||
this._options.values === false) {
|
||||
this.push(key);
|
||||
} else if (this._options.keys === false &&
|
||||
this._options.values !== false) {
|
||||
this.push(value);
|
||||
} else {
|
||||
this.push({
|
||||
key,
|
||||
value,
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
_cleanup() {
|
||||
if (this._destroyed) {
|
||||
return;
|
||||
}
|
||||
this._destroyed = true;
|
||||
|
||||
this._cursor.close(err => {
|
||||
if (err) {
|
||||
this.emit('error', err);
|
||||
return;
|
||||
}
|
||||
this.emit('close');
|
||||
});
|
||||
}
|
||||
|
||||
destroy() {
|
||||
return this._cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = MongoReadStream;
|
|
@ -0,0 +1,30 @@
|
|||
|
||||
function escape(obj) {
|
||||
return JSON.parse(JSON.stringify(obj).
|
||||
replace(/\$/g, '\uFF04').
|
||||
replace(/\./g, '\uFF0E'));
|
||||
}
|
||||
|
||||
function unescape(obj) {
|
||||
return JSON.parse(JSON.stringify(obj).
|
||||
replace(/\uFF04/g, '$').
|
||||
replace(/\uFF0E/g, '.'));
|
||||
}
|
||||
|
||||
function serialize(objMD) {
|
||||
// Tags require special handling since dot and dollar are accepted
|
||||
if (objMD.tags) {
|
||||
// eslint-disable-next-line
|
||||
objMD.tags = escape(objMD.tags);
|
||||
}
|
||||
}
|
||||
|
||||
function unserialize(objMD) {
|
||||
// Tags require special handling
|
||||
if (objMD.tags) {
|
||||
// eslint-disable-next-line
|
||||
objMD.tags = unescape(objMD.tags);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { escape, unescape, serialize, unserialize };
|
|
@ -0,0 +1,256 @@
|
|||
const errors = require('arsenal').errors;
|
||||
|
||||
const BucketClientInterface = require('./bucketclient/backend');
|
||||
const BucketFileInterface = require('./bucketfile/backend');
|
||||
const MongoClientInterface = require('./mongoclient/backend');
|
||||
const BucketInfo = require('arsenal').models.BucketInfo;
|
||||
const inMemory = require('./in_memory/backend');
|
||||
const { config } = require('../Config');
|
||||
|
||||
let CdmiMetadata;
|
||||
try {
|
||||
CdmiMetadata = require('cdmiclient').CdmiMetadata;
|
||||
} catch (err) {
|
||||
CdmiMetadata = null;
|
||||
}
|
||||
|
||||
let client;
|
||||
let implName;
|
||||
|
||||
if (config.backends.metadata === 'mem') {
|
||||
client = inMemory;
|
||||
implName = 'memorybucket';
|
||||
} else if (config.backends.metadata === 'file') {
|
||||
client = new BucketFileInterface();
|
||||
implName = 'bucketfile';
|
||||
} else if (config.backends.metadata === 'scality') {
|
||||
client = new BucketClientInterface();
|
||||
implName = 'bucketclient';
|
||||
} else if (config.backends.metadata === 'mongodb') {
|
||||
client = new MongoClientInterface();
|
||||
implName = 'mongoclient';
|
||||
} else if (config.backends.metadata === 'cdmi') {
|
||||
if (!CdmiMetadata) {
|
||||
throw new Error('Unauthorized backend');
|
||||
}
|
||||
|
||||
client = new CdmiMetadata({
|
||||
path: config.cdmi.path,
|
||||
host: config.cdmi.host,
|
||||
port: config.cdmi.port,
|
||||
readonly: config.cdmi.readonly,
|
||||
});
|
||||
implName = 'cdmi';
|
||||
}
|
||||
|
||||
const metadata = {
|
||||
createBucket: (bucketName, bucketMD, log, cb) => {
|
||||
log.debug('creating bucket in metadata');
|
||||
client.createBucket(bucketName, bucketMD, log, err => {
|
||||
if (err) {
|
||||
log.debug('error from metadata', { implName, error: err });
|
||||
return cb(err);
|
||||
}
|
||||
log.trace('bucket created in metadata');
|
||||
return cb(err);
|
||||
});
|
||||
},
|
||||
|
||||
updateBucket: (bucketName, bucketMD, log, cb) => {
|
||||
log.debug('updating bucket in metadata');
|
||||
client.putBucketAttributes(bucketName, bucketMD, log, err => {
|
||||
if (err) {
|
||||
log.debug('error from metadata', { implName, error: err });
|
||||
return cb(err);
|
||||
}
|
||||
log.trace('bucket updated in metadata');
|
||||
return cb(err);
|
||||
});
|
||||
},
|
||||
|
||||
getBucket: (bucketName, log, cb) => {
|
||||
log.debug('getting bucket from metadata');
|
||||
client.getBucketAttributes(bucketName, log, (err, data) => {
|
||||
if (err) {
|
||||
log.debug('error from metadata', { implName, error: err });
|
||||
return cb(err);
|
||||
}
|
||||
log.trace('bucket retrieved from metadata');
|
||||
return cb(err, BucketInfo.fromObj(data));
|
||||
});
|
||||
},
|
||||
|
||||
deleteBucket: (bucketName, log, cb) => {
|
||||
log.debug('deleting bucket from metadata');
|
||||
client.deleteBucket(bucketName, log, err => {
|
||||
if (err) {
|
||||
log.debug('error from metadata', { implName, error: err });
|
||||
return cb(err);
|
||||
}
|
||||
log.debug('Deleted bucket from Metadata');
|
||||
return cb(err);
|
||||
});
|
||||
},
|
||||
|
||||
putObjectMD: (bucketName, objName, objVal, params, log, cb) => {
|
||||
log.debug('putting object in metadata');
|
||||
const value = typeof objVal.getValue === 'function' ?
|
||||
objVal.getValue() : objVal;
|
||||
client.putObject(bucketName, objName, value, params, log,
|
||||
(err, data) => {
|
||||
if (err) {
|
||||
log.debug('error from metadata', { implName, error: err });
|
||||
return cb(err);
|
||||
}
|
||||
if (data) {
|
||||
log.debug('object version successfully put in metadata',
|
||||
{ version: data });
|
||||
} else {
|
||||
log.debug('object successfully put in metadata');
|
||||
}
|
||||
return cb(err, data);
|
||||
});
|
||||
},
|
||||
|
||||
getBucketAndObjectMD: (bucketName, objName, params, log, cb) => {
|
||||
log.debug('getting bucket and object from metadata',
|
||||
{ database: bucketName, object: objName });
|
||||
client.getBucketAndObject(bucketName, objName, params, log,
|
||||
(err, data) => {
|
||||
if (err) {
|
||||
log.debug('error from metadata', { implName, err });
|
||||
return cb(err);
|
||||
}
|
||||
log.debug('bucket and object retrieved from metadata',
|
||||
{ database: bucketName, object: objName });
|
||||
return cb(err, data);
|
||||
});
|
||||
},
|
||||
|
||||
getObjectMD: (bucketName, objName, params, log, cb) => {
|
||||
log.debug('getting object from metadata');
|
||||
client.getObject(bucketName, objName, params, log, (err, data) => {
|
||||
if (err) {
|
||||
log.debug('error from metadata', { implName, err });
|
||||
return cb(err);
|
||||
}
|
||||
log.debug('object retrieved from metadata');
|
||||
return cb(err, data);
|
||||
});
|
||||
},
|
||||
|
||||
deleteObjectMD: (bucketName, objName, params, log, cb) => {
|
||||
log.debug('deleting object from metadata');
|
||||
client.deleteObject(bucketName, objName, params, log, err => {
|
||||
if (err) {
|
||||
log.debug('error from metadata', { implName, err });
|
||||
return cb(err);
|
||||
}
|
||||
log.debug('object deleted from metadata');
|
||||
return cb(err);
|
||||
});
|
||||
},
|
||||
|
||||
listObject: (bucketName, listingParams, log, cb) => {
|
||||
const metadataUtils = require('./metadataUtils');
|
||||
if (listingParams.listingType === undefined) {
|
||||
// eslint-disable-next-line
|
||||
listingParams.listingType = 'Delimiter';
|
||||
}
|
||||
client.listObject(bucketName, listingParams, log, (err, data) => {
|
||||
log.debug('getting object listing from metadata');
|
||||
if (err) {
|
||||
log.debug('error from metadata', { implName, err });
|
||||
return cb(err);
|
||||
}
|
||||
log.debug('object listing retrieved from metadata');
|
||||
if (listingParams.listingType === 'DelimiterVersions') {
|
||||
// eslint-disable-next-line
|
||||
data.Versions = metadataUtils.parseListEntries(data.Versions);
|
||||
if (data.Versions instanceof Error) {
|
||||
log.error('error parsing metadata listing', {
|
||||
error: data.Versions,
|
||||
listingType: listingParams.listingType,
|
||||
method: 'listObject',
|
||||
});
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return cb(null, data);
|
||||
}
|
||||
// eslint-disable-next-line
|
||||
data.Contents = metadataUtils.parseListEntries(data.Contents);
|
||||
if (data.Contents instanceof Error) {
|
||||
log.error('error parsing metadata listing', {
|
||||
error: data.Contents,
|
||||
listingType: listingParams.listingType,
|
||||
method: 'listObject',
|
||||
});
|
||||
return cb(errors.InternalError);
|
||||
}
|
||||
return cb(null, data);
|
||||
});
|
||||
},
|
||||
|
||||
listMultipartUploads: (bucketName, listingParams, log, cb) => {
|
||||
client.listMultipartUploads(bucketName, listingParams, log,
|
||||
(err, data) => {
|
||||
log.debug('getting mpu listing from metadata');
|
||||
if (err) {
|
||||
log.debug('error from metadata', { implName, err });
|
||||
return cb(err);
|
||||
}
|
||||
log.debug('mpu listing retrieved from metadata');
|
||||
return cb(err, data);
|
||||
});
|
||||
},
|
||||
|
||||
switch: newClient => {
|
||||
client = newClient;
|
||||
},
|
||||
|
||||
getRaftBuckets: (raftId, log, cb) => {
|
||||
if (!client.getRaftBuckets) {
|
||||
return cb();
|
||||
}
|
||||
return client.getRaftBuckets(raftId, log, cb);
|
||||
},
|
||||
|
||||
checkHealth: (log, cb) => {
|
||||
if (!client.checkHealth) {
|
||||
const defResp = {};
|
||||
defResp[implName] = { code: 200, message: 'OK' };
|
||||
return cb(null, defResp);
|
||||
}
|
||||
return client.checkHealth(implName, log, cb);
|
||||
},
|
||||
|
||||
getUUID: (log, cb) => {
|
||||
if (!client.getUUID) {
|
||||
log.debug('returning empty uuid as fallback', { implName });
|
||||
return cb(null, '');
|
||||
}
|
||||
return client.getUUID(log, cb);
|
||||
},
|
||||
|
||||
getDiskUsage: (log, cb) => {
|
||||
if (!client.getDiskUsage) {
|
||||
log.debug('returning empty disk usage as fallback', { implName });
|
||||
return cb(null, {});
|
||||
}
|
||||
return client.getDiskUsage(cb);
|
||||
},
|
||||
|
||||
countItems: (log, cb) => {
|
||||
if (!client.countItems) {
|
||||
log.debug('returning zero item counts as fallback', { implName });
|
||||
return cb(null, {
|
||||
buckets: 0,
|
||||
objects: 0,
|
||||
versions: 0,
|
||||
});
|
||||
}
|
||||
return client.countItems(log, cb);
|
||||
},
|
||||
};
|
||||
|
||||
module.exports = metadata;
|
|
@ -1,4 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
'use strict'; // eslint-disable-line strict
|
||||
|
||||
require('../lib/nfs/utilities.js').createBucketWithNFSEnabled();
|
|
@ -1,108 +0,0 @@
|
|||
#!/bin/sh
|
||||
// 2>/dev/null ; exec "$(which nodejs 2>/dev/null || which node)" "$0" "$@"
|
||||
'use strict'; // eslint-disable-line strict
|
||||
|
||||
const { auth } = require('arsenal');
|
||||
const commander = require('commander');
|
||||
|
||||
const http = require('http');
|
||||
const https = require('https');
|
||||
const logger = require('../lib/utilities/logger');
|
||||
|
||||
function _performSearch(host,
|
||||
port,
|
||||
bucketName,
|
||||
query,
|
||||
listVersions,
|
||||
accessKey,
|
||||
secretKey,
|
||||
sessionToken,
|
||||
verbose, ssl) {
|
||||
const escapedSearch = encodeURIComponent(query);
|
||||
const options = {
|
||||
host,
|
||||
port,
|
||||
method: 'GET',
|
||||
path: `/${bucketName}/?search=${escapedSearch}${listVersions ? '&&versions' : ''}`,
|
||||
headers: {
|
||||
'Content-Length': 0,
|
||||
},
|
||||
rejectUnauthorized: false,
|
||||
versions: '',
|
||||
};
|
||||
if (sessionToken) {
|
||||
options.headers['x-amz-security-token'] = sessionToken;
|
||||
}
|
||||
const transport = ssl ? https : http;
|
||||
const request = transport.request(options, response => {
|
||||
if (verbose) {
|
||||
logger.info('response status code', {
|
||||
statusCode: response.statusCode,
|
||||
});
|
||||
logger.info('response headers', { headers: response.headers });
|
||||
}
|
||||
const body = [];
|
||||
response.setEncoding('utf8');
|
||||
response.on('data', chunk => body.push(chunk));
|
||||
response.on('end', () => {
|
||||
if (response.statusCode >= 200 && response.statusCode < 300) {
|
||||
logger.info('Success');
|
||||
process.stdout.write(body.join(''));
|
||||
process.exit(0);
|
||||
} else {
|
||||
logger.error('request failed with HTTP Status ', {
|
||||
statusCode: response.statusCode,
|
||||
body: body.join(''),
|
||||
});
|
||||
process.exit(1);
|
||||
}
|
||||
});
|
||||
});
|
||||
// generateV4Headers exepects request object with path that does not
|
||||
// include query
|
||||
request.path = `/${bucketName}`;
|
||||
const requestData = listVersions ? { search: query, versions: '' } : { search: query };
|
||||
auth.client.generateV4Headers(request, requestData, accessKey, secretKey, 's3');
|
||||
request.path = `/${bucketName}?search=${escapedSearch}${listVersions ? '&&versions' : ''}`;
|
||||
if (verbose) {
|
||||
logger.info('request headers', { headers: request._headers });
|
||||
}
|
||||
request.end();
|
||||
}
|
||||
|
||||
/**
|
||||
* This function is used as a binary to send a request to S3 to perform a
|
||||
* search on the objects in a bucket
|
||||
*
|
||||
* @return {undefined}
|
||||
*/
|
||||
function searchBucket() {
|
||||
// TODO: Include other bucket listing possible query params?
|
||||
commander
|
||||
.version('0.0.1')
|
||||
.option('-a, --access-key <accessKey>', 'Access key id')
|
||||
.option('-k, --secret-key <secretKey>', 'Secret access key')
|
||||
.option('-t, --session-token <sessionToken>', 'Session token')
|
||||
.option('-b, --bucket <bucket>', 'Name of the bucket')
|
||||
.option('-q, --query <query>', 'Search query')
|
||||
.option('-h, --host <host>', 'Host of the server')
|
||||
.option('-p, --port <port>', 'Port of the server')
|
||||
.option('-s', '--ssl', 'Enable ssl')
|
||||
.option('-l, --list-versions', 'List all versions of the objects that meet the search query, ' +
|
||||
'otherwise only list the latest version')
|
||||
.option('-v, --verbose')
|
||||
.parse(process.argv);
|
||||
const { host, port, accessKey, secretKey, sessionToken, bucket, query, listVersions, verbose, ssl } =
|
||||
commander;
|
||||
|
||||
if (!host || !port || !accessKey || !secretKey || !bucket || !query) {
|
||||
logger.error('missing parameter');
|
||||
commander.outputHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
_performSearch(host, port, bucket, query, listVersions, accessKey, secretKey, sessionToken, verbose,
|
||||
ssl);
|
||||
}
|
||||
|
||||
searchBucket();
|
|
@ -0,0 +1,45 @@
|
|||
---
|
||||
general:
|
||||
branches:
|
||||
ignore:
|
||||
- /^ultron\/.*/ # Ignore ultron/* branches
|
||||
artifacts:
|
||||
- coverage/
|
||||
|
||||
machine:
|
||||
node:
|
||||
version: 6.13.1
|
||||
services:
|
||||
- redis
|
||||
- docker
|
||||
ruby:
|
||||
version: "2.4.1"
|
||||
environment:
|
||||
CXX: g++-4.9
|
||||
ENABLE_LOCAL_CACHE: true
|
||||
REPORT_TOKEN: report-token-1
|
||||
hosts:
|
||||
bucketwebsitetester.s3-website-us-east-1.amazonaws.com: 127.0.0.1
|
||||
|
||||
dependencies:
|
||||
override:
|
||||
- rm -rf node_modules
|
||||
- npm install
|
||||
post:
|
||||
- sudo pip install flake8 yamllint
|
||||
- sudo pip install s3cmd==1.6.1
|
||||
# fog and ruby testing dependencies
|
||||
- gem install fog-aws -v 1.3.0
|
||||
- gem install mime-types -v 3.1
|
||||
- gem install rspec -v 3.5
|
||||
- gem install json
|
||||
# java sdk dependencies
|
||||
- sudo apt-get install -y -q default-jdk
|
||||
|
||||
|
||||
test:
|
||||
override:
|
||||
- docker run --name squid-proxy -d --net=host
|
||||
--publish 3128:3128 sameersbn/squid:3.3.8-23
|
||||
- bash tests.bash:
|
||||
parallel: true
|
|
@ -1,143 +0,0 @@
|
|||
{
|
||||
"port": 8000,
|
||||
"listenOn": [],
|
||||
"metricsPort": 8002,
|
||||
"metricsListenOn": [],
|
||||
"replicationGroupId": "RG001",
|
||||
"workers": 4,
|
||||
"restEndpoints": {
|
||||
"localhost": "us-east-1",
|
||||
"127.0.0.1": "us-east-1",
|
||||
"cloudserver-front": "us-east-1",
|
||||
"s3.docker.test": "us-east-1",
|
||||
"127.0.0.2": "us-east-1",
|
||||
"s3.amazonaws.com": "us-east-1",
|
||||
"zenko-cloudserver-replicator": "us-east-1",
|
||||
"lb": "us-east-1"
|
||||
},
|
||||
"websiteEndpoints": ["s3-website-us-east-1.amazonaws.com",
|
||||
"s3-website.us-east-2.amazonaws.com",
|
||||
"s3-website-us-west-1.amazonaws.com",
|
||||
"s3-website-us-west-2.amazonaws.com",
|
||||
"s3-website.ap-south-1.amazonaws.com",
|
||||
"s3-website.ap-northeast-2.amazonaws.com",
|
||||
"s3-website-ap-southeast-1.amazonaws.com",
|
||||
"s3-website-ap-southeast-2.amazonaws.com",
|
||||
"s3-website-ap-northeast-1.amazonaws.com",
|
||||
"s3-website.eu-central-1.amazonaws.com",
|
||||
"s3-website-eu-west-1.amazonaws.com",
|
||||
"s3-website-sa-east-1.amazonaws.com",
|
||||
"s3-website.localhost",
|
||||
"s3-website.scality.test",
|
||||
"zenkoazuretest.blob.core.windows.net"],
|
||||
"replicationEndpoints": [{
|
||||
"site": "zenko",
|
||||
"servers": ["127.0.0.1:8000"],
|
||||
"default": true
|
||||
}, {
|
||||
"site": "us-east-2",
|
||||
"type": "aws_s3"
|
||||
}],
|
||||
"backbeat": {
|
||||
"host": "localhost",
|
||||
"port": 8900
|
||||
},
|
||||
"workflowEngineOperator": {
|
||||
"host": "localhost",
|
||||
"port": 3001
|
||||
},
|
||||
"cdmi": {
|
||||
"host": "localhost",
|
||||
"port": 81,
|
||||
"path": "/dewpoint",
|
||||
"readonly": true
|
||||
},
|
||||
"bucketd": {
|
||||
"bootstrap": ["localhost:9000"]
|
||||
},
|
||||
"vaultd": {
|
||||
"host": "localhost",
|
||||
"port": 8500
|
||||
},
|
||||
"clusters": 1,
|
||||
"log": {
|
||||
"logLevel": "info",
|
||||
"dumpLevel": "error"
|
||||
},
|
||||
"healthChecks": {
|
||||
"allowFrom": ["127.0.0.1/8", "::1"]
|
||||
},
|
||||
"metadataClient": {
|
||||
"host": "localhost",
|
||||
"port": 9990
|
||||
},
|
||||
"dataClient": {
|
||||
"host": "localhost",
|
||||
"port": 9991
|
||||
},
|
||||
"pfsClient": {
|
||||
"host": "localhost",
|
||||
"port": 9992
|
||||
},
|
||||
"metadataDaemon": {
|
||||
"bindAddress": "localhost",
|
||||
"port": 9990
|
||||
},
|
||||
"dataDaemon": {
|
||||
"bindAddress": "localhost",
|
||||
"port": 9991
|
||||
},
|
||||
"pfsDaemon": {
|
||||
"bindAddress": "localhost",
|
||||
"port": 9992
|
||||
},
|
||||
"recordLog": {
|
||||
"enabled": true,
|
||||
"recordLogName": "s3-recordlog"
|
||||
},
|
||||
"mongodb": {
|
||||
"replicaSetHosts": "localhost:27018,localhost:27019,localhost:27020",
|
||||
"writeConcern": "majority",
|
||||
"replicaSet": "rs0",
|
||||
"readPreference": "primary",
|
||||
"database": "metadata"
|
||||
},
|
||||
"authdata": "authdata.json",
|
||||
"backends": {
|
||||
"auth": "file",
|
||||
"data": "file",
|
||||
"metadata": "mongodb",
|
||||
"kms": "file",
|
||||
"quota": "none"
|
||||
},
|
||||
"externalBackends": {
|
||||
"aws_s3": {
|
||||
"httpAgent": {
|
||||
"keepAlive": false,
|
||||
"keepAliveMsecs": 1000,
|
||||
"maxFreeSockets": 256,
|
||||
"maxSockets": null
|
||||
}
|
||||
},
|
||||
"gcp": {
|
||||
"httpAgent": {
|
||||
"keepAlive": true,
|
||||
"keepAliveMsecs": 1000,
|
||||
"maxFreeSockets": 256,
|
||||
"maxSockets": null
|
||||
}
|
||||
}
|
||||
},
|
||||
"requests": {
|
||||
"viaProxy": false,
|
||||
"trustedProxyCIDRs": [],
|
||||
"extractClientIPFromHeader": ""
|
||||
},
|
||||
"bucketNotificationDestinations": [
|
||||
{
|
||||
"resource": "target1",
|
||||
"type": "dummy",
|
||||
"host": "localhost:6000"
|
||||
}
|
||||
]
|
||||
}
|
|
@ -1,71 +0,0 @@
|
|||
{
|
||||
"port": 8000,
|
||||
"listenOn": [],
|
||||
"metricsPort": 8002,
|
||||
"metricsListenOn": [],
|
||||
"replicationGroupId": "RG001",
|
||||
"restEndpoints": {
|
||||
"localhost": "STANDARD",
|
||||
"127.0.0.1": "STANDARD",
|
||||
"yourhostname.ru": "STANDARD"
|
||||
},
|
||||
"websiteEndpoints": [
|
||||
"static.yourhostname.ru"
|
||||
],
|
||||
"replicationEndpoints": [ {
|
||||
"site": "zenko",
|
||||
"servers": ["127.0.0.1:8000"],
|
||||
"default": true
|
||||
} ],
|
||||
"log": {
|
||||
"logLevel": "info",
|
||||
"dumpLevel": "error"
|
||||
},
|
||||
"healthChecks": {
|
||||
"allowFrom": ["127.0.0.1/8", "::1"]
|
||||
},
|
||||
"backends": {
|
||||
"metadata": "mongodb"
|
||||
},
|
||||
"mongodb": {
|
||||
"replicaSetHosts": "127.0.0.1:27017",
|
||||
"writeConcern": "majority",
|
||||
"replicaSet": "rs0",
|
||||
"readPreference": "primary",
|
||||
"database": "s3",
|
||||
"authCredentials": {
|
||||
"username": "s3",
|
||||
"password": ""
|
||||
}
|
||||
},
|
||||
"externalBackends": {
|
||||
"aws_s3": {
|
||||
"httpAgent": {
|
||||
"keepAlive": false,
|
||||
"keepAliveMsecs": 1000,
|
||||
"maxFreeSockets": 256,
|
||||
"maxSockets": null
|
||||
}
|
||||
},
|
||||
"gcp": {
|
||||
"httpAgent": {
|
||||
"keepAlive": true,
|
||||
"keepAliveMsecs": 1000,
|
||||
"maxFreeSockets": 256,
|
||||
"maxSockets": null
|
||||
}
|
||||
}
|
||||
},
|
||||
"requests": {
|
||||
"viaProxy": false,
|
||||
"trustedProxyCIDRs": [],
|
||||
"extractClientIPFromHeader": ""
|
||||
},
|
||||
"bucketNotificationDestinations": [
|
||||
{
|
||||
"resource": "target1",
|
||||
"type": "dummy",
|
||||
"host": "localhost:6000"
|
||||
}
|
||||
]
|
||||
}
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"accounts": [{
|
||||
"name": "Bart",
|
||||
"email": "sampleaccount1@sampling.com",
|
||||
"arn": "arn:aws:iam::123456789012:root",
|
||||
"canonicalID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be",
|
||||
"shortid": "123456789012",
|
||||
"keys": [{
|
||||
"access": "accessKey1",
|
||||
"secret": "verySecretKey1"
|
||||
}]
|
||||
}, {
|
||||
"name": "Lisa",
|
||||
"email": "sampleaccount2@sampling.com",
|
||||
"arn": "arn:aws:iam::123456789013:root",
|
||||
"canonicalID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2bf",
|
||||
"shortid": "123456789013",
|
||||
"keys": [{
|
||||
"access": "accessKey2",
|
||||
"secret": "verySecretKey2"
|
||||
}]
|
||||
}]
|
||||
}
|
|
@ -23,8 +23,7 @@
|
|||
"s3-website-eu-west-1.amazonaws.com",
|
||||
"s3-website-sa-east-1.amazonaws.com",
|
||||
"s3-website.localhost",
|
||||
"s3-website.scality.test",
|
||||
"zenkoazuretest.blob.core.windows.net"],
|
||||
"s3-website.scality.test"],
|
||||
"replicationEndpoints": [{
|
||||
"site": "zenko",
|
||||
"servers": ["127.0.0.1:8000"],
|
||||
|
@ -40,7 +39,7 @@
|
|||
"readonly": true
|
||||
},
|
||||
"bucketd": {
|
||||
"bootstrap": ["localhost"]
|
||||
"bootstrap": ["localhost:9000"]
|
||||
},
|
||||
"vaultd": {
|
||||
"host": "localhost",
|
||||
|
@ -75,52 +74,8 @@
|
|||
"recordLogName": "s3-recordlog"
|
||||
},
|
||||
"mongodb": {
|
||||
"replicaSetHosts": "localhost:27017,localhost:27018,localhost:27019",
|
||||
"writeConcern": "majority",
|
||||
"replicaSet": "rs0",
|
||||
"readPreference": "primary",
|
||||
"database": "metadata"
|
||||
},
|
||||
"certFilePaths": {
|
||||
"key": "tests/unit/testConfigs/allOptsConfig/key.txt",
|
||||
"cert": "tests/unit/testConfigs/allOptsConfig/cert.txt",
|
||||
"ca": "tests/unit/testConfigs/allOptsConfig/caBundle.txt"
|
||||
},
|
||||
"outboundProxy": {
|
||||
"url": "http://test:8001",
|
||||
"caBundle": "tests/unit/testConfigs/allOptsConfig/caBundle.txt",
|
||||
"key": "tests/unit/testConfigs/allOptsConfig/key.txt",
|
||||
"cert": "tests/unit/testConfigs/allOptsConfig/cert.txt"
|
||||
},
|
||||
"localCache": {
|
||||
"name": "zenko",
|
||||
"sentinels": "localhost:6379"
|
||||
},
|
||||
"redis": {
|
||||
"name": "zenko",
|
||||
"sentinels": "localhost:6379"
|
||||
},
|
||||
"scuba": {
|
||||
"host": "localhost",
|
||||
"port": 8100
|
||||
},
|
||||
"quota": {
|
||||
"maxStalenessMS": 86400000
|
||||
},
|
||||
"utapi": {
|
||||
"redis": {
|
||||
"host": "localhost",
|
||||
"port": 6379,
|
||||
"retry": {
|
||||
"connectBackoff": {
|
||||
"min": 10,
|
||||
"max": 1000,
|
||||
"factor": 1.5,
|
||||
"jitter": 0.1,
|
||||
"deadline": 10000
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"overlayVersion": 4
|
||||
"host": "localhost",
|
||||
"port": 27018,
|
||||
"database": "metadata"
|
||||
}
|
||||
}
|
|
@ -1,112 +1,82 @@
|
|||
{
|
||||
"us-east-1": {
|
||||
"type": "file",
|
||||
"objectId": "us-east-1",
|
||||
"legacyAwsBehavior": true,
|
||||
"details": {}
|
||||
},
|
||||
"us-east-2": {
|
||||
"type": "file",
|
||||
"objectId": "us-east-2",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"us-west-1": {
|
||||
"type": "file",
|
||||
"objectId": "us-west-1",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"us-west-2": {
|
||||
"type": "file",
|
||||
"objectId": "us-west-2",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"ca-central-1": {
|
||||
"type": "file",
|
||||
"objectId": "ca-central-1",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"cn-north-1": {
|
||||
"type": "file",
|
||||
"objectId": "cn-north-1",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"ap-south-1": {
|
||||
"type": "file",
|
||||
"objectId": "ap-south-1",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"ap-northeast-1": {
|
||||
"type": "file",
|
||||
"objectId": "ap-northeast-1",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"ap-northeast-2": {
|
||||
"type": "file",
|
||||
"objectId": "ap-northeast-2",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"ap-southeast-1": {
|
||||
"type": "file",
|
||||
"objectId": "ap-southeast-1",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"ap-southeast-2": {
|
||||
"type": "file",
|
||||
"objectId": "ap-southeast-2",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"eu-central-1": {
|
||||
"type": "file",
|
||||
"objectId": "eu-central-1",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"eu-west-1": {
|
||||
"type": "file",
|
||||
"objectId": "eu-west-1",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"eu-west-2": {
|
||||
"type": "file",
|
||||
"objectId": "eu-west-2",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"EU": {
|
||||
"type": "file",
|
||||
"objectId": "EU",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"sa-east-1": {
|
||||
"type": "file",
|
||||
"objectId": "sa-east-1",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"location-dmf-v1": {
|
||||
"type": "dmf",
|
||||
"objectId": "location-dmf-v1",
|
||||
"legacyAwsBehavior": false,
|
||||
"isCold": true,
|
||||
"details": {}
|
||||
},
|
||||
"location-azure-archive-v1": {
|
||||
"type": "azure_archive",
|
||||
"objectId": "location-azure-archive-v1",
|
||||
"legacyAwsBehavior": false,
|
||||
"isCold": true,
|
||||
"details": {}
|
||||
}
|
||||
}
|
138
constants.js
138
constants.js
|
@ -39,8 +39,6 @@ const constants = {
|
|||
// once the multipart upload is complete.
|
||||
mpuBucketPrefix: 'mpuShadowBucket',
|
||||
blacklistedPrefixes: { bucket: [], object: [] },
|
||||
// GCP Object Tagging Prefix
|
||||
gcpTaggingPrefix: 'aws-tag-',
|
||||
// PublicId is used as the canonicalID for a request that contains
|
||||
// no authentication information. Requestor can access
|
||||
// only public resources
|
||||
|
@ -66,71 +64,48 @@ const constants = {
|
|||
// http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html
|
||||
minimumAllowedPartSize: 5242880,
|
||||
|
||||
// AWS sets a maximum total parts limit
|
||||
// https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html
|
||||
maximumAllowedPartCount: 10000,
|
||||
|
||||
gcpMaximumAllowedPartCount: 1024,
|
||||
|
||||
// Max size on put part or copy part is 5GB. For functional
|
||||
// testing use 110 MB as max
|
||||
maximumAllowedPartSize: process.env.MPU_TESTING === 'yes' ? 110100480 :
|
||||
5368709120,
|
||||
|
||||
// Max size allowed in a single put object request is 5GB
|
||||
// https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html
|
||||
maximumAllowedUploadSize: 5368709120,
|
||||
|
||||
// AWS states max size for user-defined metadata (x-amz-meta- headers) is
|
||||
// 2 KB: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html
|
||||
// In testing, AWS seems to allow up to 88 more bytes, so we do the same.
|
||||
maximumMetaHeadersSize: 2136,
|
||||
|
||||
// Maximum HTTP headers size allowed
|
||||
maxHttpHeadersSize: 14122,
|
||||
|
||||
// hex digest of sha256 hash of empty string:
|
||||
emptyStringHash: crypto.createHash('sha256')
|
||||
.update('', 'binary').digest('hex'),
|
||||
|
||||
// Queries supported by AWS that we do not currently support.
|
||||
// Non-bucket queries
|
||||
unsupportedQueries: [
|
||||
'accelerate',
|
||||
'analytics',
|
||||
'inventory',
|
||||
'logging',
|
||||
'metrics',
|
||||
'policyStatus',
|
||||
'publicAccessBlock',
|
||||
'notification',
|
||||
'policy',
|
||||
'requestPayment',
|
||||
'restore',
|
||||
'torrent',
|
||||
],
|
||||
|
||||
// Headers supported by AWS that we do not currently support.
|
||||
unsupportedHeaders: [
|
||||
'x-amz-server-side-encryption',
|
||||
'x-amz-server-side-encryption-customer-algorithm',
|
||||
'x-amz-server-side-encryption-aws-kms-key-id',
|
||||
'x-amz-server-side-encryption-context',
|
||||
'x-amz-server-side-encryption-customer-key',
|
||||
'x-amz-server-side-encryption-customer-key-md5',
|
||||
],
|
||||
|
||||
// user metadata header to set object locationConstraint
|
||||
objectLocationConstraintHeader: 'x-amz-storage-class',
|
||||
lastModifiedHeader: 'x-amz-meta-x-scal-last-modified',
|
||||
objectLocationConstraintHeader: 'x-amz-meta-scal-location-constraint',
|
||||
legacyLocations: ['sproxyd', 'legacy'],
|
||||
// declare here all existing service accounts and their properties
|
||||
// (if any, otherwise an empty object)
|
||||
serviceAccountProperties: {
|
||||
replication: {},
|
||||
lifecycle: {},
|
||||
gc: {},
|
||||
'md-ingestion': {
|
||||
canReplicate: true,
|
||||
},
|
||||
},
|
||||
/* eslint-disable camelcase */
|
||||
externalBackends: { aws_s3: true, azure: true, gcp: true, pfs: true, dmf: true, azure_archive: true },
|
||||
externalBackends: { aws_s3: true, azure: true, gcp: true },
|
||||
// some of the available data backends (if called directly rather
|
||||
// than through the multiple backend gateway) need a key provided
|
||||
// as a string as first parameter of the get/delete methods.
|
||||
|
@ -139,110 +114,15 @@ const constants = {
|
|||
// for external backends, don't call unless at least 1 minute
|
||||
// (60,000 milliseconds) since last call
|
||||
externalBackendHealthCheckInterval: 60000,
|
||||
versioningNotImplBackends: { azure: true, gcp: true },
|
||||
mpuMDStoredExternallyBackend: { aws_s3: true, gcp: true },
|
||||
skipBatchDeleteBackends: { azure: true, gcp: true },
|
||||
s3HandledBackends: { azure: true, gcp: true },
|
||||
hasCopyPartBackends: { aws_s3: true, gcp: true },
|
||||
versioningNotImplBackends: { azure: true },
|
||||
mpuMDStoredExternallyBackend: { aws_s3: true },
|
||||
/* eslint-enable camelcase */
|
||||
mpuMDStoredOnS3Backend: { azure: true },
|
||||
azureAccountNameRegex: /^[a-z0-9]{3,24}$/,
|
||||
base64Regex: new RegExp('^(?:[A-Za-z0-9+/]{4})*' +
|
||||
'(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$'),
|
||||
productName: 'APN/1.0 Scality/1.0 Scality CloudServer for Zenko',
|
||||
// location constraint delimiter
|
||||
zenkoSeparator: ':',
|
||||
// user metadata applied on zenko objects
|
||||
zenkoIDHeader: 'x-amz-meta-zenko-instance-id',
|
||||
bucketOwnerActions: [
|
||||
'bucketDeleteCors',
|
||||
'bucketDeleteLifecycle',
|
||||
'bucketDeletePolicy',
|
||||
'bucketDeleteReplication',
|
||||
'bucketDeleteWebsite',
|
||||
'bucketGetCors',
|
||||
'bucketGetLifecycle',
|
||||
'bucketGetLocation',
|
||||
'bucketGetPolicy',
|
||||
'bucketGetReplication',
|
||||
'bucketGetVersioning',
|
||||
'bucketGetWebsite',
|
||||
'bucketPutCors',
|
||||
'bucketPutLifecycle',
|
||||
'bucketPutPolicy',
|
||||
'bucketPutReplication',
|
||||
'bucketPutVersioning',
|
||||
'bucketPutWebsite',
|
||||
'objectDeleteTagging',
|
||||
'objectGetTagging',
|
||||
'objectPutTagging',
|
||||
'objectPutLegalHold',
|
||||
'objectPutRetention',
|
||||
],
|
||||
// response header to be sent when there are invalid
|
||||
// user metadata in the object's metadata
|
||||
invalidObjectUserMetadataHeader: 'x-amz-missing-meta',
|
||||
// Bucket specific queries supported by AWS that we do not currently support
|
||||
// these queries may or may not be supported at object level
|
||||
unsupportedBucketQueries: [
|
||||
],
|
||||
suppressedUtapiEventFields: [
|
||||
'object',
|
||||
'location',
|
||||
'versionId',
|
||||
],
|
||||
allowedUtapiEventFilterFields: [
|
||||
'operationId',
|
||||
'location',
|
||||
'account',
|
||||
'user',
|
||||
'bucket',
|
||||
],
|
||||
arrayOfAllowed: [
|
||||
'objectPutTagging',
|
||||
'objectPutLegalHold',
|
||||
'objectPutRetention',
|
||||
],
|
||||
allowedUtapiEventFilterStates: ['allow', 'deny'],
|
||||
allowedRestoreObjectRequestTierValues: ['Standard'],
|
||||
lifecycleListing: {
|
||||
CURRENT_TYPE: 'current',
|
||||
NON_CURRENT_TYPE: 'noncurrent',
|
||||
ORPHAN_DM_TYPE: 'orphan',
|
||||
},
|
||||
multiObjectDeleteConcurrency: 50,
|
||||
maxScannedLifecycleListingEntries: 10000,
|
||||
overheadField: [
|
||||
'content-length',
|
||||
'owner-id',
|
||||
'versionId',
|
||||
'isNull',
|
||||
'isDeleteMarker',
|
||||
],
|
||||
unsupportedSignatureChecksums: new Set([
|
||||
'STREAMING-UNSIGNED-PAYLOAD-TRAILER',
|
||||
'STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER',
|
||||
'STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD',
|
||||
'STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER',
|
||||
]),
|
||||
supportedSignatureChecksums: new Set([
|
||||
'UNSIGNED-PAYLOAD',
|
||||
'STREAMING-AWS4-HMAC-SHA256-PAYLOAD',
|
||||
]),
|
||||
ipv4Regex: /^(\d{1,3}\.){3}\d{1,3}(\/(3[0-2]|[12]?\d))?$/,
|
||||
ipv6Regex: /^([\da-f]{1,4}:){7}[\da-f]{1,4}$/i,
|
||||
// The AWS assumed Role resource type
|
||||
assumedRoleArnResourceType: 'assumed-role',
|
||||
// Session name of the backbeat lifecycle assumed role session.
|
||||
backbeatLifecycleSessionName: 'backbeat-lifecycle',
|
||||
actionsToConsiderAsObjectPut: [
|
||||
'initiateMultipartUpload',
|
||||
'objectPutPart',
|
||||
'completeMultipartUpload',
|
||||
],
|
||||
// if requester is not bucket owner, bucket policy actions should be denied with
|
||||
// MethodNotAllowed error
|
||||
onlyOwnerAllowed: ['bucketDeletePolicy', 'bucketGetPolicy', 'bucketPutPolicy'],
|
||||
};
|
||||
|
||||
module.exports = constants;
|
||||
|
|
|
@ -79,7 +79,7 @@ function createEncryptedBucket() {
|
|||
.option('-b, --bucket <bucket>', 'Name of the bucket')
|
||||
.option('-h, --host <host>', 'Host of the server')
|
||||
.option('-p, --port <port>', 'Port of the server')
|
||||
.option('-s, --ssl', 'Enable ssl')
|
||||
.option('-s', '--ssl', 'Enable ssl')
|
||||
.option('-v, --verbose')
|
||||
.option('-l, --location-constraint <locationConstraint>',
|
||||
'location Constraint')
|
|
@ -8,6 +8,19 @@ const inMemory = require('./in_memory/backend').backend;
|
|||
const file = require('./file/backend');
|
||||
const KMIPClient = require('arsenal').network.kmipClient;
|
||||
const Common = require('./common');
|
||||
let scalityKMS;
|
||||
let scalityKMSImpl;
|
||||
try {
|
||||
// eslint-disable-next-line import/no-unresolved
|
||||
const ScalityKMS = require('scality-kms');
|
||||
scalityKMS = new ScalityKMS(config.kms);
|
||||
scalityKMSImpl = 'scalityKms';
|
||||
} catch (err) {
|
||||
logger.warn('scality kms unavailable. ' +
|
||||
'Using file kms backend unless mem specified.');
|
||||
scalityKMS = file;
|
||||
scalityKMSImpl = 'fileKms';
|
||||
}
|
||||
|
||||
let client;
|
||||
let implName;
|
||||
|
@ -19,9 +32,8 @@ if (config.backends.kms === 'mem') {
|
|||
client = file;
|
||||
implName = 'fileKms';
|
||||
} else if (config.backends.kms === 'scality') {
|
||||
const ScalityKMS = require('scality-kms');
|
||||
client = new ScalityKMS(config.kms);
|
||||
implName = 'scalityKms';
|
||||
client = scalityKMS;
|
||||
implName = scalityKMSImpl;
|
||||
} else if (config.backends.kms === 'kmip') {
|
||||
const kmipConfig = { kmip: config.kmip };
|
||||
if (!kmipConfig.kmip) {
|
||||
|
@ -57,45 +69,52 @@ class KMS {
|
|||
/**
|
||||
*
|
||||
* @param {string} bucketName - bucket name
|
||||
* @param {object} sseConfig - SSE configuration
|
||||
* @param {object} headers - request headers
|
||||
* @param {object} log - logger object
|
||||
* @param {function} cb - callback
|
||||
* @returns {undefined}
|
||||
* @callback called with (err, serverSideEncryptionInfo: object)
|
||||
*/
|
||||
static bucketLevelEncryption(bucketName, sseConfig, log, cb) {
|
||||
static bucketLevelEncryption(bucketName, headers, log, cb) {
|
||||
const sseAlgorithm = headers['x-amz-scal-server-side-encryption'];
|
||||
const sseMasterKeyId =
|
||||
headers['x-amz-scal-server-side-encryption-aws-kms-key-id'];
|
||||
/*
|
||||
The purpose of bucket level encryption is so that the client does not
|
||||
have to send appropriate headers to trigger encryption on each object
|
||||
put in an "encrypted bucket". Customer provided keys are not
|
||||
feasible in this system because we do not want to store this key
|
||||
in the bucket metadata.
|
||||
*/
|
||||
const { algorithm, configuredMasterKeyId, mandatory } = sseConfig;
|
||||
const _mandatory = mandatory === true;
|
||||
if (algorithm === 'AES256' || algorithm === 'aws:kms') {
|
||||
return this.createBucketKey(bucketName, log, (err, masterKeyId) => {
|
||||
*/
|
||||
if (sseAlgorithm === 'AES256' ||
|
||||
(sseAlgorithm === 'aws:kms' && sseMasterKeyId === undefined)) {
|
||||
this.createBucketKey(bucketName, log, (err, masterKeyId) => {
|
||||
if (err) {
|
||||
return cb(err);
|
||||
cb(err);
|
||||
return;
|
||||
}
|
||||
|
||||
const serverSideEncryptionInfo = {
|
||||
cryptoScheme: 1,
|
||||
algorithm,
|
||||
algorithm: sseAlgorithm,
|
||||
masterKeyId,
|
||||
mandatory: _mandatory,
|
||||
mandatory: true,
|
||||
};
|
||||
|
||||
if (algorithm === 'aws:kms' && configuredMasterKeyId) {
|
||||
serverSideEncryptionInfo.configuredMasterKeyId = configuredMasterKeyId;
|
||||
}
|
||||
return cb(null, serverSideEncryptionInfo);
|
||||
cb(null, serverSideEncryptionInfo);
|
||||
});
|
||||
} else if (sseAlgorithm === 'aws:kms') {
|
||||
const serverSideEncryptionInfo = {
|
||||
cryptoScheme: 1,
|
||||
algorithm: sseAlgorithm,
|
||||
masterKeyId: sseMasterKeyId,
|
||||
mandatory: true,
|
||||
};
|
||||
cb(null, serverSideEncryptionInfo);
|
||||
} else {
|
||||
/*
|
||||
* no encryption
|
||||
*/
|
||||
cb(null, null);
|
||||
}
|
||||
/*
|
||||
* no encryption
|
||||
*/
|
||||
return cb(null, null);
|
||||
}
|
||||
|
||||
/**
|
||||
|
@ -150,18 +169,9 @@ class KMS {
|
|||
static createCipherBundle(serverSideEncryptionInfo,
|
||||
log, cb) {
|
||||
const dataKey = this.createDataKey(log);
|
||||
|
||||
const { algorithm, configuredMasterKeyId, masterKeyId: bucketMasterKeyId } = serverSideEncryptionInfo;
|
||||
|
||||
let masterKeyId = bucketMasterKeyId;
|
||||
if (configuredMasterKeyId) {
|
||||
log.debug('using user configured kms master key id');
|
||||
masterKeyId = configuredMasterKeyId;
|
||||
}
|
||||
|
||||
const cipherBundle = {
|
||||
algorithm,
|
||||
masterKeyId,
|
||||
algorithm: serverSideEncryptionInfo.algorithm,
|
||||
masterKeyId: serverSideEncryptionInfo.masterKeyId,
|
||||
cryptoScheme: 1,
|
||||
cipheredDataKey: null,
|
||||
cipher: null,
|
||||
|
@ -171,7 +181,7 @@ class KMS {
|
|||
function cipherDataKey(next) {
|
||||
log.debug('ciphering a data key');
|
||||
return client.cipherDataKey(cipherBundle.cryptoScheme,
|
||||
cipherBundle.masterKeyId,
|
||||
serverSideEncryptionInfo.masterKeyId,
|
||||
dataKey, log, (err, cipheredDataKey) => {
|
||||
if (err) {
|
||||
log.debug('error from kms',
|
||||
|
@ -286,29 +296,6 @@ class KMS {
|
|||
return cb(err, decipherBundle);
|
||||
});
|
||||
}
|
||||
|
||||
static checkHealth(log, cb) {
|
||||
if (!client.healthcheck) {
|
||||
return cb(null, {
|
||||
[implName]: { code: 200, message: 'OK' },
|
||||
});
|
||||
}
|
||||
return client.healthcheck(log, err => {
|
||||
const respBody = {};
|
||||
if (err) {
|
||||
respBody[implName] = {
|
||||
error: err.description,
|
||||
code: err.code,
|
||||
};
|
||||
} else {
|
||||
respBody[implName] = {
|
||||
code: 200,
|
||||
message: 'OK',
|
||||
};
|
||||
}
|
||||
return cb(null, respBody);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = KMS;
|
|
@ -0,0 +1,18 @@
|
|||
const utapiServer = require('utapi').UtapiServer;
|
||||
const _config = require('../Config').config;
|
||||
|
||||
// start utapi server
|
||||
if (_config.utapi) {
|
||||
const fullConfig = Object.assign({}, _config.utapi);
|
||||
if (_config.vaultd) {
|
||||
Object.assign(fullConfig, { vaultd: _config.vaultd });
|
||||
}
|
||||
if (_config.https) {
|
||||
Object.assign(fullConfig, { https: _config.https });
|
||||
}
|
||||
// copy healthcheck IPs
|
||||
if (_config.healthChecks) {
|
||||
Object.assign(fullConfig, { healthChecks: _config.healthChecks });
|
||||
}
|
||||
utapiServer(fullConfig);
|
||||
}
|
|
@ -0,0 +1,6 @@
|
|||
const UtapiReplay = require('utapi').UtapiReplay;
|
||||
const _config = require('../Config').config;
|
||||
|
||||
// start utapi server
|
||||
const replay = new UtapiReplay(_config.utapi);
|
||||
replay.start();
|
|
@ -2,72 +2,23 @@ const http = require('http');
|
|||
const https = require('https');
|
||||
const commander = require('commander');
|
||||
const { auth } = require('arsenal');
|
||||
const { UtapiClient, utapiVersion } = require('utapi');
|
||||
const { UtapiClient } = require('utapi');
|
||||
const logger = require('../utilities/logger');
|
||||
const _config = require('../Config').config;
|
||||
const { suppressedUtapiEventFields: suppressedEventFields } = require('../../constants');
|
||||
// setup utapi client
|
||||
let utapiConfig;
|
||||
|
||||
if (utapiVersion === 1 && _config.utapi) {
|
||||
if (_config.utapi.redis === undefined) {
|
||||
utapiConfig = Object.assign({}, _config.utapi, { redis: _config.redis });
|
||||
} else {
|
||||
utapiConfig = Object.assign({}, _config.utapi);
|
||||
}
|
||||
} else if (utapiVersion === 2) {
|
||||
utapiConfig = Object.assign({
|
||||
tls: _config.https,
|
||||
suppressedEventFields,
|
||||
}, _config.utapi || {});
|
||||
}
|
||||
|
||||
const utapi = new UtapiClient(utapiConfig);
|
||||
|
||||
const bucketOwnerMetrics = [
|
||||
'completeMultipartUpload',
|
||||
'multiObjectDelete',
|
||||
'abortMultipartUpload',
|
||||
'copyObject',
|
||||
'deleteObject',
|
||||
'putObject',
|
||||
'uploadPartCopy',
|
||||
'uploadPart',
|
||||
];
|
||||
|
||||
function evalAuthInfo(authInfo, canonicalID, action) {
|
||||
let accountId = authInfo.getCanonicalID();
|
||||
let userId = authInfo.isRequesterAnIAMUser() ?
|
||||
authInfo.getShortid() : undefined;
|
||||
// If action impacts 'numberOfObjectsStored' or 'storageUtilized' metric
|
||||
// only the bucket owner account's metrics should be updated
|
||||
const canonicalIdMatch = authInfo.getCanonicalID() === canonicalID;
|
||||
if (bucketOwnerMetrics.includes(action) && !canonicalIdMatch) {
|
||||
accountId = canonicalID;
|
||||
userId = undefined;
|
||||
}
|
||||
return {
|
||||
accountId,
|
||||
userId,
|
||||
};
|
||||
}
|
||||
const utapi = new UtapiClient(_config.utapi);
|
||||
|
||||
function _listMetrics(host,
|
||||
port,
|
||||
metric,
|
||||
metricType,
|
||||
timeRange,
|
||||
accessKey,
|
||||
secretKey,
|
||||
verbose,
|
||||
recent,
|
||||
ssl) {
|
||||
port,
|
||||
metric,
|
||||
metricType,
|
||||
timeRange,
|
||||
accessKey,
|
||||
secretKey,
|
||||
verbose,
|
||||
recent,
|
||||
ssl) {
|
||||
const listAction = recent ? 'ListRecentMetrics' : 'ListMetrics';
|
||||
// If recent listing, we do not provide `timeRange` in the request
|
||||
const requestObj = recent
|
||||
? { [metric]: metricType }
|
||||
: { timeRange, [metric]: metricType };
|
||||
const requestBody = JSON.stringify(requestObj);
|
||||
const options = {
|
||||
host,
|
||||
port,
|
||||
|
@ -76,7 +27,6 @@ function _listMetrics(host,
|
|||
headers: {
|
||||
'content-type': 'application/json',
|
||||
'cache-control': 'no-cache',
|
||||
'content-length': Buffer.byteLength(requestBody),
|
||||
},
|
||||
rejectUnauthorized: false,
|
||||
};
|
||||
|
@ -114,8 +64,10 @@ function _listMetrics(host,
|
|||
if (verbose) {
|
||||
logger.info('request headers', { headers: request._headers });
|
||||
}
|
||||
|
||||
request.write(requestBody);
|
||||
// If recent listing, we do not provide `timeRange` in the request
|
||||
const requestObj = recent ? {} : { timeRange };
|
||||
requestObj[metric] = metricType;
|
||||
request.write(JSON.stringify(requestObj));
|
||||
request.end();
|
||||
}
|
||||
|
||||
|
@ -138,7 +90,7 @@ function listMetrics(metricType) {
|
|||
if (metricType === 'buckets') {
|
||||
commander
|
||||
.option('-b, --buckets <buckets>', 'Name of bucket(s) with ' +
|
||||
'a comma separator if more than one');
|
||||
'a comma separator if more than one');
|
||||
} else {
|
||||
commander
|
||||
.option('-m, --metric <metric>', 'Metric type')
|
||||
|
@ -193,19 +145,6 @@ function listMetrics(metricType) {
|
|||
}
|
||||
});
|
||||
|
||||
// The string `commander[metric]` is a comma-separated list of resources
|
||||
// given by the user.
|
||||
const resources = commander[metric].split(',');
|
||||
|
||||
// Validate passed accounts to remove any canonicalIDs
|
||||
if (metric === 'accounts') {
|
||||
const invalid = resources.filter(r => !/^\d{12}$/.test(r));
|
||||
if (invalid.length > 0) {
|
||||
logger.error(`Invalid account ID: ${invalid.join(', ')}`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
const timeRange = [];
|
||||
// If recent listing, we disregard any start or end option given
|
||||
if (!recent) {
|
||||
|
@ -228,7 +167,9 @@ function listMetrics(metricType) {
|
|||
timeRange.push(numEnd);
|
||||
}
|
||||
}
|
||||
|
||||
// The string `commander[metric]` is a comma-separated list of resources
|
||||
// given by the user.
|
||||
const resources = commander[metric].split(',');
|
||||
_listMetrics(host, port, metric, resources, timeRange, accessKey, secretKey,
|
||||
verbose, recent, ssl);
|
||||
}
|
||||
|
@ -249,84 +190,13 @@ function listMetrics(metricType) {
|
|||
* @param {number} [metricObj.newByteLength] - (optional) new object size
|
||||
* @param {number|null} [metricObj.oldByteLength] - (optional) old object size
|
||||
* (obj. overwrites)
|
||||
* @param {number} [metricObj.numberOfObjects] - (optional) number of objects
|
||||
* @param {number} [metricObj.numberOfObjects] - (optional) number of obects
|
||||
* added/deleted
|
||||
* @param {boolean} [metricObject.isDelete] - (optional) Indicates whether this
|
||||
* is a delete operation
|
||||
* @return {function | undefined} - `utapi.pushMetric` or undefined if the action is
|
||||
* filtered out and not pushed to utapi.
|
||||
* @return {function} - `utapi.pushMetric`
|
||||
*/
|
||||
function pushMetric(action, log, metricObj) {
|
||||
const {
|
||||
bucket,
|
||||
keys,
|
||||
versionId,
|
||||
byteLength,
|
||||
newByteLength,
|
||||
oldByteLength,
|
||||
numberOfObjects,
|
||||
authInfo,
|
||||
canonicalID,
|
||||
location,
|
||||
isDelete,
|
||||
removedDeleteMarkers,
|
||||
} = metricObj;
|
||||
|
||||
if (utapiVersion === 2) {
|
||||
const incomingBytes = action === 'getObject' ? 0 : newByteLength;
|
||||
let sizeDelta = incomingBytes;
|
||||
if (Number.isInteger(oldByteLength) && Number.isInteger(newByteLength)) {
|
||||
sizeDelta = newByteLength - oldByteLength;
|
||||
// Include oldByteLength in conditional so we don't end up with `-0`
|
||||
} else if (action === 'completeMultipartUpload' && !versionId && oldByteLength) {
|
||||
// If this is a non-versioned bucket we need to decrement
|
||||
// the sizeDelta added by uploadPart when completeMPU is called.
|
||||
sizeDelta = -oldByteLength;
|
||||
} else if (action === 'abortMultipartUpload' && byteLength) {
|
||||
sizeDelta = -byteLength;
|
||||
} else if (action === 'putDeleteMarkerObject' && byteLength) {
|
||||
sizeDelta = -byteLength;
|
||||
}
|
||||
|
||||
let objectDelta = isDelete ? -numberOfObjects : numberOfObjects;
|
||||
// putDeleteMarkerObject does not pass numberOfObjects
|
||||
if ((action === 'putDeleteMarkerObject' && byteLength === null)
|
||||
|| action === 'replicateDelete'
|
||||
|| action === 'replicateObject') {
|
||||
objectDelta = 1;
|
||||
} else if (action === 'multiObjectDelete') {
|
||||
objectDelta = -(numberOfObjects + removedDeleteMarkers);
|
||||
}
|
||||
|
||||
const utapiObj = {
|
||||
operationId: action,
|
||||
bucket,
|
||||
location,
|
||||
objectDelta,
|
||||
sizeDelta: isDelete ? -byteLength : sizeDelta,
|
||||
incomingBytes,
|
||||
outgoingBytes: action === 'getObject' ? newByteLength : 0,
|
||||
};
|
||||
|
||||
// Any operation from lifecycle that does not change object count or size is dropped
|
||||
const isLifecycle = _config.lifecycleRoleName
|
||||
&& authInfo && authInfo.arn.endsWith(`:assumed-role/${_config.lifecycleRoleName}/backbeat-lifecycle`);
|
||||
if (isLifecycle && !objectDelta && !sizeDelta) {
|
||||
log.trace('ignoring pushMetric from lifecycle service user', { action, bucket, keys });
|
||||
return undefined;
|
||||
}
|
||||
|
||||
if (keys && keys.length === 1) {
|
||||
[utapiObj.object] = keys;
|
||||
if (versionId) {
|
||||
utapiObj.versionId = versionId;
|
||||
}
|
||||
}
|
||||
|
||||
utapiObj.account = authInfo ? evalAuthInfo(authInfo, canonicalID, action).accountId : canonicalID;
|
||||
utapiObj.user = authInfo ? evalAuthInfo(authInfo, canonicalID, action).userId : undefined;
|
||||
return utapi.pushMetric(utapiObj);
|
||||
}
|
||||
const { bucket, keys, byteLength, newByteLength,
|
||||
oldByteLength, numberOfObjects, authInfo, canonicalID } = metricObj;
|
||||
const utapiObj = {
|
||||
bucket,
|
||||
keys,
|
||||
|
@ -339,56 +209,16 @@ function pushMetric(action, log, metricObj) {
|
|||
// account-level metrics and the shortId for user-level metrics. Otherwise
|
||||
// check if the canonical ID is already provided for account-level metrics.
|
||||
if (authInfo) {
|
||||
const { accountId, userId } = evalAuthInfo(authInfo, canonicalID, action);
|
||||
utapiObj.accountId = accountId;
|
||||
utapiObj.userId = userId;
|
||||
utapiObj.accountId = authInfo.getCanonicalID();
|
||||
utapiObj.userId = authInfo.isRequesterAnIAMUser() ?
|
||||
authInfo.getShortid() : undefined;
|
||||
} else if (canonicalID) {
|
||||
utapiObj.accountId = canonicalID;
|
||||
}
|
||||
return utapi.pushMetric(action, log.getSerializedUids(), utapiObj);
|
||||
}
|
||||
|
||||
/**
|
||||
* internal: get the unique location ID from the location name
|
||||
*
|
||||
* @param {string} location - location name
|
||||
* @return {string} - location unique ID
|
||||
*/
|
||||
function _getLocationId(location) {
|
||||
return _config.locationConstraints[location].objectId;
|
||||
}
|
||||
|
||||
/**
|
||||
* Call the Utapi Client 'getLocationMetric' method with the
|
||||
* associated parameters
|
||||
* @param {string} location - name of data backend to list metric for
|
||||
* @param {object} log - werelogs logger
|
||||
* @param {function} cb - callback to call
|
||||
* @return {function} - `utapi.getLocationMetric`
|
||||
*/
|
||||
function getLocationMetric(location, log, cb) {
|
||||
const locationId = _getLocationId(location);
|
||||
return utapi.getLocationMetric(locationId, log.getSerializedUids(), cb);
|
||||
}
|
||||
|
||||
/**
|
||||
* Call the Utapi Client 'pushLocationMetric' method with the
|
||||
* associated parameters
|
||||
* @param {string} location - name of data backend
|
||||
* @param {number} byteLength - number of bytes
|
||||
* @param {object} log - werelogs logger
|
||||
* @param {function} cb - callback to call
|
||||
* @return {function} - `utapi.pushLocationMetric`
|
||||
*/
|
||||
function pushLocationMetric(location, byteLength, log, cb) {
|
||||
const locationId = _getLocationId(location);
|
||||
return utapi.pushLocationMetric(locationId, byteLength,
|
||||
log.getSerializedUids(), cb);
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
listMetrics,
|
||||
pushMetric,
|
||||
getLocationMetric,
|
||||
pushLocationMetric,
|
||||
};
|
|
@ -1,220 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# set -e stops the execution of a script if a command or pipeline has an error
|
||||
set -e
|
||||
|
||||
# modifying config.json
|
||||
JQ_FILTERS_CONFIG="."
|
||||
|
||||
# ENDPOINT var can accept comma separated values
|
||||
# for multiple endpoint locations
|
||||
if [[ "$ENDPOINT" ]]; then
|
||||
IFS="," read -ra HOST_NAMES <<< "$ENDPOINT"
|
||||
for host in "${HOST_NAMES[@]}"; do
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .restEndpoints[\"$host\"]=\"us-east-1\""
|
||||
done
|
||||
echo "Host name has been modified to ${HOST_NAMES[@]}"
|
||||
echo "Note: In your /etc/hosts file on Linux, OS X, or Unix with root permissions, make sure to associate 127.0.0.1 with ${HOST_NAMES[@]}"
|
||||
fi
|
||||
|
||||
if [[ "$LOG_LEVEL" ]]; then
|
||||
if [[ "$LOG_LEVEL" == "info" || "$LOG_LEVEL" == "debug" || "$LOG_LEVEL" == "trace" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .log.logLevel=\"$LOG_LEVEL\""
|
||||
echo "Log level has been modified to $LOG_LEVEL"
|
||||
else
|
||||
echo "The log level you provided is incorrect (info/debug/trace)"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$SSL" && "$HOST_NAMES" ]]; then
|
||||
# This condition makes sure that the certificates are not generated twice. (for docker restart)
|
||||
if [ ! -f ./ca.key ] || [ ! -f ./ca.crt ] || [ ! -f ./server.key ] || [ ! -f ./server.crt ] ; then
|
||||
# Compute config for utapi tests
|
||||
cat >>req.cfg <<EOF
|
||||
[req]
|
||||
distinguished_name = req_distinguished_name
|
||||
prompt = no
|
||||
req_extensions = s3_req
|
||||
|
||||
[req_distinguished_name]
|
||||
CN = ${HOST_NAMES[0]}
|
||||
|
||||
[s3_req]
|
||||
subjectAltName = @alt_names
|
||||
extendedKeyUsage = serverAuth, clientAuth
|
||||
|
||||
[alt_names]
|
||||
DNS.1 = *.${HOST_NAMES[0]}
|
||||
DNS.2 = ${HOST_NAMES[0]}
|
||||
|
||||
EOF
|
||||
|
||||
## Generate SSL key and certificates
|
||||
# Generate a private key for your CSR
|
||||
openssl genrsa -out ca.key 2048
|
||||
# Generate a self signed certificate for your local Certificate Authority
|
||||
openssl req -new -x509 -extensions v3_ca -key ca.key -out ca.crt -days 99999 -subj "/C=US/ST=Country/L=City/O=Organization/CN=S3 CA Server"
|
||||
# Generate a key for S3 Server
|
||||
openssl genrsa -out server.key 2048
|
||||
# Generate a Certificate Signing Request for S3 Server
|
||||
openssl req -new -key server.key -out server.csr -config req.cfg
|
||||
# Generate a local-CA-signed certificate for S3 Server
|
||||
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 99999 -sha256 -extfile req.cfg -extensions s3_req
|
||||
fi
|
||||
## Update S3Server config.json
|
||||
# This condition makes sure that certFilePaths section is not added twice. (for docker restart)
|
||||
if ! grep -q "certFilePaths" ./config.json; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .certFilePaths= { \"key\": \".\/server.key\", \"cert\": \".\/server.crt\", \"ca\": \".\/ca.crt\" }"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$LISTEN_ADDR" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .metadataDaemon.bindAddress=\"$LISTEN_ADDR\""
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .dataDaemon.bindAddress=\"$LISTEN_ADDR\""
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .pfsDaemon.bindAddress=\"$LISTEN_ADDR\""
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .listenOn=[\"$LISTEN_ADDR:8000\"]"
|
||||
fi
|
||||
|
||||
if [[ "$REPLICATION_GROUP_ID" ]] ; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .replicationGroupId=\"$REPLICATION_GROUP_ID\""
|
||||
fi
|
||||
|
||||
if [[ "$DATA_HOST" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .dataClient.host=\"$DATA_HOST\""
|
||||
fi
|
||||
|
||||
if [[ "$METADATA_HOST" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .metadataClient.host=\"$METADATA_HOST\""
|
||||
fi
|
||||
|
||||
if [[ "$PFSD_HOST" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .pfsClient.host=\"$PFSD_HOST\""
|
||||
fi
|
||||
|
||||
if [[ "$MONGODB_HOSTS" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .mongodb.replicaSetHosts=\"$MONGODB_HOSTS\""
|
||||
fi
|
||||
|
||||
if [[ "$MONGODB_RS" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .mongodb.replicaSet=\"$MONGODB_RS\""
|
||||
fi
|
||||
|
||||
if [[ "$MONGODB_DATABASE" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .mongodb.database=\"$MONGODB_DATABASE\""
|
||||
fi
|
||||
|
||||
if [ -z "$REDIS_HA_NAME" ]; then
|
||||
REDIS_HA_NAME='mymaster'
|
||||
fi
|
||||
|
||||
if [[ "$REDIS_SENTINELS" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.name=\"$REDIS_HA_NAME\""
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.sentinels=\"$REDIS_SENTINELS\""
|
||||
elif [[ "$REDIS_HOST" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.host=\"$REDIS_HOST\""
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.port=6379"
|
||||
fi
|
||||
|
||||
if [[ "$REDIS_PORT" ]] && [[ ! "$REDIS_SENTINELS" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.port=$REDIS_PORT"
|
||||
fi
|
||||
|
||||
if [[ "$REDIS_SENTINELS" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.name=\"$REDIS_HA_NAME\""
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.sentinels=\"$REDIS_SENTINELS\""
|
||||
elif [[ "$REDIS_HA_HOST" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.host=\"$REDIS_HA_HOST\""
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.port=6379"
|
||||
fi
|
||||
|
||||
if [[ "$REDIS_HA_PORT" ]] && [[ ! "$REDIS_SENTINELS" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .redis.port=$REDIS_HA_PORT"
|
||||
fi
|
||||
|
||||
if [[ "$RECORDLOG_ENABLED" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .recordLog.enabled=true"
|
||||
fi
|
||||
|
||||
if [[ "$STORAGE_LIMIT_ENABLED" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .utapi.metrics[.utapi.metrics | length]=\"location\""
|
||||
fi
|
||||
|
||||
if [[ "$CRR_METRICS_HOST" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .backbeat.host=\"$CRR_METRICS_HOST\""
|
||||
fi
|
||||
|
||||
if [[ "$CRR_METRICS_PORT" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .backbeat.port=$CRR_METRICS_PORT"
|
||||
fi
|
||||
|
||||
if [[ "$WE_OPERATOR_HOST" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .workflowEngineOperator.host=\"$WE_OPERATOR_HOST\""
|
||||
fi
|
||||
|
||||
if [[ "$WE_OPERATOR_PORT" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .workflowEngineOperator.port=$WE_OPERATOR_PORT"
|
||||
fi
|
||||
|
||||
if [[ "$HEALTHCHECKS_ALLOWFROM" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .healthChecks.allowFrom=[\"$HEALTHCHECKS_ALLOWFROM\"]"
|
||||
fi
|
||||
|
||||
# external backends http(s) agent config
|
||||
|
||||
# AWS
|
||||
if [[ "$AWS_S3_HTTPAGENT_KEEPALIVE" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.aws_s3.httpAgent.keepAlive=$AWS_S3_HTTPAGENT_KEEPALIVE"
|
||||
fi
|
||||
|
||||
if [[ "$AWS_S3_HTTPAGENT_KEEPALIVE_MS" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.aws_s3.httpAgent.keepAliveMsecs=$AWS_S3_HTTPAGENT_KEEPALIVE_MS"
|
||||
fi
|
||||
|
||||
if [[ "$AWS_S3_HTTPAGENT_KEEPALIVE_MAX_SOCKETS" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.aws_s3.httpAgent.maxSockets=$AWS_S3_HTTPAGENT_KEEPALIVE_MAX_SOCKETS"
|
||||
fi
|
||||
|
||||
if [[ "$AWS_S3_HTTPAGENT_KEEPALIVE_MAX_FREE_SOCKETS" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.aws_s3.httpAgent.maxFreeSockets=$AWS_S3_HTTPAGENT_KEEPALIVE_MAX_FREE_SOCKETS"
|
||||
fi
|
||||
|
||||
#GCP
|
||||
if [[ "$GCP_HTTPAGENT_KEEPALIVE" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.gcp.httpAgent.keepAlive=$GCP_HTTPAGENT_KEEPALIVE"
|
||||
fi
|
||||
|
||||
if [[ "$GCP_HTTPAGENT_KEEPALIVE_MS" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.gcp.httpAgent.keepAliveMsecs=$GCP_HTTPAGENT_KEEPALIVE_MS"
|
||||
fi
|
||||
|
||||
if [[ "$GCP_HTTPAGENT_KEEPALIVE_MAX_SOCKETS" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.gcp.httpAgent.maxSockets=$GCP_HTTPAGENT_KEEPALIVE_MAX_SOCKETS"
|
||||
fi
|
||||
|
||||
if [[ "$GCP_HTTPAGENT_KEEPALIVE_MAX_FREE_SOCKETS" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .externalBackends.gcp.httpAgent.maxFreeSockets=$GCP_HTTPAGENT_KEEPALIVE_MAX_FREE_SOCKETS"
|
||||
fi
|
||||
|
||||
if [[ -n "$BUCKET_DENY_FILTER" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .utapi.filter.deny.bucket=[\"$BUCKET_DENY_FILTER\"]"
|
||||
fi
|
||||
|
||||
if [[ "$TESTING_MODE" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .testingMode=true"
|
||||
fi
|
||||
|
||||
if [[ $JQ_FILTERS_CONFIG != "." ]]; then
|
||||
jq "$JQ_FILTERS_CONFIG" config.json > config.json.tmp
|
||||
mv config.json.tmp config.json
|
||||
fi
|
||||
|
||||
if test -v INITIAL_INSTANCE_ID && test -v S3METADATAPATH && ! test -f ${S3METADATAPATH}/uuid ; then
|
||||
echo -n ${INITIAL_INSTANCE_ID} > ${S3METADATAPATH}/uuid
|
||||
fi
|
||||
|
||||
# s3 secret credentials for Zenko
|
||||
if [ -r /run/secrets/s3-credentials ] ; then
|
||||
. /run/secrets/s3-credentials
|
||||
fi
|
||||
|
||||
exec "$@"
|
|
@ -0,0 +1,25 @@
|
|||
FROM node:6-slim
|
||||
MAINTAINER Giorgio Regni <gr@scality.com>
|
||||
|
||||
WORKDIR /usr/src/app
|
||||
|
||||
# Keep the .git directory in order to properly report version
|
||||
COPY ./package.json .
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y jq python git build-essential --no-install-recommends \
|
||||
&& npm install --production \
|
||||
&& apt-get autoremove --purge -y python git build-essential \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& npm cache clear \
|
||||
&& rm -rf ~/.node-gyp \
|
||||
&& rm -rf /tmp/npm-*
|
||||
|
||||
COPY ./ ./
|
||||
|
||||
VOLUME ["/usr/src/app/localData","/usr/src/app/localMetadata"]
|
||||
|
||||
ENTRYPOINT ["/usr/src/app/docker-entrypoint.sh"]
|
||||
CMD [ "npm", "start" ]
|
||||
|
||||
EXPOSE 8000
|
|
@ -7,16 +7,16 @@ COPY . /usr/src/app
|
|||
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y jq python git build-essential --no-install-recommends \
|
||||
&& yarn install --production \
|
||||
&& npm install --production \
|
||||
&& apt-get autoremove --purge -y python git build-essential \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& yarn cache clean \
|
||||
&& npm cache clear \
|
||||
&& rm -rf ~/.node-gyp \
|
||||
&& rm -rf /tmp/yarn-*
|
||||
&& rm -rf /tmp/npm-*
|
||||
|
||||
ENV S3BACKEND mem
|
||||
|
||||
ENTRYPOINT ["/usr/src/app/docker-entrypoint.sh"]
|
||||
CMD [ "yarn", "start" ]
|
||||
CMD [ "npm", "start" ]
|
||||
|
||||
EXPOSE 8000
|
|
@ -0,0 +1,107 @@
|
|||
#!/bin/bash
|
||||
|
||||
# set -e stops the execution of a script if a command or pipeline has an error
|
||||
set -e
|
||||
|
||||
# modifying config.json
|
||||
JQ_FILTERS_CONFIG="."
|
||||
|
||||
if [[ "$ENDPOINT" ]]; then
|
||||
HOST_NAME="$ENDPOINT"
|
||||
fi
|
||||
|
||||
if [[ "$HOST_NAME" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .restEndpoints[\"$HOST_NAME\"]=\"us-east-1\""
|
||||
echo "Host name has been modified to $HOST_NAME"
|
||||
echo "Note: In your /etc/hosts file on Linux, OS X, or Unix with root permissions, make sure to associate 127.0.0.1 with $HOST_NAME"
|
||||
fi
|
||||
|
||||
if [[ "$LOG_LEVEL" ]]; then
|
||||
if [[ "$LOG_LEVEL" == "info" || "$LOG_LEVEL" == "debug" || "$LOG_LEVEL" == "trace" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .log.logLevel=\"$LOG_LEVEL\""
|
||||
echo "Log level has been modified to $LOG_LEVEL"
|
||||
else
|
||||
echo "The log level you provided is incorrect (info/debug/trace)"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$SSL" && "$HOST_NAME" ]]; then
|
||||
# This condition makes sure that the certificates are not generated twice. (for docker restart)
|
||||
if [ ! -f ./ca.key ] || [ ! -f ./ca.crt ] || [ ! -f ./server.key ] || [ ! -f ./server.crt ] ; then
|
||||
# Compute config for utapi tests
|
||||
cat >>req.cfg <<EOF
|
||||
[req]
|
||||
distinguished_name = req_distinguished_name
|
||||
prompt = no
|
||||
req_extensions = s3_req
|
||||
|
||||
[req_distinguished_name]
|
||||
CN = ${HOST_NAME}
|
||||
|
||||
[s3_req]
|
||||
subjectAltName = @alt_names
|
||||
extendedKeyUsage = serverAuth, clientAuth
|
||||
|
||||
[alt_names]
|
||||
DNS.1 = *.${HOST_NAME}
|
||||
DNS.2 = ${HOST_NAME}
|
||||
|
||||
EOF
|
||||
|
||||
## Generate SSL key and certificates
|
||||
# Generate a private key for your CSR
|
||||
openssl genrsa -out ca.key 2048
|
||||
# Generate a self signed certificate for your local Certificate Authority
|
||||
openssl req -new -x509 -extensions v3_ca -key ca.key -out ca.crt -days 99999 -subj "/C=US/ST=Country/L=City/O=Organization/CN=S3 CA Server"
|
||||
# Generate a key for S3 Server
|
||||
openssl genrsa -out server.key 2048
|
||||
# Generate a Certificate Signing Request for S3 Server
|
||||
openssl req -new -key server.key -out server.csr -config req.cfg
|
||||
# Generate a local-CA-signed certificate for S3 Server
|
||||
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 99999 -sha256 -extfile req.cfg -extensions s3_req
|
||||
fi
|
||||
## Update S3Server config.json
|
||||
# This condition makes sure that certFilePaths section is not added twice. (for docker restart)
|
||||
if ! grep -q "certFilePaths" ./config.json; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .certFilePaths= { \"key\": \".\/server.key\", \"cert\": \".\/server.crt\", \"ca\": \".\/ca.crt\" }"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$LISTEN_ADDR" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .metadataDaemon.bindAddress=\"$LISTEN_ADDR\""
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .dataDaemon.bindAddress=\"$LISTEN_ADDR\""
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .listenOn=[\"$LISTEN_ADDR:8000\"]"
|
||||
fi
|
||||
|
||||
if [[ "$DATA_HOST" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .dataClient.host=\"$DATA_HOST\""
|
||||
fi
|
||||
|
||||
if [[ "$METADATA_HOST" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .metadataClient.host=\"$METADATA_HOST\""
|
||||
fi
|
||||
|
||||
if [[ "$REDIS_HOST" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.host=\"$REDIS_HOST\""
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.port=6379"
|
||||
fi
|
||||
|
||||
if [[ "$REDIS_PORT" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .localCache.port=$REDIS_PORT"
|
||||
fi
|
||||
|
||||
if [[ "$RECORDLOG_ENABLED" ]]; then
|
||||
JQ_FILTERS_CONFIG="$JQ_FILTERS_CONFIG | .recordLog.enabled=true"
|
||||
fi
|
||||
|
||||
if [[ $JQ_FILTERS_CONFIG != "." ]]; then
|
||||
jq "$JQ_FILTERS_CONFIG" config.json > config.json.tmp
|
||||
mv config.json.tmp config.json
|
||||
fi
|
||||
|
||||
# s3 secret credentials for Zenko
|
||||
if [ -r /run/secrets/s3-credentials ] ; then
|
||||
. /run/secrets/s3-credentials
|
||||
fi
|
||||
|
||||
exec "$@"
|
|
@ -27,7 +27,7 @@ including null versions and delete markers, described in the above
|
|||
links.
|
||||
|
||||
Implementation of Bucket Versioning in Zenko CloudServer
|
||||
--------------------------------------------------------
|
||||
-----------------------------------------
|
||||
|
||||
Overview of Metadata and API Component Roles
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
@ -66,7 +66,7 @@ The second section, `"Implementation of Bucket Versioning in
|
|||
API" <#implementation-of-bucket-versioning-in-api>`__, describes the way
|
||||
the metadata options are used in the API within S3 actions to create new
|
||||
versions, update their metadata, and delete them. The management of null
|
||||
versions and creation of delete markers is also described in this
|
||||
versions and creation of delete markers are also described in this
|
||||
section.
|
||||
|
||||
Implementation of Bucket Versioning in Metadata
|
||||
|
@ -179,13 +179,12 @@ PUT
|
|||
the master version with this version.
|
||||
- ``versionId: <versionId>`` create or update a specific version (for updating
|
||||
version's ACL or tags, or remote updates in geo-replication)
|
||||
|
||||
* if the version identified by ``versionId`` happens to be the latest
|
||||
- if the version identified by ``versionId`` happens to be the latest
|
||||
version, the master version will be updated as well
|
||||
* if the master version is not as recent as the version identified by
|
||||
- if the master version is not as recent as the version identified by
|
||||
``versionId``, as may happen with cross-region replication, the master
|
||||
will be updated as well
|
||||
* note that with ``versionId`` set to an empty string ``''``, it will
|
||||
- note that with ``versionId`` set to an empty string ``''``, it will
|
||||
overwrite the master version only (same as no options, but the master
|
||||
version will have a ``versionId`` property set in its metadata like
|
||||
any other version). The ``versionId`` will never be exposed to an
|
||||
|
@ -209,13 +208,10 @@ A deletion targeting the latest version of an object has to:
|
|||
- delete the specified version identified by ``versionId``
|
||||
- replace the master version with a version that is a placeholder for
|
||||
deletion
|
||||
|
||||
- this version contains a special keyword, 'isPHD', to indicate the
|
||||
master version was deleted and needs to be updated
|
||||
|
||||
- initiate a repair operation to update the value of the master
|
||||
version:
|
||||
|
||||
- involves listing the versions of the object and get the latest
|
||||
version to replace the placeholder delete version
|
||||
- if no more versions exist, metadata deletes the master version,
|
||||
|
@ -746,7 +742,7 @@ Operation
|
|||
Startup
|
||||
~~~~~~~
|
||||
|
||||
The simplest deployment is still to launch with yarn start, this will
|
||||
The simplest deployment is still to launch with npm start, this will
|
||||
start one instance of the Zenko CloudServer connector and will listen on the
|
||||
locally bound dmd ports 9990 and 9991 (by default, see below).
|
||||
|
||||
|
@ -755,22 +751,22 @@ command in the Zenko CloudServer directory:
|
|||
|
||||
::
|
||||
|
||||
yarn run start_dmd
|
||||
npm run start_dmd
|
||||
|
||||
This will open two ports:
|
||||
|
||||
- one is based on socket.io and is used for metadata transfers (9990 by
|
||||
- one is based on socket.io and is used for metadata transfers (9990 by
|
||||
default)
|
||||
|
||||
- the other is a REST interface used for data transfers (9991 by
|
||||
- the other is a REST interface used for data transfers (9991 by
|
||||
default)
|
||||
|
||||
Then, one or more instances of Zenko CloudServer without the dmd can be started
|
||||
elsewhere with:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
yarn run start_s3server
|
||||
npm run start_s3server
|
||||
|
||||
Configuration
|
||||
~~~~~~~~~~~~~
|
||||
|
@ -796,10 +792,10 @@ access:
|
|||
|
||||
To run a remote dmd, you have to do the following:
|
||||
|
||||
- change both ``"host"`` attributes to the IP or host name where the
|
||||
- change both ``"host"`` attributes to the IP or host name where the
|
||||
dmd is run.
|
||||
|
||||
- Modify the ``"bindAddress"`` attributes in ``"metadataDaemon"`` and
|
||||
- Modify the ``"bindAddress"`` attributes in ``"metadataDaemon"`` and
|
||||
``"dataDaemon"`` sections where the dmd is run to accept remote
|
||||
connections (e.g. ``"::"``)
|
||||
|
||||
|
@ -835,13 +831,13 @@ and ``createReadStream``. They more or less map the parameters accepted
|
|||
by the corresponding calls in the LevelUp implementation of LevelDB.
|
||||
They differ in the following:
|
||||
|
||||
- The ``sync`` option is ignored (under the hood, puts are gathered
|
||||
- The ``sync`` option is ignored (under the hood, puts are gathered
|
||||
into batches which have their ``sync`` property enforced when they
|
||||
are committed to the storage)
|
||||
|
||||
- Some additional versioning-specific options are supported
|
||||
- Some additional versioning-specific options are supported
|
||||
|
||||
- ``createReadStream`` becomes asynchronous, takes an additional
|
||||
- ``createReadStream`` becomes asynchronous, takes an additional
|
||||
callback argument and returns the stream in the second callback
|
||||
parameter
|
||||
|
||||
|
@ -851,10 +847,10 @@ with ``DEBUG='socket.io*'`` environment variable set.
|
|||
One parameter controls the timeout value after which RPC commands sent
|
||||
end with a timeout error, it can be changed either:
|
||||
|
||||
- via the ``DEFAULT_CALL_TIMEOUT_MS`` option in
|
||||
- via the ``DEFAULT_CALL_TIMEOUT_MS`` option in
|
||||
``lib/network/rpc/rpc.js``
|
||||
|
||||
- or in the constructor call of the ``MetadataFileClient`` object (in
|
||||
- or in the constructor call of the ``MetadataFileClient`` object (in
|
||||
``lib/metadata/bucketfile/backend.js`` as ``callTimeoutMs``.
|
||||
|
||||
Default value is 30000.
|
||||
|
@ -868,10 +864,10 @@ can tune the behavior (for better throughput or getting it more robust
|
|||
on weak networks), they have to be set in ``mdserver.js`` file directly,
|
||||
as there is no support in ``config.json`` for now for those options:
|
||||
|
||||
- ``streamMaxPendingAck``: max number of pending ack events not yet
|
||||
- ``streamMaxPendingAck``: max number of pending ack events not yet
|
||||
received (default is 5)
|
||||
|
||||
- ``streamAckTimeoutMs``: timeout for receiving an ack after an output
|
||||
- ``streamAckTimeoutMs``: timeout for receiving an ack after an output
|
||||
stream packet is sent to the client (default is 5000)
|
||||
|
||||
Data exchange through the REST data port
|
||||
|
@ -922,17 +918,17 @@ Listing Types
|
|||
We use three different types of metadata listing for various operations.
|
||||
Here are the scenarios we use each for:
|
||||
|
||||
- 'Delimiter' - when no versions are possible in the bucket since it is
|
||||
- 'Delimiter' - when no versions are possible in the bucket since it is
|
||||
an internally-used only bucket which is not exposed to a user.
|
||||
Namely,
|
||||
|
||||
1. to list objects in the "user's bucket" to respond to a GET SERVICE
|
||||
request and
|
||||
request and
|
||||
2. to do internal listings on an MPU shadow bucket to complete multipart
|
||||
upload operations.
|
||||
upload operations.
|
||||
|
||||
- 'DelimiterVersion' - to list all versions in a bucket
|
||||
- 'DelimiterMaster' - to list just the master versions of objects in a
|
||||
- 'DelimiterVersion' - to list all versions in a bucket
|
||||
- 'DelimiterMaster' - to list just the master versions of objects in a
|
||||
bucket
|
||||
|
||||
Algorithms
|
||||
|
|
|
@ -1,146 +0,0 @@
|
|||
# Bucket Policy Documentation
|
||||
|
||||
## Description
|
||||
|
||||
Bucket policy is a method of controlling access to a user's account at the
|
||||
resource level.
|
||||
There are three associated APIs:
|
||||
|
||||
- PUT Bucket policy (see https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTpolicy.html)
|
||||
- GET Bucket policy (see https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETpolicy.html)
|
||||
- DELETE Bucket policy (see https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketDELETEpolicy.html)
|
||||
|
||||
More information on bucket policies in general can be found at
|
||||
https://docs.aws.amazon.com/AmazonS3/latest/dev/using-iam-policies.html.
|
||||
|
||||
## Requirements
|
||||
|
||||
To prevent loss of access to a bucket, the root owner of a bucket will always
|
||||
be able to perform any of the three bucket policy-related operations, even
|
||||
if permission is explicitly denied.
|
||||
All other users must have permission to perform the desired operation.
|
||||
|
||||
## Design
|
||||
|
||||
On a PUTBucketPolicy request, the user provides a policy in JSON format.
|
||||
The policy is evaluated against our policy schema in Arsenal and, once
|
||||
validated, is stored as part of the bucket's metadata.
|
||||
On a GETBucketPolicy request, the policy is retrieved from the bucket's
|
||||
metadata.
|
||||
On a DELETEBucketPolicy request, the policy is deleted from the bucket's
|
||||
metadata.
|
||||
|
||||
All other APIs are updated to check if a bucket policy is attached to the bucket
|
||||
the request is made on. If there is a policy, user authorization to perform
|
||||
the requested action is checked.
|
||||
|
||||
### Differences Between Bucket and IAM Policies
|
||||
|
||||
IAM policies are attached to an IAM identity and define what actions that
|
||||
identity is allowed to or denied from doing on what resource.
|
||||
Bucket policies attach only to buckets and define what actions are allowed or
|
||||
denied for which principles on that bucket. Permissions specified in a bucket
|
||||
policy apply to all objects in that bucket unless otherwise specified.
|
||||
|
||||
Besides their attachment origins, the main structural difference between
|
||||
IAM policy and bucket policy is the requirement of a "Principal" element in
|
||||
bucket policies. This field is redundant in IAM policies.
|
||||
|
||||
### Policy Validation
|
||||
|
||||
For general guidelines for bucket policy structure, see examples here:
|
||||
https://docs.aws.amazon.com/AmazonS3/latest/dev//example-bucket-policies.html.
|
||||
|
||||
Each bucket policy statement object requires at least four keys:
|
||||
"Effect", "Principle", "Resource", and "Action".
|
||||
|
||||
"Effect" defines the effect of the policy and can have a string value of either
|
||||
"Allow" or "Deny".
|
||||
"Resource" defines to which bucket or list of buckets a policy is attached.
|
||||
An object within the bucket is also a valid resource. The element value can be
|
||||
either a single bucket or object ARN string or an array of ARNs.
|
||||
"Action" lists which action(s) the policy controls. Its value can also be either
|
||||
a string or array of S3 APIs. Each action is the API name prepended by "s3:".
|
||||
"Principle" specifies which user(s) are granted or denied access to the bucket
|
||||
resource. Its value can be a string or an object containing an array of users.
|
||||
Valid users can be identified with an account ARN, account id, or user ARN.
|
||||
|
||||
There are also two optional bucket policy statement keys: Sid and Condition.
|
||||
|
||||
"Sid" stands for "statement id". If this key is not included, one will be
|
||||
generated for the statement.
|
||||
"Condition" lists the condition under which a statement will take affect.
|
||||
The possibilities are as follows:
|
||||
|
||||
- ArnEquals
|
||||
- ArnEqualsIfExists
|
||||
- ArnLike
|
||||
- ArnLikeIfExists
|
||||
- ArnNotEquals
|
||||
- ArnNotEqualsIfExists
|
||||
- ArnNotLike
|
||||
- ArnNotLikeIfExists
|
||||
- BinaryEquals
|
||||
- BinaryEqualsIfExists
|
||||
- BinaryNotEquals
|
||||
- BinaryNotEqualsIfExists
|
||||
- Bool
|
||||
- BoolIfExists
|
||||
- DateEquals
|
||||
- DateEqualsIfExists
|
||||
- DateGreaterThan
|
||||
- DateGreaterThanEquals
|
||||
- DateGreaterThanEqualsIfExists
|
||||
- DateGreaterThanIfExists
|
||||
- DateLessThan
|
||||
- DateLessThanEquals
|
||||
- DateLessThanEqualsIfExists
|
||||
- DateLessThanIfExists
|
||||
- DateNotEquals
|
||||
- DateNotEqualsIfExists
|
||||
- IpAddress
|
||||
- IpAddressIfExists
|
||||
- NotIpAddress
|
||||
- NotIpAddressIfExists
|
||||
- Null
|
||||
- NumericEquals
|
||||
- NumericEqualsIfExists
|
||||
- NumericGreaterThan
|
||||
- NumericGreaterThanEquals
|
||||
- NumericGreaterThanEqualsIfExists
|
||||
- NumericGreaterThanIfExists
|
||||
- NumericLessThan
|
||||
- NumericLessThanEquals
|
||||
- NumericLessThanEqualsIfExists
|
||||
- NumericLessThanIfExists
|
||||
- NumericNotEquals
|
||||
- NumericNotEqualsIfExists
|
||||
- StringEquals
|
||||
- StringEqualsIfExists
|
||||
- StringEqualsIgnoreCase
|
||||
- StringEqualsIgnoreCaseIfExists
|
||||
- StringLike
|
||||
- StringLikeIfExists
|
||||
- StringNotEquals
|
||||
- StringNotEqualsIfExists
|
||||
- StringNotEqualsIgnoreCase
|
||||
- StringNotEqualsIgnoreCaseIfExists
|
||||
- StringNotLike
|
||||
- StringNotLikeIfExists
|
||||
|
||||
The value of the Condition key will be an object containing the desired
|
||||
condition name as that key. The value of inner object can be a string, boolean,
|
||||
number, or object, depending on the condition.
|
||||
|
||||
## Authorization with Multiple Access Control Mechanisms
|
||||
|
||||
In the case where multiple access control mechanisms (such as IAM policies,
|
||||
bucket policies, and ACLs) refer to the same resource, the principle of
|
||||
least-privilege is applied. Unless an action is explicitly allowed, access will
|
||||
by default be denied. An explicit DENY in any policy will trump another
|
||||
policy's ALLOW for an action. The request will only be allowed if at least one
|
||||
policy specifies an ALLOW, and there is no overriding DENY.
|
||||
|
||||
The following diagram illustrates this logic:
|
||||
|
||||
![Access_Control_Authorization_Chart](./images/access_control_authorization.png)
|
|
@ -178,7 +178,7 @@ Ruby
|
|||
~~~~
|
||||
|
||||
`AWS SDK for Ruby - Version 2 <http://docs.aws.amazon.com/sdkforruby/api/>`__
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: ruby
|
||||
|
||||
|
@ -239,7 +239,6 @@ Python
|
|||
Client integration
|
||||
|
||||
.. code:: python
|
||||
|
||||
import boto3
|
||||
|
||||
client = boto3.client(
|
||||
|
@ -254,7 +253,6 @@ Client integration
|
|||
Full integration (with object mapping)
|
||||
|
||||
.. code:: python
|
||||
|
||||
import os
|
||||
|
||||
from botocore.utils import fix_s3_host
|
||||
|
@ -295,51 +293,3 @@ Should force path-style requests even though v3 advertises it does by default.
|
|||
$client->createBucket(array(
|
||||
'Bucket' => 'bucketphp',
|
||||
));
|
||||
|
||||
Go
|
||||
~~
|
||||
|
||||
`AWS Go SDK <https://github.com/aws/aws-sdk-go>`__
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. code:: go
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/aws/endpoints"
|
||||
"github.com/aws/aws-sdk-go/aws/session"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
)
|
||||
|
||||
func main() {
|
||||
os.Setenv("AWS_ACCESS_KEY_ID", "accessKey1")
|
||||
os.Setenv("AWS_SECRET_ACCESS_KEY", "verySecretKey1")
|
||||
endpoint := "http://localhost:8000"
|
||||
timeout := time.Duration(10) * time.Second
|
||||
sess := session.Must(session.NewSession())
|
||||
|
||||
// Create a context with a timeout that will abort the upload if it takes
|
||||
// more than the passed in timeout.
|
||||
ctx, cancel := context.WithTimeout(context.Background(), timeout)
|
||||
defer cancel()
|
||||
|
||||
svc := s3.New(sess, &aws.Config{
|
||||
Region: aws.String(endpoints.UsEast1RegionID),
|
||||
Endpoint: &endpoint,
|
||||
})
|
||||
|
||||
out, err := svc.ListBucketsWithContext(ctx, &s3.ListBucketsInput{})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
} else {
|
||||
fmt.Println(out)
|
||||
}
|
||||
}
|
||||
|
|
|
@ -14,7 +14,7 @@ Got an idea? Get started!
|
|||
In order to contribute, please follow the `Contributing
|
||||
Guidelines <https://github.com/scality/Guidelines/blob/master/CONTRIBUTING.md>`__.
|
||||
If anything is unclear to you, reach out to us on
|
||||
`forum <https://forum.zenko.io/>`__ or via a GitHub issue.
|
||||
`slack <https://zenko-io.slack.com/>`__ or via a GitHub issue.
|
||||
|
||||
Don't write code? There are other ways to help!
|
||||
-----------------------------------------------
|
||||
|
|
378
docs/DOCKER.rst
378
docs/DOCKER.rst
|
@ -1,7 +1,11 @@
|
|||
Docker
|
||||
======
|
||||
|
||||
.. _environment-variables:
|
||||
- `Environment Variables <#environment-variables>`__
|
||||
- `Tunables and setup tips <#tunables-and-setup-tips>`__
|
||||
- `Examples for continuous integration with
|
||||
Docker <#continuous-integration-with-docker-hosted CloudServer>`__
|
||||
- `Examples for going in production with Docker <#in-production-with-docker-hosted CloudServer>`__
|
||||
|
||||
Environment Variables
|
||||
---------------------
|
||||
|
@ -11,27 +15,25 @@ S3DATA
|
|||
|
||||
S3DATA=multiple
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
This variable enables running CloudServer with multiple data backends, defined
|
||||
Allows you to run Scality Zenko CloudServer with multiple data backends, defined
|
||||
as regions.
|
||||
When using multiple data backends, a custom ``locationConfig.json`` file is
|
||||
mandatory. It will allow you to set custom regions. You will then need to
|
||||
provide associated rest_endpoints for each custom region in your
|
||||
``config.json`` file.
|
||||
`Learn more about multiple backends configuration <../GETTING_STARTED/#location-configuration>`__
|
||||
|
||||
For multiple data backends, a custom locationConfig.json file is required.
|
||||
This file enables you to set custom regions. You must provide associated
|
||||
rest_endpoints for each custom region in config.json.
|
||||
If you are using Scality RING endpoints, please refer to your customer
|
||||
documentation.
|
||||
|
||||
`Learn more about multiple-backend configurations <GETTING_STARTED.html#location-configuration>`__
|
||||
|
||||
If you are using Scality RING endpoints, refer to your customer documentation.
|
||||
|
||||
Running CloudServer with an AWS S3-Hosted Backend
|
||||
"""""""""""""""""""""""""""""""""""""""""""""""""
|
||||
|
||||
To run CloudServer with an S3 AWS backend, add a new section to the
|
||||
``locationConfig.json`` file with the ``aws_s3`` location type:
|
||||
Running it with an AWS S3 hosted backend
|
||||
""""""""""""""""""""""""""""""""""""""""
|
||||
To run CloudServer with an S3 AWS backend, you will have to add a new section
|
||||
to your ``locationConfig.json`` file with the ``aws_s3`` location type:
|
||||
|
||||
.. code:: json
|
||||
|
||||
(...)
|
||||
(...)
|
||||
"awsbackend": {
|
||||
"type": "aws_s3",
|
||||
"details": {
|
||||
|
@ -41,139 +43,126 @@ To run CloudServer with an S3 AWS backend, add a new section to the
|
|||
"credentialsProfile": "aws_hosted_profile"
|
||||
}
|
||||
}
|
||||
(...)
|
||||
(...)
|
||||
|
||||
Edit your AWS credentials file to enable your preferred command-line tool.
|
||||
This file must mention credentials for all backends in use. You can use
|
||||
several profiles if multiple profiles are configured.
|
||||
You will also have to edit your AWS credentials file to be able to use your
|
||||
command line tool of choice. This file should mention credentials for all the
|
||||
backends you're using. You can use several profiles when using multiple
|
||||
profiles.
|
||||
|
||||
.. code:: json
|
||||
|
||||
[default]
|
||||
aws_access_key_id=accessKey1
|
||||
aws_secret_access_key=verySecretKey1
|
||||
[aws_hosted_profile]
|
||||
aws_access_key_id={{YOUR_ACCESS_KEY}}
|
||||
aws_secret_access_key={{YOUR_SECRET_KEY}}
|
||||
[default]
|
||||
aws_access_key_id=accessKey1
|
||||
aws_secret_access_key=verySecretKey1
|
||||
[aws_hosted_profile]
|
||||
aws_access_key_id={{YOUR_ACCESS_KEY}}
|
||||
aws_secret_access_key={{YOUR_SECRET_KEY}}
|
||||
|
||||
As with locationConfig.json, the AWS credentials file must be mounted at
|
||||
run time: ``-v ~/.aws/credentials:/root/.aws/credentials`` on Unix-like
|
||||
systems (Linux, OS X, etc.), or
|
||||
Just as you need to mount your locationConfig.json, you will need to mount your
|
||||
AWS credentials file at run time:
|
||||
``-v ~/.aws/credentials:/root/.aws/credentials`` on Linux, OS X, or Unix or
|
||||
``-v C:\Users\USERNAME\.aws\credential:/root/.aws/credentials`` on Windows
|
||||
|
||||
.. note:: One account cannot copy to another account with a source and
|
||||
destination on real AWS unless the account associated with the
|
||||
accessKey/secretKey pairs used for the destination bucket has source
|
||||
bucket access privileges. To enable this, update ACLs directly on AWS.
|
||||
NOTE: One account can't copy to another account with a source and
|
||||
destination on real AWS unless the account associated with the
|
||||
access Key/secret Key pairs used for the destination bucket has rights
|
||||
to get in the source bucket. ACL's would have to be updated
|
||||
on AWS directly to enable this.
|
||||
|
||||
S3BACKEND
|
||||
~~~~~~~~~
|
||||
~~~~~~
|
||||
|
||||
S3BACKEND=file
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
For stored file data to persist, you must mount Docker volumes
|
||||
for both data and metadata. See :ref:`In Production with a Docker-Hosted CloudServer <in-production-w-a-Docker-hosted-cloudserver>`
|
||||
^^^^^^^^^^^
|
||||
When storing file data, for it to be persistent you must mount docker volumes
|
||||
for both data and metadata. See `this section <#using-docker-volumes-in-production>`__
|
||||
|
||||
S3BACKEND=mem
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
This is ideal for testing: no data remains after the container is shut down.
|
||||
^^^^^^^^^^
|
||||
This is ideal for testing - no data will remain after container is shutdown.
|
||||
|
||||
ENDPOINT
|
||||
~~~~~~~~
|
||||
|
||||
This variable specifies the endpoint. To direct CloudServer requests to
|
||||
new.host.com, for example, specify the endpoint with:
|
||||
This variable specifies your endpoint. If you have a domain such as
|
||||
new.host.com, by specifying that here, you and your users can direct s3
|
||||
server requests to new.host.com.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code:: shell
|
||||
|
||||
$ docker run -d --name cloudserver -p 8000:8000 -e ENDPOINT=new.host.com zenko/cloudserver
|
||||
docker run -d --name s3server -p 8000:8000 -e ENDPOINT=new.host.com scality/s3server
|
||||
|
||||
.. note:: On Unix-like systems (Linux, OS X, etc.) edit /etc/hosts
|
||||
to associate 127.0.0.1 with new.host.com.
|
||||
|
||||
REMOTE_MANAGEMENT_DISABLE
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
CloudServer is a part of `Zenko <https://www.zenko.io/>`__. When you run CloudServer standalone it will still try to connect to Orbit by default (browser-based graphical user interface for Zenko).
|
||||
|
||||
Setting this variable to true(1) will default to accessKey1 and verySecretKey1 for credentials and disable the automatic Orbit management:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ docker run -d --name cloudserver -p 8000:8000 -e REMOTE_MANAGEMENT_DISABLE=1 zenko/cloudserver
|
||||
Note: In your ``/etc/hosts`` file on Linux, OS X, or Unix with root
|
||||
permissions, make sure to associate 127.0.0.1 with ``new.host.com``
|
||||
|
||||
SCALITY\_ACCESS\_KEY\_ID and SCALITY\_SECRET\_ACCESS\_KEY
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
These variables specify authentication credentials for an account named
|
||||
“CustomAccount”.
|
||||
"CustomAccount".
|
||||
|
||||
Set account credentials for multiple accounts by editing conf/authdata.json
|
||||
(see below for further details). To specify one set for personal use, set these
|
||||
environment variables:
|
||||
You can set credentials for many accounts by editing
|
||||
``conf/authdata.json`` (see below for further info), but if you just
|
||||
want to specify one set of your own, you can use these environment
|
||||
variables.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code:: shell
|
||||
|
||||
$ docker run -d --name cloudserver -p 8000:8000 -e SCALITY_ACCESS_KEY_ID=newAccessKey \
|
||||
-e SCALITY_SECRET_ACCESS_KEY=newSecretKey zenko/cloudserver
|
||||
docker run -d --name s3server -p 8000:8000 -e SCALITY_ACCESS_KEY_ID=newAccessKey
|
||||
-e SCALITY_SECRET_ACCESS_KEY=newSecretKey scality/s3server
|
||||
|
||||
.. note:: This takes precedence over the contents of the authdata.json
|
||||
file. The authdata.json file is ignored.
|
||||
|
||||
.. note:: The ACCESS_KEY and SECRET_KEY environment variables are
|
||||
deprecated.
|
||||
Note: Anything in the ``authdata.json`` file will be ignored. Note: The
|
||||
old ``ACCESS_KEY`` and ``SECRET_KEY`` environment variables are now
|
||||
deprecated
|
||||
|
||||
LOG\_LEVEL
|
||||
~~~~~~~~~~
|
||||
|
||||
This variable changes the log level. There are three levels: info, debug,
|
||||
and trace. The default is info. Debug provides more detailed logs, and trace
|
||||
provides the most detailed logs.
|
||||
This variable allows you to change the log level: info, debug or trace.
|
||||
The default is info. Debug will give you more detailed logs and trace
|
||||
will give you the most detailed.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code:: shell
|
||||
|
||||
$ docker run -d --name cloudserver -p 8000:8000 -e LOG_LEVEL=trace zenko/cloudserver
|
||||
docker run -d --name s3server -p 8000:8000 -e LOG_LEVEL=trace scality/s3server
|
||||
|
||||
SSL
|
||||
~~~
|
||||
|
||||
Set true, this variable runs CloudServer with SSL.
|
||||
This variable set to true allows you to run S3 with SSL:
|
||||
|
||||
If SSL is set true:
|
||||
**Note1**: You also need to specify the ENDPOINT environment variable.
|
||||
**Note2**: In your ``/etc/hosts`` file on Linux, OS X, or Unix with root
|
||||
permissions, make sure to associate 127.0.0.1 with ``<YOUR_ENDPOINT>``
|
||||
|
||||
* The ENDPOINT environment variable must also be specified.
|
||||
|
||||
* On Unix-like systems (Linux, OS X, etc.), 127.0.0.1 must be associated with
|
||||
<YOUR_ENDPOINT> in /etc/hosts.
|
||||
|
||||
.. Warning:: Self-signed certs with a CA generated within the container are
|
||||
suitable for testing purposes only. Clients cannot trust them, and they may
|
||||
disappear altogether on a container upgrade. The best security practice for
|
||||
production environments is to use an extra container, such as
|
||||
haproxy/nginx/stunnel, for SSL/TLS termination and to pull certificates
|
||||
from a mounted volume, limiting what an exploit on either component
|
||||
can expose.
|
||||
**Warning**: These certs, being self-signed (and the CA being generated
|
||||
inside the container) will be untrusted by any clients, and could
|
||||
disappear on a container upgrade. That's ok as long as it's for quick
|
||||
testing. Also, best security practice for non-testing would be to use an
|
||||
extra container to do SSL/TLS termination such as haproxy/nginx/stunnel
|
||||
to limit what an exploit on either component could expose, as well as
|
||||
certificates in a mounted volume
|
||||
|
||||
.. code:: shell
|
||||
|
||||
$ docker run -d --name cloudserver -p 8000:8000 -e SSL=TRUE -e ENDPOINT=<YOUR_ENDPOINT> \
|
||||
zenko/cloudserver
|
||||
docker run -d --name s3server -p 8000:8000 -e SSL=TRUE -e ENDPOINT=<YOUR_ENDPOINT>
|
||||
scality/s3server
|
||||
|
||||
For more information about using ClousdServer with SSL, see `Using SSL <GETTING_STARTED.html#Using SSL>`__
|
||||
More information about how to use S3 server with SSL
|
||||
`here <https://s3.scality.com/v1.0/page/scality-with-ssl>`__
|
||||
|
||||
LISTEN\_ADDR
|
||||
~~~~~~~~~~~~
|
||||
|
||||
This variable causes CloudServer and its data and metadata components to
|
||||
listen on the specified address. This allows starting the data or metadata
|
||||
servers as standalone services, for example.
|
||||
This variable instructs the Zenko CloudServer, and its data and metadata
|
||||
components to listen on the specified address. This allows starting the data
|
||||
or metadata servers as standalone services, for example.
|
||||
|
||||
.. code:: shell
|
||||
|
||||
docker run -d --name s3server-data -p 9991:9991 -e LISTEN_ADDR=0.0.0.0
|
||||
scality/s3server yarn run start_dataserver
|
||||
scality/s3server npm run start_dataserver
|
||||
|
||||
|
||||
DATA\_HOST and METADATA\_HOST
|
||||
|
@ -185,8 +174,8 @@ Zenko CloudServer.
|
|||
|
||||
.. code:: shell
|
||||
|
||||
$ docker run -d --name cloudserver -e DATA_HOST=cloudserver-data \
|
||||
-e METADATA_HOST=cloudserver-metadata zenko/cloudserver yarn run start_s3server
|
||||
docker run -d --name s3server -e DATA_HOST=s3server-data
|
||||
-e METADATA_HOST=s3server-metadata scality/s3server npm run start_s3server
|
||||
|
||||
REDIS\_HOST
|
||||
~~~~~~~~~~~
|
||||
|
@ -196,21 +185,19 @@ localhost.
|
|||
|
||||
.. code:: shell
|
||||
|
||||
$ docker run -d --name cloudserver -p 8000:8000 \
|
||||
-e REDIS_HOST=my-redis-server.example.com zenko/cloudserver
|
||||
docker run -d --name s3server -p 8000:8000
|
||||
-e REDIS_HOST=my-redis-server.example.com scality/s3server
|
||||
|
||||
REDIS\_PORT
|
||||
~~~~~~~~~~~
|
||||
|
||||
Use this variable to connect to the Redis cache server on a port other
|
||||
than the default 6379.
|
||||
Use this variable to connect to the redis cache server on another port than
|
||||
the default 6379.
|
||||
|
||||
.. code:: shell
|
||||
|
||||
$ docker run -d --name cloudserver -p 8000:8000 \
|
||||
-e REDIS_PORT=6379 zenko/cloudserver
|
||||
|
||||
.. _tunables-and-setup-tips:
|
||||
docker run -d --name s3server -p 8000:8000
|
||||
-e REDIS_PORT=6379 scality/s3server
|
||||
|
||||
Tunables and Setup Tips
|
||||
-----------------------
|
||||
|
@ -218,57 +205,60 @@ Tunables and Setup Tips
|
|||
Using Docker Volumes
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
CloudServer runs with a file backend by default, meaning that data is
|
||||
stored inside the CloudServer’s Docker container.
|
||||
Zenko CloudServer runs with a file backend by default.
|
||||
|
||||
For data and metadata to persist, data and metadata must be hosted in Docker
|
||||
volumes outside the CloudServer’s Docker container. Otherwise, the data
|
||||
and metadata are destroyed when the container is erased.
|
||||
So, by default, the data is stored inside your Zenko CloudServer Docker
|
||||
container.
|
||||
|
||||
.. code-block:: shell
|
||||
However, if you want your data and metadata to persist, you **MUST** use
|
||||
Docker volumes to host your data and metadata outside your Zenko CloudServer
|
||||
Docker container. Otherwise, the data and metadata will be destroyed
|
||||
when you erase the container.
|
||||
|
||||
$ docker run -v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata \
|
||||
-p 8000:8000 -d zenko/cloudserver
|
||||
.. code:: shell
|
||||
|
||||
This command mounts the ./data host directory to the container
|
||||
at /usr/src/app/localData and the ./metadata host directory to
|
||||
the container at /usr/src/app/localMetaData.
|
||||
docker run -v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata
|
||||
-p 8000:8000 -d scality/s3server
|
||||
|
||||
.. tip:: These host directories can be mounted to any accessible mount
|
||||
point, such as /mnt/data and /mnt/metadata, for example.
|
||||
This command mounts the host directory, ``./data``, into the container
|
||||
at ``/usr/src/app/localData`` and the host directory, ``./metadata``, into
|
||||
the container at ``/usr/src/app/localMetaData``. It can also be any host
|
||||
mount point, like ``/mnt/data`` and ``/mnt/metadata``.
|
||||
|
||||
Adding, Modifying, or Deleting Accounts or Credentials
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
Adding modifying or deleting accounts or users credentials
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
1. Create a customized authdata.json file locally based on /conf/authdata.json.
|
||||
|
||||
2. Use `Docker volumes <https://docs.docker.com/storage/volumes/>`__
|
||||
to override the default ``authdata.json`` through a Docker file mapping.
|
||||
1. Create locally a customized ``authdata.json`` based on our ``/conf/authdata.json``.
|
||||
|
||||
2. Use `Docker
|
||||
Volume <https://docs.docker.com/engine/tutorials/dockervolumes/>`__
|
||||
to override the default ``authdata.json`` through a docker file mapping.
|
||||
For example:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code:: shell
|
||||
|
||||
$ docker run -v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json -p 8000:8000 -d \
|
||||
zenko/cloudserver
|
||||
docker run -v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json -p 8000:8000 -d
|
||||
scality/s3server
|
||||
|
||||
Specifying a Host Name
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
Specifying your own host name
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To specify a host name (for example, s3.domain.name), provide your own
|
||||
`config.json <https://github.com/scality/cloudserver/blob/master/config.json>`__
|
||||
file using `Docker volumes <https://docs.docker.com/storage/volumes/>`__.
|
||||
To specify a host name (e.g. s3.domain.name), you can provide your own
|
||||
`config.json <https://github.com/scality/S3/blob/master/config.json>`__
|
||||
using `Docker
|
||||
Volume <https://docs.docker.com/engine/tutorials/dockervolumes/>`__.
|
||||
|
||||
First, add a new key-value pair to the restEndpoints section of your
|
||||
config.json. Make the key the host name you want, and the value the default
|
||||
location\_constraint for this endpoint.
|
||||
First add a new key-value pair in the restEndpoints section of your
|
||||
config.json. The key in the key-value pair should be the host name you
|
||||
would like to add and the value is the default location\_constraint for
|
||||
this endpoint.
|
||||
|
||||
For example, ``s3.example.com`` is mapped to ``us-east-1`` which is one
|
||||
of the ``location_constraints`` listed in your locationConfig.json file
|
||||
`here <https://github.com/scality/S3/blob/master/locationConfig.json>`__.
|
||||
|
||||
For more information about location configuration, see:
|
||||
`GETTING STARTED <GETTING_STARTED.html#location-configuration>`__
|
||||
More information about location configuration
|
||||
`here <https://github.com/scality/S3/blob/master/README.md#location-configuration>`__
|
||||
|
||||
.. code:: json
|
||||
|
||||
|
@ -276,33 +266,33 @@ For more information about location configuration, see:
|
|||
"localhost": "file",
|
||||
"127.0.0.1": "file",
|
||||
...
|
||||
"cloudserver.example.com": "us-east-1"
|
||||
"s3.example.com": "us-east-1"
|
||||
},
|
||||
|
||||
Next, run CloudServer using a `Docker volume
|
||||
<https://docs.docker.com/engine/tutorials/dockervolumes/>`__:
|
||||
Then, run your Scality S3 Server using `Docker
|
||||
Volume <https://docs.docker.com/engine/tutorials/dockervolumes/>`__:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code:: shell
|
||||
|
||||
$ docker run -v $(pwd)/config.json:/usr/src/app/config.json -p 8000:8000 -d zenko/cloudserver
|
||||
docker run -v $(pwd)/config.json:/usr/src/app/config.json -p 8000:8000 -d scality/s3server
|
||||
|
||||
The local ``config.json`` file overrides the default one through a Docker
|
||||
file mapping.
|
||||
Your local ``config.json`` file will override the default one through a
|
||||
docker file mapping.
|
||||
|
||||
Running as an Unprivileged User
|
||||
Running as an unprivileged user
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
CloudServer runs as root by default.
|
||||
Zenko CloudServer runs as root by default.
|
||||
|
||||
To change this, modify the dockerfile and specify a user before the
|
||||
entry point.
|
||||
You can change that by modifing the dockerfile and specifying a user
|
||||
before the entrypoint.
|
||||
|
||||
The user must exist within the container, and must own the
|
||||
/usr/src/app directory for CloudServer to run.
|
||||
The user needs to exist within the container, and own the folder
|
||||
**/usr/src/app** for Scality Zenko CloudServer to run properly.
|
||||
|
||||
For example, the following dockerfile lines can be modified:
|
||||
For instance, you can modify these lines in the dockerfile:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code:: shell
|
||||
|
||||
...
|
||||
&& groupadd -r -g 1001 scality \
|
||||
|
@ -314,58 +304,54 @@ For example, the following dockerfile lines can be modified:
|
|||
USER scality
|
||||
ENTRYPOINT ["/usr/src/app/docker-entrypoint.sh"]
|
||||
|
||||
.. _continuous-integration-with-docker-hosted-cloudserver:
|
||||
Continuous integration with Docker hosted CloudServer
|
||||
-----------------------------------------------------
|
||||
|
||||
Continuous Integration with a Docker-Hosted CloudServer
|
||||
-------------------------------------------------------
|
||||
When you start the Docker Scality Zenko CloudServer image, you can adjust the
|
||||
configuration of the Scality Zenko CloudServer instance by passing one or more
|
||||
environment variables on the docker run command line.
|
||||
|
||||
When you start the Docker CloudServer image, you can adjust the
|
||||
configuration of the CloudServer instance by passing one or more
|
||||
environment variables on the ``docker run`` command line.
|
||||
Sample ways to run it for CI are:
|
||||
|
||||
- With custom locations (one in-memory, one hosted on AWS), and custom
|
||||
credentials mounted:
|
||||
|
||||
To run CloudServer for CI with custom locations (one in-memory,
|
||||
one hosted on AWS), and custom credentials mounted:
|
||||
.. code:: shell
|
||||
|
||||
.. code-block:: shell
|
||||
docker run --name CloudServer -p 8000:8000
|
||||
-v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json
|
||||
-v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json
|
||||
-v ~/.aws/credentials:/root/.aws/credentials
|
||||
-e S3DATA=multiple -e S3BACKEND=mem scality/s3server
|
||||
|
||||
$ docker run --name CloudServer -p 8000:8000 \
|
||||
-v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json \
|
||||
-v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json \
|
||||
-v ~/.aws/credentials:/root/.aws/credentials \
|
||||
-e S3DATA=multiple -e S3BACKEND=mem zenko/cloudserver
|
||||
- With custom locations, (one in-memory, one hosted on AWS, one file),
|
||||
and custom credentials set as environment variables
|
||||
(see `this section <#scality-access-key-id-and-scality-secret-access-key>`__):
|
||||
|
||||
To run CloudServer for CI with custom locations, (one in-memory, one
|
||||
hosted on AWS, and one file), and custom credentials `set as environment
|
||||
variables <GETTING_STARTED.html#scality-access-key-id-and-scality-secret-access-key>`__):
|
||||
.. code:: shell
|
||||
|
||||
.. code-block:: shell
|
||||
docker run --name CloudServer -p 8000:8000
|
||||
-v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json
|
||||
-v ~/.aws/credentials:/root/.aws/credentials
|
||||
-v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata
|
||||
-e SCALITY_ACCESS_KEY_ID=accessKey1
|
||||
-e SCALITY_SECRET_ACCESS_KEY=verySecretKey1
|
||||
-e S3DATA=multiple -e S3BACKEND=mem scality/s3server
|
||||
|
||||
$ docker run --name CloudServer -p 8000:8000 \
|
||||
-v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json \
|
||||
-v ~/.aws/credentials:/root/.aws/credentials \
|
||||
-v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata \
|
||||
-e SCALITY_ACCESS_KEY_ID=accessKey1 \
|
||||
-e SCALITY_SECRET_ACCESS_KEY=verySecretKey1 \
|
||||
-e S3DATA=multiple -e S3BACKEND=mem zenko/cloudserver
|
||||
In production with Docker hosted CloudServer
|
||||
--------------------------------------------
|
||||
|
||||
.. _in-production-w-a-Docker-hosted-cloudserver:
|
||||
In production, we expect that data will be persistent, that you will use the
|
||||
multiple backends capabilities of Zenko CloudServer, and that you will have a
|
||||
custom endpoint for your local storage, and custom credentials for your local
|
||||
storage:
|
||||
|
||||
In Production with a Docker-Hosted CloudServer
|
||||
----------------------------------------------
|
||||
.. code:: shell
|
||||
|
||||
Because data must persist in production settings, CloudServer offers
|
||||
multiple-backend capabilities. This requires a custom endpoint
|
||||
and custom credentials for local storage.
|
||||
|
||||
Customize these with:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ docker run -d --name CloudServer \
|
||||
-v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata \
|
||||
-v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json \
|
||||
-v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json \
|
||||
-v ~/.aws/credentials:/root/.aws/credentials -e S3DATA=multiple \
|
||||
-e ENDPOINT=custom.endpoint.com \
|
||||
-p 8000:8000 -d zenko/cloudserver \
|
||||
docker run -d --name CloudServer
|
||||
-v $(pwd)/data:/usr/src/app/localData -v $(pwd)/metadata:/usr/src/app/localMetadata
|
||||
-v $(pwd)/locationConfig.json:/usr/src/app/locationConfig.json
|
||||
-v $(pwd)/authdata.json:/usr/src/app/conf/authdata.json
|
||||
-v ~/.aws/credentials:/root/.aws/credentials -e S3DATA=multiple
|
||||
-e ENDPOINT=custom.endpoint.com
|
||||
-p 8000:8000 -d scality/s3server
|
||||
|
|
|
@ -1,221 +1,214 @@
|
|||
Getting Started
|
||||
===============
|
||||
=================
|
||||
|
||||
.. figure:: ../res/scality-cloudserver-logo.png
|
||||
:alt: Zenko CloudServer logo
|
||||
|
||||
|
||||
Dependencies
|
||||
------------
|
||||
|
||||
Building and running the Scality Zenko CloudServer requires node.js 10.x and
|
||||
yarn v1.17.x. Up-to-date versions can be found at
|
||||
`Nodesource <https://github.com/nodesource/distributions>`__.
|
||||
|CircleCI| |Scality CI|
|
||||
|
||||
Installation
|
||||
------------
|
||||
|
||||
1. Clone the source code
|
||||
Dependencies
|
||||
~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: shell
|
||||
Building and running the Scality Zenko CloudServer requires node.js 6.9.5 and
|
||||
npm v3 . Up-to-date versions can be found at
|
||||
`Nodesource <https://github.com/nodesource/distributions>`__.
|
||||
|
||||
$ git clone https://github.com/scality/cloudserver.git
|
||||
|
||||
2. Go to the cloudserver directory and use yarn to install the js dependencies.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ cd cloudserver
|
||||
$ yarn install
|
||||
|
||||
Running CloudServer with a File Backend
|
||||
---------------------------------------
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ yarn start
|
||||
|
||||
This starts a Zenko CloudServer on port 8000. Two additional ports, 9990
|
||||
and 9991, are also open locally for internal transfer of metadata and
|
||||
data, respectively.
|
||||
|
||||
The default access key is accessKey1. The secret key is verySecretKey1.
|
||||
|
||||
By default, metadata files are saved in the localMetadata directory and
|
||||
data files are saved in the localData directory in the local ./cloudserver
|
||||
directory. These directories are pre-created within the repository. To
|
||||
save data or metadata in different locations, you must specify them using
|
||||
absolute paths. Thus, when starting the server:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ mkdir -m 700 $(pwd)/myFavoriteDataPath
|
||||
$ mkdir -m 700 $(pwd)/myFavoriteMetadataPath
|
||||
$ export S3DATAPATH="$(pwd)/myFavoriteDataPath"
|
||||
$ export S3METADATAPATH="$(pwd)/myFavoriteMetadataPath"
|
||||
$ yarn start
|
||||
|
||||
Running CloudServer with Multiple Data Backends
|
||||
-----------------------------------------------
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ export S3DATA='multiple'
|
||||
$ yarn start
|
||||
|
||||
This starts a Zenko CloudServer on port 8000.
|
||||
|
||||
The default access key is accessKey1. The secret key is verySecretKey1.
|
||||
|
||||
With multiple backends, you can choose where each object is saved by setting
|
||||
the following header with a location constraint in a PUT request:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
'x-amz-meta-scal-location-constraint':'myLocationConstraint'
|
||||
|
||||
If no header is sent with a PUT object request, the bucket’s location
|
||||
constraint determines where the data is saved. If the bucket has no
|
||||
location constraint, the endpoint of the PUT request determines location.
|
||||
|
||||
See the Configuration_ section to set location constraints.
|
||||
|
||||
Run CloudServer with an In-Memory Backend
|
||||
-----------------------------------------
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ yarn run mem_backend
|
||||
|
||||
This starts a Zenko CloudServer on port 8000.
|
||||
|
||||
The default access key is accessKey1. The secret key is verySecretKey1.
|
||||
|
||||
Run CloudServer with Vault User Management
|
||||
------------------------------------------
|
||||
Clone source code
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code:: shell
|
||||
|
||||
export S3VAULT=vault
|
||||
yarn start
|
||||
git clone https://github.com/scality/S3.git
|
||||
|
||||
Note: Vault is proprietary and must be accessed separately.
|
||||
This starts a Zenko CloudServer using Vault for user management.
|
||||
Install js dependencies
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Run CloudServer for Continuous Integration Testing or in Production with Docker
|
||||
-------------------------------------------------------------------------------
|
||||
Go to the ./S3 folder,
|
||||
|
||||
Run Cloudserver with `DOCKER <DOCKER.html>`__
|
||||
.. code:: shell
|
||||
|
||||
Testing
|
||||
~~~~~~~
|
||||
npm install
|
||||
|
||||
Run unit tests with the command:
|
||||
Run it with a file backend
|
||||
--------------------------
|
||||
|
||||
.. code-block:: shell
|
||||
.. code:: shell
|
||||
|
||||
$ yarn test
|
||||
npm start
|
||||
|
||||
Run multiple-backend unit tests with:
|
||||
This starts an Zenko CloudServer on port 8000. Two additional ports 9990 and
|
||||
9991 are also open locally for internal transfer of metadata and data,
|
||||
respectively.
|
||||
|
||||
.. code-block:: shell
|
||||
The default access key is accessKey1 with a secret key of
|
||||
verySecretKey1.
|
||||
|
||||
$ CI=true S3DATA=multiple yarn start
|
||||
$ yarn run multiple_backend_test
|
||||
By default the metadata files will be saved in the localMetadata
|
||||
directory and the data files will be saved in the localData directory
|
||||
within the ./S3 directory on your machine. These directories have been
|
||||
pre-created within the repository. If you would like to save the data or
|
||||
metadata in different locations of your choice, you must specify them
|
||||
with absolute paths. So, when starting the server:
|
||||
|
||||
Run the linter with:
|
||||
.. code:: shell
|
||||
|
||||
.. code-block:: shell
|
||||
mkdir -m 700 $(pwd)/myFavoriteDataPath
|
||||
mkdir -m 700 $(pwd)/myFavoriteMetadataPath
|
||||
export S3DATAPATH="$(pwd)/myFavoriteDataPath"
|
||||
export S3METADATAPATH="$(pwd)/myFavoriteMetadataPath"
|
||||
npm start
|
||||
|
||||
$ yarn run lint
|
||||
Run it with multiple data backends
|
||||
----------------------------------
|
||||
|
||||
Running Functional Tests Locally
|
||||
.. code:: shell
|
||||
|
||||
export S3DATA='multiple'
|
||||
npm start
|
||||
|
||||
This starts an Zenko CloudServer on port 8000. The default access key is
|
||||
accessKey1 with a secret key of verySecretKey1.
|
||||
|
||||
With multiple backends, you have the ability to choose where each object
|
||||
will be saved by setting the following header with a locationConstraint
|
||||
on a PUT request:
|
||||
|
||||
.. code:: shell
|
||||
|
||||
'x-amz-meta-scal-location-constraint':'myLocationConstraint'
|
||||
|
||||
If no header is sent with a PUT object request, the location constraint
|
||||
of the bucket will determine where the data is saved. If the bucket has
|
||||
no location constraint, the endpoint of the PUT request will be used to
|
||||
determine location.
|
||||
|
||||
See the Configuration section below to learn how to set location
|
||||
constraints.
|
||||
|
||||
Run it with an in-memory backend
|
||||
--------------------------------
|
||||
|
||||
To pass AWS and Azure backend tests locally, modify
|
||||
tests/locationConfig/locationConfigTests.json so that ``awsbackend``
|
||||
specifies the bucketname of a bucket you have access to based on your
|
||||
credentials, and modify ``azurebackend`` with details for your Azure account.
|
||||
.. code:: shell
|
||||
|
||||
npm run mem_backend
|
||||
|
||||
This starts an Zenko CloudServer on port 8000. The default access key is
|
||||
accessKey1 with a secret key of verySecretKey1.
|
||||
|
||||
Run it for continuous integration testing or in production with Docker
|
||||
----------------------------------------------------------------------
|
||||
|
||||
`DOCKER <../DOCKER/>`__
|
||||
|
||||
Testing
|
||||
-------
|
||||
|
||||
You can run the unit tests with the following command:
|
||||
|
||||
.. code:: shell
|
||||
|
||||
npm test
|
||||
|
||||
You can run the multiple backend unit tests with:
|
||||
|
||||
.. code:: shell
|
||||
CI=true S3DATA=multiple npm start
|
||||
npm run multiple_backend_test
|
||||
|
||||
You can run the linter with:
|
||||
|
||||
.. code:: shell
|
||||
|
||||
npm run lint
|
||||
|
||||
Running functional tests locally:
|
||||
|
||||
For the AWS backend and Azure backend tests to pass locally,
|
||||
you must modify tests/locationConfigTests.json so that awsbackend
|
||||
specifies a bucketname of a bucket you have access to based on
|
||||
your credentials profile and modify "azurebackend" with details
|
||||
for your Azure account.
|
||||
|
||||
The test suite requires additional tools, **s3cmd** and **Redis**
|
||||
installed in the environment the tests are running in.
|
||||
|
||||
1. Install `s3cmd <http://s3tools.org/download>`__
|
||||
- Install `s3cmd <http://s3tools.org/download>`__
|
||||
- Install `redis <https://redis.io/download>`__ and start Redis.
|
||||
- Add localCache section to your ``config.json``:
|
||||
|
||||
2. Install `redis <https://redis.io/download>`__ and start Redis.
|
||||
::
|
||||
|
||||
3. Add localCache section to ``config.json``:
|
||||
|
||||
.. code:: json
|
||||
|
||||
"localCache": {
|
||||
"localCache": {
|
||||
"host": REDIS_HOST,
|
||||
"port": REDIS_PORT
|
||||
}
|
||||
}
|
||||
|
||||
where ``REDIS_HOST`` is the Redis instance IP address (``"127.0.0.1"``
|
||||
if Redis is running locally) and ``REDIS_PORT`` is the Redis instance
|
||||
port (``6379`` by default)
|
||||
where ``REDIS_HOST`` is your Redis instance IP address (``"127.0.0.1"``
|
||||
if your Redis is running locally) and ``REDIS_PORT`` is your Redis
|
||||
instance port (``6379`` by default)
|
||||
|
||||
4. Add the following to the local etc/hosts file:
|
||||
- Add the following to the etc/hosts file on your machine:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code:: shell
|
||||
|
||||
127.0.0.1 bucketwebsitetester.s3-website-us-east-1.amazonaws.com
|
||||
127.0.0.1 bucketwebsitetester.s3-website-us-east-1.amazonaws.com
|
||||
|
||||
5. Start Zenko CloudServer in memory and run the functional tests:
|
||||
- Start the Zenko CloudServer in memory and run the functional tests:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code:: shell
|
||||
|
||||
$ CI=true yarn run mem_backend
|
||||
$ CI=true yarn run ft_test
|
||||
|
||||
.. _Configuration:
|
||||
CI=true npm run mem_backend
|
||||
CI=true npm run ft_test
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
There are three configuration files for Zenko CloudServer:
|
||||
There are three configuration files for your Scality Zenko CloudServer:
|
||||
|
||||
* ``conf/authdata.json``, for authentication.
|
||||
1. ``conf/authdata.json``, described above for authentication
|
||||
|
||||
* ``locationConfig.json``, to configure where data is saved.
|
||||
2. ``locationConfig.json``, to set up configuration options for
|
||||
|
||||
* ``config.json``, for general configuration options.
|
||||
where data will be saved
|
||||
|
||||
.. _location-configuration:
|
||||
3. ``config.json``, for general configuration options
|
||||
|
||||
Location Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You must specify at least one locationConstraint in locationConfig.json
|
||||
(or leave it as pre-configured).
|
||||
You must specify at least one locationConstraint in your
|
||||
locationConfig.json (or leave as pre-configured).
|
||||
|
||||
You must also specify 'us-east-1' as a locationConstraint. If you put a
|
||||
bucket to an unknown endpoint and do not specify a locationConstraint in
|
||||
the PUT bucket call, us-east-1 is used.
|
||||
You must also specify 'us-east-1' as a locationConstraint so if you only
|
||||
define one locationConstraint, that would be it. If you put a bucket to
|
||||
an unknown endpoint and do not specify a locationConstraint in the put
|
||||
bucket call, us-east-1 will be used.
|
||||
|
||||
For instance, the following locationConstraint saves data sent to
|
||||
For instance, the following locationConstraint will save data sent to
|
||||
``myLocationConstraint`` to the file backend:
|
||||
|
||||
.. code:: json
|
||||
|
||||
"myLocationConstraint": {
|
||||
"type": "file",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
"myLocationConstraint": {
|
||||
"type": "file",
|
||||
"legacyAwsBehavior": false,
|
||||
"details": {}
|
||||
},
|
||||
|
||||
Each locationConstraint must include the ``type``, ``legacyAwsBehavior``,
|
||||
and ``details`` keys. ``type`` indicates which backend is used for that
|
||||
region. Supported backends are mem, file, and scality.``legacyAwsBehavior``
|
||||
indicates whether the region behaves the same as the AWS S3 'us-east-1'
|
||||
region. If the locationConstraint type is ``scality``, ``details`` must
|
||||
contain connector information for sproxyd. If the locationConstraint type
|
||||
is ``mem`` or ``file``, ``details`` must be empty.
|
||||
Each locationConstraint must include the ``type``,
|
||||
``legacyAwsBehavior``, and ``details`` keys. ``type`` indicates which
|
||||
backend will be used for that region. Currently, mem, file, and scality
|
||||
are the supported backends. ``legacyAwsBehavior`` indicates whether the
|
||||
region will have the same behavior as the AWS S3 'us-east-1' region. If
|
||||
the locationConstraint type is scality, ``details`` should contain
|
||||
connector information for sproxyd. If the locationConstraint type is mem
|
||||
or file, ``details`` should be empty.
|
||||
|
||||
Once locationConstraints is set in locationConfig.json, specify a default
|
||||
locationConstraint for each endpoint.
|
||||
Once you have your locationConstraints in your locationConfig.json, you
|
||||
can specify a default locationConstraint for each of your endpoints.
|
||||
|
||||
For instance, the following sets the ``localhost`` endpoint to the
|
||||
``myLocationConstraint`` data backend defined above:
|
||||
|
@ -226,24 +219,26 @@ For instance, the following sets the ``localhost`` endpoint to the
|
|||
"localhost": "myLocationConstraint"
|
||||
},
|
||||
|
||||
To use an endpoint other than localhost for Zenko CloudServer, the endpoint
|
||||
must be listed in ``restEndpoints``. Otherwise, if the server is running
|
||||
with a:
|
||||
If you would like to use an endpoint other than localhost for your
|
||||
Scality Zenko CloudServer, that endpoint MUST be listed in your
|
||||
``restEndpoints``. Otherwise if your server is running with a:
|
||||
|
||||
* **file backend**: The default location constraint is ``file``
|
||||
* **memory backend**: The default location constraint is ``mem``
|
||||
- **file backend**: your default location constraint will be ``file``
|
||||
|
||||
- **memory backend**: your default location constraint will be ``mem``
|
||||
|
||||
Endpoints
|
||||
~~~~~~~~~
|
||||
|
||||
The Zenko CloudServer supports endpoints that are rendered in either:
|
||||
Note that our Zenko CloudServer supports both:
|
||||
|
||||
* path style: http://myhostname.com/mybucket or
|
||||
* hosted style: http://mybucket.myhostname.com
|
||||
- path-style: http://myhostname.com/mybucket
|
||||
- hosted-style: http://mybucket.myhostname.com
|
||||
|
||||
However, if an IP address is specified for the host, hosted-style requests
|
||||
cannot reach the server. Use path-style requests in that case. For example,
|
||||
if you are using the AWS SDK for JavaScript, instantiate your client like this:
|
||||
However, hosted-style requests will not hit the server if you are using
|
||||
an ip address for your host. So, make sure you are using path-style
|
||||
requests in that case. For instance, if you are using the AWS SDK for
|
||||
JavaScript, you would instantiate your client like this:
|
||||
|
||||
.. code:: js
|
||||
|
||||
|
@ -252,99 +247,87 @@ if you are using the AWS SDK for JavaScript, instantiate your client like this:
|
|||
s3ForcePathStyle: true,
|
||||
});
|
||||
|
||||
Setting Your Own Access and Secret Key Pairs
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
Setting your own access key and secret key pairs
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Credentials can be set for many accounts by editing ``conf/authdata.json``,
|
||||
but use the ``SCALITY_ACCESS_KEY_ID`` and ``SCALITY_SECRET_ACCESS_KEY``
|
||||
environment variables to specify your own credentials.
|
||||
|
||||
_`scality-access-key-id-and-scality-secret-access-key`
|
||||
You can set credentials for many accounts by editing
|
||||
``conf/authdata.json`` but if you want to specify one set of your own
|
||||
credentials, you can use ``SCALITY_ACCESS_KEY_ID`` and
|
||||
``SCALITY_SECRET_ACCESS_KEY`` environment variables.
|
||||
|
||||
SCALITY\_ACCESS\_KEY\_ID and SCALITY\_SECRET\_ACCESS\_KEY
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
These variables specify authentication credentials for an account named
|
||||
“CustomAccount”.
|
||||
"CustomAccount".
|
||||
|
||||
.. note:: Anything in the ``authdata.json`` file is ignored.
|
||||
Note: Anything in the ``authdata.json`` file will be ignored.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code:: shell
|
||||
|
||||
$ SCALITY_ACCESS_KEY_ID=newAccessKey SCALITY_SECRET_ACCESS_KEY=newSecretKey yarn start
|
||||
SCALITY_ACCESS_KEY_ID=newAccessKey SCALITY_SECRET_ACCESS_KEY=newSecretKey npm start
|
||||
|
||||
.. _Using_SSL:
|
||||
|
||||
Using SSL
|
||||
~~~~~~~~~
|
||||
Scality with SSL
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To use https with your local CloudServer, you must set up
|
||||
SSL certificates.
|
||||
If you wish to use https with your local Zenko CloudServer, you need to set up
|
||||
SSL certificates. Here is a simple guide of how to do it.
|
||||
|
||||
1. Deploy CloudServer using `our DockerHub page
|
||||
<https://hub.docker.com/r/zenko/cloudserver/>`__ (run it with a file
|
||||
backend).
|
||||
Deploying Zenko CloudServer
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. Note:: If Docker is not installed locally, follow the
|
||||
`instructions to install it for your distribution
|
||||
<https://docs.docker.com/engine/installation/>`__
|
||||
First, you need to deploy **Zenko CloudServer**. This can be done very easily
|
||||
via `our **DockerHub**
|
||||
page <https://hub.docker.com/r/scality/s3server/>`__ (you want to run it
|
||||
with a file backend).
|
||||
|
||||
2. Update the CloudServer container’s config
|
||||
*Note:* *- If you don't have docker installed on your machine, here
|
||||
are the `instructions to install it for your
|
||||
distribution <https://docs.docker.com/engine/installation/>`__*
|
||||
|
||||
Add your certificates to your container. To do this,
|
||||
#. exec inside the CloudServer container.
|
||||
Updating your Zenko CloudServer container's config
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
#. Run ``$> docker ps`` to find the container’s ID (the corresponding
|
||||
image name is ``scality/cloudserver``.
|
||||
|
||||
#. Copy the corresponding container ID (``894aee038c5e`` in the present
|
||||
example), and run:
|
||||
You're going to add your certificates to your container. In order to do
|
||||
so, you need to exec inside your Zenko CloudServer container. Run a
|
||||
``$> docker ps`` and find your container's id (the corresponding image
|
||||
name should be ``scality/s3server``. Copy the corresponding container id
|
||||
(here we'll use ``894aee038c5e``, and run:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code:: sh
|
||||
|
||||
$> docker exec -it 894aee038c5e bash
|
||||
$> docker exec -it 894aee038c5e bash
|
||||
|
||||
This puts you inside your container, using an interactive terminal.
|
||||
You're now inside your container, using an interactive terminal :)
|
||||
|
||||
3. Generate the SSL key and certificates. The paths where the different
|
||||
files are stored are defined after the ``-out`` option in each of the
|
||||
following commands.
|
||||
Generate SSL key and certificates
|
||||
**********************************
|
||||
|
||||
#. Generate a private key for your certificate signing request (CSR):
|
||||
There are 5 steps to this generation. The paths where the different
|
||||
files are stored are defined after the ``-out`` option in each command
|
||||
|
||||
.. code-block:: shell
|
||||
.. code:: sh
|
||||
|
||||
$> openssl genrsa -out ca.key 2048
|
||||
# Generate a private key for your CSR
|
||||
$> openssl genrsa -out ca.key 2048
|
||||
# Generate a self signed certificate for your local Certificate Authority
|
||||
$> openssl req -new -x509 -extensions v3_ca -key ca.key -out ca.crt -days 99999 -subj "/C=US/ST=Country/L=City/O=Organization/CN=scality.test"
|
||||
|
||||
#. Generate a self-signed certificate for your local certificate
|
||||
authority (CA):
|
||||
# Generate a key for Zenko CloudServer
|
||||
$> openssl genrsa -out test.key 2048
|
||||
# Generate a Certificate Signing Request for S3 Server
|
||||
$> openssl req -new -key test.key -out test.csr -subj "/C=US/ST=Country/L=City/O=Organization/CN=*.scality.test"
|
||||
# Generate a local-CA-signed certificate for S3 Server
|
||||
$> openssl x509 -req -in test.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out test.crt -days 99999 -sha256
|
||||
|
||||
.. code:: shell
|
||||
Update Zenko CloudServer ``config.json``
|
||||
**********************************
|
||||
|
||||
$> openssl req -new -x509 -extensions v3_ca -key ca.key -out ca.crt -days 99999 -subj "/C=US/ST=Country/L=City/O=Organization/CN=scality.test"
|
||||
Add a ``certFilePaths`` section to ``./config.json`` with the
|
||||
appropriate paths:
|
||||
|
||||
#. Generate a key for the CloudServer:
|
||||
|
||||
.. code:: shell
|
||||
|
||||
$> openssl genrsa -out test.key 2048
|
||||
|
||||
#. Generate a CSR for CloudServer:
|
||||
|
||||
.. code:: shell
|
||||
|
||||
$> openssl req -new -key test.key -out test.csr -subj "/C=US/ST=Country/L=City/O=Organization/CN=*.scality.test"
|
||||
|
||||
#. Generate a certificate for CloudServer signed by the local CA:
|
||||
|
||||
.. code:: shell
|
||||
|
||||
$> openssl x509 -req -in test.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out test.crt -days 99999 -sha256
|
||||
|
||||
4. Update Zenko CloudServer ``config.json``. Add a ``certFilePaths``
|
||||
section to ``./config.json`` with appropriate paths:
|
||||
|
||||
.. code:: json
|
||||
.. code:: json
|
||||
|
||||
"certFilePaths": {
|
||||
"key": "./test.key",
|
||||
|
@ -352,36 +335,42 @@ SSL certificates.
|
|||
"ca": "./ca.crt"
|
||||
}
|
||||
|
||||
5. Run your container with the new config.
|
||||
Run your container with the new config
|
||||
****************************************
|
||||
|
||||
#. Exit the container by running ``$> exit``.
|
||||
First, you need to exit your container. Simply run ``$> exit``. Then,
|
||||
you need to restart your container. Normally, a simple
|
||||
``$> docker restart s3server`` should do the trick.
|
||||
|
||||
#. Restart the container with ``$> docker restart cloudserver``.
|
||||
Update your host config
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
6. Update the host configuration by adding s3.scality.test
|
||||
to /etc/hosts:
|
||||
Associates local IP addresses with hostname
|
||||
*******************************************
|
||||
|
||||
.. code:: bash
|
||||
In your ``/etc/hosts`` file on Linux, OS X, or Unix (with root
|
||||
permissions), edit the line of localhost so it looks like this:
|
||||
|
||||
127.0.0.1 localhost s3.scality.test
|
||||
::
|
||||
|
||||
7. Copy the local certificate authority (ca.crt in step 4) from your
|
||||
container. Choose the path to save this file to (in the present
|
||||
example, ``/root/ca.crt``), and run:
|
||||
127.0.0.1 localhost s3.scality.test
|
||||
|
||||
.. code:: shell
|
||||
Copy the local certificate authority from your container
|
||||
*********************************************************
|
||||
|
||||
$> docker cp 894aee038c5e:/usr/src/app/ca.crt /root/ca.crt
|
||||
In the above commands, it's the file named ``ca.crt``. Choose the path
|
||||
you want to save this file at (here we chose ``/root/ca.crt``), and run
|
||||
something like:
|
||||
|
||||
.. note:: Your container ID will be different, and your path to
|
||||
ca.crt may be different.
|
||||
.. code:: sh
|
||||
|
||||
Test the Config
|
||||
^^^^^^^^^^^^^^^
|
||||
$> docker cp 894aee038c5e:/usr/src/app/ca.crt /root/ca.crt
|
||||
|
||||
If aws-sdk is not installed, run ``$> yarn install aws-sdk``.
|
||||
Test your config
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
Paste the following script into a file named "test.js":
|
||||
If you do not have aws-sdk installed, run ``$> npm install aws-sdk``. In
|
||||
a ``test.js`` file, paste the following script:
|
||||
|
||||
.. code:: js
|
||||
|
||||
|
@ -421,13 +410,8 @@ Paste the following script into a file named "test.js":
|
|||
});
|
||||
});
|
||||
|
||||
Now run this script with:
|
||||
|
||||
.. code::
|
||||
|
||||
$> nodejs test.js
|
||||
|
||||
On success, the script outputs ``SSL is cool!``.
|
||||
Now run that script with ``$> nodejs test.js``. If all goes well, it
|
||||
should output ``SSL is cool!``. Enjoy that added security!
|
||||
|
||||
|
||||
.. |CircleCI| image:: https://circleci.com/gh/scality/S3.svg?style=svg
|
||||
|
|
|
@ -1,7 +1,6 @@
|
|||
# S3 Healthcheck
|
||||
|
||||
Scality S3 exposes a healthcheck route `/live` on the port used
|
||||
for the metrics (defaults to port 8002) which returns a
|
||||
Scality S3 exposes a healthcheck route `/_/healthcheck` which returns a
|
||||
response with HTTP code
|
||||
|
||||
- 200 OK
|
|
@ -4,415 +4,479 @@ Integrations
|
|||
High Availability
|
||||
=================
|
||||
|
||||
`Docker Swarm <https://docs.docker.com/engine/swarm/>`__ is a clustering tool
|
||||
developed by Docker for use with its containers. It can be used to start
|
||||
services, which we define to ensure CloudServer's continuous availability to
|
||||
end users. A swarm defines a manager and *n* workers among *n* + 1 servers.
|
||||
|
||||
This tutorial shows how to perform a basic setup with three servers, which
|
||||
provides strong service resiliency, while remaining easy to use and
|
||||
maintain. We will use NFS through Docker to share data and
|
||||
`Docker swarm <https://docs.docker.com/engine/swarm/>`__ is a
|
||||
clustering tool developped by Docker and ready to use with its
|
||||
containers. It allows to start a service, which we define and use as a
|
||||
means to ensure Zenko CloudServer's continuous availability to the end user.
|
||||
Indeed, a swarm defines a manager and n workers among n+1 servers. We
|
||||
will do a basic setup in this tutorial, with just 3 servers, which
|
||||
already provides a strong service resiliency, whilst remaining easy to
|
||||
do as an individual. We will use NFS through docker to share data and
|
||||
metadata between the different servers.
|
||||
|
||||
Sections are labeled **On Server**, **On Clients**, or
|
||||
**On All Machines**, referring respectively to NFS server, NFS clients, or
|
||||
NFS server and clients. In the present example, the server’s IP address is
|
||||
**10.200.15.113** and the client IP addresses are **10.200.15.96** and
|
||||
**10.200.15.97**
|
||||
You will see that the steps of this tutorial are defined as **On
|
||||
Server**, **On Clients**, **On All Machines**. This refers respectively
|
||||
to NFS Server, NFS Clients, or NFS Server and Clients. In our example,
|
||||
the IP of the Server will be **10.200.15.113**, while the IPs of the
|
||||
Clients will be **10.200.15.96 and 10.200.15.97**
|
||||
|
||||
1. Install Docker (on All Machines)
|
||||
Installing docker
|
||||
-----------------
|
||||
|
||||
Docker 17.03.0-ce is used for this tutorial. Docker 1.12.6 and later will
|
||||
likely work, but is not tested.
|
||||
Any version from docker 1.12.6 onwards should work; we used Docker
|
||||
17.03.0-ce for this tutorial.
|
||||
|
||||
* On Ubuntu 14.04
|
||||
Install Docker CE for Ubuntu as `documented at Docker
|
||||
<https://docs.docker.com/install/linux/docker-ce/ubuntu/>`__.
|
||||
Install the aufs dependency as recommended by Docker. The required
|
||||
commands are:
|
||||
On All Machines
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
.. code:: sh
|
||||
On Ubuntu 14.04
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
$> sudo apt-get update
|
||||
$> sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual
|
||||
$> sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
|
||||
$> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
|
||||
$> sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
|
||||
$> sudo apt-get update
|
||||
$> sudo apt-get install docker-ce
|
||||
|
||||
* On CentOS 7
|
||||
Install Docker CE as `documented at Docker
|
||||
<https://docs.docker.com/install/linux/docker-ce/centos/>`__.
|
||||
The required commands are:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> sudo yum install -y yum-utils
|
||||
$> sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
|
||||
$> sudo yum makecache fast
|
||||
$> sudo yum install docker-ce
|
||||
$> sudo systemctl start docker
|
||||
|
||||
2. Install NFS on Client(s)
|
||||
|
||||
NFS clients mount Docker volumes over the NFS server’s shared folders.
|
||||
If the NFS commons are installed, manual mounts are no longer needed.
|
||||
|
||||
* On Ubuntu 14.04
|
||||
|
||||
Install the NFS commons with apt-get:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> sudo apt-get install nfs-common
|
||||
|
||||
* On CentOS 7
|
||||
|
||||
Install the NFS utils; then start required services:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> yum install nfs-utils
|
||||
$> sudo systemctl enable rpcbind
|
||||
$> sudo systemctl enable nfs-server
|
||||
$> sudo systemctl enable nfs-lock
|
||||
$> sudo systemctl enable nfs-idmap
|
||||
$> sudo systemctl start rpcbind
|
||||
$> sudo systemctl start nfs-server
|
||||
$> sudo systemctl start nfs-lock
|
||||
$> sudo systemctl start nfs-idmap
|
||||
|
||||
3. Install NFS (on Server)
|
||||
|
||||
The NFS server hosts the data and metadata. The package(s) to install on it
|
||||
differs from the package installed on the clients.
|
||||
|
||||
* On Ubuntu 14.04
|
||||
|
||||
Install the NFS server-specific package and the NFS commons:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> sudo apt-get install nfs-kernel-server nfs-common
|
||||
|
||||
* On CentOS 7
|
||||
|
||||
Install the NFS utils and start the required services:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> yum install nfs-utils
|
||||
$> sudo systemctl enable rpcbind
|
||||
$> sudo systemctl enable nfs-server
|
||||
$> sudo systemctl enable nfs-lock
|
||||
$> sudo systemctl enable nfs-idmap
|
||||
$> sudo systemctl start rpcbind
|
||||
$> sudo systemctl start nfs-server
|
||||
$> sudo systemctl start nfs-lock
|
||||
$> sudo systemctl start nfs-idmap
|
||||
|
||||
For both distributions:
|
||||
|
||||
#. Choose where shared data and metadata from the local
|
||||
`CloudServer <http://www.zenko.io/cloudserver/>`__ shall be stored (The
|
||||
present example uses /var/nfs/data and /var/nfs/metadata). Set permissions
|
||||
for these folders for
|
||||
sharing over NFS:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> mkdir -p /var/nfs/data /var/nfs/metadata
|
||||
$> chmod -R 777 /var/nfs/
|
||||
|
||||
#. The /etc/exports file configures network permissions and r-w-x permissions
|
||||
for NFS access. Edit /etc/exports, adding the following lines:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
/var/nfs/data 10.200.15.96(rw,sync,no_root_squash) 10.200.15.97(rw,sync,no_root_squash)
|
||||
/var/nfs/metadata 10.200.15.96(rw,sync,no_root_squash) 10.200.15.97(rw,sync,no_root_squash)
|
||||
|
||||
Ubuntu applies the no\_subtree\_check option by default, so both
|
||||
folders are declared with the same permissions, even though they’re in
|
||||
the same tree.
|
||||
|
||||
#. Export this new NFS table:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> sudo exportfs -a
|
||||
|
||||
#. Edit the ``MountFlags`` option in the Docker config in
|
||||
/lib/systemd/system/docker.service to enable NFS mount from Docker volumes
|
||||
on other machines:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
MountFlags=shared
|
||||
|
||||
#. Restart the NFS server and Docker daemons to apply these changes.
|
||||
|
||||
* On Ubuntu 14.04
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> sudo service nfs-kernel-server restart
|
||||
$> sudo service docker restart
|
||||
|
||||
* On CentOS 7
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> sudo systemctl restart nfs-server
|
||||
$> sudo systemctl daemon-reload
|
||||
$> sudo systemctl restart docker
|
||||
|
||||
|
||||
4. Set Up a Docker Swarm
|
||||
|
||||
* On all machines and distributions:
|
||||
|
||||
Set up the Docker volumes to be mounted to the NFS server for CloudServer’s
|
||||
data and metadata storage. The following commands must be replicated on all
|
||||
machines:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> docker volume create --driver local --opt type=nfs --opt o=addr=10.200.15.113,rw --opt device=:/var/nfs/data --name data
|
||||
$> docker volume create --driver local --opt type=nfs --opt o=addr=10.200.15.113,rw --opt device=:/var/nfs/metadata --name metadata
|
||||
|
||||
There is no need to ``docker exec`` these volumes to mount them: the
|
||||
Docker Swarm manager does this when the Docker service is started.
|
||||
|
||||
* On a server:
|
||||
|
||||
To start a Docker service on a Docker Swarm cluster, initialize the cluster
|
||||
(that is, define a manager), prompt workers/nodes to join in, and then start
|
||||
the service.
|
||||
|
||||
Initialize the swarm cluster, and review its response:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> docker swarm init --advertise-addr 10.200.15.113
|
||||
|
||||
Swarm initialized: current node (db2aqfu3bzfzzs9b1kfeaglmq) is now a manager.
|
||||
|
||||
To add a worker to this swarm, run the following command:
|
||||
|
||||
docker swarm join \
|
||||
--token SWMTKN-1-5yxxencrdoelr7mpltljn325uz4v6fe1gojl14lzceij3nujzu-2vfs9u6ipgcq35r90xws3stka \
|
||||
10.200.15.113:2377
|
||||
|
||||
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
|
||||
|
||||
* On clients:
|
||||
|
||||
Copy and paste the command provided by your Docker Swarm init. A successful
|
||||
request/response will resemble:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> docker swarm join --token SWMTKN-1-5yxxencrdoelr7mpltljn325uz4v6fe1gojl14lzceij3nujzu-2vfs9u6ipgcq35r90xws3stka 10.200.15.113:2377
|
||||
|
||||
This node joined a swarm as a worker.
|
||||
|
||||
Set Up Docker Swarm on Clients on a Server
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Start the service on the Swarm cluster.
|
||||
The docker website has `solid
|
||||
documentation <https://docs.docker.com/engine/installation/linux/ubuntu/>`__.
|
||||
We have chosen to install the aufs dependency, as recommended by Docker.
|
||||
Here are the required commands:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> docker service create --name s3 --replicas 1 --mount type=volume,source=data,target=/usr/src/app/localData --mount type=volume,source=metadata,target=/usr/src/app/localMetadata -p 8000:8000 scality/cloudserver
|
||||
$> sudo apt-get update
|
||||
$> sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual
|
||||
$> sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
|
||||
$> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
|
||||
$> sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
|
||||
$> sudo apt-get update
|
||||
$> sudo apt-get install docker-ce
|
||||
|
||||
On a successful installation, ``docker service ls`` returns the following
|
||||
output:
|
||||
On CentOS 7
|
||||
^^^^^^^^^^^
|
||||
|
||||
The docker website has `solid
|
||||
documentation <https://docs.docker.com/engine/installation/linux/centos/>`__.
|
||||
Here are the required commands:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> sudo yum install -y yum-utils
|
||||
$> sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
|
||||
$> sudo yum makecache fast
|
||||
$> sudo yum install docker-ce
|
||||
$> sudo systemctl start docker
|
||||
|
||||
Configure NFS
|
||||
-------------
|
||||
|
||||
On Clients
|
||||
~~~~~~~~~~
|
||||
|
||||
Your NFS Clients will mount Docker volumes over your NFS Server's shared
|
||||
folders. Hence, you don't have to mount anything manually, you just have
|
||||
to install the NFS commons:
|
||||
|
||||
On Ubuntu 14.04
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
Simply install the NFS commons:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> sudo apt-get install nfs-common
|
||||
|
||||
On CentOS 7
|
||||
^^^^^^^^^^^
|
||||
|
||||
Install the NFS utils, and then start the required services:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> yum install nfs-utils
|
||||
$> sudo systemctl enable rpcbind
|
||||
$> sudo systemctl enable nfs-server
|
||||
$> sudo systemctl enable nfs-lock
|
||||
$> sudo systemctl enable nfs-idmap
|
||||
$> sudo systemctl start rpcbind
|
||||
$> sudo systemctl start nfs-server
|
||||
$> sudo systemctl start nfs-lock
|
||||
$> sudo systemctl start nfs-idmap
|
||||
|
||||
On Server
|
||||
~~~~~~~~~
|
||||
|
||||
Your NFS Server will be the machine to physically host the data and
|
||||
metadata. The package(s) we will install on it is slightly different
|
||||
from the one we installed on the clients.
|
||||
|
||||
On Ubuntu 14.04
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
Install the NFS server specific package and the NFS commons:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> sudo apt-get install nfs-kernel-server nfs-common
|
||||
|
||||
On CentOS 7
|
||||
^^^^^^^^^^^
|
||||
|
||||
Same steps as with the client: install the NFS utils and start the
|
||||
required services:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> yum install nfs-utils
|
||||
$> sudo systemctl enable rpcbind
|
||||
$> sudo systemctl enable nfs-server
|
||||
$> sudo systemctl enable nfs-lock
|
||||
$> sudo systemctl enable nfs-idmap
|
||||
$> sudo systemctl start rpcbind
|
||||
$> sudo systemctl start nfs-server
|
||||
$> sudo systemctl start nfs-lock
|
||||
$> sudo systemctl start nfs-idmap
|
||||
|
||||
On Ubuntu 14.04 and CentOS 7
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Choose where your shared data and metadata from your local `Zenko CloudServer
|
||||
<http://www.zenko.io/cloudserver/>`__ will be stored.
|
||||
We chose to go with /var/nfs/data and /var/nfs/metadata. You also need
|
||||
to set proper sharing permissions for these folders as they'll be shared
|
||||
over NFS:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> mkdir -p /var/nfs/data /var/nfs/metadata
|
||||
$> chmod -R 777 /var/nfs/
|
||||
|
||||
Now you need to update your **/etc/exports** file. This is the file that
|
||||
configures network permissions and rwx permissions for NFS access. By
|
||||
default, Ubuntu applies the no\_subtree\_check option, so we declared
|
||||
both folders with the same permissions, even though they're in the same
|
||||
tree:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> sudo vim /etc/exports
|
||||
|
||||
In this file, add the following lines:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
/var/nfs/data 10.200.15.96(rw,sync,no_root_squash) 10.200.15.97(rw,sync,no_root_squash)
|
||||
/var/nfs/metadata 10.200.15.96(rw,sync,no_root_squash) 10.200.15.97(rw,sync,no_root_squash)
|
||||
|
||||
Export this new NFS table:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> sudo exportfs -a
|
||||
|
||||
Eventually, you need to allow for NFS mount from Docker volumes on other
|
||||
machines. You need to change the Docker config in
|
||||
**/lib/systemd/system/docker.service**:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> sudo vim /lib/systemd/system/docker.service
|
||||
|
||||
In this file, change the **MountFlags** option:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
MountFlags=shared
|
||||
|
||||
Now you just need to restart the NFS server and docker daemons so your
|
||||
changes apply.
|
||||
|
||||
On Ubuntu 14.04
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
Restart your NFS Server and docker services:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> sudo service nfs-kernel-server restart
|
||||
$> sudo service docker restart
|
||||
|
||||
On CentOS 7
|
||||
^^^^^^^^^^^
|
||||
|
||||
Restart your NFS Server and docker daemons:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> sudo systemctl restart nfs-server
|
||||
$> sudo systemctl daemon-reload
|
||||
$> sudo systemctl restart docker
|
||||
|
||||
Set up your Docker Swarm service
|
||||
--------------------------------
|
||||
|
||||
On All Machines
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
On Ubuntu 14.04 and CentOS 7
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
We will now set up the Docker volumes that will be mounted to the NFS
|
||||
Server and serve as data and metadata storage for Zenko CloudServer. These two
|
||||
commands have to be replicated on all machines:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> docker volume create --driver local --opt type=nfs --opt o=addr=10.200.15.113,rw --opt device=:/var/nfs/data --name data
|
||||
$> docker volume create --driver local --opt type=nfs --opt o=addr=10.200.15.113,rw --opt device=:/var/nfs/metadata --name metadata
|
||||
|
||||
There is no need to ""docker exec" these volumes to mount them: the
|
||||
Docker Swarm manager will do it when the Docker service will be started.
|
||||
|
||||
On Server
|
||||
^^^^^^^^^
|
||||
|
||||
To start a Docker service on a Docker Swarm cluster, you first have to
|
||||
initialize that cluster (i.e.: define a manager), then have the
|
||||
workers/nodes join in, and then start the service. Initialize the swarm
|
||||
cluster, and look at the response:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> docker swarm init --advertise-addr 10.200.15.113
|
||||
|
||||
Swarm initialized: current node (db2aqfu3bzfzzs9b1kfeaglmq) is now a manager.
|
||||
|
||||
To add a worker to this swarm, run the following command:
|
||||
|
||||
docker swarm join \
|
||||
--token SWMTKN-1-5yxxencrdoelr7mpltljn325uz4v6fe1gojl14lzceij3nujzu-2vfs9u6ipgcq35r90xws3stka \
|
||||
10.200.15.113:2377
|
||||
|
||||
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
|
||||
|
||||
On Clients
|
||||
^^^^^^^^^^
|
||||
|
||||
Simply copy/paste the command provided by your docker swarm init. When
|
||||
all goes well, you'll get something like this:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> docker swarm join --token SWMTKN-1-5yxxencrdoelr7mpltljn325uz4v6fe1gojl14lzceij3nujzu-2vfs9u6ipgcq35r90xws3stka 10.200.15.113:2377
|
||||
|
||||
This node joined a swarm as a worker.
|
||||
|
||||
On Server
|
||||
^^^^^^^^^
|
||||
|
||||
Start the service on your swarm cluster!
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> docker service create --name s3 --replicas 1 --mount type=volume,source=data,target=/usr/src/app/localData --mount type=volume,source=metadata,target=/usr/src/app/localMetadata -p 8000:8000 scality/s3server
|
||||
|
||||
If you run a docker service ls, you should have the following output:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> docker service ls
|
||||
ID NAME MODE REPLICAS IMAGE
|
||||
ocmggza412ft s3 replicated 1/1 scality/cloudserver:latest
|
||||
ocmggza412ft s3 replicated 1/1 scality/s3server:latest
|
||||
|
||||
If the service does not start, consider disabling apparmor/SELinux.
|
||||
If your service won't start, consider disabling apparmor/SELinux.
|
||||
|
||||
Testing the High-Availability CloudServer
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
Testing your High Availability S3Server
|
||||
---------------------------------------
|
||||
|
||||
On all machines (client/server) and distributions (Ubuntu and CentOS),
|
||||
determine where CloudServer is running using ``docker ps``. CloudServer can
|
||||
operate on any node of the Swarm cluster, manager or worker. When you find
|
||||
it, you can kill it with ``docker stop <container id>``. It will respawn
|
||||
on a different node. Now, if one server falls, or if Docker stops
|
||||
unexpectedly, the end user will still be able to access your the local CloudServer.
|
||||
|
||||
Troubleshooting
|
||||
On All Machines
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
To troubleshoot the service, run:
|
||||
On Ubuntu 14.04 and CentOS 7
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Try to find out where your Scality Zenko CloudServer is actually running using
|
||||
the **docker ps** command. It can be on any node of the swarm cluster,
|
||||
manager or worker. When you find it, you can kill it, with **docker stop
|
||||
<container id>** and you'll see it respawn on a different node of the
|
||||
swarm cluster. Now you see, if one of your servers falls, or if docker
|
||||
stops unexpectedly, your end user will still be able to access your
|
||||
local Zenko CloudServer.
|
||||
|
||||
Troubleshooting
|
||||
---------------
|
||||
|
||||
To troubleshoot the service you can run:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> docker service ps s3docker service ps s3
|
||||
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
|
||||
0ar81cw4lvv8chafm8pw48wbc s3.1 scality/cloudserver localhost.localdomain.localdomain Running Running 7 days ago
|
||||
cvmf3j3bz8w6r4h0lf3pxo6eu \_ s3.1 scality/cloudserver localhost.localdomain.localdomain Shutdown Failed 7 days ago "task: non-zero exit (137)"
|
||||
0ar81cw4lvv8chafm8pw48wbc s3.1 scality/s3server localhost.localdomain.localdomain Running Running 7 days ago
|
||||
cvmf3j3bz8w6r4h0lf3pxo6eu \_ s3.1 scality/s3server localhost.localdomain.localdomain Shutdown Failed 7 days ago "task: non-zero exit (137)"
|
||||
|
||||
If the error is truncated, view the error in detail by inspecting the
|
||||
Docker task ID:
|
||||
If the error is truncated it is possible to have a more detailed view of
|
||||
the error by inspecting the docker task ID:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> docker inspect cvmf3j3bz8w6r4h0lf3pxo6eu
|
||||
|
||||
Off you go!
|
||||
~~~~~~~~~~~
|
||||
-----------
|
||||
|
||||
Let us know what you use this functionality for, and if you'd like any
|
||||
specific developments around it. Or, even better: come and contribute to
|
||||
our `Github repository <https://github.com/scality/s3/>`__! We look
|
||||
forward to meeting you!
|
||||
|
||||
Let us know how you use this and if you'd like any specific developments
|
||||
around it. Even better: come and contribute to our `Github repository
|
||||
<https://github.com/scality/s3/>`__! We look forward to meeting you!
|
||||
|
||||
S3FS
|
||||
====
|
||||
|
||||
You can export buckets as a filesystem with s3fs on CloudServer.
|
||||
Export your buckets as a filesystem with s3fs on top of Zenko CloudServer
|
||||
|
||||
`s3fs <https://github.com/s3fs-fuse/s3fs-fuse>`__ is an open source
|
||||
tool, available both on Debian and RedHat distributions, that enables
|
||||
you to mount an S3 bucket on a filesystem-like backend. This tutorial uses
|
||||
an Ubuntu 14.04 host to deploy and use s3fs over CloudServer.
|
||||
tool that allows you to mount an S3 bucket on a filesystem-like backend.
|
||||
It is available both on Debian and RedHat distributions. For this
|
||||
tutorial, we used an Ubuntu 14.04 host to deploy and use s3fs over
|
||||
Scality's Zenko CloudServer.
|
||||
|
||||
Deploying Zenko CloudServer with SSL
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
----------------------------
|
||||
|
||||
First, deploy CloudServer with a file backend using `our DockerHub page
|
||||
<https://hub.docker.com/r/zenko/cloudserver>`__.
|
||||
First, you need to deploy **Zenko CloudServer**. This can be done very easily
|
||||
via `our DockerHub
|
||||
page <https://hub.docker.com/r/scality/s3server/>`__ (you want to run it
|
||||
with a file backend).
|
||||
|
||||
.. note::
|
||||
*Note:* *- If you don't have docker installed on your machine, here
|
||||
are the `instructions to install it for your
|
||||
distribution <https://docs.docker.com/engine/installation/>`__*
|
||||
|
||||
If Docker is not installed on your machine, follow
|
||||
`these instructions <https://docs.docker.com/engine/installation/>`__
|
||||
to install it for your distribution.
|
||||
You also necessarily have to set up SSL with Zenko CloudServer to use s3fs. We
|
||||
have a nice
|
||||
`tutorial <https://s3.scality.com/v1.0/page/scality-with-ssl>`__ to help
|
||||
you do it.
|
||||
|
||||
You must also set up SSL with CloudServer to use s3fs. See `Using SSL
|
||||
<./GETTING_STARTED#Using_SSL>`__ for instructions.
|
||||
|
||||
s3fs Setup
|
||||
~~~~~~~~~~
|
||||
s3fs setup
|
||||
----------
|
||||
|
||||
Installing s3fs
|
||||
---------------
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
Follow the instructions in the s3fs `README
|
||||
<https://github.com/s3fs-fuse/s3fs-fuse/blob/master/README.md#installation-from-pre-built-packages>`__,
|
||||
|
||||
Check that s3fs is properly installed. A version check should return
|
||||
a response resembling:
|
||||
s3fs has quite a few dependencies. As explained in their
|
||||
`README <https://github.com/s3fs-fuse/s3fs-fuse/blob/master/README.md#installation>`__,
|
||||
the following commands should install everything for Ubuntu 14.04:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> s3fs --version
|
||||
$> sudo apt-get install automake autotools-dev g++ git libcurl4-gnutls-dev
|
||||
$> sudo apt-get install libfuse-dev libssl-dev libxml2-dev make pkg-config
|
||||
|
||||
Now you want to install s3fs per se:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> git clone https://github.com/s3fs-fuse/s3fs-fuse.git
|
||||
$> cd s3fs-fuse
|
||||
$> ./autogen.sh
|
||||
$> ./configure
|
||||
$> make
|
||||
$> sudo make install
|
||||
|
||||
Check that s3fs is properly installed by checking its version. it should
|
||||
answer as below:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> s3fs --version
|
||||
|
||||
Amazon Simple Storage Service File System V1.80(commit:d40da2c) with OpenSSL
|
||||
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
|
||||
License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>
|
||||
This is free software: you are free to change and redistribute it.
|
||||
There is NO WARRANTY, to the extent permitted by law.
|
||||
|
||||
Configuring s3fs
|
||||
----------------
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
s3fs expects you to provide it with a password file. Our file is
|
||||
``/etc/passwd-s3fs``. The structure for this file is
|
||||
``ACCESSKEYID:SECRETKEYID``, so, for CloudServer, you can run:
|
||||
``ACCESSKEYID:SECRETKEYID``, so, for S3Server, you can run:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> echo 'accessKey1:verySecretKey1' > /etc/passwd-s3fs
|
||||
$> chmod 600 /etc/passwd-s3fs
|
||||
|
||||
Using CloudServer with s3fs
|
||||
---------------------------
|
||||
Using Zenko CloudServer with s3fs
|
||||
------------------------
|
||||
|
||||
1. Use /mnt/tests3fs as a mount point.
|
||||
First, you're going to need a mountpoint; we chose ``/mnt/tests3fs``:
|
||||
|
||||
.. code:: sh
|
||||
.. code:: sh
|
||||
|
||||
$> mkdir /mnt/tests3fs
|
||||
|
||||
2. Create a bucket on your local CloudServer. In the present example it is
|
||||
named “tests3fs”.
|
||||
Then, you want to create a bucket on your local Zenko CloudServer; we named it
|
||||
``tests3fs``:
|
||||
|
||||
.. code:: sh
|
||||
.. code:: sh
|
||||
|
||||
$> s3cmd mb s3://tests3fs
|
||||
|
||||
3. Mount the bucket to your mount point with s3fs:
|
||||
*Note:* *- If you've never used s3cmd with our Zenko CloudServer, our README
|
||||
provides you with a `recommended
|
||||
config <https://github.com/scality/S3/blob/master/README.md#s3cmd>`__*
|
||||
|
||||
.. code:: sh
|
||||
Now you can mount your bucket to your mountpoint with s3fs:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> s3fs tests3fs /mnt/tests3fs -o passwd_file=/etc/passwd-s3fs -o url="https://s3.scality.test:8000/" -o use_path_request_style
|
||||
|
||||
The structure of this command is:
|
||||
``s3fs BUCKET_NAME PATH/TO/MOUNTPOINT -o OPTIONS``. Of these mandatory
|
||||
options:
|
||||
*If you're curious, the structure of this command is*
|
||||
``s3fs BUCKET_NAME PATH/TO/MOUNTPOINT -o OPTIONS``\ *, and the
|
||||
options are mandatory and serve the following purposes:
|
||||
* ``passwd_file``\ *: specifiy path to password file;
|
||||
* ``url``\ *: specify the hostname used by your SSL provider;
|
||||
* ``use_path_request_style``\ *: force path style (by default, s3fs
|
||||
uses subdomains (DNS style)).*
|
||||
|
||||
* ``passwd_file`` specifies the path to the password file.
|
||||
* ``url`` specifies the host name used by your SSL provider.
|
||||
* ``use_path_request_style`` forces the path style (by default,
|
||||
s3fs uses DNS-style subdomains).
|
||||
| From now on, you can either add files to your mountpoint, or add
|
||||
objects to your bucket, and they'll show in the other.
|
||||
| For example, let's' create two files, and then a directory with a file
|
||||
in our mountpoint:
|
||||
|
||||
Once the bucket is mounted, files added to the mount point or
|
||||
objects added to the bucket will appear in both locations.
|
||||
.. code:: sh
|
||||
|
||||
Example
|
||||
-------
|
||||
$> touch /mnt/tests3fs/file1 /mnt/tests3fs/file2
|
||||
$> mkdir /mnt/tests3fs/dir1
|
||||
$> touch /mnt/tests3fs/dir1/file3
|
||||
|
||||
Create two files, and then a directory with a file in our mount point:
|
||||
Now, I can use s3cmd to show me what is actually in S3Server:
|
||||
|
||||
.. code:: sh
|
||||
.. code:: sh
|
||||
|
||||
$> touch /mnt/tests3fs/file1 /mnt/tests3fs/file2
|
||||
$> mkdir /mnt/tests3fs/dir1
|
||||
$> touch /mnt/tests3fs/dir1/file3
|
||||
$> s3cmd ls -r s3://tests3fs
|
||||
|
||||
Now, use s3cmd to show what is in CloudServer:
|
||||
2017-02-28 17:28 0 s3://tests3fs/dir1/
|
||||
2017-02-28 17:29 0 s3://tests3fs/dir1/file3
|
||||
2017-02-28 17:28 0 s3://tests3fs/file1
|
||||
2017-02-28 17:28 0 s3://tests3fs/file2
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> s3cmd ls -r s3://tests3fs
|
||||
|
||||
2017-02-28 17:28 0 s3://tests3fs/dir1/
|
||||
2017-02-28 17:29 0 s3://tests3fs/dir1/file3
|
||||
2017-02-28 17:28 0 s3://tests3fs/file1
|
||||
2017-02-28 17:28 0 s3://tests3fs/file2
|
||||
|
||||
Now you can enjoy a filesystem view on your local CloudServer.
|
||||
Now you can enjoy a filesystem view on your local Zenko CloudServer!
|
||||
|
||||
|
||||
Duplicity
|
||||
=========
|
||||
|
||||
How to back up your files with CloudServer.
|
||||
How to backup your files with Zenko CloudServer.
|
||||
|
||||
Installing Duplicity and its Dependencies
|
||||
Installing
|
||||
-----------
|
||||
|
||||
Installing Duplicity and its dependencies
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To install `Duplicity <http://duplicity.nongnu.org/>`__,
|
||||
go to `this site <https://code.launchpad.net/duplicity/0.7-series>`__.
|
||||
Download the latest tarball. Decompress it and follow the instructions
|
||||
in the README.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> tar zxvf duplicity-0.7.11.tar.gz
|
||||
$> cd duplicity-0.7.11
|
||||
$> python setup.py install
|
||||
|
||||
You may receive error messages indicating the need to install some or all
|
||||
of the following dependencies:
|
||||
Second, you want to install
|
||||
`Duplicity <http://duplicity.nongnu.org/index.html>`__. You have to
|
||||
download `this
|
||||
tarball <https://code.launchpad.net/duplicity/0.7-series/0.7.11/+download/duplicity-0.7.11.tar.gz>`__,
|
||||
decompress it, and then checkout the README inside, which will give you
|
||||
a list of dependencies to install. If you're using Ubuntu 14.04, this is
|
||||
your lucky day: here is a lazy step by step install.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
|
@ -420,20 +484,30 @@ of the following dependencies:
|
|||
$> apt-get install python-dev python-pip python-lockfile
|
||||
$> pip install -U boto
|
||||
|
||||
Testing the Installation
|
||||
------------------------
|
||||
Then you want to actually install Duplicity:
|
||||
|
||||
1. Check that CloudServer is running. Run ``$> docker ps``. You should
|
||||
see one container named ``scality/cloudserver``. If you do not, run
|
||||
``$> docker start cloudserver`` and check again.
|
||||
.. code:: sh
|
||||
|
||||
$> tar zxvf duplicity-0.7.11.tar.gz
|
||||
$> cd duplicity-0.7.11
|
||||
$> python setup.py install
|
||||
|
||||
2. Duplicity uses a module called “Boto” to send requests to S3. Boto
|
||||
requires a configuration file located in ``/etc/boto.cfg`` to store
|
||||
your credentials and preferences. A minimal configuration
|
||||
you can fine tune `following these instructions
|
||||
<http://boto.cloudhackers.com/en/latest/getting_started.html>`__ is
|
||||
shown here:
|
||||
Using
|
||||
------
|
||||
|
||||
Testing your installation
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
First, we're just going to quickly check that Zenko CloudServer is actually
|
||||
running. To do so, simply run ``$> docker ps`` . You should see one
|
||||
container named ``scality/s3server``. If that is not the case, try
|
||||
``$> docker start s3server``, and check again.
|
||||
|
||||
Secondly, as you probably know, Duplicity uses a module called **Boto**
|
||||
to send requests to S3. Boto requires a configuration file located in
|
||||
**``/etc/boto.cfg``** to have your credentials and preferences. Here is
|
||||
a minimalistic config `that you can finetune following these
|
||||
instructions <http://boto.cloudhackers.com/en/latest/getting_started.html>`__.
|
||||
|
||||
::
|
||||
|
||||
|
@ -447,51 +521,54 @@ Testing the Installation
|
|||
# If using SSL, unmute and provide absolute path to local CA certificate
|
||||
# ca_certificates_file = /absolute/path/to/ca.crt
|
||||
|
||||
.. note:: To set up SSL with CloudServer, check out our `Using SSL
|
||||
<./GETTING_STARTED#Using_SSL>`__ in GETTING STARTED.
|
||||
*Note:* *If you want to set up SSL with Zenko CloudServer, check out our
|
||||
`tutorial <http://link/to/SSL/tutorial>`__*
|
||||
|
||||
3. At this point all requirements to run CloudServer as a backend to Duplicity
|
||||
have been met. A local folder/file should back up to the local S3.
|
||||
Try it with the decompressed Duplicity folder:
|
||||
At this point, we've met all the requirements to start running Zenko CloudServer
|
||||
as a backend to Duplicity. So we should be able to back up a local
|
||||
folder/file to local S3. Let's try with the duplicity decompressed
|
||||
folder:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$> duplicity duplicity-0.7.11 "s3://127.0.0.1:8000/testbucket/"
|
||||
|
||||
.. note:: Duplicity will prompt for a symmetric encryption passphrase.
|
||||
Save it carefully, as you will need it to recover your data.
|
||||
Alternatively, you can add the ``--no-encryption`` flag
|
||||
and the data will be stored plain.
|
||||
*Note:* *Duplicity will prompt you for a symmetric encryption
|
||||
passphrase. Save it somewhere as you will need it to recover your
|
||||
data. Alternatively, you can also add the ``--no-encryption`` flag
|
||||
and the data will be stored plain.*
|
||||
|
||||
If this command is successful, you will receive an output resembling:
|
||||
If this command is succesful, you will get an output looking like this:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
--------------[ Backup Statistics ]--------------
|
||||
StartTime 1486486547.13 (Tue Feb 7 16:55:47 2017)
|
||||
EndTime 1486486547.40 (Tue Feb 7 16:55:47 2017)
|
||||
ElapsedTime 0.27 (0.27 seconds)
|
||||
SourceFiles 388
|
||||
SourceFileSize 6634529 (6.33 MB)
|
||||
NewFiles 388
|
||||
NewFileSize 6634529 (6.33 MB)
|
||||
DeletedFiles 0
|
||||
ChangedFiles 0
|
||||
ChangedFileSize 0 (0 bytes)
|
||||
ChangedDeltaSize 0 (0 bytes)
|
||||
DeltaEntries 388
|
||||
RawDeltaSize 6392865 (6.10 MB)
|
||||
TotalDestinationSizeChange 2003677 (1.91 MB)
|
||||
Errors 0
|
||||
-------------------------------------------------
|
||||
--------------[ Backup Statistics ]--------------
|
||||
StartTime 1486486547.13 (Tue Feb 7 16:55:47 2017)
|
||||
EndTime 1486486547.40 (Tue Feb 7 16:55:47 2017)
|
||||
ElapsedTime 0.27 (0.27 seconds)
|
||||
SourceFiles 388
|
||||
SourceFileSize 6634529 (6.33 MB)
|
||||
NewFiles 388
|
||||
NewFileSize 6634529 (6.33 MB)
|
||||
DeletedFiles 0
|
||||
ChangedFiles 0
|
||||
ChangedFileSize 0 (0 bytes)
|
||||
ChangedDeltaSize 0 (0 bytes)
|
||||
DeltaEntries 388
|
||||
RawDeltaSize 6392865 (6.10 MB)
|
||||
TotalDestinationSizeChange 2003677 (1.91 MB)
|
||||
Errors 0
|
||||
-------------------------------------------------
|
||||
|
||||
Congratulations! You can now back up to your local S3 through Duplicity.
|
||||
Congratulations! You can now backup to your local S3 through duplicity
|
||||
:)
|
||||
|
||||
Automating Backups
|
||||
------------------
|
||||
Automating backups
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The easiest way to back up files periodically is to write a bash script
|
||||
and add it to your crontab. A suggested script follows.
|
||||
Now you probably want to back up your files periodically. The easiest
|
||||
way to do this is to write a bash script and add it to your crontab.
|
||||
Here is my suggestion for such a file:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
|
@ -500,33 +577,33 @@ and add it to your crontab. A suggested script follows.
|
|||
# Export your passphrase so you don't have to type anything
|
||||
export PASSPHRASE="mypassphrase"
|
||||
|
||||
# To use a GPG key, put it here and uncomment the line below
|
||||
# If you want to use a GPG Key, put it here and unmute the line below
|
||||
#GPG_KEY=
|
||||
|
||||
# Define your backup bucket, with localhost specified
|
||||
DEST="s3://127.0.0.1:8000/testbucketcloudserver/"
|
||||
DEST="s3://127.0.0.1:8000/testbuckets3server/"
|
||||
|
||||
# Define the absolute path to the folder to back up
|
||||
# Define the absolute path to the folder you want to backup
|
||||
SOURCE=/root/testfolder
|
||||
|
||||
# Set to "full" for full backups, and "incremental" for incremental backups
|
||||
# Warning: you must perform one full backup befor you can perform
|
||||
# Warning: you have to perform one full backup befor you can perform
|
||||
# incremental ones on top of it
|
||||
FULL=incremental
|
||||
|
||||
# How long to keep backups. If you don't want to delete old backups, keep
|
||||
# this value empty; otherwise, the syntax is "1Y" for one year, "1M" for
|
||||
# one month, "1D" for one day.
|
||||
# How long to keep backups for; if you don't want to delete old
|
||||
# backups, keep empty; otherwise, syntax is "1Y" for one year, "1M"
|
||||
# for one month, "1D" for one day
|
||||
OLDER_THAN="1Y"
|
||||
|
||||
# is_running checks whether Duplicity is currently completing a task
|
||||
# is_running checks whether duplicity is currently completing a task
|
||||
is_running=$(ps -ef | grep duplicity | grep python | wc -l)
|
||||
|
||||
# If Duplicity is already completing a task, this will not run
|
||||
# If duplicity is already completing a task, this will simply not run
|
||||
if [ $is_running -eq 0 ]; then
|
||||
echo "Backup for ${SOURCE} started"
|
||||
|
||||
# To delete backups older than a certain time, do it here
|
||||
# If you want to delete backups older than a certain time, we do it here
|
||||
if [ "$OLDER_THAN" != "" ]; then
|
||||
echo "Removing backups older than ${OLDER_THAN}"
|
||||
duplicity remove-older-than ${OLDER_THAN} ${DEST}
|
||||
|
@ -549,17 +626,17 @@ and add it to your crontab. A suggested script follows.
|
|||
# Forget the passphrase...
|
||||
unset PASSPHRASE
|
||||
|
||||
Put this file in ``/usr/local/sbin/backup.sh``. Run ``crontab -e`` and
|
||||
paste your configuration into the file that opens. If you're unfamiliar
|
||||
with Cron, here is a good `HowTo
|
||||
<https://help.ubuntu.com/community/CronHowto>`__. If the folder being
|
||||
backed up is a folder to be modified permanently during the work day,
|
||||
we can set incremental backups every 5 minutes from 8 AM to 9 PM Monday
|
||||
through Friday by pasting the following line into crontab:
|
||||
So let's say you put this file in ``/usr/local/sbin/backup.sh.`` Next
|
||||
you want to run ``crontab -e`` and paste your configuration in the file
|
||||
that opens. If you're unfamiliar with Cron, here is a good `How
|
||||
To <https://help.ubuntu.com/community/CronHowto>`__. The folder I'm
|
||||
backing up is a folder I modify permanently during my workday, so I want
|
||||
incremental backups every 5mn from 8AM to 9PM monday to friday. Here is
|
||||
the line I will paste in my crontab:
|
||||
|
||||
.. code:: sh
|
||||
.. code:: cron
|
||||
|
||||
*/5 8-20 * * 1-5 /usr/local/sbin/backup.sh
|
||||
|
||||
Adding or removing files from the folder being backed up will result in
|
||||
incremental backups in the bucket.
|
||||
Now I can try and add / remove files from the folder I'm backing up, and
|
||||
I will see incremental backups in my bucket.
|
||||
|
|
|
@ -1,263 +0,0 @@
|
|||
Metadata Search Documentation
|
||||
=============================
|
||||
|
||||
Description
|
||||
-----------
|
||||
|
||||
This feature enables metadata search to be performed on the metadata of objects
|
||||
stored in Zenko.
|
||||
|
||||
Requirements
|
||||
------------
|
||||
|
||||
* MongoDB
|
||||
|
||||
Design
|
||||
------
|
||||
|
||||
The Metadata Search feature expands on the existing :code:`GET Bucket` S3 API by
|
||||
enabling users to conduct metadata searches by adding the custom Zenko query
|
||||
string parameter, :code:`search`. The :code:`search` parameter is structured as a pseudo
|
||||
SQL WHERE clause, and supports basic SQL operators. For example:
|
||||
:code:`"A=1 AND B=2 OR C=3"` (complex queries can be built using nesting
|
||||
operators, :code:`(` and :code:`)`).
|
||||
|
||||
The search process is as follows:
|
||||
|
||||
* Zenko receives a :code:`GET` request.
|
||||
|
||||
.. code::
|
||||
|
||||
# regular getBucket request
|
||||
GET /bucketname HTTP/1.1
|
||||
Host: 127.0.0.1:8000
|
||||
Date: Wed, 18 Oct 2018 17:50:00 GMT
|
||||
Authorization: authorization string
|
||||
|
||||
# getBucket versions request
|
||||
GET /bucketname?versions HTTP/1.1
|
||||
Host: 127.0.0.1:8000
|
||||
Date: Wed, 18 Oct 2018 17:50:00 GMT
|
||||
Authorization: authorization string
|
||||
|
||||
# search getBucket request
|
||||
GET /bucketname?search=key%3Dsearch-item HTTP/1.1
|
||||
Host: 127.0.0.1:8000
|
||||
Date: Wed, 18 Oct 2018 17:50:00 GMT
|
||||
Authorization: authorization string
|
||||
|
||||
* If the request does *not* contain the :code:`search` query parameter, Zenko performs
|
||||
a normal bucket listing and returns an XML result containing the list of
|
||||
objects.
|
||||
* If the request *does* contain the :code:`search` query parameter, Zenko parses and
|
||||
validates the search string.
|
||||
|
||||
- If the search string is invalid, Zenko returns an :code:`InvalidArgument` error.
|
||||
|
||||
.. code::
|
||||
|
||||
<?xml version=\"1.0\" encoding=\"UTF-8\"?>
|
||||
<Error>
|
||||
<Code>InvalidArgument</Code>
|
||||
<Message>Invalid sql where clause sent as search query</Message>
|
||||
<Resource></Resource>
|
||||
<RequestId>d1d6afc64345a8e1198e</RequestId>
|
||||
</Error>
|
||||
|
||||
- If the search string is valid, Zenko parses it and generates an abstract
|
||||
syntax tree (AST). The AST is then passed to the MongoDB backend to be
|
||||
used as the query filter for retrieving objects from a bucket that
|
||||
satisfies the requested search conditions. Zenko parses the filtered
|
||||
results and returns them as the response.
|
||||
|
||||
Metadata search results have the same structure as a :code:`GET Bucket` response:
|
||||
|
||||
.. code:: xml
|
||||
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
||||
<Name>bucketname</Name>
|
||||
<Prefix/>
|
||||
<Marker/>
|
||||
<MaxKeys>1000</MaxKeys>
|
||||
<IsTruncated>false</IsTruncated>
|
||||
<Contents>
|
||||
<Key>objectKey</Key>
|
||||
<LastModified>2018-04-19T18:31:49.426Z</LastModified>
|
||||
<ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag>
|
||||
<Size>0</Size>
|
||||
<Owner>
|
||||
<ID>79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be</ID>
|
||||
<DisplayName>Bart</DisplayName>
|
||||
</Owner>
|
||||
<StorageClass>STANDARD</StorageClass>
|
||||
</Contents>
|
||||
<Contents>
|
||||
...
|
||||
</Contents>
|
||||
</ListBucketResult>
|
||||
|
||||
Performing Metadata Searches with Zenko
|
||||
---------------------------------------
|
||||
|
||||
You can perform metadata searches by:
|
||||
|
||||
+ Using the :code:`search_bucket` tool in the
|
||||
`Scality/S3 <https://github.com/scality/S3>`_ GitHub repository.
|
||||
+ Creating a signed HTTP request to Zenko in your preferred programming
|
||||
language.
|
||||
|
||||
Using the S3 Tool
|
||||
+++++++++++++++++
|
||||
|
||||
After cloning the `Scality/S3 <https://github.com/scality/S3>`_ GitHub repository
|
||||
and installing the necessary dependencies, run the following command in the S3
|
||||
project’s root directory to access the search tool:
|
||||
|
||||
.. code::
|
||||
|
||||
node bin/search_bucket
|
||||
|
||||
This generates the following output:
|
||||
|
||||
.. code::
|
||||
|
||||
Usage: search_bucket [options]
|
||||
|
||||
Options:
|
||||
|
||||
-V, --version output the version number
|
||||
-a, --access-key <accessKey> Access key id
|
||||
-k, --secret-key <secretKey> Secret access key
|
||||
-b, --bucket <bucket> Name of the bucket
|
||||
-q, --query <query> Search query
|
||||
-h, --host <host> Host of the server
|
||||
-p, --port <port> Port of the server
|
||||
-s --ssl
|
||||
-v, --verbose
|
||||
-h, --help output usage information
|
||||
|
||||
In the following examples, Zenko Server is accessible on endpoint
|
||||
:code:`http://127.0.0.1:8000` and contains the bucket :code:`zenkobucket`.
|
||||
|
||||
.. code::
|
||||
|
||||
# search for objects with metadata "blue"
|
||||
node bin/search_bucket -a accessKey1 -k verySecretKey1 -b zenkobucket \
|
||||
-q "x-amz-meta-color=blue" -h 127.0.0.1 -p 8000
|
||||
|
||||
# search for objects tagged with "type=color"
|
||||
node bin/search_bucket -a accessKey1 -k verySecretKey1 -b zenkobucket \
|
||||
-q "tags.type=color" -h 127.0.0.1 -p 8000
|
||||
|
||||
Coding Examples
|
||||
+++++++++++++++
|
||||
|
||||
Search requests can be also performed by making HTTP requests authenticated
|
||||
with one of the AWS Signature schemes: version 2 or version 4. \
|
||||
For more about authentication scheme, see:
|
||||
|
||||
* https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
|
||||
* http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
|
||||
* http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
|
||||
|
||||
You can also view examples for making requests with Auth V4 in various
|
||||
languages `here <../../../examples>`__.
|
||||
|
||||
Specifying Metadata Fields
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To search system metadata headers:
|
||||
|
||||
.. code::
|
||||
|
||||
{system-metadata-key}{supported SQL op}{search value}
|
||||
# example
|
||||
key = blueObject
|
||||
size > 0
|
||||
key LIKE "blue.*"
|
||||
|
||||
To search custom user metadata:
|
||||
|
||||
.. code::
|
||||
|
||||
# metadata must be prefixed with "x-amz-meta-"
|
||||
x-amz-meta-{user-metadata-key}{supported SQL op}{search value}
|
||||
|
||||
# example
|
||||
x-amz-meta-color = blue
|
||||
x-amz-meta-color != red
|
||||
x-amz-meta-color LIKE "b.*"
|
||||
|
||||
To search tags:
|
||||
|
||||
.. code::
|
||||
|
||||
# tag searches must be prefixed with "tags."
|
||||
tags.{tag-key}{supported SQL op}{search value}
|
||||
# example
|
||||
tags.type = color
|
||||
|
||||
Examples queries:
|
||||
|
||||
.. code::
|
||||
|
||||
# searching for objects with custom metadata "color"=blue" and are tagged
|
||||
# "type"="color"
|
||||
|
||||
tags.type="color" AND x-amz-meta-color="blue"
|
||||
|
||||
# searching for objects with the object key containing the substring "blue"
|
||||
# or (custom metadata "color"=blue" and are tagged "type"="color")
|
||||
|
||||
key LIKE '.*blue.*' OR (x-amz-meta-color="blue" AND tags.type="color")
|
||||
|
||||
Differences from SQL
|
||||
++++++++++++++++++++
|
||||
|
||||
Zenko metadata search queries are similar to SQL-query :code:`WHERE` clauses, but
|
||||
differ in that:
|
||||
|
||||
* They follow the :code:`PCRE` format
|
||||
* They do not require values with hyphens to be enclosed in
|
||||
backticks, :code:``(`)``
|
||||
|
||||
.. code::
|
||||
|
||||
# SQL query
|
||||
`x-amz-meta-search-item` = `ice-cream-cone`
|
||||
|
||||
# MD Search query
|
||||
x-amz-meta-search-item = ice-cream-cone
|
||||
|
||||
* Search queries do not support all SQL operators.
|
||||
|
||||
.. code::
|
||||
|
||||
# Supported SQL operators:
|
||||
=, <, >, <=, >=, !=, AND, OR, LIKE, <>
|
||||
|
||||
# Unsupported SQL operators:
|
||||
NOT, BETWEEN, IN, IS, +, -, %, ^, /, *, !
|
||||
|
||||
Using Regular Expressions in Metadata Search
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Regular expressions in Zenko metadata search differ from SQL in the following
|
||||
ways:
|
||||
|
||||
+ Wildcards are represented with :code:`.*` instead of :code:`%`.
|
||||
+ Regex patterns must be wrapped in quotes. Failure to do this can lead to
|
||||
misinterpretation of patterns.
|
||||
+ As with :code:`PCRE`, regular expressions can be entered in either the
|
||||
:code:`/pattern/` syntax or as the pattern itself if regex options are
|
||||
not required.
|
||||
|
||||
Example regular expressions:
|
||||
|
||||
.. code::
|
||||
|
||||
# search for strings containing word substring "helloworld"
|
||||
".*helloworld.*"
|
||||
"/.*helloworld.*/"
|
||||
"/.*helloworld.*/i"
|
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue