Compare commits

...

104 Commits

Author SHA1 Message Date
Nicolas Humbert 6e31a5aa8c ARSN-403 add tests 2024-03-04 13:23:33 +01:00
Nicolas Humbert d0606a5ce9 Merge remote-tracking branch 'origin/bugfix/ARSN-403/fix-put-metadata' into w/7.70/bugfix/ARSN-403/fix-put-metadata 2024-03-04 12:53:58 +01:00
bert-e a121810552 Merge branches 'w/7.70/improvement/ARSN-400-scuba-admin' and 'q/2224/7.10/improvement/ARSN-400-scuba-admin' into tmp/octopus/q/7.70 2024-02-26 13:59:54 +00:00
Nicolas Humbert a6f3c82827 Merge remote-tracking branch 'origin/bugfix/ARSN-392/bump' into w/7.70/bugfix/ARSN-392/bump 2024-02-21 10:01:01 +01:00
Nicolas Humbert 68204448a1 ARSN-392 Fix processVersionSpecificPut
- For backward compatibility (if isNull is undefined), add the nullVersionId field to the master update. The nullVersionId is needed for listing, retrieving, and deleting null versions.

- For the new null key implementation (if isNull is defined): add the isNull2 field and set it to true to specify that the new version is null AND has been put with a Cloudserver handling null keys (i.e., supporting S3C-7352).

- Manage scenarios in which a version is marked with the isNull attribute set to true, but without a version ID. This happens after BackbeatClient.putMetadata() is applied to a standalone null master.
2024-02-20 15:18:44 +01:00
Nicolas Humbert 40e271f7e2 ARSN-392 Import the V0 processVersionSpecificPut from Metadata
This logic is used by CRR replication feature to BackbeatClient.putMetadata on top of a null version
2024-02-20 15:18:05 +01:00
bert-e 2482fdfafc Merge branches 'w/7.70/bugfix/ARSN-392/null' and 'q/2215/7.10/bugfix/ARSN-392/null' into tmp/octopus/q/7.70 2024-02-20 14:02:11 +00:00
Nicolas Humbert b8bbdbbd81 Merge remote-tracking branch 'origin/bugfix/ARSN-392/null' into w/7.70/bugfix/ARSN-392/null 2024-02-20 14:49:31 +01:00
williamlardier 7423fac674 Merge remote-tracking branch 'origin/bugfix/ARSN-396-standardize-actionMapBP-and-chainbackend' into w/7.70/bugfix/ARSN-396-standardize-actionMapBP-and-chainbackend 2024-02-19 09:25:05 +01:00
williamlardier 9647043a02 ARSN-396: bump project 2024-02-19 09:24:27 +01:00
williamlardier f9e1f91791 Merge remote-tracking branch 'origin/development/7.70' into w/7.70/bugfix/ARSN-396-standardize-actionMapBP-and-chainbackend 2024-02-19 09:23:29 +01:00
bert-e b00378d46d Merge branch 'improvement/ARSN-400-scuba-admin' into tmp/octopus/w/7.70/improvement/ARSN-400-scuba-admin 2024-02-16 10:29:53 +00:00
Jonathan Gramain c72d8be223 ARSN-398 bump arsenal version 2024-02-15 11:23:53 -08:00
Jonathan Gramain f63cb3c762 bf: ARSN-398 DelimiterMaster: fix when gap building is disabled
- Fix the situation where gap building is disabled by
  `_saveBuildingGap()` but we attempted to reset the building gap state
  anyway.

- Introduce a new state 'Expired' that can be differentiated from
  'Disabled': it makes `getGapBuildingValidityPeriodMs()` return 0
  instead of 'null' to hint the listing backend that it should trigger
  a new listing.
2024-02-15 11:21:25 -08:00
bert-e effbf63dd4 Merge branch 'feature/ARSN-397-gapCacheClear' into q/7.70 2024-02-15 19:07:32 +00:00
bert-e 1d8ebe6a9c Merge branch 'bugfix/ARSN-394-GapCacheInvalidateStagingGaps' into q/7.70 2024-02-15 19:07:20 +00:00
Jonathan Gramain 7908654b51 ft: ARSN-397 GapCache.clear()
Add a clear() method to clear exposed and staging gaps. Retains
invalidating updates for gaps inserted after the call to clear().
2024-02-14 11:36:28 -08:00
Jonathan Gramain c4c75e976c ARSN-389 DelimiterMaster: v0 format gap skipping
Implement logic in DelimiterMaster to improve efficiency of listings
of buckets in V0 format that have a lot of current delete markers.

A GapCache instance can be attached to a DelimiterMaster instance,
which enables the following:

- Lookups in the cache to be able to restart listing directly beyond
  the cached gaps. It is done by returning FILTER_SKIP code when
  listing inside a gap, which hints the caller (RepdServer) that it is
  allowed to restart a new listing from a specific later key.

- Building gaps and cache them, when listing inside a series of current
  delete markers. This allows future listings to benefit from the gap
  information and skip over them.

An important caveat is that there is a limited time in which gaps can
be built from the current listing: it is a trade-off to guarantee the
validity of cached gaps when concurrent operations may invalidate
them. This time is set in the GapCache instance as `exposureDelayMs`,
and is the time during which concurrent operations are kept in memory
to potentially invalidate future gap creations. Because listings use a
snapshot of the database, they return entries that are older than when
the listing started. For this reason, in order to be allowed to
consistently build new gaps, it is necessary to limit the running time
of listings, and potentially redo periodically new listings (based on
time or number of listed keys), resuming from where the previous
listing stopped, instead of continuing the current listing.
2024-02-14 10:18:02 -08:00
Jonathan Gramain 1266a14253 impr: ARSN-389 change contract of skipping() API
Instead of returning a "prefix" for the listing task to skip over,
directly return the key on which to skip and continue the listing.

It is both more natural as well as needed to implement skipping over
cached "gaps" of deleted objects.

Note that it could even be more powerful to return the type of query
param to apply for the next listing ('gt' or 'gte'), but it would be
more complex to implement with little practical benefit, so instead we
add a null byte at the end of the returned key to skip to, whenever we
want a 'gt' behavior from the returned 'gte' key.

Also in this commit: clarify the API contract and always return
FILTER_ACCEPT when not allowed to skip over low-level listing
contents. A good chunk of the history of listing bugs and workarounds
comes from this confusion.
2024-02-14 10:18:02 -08:00
bert-e 29925a15ad Merge branch 'bugfix/ARSN-396-standardize-actionMapBP-and-chainbackend' into tmp/octopus/w/7.70/bugfix/ARSN-396-standardize-actionMapBP-and-chainbackend 2024-02-14 11:13:28 +00:00
Jonathan Gramain 8dc3ba7ca6 bf: ARSN-394 GapCache: invalidate staging gaps
In the GapCache._removeOverlappingGapsBeforeExpose() helper, remove
the gaps from the *staging* set that overlap with any of the staging
or frozen updates, in addition to removing the gaps from the frozen
set.

Without this extra invalidation, it's still possible to have gaps
created within the exposure delay that miss some invalidation,
resulting in stale gaps in the cache.

Modify an existing unit test to cover this case by adding extra wait
time to ensure `_removeOverlappingGapsBeforeExpose()` is called once
after the invalidating update but before the `setGap()` call.
2024-02-13 10:37:40 -08:00
Jonathan Gramain a6a76acede bf: ARSN-393 infinite loop in GapSet._coalesceGapChain()
The `GapSet._coalesceGapChain()` helper could infinite loop when
encountering a single-key gap (typically as an unchained single gap).
2024-02-12 12:00:04 -08:00
Jonathan Gramain c67331d350 ft: ARSN-391 GapCache: gap caching and invalidation
Introduce a new helper class GapCache that sits on top of a set of
GapSet instances, that delays exposure of gaps by a specific time to
guarantee atomicity wrt. invalidation from overlapping PUT/DELETE
operations.

The way it is implemented is the following:

- three update sets are used, each containing a GapSet instance and a
  series of key update batches: `staging`, `frozen`, and `exposed`

- `staging` receives the new gaps from `setGap()` calls and the
  updates from `removeOverlappingGaps()`

- `lookupGap()` only returns gaps present in `exposed`

- every `exposureDelayMs` milliseconds, the following happens:

  - the `frozen` gaps get invalidated by all key updates buffered in
    either `staging` or `frozen` update sets

  - the remainder of the `frozen` gaps is merged into `exposed` (via
    internal calls to `exposed.setGap()`)

  - the `staging` update set becomes the new `frozen` update set (both
    the gaps and the key updates)

  - a new `staging` update set is instanciated, empty

This guarantees that any gap set via `setGap()` is only exposed after
a minimum of `exposureDelayMs`, and a maximum of twice that time (plus
extra needed processing time). Also, keys passed to
`removeOverlappingGaps()` are kept in memory for at least `exposureDelayMs`
so they can invalidate new gaps that are created in this time frame.

This combined with insurance that setGap() is never called after
`exposureDelayMs` has passed since the listing process started (from a
DB snapshot), guarantees that all gaps not yet exposed have been
invalidated by any overlapping PUT/DELETE operation, hence exposed
gaps are still valid at the time they are exposed. They may still be
invalidated thereafter by future calls to removeOverlappingGaps().

The number of gaps that can be cached is bounded by the 'maxGaps'
attribute. The current strategy consists of simply not adding new gaps
when this limit is reached, solely relying on removeOverlappingGaps()
to make room for new gaps. In the future we could consider
implementing an eviction mechanism to remove less used gaps and/or
with smaller weights, but today the cost vs. benefit of doing this is
unclear.
2024-02-09 09:34:37 -08:00
Jonathan Gramain 6d6f1860ef ft: ARSN-388 implement GapSet (caching of listing gaps)
The GapSet class is intended for caching listing "gaps", which are
contiguous series of current delete markers in buckets, although the
semantics can allow for other uses in the future.

The end goal is to increase the performance of listings on V0 buckets
when a lot of delete markers are present, as a temporary solution
until buckets are migrated to V1 format.

This data structure is intented to be used by a GapCache instance,
which implements specific caching semantics (to ensure consistency
wrt. DB updates for example).
2024-02-09 09:32:49 -08:00
Mickael Bourgois 8ad0ea73a7
ARSN-390: Bump version 2024-02-05 17:45:22 +01:00
Mickael Bourgois a94040d13b
Merge remote-tracking branch 'origin/improvement/ARSN-390-scuba-arn' into w/7.70/improvement/ARSN-390-scuba-arn 2024-02-05 17:45:06 +01:00
Frédéric Meinnel 918c2c5473 Merge remote-tracking branch 'origin/bugfix/ARSN-386/fix-generate-v4-headers-for-put-with-body-requests' into w/7.70/bugfix/ARSN-386/fix-generate-v4-headers-for-put-with-body-requests 2024-01-23 12:25:28 +01:00
Frédéric Meinnel 5734d11cf1 Merge remote-tracking branch 'origin/bugfix/ARSN-385/fully-align-with-aws-on-lifecycle-configuration-dates' into w/7.70/bugfix/ARSN-385/fully-align-with-aws-on-lifecycle-configuration-dates 2024-01-16 17:47:02 +01:00
Jonathan Gramain 3b9c93be68 ARSN-381 bump arsenal version 2024-01-11 16:26:33 -08:00
Jonathan Gramain 081af3e795 ARSN-381 RPC command system between cluster workers
When using the cluster module, new processes are forked and are
dispatched workloads, usually HTTP requests. The ClusterRPC module
implements a RPC system to send commands to all cluster worker
processes at once from any particular worker, and retrieve their
individual command results, like a distributed map operation.

The existing cluster IPC channel is setup from the primary to each
worker, but not between workers, so there has to be a hop by the
primary.

How a command is treated:

- a worker sends a command message to the primary

- the primary then forwards that command to each existing worker
  (including the requestor)

- each worker then executes the command and returns a result or an
  error

- the primary gathers all workers results into an array

- finally, the primary dispatches the results array to the original
  requesting worker callback

The original use of this feature is in Metadata DBD (bucketd) to
implement a global cache refresh across worker processes.
2024-01-11 16:26:33 -08:00
bert-e 39f42d9cb4 Merge branches 'w/7.70/bugfix/ARSN-384-redirect-error-body' and 'q/2207/7.10/bugfix/ARSN-384-redirect-error-body' into tmp/octopus/q/7.70 2024-01-10 10:23:21 +00:00
Mickael Bourgois 7233ec2635
Merge remote-tracking branch 'origin/bugfix/ARSN-384-redirect-error-body' into w/7.70/bugfix/ARSN-384-redirect-error-body 2024-01-10 10:50:15 +01:00
Frédéric Meinnel aea4663ff2 Merge remote-tracking branch 'origin/bugfix/ARSN-383-lifecycle-configuration-dates-must-be-set-to-midnight' into w/7.70/bugfix/ARSN-383-lifecycle-configuration-dates-must-be-set-to-midnight 2024-01-08 15:47:01 +01:00
bert-e a0322b131c Merge branch 'bugfix/ARSN-382-redirect-root-empty' into tmp/octopus/w/7.70/bugfix/ARSN-382-redirect-root-empty 2024-01-03 08:52:08 +00:00
bert-e ddd6c87831 Merge branch 'bugfix/ARSN-382-redirect-root-empty' into tmp/octopus/w/7.70/bugfix/ARSN-382-redirect-root-empty 2024-01-02 18:09:06 +00:00
Mickael Bourgois 1efab676bc
Merge remote-tracking branch 'origin/bugfix/ARSN-382-redirect-root-empty' into w/7.70/bugfix/ARSN-382-redirect-root-empty
# Conflicts:
#	package.json
2024-01-02 11:53:05 +01:00
bert-e 2d2030dfe4 Merge branches 'w/7.70/improvement/ARSN-363-retention-day-condition' and 'q/2191/7.10/improvement/ARSN-363-retention-day-condition' into tmp/octopus/q/7.70 2023-12-26 10:55:58 +00:00
Will Toozs a7cf94d0fe
Merge remote-tracking branch 'origin/improvement/ARSN-363-retention-day-condition' into w/7.70/improvement/ARSN-363-retention-day-condition 2023-12-26 11:47:28 +01:00
Jonathan Gramain 9186643caa ARSN-379 [7.70] bump arsenal version 2023-12-22 12:35:57 -08:00
Jonathan Gramain 485a76ceb9 ARSN-379 [7.70] import FilterState and FilterReturnValue types from Delimiter 2023-12-22 12:35:44 -08:00
Jonathan Gramain 00109a2c44 ARSN-379 [7.70] adapt `DelimiterCurrent` to changes in `Delimiter`/`DelimiterMaster`
The internals of `DelimiterMaster` have changed with S3C-4682
implementation, which requires changes in the `DelimiterCurrent` class
that inherits from it.

Removed the unit test passing a key with a different prefix, because
the prefix check was removed in `DelimiterMaster` as no such key can
be passed by construction of the listing parameters.
2023-12-22 12:35:44 -08:00
Jonathan Gramain aed1247825 Merge remote-tracking branch 'origin/bugfix/ARSN-379-cherry-pick-ARSN-284-and-ARSN-293' into w/7.70/bugfix/ARSN-379-cherry-pick-ARSN-284-and-ARSN-293 2023-12-22 12:35:34 -08:00
Jonathan Gramain 2799381ef2 ARSN-380 rf: DelimiterVersions class inherits from Extension
Small refactor of DelimiterVersions class to inherit from the base
class Extension rather than Delimiter. Copy the missing fields and
methods from `Delimiter`.

This prepares for merging ARSN-379 which would otherwise cause a lot
of incompatibilities due to changes in the interface of
`DelimiterVersions` from S3C-8242.

Other minor tweaks:

- reset `nextVersionIdMarker` when skipping a common prefix

- rename `this.Contents` to `this.Versions` as we don't need to keep
  compatibility with `Delimiter`, and as it is the name used in the
  final result
2023-12-20 11:57:57 -08:00
Jonathan Gramain d08a267965 ARSN-377 bump arsenal version 2023-12-14 14:52:11 -08:00
Jonathan Gramain 063a2fb8fb ARSN-377 fix DelimiterNonCurrent and add a unit test 2023-12-14 14:51:48 -08:00
Jonathan Gramain 1bc3360daf ARSN-377 correctly handle null keys with common prefix
When encountering a null key, check for its common prefix before
including it in either the Versions array or CommonPrefixes array,
instead of always including it in the Versions array.

This commit refactors how `DelimiterVersions` works with null keys
slightly: the null key is now inserted at its correct ordered position
by the top-level `filter()` method, and the state machine handlers
only have to deal with sorted versions. Previously the individual
handlers would have to deal with the null key positioning themselves
resulting in more complex state management.
2023-12-14 14:12:26 -08:00
Jonathan Gramain 206f14bdf5 ARSN-377 improve versioned listing test
Add version IDs to delete marker metadata
2023-12-14 14:12:26 -08:00
Maha Benzekri 5ffae72693
Merge remote-tracking branch 'origin/improvement/ARSN-378-BP-authorization' into w/7.70/improvement/ARSN-378-BP-authorization 2023-12-14 11:57:20 +01:00
bert-e df4c22154e Merge branch 'improvement/ARSN-378-BP-authorization' into tmp/octopus/w/7.70/improvement/ARSN-378-BP-authorization 2023-12-14 10:55:36 +00:00
Nicolas Humbert a99a6d9d97 Merge remote-tracking branch 'origin/bugfix/ARSN-376/probe' into w/7.70/bugfix/ARSN-376/probe 2023-12-01 11:36:09 +01:00
Maha Benzekri 29ef2ef265
fixup 2023-10-30 16:51:41 +01:00
Maha Benzekri 90ab985271
Merge remote-tracking branch 'origin/improvement/ARSN-362-implicitDeny' into w/7.70/improvement/ARSN-362-implicitDeny 2023-10-30 16:35:32 +01:00
bert-e c34ad0dc31 Merge branch 'improvement/ARSN-362-implicitDeny' into tmp/octopus/w/7.70/improvement/ARSN-362-implicitDeny 2023-10-30 15:01:06 +00:00
Nicolas Humbert 79b83a9067 ARSN-369 orphan delete marker list interruption skips processed key
In the event of a listing interruption due to reaching the maximum scanned entries, the next “orphan delete marker“ listing skips the currently processed key.
2023-10-05 09:39:45 +02:00
Nicolas Humbert d84cc974d3 ARSN-366 Limit lifecycle listing on scanned entries 2023-09-27 17:19:03 +02:00
Maha Benzekri 0177fbe98f
Merge remote-tracking branch 'origin/bugfix/ARSN-367-principal-user-arn-on-policy' into w/7.70/bugfix/ARSN-367-principal-user-arn-on-policy 2023-09-25 12:17:43 +02:00
bert-e b61d178b18 Merge branch 'bugfix/ARSN-365-id-on-resource-policy' into tmp/octopus/w/7.70/bugfix/ARSN-365-id-on-resource-policy 2023-09-13 06:27:35 +00:00
Nicolas Humbert 8a7c1be2d1 ARSN-359 bump arsenal version 2023-08-08 19:50:42 -04:00
Nicolas Humbert c049df0a97 ARSN-359 Fix NextMarker calculation in listLifecycleCurrent
Please note that there are no missing entries in the listing and no extra resource used since the next listing will do the fetching anyway. The issue lies in how we determine the NextMarker. It has to be compatible with the current logic merged in Artesca.

When using the listLifecycleCurrent function, we need to calculate the NextMarker correctly. Currently, if the maximum number of keys (max-keys) is reached, the function continues fetching more entries, which is unnecessary and should be done by the next listing.

For instance, if max-keys is set to 1 and the first entry (key0) is eligible, while the following two entries (key1 and key2) are not eligible, but the fourth entry (key3) is eligible, the listing should stop at key0 and the NextMarker should be key0 instead the listing keep fetching until key3 and return the NextMarker key2.
2023-08-08 19:50:12 -04:00
Nicolas Humbert 8eb4a29c36 ARSN-358 bump version 2023-08-08 13:12:22 -04:00
Nicolas Humbert e69a97f240 add comment about this.start 2023-08-04 17:24:07 -04:00
Nicolas Humbert 81e838000f ARSN-356 List lifecycle orphan delete markers supports V0 2023-08-04 17:24:03 -04:00
Nicolas Humbert 8256d6debf ARSN-355 List lifecycle non-current versions supports V0 2023-08-04 13:02:35 -04:00
Nicolas Humbert 69c1698eb7 ARSN-354 List lifecycle current versions supports V0 bucket format 2023-08-01 11:53:37 -04:00
Nicolas Humbert c2cd90925f Adapt delimiterCurrent for S3C Metadata 2023-08-01 10:09:26 -04:00
bert-e b1723594eb Merge branch 'improvement/ARSN-351/backport' into q/7.70 2023-07-21 16:40:31 +00:00
Nicolas Humbert 49e32758fb ARSN-351 cleanup MongoDB tests 2023-07-21 08:29:16 -04:00
Nicolas Humbert e13d0f5ed8 ARSN-351 support listLifecycleObject in BucketFileInterface 2023-07-21 08:29:16 -04:00
Nicolas Humbert 0d5907956f ARSN-351 export DelimiterNonCurrent and DelimiterOrphanDeleteMarker for Metadata 2023-07-21 08:29:16 -04:00
Nicolas Humbert f0c5d60ce9 ARSN-351 export DelimiterCurrent for Metadata 2023-07-21 08:29:16 -04:00
Nicolas Humbert 8c2f4cf357 ARSN-351 support listLifecycleObject in BucketClientInterface 2023-07-21 08:29:16 -04:00
Nicolas Humbert f3f1da9bb3 ARSN-350 Missing Null Version in Lifecycle List of Non-Current Versions
Note: We only support the v1 bucket format for "list lifecycle" in Artesca.

We made the assumption that the first version key stored the current/latest version, which is true in most cases except for "null" versions. In the case of a "null" version, the current version is stored in the master key alone, rather than being stored in both the master key and a new version key. Here's an example of the key structure:

Mkey0: Represents the null version ID.
VKey0<versionID>: Represents a non-current version.

Additionally, we assumed that the versions for a given key were ordered by creation date, from newest to oldest. However, in Ring S3C, for non-current null versions, the metadata version ID is not part of the metadata key id. Therefore, the non-current null version is listed before the current version that has a version ID. Here's an example of the key ordering:

Mkey0: Master version
Vkey0: "null" non-current version
VKey0<versionID>: Current version

The listing was using only versions, but because those assumptions are incorrect, we now use both the master and the versions for each given key to ensure that we return the correct non-current versions.

(cherry picked from commit 0a4d6f862f)
2023-07-21 08:29:16 -04:00
Nicolas Humbert 036b75842e ARSN-328 Exclude keys based on their dataStoreName
(cherry picked from commit e216c9dd20)
2023-07-21 08:29:16 -04:00
Nicolas Humbert 7ac5774635 ARSN-312 Add logic to list orphan delete markers for Lifecycle
DelimiterOrphan used for listing orphan delete marker.The Metadata call returns the versions (V prefix).The MD response is then processed to only return the delete markers with zero noncurrent versions before a defined date: beforeDate.

(cherry picked from commit c9a444969b)
2023-07-17 09:06:23 -04:00
Nicolas Humbert f3b928fce0 ARSN-311 Add logic to list non-current versions for Lifecycle
DelimiterNonCurrent used for listing non-current version.The Metadata call returns the versions (V prefix).The MD response is then processed to only return the non-current versions that became non-current before a defined date: beforeDate.

(cherry picked from commit 5d018860ec)
2023-07-17 09:06:23 -04:00
Nicolas Humbert 7173a357d9 ARSN-326 Lifecycle listings should handle null version
(cherry picked from commit 4be0a06c4a)
2023-07-17 09:06:23 -04:00
bert-e 2938bb0c88 Merge branch 'improvement/ARSN-345-optimize-multiobjectdelete-api-and-batching' into q/7.70 2023-07-12 11:36:28 +00:00
williamlardier 8d758327dd
ARSN-345: bump package version 2023-07-12 13:19:38 +02:00
williamlardier be63c09624
ARSN-345: update tests and logic 2023-07-12 13:19:01 +02:00
Nicolas Humbert 4615875462 ARSN-310 Add logic to list current/master versions for Lifecycle
DelimiterCurrent used for listing current versions. The Metadata call returns the masters (M prefix) younger than a defined date: beforeDate. No extra filtering action is needed on the Metadata call response.

(cherry picked from commit ecd600ac4b)
2023-06-23 08:11:54 -04:00
bert-e a89d1d8d75 Merge branch 'improvement/ARSN-349-update-node-fcntl' into tmp/octopus/w/7.70/improvement/ARSN-349-update-node-fcntl 2023-06-20 23:12:07 +00:00
williamlardier 57e84980c8
ARSN-345: optimize InternalDeleteObject with direct deletion support 2023-06-15 13:43:27 +02:00
williamlardier 51bfd41bea
ARSN-345: optimize MultiDeleteObject with batching support 2023-06-15 13:43:27 +02:00
Nicolas Humbert cb01346d07 Merge remote-tracking branch 'origin/bugfix/ARSN-347/socketio' into w/7.70/bugfix/ARSN-347/socketio 2023-06-08 11:44:15 -04:00
bert-e e9c67f7f67 Merge branch 'bugfix/ARSN-340-bump-socket-io' into tmp/octopus/w/7.70/bugfix/ARSN-340-bump-socket-io 2023-05-30 22:49:34 +00:00
bert-e 55e68cfa17 Merge branch 'w/7.10/improvement/ARSN-335-implement-ghas' into tmp/octopus/w/7.70/improvement/ARSN-335-implement-ghas 2023-05-25 17:52:45 +00:00
bert-e 5f3540a0d5 Merge branch 'w/7.10/improvement/ARSN-335-implement-ghas' into tmp/octopus/w/7.70/improvement/ARSN-335-implement-ghas 2023-05-19 15:59:38 +00:00
Jonathan Gramain d744a709d2 ARSN-330 bump arsenal version 2023-04-05 15:40:53 -07:00
Jonathan Gramain 99e04bd6fa bf: ARSN-330 fix DelimiterVersions exception when key contains "undefined"
Fix a crash when a listed key contains the string "undefined": as the
`key.indexOf` method was used without prior checking whether a
delimiter was set, it converted the delimiter to the string
"undefined", which could be found in a key containing such string, and
causing an exception thereafter.
2023-04-05 15:35:35 -07:00
Jonathan Gramain c4cc5a2c3d ARSN-320 bump arsenal version to 7.70.4 2023-04-04 09:10:19 -07:00
Jonathan Gramain fedd0190cc impr: ARSN-320 add "isNull2" attribute to ObjectMD
This new attribute will be set whenever a Cloudserver supporting null
keys sets the "isNull" attribute to a master key, along with it.

The purpose of this attribute is to allow Cloudserver to optimize by
not having to check and delete a null versioned key when the null
master has "isNull2" set, as it is guaranteed not to exist.

We need to introduce a new attribute to keep backward compatibility,
the naming is a bit unfortunate but it has the benefit of being short
and not too specific to a particular optimization, just stating that
it is a "new" null master.
2023-04-04 09:10:19 -07:00
Jonathan Gramain 56fd4ad734 ft: ARSN-317 null key support in BucketFile backend
Support null keys in BucketFile backend - null keys are the new way to
store null versions, where a single database key with a specific empty
version ID is used instead of referencing the null version via
"nullVersionId" in object metadata.

Add relevant unit tests to check the new behavior (those were copied
and mechanically adapted from the Metadata repository).
2023-04-04 09:09:05 -07:00
Jonathan Gramain ebe6b65fcf ARSN-317 [rf] cleanup logging
Use "logger.addDefaultFields()" to set bucket, key and options to the
logs, which cleans up log calls.

Log repair errors with `log.error` unless it's ObjNotFound
2023-04-04 09:08:48 -07:00
Jonathan Gramain 3a4da1d7c0 ARSN-318 port listVersionKeys() helper for BucketFile backend
Port the listVersionKeys() helper from the Metadata backend to the
BucketFile backend, as a first step towards supporting null keys in
BucketFile.
2023-03-23 10:57:41 -07:00
Naren 5377b20ceb impr: ARSN-315 bump version to 7.70.3 2023-03-14 16:52:08 -07:00
Naren 21b329b301 Merge remote-tracking branch 'origin/improvement/ARSN-315-bump-version-7-10-46' into w/7.70/improvement/ARSN-315-bump-version-7-10-46 2023-03-14 16:49:03 -07:00
bert-e 94edf8be70 Merge branch 'improvement/ARSN-315-disable-default-metrics-collection' into tmp/octopus/w/7.70/improvement/ARSN-315-disable-default-metrics-collection 2023-03-14 23:13:08 +00:00
Jonathan Gramain 655a10ce52 ARSN-306 version bump 2023-03-09 09:57:25 -08:00
Jonathan Gramain 0c7f0e607d ARSN-306 [doc] add state chart for DelimiterVersions
And a markdown file with summary of what the listing algo does
2023-03-09 09:56:28 -08:00
Jonathan Gramain caa5d53e9b impr: ARSN-306 support null keys in versions listing
Add support for null keys in versions listing:

- when they exist, output the null keys at the appropriate position in
  the Versions array

- handle KeyMarker/VersionIdMarker appropriately as if the null keys
  were real versions. This requires the listing to start at the very
  first version of the next key each time to see the null key, then
  potentially skip over the versions below VersionIdMarker using
  skip-scan optimization.
2023-03-09 09:56:28 -08:00
Jonathan Gramain 21da975187 ARSN-306 [refactor] DelimiterVersions state machine
Use a state machine for cleaner state management in DelimiterVersions
listing algo, with Typescript for enhanced type checking

Also, fix an inefficiency with listing params generated from the
KeyMarker parameter when there is a delimiter: it was listing more
keys than necessary when the KeyMarker equals a CommonPrefix.
2023-03-09 09:56:28 -08:00
Naren 47e68a9b60 Merge remote-tracking branch 'origin/improvement/ARSN-313-upgrade-prom-client' into w/7.70/improvement/ARSN-313-upgrade-prom-client 2023-03-08 17:51:26 -08:00
Alexander Chan f33cd69e45 Merge remote-tracking branch 'origin/bugfix/ARSN-308/addLifecycleUtilsNoncurrentVersionSupport' into w/7.70/bugfix/ARSN-308/addLifecycleUtilsNoncurrentVersionSupport 2023-03-01 04:55:37 -08:00
Jonathan Gramain 10a94a0a96 ARSN-307 bump version to 7.70.0 2023-02-23 23:00:46 -08:00
47 changed files with 9389 additions and 743 deletions

View File

@ -0,0 +1,33 @@
# DelimiterVersions
The DelimiterVersions class handles raw listings from the database of a
versioned or non-versioned bucket with an optional delimiter, and
fills in a curated listing with "Versions" and "CommonPrefixes" as a
result.
## Expected Behavior
- lists individual distinct versions of versioned buckets
- only lists keys belonging to the given **prefix** (if provided)
- groups listed keys that have a common prefix ending with a delimiter
inside CommonPrefixes
- can take a **keyMarker** and optionally a **versionIdMarker** to
list from a specific key or version
- can take a **maxKeys** parameter to limit how many keys can be returned
- skips internal keys like replay keys
## State Chart
- States with grey background are *Idle* states, which are waiting for
a new listing key
- States with blue background are *Processing* states, which are
actively processing a new listing key passed by the filter()
function
![DelimiterVersions State Chart](./pics/delimiterVersionsStateChart.svg)

View File

@ -0,0 +1,50 @@
digraph {
node [shape="box",style="filled,rounded",fontsize=16,fixedsize=true,width=3];
edge [fontsize=14];
rankdir=TB;
START [shape="circle",width=0.2,label="",style="filled",fillcolor="black"]
END [shape="circle",width=0.2,label="",style="filled",fillcolor="black",peripheries=2]
node [fillcolor="lightgrey"];
"NotSkipping.Idle" [label="NotSkipping",group="NotSkipping",width=4];
"SkippingPrefix.Idle" [label="SkippingPrefix",group="SkippingPrefix"];
"WaitForNullKey.Idle" [label="WaitForNullKey",group="WaitForNullKey"];
"SkippingVersions.Idle" [label="SkippingVersions",group="SkippingVersions"];
node [fillcolor="lightblue"];
"NotSkipping.Processing" [label="NotSkipping",group="NotSkipping",width=4];
"NotSkippingV0.Processing" [label="NotSkippingV0",group="NotSkipping",width=4];
"NotSkippingV1.Processing" [label="NotSkippingV1",group="NotSkipping",width=4];
"NotSkippingCommon.Processing" [label="NotSkippingCommon",group="NotSkipping",width=4];
"SkippingPrefix.Processing" [label="SkippingPrefix",group="SkippingPrefix"];
"WaitForNullKey.Processing" [label="WaitForNullKey",group="WaitForNullKey"];
"SkippingVersions.Processing" [label="SkippingVersions",group="SkippingVersions"];
START -> "WaitForNullKey.Idle" [label="[versionIdMarker != undefined]"]
START -> "NotSkipping.Idle" [label="[versionIdMarker == undefined]"]
"NotSkipping.Idle" -> "NotSkipping.Processing" [label="filter(key, value)"]
"SkippingPrefix.Idle" -> "SkippingPrefix.Processing" [label="filter(key, value)"]
"WaitForNullKey.Idle" -> "WaitForNullKey.Processing" [label="filter(key, value)"]
"SkippingVersions.Idle" -> "SkippingVersions.Processing" [label="filter(key, value)"]
"NotSkipping.Processing" -> "NotSkippingV0.Processing" [label="vFormat='v0'"]
"NotSkipping.Processing" -> "NotSkippingV1.Processing" [label="vFormat='v1'"]
"WaitForNullKey.Processing" -> "NotSkipping.Processing" [label="master(key) != keyMarker"]
"WaitForNullKey.Processing" -> "SkippingVersions.Processing" [label="master(key) == keyMarker"]
"NotSkippingV0.Processing" -> "SkippingPrefix.Idle" [label="[key.startsWith(<ReplayPrefix>)]\n/ prefix <- <ReplayPrefix>\n-> FILTER_SKIP"]
"NotSkippingV0.Processing" -> "NotSkipping.Idle" [label="[Version.isPHD(value)]\n-> FILTER_ACCEPT"]
"NotSkippingV0.Processing" -> "NotSkippingCommon.Processing" [label="[not key.startsWith(<ReplayPrefix>)\nand not Version.isPHD(value)]"]
"NotSkippingV1.Processing" -> "NotSkippingCommon.Processing" [label="[always]"]
"NotSkippingCommon.Processing" -> END [label="[isListableKey(key, value) and\nKeys == maxKeys]\n-> FILTER_END"]
"NotSkippingCommon.Processing" -> "SkippingPrefix.Idle" [label="[isListableKey(key, value) and\nnKeys < maxKeys and\nhasDelimiter(key)]\n/ prefix <- prefixOf(key)\n/ CommonPrefixes.append(prefixOf(key))\n-> FILTER_ACCEPT"]
"NotSkippingCommon.Processing" -> "NotSkipping.Idle" [label="[isListableKey(key, value) and\nnKeys < maxKeys and\nnot hasDelimiter(key)]\n/ Contents.append(key, versionId, value)\n-> FILTER_ACCEPT"]
"SkippingPrefix.Processing" -> "SkippingPrefix.Idle" [label="[key.startsWith(prefix)]\n-> FILTER_SKIP"]
"SkippingPrefix.Processing" -> "NotSkipping.Processing" [label="[not key.startsWith(prefix)]"]
"SkippingVersions.Processing" -> "NotSkipping.Processing" [label="master(key) !== keyMarker or \nversionId > versionIdMarker"]
"SkippingVersions.Processing" -> "SkippingVersions.Idle" [label="master(key) === keyMarker and \nversionId < versionIdMarker\n-> FILTER_SKIP"]
"SkippingVersions.Processing" -> "SkippingVersions.Idle" [label="master(key) === keyMarker and \nversionId == versionIdMarker\n-> FILTER_ACCEPT"]
}

View File

@ -0,0 +1,265 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.43.0 (0)
-->
<!-- Title: %3 Pages: 1 -->
<svg width="1522pt" height="922pt"
viewBox="0.00 0.00 1522.26 922.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 918)">
<title>%3</title>
<polygon fill="white" stroke="transparent" points="-4,4 -4,-918 1518.26,-918 1518.26,4 -4,4"/>
<!-- START -->
<g id="node1" class="node">
<title>START</title>
<ellipse fill="black" stroke="black" cx="393.26" cy="-907" rx="7" ry="7"/>
</g>
<!-- NotSkipping.Idle -->
<g id="node3" class="node">
<title>NotSkipping.Idle</title>
<path fill="lightgrey" stroke="black" d="M436.26,-675C436.26,-675 172.26,-675 172.26,-675 166.26,-675 160.26,-669 160.26,-663 160.26,-663 160.26,-651 160.26,-651 160.26,-645 166.26,-639 172.26,-639 172.26,-639 436.26,-639 436.26,-639 442.26,-639 448.26,-645 448.26,-651 448.26,-651 448.26,-663 448.26,-663 448.26,-669 442.26,-675 436.26,-675"/>
<text text-anchor="middle" x="304.26" y="-653.2" font-family="Times,serif" font-size="16.00">NotSkipping</text>
</g>
<!-- START&#45;&gt;NotSkipping.Idle -->
<g id="edge2" class="edge">
<title>START&#45;&gt;NotSkipping.Idle</title>
<path fill="none" stroke="black" d="M391.06,-899.87C380.45,-870.31 334.26,-741.58 313.93,-684.93"/>
<polygon fill="black" stroke="black" points="317.12,-683.46 310.45,-675.23 310.53,-685.82 317.12,-683.46"/>
<text text-anchor="middle" x="470.76" y="-783.8" font-family="Times,serif" font-size="14.00">[versionIdMarker == undefined]</text>
</g>
<!-- WaitForNullKey.Idle -->
<g id="node5" class="node">
<title>WaitForNullKey.Idle</title>
<path fill="lightgrey" stroke="black" d="M692.26,-849C692.26,-849 500.26,-849 500.26,-849 494.26,-849 488.26,-843 488.26,-837 488.26,-837 488.26,-825 488.26,-825 488.26,-819 494.26,-813 500.26,-813 500.26,-813 692.26,-813 692.26,-813 698.26,-813 704.26,-819 704.26,-825 704.26,-825 704.26,-837 704.26,-837 704.26,-843 698.26,-849 692.26,-849"/>
<text text-anchor="middle" x="596.26" y="-827.2" font-family="Times,serif" font-size="16.00">WaitForNullKey</text>
</g>
<!-- START&#45;&gt;WaitForNullKey.Idle -->
<g id="edge1" class="edge">
<title>START&#45;&gt;WaitForNullKey.Idle</title>
<path fill="none" stroke="black" d="M399.56,-903.7C420.56,-896.05 489.7,-870.85 540.08,-852.48"/>
<polygon fill="black" stroke="black" points="541.38,-855.73 549.57,-849.02 538.98,-849.16 541.38,-855.73"/>
<text text-anchor="middle" x="608.76" y="-870.8" font-family="Times,serif" font-size="14.00">[versionIdMarker != undefined]</text>
</g>
<!-- END -->
<g id="node2" class="node">
<title>END</title>
<ellipse fill="black" stroke="black" cx="45.26" cy="-120" rx="7" ry="7"/>
<ellipse fill="none" stroke="black" cx="45.26" cy="-120" rx="11" ry="11"/>
</g>
<!-- NotSkipping.Processing -->
<g id="node7" class="node">
<title>NotSkipping.Processing</title>
<path fill="lightblue" stroke="black" d="M761.26,-558C761.26,-558 497.26,-558 497.26,-558 491.26,-558 485.26,-552 485.26,-546 485.26,-546 485.26,-534 485.26,-534 485.26,-528 491.26,-522 497.26,-522 497.26,-522 761.26,-522 761.26,-522 767.26,-522 773.26,-528 773.26,-534 773.26,-534 773.26,-546 773.26,-546 773.26,-552 767.26,-558 761.26,-558"/>
<text text-anchor="middle" x="629.26" y="-536.2" font-family="Times,serif" font-size="16.00">NotSkipping</text>
</g>
<!-- NotSkipping.Idle&#45;&gt;NotSkipping.Processing -->
<g id="edge3" class="edge">
<title>NotSkipping.Idle&#45;&gt;NotSkipping.Processing</title>
<path fill="none" stroke="black" d="M333.17,-638.98C364.86,-620.99 417.68,-592.92 466.26,-576 483.64,-569.95 502.44,-564.74 520.88,-560.34"/>
<polygon fill="black" stroke="black" points="521.83,-563.71 530.78,-558.04 520.25,-556.89 521.83,-563.71"/>
<text text-anchor="middle" x="524.26" y="-594.8" font-family="Times,serif" font-size="14.00">filter(key, value)</text>
</g>
<!-- SkippingPrefix.Idle -->
<g id="node4" class="node">
<title>SkippingPrefix.Idle</title>
<path fill="lightgrey" stroke="black" d="M662.26,-138C662.26,-138 470.26,-138 470.26,-138 464.26,-138 458.26,-132 458.26,-126 458.26,-126 458.26,-114 458.26,-114 458.26,-108 464.26,-102 470.26,-102 470.26,-102 662.26,-102 662.26,-102 668.26,-102 674.26,-108 674.26,-114 674.26,-114 674.26,-126 674.26,-126 674.26,-132 668.26,-138 662.26,-138"/>
<text text-anchor="middle" x="566.26" y="-116.2" font-family="Times,serif" font-size="16.00">SkippingPrefix</text>
</g>
<!-- SkippingPrefix.Processing -->
<g id="node11" class="node">
<title>SkippingPrefix.Processing</title>
<path fill="lightblue" stroke="black" d="M779.26,-36C779.26,-36 587.26,-36 587.26,-36 581.26,-36 575.26,-30 575.26,-24 575.26,-24 575.26,-12 575.26,-12 575.26,-6 581.26,0 587.26,0 587.26,0 779.26,0 779.26,0 785.26,0 791.26,-6 791.26,-12 791.26,-12 791.26,-24 791.26,-24 791.26,-30 785.26,-36 779.26,-36"/>
<text text-anchor="middle" x="683.26" y="-14.2" font-family="Times,serif" font-size="16.00">SkippingPrefix</text>
</g>
<!-- SkippingPrefix.Idle&#45;&gt;SkippingPrefix.Processing -->
<g id="edge4" class="edge">
<title>SkippingPrefix.Idle&#45;&gt;SkippingPrefix.Processing</title>
<path fill="none" stroke="black" d="M552.64,-101.74C543.31,-87.68 534.41,-67.95 545.26,-54 549.71,-48.29 559.34,-43.36 571.56,-39.15"/>
<polygon fill="black" stroke="black" points="572.87,-42.41 581.36,-36.07 570.77,-35.73 572.87,-42.41"/>
<text text-anchor="middle" x="603.26" y="-65.3" font-family="Times,serif" font-size="14.00">filter(key, value)</text>
</g>
<!-- WaitForNullKey.Processing -->
<g id="node12" class="node">
<title>WaitForNullKey.Processing</title>
<path fill="lightblue" stroke="black" d="M692.26,-762C692.26,-762 500.26,-762 500.26,-762 494.26,-762 488.26,-756 488.26,-750 488.26,-750 488.26,-738 488.26,-738 488.26,-732 494.26,-726 500.26,-726 500.26,-726 692.26,-726 692.26,-726 698.26,-726 704.26,-732 704.26,-738 704.26,-738 704.26,-750 704.26,-750 704.26,-756 698.26,-762 692.26,-762"/>
<text text-anchor="middle" x="596.26" y="-740.2" font-family="Times,serif" font-size="16.00">WaitForNullKey</text>
</g>
<!-- WaitForNullKey.Idle&#45;&gt;WaitForNullKey.Processing -->
<g id="edge5" class="edge">
<title>WaitForNullKey.Idle&#45;&gt;WaitForNullKey.Processing</title>
<path fill="none" stroke="black" d="M596.26,-812.8C596.26,-801.16 596.26,-785.55 596.26,-772.24"/>
<polygon fill="black" stroke="black" points="599.76,-772.18 596.26,-762.18 592.76,-772.18 599.76,-772.18"/>
<text text-anchor="middle" x="654.26" y="-783.8" font-family="Times,serif" font-size="14.00">filter(key, value)</text>
</g>
<!-- SkippingVersions.Idle -->
<g id="node6" class="node">
<title>SkippingVersions.Idle</title>
<path fill="lightgrey" stroke="black" d="M1241.26,-558C1241.26,-558 1049.26,-558 1049.26,-558 1043.26,-558 1037.26,-552 1037.26,-546 1037.26,-546 1037.26,-534 1037.26,-534 1037.26,-528 1043.26,-522 1049.26,-522 1049.26,-522 1241.26,-522 1241.26,-522 1247.26,-522 1253.26,-528 1253.26,-534 1253.26,-534 1253.26,-546 1253.26,-546 1253.26,-552 1247.26,-558 1241.26,-558"/>
<text text-anchor="middle" x="1145.26" y="-536.2" font-family="Times,serif" font-size="16.00">SkippingVersions</text>
</g>
<!-- SkippingVersions.Processing -->
<g id="node13" class="node">
<title>SkippingVersions.Processing</title>
<path fill="lightblue" stroke="black" d="M1241.26,-675C1241.26,-675 1049.26,-675 1049.26,-675 1043.26,-675 1037.26,-669 1037.26,-663 1037.26,-663 1037.26,-651 1037.26,-651 1037.26,-645 1043.26,-639 1049.26,-639 1049.26,-639 1241.26,-639 1241.26,-639 1247.26,-639 1253.26,-645 1253.26,-651 1253.26,-651 1253.26,-663 1253.26,-663 1253.26,-669 1247.26,-675 1241.26,-675"/>
<text text-anchor="middle" x="1145.26" y="-653.2" font-family="Times,serif" font-size="16.00">SkippingVersions</text>
</g>
<!-- SkippingVersions.Idle&#45;&gt;SkippingVersions.Processing -->
<g id="edge6" class="edge">
<title>SkippingVersions.Idle&#45;&gt;SkippingVersions.Processing</title>
<path fill="none" stroke="black" d="M1145.26,-558.25C1145.26,-576.77 1145.26,-606.45 1145.26,-628.25"/>
<polygon fill="black" stroke="black" points="1141.76,-628.53 1145.26,-638.53 1148.76,-628.53 1141.76,-628.53"/>
<text text-anchor="middle" x="1203.26" y="-594.8" font-family="Times,serif" font-size="14.00">filter(key, value)</text>
</g>
<!-- NotSkippingV0.Processing -->
<g id="node8" class="node">
<title>NotSkippingV0.Processing</title>
<path fill="lightblue" stroke="black" d="M436.26,-411C436.26,-411 172.26,-411 172.26,-411 166.26,-411 160.26,-405 160.26,-399 160.26,-399 160.26,-387 160.26,-387 160.26,-381 166.26,-375 172.26,-375 172.26,-375 436.26,-375 436.26,-375 442.26,-375 448.26,-381 448.26,-387 448.26,-387 448.26,-399 448.26,-399 448.26,-405 442.26,-411 436.26,-411"/>
<text text-anchor="middle" x="304.26" y="-389.2" font-family="Times,serif" font-size="16.00">NotSkippingV0</text>
</g>
<!-- NotSkipping.Processing&#45;&gt;NotSkippingV0.Processing -->
<g id="edge7" class="edge">
<title>NotSkipping.Processing&#45;&gt;NotSkippingV0.Processing</title>
<path fill="none" stroke="black" d="M573.96,-521.95C558.07,-516.64 540.84,-510.46 525.26,-504 460.22,-477.02 387.62,-439.36 343.97,-415.84"/>
<polygon fill="black" stroke="black" points="345.57,-412.72 335.11,-411.04 342.24,-418.88 345.57,-412.72"/>
<text text-anchor="middle" x="573.76" y="-462.8" font-family="Times,serif" font-size="14.00">vFormat=&#39;v0&#39;</text>
</g>
<!-- NotSkippingV1.Processing -->
<g id="node9" class="node">
<title>NotSkippingV1.Processing</title>
<path fill="lightblue" stroke="black" d="M758.26,-411C758.26,-411 494.26,-411 494.26,-411 488.26,-411 482.26,-405 482.26,-399 482.26,-399 482.26,-387 482.26,-387 482.26,-381 488.26,-375 494.26,-375 494.26,-375 758.26,-375 758.26,-375 764.26,-375 770.26,-381 770.26,-387 770.26,-387 770.26,-399 770.26,-399 770.26,-405 764.26,-411 758.26,-411"/>
<text text-anchor="middle" x="626.26" y="-389.2" font-family="Times,serif" font-size="16.00">NotSkippingV1</text>
</g>
<!-- NotSkipping.Processing&#45;&gt;NotSkippingV1.Processing -->
<g id="edge8" class="edge">
<title>NotSkipping.Processing&#45;&gt;NotSkippingV1.Processing</title>
<path fill="none" stroke="black" d="M628.91,-521.8C628.39,-496.94 627.44,-450.74 626.83,-421.23"/>
<polygon fill="black" stroke="black" points="630.32,-421.11 626.62,-411.18 623.33,-421.25 630.32,-421.11"/>
<text text-anchor="middle" x="676.76" y="-462.8" font-family="Times,serif" font-size="14.00">vFormat=&#39;v1&#39;</text>
</g>
<!-- NotSkippingV0.Processing&#45;&gt;NotSkipping.Idle -->
<g id="edge12" class="edge">
<title>NotSkippingV0.Processing&#45;&gt;NotSkipping.Idle</title>
<path fill="none" stroke="black" d="M304.26,-411.25C304.26,-455.74 304.26,-574.61 304.26,-628.62"/>
<polygon fill="black" stroke="black" points="300.76,-628.81 304.26,-638.81 307.76,-628.81 300.76,-628.81"/>
<text text-anchor="middle" x="385.76" y="-543.8" font-family="Times,serif" font-size="14.00">[Version.isPHD(value)]</text>
<text text-anchor="middle" x="385.76" y="-528.8" font-family="Times,serif" font-size="14.00">&#45;&gt; FILTER_ACCEPT</text>
</g>
<!-- NotSkippingV0.Processing&#45;&gt;SkippingPrefix.Idle -->
<g id="edge11" class="edge">
<title>NotSkippingV0.Processing&#45;&gt;SkippingPrefix.Idle</title>
<path fill="none" stroke="black" d="M448.41,-376.93C508.52,-369.95 565.63,-362.09 570.26,-357 622.9,-299.12 594.8,-196.31 577.11,-147.78"/>
<polygon fill="black" stroke="black" points="580.33,-146.4 573.53,-138.28 573.78,-148.87 580.33,-146.4"/>
<text text-anchor="middle" x="720.26" y="-297.8" font-family="Times,serif" font-size="14.00">[key.startsWith(&lt;ReplayPrefix&gt;)]</text>
<text text-anchor="middle" x="720.26" y="-282.8" font-family="Times,serif" font-size="14.00">/ prefix &lt;&#45; &lt;ReplayPrefix&gt;</text>
<text text-anchor="middle" x="720.26" y="-267.8" font-family="Times,serif" font-size="14.00">&#45;&gt; FILTER_SKIP</text>
</g>
<!-- NotSkippingCommon.Processing -->
<g id="node10" class="node">
<title>NotSkippingCommon.Processing</title>
<path fill="lightblue" stroke="black" d="M436.26,-304.5C436.26,-304.5 172.26,-304.5 172.26,-304.5 166.26,-304.5 160.26,-298.5 160.26,-292.5 160.26,-292.5 160.26,-280.5 160.26,-280.5 160.26,-274.5 166.26,-268.5 172.26,-268.5 172.26,-268.5 436.26,-268.5 436.26,-268.5 442.26,-268.5 448.26,-274.5 448.26,-280.5 448.26,-280.5 448.26,-292.5 448.26,-292.5 448.26,-298.5 442.26,-304.5 436.26,-304.5"/>
<text text-anchor="middle" x="304.26" y="-282.7" font-family="Times,serif" font-size="16.00">NotSkippingCommon</text>
</g>
<!-- NotSkippingV0.Processing&#45;&gt;NotSkippingCommon.Processing -->
<g id="edge13" class="edge">
<title>NotSkippingV0.Processing&#45;&gt;NotSkippingCommon.Processing</title>
<path fill="none" stroke="black" d="M304.26,-374.74C304.26,-358.48 304.26,-333.85 304.26,-314.9"/>
<polygon fill="black" stroke="black" points="307.76,-314.78 304.26,-304.78 300.76,-314.78 307.76,-314.78"/>
<text text-anchor="middle" x="435.26" y="-345.8" font-family="Times,serif" font-size="14.00">[not key.startsWith(&lt;ReplayPrefix&gt;)</text>
<text text-anchor="middle" x="435.26" y="-330.8" font-family="Times,serif" font-size="14.00">and not Version.isPHD(value)]</text>
</g>
<!-- NotSkippingV1.Processing&#45;&gt;NotSkippingCommon.Processing -->
<g id="edge14" class="edge">
<title>NotSkippingV1.Processing&#45;&gt;NotSkippingCommon.Processing</title>
<path fill="none" stroke="black" d="M616.43,-374.83C606.75,-359.62 590.48,-338.14 570.26,-327 549.98,-315.83 505.48,-307.38 458.57,-301.23"/>
<polygon fill="black" stroke="black" points="458.9,-297.74 448.53,-299.95 458.01,-304.69 458.9,-297.74"/>
<text text-anchor="middle" x="632.26" y="-338.3" font-family="Times,serif" font-size="14.00">[always]</text>
</g>
<!-- NotSkippingCommon.Processing&#45;&gt;END -->
<g id="edge15" class="edge">
<title>NotSkippingCommon.Processing&#45;&gt;END</title>
<path fill="none" stroke="black" d="M159.92,-279.56C109.8,-274.24 62.13,-264.33 46.26,-246 20.92,-216.72 30.42,-167.54 38.5,-140.42"/>
<polygon fill="black" stroke="black" points="41.94,-141.16 41.67,-130.57 35.27,-139.02 41.94,-141.16"/>
<text text-anchor="middle" x="152.76" y="-212.3" font-family="Times,serif" font-size="14.00">[isListableKey(key, value) and</text>
<text text-anchor="middle" x="152.76" y="-197.3" font-family="Times,serif" font-size="14.00">Keys == maxKeys]</text>
<text text-anchor="middle" x="152.76" y="-182.3" font-family="Times,serif" font-size="14.00">&#45;&gt; FILTER_END</text>
</g>
<!-- NotSkippingCommon.Processing&#45;&gt;NotSkipping.Idle -->
<g id="edge17" class="edge">
<title>NotSkippingCommon.Processing&#45;&gt;NotSkipping.Idle</title>
<path fill="none" stroke="black" d="M214.74,-304.54C146.51,-322.73 57.06,-358.99 13.26,-429 -49.27,-528.95 128.43,-602.49 233.32,-635.95"/>
<polygon fill="black" stroke="black" points="232.34,-639.31 242.93,-638.97 234.43,-632.63 232.34,-639.31"/>
<text text-anchor="middle" x="156.76" y="-492.8" font-family="Times,serif" font-size="14.00">[isListableKey(key, value) and</text>
<text text-anchor="middle" x="156.76" y="-477.8" font-family="Times,serif" font-size="14.00">nKeys &lt; maxKeys and</text>
<text text-anchor="middle" x="156.76" y="-462.8" font-family="Times,serif" font-size="14.00">not hasDelimiter(key)]</text>
<text text-anchor="middle" x="156.76" y="-447.8" font-family="Times,serif" font-size="14.00">/ Contents.append(key, versionId, value)</text>
<text text-anchor="middle" x="156.76" y="-432.8" font-family="Times,serif" font-size="14.00">&#45;&gt; FILTER_ACCEPT</text>
</g>
<!-- NotSkippingCommon.Processing&#45;&gt;SkippingPrefix.Idle -->
<g id="edge16" class="edge">
<title>NotSkippingCommon.Processing&#45;&gt;SkippingPrefix.Idle</title>
<path fill="none" stroke="black" d="M292.14,-268.23C288.18,-261.59 284.27,-253.75 282.26,-246 272.21,-207.28 255.76,-185.96 282.26,-156 293.6,-143.18 374.98,-134.02 447.74,-128.3"/>
<polygon fill="black" stroke="black" points="448.24,-131.77 457.94,-127.51 447.7,-124.79 448.24,-131.77"/>
<text text-anchor="middle" x="428.26" y="-234.8" font-family="Times,serif" font-size="14.00">[isListableKey(key, value) and</text>
<text text-anchor="middle" x="428.26" y="-219.8" font-family="Times,serif" font-size="14.00">nKeys &lt; maxKeys and</text>
<text text-anchor="middle" x="428.26" y="-204.8" font-family="Times,serif" font-size="14.00">hasDelimiter(key)]</text>
<text text-anchor="middle" x="428.26" y="-189.8" font-family="Times,serif" font-size="14.00">/ prefix &lt;&#45; prefixOf(key)</text>
<text text-anchor="middle" x="428.26" y="-174.8" font-family="Times,serif" font-size="14.00">/ CommonPrefixes.append(prefixOf(key))</text>
<text text-anchor="middle" x="428.26" y="-159.8" font-family="Times,serif" font-size="14.00">&#45;&gt; FILTER_ACCEPT</text>
</g>
<!-- SkippingPrefix.Processing&#45;&gt;SkippingPrefix.Idle -->
<g id="edge18" class="edge">
<title>SkippingPrefix.Processing&#45;&gt;SkippingPrefix.Idle</title>
<path fill="none" stroke="black" d="M681.57,-36.04C679.28,-50.54 673.9,-71.03 661.26,-84 656.4,-88.99 650.77,-93.28 644.72,-96.95"/>
<polygon fill="black" stroke="black" points="642.71,-94.06 635.6,-101.92 646.05,-100.21 642.71,-94.06"/>
<text text-anchor="middle" x="759.26" y="-72.8" font-family="Times,serif" font-size="14.00">[key.startsWith(prefix)]</text>
<text text-anchor="middle" x="759.26" y="-57.8" font-family="Times,serif" font-size="14.00">&#45;&gt; FILTER_SKIP</text>
</g>
<!-- SkippingPrefix.Processing&#45;&gt;NotSkipping.Processing -->
<g id="edge19" class="edge">
<title>SkippingPrefix.Processing&#45;&gt;NotSkipping.Processing</title>
<path fill="none" stroke="black" d="M791.46,-33.51C815.84,-38.71 837.21,-45.46 846.26,-54 868.07,-74.57 864.26,-89.02 864.26,-119 864.26,-394 864.26,-394 864.26,-394 864.26,-462.4 791.27,-499.6 726.64,-519.12"/>
<polygon fill="black" stroke="black" points="725.39,-515.84 716.77,-521.99 727.35,-522.56 725.39,-515.84"/>
<text text-anchor="middle" x="961.26" y="-282.8" font-family="Times,serif" font-size="14.00">[not key.startsWith(prefix)]</text>
</g>
<!-- WaitForNullKey.Processing&#45;&gt;NotSkipping.Processing -->
<g id="edge9" class="edge">
<title>WaitForNullKey.Processing&#45;&gt;NotSkipping.Processing</title>
<path fill="none" stroke="black" d="M599.08,-725.78C604.81,-690.67 617.89,-610.59 624.8,-568.31"/>
<polygon fill="black" stroke="black" points="628.3,-568.61 626.46,-558.18 621.39,-567.48 628.3,-568.61"/>
<text text-anchor="middle" x="707.26" y="-653.3" font-family="Times,serif" font-size="14.00">master(key) != keyMarker</text>
</g>
<!-- WaitForNullKey.Processing&#45;&gt;SkippingVersions.Processing -->
<g id="edge10" class="edge">
<title>WaitForNullKey.Processing&#45;&gt;SkippingVersions.Processing</title>
<path fill="none" stroke="black" d="M704.4,-726.26C797.32,-711.87 931.09,-691.16 1026.87,-676.33"/>
<polygon fill="black" stroke="black" points="1027.55,-679.77 1036.89,-674.78 1026.47,-672.85 1027.55,-679.77"/>
<text text-anchor="middle" x="1001.26" y="-696.8" font-family="Times,serif" font-size="14.00">master(key) == keyMarker</text>
</g>
<!-- SkippingVersions.Processing&#45;&gt;SkippingVersions.Idle -->
<g id="edge21" class="edge">
<title>SkippingVersions.Processing&#45;&gt;SkippingVersions.Idle</title>
<path fill="none" stroke="black" d="M1241.89,-638.98C1249.74,-634.29 1256.75,-628.4 1262.26,-621 1274.21,-604.96 1274.21,-592.04 1262.26,-576 1258.82,-571.38 1254.79,-567.34 1250.33,-563.82"/>
<polygon fill="black" stroke="black" points="1252.11,-560.8 1241.89,-558.02 1248.15,-566.57 1252.11,-560.8"/>
<text text-anchor="middle" x="1392.26" y="-609.8" font-family="Times,serif" font-size="14.00">master(key) === keyMarker and </text>
<text text-anchor="middle" x="1392.26" y="-594.8" font-family="Times,serif" font-size="14.00">versionId &lt; versionIdMarker</text>
<text text-anchor="middle" x="1392.26" y="-579.8" font-family="Times,serif" font-size="14.00">&#45;&gt; FILTER_SKIP</text>
</g>
<!-- SkippingVersions.Processing&#45;&gt;SkippingVersions.Idle -->
<g id="edge22" class="edge">
<title>SkippingVersions.Processing&#45;&gt;SkippingVersions.Idle</title>
<path fill="none" stroke="black" d="M1036.97,-654.38C978.97,-650.96 915.73,-642.25 897.26,-621 884.15,-605.9 884.15,-591.1 897.26,-576 914.65,-555.99 971.71,-547.1 1026.73,-543.28"/>
<polygon fill="black" stroke="black" points="1027.21,-546.76 1036.97,-542.62 1026.76,-539.77 1027.21,-546.76"/>
<text text-anchor="middle" x="1019.26" y="-609.8" font-family="Times,serif" font-size="14.00">master(key) === keyMarker and </text>
<text text-anchor="middle" x="1019.26" y="-594.8" font-family="Times,serif" font-size="14.00">versionId == versionIdMarker</text>
<text text-anchor="middle" x="1019.26" y="-579.8" font-family="Times,serif" font-size="14.00">&#45;&gt; FILTER_ACCEPT</text>
</g>
<!-- SkippingVersions.Processing&#45;&gt;NotSkipping.Processing -->
<g id="edge20" class="edge">
<title>SkippingVersions.Processing&#45;&gt;NotSkipping.Processing</title>
<path fill="none" stroke="black" d="M1037.02,-651.24C897.84,-644.67 672.13,-632.37 657.26,-621 641.04,-608.6 634.18,-586.13 631.3,-568.16"/>
<polygon fill="black" stroke="black" points="634.76,-567.68 630.02,-558.21 627.82,-568.57 634.76,-567.68"/>
<text text-anchor="middle" x="770.26" y="-602.3" font-family="Times,serif" font-size="14.00">master(key) !== keyMarker or </text>
<text text-anchor="middle" x="770.26" y="-587.3" font-family="Times,serif" font-size="14.00">versionId &gt; versionIdMarker</text>
</g>
</g>
</svg>

After

Width:  |  Height:  |  Size: 21 KiB

View File

@ -21,6 +21,7 @@ import * as retention from './lib/s3middleware/objectRetention';
import * as lifecycleHelpers from './lib/s3middleware/lifecycleHelpers'; import * as lifecycleHelpers from './lib/s3middleware/lifecycleHelpers';
export { default as errors } from './lib/errors'; export { default as errors } from './lib/errors';
export { default as Clustering } from './lib/Clustering'; export { default as Clustering } from './lib/Clustering';
export * as ClusterRPC from './lib/clustering/ClusterRPC';
export * as ipCheck from './lib/ipCheck'; export * as ipCheck from './lib/ipCheck';
export * as auth from './lib/auth/auth'; export * as auth from './lib/auth/auth';
export * as constants from './lib/constants'; export * as constants from './lib/constants';
@ -43,11 +44,16 @@ export const algorithms = {
DelimiterVersions: require('./lib/algos/list/delimiterVersions').DelimiterVersions, DelimiterVersions: require('./lib/algos/list/delimiterVersions').DelimiterVersions,
DelimiterMaster: require('./lib/algos/list/delimiterMaster').DelimiterMaster, DelimiterMaster: require('./lib/algos/list/delimiterMaster').DelimiterMaster,
MPU: require('./lib/algos/list/MPU').MultipartUploads, MPU: require('./lib/algos/list/MPU').MultipartUploads,
DelimiterCurrent: require('./lib/algos/list/delimiterCurrent').DelimiterCurrent,
DelimiterNonCurrent: require('./lib/algos/list/delimiterNonCurrent').DelimiterNonCurrent,
DelimiterOrphanDeleteMarker: require('./lib/algos/list/delimiterOrphanDeleteMarker').DelimiterOrphanDeleteMarker,
}, },
listTools: { listTools: {
DelimiterTools: require('./lib/algos/list/tools'), DelimiterTools: require('./lib/algos/list/tools'),
}, },
cache: { cache: {
GapSet: require('./lib/algos/cache/GapSet'),
GapCache: require('./lib/algos/cache/GapCache'),
LRUCache: require('./lib/algos/cache/LRUCache'), LRUCache: require('./lib/algos/cache/LRUCache'),
}, },
stream: { stream: {

363
lib/algos/cache/GapCache.ts vendored Normal file
View File

@ -0,0 +1,363 @@
import { OrderedSet } from '@js-sdsl/ordered-set';
import {
default as GapSet,
GapSetEntry,
} from './GapSet';
// the API is similar but is not strictly a superset of GapSetInterface
// so we don't extend from it
export interface GapCacheInterface {
exposureDelayMs: number;
maxGapWeight: number;
size: number;
setGap: (firstKey: string, lastKey: string, weight: number) => void;
removeOverlappingGaps: (overlappingKeys: string[]) => number;
lookupGap: (minKey: string, maxKey?: string) => Promise<GapSetEntry | null>;
[Symbol.iterator]: () => Iterator<GapSetEntry>;
toArray: () => GapSetEntry[];
};
class GapCacheUpdateSet {
newGaps: GapSet;
updatedKeys: OrderedSet<string>;
constructor(maxGapWeight: number) {
this.newGaps = new GapSet(maxGapWeight);
this.updatedKeys = new OrderedSet();
}
addUpdateBatch(updatedKeys: OrderedSet<string>): void {
this.updatedKeys.union(updatedKeys);
}
};
/**
* Cache of listing "gaps" i.e. ranges of keys that can be skipped
* over during listing (because they only contain delete markers as
* latest versions).
*
* Typically, a single GapCache instance would be attached to a raft session.
*
* The API usage is as follows:
*
* - Initialize a GapCache instance by calling start() (this starts an internal timer)
*
* - Insert a gap or update an existing one via setGap()
*
* - Lookup existing gaps via lookupGap()
*
* - Invalidate gaps that overlap a specific set of keys via removeOverlappingGaps()
*
* - Shut down a GapCache instance by calling stop() (this stops the internal timer)
*
* Gaps inserted via setGap() are not exposed immediately to lookupGap(), but only:
*
* - after a certain delay always larger than 'exposureDelayMs' and usually shorter
* than twice this value (but might be slightly longer in rare cases)
*
* - and only if they haven't been invalidated by a recent call to removeOverlappingGaps()
*
* This ensures atomicity between gap creation and invalidation from updates under
* the condition that a gap is created from first key to last key within the time defined
* by 'exposureDelayMs'.
*
* The implementation is based on two extra temporary "update sets" on top of the main
* exposed gap set, one called "staging" and the other "frozen", each containing a
* temporary updated gap set and a list of updated keys to invalidate gaps with (coming
* from calls to removeOverlappingGaps()). Every "exposureDelayMs" milliseconds, the frozen
* gaps are invalidated by all key updates coming from either of the "staging" or "frozen"
* update set, then merged into the exposed gaps set, after which the staging updates become
* the frozen updates and won't receive any new gap until the next cycle.
*/
export default class GapCache implements GapCacheInterface {
_exposureDelayMs: number;
maxGaps: number;
_stagingUpdates: GapCacheUpdateSet;
_frozenUpdates: GapCacheUpdateSet;
_exposedGaps: GapSet;
_exposeFrozenInterval: NodeJS.Timeout | null;
/**
* @constructor
*
* @param {number} exposureDelayMs - minimum delay between
* insertion of a gap via setGap() and its exposure via
* lookupGap()
* @param {number} maxGaps - maximum number of cached gaps, after
* which no new gap can be added by setGap(). (Note: a future
* improvement could replace this by an eviction strategy)
* @param {number} maxGapWeight - maximum "weight" of individual
* cached gaps, which is also the granularity for
* invalidation. Individual gaps can be chained together,
* which lookupGap() transparently consolidates in the response
* into a single large gap.
*/
constructor(exposureDelayMs: number, maxGaps: number, maxGapWeight: number) {
this._exposureDelayMs = exposureDelayMs;
this.maxGaps = maxGaps;
this._stagingUpdates = new GapCacheUpdateSet(maxGapWeight);
this._frozenUpdates = new GapCacheUpdateSet(maxGapWeight);
this._exposedGaps = new GapSet(maxGapWeight);
this._exposeFrozenInterval = null;
}
/**
* Create a GapCache from an array of exposed gap entries (used in tests)
*
* @return {GapCache} - a new GapCache instance
*/
static createFromArray(
gaps: GapSetEntry[],
exposureDelayMs: number,
maxGaps: number,
maxGapWeight: number
): GapCache {
const gapCache = new GapCache(exposureDelayMs, maxGaps, maxGapWeight);
gapCache._exposedGaps = GapSet.createFromArray(gaps, maxGapWeight)
return gapCache;
}
/**
* Internal helper to remove gaps in the staging and frozen sets
* overlapping with previously updated keys, right before the
* frozen gaps get exposed.
*
* @return {undefined}
*/
_removeOverlappingGapsBeforeExpose(): void {
for (const { updatedKeys } of [this._stagingUpdates, this._frozenUpdates]) {
if (updatedKeys.size() === 0) {
continue;
}
for (const { newGaps } of [this._stagingUpdates, this._frozenUpdates]) {
if (newGaps.size === 0) {
continue;
}
newGaps.removeOverlappingGaps(updatedKeys);
}
}
}
/**
* This function is the core mechanism that updates the exposed gaps in the
* cache. It is called on a regular interval defined by 'exposureDelayMs'.
*
* It does the following in order:
*
* - remove gaps from the frozen set that overlap with any key present in a
* batch passed to removeOverlappingGaps() since the last two triggers of
* _exposeFrozen()
*
* - merge the remaining gaps from the frozen set to the exposed set, which
* makes them visible from calls to lookupGap()
*
* - rotate by freezing the currently staging updates and initiating a new
* staging updates set
*
* @return {undefined}
*/
_exposeFrozen(): void {
this._removeOverlappingGapsBeforeExpose();
for (const gap of this._frozenUpdates.newGaps) {
// Use a trivial strategy to keep the cache size within
// limits: refuse to add new gaps when the size is above
// the 'maxGaps' threshold. We solely rely on
// removeOverlappingGaps() to make space for new gaps.
if (this._exposedGaps.size < this.maxGaps) {
this._exposedGaps.setGap(gap.firstKey, gap.lastKey, gap.weight);
}
}
this._frozenUpdates = this._stagingUpdates;
this._stagingUpdates = new GapCacheUpdateSet(this.maxGapWeight);
}
/**
* Start the internal GapCache timer
*
* @return {undefined}
*/
start(): void {
if (this._exposeFrozenInterval) {
return;
}
this._exposeFrozenInterval = setInterval(
() => this._exposeFrozen(),
this._exposureDelayMs);
}
/**
* Stop the internal GapCache timer
*
* @return {undefined}
*/
stop(): void {
if (this._exposeFrozenInterval) {
clearInterval(this._exposeFrozenInterval);
this._exposeFrozenInterval = null;
}
}
/**
* Record a gap between two keys, associated with a weight to
* limit individual gap's spanning ranges in the cache, for a more
* granular invalidation.
*
* The function handles splitting and merging existing gaps to
* maintain an optimal weight of cache entries.
*
* NOTE 1: the caller must ensure that the full length of the gap
* between 'firstKey' and 'lastKey' has been built from a listing
* snapshot that is more recent than 'exposureDelayMs' milliseconds,
* in order to guarantee that the exposed gap will be fully
* covered (and potentially invalidated) from recent calls to
* removeOverlappingGaps().
*
* NOTE 2: a usual pattern when building a large gap from multiple
* calls to setGap() is to start the next gap from 'lastKey',
* which will be passed as 'firstKey' in the next call, so that
* gaps can be chained together and consolidated by lookupGap().
*
* @param {string} firstKey - first key of the gap
* @param {string} lastKey - last key of the gap, must be greater
* or equal than 'firstKey'
* @param {number} weight - total weight between 'firstKey' and 'lastKey'
* @return {undefined}
*/
setGap(firstKey: string, lastKey: string, weight: number): void {
this._stagingUpdates.newGaps.setGap(firstKey, lastKey, weight);
}
/**
* Remove gaps that overlap with a given set of keys. Used to
* invalidate gaps when keys are inserted or deleted.
*
* @param {OrderedSet<string> | string[]} overlappingKeys - remove gaps that
* overlap with any of this set of keys
* @return {number} - how many gaps were removed from the exposed
* gaps only (overlapping gaps not yet exposed are also invalidated
* but are not accounted for in the returned value)
*/
removeOverlappingGaps(overlappingKeys: OrderedSet<string> | string[]): number {
let overlappingKeysSet;
if (Array.isArray(overlappingKeys)) {
overlappingKeysSet = new OrderedSet(overlappingKeys);
} else {
overlappingKeysSet = overlappingKeys;
}
this._stagingUpdates.addUpdateBatch(overlappingKeysSet);
return this._exposedGaps.removeOverlappingGaps(overlappingKeysSet);
}
/**
* Lookup the next exposed gap that overlaps with [minKey, maxKey]. Internally
* chained gaps are coalesced in the response into a single contiguous large gap.
*
* @param {string} minKey - minimum key overlapping with the returned gap
* @param {string} [maxKey] - maximum key overlapping with the returned gap
* @return {Promise<GapSetEntry | null>} - result of the lookup if a gap
* was found, null otherwise, as a Promise
*/
lookupGap(minKey: string, maxKey?: string): Promise<GapSetEntry | null> {
return this._exposedGaps.lookupGap(minKey, maxKey);
}
/**
* Get the maximum weight setting for individual gaps.
*
* @return {number} - maximum weight of individual gaps
*/
get maxGapWeight(): number {
return this._exposedGaps.maxWeight;
}
/**
* Set the maximum weight setting for individual gaps.
*
* @param {number} gapWeight - maximum weight of individual gaps
*/
set maxGapWeight(gapWeight: number) {
this._exposedGaps.maxWeight = gapWeight;
// also update transient gap sets
this._stagingUpdates.newGaps.maxWeight = gapWeight;
this._frozenUpdates.newGaps.maxWeight = gapWeight;
}
/**
* Get the exposure delay in milliseconds, which is the minimum
* time after which newly cached gaps will be exposed by
* lookupGap().
*
* @return {number} - exposure delay in milliseconds
*/
get exposureDelayMs(): number {
return this._exposureDelayMs;
}
/**
* Set the exposure delay in milliseconds, which is the minimum
* time after which newly cached gaps will be exposed by
* lookupGap(). Setting this attribute automatically updates the
* internal state to honor the new value.
*
* @param {number} - exposure delay in milliseconds
*/
set exposureDelayMs(exposureDelayMs: number) {
if (exposureDelayMs !== this._exposureDelayMs) {
this._exposureDelayMs = exposureDelayMs;
if (this._exposeFrozenInterval) {
// invalidate all pending gap updates, as the new interval may not be
// safe for them
this._stagingUpdates = new GapCacheUpdateSet(this.maxGapWeight);
this._frozenUpdates = new GapCacheUpdateSet(this.maxGapWeight);
// reinitialize the _exposeFrozenInterval timer with the updated delay
this.stop();
this.start();
}
}
}
/**
* Get the number of exposed gaps
*
* @return {number} number of exposed gaps
*/
get size(): number {
return this._exposedGaps.size;
}
/**
* Iterate over exposed gaps
*
* @return {Iterator<GapSetEntry>} an iterator over exposed gaps
*/
[Symbol.iterator](): Iterator<GapSetEntry> {
return this._exposedGaps[Symbol.iterator]();
}
/**
* Get an array of all exposed gaps
*
* @return {GapSetEntry[]} array of exposed gaps
*/
toArray(): GapSetEntry[] {
return this._exposedGaps.toArray();
}
/**
* Clear all exposed and staging gaps from the cache.
*
* Note: retains invalidating updates from removeOverlappingGaps()
* for correctness of gaps inserted afterwards.
*
* @return {undefined}
*/
clear(): void {
this._stagingUpdates.newGaps = new GapSet(this.maxGapWeight);
this._frozenUpdates.newGaps = new GapSet(this.maxGapWeight);
this._exposedGaps = new GapSet(this.maxGapWeight);
}
}

366
lib/algos/cache/GapSet.ts vendored Normal file
View File

@ -0,0 +1,366 @@
import assert from 'assert';
import { OrderedSet } from '@js-sdsl/ordered-set';
import errors from '../../errors';
export type GapSetEntry = {
firstKey: string,
lastKey: string,
weight: number,
};
export interface GapSetInterface {
maxWeight: number;
size: number;
setGap: (firstKey: string, lastKey: string, weight: number) => GapSetEntry;
removeOverlappingGaps: (overlappingKeys: string[]) => number;
lookupGap: (minKey: string, maxKey?: string) => Promise<GapSetEntry | null>;
[Symbol.iterator]: () => Iterator<GapSetEntry>;
toArray: () => GapSetEntry[];
};
/**
* Specialized data structure to support caching of listing "gaps",
* i.e. ranges of keys that can be skipped over during listing
* (because they only contain delete markers as latest versions)
*/
export default class GapSet implements GapSetInterface, Iterable<GapSetEntry> {
_gaps: OrderedSet<GapSetEntry>;
_maxWeight: number;
/**
* @constructor
* @param {number} maxWeight - weight threshold for each cached
* gap (unitless). Triggers splitting gaps when reached
*/
constructor(maxWeight: number) {
this._gaps = new OrderedSet(
[],
(left: GapSetEntry, right: GapSetEntry) => (
left.firstKey < right.firstKey ? -1 :
left.firstKey > right.firstKey ? 1 : 0
)
);
this._maxWeight = maxWeight;
}
/**
* Create a GapSet from an array of gap entries (used in tests)
*/
static createFromArray(gaps: GapSetEntry[], maxWeight: number): GapSet {
const gapSet = new GapSet(maxWeight);
for (const gap of gaps) {
gapSet._gaps.insert(gap);
}
return gapSet;
}
/**
* Record a gap between two keys, associated with a weight to limit
* individual gap sizes in the cache.
*
* The function handles splitting and merging existing gaps to
* maintain an optimal weight of cache entries.
*
* @param {string} firstKey - first key of the gap
* @param {string} lastKey - last key of the gap, must be greater
* or equal than 'firstKey'
* @param {number} weight - total weight between 'firstKey' and 'lastKey'
* @return {GapSetEntry} - existing or new gap entry
*/
setGap(firstKey: string, lastKey: string, weight: number): GapSetEntry {
assert(lastKey >= firstKey);
// Step 1/4: Find the closest left-overlapping gap, and either re-use it
// or chain it with a new gap depending on the weights if it exists (otherwise
// just creates a new gap).
const curGapIt = this._gaps.reverseLowerBound(<GapSetEntry>{ firstKey });
let curGap;
if (curGapIt.isAccessible()) {
curGap = curGapIt.pointer;
if (curGap.lastKey >= lastKey) {
// return fully overlapping gap already cached
return curGap;
}
}
let remainingWeight = weight;
if (!curGap // no previous gap
|| curGap.lastKey < firstKey // previous gap not overlapping
|| (curGap.lastKey === firstKey // previous gap overlapping by one key...
&& curGap.weight + weight > this._maxWeight) // ...but we can't extend it
) {
// create a new gap indexed by 'firstKey'
curGap = { firstKey, lastKey: firstKey, weight: 0 };
this._gaps.insert(curGap);
} else if (curGap.lastKey > firstKey && weight > this._maxWeight) {
// previous gap is either fully or partially contained in the new gap
// and cannot be extended: substract its weight from the total (heuristic
// in case the previous gap doesn't start at 'firstKey', which is the
// uncommon case)
remainingWeight -= curGap.weight;
// there may be an existing chained gap starting with the previous gap's
// 'lastKey': use it if it exists
const chainedGapIt = this._gaps.find(<GapSetEntry>{ firstKey: curGap.lastKey });
if (chainedGapIt.isAccessible()) {
curGap = chainedGapIt.pointer;
} else {
// no existing chained gap: chain a new gap to the previous gap
curGap = {
firstKey: curGap.lastKey,
lastKey: curGap.lastKey,
weight: 0,
};
this._gaps.insert(curGap);
}
}
// Step 2/4: Cleanup existing gaps fully included in firstKey -> lastKey, and
// aggregate their weights in curGap to define the minimum weight up to the
// last merged gap.
let nextGap;
while (true) {
const nextGapIt = this._gaps.upperBound(<GapSetEntry>{ firstKey: curGap.firstKey });
nextGap = nextGapIt.isAccessible() && nextGapIt.pointer;
// stop the cleanup when no more gap or if the next gap is not fully
// included in curGap
if (!nextGap || nextGap.lastKey > lastKey) {
break;
}
this._gaps.eraseElementByIterator(nextGapIt);
curGap.lastKey = nextGap.lastKey;
curGap.weight += nextGap.weight;
}
// Step 3/4: Extend curGap to lastKey, adjusting the weight.
// At this point, curGap weight is the minimum weight of the finished gap, save it
// for step 4.
let minMergedWeight = curGap.weight;
if (curGap.lastKey === firstKey && firstKey !== lastKey) {
// extend the existing gap by the full amount 'firstKey -> lastKey'
curGap.lastKey = lastKey;
curGap.weight += remainingWeight;
} else if (curGap.lastKey <= lastKey) {
curGap.lastKey = lastKey;
curGap.weight = remainingWeight;
}
// Step 4/4: Find the closest right-overlapping gap, and if it exists, either merge
// it or chain it with curGap depending on the weights.
if (nextGap && nextGap.firstKey <= lastKey) {
// nextGap overlaps with the new gap: check if we can merge it
minMergedWeight += nextGap.weight;
let mergedWeight;
if (lastKey === nextGap.firstKey) {
// nextGap is chained with curGap: add the full weight of nextGap
mergedWeight = curGap.weight + nextGap.weight;
} else {
// strict overlap: don't add nextGap's weight unless
// it's larger than the sum of merged ranges (as it is
// then included in `minMergedWeight`)
mergedWeight = Math.max(curGap.weight, minMergedWeight);
}
if (mergedWeight <= this._maxWeight) {
// merge nextGap into curGap
curGap.lastKey = nextGap.lastKey;
curGap.weight = mergedWeight;
this._gaps.eraseElementByKey(nextGap);
} else {
// adjust the last key to chain with nextGap and substract the next
// gap's weight from curGap (heuristic)
curGap.lastKey = nextGap.firstKey;
curGap.weight = Math.max(mergedWeight - nextGap.weight, 0);
curGap = nextGap;
}
}
// return a copy of curGap
return Object.assign({}, curGap);
}
/**
* Remove gaps that overlap with one or more keys in a given array or
* OrderedSet. Used to invalidate gaps when keys are inserted or deleted.
*
* @param {OrderedSet<string> | string[]} overlappingKeys - remove gaps that overlap
* with any of this set of keys
* @return {number} - how many gaps were removed
*/
removeOverlappingGaps(overlappingKeys: OrderedSet<string> | string[]): number {
// To optimize processing with a large number of keys and/or gaps, this function:
//
// 1. converts the overlappingKeys array to a OrderedSet (if not already a OrderedSet)
// 2. queries both the gaps set and the overlapping keys set in a loop, which allows:
// - skipping ranges of overlapping keys at once when there is no new overlapping gap
// - skipping ranges of gaps at once when there is no overlapping key
//
// This way, it is efficient when the number of non-overlapping gaps is large
// (which is the most common case in practice).
let overlappingKeysSet;
if (Array.isArray(overlappingKeys)) {
overlappingKeysSet = new OrderedSet(overlappingKeys);
} else {
overlappingKeysSet = overlappingKeys;
}
const firstKeyIt = overlappingKeysSet.begin();
let currentKey = firstKeyIt.isAccessible() && firstKeyIt.pointer;
let nRemoved = 0;
while (currentKey) {
const closestGapIt = this._gaps.reverseUpperBound(<GapSetEntry>{ firstKey: currentKey });
if (closestGapIt.isAccessible()) {
const closestGap = closestGapIt.pointer;
if (currentKey <= closestGap.lastKey) {
// currentKey overlaps closestGap: remove the gap
this._gaps.eraseElementByIterator(closestGapIt);
nRemoved += 1;
}
}
const nextGapIt = this._gaps.lowerBound(<GapSetEntry>{ firstKey: currentKey });
if (!nextGapIt.isAccessible()) {
// no more gap: we're done
return nRemoved;
}
const nextGap = nextGapIt.pointer;
// advance to the last key potentially overlapping with nextGap
let currentKeyIt = overlappingKeysSet.reverseLowerBound(nextGap.lastKey);
if (currentKeyIt.isAccessible()) {
currentKey = currentKeyIt.pointer;
if (currentKey >= nextGap.firstKey) {
// currentKey overlaps nextGap: remove the gap
this._gaps.eraseElementByIterator(nextGapIt);
nRemoved += 1;
}
}
// advance to the first key potentially overlapping with another gap
currentKeyIt = overlappingKeysSet.lowerBound(nextGap.lastKey);
currentKey = currentKeyIt.isAccessible() && currentKeyIt.pointer;
}
return nRemoved;
}
/**
* Internal helper to coalesce multiple chained gaps into a single gap.
*
* It is only used to construct lookupGap() return values and
* doesn't modify the GapSet.
*
* NOTE: The function may take a noticeable amount of time and CPU
* to execute if a large number of chained gaps have to be
* coalesced, but it should never take more than a few seconds. In
* most cases it should take less than a millisecond. It regularly
* yields to the nodejs event loop to avoid blocking it during a
* long execution.
*
* @param {GapSetEntry} firstGap - first gap of the chain to coalesce with
* the next ones in the chain
* @return {Promise<GapSetEntry>} - a new coalesced entry, as a Promise
*/
_coalesceGapChain(firstGap: GapSetEntry): Promise<GapSetEntry> {
return new Promise(resolve => {
const coalescedGap: GapSetEntry = Object.assign({}, firstGap);
const coalesceGapChainIteration = () => {
// efficiency trade-off: 100 iterations of log(N) complexity lookups should
// not block the event loop for too long
for (let opCounter = 0; opCounter < 100; ++opCounter) {
const chainedGapIt = this._gaps.find(
<GapSetEntry>{ firstKey: coalescedGap.lastKey });
if (!chainedGapIt.isAccessible()) {
// chain is complete
return resolve(coalescedGap);
}
const chainedGap = chainedGapIt.pointer;
if (chainedGap.firstKey === chainedGap.lastKey) {
// found a single-key gap: chain is complete
return resolve(coalescedGap);
}
coalescedGap.lastKey = chainedGap.lastKey;
coalescedGap.weight += chainedGap.weight;
}
// yield to the event loop before continuing the process
// of coalescing the gap chain
return process.nextTick(coalesceGapChainIteration);
};
coalesceGapChainIteration();
});
}
/**
* Lookup the next gap that overlaps with [minKey, maxKey]. Internally chained
* gaps are coalesced in the response into a single contiguous large gap.
*
* @param {string} minKey - minimum key overlapping with the returned gap
* @param {string} [maxKey] - maximum key overlapping with the returned gap
* @return {Promise<GapSetEntry | null>} - result of the lookup if a gap
* was found, null otherwise, as a Promise
*/
async lookupGap(minKey: string, maxKey?: string): Promise<GapSetEntry | null> {
let firstGap: GapSetEntry | null = null;
const minGapIt = this._gaps.reverseLowerBound(<GapSetEntry>{ firstKey: minKey });
const minGap = minGapIt.isAccessible() && minGapIt.pointer;
if (minGap && minGap.lastKey >= minKey) {
firstGap = minGap;
} else {
const maxGapIt = this._gaps.upperBound(<GapSetEntry>{ firstKey: minKey });
const maxGap = maxGapIt.isAccessible() && maxGapIt.pointer;
if (maxGap && (maxKey === undefined || maxGap.firstKey <= maxKey)) {
firstGap = maxGap;
}
}
if (!firstGap) {
return null;
}
return this._coalesceGapChain(firstGap);
}
/**
* Get the maximum weight setting for individual gaps.
*
* @return {number} - maximum weight of individual gaps
*/
get maxWeight(): number {
return this._maxWeight;
}
/**
* Set the maximum weight setting for individual gaps.
*
* @param {number} gapWeight - maximum weight of individual gaps
*/
set maxWeight(gapWeight: number) {
this._maxWeight = gapWeight;
}
/**
* Get the number of gaps stored in this set.
*
* @return {number} - number of gaps stored in this set
*/
get size(): number {
return this._gaps.size();
}
/**
* Iterate over each gap of the set, ordered by first key
*
* @return {Iterator<GapSetEntry>} - an iterator over all gaps
* Example:
* for (const gap of myGapSet) { ... }
*/
[Symbol.iterator](): Iterator<GapSetEntry> {
return this._gaps[Symbol.iterator]();
}
/**
* Return an array containing all gaps, ordered by first key
*
* NOTE: there is a toArray() method in the OrderedSet implementation
* but it does not scale well and overflows the stack quickly. This is
* why we provide an implementation based on an iterator.
*
* @return {GapSetEntry[]} - an array containing all gaps
*/
toArray(): GapSetEntry[] {
return [...this];
}
}

View File

@ -1,6 +1,6 @@
'use strict'; // eslint-disable-line strict 'use strict'; // eslint-disable-line strict
const { FILTER_SKIP, SKIP_NONE } = require('./tools'); const { FILTER_ACCEPT, SKIP_NONE } = require('./tools');
// Use a heuristic to amortize the cost of JSON // Use a heuristic to amortize the cost of JSON
// serialization/deserialization only on largest metadata where the // serialization/deserialization only on largest metadata where the
@ -92,21 +92,26 @@ class Extension {
* @param {object} entry - a listing entry from metadata * @param {object} entry - a listing entry from metadata
* expected format: { key, value } * expected format: { key, value }
* @return {number} - result of filtering the entry: * @return {number} - result of filtering the entry:
* > 0: entry is accepted and included in the result * FILTER_ACCEPT: entry is accepted and may or not be included
* = 0: entry is accepted but not included (skipping) * in the result
* < 0: entry is not accepted, listing should finish * FILTER_SKIP: listing may skip directly (with "gte" param) to
* the key returned by the skipping() method
* FILTER_END: the results are complete, listing can be stopped
*/ */
filter(entry) { filter(/* entry: { key, value } */) {
return entry ? FILTER_SKIP : FILTER_SKIP; return FILTER_ACCEPT;
} }
/** /**
* Provides the insight into why filter is skipping an entry. This could be * Provides the next key at which the listing task is allowed to skip to.
* because it is skipping a range of delimited keys or a range of specific * This could allow to skip over:
* version when doing master version listing. * - a key prefix ending with the delimiter
* - all remaining versions of an object when doing a current
* versions listing in v0 format
* - a cached "gap" of deleted objects when doing a current
* versions listing in v0 format
* *
* @return {string} - the insight: a common prefix or a master key, * @return {string} - the next key at which the listing task is allowed to skip to
* or SKIP_NONE if there is no insight
*/ */
skipping() { skipping() {
return SKIP_NONE; return SKIP_NONE;

View File

@ -1,7 +1,7 @@
'use strict'; // eslint-disable-line strict 'use strict'; // eslint-disable-line strict
const { inc, checkLimit, listingParamsMasterKeysV0ToV1, const { inc, checkLimit, listingParamsMasterKeysV0ToV1,
FILTER_END, FILTER_ACCEPT } = require('./tools'); FILTER_END, FILTER_ACCEPT, SKIP_NONE } = require('./tools');
const DEFAULT_MAX_KEYS = 1000; const DEFAULT_MAX_KEYS = 1000;
const VSConst = require('../../versioning/constants').VersioningConstants; const VSConst = require('../../versioning/constants').VersioningConstants;
const { DbPrefixes, BucketVersioningKeyFormat } = VSConst; const { DbPrefixes, BucketVersioningKeyFormat } = VSConst;
@ -163,7 +163,7 @@ class MultipartUploads {
} }
skipping() { skipping() {
return ''; return SKIP_NONE;
} }
/** /**

View File

@ -2,7 +2,7 @@
const Extension = require('./Extension').default; const Extension = require('./Extension').default;
const { checkLimit, FILTER_END, FILTER_ACCEPT, FILTER_SKIP } = require('./tools'); const { checkLimit, FILTER_END, FILTER_ACCEPT } = require('./tools');
const DEFAULT_MAX_KEYS = 10000; const DEFAULT_MAX_KEYS = 10000;
/** /**
@ -91,7 +91,7 @@ class List extends Extension {
* < 0 : listing done * < 0 : listing done
*/ */
filter(elem) { filter(elem) {
// Check first in case of maxkeys <= 0 // Check if the result array is full
if (this.keys >= this.maxKeys) { if (this.keys >= this.maxKeys) {
return FILTER_END; return FILTER_END;
} }
@ -99,7 +99,7 @@ class List extends Extension {
this.filterKeyStartsWith !== undefined) && this.filterKeyStartsWith !== undefined) &&
typeof elem === 'object' && typeof elem === 'object' &&
!this.customFilter(elem.value)) { !this.customFilter(elem.value)) {
return FILTER_SKIP; return FILTER_ACCEPT;
} }
if (typeof elem === 'object') { if (typeof elem === 'object') {
this.res.push({ this.res.push({

View File

@ -32,7 +32,7 @@ export interface DelimiterFilterState_SkippingPrefix extends FilterState {
type KeyHandler = (key: string, value: string) => FilterReturnValue; type KeyHandler = (key: string, value: string) => FilterReturnValue;
type ResultObject = { export type ResultObject = {
CommonPrefixes: string[]; CommonPrefixes: string[];
Contents: { Contents: {
key: string; key: string;
@ -305,7 +305,7 @@ export class Delimiter extends Extension {
switch (this.state.id) { switch (this.state.id) {
case DelimiterFilterStateId.SkippingPrefix: case DelimiterFilterStateId.SkippingPrefix:
const { prefix } = <DelimiterFilterState_SkippingPrefix> this.state; const { prefix } = <DelimiterFilterState_SkippingPrefix> this.state;
return prefix; return inc(prefix);
default: default:
return SKIP_NONE; return SKIP_NONE;

View File

@ -0,0 +1,127 @@
const { DelimiterMaster } = require('./delimiterMaster');
const { FILTER_ACCEPT, FILTER_END } = require('./tools');
type ResultObject = {
Contents: {
key: string;
value: string;
}[];
IsTruncated: boolean;
NextMarker ?: string;
};
/**
* Handle object listing with parameters. This extends the base class DelimiterMaster
* to return the master/current versions.
*/
class DelimiterCurrent extends DelimiterMaster {
/**
* Delimiter listing of current versions.
* @param {Object} parameters - listing parameters
* @param {String} parameters.beforeDate - limit the response to keys older than beforeDate
* @param {String} parameters.excludedDataStoreName - excluded datatore name
* @param {Number} parameters.maxScannedLifecycleListingEntries - max number of entries to be scanned
* @param {RequestLogger} logger - The logger of the request
* @param {String} [vFormat] - versioning key format
*/
constructor(parameters, logger, vFormat) {
super(parameters, logger, vFormat);
this.beforeDate = parameters.beforeDate;
this.excludedDataStoreName = parameters.excludedDataStoreName;
this.maxScannedLifecycleListingEntries = parameters.maxScannedLifecycleListingEntries;
this.scannedKeys = 0;
}
genMDParamsV0() {
const params = super.genMDParamsV0();
// lastModified and dataStoreName parameters are used by metadata that enables built-in filtering,
// a feature currently exclusive to MongoDB
if (this.beforeDate) {
params.lastModified = {
lt: this.beforeDate,
};
}
if (this.excludedDataStoreName) {
params.dataStoreName = {
ne: this.excludedDataStoreName,
}
}
return params;
}
/**
* Parses the stringified entry's value.
* @param s - sringified value
* @return - undefined if parsing fails, otherwise it contains the parsed value.
*/
_parse(s) {
let p;
try {
p = JSON.parse(s);
} catch (e: any) {
this.logger.warn(
'Could not parse Object Metadata while listing',
{ err: e.toString() });
}
return p;
}
/**
* check if the max keys count has been reached and set the
* final state of the result if it is the case
*
* specialized implementation on DelimiterCurrent to also check
* the number of scanned keys
*
* @return {Boolean} - indicates if the iteration has to stop
*/
_reachedMaxKeys(): boolean {
if (this.maxScannedLifecycleListingEntries && this.scannedKeys >= this.maxScannedLifecycleListingEntries) {
this.IsTruncated = true;
this.logger.info('listing stopped due to reaching the maximum scanned entries limit',
{
maxScannedLifecycleListingEntries: this.maxScannedLifecycleListingEntries,
scannedKeys: this.scannedKeys,
});
return true;
}
return super._reachedMaxKeys();
}
addContents(key, value) {
++this.scannedKeys;
const parsedValue = this._parse(value);
// if parsing fails, skip the key.
if (parsedValue) {
const lastModified = parsedValue['last-modified'];
const dataStoreName = parsedValue.dataStoreName;
// We then check if the current version is older than the "beforeDate" and
// "excludedDataStoreName" is not specified or if specified and the data store name is different.
if ((!this.beforeDate || (lastModified && lastModified < this.beforeDate)) &&
(!this.excludedDataStoreName || dataStoreName !== this.excludedDataStoreName)) {
super.addContents(key, value);
}
// In the event of a timeout occurring before any content is added,
// NextMarker is updated even if the object is not eligible.
// It minimizes the amount of data that the client needs to re-process if the request times out.
this.nextMarker = key;
}
}
result(): object {
const result: ResultObject = {
Contents: this.Contents,
IsTruncated: this.IsTruncated,
};
if (this.IsTruncated) {
result.NextMarker = this.nextMarker;
}
return result;
}
}
module.exports = { DelimiterCurrent };

View File

@ -5,18 +5,23 @@ import {
DelimiterFilterStateId, DelimiterFilterStateId,
DelimiterFilterState_NotSkipping, DelimiterFilterState_NotSkipping,
DelimiterFilterState_SkippingPrefix, DelimiterFilterState_SkippingPrefix,
ResultObject,
} from './delimiter'; } from './delimiter';
const Version = require('../../versioning/Version').Version; const Version = require('../../versioning/Version').Version;
const VSConst = require('../../versioning/constants').VersioningConstants; const VSConst = require('../../versioning/constants').VersioningConstants;
const { BucketVersioningKeyFormat } = VSConst; const { BucketVersioningKeyFormat } = VSConst;
const { FILTER_ACCEPT, FILTER_SKIP, FILTER_END } = require('./tools'); const { FILTER_ACCEPT, FILTER_SKIP, FILTER_END, SKIP_NONE, inc } = require('./tools');
import { GapSetEntry } from '../cache/GapSet';
import { GapCacheInterface } from '../cache/GapCache';
const VID_SEP = VSConst.VersionId.Separator; const VID_SEP = VSConst.VersionId.Separator;
const { DbPrefixes } = VSConst; const { DbPrefixes } = VSConst;
const enum DelimiterMasterFilterStateId { export const enum DelimiterMasterFilterStateId {
SkippingVersionsV0 = 101, SkippingVersionsV0 = 101,
WaitVersionAfterPHDV0 = 102, WaitVersionAfterPHDV0 = 102,
SkippingGapV0 = 103,
}; };
interface DelimiterMasterFilterState_SkippingVersionsV0 extends FilterState { interface DelimiterMasterFilterState_SkippingVersionsV0 extends FilterState {
@ -29,37 +34,121 @@ interface DelimiterMasterFilterState_WaitVersionAfterPHDV0 extends FilterState {
masterKey: string, masterKey: string,
}; };
interface DelimiterMasterFilterState_SkippingGapV0 extends FilterState {
id: DelimiterMasterFilterStateId.SkippingGapV0,
};
export const enum GapCachingState {
NoGapCache = 0, // there is no gap cache
UnknownGap = 1, // waiting for a cache lookup
GapLookupInProgress = 2, // asynchronous gap lookup in progress
GapCached = 3, // an upcoming or already skippable gap is cached
NoMoreGap = 4, // the cache doesn't have any more gaps inside the listed range
};
type GapCachingInfo_NoGapCache = {
state: GapCachingState.NoGapCache;
};
type GapCachingInfo_NoCachedGap = {
state: GapCachingState.UnknownGap
| GapCachingState.GapLookupInProgress
gapCache: GapCacheInterface;
};
type GapCachingInfo_GapCached = {
state: GapCachingState.GapCached;
gapCache: GapCacheInterface;
gapCached: GapSetEntry;
};
type GapCachingInfo_NoMoreGap = {
state: GapCachingState.NoMoreGap;
};
type GapCachingInfo = GapCachingInfo_NoGapCache
| GapCachingInfo_NoCachedGap
| GapCachingInfo_GapCached
| GapCachingInfo_NoMoreGap;
export const enum GapBuildingState {
Disabled = 0, // no gap cache or no gap building needed (e.g. in V1 versioning format)
NotBuilding = 1, // not currently building a gap (i.e. not listing within a gap)
Building = 2, // currently building a gap (i.e. listing within a gap)
Expired = 3, // not allowed to build due to exposure delay timeout
};
type GapBuildingInfo_NothingToBuild = {
state: GapBuildingState.Disabled | GapBuildingState.Expired;
};
type GapBuildingParams = {
/**
* minimum weight for a gap to be created in the cache
*/
minGapWeight: number;
/**
* trigger a cache setGap() call every N skippable keys
*/
triggerSaveGapWeight: number;
/**
* timestamp to assess whether we're still inside the validity period to
* be allowed to build gaps
*/
initTimestamp: number;
};
type GapBuildingInfo_NotBuilding = {
state: GapBuildingState.NotBuilding;
gapCache: GapCacheInterface;
params: GapBuildingParams;
};
type GapBuildingInfo_Building = {
state: GapBuildingState.Building;
gapCache: GapCacheInterface;
params: GapBuildingParams;
/**
* Gap currently being created
*/
gap: GapSetEntry;
/**
* total current weight of the gap being created
*/
gapWeight: number;
};
type GapBuildingInfo = GapBuildingInfo_NothingToBuild
| GapBuildingInfo_NotBuilding
| GapBuildingInfo_Building;
/** /**
* Handle object listing with parameters. This extends the base class Delimiter * Handle object listing with parameters. This extends the base class Delimiter
* to return the raw master versions of existing objects. * to return the raw master versions of existing objects.
*/ */
export class DelimiterMaster extends Delimiter { export class DelimiterMaster extends Delimiter {
_gapCaching: GapCachingInfo;
_gapBuilding: GapBuildingInfo;
_refreshedBuildingParams: GapBuildingParams | null;
/** /**
* Delimiter listing of master versions. * Delimiter listing of master versions.
* @param {Object} parameters - listing parameters * @param {Object} parameters - listing parameters
* @param {String} parameters.delimiter - delimiter per amazon format * @param {String} [parameters.delimiter] - delimiter per amazon format
* @param {String} parameters.prefix - prefix per amazon format * @param {String} [parameters.prefix] - prefix per amazon format
* @param {String} parameters.marker - marker per amazon format * @param {String} [parameters.marker] - marker per amazon format
* @param {Number} parameters.maxKeys - number of keys to list * @param {Number} [parameters.maxKeys] - number of keys to list
* @param {Boolean} parameters.v2 - indicates whether v2 format * @param {Boolean} [parameters.v2] - indicates whether v2 format
* @param {String} parameters.startAfter - marker per amazon v2 format * @param {String} [parameters.startAfter] - marker per amazon v2 format
* @param {String} parameters.continuationToken - obfuscated amazon token * @param {String} [parameters.continuationToken] - obfuscated amazon token
* @param {RequestLogger} logger - The logger of the request * @param {RequestLogger} logger - The logger of the request
* @param {String} [vFormat] - versioning key format * @param {String} [vFormat="v0"] - versioning key format
*/ */
constructor(parameters, logger, vFormat) { constructor(parameters, logger, vFormat?: string) {
super(parameters, logger, vFormat); super(parameters, logger, vFormat);
Object.assign(this, {
[BucketVersioningKeyFormat.v0]: {
skipping: this.skippingV0,
},
[BucketVersioningKeyFormat.v1]: {
skipping: this.skippingV1,
},
}[this.vFormat]);
if (this.vFormat === BucketVersioningKeyFormat.v0) { if (this.vFormat === BucketVersioningKeyFormat.v0) {
// override Delimiter's implementation of NotSkipping for // override Delimiter's implementation of NotSkipping for
// DelimiterMaster logic (skipping versions and special // DelimiterMaster logic (skipping versions and special
@ -77,6 +166,10 @@ export class DelimiterMaster extends Delimiter {
DelimiterMasterFilterStateId.WaitVersionAfterPHDV0, DelimiterMasterFilterStateId.WaitVersionAfterPHDV0,
this.keyHandler_WaitVersionAfterPHDV0.bind(this)); this.keyHandler_WaitVersionAfterPHDV0.bind(this));
this.setKeyHandler(
DelimiterMasterFilterStateId.SkippingGapV0,
this.keyHandler_SkippingGapV0.bind(this));
if (this.marker) { if (this.marker) {
// distinct initial state to include some special logic // distinct initial state to include some special logic
// before the first master key is found that does not have // before the first master key is found that does not have
@ -93,6 +186,176 @@ export class DelimiterMaster extends Delimiter {
} }
// in v1, we can directly use Delimiter's implementation, // in v1, we can directly use Delimiter's implementation,
// which is already set to the proper state // which is already set to the proper state
// default initialization of the gap cache and building states, can be
// set by refreshGapCache()
this._gapCaching = {
state: GapCachingState.NoGapCache,
};
this._gapBuilding = {
state: GapBuildingState.Disabled,
};
this._refreshedBuildingParams = null;
}
/**
* Get the validity period left before a refresh of the gap cache is needed
* to continue building new gaps.
*
* @return {number|null} one of:
* - the remaining time in milliseconds in which gaps can be added to the
* cache before a call to refreshGapCache() is required
* - or 0 if there is no time left and a call to refreshGapCache() is required
* to resume caching gaps
* - or null if refreshing the cache is never needed (because the gap cache
* is either not available or not used)
*/
getGapBuildingValidityPeriodMs(): number | null {
let gapBuilding;
switch (this._gapBuilding.state) {
case GapBuildingState.Disabled:
return null;
case GapBuildingState.Expired:
return 0;
case GapBuildingState.NotBuilding:
gapBuilding = <GapBuildingInfo_NotBuilding> this._gapBuilding;
break;
case GapBuildingState.Building:
gapBuilding = <GapBuildingInfo_Building> this._gapBuilding;
break;
}
const { gapCache, params } = gapBuilding;
const elapsedTime = Date.now() - params.initTimestamp;
return Math.max(gapCache.exposureDelayMs - elapsedTime, 0);
}
/**
* Refresh the gaps caching logic (gaps are series of current delete markers
* in V0 bucket metadata format). It has two effects:
*
* - starts exposing existing and future gaps from the cache to efficiently
* skip over series of current delete markers that have been seen and cached
* earlier
*
* - enables building and caching new gaps (or extend existing ones), for a
* limited time period defined by the `gapCacheProxy.exposureDelayMs` value
* in milliseconds. To refresh the validity period and resume building and
* caching new gaps, one must restart a new listing from the database (starting
* at the current listing key, included), then call refreshGapCache() again.
*
* @param {GapCacheInterface} gapCacheProxy - API proxy to the gaps cache
* (the proxy should handle prefixing object keys with the bucket name)
* @param {number} [minGapWeight=100] - minimum weight of a gap for it to be
* added in the cache
* @param {number} [triggerSaveGapWeight] - cumulative weight to wait for
* before saving the current building gap. Cannot be greater than
* `gapCacheProxy.maxGapWeight` (the value is thresholded to `maxGapWeight`
* otherwise). Defaults to `gapCacheProxy.maxGapWeight / 2`.
* @return {undefined}
*/
refreshGapCache(
gapCacheProxy: GapCacheInterface,
minGapWeight?: number,
triggerSaveGapWeight?: number
): void {
if (this.vFormat !== BucketVersioningKeyFormat.v0) {
return;
}
if (this._gapCaching.state === GapCachingState.NoGapCache) {
this._gapCaching = {
state: GapCachingState.UnknownGap,
gapCache: gapCacheProxy,
};
}
const refreshedBuildingParams: GapBuildingParams = {
minGapWeight: minGapWeight || 100,
triggerSaveGapWeight: triggerSaveGapWeight
|| Math.trunc(gapCacheProxy.maxGapWeight / 2),
initTimestamp: Date.now(),
};
if (this._gapBuilding.state === GapBuildingState.Building) {
// refreshed params will be applied as soon as the current building gap is saved
this._refreshedBuildingParams = refreshedBuildingParams;
} else {
this._gapBuilding = {
state: GapBuildingState.NotBuilding,
gapCache: gapCacheProxy,
params: refreshedBuildingParams,
};
}
}
/**
* Trigger a lookup of the closest upcoming or already skippable gap.
*
* @param {string} fromKey - lookup a gap not before 'fromKey'
* @return {undefined} - the lookup is asynchronous and its
* response is handled inside this function
*/
_triggerGapLookup(gapCaching: GapCachingInfo_NoCachedGap, fromKey: string): void {
this._gapCaching = {
state: GapCachingState.GapLookupInProgress,
gapCache: gapCaching.gapCache,
};
const maxKey = this.prefix ? inc(this.prefix) : undefined;
gapCaching.gapCache.lookupGap(fromKey, maxKey).then(_gap => {
const gap = <GapSetEntry | null> _gap;
if (gap) {
this._gapCaching = {
state: GapCachingState.GapCached,
gapCache: gapCaching.gapCache,
gapCached: gap,
};
} else {
this._gapCaching = {
state: GapCachingState.NoMoreGap,
};
}
});
}
_checkGapOnMasterDeleteMarker(key: string): FilterReturnValue {
switch (this._gapBuilding.state) {
case GapBuildingState.Disabled:
case GapBuildingState.Expired:
break;
case GapBuildingState.NotBuilding:
this._createBuildingGap(key, 1);
break;
case GapBuildingState.Building:
this._updateBuildingGap(key);
break;
}
if (this._gapCaching.state === GapCachingState.GapCached) {
const { gapCached } = this._gapCaching;
if (key >= gapCached.firstKey) {
if (key <= gapCached.lastKey) {
// we are inside the last looked up cached gap: transition to
// 'SkippingGapV0' state
this.setState(<DelimiterMasterFilterState_SkippingGapV0> {
id: DelimiterMasterFilterStateId.SkippingGapV0,
});
// cut the current gap before skipping, it will be merged or
// chained with the existing one (depending on its weight)
if (this._gapBuilding.state === GapBuildingState.Building) {
// substract 1 from the weight because we are going to chain this gap,
// which has an overlap of one key.
this._gapBuilding.gap.weight -= 1;
this._cutBuildingGap();
}
return FILTER_SKIP;
}
// as we are past the cached gap, we will need another lookup
this._gapCaching = {
state: GapCachingState.UnknownGap,
gapCache: this._gapCaching.gapCache,
};
}
}
if (this._gapCaching.state === GapCachingState.UnknownGap) {
this._triggerGapLookup(this._gapCaching, key);
}
return FILTER_ACCEPT;
} }
filter_onNewMasterKeyV0(key: string, value: string): FilterReturnValue { filter_onNewMasterKeyV0(key: string, value: string): FilterReturnValue {
@ -104,7 +367,7 @@ export class DelimiterMaster extends Delimiter {
id: DelimiterMasterFilterStateId.SkippingVersionsV0, id: DelimiterMasterFilterStateId.SkippingVersionsV0,
masterKey: key, masterKey: key,
}); });
return FILTER_ACCEPT; return this._checkGapOnMasterDeleteMarker(key);
} }
if (Version.isPHD(value)) { if (Version.isPHD(value)) {
// master version is a PHD version: wait for the first // master version is a PHD version: wait for the first
@ -116,6 +379,9 @@ export class DelimiterMaster extends Delimiter {
}); });
return FILTER_ACCEPT; return FILTER_ACCEPT;
} }
// cut the current gap as soon as a non-deleted entry is seen
this._cutBuildingGap();
if (key.startsWith(DbPrefixes.Replay)) { if (key.startsWith(DbPrefixes.Replay)) {
// skip internal replay prefix entirely // skip internal replay prefix entirely
this.setState(<DelimiterFilterState_SkippingPrefix> { this.setState(<DelimiterFilterState_SkippingPrefix> {
@ -127,6 +393,7 @@ export class DelimiterMaster extends Delimiter {
if (this._reachedMaxKeys()) { if (this._reachedMaxKeys()) {
return FILTER_END; return FILTER_END;
} }
const commonPrefix = this.addCommonPrefixOrContents(key, value); const commonPrefix = this.addCommonPrefixOrContents(key, value);
if (commonPrefix) { if (commonPrefix) {
// transition into SkippingPrefix state to skip all following keys // transition into SkippingPrefix state to skip all following keys
@ -154,6 +421,11 @@ export class DelimiterMaster extends Delimiter {
* (<key><versionIdSeparator><version>) */ * (<key><versionIdSeparator><version>) */
const versionIdIndex = key.indexOf(VID_SEP); const versionIdIndex = key.indexOf(VID_SEP);
if (versionIdIndex !== -1) { if (versionIdIndex !== -1) {
// version keys count in the building gap weight because they must
// also be listed until skipped
if (this._gapBuilding.state === GapBuildingState.Building) {
this._updateBuildingGap(key);
}
return FILTER_SKIP; return FILTER_SKIP;
} }
return this.filter_onNewMasterKeyV0(key, value); return this.filter_onNewMasterKeyV0(key, value);
@ -177,14 +449,151 @@ export class DelimiterMaster extends Delimiter {
return this.filter_onNewMasterKeyV0(key, value); return this.filter_onNewMasterKeyV0(key, value);
} }
keyHandler_SkippingGapV0(key: string, value: string): FilterReturnValue {
const { gapCache, gapCached } = <GapCachingInfo_GapCached> this._gapCaching;
if (key <= gapCached.lastKey) {
return FILTER_SKIP;
}
this._gapCaching = {
state: GapCachingState.UnknownGap,
gapCache,
};
this.setState(<DelimiterMasterFilterState_SkippingVersionsV0> {
id: DelimiterMasterFilterStateId.SkippingVersionsV0,
});
// Start a gap with weight=0 from the latest skippable key. This will
// allow to extend the gap just skipped with a chained gap in case
// other delete markers are seen after the existing gap is skipped.
this._createBuildingGap(gapCached.lastKey, 0, gapCached.weight);
return this.handleKey(key, value);
}
skippingBase(): string | undefined { skippingBase(): string | undefined {
switch (this.state.id) { switch (this.state.id) {
case DelimiterMasterFilterStateId.SkippingVersionsV0: case DelimiterMasterFilterStateId.SkippingVersionsV0:
const { masterKey } = <DelimiterMasterFilterState_SkippingVersionsV0> this.state; const { masterKey } = <DelimiterMasterFilterState_SkippingVersionsV0> this.state;
return masterKey + VID_SEP; return masterKey + inc(VID_SEP);
case DelimiterMasterFilterStateId.SkippingGapV0:
const { gapCached } = <GapCachingInfo_GapCached> this._gapCaching;
return gapCached.lastKey;
default: default:
return super.skippingBase(); return super.skippingBase();
} }
} }
result(): ResultObject {
this._cutBuildingGap();
return super.result();
}
_checkRefreshedBuildingParams(params: GapBuildingParams): GapBuildingParams {
if (this._refreshedBuildingParams) {
const newParams = this._refreshedBuildingParams;
this._refreshedBuildingParams = null;
return newParams;
}
return params;
}
/**
* Save the gap being built if allowed (i.e. still within the
* allocated exposure time window).
*
* @return {boolean} - true if the gap was saved, false if we are
* outside the allocated exposure time window.
*/
_saveBuildingGap(): boolean {
const { gapCache, params, gap, gapWeight } =
<GapBuildingInfo_Building> this._gapBuilding;
const totalElapsed = Date.now() - params.initTimestamp;
if (totalElapsed >= gapCache.exposureDelayMs) {
this._gapBuilding = {
state: GapBuildingState.Expired,
};
this._refreshedBuildingParams = null;
return false;
}
const { firstKey, lastKey, weight } = gap;
gapCache.setGap(firstKey, lastKey, weight);
this._gapBuilding = {
state: GapBuildingState.Building,
gapCache,
params: this._checkRefreshedBuildingParams(params),
gap: {
firstKey: gap.lastKey,
lastKey: gap.lastKey,
weight: 0,
},
gapWeight,
};
return true;
}
/**
* Create a new gap to be extended afterwards
*
* @param {string} newKey - gap's first key
* @param {number} startWeight - initial weight of the building gap (usually 0 or 1)
* @param {number} [cachedWeight] - if continuing a cached gap, weight of the existing
* cached portion
* @return {undefined}
*/
_createBuildingGap(newKey: string, startWeight: number, cachedWeight?: number): void {
if (this._gapBuilding.state === GapBuildingState.NotBuilding) {
const { gapCache, params } = <GapBuildingInfo_NotBuilding> this._gapBuilding;
this._gapBuilding = {
state: GapBuildingState.Building,
gapCache,
params: this._checkRefreshedBuildingParams(params),
gap: {
firstKey: newKey,
lastKey: newKey,
weight: startWeight,
},
gapWeight: (cachedWeight || 0) + startWeight,
};
}
}
_updateBuildingGap(newKey: string): void {
const gapBuilding = <GapBuildingInfo_Building> this._gapBuilding;
const { params, gap } = gapBuilding;
gap.lastKey = newKey;
gap.weight += 1;
gapBuilding.gapWeight += 1;
// the GapCache API requires updating a gap regularly because it can only split
// it once per update, by the known last key. In practice the default behavior
// is to trigger an update after a number of keys that is half the maximum weight.
// It is also useful for other listings to benefit from the cache sooner.
if (gapBuilding.gapWeight >= params.minGapWeight &&
gap.weight >= params.triggerSaveGapWeight) {
this._saveBuildingGap();
}
}
_cutBuildingGap(): void {
if (this._gapBuilding.state === GapBuildingState.Building) {
let gapBuilding = <GapBuildingInfo_Building> this._gapBuilding;
let { gapCache, params, gap, gapWeight } = gapBuilding;
// only set gaps that are significant enough in weight and
// with a non-empty extension
if (gapWeight >= params.minGapWeight && gap.weight > 0) {
// we're done if we were not allowed to save the gap
if (!this._saveBuildingGap()) {
return;
}
// params may have been refreshed, reload them
gapBuilding = <GapBuildingInfo_Building> this._gapBuilding;
params = gapBuilding.params;
}
this._gapBuilding = {
state: GapBuildingState.NotBuilding,
gapCache,
params,
};
}
}
} }

View File

@ -0,0 +1,202 @@
const { DelimiterVersions } = require('./delimiterVersions');
const { FILTER_END, FILTER_SKIP } = require('./tools');
const TRIM_METADATA_MIN_BLOB_SIZE = 10000;
/**
* Handle object listing with parameters. This extends the base class DelimiterVersions
* to return the raw non-current versions objects.
*/
class DelimiterNonCurrent extends DelimiterVersions {
/**
* Delimiter listing of non-current versions.
* @param {Object} parameters - listing parameters
* @param {String} parameters.keyMarker - key marker
* @param {String} parameters.versionIdMarker - version id marker
* @param {String} parameters.beforeDate - limit the response to keys with stale date older than beforeDate.
* stale date is the date on when a version becomes non-current.
* @param {Number} parameters.maxScannedLifecycleListingEntries - max number of entries to be scanned
* @param {String} parameters.excludedDataStoreName - exclude dataStoreName matches from the versions
* @param {RequestLogger} logger - The logger of the request
* @param {String} [vFormat] - versioning key format
*/
constructor(parameters, logger, vFormat) {
super(parameters, logger, vFormat);
this.beforeDate = parameters.beforeDate;
this.excludedDataStoreName = parameters.excludedDataStoreName;
this.maxScannedLifecycleListingEntries = parameters.maxScannedLifecycleListingEntries;
// internal state
this.prevKey = null;
this.staleDate = null;
this.scannedKeys = 0;
}
getLastModified(value) {
let lastModified;
try {
const v = JSON.parse(value);
lastModified = v['last-modified'];
} catch (e) {
this.logger.warn('could not parse Object Metadata while listing',
{
method: 'getLastModified',
err: e.toString(),
});
}
return lastModified;
}
// Overwrite keyHandler_SkippingVersions to include the last version from the previous listing.
// The creation (last-modified) date of this version will be the stale date for the following version.
// eslint-disable-next-line camelcase
keyHandler_SkippingVersions(key, versionId, value) {
if (key === this.keyMarker) {
// since the nonversioned key equals the marker, there is
// necessarily a versionId in this key
const _versionId = versionId;
if (_versionId < this.versionIdMarker) {
// skip all versions until marker
return FILTER_SKIP;
}
}
this.setState({
id: 1 /* NotSkipping */,
});
return this.handleKey(key, versionId, value);
}
filter(obj) {
if (this.maxScannedLifecycleListingEntries && this.scannedKeys >= this.maxScannedLifecycleListingEntries) {
this.IsTruncated = true;
this.logger.info('listing stopped due to reaching the maximum scanned entries limit',
{
maxScannedLifecycleListingEntries: this.maxScannedLifecycleListingEntries,
scannedKeys: this.scannedKeys,
});
return FILTER_END;
}
++this.scannedKeys;
return super.filter(obj);
}
/**
* NOTE: Each version of a specific key is sorted from the latest to the oldest
* thanks to the way version ids are generated.
* DESCRIPTION: Skip the version if it represents the master key, but keep its last-modified date in memory,
* which will be the stale date of the following version.
* The following version is pushed only:
* - if the "stale date" (picked up from the previous version) is available (JSON.parse has not failed),
* - if "beforeDate" is not specified or if specified and the "stale date" is older.
* - if "excludedDataStoreName" is not specified or if specified and the data store name is different
* The in-memory "stale date" is then updated with the version's last-modified date to be used for
* the following version.
* The process stops and returns the available results if either:
* - no more metadata key is left to be processed
* - the listing reaches the maximum number of key to be returned
* - the internal timeout is reached
* @param {String} key - The key to add
* @param {String} versionId - The version id
* @param {String} value - The value of the key
* @return {undefined}
*/
addVersion(key, versionId, value) {
this.nextKeyMarker = key;
this.nextVersionIdMarker = versionId;
// Skip the version if it represents the non-current version, but keep its last-modified date,
// which will be the stale date of the following version.
const isCurrentVersion = key !== this.prevKey;
if (isCurrentVersion) {
this.staleDate = this.getLastModified(value);
this.prevKey = key;
return;
}
// The following version is pushed only:
// - if the "stale date" (picked up from the previous version) is available (JSON.parse has not failed),
// - if "beforeDate" is not specified or if specified and the "stale date" is older.
// - if "excludedDataStoreName" is not specified or if specified and the data store name is different
let lastModified;
if (this.staleDate && (!this.beforeDate || this.staleDate < this.beforeDate)) {
const parsedValue = this._parse(value);
// if parsing fails, skip the key.
if (parsedValue) {
const dataStoreName = parsedValue.dataStoreName;
lastModified = parsedValue['last-modified'];
if (!this.excludedDataStoreName || dataStoreName !== this.excludedDataStoreName) {
const s = this._stringify(parsedValue, this.staleDate);
// check that _stringify succeeds to only push objects with a defined staleDate.
if (s) {
this.Versions.push({ key, value: s });
++this.keys;
}
}
}
}
// The in-memory "stale date" is then updated with the version's last-modified date to be used for
// the following version.
this.staleDate = lastModified || this.getLastModified(value);
return;
}
/**
* Parses the stringified entry's value and remove the location property if too large.
* @param {string} s - sringified value
* @return {object} p - undefined if parsing fails, otherwise it contains the parsed value.
*/
_parse(s) {
let p;
try {
p = JSON.parse(s);
if (s.length >= TRIM_METADATA_MIN_BLOB_SIZE) {
delete p.location;
}
} catch (e) {
this.logger.warn('Could not parse Object Metadata while listing', {
method: 'DelimiterNonCurrent._parse',
err: e.toString(),
});
}
return p;
}
_stringify(parsedMD, staleDate) {
const p = parsedMD;
let s = undefined;
p.staleDate = staleDate;
try {
s = JSON.stringify(p);
} catch (e) {
this.logger.warn('could not stringify Object Metadata while listing', {
method: 'DelimiterNonCurrent._stringify',
err: e.toString(),
});
}
return s;
}
result() {
const { Versions, IsTruncated, NextKeyMarker, NextVersionIdMarker } = super.result();
const result = {
Contents: Versions,
IsTruncated,
};
if (NextKeyMarker) {
result.NextKeyMarker = NextKeyMarker;
}
if (NextVersionIdMarker) {
result.NextVersionIdMarker = NextVersionIdMarker;
}
return result;
}
}
module.exports = { DelimiterNonCurrent };

View File

@ -0,0 +1,204 @@
const DelimiterVersions = require('./delimiterVersions').DelimiterVersions;
const { FILTER_END } = require('./tools');
const TRIM_METADATA_MIN_BLOB_SIZE = 10000;
/**
* Handle object listing with parameters. This extends the base class DelimiterVersions
* to return the orphan delete markers. Orphan delete markers are also
* refered as expired object delete marker.
* They are delete marker with zero noncurrent versions.
*/
class DelimiterOrphanDeleteMarker extends DelimiterVersions {
/**
* Delimiter listing of orphan delete markers.
* @param {Object} parameters - listing parameters
* @param {String} parameters.beforeDate - limit the response to keys older than beforeDate
* @param {Number} parameters.maxScannedLifecycleListingEntries - max number of entries to be scanned
* @param {RequestLogger} logger - The logger of the request
* @param {String} [vFormat] - versioning key format
*/
constructor(parameters, logger, vFormat) {
const {
marker,
maxKeys,
prefix,
beforeDate,
maxScannedLifecycleListingEntries,
} = parameters;
const versionParams = {
// The orphan delete marker logic uses the term 'marker' instead of 'keyMarker',
// as the latter could suggest the presence of a 'versionIdMarker'.
keyMarker: marker,
maxKeys,
prefix,
};
super(versionParams, logger, vFormat);
this.maxScannedLifecycleListingEntries = maxScannedLifecycleListingEntries;
this.beforeDate = beforeDate;
// this.prevKeyName is used as a marker for the next listing when the current one reaches its entry limit.
// We cannot rely on this.keyName, as it contains the name of the current key.
// In the event of a listing interruption due to reaching the maximum scanned entries,
// relying on this.keyName would cause the next listing to skip the current key because S3 starts
// listing after the marker.
this.prevKeyName = null;
this.keyName = null;
this.value = null;
this.scannedKeys = 0;
}
_reachedMaxKeys() {
if (this.keys >= this.maxKeys) {
return true;
}
return false;
}
_addOrphan() {
const parsedValue = this._parse(this.value);
// if parsing fails, skip the key.
if (parsedValue) {
const lastModified = parsedValue['last-modified'];
const isDeleteMarker = parsedValue.isDeleteMarker;
// We then check if the orphan version is a delete marker and if it is older than the "beforeDate"
if ((!this.beforeDate || (lastModified && lastModified < this.beforeDate)) && isDeleteMarker) {
// Prefer returning an untrimmed data rather than stopping the service in case of parsing failure.
const s = this._stringify(parsedValue) || this.value;
this.Versions.push({ key: this.keyName, value: s });
this.nextKeyMarker = this.keyName;
++this.keys;
}
}
}
/**
* Parses the stringified entry's value and remove the location property if too large.
* @param {string} s - sringified value
* @return {object} p - undefined if parsing fails, otherwise it contains the parsed value.
*/
_parse(s) {
let p;
try {
p = JSON.parse(s);
if (s.length >= TRIM_METADATA_MIN_BLOB_SIZE) {
delete p.location;
}
} catch (e) {
this.logger.warn('Could not parse Object Metadata while listing', {
method: 'DelimiterOrphanDeleteMarker._parse',
err: e.toString(),
});
}
return p;
}
_stringify(value) {
const p = value;
let s = undefined;
try {
s = JSON.stringify(p);
} catch (e) {
this.logger.warn('could not stringify Object Metadata while listing',
{
method: 'DelimiterOrphanDeleteMarker._stringify',
err: e.toString(),
});
}
return s;
}
/**
* The purpose of _isMaxScannedEntriesReached is to restrict the number of scanned entries,
* thus controlling resource overhead (CPU...).
* @return {boolean} isMaxScannedEntriesReached - true if the maximum limit on the number
* of entries scanned has been reached, false otherwise.
*/
_isMaxScannedEntriesReached() {
return this.maxScannedLifecycleListingEntries && this.scannedKeys >= this.maxScannedLifecycleListingEntries;
}
filter(obj) {
if (this._isMaxScannedEntriesReached()) {
this.nextKeyMarker = this.prevKeyName;
this.IsTruncated = true;
this.logger.info('listing stopped due to reaching the maximum scanned entries limit',
{
maxScannedLifecycleListingEntries: this.maxScannedLifecycleListingEntries,
scannedKeys: this.scannedKeys,
});
return FILTER_END;
}
++this.scannedKeys;
return super.filter(obj);
}
/**
* NOTE: Each version of a specific key is sorted from the latest to the oldest
* thanks to the way version ids are generated.
* DESCRIPTION: For a given key, the latest version is kept in memory since it is the current version.
* If the following version reference a new key, it means that the previous one was an orphan version.
* We then check if the orphan version is a delete marker and if it is older than the "beforeDate"
* The process stops and returns the available results if either:
* - no more metadata key is left to be processed
* - the listing reaches the maximum number of key to be returned
* - the internal timeout is reached
* NOTE: we cannot leverage MongoDB to list keys older than "beforeDate"
* because then we will not be able to assess its orphanage.
* @param {String} key - The object key.
* @param {String} versionId - The object version id.
* @param {String} value - The value of the key
* @return {undefined}
*/
addVersion(key, versionId, value) {
// For a given key, the youngest version is kept in memory since it represents the current version.
if (key !== this.keyName) {
// If this.value is defined, it means that <this.keyName, this.value> pair is "allowed" to be an orphan.
if (this.value) {
this._addOrphan();
}
this.prevKeyName = this.keyName;
this.keyName = key;
this.value = value;
return;
}
// If the key is not the current version, we can skip it in the next listing
// in the case where the current listing is interrupted due to reaching the maximum scanned entries.
this.prevKeyName = key;
this.keyName = key;
this.value = null;
return;
}
result() {
// Only check for remaining last orphan delete marker if the listing is not interrupted.
// This will help avoid false positives.
if (!this._isMaxScannedEntriesReached()) {
// The following check makes sure the last orphan delete marker is not forgotten.
if (this.keys < this.maxKeys) {
if (this.value) {
this._addOrphan();
}
// The following make sure that if makeKeys is reached, isTruncated is set to true.
// We moved the "isTruncated" from _reachedMaxKeys to make sure we take into account the last entity
// if listing is truncated right before the last entity and the last entity is a orphan delete marker.
} else {
this.IsTruncated = this.maxKeys > 0;
}
}
const result = {
Contents: this.Versions,
IsTruncated: this.IsTruncated,
};
if (this.IsTruncated) {
result.NextMarker = this.nextKeyMarker;
}
return result;
}
}
module.exports = { DelimiterOrphanDeleteMarker };

View File

@ -1,304 +0,0 @@
'use strict'; // eslint-disable-line strict
const Delimiter = require('./delimiter').Delimiter;
const Version = require('../../versioning/Version').Version;
const VSConst = require('../../versioning/constants').VersioningConstants;
const { inc, FILTER_END, FILTER_ACCEPT, FILTER_SKIP, SKIP_NONE } =
require('./tools');
const VID_SEP = VSConst.VersionId.Separator;
const { DbPrefixes, BucketVersioningKeyFormat } = VSConst;
/**
* Handle object listing with parameters
*
* @prop {String[]} CommonPrefixes - 'folders' defined by the delimiter
* @prop {String[]} Contents - 'files' to list
* @prop {Boolean} IsTruncated - truncated listing flag
* @prop {String|undefined} NextMarker - marker per amazon format
* @prop {Number} keys - count of listed keys
* @prop {String|undefined} delimiter - separator per amazon format
* @prop {String|undefined} prefix - prefix per amazon format
* @prop {Number} maxKeys - number of keys to list
*/
class DelimiterVersions extends Delimiter {
constructor(parameters, logger, vFormat) {
super(parameters, logger, vFormat);
// specific to version listing
this.keyMarker = parameters.keyMarker;
this.versionIdMarker = parameters.versionIdMarker;
// internal state
this.masterKey = undefined;
this.masterVersionId = undefined;
// listing results
this.NextMarker = parameters.keyMarker;
this.NextVersionIdMarker = undefined;
this.inReplayPrefix = false;
Object.assign(this, {
[BucketVersioningKeyFormat.v0]: {
genMDParams: this.genMDParamsV0,
filter: this.filterV0,
skipping: this.skippingV0,
},
[BucketVersioningKeyFormat.v1]: {
genMDParams: this.genMDParamsV1,
filter: this.filterV1,
skipping: this.skippingV1,
},
}[this.vFormat]);
}
genMDParamsV0() {
const params = {};
if (this.parameters.prefix) {
params.gte = this.parameters.prefix;
params.lt = inc(this.parameters.prefix);
}
if (this.parameters.keyMarker) {
if (params.gte && params.gte > this.parameters.keyMarker) {
return params;
}
delete params.gte;
if (this.parameters.versionIdMarker) {
// versionIdMarker should always come with keyMarker
// but may not be the other way around
params.gt = this.parameters.keyMarker
+ VID_SEP
+ this.parameters.versionIdMarker;
} else {
params.gt = inc(this.parameters.keyMarker + VID_SEP);
}
}
return params;
}
genMDParamsV1() {
// return an array of two listing params sets to ask for
// synchronized listing of M and V ranges
const params = [{}, {}];
if (this.parameters.prefix) {
params[0].gte = DbPrefixes.Master + this.parameters.prefix;
params[0].lt = DbPrefixes.Master + inc(this.parameters.prefix);
params[1].gte = DbPrefixes.Version + this.parameters.prefix;
params[1].lt = DbPrefixes.Version + inc(this.parameters.prefix);
} else {
params[0].gte = DbPrefixes.Master;
params[0].lt = inc(DbPrefixes.Master); // stop after the last master key
params[1].gte = DbPrefixes.Version;
params[1].lt = inc(DbPrefixes.Version); // stop after the last version key
}
if (this.parameters.keyMarker) {
if (params[1].gte <= DbPrefixes.Version + this.parameters.keyMarker) {
delete params[0].gte;
delete params[1].gte;
params[0].gt = DbPrefixes.Master + inc(this.parameters.keyMarker + VID_SEP);
if (this.parameters.versionIdMarker) {
// versionIdMarker should always come with keyMarker
// but may not be the other way around
params[1].gt = DbPrefixes.Version
+ this.parameters.keyMarker
+ VID_SEP
+ this.parameters.versionIdMarker;
} else {
params[1].gt = DbPrefixes.Version
+ inc(this.parameters.keyMarker + VID_SEP);
}
}
}
return params;
}
/**
* Used to synchronize listing of M and V prefixes by object key
*
* @param {object} masterObj object listed from first range
* returned by genMDParamsV1() (the master keys range)
* @param {object} versionObj object listed from second range
* returned by genMDParamsV1() (the version keys range)
* @return {number} comparison result:
* * -1 if master key < version key
* * 1 if master key > version key
*/
compareObjects(masterObj, versionObj) {
const masterKey = masterObj.key.slice(DbPrefixes.Master.length);
const versionKey = versionObj.key.slice(DbPrefixes.Version.length);
return masterKey < versionKey ? -1 : 1;
}
/**
* Add a (key, versionId, value) tuple to the listing.
* Set the NextMarker to the current key
* Increment the keys counter
* @param {object} obj - the entry to add to the listing result
* @param {String} obj.key - The key to add
* @param {String} obj.versionId - versionId
* @param {String} obj.value - The value of the key
* @return {Boolean} - indicates if iteration should continue
*/
addContents(obj) {
if (this._reachedMaxKeys()) {
return FILTER_END;
}
this.Contents.push({
key: obj.key,
value: this.trimMetadata(obj.value),
versionId: obj.versionId,
});
this.NextMarker = obj.key;
this.NextVersionIdMarker = obj.versionId;
++this.keys;
return FILTER_ACCEPT;
}
/**
* Add a Common Prefix in the list
* @param {String} key - object name
* @param {Number} index - after prefix starting point
* @return {Boolean} - indicates if iteration should continue
*/
addCommonPrefix(key, index) {
const commonPrefix = key.substring(0, index + this.delimiter.length);
if (this.CommonPrefixes.indexOf(commonPrefix) === -1
&& this.NextMarker !== commonPrefix) {
if (this._reachedMaxKeys()) {
return FILTER_END;
}
this.CommonPrefixes.push(commonPrefix);
this.NextMarker = commonPrefix;
++this.keys;
return FILTER_ACCEPT;
}
return FILTER_SKIP;
}
/**
* Filter to apply on each iteration if bucket is in v0
* versioning key format, based on:
* - prefix
* - delimiter
* - maxKeys
* The marker is being handled directly by levelDB
* @param {Object} obj - The key and value of the element
* @param {String} obj.key - The key of the element
* @param {String} obj.value - The value of the element
* @return {number} - indicates if iteration should continue
*/
filterV0(obj) {
if (obj.key.startsWith(DbPrefixes.Replay)) {
this.inReplayPrefix = true;
return FILTER_SKIP;
}
this.inReplayPrefix = false;
if (Version.isPHD(obj.value)) {
// return accept to avoid skipping the next values in range
return FILTER_ACCEPT;
}
return this.filterCommon(obj.key, obj.value);
}
/**
* Filter to apply on each iteration if bucket is in v1
* versioning key format, based on:
* - prefix
* - delimiter
* - maxKeys
* The marker is being handled directly by levelDB
* @param {Object} obj - The key and value of the element
* @param {String} obj.key - The key of the element
* @param {String} obj.value - The value of the element
* @return {number} - indicates if iteration should continue
*/
filterV1(obj) {
if (Version.isPHD(obj.value)) {
// return accept to avoid skipping the next values in range
return FILTER_ACCEPT;
}
// this function receives both M and V keys, but their prefix
// length is the same so we can remove their prefix without
// looking at the type of key
return this.filterCommon(obj.key.slice(DbPrefixes.Master.length),
obj.value);
}
filterCommon(key, value) {
if (this.prefix && !key.startsWith(this.prefix)) {
return FILTER_SKIP;
}
let nonversionedKey;
let versionId = undefined;
const versionIdIndex = key.indexOf(VID_SEP);
if (versionIdIndex < 0) {
nonversionedKey = key;
this.masterKey = key;
this.masterVersionId =
Version.from(value).getVersionId() || 'null';
versionId = this.masterVersionId;
} else {
nonversionedKey = key.slice(0, versionIdIndex);
versionId = key.slice(versionIdIndex + 1);
// skip a version key if it is the master version
if (this.masterKey === nonversionedKey && this.masterVersionId === versionId) {
return FILTER_SKIP;
}
this.masterKey = undefined;
this.masterVersionId = undefined;
}
if (this.delimiter) {
const baseIndex = this.prefix ? this.prefix.length : 0;
const delimiterIndex = nonversionedKey.indexOf(this.delimiter, baseIndex);
if (delimiterIndex >= 0) {
return this.addCommonPrefix(nonversionedKey, delimiterIndex);
}
}
return this.addContents({ key: nonversionedKey, value, versionId });
}
skippingV0() {
if (this.inReplayPrefix) {
return DbPrefixes.Replay;
}
if (this.NextMarker) {
const index = this.NextMarker.lastIndexOf(this.delimiter);
if (index === this.NextMarker.length - 1) {
return this.NextMarker;
}
}
return SKIP_NONE;
}
skippingV1() {
const skipV0 = this.skippingV0();
if (skipV0 === SKIP_NONE) {
return SKIP_NONE;
}
// skip to the same object key in both M and V range listings
return [DbPrefixes.Master + skipV0,
DbPrefixes.Version + skipV0];
}
/**
* Return an object containing all mandatory fields to use once the
* iteration is done, doesn't show a NextMarker field if the output
* isn't truncated
* @return {Object} - following amazon format
*/
result() {
/* NextMarker is only provided when delimiter is used.
* specified in v1 listing documentation
* http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html
*/
return {
CommonPrefixes: this.CommonPrefixes,
Versions: this.Contents,
IsTruncated: this.IsTruncated,
NextKeyMarker: this.IsTruncated ? this.NextMarker : undefined,
NextVersionIdMarker: this.IsTruncated ?
this.NextVersionIdMarker : undefined,
Delimiter: this.delimiter,
};
}
}
module.exports = { DelimiterVersions };

View File

@ -0,0 +1,530 @@
'use strict'; // eslint-disable-line strict
const Extension = require('./Extension').default;
import {
FilterState,
FilterReturnValue,
} from './delimiter';
const Version = require('../../versioning/Version').Version;
const VSConst = require('../../versioning/constants').VersioningConstants;
const { inc, FILTER_END, FILTER_ACCEPT, FILTER_SKIP, SKIP_NONE } =
require('./tools');
const VID_SEP = VSConst.VersionId.Separator;
const { DbPrefixes, BucketVersioningKeyFormat } = VSConst;
export const enum DelimiterVersionsFilterStateId {
NotSkipping = 1,
SkippingPrefix = 2,
SkippingVersions = 3,
};
export interface DelimiterVersionsFilterState_NotSkipping extends FilterState {
id: DelimiterVersionsFilterStateId.NotSkipping,
};
export interface DelimiterVersionsFilterState_SkippingPrefix extends FilterState {
id: DelimiterVersionsFilterStateId.SkippingPrefix,
prefix: string;
};
export interface DelimiterVersionsFilterState_SkippingVersions extends FilterState {
id: DelimiterVersionsFilterStateId.SkippingVersions,
gt: string;
};
type KeyHandler = (key: string, versionId: string | undefined, value: string) => FilterReturnValue;
type ResultObject = {
CommonPrefixes: string[],
Versions: {
key: string;
value: string;
versionId: string;
}[];
IsTruncated: boolean;
Delimiter ?: string;
NextKeyMarker ?: string;
NextVersionIdMarker ?: string;
};
type GenMDParamsItem = {
gt ?: string,
gte ?: string,
lt ?: string,
};
/**
* Handle object listing with parameters
*
* @prop {String[]} CommonPrefixes - 'folders' defined by the delimiter
* @prop {String[]} Contents - 'files' to list
* @prop {Boolean} IsTruncated - truncated listing flag
* @prop {String|undefined} NextMarker - marker per amazon format
* @prop {Number} keys - count of listed keys
* @prop {String|undefined} delimiter - separator per amazon format
* @prop {String|undefined} prefix - prefix per amazon format
* @prop {Number} maxKeys - number of keys to list
*/
export class DelimiterVersions extends Extension {
state: FilterState;
keyHandlers: { [id: number]: KeyHandler };
constructor(parameters, logger, vFormat) {
super(parameters, logger);
// original listing parameters
this.delimiter = parameters.delimiter;
this.prefix = parameters.prefix;
this.maxKeys = parameters.maxKeys || 1000;
// specific to version listing
this.keyMarker = parameters.keyMarker;
this.versionIdMarker = parameters.versionIdMarker;
// internal state
this.masterKey = undefined;
this.masterVersionId = undefined;
this.nullKey = null;
this.vFormat = vFormat || BucketVersioningKeyFormat.v0;
// listing results
this.CommonPrefixes = [];
this.Versions = [];
this.IsTruncated = false;
this.nextKeyMarker = parameters.keyMarker;
this.nextVersionIdMarker = undefined;
this.keyHandlers = {};
Object.assign(this, {
[BucketVersioningKeyFormat.v0]: {
genMDParams: this.genMDParamsV0,
getObjectKey: this.getObjectKeyV0,
skipping: this.skippingV0,
},
[BucketVersioningKeyFormat.v1]: {
genMDParams: this.genMDParamsV1,
getObjectKey: this.getObjectKeyV1,
skipping: this.skippingV1,
},
}[this.vFormat]);
if (this.vFormat === BucketVersioningKeyFormat.v0) {
this.setKeyHandler(
DelimiterVersionsFilterStateId.NotSkipping,
this.keyHandler_NotSkippingV0.bind(this));
} else {
this.setKeyHandler(
DelimiterVersionsFilterStateId.NotSkipping,
this.keyHandler_NotSkippingV1.bind(this));
}
this.setKeyHandler(
DelimiterVersionsFilterStateId.SkippingPrefix,
this.keyHandler_SkippingPrefix.bind(this));
this.setKeyHandler(
DelimiterVersionsFilterStateId.SkippingVersions,
this.keyHandler_SkippingVersions.bind(this));
if (this.versionIdMarker) {
this.state = <DelimiterVersionsFilterState_SkippingVersions> {
id: DelimiterVersionsFilterStateId.SkippingVersions,
gt: `${this.keyMarker}${VID_SEP}${this.versionIdMarker}`,
};
} else {
this.state = <DelimiterVersionsFilterState_NotSkipping> {
id: DelimiterVersionsFilterStateId.NotSkipping,
};
}
}
genMDParamsV0() {
const params: GenMDParamsItem = {};
if (this.prefix) {
params.gte = this.prefix;
params.lt = inc(this.prefix);
}
if (this.keyMarker && this.delimiter) {
const commonPrefix = this.getCommonPrefix(this.keyMarker);
if (commonPrefix) {
const afterPrefix = inc(commonPrefix);
if (!params.gte || afterPrefix > params.gte) {
params.gte = afterPrefix;
}
}
}
if (this.keyMarker && (!params.gte || this.keyMarker >= params.gte)) {
delete params.gte;
if (this.versionIdMarker) {
// start from the beginning of versions so we can
// check if there's a null key and fetch it
// (afterwards, we can skip the rest of versions until
// we reach versionIdMarker)
params.gte = `${this.keyMarker}${VID_SEP}`;
} else {
params.gt = `${this.keyMarker}${inc(VID_SEP)}`;
}
}
return params;
}
genMDParamsV1() {
// return an array of two listing params sets to ask for
// synchronized listing of M and V ranges
const v0Params: GenMDParamsItem = this.genMDParamsV0();
const mParams: GenMDParamsItem = {};
const vParams: GenMDParamsItem = {};
if (v0Params.gt) {
mParams.gt = `${DbPrefixes.Master}${v0Params.gt}`;
vParams.gt = `${DbPrefixes.Version}${v0Params.gt}`;
} else if (v0Params.gte) {
mParams.gte = `${DbPrefixes.Master}${v0Params.gte}`;
vParams.gte = `${DbPrefixes.Version}${v0Params.gte}`;
} else {
mParams.gte = DbPrefixes.Master;
vParams.gte = DbPrefixes.Version;
}
if (v0Params.lt) {
mParams.lt = `${DbPrefixes.Master}${v0Params.lt}`;
vParams.lt = `${DbPrefixes.Version}${v0Params.lt}`;
} else {
mParams.lt = inc(DbPrefixes.Master);
vParams.lt = inc(DbPrefixes.Version);
}
return [mParams, vParams];
}
/**
* check if the max keys count has been reached and set the
* final state of the result if it is the case
* @return {Boolean} - indicates if the iteration has to stop
*/
_reachedMaxKeys(): boolean {
if (this.keys >= this.maxKeys) {
// In cases of maxKeys <= 0 -> IsTruncated = false
this.IsTruncated = this.maxKeys > 0;
return true;
}
return false;
}
/**
* Used to synchronize listing of M and V prefixes by object key
*
* @param {object} masterObj object listed from first range
* returned by genMDParamsV1() (the master keys range)
* @param {object} versionObj object listed from second range
* returned by genMDParamsV1() (the version keys range)
* @return {number} comparison result:
* * -1 if master key < version key
* * 1 if master key > version key
*/
compareObjects(masterObj, versionObj) {
const masterKey = masterObj.key.slice(DbPrefixes.Master.length);
const versionKey = versionObj.key.slice(DbPrefixes.Version.length);
return masterKey < versionKey ? -1 : 1;
}
/**
* Parse a listing key into its nonversioned key and version ID components
*
* @param {string} key - full listing key
* @return {object} obj
* @return {string} obj.key - nonversioned part of key
* @return {string} [obj.versionId] - version ID in the key
*/
parseKey(fullKey: string): { key: string, versionId ?: string } {
const versionIdIndex = fullKey.indexOf(VID_SEP);
if (versionIdIndex === -1) {
return { key: fullKey };
}
const nonversionedKey: string = fullKey.slice(0, versionIdIndex);
let versionId: string = fullKey.slice(versionIdIndex + 1);
return { key: nonversionedKey, versionId };
}
/**
* Include a key in the listing output, in the Versions or CommonPrefix result
*
* @param {string} key - key (without version ID)
* @param {string} versionId - version ID
* @param {string} value - metadata value
* @return {undefined}
*/
addKey(key: string, versionId: string, value: string) {
// add the subprefix to the common prefixes if the key has the delimiter
const commonPrefix = this.getCommonPrefix(key);
if (commonPrefix) {
this.addCommonPrefix(commonPrefix);
// transition into SkippingPrefix state to skip all following keys
// while they start with the same prefix
this.setState(<DelimiterVersionsFilterState_SkippingPrefix> {
id: DelimiterVersionsFilterStateId.SkippingPrefix,
prefix: commonPrefix,
});
} else {
this.addVersion(key, versionId, value);
}
}
/**
* Add a (key, versionId, value) tuple to the listing.
* Set the NextMarker to the current key
* Increment the keys counter
* @param {String} key - The key to add
* @param {String} versionId - versionId
* @param {String} value - The value of the key
* @return {undefined}
*/
addVersion(key: string, versionId: string, value: string) {
this.Versions.push({
key,
versionId,
value: this.trimMetadata(value),
});
this.nextKeyMarker = key;
this.nextVersionIdMarker = versionId;
++this.keys;
}
getCommonPrefix(key: string): string | undefined {
if (!this.delimiter) {
return undefined;
}
const baseIndex = this.prefix ? this.prefix.length : 0;
const delimiterIndex = key.indexOf(this.delimiter, baseIndex);
if (delimiterIndex === -1) {
return undefined;
}
return key.substring(0, delimiterIndex + this.delimiter.length);
}
/**
* Add a Common Prefix in the list
* @param {String} commonPrefix - common prefix to add
* @return {undefined}
*/
addCommonPrefix(commonPrefix: string): void {
// add the new prefix to the list
this.CommonPrefixes.push(commonPrefix);
++this.keys;
this.nextKeyMarker = commonPrefix;
this.nextVersionIdMarker = undefined;
}
/**
* Cache the current null key, to save it for outputting it later at
* the correct position
*
* @param {String} key - nonversioned key of the null key
* @param {String} versionId - real version ID of the null key
* @param {String} value - value of the null key
* @return {undefined}
*/
cacheNullKey(key: string, versionId: string, value: string): void {
this.nullKey = { key, versionId, value };
}
getObjectKeyV0(obj: { key: string }): string {
return obj.key;
}
getObjectKeyV1(obj: { key: string }): string {
return obj.key.slice(DbPrefixes.Master.length);
}
/**
* Filter to apply on each iteration, based on:
* - prefix
* - delimiter
* - maxKeys
* The marker is being handled directly by levelDB
* @param {Object} obj - The key and value of the element
* @param {String} obj.key - The key of the element
* @param {String} obj.value - The value of the element
* @return {number} - indicates if iteration should continue
*/
filter(obj: { key: string, value: string }): FilterReturnValue {
const key = this.getObjectKey(obj);
const value = obj.value;
const { key: nonversionedKey, versionId: keyVersionId } = this.parseKey(key);
if (this.nullKey) {
if (this.nullKey.key !== nonversionedKey
|| this.nullKey.versionId < <string> keyVersionId) {
this.handleKey(
this.nullKey.key, this.nullKey.versionId, this.nullKey.value);
this.nullKey = null;
}
}
if (keyVersionId === '') {
// null key
this.cacheNullKey(nonversionedKey, Version.from(value).getVersionId(), value);
if (this.state.id === DelimiterVersionsFilterStateId.SkippingVersions) {
return FILTER_SKIP;
}
return FILTER_ACCEPT;
}
return this.handleKey(nonversionedKey, keyVersionId, value);
}
setState(state: FilterState): void {
this.state = state;
}
setKeyHandler(stateId: number, keyHandler: KeyHandler): void {
this.keyHandlers[stateId] = keyHandler;
}
handleKey(key: string, versionId: string | undefined, value: string): FilterReturnValue {
return this.keyHandlers[this.state.id](key, versionId, value);
}
keyHandler_NotSkippingV0(key: string, versionId: string | undefined, value: string): FilterReturnValue {
if (key.startsWith(DbPrefixes.Replay)) {
// skip internal replay prefix entirely
this.setState(<DelimiterVersionsFilterState_SkippingPrefix> {
id: DelimiterVersionsFilterStateId.SkippingPrefix,
prefix: DbPrefixes.Replay,
});
return FILTER_SKIP;
}
if (Version.isPHD(value)) {
return FILTER_ACCEPT;
}
return this.filter_onNewKey(key, versionId, value);
}
keyHandler_NotSkippingV1(key: string, versionId: string | undefined, value: string): FilterReturnValue {
return this.filter_onNewKey(key, versionId, value);
}
filter_onNewKey(key: string, versionId: string | undefined, value: string): FilterReturnValue {
if (this._reachedMaxKeys()) {
return FILTER_END;
}
if (versionId === undefined) {
this.masterKey = key;
this.masterVersionId = Version.from(value).getVersionId() || 'null';
this.addKey(this.masterKey, this.masterVersionId, value);
} else {
if (this.masterKey === key && this.masterVersionId === versionId) {
// do not add a version key if it is the master version
return FILTER_ACCEPT;
}
this.addKey(key, versionId, value);
}
return FILTER_ACCEPT;
}
keyHandler_SkippingPrefix(key: string, versionId: string | undefined, value: string): FilterReturnValue {
const { prefix } = <DelimiterVersionsFilterState_SkippingPrefix> this.state;
if (key.startsWith(prefix)) {
return FILTER_SKIP;
}
this.setState(<DelimiterVersionsFilterState_NotSkipping> {
id: DelimiterVersionsFilterStateId.NotSkipping,
});
return this.handleKey(key, versionId, value);
}
keyHandler_SkippingVersions(key: string, versionId: string | undefined, value: string): FilterReturnValue {
if (key === this.keyMarker) {
// since the nonversioned key equals the marker, there is
// necessarily a versionId in this key
const _versionId = <string> versionId;
if (_versionId < this.versionIdMarker) {
// skip all versions until marker
return FILTER_SKIP;
}
if (_versionId === this.versionIdMarker) {
// nothing left to skip, so return ACCEPT, but don't add this version
return FILTER_ACCEPT;
}
}
this.setState(<DelimiterVersionsFilterState_NotSkipping> {
id: DelimiterVersionsFilterStateId.NotSkipping,
});
return this.handleKey(key, versionId, value);
}
skippingBase(): string | undefined {
switch (this.state.id) {
case DelimiterVersionsFilterStateId.SkippingPrefix:
const { prefix } = <DelimiterVersionsFilterState_SkippingPrefix> this.state;
return inc(prefix);
case DelimiterVersionsFilterStateId.SkippingVersions:
const { gt } = <DelimiterVersionsFilterState_SkippingVersions> this.state;
// the contract of skipping() is to return the first key
// that can be skipped to, so adding a null byte to skip
// over the existing versioned key set in 'gt'
return `${gt}\0`;
default:
return SKIP_NONE;
}
}
skippingV0() {
return this.skippingBase();
}
skippingV1() {
const skipTo = this.skippingBase();
if (skipTo === SKIP_NONE) {
return SKIP_NONE;
}
// skip to the same object key in both M and V range listings
return [
`${DbPrefixes.Master}${skipTo}`,
`${DbPrefixes.Version}${skipTo}`,
];
}
/**
* Return an object containing all mandatory fields to use once the
* iteration is done, doesn't show a NextMarker field if the output
* isn't truncated
* @return {Object} - following amazon format
*/
result() {
// Add the last null key if still in cache (when it is the
// last version of the last key)
//
// NOTE: _reachedMaxKeys sets IsTruncated to true when it
// returns true. Here we want this because either:
//
// - we did not reach the max keys yet so the result is not
// - truncated, and there is still room for the null key in
// - the results
//
// - OR we reached it already while having to process a new
// key (so the result is truncated even without the null key)
//
// - OR we are *just* below the limit but the null key to add
// does not fit, so we know the result is now truncated
// because there remains the null key to be output.
//
if (this.nullKey) {
this.handleKey(this.nullKey.key, this.nullKey.versionId, this.nullKey.value);
}
const result: ResultObject = {
CommonPrefixes: this.CommonPrefixes,
Versions: this.Versions,
IsTruncated: this.IsTruncated,
};
if (this.delimiter) {
result.Delimiter = this.delimiter;
}
if (this.IsTruncated) {
result.NextKeyMarker = this.nextKeyMarker;
if (this.nextVersionIdMarker) {
result.NextVersionIdMarker = this.nextVersionIdMarker;
}
};
return result;
}
}
module.exports = { DelimiterVersions };

View File

@ -6,4 +6,7 @@ module.exports = {
DelimiterMaster: require('./delimiterMaster') DelimiterMaster: require('./delimiterMaster')
.DelimiterMaster, .DelimiterMaster,
MPU: require('./MPU').MultipartUploads, MPU: require('./MPU').MultipartUploads,
DelimiterCurrent: require('./delimiterCurrent').DelimiterCurrent,
DelimiterNonCurrent: require('./delimiterNonCurrent').DelimiterNonCurrent,
DelimiterOrphanDeleteMarker: require('./delimiterOrphanDeleteMarker').DelimiterOrphanDeleteMarker,
}; };

View File

@ -52,21 +52,21 @@ class Skip {
assert(this.skipRangeCb); assert(this.skipRangeCb);
const filteringResult = this.extension.filter(entry); const filteringResult = this.extension.filter(entry);
const skippingRange = this.extension.skipping(); const skipTo = this.extension.skipping();
if (filteringResult === FILTER_END) { if (filteringResult === FILTER_END) {
this.listingEndCb(); this.listingEndCb();
} else if (filteringResult === FILTER_SKIP } else if (filteringResult === FILTER_SKIP
&& skippingRange !== SKIP_NONE) { && skipTo !== SKIP_NONE) {
if (++this.streakLength >= MAX_STREAK_LENGTH) { if (++this.streakLength >= MAX_STREAK_LENGTH) {
let newRange; let newRange;
if (Array.isArray(skippingRange)) { if (Array.isArray(skipTo)) {
newRange = []; newRange = [];
for (let i = 0; i < skippingRange.length; ++i) { for (let i = 0; i < skipTo.length; ++i) {
newRange.push(this._inc(skippingRange[i])); newRange.push(skipTo[i]);
} }
} else { } else {
newRange = this._inc(skippingRange); newRange = skipTo;
} }
/* Avoid to loop on the same range again and again. */ /* Avoid to loop on the same range again and again. */
if (newRange === this.gteParams) { if (newRange === this.gteParams) {
@ -79,16 +79,6 @@ class Skip {
this.streakLength = 0; this.streakLength = 0;
} }
} }
_inc(str) {
if (!str) {
return str;
}
const lastCharValue = str.charCodeAt(str.length - 1);
const lastCharNewValue = String.fromCharCode(lastCharValue + 1);
return `${str.slice(0, str.length - 1)}${lastCharNewValue}`;
}
} }

View File

@ -0,0 +1,514 @@
import cluster, { Worker } from 'cluster';
import * as werelogs from 'werelogs';
import { default as errors } from '../../lib/errors';
const rpcLogger = new werelogs.Logger('ClusterRPC');
/**
* Remote procedure calls support between cluster workers.
*
* When using the cluster module, new processes are forked and are
* dispatched workloads, usually HTTP requests. The ClusterRPC module
* implements a RPC system to send commands to all cluster worker
* processes at once from any particular worker, and retrieve their
* individual command results, like a distributed map operation.
*
* The existing nodejs cluster IPC channel is setup from the primary
* to each worker, but not between workers, so there has to be a hop
* by the primary.
*
* How a command is treated:
*
* - a worker sends a command message to the primary
*
* - the primary then forwards that command to each existing worker
* (including the requestor)
*
* - each worker then executes the command and returns a result or an
* error
*
* - the primary gathers all workers results into an array
*
* - finally, the primary dispatches the results array to the original
* requesting worker
*
*
* Limitations:
*
* - The command payload must be serializable, which means that:
* - it should not contain circular references
* - it should be of a reasonable size to be sent in a single RPC message
*
* - The "toWorkers" parameter of value "*" targets the set of workers
* that are available at the time the command is dispatched. Any new
* worker spawned after the command has been dispatched for
* processing, but before the command completes, don't execute
* the command and hence are not part of the results array.
*
*
* To set it up:
*
* - On the primary:
* if (cluster.isPrimary) {
* setupRPCPrimary();
* }
*
* - On the workers:
* if (!cluster.isPrimary) {
* setupRPCWorker({
* handler1: (payload: object, uids: string, callback: HandlerCallback) => void,
* handler2: ...
* });
* }
* Handler functions will be passed the command payload, request
* serialized uids, and must call the callback when the worker is done
* processing the command:
* callback(error: Error | null | undefined, result?: any)
*
* When this setup is done, any worker can start sending commands by calling
* the async function sendWorkerCommand().
*/
// exported types
export type ResultObject = {
error: Error | null;
result: any;
};
/**
* saved Promise for sendWorkerCommand
*/
export type CommandPromise = {
resolve: (results?: ResultObject[]) => void;
reject: (error: Error) => void;
timeout: NodeJS.Timer | null;
};
export type HandlerCallback = (error: Error | null | undefined, result?: any) => void;
export type HandlerFunction = (payload: object, uids: string, callback: HandlerCallback) => void;
export type HandlersMap = {
[index: string]: HandlerFunction;
};
// private types
type RPCMessage<T extends string, P> = {
type: T;
uids: string;
payload: P;
};
type RPCCommandMessage = RPCMessage<'cluster-rpc:command', any> & {
toWorkers: string;
toHandler: string;
};
type MarshalledResultObject = {
error: string | null;
result: any;
};
type RPCCommandResultMessage = RPCMessage<'cluster-rpc:commandResult', MarshalledResultObject>;
type RPCCommandResultsMessage = RPCMessage<'cluster-rpc:commandResults', {
results: MarshalledResultObject[];
}>;
type RPCCommandErrorMessage = RPCMessage<'cluster-rpc:commandError', {
error: string;
}>;
/**
* In primary: store worker IDs that are waiting to be dispatched
* their command's results, as a mapping.
*/
const uidsToWorkerId: {
[index: string]: number;
} = {};
/**
* In primary: store worker responses for commands in progress as a
* mapping.
*
* Result objects are 'null' while the worker is still processing the
* command. When a worker finishes processing it stores the result as:
* {
* error: string | null,
* result: any
* }
*/
const uidsToCommandResults: {
[index: string]: {
[index: number]: MarshalledResultObject | null;
};
} = {};
/**
* In workers: store promise callbacks for commands waiting to be
* dispatched, as a mapping.
*/
const uidsToCommandPromise: {
[index: string]: CommandPromise;
} = {};
function _isRpcMessage(message) {
return (message !== null &&
typeof message === 'object' &&
typeof message.type === 'string' &&
message.type.startsWith('cluster-rpc:'));
}
/**
* Setup cluster RPC system on the primary
*
* @return {undefined}
*/
export function setupRPCPrimary() {
cluster.on('message', (worker, message) => {
if (_isRpcMessage(message)) {
_handlePrimaryMessage(worker?.id, message);
}
});
}
/**
* Setup RPCs on a cluster worker process
*
* @param {object} handlers - mapping of handler names to handler functions
* handler function:
* handler({object} payload, {string} uids, {function} callback)
* handler callback must be called when worker is done with the command:
* callback({Error|null} error, {any} [result])
* @return {undefined}
* }
*/
export function setupRPCWorker(handlers: HandlersMap) {
if (!process.send) {
throw new Error('fatal: cannot setup cluster RPC: "process.send" is not available');
}
process.on('message', (message: RPCCommandMessage | RPCCommandResultsMessage) => {
if (_isRpcMessage(message)) {
_handleWorkerMessage(message, handlers);
}
});
}
/**
* Send a command for workers to execute in parallel, and wait for results
*
* @param {string} toWorkers - which workers should execute the command
* Currently the only supported value is "*", meaning all workers will
* execute the command
* @param {string} toHandler - name of handler that will execute the
* command in workers, as declared in setupRPCWorker() parameter object
* @param {string} uids - unique identifier of the command, must be
* unique across all commands in progress
* @param {object} payload - message payload, sent as-is to the handler
* @param {number} [timeoutMs=60000] - timeout the command with a
* "RequestTimeout" error after this number of milliseconds - set to 0
* to disable timeouts (the command may then hang forever)
* @returns {Promise}
*/
export async function sendWorkerCommand(
toWorkers: string,
toHandler: string,
uids: string,
payload: object,
timeoutMs: number = 60000
) {
if (typeof uids !== 'string') {
rpcLogger.error('missing or invalid "uids" field', { uids });
throw errors.MissingParameter;
}
if (uidsToCommandPromise[uids] !== undefined) {
rpcLogger.error('a command is already in progress with same uids', { uids });
throw errors.OperationAborted;
}
rpcLogger.info('sending command', { toWorkers, toHandler, uids, payload });
return new Promise((resolve, reject) => {
let timeout: NodeJS.Timer | null = null;
if (timeoutMs) {
timeout = setTimeout(() => {
delete uidsToCommandPromise[uids];
reject(errors.RequestTimeout);
}, timeoutMs);
}
uidsToCommandPromise[uids] = { resolve, reject, timeout };
const message: RPCCommandMessage = {
type: 'cluster-rpc:command',
toWorkers,
toHandler,
uids,
payload,
};
return process.send?.(message);
});
}
/**
* Get the number of commands in flight
* @returns {number}
*/
export function getPendingCommandsCount() {
return Object.keys(uidsToCommandPromise).length;
}
function _dispatchCommandResultsToWorker(
worker: Worker,
uids: string,
resultsArray: MarshalledResultObject[]
): void {
const message: RPCCommandResultsMessage = {
type: 'cluster-rpc:commandResults',
uids,
payload: {
results: resultsArray,
},
};
worker.send(message);
}
function _dispatchCommandErrorToWorker(
worker: Worker,
uids: string,
error: Error,
): void {
const message: RPCCommandErrorMessage = {
type: 'cluster-rpc:commandError',
uids,
payload: {
error: error.message,
},
};
worker.send(message);
}
function _handlePrimaryCommandMessage(
fromWorkerId: number,
logger: any,
message: RPCCommandMessage
): void {
const { toWorkers, toHandler, uids, payload } = message;
if (toWorkers === '*') {
if (uidsToWorkerId[uids] !== undefined) {
logger.warn('new command already has a waiting worker with same uids', {
uids, workerId: uidsToWorkerId[uids],
});
return undefined;
}
const commandResults = {};
for (const workerId of Object.keys(cluster.workers || {})) {
commandResults[workerId] = null;
}
uidsToWorkerId[uids] = fromWorkerId;
uidsToCommandResults[uids] = commandResults;
for (const [workerId, worker] of Object.entries(cluster.workers || {})) {
logger.debug('sending command message to worker', {
workerId, toHandler, payload,
});
if (worker) {
worker.send(message);
}
}
} else {
logger.error('unsupported "toWorkers" field from worker command message', {
toWorkers,
});
const fromWorker = cluster.workers?.[fromWorkerId];
if (fromWorker) {
_dispatchCommandErrorToWorker(fromWorker, uids, errors.NotImplemented);
}
}
}
function _handlePrimaryCommandResultMessage(
fromWorkerId: number,
logger: any,
message: RPCCommandResultMessage
): void {
const { uids, payload } = message;
const commandResults = uidsToCommandResults[uids];
if (!commandResults) {
logger.warn('received command response message from worker for command not in flight', {
workerId: fromWorkerId,
uids,
});
return undefined;
}
if (commandResults[fromWorkerId] === undefined) {
logger.warn('received command response message with unexpected worker ID', {
workerId: fromWorkerId,
uids,
});
return undefined;
}
if (commandResults[fromWorkerId] !== null) {
logger.warn('ignoring duplicate command response from worker', {
workerId: fromWorkerId,
uids,
});
return undefined;
}
commandResults[fromWorkerId] = payload;
const commandResultsArray = Object.values(commandResults);
if (commandResultsArray.every(response => response !== null)) {
logger.debug('all workers responded to command', { uids });
const completeCommandResultsArray = <MarshalledResultObject[]> commandResultsArray;
const toWorkerId = uidsToWorkerId[uids];
const toWorker = cluster.workers?.[toWorkerId];
delete uidsToCommandResults[uids];
delete uidsToWorkerId[uids];
if (!toWorker) {
logger.warn('worker shut down while its command was executing', {
workerId: toWorkerId, uids,
});
return undefined;
}
// send back response to original worker
_dispatchCommandResultsToWorker(toWorker, uids, completeCommandResultsArray);
}
}
function _handlePrimaryMessage(
fromWorkerId: number,
message: RPCCommandMessage | RPCCommandResultMessage
): void {
const { type: messageType, uids } = message;
const logger = rpcLogger.newRequestLoggerFromSerializedUids(uids);
logger.debug('primary received message from worker', {
workerId: fromWorkerId, rpcMessage: message,
});
if (messageType === 'cluster-rpc:command') {
return _handlePrimaryCommandMessage(fromWorkerId, logger, message);
}
if (messageType === 'cluster-rpc:commandResult') {
return _handlePrimaryCommandResultMessage(fromWorkerId, logger, message);
}
logger.error('unsupported message type', {
workerId: fromWorkerId, messageType, uids,
});
return undefined;
}
function _sendWorkerCommandResult(
uids: string,
error: Error | null | undefined,
result?: any
): void {
const message: RPCCommandResultMessage = {
type: 'cluster-rpc:commandResult',
uids,
payload: {
error: error ? error.message : null,
result,
},
};
process.send?.(message);
}
function _handleWorkerCommandMessage(
logger: any,
message: RPCCommandMessage,
handlers: HandlersMap
): void {
const { toHandler, uids, payload } = message;
const cb: HandlerCallback = (err, result) => _sendWorkerCommandResult(uids, err, result);
if (toHandler in handlers) {
return handlers[toHandler](payload, uids, cb);
}
logger.error('no such handler in "toHandler" field from worker command message', {
toHandler,
});
return cb(errors.NotImplemented);
}
function _handleWorkerCommandResultsMessage(
logger: any,
message: RPCCommandResultsMessage,
): void {
const { uids, payload } = message;
const { results } = payload;
const commandPromise: CommandPromise = uidsToCommandPromise[uids];
if (commandPromise === undefined) {
logger.error('missing promise for command results', { uids, payload });
return undefined;
}
if (commandPromise.timeout) {
clearTimeout(commandPromise.timeout);
}
delete uidsToCommandPromise[uids];
const unmarshalledResults = results.map(workerResult => {
let workerError: Error | null = null;
if (workerResult.error) {
if (workerResult.error in errors) {
workerError = errors[workerResult.error];
} else {
workerError = new Error(workerResult.error);
}
}
const unmarshalledResult: ResultObject = {
error: workerError,
result: workerResult.result,
};
return unmarshalledResult;
});
return commandPromise.resolve(unmarshalledResults);
}
function _handleWorkerCommandErrorMessage(
logger: any,
message: RPCCommandErrorMessage,
): void {
const { uids, payload } = message;
const { error } = payload;
const commandPromise: CommandPromise = uidsToCommandPromise[uids];
if (commandPromise === undefined) {
logger.error('missing promise for command results', { uids, payload });
return undefined;
}
if (commandPromise.timeout) {
clearTimeout(commandPromise.timeout);
}
delete uidsToCommandPromise[uids];
let commandError: Error | null = null;
if (error in errors) {
commandError = errors[error];
} else {
commandError = new Error(error);
}
return commandPromise.reject(<Error> commandError);
}
function _handleWorkerMessage(
message: RPCCommandMessage | RPCCommandResultsMessage | RPCCommandErrorMessage,
handlers: HandlersMap
): void {
const { type: messageType, uids } = message;
const workerId = cluster.worker?.id;
const logger = rpcLogger.newRequestLoggerFromSerializedUids(uids);
logger.debug('worker received message from primary', {
workerId, rpcMessage: message,
});
if (messageType === 'cluster-rpc:command') {
return _handleWorkerCommandMessage(logger, message, handlers);
}
if (messageType === 'cluster-rpc:commandResults') {
return _handleWorkerCommandResultsMessage(logger, message);
}
if (messageType === 'cluster-rpc:commandError') {
return _handleWorkerCommandErrorMessage(logger, message);
}
logger.error('unsupported message type', {
workerId, messageType,
});
return undefined;
}

View File

@ -126,6 +126,7 @@ export const supportedLifecycleRules = [
// Maximum number of buckets to cache (bucket metadata) // Maximum number of buckets to cache (bucket metadata)
export const maxCachedBuckets = process.env.METADATA_MAX_CACHED_BUCKETS ? export const maxCachedBuckets = process.env.METADATA_MAX_CACHED_BUCKETS ?
Number(process.env.METADATA_MAX_CACHED_BUCKETS) : 1000; Number(process.env.METADATA_MAX_CACHED_BUCKETS) : 1000;
export const maxBatchingConcurrentOperations = 5;
/** For policy resource arn check we allow empty account ID to not break compatibility */ /** For policy resource arn check we allow empty account ID to not break compatibility */
export const policyArnAllowedEmptyAccountId = ['utapi', 'scuba']; export const policyArnAllowedEmptyAccountId = ['utapi', 'scuba'];

View File

@ -56,9 +56,10 @@ export type ObjectMDData = {
acl: ACL; acl: ACL;
key: string; key: string;
location: null | Location[]; location: null | Location[];
// versionId, isNull, nullVersionId and isDeleteMarker // versionId, isNull, isNull2, nullVersionId and isDeleteMarker
// should be undefined when not set explicitly // should be undefined when not set explicitly
isNull?: boolean; isNull?: boolean;
isNull2?: boolean;
nullVersionId?: string; nullVersionId?: string;
nullUploadId?: string; nullUploadId?: string;
isDeleteMarker?: boolean; isDeleteMarker?: boolean;
@ -180,6 +181,7 @@ export default class ObjectMD {
// versionId, isNull, nullVersionId and isDeleteMarker // versionId, isNull, nullVersionId and isDeleteMarker
// should be undefined when not set explicitly // should be undefined when not set explicitly
isNull: undefined, isNull: undefined,
isNull2: undefined,
nullVersionId: undefined, nullVersionId: undefined,
nullUploadId: undefined, nullUploadId: undefined,
isDeleteMarker: undefined, isDeleteMarker: undefined,
@ -692,6 +694,31 @@ export default class ObjectMD {
return this._data.isNull || false; return this._data.isNull || false;
} }
/**
* Set metadata isNull2 value
*
* @param isNull2 - Whether new version is null or not AND has
* been put with a Cloudserver handling null keys (i.e. supporting
* S3C-7352)
* @return itself
*/
setIsNull2(isNull2: boolean) {
this._data.isNull2 = isNull2;
return this;
}
/**
* Get metadata isNull2 value
*
* @return isNull2 - Whether new version is null or not AND has
* been put with a Cloudserver handling null keys (i.e. supporting
* S3C-7352)
*/
getIsNull2() {
return this._data.isNull2 || false;
}
/** /**
* Set metadata nullVersionId value * Set metadata nullVersionId value
* *

View File

@ -51,6 +51,36 @@ function _parseListEntries(entries) {
}); });
} }
/** _parseLifecycleListEntries - parse the values returned in a lifeycle listing by metadata
* @param {object[]} entries - Version or Content entries in a metadata listing
* @param {string} entries[].key - metadata key
* @param {string} entries[].value - stringified object metadata
* @return {object} - mapped array with parsed value or JSON parsing err
*/
function _parseLifecycleListEntries(entries) {
return entries.map(entry => {
const tmp = JSON.parse(entry.value);
return {
key: entry.key,
value: {
Size: tmp['content-length'],
ETag: tmp['content-md5'],
VersionId: tmp.versionId,
IsNull: tmp.isNull,
LastModified: tmp['last-modified'],
Owner: {
DisplayName: tmp['owner-display-name'],
ID: tmp['owner-id'],
},
StorageClass: tmp['x-amz-storage-class'],
tags: tmp.tags,
staleDate: tmp.staleDate,
dataStoreName: tmp.dataStoreName,
},
};
});
}
/** parseListEntries - parse the values returned in a listing by metadata /** parseListEntries - parse the values returned in a listing by metadata
* @param {object[]} entries - Version or Content entries in a metadata listing * @param {object[]} entries - Version or Content entries in a metadata listing
* @param {string} entries[].key - metadata key * @param {string} entries[].key - metadata key
@ -213,6 +243,25 @@ class MetadataWrapper {
}); });
} }
getObjectsMD(bucketName, objNamesWithParams, log, cb) {
if (typeof this.client.getObjects !== 'function') {
log.debug('backend does not support get object metadata with batching', {
implName: this.implName,
});
return cb(errors.NotImplemented);
}
log.debug('getting objects from metadata', { objects: objNamesWithParams });
return this.client.getObjects(bucketName, objNamesWithParams, log, (err, data) => {
if (err) {
log.debug('error getting objects from metadata', { implName: this.implName, objects: objNamesWithParams,
err });
return cb(err);
}
log.debug('objects retrieved from metadata', { objects: objNamesWithParams });
return cb(err, data);
});
}
getObjectMD(bucketName, objName, params, log, cb) { getObjectMD(bucketName, objName, params, log, cb) {
log.debug('getting object from metadata'); log.debug('getting object from metadata');
this.client.getObject(bucketName, objName, params, log, (err, data) => { this.client.getObject(bucketName, objName, params, log, (err, data) => {
@ -279,6 +328,29 @@ class MetadataWrapper {
}); });
} }
listLifecycleObject(bucketName, listingParams, log, cb) {
log.debug('getting object listing for lifecycle from metadata');
this.client.listLifecycleObject(bucketName, listingParams, log, (err, data) => {
if (err) {
log.error('error from metadata', { implName: this.implName,
err });
return cb(err);
}
log.debug('object listing for lifecycle retrieved from metadata');
// eslint-disable-next-line no-param-reassign
data.Contents = parseListEntries(data.Contents, _parseLifecycleListEntries);
if (data.Contents instanceof Error) {
log.error('error parsing metadata listing for lifecycle', {
error: data.Contents,
listingType: listingParams.listingType,
method: 'listLifecycleObject',
});
return cb(errors.InternalError);
}
return cb(null, data);
});
}
listMultipartUploads(bucketName, listingParams, log, cb) { listMultipartUploads(bucketName, listingParams, log, cb) {
this.client.listMultipartUploads(bucketName, listingParams, log, this.client.listMultipartUploads(bucketName, listingParams, log,
(err, data) => { (err, data) => {

View File

@ -110,6 +110,17 @@ class BucketClientInterface {
return null; return null;
} }
listLifecycleObject(bucketName, params, log, cb) {
this.client.listObject(bucketName, log.getSerializedUids(), params,
(err, data) => {
if (err) {
return cb(err);
}
return cb(null, JSON.parse(data));
});
return null;
}
listMultipartUploads(bucketName, params, log, cb) { listMultipartUploads(bucketName, params, log, cb) {
this.client.listObject(bucketName, log.getSerializedUids(), params, this.client.listObject(bucketName, log.getSerializedUids(), params,
(err, data) => { (err, data) => {

View File

@ -325,6 +325,10 @@ class BucketFileInterface {
return this.internalListObject(bucketName, params, log, cb); return this.internalListObject(bucketName, params, log, cb);
} }
listLifecycleObject(bucketName, params, log, cb) {
return this.internalListObject(bucketName, params, log, cb);
}
listMultipartUploads(bucketName, params, log, cb) { listMultipartUploads(bucketName, params, log, cb) {
return this.internalListObject(bucketName, params, log, cb); return this.internalListObject(bucketName, params, log, cb);
} }

View File

@ -318,6 +318,10 @@ const metastore = {
}); });
}, },
listLifecycleObject(bucketName, params, log, cb) {
return process.nextTick(cb, errors.NotImplemented);
},
listMultipartUploads(bucketName, listingParams, log, cb) { listMultipartUploads(bucketName, listingParams, log, cb) {
process.nextTick(() => { process.nextTick(() => {
metastore.getBucketAttributes(bucketName, log, (err, bucket) => { metastore.getBucketAttributes(bucketName, log, (err, bucket) => {

View File

@ -965,6 +965,86 @@ class MongoClientInterface {
], cb); ], cb);
} }
/**
* gets object metadata for a list of objects
* @param {String} bucketName bucket name
* @param {Array} objects array of objects
* @param {Object} log logger
* @param {Function} callback callback
* @return {undefined}
*/
getObjects(bucketName, objects, log, callback) {
const c = this.getCollection(bucketName);
let vFormat = null;
if (!Array.isArray(objects)) {
return callback(errors.InternalError.customizeDescription('objects must be an array'));
}
// We do not accept more than 1000 keys in a single request
if (objects.length > 1000) {
return callback(errors.InternalError.customizeDescription('cannot get more than 1000 objects'));
}
// Function to process each document
const processDoc = (doc, objName, params, key, cb) => {
const versionIdValue = params && params.versionId ? params.versionId : undefined;
if (!doc && versionIdValue) {
// If no document and a version ID is provided, return an error.
return cb(null, {
err: errors.NoSuchKey,
doc: null,
versionId: versionIdValue,
key,
});
}
// If no master found then object is either non existent or last
// version is delete marker
if (!doc || doc.value.isPHD) {
return this.getLatestVersion(c, objName, vFormat, log, (err, _doc) => cb(null, {
err,
doc: _doc || null,
versionId: versionIdValue,
key,
}));
}
MongoUtils.unserialize(doc.value);
return cb(null, {
err: null,
doc: doc.value,
versionId: versionIdValue,
key,
});
};
return this.getBucketVFormat(bucketName, log, (err, _vFormat) => {
if (err) {
return callback(err);
}
vFormat = _vFormat;
const keys = objects.map(({ key: objName, params }) => (params && params.versionId
? formatVersionKey(objName, params.versionId, vFormat)
: formatMasterKey(objName, vFormat)));
return c.find({
_id: { $in: keys },
$or: [
{ 'value.deleted': { $exists: false } },
{ 'value.deleted': { $eq: false } },
],
}).toArray((err, docs) => {
if (err) {
return callback(err);
}
// Create a Map to quickly find docs by their keys
const docByKey = new Map(docs.map(doc => [doc._id, doc]));
// Process each document using associated context (objName, params)
return async.mapLimit(objects, constants.maxBatchingConcurrentOperations,
({ key: objName, params }, cb) => {
const key = params && params.versionId
? formatVersionKey(objName, params.versionId, vFormat)
: formatMasterKey(objName, vFormat);
const doc = docByKey.get(key);
processDoc(doc, objName, params, key, cb);
}, callback);
});
});
}
/** /**
* This function return the latest version of an object * This function return the latest version of an object
* by getting all keys related to an object's versions, ordering them * by getting all keys related to an object's versions, ordering them
@ -1125,7 +1205,7 @@ class MongoClientInterface {
'value.isPHD': true, 'value.isPHD': true,
'value.versionId': mst.versionId, 'value.versionId': mst.versionId,
}; };
this.internalDeleteObject(c, bucketName, masterKey, filter, log, err => { this.internalDeleteObject(c, bucketName, masterKey, filter, null, log, err => {
if (err) { if (err) {
// the PHD master might get updated when a PUT is performed // the PHD master might get updated when a PUT is performed
// before the repair is done, we don't want to return an error // before the repair is done, we don't want to return an error
@ -1195,7 +1275,7 @@ class MongoClientInterface {
next, next,
), ),
// delete version // delete version
next => this.internalDeleteObject(c, bucketName, versionKey, {}, log, next => this.internalDeleteObject(c, bucketName, versionKey, {}, params, log,
err => { err => {
// we don't return an error in case we don't find // we don't return an error in case we don't find
// a version as we expect this case when dealing with // a version as we expect this case when dealing with
@ -1231,7 +1311,7 @@ class MongoClientInterface {
*/ */
deleteObjectVerNotMaster(c, bucketName, objName, params, log, cb) { deleteObjectVerNotMaster(c, bucketName, objName, params, log, cb) {
const versionKey = formatVersionKey(objName, params.versionId, params.vFormat); const versionKey = formatVersionKey(objName, params.versionId, params.vFormat);
this.internalDeleteObject(c, bucketName, versionKey, {}, log, err => { this.internalDeleteObject(c, bucketName, versionKey, {}, params, log, err => {
if (err) { if (err) {
if (err.is.NoSuchKey) { if (err.is.NoSuchKey) {
log.error( log.error(
@ -1321,7 +1401,7 @@ class MongoClientInterface {
*/ */
deleteObjectNoVer(c, bucketName, objName, params, log, cb) { deleteObjectNoVer(c, bucketName, objName, params, log, cb) {
const masterKey = formatMasterKey(objName, params.vFormat); const masterKey = formatMasterKey(objName, params.vFormat);
this.internalDeleteObject(c, bucketName, masterKey, {}, log, err => { this.internalDeleteObject(c, bucketName, masterKey, {}, params, log, err => {
if (err) { if (err) {
// Should not return an error when no object is found // Should not return an error when no object is found
if (err.is.NoSuchKey) { if (err.is.NoSuchKey) {
@ -1344,12 +1424,29 @@ class MongoClientInterface {
* @param {string} bucketName bucket name * @param {string} bucketName bucket name
* @param {string} key Key of the object to delete * @param {string} key Key of the object to delete
* @param {object} filter additional query filters * @param {object} filter additional query filters
* @param {object} params request params
* @param {Logger} log logger instance * @param {Logger} log logger instance
* @param {Function} cb callback containing error * @param {Function} cb callback containing error
* and BulkWriteResult * and BulkWriteResult
* @return {undefined} * @return {undefined}
*/ */
internalDeleteObject(collection, bucketName, key, filter, log, cb) { internalDeleteObject(collection, bucketName, key, filter, params, log, cb) {
// filter used when deleting object
const deleteFilter = Object.assign({
_id: key,
}, filter);
if (params && params.doesNotNeedOpogUpdate) {
// If flag is true, directly delete object
return collection.deleteOne(deleteFilter)
.then(() => cb(null))
.catch(err => {
log.error('internalDeleteObject: error deleting object',
{ bucket: bucketName, object: key, error: err.message });
return cb(errors.InternalError);
});
}
// filter used when finding and updating object // filter used when finding and updating object
const findFilter = Object.assign({ const findFilter = Object.assign({
_id: key, _id: key,
@ -1358,12 +1455,12 @@ class MongoClientInterface {
{ 'value.deleted': { $eq: false } }, { 'value.deleted': { $eq: false } },
], ],
}, filter); }, filter);
// filter used when deleting object
const updateDeleteFilter = Object.assign({ const updateDeleteFilter = Object.assign({
'_id': key, '_id': key,
'value.deleted': true, 'value.deleted': true,
}, filter); }, filter);
async.waterfall([ return async.waterfall([
// Adding delete flag when getting the object // Adding delete flag when getting the object
// to avoid having race conditions. // to avoid having race conditions.
next => collection.findOneAndUpdate(findFilter, { next => collection.findOneAndUpdate(findFilter, {
@ -1387,7 +1484,8 @@ class MongoClientInterface {
const obj = doc.value; const obj = doc.value;
const objMetadata = new ObjectMD(obj.value); const objMetadata = new ObjectMD(obj.value);
objMetadata.setOriginOp('s3:ObjectRemoved:Delete'); objMetadata.setOriginOp('s3:ObjectRemoved:Delete');
objMetadata.setDeleted(true); // Not supported in 7.x
// objMetadata.setDeleted(true);
return next(null, objMetadata.getValue()); return next(null, objMetadata.getValue());
}), }),
// We update the full object to get the whole object metadata // We update the full object to get the whole object metadata
@ -1607,6 +1705,43 @@ class MongoClientInterface {
}); });
} }
/**
* lists current version, non-current version and orphan delete markers in a bucket
* @param {String} bucketName bucket name
* @param {Object} params params
* @param {String} params.listingType type of algorithm to use
* @param {Number} [params.maxKeys] maximum number of keys to list
* @param {String} [params.prefix] prefix of objects to use
* @param {Object} log logger
* @param {Function} cb callback
* @return {undefined}
*/
listLifecycleObject(bucketName, params, log, cb) {
return this.getBucketVFormat(bucketName, log, (err, vFormat) => {
if (err) {
return cb(err);
}
if (vFormat !== BUCKET_VERSIONS.v1) {
log.error('not supported bucket format version',
{ method: 'listLifecycleObject', bucket: bucketName, vFormat });
return cb(errors.NotImplemented.customizeDescription('Not supported bucket format version'));
}
const extName = params.listingType;
const extension = new listAlgos[extName](params, log, vFormat);
const extensionParams = extension.genMDParams();
const internalParams = {
mainStreamParams: Array.isArray(extensionParams) ? extensionParams[0] : extensionParams,
secondaryStreamParams: Array.isArray(extensionParams) ? extensionParams[1] : null,
};
return this.internalListObject(bucketName, internalParams, extension, vFormat, log, cb);
});
}
/** /**
* lists versionned and non versionned objects in a bucket * lists versionned and non versionned objects in a bucket
* @param {String} bucketName bucket name * @param {String} bucketName bucket name
@ -2245,7 +2380,7 @@ class MongoClientInterface {
}); });
return cb(errors.InternalError); return cb(errors.InternalError);
} }
return this.internalDeleteObject(c, bucketName, masterKey, filter, log, return this.internalDeleteObject(c, bucketName, masterKey, filter, null, log,
err => { err => {
if (err) { if (err) {
// unable to find an object that matches the conditions // unable to find an object that matches the conditions

View File

@ -55,6 +55,22 @@ class MongoReadStream extends Readable {
} }
} }
if (options.lastModified) {
query['value.last-modified'] = {};
if (options.lastModified.lt) {
query['value.last-modified'].$lt = options.lastModified.lt;
}
}
if (options.dataStoreName) {
query['value.dataStoreName'] = {};
if (options.dataStoreName.ne) {
query['value.dataStoreName'].$ne = options.dataStoreName.ne;
}
}
if (!Object.keys(query._id).length) { if (!Object.keys(query._id).length) {
delete query._id; delete query._id;
} }

View File

@ -3,7 +3,7 @@ import { VersioningConstants } from './constants';
const VID_SEP = VersioningConstants.VersionId.Separator; const VID_SEP = VersioningConstants.VersionId.Separator;
/** /**
* Class for manipulating an object version. * Class for manipulating an object version.
* The format of a version: { isNull, isDeleteMarker, versionId, otherInfo } * The format of a version: { isNull, isNull2, isDeleteMarker, versionId, otherInfo }
* *
* @note Some of these functions are optimized based on string search * @note Some of these functions are optimized based on string search
* prior to a full JSON parse/stringify. (Vinh: 18K op/s are achieved * prior to a full JSON parse/stringify. (Vinh: 18K op/s are achieved
@ -13,24 +13,31 @@ const VID_SEP = VersioningConstants.VersionId.Separator;
export class Version { export class Version {
version: { version: {
isNull?: boolean; isNull?: boolean;
isNull2?: boolean;
isDeleteMarker?: boolean; isDeleteMarker?: boolean;
versionId?: string; versionId?: string;
isPHD?: boolean; isPHD?: boolean;
nullVersionId?: string;
}; };
/** /**
* Create a new version instantiation from its data object. * Create a new version instantiation from its data object.
* @param version - the data object to instantiate * @param version - the data object to instantiate
* @param version.isNull - is a null version * @param version.isNull - is a null version
* @param version.isNull2 - Whether new version is null or not AND has
* been put with a Cloudserver handling null keys (i.e. supporting
* S3C-7352)
* @param version.isDeleteMarker - is a delete marker * @param version.isDeleteMarker - is a delete marker
* @param version.versionId - the version id * @param version.versionId - the version id
* @constructor * @constructor
*/ */
constructor(version?: { constructor(version?: {
isNull?: boolean; isNull?: boolean;
isNull2?: boolean;
isDeleteMarker?: boolean; isDeleteMarker?: boolean;
versionId?: string; versionId?: string;
isPHD?: boolean; isPHD?: boolean;
nullVersionId?: string;
}) { }) {
this.version = version || {}; this.version = version || {};
} }
@ -166,6 +173,19 @@ export class Version {
return this.version.isNull ?? false; return this.version.isNull ?? false;
} }
/**
* Check if a version is a null version and has
* been put with a Cloudserver handling null keys (i.e. supporting
* S3C-7352).
*
* @return - stating if the value is a null version and has
* been put with a Cloudserver handling null keys (i.e. supporting
* S3C-7352).
*/
isNull2Version(): boolean {
return this.version.isNull2 ?? false;
}
/** /**
* Check if a stringified object is a delete marker. * Check if a stringified object is a delete marker.
* *
@ -235,6 +255,19 @@ export class Version {
return this; return this;
} }
/**
* Mark that the null version has been put with a Cloudserver handling null keys (i.e. supporting S3C-7352)
*
* If `isNull2` is set, `isNull` is also set to maintain consistency.
* Explicitly setting both avoids misunderstandings and mistakes in future updates or fixes.
* @return - the updated version
*/
setNull2Version() {
this.version.isNull2 = true;
this.version.isNull = true;
return this;
}
/** /**
* Serialize the version. * Serialize the version.
* *

View File

@ -22,11 +22,11 @@ function getPrefixUpperBoundary(prefix: string): string {
return prefix; return prefix;
} }
function formatVersionKey(key: string, versionId: string) { function formatVersionKey(key: string, versionId: string): string {
return `${key}${VID_SEP}${versionId}`; return `${key}${VID_SEP}${versionId}`;
} }
function formatCacheKey(db: string, key: string) { function formatCacheKey(db: string, key: string): string {
// using double VID_SEP to make sure the cache key is unique // using double VID_SEP to make sure the cache key is unique
return `${db}${VID_SEP}${VID_SEP}${key}`; return `${db}${VID_SEP}${VID_SEP}${key}`;
} }
@ -89,8 +89,10 @@ export default class VersioningRequestProcessor {
callback: (error: ArsenalError | null, data?: any) => void, callback: (error: ArsenalError | null, data?: any) => void,
) { ) {
const { db, key, options } = request; const { db, key, options } = request;
logger.addDefaultFields({ bucket: db, key, options });
if (options && options.versionId) { if (options && options.versionId) {
const versionKey = formatVersionKey(key, options.versionId); const keyVersionId = options.versionId === 'null' ? '' : options.versionId;
const versionKey = formatVersionKey(key, keyVersionId);
return this.wgm.get({ db, key: versionKey }, logger, callback); return this.wgm.get({ db, key: versionKey }, logger, callback);
} }
return this.wgm.get(request, logger, (err, data) => { return this.wgm.get(request, logger, (err, data) => {
@ -101,13 +103,82 @@ export default class VersioningRequestProcessor {
if (!Version.isPHD(data)) { if (!Version.isPHD(data)) {
return callback(null, data); return callback(null, data);
} }
logger.debug('master version is a PHD, getting the latest version', logger.debug('master version is a PHD, getting the latest version');
{ db, key });
// otherwise, need to search for the latest version // otherwise, need to search for the latest version
return this.getByListing(request, logger, callback); return this.getByListing(request, logger, callback);
}); });
} }
/**
* Helper that lists version keys for a certain object key,
* sorted by version ID. If a null key exists for this object, it is
* sorted at the appropriate position by its internal version ID and
* its key will be appended its internal version ID.
*
* @param {string} db - bucket name
* @param {string} key - object key
* @param {object} [options] - options object
* @param {number} [options.limit] - max version keys returned
* (returns all object version keys if not specified)
* @param {object} logger - logger of the request
* @param {function} callback - callback(err, {object|null} master, {array} versions)
* master: { key, value }
* versions: [{ key, value }, ...]
* @return {undefined}
*/
listVersionKeys(db, key, options, logger, callback) {
const { limit } = options || {};
const listingParams: any = {};
let nullKeyLength;
// include master key in v0 listing
listingParams.gte = key;
listingParams.lt = `${key}${VID_SEPPLUS}`;
if (limit !== undefined) {
// may have to skip master + null key, so 2 extra to list in the worst case
listingParams.limit = limit + 2;
}
nullKeyLength = key.length + 1;
return this.wgm.list({
db,
params: listingParams,
}, logger, (err, rawVersions) => {
if (err) {
return callback(err);
}
if (rawVersions.length === 0) {
// object does not have any version key
return callback(null, null, []);
}
let versions = rawVersions;
let master;
// in v0 there is always a master key before versions
master = versions.shift();
if (versions.length === 0) {
return callback(null, master, []);
}
const firstItem = versions[0];
if (firstItem.key.length === nullKeyLength) {
// first version is the null key
const nullVersion = Version.from(firstItem.value);
const nullVersionKey = formatVersionKey(key, <string> nullVersion.getVersionId());
// find null key's natural versioning order in the list
let nullPos = versions.findIndex(item => item.key > nullVersionKey);
if (nullPos === -1) {
nullPos = versions.length;
}
// move null key at the correct position and append its real version ID to the key
versions = versions.slice(1, nullPos)
.concat([{ key: nullVersionKey, value: firstItem.value, isNullKey: true }])
.concat(versions.slice(nullPos));
}
if (limit !== undefined) {
// truncate versions to 'limit' entries
versions.splice(limit);
}
return callback(null, master, versions);
});
}
/** /**
* Get the latest version of an object when the master version is a place * Get the latest version of an object when the master version is a place
* holder for deletion. For any given pair of db and key, only a * holder for deletion. For any given pair of db and key, only a
@ -132,39 +203,39 @@ export default class VersioningRequestProcessor {
if (!this.enqueueGet(request, logger, callback)) { if (!this.enqueueGet(request, logger, callback)) {
return null; return null;
} }
logger.info('start listing latest versions', { request }); logger.info('start listing latest versions');
// otherwise, search for the latest version // otherwise, search for the latest version
const cacheKey = formatCacheKey(request.db, request.key); const cacheKey = formatCacheKey(request.db, request.key);
clearTimeout(this.repairing[cacheKey]); clearTimeout(this.repairing[cacheKey]);
delete this.repairing[cacheKey]; delete this.repairing[cacheKey];
const req = { db: request.db, params: { return this.listVersionKeys(request.db, request.key, {
gte: request.key, lt: `${request.key}${VID_SEPPLUS}`, limit: 2 } }; limit: 1,
return this.wgm.list(req, logger, (err, list) => { }, logger, (err, master, versions) => {
logger.info('listing latest versions done', { err, list }); logger.info('listing latest versions done', { err, master, versions });
if (err) { if (err) {
return this.dequeueGet(request, err); return this.dequeueGet(request, err);
} }
// the complete list of versions is always: mst, v1, v2, ... if (!master) {
if (list.length === 0) {
return this.dequeueGet(request, errors.ObjNotFound); return this.dequeueGet(request, errors.ObjNotFound);
} }
if (!Version.isPHD(list[0].value)) { if (!Version.isPHD(master.value)) {
return this.dequeueGet(request, null, list[0].value); return this.dequeueGet(request, null, master.value);
} }
if (list.length === 1) { if (versions.length === 0) {
logger.info('no other versions', { request }); logger.info('no other versions');
this.dequeueGet(request, errors.ObjNotFound); this.dequeueGet(request, errors.ObjNotFound);
return this.repairMaster(request, logger, return this.repairMaster(request, logger,
{ type: 'del', { type: 'del', value: master.value });
value: list[0].value });
} }
// need repair // need repair
logger.info('update master by the latest version', { request }); logger.info('update master by the latest version');
const nextValue = list[1].value; const next = {
this.dequeueGet(request, null, nextValue); value: versions[0].value,
isNullKey: versions[0].isNullKey,
};
this.dequeueGet(request, null, next.value);
return this.repairMaster(request, logger, return this.repairMaster(request, logger,
{ type: 'put', value: list[0].value, { type: 'put', value: master.value, next });
nextValue });
}); });
} }
@ -227,42 +298,60 @@ export default class VersioningRequestProcessor {
* RepdConnection format { db, key * RepdConnection format { db, key
* [, value][, type], method, options } * [, value][, type], method, options }
* @param logger - logger * @param logger - logger
* @param hints - storing reparing hints * @param {object} data - storing reparing hints
* @param hints.type - type of repair operation ('put' or 'del') * @param {string} data.value - existing value of the master version (PHD)
* @param hints.value - existing value of the master version (PHD) * @param {object} data.next - the suggested latest version
* @param hints.nextValue - the suggested latest version * @param {string} data.next.value - the suggested latest version value
(for 'put') * @param {boolean} data.next.isNullKey - whether the suggested
* latest version is a null key
* @return - to finish the call * @return - to finish the call
*/ */
repairMaster(request: any, logger: RequestLogger, hints: { repairMaster(request: any, logger: RequestLogger, data: {
type: 'put' | 'del'; type: 'put' | 'del';
value: string; value: string;
nextValue?: string; next?: {
value: string;
isNullKey: boolean;
};
}) { }) {
const { db, key } = request; const { db, key } = request;
logger.info('start repair process', { request }); logger.info('start repair process');
this.writeCache.get({ db, key }, logger, (err, value) => { this.writeCache.get({ db, key }, logger, (err, value) => {
// error or the new version is not a place holder for deletion // error or the new version is not a place holder for deletion
if (err) { if (err) {
return logger.info('error repairing', { request, error: err }); if (err.is.ObjNotFound) {
return logger.debug('did not repair master: PHD was deleted');
} else {
return logger.error('error repairing', { error: err });
}
} }
if (!Version.isPHD(value)) { if (!Version.isPHD(value)) {
return logger.debug('master is updated already', { request }); return logger.debug('master is updated already');
} }
// the latest version is the same place holder for deletion // the latest version is the same place holder for deletion
if (hints.value === value) { if (data.value === value) {
// update the latest version with the next version // update the latest version with the next version
const ops: any = [];
if (data.next) {
ops.push({ key, value: data.next.value });
// cleanup the null key if it is the new master
if (data.next.isNullKey) {
ops.push({ key: formatVersionKey(key, ''), type: 'del' });
}
} else {
ops.push({ key, type: 'del' });
}
const repairRequest = { const repairRequest = {
db, db,
array: [ array: ops,
{ type: hints.type, key, value: hints.nextValue }, };
] };
logger.info('replicate repair request', { repairRequest }); logger.info('replicate repair request', { repairRequest });
return this.writeCache.batch(repairRequest, logger, () => {}); return this.writeCache.batch(repairRequest, logger, () => {});
} }
// The latest version is an updated place holder for deletion, // The latest version is an updated place holder for deletion,
// repeat the repair process from listing for latest versions. // repeat the repair process from listing for latest versions.
// The queue will ensure single repair process at any moment. // The queue will ensure single repair process at any moment.
logger.info('latest version is an updated PHD');
return this.getByListing(request, logger, () => {}); return this.getByListing(request, logger, () => {});
}); });
} }
@ -284,6 +373,7 @@ export default class VersioningRequestProcessor {
callback: (error: ArsenalError | null, data?: any) => void, callback: (error: ArsenalError | null, data?: any) => void,
) { ) {
const { db, key, value, options } = request; const { db, key, value, options } = request;
logger.addDefaultFields({ bucket: db, key, options });
// valid combinations of versioning options: // valid combinations of versioning options:
// - !versioning && !versionId: normal non-versioning put // - !versioning && !versionId: normal non-versioning put
// - versioning && !versionId: create a new version // - versioning && !versionId: create a new version
@ -337,6 +427,7 @@ export default class VersioningRequestProcessor {
versionId: string, versionId: string,
) => void, ) => void,
) { ) {
logger.info('process new version put');
// making a new versionId and a new version key // making a new versionId and a new version key
const versionId = this.generateVersionId(); const versionId = this.generateVersionId();
const versionKey = formatVersionKey(request.key, versionId); const versionKey = formatVersionKey(request.key, versionId);
@ -365,12 +456,22 @@ export default class VersioningRequestProcessor {
logger: RequestLogger, logger: RequestLogger,
callback: (err: ArsenalError | null, data?: any, versionId?: string) => void, callback: (err: ArsenalError | null, data?: any, versionId?: string) => void,
) { ) {
logger.info('process version specific put');
const { db, key } = request; const { db, key } = request;
// versionId is empty: update the master version // versionId is empty: update the master version
if (request.options.versionId === '') { if (request.options.versionId === '') {
const versionId = this.generateVersionId(); const versionId = this.generateVersionId();
const value = Version.appendVersionId(request.value, versionId); const value = Version.appendVersionId(request.value, versionId);
return callback(null, [{ key, value }], versionId); const ops: any = [{ key, value }];
if (request.options.deleteNullKey) {
const nullKey = formatVersionKey(key, '');
ops.push({ key: nullKey, type: 'del' });
}
return callback(null, ops, versionId);
}
if (request.options.versionId === 'null') {
const nullKey = formatVersionKey(key, '');
return callback(null, [{ key: nullKey, value: request.value }], 'null');
} }
// need to get the master version to check if this is the master version // need to get the master version to check if this is the master version
this.writeCache.get({ db, key }, logger, (err, data) => { this.writeCache.get({ db, key }, logger, (err, data) => {
@ -378,43 +479,112 @@ export default class VersioningRequestProcessor {
return callback(err); return callback(err);
} }
const versionId = request.options.versionId; const versionId = request.options.versionId;
const versionKey = formatVersionKey(request.key, versionId); const versionKey = formatVersionKey(key, versionId);
const ops = [{ key: versionKey, value: request.value }]; const ops: any = [];
const masterVersion = data !== undefined && const masterVersion = data !== undefined &&
Version.from(data); Version.from(data);
// push a version key if we're not updating the null
// version (or in legacy Cloudservers not sending the
// 'isNull' parameter, but this has an issue, see S3C-7526)
if (request.options.isNull !== true) {
const versionOp = { key: versionKey, value: request.value };
ops.push(versionOp);
}
if (masterVersion) { if (masterVersion) {
const versionIdFromMaster = masterVersion.getVersionId();
// master key exists // master key exists
// note that older versions have a greater version ID
const versionIdFromMaster = masterVersion.getVersionId();
if (versionIdFromMaster === undefined || if (versionIdFromMaster === undefined ||
versionIdFromMaster >= versionId) { versionIdFromMaster >= versionId) {
// master key is not newer than the put version
let masterVersionId;
let value = request.value; let value = request.value;
logger.debug('version to put is not older than master');
// Delete the deprecated, null key for backward compatibility
// to avoid storing both deprecated and new null keys.
// If master null version was put with an older Cloudserver (or in compat mode),
// there is a possibility that it also has a null versioned key
// associated, so we need to delete it as we write the null key.
// Deprecated null key gets deleted when the new CloudServer:
// - updates metadata of a null master (options.isNull=true)
// - puts metadata on top of a master null key (options.isNull=false)
if (request.options.isNull !== undefined && // new null key behavior when isNull is defined.
masterVersion.isNullVersion() && // master is null
!masterVersion.isNull2Version()) { // master does not support the new null key behavior yet.
const masterNullVersionId = masterVersion.getVersionId();
// The deprecated null key is referenced in the "versionId" property of the master key.
if (masterNullVersionId) {
const oldNullVersionKey = formatVersionKey(key, masterNullVersionId);
ops.push({ key: oldNullVersionKey, type: 'del' });
}
}
// new behavior when isNull is defined is to only
// update the master key if it is the latest
// version, old behavior needs to copy master to
// the null version because older Cloudservers
// rely on version-specific PUT to copy master
// contents to a new null version key (newer ones
// use special versionId="null" requests for this
// purpose).
if (versionIdFromMaster !== versionId ||
request.options.isNull === undefined) {
// master key is strictly older than the put version
let masterVersionId;
if (masterVersion.isNullVersion() && versionIdFromMaster) { if (masterVersion.isNullVersion() && versionIdFromMaster) {
// master key is a null version logger.debug('master key is a null version');
masterVersionId = versionIdFromMaster; masterVersionId = versionIdFromMaster;
} else if (versionIdFromMaster === undefined) { } else if (versionIdFromMaster === undefined) {
logger.debug('master key is nonversioned');
// master key does not have a versionID // master key does not have a versionID
// => create one with the "infinite" version ID // => create one with the "infinite" version ID
masterVersionId = getInfVid(this.replicationGroupId); masterVersionId = getInfVid(this.replicationGroupId);
masterVersion.setVersionId(masterVersionId); masterVersion.setVersionId(masterVersionId);
} else {
logger.debug('master key is a regular version');
} }
if (masterVersionId) { if (request.options.isNull === true) {
// => create a new version key from the master version if (!masterVersionId) {
const masterVersionKey = formatVersionKey(key, masterVersionId); // master is a regular version: delete the null key that
// may exist (older null version)
logger.debug('delete null key');
const nullKey = formatVersionKey(key, '');
ops.push({ key: nullKey, type: 'del' });
}
} else if (masterVersionId) {
logger.debug('create version key from master version');
// isNull === false means Cloudserver supports null keys,
// so create a null key in this case, and a version key otherwise
const masterKeyVersionId = request.options.isNull === false ?
'' : masterVersionId;
const masterVersionKey = formatVersionKey(key, masterKeyVersionId);
masterVersion.setNullVersion();
// isNull === false means Cloudserver supports null keys,
// so create a null key with the isNull2 flag
if (request.options.isNull === false) {
masterVersion.setNull2Version();
// else isNull === undefined means Cloudserver does not support null keys,
// and versionIdFromMaster !== versionId means that a version is PUT on top of a null version
// hence set/update the new master nullVersionId for backward compatibility
} else if (versionIdFromMaster !== versionId) {
// => set the nullVersionId to the master version if put version on top of null version. // => set the nullVersionId to the master version if put version on top of null version.
if (versionIdFromMaster !== versionId) {
value = Version.updateOrAppendNullVersionId(request.value, masterVersionId); value = Version.updateOrAppendNullVersionId(request.value, masterVersionId);
} }
masterVersion.setNullVersion();
ops.push({ key: masterVersionKey, ops.push({ key: masterVersionKey,
value: masterVersion.toString() }); value: masterVersion.toString() });
} }
// => update the master key, note that older } else {
// versions have a greater version ID logger.debug('version to put is the master');
ops.push({ key, value }); }
ops.push({ key, value: value });
} else {
logger.debug('version to put is older than master');
if (request.options.isNull === true && !masterVersion.isNullVersion()) {
logger.debug('create or update null key');
const nullKey = formatVersionKey(key, '');
const nullKeyOp = { key: nullKey, value: request.value };
ops.push(nullKeyOp);
// for backward compatibility: remove null version key
ops.push({ key: versionKey, type: 'del' });
}
} }
// otherwise, master key is newer so do not update it
} else { } else {
// master key does not exist: create it // master key does not exist: create it
ops.push({ key, value: request.value }); ops.push({ key, value: request.value });
@ -431,8 +601,10 @@ export default class VersioningRequestProcessor {
callback: (err: ArsenalError | null, data?: any) => void, callback: (err: ArsenalError | null, data?: any) => void,
) { ) {
const { db, key, options } = request; const { db, key, options } = request;
logger.addDefaultFields({ bucket: db, key, options });
// no versioning or versioning configuration off // no versioning or versioning configuration off
if (!(options && options.versionId)) { if (!(options && options.versionId)) {
logger.info('process non-versioned delete');
return this.writeCache.batch({ db, return this.writeCache.batch({ db,
array: [{ key, type: 'del' }] }, array: [{ key, type: 'del' }] },
logger, callback); logger, callback);
@ -470,7 +642,12 @@ export default class VersioningRequestProcessor {
versionId?: string, versionId?: string,
) => void, ) => void,
) { ) {
logger.info('process version specific delete');
const { db, key, options } = request; const { db, key, options } = request;
if (options.versionId === 'null') {
const nullKey = formatVersionKey(key, '');
return callback(null, [{ key: nullKey, type: 'del' }], 'null');
}
// deleting a specific version // deleting a specific version
this.writeCache.get({ db, key }, logger, (err, data) => { this.writeCache.get({ db, key }, logger, (err, data) => {
if (err && !err.is.ObjNotFound) { if (err && !err.is.ObjNotFound) {
@ -478,7 +655,8 @@ export default class VersioningRequestProcessor {
} }
// delete the specific version // delete the specific version
const versionId = options.versionId; const versionId = options.versionId;
const versionKey = formatVersionKey(key, versionId); const keyVersionId = options.isNull ? '' : versionId;
const versionKey = formatVersionKey(key, keyVersionId);
const ops: any = [{ key: versionKey, type: 'del' }]; const ops: any = [{ key: versionKey, type: 'del' }];
// update the master version as PHD if it is the deleting version // update the master version as PHD if it is the deleting version
if (Version.isPHD(data) || if (Version.isPHD(data) ||

View File

@ -3,7 +3,7 @@
"engines": { "engines": {
"node": ">=16" "node": ">=16"
}, },
"version": "7.10.59", "version": "7.70.26",
"description": "Common utilities for the S3 project components", "description": "Common utilities for the S3 project components",
"main": "build/index.js", "main": "build/index.js",
"repository": { "repository": {
@ -17,6 +17,7 @@
}, },
"homepage": "https://github.com/scality/Arsenal#readme", "homepage": "https://github.com/scality/Arsenal#readme",
"dependencies": { "dependencies": {
"@js-sdsl/ordered-set": "^4.4.2",
"@types/async": "^3.2.12", "@types/async": "^3.2.12",
"@types/utf8": "^3.0.1", "@types/utf8": "^3.0.1",
"JSONStream": "^1.0.0", "JSONStream": "^1.0.0",
@ -83,7 +84,7 @@
"build": "tsc", "build": "tsc",
"prepare": "yarn build", "prepare": "yarn build",
"ft_test": "jest tests/functional --testTimeout=120000 --forceExit", "ft_test": "jest tests/functional --testTimeout=120000 --forceExit",
"build_doc": "cd documentation/listingAlgos/pics; dot -Tsvg delimiterStateChart.dot > delimiterStateChart.svg; dot -Tsvg delimiterMasterV0StateChart.dot > delimiterMasterV0StateChart.svg" "build_doc": "cd documentation/listingAlgos/pics; dot -Tsvg delimiterStateChart.dot > delimiterStateChart.svg; dot -Tsvg delimiterMasterV0StateChart.dot > delimiterMasterV0StateChart.svg; dot -Tsvg delimiterVersionsStateChart.dot > delimiterVersionsStateChart.svg"
}, },
"private": true, "private": true,
"jest": { "jest": {

View File

@ -0,0 +1,309 @@
const async = require('async');
const cluster = require('cluster');
const http = require('http');
const errors = require('../../../build/lib/errors').default;
const {
setupRPCPrimary,
setupRPCWorker,
sendWorkerCommand,
getPendingCommandsCount,
} = require('../../../build/lib/clustering/ClusterRPC');
/* eslint-disable prefer-const */
let SERVER_PORT;
let N_WORKERS;
/* eslint-enable prefer-const */
/* eslint-disable no-console */
function genUIDS() {
return Math.trunc(Math.random() * 0x10000).toString(16);
}
// for testing robustness: regularly pollute the message channel with
// unrelated IPC messages
function sendPollutionMessage(message) {
if (cluster.isPrimary) {
const randomWorker = Math.trunc(Math.random() * cluster.workers.length);
const worker = cluster.workers[randomWorker];
if (worker) {
worker.send(message);
}
} else {
process.send(message);
}
}
const ipcPolluterIntervals = [
setInterval(
() => sendPollutionMessage('string pollution'), 1500),
setInterval(
() => sendPollutionMessage({ pollution: 'bar' }), 2321),
setInterval(
() => sendPollutionMessage({ type: 'pollution', foo: { bar: 'baz' } }), 2777),
];
function someTestHandlerFunc(payload, uids, callback) {
setTimeout(() => callback(null, { someResponsePayload: 'bar' }), 10);
}
function testHandlerWithFailureFunc(payload, uids, callback) {
setTimeout(() => {
// exactly one of the workers fails to execute this command
if (cluster.worker.id === 1) {
callback(errors.ServiceFailure);
} else {
callback(null, { someResponsePayload: 'bar' });
}
}, 10);
}
const rpcHandlers = {
SomeTestHandler: someTestHandlerFunc,
TestHandlerWithFailure: testHandlerWithFailureFunc,
TestHandlerWithNoResponse: () => {},
};
function respondOnTestFailure(message, error, results) {
console.error('After sendWorkerCommand() resolve/reject: ' +
`${message}, error=${error}, results=${JSON.stringify(results)}`);
console.trace();
throw errors.InternalError;
}
async function successfulCommandTestGeneric(nWorkers) {
try {
const results = await sendWorkerCommand('*', 'SomeTestHandler', genUIDS(), {});
if (results.length !== nWorkers) {
return respondOnTestFailure(
`expected ${nWorkers} worker results, got ${results.length}`,
null, results);
}
for (const result of results) {
if (typeof result !== 'object' || result === null) {
return respondOnTestFailure('not all results are objects', null, results);
}
if (result.error !== null) {
return respondOnTestFailure(
'one or more workers had an unexpected error',
null, results);
}
if (typeof result.result !== 'object' || result.result === null) {
return respondOnTestFailure(
'one or more workers did not return a result object',
null, results);
}
if (result.result.someResponsePayload !== 'bar') {
return respondOnTestFailure(
'one or more workers did not return the expected payload',
null, results);
}
}
return undefined;
} catch (err) {
return respondOnTestFailure(`returned unexpected error ${err}`, err, null);
}
}
async function successfulCommandTest() {
return successfulCommandTestGeneric(N_WORKERS);
}
async function successfulCommandWithExtraWorkerTest() {
return successfulCommandTestGeneric(N_WORKERS + 1);
}
async function unsupportedToWorkersTest() {
try {
const results = await sendWorkerCommand('badToWorkers', 'SomeTestHandler', genUIDS(), {});
return respondOnTestFailure('expected an error', null, results);
} catch (err) {
if (!err.is.NotImplemented) {
return respondOnTestFailure('expected a NotImplemented error', err, null);
}
return undefined;
}
}
async function unsupportedHandlerTest() {
try {
const results = await sendWorkerCommand('*', 'AWrongTestHandler', genUIDS(), {});
if (results.length !== N_WORKERS) {
return respondOnTestFailure(
`expected ${N_WORKERS} worker results, got ${results.length}`,
null, results);
}
for (const result of results) {
if (typeof result !== 'object' || result === null) {
return respondOnTestFailure('not all results are objects', null, results);
}
if (result.error === null || !result.error.is.NotImplemented) {
return respondOnTestFailure(
'one or more workers did not return the expected NotImplemented error',
null, results);
}
}
return undefined;
} catch (err) {
return respondOnTestFailure(`returned unexpected error ${err}`, err, null);
}
}
async function missingUidsTest() {
try {
const results = await sendWorkerCommand('*', 'SomeTestHandler', undefined, {});
return respondOnTestFailure('expected an error', null, results);
} catch (err) {
if (!err.is.MissingParameter) {
return respondOnTestFailure('expected a MissingParameter error', err, null);
}
return undefined;
}
}
async function duplicateUidsTest() {
const dupUIDS = genUIDS();
const promises = [
sendWorkerCommand('*', 'SomeTestHandler', dupUIDS, {}),
sendWorkerCommand('*', 'SomeTestHandler', dupUIDS, {}),
];
const results = await Promise.allSettled(promises);
if (results[1].status !== 'rejected') {
return respondOnTestFailure('expected an error from the second call', null, null);
}
if (!results[1].reason.is.OperationAborted) {
return respondOnTestFailure(
'expected a OperationAborted error', results[1].reason, null);
}
return undefined;
}
async function unsuccessfulWorkerTest() {
try {
const results = await sendWorkerCommand('*', 'TestHandlerWithFailure', genUIDS(), {});
if (results.length !== N_WORKERS) {
return respondOnTestFailure(
`expected ${N_WORKERS} worker results, got ${results.length}`,
null, results);
}
const nServiceFailures = results.filter(result => (
result.error && result.error.is.ServiceFailure
)).length;
if (nServiceFailures !== 1) {
return respondOnTestFailure(
'expected exactly one worker result to be ServiceFailure error',
null, results);
}
return undefined;
} catch (err) {
return respondOnTestFailure(`returned unexpected error ${err}`, err, null);
}
}
async function workerTimeoutTest() {
try {
const results = await sendWorkerCommand(
'*', 'TestHandlerWithNoResponse', genUIDS(), {}, 1000);
return respondOnTestFailure('expected an error', null, results);
} catch (err) {
if (!err.is.RequestTimeout) {
return respondOnTestFailure('expected a RequestTimeout error', err, null);
}
return undefined;
}
}
const TEST_URLS = {
'/successful-command': successfulCommandTest,
'/successful-command-with-extra-worker': successfulCommandWithExtraWorkerTest,
'/unsupported-to-workers': unsupportedToWorkersTest,
'/unsupported-handler': unsupportedHandlerTest,
'/missing-uids': missingUidsTest,
'/duplicate-uids': duplicateUidsTest,
'/unsuccessful-worker': unsuccessfulWorkerTest,
'/worker-timeout': workerTimeoutTest,
};
if (process.argv.length !== 4) {
console.error('ClusterRPC test server: GET requests on test URLs trigger test runs\n\n' +
'Usage: node ClusterRPC-test-server.js <port> <nb-workers>\n\n' +
'Available test URLs:');
console.error(`${Object.keys(TEST_URLS).map(url => `- ${url}\n`).join('')}`);
process.exit(2);
}
/* eslint-disable prefer-const */
[
SERVER_PORT,
N_WORKERS,
] = process.argv.slice(2, 4).map(value => Number.parseInt(value, 10));
/* eslint-enable prefer-const */
let server;
if (cluster.isPrimary) {
async.timesSeries(
N_WORKERS,
(i, wcb) => cluster.fork().on('online', wcb),
() => {
setupRPCPrimary();
},
);
} else {
// in worker
server = http.createServer((req, res) => {
if (req.url in TEST_URLS) {
return TEST_URLS[req.url]().then(() => {
if (getPendingCommandsCount() !== 0) {
console.error(`There are still ${getPendingCommandsCount()} pending ` +
`RPC commands after test ${req.url} completed`);
throw errors.InternalError;
}
res.writeHead(200);
res.end();
}).catch(err => {
res.writeHead(err.code);
res.end(err.message);
});
}
console.error(`Invalid test URL ${req.url}`);
res.writeHead(400);
res.end();
return undefined;
});
server.listen(SERVER_PORT);
server.on('listening', () => {
console.log('Worker is listening');
});
setupRPCWorker(rpcHandlers);
}
function stop(signal) {
if (cluster.isPrimary) {
console.log(`Handling signal ${signal}`);
for (const worker of Object.values(cluster.workers)) {
worker.kill(signal);
worker.on('exit', () => {
console.log(`Worker ${worker.id} exited`);
});
}
}
for (const interval of ipcPolluterIntervals) {
clearInterval(interval);
}
}
process.on('SIGTERM', stop);
process.on('SIGINT', stop);
process.on('SIGPIPE', () => {});
// for testing: spawn a new worker each time SIGUSR1 is received
function spawnNewWorker() {
if (cluster.isPrimary) {
cluster.fork();
}
}
process.on('SIGUSR1', spawnNewWorker);

View File

@ -0,0 +1,109 @@
'use strict'; // eslint-disable-line
const http = require('http');
const readline = require('readline');
const spawn = require('child_process').spawn;
const TEST_SERVER_PORT = 8800;
const NB_WORKERS = 4;
let testServer = null;
/*
* jest tests don't correctly support cluster mode with child forked
* processes, instead we use an external test server that launches
* each test based on the provided URL, and returns either 200 for
* success or 500 for failure. A crash would also cause a failure
* from the client side.
*/
function startTestServer(done) {
testServer = spawn('node', [
`${__dirname}/ClusterRPC-test-server.js`,
TEST_SERVER_PORT,
NB_WORKERS,
]);
// gather server stderr to display test failures info
testServer.stdout.pipe(process.stdout);
testServer.stderr.pipe(process.stderr);
const rl = readline.createInterface({
input: testServer.stdout,
});
let nbListeningWorkers = 0;
rl.on('line', line => {
if (line === 'Worker is listening') {
nbListeningWorkers++;
if (nbListeningWorkers === NB_WORKERS) {
rl.close();
done();
}
}
});
}
function stopTestServer(done) {
testServer.kill('SIGTERM');
testServer.on('close', done);
}
function runTest(testUrl, cb) {
const req = http.request(`http://localhost:${TEST_SERVER_PORT}/${testUrl}`, res => {
res
.on('data', () => {})
.on('end', () => {
expect(res.statusCode).toEqual(200);
cb();
})
.on('error', err => cb(err));
});
req
.end()
.on('error', err => cb(err));
}
describe('ClusterRPC', () => {
beforeAll(done => startTestServer(done));
afterAll(done => stopTestServer(done));
it('should send a successful command to all workers', done => {
runTest('successful-command', done);
});
it('should error if "toWorkers" field is not "*"', done => {
runTest('unsupported-to-workers', done);
});
it('should error if handler name is not known', done => {
runTest('unsupported-handler', done);
});
it('should error if "uids" field is not passed', done => {
runTest('missing-uids', done);
});
it('should error if two simultaneous commands with same "uids" field are sent', done => {
runTest('duplicate-uids', done);
});
it('should timeout if one or more workers don\'t respond in allocated time', done => {
runTest('worker-timeout', done);
});
it('should return worker errors in results array', done => {
runTest('unsuccessful-worker', done);
});
it('should send a successful command to all workers after an extra worker is spawned', done => {
const rl = readline.createInterface({
input: testServer.stdout,
});
rl.on('line', line => {
if (line === 'Worker is listening') {
rl.close();
runTest('successful-command-with-extra-worker', done);
}
});
// The test server spawns a new worker when it receives SIGUSR1
testServer.kill('SIGUSR1');
});
});

265
tests/unit/algos/cache/GapCache.spec.ts vendored Normal file
View File

@ -0,0 +1,265 @@
import GapCache from '../../../../lib/algos/cache/GapCache';
describe('GapCache', () => {
let gapCache;
beforeEach(() => {
// exposureDelayMs=100, maxGaps=10, maxGapWeight=100
gapCache = new GapCache(100, 10, 100);
gapCache.start();
});
afterEach(() => {
gapCache.stop();
});
describe('getters and setters', () => {
it('maxGapWeight getter', () => {
expect(gapCache.maxGapWeight).toEqual(100);
});
it('maxGapWeight setter', () => {
gapCache.maxGapWeight = 123;
expect(gapCache.maxGapWeight).toEqual(123);
// check that internal gap sets have also been updated
expect(gapCache._stagingUpdates.newGaps.maxWeight).toEqual(123);
expect(gapCache._frozenUpdates.newGaps.maxWeight).toEqual(123);
});
it('exposureDelayMs getter', () => {
expect(gapCache.exposureDelayMs).toEqual(100);
});
it('exposureDelayMs setter', async () => {
// insert a first gap
gapCache.setGap('bar', 'baz', 10);
// change the exposure delay to 50ms
gapCache.exposureDelayMs = 50;
expect(gapCache.exposureDelayMs).toEqual(50);
gapCache.setGap('qux', 'quz', 10);
// wait for more than twice the new exposure delay
await new Promise(resolve => setTimeout(resolve, 200));
// only the second gap should have been exposed, due to the change of
// exposure delay subsequent to the first call to setGap()
expect(await gapCache.lookupGap('ape', 'zoo')).toEqual(
{ firstKey: 'qux', lastKey: 'quz', weight: 10 }
);
});
});
describe('clear()', () => {
it('should clear all exposed gaps', async () => {
gapCache.setGap('bar', 'baz', 10);
gapCache.setGap('qux', 'quz', 20);
await new Promise(resolve => setTimeout(resolve, 300));
expect(await gapCache.lookupGap('ape', 'zoo')).toEqual(
{ firstKey: 'bar', lastKey: 'baz', weight: 10 }
);
gapCache.clear();
expect(await gapCache.lookupGap('ape', 'zoo')).toBeNull();
});
it('should clear all staging gaps', async () => {
gapCache.setGap('bar', 'baz', 10);
gapCache.setGap('qux', 'quz', 20);
gapCache.clear();
await new Promise(resolve => setTimeout(resolve, 300));
expect(await gapCache.lookupGap('ape', 'zoo')).toBeNull();
});
it('should keep existing invalidating updates against the next new gaps', async () => {
// invalidate future gaps containing 'dog'
expect(gapCache.removeOverlappingGaps(['dog'])).toEqual(0);
// then, clear the cache
gapCache.clear();
// wait for 50ms (half of exposure delay of 100ms) before
// setting a new gap overlapping with 'dog'
await new Promise(resolve => setTimeout(resolve, 50));
gapCache.setGap('cat', 'fox', 10);
// also set a non-overlapping gap to make sure it is not invalidated
gapCache.setGap('goat', 'hog', 20);
// wait an extra 250ms to ensure all valid gaps have been exposed
await new Promise(resolve => setTimeout(resolve, 250));
// the next gap is indeed 'goat'... because 'cat'... should have been invalidated
expect(await gapCache.lookupGap('bat', 'zoo')).toEqual(
{ firstKey: 'goat', lastKey: 'hog', weight: 20 });
});
});
it('should expose gaps after at least exposureDelayMs milliseconds', async () => {
gapCache.setGap('bar', 'baz', 10);
expect(await gapCache.lookupGap('ape', 'cat')).toBeNull();
// wait for 50ms which is half of the minimum time to exposure
await new Promise(resolve => setTimeout(resolve, 50));
// the gap should not be exposed yet
expect(await gapCache.lookupGap('ape', 'cat')).toBeNull();
// wait for an extra 250ms (total 300ms): the upper bound for exposure of any
// setGap() call is twice the exposureDelayMs value, so 200ms, wait an extra
// 100ms to cope with scheduling uncertainty and GapSet processing time, after
// which the gap introduced by setGap() should always be exposed.
await new Promise(resolve => setTimeout(resolve, 250));
expect(await gapCache.lookupGap('ape', 'cat')).toEqual(
{ firstKey: 'bar', lastKey: 'baz', weight: 10 });
// check getters
expect(gapCache.maxGaps).toEqual(10);
expect(gapCache.maxGapWeight).toEqual(100);
expect(gapCache.size).toEqual(1);
// check iteration over the exposed gaps
let nGaps = 0;
for (const gap of gapCache) {
expect(gap).toEqual({ firstKey: 'bar', lastKey: 'baz', weight: 10 });
nGaps += 1;
}
expect(nGaps).toEqual(1);
// check toArray()
expect(gapCache.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
]);
});
it('removeOverlappingGaps() should invalidate all overlapping gaps that are already exposed',
async () => {
gapCache.setGap('cat', 'fox', 10);
gapCache.setGap('lion', 'seal', 20);
// wait for 3x100ms to ensure all setGap() calls have been exposed
await new Promise(resolve => setTimeout(resolve, 300));
// expect 0 gap removed because 'hog' is not in any gap
expect(gapCache.removeOverlappingGaps(['hog'])).toEqual(0);
// expect 1 gap removed because 'cat' -> 'fox' should be already exposed
expect(gapCache.removeOverlappingGaps(['dog'])).toEqual(1);
// the gap should have been invalidated permanently
expect(await gapCache.lookupGap('dog', 'fox')).toBeNull();
// the other gap should still be present
expect(await gapCache.lookupGap('rat', 'tiger')).toEqual(
{ firstKey: 'lion', lastKey: 'seal', weight: 20 });
});
it('removeOverlappingGaps() should invalidate all overlapping gaps that are not yet exposed',
async () => {
gapCache.setGap('cat', 'fox', 10);
gapCache.setGap('lion', 'seal', 20);
// make the following calls asynchronous for the sake of the
// test, but not waiting for the exposure delay
await new Promise(resolve => setImmediate(resolve));
// expect 0 gap removed because 'hog' is not in any gap
expect(gapCache.removeOverlappingGaps(['hog'])).toEqual(0);
// expect 0 gap removed because 'cat' -> 'fox' is not exposed yet,
// but internally it should have been removed from the staging or
// frozen gap set
expect(gapCache.removeOverlappingGaps(['dog'])).toEqual(0);
// wait for 3x100ms to ensure all non-invalidated setGap() calls have been exposed
await new Promise(resolve => setTimeout(resolve, 300));
// the gap should have been invalidated permanently
expect(await gapCache.lookupGap('dog', 'fox')).toBeNull();
// the other gap should now be exposed
expect(await gapCache.lookupGap('rat', 'tiger')).toEqual(
{ firstKey: 'lion', lastKey: 'seal', weight: 20 });
});
it('removeOverlappingGaps() should invalidate gaps created later by setGap() but ' +
'within the exposure delay', async () => {
// wait for 80ms (slightly less than exposure delay of 100ms)
// before calling removeOverlappingGaps(), so that the next
// exposure timer kicks in before the call to setGap()
await new Promise(resolve => setTimeout(resolve, 80));
// there is no exposed gap yet, so expect 0 gap removed
expect(gapCache.removeOverlappingGaps(['dog'])).toEqual(0);
// wait for 50ms (half of exposure delay of 100ms) before
// setting a new gap overlapping with 'dog'
await new Promise(resolve => setTimeout(resolve, 50));
gapCache.setGap('cat', 'fox', 10);
// also set a non-overlapping gap to make sure it is not invalidated
gapCache.setGap('goat', 'hog', 20);
// wait an extra 250ms to ensure all valid gaps have been exposed
await new Promise(resolve => setTimeout(resolve, 250));
// the next gap is indeed 'goat'... because 'cat'... should have been invalidated
expect(await gapCache.lookupGap('bat', 'zoo')).toEqual(
{ firstKey: 'goat', lastKey: 'hog', weight: 20 });
});
it('removeOverlappingGaps() should not invalidate gaps created more than twice ' +
'the exposure delay later', async () => {
// there is no exposed gap yet, so expect 0 gap removed
expect(gapCache.removeOverlappingGaps(['dog'])).toEqual(0);
// wait for 250ms (more than twice the exposure delay of 100ms) before
// setting a new gap overlapping with 'dog'
await new Promise(resolve => setTimeout(resolve, 250));
gapCache.setGap('cat', 'fox', 10);
// also set a non-overlapping gap to make sure it is not invalidated
gapCache.setGap('goat', 'hog', 20);
// wait for an extra 250ms to ensure the new gap is exposed
await new Promise(resolve => setTimeout(resolve, 250));
// should find the inserted gap as it should not have been invalidated
expect(await gapCache.lookupGap('bat', 'zoo')).toEqual(
{ firstKey: 'cat', lastKey: 'fox', weight: 10 });
});
it('exposed gaps should be merged when possible', async () => {
gapCache.setGap('bar', 'baz', 10);
gapCache.setGap('baz', 'qux', 10);
// wait until the merged gap is exposed
await new Promise(resolve => setTimeout(resolve, 300));
expect(await gapCache.lookupGap('ape', 'cat')).toEqual(
{ firstKey: 'bar', lastKey: 'qux', weight: 20 });
});
it('exposed gaps should be split when above maxGapWeight', async () => {
gapCache.setGap('bar', 'baz', gapCache.maxGapWeight - 1);
gapCache.setGap('baz', 'qux', 10);
// wait until the gaps are exposed
await new Promise(resolve => setTimeout(resolve, 300));
expect(await gapCache.lookupGap('cat', 'dog')).toEqual(
{ firstKey: 'baz', lastKey: 'qux', weight: 10 });
});
it('gaps should not be exposed when reaching the maxGaps limit', async () => {
const gapsArray = new Array(gapCache.maxGaps).fill(undefined).map(
(_, i) => {
const firstKey = `0000${i}`.slice(-4);
return {
firstKey,
lastKey: `${firstKey}foo`,
weight: 10,
};
}
);
for (const gap of gapsArray) {
gapCache.setGap(gap.firstKey, gap.lastKey, gap.weight);
}
// wait until the gaps are exposed
await new Promise(resolve => setTimeout(resolve, 300));
expect(gapCache.size).toEqual(gapCache.maxGaps);
gapCache.setGap('noroomforthisgap', 'noroomforthisgapfoo');
// wait until the gaps are exposed
await new Promise(resolve => setTimeout(resolve, 300));
// the number of gaps should still be 'maxGaps'
expect(gapCache.size).toEqual(gapCache.maxGaps);
// the gaps should correspond to the original array
expect(gapCache.toArray()).toEqual(gapsArray);
});
});

878
tests/unit/algos/cache/GapSet.spec.ts vendored Normal file
View File

@ -0,0 +1,878 @@
import { OrderedSet } from '@js-sdsl/ordered-set';
import GapSet from '../../../../lib/algos/cache/GapSet';
function genRandomKey(): string {
const CHARS = 'abcdefghijklmnopqrstuvwxyz0123456789';
return new Array(16).fill(undefined).map(
() => CHARS[Math.trunc(Math.random() * CHARS.length)]
).join('');
}
function genRandomUnchainedGaps(nGaps) {
const gapBounds = new Array(nGaps * 2).fill(undefined).map(
() => genRandomKey()
);
gapBounds.sort();
const gapsArray = new Array(nGaps).fill(undefined).map(
(_, i) => ({
firstKey: gapBounds[2 * i],
lastKey: gapBounds[2 * i + 1],
weight: 10,
})
);
return gapsArray;
}
function genRandomChainedGaps(nGaps) {
const gapBounds = new Array(nGaps + 1).fill(undefined).map(
() => genRandomKey()
);
gapBounds.sort();
const gapsArray = new Array(nGaps).fill(undefined).map(
(_, i) => ({
firstKey: gapBounds[i],
lastKey: gapBounds[i + 1],
weight: 10,
})
);
return gapsArray;
}
/**
* Shuffle an array in-place
*
* @param {any[]} - The array to shuffle
* @return {undefined}
*/
function shuffleArray(array) {
for (let i = array.length - 1; i > 0; i--) {
const randIndex = Math.trunc(Math.random() * (i + 1));
/* eslint-disable no-param-reassign */
const randIndexVal = array[randIndex];
array[randIndex] = array[i];
array[i] = randIndexVal;
/* eslint-enable no-param-reassign */
}
}
describe('GapSet', () => {
const INITIAL_GAPSET = [
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
];
const INITIAL_GAPSET_WITH_CHAIN = [
// single-key gap
{ firstKey: 'ape', lastKey: 'ape', weight: 1 },
// start of chain
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'baz', lastKey: 'qux', weight: 15 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
{ firstKey: 'quz', lastKey: 'rat', weight: 25 },
{ firstKey: 'rat', lastKey: 'yak', weight: 30 },
// end of chain
]
let gapsArray;
let gapSet;
let gapsArrayWithChain;
let gapSetWithChain;
beforeEach(() => {
gapsArray = JSON.parse(
JSON.stringify(INITIAL_GAPSET)
);
gapSet = GapSet.createFromArray(gapsArray, 100);
gapsArrayWithChain = JSON.parse(
JSON.stringify(INITIAL_GAPSET_WITH_CHAIN)
);
gapSetWithChain = GapSet.createFromArray(gapsArrayWithChain, 100);
});
describe('GapSet::size', () => {
it('should return 0 for an empty gap set', () => {
const emptyGapSet = new GapSet(100);
expect(emptyGapSet.size).toEqual(0);
});
it('should return the size of the gap set', () => {
expect(gapSet.size).toEqual(2);
});
it('should reflect the new size after removal of gaps', () => {
gapSet._gaps.eraseElementByKey({ firstKey: 'bar' });
expect(gapSet.size).toEqual(1);
});
});
describe('GapSet::maxWeight', () => {
it('getter', () => {
const emptyGapSet = new GapSet(123);
expect(emptyGapSet.maxWeight).toEqual(123);
});
it('setter', () => {
const emptyGapSet = new GapSet(123);
emptyGapSet.maxWeight = 456;
expect(emptyGapSet.maxWeight).toEqual(456);
});
});
describe('GapSet::setGap()', () => {
it('should start a gap with a single key in empty gap set', () => {
const emptyGapSet = new GapSet(100);
const gap = emptyGapSet.setGap('foo', 'foo', 1);
expect(gap).toEqual({ firstKey: 'foo', lastKey: 'foo', weight: 1 });
expect(emptyGapSet.toArray()).toEqual([
{ firstKey: 'foo', lastKey: 'foo', weight: 1 },
]);
});
it('should start a gap with a single key in non-empty gap set', () => {
const gap = gapSet.setGap('foo', 'foo', 1);
expect(gap).toEqual({ firstKey: 'foo', lastKey: 'foo', weight: 1 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'foo', lastKey: 'foo', weight: 1 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
it('should start a gap with multiple keys in empty gap set', () => {
const emptyGapSet = new GapSet(100);
const gap = emptyGapSet.setGap('foo', 'qux', 5);
expect(gap).toEqual({ firstKey: 'foo', lastKey: 'qux', weight: 5 });
expect(emptyGapSet.toArray()).toEqual([
{ firstKey: 'foo', lastKey: 'qux', weight: 5 },
]);
});
it('should return a new object rather than a gap managed by GapSet', () => {
const emptyGapSet = new GapSet(100);
const gap = emptyGapSet.setGap('foo', 'qux', 5);
gap.lastKey = 'quz';
// check that modifying the returned gap doesn't affect the GapSet
expect(emptyGapSet.toArray()).toEqual([
{ firstKey: 'foo', lastKey: 'qux', weight: 5 },
]);
});
it('should return an existing gap that includes the wanted gap', () => {
const gap = gapSet.setGap('bat', 'bay', 5);
expect(gap).toEqual({ firstKey: 'bar', lastKey: 'baz', weight: 10 });
expect(gapSet.toArray()).toEqual(INITIAL_GAPSET);
});
it('should return an existing gap that starts with the wanted gap first key', () => {
const gap = gapSet.setGap('bar', 'bay', 5);
expect(gap).toEqual({ firstKey: 'bar', lastKey: 'baz', weight: 10 });
expect(gapSet.toArray()).toEqual(INITIAL_GAPSET);
});
it('should return an existing gap that ends with the wanted gap last key', () => {
const gap = gapSet.setGap('bat', 'baz', 5);
expect(gap).toEqual({ firstKey: 'bar', lastKey: 'baz', weight: 10 });
expect(gapSet.toArray()).toEqual(INITIAL_GAPSET);
});
it('should return the existing chained gap that starts with the first key', () => {
const gap = gapSetWithChain.setGap('baz', 'quo', 10);
expect(gap).toEqual({ firstKey: 'baz', lastKey: 'qux', weight: 15 });
expect(gapSetWithChain.toArray()).toEqual(INITIAL_GAPSET_WITH_CHAIN);
});
it('should extend a single-key gap with no other gap', () => {
const singleKeyGap = { firstKey: 'foo', lastKey: 'foo', weight: 1 };
const singleKeyGapSet = GapSet.createFromArray([singleKeyGap], 100);
const extendedGap = singleKeyGapSet.setGap('foo', 'qux', 30);
expect(extendedGap).toEqual({ firstKey: 'foo', lastKey: 'qux', weight: 31 });
expect(singleKeyGapSet.toArray()).toEqual([
{ firstKey: 'foo', lastKey: 'qux', weight: 31 },
]);
});
it('should extend a gap with no next gap', () => {
// existing gap: 'qux' -> 'quz'
const extendedGap = gapSet.setGap('qux', 'rat', 25);
expect(extendedGap).toEqual({ firstKey: 'qux', lastKey: 'rat', weight: 25 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'qux', lastKey: 'rat', weight: 25 },
]);
});
it('should extend a gap without overlap with next gap', () => {
// existing gap: 'bar' -> 'baz'
const extendedGap = gapSet.setGap('bar', 'dog', 15);
expect(extendedGap).toEqual({ firstKey: 'bar', lastKey: 'dog', weight: 15 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'dog', weight: 15 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
it('should extend a gap starting from its last key', () => {
// existing gap: 'qux' -> 'quz'
const extendedGap = gapSet.setGap('quz', 'rat', 5);
expect(extendedGap).toEqual({ firstKey: 'qux', lastKey: 'rat', weight: 25 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'qux', lastKey: 'rat', weight: 25 },
]);
});
it('should merge with next gap with single-key overlap if total weight is ' +
'under maxWeight', () => {
const extendedGap = gapSet.setGap('bar', 'qux', 80);
// updated weight is accurately set as the sum of
// overlapping individual gap weights
expect(extendedGap).toEqual({ firstKey: 'bar', lastKey: 'quz', weight: 80 + 20 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'quz', weight: 80 + 20 },
]);
});
it('should chain with next gap with single-key overlap if total weight is ' +
'above maxWeight', () => {
const extendedGap = gapSet.setGap('bar', 'qux', 90);
expect(extendedGap).toEqual({ firstKey: 'qux', lastKey: 'quz', weight: 20 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'qux', weight: 90 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
it('should merge with both previous and next gap if bounds overlap by a ' +
'single key and total weight is under maxWeight', () => {
const extendedGap = gapSet.setGap('baz', 'qux', 30);
// updated weight is accurately set as the sum of
// overlapping individual gap weights
expect(extendedGap).toEqual({ firstKey: 'bar', lastKey: 'quz', weight: 10 + 30 + 20 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'quz', weight: 10 + 30 + 20 },
]);
});
it('should merge with previous gap and chain with next gap if bounds overlap by a ' +
'single key on either side and weight is above maxWeight when merging on right side', () => {
const extendedGap = gapSet.setGap('baz', 'qux', 90);
expect(extendedGap).toEqual({ firstKey: 'qux', lastKey: 'quz', weight: 20 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'qux', weight: 100 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
it('should chain with previous gap and merge with next gap if bounds overlap by a ' +
'single key on either side and weight is above maxWeight when merging on left side', () => {
// modified version of the common test set with increased weight
// for 'bar' -> 'baz'
const gapSet = GapSet.createFromArray([
{ firstKey: 'bar', lastKey: 'baz', weight: 80 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
], 100);
const extendedGap = gapSet.setGap('baz', 'qux', 70);
expect(extendedGap).toEqual({ firstKey: 'baz', lastKey: 'quz', weight: 90 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'baz', weight: 80 },
{ firstKey: 'baz', lastKey: 'quz', weight: 90 },
]);
});
it('should merge with both previous and next gap if left bound overlaps by a ' +
'single key and total weight is under maxWeight', () => {
const extendedGap = gapSet.setGap('baz', 'quxxx', 40);
// updated weight is heuristically set as the sum of the
// previous chained gap's weight and the new weight
// (excluding the overlapping gap on right side)
expect(extendedGap).toEqual({ firstKey: 'bar', lastKey: 'quz', weight: 10 + 40 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'quz', weight: 10 + 40 },
]);
});
it('should chain with previous gap and merge with next gap if left bound overlaps by a ' +
'single key and total weight is above maxWeight', () => {
const extendedGap = gapSet.setGap('baz', 'quxxx', 95);
// updated weight is accurately set as the sum of
// overlapping individual gap weights
expect(extendedGap).toEqual({ firstKey: 'baz', lastKey: 'quz', weight: 95 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'baz', lastKey: 'quz', weight: 95 },
]);
});
it('should extend a gap with overlap with next gap and large weight', () => {
const extendedGap = gapSet.setGap('bar', 'quxxx', 80);
// updated weight is heuristically chosen to be the new
// gap weight which is larger than the sum of the existing merged
// gap weights
expect(extendedGap).toEqual({ firstKey: 'bar', lastKey: 'quz', weight: 80 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'quz', weight: 80 },
]);
});
it('should extend a gap with overlap with next gap and small weight', () => {
const extendedGap = gapSet.setGap('bar', 'quxxx', 11);
// updated weight is heuristically chosen to be the sum of the existing merged
// gap weights which is larger than the new gap weight
expect(extendedGap).toEqual({ firstKey: 'bar', lastKey: 'quz', weight: 10 + 20 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'quz', weight: 10 + 20 },
]);
});
it('should extend a gap with overlap beyond last key of next gap', () => {
const extendedGap = gapSet.setGap('bar', 'rat', 80);
// updated weight is the new gap weight
expect(extendedGap).toEqual({ firstKey: 'bar', lastKey: 'rat', weight: 80 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'rat', weight: 80 },
]);
});
it('should extend a gap with overlap beyond last key of next gap with a chained gap ' +
'if above maxWeight', () => {
// gapSet was initialized with maxWeight=100
const extendedGap = gapSet.setGap('bar', 'rat', 105);
// returned new gap is the right-side chained gap
// updated weight is the new gap weight minus the left-side chained gap's weight
expect(extendedGap).toEqual({ firstKey: 'baz', lastKey: 'rat', weight: 105 - 10 });
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'baz', lastKey: 'rat', weight: 105 - 10 },
]);
});
it('should extend a single-key gap with overlap on chained gaps', () => {
// existing gap: 'ape' -> 'ape' (weight=1)
const extendedGap = gapSetWithChain.setGap('ape', 'dog', 30);
// updated weight heuristically including the new gap
// weight, which is larger than the overlapping gaps cumulated
// weights (10+15=25)
expect(extendedGap).toEqual({ firstKey: 'ape', lastKey: 'qux', weight: 30 });
expect(gapSetWithChain.toArray()).toEqual([
{ firstKey: 'ape', lastKey: 'qux', weight: 30 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
{ firstKey: 'quz', lastKey: 'rat', weight: 25 },
{ firstKey: 'rat', lastKey: 'yak', weight: 30 },
]);
});
it('should merge and extend + update weight a gap with overlap not past end of chained gaps',
() => {
const extendedGap = gapSetWithChain.setGap('baz', 'sea', 80);
expect(extendedGap).toEqual({ firstKey: 'baz', lastKey: 'yak', weight: 90 });
expect(gapSetWithChain.toArray()).toEqual([
{ firstKey: 'ape', lastKey: 'ape', weight: 1 },
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'baz', lastKey: 'yak', weight: 90 },
]);
});
it('should merge and extend + update weight a gap with overlap past end of chained gaps',
() => {
const extendedGap = gapSetWithChain.setGap('baz', 'zoo', 95);
expect(extendedGap).toEqual({ firstKey: 'baz', lastKey: 'zoo', weight: 95 });
expect(gapSetWithChain.toArray()).toEqual([
{ firstKey: 'ape', lastKey: 'ape', weight: 1 },
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'baz', lastKey: 'zoo', weight: 95 },
]);
});
it('should extend gap + update weight with overlap past end of chained gaps and ' +
'above maxWeight', () => {
const extendedGap = gapSetWithChain.setGap('baz', 'zoo', 105);
// updated weight is the new gap weight minus the left-side chained gap's weight
expect(extendedGap).toEqual({ firstKey: 'qux', lastKey: 'zoo', weight: 105 - 15 });
expect(gapSetWithChain.toArray()).toEqual([
{ firstKey: 'ape', lastKey: 'ape', weight: 1 },
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'baz', lastKey: 'qux', weight: 15 },
{ firstKey: 'qux', lastKey: 'zoo', weight: 105 - 15 },
]);
});
it('should return existing chained gap with overlap above maxWeight', () => {
const chainedGapsArray = [
{ firstKey: 'ant', lastKey: 'cat', weight: 90 },
{ firstKey: 'cat', lastKey: 'fox', weight: 40 },
];
const chainedGapsSet = GapSet.createFromArray(chainedGapsArray, 100);
const extendedGap = chainedGapsSet.setGap('bat', 'dog', 105);
expect(extendedGap).toEqual({ firstKey: 'cat', lastKey: 'fox', weight: 40 });
expect(chainedGapsSet.toArray()).toEqual([
{ firstKey: 'ant', lastKey: 'cat', weight: 90 },
{ firstKey: 'cat', lastKey: 'fox', weight: 40 },
]);
});
it('should merge but not extend nor update weight with overlap on chained gaps', () => {
// existing chained gap: 'baz' -> 'qux'
const extendedGap = gapSetWithChain.setGap('baz', 'quxxx', 25);
// updated weight is the sum of the two merged gap's weights
expect(extendedGap).toEqual({ firstKey: 'baz', lastKey: 'quz', weight: 15 + 20 });
expect(gapSetWithChain.toArray()).toEqual([
{ firstKey: 'ape', lastKey: 'ape', weight: 1 },
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'baz', lastKey: 'quz', weight: 15 + 20 },
{ firstKey: 'quz', lastKey: 'rat', weight: 25 },
{ firstKey: 'rat', lastKey: 'yak', weight: 30 },
]);
});
});
describe('GapSet::removeOverlappingGaps()', () => {
describe('with zero key as parameter', () => {
it('passed in an array: should not remove any gap', () => {
const nRemoved = gapSet.removeOverlappingGaps([]);
expect(nRemoved).toEqual(0);
expect(gapSet.toArray()).toEqual(INITIAL_GAPSET);
});
it('passed in a OrderedSet: should not remove any gap', () => {
const nRemoved = gapSet.removeOverlappingGaps(new OrderedSet());
expect(nRemoved).toEqual(0);
expect(gapSet.toArray()).toEqual(INITIAL_GAPSET);
});
});
describe('with an array of one key as parameter', () => {
it('should not remove any gap if no overlap', () => {
const nRemoved = gapSet.removeOverlappingGaps(['rat']);
expect(nRemoved).toEqual(0);
expect(gapSet.toArray()).toEqual(INITIAL_GAPSET);
});
it('should remove a single gap if overlaps', () => {
const nRemoved = gapSet.removeOverlappingGaps(['bat']);
expect(nRemoved).toEqual(1);
expect(gapSet.toArray()).toEqual([
// removed: { firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
it('should remove a single gap if overlaps with first key of first gap', () => {
const nRemoved = gapSet.removeOverlappingGaps(['bar']);
expect(nRemoved).toEqual(1);
expect(gapSet.toArray()).toEqual([
// removed: { firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
it('should remove a single gap if overlaps with first key of non-first gap', () => {
const nRemoved = gapSet.removeOverlappingGaps(['qux']);
expect(nRemoved).toEqual(1);
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
// removed: { firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
it('should remove a single gap if overlaps with last key', () => {
const nRemoved = gapSet.removeOverlappingGaps(['quz']);
expect(nRemoved).toEqual(1);
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
// removed: { firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
it('should remove a single gap in chain if overlaps with one chained gap', () => {
const nRemoved = gapSetWithChain.removeOverlappingGaps(['dog']);
expect(nRemoved).toEqual(1);
expect(gapSetWithChain.toArray()).toEqual([
{ firstKey: 'ape', lastKey: 'ape', weight: 1 },
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
// removed: { firstKey: 'baz', lastKey: 'qux', weight: 15 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
{ firstKey: 'quz', lastKey: 'rat', weight: 25 },
{ firstKey: 'rat', lastKey: 'yak', weight: 30 },
]);
});
it('should remove two gaps in chain if overlaps with two chained gap', () => {
const nRemoved = gapSetWithChain.removeOverlappingGaps(['qux']);
expect(nRemoved).toEqual(2);
expect(gapSetWithChain.toArray()).toEqual([
{ firstKey: 'ape', lastKey: 'ape', weight: 1 },
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
// removed: { firstKey: 'baz', lastKey: 'qux', weight: 15 },
// removed: { firstKey: 'qux', lastKey: 'quz', weight: 20 },
{ firstKey: 'quz', lastKey: 'rat', weight: 25 },
{ firstKey: 'rat', lastKey: 'yak', weight: 30 },
]);
});
});
describe('with an array of two keys as parameter', () => {
it('should not remove any gap if no overlap', () => {
const nRemoved = gapSet.removeOverlappingGaps(['rat', `rat\0v100`]);
expect(nRemoved).toEqual(0);
expect(gapSet.toArray()).toEqual(INITIAL_GAPSET);
});
it('should remove a single gap if both keys overlap', () => {
const nRemoved = gapSet.removeOverlappingGaps(['bat', 'bat\0v100']);
expect(nRemoved).toEqual(1);
expect(gapSet.toArray()).toEqual([
// removed: { firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
it('should remove a single gap if min key overlaps with first key of first gap', () => {
const nRemoved = gapSet.removeOverlappingGaps(['bar\0v100', 'bar']);
expect(nRemoved).toEqual(1);
expect(gapSet.toArray()).toEqual([
// removed: { firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
it('should remove a single gap if max key overlaps with first key of first gap', () => {
const nRemoved = gapSet.removeOverlappingGaps(['ape', 'bar']);
expect(nRemoved).toEqual(1);
expect(gapSet.toArray()).toEqual([
// removed: { firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
it('should not remove any gap if both keys straddle an existing gap without overlap',
() => {
const nRemoved = gapSet.removeOverlappingGaps(['cow', 'ape']);
expect(nRemoved).toEqual(0);
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
it('should remove the two last gaps in chained gaps if last gap bounds match ' +
'the two keys', () => {
const nRemoved = gapSetWithChain.removeOverlappingGaps(['yak', 'rat']);
expect(nRemoved).toEqual(2);
expect(gapSetWithChain.toArray()).toEqual([
{ firstKey: 'ape', lastKey: 'ape', weight: 1 },
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'baz', lastKey: 'qux', weight: 15 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
// removed: { firstKey: 'quz', lastKey: 'rat', weight: 25 },
// removed: { firstKey: 'rat', lastKey: 'yak', weight: 30 },
]);
});
it('should remove first and last gap in chained gaps if their bounds match ' +
'the two keys', () => {
const nRemoved = gapSetWithChain.removeOverlappingGaps(['yak', 'bar']);
expect(nRemoved).toEqual(2);
expect(gapSetWithChain.toArray()).toEqual([
{ firstKey: 'ape', lastKey: 'ape', weight: 1 },
// removed: { firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'baz', lastKey: 'qux', weight: 15 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
{ firstKey: 'quz', lastKey: 'rat', weight: 25 },
// removed: { firstKey: 'rat', lastKey: 'yak', weight: 30 },
]);
});
});
describe('with an array of three keys as parameter', () => {
it('should remove a single gap if only median key overlaps with gap', () => {
const nRemoved = gapSet.removeOverlappingGaps(['ape', 'bat', 'cow']);
expect(nRemoved).toEqual(1);
expect(gapSet.toArray()).toEqual([
// removed: { firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
it('should remove a single-key gap and two contiguous chained gaps each overlapping' +
'with one key', () => {
const nRemoved = gapSetWithChain.removeOverlappingGaps(['ape', 'bat', 'cow']);
expect(nRemoved).toEqual(3);
expect(gapSetWithChain.toArray()).toEqual([
// removed: { firstKey: 'ape', lastKey: 'ape', weight: 1 },
// removed: { firstKey: 'bar', lastKey: 'baz', weight: 10 },
// removed: { firstKey: 'baz', lastKey: 'qux', weight: 15 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
{ firstKey: 'quz', lastKey: 'rat', weight: 25 },
{ firstKey: 'rat', lastKey: 'yak', weight: 30 },
]);
});
it('should not remove any gap if all keys are intermingled but do not overlap', () => {
const nRemoved = gapSet.removeOverlappingGaps(['ape', 'rat', 'cow']);
expect(nRemoved).toEqual(0);
expect(gapSet.toArray()).toEqual([
{ firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
it('should remove three discontiguous chained gaps each overlapping with one key', () => {
const nRemoved = gapSetWithChain.removeOverlappingGaps(['bat', 'quxxx', 'tiger']);
expect(nRemoved).toEqual(3);
expect(gapSetWithChain.toArray()).toEqual([
{ firstKey: 'ape', lastKey: 'ape', weight: 1 },
// removed: { firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'baz', lastKey: 'qux', weight: 15 },
// removed: { firstKey: 'qux', lastKey: 'quz', weight: 20 },
{ firstKey: 'quz', lastKey: 'rat', weight: 25 },
// { firstKey: 'rat', lastKey: 'yak', weight: 30 },
]);
});
});
describe('with a OrderedSet of three keys as parameter', () => {
it('should remove a single gap if only median key overlaps with gap', () => {
const nRemoved = gapSet.removeOverlappingGaps(
new OrderedSet(['ape', 'bat', 'cow']));
expect(nRemoved).toEqual(1);
expect(gapSet.toArray()).toEqual([
// removed: { firstKey: 'bar', lastKey: 'baz', weight: 10 },
{ firstKey: 'qux', lastKey: 'quz', weight: 20 },
]);
});
});
// this helper checks that:
// - the gaps not overlapping with any key are still present in newGapsArray
// - and the gaps overlapping with at least one key have been removed from oldGapsArray
// NOTE: It uses a sorted list of keys for efficiency, otherwise it would require
// O(n^2) compute time which would be expensive with 50K keys.
function checkOverlapInvariant(sortedKeys, oldGapsArray, newGapsArray) {
let oldGapIdx = 0;
let newGapIdx = 0;
for (const key of sortedKeys) {
// for all gaps not overlapping with any key in 'sortedKeys',
// check that they are still in 'newGapsArray'
while (oldGapIdx < oldGapsArray.length &&
oldGapsArray[oldGapIdx].lastKey < key) {
expect(oldGapsArray[oldGapIdx]).toEqual(newGapsArray[newGapIdx]);
oldGapIdx += 1;
newGapIdx += 1;
}
// for the gap(s) overlapping with the current key,
// check that they have been removed from 'newGapsArray'
while (oldGapIdx < oldGapsArray.length &&
oldGapsArray[oldGapIdx].firstKey <= key) {
if (newGapIdx < newGapsArray.length) {
expect(oldGapsArray[oldGapIdx]).not.toEqual(newGapsArray[newGapIdx]);
}
++oldGapIdx;
}
}
// check the range after the last key in 'sortedKeys'
while (oldGapIdx < oldGapsArray.length) {
expect(oldGapsArray[oldGapIdx]).toEqual(newGapsArray[newGapIdx]);
oldGapIdx += 1;
newGapIdx += 1;
}
// check that no extra range is in newGapsArray
expect(newGapIdx).toEqual(newGapsArray.length);
}
[false, true].forEach(chained => {
describe(`with 10K random ${chained ? 'chained' : 'unchained'} gaps`, () => {
let largeGapsArray;
let largeGapSet;
beforeEach(() => {
largeGapsArray = chained ?
genRandomChainedGaps(10000) :
genRandomUnchainedGaps(10000);
largeGapSet = GapSet.createFromArray(largeGapsArray, 100);
});
[{
desc: 'equal to their first key',
getGapKey: gap => gap.firstKey,
}, {
desc: 'equal to their last key',
getGapKey: gap => gap.lastKey,
}, {
desc: 'neither their first nor last key',
getGapKey: gap => `${gap.firstKey}/foo`,
}].forEach(testCase => {
it(`should remove the overlapping gap(s) with one key ${testCase.desc}`, () => {
const gapIndex = 5000;
const gap = largeGapsArray[gapIndex];
const overlappingKey = testCase.getGapKey(gap);
const nRemoved = largeGapSet.removeOverlappingGaps([overlappingKey]);
let firstRemovedGapIndex, lastRemovedGapIndex;
if (chained && overlappingKey === gap.firstKey) {
expect(nRemoved).toEqual(2);
[firstRemovedGapIndex, lastRemovedGapIndex] = [4999, 5000];
} else if (chained && overlappingKey === gap.lastKey) {
expect(nRemoved).toEqual(2);
[firstRemovedGapIndex, lastRemovedGapIndex] = [5000, 5001];
} else {
expect(nRemoved).toEqual(1);
[firstRemovedGapIndex, lastRemovedGapIndex] = [5000, 5000];
}
const expectedGaps = [
...largeGapsArray.slice(0, firstRemovedGapIndex),
...largeGapsArray.slice(lastRemovedGapIndex + 1)
];
const newGaps = largeGapSet.toArray();
expect(newGaps).toEqual(expectedGaps);
});
it(`should remove all gaps when they all overlap with one key ${testCase.desc}`,
() => {
// simulate a scenario made of 200 batches of 50 operations, each with
// random keys scattered across all gaps that each overlaps a distinct gap
// (supposedly a worst-case performance scenario for such batch sizes)
const overlappingKeys = largeGapsArray.map(testCase.getGapKey);
shuffleArray(overlappingKeys);
for (let i = 0; i < overlappingKeys.length; i += 50) {
const nRemoved = largeGapSet.removeOverlappingGaps(
overlappingKeys.slice(i, i + 50));
// with unchained gaps, we expect to have removed exactly
// 50 gaps (the size of 'overlappingKeys').
if (!chained) {
expect(nRemoved).toEqual(50);
}
}
const newGaps = largeGapSet.toArray();
expect(newGaps).toEqual([]);
});
});
it('should remove only and all overlapping gaps with 50K randomized keys', () => {
const randomizedKeys = new Array(50000).fill(undefined).map(
() => genRandomKey()
);
for (let i = 0; i < randomizedKeys.length; i += 50) {
largeGapSet.removeOverlappingGaps(
randomizedKeys.slice(i, i + 50));
}
const newGaps = largeGapSet.toArray();
randomizedKeys.sort();
checkOverlapInvariant(randomizedKeys, largeGapsArray, newGaps);
});
});
});
});
describe('GapSet::_coalesceGapChain()', () => {
afterEach(() => {
// check that the gap sets were not modified by the operation
expect(gapSet.toArray()).toEqual(INITIAL_GAPSET);
expect(gapSetWithChain.toArray()).toEqual(INITIAL_GAPSET_WITH_CHAIN);
});
it('should not coalesce if gaps are not chained', async () => {
const gap = { firstKey: 'bar', lastKey: 'baz', weight: 10 };
const coalescedGap = await gapSet._coalesceGapChain(gap);
expect(coalescedGap).toEqual({ firstKey: 'bar', lastKey: 'baz', weight: 10 });
});
it('should coalesce one chained gap', async () => {
const gap = { firstKey: 'quz', lastKey: 'rat', weight: 25 };
const coalescedGap = await gapSetWithChain._coalesceGapChain(gap);
expect(coalescedGap).toEqual({ firstKey: 'quz', lastKey: 'yak', weight: 55 });
});
it('should coalesce a chain of five gaps', async () => {
const gap = { firstKey: 'bar', lastKey: 'baz', weight: 10 };
const coalescedGap = await gapSetWithChain._coalesceGapChain(gap);
expect(coalescedGap).toEqual({ firstKey: 'bar', lastKey: 'yak', weight: 100 });
});
it('should coalesce a chain of one thousand gaps', async () => {
const getKey = i => `000${i}`.slice(-4);
const thousandGapsArray = new Array(1000).fill(undefined).map(
(_, i) => ({ firstKey: getKey(i), lastKey: getKey(i + 1), weight: 10 })
);
const thousandGapsSet = GapSet.createFromArray(thousandGapsArray, 100);
const gap = { firstKey: '0000', lastKey: '0001', weight: 10 };
const coalescedGap = await thousandGapsSet._coalesceGapChain(gap);
expect(coalescedGap).toEqual({ firstKey: '0000', lastKey: '1000', weight: 10000 });
});
it('should coalesce a single-key gap', async () => {
const singleKeyGapSet = GapSet.createFromArray([
{ firstKey: '0000', lastKey: '0000', weight: 1 },
], 100);
const gap = { firstKey: '0000', lastKey: '0000', weight: 1 };
const coalescedGap = await singleKeyGapSet._coalesceGapChain(gap);
expect(coalescedGap).toEqual({ firstKey: '0000', lastKey: '0000', weight: 1 });
});
it('should coalesce a chain of two gaps ending with a single-key gap', async () => {
const singleKeyGapSet = GapSet.createFromArray([
{ firstKey: '0000', lastKey: '0003', weight: 9 },
{ firstKey: '0003', lastKey: '0003', weight: 1 },
], 100);
const gap = { firstKey: '0000', lastKey: '0003', weight: 9 };
const coalescedGap = await singleKeyGapSet._coalesceGapChain(gap);
expect(coalescedGap).toEqual({ firstKey: '0000', lastKey: '0003', weight: 9 });
});
});
describe('GapSet::lookupGap()', () => {
afterEach(() => {
// check that the gap sets were not modified by the operation
expect(gapSet.toArray()).toEqual(INITIAL_GAPSET);
expect(gapSetWithChain.toArray()).toEqual(INITIAL_GAPSET_WITH_CHAIN);
});
it('should return null with empty cache', async () => {
const emptyGapSet = new GapSet(100);
const gap = await emptyGapSet.lookupGap('cat', 'dog');
expect(gap).toBeNull();
});
it('should return null if no gap overlaps [minKey, maxKey]', async () => {
const gap = await gapSet.lookupGap('cat', 'dog');
expect(gap).toBeNull();
});
it('should return the first gap that overlaps if all gaps overlap', async () => {
const gap = await gapSet.lookupGap('ape', 'zoo');
expect(gap).toEqual({ firstKey: 'bar', lastKey: 'baz', weight: 10 });
});
it('should return an existing gap that contains [minKey, maxKey]', async () => {
const gap1 = await gapSet.lookupGap('bat', 'bay');
expect(gap1).toEqual({ firstKey: 'bar', lastKey: 'baz', weight: 10 });
const gap2 = await gapSet.lookupGap('quxxx', 'quy');
expect(gap2).toEqual({ firstKey: 'qux', lastKey: 'quz', weight: 20 });
});
it('should return an existing gap that overlaps with minKey but not maxKey', async () => {
const gap = await gapSet.lookupGap('ape', 'bat');
expect(gap).toEqual({ firstKey: 'bar', lastKey: 'baz', weight: 10 });
});
it('should return an existing gap that overlaps just with minKey when no maxKey is provided',
async () => {
const gap = await gapSet.lookupGap('ape');
expect(gap).toEqual({ firstKey: 'bar', lastKey: 'baz', weight: 10 });
});
it('should return an existing gap that overlaps with maxKey but not minKey', async () => {
const gap = await gapSet.lookupGap('bat', 'cat');
expect(gap).toEqual({ firstKey: 'bar', lastKey: 'baz', weight: 10 });
});
it('should return an existing gap that is contained in [minKey, maxKey] strictly', async () => {
const gap = await gapSet.lookupGap('dog', 'rat');
expect(gap).toEqual({ firstKey: 'qux', lastKey: 'quz', weight: 20 });
});
it('should return a coalesced gap from chained gaps that fully overlaps [minKey, maxKey]', async () => {
const gap = await gapSetWithChain.lookupGap('bat', 'zoo');
expect(gap).toEqual({ firstKey: 'bar', lastKey: 'yak', weight: 100 });
});
it('should return a coalesced gap from chained gaps that contain [minKey, maxKey] strictly',
async () => {
const gap = await gapSetWithChain.lookupGap('bog', 'dog');
expect(gap).toEqual({ firstKey: 'baz', lastKey: 'yak', weight: 90 });
});
});
});

View File

@ -727,7 +727,7 @@ function getTestListing(mdParams, data, vFormat) {
}); });
} }
assert.strictEqual(delimiter.skipping(), assert.strictEqual(delimiter.skipping(),
`${vFormat === 'v1' ? DbPrefixes.Master : ''}foo/`); `${vFormat === 'v1' ? DbPrefixes.Master : ''}foo0`);
}); });
tests.forEach(test => { tests.forEach(test => {

View File

@ -0,0 +1,430 @@
'use strict'; // eslint-disable-line strict
const assert = require('assert');
const DelimiterCurrent =
require('../../../../lib/algos/list/delimiterCurrent').DelimiterCurrent;
const {
FILTER_ACCEPT,
FILTER_END,
} = require('../../../../lib/algos/list/tools');
const VSConst =
require('../../../../lib/versioning/constants').VersioningConstants;
const { DbPrefixes } = VSConst;
const fakeLogger = {
trace: () => {},
debug: () => {},
info: () => {},
warn: () => {},
error: () => {},
fatal: () => {},
};
function getListingKey(key, vFormat) {
if (vFormat === 'v0') {
return key;
}
if (vFormat === 'v1') {
return `${DbPrefixes.Master}${key}`;
}
return assert.fail(`bad vFormat ${vFormat}`);
}
['v0', 'v1'].forEach(v => {
describe(`DelimiterCurrent with ${v} bucket format`, () => {
it('should return expected metadata parameters', () => {
const prefix = 'pre';
const marker = 'premark';
const beforeDate = '1970-01-01T00:00:00.005Z';
const excludedDataStoreName = 'location1';
const maxScannedLifecycleListingEntries = 2;
const delimiter = new DelimiterCurrent({
prefix,
marker,
beforeDate,
excludedDataStoreName,
maxScannedLifecycleListingEntries,
}, fakeLogger, v);
const expectedParams = {
dataStoreName: {
ne: excludedDataStoreName,
},
lastModified: {
lt: beforeDate,
},
gt: getListingKey('premark', v),
lt: getListingKey('prf', v),
};
assert.deepStrictEqual(delimiter.genMDParams(), expectedParams);
assert.strictEqual(delimiter.maxScannedLifecycleListingEntries, 2);
});
it('should accept entry starting with prefix', () => {
const delimiter = new DelimiterCurrent({ prefix: 'prefix' }, fakeLogger, v);
const masterKey = 'prefix1';
const date1 = '1970-01-01T00:00:00.001Z';
const value1 = `{"last-modified": "${date1}"}`;
assert.strictEqual(delimiter.filter({ key: getListingKey(masterKey, v), value: value1 }), FILTER_ACCEPT);
const expectedResult = {
Contents: [
{
key: masterKey,
value: value1,
},
],
IsTruncated: false,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should accept a master and return it', () => {
const delimiter = new DelimiterCurrent({ }, fakeLogger, v);
const masterKey = 'key';
const date1 = '1970-01-01T00:00:00.001Z';
const value1 = `{"last-modified": "${date1}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey, v),
value: value1,
}), FILTER_ACCEPT);
const expectedResult = {
Contents: [
{
key: masterKey,
value: value1,
},
],
IsTruncated: false,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should accept the first master and return the truncated content', () => {
const delimiter = new DelimiterCurrent({ maxKeys: 1 }, fakeLogger, v);
const masterKey1 = 'key1';
const date1 = '1970-01-01T00:00:00.001Z';
const value1 = `{"last-modified": "${date1}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey1, v),
value: value1,
}), FILTER_ACCEPT);
const masterKey2 = 'key2';
const date2 = '1970-01-01T00:00:00.000Z';
const value2 = `{"last-modified": "${date2}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey2, v),
value: value2,
}), FILTER_END);
const expectedResult = {
Contents: [
{
key: masterKey1,
value: value1,
},
],
NextMarker: masterKey1,
IsTruncated: true,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should return the object created before beforeDate', () => {
const beforeDate = '1970-01-01T00:00:00.003Z';
const delimiter = new DelimiterCurrent({ beforeDate }, fakeLogger, v);
const masterKey1 = 'key1';
const date1 = '1970-01-01T00:00:00.004Z';
const value1 = `{"last-modified": "${date1}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey1, v),
value: value1,
}), FILTER_ACCEPT);
const masterKey2 = 'key2';
const date2 = '1970-01-01T00:00:00.000Z';
const value2 = `{"last-modified": "${date2}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey2, v),
value: value2,
}), FILTER_ACCEPT);
const expectedResult = {
Contents: [
{
key: masterKey2,
value: value2,
},
],
IsTruncated: false,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should return an empty list if last-modified is an empty string', () => {
const beforeDate = '1970-01-01T00:00:00.003Z';
const delimiter = new DelimiterCurrent({ beforeDate }, fakeLogger, v);
const masterKey0 = 'key0';
const value0 = '{"last-modified": ""}';
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey0, v),
value: value0,
}), FILTER_ACCEPT);
const expectedResult = {
Contents: [],
IsTruncated: false,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should return an empty list if last-modified is undefined', () => {
const beforeDate = '1970-01-01T00:00:00.003Z';
const delimiter = new DelimiterCurrent({ beforeDate }, fakeLogger, v);
const masterKey0 = 'key0';
const value0 = '{}';
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey0, v),
value: value0,
}), FILTER_ACCEPT);
const expectedResult = {
Contents: [],
IsTruncated: false,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should return the object with dataStore name that does not match', () => {
const beforeDate = '1970-01-01T00:00:00.005Z';
const excludedDataStoreName = 'location-excluded';
const delimiter = new DelimiterCurrent({ beforeDate, excludedDataStoreName }, fakeLogger, v);
const masterKey1 = 'key1';
const date1 = '1970-01-01T00:00:00.004Z';
const value1 = `{"last-modified": "${date1}", "dataStoreName": "valid"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey1, v),
value: value1,
}), FILTER_ACCEPT);
const masterKey2 = 'key2';
const date2 = '1970-01-01T00:00:00.000Z';
const value2 = `{"last-modified": "${date2}", "dataStoreName": "${excludedDataStoreName}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey2, v),
value: value2,
}), FILTER_ACCEPT);
const expectedResult = {
Contents: [
{
key: masterKey1,
value: value1,
},
],
IsTruncated: false,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should stop fetching entries if the max keys are reached and return the accurate next marker', () => {
const beforeDate = '1970-01-01T00:00:00.005Z';
const excludedDataStoreName = 'location-excluded';
const delimiter = new DelimiterCurrent({ beforeDate, excludedDataStoreName, maxKeys: 1 }, fakeLogger, v);
const masterKey1 = 'key1';
const date1 = '1970-01-01T00:00:00.004Z';
const value1 = `{"last-modified": "${date1}", "dataStoreName": "valid"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey1, v),
value: value1,
}), FILTER_ACCEPT);
const masterKey2 = 'key2';
const date2 = '1970-01-01T00:00:00.000Z';
const value2 = `{"last-modified": "${date2}", "dataStoreName": "${excludedDataStoreName}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey2, v),
value: value2,
}), FILTER_END);
const expectedResult = {
Contents: [
{
key: masterKey1,
value: value1,
},
],
IsTruncated: true,
NextMarker: masterKey1,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should return the object created before beforeDate and with dataStore name that does not match', () => {
const beforeDate = '1970-01-01T00:00:00.003Z';
const excludedDataStoreName = 'location-excluded';
const delimiter = new DelimiterCurrent({ beforeDate, excludedDataStoreName }, fakeLogger, v);
const masterKey1 = 'key1';
const date1 = '1970-01-01T00:00:00.004Z';
const value1 = `{"last-modified": "${date1}", "dataStoreName": "valid"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey1, v),
value: value1,
}), FILTER_ACCEPT);
const masterKey2 = 'key2';
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"last-modified": "${date2}", "dataStoreName": "valid"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey2, v),
value: value2,
}), FILTER_ACCEPT);
const masterKey3 = 'key3';
const date3 = '1970-01-01T00:00:00.000Z';
const value3 = `{"last-modified": "${date3}", "dataStoreName": "${excludedDataStoreName}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey3, v),
value: value3,
}), FILTER_ACCEPT);
const expectedResult = {
Contents: [
{
key: masterKey2,
value: value2,
},
],
IsTruncated: false,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should return the objects pushed before max scanned entries value is reached', () => {
const beforeDate = '1970-01-01T00:00:00.003Z';
const maxScannedLifecycleListingEntries = 2;
const delimiter = new DelimiterCurrent({ beforeDate, maxScannedLifecycleListingEntries }, fakeLogger, v);
const masterKey1 = 'key1';
const date1 = '1970-01-01T00:00:00.000Z';
const value1 = `{"last-modified": "${date1}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey1, v),
value: value1,
}), FILTER_ACCEPT);
const masterKey2 = 'key2';
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"last-modified": "${date2}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey2, v),
value: value2,
}), FILTER_ACCEPT);
const masterKey3 = 'key3';
const date3 = '1970-01-01T00:00:00.002Z';
const value3 = `{"last-modified": "${date3}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey3, v),
value: value3,
}), FILTER_END);
const expectedResult = {
Contents: [
{
key: masterKey1,
value: value1,
},
{
key: masterKey2,
value: value2,
},
],
NextMarker: masterKey2,
IsTruncated: true,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should return empty content after max scanned entries value is reached', () => {
const beforeDate = '1970-01-01T00:00:00.003Z';
const maxScannedLifecycleListingEntries = 2;
const delimiter = new DelimiterCurrent({ beforeDate, maxScannedLifecycleListingEntries }, fakeLogger, v);
const masterKey1 = 'key1';
const date1 = '1970-01-01T00:00:00.004Z';
const value1 = `{"last-modified": "${date1}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey1, v),
value: value1,
}), FILTER_ACCEPT);
const masterKey2 = 'key2';
const date2 = '1970-01-01T00:00:00.005Z';
const value2 = `{"last-modified": "${date2}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey2, v),
value: value2,
}), FILTER_ACCEPT);
const masterKey3 = 'key3';
const date3 = '1970-01-01T00:00:00.006Z';
const value3 = `{"last-modified": "${date3}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(masterKey3, v),
value: value3,
}), FILTER_END);
const expectedResult = {
Contents: [],
NextMarker: masterKey2,
IsTruncated: true,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
});
});

View File

@ -2,8 +2,12 @@
const assert = require('assert'); const assert = require('assert');
const DelimiterMaster = import {
require('../../../../lib/algos/list/delimiterMaster').DelimiterMaster; DelimiterMaster,
DelimiterMasterFilterStateId,
GapCachingState,
GapBuildingState,
} from '../../../../lib/algos/list/delimiterMaster';
const { const {
FILTER_ACCEPT, FILTER_ACCEPT,
FILTER_SKIP, FILTER_SKIP,
@ -11,6 +15,8 @@ const {
SKIP_NONE, SKIP_NONE,
inc, inc,
} = require('../../../../lib/algos/list/tools'); } = require('../../../../lib/algos/list/tools');
import { default as GapSet, GapSetEntry } from '../../../../lib/algos/cache/GapSet';
import { GapCacheInterface } from '../../../../lib/algos/cache/GapCache';
const VSConst = const VSConst =
require('../../../../lib/versioning/constants').VersioningConstants; require('../../../../lib/versioning/constants').VersioningConstants;
const Version = require('../../../../lib/versioning/Version').Version; const Version = require('../../../../lib/versioning/Version').Version;
@ -167,7 +173,7 @@ function getListingKey(key, vFormat) {
}); });
if (vFormat === 'v0') { if (vFormat === 'v0') {
it('should return <key><VersionIdSeparator> for DelimiterMaster when ' + it('skipping() should return <key>inc(<VersionIdSeparator>) for DelimiterMaster when ' +
'NextMarker is set and there is a delimiter', () => { 'NextMarker is set and there is a delimiter', () => {
const key = 'key'; const key = 'key';
const delimiter = new DelimiterMaster( const delimiter = new DelimiterMaster(
@ -178,14 +184,10 @@ function getListingKey(key, vFormat) {
const listingKey = getListingKey(key, vFormat); const listingKey = getListingKey(key, vFormat);
delimiter.filter({ key: listingKey, value: '' }); delimiter.filter({ key: listingKey, value: '' });
assert.strictEqual(delimiter.nextMarker, key); assert.strictEqual(delimiter.nextMarker, key);
assert.strictEqual(delimiter.skipping(), `${listingKey}${inc(VID_SEP)}`);
/* With a delimiter skipping should return previous key + VID_SEP
* (except when a delimiter is set and the NextMarker ends with the
* delimiter) . */
assert.strictEqual(delimiter.skipping(), listingKey + VID_SEP);
}); });
it('should return <key><VersionIdSeparator> for DelimiterMaster when ' + it('skipping() should return <key>inc(<VersionIdSeparator>) for DelimiterMaster when ' +
'NextContinuationToken is set and there is a delimiter', () => { 'NextContinuationToken is set and there is a delimiter', () => {
const key = 'key'; const key = 'key';
const delimiter = new DelimiterMaster( const delimiter = new DelimiterMaster(
@ -197,7 +199,7 @@ function getListingKey(key, vFormat) {
delimiter.filter({ key: listingKey, value: '' }); delimiter.filter({ key: listingKey, value: '' });
assert.strictEqual(delimiter.nextMarker, key); assert.strictEqual(delimiter.nextMarker, key);
assert.strictEqual(delimiter.skipping(), listingKey + VID_SEP); assert.strictEqual(delimiter.skipping(), `${listingKey}${inc(VID_SEP)}`);
}); });
it('should accept a PHD version as first input', () => { it('should accept a PHD version as first input', () => {
@ -446,7 +448,7 @@ function getListingKey(key, vFormat) {
}), }),
FILTER_SKIP); FILTER_SKIP);
// ...it should skip the whole replay prefix // ...it should skip the whole replay prefix
assert.strictEqual(delimiter.skipping(), DbPrefixes.Replay); assert.strictEqual(delimiter.skipping(), inc(DbPrefixes.Replay));
// simulate a listing that reaches regular object keys // simulate a listing that reaches regular object keys
// beyond the replay prefix, ... // beyond the replay prefix, ...
@ -461,8 +463,8 @@ function getListingKey(key, vFormat) {
// as usual // as usual
assert.strictEqual(delimiter.skipping(), assert.strictEqual(delimiter.skipping(),
delimiterChar ? delimiterChar ?
`${inc(DbPrefixes.Replay)}foo/` : `${inc(DbPrefixes.Replay)}foo0` :
`${inc(DbPrefixes.Replay)}foo/bar${VID_SEP}`); `${inc(DbPrefixes.Replay)}foo/bar${inc(VID_SEP)}`);
}); });
}); });
} }
@ -488,12 +490,12 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
key: `foo/deleted${VID_SEP}v1`, key: `foo/deleted${VID_SEP}v1`,
isDeleteMarker: true, isDeleteMarker: true,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/deleted${VID_SEP}`, skipping: `foo/deleted${inc(VID_SEP)}`,
}, },
{ {
key: `foo/deleted${VID_SEP}v2`, key: `foo/deleted${VID_SEP}v2`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/deleted${VID_SEP}`, skipping: `foo/deleted${inc(VID_SEP)}`,
}, },
{ {
key: 'foo/notdeleted', key: 'foo/notdeleted',
@ -502,7 +504,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
{ {
key: `foo/notdeleted${VID_SEP}v1`, key: `foo/notdeleted${VID_SEP}v1`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/notdeleted${VID_SEP}`, skipping: `foo/notdeleted${inc(VID_SEP)}`,
}, },
{ {
key: 'foo/subprefix/key-1', key: 'foo/subprefix/key-1',
@ -511,7 +513,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
{ {
key: `foo/subprefix/key-1${VID_SEP}v1`, key: `foo/subprefix/key-1${VID_SEP}v1`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/subprefix/key-1${VID_SEP}`, skipping: `foo/subprefix/key-1${inc(VID_SEP)}`,
}, },
], ],
result: { result: {
@ -542,7 +544,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
key: `foo/01${VID_SEP}v1`, key: `foo/01${VID_SEP}v1`,
isDeleteMarker: true, isDeleteMarker: true,
res: FILTER_SKIP, // versions get skipped after master res: FILTER_SKIP, // versions get skipped after master
skipping: `foo/01${VID_SEP}`, skipping: `foo/01${inc(VID_SEP)}`,
}, },
{ {
key: 'foo/02', key: 'foo/02',
@ -553,7 +555,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
key: `foo/02${VID_SEP}v1`, key: `foo/02${VID_SEP}v1`,
isDeleteMarker: true, isDeleteMarker: true,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/02${VID_SEP}`, skipping: `foo/02${inc(VID_SEP)}`,
}, },
{ {
key: 'foo/03', key: 'foo/03',
@ -562,7 +564,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
{ {
key: `foo/03${VID_SEP}v1`, key: `foo/03${VID_SEP}v1`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/03${VID_SEP}`, skipping: `foo/03${inc(VID_SEP)}`,
}, },
], ],
result: { result: {
@ -592,7 +594,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
key: `foo/bar/01${VID_SEP}v1`, key: `foo/bar/01${VID_SEP}v1`,
isDeleteMarker: true, isDeleteMarker: true,
res: FILTER_SKIP, // versions get skipped after master res: FILTER_SKIP, // versions get skipped after master
skipping: `foo/bar/01${VID_SEP}`, skipping: `foo/bar/01${inc(VID_SEP)}`,
}, },
{ {
key: 'foo/bar/02', key: 'foo/bar/02',
@ -603,12 +605,12 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
key: `foo/bar/02${VID_SEP}v1`, key: `foo/bar/02${VID_SEP}v1`,
isDeleteMarker: true, isDeleteMarker: true,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/bar/02${VID_SEP}`, skipping: `foo/bar/02${inc(VID_SEP)}`,
}, },
{ {
key: `foo/bar/02${VID_SEP}v2`, key: `foo/bar/02${VID_SEP}v2`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/bar/02${VID_SEP}`, skipping: `foo/bar/02${inc(VID_SEP)}`,
}, },
{ {
key: 'foo/bar/03', key: 'foo/bar/03',
@ -618,19 +620,19 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
key: `foo/bar/03${VID_SEP}v1`, key: `foo/bar/03${VID_SEP}v1`,
res: FILTER_SKIP, res: FILTER_SKIP,
// from now on, skip the 'foo/bar/' prefix because we have already seen it // from now on, skip the 'foo/bar/' prefix because we have already seen it
skipping: 'foo/bar/', skipping: 'foo/bar0',
}, },
{ {
key: 'foo/bar/04', key: 'foo/bar/04',
isDeleteMarker: true, isDeleteMarker: true,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: 'foo/bar/', skipping: 'foo/bar0',
}, },
{ {
key: `foo/bar/04${VID_SEP}v1`, key: `foo/bar/04${VID_SEP}v1`,
isDeleteMarker: true, isDeleteMarker: true,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: 'foo/bar/', skipping: 'foo/bar0',
}, },
{ {
key: 'foo/baz/01', key: 'foo/baz/01',
@ -640,7 +642,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
key: `foo/baz/01${VID_SEP}v1`, key: `foo/baz/01${VID_SEP}v1`,
res: FILTER_SKIP, res: FILTER_SKIP,
// skip the 'foo/baz/' prefix because we have already seen it // skip the 'foo/baz/' prefix because we have already seen it
skipping: 'foo/baz/', skipping: 'foo/baz0',
}, },
], ],
result: { result: {
@ -669,7 +671,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
{ {
key: `foo/${VID_SEP}v1`, key: `foo/${VID_SEP}v1`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/${VID_SEP}`, skipping: `foo/${inc(VID_SEP)}`,
}, },
{ {
key: 'foo/deleted', key: 'foo/deleted',
@ -680,12 +682,12 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
key: `foo/deleted${VID_SEP}v1`, key: `foo/deleted${VID_SEP}v1`,
isDeleteMarker: true, isDeleteMarker: true,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/deleted${VID_SEP}`, skipping: `foo/deleted${inc(VID_SEP)}`,
}, },
{ {
key: `foo/deleted${VID_SEP}v2`, key: `foo/deleted${VID_SEP}v2`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/deleted${VID_SEP}`, skipping: `foo/deleted${inc(VID_SEP)}`,
}, },
{ {
key: 'foo/notdeleted', key: 'foo/notdeleted',
@ -694,7 +696,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
{ {
key: `foo/notdeleted${VID_SEP}v1`, key: `foo/notdeleted${VID_SEP}v1`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/notdeleted${VID_SEP}`, skipping: `foo/notdeleted${inc(VID_SEP)}`,
}, },
{ {
key: 'foo/subprefix/key-1', key: 'foo/subprefix/key-1',
@ -703,17 +705,17 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
{ {
key: `foo/subprefix/key-1${VID_SEP}v1`, key: `foo/subprefix/key-1${VID_SEP}v1`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: 'foo/subprefix/', skipping: 'foo/subprefix0',
}, },
{ {
key: 'foo/subprefix/key-2', key: 'foo/subprefix/key-2',
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: 'foo/subprefix/', skipping: 'foo/subprefix0',
}, },
{ {
key: `foo/subprefix/key-2${VID_SEP}v1`, key: `foo/subprefix/key-2${VID_SEP}v1`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: 'foo/subprefix/', skipping: 'foo/subprefix0',
}, },
], ],
result: { result: {
@ -744,7 +746,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
{ {
key: `foo${VID_SEP}v1`, key: `foo${VID_SEP}v1`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo${VID_SEP}`, skipping: `foo${inc(VID_SEP)}`,
}, },
{ {
key: 'foo/deleted', key: 'foo/deleted',
@ -755,12 +757,12 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
key: `foo/deleted${VID_SEP}v1`, key: `foo/deleted${VID_SEP}v1`,
isDeleteMarker: true, isDeleteMarker: true,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/deleted${VID_SEP}`, skipping: `foo/deleted${inc(VID_SEP)}`,
}, },
{ {
key: `foo/deleted${VID_SEP}v2`, key: `foo/deleted${VID_SEP}v2`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/deleted${VID_SEP}`, skipping: `foo/deleted${inc(VID_SEP)}`,
}, },
{ {
key: 'foo/notdeleted', key: 'foo/notdeleted',
@ -769,17 +771,17 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
{ {
key: `foo/notdeleted${VID_SEP}v1`, key: `foo/notdeleted${VID_SEP}v1`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: 'foo/', skipping: 'foo0',
}, },
{ {
key: 'foo/subprefix/key-1', key: 'foo/subprefix/key-1',
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: 'foo/', skipping: 'foo0',
}, },
{ {
key: `foo/subprefix/key-1${VID_SEP}v1`, key: `foo/subprefix/key-1${VID_SEP}v1`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: 'foo/', skipping: 'foo0',
}, },
], ],
result: { result: {
@ -811,7 +813,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
key: `foo/${VID_SEP}v1`, key: `foo/${VID_SEP}v1`,
isDeleteMarker: true, isDeleteMarker: true,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/${VID_SEP}`, skipping: `foo/${inc(VID_SEP)}`,
}, },
{ {
key: 'foo/subprefix', key: 'foo/subprefix',
@ -824,7 +826,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
{ {
key: 'foo/subprefix/02', key: 'foo/subprefix/02',
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: 'foo/subprefix/', // already added to common prefix skipping: 'foo/subprefix0', // already added to common prefix
}, },
], ],
result: { result: {
@ -859,7 +861,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
{ {
key: `foo/01${VID_SEP}v2`, key: `foo/01${VID_SEP}v2`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/01${VID_SEP}`, skipping: `foo/01${inc(VID_SEP)}`,
}, },
{ {
key: 'foo/02', key: 'foo/02',
@ -870,12 +872,12 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
key: `foo/02${VID_SEP}v1`, key: `foo/02${VID_SEP}v1`,
isDeleteMarker: true, isDeleteMarker: true,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/02${VID_SEP}`, skipping: `foo/02${inc(VID_SEP)}`,
}, },
{ {
key: `foo/02${VID_SEP}v2`, key: `foo/02${VID_SEP}v2`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `foo/02${VID_SEP}`, skipping: `foo/02${inc(VID_SEP)}`,
}, },
], ],
result: { result: {
@ -950,7 +952,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
{ {
key: `${DbPrefixes.Master}foo/subprefix/key-2`, key: `${DbPrefixes.Master}foo/subprefix/key-2`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `${DbPrefixes.Master}foo/subprefix/`, skipping: `${DbPrefixes.Master}foo/subprefix0`,
}, },
], ],
result: { result: {
@ -985,7 +987,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
{ {
key: `${DbPrefixes.Master}foo/subprefix/key-1`, key: `${DbPrefixes.Master}foo/subprefix/key-1`,
res: FILTER_SKIP, res: FILTER_SKIP,
skipping: `${DbPrefixes.Master}foo/`, skipping: `${DbPrefixes.Master}foo0`,
}, },
], ],
result: { result: {
@ -1005,7 +1007,7 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
it(`vFormat=${testCase.vFormat}: ${testCase.desc}`, () => { it(`vFormat=${testCase.vFormat}: ${testCase.desc}`, () => {
const delimiter = new DelimiterMaster(testCase.params, fakeLogger, testCase.vFormat); const delimiter = new DelimiterMaster(testCase.params, fakeLogger, testCase.vFormat);
const resultEntries = testCase.entries.map(testEntry => { const resultEntries = testCase.entries.map(testEntry => {
const entry = { const entry: any = {
key: testEntry.key, key: testEntry.key,
}; };
if (testEntry.isDeleteMarker) { if (testEntry.isDeleteMarker) {
@ -1029,3 +1031,567 @@ describe('DelimiterMaster listing algorithm: sequence of filter() scenarii', ()
}); });
}); });
}); });
/**
* Test class that provides a GapCache-compatible interface via a
* GapSet implementation, i.e. without introducing a delay to expose
* gaps like the GapCache class does, so tests can check more easily
* which gaps have been updated.
*/
class GapCacheAsSet extends GapSet implements GapCacheInterface {
exposureDelayMs: number;
constructor(maxGapWeight: number) {
super(maxGapWeight);
this.exposureDelayMs = 1000;
}
static createFromArray(gaps: GapSetEntry[], maxWeight: number): GapCacheAsSet {
const gs = new GapCacheAsSet(maxWeight);
for (const gap of gaps) {
gs._gaps.insert(gap);
}
return gs;
}
get maxGapWeight(): number {
return super.maxWeight;
}
}
type FilterEntriesResumeState = {
i: number,
version: number,
};
/**
* Convenience test helper to build listing entries and pass them to
* the DelimiterMaster.filter() function in order, and checks the
* return code. It is also useful to check the state of the gap cache
* afterwards.
*
* The first object key is "pre/0001" and is incremented on each master key.
*
* The current object version is "v100" and the version is then incremented
* for each noncurrent version ("v101" etc.).
*
* @param {DelimiterMaster} listing - listing algorithm instance
* @param {string} pattern - pattern of keys to create:
* - an upper-case letter is a master key
* - a lower-case letter is a version key
* - a 'd' (or 'D') letter is a delete marker
* - any other letter (e.g. 'v' or 'V') is a regular version
* - space characters ' ' are allowed and must be matched by
* a space character at the same position in 'expectedCodes'
* @param {string} expectedCodes - string of expected codes from filter()
* matching each entry from 'pattern':
* - 'a' stands for FILTER_ACCEPT
* - 's' stands for FILTER_SKIP
* - 'e' stands for FILTER_END
* - ' ' must be matched by a space character in 'pattern'
* @return {FilterEntriesResumeState} - a state that can be passed in
* the next call as 'resumeFromState' to resume filtering the next
* keys
*/
function filterEntries(
listing: DelimiterMaster,
pattern: string,
expectedCodes: string,
resumeFromState?: FilterEntriesResumeState,
): FilterEntriesResumeState {
const ExpectedCodeMap: string[] = [];
ExpectedCodeMap[FILTER_ACCEPT] = 'a';
ExpectedCodeMap[FILTER_SKIP] = 's';
ExpectedCodeMap[FILTER_END] = 'e';
let { i, version } = resumeFromState || { i: 0, version: 100 };
const obtainedCodes = pattern.split('').map(p => {
if (p === ' ') {
return ' ';
}
if (p.toUpperCase() === p) {
// master key
i += 1;
version = 100;
}
const keyId = `0000${i}`.slice(-4);
const key = `pre/${keyId}`;
const md: any = ('Dd'.includes(p)) ? { isDeleteMarker: true } : {};
md.versionId = `v${version}`;
const value = JSON.stringify(md);
const entry = (p.toUpperCase() === p) ? { key, value } : { key: `${key}\0v${version}`, value };
const ret = listing.filter(entry);
if (p.toLowerCase() === p) {
// version key
version += 1;
}
return ExpectedCodeMap[<number> <unknown> ret];
}).join('');
expect(obtainedCodes).toEqual(expectedCodes);
return { i, version };
}
describe('DelimiterMaster listing algorithm: gap caching and lookup', () => {
it('should not cache a gap of weight smaller than minGapWeight', () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapCache = new GapCacheAsSet(100);
listing.refreshGapCache(gapCache, 7); // minGapWeight=7
filterEntries(listing, 'Vv Ddv Ddv Vv Ddv', 'as ass ass as ass');
expect(gapCache.toArray()).toEqual([]);
});
it('should cache a gap of weight equal to minGapWeight', () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapCache = new GapCacheAsSet(100);
listing.refreshGapCache(gapCache, 9); // minGapWeight=9
filterEntries(listing, 'Vv Ddv Ddv Ddv Vv Ddv', 'as ass ass ass as ass');
expect(gapCache.toArray()).toEqual([
{ firstKey: 'pre/0002', lastKey: `pre/0004${VID_SEP}v101`, weight: 9 },
]);
});
it('should cache a gap of weight equal to maxWeight in a single gap', () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapCache = new GapCacheAsSet(13); // maxWeight=13
listing.refreshGapCache(gapCache, 5); // minGapWeight=5
filterEntries(listing, 'Vv Ddv Ddvv Ddv Ddv Vv Ddv', 'as ass asss ass ass as ass');
expect(gapCache.toArray()).toEqual([
{ firstKey: 'pre/0002', lastKey: `pre/0005${VID_SEP}v101`, weight: 13 },
]);
});
it('should not cache a gap if listing has been running for more than exposureDelayMs',
async () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapsArray = [
{ firstKey: 'pre/0006', lastKey: `pre/0007${VID_SEP}v100`, weight: 6 },
];
const gapCache = GapCacheAsSet.createFromArray(JSON.parse(
JSON.stringify(gapsArray)
), 100);
listing.refreshGapCache(gapCache, 1, 1);
let resumeFromState = filterEntries(listing, 'Vv', 'as');
let validityPeriod = listing.getGapBuildingValidityPeriodMs();
expect(validityPeriod).toBeGreaterThan(gapCache.exposureDelayMs - 10);
expect(validityPeriod).toBeLessThan(gapCache.exposureDelayMs + 10);
await new Promise(resolve => setTimeout(resolve, gapCache.exposureDelayMs + 10));
validityPeriod = listing.getGapBuildingValidityPeriodMs();
expect(validityPeriod).toEqual(0);
resumeFromState = filterEntries(listing, 'Ddv Ddv Ddv Vvv', 'ass ass ass ass',
resumeFromState);
expect(gapCache.toArray()).toEqual(gapsArray);
// gap building should be in expired state
expect(listing._gapBuilding.state).toEqual(GapBuildingState.Expired);
// remaining validity period should still be 0 because gap building has expired
validityPeriod = listing.getGapBuildingValidityPeriodMs();
expect(validityPeriod).toEqual(0);
// we should still be able to skip over the existing cached gaps
expect(listing._gapCaching.state).toEqual(GapCachingState.GapLookupInProgress);
await new Promise(resolve => setTimeout(resolve, 1));
expect(listing._gapCaching.state).toEqual(GapCachingState.GapCached);
filterEntries(listing, 'Ddv Ddv Ddv', 'sss sss ass', resumeFromState);
});
[1, 3, 5, 10].forEach(triggerSaveGapWeight => {
it('should cache a gap of weight maxWeight + 1 in two chained gaps ' +
`(triggerSaveGapWeight=${triggerSaveGapWeight})`, () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapCache = new GapCacheAsSet(12); // maxWeight=12
listing.refreshGapCache(gapCache, 5, triggerSaveGapWeight);
filterEntries(listing, 'Vv Ddv Ddvv Ddv Ddv Vv Ddv', 'as ass asss ass ass as ass');
if (triggerSaveGapWeight === 1) {
// trigger=1 guarantees that the weight of split gaps is maximized
expect(gapCache.toArray()).toEqual([
{ firstKey: 'pre/0002', lastKey: `pre/0005${VID_SEP}v100`, weight: 12 },
{ firstKey: `pre/0005${VID_SEP}v100`, lastKey: `pre/0005${VID_SEP}v101`, weight: 1 },
]);
} else if (triggerSaveGapWeight === 3) {
// - the first trigger happens after 'minGapWeight' listing entries, so 5
// - the second and third triggers happen after 'triggerSaveGapWeight' listing
// entries, so 3 then 3 - same gap because 5+3+3=11 and 11 <= 12 (maxWeight)
// - finally, 2 more entries to complete the gap, at which point the
// entry is split, hence we get two entries weights 11 and 2 respectively.
expect(gapCache.toArray()).toEqual([
{ firstKey: 'pre/0002', lastKey: 'pre/0005', weight: 11 },
{ firstKey: 'pre/0005', lastKey: `pre/0005${VID_SEP}v101`, weight: 2 },
]);
} else {
// trigger=5|10
expect(gapCache.toArray()).toEqual([
{ firstKey: 'pre/0002', lastKey: `pre/0004${VID_SEP}v101`, weight: 10 },
{ firstKey: `pre/0004${VID_SEP}v101`, lastKey: `pre/0005${VID_SEP}v101`, weight: 3 },
]);
}
});
});
[1, 2, 3].forEach(triggerSaveGapWeight => {
it('should cache a gap of weight more than twice maxWeight in as many chained gaps ' +
`as needed (triggerSaveGapWeight=${triggerSaveGapWeight})`, () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapCache = new GapCacheAsSet(5); // maxWeight=5
// minGapWeight=4 prevents the last gap starting at "0008" from being cached
listing.refreshGapCache(gapCache, 4, triggerSaveGapWeight);
filterEntries(listing, 'Vv Ddv Ddvv Ddv Ddv Ddv Vv Ddv', 'as ass asss ass ass ass as ass');
// the slight differences in weight between different values of
// 'triggerSaveGapWeight' are due to the combination of the trigger
// frequency and the 'minGapWeight' value (3), but in all cases a
// reasonable splitting job should be obtained.
//
// NOTE: in practice, the default trigger is half the maximum weight, any value
// equal or lower should yield gap weights close enough to the maximum allowed.
if (triggerSaveGapWeight === 1) {
// a trigger at every key guarantees gaps to maximize their weight
expect(gapCache.toArray()).toEqual([
{ firstKey: 'pre/0002', lastKey: `pre/0003${VID_SEP}v100`, weight: 5 },
{ firstKey: `pre/0003${VID_SEP}v100`, lastKey: `pre/0004${VID_SEP}v101`, weight: 5 },
{ firstKey: `pre/0004${VID_SEP}v101`, lastKey: `pre/0006${VID_SEP}v100`, weight: 5 },
{ firstKey: `pre/0006${VID_SEP}v100`, lastKey: `pre/0006${VID_SEP}v101`, weight: 1 },
]);
} else if (triggerSaveGapWeight === 2) {
expect(gapCache.toArray()).toEqual([
{ firstKey: 'pre/0002', lastKey: 'pre/0003', weight: 4 },
{ firstKey: 'pre/0003', lastKey: 'pre/0004', weight: 4 },
{ firstKey: 'pre/0004', lastKey: `pre/0005${VID_SEP}v100`, weight: 4 },
{ firstKey: `pre/0005${VID_SEP}v100`, lastKey: `pre/0006${VID_SEP}v101`, weight: 4 },
]);
} else {
// trigger=3
expect(gapCache.toArray()).toEqual([
{ firstKey: 'pre/0002', lastKey: 'pre/0003', weight: 4 },
{ firstKey: 'pre/0003', lastKey: `pre/0003${VID_SEP}v102`, weight: 3 },
{ firstKey: `pre/0003${VID_SEP}v102`, lastKey: `pre/0004${VID_SEP}v101`, weight: 3 },
{ firstKey: `pre/0004${VID_SEP}v101`, lastKey: `pre/0005${VID_SEP}v101`, weight: 3 },
{ firstKey: `pre/0005${VID_SEP}v101`, lastKey: `pre/0006${VID_SEP}v101`, weight: 3 },
]);
}
});
});
it('should cut the current gap when seeing a non-deleted object, and start a new ' +
'gap on the next deleted object', () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapCache = new GapCacheAsSet(100);
listing.refreshGapCache(gapCache, 2); // minGapWeight=2
filterEntries(listing, 'Vv Ddv Vv Ddv Vv', 'as ass as ass as');
expect(gapCache.toArray()).toEqual([
{ firstKey: 'pre/0002', lastKey: `pre/0002${VID_SEP}v101`, weight: 3 },
{ firstKey: 'pre/0004', lastKey: `pre/0004${VID_SEP}v101`, weight: 3 },
]);
});
it('should complete the current gap when returning a result', () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapCache = new GapCacheAsSet(100);
listing.refreshGapCache(gapCache, 2); // ensure the gap above minGapWeight=2 gets saved
filterEntries(listing, 'Vv Ddv Ddv', 'as ass ass');
const result = listing.result();
expect(result).toEqual({
CommonPrefixes: [],
Contents: [
{ key: 'pre/0001', value: '{"versionId":"v100"}' },
],
Delimiter: undefined,
IsTruncated: false,
NextMarker: undefined,
});
expect(gapCache.toArray()).toEqual([
{ firstKey: 'pre/0002', lastKey: `pre/0003${VID_SEP}v101`, weight: 6 },
]);
});
it('should refresh the building params when refreshGapCache() is called in NonBuilding state',
() => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapCache = new GapCacheAsSet(100);
// ensure the first gap with weight=9 gets saved
listing.refreshGapCache(gapCache, 9);
let resumeFromState = filterEntries(listing, 'Vv', 'as');
// refresh with a different value for minGapWeight (12)
listing.refreshGapCache(gapCache, 12);
resumeFromState = filterEntries(listing, 'Ddv Ddv Ddv Vv', 'ass ass ass as',
resumeFromState);
// for the building gap, minGapWeight should have been updated to 12, hence the
// gap should not have been created
expect(gapCache.toArray()).toEqual([]);
filterEntries(listing, 'Ddv Ddv Ddv Ddv Vv', 'ass ass ass ass as', resumeFromState);
// there should now be a new gap with weight=12
expect(gapCache.toArray()).toEqual([
{ firstKey: 'pre/0006', lastKey: `pre/0009${VID_SEP}v101`, weight: 12 },
]);
});
it('should save the refreshed building params when refreshGapCache() is called in Building state',
() => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapCache = new GapCacheAsSet(100);
// ensure the first gap with weight=9 gets saved
listing.refreshGapCache(gapCache, 9);
let resumeFromState = filterEntries(listing, 'Vv Ddv Ddv', 'as ass ass');
// refresh with a different value for minGapWeight (12)
listing.refreshGapCache(gapCache, 12);
resumeFromState = filterEntries(listing, 'Ddv Vv', 'ass as', resumeFromState);
// for the building gap, minGapWeight should still be 9, hence the gap should
// have been created
expect(gapCache.toArray()).toEqual([
{ firstKey: 'pre/0002', lastKey: `pre/0004${VID_SEP}v101`, weight: 9 },
]);
filterEntries(listing, 'Ddv Ddv Ddv Vv', 'ass ass ass as', resumeFromState);
// there should still be only one gap because the next gap's weight is 9 and 9 < 12
expect(gapCache.toArray()).toEqual([
{ firstKey: 'pre/0002', lastKey: `pre/0004${VID_SEP}v101`, weight: 9 },
]);
});
it('should not build a new gap when skipping a prefix', () => {
const listing = new DelimiterMaster({
delimiter: '/',
}, fakeLogger, 'v0');
const gapCache = new GapCacheAsSet(100);
// force immediate creation of gaps with 1, 1
listing.refreshGapCache(gapCache, 1, 1);
// prefix should be skipped, but no new gap should be created
filterEntries(listing, 'Vv Ddv Ddv Ddv', 'as sss sss sss');
expect(gapCache.toArray()).toEqual([]);
});
it('should trigger gap lookup and continue filtering without skipping when encountering ' +
'a delete marker', () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapsArray = [
{ firstKey: 'pre/0002', lastKey: `pre/0003${VID_SEP}v100`, weight: 6 },
];
const gapCache = GapCacheAsSet.createFromArray(JSON.parse(
JSON.stringify(gapsArray)
), 100);
listing.refreshGapCache(gapCache);
let resumeState = filterEntries(listing, 'Vv', 'as');
// state should still be UnknownGap since no delete marker has been seen yet
expect(listing._gapCaching.state).toEqual(GapCachingState.UnknownGap);
resumeState = filterEntries(listing, 'D', 'a', resumeState);
// since the lookup is asynchronous (Promise-based), it should now be in progress
expect(listing._gapCaching.state).toEqual(GapCachingState.GapLookupInProgress);
filterEntries(listing, 'dv Ddv Vv Ddv', 'ss ass as ass', resumeState);
// the lookup should still be in progress
expect(listing._gapCaching.state).toEqual(GapCachingState.GapLookupInProgress);
// the gap cache shouldn't have been updated
expect(gapCache.toArray()).toEqual(gapsArray);
});
it('should cache a gap after lookup completes, and use it to skip over keys ' +
'within the gap range', async () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapsArray = [
{ firstKey: 'pre/0002', lastKey: `pre/0006${VID_SEP}v101`, weight: 14 },
];
const gapCache = GapCacheAsSet.createFromArray(JSON.parse(
JSON.stringify(gapsArray)
), 100);
listing.refreshGapCache(gapCache);
let resumeState = filterEntries(listing, 'Vv D', 'as a');
// since the lookup is asynchronous (Promise-based), it should now be in progress
expect(listing._gapCaching.state).toEqual(GapCachingState.GapLookupInProgress);
expect(listing.state.id).toEqual(DelimiterMasterFilterStateId.SkippingVersionsV0);
// wait until the lookup completes (should happen in the next
// event loop iteration so always quicker than a non-immediate timer)
await new Promise(resolve => setTimeout(resolve, 1));
// the lookup should have completed now and the next gap should be cached
expect(listing._gapCaching.state).toEqual(GapCachingState.GapCached);
expect(listing.state.id).toEqual(DelimiterMasterFilterStateId.SkippingVersionsV0);
// the state should stay in SkippingVersionsV0 until filter() is called with
// a new master delete marker
resumeState = filterEntries(listing, 'dvvv', 'ssss', resumeState);
expect(listing.state.id).toEqual(DelimiterMasterFilterStateId.SkippingVersionsV0);
// here comes the next master delete marker, it should be skipped as it is still within
// the cached gap's range (its key is "0003" and version "v100")
resumeState = filterEntries(listing, 'D', 's', resumeState);
// the listing algorithm should now be actively skipping the gap
expect(listing.state.id).toEqual(DelimiterMasterFilterStateId.SkippingGapV0);
// the skipping() function should return the gap's last key.
// NOTE: returning a key to jump to that is the actual gap's last key
// (instead of a key just after) allows the listing algorithm to build
// a chained gap when the database listing is restarted from that point
// and there are more delete markers to skip.
expect(listing.skipping()).toEqual(`pre/0006${VID_SEP}v101`);
// - The next master delete markers with key "0004" and "0005" are still within the
// gap's range, so filter() should return FILTER_SKIP ('s')
//
// - Master key "0006" is NOT a delete marker, although this means that the update
// happened after the gap was looked up and the listing is allowed to skip it as
// well (it actually doesn't even check so doesn't know what type of key it is).
//
// - The following master delete marker "0007" is past the gap so returns
// FILTER_ACCEPT ('a') and should have triggered a new cache lookup, and
// the listing state should have been switched back to SkippingVersionsV0.
resumeState = filterEntries(listing, 'dv Ddv Ddv Vvvv Ddv', 'ss sss sss ssss ass',
resumeState);
expect(listing._gapCaching.state).toEqual(GapCachingState.GapLookupInProgress);
expect(listing.state.id).toEqual(DelimiterMasterFilterStateId.SkippingVersionsV0);
// the gap cache must not have been updated in the process
expect(gapCache.toArray()).toEqual(gapsArray);
});
it('should extend a cached gap forward if current delete markers are listed beyond',
async () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapsArray = [
{ firstKey: 'pre/0002', lastKey: `pre/0005${VID_SEP}v100`, weight: 12 },
];
const gapCache = GapCacheAsSet.createFromArray(JSON.parse(
JSON.stringify(gapsArray)
), 100);
listing.refreshGapCache(gapCache, 2);
let resumeState = filterEntries(listing, 'Vv D', 'as a');
// wait until the lookup completes (should happen in the next
// event loop iteration so always quicker than a non-immediate timer)
await new Promise(resolve => setTimeout(resolve, 1));
// the lookup should have completed now and the next gap should be cached,
// continue with filtering
resumeState = filterEntries(listing, 'dv Ddv Ddv Ddv Ddv Ddvvv Vv Ddv Vv',
'ss sss sss sss ass assss as ass as',
resumeState);
// the cached gap should be extended to the last key before the last regular
// master version ('V')
expect(gapCache.toArray()).toEqual([
// this gap has been extended forward up to right before the first non-deleted
// current version following the gap, and its weight updated with how many
// extra keys are skippable
{ firstKey: 'pre/0002', lastKey: `pre/0007${VID_SEP}v103`, weight: 21 },
// this gap has been created from the next deleted current version
{ firstKey: 'pre/0009', lastKey: `pre/0009${VID_SEP}v101`, weight: 3 },
]);
});
it('should extend a cached gap backwards if current delete markers are listed ahead, ' +
'and forward if more skippable keys are seen', async () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapsArray = [
{ firstKey: 'pre/0004', lastKey: `pre/0005${VID_SEP}v100`, weight: 4 },
];
const gapCache = GapCacheAsSet.createFromArray(JSON.parse(
JSON.stringify(gapsArray)
), 100);
listing.refreshGapCache(gapCache, 2);
let resumeState = filterEntries(listing, 'Vv D', 'as a');
// wait until the lookup completes (should happen in the next
// event loop iteration so always quicker than a non-immediate timer)
await new Promise(resolve => setTimeout(resolve, 1));
// the lookup should have completed now and the next gap should be cached,
// continue with filtering
expect(listing._gapCaching.state).toEqual(GapCachingState.GapCached);
resumeState = filterEntries(listing, 'dv Ddv Ddv Ddv Vv Ddv Vv',
'ss ass sss sss as ass as', resumeState);
// the cached gap should be extended to the last key before the last regular
// master version ('V')
expect(gapCache.toArray()).toEqual([
// this gap has been extended:
// - backwards up to the first listed delete marker
// - forward up to the last skippable key
// and its weight updated with how many extra keys are skippable
{ firstKey: 'pre/0002', lastKey: `pre/0005${VID_SEP}v101`, weight: 11 },
// this gap has been created from the next deleted current version
{ firstKey: 'pre/0007', lastKey: `pre/0007${VID_SEP}v101`, weight: 3 },
]);
});
it('should not extend a cached gap forward if extension weight is 0',
async () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapsArray = [
{ firstKey: 'pre/0002', lastKey: `pre/0005${VID_SEP}v101`, weight: 13 },
];
const gapCache = GapCacheAsSet.createFromArray(JSON.parse(
JSON.stringify(gapsArray)
), 100);
listing.refreshGapCache(gapCache, 2);
let resumeState = filterEntries(listing, 'Vv D', 'as a');
// wait until the lookup completes (should happen in the next
// event loop iteration so always quicker than a non-immediate timer)
await new Promise(resolve => setTimeout(resolve, 1));
// the lookup should have completed now and the next gap should
// be cached, simulate a concurrent invalidation by removing the
// existing gap immediately, then continue with filtering
resumeState = filterEntries(listing, 'dv Ddv Ddv Ddv',
'ss sss sss sss', resumeState);
gapCache.removeOverlappingGaps(['pre/0002']);
resumeState = filterEntries(listing, 'Vv', 'as', resumeState);
// no new gap should have been added
expect(gapCache.toArray()).toEqual([]);
});
it('should ignore gap with 0 listed key in it (e.g. due to skipping a prefix)', async () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v0');
const gapsArray = [
{ firstKey: 'pre/0004/a', lastKey: 'pre/0004/b', weight: 10 },
];
const gapCache = GapCacheAsSet.createFromArray(JSON.parse(
JSON.stringify(gapsArray)
), 100);
listing.refreshGapCache(gapCache);
let resumeState = filterEntries(listing, 'Dd Vv Vv', 'as as as');
// wait until the lookup completes (should happen in the next
// event loop iteration so always quicker than a non-immediate timer)
await new Promise(resolve => setTimeout(resolve, 1));
expect(listing._gapCaching.state).toEqual(GapCachingState.GapCached);
expect(gapCache.toArray()).toEqual([
{ firstKey: 'pre/0004/a', lastKey: 'pre/0004/b', weight: 10 },
]);
// "0004" keys are still prior to the gap's first key
resumeState = filterEntries(listing, 'Ddv', 'ass', resumeState);
expect(listing._gapCaching.state).toEqual(GapCachingState.GapCached);
// the next delete marker "0005" should trigger a new lookup...
resumeState = filterEntries(listing, 'D', 'a', resumeState);
expect(listing._gapCaching.state).toEqual(GapCachingState.GapLookupInProgress);
await new Promise(resolve => setTimeout(resolve, 1));
// ...which returns 'null' and sets the state to NoMoreGap
expect(listing._gapCaching.state).toEqual(GapCachingState.NoMoreGap);
filterEntries(listing, 'dv Vv', 'ss as', resumeState);
});
it('should disable gap fetching and building if using V1 format', async () => {
const listing = new DelimiterMaster({}, fakeLogger, 'v1');
const gapCache = new GapCacheAsSet(100);
listing.refreshGapCache(gapCache);
expect(listing.getGapBuildingValidityPeriodMs()).toBeNull();
expect(listing._gapCaching.state).toEqual(GapCachingState.NoGapCache);
// mimic V1 listing of master prefix
filterEntries(listing, 'V V', 'a a');
expect(listing._gapBuilding.state).toEqual(GapBuildingState.Disabled);
});
});

View File

@ -0,0 +1,452 @@
'use strict'; // eslint-disable-line strict
const assert = require('assert');
const DelimiterNonCurrent =
require('../../../../lib/algos/list/delimiterNonCurrent').DelimiterNonCurrent;
const {
FILTER_ACCEPT,
FILTER_END,
} = require('../../../../lib/algos/list/tools');
const VSConst =
require('../../../../lib/versioning/constants').VersioningConstants;
const { DbPrefixes } = VSConst;
const VID_SEP = VSConst.VersionId.Separator;
const EmptyResult = {
Contents: [],
IsTruncated: false,
};
const fakeLogger = {
trace: () => {},
debug: () => {},
info: () => {},
warn: () => {},
error: () => {},
fatal: () => {},
};
function getListingKey(key, vFormat) {
if (vFormat === 'v0') {
return key;
}
if (vFormat === 'v1') {
const keyPrefix = key.includes(VID_SEP) ?
DbPrefixes.Version : DbPrefixes.Master;
return `${keyPrefix}${key}`;
}
return assert.fail(`bad format ${vFormat}`);
}
['v0', 'v1'].forEach(v => {
describe(`DelimiterNonCurrent with ${v} bucket format`, () => {
it('should return expected metadata parameters', () => {
const prefix = 'pre';
const keyMarker = 'premark';
const versionIdMarker = 'vid1';
const maxScannedLifecycleListingEntries = 2;
const delimiter = new DelimiterNonCurrent({
prefix,
keyMarker,
versionIdMarker,
maxScannedLifecycleListingEntries,
}, fakeLogger, v);
let expectedParams;
if (v === 'v0') {
expectedParams = { gte: `${keyMarker}${VID_SEP}`, lt: 'prf' };
} else {
expectedParams = [
{
gte: `${DbPrefixes.Master}${keyMarker}${VID_SEP}`,
lt: `${DbPrefixes.Master}prf`,
},
{
gte: `${DbPrefixes.Version}${keyMarker}${VID_SEP}`,
lt: `${DbPrefixes.Version}prf`,
},
];
}
assert.deepStrictEqual(delimiter.genMDParams(), expectedParams);
assert.strictEqual(delimiter.maxScannedLifecycleListingEntries, 2);
});
it('should accept entry starting with prefix', () => {
const delimiter = new DelimiterNonCurrent({ prefix: 'prefix' }, fakeLogger, v);
const listingKey = getListingKey('prefix1', v);
assert.strictEqual(delimiter.filter({ key: listingKey, value: '' }), FILTER_ACCEPT);
assert.deepStrictEqual(delimiter.result(), EmptyResult);
});
it('should accept a version and return an empty content', () => {
const delimiter = new DelimiterNonCurrent({ }, fakeLogger, v);
const masterKey = 'key';
const versionId1 = 'version1';
const versionKey1 = `${masterKey}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.001Z';
const value1 = `{"versionId":"${versionId1}", "last-modified": "${date1}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
assert.deepStrictEqual(delimiter.result(), EmptyResult);
});
it('should accept two versions and return the noncurrent version', () => {
const delimiter = new DelimiterNonCurrent({ }, fakeLogger, v);
const masterKey = 'key';
// filter first version
const versionId1 = 'version1';
const versionKey1 = `${masterKey}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.002Z';
const value1 = `{"versionId":"${versionId1}", "last-modified": "${date1}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
// filter second version
const versionId2 = 'version2';
const versionKey2 = `${masterKey}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"versionId":"${versionId2}", "last-modified": "${date2}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_ACCEPT);
const expectedResult = {
Contents: [
{
key: masterKey,
value: `{"versionId":"${versionId2}","last-modified":"${date2}","staleDate":"${date1}"}`,
},
],
IsTruncated: false,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should accept three versions and return the noncurrent version which stale date before beforeDate', () => {
const beforeDate = '1970-01-01T00:00:00.002Z';
const delimiter = new DelimiterNonCurrent({ beforeDate }, fakeLogger, v);
const masterKey = 'key';
// filter first version
const versionId1 = 'version1';
const versionKey1 = `${masterKey}${VID_SEP}${versionId1}`;
const date1 = beforeDate;
const value1 = `{"versionId":"${versionId1}", "last-modified": "${date1}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
// filter second version
const versionId2 = 'version2';
const versionKey2 = `${masterKey}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"versionId":"${versionId2}", "last-modified": "${date2}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_ACCEPT);
// filter third version
const versionId3 = 'version3';
const versionKey3 = `${masterKey}${VID_SEP}${versionId3}`;
const date3 = '1970-01-01T00:00:00.000Z';
const value3 = `{"versionId":"${versionId3}", "last-modified": "${date3}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey3, v),
value: value3,
}), FILTER_ACCEPT);
const expectedResult = {
Contents: [
{
key: masterKey,
value: `{"versionId":"${versionId3}","last-modified":"${date3}","staleDate":"${date2}"}`,
},
],
IsTruncated: false,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should accept one delete marker and one version and return the noncurrent version', () => {
const delimiter = new DelimiterNonCurrent({ }, fakeLogger, v);
// const version = new Version({ isDeleteMarker: true });
const masterKey = 'key';
// filter delete marker
const versionId1 = 'version1';
const versionKey1 = `${masterKey}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.002Z';
const value1 = `{"versionId":"${versionId1}", "last-modified": "${date1}", "isDeleteMarker": true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
// filter second version
const versionId2 = 'version2';
const versionKey2 = `${masterKey}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"versionId":"${versionId2}", "last-modified": "${date2}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_ACCEPT);
const expectedResult = {
Contents: [
{
key: masterKey,
value: `{"versionId":"${versionId2}","last-modified":"${date2}","staleDate":"${date1}"}`,
},
],
IsTruncated: false,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should end filtering if max keys reached', () => {
const delimiter = new DelimiterNonCurrent({ maxKeys: 1 }, fakeLogger, v);
const masterKey = 'key';
// filter delete marker
const versionId1 = 'version1';
const versionKey1 = `${masterKey}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.002Z';
const value1 = `{"versionId":"${versionId1}", "last-modified": "${date1}", "isDeleteMarker": true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
// filter second version
const versionId2 = 'version2';
const versionKey2 = `${masterKey}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"versionId":"${versionId2}", "last-modified": "${date2}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_ACCEPT);
// filter third version
const versionId3 = 'version3';
const versionKey3 = `${masterKey}${VID_SEP}${versionId3}`;
const date3 = '1970-01-01T00:00:00.000Z';
const value3 = `{"versionId":"${versionId3}", "last-modified": "${date3}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey3, v),
value: value3,
}), FILTER_END);
const expectedResult = {
Contents: [
{
key: masterKey,
value: `{"versionId":"${versionId2}","last-modified":"${date2}","staleDate":"${date1}"}`,
},
],
IsTruncated: true,
NextKeyMarker: masterKey,
NextVersionIdMarker: versionId2,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should return the non-current versions pushed before max scanned entries value is reached', () => {
const maxScannedLifecycleListingEntries = 2;
const delimiter = new DelimiterNonCurrent({ maxScannedLifecycleListingEntries }, fakeLogger, v);
const masterKey = 'key';
// filter delete marker
const versionId1 = 'version1';
const versionKey1 = `${masterKey}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.002Z';
const value1 = `{"versionId":"${versionId1}", "last-modified": "${date1}", "isDeleteMarker": true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
// filter second version
const versionId2 = 'version2';
const versionKey2 = `${masterKey}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"versionId":"${versionId2}", "last-modified": "${date2}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_ACCEPT);
// filter third version
const versionId3 = 'version3';
const versionKey3 = `${masterKey}${VID_SEP}${versionId3}`;
const date3 = '1970-01-01T00:00:00.000Z';
const value3 = `{"versionId":"${versionId3}", "last-modified": "${date3}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey3, v),
value: value3,
}), FILTER_END);
const expectedResult = {
Contents: [
{
key: masterKey,
value: `{"versionId":"${versionId2}","last-modified":"${date2}","staleDate":"${date1}"}`,
},
],
IsTruncated: true,
NextKeyMarker: masterKey,
NextVersionIdMarker: versionId2,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should return empty content after max scanned entries value is reached', () => {
const maxScannedLifecycleListingEntries = 2;
const delimiter = new DelimiterNonCurrent({ maxScannedLifecycleListingEntries }, fakeLogger, v);
// filter current version
const masterKey1 = 'key1';
const versionId1 = 'version1';
const versionKey1 = `${masterKey1}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.002Z';
const value1 = `{"versionId":"${versionId1}", "last-modified": "${date1}"`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
// filter current version
const masterKey2 = 'key2';
const versionId2 = 'version2';
const versionKey2 = `${masterKey2}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"versionId":"${versionId2}", "last-modified": "${date2}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_ACCEPT);
// filter current version
const masterKey3 = 'key3';
const versionId3 = 'version3';
const versionKey3 = `${masterKey3}${VID_SEP}${versionId3}`;
const date3 = '1970-01-01T00:00:00.000Z';
const value3 = `{"versionId":"${versionId3}", "last-modified": "${date3}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey3, v),
value: value3,
}), FILTER_END);
const expectedResult = {
Contents: [],
IsTruncated: true,
NextKeyMarker: masterKey2,
NextVersionIdMarker: versionId2,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should return noncurrent versions starting from a marker', () => {
const delimiter = new DelimiterNonCurrent({
keyMarker: 'key',
versionIdMarker: 'version1',
}, fakeLogger, v);
const masterKey = 'key';
// filter first version
const versionId1 = 'version1';
const versionKey1 = `${masterKey}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.002Z';
const value1 = `{"versionId":"${versionId1}", "last-modified": "${date1}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
// filter second version
const versionId2 = 'version2';
const versionKey2 = `${masterKey}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"versionId":"${versionId2}", "last-modified": "${date2}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_ACCEPT);
// filter third version
const versionId3 = 'version3';
const versionKey3 = `${masterKey}${VID_SEP}${versionId3}`;
const date3 = '1970-01-01T00:00:00.000Z';
const value3 = `{"versionId":"${versionId3}", "last-modified": "${date3}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey3, v),
value: value3,
}), FILTER_ACCEPT);
const expectedResult = {
Contents: [
{
key: masterKey,
value: `{"versionId":"${versionId2}","last-modified":"${date2}","staleDate":"${date1}"}`,
},
{
key: masterKey,
value: `{"versionId":"${versionId3}","last-modified":"${date3}","staleDate":"${date2}"}`,
},
],
IsTruncated: false,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
});
});

View File

@ -0,0 +1,493 @@
'use strict'; // eslint-disable-line strict
const assert = require('assert');
const DelimiterOrphanDeleteMarker =
require('../../../../lib/algos/list/delimiterOrphanDeleteMarker').DelimiterOrphanDeleteMarker;
const {
FILTER_ACCEPT,
FILTER_END,
inc,
} = require('../../../../lib/algos/list/tools');
const VSConst =
require('../../../../lib/versioning/constants').VersioningConstants;
const { DbPrefixes } = VSConst;
const VID_SEP = VSConst.VersionId.Separator;
const EmptyResult = {
Contents: [],
IsTruncated: false,
};
const fakeLogger = {
trace: () => {},
debug: () => {},
info: () => {},
warn: () => {},
error: () => {},
fatal: () => {},
};
function getListingKey(key, vFormat) {
if (vFormat === 'v0') {
return key;
}
if (vFormat === 'v1') {
const keyPrefix = key.includes(VID_SEP) ?
DbPrefixes.Version : DbPrefixes.Master;
return `${keyPrefix}${key}`;
}
return assert.fail(`bad format ${vFormat}`);
}
['v0', 'v1'].forEach(v => {
describe(`DelimiterOrphanDeleteMarker with ${v} bucket format`, () => {
it('should return expected metadata parameters', () => {
const prefix = 'pre';
const marker = 'premark';
const maxScannedLifecycleListingEntries = 2;
const delimiter = new DelimiterOrphanDeleteMarker({
prefix,
marker,
maxScannedLifecycleListingEntries,
}, fakeLogger, v);
let expectedParams;
if (v === 'v0') {
expectedParams = { gt: `premark${inc(VID_SEP)}`, lt: 'prf' };
} else {
expectedParams = [
{
gt: `${DbPrefixes.Master}premark${inc(VID_SEP)}`,
lt: `${DbPrefixes.Master}prf`,
},
{
gt: `${DbPrefixes.Version}premark${inc(VID_SEP)}`,
lt: `${DbPrefixes.Version}prf`,
},
];
}
assert.deepStrictEqual(delimiter.genMDParams(), expectedParams);
assert.strictEqual(delimiter.maxScannedLifecycleListingEntries, 2);
});
it('should accept entry starting with prefix', () => {
const delimiter = new DelimiterOrphanDeleteMarker({ prefix: 'prefix' }, fakeLogger, v);
const listingKey = getListingKey('prefix1', v);
assert.strictEqual(delimiter.filter({ key: listingKey, value: '' }), FILTER_ACCEPT);
assert.deepStrictEqual(delimiter.result(), EmptyResult);
});
it('should accept a version and return an empty content', () => {
const delimiter = new DelimiterOrphanDeleteMarker({ }, fakeLogger, v);
const masterKey = 'key';
const versionId1 = 'version1';
const versionKey1 = `${masterKey}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.001Z';
const value1 = `{"versionId":"${versionId1}","last-modified":"${date1}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
assert.deepStrictEqual(delimiter.result(), EmptyResult);
});
it('should accept an orphan delete marker and return it from the content', () => {
const delimiter = new DelimiterOrphanDeleteMarker({ }, fakeLogger, v);
const masterKey = 'key';
const versionId1 = 'version1';
const versionKey1 = `${masterKey}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.001Z';
const value1 = `{"versionId":"${versionId1}","last-modified":"${date1}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
const expectedResult = {
Contents: [
{
key: masterKey,
value: value1,
},
],
IsTruncated: false,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should accept two orphan delete markers and return them from the content', () => {
const delimiter = new DelimiterOrphanDeleteMarker({ }, fakeLogger, v);
// filter the first orphan delete marker
const masterKey1 = 'key1';
const versionId1 = 'version1';
const versionKey1 = `${masterKey1}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.002Z';
const value1 = `{"versionId":"${versionId1}","last-modified":"${date1}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
// filter the second orphan delete marker
const masterKey2 = 'key2';
const versionId2 = 'version2';
const versionKey2 = `${masterKey2}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"versionId":"${versionId2}","last-modified":"${date2}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_ACCEPT);
const expectedResult = {
Contents: [
{
key: masterKey1,
value: value1,
},
{
key: masterKey2,
value: value2,
},
],
IsTruncated: false,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should accept two orphan delete markers and return truncated content with one', () => {
const delimiter = new DelimiterOrphanDeleteMarker({ maxKeys: 1 }, fakeLogger, v);
// filter the first orphan delete marker
const masterKey1 = 'key1';
const versionId1 = 'version1';
const versionKey1 = `${masterKey1}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.002Z';
const value1 = `{"versionId":"${versionId1}","last-modified":"${date1}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
// filter the second orphan delete marker
const masterKey2 = 'key2';
const versionId2 = 'version2';
const versionKey2 = `${masterKey2}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"versionId":"${versionId2}","last-modified":"${date2}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_ACCEPT);
const expectedResult = {
Contents: [
{
key: masterKey1,
value: value1,
},
],
NextMarker: masterKey1,
IsTruncated: true,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should accept two orphan delete markers and return the one created before the beforeDate', () => {
const date1 = '1970-01-01T00:00:00.002Z';
const delimiter = new DelimiterOrphanDeleteMarker({ beforeDate: date1 }, fakeLogger, v);
// filter the first orphan delete marker
const masterKey1 = 'key1';
const versionId1 = 'version1';
const versionKey1 = `${masterKey1}${VID_SEP}${versionId1}`;
const value1 = `{"versionId":"${versionId1}","last-modified":"${date1}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
// filter the second orphan delete marker
const masterKey2 = 'key2';
const versionId2 = 'version2';
const versionKey2 = `${masterKey2}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"versionId":"${versionId2}","last-modified":"${date2}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_ACCEPT);
const expectedResult = {
Contents: [
{
key: masterKey2,
value: value2,
},
],
IsTruncated: false,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should end filtering if max keys reached', () => {
const delimiter = new DelimiterOrphanDeleteMarker({ maxKeys: 1 }, fakeLogger, v);
// filter the first orphan delete marker
const masterKey1 = 'key1';
const versionId1 = 'version1';
const versionKey1 = `${masterKey1}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.002Z';
const value1 = `{"versionId":"${versionId1}","last-modified":"${date1}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
// filter the second orphan delete marker
const masterKey2 = 'key2';
const versionId2 = 'version2';
const versionKey2 = `${masterKey2}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"versionId":"${versionId2}","last-modified":"${date2}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_ACCEPT);
// filter the third orphan delete marker
const masterKey3 = 'key3';
const versionId3 = 'version3';
const versionKey3 = `${masterKey3}${VID_SEP}${versionId3}`;
const date3 = '1970-01-01T00:00:00.000Z';
const value3 = `{"versionId":"${versionId3}","last-modified":"${date3}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey3, v),
value: value3,
}), FILTER_END);
const expectedResult = {
Contents: [
{
key: masterKey1,
value: value1,
},
],
NextMarker: masterKey1,
IsTruncated: true,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should end filtering if max scanned entries value is reached', () => {
const maxScannedLifecycleListingEntries = 2;
const delimiter = new DelimiterOrphanDeleteMarker({ maxScannedLifecycleListingEntries }, fakeLogger, v);
// filter the first orphan delete marker
const masterKey1 = 'key1';
const versionId1 = 'version1';
const versionKey1 = `${masterKey1}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.002Z';
const value1 = `{"versionId":"${versionId1}","last-modified":"${date1}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
// filter the second orphan delete marker
const masterKey2 = 'key2';
const versionId2 = 'version2';
const versionKey2 = `${masterKey2}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"versionId":"${versionId2}","last-modified":"${date2}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_ACCEPT);
// filter the third orphan delete marker
const masterKey3 = 'key3';
const versionId3 = 'version3';
const versionKey3 = `${masterKey3}${VID_SEP}${versionId3}`;
const date3 = '1970-01-01T00:00:00.001Z';
const value3 = `{"versionId":"${versionId3}","last-modified":"${date3}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey3, v),
value: value3,
}), FILTER_END);
const expectedResult = {
Contents: [
{
key: masterKey1,
value: value1,
},
],
NextMarker: masterKey1,
IsTruncated: true,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should not consider the last delete marker scanned as an orphan if listing interrupted', () => {
const maxScannedLifecycleListingEntries = 1;
const delimiter = new DelimiterOrphanDeleteMarker({ maxScannedLifecycleListingEntries }, fakeLogger, v);
// filter the delete marker (not orphan)
const masterKey1 = 'key1';
const versionId1 = 'version1';
const versionKey1 = `${masterKey1}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.001Z';
const value1 = `{"versionId":"${versionId1}","last-modified":"${date1}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
const versionId2 = 'version2';
const versionKey2 = `${masterKey1}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.002Z';
const value2 = `{"versionId":"${versionId2}","last-modified":"${date2}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_END);
const expectedResult = {
Contents: [],
NextMarker: null,
IsTruncated: true,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should end filtering with empty content if max scanned entries value is reached', () => {
const maxScannedLifecycleListingEntries = 2;
const delimiter = new DelimiterOrphanDeleteMarker({ maxScannedLifecycleListingEntries }, fakeLogger, v);
// not a delete marker
const masterKey1 = 'key1';
const versionId1 = 'version1';
const versionKey1 = `${masterKey1}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.002Z';
const value1 = `{"versionId":"${versionId1}","last-modified":"${date1}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
// not a delete marker
const masterKey2 = 'key2';
const versionId2 = 'version2';
const versionKey2 = `${masterKey2}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"versionId":"${versionId2}","last-modified":"${date2}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_ACCEPT);
// orphan delete marker
const masterKey3 = 'key3';
const versionId3 = 'version3';
const versionKey3 = `${masterKey3}${VID_SEP}${versionId3}`;
const date3 = '1970-01-01T00:00:00.000Z';
const value3 = `{"versionId":"${versionId3}","last-modified":"${date3}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey3, v),
value: value3,
}), FILTER_END);
const expectedResult = {
Contents: [],
NextMarker: masterKey1,
IsTruncated: true,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
it('should return NextMarker when the max scanned entries is reached while processing a non-orphan key', () => {
// This approach prevents us from starting the next listing from the non-orphan key and, as a result,
// avoids the need to revisit all its versions unnecessarily.
const maxScannedLifecycleListingEntries = 2;
const delimiter = new DelimiterOrphanDeleteMarker({ maxScannedLifecycleListingEntries }, fakeLogger, v);
// key 1 is not an orphan
const masterKey1 = 'key1';
const versionId1 = 'version1';
const versionKey1 = `${masterKey1}${VID_SEP}${versionId1}`;
const date1 = '1970-01-01T00:00:00.002Z';
const value1 = `{"versionId":"${versionId1}","last-modified":"${date1}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey1, v),
value: value1,
}), FILTER_ACCEPT);
const versionId2 = 'version2';
const versionKey2 = `${masterKey1}${VID_SEP}${versionId2}`;
const date2 = '1970-01-01T00:00:00.001Z';
const value2 = `{"versionId":"${versionId2}","last-modified":"${date2}"}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey2, v),
value: value2,
}), FILTER_ACCEPT);
// orphan delete marker
const masterKey3 = 'key3';
const versionId3 = 'version3';
const versionKey3 = `${masterKey3}${VID_SEP}${versionId3}`;
const date3 = '1970-01-01T00:00:00.000Z';
const value3 = `{"versionId":"${versionId3}","last-modified":"${date3}","isDeleteMarker":true}`;
assert.strictEqual(delimiter.filter({
key: getListingKey(versionKey3, v),
value: value3,
}), FILTER_END);
const expectedResult = {
Contents: [],
NextMarker: masterKey1,
IsTruncated: true,
};
assert.deepStrictEqual(delimiter.result(), expectedResult);
});
});
});

File diff suppressed because it is too large Load Diff

View File

@ -116,7 +116,7 @@ describe('Skip Algorithm', () => {
// Skipping algo params // Skipping algo params
const extension = { const extension = {
filter: () => FILTER_SKIP, filter: () => FILTER_SKIP,
skipping: () => 'entry0', skipping: () => 'entry1',
}; };
const gte = 'some-other-entry'; const gte = 'some-other-entry';
// Setting spy functions // Setting spy functions
@ -138,7 +138,7 @@ describe('Skip Algorithm', () => {
// Skipping algo params // Skipping algo params
const extension = { const extension = {
filter: () => FILTER_SKIP, filter: () => FILTER_SKIP,
skipping: () => ['first-entry-0', 'second-entry-0'], skipping: () => ['first-entry-1', 'second-entry-1'],
}; };
const gte = 'some-other-entry'; const gte = 'some-other-entry';
// Setting spy functions // Setting spy functions
@ -160,7 +160,7 @@ describe('Skip Algorithm', () => {
// Skipping algo params // Skipping algo params
const extension = { const extension = {
filter: () => FILTER_SKIP, filter: () => FILTER_SKIP,
skipping: () => 'entry-0', skipping: () => 'entry-1',
}; };
const gte = 'entry-1'; const gte = 'entry-1';
// Setting spy functions // Setting spy functions

View File

@ -67,6 +67,8 @@ describe('ObjectMD class setters/getters', () => {
['Location', ['location1']], ['Location', ['location1']],
['IsNull', null, false], ['IsNull', null, false],
['IsNull', true], ['IsNull', true],
['IsNull2', null, false],
['IsNull2', true],
['NullVersionId', null, undefined], ['NullVersionId', null, undefined],
['NullVersionId', '111111'], ['NullVersionId', '111111'],
['NullUploadId', null, undefined], ['NullUploadId', null, undefined],
@ -335,6 +337,7 @@ describe('getAttributes static method', () => {
'key': true, 'key': true,
'location': true, 'location': true,
'isNull': true, 'isNull': true,
'isNull2': true,
'nullVersionId': true, 'nullVersionId': true,
'nullUploadId': true, 'nullUploadId': true,
'isDeleteMarker': true, 'isDeleteMarker': true,

View File

@ -70,7 +70,7 @@ describe('MongoClientInterface:delObject', () => {
it('deleteObjectNoVer:: should fail when internalDeleteObject fails', done => { it('deleteObjectNoVer:: should fail when internalDeleteObject fails', done => {
const internalDeleteObjectStub = sinon.stub(client, 'internalDeleteObject') const internalDeleteObjectStub = sinon.stub(client, 'internalDeleteObject')
.callsArgWith(5, errors.InternalError); .callsArgWith(6, errors.InternalError);
client.deleteObjectNoVer(null, 'example-bucket', 'example-object', {}, logger, err => { client.deleteObjectNoVer(null, 'example-bucket', 'example-object', {}, logger, err => {
assert(internalDeleteObjectStub.calledOnce); assert(internalDeleteObjectStub.calledOnce);
assert(err.is.InternalError); assert(err.is.InternalError);
@ -79,7 +79,7 @@ describe('MongoClientInterface:delObject', () => {
}); });
it('deleteObjectNoVer:: should not fail', done => { it('deleteObjectNoVer:: should not fail', done => {
sinon.stub(client, 'internalDeleteObject').callsArgWith(5, null, { ok: 1 }); sinon.stub(client, 'internalDeleteObject').callsArgWith(6, null, { ok: 1 });
client.deleteObjectNoVer(null, 'example-bucket', 'example-object', {}, logger, err => { client.deleteObjectNoVer(null, 'example-bucket', 'example-object', {}, logger, err => {
assert.deepStrictEqual(err, null); assert.deepStrictEqual(err, null);
return done(); return done();
@ -140,7 +140,7 @@ describe('MongoClientInterface:delObject', () => {
}); });
it('deleteObjectVerNotMaster:: should fail when findOneAndDelete fails', done => { it('deleteObjectVerNotMaster:: should fail when findOneAndDelete fails', done => {
sinon.stub(client, 'internalDeleteObject').callsArgWith(5, errors.InternalError); sinon.stub(client, 'internalDeleteObject').callsArgWith(6, errors.InternalError);
client.deleteObjectVerNotMaster(null, 'example-bucket', 'example-object', {}, logger, err => { client.deleteObjectVerNotMaster(null, 'example-bucket', 'example-object', {}, logger, err => {
assert(err.is.InternalError); assert(err.is.InternalError);
return done(); return done();
@ -151,7 +151,7 @@ describe('MongoClientInterface:delObject', () => {
const collection = { const collection = {
updateOne: (filter, update, params, cb) => cb(null), updateOne: (filter, update, params, cb) => cb(null),
}; };
sinon.stub(client, 'internalDeleteObject').callsArg(5); sinon.stub(client, 'internalDeleteObject').callsArg(6);
sinon.stub(client, 'deleteOrRepairPHD').callsFake((...args) => args[6](errors.InternalError)); sinon.stub(client, 'deleteOrRepairPHD').callsFake((...args) => args[6](errors.InternalError));
client.deleteObjectVerMaster(collection, 'example-bucket', 'example-object', {}, logger, err => { client.deleteObjectVerMaster(collection, 'example-bucket', 'example-object', {}, logger, err => {
assert(err.is.InternalError); assert(err.is.InternalError);
@ -163,7 +163,7 @@ describe('MongoClientInterface:delObject', () => {
const collection = { const collection = {
updateOne: (filter, update, params, cb) => cb(null), updateOne: (filter, update, params, cb) => cb(null),
}; };
sinon.stub(client, 'internalDeleteObject').callsArg(5); sinon.stub(client, 'internalDeleteObject').callsArg(6);
sinon.stub(client, 'deleteOrRepairPHD').callsArg(6); sinon.stub(client, 'deleteOrRepairPHD').callsArg(6);
client.deleteObjectVerMaster(collection, 'example-bucket', 'example-object', {}, logger, err => { client.deleteObjectVerMaster(collection, 'example-bucket', 'example-object', {}, logger, err => {
assert.deepStrictEqual(err, undefined); assert.deepStrictEqual(err, undefined);
@ -174,7 +174,7 @@ describe('MongoClientInterface:delObject', () => {
it('deleteOrRepairPHD:: should not fail', done => { it('deleteOrRepairPHD:: should not fail', done => {
sinon.useFakeTimers(); sinon.useFakeTimers();
sinon.stub(client, 'getLatestVersion').callsFake((...args) => args[4](null, { isDeleteMarker: false })); sinon.stub(client, 'getLatestVersion').callsFake((...args) => args[4](null, { isDeleteMarker: false }));
sinon.stub(client, 'internalDeleteObject').callsArg(5); sinon.stub(client, 'internalDeleteObject').callsArg(6);
sinon.stub(client, 'asyncRepair').callsArg(5); sinon.stub(client, 'asyncRepair').callsArg(5);
client.deleteOrRepairPHD({}, 'example-bucket', 'example-object', {}, 'v0', logger, err => { client.deleteOrRepairPHD({}, 'example-bucket', 'example-object', {}, 'v0', logger, err => {
assert.deepStrictEqual(err, null); assert.deepStrictEqual(err, null);
@ -207,11 +207,41 @@ describe('MongoClientInterface:delObject', () => {
const collection = { const collection = {
findOneAndUpdate: sinon.stub().callsArgWith(3, null, {}), findOneAndUpdate: sinon.stub().callsArgWith(3, null, {}),
}; };
client.internalDeleteObject(collection, 'example-bucket', 'example-object', null, logger, err => { client.internalDeleteObject(collection, 'example-bucket', 'example-object', null, null, logger, err => {
assert(err.is.NoSuchKey); assert(err.is.NoSuchKey);
return done(); return done();
}); });
}); });
it('internalDeleteObject:: should directly delete object if params.doesNotNeedOpogUpdate is true', done => {
const collection = {
deleteOne: sinon.stub().returns(Promise.resolve()),
};
const params = {
doesNotNeedOpogUpdate: true,
};
client.internalDeleteObject(collection, 'example-bucket', 'example-object', null, params, logger, err => {
assert.deepEqual(err, null);
assert(collection.deleteOne.calledOnce);
return done();
});
});
it('internalDeleteObject:: should go through the normal flow if params is null', done => {
const findOneAndUpdate = sinon.stub().callsArgWith(3, null, { value: { value: objMD } });
const bulkWrite = sinon.stub().callsArgWith(2, null);
const collection = {
findOneAndUpdate,
bulkWrite,
};
client.internalDeleteObject(collection, 'example-bucket', 'example-object', null, null, logger, err => {
assert.deepEqual(err, null);
assert(findOneAndUpdate.calledOnce);
assert(bulkWrite.calledOnce);
return done();
});
});
// incompatible with 7.x ObjectMD // incompatible with 7.x ObjectMD
it.skip('internalDeleteObject:: should get PHD object with versionId', done => { it.skip('internalDeleteObject:: should get PHD object with versionId', done => {
const findOneAndUpdate = sinon.stub().callsArgWith(3, null, { value: { value: objMD } }); const findOneAndUpdate = sinon.stub().callsArgWith(3, null, { value: { value: objMD } });
@ -223,7 +253,7 @@ describe('MongoClientInterface:delObject', () => {
'value.isPHD': true, 'value.isPHD': true,
'value.versionId': '1234', 'value.versionId': '1234',
}; };
client.internalDeleteObject(collection, 'example-bucket', 'example-object', filter, logger, err => { client.internalDeleteObject(collection, 'example-bucket', 'example-object', filter, null, logger, err => {
assert.deepEqual(err, undefined); assert.deepEqual(err, undefined);
assert(findOneAndUpdate.args[0][0]['value.isPHD']); assert(findOneAndUpdate.args[0][0]['value.isPHD']);
assert.strictEqual(findOneAndUpdate.args[0][0]['value.versionId'], '1234'); assert.strictEqual(findOneAndUpdate.args[0][0]['value.versionId'], '1234');

View File

@ -0,0 +1,342 @@
const assert = require('assert');
const werelogs = require('werelogs');
const logger = new werelogs.Logger('MongoClientInterface', 'debug', 'debug');
const errors = require('../../../../../lib/errors').default;
const sinon = require('sinon');
const MongoClientInterface =
require('../../../../../lib/storage/metadata/mongoclient/MongoClientInterface');
const utils = require('../../../../../lib/storage/metadata/mongoclient/utils');
describe('MongoClientInterface:getObjects', () => {
let client;
beforeAll(done => {
client = new MongoClientInterface({});
return done();
});
afterEach(done => {
sinon.restore();
return done();
});
it('should fail if not an array', done => {
const collection = {
findOne: Promise.resolve({}),
};
sinon.stub(client, 'getCollection').callsFake(() => collection);
client.getObjects('example-bucket', {}, logger, err => {
assert.deepStrictEqual(err, errors.InternalError);
return done();
});
});
it('should fail when getBucketVFormat fails', done => {
const collection = {
findOne: (filter, params, cb) => cb(null, {}),
};
sinon.stub(client, 'getCollection').callsFake(() => collection);
const objects = [{ key: 'example-object', params: { versionId: '1' } }];
sinon.stub(client, 'getBucketVFormat').callsFake((bucketName, log, cb) => cb(errors.InternalError));
client.getObjects('example-bucket', objects, logger, err => {
assert.deepStrictEqual(err, errors.InternalError);
return done();
});
});
it('should fail when find fails', done => {
const objects = [{ key: 'example-object', params: { versionId: '1' } }];
const collection = {
find: () => ({
toArray: (cb) => cb(errors.InternalError),
}),
};
sinon.stub(client, 'getCollection').callsFake(() => collection);
sinon.stub(client, 'getBucketVFormat').callsFake((bucketName, log, cb) => cb(null, 'v0'));
client.getObjects('example-bucket', objects, logger, err => {
assert.deepStrictEqual(err, errors.InternalError);
return done();
});
});
it('should fail when getLatestVersion fails', done => {
const objects = [{ key: 'example-object', params: { versionId: '1' } }];
const collection = {
find: () => ({
toArray: (cb) => cb(errors.InternalError, []),
}),
};
sinon.stub(client, 'getCollection').callsFake(() => collection);
sinon.stub(client, 'getBucketVFormat').callsFake((bucketName, log, cb) => cb(null, 'v0'));
sinon.stub(client, 'getLatestVersion').callsFake((bucketName, data, log, cb) => cb(errors.InternalError));
client.getObjects('example-bucket', objects, logger, err => {
assert.deepStrictEqual(err, errors.InternalError);
return done();
});
});
it('should return empty document if version is set and not found', done => {
const objects = [{ key: 'example-object', params: { versionId: '1' } }];
const doc = {
_id: 'example-key1',
value: {
isPHD: true,
last: true,
},
};
const collection = {
find: () => ({
toArray: (cb) => cb(null, [doc]),
}),
};
const bucketVFormat = 'v0';
sinon.stub(client, 'getCollection').callsFake(() => collection);
sinon.stub(client, 'getBucketVFormat').callsFake((bucketName, log, cb) => cb(null, bucketVFormat));
doc.value.last = true;
sinon.stub(client, 'getLatestVersion').callsFake((c, objName, vFormat, log, cb) => cb(null, doc.value));
client.getObjects('example-bucket', objects, logger, (err, res) => {
assert.deepStrictEqual(err, null);
assert.deepStrictEqual(res[0], {
doc: null,
key: utils.formatVersionKey(objects[0].key, objects[0].params.versionId, bucketVFormat),
versionId: objects[0].params.versionId,
err: errors.NoSuchKey,
});
return done();
});
});
it('should return empty document if version is not set and not found', done => {
const objects = [{ key: 'example-object', params: { } }];
const doc = {
_id: 'example-key1',
value: {
isPHD: false,
last: true,
},
};
const collection = {
find: () => ({
toArray: (cb) => cb(null, [doc]),
}),
};
sinon.stub(client, 'getCollection').callsFake(() => collection);
sinon.stub(client, 'getBucketVFormat').callsFake((bucketName, log, cb) => cb(null, 'v0'));
doc.value.last = true;
sinon.stub(client, 'getLatestVersion').callsFake((c, objName, vFormat, log, cb) => cb(null, doc.value));
client.getObjects('example-bucket', objects, logger, (err, res) => {
assert.deepStrictEqual(err, null);
assert.deepStrictEqual(res[0], {
doc: doc.value,
key: objects[0].key,
versionId: undefined,
err: null,
});
return done();
});
});
it('should return latest version if version is found and master is PHD', done => {
const objects = [{ key: 'example-object', params: { } }];
const doc = {
_id: 'example-key1',
value: {
isPHD: true,
},
};
const collection = {
find: () => ({
toArray: (cb) => cb(null, [doc]),
}),
};
const bucketVFormat = 'v0';
sinon.stub(client, 'getCollection').callsFake(() => collection);
sinon.stub(client, 'getBucketVFormat').callsFake((bucketName, log, cb) => cb(null, bucketVFormat));
sinon.stub(client, 'getLatestVersion').callsFake((c, objName, vFormat, log, cb) => cb(null, doc.value));
client.getObjects('example-bucket', objects, logger, (err, res) => {
assert.deepStrictEqual(err, null);
assert.deepStrictEqual(res, [{
doc: doc.value,
key: objects[0].key,
versionId: undefined,
err: null,
}]);
return done();
});
});
it('should return master', done => {
const objects = [{ key: 'example-object', params: {} }];
const doc = {
_id: 'example-key1',
value: {
isPHD: false,
},
};
const collection = {
find: () => ({
toArray: (cb) => cb(null, [doc]),
}),
};
const bucketVFormat = 'v0';
sinon.stub(client, 'getCollection').callsFake(() => collection);
sinon.stub(client, 'getBucketVFormat').callsFake((bucketName, log, cb) => cb(null, bucketVFormat));
doc.value.last = true;
sinon.stub(client, 'getLatestVersion').callsFake((c, objName, vFormat, log, cb) => cb(null, doc.value));
client.getObjects('example-bucket', objects, logger, (err, res) => {
assert.deepStrictEqual(err, null);
assert.deepStrictEqual(res[0], {
doc: doc.value,
key: objects[0].key,
versionId: undefined,
err: null,
});
return done();
});
});
it('should return many objects', done => {
const N = 5;
const objects = [];
const bucketVFormat = 'v0';
for (let i = 1; i <= N; i++) {
objects.push({ key: `example-object-${i}`, params: { versionId: `${i}` } });
}
const docTemplate = {
_id: 'example-key',
value: {
isPHD: false,
},
};
const collection = {
find: () => ({
toArray: (cb) => {
const docs = [];
for (let i = 1; i <= N; i++) {
const newDoc = JSON.parse(JSON.stringify(docTemplate));
newDoc._id = utils.formatVersionKey(`example-object-${i}`, `${i}`, bucketVFormat);
docs.push(newDoc);
}
cb(null, docs);
},
}),
};
sinon.stub(client, 'getCollection').callsFake(() => collection);
sinon.stub(client, 'getBucketVFormat').callsFake((bucketName, log, cb) => cb(null, bucketVFormat));
client.getObjects('example-bucket', objects, logger, (err, res) => {
assert.deepStrictEqual(err, null);
const expectedResults = [];
for (let i = 0; i < N; i++) {
expectedResults.push({
doc: docTemplate.value,
key: utils.formatVersionKey(objects[i].key, objects[i].params.versionId),
versionId: `${i + 1}`,
err: null,
});
}
assert.deepStrictEqual(res, expectedResults);
return done();
});
});
it('should return multiple objects and null documents if one object is not found', done => {
const N = 5;
const objects = [];
const bucketVFormat = 'v0';
for (let i = 1; i <= N; i++) {
objects.push({ key: `example-object-${i}`, params: { versionId: `${i}` } });
}
const docTemplate = {
_id: 'example-key',
value: {
isPHD: false,
},
};
const collection = {
find: () => ({
toArray: (cb) => {
const docs = [];
for (let i = 1; i < N; i++) {
const newDoc = JSON.parse(JSON.stringify(docTemplate));
newDoc._id = utils.formatVersionKey(`example-object-${i}`, `${i}`, bucketVFormat);
docs.push(newDoc);
}
cb(null, docs);
},
}),
};
sinon.stub(client, 'getCollection').callsFake(() => collection);
sinon.stub(client, 'getBucketVFormat').callsFake((bucketName, log, cb) => cb(null, bucketVFormat));
client.getObjects('example-bucket', objects, logger, (err, res) => {
assert.deepStrictEqual(err, null);
const expectedResults = [];
for (let i = 0; i < N; i++) {
expectedResults.push({
doc: i === N - 1 ? null : docTemplate.value,
key: utils.formatVersionKey(objects[i].key, objects[i].params.versionId),
versionId: `${i + 1}`,
err: i === N - 1 ? errors.NoSuchKey : null,
});
}
assert.deepStrictEqual(res, expectedResults);
return done();
});
});
it('should return multiple objects and errors if one object latest version retrieval fails', done => {
const N = 5;
const objects = [];
const bucketVFormat = 'v0';
for (let i = 1; i <= N; i++) {
objects.push({ key: `example-object-${i}`, params: { } });
}
const docTemplate = {
_id: 'example-key',
value: {
isPHD: false,
},
};
const collection = {
find: () => ({
toArray: (cb) => {
const docs = [];
for (let i = 1; i < N; i++) {
const newDoc = JSON.parse(JSON.stringify(docTemplate));
newDoc._id = `example-object-${i}`;
docs.push(newDoc);
}
cb(null, docs);
},
}),
};
sinon.stub(client, 'getCollection').callsFake(() => collection);
sinon.stub(client, 'getBucketVFormat').callsFake((bucketName, log, cb) => cb(null, bucketVFormat));
sinon.stub(client, 'getLatestVersion').callsFake((c, objName, vFormat, log, cb) => {
if (objName === objects[N - 1].key) {
return cb(errors.InternalError);
}
return cb(null, docTemplate.value);
});
client.getObjects('example-bucket', objects, logger, (err, res) => {
assert.deepStrictEqual(err, null);
const expectedResults = [];
for (let i = 0; i < N; i++) {
expectedResults.push({
doc: i === N - 1 ? null : docTemplate.value,
key: objects[i].key,
versionId: undefined,
err: i === N - 1 ? errors.InternalError : null,
});
}
assert.deepStrictEqual(res, expectedResults);
return done();
});
});
});

View File

@ -99,7 +99,7 @@ describe('MongoClientInterface:deleteObjectWithCond', () => {
sinon.stub(client, 'getCollection').callsFake(() => {}); sinon.stub(client, 'getCollection').callsFake(() => {});
sinon.stub(client, 'getBucketVFormat').callsFake((bucketName, log, cb) => cb(null)); sinon.stub(client, 'getBucketVFormat').callsFake((bucketName, log, cb) => cb(null));
sinon.stub(utils, 'translateConditions').callsFake(() => null); sinon.stub(utils, 'translateConditions').callsFake(() => null);
sinon.stub(client, 'internalDeleteObject').callsArgWith(5, errors.InternalError); sinon.stub(client, 'internalDeleteObject').callsArgWith(6, errors.InternalError);
client.deleteObjectWithCond('example-bucket', 'example-object', {}, logger, err => { client.deleteObjectWithCond('example-bucket', 'example-object', {}, logger, err => {
assert(err.is.InternalError); assert(err.is.InternalError);
return done(); return done();

File diff suppressed because it is too large Load Diff

View File

@ -1200,6 +1200,11 @@
"@jridgewell/resolve-uri" "^3.0.3" "@jridgewell/resolve-uri" "^3.0.3"
"@jridgewell/sourcemap-codec" "^1.4.10" "@jridgewell/sourcemap-codec" "^1.4.10"
"@js-sdsl/ordered-set@^4.4.2":
version "4.4.2"
resolved "https://registry.yarnpkg.com/@js-sdsl/ordered-set/-/ordered-set-4.4.2.tgz#ab857eb63cf358b5a0f74fdd458b4601423779b7"
integrity sha512-ieYQ8WlBPKYzEo81H3q0DFbd8WtFRXXABb4+vRCF0AO3WWtJZFxYvRGdipUXGrd6tlSySmqhcPuO3J6SCodCxg==
"@npmcli/fs@^1.0.0": "@npmcli/fs@^1.0.0":
version "1.1.1" version "1.1.1"
resolved "https://registry.yarnpkg.com/@npmcli/fs/-/fs-1.1.1.tgz#72f719fe935e687c56a4faecf3c03d06ba593257" resolved "https://registry.yarnpkg.com/@npmcli/fs/-/fs-1.1.1.tgz#72f719fe935e687c56a4faecf3c03d06ba593257"