Merge pull request #1 from leanix/use-keys

TNDO-2893 Fork redisdown and switch to using individual keys for values

LICENSE

@@ -1,4 +1,5 @@
 Copyright (c) 2014 Sutoiku, Inc.
+Copyright (c) 2021 LeanIX GmbH
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
 documentation files (the "Software"), to deal in the Software without restriction, including without limitation the

README.md

@@ -1,49 +1,43 @@
-# RedisDOWN [![Build Status](https://travis-ci.org/hmalphettes/redisdown.svg?branch=master)](https://travis-ci.org/hmalphettes/redisdown)
+# redis-key-down
 
 Redis backend for [LevelUP](https://github.com/rvagg/node-levelup)
 
 Requirements:
 * redis-2.8 or more recent.
 * node-12.x
 
-Uses a sorted-set to order the keys and a hash to store the values.
+This is a fork of the excellent [redisdown](https://github.com/hmalphettes/redisdown).
+Like the original, it uses a sorted-set to order the keys, but the values are stored
+as individual keys to make sure that key eviction will not flush all the data when
+using Redis as an [LRU cache](https://redis.io/topics/lru-cache).
 
 Fetches the ordered key value pairs during iterations with a single redis lua call.
 
-[Abstract-LevelDOWN](https://github.com/rvagg/abstract-leveldown) testsuite is green
-except for the ['implicit iterator snapshot'](https://github.com/hmalphettes/redisdown/issues/10).
-
-# Warning: data migration from redisdown-v0.1.9
-redisdown > v0.1.10 no longer JSON encode/decode itself. Levelup take care of that.
-JSON Data written in v0.1.9 is not correctly decoded back to a javascript object in v0.1.10 and above.
-
-This change was introduced to support binary values in redisdown and escaped the fact that it was breaking backward compatibility for the data.
-
-Workaround: https://github.com/hmalphettes/redisdown/issues/24#issuecomment-193076281
+[Abstract-LevelDOWN](https://github.com/rvagg/abstract-leveldown) testsuite is green.
 
 # Example
 
 Copied and pasted from the levelup documentation.
-Added the db option when creating the db to use redisdown.
+Added the db option when creating the db to use redis-key-down.
 
 ```
 var levelup = require('levelup')
-var redisdown = require('redisdown')
+var redisKeyDown = require('redis-key-down')
 
 // 1) Create our database, supply location and options.
 //    This will create or open the underlying LevelDB store.
-var db = levelup('mydb', { db: redisdown, host: 'localhost', port: 6379 })
+var db = levelup('mydb', { db: redisKeyDown, host: 'localhost', port: 6379 })
 
 // If you use sentinel/cluster mode, you must use a single slot to store the values thanks to a `{hash}`
-//var db = levelup('{thehash}path', { db: redisdown });
+//var db = levelup('{thehash}path', { db: redisKeyDown });
 
 // if you already have a redis client
-//var db = levelup('mydb', { db: redisdown, redis: redisClient })
+//var db = levelup('mydb', { db: redisKeyDown, redis: redisClient })
 
 // if you use an URL environment variable
-//var db = levelup('mydb', { db: redisdown, url: process.env.REDIS_URL })
+//var db = levelup('mydb', { db: redisKeyDown, url: process.env.REDIS_URL })
 // if you use Redis Cloud on Heroku
-//var db = levelup('mydb', { db: redisdown, url: process.env.REDISCLOUD_URL })
+//var db = levelup('mydb', { db: redisKeyDown, url: process.env.REDISCLOUD_URL })
 
 // 2) put a key & value
 db.put('name', 'LevelUP', function (err) {
@@ -62,33 +56,33 @@ db.put('name', 'LevelUP', function (err) {
 # API
 --------------------------------------------------------
 <a name="ctor"></a>
-### redisdown(location)
-<code>redisdown(location)</code> returns a new **RedisDOWN** instance. `location` is a String pointing at the root namespace of the data in redis.
+### redisKeyDown(location)
+<code>redisKeyDown(location)</code> returns a new **RedisDOWN** instance. `location` is a String pointing at the root namespace of the data in redis.
 
-* `location+':h'` is the hash where the values are stored.
+* `location+'$'+key` is where the values are stored.
 * `location+':z'` is the set where the keys are sorted.
 
 --------------------------------------------------------
-<a name="redisdown_open"></a>
-### redisdown#open([options, ]callback)
+<a name="redisKeyDown_open"></a>
+### redisKeyDown#open([options, ]callback)
 <code>open()</code> is an instance method on an existing database object.
 
 options is a hash that is passed to the redis library to create a redis client:
 
 * `highWaterMark` number of values to fetch in one redis call for iteration. Defaults to 256.
 * `port` redis port. Defaults to '127.0.0.1'
 * `host` redis host. Defaults to 6379
-* `redis` already configured redis client. redisDown will not open or close it. host and port and all other redis options are ignored.
+* `redis` already configured redis client. redisKeyDown will not open or close it. host and port and all other redis options are ignored.
 * Other options: https://github.com/mranney/node_redis#rediscreateclientport-host-options
 
 -----------------------------------
-<a name="redisdown_destroy"></a>
-### redisdown.destroy(location, [options, ]callback)
+<a name="redisKeyDown_destroy"></a>
+### redisKeyDown.destroy(location, [options, ]callback)
 <code>destroy(location)</code> is used to completely delete all data in redis related to the location.
 
 -----------------------------------
-<a name="redisdown_batch_prefixes"></a>
-### redisdown#batch([{type: 'put', key: 'foo1', value: 'bar1' [, prefix: 'subsection']}, ...])
+<a name="redisKeyDown_batch_prefixes"></a>
+### redisKeyDown#batch([{type: 'put', key: 'foo1', value: 'bar1' [, prefix: 'subsection']}, ...])
 <code>batch()</code> supports an exra property `prefix` to store/retrieve/delete a key in a specific namespace of the redis DB.
 It is useful to support sublevel-ish batch operations: https://github.com/dominictarr/level-sublevel#batches
 and is well supported by redis.
@@ -103,12 +97,3 @@ Passing a levelup instance is demonstrated here: https://github.com/hmalphettes/
 
 The script will install the extra required dependencies.
 It works for me.
-
-# LICENSE
-redisdown is freely distributable under the term of the MIT License.
-Copyright: Sutoiku Inc 2014.
-
-If you need something different, let me know.
-
-# HELP Wanted
-- Collation: do we need to worry about this?

index.js

@@ -110,8 +110,16 @@ RedisDown.prototype._open = function (options, callback) {
     }
 };
 
+RedisDown.prototype._serializeKey = function (key) {
+    return toBufferOrString(key);
+}
+
+RedisDown.prototype._serializeValue = function (value) {
+    return toBufferOrString(value);
+}
+
 RedisDown.prototype._get = function (key, options, callback) {
-    this.db.hget(this.location + ':h', cleanKey(key), function (e, v) {
+    this.db.get(valueKey(this.location, key), function (e, v) {
         if (e) {
             return setImmediate(function callNext() {
                 return callback(e);
@@ -134,11 +142,11 @@ RedisDown.prototype._get = function (key, options, callback) {
     });
 };
 
-RedisDown.prototype._put = function (key, rawvalue, opt, callback) {
-    if (typeof rawvalue === 'undefined' || rawvalue === null) {
-        rawvalue = '';
+RedisDown.prototype._put = function (key, value, opt, callback) {
+    if (typeof value === 'undefined' || value === null) {
+        value = '';
     }
-    this.__exec(this.__appendPutCmd([], key, rawvalue.toString()), callback);
+    this.__exec(this.__appendPutCmd([], key, value), callback);
 };
 
 RedisDown.prototype._del = function (key, opt, cb) {
@@ -166,16 +174,14 @@ RedisDown.prototype.__getPrefix = function (prefix) {
 
 RedisDown.prototype.__appendPutCmd = function (commandList, key, value, prefix) {
     var resolvedPrefix = this.__getPrefix(prefix);
-    key = cleanKey(key);
-    commandList.push(['hset', resolvedPrefix + ':h', key, value === undefined ? '' : value]);
+    commandList.push(['set', valueKey(resolvedPrefix, key), value === undefined ? '' : value]);
     commandList.push(['zadd', resolvedPrefix + ':z', 0, key]);
     return commandList;
 };
 
 RedisDown.prototype.__appendDelCmd = function (commandList, key, prefix) {
-    key = cleanKey(key);
     var resolvedPrefix = this.__getPrefix(prefix);
-    commandList.push(['hdel', resolvedPrefix + ':h', key]);
+    commandList.push(['del', valueKey(resolvedPrefix, key)]);
     commandList.push(['zrem', resolvedPrefix + ':z', key]);
     return commandList;
 };
@@ -232,7 +238,7 @@ RedisDown.destroy = function (location, options, callback) {
     }
     var sanitizedLocation = sanitizeLocation(location);
     var client = redisLib.createClient(options.port, options.host, options);
-    client.del(sanitizedLocation + ':h', sanitizedLocation + ':z', function (e) {
+    client.del(sanitizedLocation + ':z', function (e) {
         client.quit();
         callback(e);
     });
@@ -246,7 +252,7 @@ RedisDown.prototype.destroy = function (doClose, callback) {
         doClose = true;
     }
     var self = this;
-    this.db.del(this.location + ':h', this.location + ':z', function (e) {
+    this.db.del(this.location + ':z', function (e) {
         if (doClose) {
             self.close(callback);
         } else {
@@ -261,7 +267,7 @@ RedisDown.prototype.destroy = function (doClose, callback) {
  * when the identifier is identical, it is safe to reuse the same client.
  */
 function _makeRedisId(location, options) {
-    var redisIdOptions = ['host', 'port',
+    var redisIdOptions = ['host', 'port', 'tls', 'password',
         'parser', 'return_buffers', 'detect_buffers', 'socket_nodelay', 'no_ready_check',
         'enable_offline_queue', 'retry_max_delay', 'connect_timeout', 'max_attempts'
     ];
@@ -300,14 +306,23 @@ function sanitizeLocation(location) {
     return location;
 }
 
-function cleanKey(key) {
+function toBufferOrString(key) {
     if (Buffer.isBuffer(key)) {
         return key;
     } else {
         return key.toString();
     }
 }
 
+function valueKey(location, key) {
+    var prefix = location + '$';
+    if (Buffer.isBuffer(key)) {
+        return Buffer.concat([Buffer.from(prefix), key]);
+    } else {
+        return prefix + key;
+    }
+}
+
 RedisDown.reset = function (callback) {
     for (var k in RedisDown.dbs) {
         if (RedisDown.dbs.hasOwnProperty(k)) {

luascripts/zhpairs.lua

@@ -9,10 +9,14 @@ local keys = redis.call('zrangebylex', KEYS[1]..':z', unpack(ARGV))
 if #keys == 0 then
   return keys
 end
-local values = redis.call('hmget',KEYS[1]..':h',unpack(keys))
+local prefixkeys = {}
+for i,v in ipairs(keys) do
+  prefixkeys[i] = KEYS[1]..'$'..v
+end
+local values = redis.call('mget', unpack(prefixkeys))
 local result = {}
 for i,v in ipairs(keys) do
   result[i*2-1] = v
   result[i*2] = values[i]
 end
-return result
+return result

luascripts/zhrevpairs.lua

@@ -9,7 +9,11 @@ local keys = redis.call('zrevrangebylex', KEYS[1]..':z', unpack(ARGV))
 if #keys == 0 then
   return keys
 end
-local values = redis.call('hmget',KEYS[1]..':h',unpack(keys))
+local prefixkeys = {}
+for i,v in ipairs(keys) do
+  prefixkeys[i] = KEYS[1]..'$'..v
+end
+local values = redis.call('mget', unpack(prefixkeys))
 local result = {}
 for i,v in ipairs(keys) do
   result[i*2-1] = v

luascripts/zhrevvalues-lk.lua

@@ -10,6 +10,10 @@ local keys = redis.call('zrevrangebylex', KEYS[1]..':z', unpack(ARGV))
 if #keys == 0 then
   return keys
 end
-local values = redis.call('hmget', KEYS[1]..':h', unpack(keys))
+local prefixkeys = {}
+for i,v in ipairs(keys) do
+  prefixkeys[i] = KEYS[1]..'$'..v
+end
+local values = redis.call('mget', unpack(prefixkeys))
 values[#values+1] = keys[#keys]
 return values

luascripts/zhvalues-lk.lua

@@ -10,6 +10,10 @@ local keys = redis.call('zrangebylex', KEYS[1]..':z', unpack(ARGV))
 if #keys == 0 then
   return keys
 end
-local values = redis.call('hmget', KEYS[1]..':h', unpack(keys))
+local prefixkeys = {}
+for i,v in ipairs(keys) do
+  prefixkeys[i] = KEYS[1]..'$'..v
+end
+local values = redis.call('mget', unpack(prefixkeys))
 values[#values+1] = keys[#keys]
 return values

package.json

@@ -1,10 +1,10 @@
 {
-  "name": "redisdown",
-  "description": "redis adapter for levelup",
-  "version": "0.1.13",
+  "name": "redis-key-down",
+  "description": "redis adapter for levelup that uses individual keys for values",
+  "version": "0.1.0",
   "repository": {
     "type": "git",
-    "url": "https://github.com/hmalphettes/redisdown.git"
+    "url": "https://github.com/leanix/redis-key-down.git"
   },
   "main": "index.js",
   "scripts": {