Merge pull request #8 from hiett/sh/response-encoding

Response encoding, and automated testing with @upstash/redis

.github/workflows/test.yml

@@ -0,0 +1,37 @@
+name: Test @upstash/redis compatability
+on:
+  workflow_dispatch:
+  push:
+    paths:
+      - 'lib/**'
+  schedule:
+    - cron: '0 12 * * *'
+
+env:
+  SRH_TOKEN: example_token
+
+jobs:
+  container-job:
+    runs-on: ubuntu-latest
+    container: denoland/deno
+    services:
+      redis:
+        image: redis/redis-stack-server:6.2.6-v6 # 6.2 is the Upstash compatible Redis version
+      srh:
+        image: hiett/serverless-redis-http:0.0.5-alpha
+        env:
+          SRH_MODE: env
+          SRH_TOKEN: ${{ env.SRH_TOKEN }}
+          SRH_CONNECTION_STRING: redis://redis:6379
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+        with:
+          repository: upstash/upstash-redis
+
+      - name: Run @upstash/redis Test Suite
+        run: deno test -A ./pkg
+        env:
+          UPSTASH_REDIS_REST_URL: http://srh:80
+          UPSTASH_REDIS_REST_TOKEN: ${{ env.SRH_TOKEN }}

.gitignore

@@ -29,6 +29,4 @@ srh-*.tar
 
 *.iml
 
-srh-config/
-
-example/
+srh-config/

HOW_TO_BUILD.md

@@ -0,0 +1,7 @@
+## Building the Docker image
+
+To build both an amd64 image and an arm64 image, on an M1 Mac:
+
+```
+docker buildx build --platform linux/amd64,linux/arm64 --push -t hiett/serverless-redis-http:0.0.5-alpha
+```

README.md

@@ -1,102 +1,145 @@
-# SRH: Serverless Redis HTTP
+# Serverless Redis HTTP (SRH)
 
----
+A Redis proxy and connection pooler that uses HTTP rather than the Redis binary protocol.\
+The aim of this project is to be entirely compatible with Upstash, and work with any Upstash supported Redis version.
 
-**TLDR: If you want to run a local Upstash-compatible HTTP layer in front of your Redis:**
+Use cases for SRH:
+- For usage in your CI pipelines, creating Upstash databases is tedious, or you have lots of parallel runs.
+    - See [Using in GitHub Actions](#in-github-actions) on how to quickly get SRH setup for this context.
+- For usage inside of Kubernetes, or any network whereby the Redis server is not exposed to the internet.
+    - See [Using in Docker Compose](#via-docker-compose) for the various setup options directly using the Docker Container.
+- For local development environments, where you have a local Redis server running, or require offline access.
+    - See [Using the Docker Command](#via-docker-command), or [Using Docker Compose](#via-docker-compose).
 
-0) Have a locally running Redis instance - in this example bound to the default port 6379
-1) create a json file called tokens.json in a folder called srh-config (`srh-config/tokens.json`)
-2) paste this in:
-  ```json
-{
-    "example_token": {
-        "srh_id": "some_unique_identifier",
-        "connection_string": "redis://localhost:6379",
-        "max_connections": 3
-    } 
-}
-  ```
-3) Run this command:
-`docker run -it -d -p 8079:80 --name srh --mount type=bind,source=$(pwd)/srh-config/tokens.json,target=/app/srh-config/tokens.json hiett/serverless-redis-http:latest`
-4) Set this as your Upstash configuration
-```js
+## Differences between Upstash and Redis to note
+SRH tests are ran nightly against the `@upstash/redis` JavaScript package. However, there are some minor differences between Upstash's implementation of Redis and the official Redis code.
+
+- The `UNLINK` command will not throw an error when 0 keys are given to it. In Redis, and as such SRH, an error will be thrown.
+- In the `ZRANGE` command, in Upstash you are not required to provide `BYSCORE` or `BYLEX` in order to use the `LIMIT` argument. With Redis/SRH, this will throw an error if not provided.
+- The Upstash implementation of `RedisJSON` contains a number of subtle differences in what is returned in responses. For this reason, **it is not advisable to use SRH with Redis Stack if you are testing your Upstash implementation that uses JSON commands**. If you don't use any JSON commands, then all is good :)
+- **SRH does not implement commands via paths, or accepting the token via a query param**. Only the body method is implemented, which the `@upstash/redis` SDK uses.
+
+### Similarities to note:
+
+Pipelines and Transaction endpoints are also implemented, also using the body data only. You can read more about the RestAPI here: [Upstash Docs on the Rest API](https://docs.upstash.com/redis/features/restapi)
+
+Response encoding is also fully implemented. This is enabled by default by the `@upstash/redis` SDK. You can read more about that here: [Upstash Docs on Hashed Responses](https://docs.upstash.com/redis/sdks/javascriptsdk/troubleshooting#hashed-response)
+
+## How to use with the `@upstash/redis` SDK
+Simply set the REST URL and token to where the SRH instance is running. For example:
+```ts
 import {Redis} from '@upstash/redis';
 
 export const redis = new Redis({
     url: "http://localhost:8079",
     token: "example_token",
-    responseEncoding: false, // IMPORTANT: Upstash has recently added response encoding, but SRH does not support it yet.
 });
 ```
----
 
-A Redis connection pooler for serverless applications. This allows your serverless functions to talk to Redis via HTTP,
-while also not having to worry about the Redis max connection limits.
-
-The idea is you host this alongside your Redis server, to minimise latency. Your serverless functions can then talk to 
-this via HTTP.
-
-## Features
-- Allows you to talk to redis via HTTP
-- Pools redis connections
-- Automatically kills redis connections after inactivity
-- Supports multiple redis instances, and you can configure unique tokens for each
-- Fully supports the `@upstash/redis` TypeScript library.
+# Setting up SRH
+## Via Docker command
+If you have a locally running Redis server, you can simply start an SRH container that connects to it.
+In this example, SRH will be running on port `8080`.
+
+```bash
+docker run \
+    -it -d -p 8080:80 --name srh \
+    -e SRH_MODE=env \
+    -e SRH_TOKEN=your_token_here \
+    -e SRH_CONNECTION_STRING="redis://your_server_here:6379" \
+    hiett/serverless-redis-http:latest
+```
 
-## Client usage
-This will not work with regular Redis clients, as it is over HTTP and not the redis protocol.
-However, to try and keep the project as "standardised" as possible, you can use the `@upstash/redis` TypeScript library.
-You can read about it here: [Upstash Redis GitHub](https://github.com/upstash/upstash-redis)
+## Via Docker Compose
+If you wish to run in Kubernetes, this should contain all the basics would need to set that up. However, be sure to read the Configuration Options, because you can create a setup whereby multiple Redis servers are proxied.
+```yml
+version: '3'
+services:
+  redis:
+    image: redis
+    ports:
+      - '6379:6379'
+  serverless-redis-http:
+    ports:
+      - '8079:80'
+    image: hiett/serverless-redis-http:latest
+    environment:
+      SRH_MODE: env
+      SRH_TOKEN: example_token
+      SRH_CONNECTION_STRING: 'redis://redis:6379' # Using `redis` hostname since they're in the same Docker network.
+```
 
-Soon I will add specific documentation for the endpoints so you can implement clients in other languages.
+## In GitHub Actions
+
+SRH works nicely in GitHub Actions because you can run it as a container in a job's services. Simply start a Redis server, and then
+SRH alongside it. You don't need to worry about a race condition of the Redis instance not being ready, because SRH doesn't create a Redis connection until the first command comes in.
+
+```yml
+name: Test @upstash/redis compatability
+on:
+  push:
+  workflow_dispatch:
+
+env:
+  SRH_TOKEN: example_token
+
+jobs:
+  container-job:
+    runs-on: ubuntu-latest
+    container: denoland/deno
+    services:
+      redis:
+        image: redis/redis-stack-server:6.2.6-v6 # 6.2 is the Upstash compatible Redis version
+      srh:
+        image: hiett/serverless-redis-http:latest
+        env:
+          SRH_MODE: env # We are using env mode because we are only connecting to one server.
+          SRH_TOKEN: ${{ env.SRH_TOKEN }}
+          SRH_CONNECTION_STRING: redis://redis:6379
+
+    steps:
+      # You can place your normal testing steps here. In this example, we are running SRH against the upstash/upstash-redis test suite.
+      - name: Checkout code
+        uses: actions/checkout@v3
+        with:
+          repository: upstash/upstash-redis
+
+      - name: Run @upstash/redis Test Suite
+        run: deno test -A ./pkg
+        env:
+          UPSTASH_REDIS_REST_URL: http://srh:80
+          UPSTASH_REDIS_REST_TOKEN: ${{ env.SRH_TOKEN }}
+```
 
-## Installation
-You have to options to run this:
-- Via docker: `docker pull hiett/serverless-redis-http:latest` [Docker Hub link](https://hub.docker.com/r/hiett/serverless-redis-http)
-- Via elixir: `(clone this repo)` -> `mix deps.get` -> `iex -S mix`
+# Configuration Options
 
-If you are running via Docker, you will need to mount the configuration file to `/app/srh-config/tokens.json`.\
-An example of a run command is the following:
+SRH works with multiple Redis servers, and can pool however many connections you wish it to. It will shut down un-used pools after 15 minutes of inactivity. Upon the next command, it will re-build the pool.
 
-`docker run -it -d -p 8080:80 --name srh --mount type=bind,source=$(pwd)/srh-config/tokens.json,target=/app/srh-config/tokens.json hiett/serverless-redis-http:latest`
+## Connecting to multiple Redis servers at the same time
 
-*Note that it is running on port 80*
+The examples above use environment variables in order to tell SRH which Redis server to connect to. However, you can also use a configuration JSON file, which lets you create as many connections as you wish. The token provided in each request will decide which pool is used.
 
-To configure Redis targets:\
-Create a file: `srh-config/tokens.json`
+Create a JSON file, in this example called `tokens.json`:
 ```json
 {
     "example_token": {
         "srh_id": "some_unique_identifier",
         "connection_string": "redis://localhost:6379",
         "max_connections": 3
-    } 
+    }
 }
 ```
+You can provide as many entries to the base object as you wish, and configure the number of max connections per pool. The `srh_id` is used internally to keep track of instances. It can be anything you want.
 
-### Docker Compose
-You'll want the above `tokens.json` file but use this as your connection string:
-```json
-"connection_string": "redis://redis:6379"
-```
-docker-compose.yaml
-```yaml
-version: '3'
-services:
-  redis:
-    image: redis
-    ports:
-      - '6379:6379'
-  serverless-redis-http:
-    ports:
-      - '8079:80'
-    image: hiett/serverless-redis-http:latest
-    volumes:
-      - ./path/to/tokens.json:/app/srh-config/tokens.json
-```
+Once you have created this, mount it to the docker container to the `/app/srh-config/tokens.json` file. Here is an example docker command:
+
+`docker run -it -d -p 8079:80 --name srh --mount type=bind,source=$(pwd)/tokens.json,target=/app/srh-config/tokens.json hiett/serverless-redis-http:latest`
+
+## Environment Variables
 
-Notes: 
-- Srh_id can be anything you want, as long as it's a string, and unique.
-- `max_connections` is the maximum number of connections for the pool.
-  - If there is inactivity, the pool will kill these connections. They will only be open while the pool is alive. The pool will re-create these connections when commands come in.
-- You can add more redis instances to connect to by adding more tokens and connection configurations. Based on the header in each request, the correct pool/connection info will be used.
+| Name | Default Value | Notes |
+| ---- | ------------- | ----- |
+| SRH_MODE | `file` | Can be `env` or `file`. If `file`, see [Connecting to multiple Redis servers](#connecting-to-multiple-redis-servers-at-the-same-time). If set to `env`, you are required to provide the following environment variables: |
+| SRH_TOKEN | `<required if SRH_MODE = env>` | Set the token that the Rest API will require |
+| SRH_CONNECTION_STRING | `<required if SRH_MODE = env>` | Sets the connection string to the Redis server. |
+| SRH_MAX_CONNECTIONS | `3` | Only used if `SRH_MODE=env`.

config/test.exs

@@ -0,0 +1 @@
+import Config

example/.gitignore

@@ -0,0 +1,2 @@
+node_modules/
+dist/

example/package.json

@@ -0,0 +1,19 @@
+{
+  "name": "srh-example",
+  "version": "1.0.0",
+  "main": "index.js",
+  "author": "Scott Hiett",
+  "license": "MIT",
+  "private": false,
+  "scripts": {
+    "start": "ts-node src/index.ts"
+  },
+  "dependencies": {
+    "@upstash/redis": "^1.20.2"
+  },
+  "devDependencies": {
+    "@types/node": "^18.15.11",
+    "ts-node": "^10.9.1",
+    "typescript": "^5.0.2"
+  }
+}

example/src/index.ts

@@ -0,0 +1,39 @@
+import {Redis} from "@upstash/redis";
+
+const redis = new Redis({
+  // The URL of the SRH instance
+  url: "http://127.0.0.1:8080",
+
+  // The token you defined in tokens.json
+  token: "example_token",
+
+  // Response encoding is supported (this is enabled by default)
+  responseEncoding: true,
+});
+
+(async () => {
+  await redis.set("foo", "bar");
+  const value = await redis.get("foo");
+  console.log(value);
+
+  // Run a pipeline operation
+  const pipelineResponse = await redis.pipeline()
+    .set("amazing-key", "bar")
+    .get("amazing-key")
+    .del("amazing-other-key")
+    .del("random-key-that-doesnt-exist")
+    .srandmember("random-key-that-doesnt-exist")
+    .sadd("amazing-set", "item1", "item2", "item3", "bar", "foo", "example")
+    .smembers("amazing-set")
+    .get("foo")
+    .exec();
+
+  console.log(pipelineResponse);
+
+  const multiExecResponse = await redis.multi()
+    .set("example", "value")
+    .get("example")
+    .exec();
+
+  console.log(multiExecResponse);
+})();

example/srh-config.json

@@ -0,0 +1,7 @@
+{
+  "example_token": {
+    "srh_id": "some_unique_identifier",
+    "connection_string": "redis://redis:6379",
+    "max_connections": 3
+  }
+}

example/tsconfig.json

@@ -0,0 +1,109 @@
+{
+  "compilerOptions": {
+    /* Visit https://aka.ms/tsconfig to read more about this file */
+
+    /* Projects */
+    // "incremental": true,                              /* Save .tsbuildinfo files to allow for incremental compilation of projects. */
+    // "composite": true,                                /* Enable constraints that allow a TypeScript project to be used with project references. */
+    // "tsBuildInfoFile": "./.tsbuildinfo",              /* Specify the path to .tsbuildinfo incremental compilation file. */
+    // "disableSourceOfProjectReferenceRedirect": true,  /* Disable preferring source files instead of declaration files when referencing composite projects. */
+    // "disableSolutionSearching": true,                 /* Opt a project out of multi-project reference checking when editing. */
+    // "disableReferencedProjectLoad": true,             /* Reduce the number of projects loaded automatically by TypeScript. */
+
+    /* Language and Environment */
+    "target": "es2016",                                  /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */
+    // "lib": [],                                        /* Specify a set of bundled library declaration files that describe the target runtime environment. */
+    // "jsx": "preserve",                                /* Specify what JSX code is generated. */
+    // "experimentalDecorators": true,                   /* Enable experimental support for legacy experimental decorators. */
+    // "emitDecoratorMetadata": true,                    /* Emit design-type metadata for decorated declarations in source files. */
+    // "jsxFactory": "",                                 /* Specify the JSX factory function used when targeting React JSX emit, e.g. 'React.createElement' or 'h'. */
+    // "jsxFragmentFactory": "",                         /* Specify the JSX Fragment reference used for fragments when targeting React JSX emit e.g. 'React.Fragment' or 'Fragment'. */
+    // "jsxImportSource": "",                            /* Specify module specifier used to import the JSX factory functions when using 'jsx: react-jsx*'. */
+    // "reactNamespace": "",                             /* Specify the object invoked for 'createElement'. This only applies when targeting 'react' JSX emit. */
+    // "noLib": true,                                    /* Disable including any library files, including the default lib.d.ts. */
+    // "useDefineForClassFields": true,                  /* Emit ECMAScript-standard-compliant class fields. */
+    // "moduleDetection": "auto",                        /* Control what method is used to detect module-format JS files. */
+
+    /* Modules */
+    "module": "commonjs",                                /* Specify what module code is generated. */
+     "rootDir": "./src",                                  /* Specify the root folder within your source files. */
+    // "moduleResolution": "node10",                     /* Specify how TypeScript looks up a file from a given module specifier. */
+    // "baseUrl": "./",                                  /* Specify the base directory to resolve non-relative module names. */
+    // "paths": {},                                      /* Specify a set of entries that re-map imports to additional lookup locations. */
+    // "rootDirs": [],                                   /* Allow multiple folders to be treated as one when resolving modules. */
+    // "typeRoots": [],                                  /* Specify multiple folders that act like './node_modules/@types'. */
+    // "types": [],                                      /* Specify type package names to be included without being referenced in a source file. */
+    // "allowUmdGlobalAccess": true,                     /* Allow accessing UMD globals from modules. */
+    // "moduleSuffixes": [],                             /* List of file name suffixes to search when resolving a module. */
+    // "allowImportingTsExtensions": true,               /* Allow imports to include TypeScript file extensions. Requires '--moduleResolution bundler' and either '--noEmit' or '--emitDeclarationOnly' to be set. */
+    // "resolvePackageJsonExports": true,                /* Use the package.json 'exports' field when resolving package imports. */
+    // "resolvePackageJsonImports": true,                /* Use the package.json 'imports' field when resolving imports. */
+    // "customConditions": [],                           /* Conditions to set in addition to the resolver-specific defaults when resolving imports. */
+    // "resolveJsonModule": true,                        /* Enable importing .json files. */
+    // "allowArbitraryExtensions": true,                 /* Enable importing files with any extension, provided a declaration file is present. */
+    // "noResolve": true,                                /* Disallow 'import's, 'require's or '<reference>'s from expanding the number of files TypeScript should add to a project. */
+
+    /* JavaScript Support */
+    // "allowJs": true,                                  /* Allow JavaScript files to be a part of your program. Use the 'checkJS' option to get errors from these files. */
+    // "checkJs": true,                                  /* Enable error reporting in type-checked JavaScript files. */
+    // "maxNodeModuleJsDepth": 1,                        /* Specify the maximum folder depth used for checking JavaScript files from 'node_modules'. Only applicable with 'allowJs'. */
+
+    /* Emit */
+    // "declaration": true,                              /* Generate .d.ts files from TypeScript and JavaScript files in your project. */
+    // "declarationMap": true,                           /* Create sourcemaps for d.ts files. */
+    // "emitDeclarationOnly": true,                      /* Only output d.ts files and not JavaScript files. */
+    // "sourceMap": true,                                /* Create source map files for emitted JavaScript files. */
+    // "inlineSourceMap": true,                          /* Include sourcemap files inside the emitted JavaScript. */
+    // "outFile": "./",                                  /* Specify a file that bundles all outputs into one JavaScript file. If 'declaration' is true, also designates a file that bundles all .d.ts output. */
+     "outDir": "./dist",                                   /* Specify an output folder for all emitted files. */
+    // "removeComments": true,                           /* Disable emitting comments. */
+    // "noEmit": true,                                   /* Disable emitting files from a compilation. */
+    // "importHelpers": true,                            /* Allow importing helper functions from tslib once per project, instead of including them per-file. */
+    // "importsNotUsedAsValues": "remove",               /* Specify emit/checking behavior for imports that are only used for types. */
+    // "downlevelIteration": true,                       /* Emit more compliant, but verbose and less performant JavaScript for iteration. */
+    // "sourceRoot": "",                                 /* Specify the root path for debuggers to find the reference source code. */
+    // "mapRoot": "",                                    /* Specify the location where debugger should locate map files instead of generated locations. */
+    // "inlineSources": true,                            /* Include source code in the sourcemaps inside the emitted JavaScript. */
+    // "emitBOM": true,                                  /* Emit a UTF-8 Byte Order Mark (BOM) in the beginning of output files. */
+    // "newLine": "crlf",                                /* Set the newline character for emitting files. */
+    // "stripInternal": true,                            /* Disable emitting declarations that have '@internal' in their JSDoc comments. */
+    // "noEmitHelpers": true,                            /* Disable generating custom helper functions like '__extends' in compiled output. */
+    // "noEmitOnError": true,                            /* Disable emitting files if any type checking errors are reported. */
+    // "preserveConstEnums": true,                       /* Disable erasing 'const enum' declarations in generated code. */
+    // "declarationDir": "./",                           /* Specify the output directory for generated declaration files. */
+    // "preserveValueImports": true,                     /* Preserve unused imported values in the JavaScript output that would otherwise be removed. */
+
+    /* Interop Constraints */
+    // "isolatedModules": true,                          /* Ensure that each file can be safely transpiled without relying on other imports. */
+    // "verbatimModuleSyntax": true,                     /* Do not transform or elide any imports or exports not marked as type-only, ensuring they are written in the output file's format based on the 'module' setting. */
+    // "allowSyntheticDefaultImports": true,             /* Allow 'import x from y' when a module doesn't have a default export. */
+    "esModuleInterop": true,                             /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables 'allowSyntheticDefaultImports' for type compatibility. */
+    // "preserveSymlinks": true,                         /* Disable resolving symlinks to their realpath. This correlates to the same flag in node. */
+    "forceConsistentCasingInFileNames": true,            /* Ensure that casing is correct in imports. */
+
+    /* Type Checking */
+    "strict": true,                                      /* Enable all strict type-checking options. */
+    // "noImplicitAny": true,                            /* Enable error reporting for expressions and declarations with an implied 'any' type. */
+    // "strictNullChecks": true,                         /* When type checking, take into account 'null' and 'undefined'. */
+    // "strictFunctionTypes": true,                      /* When assigning functions, check to ensure parameters and the return values are subtype-compatible. */
+    // "strictBindCallApply": true,                      /* Check that the arguments for 'bind', 'call', and 'apply' methods match the original function. */
+    // "strictPropertyInitialization": true,             /* Check for class properties that are declared but not set in the constructor. */
+    // "noImplicitThis": true,                           /* Enable error reporting when 'this' is given the type 'any'. */
+    // "useUnknownInCatchVariables": true,               /* Default catch clause variables as 'unknown' instead of 'any'. */
+    // "alwaysStrict": true,                             /* Ensure 'use strict' is always emitted. */
+    // "noUnusedLocals": true,                           /* Enable error reporting when local variables aren't read. */
+    // "noUnusedParameters": true,                       /* Raise an error when a function parameter isn't read. */
+    // "exactOptionalPropertyTypes": true,               /* Interpret optional property types as written, rather than adding 'undefined'. */
+    // "noImplicitReturns": true,                        /* Enable error reporting for codepaths that do not explicitly return in a function. */
+    // "noFallthroughCasesInSwitch": true,               /* Enable error reporting for fallthrough cases in switch statements. */
+    // "noUncheckedIndexedAccess": true,                 /* Add 'undefined' to a type when accessed using an index. */
+    // "noImplicitOverride": true,                       /* Ensure overriding members in derived classes are marked with an override modifier. */
+    // "noPropertyAccessFromIndexSignature": true,       /* Enforces using indexed accessors for keys declared using an indexed type. */
+    // "allowUnusedLabels": true,                        /* Disable error reporting for unused labels. */
+    // "allowUnreachableCode": true,                     /* Disable error reporting for unreachable code. */
+
+    /* Completeness */
+    // "skipDefaultLibCheck": true,                      /* Skip type checking .d.ts files that are included with TypeScript. */
+    "skipLibCheck": true                                 /* Skip type checking all .d.ts files. */
+  }
+}

example/yarn.lock

@@ -0,0 +1,162 @@
+# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
+# yarn lockfile v1
+
+
+"@cspotcode/source-map-support@^0.8.0":
+  version "0.8.1"
+  resolved "https://registry.yarnpkg.com/@cspotcode/source-map-support/-/source-map-support-0.8.1.tgz#00629c35a688e05a88b1cda684fb9d5e73f000a1"
+  integrity sha512-IchNf6dN4tHoMFIn/7OE8LWZ19Y6q/67Bmf6vnGREv8RSbBVb9LPJxEcnwrcwX6ixSvaiGoomAUvu4YSxXrVgw==
+  dependencies:
+    "@jridgewell/trace-mapping" "0.3.9"
+
+"@jridgewell/resolve-uri@^3.0.3":
+  version "3.1.0"
+  resolved "https://registry.yarnpkg.com/@jridgewell/resolve-uri/-/resolve-uri-3.1.0.tgz#2203b118c157721addfe69d47b70465463066d78"
+  integrity sha512-F2msla3tad+Mfht5cJq7LSXcdudKTWCVYUgw6pLFOOHSTtZlj6SWNYAp+AhuqLmWdBO2X5hPrLcu8cVP8fy28w==
+
+"@jridgewell/sourcemap-codec@^1.4.10":
+  version "1.4.14"
+  resolved "https://registry.yarnpkg.com/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.4.14.tgz#add4c98d341472a289190b424efbdb096991bb24"
+  integrity sha512-XPSJHWmi394fuUuzDnGz1wiKqWfo1yXecHQMRf2l6hztTO+nPru658AyDngaBe7isIxEkRsPR3FZh+s7iVa4Uw==
+
+"@jridgewell/[email protected]":
+  version "0.3.9"
+  resolved "https://registry.yarnpkg.com/@jridgewell/trace-mapping/-/trace-mapping-0.3.9.tgz#6534fd5933a53ba7cbf3a17615e273a0d1273ff9"
+  integrity sha512-3Belt6tdc8bPgAtbcmdtNJlirVoTmEb5e2gC94PnkwEW9jI6CAHUeoG85tjWP5WquqfavoMtMwiG4P926ZKKuQ==
+  dependencies:
+    "@jridgewell/resolve-uri" "^3.0.3"
+    "@jridgewell/sourcemap-codec" "^1.4.10"
+
+"@tsconfig/node10@^1.0.7":
+  version "1.0.9"
+  resolved "https://registry.yarnpkg.com/@tsconfig/node10/-/node10-1.0.9.tgz#df4907fc07a886922637b15e02d4cebc4c0021b2"
+  integrity sha512-jNsYVVxU8v5g43Erja32laIDHXeoNvFEpX33OK4d6hljo3jDhCBDhx5dhCCTMWUojscpAagGiRkBKxpdl9fxqA==
+
+"@tsconfig/node12@^1.0.7":
+  version "1.0.11"
+  resolved "https://registry.yarnpkg.com/@tsconfig/node12/-/node12-1.0.11.tgz#ee3def1f27d9ed66dac6e46a295cffb0152e058d"
+  integrity sha512-cqefuRsh12pWyGsIoBKJA9luFu3mRxCA+ORZvA4ktLSzIuCUtWVxGIuXigEwO5/ywWFMZ2QEGKWvkZG1zDMTag==
+
+"@tsconfig/node14@^1.0.0":
+  version "1.0.3"
+  resolved "https://registry.yarnpkg.com/@tsconfig/node14/-/node14-1.0.3.tgz#e4386316284f00b98435bf40f72f75a09dabf6c1"
+  integrity sha512-ysT8mhdixWK6Hw3i1V2AeRqZ5WfXg1G43mqoYlM2nc6388Fq5jcXyr5mRsqViLx/GJYdoL0bfXD8nmF+Zn/Iow==
+
+"@tsconfig/node16@^1.0.2":
+  version "1.0.3"
+  resolved "https://registry.yarnpkg.com/@tsconfig/node16/-/node16-1.0.3.tgz#472eaab5f15c1ffdd7f8628bd4c4f753995ec79e"
+  integrity sha512-yOlFc+7UtL/89t2ZhjPvvB/DeAr3r+Dq58IgzsFkOAvVC6NMJXmCGjbptdXdR9qsX7pKcTL+s87FtYREi2dEEQ==
+
+"@types/node@^18.15.11":
+  version "18.15.11"
+  resolved "https://registry.yarnpkg.com/@types/node/-/node-18.15.11.tgz#b3b790f09cb1696cffcec605de025b088fa4225f"
+  integrity sha512-E5Kwq2n4SbMzQOn6wnmBjuK9ouqlURrcZDVfbo9ftDDTFt3nk7ZKK4GMOzoYgnpQJKcxwQw+lGaBvvlMo0qN/Q==
+
+"@upstash/redis@^1.20.2":
+  version "1.20.2"
+  resolved "https://registry.yarnpkg.com/@upstash/redis/-/redis-1.20.2.tgz#f797915c90054764b26d2289f5da5dd4b68d8480"
+  integrity sha512-9QS/SypDxeeh672H7dEEmuYOX5TtPYnaDLlhxWJEPd8LzcEQ6hohwDJuojpqGkvvvrK58mlWOkN1GrMxbXPTeQ==
+  dependencies:
+    isomorphic-fetch "^3.0.0"
+
+acorn-walk@^8.1.1:
+  version "8.2.0"
+  resolved "https://registry.yarnpkg.com/acorn-walk/-/acorn-walk-8.2.0.tgz#741210f2e2426454508853a2f44d0ab83b7f69c1"
+  integrity sha512-k+iyHEuPgSw6SbuDpGQM+06HQUa04DZ3o+F6CSzXMvvI5KMvnaEqXe+YVe555R9nn6GPt404fos4wcgpw12SDA==
+
+acorn@^8.4.1:
+  version "8.8.2"
+  resolved "https://registry.yarnpkg.com/acorn/-/acorn-8.8.2.tgz#1b2f25db02af965399b9776b0c2c391276d37c4a"
+  integrity sha512-xjIYgE8HBrkpd/sJqOGNspf8uHG+NOHGOw6a/Urj8taM2EXfdNAH2oFcPeIFfsv3+kz/mJrS5VuMqbNLjCa2vw==
+
+arg@^4.1.0:
+  version "4.1.3"
+  resolved "https://registry.yarnpkg.com/arg/-/arg-4.1.3.tgz#269fc7ad5b8e42cb63c896d5666017261c144089"
+  integrity sha512-58S9QDqG0Xx27YwPSt9fJxivjYl432YCwfDMfZ+71RAqUrZef7LrKQZ3LHLOwCS4FLNBplP533Zx895SeOCHvA==
+
+create-require@^1.1.0:
+  version "1.1.1"
+  resolved "https://registry.yarnpkg.com/create-require/-/create-require-1.1.1.tgz#c1d7e8f1e5f6cfc9ff65f9cd352d37348756c333"
+  integrity sha512-dcKFX3jn0MpIaXjisoRvexIJVEKzaq7z2rZKxf+MSr9TkdmHmsU4m2lcLojrj/FHl8mk5VxMmYA+ftRkP/3oKQ==
+
+diff@^4.0.1:
+  version "4.0.2"
+  resolved "https://registry.yarnpkg.com/diff/-/diff-4.0.2.tgz#60f3aecb89d5fae520c11aa19efc2bb982aade7d"
+  integrity sha512-58lmxKSA4BNyLz+HHMUzlOEpg09FV+ev6ZMe3vJihgdxzgcwZ8VoEEPmALCZG9LmqfVoNMMKpttIYTVG6uDY7A==
+
+isomorphic-fetch@^3.0.0:
+  version "3.0.0"
+  resolved "https://registry.yarnpkg.com/isomorphic-fetch/-/isomorphic-fetch-3.0.0.tgz#0267b005049046d2421207215d45d6a262b8b8b4"
+  integrity sha512-qvUtwJ3j6qwsF3jLxkZ72qCgjMysPzDfeV240JHiGZsANBYd+EEuu35v7dfrJ9Up0Ak07D7GGSkGhCHTqg/5wA==
+  dependencies:
+    node-fetch "^2.6.1"
+    whatwg-fetch "^3.4.1"
+
+make-error@^1.1.1:
+  version "1.3.6"
+  resolved "https://registry.yarnpkg.com/make-error/-/make-error-1.3.6.tgz#2eb2e37ea9b67c4891f684a1394799af484cf7a2"
+  integrity sha512-s8UhlNe7vPKomQhC1qFelMokr/Sc3AgNbso3n74mVPA5LTZwkB9NlXf4XPamLxJE8h0gh73rM94xvwRT2CVInw==
+
+node-fetch@^2.6.1:
+  version "2.6.9"
+  resolved "https://registry.yarnpkg.com/node-fetch/-/node-fetch-2.6.9.tgz#7c7f744b5cc6eb5fd404e0c7a9fec630a55657e6"
+  integrity sha512-DJm/CJkZkRjKKj4Zi4BsKVZh3ValV5IR5s7LVZnW+6YMh0W1BfNA8XSs6DLMGYlId5F3KnA70uu2qepcR08Qqg==
+  dependencies:
+    whatwg-url "^5.0.0"
+
+tr46@~0.0.3:
+  version "0.0.3"
+  resolved "https://registry.yarnpkg.com/tr46/-/tr46-0.0.3.tgz#8184fd347dac9cdc185992f3a6622e14b9d9ab6a"
+  integrity sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw==
+
+ts-node@^10.9.1:
+  version "10.9.1"
+  resolved "https://registry.yarnpkg.com/ts-node/-/ts-node-10.9.1.tgz#e73de9102958af9e1f0b168a6ff320e25adcff4b"
+  integrity sha512-NtVysVPkxxrwFGUUxGYhfux8k78pQB3JqYBXlLRZgdGUqTO5wU/UyHop5p70iEbGhB7q5KmiZiU0Y3KlJrScEw==
+  dependencies:
+    "@cspotcode/source-map-support" "^0.8.0"
+    "@tsconfig/node10" "^1.0.7"
+    "@tsconfig/node12" "^1.0.7"
+    "@tsconfig/node14" "^1.0.0"
+    "@tsconfig/node16" "^1.0.2"
+    acorn "^8.4.1"
+    acorn-walk "^8.1.1"
+    arg "^4.1.0"
+    create-require "^1.1.0"
+    diff "^4.0.1"
+    make-error "^1.1.1"
+    v8-compile-cache-lib "^3.0.1"
+    yn "3.1.1"
+
+typescript@^5.0.2:
+  version "5.0.2"
+  resolved "https://registry.yarnpkg.com/typescript/-/typescript-5.0.2.tgz#891e1a90c5189d8506af64b9ef929fca99ba1ee5"
+  integrity sha512-wVORMBGO/FAs/++blGNeAVdbNKtIh1rbBL2EyQ1+J9lClJ93KiiKe8PmFIVdXhHcyv44SL9oglmfeSsndo0jRw==
+
+v8-compile-cache-lib@^3.0.1:
+  version "3.0.1"
+  resolved "https://registry.yarnpkg.com/v8-compile-cache-lib/-/v8-compile-cache-lib-3.0.1.tgz#6336e8d71965cb3d35a1bbb7868445a7c05264bf"
+  integrity sha512-wa7YjyUGfNZngI/vtK0UHAN+lgDCxBPCylVXGp0zu59Fz5aiGtNXaq3DhIov063MorB+VfufLh3JlF2KdTK3xg==
+
+webidl-conversions@^3.0.0:
+  version "3.0.1"
+  resolved "https://registry.yarnpkg.com/webidl-conversions/-/webidl-conversions-3.0.1.tgz#24534275e2a7bc6be7bc86611cc16ae0a5654871"
+  integrity sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ==
+
+whatwg-fetch@^3.4.1:
+  version "3.6.2"
+  resolved "https://registry.yarnpkg.com/whatwg-fetch/-/whatwg-fetch-3.6.2.tgz#dced24f37f2624ed0281725d51d0e2e3fe677f8c"
+  integrity sha512-bJlen0FcuU/0EMLrdbJ7zOnW6ITZLrZMIarMUVmdKtsGvZna8vxKYaexICWPfZ8qwf9fzNq+UEIZrnSaApt6RA==
+
+whatwg-url@^5.0.0:
+  version "5.0.0"
+  resolved "https://registry.yarnpkg.com/whatwg-url/-/whatwg-url-5.0.0.tgz#966454e8765462e37644d3626f6742ce8b70965d"
+  integrity sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw==
+  dependencies:
+    tr46 "~0.0.3"
+    webidl-conversions "^3.0.0"
+
+[email protected]:
+  version "3.1.1"
+  resolved "https://registry.yarnpkg.com/yn/-/yn-3.1.1.tgz#1e87401a09d767c1d5eab26a6e4c185182d2eb50"
+  integrity sha512-Ux4ygGWsu2c7isFWe8Yu1YluJmqVhxqK2cLXNQA5AcC3QfbGNpM7fu0Y8b/z16pXLnFxZYvWhd3fhBY9DLmC6Q==

lib/srh/auth/token_resolver.ex

@@ -1,7 +1,6 @@
 defmodule Srh.Auth.TokenResolver do
   use GenServer
 
-  @mode Application.fetch_env!(:srh, :mode)
   @file_path Application.fetch_env!(:srh, :file_path)
 
   @ets_table_name :srh_token_resolver
@@ -25,7 +24,7 @@ defmodule Srh.Auth.TokenResolver do
     table = :ets.new(@ets_table_name, [:named_table, read_concurrency: true])
 
     # Populate the ETS table with data from storage
-    do_init_load(@mode)
+    do_init_load(get_token_loader_mode())
 
     {
       :ok,
@@ -36,7 +35,7 @@ defmodule Srh.Auth.TokenResolver do
   end
 
   def resolve(token) do
-    do_resolve(@mode, token)
+    do_resolve(get_token_loader_mode(), token)
   end
 
   # Server methods
@@ -49,6 +48,10 @@ defmodule Srh.Auth.TokenResolver do
   end
 
   # Internal server
+  defp get_token_loader_mode() do
+    System.get_env("SRH_MODE", "file")
+  end
+
   defp do_init_load("file") do
     config_file_data = Jason.decode!(File.read!(@file_path))
     IO.puts("Loaded config file from disk. #{map_size(config_file_data)} entries.")
@@ -59,6 +62,25 @@ defmodule Srh.Auth.TokenResolver do
     )
   end
 
+  defp do_init_load("env") do
+    srh_token = System.get_env("SRH_TOKEN")
+    srh_connection_string = System.get_env("SRH_CONNECTION_STRING")
+
+    # Returns an error if fails, first tuple value is the number
+    {srh_max_connections, ""} = Integer.parse(System.get_env("SRH_MAX_CONNECTIONS", "3"))
+
+    # Create a config-file-like structure that the ETS layout expects, with just one entry
+    config_file_data = Map.put(%{}, srh_token, %{
+      "srh_id" => "env_config_connection", # Jason.parse! expects these keys to be strings, not atoms, so we need to replicate that setup
+      "connection_string" => srh_connection_string,
+      "max_connections" => srh_max_connections
+    })
+
+    IO.puts("Loaded config from env. #{map_size(config_file_data)} entries.")
+    # Load this into ETS
+    Enum.each(config_file_data, &:ets.insert(@ets_table_name, &1))
+  end
+
   defp do_init_load(_), do: :ok
 
   # Internal, but client side, methods. These are client side to prevent GenServer lockup
@@ -73,6 +95,9 @@ defmodule Srh.Auth.TokenResolver do
     end
   end
 
+  # The env strategy uses the same ETS table as the file strategy, so we can fall back on that
+  defp do_resolve("env", token), do: do_resolve("file", token)
+
   defp do_resolve("redis", _token) do
     {
       :ok,

lib/srh/http/base_router.ex

@@ -2,6 +2,7 @@ defmodule Srh.Http.BaseRouter do
   use Plug.Router
   alias Srh.Http.RequestValidator
   alias Srh.Http.CommandHandler
+  alias Srh.Http.ResultEncoder
 
   plug(:match)
   plug(Plug.Parsers, parsers: [:json], pass: ["application/json"], json_decoder: Jason)
@@ -12,27 +13,30 @@ defmodule Srh.Http.BaseRouter do
   end
 
   post "/" do
-    conn
-    |> handle_extract_auth(&CommandHandler.handle_command(conn, &1))
-    |> handle_response(conn)
+    do_command_request(conn, &CommandHandler.handle_command(&1, &2))
   end
 
   post "/pipeline" do
-    conn
-    |> handle_extract_auth(&CommandHandler.handle_command_array(conn, &1))
-    |> handle_response(conn)
+    do_command_request(conn, &CommandHandler.handle_command_array(&1, &2))
   end
 
   post "/multi-exec" do
-    conn
-    |> handle_extract_auth(&CommandHandler.handle_command_transaction_array(conn, &1))
-    |> handle_response(conn)
+    do_command_request(conn, &CommandHandler.handle_command_transaction_array(&1, &2))
   end
 
   match _ do
     send_resp(conn, 404, "Endpoint not found")
   end
 
+  defp do_command_request(conn, success_lambda) do
+    encoding_enabled = handle_extract_encoding?(conn)
+
+    conn
+    |> handle_extract_auth(&success_lambda.(conn, &1))
+    |> handle_encoding_step(encoding_enabled)
+    |> handle_response(conn)
+  end
+
   defp handle_extract_auth(conn, success_lambda) do
     case conn
          |> get_req_header("authorization")
@@ -45,6 +49,24 @@ defmodule Srh.Http.BaseRouter do
     end
   end
 
+  defp handle_extract_encoding?(conn) do
+    case conn
+         |> get_req_header("upstash-encoding")
+         |> RequestValidator.validate_encoding_header() do
+      {:ok, _encoding_enabled} -> true
+      {:error, _} -> false # it's not required to be present
+    end
+  end
+
+  defp handle_encoding_step(response, encoding_enabled) do
+    case encoding_enabled do
+      true ->
+        # We need to use the encoder to
+        ResultEncoder.encode_response(response)
+      false -> response
+    end
+  end
+
   defp handle_response(response, conn) do
     %{code: code, message: message, json: json} =
       case response do

lib/srh/http/request_validator.ex

@@ -26,6 +26,21 @@ defmodule Srh.Http.RequestValidator do
 
   defp do_validate_pipeline_item(_), do: :error
 
+  def validate_encoding_header(header_value_array) when is_list(header_value_array) do
+    do_validate_encoding_header(header_value_array)
+  end
+
+  # This has been broken up like this to future-proof different encoding modes in the future
+  defp do_validate_encoding_header([first_item | rest]) do
+    case first_item do
+      "base64" -> {:ok, true}
+
+      _ -> do_validate_encoding_header(rest)
+    end
+  end
+
+  defp do_validate_encoding_header([]), do: {:error, :not_found}
+
   def validate_bearer_header(header_value_array) when is_list(header_value_array) do
     do_validate_bearer_header(header_value_array)
   end

lib/srh/http/result_encoder.ex

@@ -0,0 +1,66 @@
+defmodule Srh.Http.ResultEncoder do
+
+  # Errors don't get encoded, we need to skip over those
+  def encode_response({:redis_error, error_result_map}) do
+    {:redis_error, error_result_map}
+  end
+
+  # List-based responses, they will contain multiple entries
+  # It's important to note that this is DIFFERENT from a list of values,
+  # as it's a list of separate command responses. Each is a map that either
+  # Contains a result or an error
+  def encode_response({:ok, result_list}) when is_list(result_list) do
+    # Each one of these entries needs to be encoded
+    {:ok, encode_response_list(result_list, [])}
+  end
+
+  # Single item response
+  def encode_response({:ok, %{result: result_value}}) do
+    {:ok, %{result: encode_result_value(result_value)}}
+  end
+
+  ## RESULT LIST ENCODING ##
+
+  defp encode_response_list([current | rest], encoded_responses) do
+    encoded_current_entry = case current do
+      %{result: value} ->
+        %{result: encode_result_value(value)} # Encode the value
+      %{error: error_message} ->
+        %{error: error_message} # We don't encode errors
+    end
+
+    encode_response_list(rest, [encoded_current_entry | encoded_responses])
+  end
+
+  defp encode_response_list([], encoded_responses) do
+    Enum.reverse(encoded_responses)
+  end
+
+  ## RESULT VALUE ENCODING ##
+
+  # Numbers are ignored
+  defp encode_result_value(value) when is_number(value), do: value
+
+  # Null/nil is ignored
+  defp encode_result_value(value) when is_nil(value), do: value
+
+  # Strings / blobs (any binary data) is encoded to Base64
+  defp encode_result_value(value) when is_binary(value), do: Base.encode64(value)
+
+  defp encode_result_value(arr) when is_list(arr) do
+    encode_result_value_list(arr, [])
+  end
+
+  ## RESULT VALUE LIST ENCODING ##
+
+  # Arrays can have values that are encoded, or aren't, based on whats laid out above
+  defp encode_result_value_list([current | rest], encoded_responses) do
+    encoded_value = encode_result_value(current)
+    encode_result_value_list(rest, [encoded_value | encoded_responses])
+  end
+
+  defp encode_result_value_list([], encoded_responses) do
+    # There are no responses left, and since we add them backwards, we need to flip the list
+    Enum.reverse(encoded_responses)
+  end
+end