feat(endo): Add Endo README and DESIGN
ADD
packages/endo/DESIGN.md
+
+# Endo Design
+
+Each of the workflows Endo executes a portion of one sequence of underlying
+internals.
+
+* search (`search.js`): Scan the parent directories of a given `moduleLocation`
+ until successfully finding and reading a `package.json` for the containing
+ application.
+* map compartments (`compartmap.js`): Find and gather all the `package.json`
+ files for the application's transitive dependencies.
+ Use these to construct a compartment map describing how to construct a
+ `Compartment` for each application package and how to link the modules each
+ exports in the compartments that import them.
+* load compartments (`archive.js`): Using `compartment.load`, or
+ implicitly through `compartment.import`, create a module graph for the
+ application's entire working set.
+ When creating an archive, this does not execute any of the modules. Endo
+ uses the compartments and a special `importHook` that records the text of
+ every module the main module needed.
+* import modules (`import.js`, `import-archive.js`): Actually
+ execute the working set.
+
+Around this sequence, we can enter late or depart early to store or retrieve an
+archive.
+Endo provides workflows that use `read` and `write` hooks when interacting
+with a filesystem or work with the archive bytes directly.
+
+This diagram represents the the workflows of each of the public methods like
+`importLocation`.
+Each column of pipes `|` is a workflow from top to bottom.
+Each asterisk `*` denotes a step that is taken by that workflow.
+The functions `loadArchive` and `parseArchive` partially apply `importArchive`,
+so the pluses `+` indicate the steps taken in continuation.
+The dotted lines `.'. : '.'` indicate carrying an archive file from the end of
+one workflow to the beginning of another, either as bytes or a location.
+
+```
+ loadLocation writeArchive
+ importLocation | | makeArchive
+ | | | |
+ | | | | parseArchive
+ | | | | | loadArchive
+ | | | | | | importArchive
+ | | | | | | |...
+ search -> * * * * | |'| . '
+ map compartments -> * * * * .'.| | |' : :
+ read archive -> | | | | ' * * * : :
+ unpack archive -> | | | | : * * * : :
+assemble compartments -> * * * * : + + * : : <- endowments
+ load compartments -> * * * * : + + * : :
+ import modules -> * | | : + + * : :
+ pack archive -> * * ' : :
+ write archive -> * '.' <- data : :
+ '..............' : <- files
+ '...............'
+```
+
+> TODO
+>
+> A future version of Endo may introduce a command line for:
+>
+> * Sandboxing Node.js applications by using `endo` just as you would use
+> `node`.
+> * Passing additional `endo` command line arguments to grant common
+> attenuated powers like the built-in `fs` module with limited read access
+> `--read .` the `net` and `http` modules with limited authority `--net
+> example.com`, or the standard input and output.
+> * Allowing application authors to express policies that extend
+> the availability of limited powers like access to the `fs` module
+> or standard input and output streams to third-party packages.
+> * Creating application archives and running them, with `endo --pack app.agar
+> app.js` and `endo --archive app.agar`.
+
+> TODO
+>
+> A future version of Endo may introduce support for web clients.
+> This would support separate modes for development (without a build step) and
+> production (demarcated by a build step).
+>
+> For development, Endo would provide a `compartmap` tool that would build
+> a `compartmap.json` each time the dependency graph changes or when
+> endowment policies change.
+> To run an Endo application on the web, the developer would introduce
+> a `<script src="node_modules/endo/web.js" import="./app.js"
+> compartmap="compartmap.json">`.
+> The `import` and `compartmap` might have sensible defaults.
+>
+> For production, Endo would provide a build step similar to the archiver,
+> but without the archive envelope.
+
+> TODO
+>
+> A future version of Endo may add support for generalized workers: workers
+> that would be introduced to compartments as endowments and have the same usage
+> but different implementations when running in Node.js, in Node.js from an
+> archive, or on the web.
+> The developer would introduce the worker entry module and desired name in
+> their `package.json`.
+>
+> For this example, we presume the existence of a "promise worker" calling
+> convention, where the parent worker gets a promise for the "remote presence"
+> of the worker's exported namespace.
+>
+> ```json
+> {
+> "promise-workers": {
+> "MyPromiseWorker": "./my-worker.js"
+> }
+> }
+> ```
+>
+> In that package, Endo would introduce the worker as a global.
+>
+> ```js
+> const worker = new MyPromiseWorker();
+> ```
+
+
+## Compartment Maps
+
+Endo at the command line works by generating a _compartment map_ from
+your application workspace and all of the `node_modules` it needs.
+A compartment map is similar to a lock file because it collects information
+from all of the installed modules.
+A compartment map describes how to construct compartments for each
+package in your application and link their module namespaces.
+
+> TODO
+>
+> For browser applications, a future version of Endo will ship
+> a `compartmap` tool that can be used as a `postinall` script
+> to generate a compartment map applications can use _during development_
+> to run on the client side without a build step.
+>
+> This compartment map will include the `browser` tag and all corresponding
+> module aliases from the `exports` field in `package.json`.
+> The Endo command line tool will always generate a compartment map
+> of its own without the `browser` tag and will never use the `compartmap.json`
+> file on disk.
+>
+> A compartment map is similar in spirit to an import map or a package lock.
+> The object fully describes how to compose a compartment DAG to run an
+> application.
+> A compartment map is intended to be generated only when dependencies or
+> compartment policies change, not to be drafted by hand, and certainly not to be
+> both manually and automatically maintained.
+>
+> As with an importmap, we can add a postinstall hook to package.json that will
+> generate a new compartmap each time the dependency graph of an application
+> changes.
+>
+> ```json
+> {
+> "scripts": {
+> "postinstall": "compartmap"
+> }
+> }
+> ```
+>
+> We can include this in the scaffolding for compartmentalized applications.
+>
+> The `compartmap` tool generates a `compartmap.json` file, merging what
+> it finds in a `package-lock.json`, `yarn.lock` or some such shrinkwrap,
+> or just crawls `node_modules` and finds all the `package.json` files
+> as an input.
+> Then, it merges a `compolicy.json` if it finds one.
+> The default compartment policy is empty, which implies:
+>
+> * endowments only pass to the main compartment.
+> * no package has access to any built-in modules.
+
+The compartment map shape:
+
+```ts
+// CompartmentMap describes how to prepare compartments
+// to run an application.
+type CompartmentMap = {
+ tags: Tags,
+ main: CompartmentName,
+ compartments: Object<CompartmentName, Compartment>,
+ realms: Object<RealmName, Realm>, // TODO
+};
+
+// Tags are the build tags for the compartment.
+// These may include terms like "browser", meaning
+// each compartment uses the implementation of each
+// module suitable for use in a browser environment.
+type Tags = Array<Tag>;
+type Tag = string;
+
+// CompartmentName is an arbitrary string to name
+// a compartment for purposes of inter-compartment linkage.
+type CompartmentName = string;
+
+// Compartment describes where to find the modules
+// for a compartment and how to link the compartment
+// to modules in other compartments, or to built-in modules.
+type Compartment = {
+ root: Root,
+ modules: ModuleMap
+ // The name of the realm to run the compartment within.
+ // The default is a single frozen realm that has no name.
+ realm: RealmName? // TODO
+};
+
+// Root is the URL relative to the compartmap.json's
+// containing location to the compartment's files.
+type Root string;
+
+// ModuleMap describes modules available in the compartment
+// that do not correspond to source files in the same compartment.
+type ModuleMap = Object<InternalModuleSpecifier, Module>;
+
+// Module describes a module that isn't in the same
+// compartment and how to introduce it to the compartment's
+// module namespace.
+type Module = {
+ // The name of the foreign compartment:
+ // TODO an absent compartment name may imply either
+ // that the module is an internal alias of the
+ // same compartment, or given by the user.
+ compartment: CompartmentName?,
+ // The name of the module in the foreign compartment's
+ // module namespace:
+ module: ExternalModuleSpecifier?,
+ // Alternately, that this module is not from
+ // any compartment and must be expressly passed
+ // into the compartment graph from the user.
+ parameter: ModuleParameter?, // TODO
+};
+
+// InternalModuleSpecifier is the module specifier
+// in the namespace of the native compartment.
+type InternalModuleSpecifier string;
+
+// ExternalModuleSpecifier is the module specifier
+// in the namespace of the foreign compartment.
+type ExternalModuleSpecifier string;
+
+// TODO everything hereafter...
+
+// Realm describes another realm to contain one or more
+// compartments.
+// The default realm is frozen by lockdown with no
+// powerful references.
+type Realm = {
+ // TODO lockdown options
+};
+
+// RealmName is an arbitrary identifier for realms
+// for reference from any Compartment description.
+// No names are reserved; the default realm has no name.
+type RealmName string;
+
+// ModuleParameter indicates that the module does not come from
+// another compartment but must be passed expressly into the
+// application by the user.
+// For example, the Node.js `fs` built-in module provides
+// powers that must be expressly granted to an application
+// and may be attenuated or limited by Endo on behalf of the user.
+// The string value is the name of the module to be provided
+// in the application's given module map.
+type ModuleParameter string;
+```
+
+
+## Compartment Map Policies
+
+> TODO
+>
+> At time of writing, Endo only empowers application code to execute
+> and log results to the command line.
+> A compartment map policy is a file that will sit beside an application that
+> expresses what powerful objects should pass into the compartment for each
+> package of an application.
+>
+> MetaMask's [LavaMoat] generates a `lavalmoat.config.json` file that serves
+> the same purposes, using a tool called TOFU: _trust on first use_.
+
+ [LavaMoat]: https://github.com/LavaMoat/lavamoat

ADD
packages/endo/README.md
+# Endo
+
+Endo runs Node.js packages in a sandbox.
+
+Each Node.js package runs in a separate [compartment][Compartments] of a single
+immutable realm.
+Each compartment may be given limited access to powerful
+modules like `fs` and powerful globals like `fetch`.
+
+Endo can also construct an archive that bundles all of the packages and modules
+needed to run an Endo command, then run an application directly from an
+archive.
+
+Currently Endo supports running packages consisting entirely of ECMAScript
+modules (ESM) that do not depend upon any powerful built-in modules or
+facilities of global scope at the command line.
+
+Future versions may support:
+
+* CommonJS modules,
+* Passing powerful and powerless built-in modules and globals into specific
+ compartments based on application policies,
+* Realm isolation,
+* Running vetted shims in a realm before [lockdown].
+* Threading endowments,
+* Threading built-in modules,
+* Membranes or tamper-proofing between packages,
+* Extensions for module and program translation scoped to each compartment,
+* Running Endo applications on the web without a build step.
+* Generalized support for workers under Node or on the web.
+* An `importmap` superset.
+
+ [compartment]: https://github.com/Agoric/SES-shim/blob/master/packages/ses/README.md#compartment
+ [lockdown]: https://github.com/Agoric/SES-shim/blob/master/packages/ses/README.md#lockdown
+
+
+## Usage
+
+Endo runs your application in a single frozen realm, endowing only the main
+compartment with the given powerful global properties and built-in modules.
+
+The `importLocation` function runs a compartmentalized application off the file
+system.
+The `endowments` are properties to add to the `globalThis` in the global scope
+of the application's main package compartment.
+The `modules` are built-in modules to grant the application's main package
+compartment.
+
+```js
+import fs from "fs";
+import { importLocation } from "endo";
+
+// ...
+
+const modules = { fs };
+const endowments = { console };
+
+const read = async location =>
+ fs.promises.readFile(new URL(location).pathname);
+
+const { namespace } = await importLocation(
+ read,
+ moduleLocation,
+ endowments,
+ modules
+);
+```
+
+Endo does nothing to arrange for the realm to be frozen.
+The application using Endo is responsible for applying the [SES] shim (if
+necessary) and calling `lockdown` to freeze the realm (if necessary).
+Endo is also not coupled to Node.js IO and does not import any powerful
+modules like `fs`.
+The user must provide `read` and `write` functions from whatever IO
+powers they have.
+
+> TODO
+>
+> A future version will allow application authors to distribute
+> their global endowments and granted built-in modules to third-party
+> packages within the application, as with [LavaMoat].
+
+The `importLocation` function uses `loadLocation`.
+Using `loadLocation` directly allows for deferred execution or multiple runs
+with different endowments or modules.
+Calling `loadLocation` returns an `Application` object with an
+`execute(endowments?, modules?)` method.
+
+Use `writeArchive` to capture an application in an archival format.
+Archives are `zip` files with a `compartmap.json` manifest file.
+
+```js
+import fs from "fs";
+import { writeArchive } from "endo";
+
+const read = async location =>
+ fs.promises.readFile(new URL(location).pathname);
+const write = async (location, content) =>
+ fs.promises.writeFile(new URL(location).pathname, content);
+
+await writeArchive(
+ write,
+ read,
+ new URL('app.agar', import.meta.url).toString(), // the archive to write
+ new URL('app.js', import.meta.url).toString() // the application to capture
+);
+```
+
+The `writeArchive` function uses `makeArchive`.
+Using `makeArchive` directly gives you the archive bytes.
+
+Use `importArchive` to run an application from an archive.
+Note the similarity to `importLocation`.
+
+```js
+import fs from "fs";
+import { importArchive } from "endo";
+
+// ...
+
+const modules = { fs };
+const endowments = { console };
+
+const read = async location =>
+ fs.promises.readFile(new URL(location).pathname);
+
+const { namespace } = await importArchive(
+ read,
+ archiveLocation,
+ endowments,
+ modules
+);
+```
+
+The `importArchive` function composes `loadArchive` and `parseArchive`.
+Use `loadArchive` to defer execution or run multiple times with varying
+endowments.
+Use `parseArchive` to construct a runner from the bytes of an archive.
+The `loadArchive` and `parseArchive` functions return an `Application`
+object with an `execute(endowments?, modules?)` method.
+
+
+## Ruminations on the Name
+
+* In Latin, "endo-" means "internal" or "within".
+ This is fitting because Endo runs Node _within_ a safe sandbox.
+ This is fitting in turn because Endo is built on the legacy of Google Caja.
+ In Spanish, "caja" means "box" and is related to the Latin word "capsum" and
+ English "capsule", as in "encapsulate".
+* Endo is an anagram of Node and Deno.
+ That is to say, we are not Done yet.
+* The `endo` command, like the `sudo` command, is a "do" command.
+ However, instead of escalating priviliedge, it encapsulates priviledge.
+* Endo lets applications endow packages with limited powerful objects and
+ modules. As they say, you can't spell "endow" without "endo"!
+* So, "E.N.Do" forms the acronym "Encapsulated Node Do".
+
+So, just as "soo-doo" (super user do) and "soo-doh" (like "pseudo") are valid
+pronunciations of `sudo`, "en-doh" and "en-doo" are both valid pronunciations of
+`endo`.
+
+
+## Package Descriptors
+
+Endo uses [Compartments] from the [SES] shim, one for each Node.js package
+your application needs.
+Endo generates a compartment graph from Node.js packaged module descriptors:
+the `package.json` files of the application and all its dependencies.
+Consequently, an application must have a `package.json`.
+
+Each package has its own descriptor, `package.json`.
+Some standard properties of the descriptor are relevant and used by a
+compartment map.
+
+* `name`
+* `type`
+* `main`
+* `exports`
+* `browser`
+* `dependencies`
+* `files`
+
+The compartment map will contain one compartment for each `package.json`
+necessary to build the application.
+Like Node.js, Endo trusts the package manager to arrange the packages such that
+a satisfactory version of every package's dependencies rests in a parent
+directory, under `node_modules`.
+
+The `main`, `browser`, and `exports` properties determine the modules each
+package exports to other compartments.
+
+The `exports` property describes [package entry points] and can be influenced
+by build _tags_.
+Currently, the only tag supported by Endo is `import`, which indicates that the
+module map should use ESM modules over CommonJS modules or other variants.
+
+> TODO
+>
+> A future version may reveal other tags like `browser` to prepare an
+> application for use in a web client.
+> For this case, Endo would prepare a JSON manifest like an `importmap` (if not
+> precisely an `importmap`).
+> The "compartment map" would be consistent except when the dependency graph
+> changes so updates could be automated with a `postinstall` script.
+> Preparing a web application for production would follow a process similar to
+> creating an archive, but with the `browser` build tag.
+
+The `browser` and `require` tags are well-known but not yet supported.
+The `browser` tag will apply for compartment maps generated for use on the web.
+The `require` tag is a fallback for environments that do not support ESM and
+will never apply.
+
+If no `exports` apply to the root of the compartment namespace (`"."`),
+the `main` property serves as a default.
+
+> TODO
+>
+> The absence of `exports` implies that _all_ of the modules in the package are
+> valid entries.
+> Endo does not yet support packages that do not name all of their in
+> `package.json`, which is unfortunately a significant portion of packages in
+> `npm`.
+
+The `files` property indicates all of the files in the package that
+should be vended out to applications.
+The file set implicitly includes all `**.js`, `**.mjs`, and `**.cjs` files.
+The file set implicitly excludes anything under `node_modules`.
+
+> TODO
+>
+> In Node.js, a module specifier that corresponds to a directory implicitly
+> redirects to the underlying `index.js` file.
+> Capturing a full list of `files` in a browser compartment map would allow
+> a compartment to follow these redirects without chancing a wasted round trip
+> to get a File not Found error.
+> This could alternately be solved by inferring the redirect when an internal
+> module specifier has no extension.
+
+> TODO
+>
+> Endo does not yet do anything with the `files` globs but a future version
+> of Endo will collect these in archives.
+> Endo should eventually provide the means for any compartment to access its
+> own files using an attenuated `fs` module or `fetch` global, in conjunction
+> with usable values for `import.meta.url` in ECMAScript modules or `__dirname`
+> and `__filename` in CommonJS modules.
+
+Officially beginning with Node.js 14, Node.js treats `.mjs` files as ECMAScript
+modules and `.cjs` files as CommonJS modules.
+The `.js` extension indicates a CommonJS module by default, to maintain
+backward compatibility.
+However, packages that have a `type` property that explicitly says `module`
+will treat a `.js` file as an ECMAScript module.
+
+> TODO
+>
+> Endo does not yet respect the `module` field as it currently
+> only recognizes ECMAScript modules.
+> For backard compatibility, be sure to indicate that any package is of `type`
+> `module` if it uses the `.js` extension for ECMAScript modules.
+>
+> A future version of Endo will support CommonJS modules and enforce this
+> behavior.
+
+> TODO
+>
+> Endo may elect to respect some properties specified for import maps.
+
+> TODO
+>
+> Many Node.js applications using CommonJS modules expect to
+> be able to `require` a JSON file like `package.json`.
+> As of Node.js 14, Node does not support importing JSON
+> using ECMAScript `import` directives.
+
+> TODO
+>
+> A future version of Endo may add support for source-to-source
+> translation in the scope of a package or compartment.
+> This would be expressed in `package.json` using a property like
+> `translate` that would contain a map from file extension
+> to a module that exports a suitable translator.
+>
+> For browser applications, Endo would use the translator modules
+> in two modes.
+> During development, Endo would be able to load the translator
+> in the client, with the `browser` tag.
+> Endo would also be able to run the translator in a separate
+> non-browser compartment during bundling, so the translator
+> can be excluded from the production application and archived applications.
+
+> TODO
+>
+> Endo may also add support for compartment map plugins that would recognize
+> packages in `devDependencies` that need to introduce globals.
+> For example, _packages_ that use JSX and a virtual DOM would be able to add a
+> module-to-module translator and endow the compartment with the `h` the
+> translated modules need.
+
+ [Compartments]: https://github.com/Agoric/SES-shim/blob/master/packages/ses/README.md#compartment
+ [SES]: https://github.com/Agoric/SES-shim/blob/master/packages/ses/README.md
+ [LavaMoat]: https://github.com/LavaMoat/lavamoat
+ [package entry points]: https://nodejs.org/api/esm.html#esm_package_entry_points

feat(endo): Add Endo README and DESIGN (#337)