Dynamicland Geokit: Augmented Reality Maps
Notes from Dynamicland: Geokit

      
Hey ___,
I know you've been curious about some of the stuff I've done at
Dynamicland. It might be a while until you
can make it out here and visit in person, so I'll try to explain one
project of mine, Geokit, from top to bottom, to give you some of the
flavor.
I'll discuss what Geokit actually is and what you do with it
first, and then I'll explain how it's made. That explanation might
help you understand how people make things in Dynamicland in
general. It's the kind of detail that isn't covered in the zine or on
the Dynamicland website.
I sort of hope you read this, come here yourself, and reimplement
Geokit better.
What Geokit is


Geokit is a 'kit' or 'library' for building and viewing maps.
One Dynamicland researcher, Luke
Iannini, writes about his experiences
with Geokit:

The real estate group that owns the building Dynamicland lives in
uses their purchasing power to influence the city to improve
infrastructure to the underserved areas of Oakland they develop
in. The head of this initiative was visiting the space, and she
began describing her work while we were gathered around the lunch
table, which happened to have Geokit spread across it looking at
Oakland. Without breaking the flow of conversation, I dealt the
transit card to display the bus routes, and she grabbed the “zoom
and pan dial”1 without any instruction to zoom in on a portion
of West Oakland — noticing a huge hole in route coverage she’d never
seen before where she knew there were tons of working families. We
spent the next 15 minutes exploring as she taught me more about the
city's transit details than I ever knew I wanted to know.
In another case at a party I learned that a person I’d just met grew
up in the same city as me (Tucson, AZ) when she searched for it to
print it out, and we then bonded deeply over looking at the
incredibly stark racial divisions in North and South Tucson that we
knew from experience but had never seen laid out in raw data.
I’ve never had experiences like these in any other medium. Buildings
don’t tend to have detailed racial, transit and elevation maps of
every city in the US on the walls. In theory you could pull this
data up on your personal iPad, praying there aren’t any embarrassing
notifications and hopefully remembering what you were even going to
look at by the time you’re past the lock screen looking at 24
rainbow-gradient icons and red bubbles, but I am personally not in
the habit of pulling out my devices in the middle of parties. In
both of these cases these just-dynamic-enough maps were places at
the party, just like the appetizers table and the piano, where
people could casually gather, play and converse.

Geokit lives on a shelf when it's not in use, so I'll take it off the
shelf and lay it out on a table:


Taking Geokit off the shelf. Some other kits below it; general tools on
the right

Geokit in action
I'm going to pull up a map of San Francisco and zoom into Dolores
Park.
I'll start by typing into the "Geomap above from place search" page,
which turns the giant blank page above it into a map.


  

Typing places into Geokit and seeing maps

If you were here in person, I'd hand you the keyboard, and I bet
you'd want to type your hometown in. But I'll just use San Francisco.
OK, now I want to zoom in to the park. I have a tool for zooming, the
"Map magnifier with IR (infrared) dial" page (the smaller page on the
right). When I put that page on the table, it becomes a zoomed-in
'inset map.'

Inset map (right) shows a zoomed-in view of the dial's
location on the big map (left)

I also have to put this rotating dial on the big map of SF, which
controls both the location and zoom level of the inset map. Moving the
dial pans the inset map, and rotating the dial zooms it in and
out. Being able to
directly
move the map position is really satisfying.
In this next clip, I put the dial down and then pan over and zoom the
inset map to Dolores Park in the Mission.


  

Using the dial to pan and zoom the inset map to Dolores Park

Now I'm curious about this part of the neighborhood. How diverse is
it?  How has it changed over time? Are there stark dividing lines?
I'll apply some data layers: a color-coded demographic map ("racial
dot map"), and then street
lines and labels on top so the map is more understandable. I point
them at the big map first, then point them at the zoomed map (just to
show that both work).


  

Applying tilelayer overlays to the big map and then
just to the inset map


Seeing demographics around Dolores Park by pointing
the "Demographic dots tilelayer" page at the inset map. The color legend is
written on that tilelayer page – an advantage of using real paper
rather than a screen

Check out how the west (toward Castro) is bluer on the inset map,
while the east (toward Mission) is more orange.2 Imagine
having multiple people at the table: each person might have their
personal view on smaller pages, but the table can all share the big
map at the center.
Zoom lenses
To be honest, this whole project started with a very clear
image I had in my head. I wanted to make zoom
lenses, where you could point a lens at a map to zoom into it, then
point a lens at that lens... so I want to show you those lenses now.


Applying a zoom lens to the inset map, then
applying a zoom lens to that zoom lens

You can see me playing with the OpenPTMap (public transport) tilelayer
as I mess with zoom, applying it here and there at various zoom levels
(it propagates down from wherever I put it). The layer shows some bus
lines, as well as the J-Church light rail line.3 I'm
trying to get the flexibility of these tools across to you: layers
work everywhere and zoom works everywhere, and you can pull a tool out
and use it wherever and whenever you want.
Here's my play on Powers of
Ten in Geokit:
starting from the Earth, I apply a zoom lens, then a lens to that
lens, and so on, until I get down to our neighborhood in Oakland.

I can even print a map. I have a printed map of San Francisco here,
for instance. I use exactly the same tools on this printout: I look at
public transit lines, then demographics, then I zoom and pan around
east of Golden Gate Park.


Working with a printed map: applying tilelayers
and zoom lens

Even if Geokit isn't drawing any tiles on a map and it's just a
printout, it's still a fully functioning map.

…and a "map" page is just any page which represents a (lon1, lat1, lon2, lat2) geographic box, even if it doesn't render any tiles!(e.g. the printed maps are still meaningful maps even though there is no projection on them)— Omar Rizwan (@rsnous) April 29, 2018

I've been using San Francisco as an example, but I could have entered
any place in the world, and everything would have worked exactly the
same.4
The one-page 'engine' page makes all these maps and tools work, and
that little engine doesn't need to know anything about the stuff built
above it. Basically, the engine takes a page representing some
geographic coordinates, and it draws some map tiles from the Internet
on top. I'll show how you might build that engine from scratch here in
Dynamicland, step by step.
All the parts
These are all the 'game pieces' inside the kit. Each page is both a
little computer program and an object that people and other pages
can see and work with. Some pages are face-down, but most of them are
face-up, which means they're running right now. (Face-up is the side
with the colored dots.)

You often point one page at another to make it do something, like
how the zoom lens points at the printed map, or how the keyboard
points at the text box which points at the big map.
We have a few ways to get a map of a place:

⌨️type a place name into the text box to search for it
use a premade map like my 📄 printed map of SF
apply the 🔍 zoom lens to an existing map: the lens becomes a zoomed-in map
of the green region it points at
use the 🎛 zoom & pan dial ('loupe') page on an existing map: it zooms into
wherever the dial is, and you rotate the dial to change the zoom
level

These parts are the ones in the kit right now, but anyone here can
make new parts: you could come here and write up a new kind of
tilelayer or a new way to make maps.
That's why I call Geokit a kit and not an application: it's naturally
extensible. You could come here, pull a keyboard off the shelf, point
it at any part of Geokit, and just start editing it and printing new
stuff for your own use. You might provide a new tileset to answer some
question you have, or come up with a new zoom mechanism to derive
maps. (I'll explain how to do both of those things in this writeup!)
You have the same status that I do, and you don't have to open your
laptop and download and recompile some source code if you want to
change something.
Background on Realtalk
Dynamicland is a place – a land – in
Oakland, California.
Realtalk is the programming
system that runs inside Dynamicland. It includes a superset of the
Lua programming
language. You
do 'pure' computation in Lua, but you use special Realtalk extensions
to do all communication with other pages and with the outside world.

An overview of Realtalk concepts and syntax, by
Tabitha Yong

The Realtalk 'operating system' already knows about some wishes as
part of its standard library, so we can say things like Wish (you) is labelled "Bella". and Wish (you) is highlighted "green". and they
work right away. But we can also make up our own claims and wishes:
-- Made-up claim demo
Claim (you) blahblahblah.

-- Made-up claim when demo
When /page/ blahblahblah:
    Wish (page) is highlighted "blue".
End

Acting on claim syntax that I just made up (full size)

Why is "Made-up claim demo" highlighted in blue? What will happen if I
flip "Made-up claim when demo" face-down?
Realtalk and the rest of Dynamicland
I'm mainly going to discuss Realtalk here, so I'm writing in a form
that will look almost like any other tutorial about how to do a
programming project.
I'm not going to talk about the social experience of
Dynamicland,
the community, or things particular to the physical lab, but those
things are integral to the Dynamicland concept! Knowledge transfer in
Dynamicland usually happens those ways, not through writing like
this. I just think others can do a better job of describing those
things, and I want to fill the gap of documenting 'programming.'
Check out @Dynamicland1 on Twitter
for a broader view.
Making Geokit
I'll start by creating a map page representing San Francisco, a map
page which won't actually do anything at first.
Then I'll build up the Geokit 'engine', which is about a page of code,
in four steps. (The engine will make the map page work.)


A starter three-line Geokit 'engine page' that just paints that map blue.


A better engine which downloads a map tile from OpenStreetMap and
displays it.


Support for multiple tile layers, from different sources, on one
map.


Stitching more than one tile together to get exactly the right map
view to fit the map page's area.


A map page that doesn't do anything (yet)
First, I'll make a San Francisco map page for Geokit to act on.
I want to make a claim that my map page represents the geographical
area of San Francisco, so that Geokit knows it's supposed to draw San
Francisco on it. I'm going to type in a claim whose syntax I made up:
Claim (you) is geomap of bbox (... coordinates ...).
At Dynamicland, you make a new page by

pointing a keyboard at an existing page someone else wrote (ideally
with code you want to reuse),
then patching its code in place to do what you want,
and then printing it out.

I'll do that to make my map page, although I'm not keeping any of the
original code in this case. I also add a bunch of linebreaks so the
printout covers the whole page.



Is this a map? Well, it's a page – a program, basically – which is
making a claim about itself. (See Tabitha Yong's cheat sheet, shown
earlier, for more about claims.)

Our map page (full size). It's
the 17,595th page!

-- San Francisco map
Claim (you) is geomap of bbox ({left=-122.527, bottom=37.664, right=-122.348, top=37.851}).
You can think of it as setting a dynamic 'property' that other pages
can see: "I claim I'm a map representing the box with corners
(122.527° W, 37.664° N) and (122.348° W, 37.851° N)." (Those
longitude-latitude pairs are, very roughly, the bounds of San
Francisco.)

A debug aid: showing claims made by each page on
the table (full size)

You can make whatever claims you want, but there's no guarantee that
anything will happen as a result. Right now, nobody cares about my
geomap claim, so the map is inert. But soon, the Geokit 'engine' will
be querying for all the pages that make that kind of claim, and it
will give them map behavior.
1. Starting on the 'engine' page
Next, I'll start building the Geokit 'engine', which is a separate
page: it'll look around for maps like the one I just made, then draw
on them.
My first attempt is a minimal test to show how I can recognize
and act on maps: it colors all maps solid blue.

Geokit engine page: colors all maps blue

-- Geokit 1st try: blue
-- On each geomap, draws blue.

When /map/ is geomap of box /bbox/:

    Wish (map) is highlighted "blue".

End
I'm using a wish here, which is a sort of broadcast request for
something in the world to be different: in this case, whenever I see a
map, I wish for it to be highlighted in blue.
Some other page happens to notice that wish and fulfill it using a
projector, but I don't know or care about that
mechanism.
I'm not addressing the wish to anyone in particular. When I put this
page together with the map, sure enough, the map turns blue.

Turning the engine on and coloring SF blue

To show you that the engine responds to every map, not just one, I
made a map of New York as well. See for yourself: it turns blue the
same way.


  

Making both New York and SF blue

2. Drawing one map tile
OK. Now I want to draw actual map tiles from
OpenStreetMap instead of
that placeholder blue. I'll start with something small: draw one
tile of SF at a fixed
level of detail, big enough that it fits the page.

Drawing a map of SF!

I'll show you (an abbreviated version of) how I edited our previous
"paint maps blue" 1st try and ended up with this 2nd try, which draws
an actual map tile. I'll show you the code next, so don't worry about
squinting to read all the code in the video. (The projection is much
clearer in person!)
I start by changing "blue" to "orange" and then "red" to show
you how I can edit the page in place. Then I extend the code so that
it downloads and draws a hard-coded tile at (11, 327, 791), which is
San Francisco. Finally, I make the code more general. I copy and paste
the utility function deg2num from another page, which I use to
compute a tile based on the bbox encoded in the map page, along with
a drawing size for that tile based on the page's physical height and
width. Once it works, I print the page out, so I have a permanent,
physical version.


Editing Geokit 1st try into Geokit 2nd try, which
actually renders a map. Printing the final 2nd try out at the end


Geokit engine page, 2nd try: draws one OSM tile on
each map. Key When block is highlighted

This code is some of the most 'traditional' and complicated in the
whole kit. Honestly, it's a little unidiomatic to have so much
'normal' code bunched together. It doesn't help that it has to
interface with the outside world (the Internet and filesystem).
You might start by looking at the code block I highlighted, the When /map/ is geomap... block at the bottom, which runs for each map on
the table:
local URL_TEMPLATE = "https://tile.openstreetmap.org/%d/%d/%d.png"
local ZOOM = 11
When /map/ is geomap of bbox /bbox/,
     /map/ has width /width/, /map/ has height /height/:

    local xtile, ytile = deg2num((bbox.left + bbox.right)/2,
                                 (bbox.top + bbox.bottom)/2,
                                 ZOOM)
    local url, path = get_tile_url_and_path(URL_TEMPLATE, ZOOM,
                                            math.floor(xtile),
                                            math.floor(ytile))
    local tilesize = math.min(width, height)
    When tile URL (url) is downloaded to path (path):
        local ill = new_illumination()
        ill:image { filename=path, scale=1,
                    x=0, y=0,
                    width=tilesize, height=tilesize }
        Wish (map) has illumination (ill).
    End

End


Have the When bind variables:

map: reference to the map page
bbox: the left, right, top, bottom coordinates of this geomap
width: the map page's usable width in inches (7.5)
height: the map page's usable height in inches (10)



Compute the geographic center of bbox and use
deg2num
to find the corresponding OSM tile coordinates xtile and ytile
(327 and 791, respectively, in this case)


Construct the URL https://tile.openstreetmap.org/11/327/791.png and
the download path
shared/geokit-tiles/https___tile_openstreetmap_org_11_327_791_png.png


Compute the tilesize at which to render the tile image, which is
the minimum of the page width and height. In this case, it's about
7.5 inches.


Once the tile is downloaded5, build an illumination (the
built-in graphics library) that draws the tile image. Then Wish to
draw that illumination on the map page.


I patched in some log statements to show you the live values of
bbox, width = 7.491, height = 10.001, xtile = 327, and ytile = 791:

Viewing live values with log statements

3. Modular tile layers
I can make a ten-line change to the engine and get a huge amount more
flexibility. Instead of hard-coding that URL_TEMPLATE = "https://tile.openstreetmap.org/%d/%d/%d.png" in Geokit, I'll let
pages make Wishes to draw any tilesets from any URLs they want.
To let the user specify what layers to draw, I've made individual
tilelayer pages. Each page represents a layer: when you point a layer
page at a map page, the layer wishes to draw that layer on the map.

Here's the source code for the "Demographic dots tilelayer", for
example:
When (you) points "up" at /map/:
    Wish (map) shows tilelayer
        "http://demographics.virginia.edu/DotMap/tiles4/%d/%d/%d.png"
        with options ({max_zoom=13}).
End
I edited the engine so that it responds to these Wishes, rather than
looking for any geomap and drawing that hard-coded URL_TEMPLATE
from last time.
Here are all four tilelayers in action on the San Francisco
map. Notice how I chain layers to put them on top of each other!

And here is the new engine page:

Geokit engine page: allows custom tilelayers for
each map. Changes from 2nd try are highlighted

This 3rd-try Geokit is almost unchanged from the 2nd try, except for
this section, from line 34 down to the extra clause in the key When
block that handles maps:
When /someone/ wishes /page/ shows tilelayer /url_template/ with options /opts/,
     /page/ wishes /target/ shows tilelayer /something/ with options /something/,
     page ~= target:
    opts = rt.table(opts) -- Make a mutable copy of opts
    opts.priority = 1 + (opts.priority or 1)
    Wish (target) shows tilelayer (url_template) with options (opts).
End

local ZOOM = 11
When /someone/ wishes /map/ shows tilelayer /url_template/ with options /opts/,
     /map/ is geomap of bbox /bbox/,
     /map/ has width /width/, /map/ has height /height/:
Start from the bottom: the added When /someone/ wishes /map/ shows tilelayer /url_template/ with options /opts/ clause makes my
map-drawing code run separately, for every tilelayer wish for each
map, rather than running only once for each map.
It binds url_template to whatever the tilelayer page said, whereas I
had a hard-coded URL_TEMPLATE constant last time. The tilelayer
page code makes a wish like Wish (map) shows tilelayer "https://tile.openstreetmap.org/%d/%d/%d.png" with options ({}).", which results in url_template = "https://tile.openstreetmap.org/%d/%d/%d.png" and opts = {}.
The first When block (When /someone/ wishes /page/ shows tilelayer...), starting on line 34, is really interesting. It's what
lets me put layers on top of each other by chaining them. When a
tilelayer page (someone) is pointed at another tilelayer page
(page), and that page is itself pointed at target, then
someone's tilelayer wish is 'forwarded' to target, but with its
priority boosted by 1 (higher priority means drawn on top, so
someone's tilelayer gets drawn above page's tilelayer). That means
that a tilelayer pointed directly at the map is drawn at the bottom, a
tilelayer pointed at that tilelayer is in front of it, and so
on. The outermost tilelayer is drawn on top.



Chaining layers to put one on top of
another. Center labeled to show 'tilelayer forwarding'

A note on the Realtalk model
Suppose you were implementing this tilelayer manipulation in
traditional software. You'd probably make a list of tilelayers, with
(+) and (-) buttons. You might put a check box next to each
tilelayer. You'd have to implement drag and drop, and you'd have to
handle events every time the user reordered the
layers.6
This step starts to illustrate the power of the Realtalk model, in a
way that I've tweeted about
before. Look
at how I've handled layers. Each layer is a page, and you put the
layer down and point it to enable it and set its position. Nobody had
to implement a checkbox or handle a drag event or synchronize an array
with a UI view.
When I first showed up at Dynamicland, I did a lot of what I think of
as 'traditional programming.' That means using a page essentially as a
computer monitor: you hack out a lot of Lua code, print it, and it
renders something cool on that page. Maybe it takes input based on
different regions of the page. I've seen programmers do this over and
over on their first exposure to the system, and I fell into it too: I
would say it took me about four months to understand how to really
take advantage of Realtalk.
One way to think about Realtalk is that a lot of code in the outside
world is about displaying lists and letting people pick from
lists.


Mapping software has to show a list of layers, and make them
checkable and uncheckable, and make them draggable, and handle all the
events, and remember what the state is.


A photo app shows you a list of photos, and then you open one, and
then you see a toolbox of operations you can apply, and then you pick
one...


Even command-line programs let you ask for a list of source branches
or a list of disk partitions or a list of files or whatever, and have
some set of subcommands you can apply to your object of choice.


In Realtalk, most of that code falls away – provided that you
represent your objects as individual physical pages. Once you have
physical pages, you automatically get the operations of the physical
world: placing and picking up objects, moving and grouping objects
in space, pointing objects at each other, and so on.

another purpose of this dramatic simplification is to enable STEPS-like human-scale computing systems, and the purpose of *that* is to enable Nile Viewer-like representations that let people to see and understand the behavior of the entire system at every level of abstraction— Bret Victor (@worrydream) July 15, 2018

We have a rule of thumb that no program should be longer than a page
(73 lines!) If you have a longer program, you're probably doing
something wrong, and you should split it up into little pages that
work together.
And there's a lighter rule of thumb floating around, that most code
should be 'purple text' – Whens and Claims and Wishes, which the
editor highlights in purple – not gobs of plain Lua code.
These rules suggest that there's a distinct Realtalk idiom, which is
very different from just writing Lua for projectors and cameras. It's
something like objects or actors (where the pages are objects), and
reactive programming, and reusing functionality from the real world
wherever possible.
I don't claim that Geokit is an ideal example: the engine page is
pretty complex, and even my use of the projector to render maps is
questionable, since any static map could have been printed
instead. But the layer pages show a little bit of this power.
4. Fitting to exact coordinates
Finally, I want to get rid of that one-tile hack. If a map page is
supposed to represent the box with corners (122.527° W, 37.664° N) and
(122.348° W, 37.851° N), I want it to display a map of that exact
area, not just some nearby tile image that OpenStreetMap serves
up.


Comparing our previous engine (left) to the
behavior I want (right). Hover over right to see a patched view with tile boundaries
and coordinates.

We need to figure out the right level of detail, get multiple tiles at
that level, and then stitch the map together.

Geokit engine page: draws tiles to fit the map's
bounding box. Changes from 3rd try are highlighted

I've made this engine development into a GitHub
gist
as well, so you can look at the
revisions
to see how I got from step 1 to here.
I've only changed the bottom When block from the previous engine:

When /someone/ wishes /map/ shows tilelayer /url_template/ with options /opts/,
     /map/ is geomap of bbox /bbox/,
     /map/ has width /width/, /map/ has height /height/:

    -- Made-up formula to guess a good zoom from width of bbox.
    local zoom = math.log(360 / (bbox.right - bbox.left)) / math.log(2)
    zoom = math.ceil(zoom + width/16 - 1) -- experimentally determined width factor
    local max_zoom = opts.max_zoom or 17
    if zoom > max_zoom then zoom = max_zoom end
  
    local xtile_left, ytile_top = deg2num(bbox.left, bbox.top, zoom)
    local xtile_right, ytile_bottom = deg2num(bbox.right, bbox.bottom, zoom)

    local tilesize = width / (xtile_right - xtile_left)

    for xtile=math.floor(xtile_left),math.floor(xtile_right) do
        for ytile=math.floor(ytile_top),math.floor(ytile_bottom) do
            local url, path = get_tile_url_and_path(url_template, zoom, xtile, ytile)
            When tile URL (url) is downloaded to path (path):
                local ill = new_illumination()
                ill:image { filename=path, scale=1,
                            x=tilesize * (xtile - xtile_left),
                            y=tilesize * (ytile - ytile_top),
                            width=tilesize, height=tilesize }
                Wish (map) has illumination (ill) with priority (opts.priority or 1).
            End
        end
    end
End


Determine the zoom
level
zoom – higher zoom means more tiles are used to represent the
same area. I
should point out that there's no single correct zoom level to
draw a particular map: it depends on how much detail you want to
show. You probably want higher zoom the higher your display
resolution is.
Here, I take the logarithm of (360 / the distance between the left
longitude and right longitude of the map), so a map covering a
smaller geographical area gets higher zoom. Then I increase the
zoom by a bit, based on the physical size of the map page (width / 16). For this San Francisco map, I end up with zoom = 11.


Compute the tilesize in inches at which to render each tile, such
that the tiles exactly fit the page.7 In this case, we get something
like tilesize = 7.44 inches.


Loop through all the tiles and download and render each one, just
as we did when we drew a single tile previously.


With perfect fit working, maps of different sizes also work
great. Here's the New York map I made earlier.

New York and SF. Hover to see tile boundaries


I showed a lot of tools, like zooming and searching for places by
name, at the beginning; how do they work?
You can implement all of them without changing the engine at all. I
won't go into as much detail about them – maybe some other time – but
they all fundamentally come down to the claim that a page is geomap of bbox (...something...). I've used constant coordinates for the
bbox so far, but they can also be dynamic values, which is how we
get dynamic maps.
Zoom lens

Applying 2 zoom lenses to get from the whole US to
Northern California

The zoom lens is surprisingly simple to implement, and it was the
first tool I made. Wanting to make a zoom lens inspired the whole
project, after all.

Zoom lens page: zooms into map to its left

The lens draws the green rectangle on the map it's zooming into, then
computes a bbox so it can make its own claim to be a geomap.
It inherits tilelayers from its parent, but scales their priorities
down so the user can easily put layers on top. It also inherits
highlighted points, which are an addon I'll discuss next.
Addons
I need some addons to support the rest of the tools.
Wish (...) is geomap containing bbox (...)

Addon to handle geomap wish

This addon handles Wish (...) is geomap containing bbox (...). It
lets a page become a map of an area that doesn't exactly match its
aspect ratio. I'll need this functionality soon, when I want to map an
arbitrary place that the user types in.
For example, if you have a page with a 2:1 aspect ratio, but you
want to make page a map of square San Francisco, you can do Wish (page) is geomap containing bbox (...SF coordinates...), and this
addon will give page a claim for a wider bbox which still contains
San Francisco.
If you already knew a bbox that matched the page's 2:1 aspect
ratio, then you wouldn't need this wish addon: you could simply make
the original Claim (page) is geomap of bbox (...). that we
implemented in the engine.
Wish (...) highlights geopoint (...) with label (...)

Addon to handle highlighting points

This addon handles Wish (...) highlights geopoint (...) with label (...). You can label specific places on a map with this
Wish.
Place search

Typing in "USA" and getting a map

Place search goes to Google Maps with whatever you typed, gets a
suggested viewport back, and then makes a Wish (to the geomap-wish
addon from earlier) to construct a map containing that viewport. It
also sets the OSM tilelayer as a default.

"Geomap above from place search": point keyboard
at it and type, and this page turns the page above it into a map

Zoom and pan dial

Zooming into part of Queens with the "IR
dial." Notice the 4 retroreflective dots on the dial, which reflect
infrared light back to a camera, so I can track the dial's position and angle

This dial pan-and-zoom mechanic was suggested and implemented by some
visiting designers from a technology company you've heard of. We
showed off a few things in the morning, including a generic
dial and my
Geokit zoom lenses, and one designer (who "didn't program")
immediately thought about how to combine the two.
How could we control the zoomed-in map directly and in context,
instead of having the lens (which sits at a distance from the original
map)? With that interaction idea and a couple of hours of work in the
afternoon, we had a working prototype for this dial. And we were
working at a table together, with pieces of paper, a marker for
scribbling, a dial, and a keyboard, not with laptops.
I love the direct manipulation; just as hoped, moving the dial on the
map to pan feels much nicer than moving the zoom lens out on the side.
The green number on the dial (from 0.0 to 1.0) is the zoom level,
where 1.0 is the most zoomed-in and 0.0 is the least.
There's also a green rectangle, which represents the exact area that
the inset map is showing. When I increase the zoom level, the green
rectangle shrinks and the inset map zooms in.

"Map magnifier with IR dial": zooms into map to its
left. Expects IR dial to be on that page; turn the dial to zoom and
move it to pan

The original dial was just a (small) normal page, which you'd put on
top of the map page. But the light from the projector interfered with
dot detection, especially with the racial dot tilelayer's bright
colors in place, so it was unreliable. To avoid that problem, I'm
using an IR (infrared) dot detector (/map/ contains tracking dots /dots/) which tracks the gray-looking retroreflective dots. (Ideally,
the dial would be a normal page, though.)


  

Looking at the underside of the IR dial
peripheral. How does it spin?

One of the best things about Dynamicland is making peripherals and
using them with the computer right away. We don't need to implement
simulated inertia or make a virtual drag handle, since we simply track
the physical dial.
The dot furthest from the average of the dots is the 'head' of the
dial, which I use to figure out the dial's angle.
It'd also be interesting to draw the zoomed map on the dial itself;
we've played with this a few times, but there are still some issues
(interference, performance, rotating view properly).
Printing maps

A printed map of SF. Notice the claim at the top

Printing is very much a work in progress. You point the "Print
static map" page at a map page, and it prints a static map of that
place, with a Google Maps view on it.

"Print static map"

There's a lot of fiddly calibration to do here, and it's not perfect
right now. It goes to Google's Maps API and gets an image around the
center of the map it's pointed at, guessing the right radii around
that image to line up with the existing map. It injects a claim, so
the printed map will be a valid Geokit map too.
It's been surprisingly useful as a debugging tool, since I can turn a
misbehaving dynamic map, which is hard to keep stable, into a static,
printed test case.
Conclusion
These pages are all there is! Everything I showed you at the beginning
of this writeup was implemented by the actual pieces of paper in front
of me, and now you've seen how all that code works.
Yeah, I'm relying on some 'built-in' functionality, but all that
behavior is also implemented on pages here in Dynamicland. When
you're here, you can walk over to a board and read through those
system libraries. There is no underlying stack hidden on a computer
hard drive or on GitHub. On this whiteboard, you can find the page
that renders to the projector when I Wish (map) has illumination...,
the page that makes a GET request when I query When URL (...) has contents /contents/:, and the page that parses those Claims and
Wishes and Whens in the first place:

In Dynamicland, software is supposed to be this simple or
simpler. Software in Dynamicland is meant to be continually read and
tweaked and discarded by users; that's why each page has its source
code printed on it.
Why maps?
Why maps? Personally, I've been interested in maps and cities and
public transportation and infrastructure for a few years, and I
haven't gotten to do much original work around that interest, so
Dynamicland seemed like a good place to explore it. I was excited to
build an idiomatic-feeling 'kit' with parts, instead of a one-off
program like my earliest work here.
I think maps highlight the advantages of breaking out of the
screen. We insult an entire tradition of large, information-rich maps
when we cram them onto tiny phone screens, then crowd around them and
make finger
gestures
to squeeze answers out bit by bit.
I figured that in Dynamicland, we could have maps that actually looked
like maps. We could have multiple people sitting around a map at a
table, sharing a large view, but also applying computational filters
and derivations to get at their individual interests. You could
explore many places and ideas at once, and keep track of them on the
table in front of you, instead of constantly sacrificing context on a
limited screen.
Dynamicland is great at spatial interfaces, and maps are
spatial. You get tools like dot tracking, page tracking, whiskers, and
rotation, as well as whatever else you can tie in from the physical
world.
And maps involve real information from the world, which makes them
more relatable than a visualization of some mathematical model. I've
seen plenty of people whip out lenses and turn zoom knobs and poke at
layers in an attempt to learn about the place where they grew up. Maps
are an early opportunity to use this technology to ask real questions
about the world, the same way that I use my phone or laptop to ask
questions.
On documentation
I think a lot of the material and discussion about Dynamicland has
been either too vague or too focused on the wrong details. It's been
hard to talk about the place to people who haven't visited. You lapse
into phrases that sound like platitudes.
I'm taking a different approach here. I want to be concrete about how
you make things and about the details of the programming
experience. I'm hoping that this detail will actually do a better job
of communicating the bigger ideas.
I think there are a few reasons why there hasn't been much written
about Realtalk, especially writing beyond 'first impressions':


Written essays feel like an awkward medium for talking about
Dynamicland. Because it's a physical space, its inhabitants are used
to communicating in person, with demonstrations at hand. You
wouldn't ordinarily write or read a document like this one; if you
want to know how something works, you put it out on the table and
read the code on the pages, and maybe ask someone in the space
(since it is a shared space, people will be there!)


Building external attention hasn't been a big goal: the focus has
been on in-person outreach, fundraising, and development within the
space.


Even among people who've visited Dynamicland, not many actually know
Realtalk well, so not many can write this kind of overview. As with
any creative tool, becoming fluent takes time. It took me at least a
few months.


There may also be some embarrassment about having a traditional
text-editor-oriented environment, given the earlier work of
Bret and other researchers in the lab.


There was a fear that people would fixate on details and copy them
blindly while missing the values, as happened to the Xerox Alto GUI.


But focusing on these programming details still seems better than
focusing on things like the dots and projectors and computer vision,
which are obvious to anyone who visits and so get a ton of
discussion. I see that stuff – dots, the nature of the page database,
cameras and projectors, occlusion, comparisons to augmented reality –
purely as implementation details, and I don't spend much time thinking
about it; I haven't discussed it in this writeup at all!
The pages/wishes/claims/whens programming model is the true 'spirit of
Realtalk', even if ideally we'd change programming practice in even
more radical ways: get rid of the keyboards as well, or add more
direct manipulation, or introduce better visibility for
debugging... or abolish programming entirely?

One suggestion for what 'programming' in
Dynamicland could look like,
by Nicky Case (full size)

In fact, I'd argue that the programming model is very
under-discussed. It is more important than you may think from what you
see online. A lot of what's new about Dynamicland is that it's a
programming system! Compare projection-mapping demos you see from the
1990s and 2000s, from various academic and industrial labs. They may
have similar interface techniques, and they may demo cleaner, but the
implementation is some big blob of C++ that is written backstage. You
can't walk in and do your own program. The pieces of paper in
Dynamicland are not just display surfaces for a demo someone wrote on
their PC: they're used to program the system directly.
You could implement Geokit in JavaScript or C++, and build your own
projection mapping and computer vision rig, and you could implement
any Dynamicland program that way, but it would be a lot of work. I
certainly would never have done it. Dynamicland lets you take all that
as given.
There's also lots of Dynamicland coverage that is just out of date!
Using the system today (August, 2018) is a very different experience
compared to using the system a year or even six months ago: there are
thousands of example programs, computing surfaces are everywhere in
the space, projectors are high-resolution enough to display detailed
data and text, everything runs way faster, you edit with keyboards
within the system instead of laptops – the fundamental design is
intact, but the subjective experience has changed a lot, especially
when you're making things.
I hope this writeup has given you a better idea of how Dynamicland
works and what's powerful and new about it. I've talked about some of
these ideas @rsnous on Twitter, and I'd
like to write and release more in public – let me know what you want
to hear about.
best,
Omar
My thanks to John Backus, Jason Brennan, Nicky Case, Mary Rose Cook,
Daniel Feichtinger, Katie Gu, Jacob Haip, Joshua Horowitz, Luke
Iannini, Lauren Killingsworth, Tristan Navarro, Lisa Neigut, Paul
Sonnentag, Bret Victor, Roshan Vid, Elizabeth Yang, and Andy Zhao for
their thoughtful comments and feedback on this writeup.





as I'll discuss later, made by a visiting designer who
"didn't program" ↩︎


back during the 2010 U.S. Census,
at least ↩︎


The blank white areas you see on some of the lenses are
a bug in Geokit: they're areas where no public transport runs, so
OpenPTMap doesn't provide any tile images
(good vs.
bad). If
Geokit gets a 404 instead of a tile image, it renders the 404 as
though it's a valid PNG – which appears as an opaque white tile,
blocking out the tilelayers underneath. Those tilelayers
underneath are supposed to still show, so that's a bug. ↩︎


except the U.S. Racial Dot Map,
outside the U.S. ↩︎


Maybe you're wondering why we need to save the tile to a
path on disk. Why not render the PNG data in the contents
variable on line 20 directly? That's probably the right way; it's
a quirk of the current illumination API that you have to provide
the rendering engine an image filename instead of data. ↩︎


The operation of the word user is interesting when
we talk about traditional software. For example, in a traditional
application, if the programmer doesn't explicitly provide a
dialog to add a new kind of tilelayer, the user can never add
their own tilelayer. Even if the application is open-source, how
many people are able or willing to go in and make that change?
There's something fundamentally authoritarian going on: the
programmer makes the application, and the user uses it. ↩︎


I'm assuming that the bounding box declared by a map page
matches the map page's aspect ratio. That is, if you say you
represent a 16:9 geographical area, you should be a 16:9 page. You
could imagine a design that automatically refits the drawn region
to the page size, but the Geokit core should be as simple as
possible: that function can be built on top later. I have an
add-on, "Wish is geomap containing", which does the automatic
refitting for applications that need it. ↩︎




    