text: .
  For the last 15 months or so our small team has been writing applications using Linked Data as a fundamental architectural pattern. Our applications are not specialized semantic web applications - they are fairly typical business applications. Our applications do not function stand-alone – each is a component of a larger system that appears to an end-user as a single, coherent (hopefully) web site. Our interest in the Linked Data model is primarily for its ability to enable multiple independently-deployable applications to integrate as a larger system in this manner.
  Our server applications are written in Python and our client applications are written in JavaScript. Our interest in JSON – a media type - is really secondary to our need for in-memory representations of data that are suitable for programming on both the client and the server. The value of JSON for us is primarily the ease with which it can be converted to and from in-memory data structures that are convenient to programmers. Our experiences programming with our data have led us to define and use 3 different in-memory organizations of the same data for different purposes. I will call these
  1. Object graphs
  2. Sets of triples
  3. Object lists
  Adopting RDF was not in itself a goal for us – it was a pragmatic way to represent our business data. We were already pretty keen on RDF when we started for various reasons, but our experiences with using these 3 data organizations, mapping between them, and translating them to and from wire formats have redoubled our enthusiasm for RDF. The RDF model provides us with a single semantic model that we can use to reliably convert between different in-memory organizations and wire formats using standardized implementations without any loss of meaning, and we have found this to be very powerful, as well as convenient.
  The primary purpose of the ‘object graph’ programming data structure is to make it easy for programmers who have knowledge of the application data model to write clients - especially, but not exclusively, UIs. This is the normal use of JSON. This goal has some consequences:
  1. We do not use URIs for predicates in this organization. Using URIs for predicates is important for clients that don’t know the application data model, but in fact is a major nuisance if you do. It is much easier to code using 'dc_title' as the name of a predicate than ''. [Of course, the underlying RDF model of our application still uses URLs for predicates – this is important for integration.] This means that the ‘object graph’ data structure would not be suitable for a generic RDF client that did not have application knowledge. That is fine – our system has good support for those sorts of clients too – the ‘object graph’ structure just isn’t the one for them.'}
  I deliberately chose to use JavaScript syntax here instead of JSON syntax to emphasize that our primary interest is in the programming data structures, not the wire format. The _subject property is conceptually the same as the @id property described in JSON-LD. We do not use @id because we came across situations where a popular programming library would fail when @id was used as a property name. ‘Object graph’ property names are simple names - we do not use the ':' character in property names either because some popular programming libraries will choke on those too.
  Although we don’t use URIs in the predicates in this data structure, we use proper URL predicates in the messages we exchange between clients and servers. We developed a simple JavaScript function that will convert the ‘object graph’ data organization to any of the other organizations, and also to various wire formats (media types). This function knows how to convert the ‘dc_’ prefix on predicate names to and from proper URL prefixes. Each instance of the function is customizable with mappings for namespaces it doesn’t already know (this is similar to the ‘context’ information of JSON-LD). It is important to us that the mapping information is in the conversion function, not in the programming data structure.
  Notice also that URIs property values are represented by instances of a URI class that we wrote. Since we assume that the programmers using this object have an understanding of the application data model, you might reasonably argue that we could use simple strings for URLs, which is what most JavaScript programmers are used to. [I will resist the temptation to rant about the fact that none of the popular programming languages has built-in support for the most important data type of the internet.] This is a reasonable argument, but we found that it is hard to write generic library functions if you rely on application knowledge to know what type something is, so we somewhat reluctantly changed our design to represent URIs as instances of an URI class. We do the same thing for dates, using JavaScript’s standard Date class, for similar reasons.';
  While many of our resource representations are simple like this, the most interesting and useful ones for JavaScript UI programming are not. Many of the more useful resource representations contain information about more than one resource. In the ‘Object graph’ data organization, these might look like]
  This example illustrates that when one object contains a URI reference, and the overall resource representation also contains information about the resource identified by that URI, then instead of making the value of the property be a simple URI object, we make it be a new programming object that contains all the available information for that URI. The example also shows that, unlike with naïve JSON, these objects form a graph, not a tree – in this example the same object appears in the items and back__ordered_items arrays.
  It turns out that the ‘object graph’ organization of data is close to ideal for JavaScript UI programming using popular UI frameworks like Knockout, and popular template technologies, like Moustache, and so the ‘object graph’ data organization is the one that most of our JavaScript client code uses. However, the ‘object graph’ organization of data is not ideal for every purpose – for example we calculate differences between graphs in several places in the UI processing, and that calculation is much easier to do on the ‘sets of triples’ organization described below. Because these organizations are semantically equivalent in the RDF model, we can freely convert between them and use whatever organization fits the current programming task best.
  Early on, we wrote software that knows how to serialize and de-serialize the ‘object graph’ organization to and from JSON in a fairly direct fashion, and we put it on the wire in HTTP requests and responses. In other words ‘object graph’ became a new media type for us. At the time, we thought this media type would be important, and we thought that we might use JSON-LD as a standard for it. In fact, it turned out that this media type is not the most useful one and we don’t use it any more. Here is why:
  1. JSON cannot handle graphs directly.
  Since the objects form a graph, and JSON can only handle trees, some special coding is required. This coding isn’t terribly difficult, but it is much too complex to put in-line in a program and so has to be put in a library function. We also want library support for mapping URL prefixes and for handling URI and Date types. Once the programmer has to use a library function to serialize and de-serialize the objects, and the programming objects are no longer 1-1 with the serialized JSON, the details of the wire format are not so important to the programmer any more – in the extreme, it wouldn’t really matter much if the library produced JSON or Turtle or RDF/XML. There is some pedagogic value in using JSON, since more people know how to read JSON than Turtle, and the implementation of our library functions is easier with JSON, but the direct equivalence to the programming structure is lost..
  Our original direction was to try to solve these problems by adding complexities to our media type and its processing, but we quickly decided there is a better way. What we do instead is break the object graph down into its constituent triples and serialize the triples in a format close to RDF/JSON. In the other direction, we build the ‘object graph’ format on the client from the triples. This solves the second and third problems above, and is no worse from the point of view of the first (requires use of a library function). The library functions that do this are not complex – it takes about 150 lines of straightforward JavaScript for bi-directional conversion.
  The fact that we produce the ‘object graph’ data format from triples on the client (or server), rather than exchanging it between the client and server, is a major protocol simplification compared with JSON-LD, because our equivalent of the JSON-LD ‘context’ information is no longer part of the contract between server and client – it is a purely local concern. The interface with the programmer is also simplified, because the mapping information is in the conversion function, not in the programming data structure.
  Since a ‘graph’ of RDF triples is the canonical model that we use to convert between in-memory data organizations and to create wire formats, it’s probably no surprise that we have one data organization that is very close to the native triples. There are at least two obvious options for a programming data structure that is close to triples – one is [[Subject, Predicate, Object]] and the other one is {Subject : { Predicate : [Object]}}, which is the data structure organization corresponding to the media type described in the RDF/JSON specification. Our experience is that the RDF/JSON-inspired structure is more useful – you can iterate over it almost as easily as you can the other structure, but you can often find what you want directly by lookup, whereas the first format always requires you to iterate. to be a simple value in the case of numbers, strings, and Booleans, a URI instance for URIs, a Date instance for dates, and an RDF/JSON-style structure only for other cases (which are rare enough that a lot of programmers might never see them). We also allow a single Object instead of an array of Objects. When we put these structures on the wire as JSON, we use simple values for those types supported by JSON (string, number, Boolean) and RDF/JSON-style structures for the rest, including dates and URIs. (This means we are not really RDF/JSON compliant currently, but we could easily change the wire format to comply if we had a reason to do so.)
  We also deviate from RDF/JSON in our interpretation of the semantic of the array of Objects. In RDF/JSON, the array is interpreted as an open set, but in our programming model array is interpreted as a closed List. We could bring ourselves into conformance with RDF/JSON by changing this:
  {'': {'': ['The Merchant of Venice', 'The Jew of Venice']}}
  To something like this:
  {'': {'': [{'type':'', 'value':['The Merchant of Venice', 'The Jew of Venice']}]}}
  Our library code would hide this detail from programmers using any of our programming formats.
  In our applications, we have been storing our data in a JSON database (currently MongoDB). The ‘Object Graph’ format we described above is good for programming, but we don’t like it as much for database storage and query. We see several problems with it for databases:
  1. Nested structures are always more complex to query and update
  2. Any nested organization you pick will be better to query for some use-cases and worse for others, while a flatter structure will be fair to all. This reminds me of a primary lesson we learned 30-40 years ago when we transitioned from hierarchical databases to relational ones – flatter is better for database data.
  3. There is no way in a single-rooted hierarchical structure to represent sets of triples that form multiple disconnected graphs rather than a single one. [I wish the RDF specifications had not chosen the word ‘graph’ to describe a set of RDF triples.]
  You might intuitively think that the ‘set of triples’ data organization would be a good one for a database, but it has a different problem. What databases know how to do is query on the value of predicates, and in the ‘set of triples’ organization, the ‘subject URLs’ are not the value of any predicate, and so are not queriable..
  This format is never seen by our programmers – it is entirely hidden behind our database access methods, and you will only see it if you inspect one of our databases using database tools. Our database access methods accept and return data in the ‘set of triples’ format, and even queries are expressed in a slightly extended form of the ‘set of triples’ format.
  The important thing about JSON itself, and the reason for its popularity, is that a JSON message also defines a data structure in popular programming languages. When you design a JSON message, you are really designing a programming API, and the popularity of your JSON will be determined by the attractiveness of the programming API it defines, not by the wire format.
  We also experimented with designing a solution that satisfies these goals, but ultimately abandoned it. Our servers already produce multiple RDF formats – e.g. Turtle, RDF/XML, RDFa and RDF/JSON – so we don’t really have a need for another one. We ran into the issues I described above (handling of object graphs, URL fragments, and multi-graph resources) and we found that our solution was becoming more and more complex. [We note that the JSON-LD chair appears to agree on the difficulty of reconciling these goals (), although we don’t agree with his conclusions.] Most of all we began to question the basic premise of JSON itself that the server should try to design the programming data structure for the client. We learned that our clients often wanted more than one data organization, even within the same request, and we discovered that if we let the server supply the data in a ‘flat’ format and we let the client construct a hierarchical object structure from that, then several significant difficulties and complexities disappeared.
  Both our ‘object graph’ and our ‘object list’ data organizations have some similarities with one of the many variants of JSON-LD that are described in the specification. While this is true, it is not really interesting since we never actually serialize either of these data organizations – we construct them in memory from the lower-level RDF/JSON format. [RDF/JSON could have been called ‘NTuples-over-JSON’].
  1. Using RDF as the underlying model for data has been very powerful for us. In our experience many projects get bogged down in hopeless arguments trying to pin down a single organization for their data, not realizing that it is impossible to define one that will satisfy all use-cases. RDF has freed us to support multiple alternative data organizations for the same data for different programming tasks and also serialize the data in different media types, while agreeing only on the core concepts of our data.
  2. RDF/JSON is a useful wire format. Our serialization of the ‘set of triples’ data format is now quite close to RDF/JSON and could be made compliant with little effort. Even better would be if RDF/JSON were officially extended as we have done to use JSON numbers, strings and Booleans directly rather than wrapping them in structures, allow single object values as well as arrays of objects, and provide support for RDF List. Our extended version of RDF/JSON and RDFa are the two representations we use most heavily, although all our servers also support Turtle and RDF/XML.
  3. We have found no need for JSON-LD, although we have used one or two good ideas from it. JSON-LD does not solve any problems that we have encountered, and it is complex
  I've tended to avoid blank nodes, because I have not been totally comfortable with them - I didn't have a strong model for their meaning and usage. Recently I have formed a view of what I think they mean and how they should be used, and I have also found some use-cases in our applications for them. In addition, I have decided that blank nodes provide a simpler way of thinking about other parts of RDF. Some of these thoughts were inspired by a blog post on RDF that I ran into and can no longer find.
  At one time I viewed blank nodes as being sort of "anonymous resources" or "nested resources/objects". I now think this is wrong - it is a carry-over from my OO past.. Let me try to explain.
  I will claim that the most fundamental construct in RDF is the one that allows you to link two URI references with a predicate. In English, we might make a statement like "International Business Machines employs Martin Paul Nally". In RDF, this would translate to
  <> xxx:employs <>.
  This works well when there is a URL for Martin Nally, which happens to be true in my case. In English, when we don't have a suitable name to use to identify something, we instead use a subordinate clause to identify the object. So we might say 'International Business Machines employs that person whose first name is Martin and last name is Nally and was born in Scotland in January 1957'. I added emphasis to highlight the subordinate clause. I now believe that the proper use of blank nodes in RDF is to express these subordinate clauses. In other words, a blank node is not a nested object or an anonymous resource, it is a reference to a resource. It's like an URL - it references an existing resource - it is not a special type of resource. Using the example above, we might say:
  <>"
  ].
  Although this seems more correct, it's debatable if it's worth the extra overhead to 'do it right', rather than do it the easy way. If the shortcut of RDF literals had not been available to us, we might have preferred to write it correctly rather than writing
  a xsd:String;
  rdf:value "California"
  I.
  For most of our project, we have been relying on a PaaS (primarily Openshift, but also Heroku and CloudFoundry) to do deployment for us. [We did once, very early on, write a script to deploy directly to Amazon EC2, but we did not maintain that script, and I'm not even sure it ever really worked.] Recently, I've been thinking about doing our own deployments again, without a PaaS. While thinking about this, I have come to realize that there are some other approaches that are quite different from what we are used to at Openshift or on our development machines. When you create a new application at OpenShift, I believe it actually creates a new [partition of a] virtual machine on Amazon EC2. Openshift controls the software stack of the application VM (OS, middle-ware) although you have some control of the middle-ware stack through the 'cartridges' you select. The 'Application creation' operation is one of the few times you are really interacting with OpenShift servers directly - when you deploy your application code via a Git push, I believe you are pushing it directly to your application's own Amazon VM where it triggers an OpenShift script written by RedHat. This script will stop the currently-running application instance, update the application code (including installing new prerequisite packages from the internet if necessary), and then restart the application with the new code. I do not know how, when or whether the OS, middleware or RedHat-written Openshift scripts are updated on your application VM. I am 80% sure this is an accurate description of what OpenShift does, and 60% sure that Heroku and CloudFoundry do essentially the same thing. We have seen that this approach is not without problems - these scripts can (and do sometimes) fail, leaving your application 'down' until you can figure out how to fix your end of the script or wait for RedHat to fix their end, and even if they run correctly, your application is 'down' while the scripts run.
  It occurred to me that there are other approaches to application deployment. For example, if I separate my load-balancer, application and database into 3 different [virtual] machines, I can deploy new versions of the application without ever 'patching' a [virtual] machine during deployment. In this approach, the output of my application build would be a complete virtual machine image, not just application files or packages or a git commit. To deploy this new application version, I would simply start a new virtual machine from the new image, and adjust the load-balancer tables to point to the new machine. I would still need a small amount of configuration of the new VM, for example to teach it to find the database. I might leave the old version running for a while to complete any outstanding requests and for potential roll-back. If there is a problem with the new version, I just need to change the load-balancer tables to point back to the old VM, which is still running, an action that should take almost no time at all, and I could also split traffic between old and new versions until I gain more confidence in the new one. There should be zero visible downtime for my application in all cases. The deployment script is much simpler, and so should be significantly more robust. Some corresponding complexity got added to my build scripts to create VM images, but that is a good trade, because failure there does not hurt me nearly as much. It's also fairly clear with this approach how to construct a system test environment that is guaranteed to be faithful to the production one - just run the same VM image. If the new version introduces a significant change to the data model, some of these scenarios become harder, but not harder than they are with the patch-in-place approach. Obviously this approach takes many more resources, so it might not be attractive for small 'casual' applications.
  As a complete beginner in this field, I can't really comment yet on which of these approaches is better for us, but it's interesting that quite different choices are clearly possible, and the approach chosen for us by our PaaS may not be the ideal one for us. For our project, rather than implement the OpenShift approach on EC2 with my own scripts, it seems much more interesting to explore the alternate approach that avoids VM patching altogether. As Mae West said, “when choosing between two evils, I always like to try the one I've never tried before.”
  One thing we have discovered - which should come as no surprise - is that it is much harder to design a system that is composed of a set of semi-independent applications/services than it is to design a single monolithic application. Recently I have been trying to clean up the dependencies between our semi-independent applications and document them. This rest of this post comprises my first draft of a document on this. It is specific to our set of applications, but maybe the designs we use will be of interest to you, or maybe you will be able to suggest better designs to us. It's a bit long, so skip it if you find it boring.
  The LeanScheme API is a web of HTTP resources whose data [meta-] model is RDF. There is a set of 'well-known URLs' that clients can use to start navigating the web, and to 'jump' between sub-webs.
  With hindsight, it's probably not surprising that having our team use ALM as an example problem domain to work with did not generate any positive excitement. After all, why go back and redo in a slightly different way something that our current tools already do perfectly well? Given this tepid reception, we went off looking for a different problem to use as our guinea-pig. At Innovate I listened to Eric Reiss present his "Lean Start-up" ideas, and I also read Steve Blank's "Four Steps to the Epiphany" and Ash Maurya's "Running Lean". I also read quite a few blogs on companies that do continuous deployment (as mentioned in a previous post). Those ideas seemed like they could form the foundation of a tool that at least had the merit of being different from things we have already.
  We noodled on this a bit, and Dave Tropeano and I put together a couple of short "marketing" videos to try to explain our tool idea. Dave's is shorter and slicker than mine (and so much better), but mine goes into more depth. Here are links to the videos:
  Dave's:
  Mine:
  These ideas may or may not have potential to evolve into something useful - time will tell - but in the meantime it gives us an example to work on that isn't just rehashing our history.
  Until recently, all the data I'd looked at in our applications had fit nicely into RDF (with the possible minor exception of RDF/JSON's difficulty representing Lists). I ran in to a problem a few weeks back when I implemented a history feature. The idea is that each time we update a resource in the database, we snapshot the old values. The UI gives you a history of the differences over time. I wanted to have a single resource that would give me all the previous versions of a particular resource. Now my problem is "what is the data model for this resource whose value is a set of previous graphs for the same resource". I obviously cannot put all the triples from all the graphs in a single graph, because they will conflict and I won't be able to tell which triple came from which graph. I can imagine 3 solutions:
  The one I think I like the best is actually 2. It's a bit like having an equivalent for Trig of what RDF/JSON does for Turtle.
  The one I actually implemented was 3. The reasons were pragmatic. For one thing, I needed to be able to return this structure in multiple representations, especially HTML/RDFa, and it was looking like a lot more work to implement option 2 for multiple representations. I would probably also have had to invent new media types (Trig/JSON? RDFb?). Just inventing a new datatype within the existing representations seemed easier.
  If I get the time and energy I might go back and look at this again, since I'm not totally happy with what I did. Anyone got any bright ideas?
  You.
  I.