My name is Dave Browning.
I'm on the CloudKit team here at Apple, and I'm super excited to talk to you about some new things in CloudKit, especially a brand-new CloudKit Dashboard.
If you're not familiar with it, the Dashboard is a web application designed to help you as you're adding CloudKit functionality to your apps.
So first, I'd like to talk about some of our goals behind the things we're announcing today, and at a high level, as the title of this session suggests, we really want to help you build better applications on top of CloudKit, and we want to do that in a number of different ways.
First, we want to help you through all stages of your app's lifecycle.
So, when you're first learning about CloudKit, when you're starting to build your functionality into your application, when you're beta testing with users, when you're getting your app out into the App Store, going to production, scaling up, really honing in your user experience, and then finally, potentially supporting customer problems in the wild.
We want to give you the tools necessary to help you throughout all of those phases.
We also want to give you a way to experiment with the entire API.
Now, obviously, you can jump right in to Xcode and start using our API and figuring out how it works, but we want to give you something to sit alongside Xcode that lets you visually play with that same API so that you can experiment with it, understand how it works, while you're building that functionality into your application.
We also want to give you visibility into all events across the system, across all of your users, and especially once you're out in production in the App Store and growing your user base, we want to help you understand the aggregate behavior, the communication among all those users back at the CloudKit server.
Alright. So, first, some CloudKit concepts.
At the end of the day, you're using the API to store records back to the CloudKit server.
This is your structured data, your keys and values, and as a reminder, the values can be different types, strings, ints [phonetic], doubles, an asset, which is a binary file.
When you store a record, it always exists inside of what we call a zone.
A zone is basically a bucket of records, and it's a foundational piece of some of our APIs that we'll talk about in a bit.
A zone always exists in a database, and some databases allow you to create different zones, if you need to bucket records in different ways.
All of your users have their own private database.
This is where you store their private data that only they can see.
They also have their own shared database, and this is because last year, we launched CloudKit Sharing, and if one of your users shares data with another one, that data shows up in their shared database.
You can think of it as a proxy back to the owner's private data.
And then finally, there's a public database which everyone can read and write to, and there's one of those.
So, we give you a way to promote your schema changes to the production environment, and then in production, all of your users have their own private, shared, and one public database there, separate from the development environment.
And then, all of this lives in the highest-level concept, which is a container.
Container has a unique identifier, and it usually maps one to one with your application.
Okay, so that was the CloudKit concepts.
So, I mentioned that we built an example application on top of CloudKit that I want to use today during the demo.
It's a to-do list app, the age-old example.
The idea is, users can create to-do lists, and they can have items inside of there.
So, let's talk about the data model that we'll be using, since we'll be storing this data back to CloudKit.
So, every time your user creates a to-do list in the app, we'll create a List record on the server.
Whenever they create items within the list, we'll have an item record for each, and we need a way to point the item back to the list to which it belongs, and in CloudKit, you do that with a reference, which is basically a pointer back to another record via its identifier.
And, in this case, we're going to use what's called the parent reference.
The parent reference is a system field that exists on every record.
It's there for you by default, and it's the way in CloudKit that you logically tell the server when something has a parent.
Now, we also, in this example app, wanted to allow our users to securely share to-do lists with other users using CloudKit Sharing.
So, here's how that will work in the data model.
Our List record will be what we call a Root record, and it will point to a Share record.
A share is where you define the set of participants and their permissions, and because we use the parent reference to point items back to a list, they automatically get included in the share for us.
You don't have to go and share them separately.
To learn more about sharing, make sure to go back and check out last year's session, What's New with CloudKit.
Okay, so that's how we're going to be storing the data.
Now, let's talk about the APIs we're going to be using, how we'll communicate back to the server.
So, the way we wanted this application to work is that if the user opens it up and creates to-do lists with items on their iPhone, but they also have an iPad, and they run the app there, we want that data to be synchronized.
And, because we're using sharing, if I share a list with someone else, and either of us edits it, we want the other to see those edits.
So, we have a core data, we're locally storing all of our data locally in core data, and we do this for quick reads and writes, and for offline access.
But, every time there's a modification, when there's network availability, we want to send that back to the server.
The server holds the truth.
And, if we use in CloudKit what's called subscriptions, we can have the server send push notifications to my other devices or to other users' devices whenever data they care about changes.
So, here's how we're going to make that work in the example app.
The first time it runs, we're going to create a CKDatabaseSubscription on the server for that user's private database.
That tells the server to send push notifications to this user's devices whenever their private data changes, anything inside their private database.
So, CloudKit will now send push notifications to this user's devices when data changes in either of those places.
Now, once we receive one of those push notifications in our app, or when it launches, we need a way to ask the server for the changes that exist that we don't have yet, and we're going to do that with CKFetchDatabaseChangesOperation.
This asks the server, "Please tell me what zones have changed," remember, we mentioned zones earlier, "inside of this database."
And, if any of them did, we can then turn around and ask the server what records changed inside of that zone, or those zones, with CKFetchRecordZone ChangesOperation.
To learn more about this, check our last year's session, CloudKit Best Practices, and because it is such a commonly used workflow, we've added new documentation called "Maintaining a Local Cache of CloudKit Records," that walks you through this workflow and gives you a bunch of Swift code so that you can build it into your application.
Now, because fetching changes is such a common thing, we've built the ability to play and experiment with it right inside the Dashboard.
So, let's go take a look.
Alright. So, up on the screen, on the left side, you can see the new CloudKit Dashboard, and on the right side, I'm sharing the screen of this iPhone running that example to-do list application that I mentioned before.
Now, if you look in the Dashboard, we're on the home page.
I'm signed in with my developer account, and the first thing you can see are all of your developer teams, all of the teams that you're a part of, because you may be a part of more than one team.
For each team, you can see the CloudKit containers that that teams owns, and you can quickly see which ones have been deployed to production and which ones are still in development.
If you have a bunch of teams, or a bunch of containers, you can always filter down the list up at the top.
Now, in this case, this is the container I'm using for this example app that I'm demoing, so let's click in.
When you click into a container, you see the development and production environments that I mentioned earlier, side by side.
And, in this case, we're using the development environment, because I'm still building this example app.
This lets you play with the zone API in CloudKit.
You can ask the server for zone information inside of a specific database.
Now, in this case, I want to show you, I'm logged into the Dashboard with my developer account, and that Apple ID is the same one on my device.
It's my personal iCloud account, and my developer account.
So here, the Dashboard asks you, "Would you like to load zones from a private database for this account, a shared database, or the public database?"
And, let's do the private database.
Because we want to use the fetch changes APIs, we're going to check this box to fetch the zones that have changed inside of this database since a certain point in time.
Now, the way this API works is, it allows you to specify a change token which marks where in history you have synchronized up to.
You can apply that here, and we're going to leave it blank, which tells the server we want all changes inside of this database since the beginning of time, and if I fetch the changes, you see we get a result.
This To-dos zone is the zone that my application created the first time it ran on this device, and it gives us a place to store the future records that we'll be saving as the user creates to-do lists and items.
Notice that the server returned a change token which, it's populated automatically in this field, marking that we've moved forward in time.
So, if I fetch changes again with this token now, the server says, "No zones have changed since then," so we're up to date.
Now, if I clear that token and we go back since the beginning of time, we'll see my To-dos zone.
If I hover this row, the Dashboard gives us a handy little link to then go and fetch the record changes inside of this zone.
Notice it's taken us over to the Records tab.
It's chosen my private database, the To-dos zone that I clicked on, and it said we want to load record using the fetch changes APIs.
And, we see two results over on the right.
Well, the good news is, oh, there it goes.
Alright. And, let's name this one Chores, because unfortunately I'll have to do those.
Alright. So, that technically saved back two lists to the server, so we have three here in our iOS application.
Now, let's jump back to the Dashboard and fetch those changes inside of this zone since the last time, when it said there weren't any.
So now, when I fetch changes, we see those two List records that I just created in the app.
And, if I tap on the record name for that second one, it opens up what we call the Record Editor.
This shows you information about that specific record, and in this case, we can see its unique record name.
We can see the record type, and in this case, it's a List record type, like I mentioned before, when we were talking about the data model.
It's in my private database in the To-dos zone, and it was created and modified by me just a second ago.
So, I've decided, chores don't sound very exciting, so let's rename this to Movies to Watch, and change the to-do list.
Now, I'm going to save this back to the Server view of the Dashboard, and I want you to watch the iOS application on the right side.
The server saw that I had a subscription for my private database, sent a push notification to my device.
My application saw that push, turned around, fetched the new changes, updated its local core data cache, and now you can see the UI reflects the fact that we've changed that list to Movies to Watch.
So, hopefully that walks you through some of the things that you can do now with the fetch changes API.
Let's jump back to slides and recap what we just talked about.
So, in the Dashboard, when we use the Record Editor to modify that record to update it on the server, it used the publicly available CloudKit Web Service APIs that all of you can use.
It sent an HTTP post to the records modify end point, and it sent a JSON body saying, "I would like to update this record," and set its name field to Movies to Watch.
That corresponds to the CKModifyRecordsOperation in the iOS, the native API, which we're using in the example iOS application up here.
When we fetched zone changes inside of our To-dos zone, that used the publicly available records changes end point, and it said, "Server, please tell me the set of records that have changed in the To-dos zone for days of count since this change token."
And, that corresponds to the CKFetchRecordZone ChangesOperation in the native API, which we're using in our app.
So, the point here is that the Dashboard now lets you play with the exact same APIs that you're using in your native and web applications, hopefully allowing you to understand and debug functionality as you're building it into your application.
Now, I mentioned that part of getting all of this data synchronization to work is subscriptions and push notifications, so let's jump back into the Dashboard and see how it's given us the ability to experiment and understand more of this behavior.
So, I'm going to close out the Record Editor, and as a reminder, we're in the development environment data section.
There's a new tab here for Subscriptions.
If I click it, this lets me use the subscriptions API in CloudKit to fetch down subscriptions that exist inside of a specific database.
So, I'm going to ask what subscriptions exist in my private database, and we see that our server returned a result.
It's a database subscription, and I've given it the ID of private changes, which is something I chose in my code.
So, that's a subscription that exists that will tell the server to send push notifications whenever my private data changes.
If we look in the shared database, we also see a database subscription there, like I mentioned, with the ID, shared changes.
Now, there are subscriptions and push notifications and you're fetching changes, potentially across multiple devices.
Wouldn't it be awesome to see a log of all of these events as they're happening on the server?
If I click back up to the container, thanks, there's a new section called Logs that I'm going to click into for the development environment.
This starts on a feature we call Live Log.
It opens up a real-time connection back to the CloudKit server, and the server will push events as they happen right to your browser.
It loads a bit of history, so here we can see some requests we were doing from iOS, from the app up here, as well as from the Web Service API, because that's what the CloudKit Dashboard was using.
We can see we were doing some record modifies, some zone fetches, etc. Now, let me clear this out, and let's create a new list, and watch what happens in the log.
So, I'm going to create a list, then we see the event pop in, or boom.
So, here we can see some information about the specific event, but let's jump back into slides and talk about all of the things you might see show up in your log.
So, here's an example row, like we just saw in the user interface.
Let's walk through each column and see what shows up.
The first one is the time.
This is the time at which the event happened on the server.
The next column is the platform.
This tells you which platform the request came from.
Because CloudKit is available iOS macOS, tvOS, and watchOS, you may see all of those show up here, if you're building apps on those platforms.
You also see the version of the platform, and if you're using the Web Service API to build a web application or to extend your app to another platform, then you'll see it show up here as web.
The next column is the user.
This will show you the CloudKit user record ID.
Now, in this case, if the person that sent this event is on your developer team, we know their name, and we'll show it.
So, we knew my name, because I'm logged into Dashboard on my team, and this is handy as you're testing things out with people on your team in development.
But, for everyone else, for all of your normal users, it will show you the CloudKit user record ID.
The next column is the type of the event.
This is most commonly database, as you're interacting with the database API, but if you have subscriptions and CloudKit is sending push notifications on your behalf, you will see pushes show up right here in this log as they happen.
You will also see sharing events called out separately.
The next column is the operation ID.
So, as of iOS 10.3, and the newest version of macOS Sierra, CloudKit will automatically create unique IDs for every operation that you're issuing in the native API, and you will see them show up here.
The next column is operation group name, which we'll get back to in just a second, and the final column are the set of details specific to this type of event.
Now, in this case, it was a database event, and so we can see the type of database operation that we issued back to the server.
This one was a zone fetch, but you'll see things like record modify, database changes, depending on the type of operations you're submitting.
In this case, because it's a database operation, we could see it was in the private database, and in the To-dos zone.
We also see the server latency.
Now, this is how much time it took the CloudKit server to process this, but note that it doesn't include internet latency or the time spent getting from the client to the server and back.
You can see the request size, and the response size.
You can also see the hardware identifier, and no, this is not a new iPhone.
This is actually the identifier for an iPhone 6s Plus, like I'm running up here.
Some more information about operation groups, which we'll talk about in just a second, and finally, the request ID.
If an event leads to an error, you will see it called out in red, and it will tell you the specific type of error.
In this case, I tried to fetch a zone that didn't exist, and so the server said this was a zone not found.
Alright. So, I mentioned operation groups a couple times, so what's that about?
Well, new in iOS 11 and all of the other matching platforms, we've provided, as you probably guessed, the ability to group operations based on application logic.
In this example to-do list app, there are a number of things that it needs to do when it first launches, in the initialization logic.
It needed to create a zone on the server within which to store our records.
It needs to create two subscriptions.
Remember, I mentioned the database subscription in the private and shared database.
And, it needs to fetch down any existing changes in the private and shared database, and then potentially fetch zone changes.
So, there's a number of operations, all of which encompass the initialization logic, and so we now have a way to group all of that logic together.
Now, as you probably know, there are a number of Apple applications built on top of CloudKit, and I'd like to give you some examples of how we use operation group names in our apps.
So, iCloud Backup is built on top of CloudKit.
So, every night when your iPhone or iPad is plugged in and on Wi-Fi, it'll automatically back itself up.
That may take a number of CloudKit operations to do, and so we can group those up inside of an operation group named Automated Backup.
If the user triggers one manually, we can call that out separately as a Manual Backup.
And, when they go to restore the data on a new device, we can have an operation group that encompasses all of the operations within.
iCloud Photo Library is built on top of CloudKit, and they use operation groups to designate when they're setting up your library, downloading thumbnails, or fetching a movie that you tapped on.
iCloud Drive is built on CloudKit, and so they have operation groups to call out when they're initially pulling down any changes from a server, when they do so after a push because of a subscription, or when you tap into a file to download it.
So, let's look at the API.
So, there's a new class, CKOperationGroup.
It provides you an operation group ID set for you by the system.
It allows you to specify a CKOperationConfiguration, which we'll talk about in just a second.
It allows you to provide a name.
This is any string that makes sense in your application, and the names on the previous slide were ones that we use, and you want to be careful not to put personally identifiable information into this.
This is your app logic, what it's doing.
The next property is quantity.
This is an integer that you can set, and it's completely up to your application.
So, some examples from our apps might be when we're backing something up in iCloud Backup, we might designate how many files we're backing up.
Or, in iCloud Photo Library, when they're downloading thumbnails, we could use it to say how many thumbnails are being downloaded in this group.
You could also set the expected send and receive sizes, and this tells the server, I'm sorry, tells the client how much data you think you're estimating will be sent back and forth between you and the server.
And, notice the type is CKOperationGroupTransferSize, and this is an enum which allows you to specify an order of magnitude.
So, the point is, is that it's an order-of-magnitude estimate.
It doesn't have to be perfect exact byte counts.
And, by setting things like these properties, as well as properties like quality of service, this allows the system to optimize when network calls are sent back to the server based on the network conditions of your user's device.
Finally, once you've configured an operation group, you can add an operation to it by applying it to its group property.
Now, I mentioned CKOperationConfiguration, so let's talk about that for a second.
In the past, you used to define properties like qualityOfService, allowsCellularAccess, isLongLived on an operation.
That is now deprecated, and instead, you apply it to a CKOperationConfiguration, and the reason for that is because we found that it was very easy to forget to apply these to all of the operations that you're issuing in your client.
So now, you can set it up maybe once or a few times.
You can apply it specifically to an operation, if that makes sense, or hopefully more likely, to an entire operation group, which applies it to all of the operations within, and that was that default configuration property on operation group that we saw before.
OK. So, if you look back at that example log event from before, let's call out some of the things that we saw around operation groups.
So here, you can see in the log the operation group name, so I can see this event happened because of my applications initialization logic.
You can also see its unique operation group ID and a quantity property, if you set it.
Now, at every event, you can see the operation group that it applied to.
You can then see the specific operation that applies, based on the operation ID, and if it leads to multiple requests, you can always see the specific request ID.
Alright. So, with this new log feature and the things being exposed, let's take a minute to talk about privacy.
As you know, Apple cares very deeply about our users' privacy, but with CloudKit, we also understand that you're working with a system that you don't own or have full control over, yet you still need to debug customer problems.
So, with both of those things in mind, here's how it works.
But, you cannot see the private and shared data of other users.
You can see the public data in the public database, because it's meant to be public, and you can now see log events for you and every other user.
However, log events do not include the data.
So, for example, some of the events we would see in the case of this to-do list app, you could see that my account was issuing record modifies, but you couldn't see that I was creating a list named Vacation Ideas, or Chores, or Movies to Watch.
So, we hope that this balance really allows you to debug problems as they happen, while still keeping your customers' data safe and private.
OK, so I mentioned earlier that we wanted to add the ability to share in this example application.
We want to let our users securely share a to-do list with other users so that they can collaborate.
As a reminder, this was our data model.
We point items back to a list via the parent reference, and we have the list as the root record in a share.
Now, we've added some of the ability to debug and explore more of this in CloudKit Dashboard, so let's go take a look.
Now, on the right side of the screen, I want you to look at the iOS app.
I'm going to tap into that top Vacation Ideas List record, and you'll notice that it shows us some information saying that this is already shared.
So, what happened was, I created this to-do list on my phone, which means it's in my private database, so I'm the owner, but I invited my friend, Emily Parker, to share this, because she's a travel expert, and I wanted her to give me some ideas on where to go on vacation.
So, in the Dashboard, if we go back to the data section, remember, the last time we were looking at the records in my private database in the To-dos zone, and if we fetch changes until we get to a point where nothing has changed, so the Dashboard is up to date, let's add an item in the iOS application.
So, let's say I would love to go to Paris.
So, my iOS app has stored that record as an Item record back to the server, and if I fetch changes in the Dashboard again, we see it show up.
I'm going to tap on the record name and open up the Record Editor.
Again, we can see this is of the Item record type.
It's in my private database in the To-dos zone, and down under the Field section, we can see Paris, the name that I provided in the iOS app.
Now, if I zoom in a bit, you'll notice there's a section here labeled Sharing, and it tells us that this record is a descendant.
That means that it points to a parent via its parent reference.
The Dashboard tells us the record name for that parent, and provides us a little jump icon to load that parent up inside the Record Editor.
So now, we're looking at a List record, and this is the Vacation Ideas List record within which we created the Paris item.
In the Sharing section, we can see that this is a Root record in a share.
It points to a specific share.
It tells us what that name is, and we can open up that share.
So now, we're looking at a CloudKit.share record, and down below, you'll see on the share, you can actually see information about the sharing participants.
You can add and remove participants right here to test out how your app behaves when that data is modified on the server.
This allows you to browse relationships, and if the Record Editor detects that there are any other types of references in your record, it will also let you jump to them, and we have a bit of back and forward History buttons up here in the top left.
So, that shows you a bit of how you can view and navigate the data relationships going on with sharing.
Let's take a look at how logs can help you out when you're doing sharing functionality in your app.
So, I'm going to load up the logs.
Let's create another item in my iOS application.
So, let's say I'd also really like to go to Vienna, Austria.
Now, I'm going to hit Return.
I want you to watch the Dashboard log.
So first, we see a request come in for the record modify.
We can see, notice my operation group name.
I'm using one saying, "Oh, this is logic for creating an item in the application.
It was a record modify in Dave's private database."
After that, we can subsequently see that CloudKit sent a push notification to Emily.
It was in her shared database because of a subscription called Shared Changes that we set up on her device when she first ran the app.
The push notification made it to Emily's device, and then her app turned around and fetched database changes, and then zone changes, and notice the operation group name, fetching changes after notification.
So, we really hope that this will help you as you're debugging and trying to understand what's happening across multiple devices and potentially, multiple users in sharing cases.
Now, in this case, we were in the Live Log, but we also have a feature called Historical Log.
This allows you to go back and view events that happened in the past.
We store events for up to seven days, and you can do things like, show me all the events that happened yesterday, let's say, between 9 and 10, and you can provide a bunch more filters, and this will return to you all the events matching those filters.
Alright. The last thing that I would like to talk about is telemetry.
So, last year we offered CloudKit Telemetry, and this year we've expanded the offering.
As a reminder, telemetry is your way to understand aggregate behavior happening across all of your users' devices coming into your container.
In this case, we're looking at the development environment, so we're seeing information that I've been playing with from this app onstage.
You can choose a time filter, so you can choose the last year, the last 30 days, the last day, or the last hour, and let's look at the last hour, since that's when we've been onstage playing with this app.
You can ask for telemetry information that applies to everyone's private database, everyone's shared database, or just the public database, and let's leave it on private.
You can also view telemetry information around all operations within that time window for the private databases, and we'll tell you the specific types of operations, and how many of each came in during that time window.
So, you could say, "I would like to see telemetry information only for record modify operations."
But, in this case, let's look at it for all operations.
The first graph you see at the top is called Requests.
This shows you all of the requests coming into the server, again, during this specific time window, and for the private databases, and how many requests for each type came in at a time slice.
Notice it's called requests, and not operations, and that's because, as I said before, as you're issuing operations from the API, in some cases, that leads to multiple requests, and we want to give you visibility into all of the requests.
So, here we can see that I was doing record modifies, database changes, subscription modifies, and how many of each.
We hope that this helps you, during development, understand how many types of requests you're sending in, as you start to test and build your app.
If I scroll down a bit, the next graph is called Server Latency.
Here, we're exposing to you exactly how long, again, it took the server to process these events.
We show you the 50th and the 95th percentile, and as a reminder, you can jump in and see how long it takes for certain types of operations.
And, the reason we want to give you this information is so that it helps you understand how long it takes to process the type of operation that you're adding to your application, so that you can choose the right one, depending on the experience that you're trying to build.
The next graph is Error Counts.
This used to be called Error Rate, so we used to show you what percentage of requests failed.
Now we show you how many exact ones failed, and how many users it affected, and the specific types of errors.
So, in this case, you can see at this time slice, there was one error.
It was a conflict, which meant I tried to update something that was already updated on the server, and it affected one user.
So, we hope that this information will allow you to detect when you might have problems in your app causing more errors, and to understand what percentage of your user base it might be affecting.
And finally, you still have access to your average request size, which shows you how much data is coming in to the server.
Now again, this was my development environment.
Let's jump back to slides and take a look at some example screenshots of these graphs in the wild.
So, here's a Request graph from one of your containers in the wild, and you can see, on any one day, they do about 2.3 million record fetch operations across all of their users in the private database.
And, what's interesting is if you look at this scale, you can actually start to see trends across weekday and weekends.
And so, these developers can hopefully use this to, like I said, monitor for spikes or drops, changes in this behavior, as they release new versions.
And, of course, if you change how you interact with CloudKit, then maybe you expect a change.
Here's an example of an Errors graph in the wild.
Notice that they have about, oh, on any one day, 1500 zone-not-found errors, but it's only affecting about 55 users, which could be a small percentage of their total user base.
So, it might mean that there's an edge case in their application where it gets into a weird state where it thinks a zone exists that doesn't actually exist in the server.
We have CKError documentation that used to tell you all the types of errors you might see.
We've updated that to explain what each error means, and how you might handle that in your application.
So, the errors you'll see show up in this graph map back to that documentation.
We also give you some telemetry around the push notifications that CloudKit is sending for your application in the wild.
It's separated based on the subscriptions in the private, public, and shared database, and this is sort of a sanity check.
If a customer complains that data is not synchronizing, you can go here and feel good that CloudKit is still sending push notifications, that subscriptions are still configured right in your application.
OK. So, what did we cover today?
Well, we've launched a brand-new CloudKit Dashboard.
We hope you check it out, try it out.
Please give us your feedback, let us know what works, what doesn't work, what else you'd like to see there.
We've launched a new CKOperationGroup API.
We hope you've started to see the help it can provide you, especially when looking in the Dashboard Log, to understand exactly what code led to what network request being sent back to the server.
Like I said, we want all of your feedback.
It really does guide what we do, so follow Radars, post on the forums, let us know.
For more information, here's a link back to this specific session, and we also have an email address, cloudkit@apple.
And with that, thank you very much.