Al Martin: Hi, guys. This is Al Martin Making Data Simple Series. Today we're going to talk about data movement. And I've got two expert speakers in the room with me. I've got Michael and Rajani. And why don't I just start, let you guys introduce yourself and then I've got plenty of questions around data movement. Go ahead, Michael.
Michael Springgay: Okay, so, I'm Mike Springgay. I'm an architect within the Db2 warehouse development organization focused on data movement and compatibility.
Rajani Maindiratta: Hi, I am Rajani Maindiratta. I manage the load team in Db2 data warehouse on cloud, now moving to data other areas well.
Al Martin: Fantastic. So, look, I deal with all, you know, unified governance, fiber data management, visualization in our entire point of view, our entire strategy. I think the difficult two we have today is semi-structured, structured, unstructured data, Skypes, on-premise, cloud, transactional workloads but — oh well and I could go on saying look we're on cloud. We're on hosted. We're on private cloud. We have an appliance.
This all doesn't work without data movements. That's how important data movement is. And you can't move data you got none of that, right? So I guess the first question, Rajani, and I'll go with you first if that's all right, what is data movement in your definition and what is it important?
Rajani Maindiratta: Okay, so kind of what you explained earlier, I see it as just one of the challenges around data in general. As you said, companies — organizations have — are faced with the deluge of data and something you can manage. But they need to understand that they like to — would like to mine and to make business decisions.
And there's three or four different difficulties with that. One is the volume, the sheer volume of data that's coming in from different aspects day to day transactions, e-commerce transactions, IoT, social mobile data, all of that's coming into the organization. So they need to understand what's coming in.
They also need to deal with the speed in which it's coming in and also it's coming from a number of different sources. So because the businesses would like to, organizations (both) would like to try to make business decisions to potentially increase revenue, work with — improve efficiencies in the organization, perhaps solve a business problem they need to have a repository through where this data resides coming from multiple sources where they can like do analytics, run the reports, understand what's happening in their organization so they can make those decisions.
Data movement comes into play because they need to be able to do: get the data into the repository with data warehouse which is simply repository different sources of data. They need to efficiently be able to move the data so that they can quickly react to what the data is telling them whether, you know, maybe they need to focus an area because they're somewhere else or it — the possibilities are really endless.
Al Martin: What are the most common customer stories or use cases you're working with today? And I guess or if there's a unique one I'd be curious as to that as well, I mean one of the strangest ones you have. But what is a common customer story?
Rajani Maindiratta: Customer use cases, there's been a few that we encountered. Healthcare has been one where we had a — we have customers who, you know, daily they need to mine physician records test results, things like that at a daily basis right? They're constantly analyzing the data to know what's happening and from a healthcare perspective.
Al Martin: And what are they moving data from into where?
Rajani Maindiratta: Oh okay. So we see different sources right Oracle is one the customer is moving data from Oracle to DashDB or Db2 for (health) on cloud for example.
Rajani Maindiratta: Yes on-prem into warehouse on cloud.
Rajani Maindiratta: So yes so that's why in use case there's the Netezza use case right the use case of course.
Rajani Maindiratta: It's the appliance use case?
Al Martin: The appliance use case one in the cloud as well?
Rajani Maindiratta: That's right yes.
Rajani Maindiratta: That's happening as well.
Al Martin: So, Mike, where are you spending most of your time at today? I mean I know you've got a lot of passion. We talk a lot about data movement. You know, kind of carrying on the conversation we just had, you know, what do you see as kind of the biggest challenge? I'm kind of — well, I got a multi-part question here.
What do you see as the biggest challenge that you're facing? Are you doing more on-prem to cloud, on-prem to on-prem in terms of data movement? What do — how do those challenges differ and what kind of speeds can — what are the sides of the data that many of these clients are moving?
Michael Springgay: Right, So I think your last point is really where the biggest challenge is is the size of the data has, you know, expanding tremendously in the, you know, the last few years. And that certainly causes challenges on premise but it also significantly challenges you when you talk about moving into the cloud. So a lot of the area of focus is really in both spaces.
Most in customers, companies are in a hybrid situation. And they really are moving everything off of on-premise. They still have on-premise for their core applications but they want to move some lower priority workloads into the cloud but they need to get the data there and the data has to get there quickly. And that's a challenge for most cloud providers.
And in the industry most people are looking at disk shipping type technology with large appliance based products for shipping data. IBM is no different in that space, you know, for really large terabytes of data when I mean the over 25 terabytes of data and petabytes, you know, you want to ship the disk. But the challenge with that is you still have the legacy because you, you know, unload it. You got to purolate it or ship it with, you know, FedEx or whatever to get the data into the data center. It's always a delay.
So one of the interesting things that, you know, IBM is doing is that we bought a company called Aspera and we're using that Aspera technology to allow us to transfer over the Internet our larger volumes of data so we can get up into that 20, 25 terabyte range and still reasonably quickly move the data from on-premise into cloud.
Al Martin: When they're going that on-premise into the cloud though where is the line of distinction between hey I can move that data to a great technology like Aspera and what kind of speed are you talking by the way? And then or for you got to go get FedEx or something to ship the data. What's the line of distinction there?
Michael Springgay: So usually we recommend around that 20 to 25 terabyte range. I mean obviously it's going to be somewhat dependent on the customer's bandwidth and their own willingness to use that bandwidth to move the data or even the time they're willing to wait. But at that point you sort of after 25 terabytes you're probably in the breaking zone of the amount of time it would take it's probably faster to put it onto an appliance ship appliance can get it unloaded into the data center.
The speed we're seeing is, you know, typically with Aspera we can get up into the 200 gigabytes an hour transfer rates assuming we can saturate the whole network. Without Aspera though, you know, most customers would see probably a 5 or 20 gigabyte an hour. So it's really not that usable without that technology to move data in the 20 to 25 terabytes because you’re talking days.
And when you're talking about the Internet it's also not necessarily the most robust connection so, you know, you have to assume some amount of drop. So if you're moving a terabyte file and you lose your connection then you better start over. So it's being able to get that through with the reliancy of Aspera and speeds to make it more viable to where viable to move larger data.
Al Martin: So, Rajani, so that's good. Thank you but Rajani do you — where are the inhibitors that you face most? I mean what's the biggest challenge? In other words is it the network? Is — do we find ourselves in a situation where even if we could use a spare by example that we end up, you know, we end up, you know, using FedEx or something because of a firewall issue and they don't want to like open it up for what a couple days in terms of driving this? Is that the biggest issue or do you see the biggest issue is some other form of technology?
Al Martin: Well, how does lift solve that? I know I'm always pushing like single like a one button experience. Where we at?
Michael Springgay: ...to load into the Db2 warehouse. So it really makes it an end-to-end process, maybe not exactly one click, but very seamless options the customer has.
Al Martin: Well how close are we at a one click?
Michael Springgay: I know that that's easy for executive to come in and say, "We need pushbutton data movement."
Al Martin: Right, but what's real? What's fact? What's fiction?
Michael Springgay: Well, I mean, I think with data movement there is challenges with simple one click because you have to understand the source and the target of your extract. So there is some knowledge of shaping the data that you have that has to be presented.
But I think we're at the point where really you do just provide the shape of the data. So that's knowledge you have typically. And then really is you push go and it does the extract and moves it up.
Al Martin: So, in short, there is pushbutton, but you've got to know the source, the destination and you've got to set things up to do it correctly. Are you doing a lot of ground or cloud right now?
Michael Springgay: Yes. There is still a lot of customers looking to move to the cloud. And all of them are using — most of them are using at least the Lift technology to drive that.
Al Martin: And when they do I mean do you get more requests that's just ground to ground or do you get a lot of clients that are coming in and say because of, you know, the difficulty in ground to Cloud and in meeting a technology like Aspera? Is that where most the urgent requests or the main queries come in from clients?
Michael Springgay: I think because it is more challenging it is where we seen our focus to help them deal with it. Certainly on the ground to ground the customers are asking or if there are more traditional tools that people have been using that hasn't really changed all that much. So there they're really just looking at speed is important because even in ground to ground the distance between data centers is growing.
It used to be that people were moving data probably from box A to box B and they sat beside each other. I mean now people are moving data across data centers that are probably on the other side of the country. So even there you see the challenges of network coming into play.
Michael Springgay: Yes. I mean, I think IBM really takes the value is that we're trying to bring the value and charge you for what value you can get out of your data \ movement around of data. I would certainly see with competitors that they maybe make it easy to get it in but then they charge you either to move it between services which, you know, is sort of like double charging it because you had already got it up there and you're charging for that as well as to get it out. They don't really want to move the data back out.
Al Martin: Now and there I guess the other question I had that kind of hits me, you know, is it would seem to me like we've got a campaign what's best (about the) work. It's a revolution to move data from on-prem to the cloud right and get it there. But having said that.
You know, there's a revolution of moving data to the cloud but I think — I don't know what your experience is. To me it's a difficult concept. Even though clients are actually moving to the cloud they've got to open up their firewall — whatever the case may be to get that data from on-prem to the Cloud. Are they willing to do that? Do you see that or do you see a lot of push back or do you see all right now we're going to — as the results we're going to throw it into FedEx because we're uncomfortable with that?
Michael Springgay: Yes I think there are certainly challenges and typically there's internal struggles between the different groups and the (unintelligible) you have to put firewalls open. I find that most cases that can happen. The customers understand. They go to cloud they have to open up some of the network.
But there definitely is a challenge about how much of the bandwidth of the Internet connection they pay for that they're willing to use to move data. So that often is a challenge and then, you know, you fall back to FedEx because you can't use all the bandwidth.
Al Martin: How often are — does that fall back to does this happen?
Michael Springgay: You know, not all the time. It really depends on how much data we're moving. So I do think it probably, you know, we can probably move better data with Aspera than maybe a customer is willing to take. And so they'll fall back to FedEx earlier then maybe we would say that you need to.
Rajani Maindiratta: So we — Aspera we just sent out with our product for almost two years now. So it's been IBM's only technology in (unintelligible) cases.
Michael Springgay: You know what we did? I think IBM acquired Aspera like a year when nobody knows it right, remember?
Michael Springgay: Couple years at least.
Rajani Maindiratta: Yes and then we integrated it into data on cloud.
Al Martin: So it's kind of the backbone of everything we do at this point in time.
Rajani Maindiratta: It is yes.
Al Martin: All right, what separates that from just any other technology? Why would we do that?
Michael Springgay: Well I think, you know, it's the commercial version of some technology that's been out there and research. But they've really made it simple, you know, because it's all there. It's set up for you as part of the product. We don't have to just install a client. You don't have to (unintelligible) a server or we're managing the resiliency of it which is some of the other (unintelligible).
Al Martin: But the speed, you know, as I'm working with a team, speed is amazing. So (unintelligible) on difference than just the normal data movements. I mean there's some magic in there some place.
Michael Springgay: Yes so they're using a UDB versus TCB. So they're building in on top of the network. They're not relying on sort of your operating system for the failure. So they have their own algorithms to figure out how to split the file up and send it in parallel and put it back together and make sure the pieces all come correctly.
Al Martin: So this is like Silicon Valley. And I know you've probably never seen that show.
Al Martin: It's the same thing.
Michael Springgay: Yes it's the — yes. They are basically applying heuristics at a higher level to basically exceed the standard bandwidth limitations.
Al Martin: Because we've talked a little — well a lot of what we're — I think some of it's just an overall point of view but some of its IBM technology. What do you — is there any more you can say about the industry in this space, I mean things that where the industry is, where you think it's going where — you know, I mean whether — where Amazon is, where Google is? Is your — any comments to any of those areas?
Michael Springgay: Amazon for sure is certainly marketing the whole disk transfer service. Like they have names and everything and they definitely grow in the size of which appliances they have. They have appliances for small, medium and large whereas, you know, we're trying to at IBM at least not have to have a smaller version available through the Aspera technology.
Google is I think in the same area but they're probably not in (cure) in the disk transfer. They do have disk transfer services but it's not quite as robust as say AWS for sure. I think everyone's trying to figure out the network bandwidth solution. I know AWS recommends some open source technology that is similar but not quite as sophisticated as Aspera do the technology but they haven't really necessarily invested directly in how to do that to make it easy for the customer, sort of that one button click quick thing they're telling you how to basically build up different networks and servers to solve a problem.
Rajani Maindiratta: (Unintelligible) commented (unintelligible). He didn't — he doesn't get overall that they're moving into the warehouse, updates of the (unintelligible). That's a huge focus.
Rajani Maindiratta: Compression is a (unintelligible) yes, just more efficient, effective use of parallelism as well is something — an area to focus on as well (unintelligible).
Michael Springgay: I think the other thing that people are less interested once they get the data in to moving it back out. So, you know, you would see an ETL process taken in the past where the data qualities weren't that big so people would bring it out into a middle tier tool to do transformations and put a pack in. Now you really see customers pushing hard to get all of those transformations done as much within the database server so that the data movement piece is minimized.
So one other aspect is trying to avoid it (unintelligible) because you can keep the data in as many locations as they are and only move the smaller pieces that you need to make business logic grow.
Michael Springgay: I think it's really around ease of use and speed. I mean really with the embedded Aspera in our cloud technology we're trying to make that simple versus, you know, I mention Amazon who sort of points you in places where you can speed up your networks but you have to set up all the pieces. So I think we're trying to make it simple.
Al Martin: Simple. Any other questions or any other answer that you'd give Rajani?
Rajani Maindiratta: Yes simple and just understanding our customer as well too I think is something we're going to have to (unintelligible) in the sense of we take feedback from our customers and drive improvements within the product.
Al Martin: If I'm a client what kind of speed can I — I mean I know it's dependent on compression, data size or lots of different things. What kind of speed? You may have hit this before but what can I expect?
Michael Springgay: So I think from, you know, around the cloud we're looking about 200 gigabits an hour range with the (lift) technology and in Aspera. On-premise though we can see, you know, 4 or 5 terabytes an hour if we have — has enough network storage attached to the machine with the (unintelligible).
Al Martin: Good. So if the listeners want to learn more about, you know, data movement, our point of view, IBM's point of view on data movement where would they go?
Rajani Maindiratta: They — we have some kind of knowledge center I believe has a lot of information there. Some of our recent announcements around the mass data movement advice and the (list field) I think there's announcement. If you go look at the IBM announcements page there's a blog around those technologies that we released.
Al Martin: Good. Well any ones that she might have missed Michael?
Michael Springgay: ...IBM Cloud is probably the best place to go first. That's where we want you to go and that's going to make it the simplest one. But or closest to one click that we (unintelligible).
Al Martin: ...because I get to know people. And also I'm going to ask what book you're reading or what you would recommend because I read a lot of books and so I just know that I'm going to go there. But anything left on the data movement that you'd say that you feel like we missed or left unsaid or did we get it?
Rajani Maindiratta: I have a question. What was everyone else doing at IBM (unintelligible) that we may want to be doing then?
Michael Springgay: That's a good question.
Michael Springgay: ...IBM is focused on that bigger device, the one that we may need is smaller things like the Snowball.
Al Martin: Snowball, explain that.
Michael Springgay: So Amazon Snowball is a 50 terabyte device. It's pretty neat because it's a small box that you can (unintelligible) versus, you know, our last bit of — and so it's a fairly large appliance. So I think just from the ease of use for those smaller customers that aren't in that huge terabyte, 100 terabytes, petabyte system it's a slightly more convenient.
Michael Springgay: I think the challenge is is that the customers we're working with will ultimately need (masked in) movement. But when you're talking about a POC or something like that when you're starting area is a much smaller dataset that they're trying to move over. And so just maybe gives us more flexibility in the early stages.
So look I like to do a little lightening round and then Rajani I'll start with you.
Al Martin: What's the most exciting thing you're working on right now? Anything that you'd call out that'll get you up every morning?
Rajani Maindiratta: I think it's around this area (unintelligible).
Al Martin: Yes, but give me some specific in terms of around this area the data movement.
Rajani Maindiratta: Update performance. That’s kind of the area I'm focused on.
Rajani Maindiratta: It's very similar kind of development space. It's just that I think one of our benchmarks has been against the appliance. And we — we're trying to get our speeds to kind of up — on par or better than that. So there's challenges just like as in the load area but I wouldn't say it's worse - any worse or better.
Al Martin: Fair enough, fair enough. How about you, Michael? What gets you up in the morning?
Michael Springgay: Yes, so, you know, my day job is to focus on both (external) tables and compatibility. But I find external tables very interesting in the sense that there's a lot of different file formats out there that people need to shape. And the more we can broadly support exactly what they have without having them make them do transformations is makes some so, you know, making sure we can figure out how to get the right set of options to support the most broad set of data that there is.
Al Martin: Is it — I know you'd be studied. In other words you do your research. Where do you - where does the data movement expert do his research?
Michael Springgay: A lot of it comes from customers. You know, I work with a lot of customers. So we see what their data is and understand exactly how they have laid out their data.
Al Martin: What Rajani what if I want to go do a crash course on data movement? Any place you have me — direct me outside of the locations you already have at IBM?
Al Martin: And best practice?
Rajani Maindiratta: ...and then the big movement space.
Al Martin: How about books?
Al Martin: That's all right.
Michael Springgay: ...college many years earlier so I like — I want old detective books.
Michael Springgay: I'm reading the Bosch series right now.
Al Martin: The Bosch series?
Al Martin: What have you learned out of that?
Rajani Maindiratta: Oh just kind of perseverance.
Al Martin: Would you recommend it?
Al Martin: Yes, I recommend.
Rajani Maindiratta: Yes. And there's a coaching famous leadership book that I forgot about.
Al Martin: Fantastic. I like books. I got a list. I can — you know, my list is never exhausted but I keep going at it.
Al Martin: All right, fair enough guys. I thank you for taking your time today. I appreciate it but talk to you next time.