《代码英雄》第二季（6）：数据大爆炸
大数据 代码英雄

> 
> 代码英雄讲述了开发人员、程序员、黑客、极客和开源反叛者如何彻底改变技术前景的真实史诗。
> 
> 
> 



> 
> **什么是《代码英雄》**
> 
> 
> <ruby> 代码英雄 <rt>  Command Line Heroes </rt></ruby>是世界领先的企业开源软件解决方案供应商红帽（Red Hat）精心制作的原创音频播客，讲述开发人员、程序员、黑客、极客和开源反叛者如何彻底改变技术前景的真实史诗。该音频博客邀请到了谷歌、NASA 等重量级企业的众多技术大牛共同讲述开源、操作系统、容器、DevOps、混合云等发展过程中的动人故事。
> 
> 
> 


![](/data/attachment/album/202009/25/061607owaw9spklswpqqaj.jpg)


本文是《[代码英雄](https://www.redhat.com/en/command-line-heroes)》系列播客[第二季（6）：数据大爆炸](https://www.redhat.com/en/command-line-heroes/season-2/the-data-explosion)的[音频](https://dts.podtrac.com/redirect.mp3/audio.simplecast.com/4c5f676c.mp3)脚本。




> 
> 导语：大数据将有助于解决大问题：我们如何种植粮食、如何向需要的人运送物资、如何治疗疾病。但首先，我们需要弄清楚如何处理它。
> 
> 
> 现代生活充满了相互联系的组件。我们现在一天产生的数据比几千年来的数据还要多。 Kenneth Cukier 解释了数据是如何发生变化的，以及它是如何开始改变我们的。Ellen Grant 博士告诉我们波士顿儿童医院是如何使用开源软件将堆积如山的数据转化为个性化治疗方法。Sage Weil 则分享了 Ceph 的可扩展和弹性云存储如何帮助我们管理数据洪流。
> 
> 
> 收集信息是了解我们周围世界的关键。大数据正在帮助我们拓展永不停歇的探索使命。
> 
> 
> 


**00:00:03 - Saron Yitbarek**：


如果你把从人类历史早期到 2003 年创建的所有数据计算在内，你将得到大约 500 万 GB 的数据。我们昨天创建了多少 GB 数据？


**00:00:15 - 问卷调查答案 1**：


哦，天哪，10 万。


**00:00:21 - 问卷调查答案 2**：


可能是 500 万 GB 数据。


**00:00:23 - 问卷调查答案 3**：


我们在昨天一天之内创建了多少 GB 的数据？1000 万 GB 数据？


**00:00:31 - 问卷调查答案 4**：


我不太知道，可能是 200 万 GB 数据？


**00:00:36 - 问卷调查答案 5**：


也许一天就有 100 万 GB 数据？


**00:00:40 - Saron Yitbarek**：


答案是？超过 25 亿 GB 数据！


**00:00:44 - 问卷调查答案 1**：


哇哦。


**00:00:44 - 问卷调查答案 2**：


25 亿？


**00:00:45 - 问卷调查答案 3**：


所以，我们已经打破了世界纪录。


**00:00:45 - 问卷调查答案 4**：


那可真是很多个 G 啊。


**00:00:45 - 问卷调查答案 5**：


我都不敢相信那有这么多的数据。


**00:00:52 - Saron Yitbarek**：


在 2016 年，我们的年度在线数据流量首次超过了 1ZB。一个 ZB 是 1000^7 字节。好，记住这个数字了吗？现在把它乘以 3，因为那是我们将在 2021 年拥有的数据量大小。


**00:01:10**：


我知道，大脑不是以 ZB 为单位进行思考的，但请你至少暂时记住这个数。 我们的 IP 流量将在五年内翻三番。 这是数据的洪流，而我们正处于其中。


**00:01:24**：


在刚过去的一分钟里，人们发出了 1600 万条短信；与此同时，在我说出这句话的时间里，谷歌处理了 20 万条搜索。


**00:01:37**：


如果我们能在数据洪流来临时做好准备、站稳脚跟，那么隐藏在其中的模式、答案和秘密可以极大地改善我们的生活。


**00:01:50**：


我是 Saron Yitbarek，这里是《代码英雄》，一款红帽公司原创的播客节目。浪潮近在眼前。这里是第二季第六集，数据大爆炸。


**00:02:17**：


我们如何处理如此大量的数据？ 采集到这些数据后，我们如何利用它们？ 大数据将为我们解决一些最复杂的问题：


**00:02:29**：


如何管理交通、如何种植粮食、如何向需要的人提供物资，但这一切的前提是，我们必须弄清楚该怎么使用这些数据、以及该怎么在短得不能再短的时间内完成对它们的处理。


**00:02:43 - Kenneth Cukier**：


通过获取更多的数据，我们可以深入到这些子群体、这些细节，而这是我们以前从来没有过的方式。


**00:02:53 - Saron Yitbarek**：


Kenneth Cukier 是《<ruby> 经济学人 <rt>  The Economist </rt></ruby>》的高级编辑，他也和我都在科技播客《Babbage》里。


**00:03:01 - Kenneth Cukier**：


这并不是说我们以前无法收集数据。我们可以，但这真的、真的很昂贵。真正的革命性突破是，我们使数据搜集变得十分容易。


**00:03:10**：


现在收集数据的成本极低，而且处理起来也超级简单，因为都是由电脑完成的。这已经成为我们这个时代的巨大革命，它可能是现代生活最显著的特征，在未来几十年甚至下个世纪都会如此。这也是大数据如此重要的原因。


**00:03:33 - Saron Yitbarek**：


历史可以提醒我们这种变化是翻天覆地的。想想看，4000 年前，我们把所有的数据都刻在了干泥板上。


**00:03:46 - Kenneth Cukier**：


这些黏土盘很重。黏土盘被烤干后，刻在其中的数据就无法更改。从古至今，信息处理、存储、传输、创建的方式都发生了变化，对吗？


**00:04:04 - Saron Yitbarek**：


时代巨变。大约在 1450 年，印刷机的发明带来了第一次信息革命。今天，我们也迎来了一场革命。


**00:04:16 - Kenneth Cukier**：


现在的存储介质很轻巧。信息的修改变得极度简单，借助现有的处理器，我们只需要使用删除键就能修改我们所拥有的信息实例，无论那是储存在磁带上还是晶体管里。我们可以以光速传输数据，不用携带什么黏土盘。


**00:04:37 - Saron Yitbarek**：


在 15 世纪，借助印刷机，大量的数据得以传播。这些知识提升了人们对事物的认识，并促成了启蒙运动。


**00:04:45**：


今天，大数据可以再次提升我们的知识水平，但我们必须要想办法充分利用这些数据。唯有修好大坝、备好轮机，才能让浪潮为人所用。


**00:05:00 - Kenneth Cukier**：


当下，人们远没有做到对数据的充分利用。这一点非常重要，因为我们已经看到，数据中存在这种潜在的价值，而收集、存储和处理数据的成本在近百年来，乃至于近十年来，已经显著地降低了。


**00:05:22 - Kenneth Cukier**：


这很振奋人心。但问题是，我们在文化上、在我们的组织流程上，甚至我们的 CFO 和 CIO 们拨给相关方面的预算中，并不重视这种价值，


**00:05:35 - Saron Yitbarek**：


想着这种事肯定让人极度沮丧。启蒙运动在敲门，却无人应答。然而，我们不回答的部分原因是：门后到底是谁？这些数据能带来什么？


**00:05:51**：


Kenneth 认为，某些公司不采用大数据，乃是因为它太过于新奇。


**00:05:56 - Kenneth Cukier**：


一旦你收集了大量数据后，你能拿它干什么？我就直说吧，只有傻子才会以为自己知道。你绝对无法设想你今天收集到的数据明天能拿来做什么用。


**00:06:12**：


最重要的是要有数据，并以开放的思想对待所有可以使用的方式。


**00:06:18 - Saron Yitbarek**：


如果我们按照 Kenneth 说的那样以正确的方式对待大数据，我们将会发现一切的全新可能性。这将是一个人人 —— 而不只是数据科学家 —— 都能洞察趋势、分析细节的世界。


**00:06:33 - Kenneth Cukier**：


如果我们能意识到，这个世界是可以通过收集经验证据来理解、改变和改善的，并且可以用一种自动化的方式进行改善，我们将会得到看待它的全新角度。我个人认为，现如今，在世界各地，上至政策制定者、下至星巴克咖啡师，都在经历这种引人深思的文化上或心理上的变化。


**00:07:00**：


各行各业的人都有点数据基因，就像是被感染了似的。现在，无论他们专注于什么方面，他们都以大数据的方式思考。


**00:07:15 - Saron Yitbarek**：


Kenneth Cukier 给我们讲了一个简短的故事来展现这种新数据思维方式的力量。微软的一些研究人员开始着手研究胰腺癌问题。


**00:07:27**：


人们发现胰腺癌往往为时已晚，早期发现可以挽救生命。因此，研究人员开始询问这些患者，在开始搜索有关胰腺癌的信息之前几个月，他们搜索了什么？ 而早在发现前数年，他们又搜索了什么？


**00:07:46**：


研究人员开始寻找埋藏在所有搜索数据中的线索和模式。


**00:07:54 - Kenneth Cukier**：


他们有了重大发现。通过分析患者在最终开始搜索“胰腺癌”之前的这段时间中所搜索的关键词，他们识别出了一套规律，可以非常准确地预测搜索者是否患有胰腺癌。


**00:08:09**：


在这里，我们能学到一点：想象力与数据中潜在规律的结合，是可以挽救生命的。他们现在所要做的就是找到一种方法，通过方法来解释这一发现，这样当人们在搜索这些术语时，他们可以以一种微妙的方式干预，说，“你可能要去诊所检查一下。”


**00:08:29**：


像这样使用数据，就能救人于水火之中。


**00:08:37 - Saron Yitbarek**：


研究人员偶然发现的是一种新的癌症筛查方式，通过这种方法，患者可以提前一个月得知自己可能患癌。利用数据不仅仅是一个利润或效率最大化的问题。


**00:08:52**：


它的意义远不止于此。对于人类而言，这些数据中确确实实存在着大量的潜在利好。抗拒使用大数据可能只是自欺欺人。接下来，我们要关注的是，这场将数据投入工作的持久战。


**00:09:18**：


哈佛医学院的波士顿儿童医院去年完成了 26000 多台手术，进行约 25 万人次的儿童放射检查。


**00:09:31**：


医护人员的表现令人称道，但有一个巨大的障碍挡在他们面前。


**00:09:37 - Ellen Grant**：


在医院的环境中，尤其是作为医生，我们经常会遇到难以获取数据的问题。


**00:09:45 - Saron Yitbarek**：


这位是 Ellen Grant 医生，她是波士顿儿童医院的儿科神经放射科医生，她在诊疗时依靠访问数据和分析医学图像。


**00:09:56 - Ellen Grant**：


如果没有专门设置的环境，想要从 packs 里存储的图像进行额外的数据分析绝非易事。当你在一个只提供了普通的医院电脑的读片室里时，要做到这一点并不容易。


**00:10:14**：


获取数据实际上是有障碍的。


**00:10:17 - Saron Yitbarek**：


其实许多医院都会大量抛弃数据，因为存储它们的成本实在过于高昂。这部分数据就像这样丢失了。像 Grant 这样的放射科医生可能是第一批因为数据实在太多而感到沮丧的医务人员。


**00:10:33**：


当医院走向数字化后，他们开始创造大量的数据，很快，这个量就大到无法处理了。


**00:10:41 - Ellen Grant**：


我，作为一名临床医生，在读片室里的时候希望能将所有复杂的分析工作在研究环境中做完。但我无法随便就从 packs 中拿出来图像，拿到一些可以进行分析的地方，再拿回到我手里。


**00:10:59 - Saron Yitbarek**：


顺便说一句，packs 就是医院存储其图像的数据仓库。Grant 医生知道有一些工具可以让这些图像 packs 发挥更大的功能，但成本太高。


**00:11:12 - Ellen Grant**：


随着机器学习和 AI 时代的到来，数据的生产量将会日渐加大，我们会需要更多计算资源来进行这类大规模的数据库分析。


**00:11:27 - Saron Yitbarek**：


数据已经堆积如山了，但处理能力却没有相称的增长。在这一前提下，对数据的彻底处理将变得遥不可及。而复杂、昂贵的超级计算机并不是医院的选择。


**00:11:41**：


Grant 医生深感沮丧。


**00:11:44 - Ellen Grant**：


我们能不能想出一个更好的办法，让我把数据拿到这里来，分析一下，然后放回去，这样我就可以在会诊的时候，一边解释临床图像，一边把分析做完，因为我希望可以在会诊上展示数据，在此同时进行快速分析。


**00:11:56**：


我可不想在不同的电脑和存储器之间把这些数据挪来挪去，这不是我的工作。我的工作是理解非常复杂的医学疾病，并把相关的事实真相记在脑子里。


**00:12:10**：


我想专注于我的技术领域，在此同时利用计算机领域的新兴技术；而不必这方面过于深入钻研。


**00:12:21 - Saron Yitbarek**：


Grant 和世界各地的放射科医生们需要的是一种方法，只要点击图像就能运行详细分析，并让这一切都发生在云端，这样医院就不必建立自己的服务器场地，也不必把医务人员变成程序员。


**00:12:40**：


他们需要一种方法来使他们的数据尽可能地拯救生命。这正是 Grant 医生和几位代码英雄决定去做的事。


**00:12:55**：


Grant 在波士顿儿童医院的团队正在与红帽和马萨诸塞州开放云（MOC）合作。关于 MOC 的更多内容稍后再说。首先，我们需要请出 Rudolph Pienaar，他是医院的一名生物化学工程师，来描述一下他们的解决方案。它是一个开源的、基于容器的成像平台。


**00:13:15 - Saron Yitbarek**：


它完全是在云端运行的，所以你不受医院本身计算能力的限制。他们称这一作品为 ChRIS。


**00:13:24 - Rudolph Pienaar**：


ChRIS 有一个后台数据库，其实就是一个 Django Python 机器。它可以跟踪用户，并跟踪这些用户使用过的数据以及分析结果。


**00:13:35**：


围绕这个数据库，有大量的服务群，这些服务都是作为自己的实例存在于容器中。它们处理与医院资源的通信，比如与医院数据库的通信。这些服务从资源中提取复杂的数据，将其推送给云端的、或者另一个实验室的、或者别的什么地方的其他服务处理。在计算数据的地方，有 Kubernetes 之类的编排服务，以及你需要使用的分析程序。数据处理结束之后，结果就会被发送回来。


**00:14:11 - Saron Yitbarek**：


对于 Grant 医生来说，ChRIS 成像平台是一种让数据活起来的方法。更重要的是，这种数据处理方式能让她成为更好的医生。


**00:14:21 - Ellen Grant**：


优秀的医生之所以优秀，是因为他们在一生中积累了丰富的从业经验。如果我们能把这一点融入到数据分析中，以此来获得更多的信息，我们就能知道得更多，并更有效地整合这些经验。


**00:14:39**：


例如，我对特定病患的特定受伤方式的认识，取决于我的从医经验和对这些经验的整体理解。


**00:14:52**：


现在，我可以根据真实数据创建受伤症状分布的概率图，并将其公之于众；我也可以寻找有相似模式的患者，并告诉他们在接受治疗时，什么对他们最有效，以便更接近精准医疗。


**00:15:10**：


整合大量的数据，尝试探索我们过去的知识，并尽你所能，点明治疗病人的最佳方式。


**00:15:21 - Saron Yitbarek**：


这对被送到医院的孩子意味着什么？Grant 医生说，ChRIS 平台能提供更有针对性的诊断和更个性化的护理。


**00:15:31 - Ellen Grant**：


如果我们拥有更复杂的数据库，我们就能更好地理解信息之间繁杂的相互作用，因此就能更好地指导每个患者。我认为 ChRIS 就像是我进入超级大脑的接口，它能让我比平时更聪明，因为我不能一次把所有数据保存在我的大脑中。


**00:15:53 - Saron Yitbarek**：


当赌注如此沉重时，我们要突破人类大脑的极限。这位是 Máirín Duffy。她是红帽团队中的设计师，她让 ChRIS 成为现实，而根据个人经验，她知道这件事其中的风险。


**00:16:15 - Máirín Duffy**：


我父亲中风了，所以我一直作为病人家属等待医疗技术诊断，因为当一个人中风并被送到医院之后，医务人员必须弄清楚是哪种类型的中风。根据中风类型，有不同的治疗方法。


**00:16:31**：


如果使用了错误的治疗方案，就可能发生极其糟糕的事。所以，在这种情况下，你能越快地把病人送来做核磁共振，就能越快地得到治疗方案。速度越快就有可能挽救他们的生命。


**00:16:43**：


想想看，仅仅是把图像处理从云端推送出来，并行化处理，就能让它快很多。这样就能将这个过程从几小时、几天，缩短到几分钟。


**00:16:55 - Saron Yitbarek**：


医学可能正迎来一个新的拐点。一个不是由药理学驱动，而是由计算机科学驱动的拐点。另外，想想像 ChRIS 这种东西的拓展性。


**00:17:08**：


发展中国家的医生也可以受益于波士顿儿童医院的专业知识和数据集。任何有手机服务的人都可以通过网络访问能够拯救生命的数据和计算结果。


**00:17:24**：


除了医学，很多其他领域也可能出现类似的拐点。但前提是，人们得知道如何从自己的数据中找到隐藏信息。为了做到这一点，他们需要探索一个全新的计算领域。


**00:17:46**：


世界各地的人们都在学习如何利用数据。就像在波士顿儿童医院一样，将数据洪流导向目标。


**00:17:56**：


换句话说，我们在处理这些数据。但我们之所以能做到这一点，是因为新一代的云计算使之成为可能。


**00:18:11**：


对于像 ChRIS 这样的平台来说，一个关键因素是基于云计算的新型存储方式。请记住，很多医院都会把收集到的数据扔掉，因为他们根本无法容纳所有数据。


**00:18:25**：


这就是我想重点讨论的数据泛滥的最后一块拼图：存储解决方案。对于 ChRIS 来说，存储解决方案是一个叫 Ceph 的开源项目。它使用的马萨诸塞州开放云，就基于 Ceph。


**00:18:45**：


我和 Ceph 的创建者 Sage Weil 聊了聊，想了解更多关于像波士顿儿童医院这样的地方是如何在闪电般的时间内处理海量数据的。以下是我与 Sage 的对话。我认为，第一个重要问题是，什么是 Ceph，它能做什么？


**00:19:05 - Sage Weil**：


当然，Ceph 是一个由软件定义的存储系统，它允许你提供可靠的存储服务，并在不可靠的硬件上提供各种协议。


**00:19:14**：


它的设计从开始就是满足可扩展性，所以你可以拥有非常非常大的存储系统、非常大的数据集。于此同时，系统对硬件故障和网络故障有优秀的容忍性，所以即使出现了一些这类问题，存储中的数据仍然不会变得难于访问。


**00:19:29 - Saron Yitbarek**：


现在，数据太多了。


**00:19:31 - Sage Weil**：


是的。


**00:19:33 - Saron Yitbarek**：


如此大的工作量。要处理的东西实在是太多了。你认为这个解决方案出现得是时候吗？


**00:19:39 - Sage Weil**：


是的，肯定是这样。在当时，行业中这方面的严重不足是显而易见的。没有开源的解决方案可以解决可扩展的存储问题。所以，我们显然得造个轮子。


**00:19:53 - Saron Yitbarek**：


考虑到我们每天要处理的数据量，以及它将来只会越来越多、越来越难管理的事实，你认为当今该怎么做才能解决这种日益增长的需求？


**00:20:09 - Sage Weil**：


我认为有几方面。一方面，有令人难以置信的数据量正在产生，所以你需要可扩展的系统。它不仅可以在硬件和数据规模上进行扩展，而且，它的管理成本应该是一定的，至少应该基本固定。


**00:20:25 - Saron Yitbarek**：


嗯。


**00:20:26 - Sage Weil**：


你不会想就为每多 10PB 存储空间或类似的东西就多雇一个员工吧？我认为这套系统在运维上也必须可扩展。


**00:20:33 - Saron Yitbarek**：


是的。


**00:20:35 - Sage Weil**：


这是其中的一部分。我认为，人们利用存储空间的方式也在改变。一开始，都是文件存储，然后我们有了虚拟机的块存储，我觉得对象存储在某种程度上是行业的重要趋势。


**00:20:51**：


我认为，下一个阶段的目标并不局限于提供一个对象存储端点，并将数据存储在集群中；我们需要将解决方案进一步升级，好让它能管理集群的集群，抑或是对分布于不同地理位置的云空间及私有数据中心储存空间中的数据进行管理。


**00:21:13**：


例如说，你现在将数据写入一个位置，随着时间的推移，你可能会想将数据分层到到其他位置，因为它更便宜、或者服务器离你更近；或者，一旦数据太老、不会频繁使用了，你就需要将其移动到性能更低、容量更大的层次上，以保证存储的成本较低。


**00:21:27**：


你可能也会为了遵循地方法规而移动数据。在欧洲的一些地区接收数据时，数据来源必须保持在特定的政治边界内。


**00:21:39**：


在某些行业，像 HIPAA 这样的东西限制了数据的移动方式。我认为，随着现代 IT 组织越来越多地分布在不同的数据中心、公有和私有云中，统一地、自动化地管理它们的能力正变得越加重要。


**00:21:58 - Saron Yitbarek**：


当你想到未来我们要如何管理和存储数据，以及如何处理数据的时候，开源在其中扮演了怎样的角色？你曾提到，你之所以要创建一个开源的解决方案，是因为你个人的理念和你对自由和开源软件的强烈感情。


**00:22:16**：


你如何看待开源对未来其他解决方案的影响？


**00:22:21 - Sage Weil**：


我认为，特别是在基础设施领域，解决方案正在向开源靠拢。我认为原因是基础设施领域的成本压力很大，特别是对于构建软件即服务（SaaS）或云服务的人来说，低成本的基础设施很重要，从他们的角度来看，开源显然是一个非常好的方法。


**00:22:48**：


第二个原因更多地是社会因素，在这个快速发展的领域里有如此多新的工具、新的框架、新的协议、新的数据思维方式，这个领域中有这么多创新和变化，有这么多不同的产品和项目在相互作用，所以很难以传统方式做到这一点，比如说，让不同的公司互相签订合作协议，共同开发。


**00:23:20**：


开源可以消除此事上的所有阻力。


**00:23:28 - Saron Yitbarek**：


Sage Weil 是红帽公司的高级咨询工程师，也是 Ceph 项目的负责人。我要绕回到《经济学人》的 Kenneth Cukier，以从一个更整体的视角上进行讨论，因为我希望，我们能够记住他关于人与数据之间关系的看法，以及我们从泥板，到印刷机，再到像 Sage 打造的云端奇迹的进步历程。


**00:23:55 - Kenneth Cukier**：


这关乎人类的进步，关乎我们如何更好地理解世界，如何从现实中总结经验，以及如何改善世界。这进步也是人类一直以来的使命。


**00:24:08 - Saron Yitbarek**：


使命永无止境。但是，与此同时，学会处理我们收集到的数据并将其投入使用，是整整一代人的开源任务。我们将在田纳西州的<ruby> 橡树岭国家实验室 <rt>  Oak Ridge National Laboratory </rt></ruby>短暂停留，结束我们的数据之旅。它是世界上最快的超级计算机 Summit 的所在地，至少在 2018 年是最快的超级计算机。


**00:24:43**：


这台机器每秒能处理 20 万亿次计算。换个计量单位，就是 200 petaflops。这样的处理速度，对于医院、银行或者今天所有受益于高性能计算的成千上万的组织来说并不现实。


**00:25:04**：


像 Summit 这样的超级计算机更多的是留给强子对撞机的领域。不过话说回来，我们曾经在泥板上记录的只是 100 字节的信息。


**00:25:16**：


在数据存储和数据处理的领域中，非凡的壮举不断成为新的常态。有一天，我们或许能将 Summit 级别的超级计算机装进口袋。想一想，到时候我们能够搜索到的答案。


**00:25:42**：


下一集，我们聊聊无服务器。第 7 集将会讲述我们与基于云的开发之间不断发展的关系。我们将会探究，在我们的工作中有多少可以抽象化的部分，以及在这个过程可能会失去的东西。


**00:25:58 - Saron Yitbarek**：


同时，如果你想深入了 ChRIS 的故事，请访问 [redhat.com/chris](https://www.redhat.com/chris) ，了解它是如何构建的，以及如何为项目本身做出贡献。


**00:26:12 - Saron Yitbarek**：


《代码英雄》是一款红帽公司原创的播客。你可以在 Apple Podcast、Google Podcast 或任何你想做的事情上免费收听。


**00:26:24 - Saron Yitbarek**：


我是 Saron Yitbarek。坚持编程，下期再见。



> 
> **什么是 LCTT SIG 和 LCTT LCRH SIG**
> 
> 
> LCTT SIG 是 LCTT <ruby> 特别兴趣小组 <rt>  Special Interest Group </rt></ruby>，LCTT SIG 是针对特定领域、特定内容的翻译小组，翻译组成员将遵循 LCTT 流程和规范，参与翻译，并获得相应的奖励。LCRH SIG 是 LCTT 联合红帽（Red Hat）发起的 SIG，当前专注任务是《代码英雄》系列播客的脚本汉化，已有数十位贡献者加入。敬请每周三、周五期待经过我们精心翻译、校对和发布的译文。
> 
> 
> 欢迎[加入 LCRH SIG](/article-12436-1.html) 一同参与贡献，并领取红帽（Red Hat）和我们联合颁发的专属贡献者证书。
> 
> 
> 




---


via: <https://www.redhat.com/en/command-line-heroes/season-2/the-data-explosion>


作者：[Red Hat](https://www.redhat.com/en/command-line-heroes) 选题：[bestony](https://github.com/bestony) 译者：[TimeBear](https://github.com/TimeBear) 校对：[Northurland](https://github.com/Northurland), [wxy](https://github.com/wxy)


本文由 [LCRH](https://github.com/LCTT/LCRH) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出


OK
Big data is going to help solve big problems: how we grow food; how we deliver supplies to those in need; how we cure disease. But first, we need to figure out how to handle it.

Modern life is filled with connected gadgets. We now produce more data in a day than we did over thousands of years. Kenneth Cukier explains how data has changed, and how it's beginning to change us. Dr. Ellen Grant tells us how Boston Children's Hospital is using open source software to transform mountains of data into individualized treatments. And Sage Weil shares how Ceph's scalable and resilient cloud storage helps us manage the data flood.

Gathering information is key to understanding the world around us. Big data is helping us expand our never ending mission of discovery.

**00:03** - *Saron Yitbarek*

If you take all human data created from the dawn of time to 2003, you'd get about five million gigabytes of data. How many gigabytes of data did we create yesterday?

**00:15** - *Poll response 1*

Oh gosh, 100,000.

**00:21** -*Poll response 2*

Like five million gigabytes.

**00:23** - *Poll response 3*

How many gigabytes of data did we create yesterday in one day? 10 million gigabytes?

**00:31** - *Poll response 4*

I would say, I don't know, like two million maybe?

**00:36** - *Poll response 5*

Maybe a million gigabytes in a day?

**00:40** - *Saron Yitbarek*

Answer? More than 2.5 billion.

**00:44** - *Poll response 1*

Oh wow.

**00:44** - *Poll response 2*

2.5 billion?

**00:45** - *Poll response 3*

So, we already beat the world record.

**00:45** - *Poll response 4*

That's a lot of gigabytes.

**00:45** - *Poll response 5*

That's a lot of data right there. I don't believe it.

**00:52** - *Saron Yitbarek*

In 2016, our annual data traffic online passed one zetabyte for the first time. That's one sextillion bites, if it helps. Okay, got that number in your head? Now triple it, because that's how much data we'll have by 2021.

**01:10** - *Saron Yitbarek*

I know, the brain wasn't made to think in zetabytes, but just hold onto this one little fact for a second. Our IP traffic will triple in five years. It's a data flood and we're in the middle of it.

**01:24** - *Saron Yitbarek*

That last minute that went by, people sent 16 million text messages and in the time it took me to say that sentence, Google processed 200,000 searches.

**01:37** - *Saron Yitbarek*

Hidden inside that data flood are patterns, answers, and secrets that can massively improve our lives if we can just keep ourselves standing upright when the flood comes in.

**01:50** - *Saron Yitbarek*

I'm Saron Yitbarek and this is Command Line Heroes, an original podcast from Red Hat. The tidal waves are on the horizon. This is episode 6 of season 2, the data flood.

**02:17** - *Saron Yitbarek*

How do we handle such enormous amounts of data? How will we make use of that data once it's captured? Big data is going to solve some of our most complicated problems:

**02:29** - *Saron Yitbarek*

How we manage traffic. How we grow food. How we deliver supplies to those in need. But only once we figure out how to work with all that data, how to process it, and at breakneck speed.

**02:43** - *Kenneth Cukier*

By having more data, we can drill down into these sub-groups, these particulars, and these details in ways that we never could before.

**02:53** - *Saron Yitbarek*

Kenneth Cukier is a senior editor at The Economist and he's also the Saron Yitbarek of their tech podcast called Babbage.

**03:01** - *Kenneth Cukier*

It's not to say that we couldn't collect the data before. We could, it just was really, really expensive. The real revolution is that we can collect this data very easily.

**03:10** - *Kenneth Cukier*

It's very inexpensive and the processing is super simple because it's all done by a computer. This has become the huge revolution of our era and it is probably the most defining aspect of modern life and will be for the next several decades, if not the next century. That's why big data's such a big deal.

**03:33** - *Saron Yitbarek*

A little history can remind us how radical that change has been. Think about it, 4,000 years ago, we were scratching all our data into dried slabs of mud.

**03:46** - *Kenneth Cukier*

These clay disks were heavy. The data that is imprinted in them once they're baked can't be changed. All of these features of how information was processed, stored, transferred, created has changed, right?

**04:04** - *Saron Yitbarek*

Changed big time. Around the year 1450, you get the first information revolution with the invention of the printing press. And today, we have our own revolution.

**04:16** - *Kenneth Cukier*

It's lightweight. It can be changed super simply because we can just use the delete key and change the instantiation of the information that we have in whether it's the magnetic tape or in the transition of the electronic transistors and the processors that we have. We can transport it at the speed of light, unlike say a clay disk that you have to carry.

**04:37** - *Saron Yitbarek*

The printing press levelled up our understanding of things with the 15th-century flood of data that ushered in the enlightenment.

**04:45** - *Saron Yitbarek*

Today, big data can level us up again. But only if we figure out how to take advantage of all that data. Only if we build the dams and the turbines that will put the flood to work.

**05:00** - *Kenneth Cukier*

There is a huge gap between what is possible and what companies and individuals and organizations are doing. That's really important because we could already see that there's this latent value in the data and that the cost of collecting, storing, and processing the data has really dropped down considerably to where it was, of course, a hundred years ago but even just ten years ago.

**05:22** - *Kenneth Cukier*

That's really exciting. But the problem is that, culturally, and in our organizational processes and even in the budgets that our CFOs and our CIOs allot to data, we're not there yet.

**05:35** - *Saron Yitbarek*

Super frustrating when you think about it. Enlightenment knocking at the door and nobody's answering. Part of the reason we're not answering though is that, well, who's actually behind the door? What's all this data gonna deliver?

**05:51** - *Saron Yitbarek*

Kenneth figures the newness of big data keeps some companies from taking that leap.

**05:56** - *Kenneth Cukier*

What is the value of the data once you collect a lot of it? The most honest answer is if you think you know, you're a fool. Because you can never identify today all the ways with which we're going to put the data to uses tomorrow.

**06:12** - *Kenneth Cukier*

The most important thing is to have the data and to have an open mind in all the ways that it can be used.

**06:18** - *Saron Yitbarek*

What Kenneth’s envisioning if we get big data right, is a wholesale transformation of our attitudes towards what's possible. A world where everybody, not just data scientists, can see the potential and gain insight.

**06:33** - *Kenneth Cukier*

By understanding that the world is one in which we can collect empirical evidence about it in order to understand it, and in order to change it, and improve it, and that improvements can happen in an automated fashion, we will see the world in a different way, and I think that's the really interesting change that's happening now culturally or psychologically around the world with policy makers and business people and Starbucks baristas.

**07:00** - *Kenneth Cukier*

Everyone in all walks of life sort of have the data gene. They got it. They've sort of gotten the inoculation and now, everywhere they look, they think in a data mindset.

**07:15** - *Saron Yitbarek*

Kenneth Cukier told us a quick story to illustrate the power of that new data mindset. Some researchers at Microsoft began thinking about pancreatic cancer.

**07:27** - *Saron Yitbarek*

People often find out about pancreatic cancer too late and early detection could save lives. So, the researchers began asking before these patients start searching for information on pancreatic cancer, what were they searching for in the previous few months? And in the previous years.

**07:46** - *Saron Yitbarek*

They began looking for clues buried inside all that search data. They began looking for patterns.

**07:54** - *Kenneth Cukier*

They struck pay dirt. They saw that they can identify patterns in the search terms leading up to the moment where people started searching for pancreatic cancer that predicted very accurately that people had pancreatic cancer.

**08:09** - *Kenneth Cukier*

The lesson here is that by using their imagination in terms of the latent knowledge inside of data, they can save lives. All they need now to do is to find a way to instrument through policy this finding, so that when people are searching for these terms, they can intervene in a subtle way to say, "You might wanna go to a healthcare clinic and get this checked out."

**08:29** - *Kenneth Cukier*

If they start doing that, people's lives will be saved.

**08:37** - *Saron Yitbarek*

What the researchers stumbled upon is a new form of cancer screening, a process that could alert patients month earlier. Making use of data isn't just a question of maximizing profits or efficiency.

**08:52** - *Saron Yitbarek*

It's about so much more than that. Hiding in all that data are real, enormous positives for humankind. If we don't use that data, we could be cheating ourselves. It's that epic struggle to put data to work that we're focusing on next.

**09:18** - *Saron Yitbarek*

Boston Children's Hospital at Harvard Medical School performed more than 26,000 surgeries last year. Kids walk through their doors for about a quarter million radiological exams.

**09:31** - *Saron Yitbarek*

The staff is doing incredible work, but there's a huge roadblock standing in their way.

**09:37** - *Ellen Grant*

A lot of the problems that we have in a hospital environment, especially as a physician, is how to get access to the data.

**09:45** - *Saron Yitbarek*

That's Dr. Ellen Grant. She's a pediatric neuroradiologist at Boston Children's and she depends on accessing data and analyzing medical images.

**09:56** - *Ellen Grant*

It's not simple to get access into say a pax archive where the images are stored to do additional data analysis unless you set up an environment. That's not easy to do when you're in a reading room where there are just standard hospital PCs provided.

**10:14** - *Ellen Grant*

There's a barrier to actually get to the data.

**10:17** - *Saron Yitbarek*

Hospitals actually dump a lot of their data because they can't afford to store it. So, that data's just lost. Radiologists like Dr. Grant may have been the first group of doctors to feel the frustration of data overload.

**10:33** - *Saron Yitbarek*

When they went digital, they began creating enormous amounts of data and that quickly became impossible to handle.

**10:41** - *Ellen Grant*

I, as a clinician, in the reading room wanted to be able to do all the fancy analysis that could be done in a research environment. But there's no way to easily get images off of the packs and get them to some place where the analysis could be done and get them back into my hands.

**10:59** - *Saron Yitbarek*

Packs by the way are what hospitals call the data banks that store their images. Dr. Grant knew there were tools that could make those packs of images more functional, but costs were prohibitive.

**11:12** - *Ellen Grant*

As we're entering into this era of machine learning and AI, there's more of that gonna happen that we need these larger computational resources to really start to do the large database analysis we wanna do.

**11:27** - *Saron Yitbarek*

The data's been piling up, but not so much the processing. On premise data processing would be out of reach. And elaborate, expensive supercomputers aren't an option for hospitals.

**11:41** - *Saron Yitbarek*

Dr. Grant became deeply frustrated.

**11:44** - *Ellen Grant*

Can't we figure out a better way for me to just get data over here, analyze it, get it back so I can do it at the console where I'm interpreting clinical images cause I wanna have that data there and analyzed there quickly.

**11:56** - *Ellen Grant*

I don't wanna have to move to different computers and memorize all this line code when that's not my job. My job is trying to understand very complex medical diseases and keep all those facts in my head.

**12:10** - *Ellen Grant*

I wanted to keep my focus on where my skill set was but exploit what is emerging in the computational side without having to dive that deep into it.

**12:21** - *Saron Yitbarek*

What Dr. Grant and radiologists around the world needed was a way to click on imagery, run detailed analysis, and have it all happen on the cloud so the hospital didn't have to build their own server farm and didn't have to turn the medical staff into programmers.

**12:40** - *Saron Yitbarek*

They needed a way to make their data save all the lives it could and so, that's exactly what Dr. Grant and a few command line heroes decided to build.

**12:55** - *Saron Yitbarek*

Dr. Grant's team at Boston Children's Hospital worked with Red Hat and the Massachusetts Open Cloud (MOC). More on the MOC a little later. First though, here's Rudolph Pienaar, a biomedical engineer at the hospital describing their solution. It's an open source, container-based imaging platform.

**13:15** - *Saron Yitbarek*

It's all run on the cloud, too. So you're not limited by the computing power at the hospital itself. They call their creation ChRIS.

**13:24** - *Rudolph Pienaar*

There's a backend database that's a Django Python machine really and that keeps track of users. It keeps track of the data they've processed. It keeps track of results.

**13:35** - *Rudolph Pienaar*

Then there are a whole bunch of constellation of services around this database that all exist as their own instances in containers. These deal with communicating with hospital resources like databases. They deal with the intricacies of pulling data from these resources and then pushing them out to other services that exist on a cloud or another lab or whatever it might be. On the place where data is computed, there's all these services like Kubernetes, the schedule, the actual analysis of the data that you want to be doing. Then, pulling it back again.

**14:11** - *Saron Yitbarek*

For Dr. Grant, the ChRIS imaging platform is a way to make data come to life. More than that, it's a way for data to make her a better doctor.

**14:21** - *Ellen Grant*

What makes a person a good physician is the experience they've had over a lifetime of practicing. But if I can kind of embody that into the data analysis and access more of that information, we just all know more and can combine the information better.

**14:39** - *Ellen Grant*

For example, I have a perception of what a certain pattern of injury looks like in a certain patient population built on my Gestalt understanding from the memories that I have.

**14:52** - *Ellen Grant*

I can now actually create probability maps of their distributions and inform everybody based on real data, or I can look for similar patients who had similar patterns and say what works best with them when they were treated to try to get closer to precision medicine.

**15:10** - *Ellen Grant*

Integrating the large amount of data and trying to exploit our past knowledge and to best inform how to treat any individual the best you can.

**15:21** - *Saron Yitbarek*

What does that mean for the children that are brought to the hospital? Dr. Grant says the ChRIS platform delivers more targeted diagnoses and more individualized care.

**15:31** - *Ellen Grant*

If we have more complex databases, we can understand complex interactions better and hopefully guide individual patients better. I think of ChRIS basically as my portal into multiple accessory lobes so I can be a lot smarter than I can on my own, ‘cause I cannot keep all this data in my brain at one time.

**15:53** - *Saron Yitbarek*

When the stakes are this high, we wanna push past the limits of the human brain. Here's Máirín Duffy. She's a designer on the Red Hat team that makes ChRIS happen and she knows from personal experience what's at stake.

**16:15** - *Máirín Duffy*

My father had a stroke, so I've been there sort of in the patient's family side of waiting for medical technology, ‘cause when someone has a stroke, they bring you in the hospital and they have to figure out what type of stroke it is. Based on the type, there's different treatments.

**16:31** - *Máirín Duffy*

If you give the wrong treatment, then really bad things can happen. So, the faster you can get the patient in for an MRI, the faster you can interpret the results in that situation. The faster you can potentially save their life.

**16:43** - *Máirín Duffy*

Just think about just the fact of getting that image processing pushed out of the cloud, parallelized, make it so much faster. So instead of being hours or days, it's minutes.

**16:55** - *Saron Yitbarek*

Medicine may be arriving at a new inflection point. One not driven by pharmacology, but by computer science. Also, think about the scalability of something like ChRIS.

**17:08** - *Saron Yitbarek*

You could have doctors in developing countries benefiting from the expertise and data sets at Boston Children's Hospital. Anybody with cell service could access web-based computing and data that might save lives.

**17:24** - *Saron Yitbarek*

Besides medicine, lots of other fields could be witnessing a similar inflection point. But only if they figure out how to make their data collections sing. To do that, they need to discover a whole new territory of computing.

**17:46** - *Saron Yitbarek*

All around the world, we're learning to make use of our data. Diverting those data floods towards our own goals, like at Boston Children's Hospital.

**17:56** - *Saron Yitbarek*

In other words, we're processing that data. But we can only do that because a new generation of cloud-based computing makes the processing possible.

**18:11** - *Saron Yitbarek*

For platforms like ChRIS, a key ingredient is that cloud-based computing is a new kind of storage. Remember that lots of hospitals throw out the data they gather because they literally can't hold it all.

**18:25** - *Saron Yitbarek*

That's what I wanna focus on as a last piece of the data flood puzzle. The storage solution. For ChRIS, the storage solution came in the form of an open source project called Ceph. The Massachusetts Open Cloud, which ChRIS uses, depends on Ceph.

**18:45** - *Saron Yitbarek*

So, I got chatting with its creator, Sage Weil, to learn more about how places like Boston Children's can process enormous amounts of data in lightning time. Here's my conversation with Sage. I think a great first question is what is Ceph and what does it do?

**19:05** - *Sage Weil*

Sure. Ceph is a software-defined storage system that allows you to provide a reliable storage service, providing various protocols across unreliable hardware.

**19:14** - *Sage Weil*

It's designed from the ground up to be scalable, so you can have very, very large storage systems, very large data sets and you can make them available and tolerate hardware failures and network failures and so on without compromising availability.

**19:29** - *Saron Yitbarek*

Nowadays, there's just so much data.

**19:31** - *Sage Weil*

Yes.

**19:33** - *Saron Yitbarek*

So much consumption. There's just so much to get a handle on. Do you feel like the timing of it was part of the need for the solution?

**19:39** - *Sage Weil*

Yes, definitely. At the time, it just seemed painfully obvious that there's this huge gap in the industry. There was no open source solution that would address the scalable storage problem. So it was obvious that we needed to build something.

**19:53** - *Saron Yitbarek*

When we're thinking about the amount of data we're dealing with on a daily basis and the fact that it's only growing, it's only getting bigger and harder to manage, what do you see that's being worked on today that will maybe address this growing need?

**20:09** - *Sage Weil*

I think there are sort of several pieces of it. The first thing is that there's incredible amount of data being generated, of course, so you need scalable systems that can scale not just in the amount of hardware and data that you're storing but also have a sort of fixed or nearly fixed operational overhead.

**20:25** - *Saron Yitbarek*

Mm-hmm (affirmative).

**20:26** - *Sage Weil*

You don't wanna pay another person per 10 petabytes or something like that. They have to be operationally scalable, I guess would be the way to put it.

**20:33** - *Saron Yitbarek*

Yeah.

**20:35** - *Sage Weil*

That's part of it. I think also the way that people interact with storage is changing as well. In the beginning, it was all file storage and then we have block storage for VMs, object storage is sort of, I think, a critical trend in the industry.

**20:51** - *Sage Weil*

I think really the next phase of this is not so much around just providing an object storage endpoint and being able to store data in one cluster, but really taking this sort of the level up and having clusters of clusters, geographically distributed mesh of cloud footprints or private datacenter footprints where data is stored and being able to manage the data that's distributed across those.

**21:13** - *Sage Weil*

Maybe you write the data today in one location, you tier it off to somewhere else over time because it's cheaper, or it's closer, or the data's older, and you need to move it to a lower-performance, higher-capacity tier for pricing reasons.

**21:27** - *Sage Weil*

Dealing with things like compliance so that when you ingest data in one, in Europe, it has to stay within certain political boundaries in order to comply with regulation.

**21:39** - *Sage Weil*

In certain industries, you have things like HIPAA that restricts the way that data's moved around. I think as modern IT organizations are increasingly spread across lots of different datacenters and lots of public clouds and their own private cloud infrastructure, being able to manage all this data and automate that management is becoming increasingly important.

**21:58** - *Saron Yitbarek*

When you think about how we're going to manage and store data in the future, and process data in the future, how does open source play a role in that? You mentioned that you wanted to create an open source solution because of your personal philosophy and your strong feelings on free and open software.

**22:16** - *Saron Yitbarek*

How do you see open source affecting other solutions in the future?

**22:21** - *Sage Weil*

I think that, particularly in the infrastructure space, solutions are converging towards open source. I think the reason for that is there are high cost pressures in the infrastructure space and particularly for people building software-as-a-service or cloud services, it's important that they keep their infrastructure very cheap and open source is obviously a very good way to do that from their perspective.

**22:48** - *Sage Weil*

I think the second reason is more of a, I think, a social reason and that it's such a fast-moving field where you have new tools, new frameworks, new protocols, new ways of thinking about data and there's so much innovation and change happening in that space and so many different products and projects that are interacting, that it's very hard to do that in a way that is sort of based on the traditional model, where you have different companies having partnership agreements and co-development or whatever.

**23:20** - *Sage Weil*

Open source removes all of that friction.

**23:28** - *Saron Yitbarek*

Sage Weil is a senior consulting engineer at Red Hat and the Ceph project lead. I'm gonna circle back to Kenneth Cukier from The Economist so we can zoom out a bit, because I want us to remember that vision he had about our relationship with data and how we've progressed from clay tablets to the printing press to cloud-based wonders like the one Sage built.

**23:55** - *Kenneth Cukier*

This is about human progress and it is about how we can understand the world and the empirical evidence of the world better to improve the world. It is the same mission of progress that humans have always been on.

**24:08** - *Saron Yitbarek*

The mission never ends. But, in the meantime, learning to process the data we've gathered and put that flood to work, that's an open source mission for a whole generation. We're ending our data journey with a quick stop at the Oak Ridge National Laboratory in Tennessee. It's home to Summit, the world's fastest supercomputer or at least fastest as of 2018.

**24:43** - *Saron Yitbarek*

This machine processes 200,000 trillion calculations per second. That's 200 petaflops, if you're counting. Processing speed like that isn't practical for hospitals or banks or all the thousands of organizations that benefit from high high performance computing today.

**25:04** - *Saron Yitbarek*

Supercomputers like Summit are reserved more for Hadron Collider territory. But then again, we were once recording just a hundred bytes of info on clay tablets.

**25:16** - *Saron Yitbarek*

The story of data storage and data processing is one where extraordinary feats keep becoming the new normal. One day, we might all have Summit-sized supercomputers in our pockets. Think of the answers we'll be able to search for then.

**25:42** - *Saron Yitbarek*

Next episode, we're going serverless. Or are we? Episode 7 is all about our evolving relationship with cloud-based development. We're figuring out how much of our work we can abstract and what we might be giving up in the process.

**25:58** - *Saron Yitbarek*

Meantime, if you wanna dive deeper into the ChRIS story, visit [redhat.com/chris](https://www.redhat.com/chris) to learn more about how it was built and how you can contribute to the project itself.

**26:12** - *Saron Yitbarek*

Command Line Heroes is an original podcast from Red Hat. Listen for free on Apple Podcasts, Google Podcasts, or wherever you do your thing.

**26:24** - *Saron Yitbarek*

I'm Saron Yitbarek. Until next time, keep on coding.

### Keep going

### Why our big data questions require scalable, open answers

Data use is too universal for closed source solutions

### Creating ChRIS

Boston Children's Hospital needed better access to their data. So they teamed up with Red Hat and the Mass Open Cloud.

### File, block, or object storage?

We can store worlds of data—and how we do that matters.

### Featured in this episode

### Also in this episode

### Red Hat legal and privacy links

[About Red Hat](/en/about/company)[Jobs](/en/jobs)[Events](/en/events)[Locations](/en/about/office-locations)[Contact Red Hat](/en/contact)[Red Hat Blog](/en/blog)[Diversity, equity, and inclusion](/en/about/our-culture/diversity-equity-inclusion)[Cool Stuff Store](https://coolstuff.redhat.com/)[Red Hat Summit](https://www.redhat.com/en/summit)