The data is in—massive amounts of it, and high computing power can help enterprises make some sense out of it. For a technology that has gone through ebbs and flows in popularity, high performance computing (HPC) may be expanding to use cases beyond those found in scientific research as more industries can tap into valuable insights gained from artificial intelligence, machine learning, and other emerging technologies.
So, what does this mean to your organization? If you’re increasingly facing the need to translate large amounts of consumer data to track trends or calculate thousands of financial transactions a day to support business growth, is HPC something you should be considering?
Consider this post your beginner’s guide to HPC. We’ll break down some of the basics of HPC so that when you’re ready to take the plunge, you’ll have a better idea of what you’re getting into.
What is HPC?
By aggregating compute power of hundreds or even thousands of compute servers (more on these later), high performance computing is the ability to carry out data-intensive calculations at much higher speeds than a regular desktop computer. We’re talking about computational power that can be measured in petaflops—that’s millions of times faster than a “regular” computer’s power—or even exaflops. This kind of exascale computing is necessary to advance work in countless industries and specialties, from genome sequencing to storm tracking.
Why should I know about HPC?
HPC capabilities were historically used to advance academic research, but with the rise of artificial intelligence and other data-intensive use cases, other industries may begin to realize the advantages of utilizing HPC-like infrastructure and supercomputers, the most powerful category of HPC solutions.
Though you might hear HPC and supercomputing terms used interchangeably, HPC generally refers to the robust processing ability we’ve already covered, while a supercomputer is a single system made up of many computers, interconnected via a very fast network, and the associated storage repository that feeds the data into the processors.
Given the upfront cost, specialized infrastructure and maintenance expenses they require, many organizations consider supercomputers out of reach. This is understandable when you consider that this type of compute power was historically reserved for national research laboratories.
However, with the introduction of open source approaches, HPC solutions have become more accessible. For example, by the 1990s, clusters of massively parallel computers built from industry-standard components began to emerge alongside proprietary machines. This is when Linux and open source software adoption in HPC skyrocketed leading to ubiquitous presence at all high performance computing sites. And recently, since the late 2000s, some of the traditional HPC workloads began to move to cloud resources. In place of monolithic, on-premise supercomputers, the industry has moved towards flexible infrastructure based on open standards, with workloads decoupled from specific hardware.
And now, HPC is also at a point where it’s converging with big data as more and more companies, regardless of the industry, are facing large-scale computational problems stemming from the flood of data and the need to process it within time and cost parameters. Greater compute power allows larger datasets to be ingested more quickly and more permutations to be performed in the same amount of time, ultimately allowing problems to be solved more efficiently.
What makes up an HPC system?
HPC systems are actually groups—or clusters—of computers. The main elements in an HPC system are network, storage and compute. The compute part of the system takes the data given to it via the network and spins up results.
An HPC cluster consists of hundreds or even thousands of compute servers—or nodes—that are networked together. The nodes work in parallel with each other, running smaller parts of a large job simultaneously, which reduces the time it takes to solve one problem. If you’ve heard the term parallel processing, this is where it comes from. Therein lies the main advantage of HPC, although, we are starting to see an increasing number of workflows that use hardware and software resources in a predefined sequence or in multiple consecutive stages.
The nodes need to be able to talk to one another to work in harmony—and computers talk over networks. Networking makes it possible for the cluster to talk to data storage, and it must be able to support high-speed transfers of data between these components.
And finally, the storage component, which is critical to the performance of HPC applications, must be able to feed and ingest data to and from the compute servers as quickly as it is processed.
Open source and Linux power the world’s top supercomputers
Your HPC cluster won’t run without software. Open source software takes down adoption barriers and makes it easier for the scientific community to collaborate.
One of the most popular choices for running HPC clusters is Linux and Red Hat Enterprise Linux (RHEL), is the operating system of choice for some of the top supercomputers in the world. The list of 500 of the world’s most powerful computer systems is revealed twice a year (in June and November). For almost 30 years, the TOP500 ranking has provided a pulse on the HPC market while encouraging the exchange of data and software throughout the tech industry and beyond.
You can read more about the TOP500 and how Red Hat is paving the way for next-generation HPC based on containers here.
Containers and what’s next for HPC
Containers provide the ability to package application code, its dependencies and user data, simplifying sharing of insights and data across global communities and making migrating applications between distributed sites and clouds easier.
These capabilities make containers relevant for HPC. Podman is just one of the container engines that is showing a lot of promise in the HPC market and as the adoption of containers continues to grow, the need for HPC-aware container orchestration platforms, like the Kubernetes-based Red Hat OpenShift, will come into focus.
Open source software, off-the-shelf hardware, and community-driven standards are critical to HPC’s sustainability. Innovation in the HPC world has been community-driven, and recent adoption of cloud technologies can help make HPC capabilities applicable to organizations outside the realm of academia. Collaboration among the tech industry, academia and other public agencies will ultimately drive expanded capabilities for HPC and clouds alike.
Before the world of digital computers, the term “computer” was used to describe people who performed these mathematical calculations. While humans sure can do a whole lot, computers make it possible to process information beyond our reach. And the computational power of a laptop or desktop computer pales in comparison to high performance computing systems, which can be used to solve some of today’s most complex and important scientific problems.
With open source approaches, HPC infrastructure and applications can span far beyond science, bleeding into enterprise datacenters and moving to clouds. You can take the next step by exploring what Red Hat is doing in HPC.
About the author
As the Managing Editor of the Red Hat Blog, Thanh Wong works with technical subject matter experts to develop and edit content for publication. She is fascinated with learning about new technologies and processes, and she's vested in sharing how they can help solve problems for enterprise environments. Outside of Red Hat, Wong hears a lot about the command line from her system administrator husband. Together, they're raising a young daughter and live in Maryland.
Takwimu ni katika kiasi kikubwa cha hayo, na nguvu ya juu ya kompyuta inaweza kusaidia makampuni kufanya baadhi ya maana nje yake. Kwa teknolojia ambayo imepitia ebbs na mtiririko katika umaarufu, kompyuta ya utendaji wa juu (HPC) inaweza kupanuka kwa kutumia kesi zaidi ya zile zilizopatikana katika utafiti wa kisayansi, kwani viwanda zaidi vinaweza kutumia ufahamu muhimu uliopatikana kutoka kwa akili bandia, kujifunza mashine, na teknolojia zingine zinazojitokeza. Hii inamaanisha nini kwa shirika lako? Ikiwa unakabiliwa na mahitaji ya kuhamisha kiasi kikubwa cha data ya watumiaji kufuatilia mwenendo au kuhesabu maelfu ya shughuli za kifedha kwa siku ili kusaidia ukuaji wa biashara, ni HPC kitu unapaswa kuzingatia? Fikiria chapisho hili mwongozo wako wa mwanzoni kwa HPC. Tutavunja chini baadhi ya misingi ya HPC ili wakati wewe ni tayari kuchukua plunge, utakuwa na wazo bora ya nini wewe ni kupata katika. HPC ni nini? Kwa kuunganisha nguvu ya kompyuta ya mamia au hata maelfu ya seva za kompyuta, kompyuta ya utendaji wa juu ni uwezo wa kufanya hesabu za data kwa kasi kubwa kuliko kompyuta ya kawaida ya desktop. Sisi ni kuzungumza juu ya nguvu ya hesabu ambayo inaweza kupimwa katika petaflops - kwamba ni mamilioni ya mara kwa kasi kuliko nguvu ya "kawaida" kompyuta au hata exaflops. "Kama ilivyoelezwa hapo juu, ""Exascale Computing"" ni aina ya kompyuta muhimu kwa maendeleo ya kazi katika viwanda na utaalam usio na hesabu, kutoka kwa mpangilio wa genome hadi kufuatilia dhoruba." Kwa nini unapaswa kujua kuhusu HPC? Uwezo wa HPC umekuwa ukitumika kwa maendeleo ya utafiti wa kitaaluma, lakini kwa kuongezeka kwa akili bandia na kesi nyingine za matumizi ya data, viwanda vingine vinaweza kuanza kutambua faida za kutumia miundombinu ya HPC na supercomputers, jamii yenye nguvu zaidi ya suluhisho la HPC. Ingawa unaweza kusikia HPC na supercomputing maneno kutumika interchangeably, HPC kwa ujumla inahusu uwezo wa usindikaji imara sisi tayari kufunikwa, wakati supercomputer ni mfumo mmoja ulioundwa na kompyuta nyingi, interconnected kupitia mtandao wa kasi sana, na associated kuhifadhi hazina ambayo feeds data katika processors. Kwa kuzingatia gharama ya awali, miundombinu maalumu na gharama za matengenezo wanayohitaji, mashirika mengi yanaona kompyuta kubwa kuwa haiwezekani. Hii inaeleweka wakati wewe kuzingatia kwamba aina hii ya nguvu ya hesabu ilikuwa kihistoria kuhifadhiwa kwa ajili ya maabara ya utafiti wa kitaifa. Hata hivyo, kwa kuanzishwa kwa mbinu za chanzo wazi, ufumbuzi wa HPC umekuwa rahisi zaidi. Kwa mfano, kufikia miaka ya 1990, makundi ya kompyuta za sambamba sana zilizojengwa kutoka kwa vipengele vya kiwango cha sekta yalianza kuibuka pamoja na mashine za wamiliki. Wakati huo, matumizi ya Linux na programu ya chanzo wazi katika HPC iliongezeka sana na kusababisha uwepo wa kila mahali katika maeneo yote ya kompyuta ya utendaji wa juu. Tangu mwishoni mwa miaka ya 2000, kazi za HPC za jadi zimeanza kuhamia kwenye wingu. Badala ya monolithic, juu ya eneo supercomputers, sekta imehamia kuelekea miundombinu rahisi kulingana na viwango wazi, na kazi uncoupled kutoka vifaa maalum. HPC pia iko katika hatua ambapo inakaribia na data kubwa, kwani makampuni mengi, bila kujali sekta, yanakabiliwa na matatizo makubwa ya kompyuta yanayotokana na mafuriko ya data na haja ya kuifanya ndani ya muda na vigezo vya gharama. Nguvu kubwa ya kompyuta inaruhusu seti kubwa za data kuchukuliwa haraka zaidi na permutations zaidi kufanywa katika kiasi sawa cha muda, hatimaye kuruhusu matatizo kutatuliwa kwa ufanisi zaidi. Ni nini kinachotengeneza mfumo wa HPC? HPC mifumo ni kweli makundi <unk> au makundi <unk> ya kompyuta. Sehemu kuu za mfumo wa HPC ni mtandao, kuhifadhi na kuhesabu. Sehemu ya kompyuta ya mfumo inachukua data iliyotolewa kwake kupitia mtandao na huzunguka matokeo. HPC kundi lina mamia au hata maelfu ya seva za hesabu au nodes ambazo zimeunganishwa pamoja. Node hufanya kazi sambamba na kila mmoja, ikiendesha sehemu ndogo za kazi kubwa kwa wakati mmoja, ambayo hupunguza wakati unachukuliwa kutatua tatizo moja. Kama umesikia neno usindikaji sambamba, hii ni ambapo inakuja kutoka. Hili ni faida kuu ya HPC, ingawa tunaanza kuona idadi inayoongezeka ya mtiririko wa kazi ambayo hutumia vifaa na rasilimali za programu katika mfululizo uliowekwa au katika hatua nyingi mfululizo. Node zinahitaji kuwa na uwezo wa kuzungumza na kila mmoja kufanya kazi kwa upatano - na kompyuta kuzungumza juu ya mitandao. Mtandao hufanya iwezekane kwa cluster kuzungumza na kuhifadhi data, na lazima iweze kuunga mkono uhamisho wa kasi wa data kati ya vipengele hivi. Mwishowe, kipengele cha kuhifadhi, ambacho ni muhimu kwa utendaji wa maombi ya HPC, lazima iweze kulisha na kumeza data kutoka na kwenda kwa seva za kompyuta haraka kama inavyofanyiwa kazi. Open source na Linux nguvu supercomputers juu ya dunia yako HPC kundi si kukimbia bila programu. Programu ya chanzo wazi hupunguza vizuizi vya kupitishwa na inafanya iwe rahisi kwa jamii ya kisayansi kushirikiana. Hifadhi ya HPC ya Red Hat Enterprise Linux (RHEL) ni moja ya mifumo ya uendeshaji ya msingi ya kompyuta za juu duniani. Orodha ya 500 ya mifumo ya kompyuta yenye nguvu zaidi ulimwenguni hufunuliwa mara mbili kwa mwaka (mwezi Juni na Novemba). Kwa karibu miaka 30, orodha ya TOP500 imetoa msukumo kwenye soko la HPC wakati wa kuhamasisha kubadilishana kwa data na programu katika sekta ya teknolojia na zaidi. Unaweza kusoma zaidi juu ya Top500 na jinsi Red Hat inavyofungua njia kwa HPC ya kizazi kijacho kulingana na vyombo hapa. Vifaa vya HPC hutoa uwezo wa kufunga nambari ya maombi, utegemezi wake na data ya mtumiaji, kurahisisha kushirikiana kwa ufahamu na data katika jamii za kimataifa na kufanya maombi ya kuhamia kati ya maeneo ya kusambazwa na wingu rahisi. Uwezo huu kufanya vyombo muhimu kwa HPC. Podman ni moja tu ya injini za kontena ambazo zinaonyesha ahadi nyingi katika soko la HPC, na kama kupitishwa kwa kontena kunaendelea kukua, hitaji la majukwaa ya orchestration ya kontena ya HPC, kama vile Red Hat OpenShift ya Kubernetes, itazingatia. Programu ya chanzo wazi, vifaa vya shelf, na viwango vya jamii ni muhimu kwa uendelevu wa HPC. Ubunifu katika ulimwengu wa HPC umekuwa unaendeshwa na jamii, na kupitishwa kwa hivi karibuni kwa teknolojia za wingu kunaweza kusaidia kufanya uwezo wa HPC utumike kwa mashirika nje ya ulimwengu wa kitaaluma. Ushirikiano kati ya sekta ya teknolojia, wasomi na mashirika mengine ya umma hatimaye itasababisha uwezo ulioongezwa kwa HPC na mawingu sawa. Kabla ya ulimwengu wa kompyuta za dijiti, neno "kompyuta" lilitumiwa kuelezea watu waliofanya hesabu hizo za hisabati. Ingawa wanadamu wanaweza kufanya mambo mengi, kompyuta zinawezesha sisi kuchanganua habari ambazo hatuwezi kuzifikia. Na nguvu ya kompyuta ya kompyuta ndogo au kompyuta ya desktop ni ndogo ikilinganishwa na mifumo ya kompyuta ya utendaji wa juu, ambayo inaweza kutumika kutatua baadhi ya matatizo ya kisayansi ya leo ngumu na muhimu. Kwa njia za chanzo wazi, miundombinu ya HPC na maombi yanaweza kuenea mbali zaidi ya sayansi, kuvuja katika vituo vya data vya biashara na kuhamia kwenye mawingu. Kwa hivyo, kwa sababu ya hali hii, unaweza kuamua kuanzisha Red Hat HPC. Kama Mhariri Mtendaji wa Red Hat Blog, Thanh Wong anafanya kazi na wataalam wa mada ya kiufundi kuendeleza na kuhariri maudhui kwa ajili ya kuchapishwa. Yeye ni fascinated na kujifunza kuhusu teknolojia mpya na michakato, na yeye ni vested katika kushiriki jinsi wanaweza kusaidia kutatua matatizo kwa mazingira ya biashara. Nje ya Red Hat, Wong anasikia mengi kuhusu mstari wa amri kutoka kwa mume wake msimamizi wa mfumo. Pamoja, wanalea binti mdogo na wanaishi Maryland.