Interviewing my mother, a mainframe COBOL programmer - <AUTHOR> https://medium.com/@Svenskunganka/interviewing-my-mother-a-mainframe-cobol-programmer-c693d40d88f7#.66qbnhd8p <comment> c-smile Nice article.

Reminded me my own experience of creating bank software system for a State Bank of one of republics that was in the middle of separating from former USSR. "Perestroika" they named that process.

No mainframes, where we would get those at that time? Only i386 as a server and PC XTs as terminals. Connected by ropes essentially, but NetBIOS was working over them. Plus 1200 bod or so modems to connect with other banks in the republic.

But we (team of two developers) did it in 9 months.

Excellent Zortech C++ by Walter Bright (creator of D language).

Editor - MultiEdit the Great. What we would do without it - I don't know.

Database - dbVista (later known as dbRaima).

Everything was working on PTDOS (Russian MSDOS alike thing)

Yet multitasking C library providing cooperative multitasking (same execution model Node.js is using).

And everything in 640k of RAM that "should be enough for anybody".

We were young and nobody hadn't even a chance to tell us that this is "too serious", "impossible" and the like.

A lot of manual input by operators, so usability requirements were surprisingly high: [http://www.terrainformatica.com/2016/02/ui-usability- solutio...](http://www.terrainformatica.com/2016/02/ui-usability-solution-i- am-especially-proud-of/)

:)

<comment><USER4> I would _love_ to read a writeup of that project.

<comment> c-smile Hard to recall actually, many waters had flown away since that time.

Just one fact: the system survived monetary reform in USSR of 1991 when in three days the whole country suddenly changed all banknotes ( [https://en.wikipedia.org/wiki/Monetary_reform_in_the_Soviet_...](https://en.wikipedia.org/wiki/Monetary_reform_in_the_Soviet_Union,_1991) ). History of USSR is just one big social experiment of things like this.

Number of transactions in these three days was astonishing as you can imagine. We've spent all three days in the bank worrying mostly about hardware. 12 keyboards and 3 workstations from twenty-something were lost in the battle :)

But the most crazy thing that I saw was on the next day after the reform. It was January 26th, and it was snowy all three days. Plaza in front of bank's building was covered by snow and banknotes on top of it, surreal. People who were late or exceeded exchange quota were just throwing money away.

But our system did handle it surprisingly well.

My pardon for hijacking the COBOL topic. C++ rulez forever!

<comment><USER4> Thank you for the details :)

</comment></comment></comment></comment><comment><USER5> My grandmother who is in her 80s was one of the US military's first female programmers... and she was a widow... and she was German expatriate (as you can imagine the difficulty of that post WWII). They programmed in punch cards and drank a lot of coffee.

If it sounds like I am bragging.. I am.. I am very proud of her. What I'm not proud of is not spending enough time with her to extract more details. I did ask her a similar question about being a German widow working with predominately males and she basically had the same answer as the articles mother... "no problems what so ever".

This article is a great reminder that I need to call my grandmother more often :)

<comment><USER6> Was her name Grace Hopper?

<comment><USER5> No it was not. I wish I could provide an official link of her work but I can't seem to google anything. She isn't famous by any means and is exceedingly humble (hence the difficulty on finding details indirectly).

Interestingly though she absolutely loves all things sci-fi. She records almost everything the SyFy channel produces and loves Babylon 5, Dr. Who, etc...

She is exceedingly good at board games. Even ones she has never played before she often wins. I remember brining Chinese checkers once to her with my wife. She had never played before but she absolutely trashed my wife and me in multiple games.

She grew up in Germany and I have often wondered if that had any affect on her non-traditional-gender hobbies and likes. Or maybe it is just her personality (as I love board games and SciFi as well).

</comment></comment></comment><comment><USER7> "We used to have personal desks, but now we have this "pick whatever spot is available" open area. I dislike it a lot."

This fits in line with what nearly everybody I know thinks. Does anyone _actually_ like open-plan offices without dedicated desks? How long until that trend passes?

<comment><USER8> When we were moved to an open office plan they were quite open about it - <USER2> one and only reason for doing so is they can squeeze more people into a smaller footprint. All the garbage about "facilitating communication" is just that, garbage.

<comment><USER9> Don't forget about open-back seating, so everybody can see what you're working on.

<comment><USER10> I don't particularly mind that people see my monitors. If you work in a decent place, no one would care if you read some news once in a while. If someone spends the day idling, then it's noticed anyway.

Once in a while I get a sensitive or confidential email, but that usually happens when the computer is connected to a big screen in a conference room full of people anyway...

I DO mind not finding anywhere to sit, not knowing where people are when I want to talk to them, not having two or three huge monitors to work on, and not having my Model-M to type on...

<comment><USER9> Wait, are you saying there are still companies attempting to hotdesk, "not finding anywhere to sit?"

<comment><USER10> Yes? A rather large telecom company for example.

I guess there are some stools and tables available somewhere, and technically you can sit on the floor I suppose, but people that are late more or less have to go home again.

They are kind of in a downwards spiral anyway, so maybe it will sort it self out.

</comment></comment></comment></comment></comment></comment><comment> nevi-me Interesting article! I've sometimes wondered if I should learn more COBOL. I'm 26, one of my gigs was to reverse-engineer loads of COBOL code whose functionality was being migrated to something modern.

There was a few of these very long scripts, written in the 70's. At the top was "Here be Dragons".

It felt cool after I figured out what interest calculations it was performing, I ended up leaving a comment "Dragon Slayer was Here" at the end of the script.

<comment><USER11> you should learn the domain knowledge there while the experts is still around, then migrate it out to ruby or .NET or whatever

<comment> nevi-me I think most if not all Mainframes run Java, so I think saner would be to learn so I can port to Java.

There are still younger guys in our banks who COBOL, but the domain expertise sits with the much older guys. It's lately common for the older guys who go on pension to come back as "independent consultants". They earn a lot of course.

<comment><USER11> eh, modern mainframe even runs Linux, so not really a problem there technically

Just find one of the experts that are willing to answer whatever question you have on the legacy COBOL code

</comment></comment></comment></comment><comment><USER12> The COBOL programmer shortage is solvable if they simply pay people enough money. COBOL programmers do not make SV salaries, far from it. Yet the code they write generally is handling more critical tasks than most of the stuff SV startups are producing (bank balances, payroll, insurance claims, etc.)

Though I found it to be stifling, I would work with mainframes again if the price was right.

<comment><USER13> It's well known that Financial companies are generally among the worst places for programmers to work. Learning COBOL just sets you up for a lot of that.

<comment><USER14> Finance is a very very broad industry. Everything from Xero to your local credit union/building society to bulge bracket investment banks fall into 'finance'. I work for a bulge bracket IB, and all of the developers enjoy their work because it's what you would expect working at a technology company; maintaining an in house document store, implementing complex mathematical models and implementing distributed graph based pricing systems for real time data are all things I've had the opportunity to do here, and everyone is payed at market rate or higher for their experience.

It's a great industry to be in if you work in the right area, just like any industry.

</comment></comment></comment><comment><USER15> You know you have job security when you fuck up so badly the government steps in and you still don't get fired. Most of us can only dream of ever achieving that kind of job security.

Very nice article. Captivating read. I was interested about what kind of 'previous computer experience' your mom had. That era of computing is completely alien to me.

<comment><USER16> Also comes down to Sweden having a "how do we fix this and make sure it never happens again" culture rather than a US-style blame culture. You can get fired in Sweden if you're grossly negligent or disloyal, but not for a simple mistake that anyone could have made.

<comment><USER17> This is a very good attitude. In general, the impression I get of Nordic countries is of a more egalitarian social contract with people looking out for each other rather than the culture of personal progress at any cost that we see elsewhere.

<comment><USER18> I suspect the climate has something to do with it, over hundreds if not thousands of years.

</comment></comment></comment></comment><comment><USER19> >I can only imagine the fat paycheck a 20-year old mainframe programmer would get though, because your age in this case would be invaluable.

I have a sneaking suspicion that this is not the case. Companies that want top talent and are willing to pay for it are not shy about that fact. Seeing a legit job posting for a position that pays double your current salary is a solid incentive to apply.

I'm guessing that the subject of this interview probably makes a less-than- competitive salary. Programmer pay has handily out-paced inflation for decades now. She probably started at around $30,000/yr in 1991, if she managed to get a 3% raise every year, that would only bring her to like $63,000/yr.

Naturally, I'd love to be wrong about this. This woman has a demanding and important job and deserves a generous salary for it.

<comment><USER6> You're not wrong. COBOL programmers aren't paid well AT ALL.

<comment><USER20> I've got a uncle whose a COBOL solutions architect at a large company and he makes less than 125K. It's a decent living but he doesnt really have the ability to make a move like many of his .net based colleagues.

<comment><USER21> > It's a decent living

It's over double the average family income in the USA.

<comment><USER22> So?

Developers straight out of college can sometimes make _triple_ the average household income.

<comment><USER23> Depends on where you live. In SV, sure, but they have the highest cost of living in the world.

In Dallas, a fresh out of college CS grad can probably expect a salary in the lower 40s. And the cost of living is cheap enough you can afford a house on that.

<comment><USER24> Where in DFW can you purchase a nice home on 40k?

<comment><USER23> I'm talking about renting, not purchasing.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER25> I'd love to have heard some actual quotes from your mother; how did she get into it, what are the best/worst/strangest parts of her day, how does she think the future will look given the lack of young COBOLists, etc.

<comment><AUTHOR> You are right, I apologize, I'll ask her about how her days looks like and her best/worst/strangest parts. I'll add that directly to the post and reply to you when it's done!

EDIT: I've added some Q&A at the bottom, will add more as more questions come in.

<comment><USER26> The q and a bit was my best part, it sounded more life direct quotes.

</comment></comment></comment><comment><USER27> Hm, my mom programmed on a mainframe with the pre-Fortran "Michigan Analog Decoder", is that interview-worthy?

<comment><USER28> I found lots of analog companies Googling for it but no references to Michelin being in the business. So, yeah, I'd be interested in a link about it and/or the story.

<comment><USER27> K, turns out it was actually Michigan _Algorithm_ Decoder:

[https://en.wikipedia.org/wiki/MAD_(programming_language)](https://en.wikipedia.org/wiki/MAD_\(programming_language\))

So, not quite as old-school as the analog days :-p

<comment><USER28> I was trying to imagine how an analogue component could be anything but a sensor for a mainframe in a factory or something. A programming language called MAD makes so much more sense. :)

</comment></comment></comment></comment><comment><USER29> It's not that COBOL is particularly good or bad. It's that COBOL-era database technology is painfully-obsolete. Flat files, ISAM, and IMS are all pre-SQL, and old shops are likely to still be running some of that stuff. DB2 isn't too bad; it speaks SQL and runs on various platforms including Linux.

With the older systems, the database update logic and the business logic are in the same programs. This is not good.

<comment><USER18> _Flat files, ISAM, and IMS_

Those were all good ideas at the time.

Infact IMS was such a good idea the hipsters just reinvented about 1% of its functionality and called it MongoDB!

<comment><USER30> I was explaining NoSQL to an older coworker of mine who bemusedly humored me for 20 minutes or so before explaining she'd started her career off with Pick in the 1970s...

<comment><USER11> this?

[https://en.m.wikipedia.org/wiki/Pick_operating_system](https://en.m.wikipedia.org/wiki/Pick_operating_system)

<comment><USER30> Yes

</comment></comment></comment></comment></comment><comment><USER31> At the end of the 1980s, I was in a systems administration class for a now gone minicomputer line. There were two women in it who seemed quite mature, but must have been younger than I am now. One told a story about her introduction to computing that has stuck with me. She passed an aptitude test to become a programmer: by now I have forgotten whether she worked for a computer manufacturer or a customer the size of a utility company. Whichever it was, the company had a machine room with a glass window opening onto a busy street in Manhattan. In those days, programs mostly resided on punch cards, and data mostly resided on tapes. One would bring one's cards and I suppose tapes to the operators. They would hang the tapes, run the cards through the reader, and the tapes would commence to spin back and forth. If all went well, the program ended with the tapes spinning on back to the reel. In case of an ABEND, everything stopped, and every New Yorker on the sidewalk could say that the program didn't work. It struck me as being far more public a test than most of us my age and younger have ever had to deal with.

</comment><comment><USER32> The statements about batch processing are interesting. For example, when you transfer money to someone at a different bank, one way is for the banks to have correspondent accounts with each other, so the transfer is just within accounts at one bank. When one of the correspondent accounts is out of money, they have to do a real inter-bank transfer, probably via the fed or some larger bank.

Bottom line is that there is parked money in these accounts. The banks should have every incentive in the world to minimize this parked money (either to get access to their own money or to avoid paying interest on someone else's). So there should be pressure to make the systems as real-time as possible, not batched.

<comment> nevi-me Banks already do some of that locally and internationally.

In South Africa we have BankServ, which is a local interchange and clearing house. Transactions are still in batches, but the settlement between banks happens nett between the local banks.

Internationally, MasterCard, VISA, Amex etc. manage the clearing. Banks then have to pay within 2 days via Swift to either other banks or their designated international settling bank.

Going back to your suggestion, it could work because the banks could just pay each other when they run out of funds. The challenge becomes fraud and other legal issues, which often mandate settling houses.

The real-time problem is a legacy hardware problem as well as the settling issue. On your last point, money is normally parked at the Central Banks, mostly as a legal requirement. From my experience, banks in healthy financial economies like mine don't keep a lot more than the minimum cash reserve with the Central Bank.

<comment><USER16> > Transactions are still in batches, but the settlement between banks happens > nett between the local banks.

Same thing in Sweden, I believe. Cleared through Bankgirot.

[https://en.wikipedia.org/wiki/Bankgirot](https://en.wikipedia.org/wiki/Bankgirot)

</comment></comment></comment><comment><USER33> First off, great post.

It's very nostalgic for me because the author is fascinated by the technology of the day. COBOL, hierarchical databases, batch processing were still a norm in IT when I got into the field. My first programming job was around 1992-93 on a Vax using SQR and PL/SQL for an accounting department. We also had to be able to read COBOL. By contrast, my friend got into the field 5 years ahead of me and was hired into an insurance company. His IT department was just switching from assembly and IMS on an IBM mainframe to the "newfangled" COBOL going against DB2.

I left IBM after a year, stayed in my first programming job for a year, and then started a 5-year stint with startups. When I hit the startup scene I realized how a new norm was developing. Small servers, C/C++, relational databases, and OLTP were the dominant paradigm in smaller organizations.

The author did a nice job describing this period in the history of computing.

</comment><comment><USER34> If you're interested in knowing how banking works between banks, Planet Money had a podcast on it few years ago. Apparently bankers met at a parking lot in New York and traded bags of paper checks back in the days.

[http://www.npr.org/sections/money/2013/10/04/229224964/episo...](http://www.npr.org/sections/money/2013/10/04/229224964/episode-489-the- invisible-plumbing-of-our-economy)

</comment><comment><USER35> Andrew Turley has given a nice COBOL talk; the title alone - "What We Can Learn from COBOL" \- is worth the price of admission:

[http://confreaks.tv/videos/goruco2014-what-we-can-learn- from...](http://confreaks.tv/videos/goruco2014-what-we-can-learn-from-cobol)

</comment><comment><USER36> Really interesting! Thank you both.

I would love to know what a 'typical day' and the general working environment are like. (A vague question, I know, but sort-of on purpose.)

</comment><comment><USER32> My mother indirectly used COBOL in the microcomputer era: she used Real World Accounting on 68000 based Radio Shack Model-16's under something called Ryan McFarland COBOL Operating System (RM/COS). Eventually this became available on Xenix and SCO UNIX on PCs.

My one and so far only COBOL program was an ASCII-graphics game of checkers for the TRS-80 model 16.

<comment><USER28> That's interesting because (a) it's first I've heard of COS and (b) I cant find jack on it. A bunch of ComputerWorld articles and a patent reference it with a description it's a custom OS for COBOL apps. A web site says it's for an abstract, C machine but nothing to confirm. BitSavers doesnt have anything that stands out to me. Checked jnder Tandy as adds indicated it was licensed to them.

I'd be interested in links to guides on this product or scanned pages that show whether OS itself was COBOL. I'll put it on Wikipedia's Timeline of Operating Systems list if I get the info. I got quite a few added there so far.

Edit: Some paydirt showing ghe company along with at least compilers absorbed by MicroFocus. Tech might still live on somehow.

[https://en.m.wikipedia.org/wiki/Digitek](https://en.m.wikipedia.org/wiki/Digitek)

Looking with Digitek, MicroFocus, and Liant still didnt get me jack. This is some phantom of a product.

<comment><USER32> Search for "rm/cos 68000"

I remember a little about it:

You could set up "partitions", which mean reserve fixed amounts of RAM for different applications.

It ran on the (8 MB I think) external hard drive for the TRS-80 model 16.

An advantage compared with Xenix is that it was smaller (used less of the hard drive).

Also it had just a single directory, but had a form of hierarchy in that you could use any number of '.'s as separators: like usr.bin.ls. The total name could be long. I think OS-360 worked something like this.

I think the last line of the screen was always the command line, and the other lines were for command (or cobol menu screen) output and worked like a pager.

You could definitely hook up serial terminals.

I found this:

[https://books.google.com/books?id=05wAGZQlo9QC&pg=PA694&lpg=...](https://books.google.com/books?id=05wAGZQlo9QC&pg=PA694&lpg=PA694&dq=rm/cos+68000&source=bl&ots=gesmgDuQyt&sig=yiG4elq9cIWxcQua_Nxgf1YUOX8&hl=en&sa=X&ved=0ahUKEwin6qbqi_bNAhWEeSYKHTOwBDAQ6AEIHzAA#v=onepage&q=rm%2Fcos%2068000&f=false)

[http://www.1000bit.it/ad/bro/altos/Altos-68000-Systems.pdf](http://www.1000bit.it/ad/bro/altos/Altos-68000-Systems.pdf)

It's the same as "COS990", but for 68K.

<comment><USER28> Thanks! That last link is helpful! It's actually quite an interesting and capable OS. Has file & software protection mechanisms. I recall UNIX having about no reliability at the time. Also, coding, compiling, and directly running without linking the apps is closer to a LISP machine than regular OS of the time.

Might give it at least a brief Wikipedia article with these links.

</comment></comment></comment></comment><comment><USER37> I think you are overgeneralizing IMS performance. "IMS is extremely old and very slow. Searching for data can take hours." Especially when you later point out that they have 10TB of DB2 data and probably more than that in IMS. Not all IMS databases are slow. Not all MySQL / Postgres, etc. databases are fast.

<comment><AUTHOR> Definitely not! I'm mostly quoting what she said, but IBM has a lot of nice tech, but you have to understand that these are very old versions of IMS and the volume of data they have does not make the case better. Hierarchical databases are not always slow, but once you want to find stuff at the very bottom of the hierarchy, it'll start take some time.

<comment><USER38> Um, what about indexes?

<comment><USER39> These older databases often allow the use of indexes as an option. Of course ISAM (Indexed Sequential Access Method) is indexed. But also it is often possible to add indexes to the networked database internal structures. An alternative is to use hash tables with overflow. In any case you must usually periodically scan through the database reorganizing the indices and/or location of records, which can take up a lot of time and usually requires taking the database offline. Here's a good read(first is PDF):

[https://www.google.com/url?q=http://advsys.net/ghs/publicati...](https://www.google.com/url?q=http://advsys.net/ghs/publications/DBReorgPAP.pdf&sa=U&ved=0ahUKEwiznqms3vTNAhVC7GMKHdpyC7cQFggUMAA&usg=AFQjCNGMpZbH9wAWbDq-6FiMhtn1fBIg4w)

or the same paper at Google Books:

[https://books.google.com/books?id=YqM3pqFVt8IC&pg=PA8&lpg=PA...](https://books.google.com/books?id=YqM3pqFVt8IC&pg=PA8&lpg=PA4&ots=ehTAPwbseo&focus=viewport&dq=CODASYL++REORG)

</comment></comment></comment></comment><comment><USER40> Not compeltely related to article but I've often heard that banks argue that COBOL is more reliable and maintainable then any "modern" language.

I've always wondered why this is the case. Is it because people are way more careful when entering code? The article points out that their IDE is connected to the Mainframe directly which makes me believe an error is much more severe. I'm more confident that the compile and unit tests can catch my mistake than I'm in my own coding skills (which is why I am not always afraid of changing tightly coupled systems). And even if worse comes to worse, i'd still be confident that some system will detect that change and roll back my commit before all goes south.

With COBOLT (in such old environments), it seems that things must not break at any point so developer are a lot more careful with what they do, which makes the code do what it is designed for very well. The problem with such an approach is of course upgradability (I would not want to touch undocumented code that has not been updated for 20 years because some government decided to replace an old "protocol".

Or maybe, COBOLT is considered more reliable (by Banks, etc) because managers just see the fact "this system has been running for 15 years" while ignoring the problems that come with it.

But I'd like to hear your opinion about it.

<comment><USER41> COBOL development had (has?) a completely different development and deployment experience. Sure, you keyed your code directly into the mainframe, but your code and your session were isolated - you, the developer, had limited privileges. Only once your code took the prescribed input and produce the prescribed output was it allowed near "production" data.

That said, the language itself was (is?) so limited in scope and ability that stability was nearly guaranteed. You READ and PRINT using tape or disk and printers, but the programmer didn't worry about what devices were available - some system admin pointed these devices at a job, and READ came from that tape and PRINT went to that printer. The setup and management of the jobs that ran your code was a very manual process. No pointers, no memory management ... nothing low-level at all. Just high-level abstraction.

Then there's hardware reliability. As an example: you can walk inside the mainframe, pull a CPU, and the thing keeps working. Replace that CPU, and the system makes use of it. None of the system's users ever know that it happened.

<comment><USER42> > Then there's hardware reliability. As an example: you can walk inside the > mainframe, pull a CPU, and the thing keeps working. Replace that CPU, and > the system makes use of it. None of the system's users ever know that it > happened.

From what I understand, until Tandem came along and started to seriously eat away at the market in the late 1970s and forced all the other manufacturers to follow, mainframes were not known for reliability.

<comment><USER28> Im still looking for data myself on reliability over time. I recall one OS for IBM mainframes bragging about its reliability because it ran six months or so without a reboot. I was think, "Wow... some good numbers there..."

</comment></comment></comment></comment><comment><USER43> I am sad to see the mainframe platform go away. There are still some things that modern platforms could learn from, my biggest favorite is probably the heavy system instrumentation that comes with it (system traces, SMF and RMF). And WLM is also neat.

I am afraid it became a victim of its own success. IBM and other SW companies got used to so huge margins so they stopped innovating enough, in order to protect the revenue.

</comment><comment><USER44> I remember a professor at Stanford for an early CS class (I'm pretty sure it was Jerry Cain) mentioning COBOL programming and drawing a graph with salary vs time for a given language that manages to be adopted widely. First when the language is rare there's a great ascent up, then as more people gain knowledge the rates drop to some minimum as the knowledge supply broadens. Then as the language shifts into a legacy context the graph ramps up again. If I remember right he claimed the ramp exceeded the initial peak in the earliest days of the language. I forget what the context was but I think it started out with a question about what languages paid the most. I think I recall him quoting a $300/hr figure for COBOL because it's so old school and the knowledgeable population so small. Then come to think of it I think he said the rates essentially drop to zero when the language is all but abandoned and transitions to modern equivalents have befallen.

<comment><USER45> It's funny what most of us consider "a lot of money". An uber driver yesterday mentioned overhearing two organ replacement surgeons talking at his gym in Palo Alto. One said, "man, the pay sucks. It's only $13,000,000 a year :-(". And the other one responded "sheesh, come work for us and we could at least get you up to $19,000,000 a year." Kind of puts things in perspective...

<comment><USER44> Yah, the free market is an amazing thing where free individuals freely agree upon rates even as astounding as those figures. The even cooler thing is that there even exist institutions that can fit that into their payroll! Maybe cooler still is that economies exist that even hold institutions such as those.

I remember going to this Bay.net talk with Juval Lowy who beat down on everyone calling devs code monkeys who got paid peanuts. Haha, ~100K range is definitely peanuts compared to a seven figure salary. What a world.

<comment><USER46> Almost nothing about healthcare in the United States resembles a functioning market, much less one worth marveling at because of all the freely available choices being made.

</comment></comment></comment></comment><comment><USER47> We have the full suite of IBM systems at work, z, i, and p, plus a multitude of other VM systems and windows servers. They have been talking about retiring the z-series for the last twenty years yet have yet to do so. The reason is probably similar to that of the banks, not that there isn't sufficient talent to do so but just because it works, works well, and damn if every silly project a portion off the z, i, or p, doesn't go wrong in some form or another.

the new thing is wrappers, extending the life of the applications and their data but allowing modern interfaces via the web and hand held devices. if anything its the best example of code reuse there is.

on another note, I don't know if we have anyone under fifty who knows COBOL well enough to do new work but a few can do maintenance just fine

</comment><comment><USER48> Very interesting article, thank you very much!

I spent about 9 months in total in a mainframe team during my training, but that team mainly took care of hooking up the mainframes to the corporate network, so I did not do any coding except for small Perl script. (I came close to having to code in REXX, but I dodged that bullet. I played around with REXX at home and did not like it very much.)

But it was very interesting to see all the examples of parallel evolution in the mainframe world, at least where the terminology was concerned. A reboot was an IPL (also used as a verb), the OS kernel was a nucleus (a term I really liked), they also had a term for network interfaces I forgot about.

A strange but interesting world, it's a shame those beasts are so expensive.

</comment><comment><USER49> Was expecting a Question and A...

<comment><AUTHOR> Yeah, sorry about that! I'll add some direct Q&A at the bottom of the post!

<comment><USER50> Yeah, please do! Here's some questions I would ask:

1\. How/when did you decide you wanted to be a programmer?

2\. What were the challenges you've faced as a female programmer that started in the 90's?

3\. You've been working for the same entity and possibly the same code base for more than 20 years. Does it ever gets old?

4\. How scary is it to write code for a bank?

I'm certain I would have 100's more questions to ask, but I'll be satisfied with at least these :-)

<comment><AUTHOR> There! I've added those to the post at the bottom, with one more quite funny story that I didn't knew about myself. Thank you all for asking questions, there are many questions that I couldn't come up with myself that I'm very happy to get answers from!

</comment></comment></comment></comment><comment><USER51> About the IMS to DB2 migration, intuitively I would try to write a translation layer between the two databases. Do you know if this was already attempted or deemed undoable (or silly) from the beginning ?

<comment><USER52> DB2 is a relational database and IMS is not.

The programs that use IMS probably do lots of things that are just _not done_ in relational databases for good reason, so if you just translate it you will either get errors in relational constraint or you will configure DB2 to stop enforcing relations, losing most of the advantages DB2 has in the first place.

</comment></comment><comment><USER53> Interesting personal perspective, thanks for posting!

It sounds like other than the very slow database, the biggest impediment in this space is the lack of new talent familiar with the environment.

<comment><USER54> Absolutely. In many cases, universities can't afford their own IBM mainframe environment to allow students to learn how to use it. Many folks, such as myself, have to have on-the-job training in order to learn COBOL and how to use the IBM mainframe itself.

<comment><USER53> Is that environment something that could be virtualized, were it not for intellectual property reasons?

Almost seems to be more of a vendor lock-in issue than an a mere 'old and scarce platform' issue, albeit that's significant also.

<comment><USER55> There's already an emulator called Hercules, but you need a license from IBM to run the operating system all of this runs on top of.

<comment><USER56> Maybe telling IBM about your interest would lead to them licence it free to emulators. They wouldn't like their platform to die because nobody knows how to use it.

<comment><USER57> you'd think that wouldn't you , instead they've sued them in the past [http://arstechnica.com/information-technology/2010/04/ibm- br...](http://arstechnica.com/information-technology/2010/04/ibm-breaks-oss- patent-promise-targets-mainframe-emulator/)

<comment><USER56> That's because they've been a bit naughty though, not because of mere corporate evil. Quoting the article you linked:

\-----8<\-----

A well-designed System Z emulator that allows users to migrate their own mainframe applications to commodity hardware would obviously pose a serious threat to IBM's mainframe business, but IBM's software licensing terms have historically prevented such a threat from materializing.

...

In many ways, the project arguably benefits IBM by encouraging interest in the mainframe platform. [...] What brought about IBM's change in perspective was an unexpected effort by the TurboHercules company to commercialize the project in some unusual ways.

TurboHercules came up with a bizarre method to circumvent the licensing restrictions and monetize the emulator. IBM allows customers to transfer the operating system license to another machine in the event that their mainframe suffers an outage. Depending on how you choose to interpret that part of the license, it could make it legally permissible to use IBM's mainframe operating system with Hercules in some cases.

Exploiting that loophole in the license, TurboHercules promotes the Hercules emulator as a "disaster recovery" solution that allows mainframe users to continue running their mainframe software on regular PC hardware when their mainframe is inoperable or experiencing technical problems. This has apparently opened up a market for commercial Hercules support with a modest number of potential customers, such as government entities that are required to have redundant failover systems for emergencies, but can't afford to buy a whole additional mainframe.

\-----8<\-----

<comment><USER58> Sure naughty or not naughty though, having IBM be actively hostile to the only cheap way to get access to a mainframe like environment is disasterous for their skill pool.

So if they don't like herculeas' efforts they should've brought out their own, or come to some kind of arrangement with them.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER59> From a technology perspective, the mainframe software described in this interview is clearly an evolutionary dead-end. It was written in an era when every bank had its own programmers and wrote its own proprietary software.

Because of the widespread use of these homegrown systems in the banking industry, however, it appears that management is not yet ready for a total break with the past. The investors that own these companies need to demand change and indicate they are willing to pay the cost.

</comment><comment><USER60> Where is the interview? I want the interview!

</comment><comment><USER61> Is it possible to get into COBOL still? I don't know anything about it, most people seem to say it is horrible to use. Would it be career suicide? Will it be totally gone in X years?

Given the chance, I'd probably jump onto the opportunity to work with it.

For reference I do iOS development now and I have ZERO experience with COBOL.

</comment><comment><USER62> I love this article. My own stepmother just retired from a 30+ year old career as a COBOL programmer. One of my first gigs was actually doing a bit of COBOL myself - it's an interesting language in that the database access is built into the language itself, rather than being a library or add-on.

</comment><comment><USER63> There are tons of (young) Mainframe programmers and experts from India. Banks are facing no such problems. And no, they don't necessarily earn fat paychecks.

And I think you have broken a security rule if your screen shot of ISFP is from her bank. This whole article seems risky for her job.

<comment><AUTHOR> That's not her ISFP, she got much nicer colors! That's an image I found of an ISFP.

</comment></comment><comment><USER64> Awesome story, thanks for sharing! It's important to take a step back and appreciate history.

<comment><USER25> I'm not sure it's fair to call it history -- it is _because_ of history (what isn't?), but what is described is the way things are _right now_ , eh?

</comment></comment><comment><USER65> Japan's Mizuho bank is notorious for struggling with the system update. Big mergers bring big office politics.

They were in news in 2002, 2011, and also right now, where they are struggling to finally merge the 3 systems of their pre-merge banks together.

</comment><comment><USER66>
      I can only imagine the fat paycheck a 20-year old
      mainframe programmer would get though, because your age
      in this case would be invaluable.

As long as you're on the topic of compensation, how much does she make right now?

<comment><USER67> Not OP; but working in an investment bank in Western Europe as we speak. The mainframe guys next to me are making around 1-2k EUR a day as contractors.

</comment></comment><comment><USER68> Thank you for writing this! This was a very fascinating read - <USER3> insightful.

</comment><comment> david-given You _can_ interface Cobol with modern technologies. See, for example, this web framework:

[http://www.coboloncogs.org/](http://www.coboloncogs.org/)

</comment><comment><USER69> This tech stack is basically the same as what most insurance companies in Australia use! IBM mainframes, IMS, DB2, Endeavour etc. there's a lot of this type of work still out there.

</comment><comment><USER70> Is there really a market for my COBOL knowledge still? It was one of my favorite languages and honestly it is what got me into SQL and other data analysis languages.

</comment><comment><USER71> Thank you for sharing this. Off topic, but I thought the article was very well written!

</comment><comment><USER72> I studied Fortran, COBOL and RPG II later doing RPG II programming on IBM S/36 computers for 4 years until I went to the PC.

</comment><comment><USER73> >There's nothing wrong with the language itself, the problem is that barely anyone knows it — at least not in the context of mainframe programming.

What data did you use to determine that barely anyone knows mainframe COBOL? There are a significant amount of companies that still use it because it just works and they're in no need to upgrade to current technologies.

<comment><USER74> I don't have any data for that point, but I get the sense that this shortage of labor is caused more by aging workforce than obscurity. My dad is an engineering manager for a large credit card processor that uses a very similar mainframe setup, and they're constantly putting enormous pressure on their offshore contracting firms to provide young COBOL programmers because they can barely find them here. It's a job that pays people enough that they can retire well if they're good with money, and they're retiring constantly. At 57, he's one of the youngest on the team.

<comment><USER57> unfortunately even assuming the outsourcer can find people who really have COBOL experience there's a world of difference between knowing COBOL and being able to maintain the 30+ years of hacks that many large companies will have built up in these systems.

Unfortunately the alternative (re-write the mainframe/COBOL systems in a more modern platform) is a risky and costly process....

<comment><USER75> > being able to maintain the 30+ years of hacks that many large companies will > have built up in these systems.

It's not even the hacks. It's that "how the business works" was automated into COBOL 30 or 40 years ago, everyone in the "business" who knew how and why things happened retired or got laid off, and the COBOL programmers are the only ones who remember the business rules.

</comment></comment></comment></comment><comment><USER76> I can understand why the big banks are reluctant to re-write the big systems now that they have painted themselves so far into the corner, but what I don't understand is why they seemed so disinterested in the massive advances that were made in the 1970s that would have made COBOL and IMS long obsolete already then.

<comment><USER18> Reinventing the wheel is not the same as advancing.

</comment></comment><comment><USER77> I knew someone who just retired from being a mainframe programmer (not just COBOL). She reckons she could have kept working for another 20 years if she wanted.

</comment><comment><USER78> nice to see such interview, especially it's from parent

</comment><comment><USER79> COBOL is one of the "big 3" compiled languages developed in the late 1950s; the other two FORTRAN and LISP. All three are still in use.

COBOLs fortee was you instructed the computer in English-like sentences and paragraphs.</comment>


LuaJIT 2.0 Beta released, making Lua the fastest interpreted language - <AUTHOR> http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=all&d=data&gpp=on&java=on&luajit=on&v8=on&calc=calculate&box=1

<comment><AUTHOR> Link goes to the benchmark shootout, the actual announcement it here:

<http://article.gmane.org/gmane.comp.lang.lua.general/58814>

Congrats to Mike Pall on an impressive solo achievement.

</comment><comment><USER2> I'm not sure what this link is trying to tell me. Faster than JavaScript but slower than Java is "the fastest interpreted language"?</comment>


Trial Without Error: Towards Safe RL with Human Intervention - <AUTHOR> https://owainevans.github.io/blog/hirl_blog.html <comment><USER2> Wicked!</comment>


Software Project Success – 5 Keys of Responsibility - vasya-gh https://perceptionbox.io/business/software-project-success-5-keys-of-responsibility/ <comment> vasya-gh Who is responsible for software project success? What about failure? It's necessary to divide the outcome into two parts. Your executives or senior managers commissioning the software are responsible for ensuring it meets your business requirements.</comment>


Blendle has received $3.8M from The New York Times and Axel Springer - <AUTHOR> https://medium.com/on-blendle/today-our-journalistic-startup-got-backed-by-the-new-york-times-and-axel-springer-8857c70832e4

<comment><USER2> Good for Blendle! Not the first Dutch start-up to be funded by US investors, but one of the most well known (in the Netherlands anyway).

As far as I am aware, there is no US-based Blendle yet? (An article aggregator where you pay per article). If so, they have a large market ahead of them!

<comment><USER3> On behalf of the team: Thanks! (I only recently joined, so I can't take any credit myself).

As far as we know, there's no other company in the world that offers this 'pay-per-article-service'.

Blendle has been tremendously succesful in the Netherlands, proving to us that people are still interested in high quality journalism. It's just that until Blendle came along, the way journalism is offered to readers was outdated to the point of it being ridiculous. We're happy to provide an easy alternative!

</comment></comment><comment><USER4> Grats to Blendle, but I don't really get the big deal. Isn't this just micro payments on bite sized content? Doesn't this just devolve into a reinforced bubble contest of which articles can get the most people to read them and thus editors (and data science people) know which articles/journalists to invest in? How do you publish unpopular stories in a world where only the popular get paid for?

<comment><USER3> Thanks for the grats!

Well, when you get right down to it, all internet startups essentially focused on 'reading words that were typed into such a box by someone else previously.' [0] :)

The reinforced bubble you speak of is not what we're seeing in the articles being bought, but it is something that has our attention. We love great investigative reporting and journalism. Keeping that alive and nurturing that is our raison d'être.

So if that happens and if we can, we're going to prevent that bubble.

[0] [https://plus.google.com/117114202722218150209/posts/KhMRNmgG...](https://plus.google.com/117114202722218150209/posts/KhMRNmgGniP?hl=en)

</comment></comment><comment><USER5> Great news for blendle, but their irritating widget has already turned me off a few sites that use it and I was a pretty regular visitor there.

<comment><USER6> Anything we can do to improve your experience?

<comment><USER5> Make your content stand out as different rather than trying really hard to embed it in such a way that it becomes a seamless part of the surroundings. I can see how that improves your bottom line but it seems like an anti-pattern to me.

<comment><USER3> Thanks for your feedback!

I'll put it in the feature-request-backlog, but things are kindof exploding right now, so it'll be a while before it will get any attention.</comment></comment></comment></comment>


GAN – Generative Adversary Networks (2018) - <AUTHOR> https://medium.com/@jonathan_hui/gan-whats-generative-adversarial-networks-and-its-application-f39ed278ef09 <comment><USER2> While I appreciate people trying to publish their own explanations of how various NN/Deep Learning architectures work, I think a fully coded article would be much better for the people these are aimed at.

It's like with research papers: The intuitive part isn't _that_ hard to grok (sometimes they are remarkably intuitive) - it's the translation from equations to working code.

I have one friend that's trying to learn ML, and he has a solid mathematical background, but mostly on a theoretical level (i.e not applied). He could explain perfectly well how various neural networks worked, but he really, really struggled with coding them.

In fact, I wish every researcher would put a link to their code repository

<comment><USER3> It depends on where you are in terms of your learning and background. Some people will find value in intuitive blogposts which complement research papers. Others can "grok" this as you say and will find more value in code. Yet others can "grok" the code if only they had the intuitive understanding. You can see where I am getting: everyone has different backgrounds and needs and different articles will appeal to different audiences. There is no one audience to appeal to.

On the note of code: this tends to be more subjective than the maths in a paper. There are lots of design decisions that can affect the overall structure of the code. Then there is the trade off between readability and performance. Readable code for blogs is probably not the code you would use in practice.

Personally I find it easier to establish the ground truth in the mathematics than to try and figure out the underlying truth from the code. And if you can't do it yourself, then that's where the intuitive blogposts help. After that you can choose your own libraries, frameworks, programming design decisions and teachers (including other blogposts) to help you get the code.

</comment></comment><comment><USER4> I was thinking, can we invent a more formal, common graphical representation of how neural networks work, just like how we have a graphical representation of electronic networks?

<comment><USER5> One of the major selling points of neural networks is that you can model/learn relationships that are very hard to explicitly write down. Also keep in mind that neural networks often work with floating point values to scale input values. Sure, you can write down these floating point values in a diagram, but the numbers themselves will not tell you why they have the magnitude that they have.

With that being said, there is research happening that tries to find better tools to understand the reasoning of neural networks. And sometimes we might actually see and understand a general decision pattern of a NN that we could adopt for a simpler model. But in many cases it will not be possible to come up with a simpler model because some phenomena have an intrinsic complexity.

<comment><USER6> On a related note, there's a "Interpretable Machine Learning" approach that tries to provide more insight into the "whys" of ML. (I'm not en expert and I've learnt about this quite recently)

</comment></comment></comment><comment><USER7> Nice article. I specially enjoyed your collection of applications linked at the bottom.

</comment><comment><USER8> Dear Medium developers, let me summarize my experience with Medium in one image:

[https://i.imgur.com/E0MYinp.png](https://i.imgur.com/E0MYinp.png)

To blog writers, please reconsider hosting your blog on Medium.

<comment><USER9> You may have posted personal info there.

<comment><USER8> I'm aware it contains my name and email, that's okay.

<comment><USER8> And someone took the opportunity to sign me up for a bunch of unwanted newspapers... My email hasn't been private for a long time, and so are many other people's emails. That is not an invitation to be an asshole, nor am I a 'stupid fuck'.</comment></comment></comment></comment>


How Ridiculous Is It That Apple Maps Redirect To Google Maps On The Web? - <AUTHOR> http://techcrunch.com/2012/09/22/mapsception/

<comment><USER2> Why does Apple Maps redirect to Google maps on the web? Maybe it has something to do with the fact that there is no Apple Maps website? Where do you think it should redirect?

Also, what does womp mean?

</comment><comment><USER3> Almost as ridiculous as the number of puff pieces AOL has spewed out about Apple Maps!</comment>


Agent 008 Ball - <AUTHOR> http://agent8ball.com/

<comment><USER2> Only weak point is that once you line up on a shot and pull the cue back, you can't alter the angle, you're committed to the one you chose. But then it's meant to be played for speed rather than accuracy.

</comment><comment><USER3> Was fun :) I found a bug. You have a feature that gives you the possibility to do the next shot while your white ball is still moving. Well, if it is the case (you see the stick and can shot) but during that moment the white ball gets in a hole, the game doesn't give it back to you like normally. Sorry for my english, I don't know the English expression for (loose the white ball in a hole) ^^

<comment><USER4> If the white ball goes in it's called a "scratch."

<comment><USER3> Cool thanks! I should remember it :)

</comment></comment></comment><comment><USER5> Here's the website to the company that made it:

<http://www.thinkpixellab.com>

They did it for an IE9 promotion a couple years ago. Has help up surprisingly well.

</comment><comment><USER6> Well that killed an hour of my day, but I can say that I have potted all 15 balls in one game.

</comment><comment><USER7> This is great! The balls are a little pixelated on a retina screen though.

</comment><comment><USER8> All I see is a black page (running Chromium). What am I missing?

<comment><USER9> It's a (very nice) html game. Works on Chrome, FWIW

<comment><USER8> Ah well, I shouldn't be playing at work anyway, so I guess I'll check it out when I get home. Thanks!

</comment></comment></comment><comment><USER10> Kinda fun, but too easy.</comment>


Nginx module by Google for rewriting web pages to reduce latency and bandwidth - <AUTHOR> http://ngxpagespeed.com/ <comment><USER2> Hello from the PageSpeed team.

Image compression and device-sensitive transcoding are important for mobile and emerging markets, and are somewhat challenging to deploy on your own. For larger sites we acknowledge there is some tuning & thought that must be applied, and we'd be happy to help via the discussion groups (mod-pagespeed- discuss, ngx-pagespeed-discuss). We are also working on a new release that will make this easier. For smaller sites, things should work out of the box. As with all PageSpeed issues, please let us know if they don't, or you think there's something specific to improve.

We've heard user concerns about SEO, and are happy to report that this should be improved with our latest beta release (1.0), which adds 'link canonical' attributes to image response headers. This allows search engines to point back to the origin image.

Please give us a shout on our discussion group or bug-list and we'll try to help sort through any others issues that you see: [https://github.com/pagespeed/ngx_pagespeed/issues?q=is%3Aope...](https://github.com/pagespeed/ngx_pagespeed/issues?q=is%3Aopen)

<comment><USER3> its really tricky for people to figure out how to build this and deploy it production.

I have a nginx build script with pagespeed that I use for my docker images. I dont think I leverage pagespeed enough - it would be great if you could point out what needs to change (and others could learn from it)

[https://gist.github.com/redcarpetup/e52cbddb907effe70bd0](https://gist.github.com/redcarpetup/e52cbddb907effe70bd0)

EDIT: in fact that the first bug I see on your bug list is [https://github.com/pagespeed/ngx_pagespeed/issues/1106](https://github.com/pagespeed/ngx_pagespeed/issues/1106) "Nginx cannot find module when building inside Docker container"

<comment><USER4> Better yet, an nginx .deb repo with pagespeed already compiled in. Like the one nginx provides[1].

I'm very reluctant to use anything that doesn't come packaged. Compiling your own software only really works when you're constantly maintaining the same project and its stack. In every other situation, it's a hassle, and I always fall behind.

[1] [http://nginx.org/en/linux_packages.html](http://nginx.org/en/linux_packages.html)

</comment></comment></comment><comment><USER5> Seems as if people have had a few negative experiences of 'pagespeed'. Why is this?

Well, not every Pagespeed feature works with every site. However that tends to get learned the hard way. Those lazy loading images were working a treat on Friday, come monday - ah, no Apple users could see any images, consequently in an instant Pagespeed gets removed.

To take an automotive analogy, Pagespeed is a bit like doing things to a car for improved 'performance'. Let's save weight, let's put some active suspension in there, let's remove a few bits to improve the aero, let's put some 100 octane fuel in the tank and chuck in some twin turbo. All these improvements - let's keep going there are 50 more tweaks we can do in the garage, why not try 38 of them at once? What could possibly go wrong?

So, the main problem with Pagespeed is not how it works it is just that it encourages people to keep going with it, layering on more 'improvements' until the website does the computing equivalent of leaving the road to spontaneously burst into flames.

<comment><USER6> That sums up my experience pretty well. I tried installing it for a couple of sites and seemed really cool at first. But just weird things started to happen with styles and such and it seemed like it was going to take a lot of research and debugging to get everything dialed in. Not really having time to do that, I disabled it and put it on the list for later. I wouldn't call it a "negative" experience necessarily, just not the easy win that I was hoping for.

I do think it would be awesome for some legacy code if it was critical to get the performance tuned up and you have a day or so to get everything configured.

</comment></comment><comment><USER7> > Recognizes common JavaScript libraries and redirects each to a canonical > URL.

> Inserts Google Analytics javascript snippet.

While these are useful (and what appears to be optional) features, it leaves a bad taste in my mouth to think that a business reason for Google to support this library could be to lower the barrier of entry for developers to allow Google to track their users.

<comment><USER8> I would argue that adding a Google Analytics JS snippet is the easiest thing possible for an average user while adding an Nginx plugin is infinitely more difficult.

<comment><USER9> In most cases yes. I have a feeling there's a customer somewhere running an old CMS or has a big jungle of HTML documents but not the resources to modify them all accordingly.

<comment><AUTHOR> Yes, this is one the main utilities of PageSpeed - you can optimize and tweak content on pages on which you don't have control over.

</comment></comment></comment></comment><comment><USER10> We ran this in production from its initial release over two years ago. Last month, we moved to a new datacenter and decided to forgo ngxpagespeed entirely.

The results have long been mixed - it does a great job minifying content and reducing requests, but the net results are highly questionable. While we reduced our network transfers and round-trips, the amount of time it took to serve requests was significantly increased to a point where the net gains were minimal. We played around with enabling and disabling the various filters and changing the settings and configurations to a great extent, and filed quite a few issues with the (ngx)pagespeed teams as we came across them.

If you do use this module, don't enable image recompression. It'll use up a lot of CPU, but worse, it'll also add significant latency to your requests. The caches on disk will always be full, and your disk IO will increase across the board. My personal favorite filter was the CSS merger which pulls various CSS resources into a single file. My favorite feature that always let me down was the "prioritize critical CSS" filter which (in theory) inlines CSS needed for above-the-fold content to display immediately. It never worked reliably for us, and broke our CSS more times than I can count. We always went back every few months to try it and always came away disappointed.

ngxpagespeed does not play well with caching (it must be enabled on the outer- most node) as it relies on realtime metrics collected via JS from served pages to make decisions about content, caching, inlining, and more. It also doesn't work well for SEO without significant manual tuning as it rewrites content URLs (I hear there is a way around this on Apache, but not on nginx) to pass back information to nginx, appending .pagespeed.xxxxx to your non-HTML URLs, including images that will be picked up by Google's bots. If you disable pagespeed later, all these will be broken. There's also no guarantee that these links will remain valid across pagespeed versions or if you enable/disable certain ngxpagespeed modules/filters, though I can't say I've been bitten by that (solution: you need to hack your nginx configuration to always include a Link...canonical header in your outgoing requests).

There are open issues galore regarding HTTPS and ngxpagespeed - it's an ongoing battle to make it play nicely with HTTPS upstreams for caching. It also basically proxies all requests made to your site back to your site once more so it can cache and optimize them, adding an entire roundtrip to initial or uncached requests. By default it won't cache HTML, so all HTML pages get proxied an extra time. There's an option to disable this for static resources by loading them from disk instead, but good luck getting that to play nicely with multiple upstreams and overlapping paths taking advantage of rewrite rules.

In the end, it just wasn't worth it for us. We've configured our webapp to pull in, minify, and inline its own (entire) CSS on first page load (about 20 lines of code w/ the Yahoo minify library doing the crunching), then configured nginx to reduce latency (http2, ssl_buffer_size, ssl session caching, smart keepalives, gzip, ssl_cipher optimizations (hint: bigger isn't better), proper expire headers, MTU optimization, etc) and the results have been better than what ngxpagespeed was giving us.

Sorry for how unstructured this comment is, I've been remembering stuff as I go along. By all means, ngxpagespeed is an awesome effort. As a primarily non- web developer, I think this is what browsers and http servers should do by default: act as a compiler of sorts for your content and find and implement optimizations where possible because your job isn't to figure out how to minimize network requests but rather to write code that works. However, it's the early days of "web compilers" and the intelligence and logic is far from being any good just yet. Just like 30 years ago no compiler could come close to even an average developer's hand-crafted (not even tuned!) assembly, that's how I see the state of "optimizing" proxies today. Give it another decade or two, I suppose :)

<comment><USER11> Thanks a lot for this comment!

For me, this really underline how an already optimised website won't benefit much from it (especially the image recompression bit), and it may even be harmful.

On the other hand, it could work great by being properly configured for small websites that don't get much optimisations, and could really benefit from generic optimisation.

<comment><AUTHOR> You still could benefit, because the optimizations are user agent-specific, and if you aggressively use caching with Nginx, then you don't need to make as many live calls to the backend to segment the cache per user agent type. PageSpeed is written in C and highly optimized so it has performance benefits in your case as well, I think. For example, there are user agents for which PageSpeed may decide to deliver your CSS as inline and small images - as data URIs. It also automatically (but optionally) does lazy loading of images, etc. It has beacons that are also implemented with it. It's perfect for CMS websites like it is in my usecase. It's not a panacea though.

</comment></comment></comment><comment><USER12> Why is this so much upvoted? This is old news. IMHO ngxpagespeed is a good option if you have a lot of old legacy stuff which cannot be converted for a good web compression nowadays.

But compiling your own nginx is not trivial and I would not recommend it for everyone.

It would be great if there would be a great precompiled nginx with newest security fixes for my Ubuntu which can enable ngxpagespeed or the new brotli compression: This is missing in the open source world.

<comment><USER13>
       compiling your own nginx is not trivial and I would
       not recommend it for everyone

Yes, this is frustrating as an nginx module developer as well. Apache has a loadable module system and we can ship binaries, but to add a module to nginx you have to recompile.

Nginx's developers have been hinting that they're going to add support for loading .so modules, and I really hope they do!

<comment><USER14> tengine (a fork of nginx) fixes this problem.

<comment><AUTHOR> But is, unfortunately, a few generations behind.

</comment></comment></comment></comment><comment><USER15> Cool to see this featured. If anyone's interested in actually playing around with ngx_pagespeed, I wrote a pretty straightforward tutorial on my blog to get up and running (granted, I last updated it in 2013, so some things may be outdated).

[https://dlo.me/archives/2013/05/14/nginx-pagespeed- cloudfron...](https://dlo.me/archives/2013/05/14/nginx-pagespeed-cloudfront/)

</comment><comment><USER16> This made me chuckle. I've been doing this type of stream editing for years, on the client side. Of course I strip out Javascript. In addition to blocking google-analytics, doubleclick, etc. client-side, via DNS. Google reminds me of Goldman Sachs: trying to take both sides of a bet. But Google can never deliver the best experience for the user, because the best experience does not include redirects to set cookies, tracking, ads and other nonsense that serves no benefit to the user. Either Google is for advertisers or it is for users. Users do not pay Google. Draw your own conclusions.

<comment><USER17> What made you chuckle?

ngxpagespeed and its apache brother mod_pagespeed are absolutely incredible tools for improving performances of any website. It automates lots of work that used to have to be coded manually in your application, and moves the responsibility to the webserver. It, in the end, allows you with a dead simple configuration file, to minify, shard, optimize, and make many other performance optimization, without you ever worrying about implementing those yourself.

> Either Google is for advertisers or it is for users.

What does that have to do with ngxpagespeed?

<comment><USER18> I think the OP is confusing strategic and tactical decisions by Google. On a strategic board-level view, yes Google is an ad company and reducing web page size without stripping ads is a bit disingenuous. Strategically Google will never drop ads, so the next best step for this project probably won't happen

However tactically Google benefits from paying clever people to make great products irrespective of their strategic fit - and this seems a good product.

I had never heard of it, and will build it in my pipeline RSN.

<comment><USER19> > Strategically Google will never drop ads, so the next best step for this > project

I'm curious what this step would be?

NGINX pagespeed is a server side module that you run before you send data to the end-user.

Many people who own servers want to extract revenue from their end-users, and yes, that means ads.

If pagespeed stripped ads from a person's website, I think they'd be very unappreciative of it.

</comment></comment></comment></comment><comment><USER20> Pagespeed is absolutely one of the worst plugins in the world. mod_pagespeed for apache is garbage, and permanently breaks some websites when removed. I see no reason why the nginx version would be any better.

<comment><USER13> I'm sorry it didn't work out! Permanently breaking a website when removed is very surprising, since if you uninstall the module it's not there anymore to have an effect. Would you be up for explaining more?

(I work on mod_pagespeed, and if we're breaking sites on removal that's something I want to look into.)

<comment><USER17> This kind of answer from Jeff (and other people) from the MPS team is in part what made me enjoy using MPS. Should you have _any_ issue with MPS, if you send an email to the mailing list, you will always receive a kind answer, even in those cases were it is obvious you didn't read the doc.

I "worked" for _months_ with Jeff and his team to find a real tricky bug to catch (well, they mostly did the work, I sent the bug reports) and they went above and beyond to find and fix that bug. (Remember the #1048 issue?)

I definitely recommend MPS, it will not "break your website".

<comment><AUTHOR> I agree, Jeff and Otto have an amazing ethic and truly care for the project. I wish more project are lucky to have such leadership. Otto is no even with Google but has spent endless hours trying to help me and never tried to pitch consulting services although it seems that this is his business.

</comment></comment></comment></comment><comment><USER21> Does image spriting and js/css concatenation still make any sense if you're already using HTTP/2 in Nginx? The requests should already be multiplexed.

</comment><comment><USER22> Are there any precompiled packages for Nginx for this? Looks like an awesome project, but I'm not a huge fan of compiling from source because it makes maintainability difficult in future. Having to compile Nginx from source just so I can use PageSpeed isn't something I look forward to.

<comment><USER23> just script the compilation so it's easier over time. That's what i do for my LEMP stack's installer and source compiled nginx. There's a shell based menu option i can run to just recompile nginx for upgrade, downloads or recompiles [http://centminmod.com/menu.html](http://centminmod.com/menu.html) :)

automation is the key ^_^

</comment></comment><comment><USER24> Ahh, Kloudsec [0] uses a fork of this in our free CDN service for our users. The issue with something like this is that

a) most website owners who want to get things done will find it hard to recompile, optimize, and deploy in production.

b) most websites in the web are really just WP sites hosted on X hosting provider

And yet we truly believe that sth like Pagespeed is a worthwhile investment (if done by 1, deployed on many). So we built Kloudsec to make optimizations like this available for free in our CDN layer (which is also free).

\---

Most importantly, it is super easy to deploy. No programming/dev-ops required. Register an account, add your domain, then update DNS records. (You get to keep your fav nameservers)

[0]: [https://kloudsec.com](https://kloudsec.com)

</comment><comment><USER17> I've been a heavy user of mod_pagespeed (the Apache version of Google's plugin) for about 5 years now. It's been an absolutely incredible tool.

I am now, however, wondering what the future of MPS is with the advent of HTTP/2? A lot of the work MPS is doing, and was designed for, is optimizing for HTTP/1.1 shortcomings, and most of MPS's work seems irrelevant in the HTTP2 world, and even detrimental in some cases (take URL sharding for example).

<comment><AUTHOR> It still can add tremendous value. For example, I do all my CDN offloading with it, domain sharding, making all paths relative, etc. It has a growing number of filters. Combining assets is just one of its most basic feature. It has a lot. Minification is still gonna be useful with HTTP/2.

</comment></comment><comment><USER23> Also don't forgot about ngx_pagespeed's admin console lots of useful stats there too [https://developers.google.com/speed/pagespeed/module/console](https://developers.google.com/speed/pagespeed/module/console) & [https://developers.google.com/speed/pagespeed/module/admin](https://developers.google.com/speed/pagespeed/module/admin)

screenshot examples [https://community.centminmod.com/threads/admin-stats- console...](https://community.centminmod.com/threads/admin-stats-console- revamp-in-ngx_pagespeed-1-9-32-1-beta.1669/)

</comment><comment><USER25> Beware if you're using this BEHIND a separate caching layer such as Varnish. That's not well-supported (and helped contribute to an outage for us. we used auto-scaling groups and that shot us in the foot when first trying out pagespeed).

In itself this module is pretty cool, but you're actually better off simply internalizing the various practices within your own code & infrastructure, with a CDN in front.

None of this is to knock the Pagespeed plugin. It IS really cool, and in an organization with many silos (like Google) it has many benefits.

<comment><AUTHOR> Actually, it is supported and this is how I've been using it. It's documented here [0].

[0]: [https://developers.google.com/speed/pagespeed/module/downstr...](https://developers.google.com/speed/pagespeed/module/downstream- caching)

<comment><USER25> "Note: this feature is currently experimental" and that's a true statement.

<comment><AUTHOR> The whole thing is a beta. I've been using Downstream Caching in production for years.

</comment></comment></comment></comment><comment><USER26> Reduces latency and greatly increases the disk writing and reading

</comment><comment><USER27> We used this a lot when we had a old project to maintain and it's great! Today, having a new version where everything is optimized before we turned it off.

</comment><comment><USER28> Does anyone know if that works with sites behind a nginx proxy? I have Apache and Nginx and if that works for proxied content.

<comment><USER29> yes, it does :)

<comment><USER28> Thanks! So it works on the post-cached content? If I choose the minification or CSS unification.. Or is that on the fly?

Anyone familiar with that?

</comment></comment></comment><comment><USER30> Is there any standalone program that use PSOL library to post process static page? I only read about modules for http servers.

</comment><comment><USER31> Wow. Is this compatible with AWS, Load Balancers, etc.?

</comment><comment><USER32> Ugg... Apache httpd module been around for awhile, no news.

Nginx module re-announced, and big news.

Holy crap... is everything Nginx-related somehow important and note worth and everything Apache httpd related delegated to the trash bin?

Lest anyone forget: Nginx doesn't even have _loadable modules_ so this requires rebuilding Nginx

</comment><comment><USER33> Anything that impacts the readability of the markup sent to the client is an atrocity - not that Google would care about this since they are notorious abusers of js obfuscation. Don't be evil indeed.

<comment><USER13> If you're visiting a website that runs PageSpeed and you want to see the original markup, add ?PageSpeed=off to the url.

For example, compare [http://www.jefftk.com/index](http://www.jefftk.com/index) to [http://www.jefftk.com/index?PageSpeed=off](http://www.jefftk.com/index?PageSpeed=off)</comment></comment>


Why Is Storage on Kubernetes So Hard? - <AUTHOR> https://softwareengineeringdaily.com/2019/01/11/why-is-storage-on-kubernetes-is-so-hard/ <comment><USER2> I don't really think this is a Kubernetes-specific problem. If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work. But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 99.9%-ile latency) and that that last 0.1% is always the most expensive. Sometimes you need it, and I get that, but it's not a problem that everyone has.

Most applications are not so IOPS limited that they depend on the difference between PCIE request latency and going out over the network to get stuff that's actually stored on a nearby rack. And in that case what Kubernetes offers (with the help of cloud providers) is fine. You make a storage class. You make a persistent volume claim. Your pod mounts that. Not all that hard. If the performance isn't good enough, though, then you have to build something yourself.

I am used to a completely different model that we had at Google. You could not get durable storage in your job allocation. Everything went through some other controller that did not give you a block device or even POSIX semantics, and you designed your app around that. If you needed more IOPS you talked to more backends and duplicated more data.

Meanwhile in the public cloud world, you get to have a physical block device with an ext4 filesystem that can magically appear in any of your 5 availability zones, provisioned on the type of disk you specify with a guaranteed number of IOPS. It's honestly pretty good for 90% or even 99% of the things people are using disks for. (In my last production environment I actually ran stuff like InfluxDB against EFS, the fully-managed POSIX filesystem that Amazon provides. It did fine.)

<comment><USER3> For databases, local storage beats SAN storage massively. I can get better performance and IOPS from an Intel NUC with a decent PCIe SSD than I can get from an AWS RDS instance that costs as much per month as the NUC did to buy. If I optimize a proper rack mount server for database I can get an insane amount and storage and performance compared to even a few months of RDS. Even a pair of 40 Gbps SAN links would not compare.

<comment><USER4> I'm not sure this is necessarily true. 2 x 40Gbps is bandwidth which is typically not the limiting factor. If it was you can go with higher bandwidth like FC. RDS is a database service, not SAN. Look at SAN boxes from storage companies, you can get things like 60 SSDs in a single box. To match the IOPs of that you'd need a lot of servers with local storage. I think in general dis-aggregating compute and storage is the optimal approach, whether some particular solution is better/price effective is a different question. Having all your drives in a box means they're easier to share, easier to service, easier to replace the servers as well ... lots of wins there.

<comment><USER5> If you're doing it locally, you'd ideally use RAID with a battery backed unit and your fsync essentially end up going to memory.

<comment><USER6> Until a battery relearn takes your app down in the middle of the night. I don't miss those.

</comment></comment></comment></comment></comment><comment><USER7> Kubernetes has a few good storage solutions that are built on well-known technology...

[https://rook.io](https://rook.io) (built on ceph)

[https://www.openebs.io](https://www.openebs.io) (built on Jiva/cStor)

The _distributed storage_ problem is difficult on it's own, this isn't a kubernetes specific issue. IMO Kubernetes is improving on the state of the art by dealing with both the distributed storage problem _and_ dynamic provisioning, and that's why it might seem somewhat flustered.

What isn't shown here is how effortless it feels when you _do_ have a solution like rook or openEBS in place -- we've never seen ergonomics like this before for deploying applications. Also, the CSI (Container Storage Interface) that they're building and refining as they go will be extremely valuable to the community going forward.

BTW, if you want to do things in a static provisioning sense, support for local volumes (and hostPaths) have been around for a very long time -- just use those and handle your storage how you would have handled it before kubernetes existed.

Shameless plug I've also written about this on a fair number of occasions:

[https://vadosware.io/post/kicking-the-tires-on-openebs- for-c...](https://vadosware.io/post/kicking-the-tires-on-openebs-for-cluster- storage/)

[https://vadosware.io/post/disassembling-raid-on-hetzner- with...](https://vadosware.io/post/disassembling-raid-on-hetzner-without- rescue-mode/)

I've gone from a hostPath -> Rook (Ceph) -> hostPath (undoing RAID had issues) -> OpenEBS, and now I have easy to spin up, dynamic storage & resiliency on my tiny kubernetes cluster.

All of this being capable as someone who is not a sysadmin by trade, should not be understated. The bar is being lowered -- I learned enough about ceph to be dangerous, enough about openebs to be dangerous, and got resiliency and ease of use/integration from kubernetes.

<comment><USER8> OpenEBS looked interesting, but it turns out they're spammers. :(

Starred the GitHub repo... and a few minutes later received email spam from them to my personal email (it's in my GitHub profile). :(

Completely lost interest in their project at that point.

Probably best to skip it, as rewarding spammers doesn't lead to good things. :(

<comment><USER7> I had no idea they did that, just made an issue about it[0].

Also, I wouldn't be so quick to write them off -- their solution is based on Container Attached Storage (CAS), and is the only relatively mature solution so far I've seen (I haven't seen any others that do CAS) that sort of take the Ceph model and turn it inside out -- pods talk to volumes over iSCSI via "controller" pods, and writes are replicated amongst these controller pods (controller pods have anti-affinity to ensure they end up on separate machines).

I have yet to do any performance testing on ceph vs openebs but I can tell you it was easier to wrap my head around than Ceph (though of course ceph is a pretty robust system), and way easier to debug/trace through the system.

[0]: [https://github.com/openebs/openebs/issues/2345](https://github.com/openebs/openebs/issues/2345)

<comment><USER8> Thanks for doing that. I have personal strong aversion to spammers. To the point where their product doesn't matter, I'll just use/help a competitor instead.

Reaching out to the occasional person who stars a project, if there's some strong overlap of stuff then maybe sure. An automatic spam approach though... that's not on. That makes starring projects a "dangerous" thing for end users, as they'd have to be open to emails for every one. :/

And yeah, it did look useful up until this point. Lets see what their response to the GH issue is like. :)

<comment><USER9> Thank you for the feedback. Feedback like this will help us understand the best practices in community building. We have disabled the email trigger on starring. Thanks again for this open feedback.

<comment><USER8> No worries Uma. The project looks interesting, and it's even written in Go, so I'll probably take it for a spin in the next few weeks. :)

</comment></comment></comment></comment></comment></comment><comment><USER10> Here's the root of the problem.

> Static provisioning also goes against the mindset of Kubernetes

Then the mindset of Kubernetes is wrong. Or at least incomplete. Persistent data is an essential part of computing. In some ways it's the most important part. You could swap out every compute element in your system and be running exactly the way you were very quickly. Now try it with your storage elements. Oops, screwed. The data is the identity of your system.

The problem is that storage is not trivially relocatable like compute is, and yet every single orchestration system I've seen seems to assume otherwise. The people who write them develop models to represent the easy case, then come back to the harder one as an afterthought. A car that's really a boat with wheels bolted on isn't going to have great handling, but that's pretty much where we are with storage in Kubernetes.

<comment><USER11> AFAICT, the problem is have a separate, generic, distributed storage layer that is also performant is a solution very few have (like Google), and the rest of us have tried to hack things together with StatefulSets, GlusterFS, NFS, Persistent Volumes, etc.

The Borg/Omega model kind of assumes you have a Google-like storage tier interconnected with 10GbE links that is automatically replicated and accessible from potentially anywhere. Once you have that, then storage on k8s is "easy".

<comment><USER12> Unless I totally misread the "colossus" papers, Google does not in fact have an "accessible from anywhere" storage layer. GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications.

Presumably that means that on every Google machine there is a directory that is shared for the entire cluster BUT:

1) It is per-cluster, not global (given Google's cluster sizes one imagine's that's not that much of a limitation, but then again, it seems like that is a clear limit)

2) you can only create or append to files (or delete them I guess) (and this is therefore non-posix, and does not support things like mysql or postgres)

3) this is very different from what GlusterFS, NFS, persistent volumes, etc provide. Therefore disks on google cloud are presumably very much not just files on this GFS/colossus thing.

4b) it was single master at least until 2004. Maybe until 2010. Apparently that can work.

[https://cloud.google.com/files/storage_architecture_and_chal...](https://cloud.google.com/files/storage_architecture_and_challenges.pdf)

[https://static.googleusercontent.com/media/research.google.c...](https://static.googleusercontent.com/media/research.google.com/en/us/archive/gfs- sosp2003.pdf)

<comment><USER13> > GFS/Colossus is a per-cluster filesystem capable of read and append only. It > is very much NOT generic and only supports custom applications.

For the most part, google only has custom applications. Just about everything is written in-house, and takes advantage of things like being able to open a file from the local disk file just as easily as opening one in Colossus or their equivalent of Zookeeper.

</comment></comment></comment></comment><comment><USER14>
      Why is it so hard to hammer
      a nail with a screwdriver?

Because stateful storage isn't the problem the system was developed to solve. The author is conflating what K8s is (a stateless container orchestrator) with what he wants it to be (a full service devops guy).

Now, if you simply must have stateful storage, I've had a pretty good time with PVCs pointing to Plain Jane NFS volumes combined with node tainting and performing local replication of high need data on the pods as needed.

If your pods are scaling up and down or dying so quickly that this seems untenable, you have other fish to fry.

</comment><comment><USER15> I think the premise of this article is misleading. It's true that creating a storage subsystem in Kubernetes is exceedingly difficult, however using Kubernetes for stateful applications isn't any more difficult than using instances with persistent disk in any cloud provider.

For applications deployed in cloud infrastructure, most folks are using persistent disk simply for the ease of management. Some folks end up going to local disk for scale and performance reasons, at which point they end up having exactly the same problem one would have trying to do the same with Kubernetes.

</comment><comment><USER16> Microsoft had the same problems when launching their microservice platform (Service Fabric). They relialized most of their problems went away if they gave their stateless service the notion of state and so they did. Most MS services, including Azure itself, now runs of something similar to K8 but that is state full.

Service Fabric powers many Microsoft services today, including Azure SQL Database, Azure Cosmos DB, Cortana, Microsoft Power BI, Microsoft Intune, Azure Event Hubs, Azure IoT Hub, Dynamics 365, Skype for Business, and many core Azure services.

[https://blogs.msdn.microsoft.com/azuredev/2018/08/15/service...](https://blogs.msdn.microsoft.com/azuredev/2018/08/15/service- fabric-and-kubernetes-comparison-part-1-distributed-systems-architecture/)

</comment><comment><USER17> Netapp has Kubernetes-as-a-Service that includes their own persistent volume. As a consultant I had it foisted on me for a project, but it worked pretty well - I never had to worry about storage - it just worked - so I guess I never realized it was "hard"? Definitely something I'm glad I let someone else manage.

[https://cloud.netapp.com/kubernetes- service](https://cloud.netapp.com/kubernetes-service)

</comment><comment><USER18> I could not figure out cluster storage with docker swarm at least. With containers springing up and going away on multiple computers I could not figure out where the volumes were supposed to "live"

The only thing that at least seemed possible without additional software was nfs mounts but I was thinking it does not help the high availability cause to see the one and only storage node go down. Nor could I see any benefits to a cluster that used the same network for file access as external network requests. Say I wanted to build a horizontally scalable video sharing site I don't see how I could test the real world performance of it before it went live.

<comment><USER19> You could use multiple network interfaces, one for storage, the other for the rest.

Not sure if this is in a datacenter or not. In a datacenter, you could use something like 3-PAR to have network attached storage.

But storage is hard. This is one of the advantages of using cloud providers, they have this part figured out for you. AWS's EBS volumes are network- attached storage.

Then there's another layer of abstraction when you are using containers. Ok, so you have this volume accessible from the network now, in whatever form. How do you attach it to running containers? That's what this article is about, mostly. Not with the underlying storage mechanisms.

<comment><USER18> I am Leary of Amazon because of the potential for sudden ruinous expenses. I looked over digital ocean and they say they are working on kubernetes but I don't see how block storage is going to make it work. I'm just developing this on my home server for now. I'm just trying to learn the basics. Seems like cluster computing isn't something that can be roll your own like ordinary docker can be.

Thanks for the lead though.

<comment><USER20> _> I looked over digital ocean and they say they are working on kubernetes_

FYI, Digital Ocean Kubernetes opened to everyone on December 11th.

</comment></comment></comment></comment><comment><USER21> I ended up making this: [https://github.com/gluster/piragua](https://github.com/gluster/piragua) to make connecting glusterfs to kubes easier/scalable.

</comment><comment><USER22> A few observations here

o managing state is hard. Kubernetes makes it look simple because we have been moving the state from apps to databases/messaging queues o storing data durably at scale, with speed and consistently is a CAP problem o distributed block storage is slow, complex and resource hungry o distributed storage with metadata almost always has a metadata speed problem (again CAP) o managing your own high performance storage system is bankruptingly hard

Ceph/rook is almost certainly not the answer. Ceph has been pushed with openstack, which is a horridly mess of complexity. Ceph is slow hard to mange and eats resources. if you are running k8s on real steel, then use a SAN. if you're on AWS/google use the block primitives provided.

Firstly, there is no general storage backend that is a good fit for all workloads. Some things need low latency (either direct attached, or local SAN) some can cope with single instances of small EBS volumes. Some need shared block sorage, some need shared posix.

With AWS, there is no share block storage publicly available. Mapping volumes to random containers is pretty simple. Failing that there is EFS, which despite terrible metadata performance, kicks out a boat load of data. If you app can store all its state in one large file, this might be for you.

Google has the same, although I've not tried their new EFS/NFS service.

</comment><comment><USER23> While storage generally not a simple problem, Kubernetes' design makes it even harder. Why Kubernetes went this way is not entirely clear to me. One of the reasons could be its owners' focus on cloud infrastructures and tutorial-level developer use cases, where you want to show how to dynamically create and consume a file system backed by GCE persistent disks or EBS. Also getting a secure and usable Kubernetes setup running is currently a lucrative business model.

But as the OP writes, the standard case it should support is secure access to existing storage, and Kubernetes fails there on all levels. Simple things should be simple, hard things should be possible.

In its API design, it hides storage under abstractions like PVC, PV, StorageClass that most users have not seen before in their career (where do they come from btw?) and which do not re-use general system architecture concepts. Worse, these abstractions do not have exact definitions as their semantics can be massaged with access control rules.

Consider the most simple case: mounting an existing file systems and authenticate with a Kubernetes secret. You should be able to do this in 3-4 lines in a pod defnition. Instead you need to create several yaml files, whose content is strongly dependent on how your Kubernetes was set up, so no general tutorial will help you (but others are, see above).

And this is just for getting the basics going. Secure access / user authentication is still unsolved after two years ([https://www.quobyte.com/blog/2017/03/17/the-state-of- secure-...](https://www.quobyte.com/blog/2017/03/17/the-state-of-secure- storage-access-in-container-infrastructures/)), and does not seem to be high on the agenda neither in Kubernetes nor CSI.

There are other basics missing (like file systems do not have necessarily a "size"), but let's not go into detail there.

</comment><comment><USER24> Storage itself is a complicated problem domain, one that is easily dismissed until you actually have to deal with it in gory detail.

K8s grew organically here, and thus the seams show. It's getting better every release, particularly being able able to dynamically provision and schedule storage to pods in a "zone aware" manner, which has been tricky.

The deeper issue issue is we are spoiled for choice on storage engines. Using Ceph for random access r/w low latency block storage I wouldn't wish on my worst enemy, for example. But it's hard to distinguish hard numbers for comparative purposes.

</comment><comment><USER25> The old fashioned approach of using a database server in a co-hosted setup with a locally hosted storage on SSD with another couple of servers in active or standby type of replication is the best option for the db side.

The rest of the stateless compute cores can run on kubernetes.

</comment><comment><USER26> There are solutions like Arrikto's Rok [1] that solve the problem of handling stateful workloads in Kubernetes.

[1] www.arrikto.com

</comment><comment><USER27> Kind of feels like the author doesn't know kubernetes? Stateful sets make all this very easy. I only have a user-end knowledge of k8s from GCE and it seems like k8s makes storage very easy.

EDIT K8s makes storage very easy. No "seems."

<comment><USER28> > k8s from GCE and it seems like k8s makes storage very easy.

or, using storage in an environment where the real storage is completely managed for you makes storage seem easy..

</comment></comment><comment><USER29> I saw many comments about stateful workloads. I am not sure it is a necessary issue for cloud environment.

Within a zone or a cluster, the latency is about 1ms, which is faster than most hard disks. The network bandwidth is on par with disk throughput. What we really need is a faster database and a faster object storage that can match the network performance (1ms and 10Gbps), then all workloads can be stateless.

If one uses a VM on GCP, the VM has no local storage besides the small local SSDs. Practically even the VM is stateless besides some cache.

<comment><USER30> From a database perspective, 1ms to disk is an eternity.

A good disk subsystem had less write latency than that in the early 90's.

<comment><USER31> I don't think that's true. I recall ~5ms seek times being top of the line.

<comment><USER28> he said 'subsystem' not 'disk'-

what was the latency to the controller with ram cache?

using seek time as a measure is also somewhat worst case - controllers/filesystems also queue(d) according to drive geometry.</comment></comment></comment></comment>