Your live coding demo is boring me (2015) - <AUTHOR> https://inconshreveable.com/11-13-2015/your-live-coding-demo-is-boring/ <comment><USER2> There are a lot of caveats, good live coding demos seem share some of the same properties:

\- the presentation is focused on the experience around writing the code. Debugging, tooling, etc. No, I don't want to see you fumble around coding. Yes, I do want to see you interact with your tools and libraries and help me learn something in the process.

\- the presentation is polished and the presenter knows every single line by heart (and have complete files they can just cp into to place if needed).

\- concepts of what we are going to see are established in the presentation before they are demo'd.

\- as always, having the video backup won't hurt in case of network or other dependency failures. If your demo is busted and you can't recover in 30 seconds, bail. Your audience gets it and will thank you.

<comment><USER3> I feel like the best way to do a live coding demo is to compromise on either the live or the coding part. For example, by preparing the code ahead of time, or playing a prerecorded video of instead. This convinces me that live coding is an inherently flawed concept.

<comment><USER4> The only live coding I have seen that was interesting was Notch working on a new game a few years ago. It was interesting because "debugging" was looking at what was rendered in the game and changing it. The experience for him was as visual as it was for us instead of inside his head.

<comment><USER5> You wouldn't happen to have a link, would you?

<comment><USER6> [https://www.youtube.com/watch?v=rhN35bGvM8c](https://www.youtube.com/watch?v=rhN35bGvM8c)

Start around the 8 min mark

</comment></comment></comment></comment></comment><comment><USER7> Someone steal my idea:

A text editor but with "record" and "stop" buttons. Click "record", write all your code, then click "stop". Now you can go back in time and play it all from the beginning, and you can also select certain stopping points as frames.

Then in your live demo, just navigate from one frame to the next, and everyone can watch the pre-recorded code being typed as if it were being done in real time.

Bam. Problem solved.

<comment><USER8> I once saw a presentation that used a Python shell. The guy typed _super_ fast and didn't make any mistakes. It was pretty impressive.

But then afterwards he told us that he had written a tool that typed out something prewritten as he mashed the keyboard.

<comment><USER9> Hi is me:

[https://www.youtube.com/watch?v=NSLvERZQSok#t=965](https://www.youtube.com/watch?v=NSLvERZQSok#t=965)

That later turned into

[http://doitlive.readthedocs.org/en/latest/](http://doitlive.readthedocs.org/en/latest/)

and

[https://www.youtube.com/watch?v=u4_UhFloISY](https://www.youtube.com/watch?v=u4_UhFloISY)

<comment><USER8> Very cool, though what I saw was someone else, probably over a decade ago, who must have had the same idea independently. Though I don't know that he released it as you did.

<comment><USER9> Ah, probably using playerpiano. Same idea, but I wanted it for bash.

<comment><USER8> Ah, yes, and in fact the presentation was from Pete Fein himself! I guess he did release it after all.

</comment></comment></comment></comment></comment></comment><comment><USER10> I'm not sure I agree with this post. The now-legendary "Build a blog in rails in 30 minutes" thing was one of those "holy crap" experiences for me. Seeing how, start-to-finish, you could have a fully functioning blog with authentication and everything was sort of magical, and I don't think having a slide showing Rails commands would have been as interesting.

<comment><USER11> _The now-legendary "Build a blog in rails in 30 minutes"_

I'm sure the Gettysburg Address was riveting when originally given. That does not mean I enjoy every speech I hear. Quite the contrary, the exception proves the rule.

</comment></comment><comment><USER12> Counterpoint, there are also some great things about live demos:

\- you can see the thinking process progressing in a certain order. I know it helps me understand it, because I can see something unfold piece by piece instead of being dumped as a big fat pile of info. But also because things have a relationship, and decisions have a cause, which a good live demo will highlight.

\- I can see the person interract with tooling. I like tooling.

\- it gives me a sense of the time and energy necessary to put the code in motion. Yes I know the person knows the damn thing by heart, but I take it in consideration. If you use slides, or pre-made scripst, I can't even make an approximations of the time it takes to get there, because I'm missing tons of steps, outputs and links.

\- it's reassuring. The thing doesn't seems so abstract anymore, as you can see it coming to life, instead of "in theory, it works like this". It makes me want to try the thing.

\- it's easy to follow. A lot of coders suck at making presentations that lead me from their A to their Z, and I have to adapt to every style, every slide, etc. But most coders on stage know how to code, and I understand their process. It's like peaking into the box to see the product instead of reading the packaging. Plus, so many speakers suck, either because they are too shy, or try to be too much. But live coding is just natural to me.

Now, I don't always live code during conferences. Sometime I use slides, sometime I use scripts, sometime I code, and often I mix and match, with hand drawing, pictures, schemas and every tool I get. I also speak, stand alone, about something unrelated to the computer screen, to maintain attention and create a more personnal interraction.

But those are hard to do for most people, not trained to speak in public, not used to explain their though process or understand how people will interpret it. It's a skill that needs a lot of practice. And not everybody will get such practice before their conf. Live coding can help with this, as it's a nice compromise.

</comment><comment><USER13> Venkat Subramaniam has a signature presentation style built around live coding interspersed with live construction of his discussion topics inside his editor. The first recent example of this live coding style I found is

[https://www.youtube.com/watch?v=kG2SEcl1aMM](https://www.youtube.com/watch?v=kG2SEcl1aMM)

I have never been bored in these talks!

<comment><USER14> +1 For Venkat. I haven't come across another speaker who can code and talk and teach all at the same time like he can. He even manages to be funny. I have a lot of love for this guy

</comment></comment><comment><USER15> If you're just going to show a video, why not skip the whole presentation and just make a video? The whole point of a presentation is to be able to read the room, slow down or clarify points, and to respond to feedback in real time.

</comment><comment><USER16> Sure, a live coding talk without preparation is gonna get people to check out. But honestly, if you throw up 45 minutes of slides you're gonna put an audience to sleep too.

According to audience feedback I've collected, my talks where I do a lot of live coding are engaging and fun. But that's cos I practice, and I put effort into the talk.

The key to a good presentation is to polish it and engage the audience. It's part teaching and part performance, and if you don't do both of those well, you'll lose people. Ask the audience questions to get them involved at key places. Practice so you know the timing of things.

If you just show slides or just live code, you're probably better off recording a youtube video, tweeting it out, and spend conferences enjoying the hallway track. :)

<comment><USER17> I just finished my first talk. I had not prepared half the slides necessary the day of, nor practiced anything but the intro. I thought it would take 45 minutes but I went over to 1hr30mins. Only 10 mins of that went into livecoding, which I rehearsed a dozen times.

I had one part where I intended to livecode a hello world app for threeJS. The whole code was printed in my hand, so I read it through and typed it out. 3 minutes later, it didn't compile and I could tell my audience was getting bored, so I bailed and showed the end result.

At the end of it, I was told it was one of the better talks given at my local meetups and it was more like 3 talks in one. So that was a huge compliment. I didn't catch anyone falling asleep or leaving either, so that's a good sign. I kept it engaging by asking a question every 10-15minutes to test if my audience knew anything about the topic, at the 20% 40% 60% 80% and 100% points of the talk. I tried to tailor analogies based off things my audience would know, or the responses I got during q/a.

This past weekend I went to a google cloud conference and watched a dozen presentations. Some were good and informative, others way too slow-paced and/or hard to read on screen. Quality varied greatly. None of the presenters asked questions during talk. I feel like there was a missed oppurtunity here

I personally think the 80/20 rule applies here - 80% slides, 20% livecoding + asking questions to the audience. It's boring listening to one person talk the entire time, no matter how engaging they might be. The best presenters takes feedback from the audience straight into code. Those are always fun.

As a side note, the livecoding should really happen somewhere halfway during the presentation. Ideally, everything should be recorded on video anyhow as a backup.

I think it's also important to know your environment as well. I have a natural tendency to want to walk around if possible while presenting, because standing still is boring for the audience. Sometimes, the projector is incredibly hard to read due to poor lightning as well. So it's important to consider what color your slides are going to be. Font sizes, transitions, content per slide, opacity for highlighted code sections, etc should also be considered. Pure black backgrounds aren't that great, personally a linear-gradient darkgrey is more inviting. Important transition points should use a different slide color.

Lastly, bringing tools with you the day of talk. Laser pointer, extension pole, pointing to the slides, your phone + notes linked to slides.com, hardware for recording, tripod, lapel mic, camera, backup chargers, a mouse, downloaded slide assets for offline viewing, ready to show-code on vscode, memorized shortcut keys and/or extensions to show what keys are pressed, printed notes, etc. There's a surprisingly huge checklist of items you should have available as a presenter, at least if you treat it seriously.

I wrote a post about my first talk and things I wish I did right. [https://vincentntang.com/things-wish-knew-first-tech- talk/](https://vincentntang.com/things-wish-knew-first-tech-talk/)

</comment></comment><comment><USER18> Author here. After four years, I still stand by this advice/critique. As everyone likes to point out, there are a few presenters that are an exception to the rule. That's true! I'm certainly not one of them, though.

I would also love to see better tooling made for creating presentations from code. I usually end up painstakingly custom highlighting code so that it can be stepped through line-by-line on slides which is very ungrateful and frustrating work. I also haven't done this in a while, so if folks have found or made tools that make this easier, please send me recommendations!

</comment><comment><USER19> This was the format I followed recently for an R workshop.

-Presentation rendered made from Rmarkdown shared with evertone, so they can copy paste from it and run it on the Console as I take them through the slides

\- The source Rmd used was also nice along with the entire data, which means the chunks of R code in the presentation is available within Rstudio so they just have to click the run button at each section and get the output

There was some live coding but predominantly it was the existing prepared code edit and rerunning. It worked!

Presentation for reference : [https://amrrs.github.io/r_beginners_workshop/presentation.ht...](https://amrrs.github.io/r_beginners_workshop/presentation.html#1)

</comment><comment><USER20> I feel like the biggest criticism here is that live-coding in presentations tend to be unrehearsed. That can be fixed by rehearsing it and shaking out the bugs ahead-of-time, thus avoiding the "impromptu debugging session".

<comment><USER21> Totally agree that it needs to be well-rehearsed. An alternative to being super well rehearsed is to record a video of the live coding and talk live over that, a technique I've used successfully in the past.

</comment></comment><comment><USER22> Counterexample of a non-boring "live" coding demo (OK, it is recorded and played in a faster tempo): Some of Bisqwit's videos. Some of my favorites:

OpenGL programming, simple FPS style walking scene (DOS, 256 colors, dithering, OSMesa): [https://youtu.be/vkUwT9U1GzA](https://youtu.be/vkUwT9U1GzA)

Creating a MicroBlaze emulator in C++11 (runs Linux!): [https://www.youtube.com/watch?v=1e0OCwlaEZM](https://www.youtube.com/watch?v=1e0OCwlaEZM)

Tandy 1000 Soundchip revisited: ASM programming example: [https://www.youtube.com/watch?v=gjuHxjdv3ZE](https://www.youtube.com/watch?v=gjuHxjdv3ZE)

Creating a DCPU-16 emulator in C++11 [here Bisqwit also gives oral explanations]: [https://www.youtube.com/watch?v=MvDtr3cNaLU](https://www.youtube.com/watch?v=MvDtr3cNaLU)

</comment><comment><USER23> I have watched a couple (out of hundreds) of magical coding demos, one that comes to mind was Steven Sanderson demo of Knockout.js

[https://channel9.msdn.com/Events/MIX/MIX11/FRM08](https://channel9.msdn.com/Events/MIX/MIX11/FRM08)

I just felt really old...

<comment><USER24> Imo the author isn't saying that live coding talks can't be high-quality or even 'magical' \- just pointing out that the practice is more likely to detract from a presentation than improve it.

<comment><USER23> No I guess I meant to say out of hundreds of live coding talks, a couple were very good.

Most of the time there is this tension in the audience as you see the poor person make a small typo and you wait the longest, hoping they will notice it themselves, then they try to run it and they can't figure it out and you sit there trying to explain it and they are stressed out of their mind and can't understand what you are saying because they are reading a different file in the ide than the one you are referring to and aaaaaaaahhh.... the authors pain is real.

</comment></comment></comment><comment><USER25> I once did a live coding series about learning a new language (Elm in this case) where I tried to show how to apply general knowledge of programming into learning a new language. I edited it heavily. It was still horrible :-) I quit after 5 episodes, I think. I also did a series (about 60) of Ruby TDD (which I called "One Pomodoro a Day") where I did one single pomodoro every day, unedited and recorded it with Asciinema (no audio). For me, this worked pretty well and I think it's quite watchable at 2-4x speed. However, it's still me fumbling about in my normal "Oh heck, that was a stupid idea, let's try X" way. I've shown it to a few people and it never got any traction, so I shelved it.

I'd really love to do another series, but I don't really like the idea of "And oh, BTW, here's one we prepared earlier". However, it seems to be what people want to watch (including myself!)

The other thing that I find a bit ironic is that I really _want_ to share programming with people. I love it. However, I'm really embarrassed about my programming. It's never good enough -- and that's one of the things that I think makes me a better programmer. But it's one thing to share your code with yourself or your team and laugh about the stupid things you do. It's another to show it to the world. Even now, I'm loathe to share links to my series (though Google will help you find "One Pomodoro A Day" if you really want to find it ;-) ). It's like, "And now _everybody_ will know that I'm a goof and I forget basic syntax and I look up basic library calls in the documentation and I make mistakes with TDD _all_ the time and my design is sometimes really naive and..." well, you get the point. It's terrifying! But compelling at the same time.

I've always identified with being an artist and I think programming really is a performance art. I usually don't care what I program on -- it's the programming that matters to me. The fact that I often suck is disappointing, but I'm always thinking, "What's the point of writing this if nobody will ever read it"? I wonder if other people feel the same way...

</comment><comment><USER26> It takes longer, but the best presentation of this kind that I've given was from taking video of a live coding exercise and editing it extensively (removing typos, adjusting speed). Then split it into sections and made a sequence of short clips out of them. Then put that into a presentation with one clip per slide. That way if I was a little slower when talking than planned, the video wouldn't get ahead of me, it would reach the end of the current clip and advance once I brought up the next slide.

It's a lot of work but it comes out looking effortless, and without wasting any of the audience's time.

</comment><comment><USER27> I've enjoyed a lot of emacs presentations on YouTube where they're doing it all live and talking through it. Often it teaches me something new and makes me want to use emacs more. (I still mostly use vim)

<comment><USER28> Have you given Spacemacs a try? I've been using it for about a week now and it's pretty revelatory.

</comment></comment><comment><USER29> More youtube tutorials should take this into account too. I really don't need to see someone type out the code. A walk-through of pre-written code is much more interesting.

<comment><USER30> It depends. Usually the code evolves in a live coding session and a lot of code is deleted or changed to show intermediate steps to make a point. Just seeing the finished product isn't the same thing.

I think a great deal of the actual "typing time" isn't necessary, but showing the intermediate steps usually is. I think the authors show it just as a means of avoiding the very time consuming process of editing a video. So there's some give and take. They save a considerable amount of their time that would have been an incredibly boring and time consuming process, and you loose a little bit of your time but get to watch a video that wouldn't have been made otherwise.

</comment></comment><comment><USER31> [Disclaimer - I work at Microsoft and have worked on developers events for many years] now with that said, I have seen both some amazing sessions, and some terrible ones. Live coding alone is typically not the single biggest problem, but an important element of the bigger picture. Simply put, a great presentation is certainly not easy and involves balancing many things.

If anyone cares to see my example (as-in, in my opinion) of one of the best session we did recently, skim this 1-hour Connect(); 2018 Scott Hanselman keynote and I would honestly love to hear what you think of the approach we took.

[https://www.youtube.com/watch?v=5_iE7azx7Vo](https://www.youtube.com/watch?v=5_iE7azx7Vo)

</comment><comment><USER32> As usual leave it up to Jeff Dean to be an outlier, this[0] is his talk/demo at GCP Next 2016 - I was on edge of my seat as I watched him talk and code things up at the same time. He is awesome. Oh and Gary Bernhardt's talk WAT[1] which happens to be one of my favorite talk of all time.

[0] [https://youtu.be/HgWHeT_OwHc?t=7159](https://youtu.be/HgWHeT_OwHc?t=7159)

[1] [https://www.destroyallsoftware.com/talks/wat](https://www.destroyallsoftware.com/talks/wat)

<comment><USER33> I'm as much a Jeff Dean fan as the next guy, but he doesn't live-code in the talk you linked. The article says code/demo is great, just live-coding isn't.

</comment></comment><comment><USER34> I agree except for shaders and procedural generation. It's not boring then, since you look at the output at the same time. But it can be boring too if it's too repetitive.

</comment><comment><USER35> Unless you're David Beazley.

<comment><USER36> exactly. [https://www.youtube.com/watch?v=MCs5OvhV9S4](https://www.youtube.com/watch?v=MCs5OvhV9S4)

</comment></comment><comment><USER37> I generally agree, but there are definitely exceptions. The most impressive I've seen is Haoyi Li parsing a decent chunk of Haskell using his FastParse library in ~10 minutes: [https://youtu.be/mARO- qchsKM?t=2386](https://youtu.be/mARO-qchsKM?t=2386)

</comment><comment><USER38> There is nothing worse than watching a live debugging session. On the flipside, DeepMind did a great job with their StarCraft presentation yesterday, showing pre-recorded commentary/analysis as well as select highlights. These types of talks (with animations, etc.) should definitely supplant live coding.

</comment><comment><USER39> From TFA

>> I can think faster than you can type

No, you can't. Just because the author operates at XYZ skill level, does not mean the rest of us are similarly impaired. I'm sorry he had to sit through so many boring coding presentations _that he purposefully chose to attend_.

<comment><USER40> So... you're saying that you're some god-level typist and logician who can push out multiple 500 character Python 1-liners in 2 seconds or ... something?

Most of us can read and understand code faster than we can type it out.

<comment><USER39> No I'm an ordinary professional. I don't start typing until I've thought it out, and then it's a very mediocre 100-115wpm. I'm saying the article author must be an exceptionally _slow_ typist, or prone to mistakes - not that I am anything more than average.

</comment></comment></comment><comment><USER41> I would go a step further. Please don't show me code. Show me runtimes, interesting problems, and MAYBE a line that shows "Aw gee! Thats easy!"

</comment><comment><USER42> The flaws of live coding demos apply to remote code-pairing challenges as well, which is why I hate them with a passion.

</comment><comment><USER43> The original docker demo was impressive to me in that it demonstrated how fast it was to spin up containers.

</comment><comment><USER44> I like seeing code being ran live though, even if the change is a couple of characters.

</comment><comment><USER45> There is a audience for live coding? Search at twitch?

It just has near empty overlap with devs?</comment>


The Secret Goldman Sachs Tapes - <AUTHOR> http://www.bloombergview.com/articles/2014-09-26/the-secret-goldman-sachs-tapes <comment><USER2> If the tapes only contain content describing a general culture of discouragement of rigorous oversight, it's not totally surprising or any more jaw-droppingly incriminating than previous knowledge concerning the Fed's relationship with the banks. Basically this is what we already knew.

The question isn't whether or not regulation is lax and whether or not this happened, it's how to fix it. And this poses a big problem because of the revolving door between regulatory agencies and private banks. No one wants to regulate a company that might offer them a million dollar job one day. It's kind of a tough problem to fix, since the only people who know enough about financial industry to regulate it are the people who work for the industry. And if we prevented people from working for the industry they regulate in any capacity for say, up to 10 years after they leave their government job (is this even Constitutional?), much fewer people would want to work for the government or the Fed. But even so I think that kind of reform is necessary - because it will attract to government service only people who are serious about regulation.

<comment><USER3> >>The question isn't whether or not regulation is lax and whether or not this happened, it's how to fix it.

In my opinion the only way to reliably fix this is to create a group similar to the one that the federal government created to end Al Capone's illegal activities during the Prohibition.

That group was nicknamed "The Untouchables" because they were fearless and incorruptible (by reputation, which is what's important). They were hand- picked by Eliot Ness, who reportedly went through the records of hundreds of Prohibition agents and personally identified a small group of them who were known for their honesty and integrity (as well as special skills, such as car tailing). What's remarkable about this group is that they were not only stubborn as hell but also extremely resilient to Capone's attempts at bribery and intimidation. Ness himself declined a bribe that was equal to his salary but would be paid to him monthly. They had a huge impact on bringing down the kingpin.

It's pretty clear that the regulators in charge of overseeing the financial industry today don't have this type of drive. To them, it's just a job. They are normal men and women who are up against groups of massive political power. As such, they are not in a position to do their jobs effectively. That's why the system is rife with rampant corruption.

Before you say I'm dramatizing, consider the possibility that Enron was just the tip of the iceberg, and under the surface there is ten Enrons that are propping up huge sectors of the economy with lies and deceit. Maybe ignorance is bliss and we should just let it operate like that. But truth has a habit of coming out sooner or later - and the sooner it comes out and everything comes crashing down, the sooner we can rebuild the system.

<comment><USER4> The US Constitution grants us the legal right to pursue happiness, and for some, that was consuming and/or making a living brewing alcohol. Elliot Ness and his goons were a bunch of arrogant asshats who wanted to trample on that.

They go down in the historical category of bad people who get off by forcibly inflicting their immoral, mistaken ideology on others.

Comparing this persent witch hunt to that one is, ironically, very appropriate, I think.

<comment><USER5> I'm confused. Al Capone wasn't a freedom fighter that was looking out for your freedoms during the time of an unjust law. He was a gangster that capitalized an opportunity created by poorly thought out government policies. Just because the government created the situation that made him rich, doesn't mean he gets some sort of 'free pass' on being a kingpin that killed people. Do you feel that Al Capone would have just been an honest businessman if only Prohibition didn't exist?

If people inside financial institutions are, as common practice / policy, picking the winners/losers and hiding conflicts of interest, do you think that going after such people is a 'witch hunt?' Do you feel that when regulators meet with the people that they are regulating that they should have a "he looks honest, I'll just trust whatever he says" attitude?

<comment><USER4> The point about Al Capone is irrelevant. What is relevant is all the honest businessmen that Prohibition violated.

Yes, I think it's a witch hunt. And any system with regulators "embedded" is a sick one. The proper thing to do is let the financial system self-regulate.

Of course at this point, that's like saying the proper thing for the Soviet Union to do when it fell was to suddenly become capitalist. Doing the right thing here would require an (additional and probably worse) period of sickness because we've been doing the unhealthy thing for decades.

<comment><USER2> 'The proper thing to do is let the financial system self-regulate' \- What are you kidding me? Just six years ago, a lack of enforced rules - in other words, lax regulation - and a complete lack of regulation of derivatives led to a runaway housing bubble that practically destroyed the entire economy, killed thousands of people (through suicide/inability to afford medical bills) and lost 8.2 million people their homes. You can't let something which has all of our welfares wrapped up in it hinge on the momentary greed of a few people who have every incentive to take short cuts to greater success. If this "creative destruction" is acceptable to you, then I guess we have vastly different views of acceptability.

Markets are great at solving lots of problems but they have blind spots. And one of them is the welfare of the people who are systemically connected, but not directly participating in the market. Human beings are not perfectly rational, and so neither are the markets that they make up. Until humans and markets are perfectly rational, we will need regulation to prevent gratuitous human suffering.

And if like most economic libertarians you're going to come back at with me with the absurd "It was too much regulation, not too little" argument (which I assume you will), we have very little to discuss.

<comment><USER5> I think that @javert would have a vastly different view if it was _his /her_ welfare that was 'creatively' destroyed... As such, @javert is probably well- removed from needing to deal with such issues, so it's easy for those people to be a statistic.

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> Audio: [http://www.thisamericanlife.org/radio- archives/episode/536/t...](http://www.thisamericanlife.org/radio- archives/episode/536/the-secret-recordings-of-carmen-segarra)

Transcript: [http://www.thisamericanlife.org/sites/default/files/TAL_536_...](http://www.thisamericanlife.org/sites/default/files/TAL_536_transcript.pdf)

</comment><comment><USER7> I worked at the Federal Reserve for some time, so I know that the culture is as toxic as the article says it to be. Professionally, this experience has been the most aggravating of my career.

Oh, let me tell you about the laziness! One of the developers I worked with would come to work at 11 am every single day and leave at 5 PM—sharp. He would then take a two hour lunch break.

It wouldn't be so bad if he actually did some work, but there was none. I remember vividly this one incident: we were trying hard to meet a deadline and he was responsible for an important component. Every week, he would paint a rosy picture of his progress, saying he was THIS close to finishing the project. The week of the deadline comes and it's time for him to lay down his cards—and surprise, surprise—there was no working code! All he produced was a jumbling mishmash of code that had no hope of working. I spent the next week in the office writing this component.

Close to 70 hours. Unreal.

[Oh yeah, this guy was getting paid $130K, not to mention those sweet government bennies!]

His direct manager would complain about him all the time, and many meetings were asking him to improve his output. And yet, he was never punished. The truth is, when it comes to working at the Federal Reserve, your work output is not as important as who your friends are. And this guy had friends in high(er) places.

This guy is just one employee, but I believe it is just endemic of the kind of culture the Fed breeds. One of nepotism, of indolence, of apathy. I am truly scared for the future of this country knowing that these are the people behind the wheel.

</comment><comment><USER8> There's also a print version of the story at ProPublica: [http://www.propublica.org/article/carmen-segarras-secret- rec...](http://www.propublica.org/article/carmen-segarras-secret-recordings- from-inside-new-york-fed)

</comment><comment><USER9> _The Ray Rice video for the financial sector has arrived._

Not even close. That metaphor would only be apt if there was literally video of a Federal Reserve employee accepting cash from an employee of a major bank with an explicit statement from the bank saying something along the lines of

"So you're going to keep the interest rates low so we can keep doing bond arbitrage instead of lowering rates for consumer lending right?"

Even then most people wouldn't care because they have no idea how financial institutions work.

<comment><USER10> With your hypothetical there, I think you have just described fairly accurately how I think most financial institutions work

</comment></comment><comment><USER11> There are times when I can't tell whether people are stupid or not, making comments like stopping employees of regulatory bodies seeking jobs at the companies they regulate would amount to restriction of trade or non-compete clause or whatever.

How about saying that if FBI or DEA agents were banned from working or consulting for drug cartels and crime bosses after retiring that would be restriction of their employment prospects.

Are we living in some kind of twilight zone or is that what happens when debates a left to guys on HN?

Why can't American media simply call spades spades and simply come out and rather than say that Fed employees and bosses are simply weak and spineless, the simply say that they are corrupt and lacking in integrity?

In fact the media are the biggest problem here, because instead of calling out things at the right time they weasel around the whole issue and belatedly come out with 'startling new revelations' after the horse has bolted.

Why couldn't the Bloomberg journalists who revealed this information simply ask their employer about this? He could have told them about the Fed's dubiousness 30 years ago.

</comment><comment><USER12> I wonder if there isn't a way to deal with this problem through something like the South African divestment movement of the 80's. If institutions like universities, pensions, and corporations were under pressure from their constituents to not do business from Goldman Sachs, Goldman Sachs would start feeling the pressure to not capture regulators.

It is not clear that banking advice from GS is that great. Often times their customers are effectively the victim. If there was popular pressure to not deal with them they might clean up their act.

<comment><USER13> I'll rephrase your suggestion:

Since the regulator responsible for monitoring private financial institutions has failed, we along with universities, pensions, etc should stop investing in those financial institutions.

The above strikes me as very backwards. Isn't the appropriate response to fire the heads of the regulators, rebuild the agency to act on the recommendations that an outside body proposed to the regulators, and ensure that congress follows up to ensure changes have occurred.

Oh wait. Therein lies the problem, congress has been captured by the finance sector's lobbying, so I'm being idealistic and your proposal is not as backwards as I first thought. I'm starting to understand Lawrence Lessig's focus on removing money from election campaigns, so that we can have real government policy because that's the real root cause of issues like this.

</comment></comment><comment><USER14> I will definitely listen to the broadcast. It surprises me that this kind of thing doesn't happen more often, given how easy it is to record conversations these days, and the liberality of the relevant laws in most states (one party consent).

</comment><comment><USER15> The environment of the FED sounds more like the paternal architecture of the military:

        Just the opposite: The Fed encourages its employees to
        keep their heads down, to obey their managers and to
        appease the banks. That is, bank regulators failed to do
        their jobs properly not because they lacked the tools but
        because they were discouraged from using them.

</comment><comment><USER11> The Fed can do whatever it bloody wants so long as it uses its own money to bail out the banks it is supposed to regulate, rather than taxpayers money.

So long as they and the banks they are supposed to supervise stick to that they can do whatever they bloody want.

This is where the quality of debates on HN become so sucky and surreal.

Given that more than a few of the Treasury Secretaries, SEC bosses and what nots are ex Goldman Sachs et al employees, why do any intelligent people expect the Fed and other government agencies to regulate the banks properly, if the failure of the institutions they are supposed to regulate would result in the loss of the pensions the continue to receive from such institutions?

</comment><comment><USER16> Story posted late this afternoon:

[http://www.bloomberg.com/news/2014-09-26/goldman-sachs- said-...](http://www.bloomberg.com/news/2014-09-26/goldman-sachs-said-to- prohibit-bankers-from-buying-stocks.html)

> Goldman Sachs Group Inc. (GS), the top adviser on corporate takeovers, is > changing its conflict-of-interest policy to bar investment bankers from > trading individual stocks and bonds, a person with direct knowledge of the > matter said.

</comment><comment><USER17> Here's one more secret Goldman Sachs tape)) [http://www.youtube.com/watch?v=sX7TWstSV_E](http://www.youtube.com/watch?v=sX7TWstSV_E)

</comment><comment><USER18> This stuff is fairly innocuous if that's all Lewis can come up with after reviewing 46 hours of tapes. "I'm not going to comment until I know what my boss thinks?" That's something nearly every underling has or will say at some point.

The bigger issue is that being a regulator in the U.S. is a shitty job. Half the country thinks we shouldn't even have regulators, and the other half blames regulators for not understanding what they're doing out of one side of their mouth while condemning anyone with industry experience who does know out of the other side. So no surprise that the regulators are deferential to the banks. At any given time they're one presidential election away from losing what little leverage they have.

Our regulatory system is the result of a clash of cultural values. One one hand we aren't comfortable with doing away with regulation. To the contrary, we demand it. On the other hand we think of regulatory bodies as leeching off "value-creating businesses" and condemn anyone who undertakes government service. Unsurprisingly, we get regulatory bodies that try not to rock the boat.

<comment><USER6> I guess you haven't read or listened to the whole thing?

"Goldman had advised one energy company, El Paso Corp., as it sold itself to another energy company, Kinder Morgan, in which Goldman actually owned a $4 billion stake, and a Goldman banker had a big personal investment. The incident forced the Fed to ask Goldman to see its conflict of interest policy. It turned out that Goldman had no conflict of interest policy -- but when Segarra insisted on saying as much in her report, her bosses tried to get her to change her report. Under pressure, she finally agreed to change the language in her report, but she couldn't resist telling her boss that she wouldn't be changing her mind. Shortly after that encounter, she was fired.)"

<comment><USER18> I did skim the examples. The one you quoted isn't damning either. Its not illegal for M&A professionals to have conflicts of interest. That's the rule in the business world. The exceptions (e.g. fiduciary duties of corporate officers) are provided by law.

<comment><USER19> Sure it's not illegal, but do you think it's ethical?

Do you think it's a thoughtful policy for Goldman Sachs?

I know if I were one of the M&A parties, I would irate to find out that the supposed advisor had a personal stake that could have influenced how things shook out.

<comment><USER18> Contemporary understanding is that outside of special cases, ethics is dictated by the market. If Goldman's clients have a problem with with bankers having personal stakes, they'll not use Goldman's services. Its not like they don't know. Its very common for investment professionals to get in on the deals they work on.

In any case, regulators are entrusted with making sure people follow the law, not some ethical code that differs from person to person.

<comment><USER20> > ethics is dictated by the market

Maybe we can start trading in ethics? /s

</comment></comment></comment></comment></comment></comment><comment><USER21> Here's the original story at This American Life (no slight intended to Bloomberg, as the attribution and respect is crystal clear in the article):

[http://www.thisamericanlife.org/radio- archives/episode/536/t...](http://www.thisamericanlife.org/radio- archives/episode/536/the-secret-recordings-of-carmen-segarra)

</comment><comment><USER22> Germane to the topic:

Century of Enslavement: The History of The Federal Reserve

[http://m.youtube.com/watch?v=5IJeemTQ7Vk](http://m.youtube.com/watch?v=5IJeemTQ7Vk)

<comment><USER23> Sorry, can't watch due to all the chemtrail smoke in my eyes. And once it clears my mind will be controlled by them anyway, so I won't want to watch anymore.

<comment><USER22> It sounds like your brain has already died from a steady diet of Statist pablum. Perhaps when you grow up you'll be able to masticate actual history.

</comment></comment></comment><comment><USER24> So after listening to the entire show, I'd have to say she doesn't come across as being in the right at all.

The big showdown on whether Goldman had a conflicts of interest policy came down to a difference of opinion that she was clearly in the wrong on. Her boss's request to amend her statement to say that they had one but it was totally inadequate should have been an easy one to agree to. Especially if the Fed's legal department was the one objecting to saying there was no policy.

Instead she comes across as extremely stubborn and unable to compromise or see other viewpoints besides her own. I would have fired her too just because she was so difficult to work with.

Silva does seem spineless in going out of the way to not verbally offend Goldman, but IMO that is more of a personality trait that him ignoring evidence.

Now she's suing wanting to be vindicated that she was really right all along and everyone else was wrong, and releasing the tapes to support that. Not a Goldman fan, but for this report there is really no moral outrage to be had.

<comment><USER25> >"Instead she comes across as extremely stubborn and unable to compromise or see other viewpoints besides her own. I would have fired her too just because she was so difficult to work with."

And frankly, that would be a reason for you not to be a manager of Fed regulators.

The Beim report specifically, _specifically_ , says the kind of dissent she represented is exactly what the Fed needs.

And she did offer options, that she could simply be overruled if her superiors disagreed. This is how it should work. If they disagree, they need to personally be taking responsibility for that disagreement, not forcing the employee to change their views.

Not to mention it's pretty clear Goldman had no _operative_ conflict of interest policy. They've simply been in blatant conflict too often for it to be believable there's anything limiting it.

<comment><USER24> The dissent is totally acceptable if there were degrees or shades of opinion (i.e. is the policy acceptable or not, does it meet established requirements, is it being enforced).

The issue is when she knowingly states as fact something that is not accurate (i.e. there is NO policy). That crosses the line into slander since it can cause reputational and financial harm to the institution being investigated.

Even more importantly, by one regulator giving the appearance of "gunning" for Goldman it jeopardizes the work of countless others who also have findings and research that may actually be true.</comment></comment></comment>


Pragmatic Go use at bit.ly - <AUTHOR> http://www.reddit.com/r/golang/comments/ydcfi/pragmatic_go_use_at_bitly/ <comment><AUTHOR> I'm sorry for linking to the reddit discussion, but HN autokills submissions to bit.ly URLs.</comment>


Approving a Muscular Dystrophy Drug Ignites a Civil War at the FDA - <AUTHOR> http://www.forbes.com/sites/matthewherper/2016/09/20/approving-a-muscular-dystrophy-drug-ignites-civil-war-at-the-fda/#38899d54353d <comment><USER2> Derek Lowe -- Of the fantastic "In The Pipeline" blog with Science disagrees with the FDA approval;

[http://blogs.sciencemag.org/pipeline/archives/2016/09/20/sar...](http://blogs.sciencemag.org/pipeline/archives/2016/09/20/sarepta- gets-an-approval-unfortunately)

He's an excellent read if anyone wants to keep up with things in the Pharma and regulatory world. He's probably more famous for his "Things I Won't Work With" series that I've seen linked to a few times on HN.

> And he's just getting warmed up, if that's the right phrase to use for > something that detonates things at -180C (that's -300 Fahrenheit, if you > only have a kitchen thermometer). The great majority of Streng's reactions > have surely never been run again. The paper goes on to react FOOF with > everything else you wouldn't react it with: ammonia ("vigorous", this at > 100K), water ice (explosion, natch), chlorine ("violent explosion", so he > added it more slowly the second time), red phosphorus (not good), bromine > fluoride, chlorine trifluoride (say what?), perchloryl fluoride (!), > tetrafluorohydrazine (how on Earth. . .), and on, and on. If the paper > weren't laid out in complete grammatical sentences and published in JACS, > you'd swear it was the work of a violent lunatic.

[http://blogs.sciencemag.org/pipeline/archives/2010/02/23/thi...](http://blogs.sciencemag.org/pipeline/archives/2010/02/23/things_i_wont_work_with_dioxygen_difluoride)

</comment><comment><USER3> This article has many scientific gaffes:

 _because the genetic defect that causes Duchenne is on the X chromosome, it is a disease of boys._

It is because it is an X-Linked-Recessive trait that it a disease of boys. Saying "because [it] is on the X chromosome" adds no value by itself. If it were "because [it] is on the Y chromosome", then it would sufficiently follow that it was restricted to boys (with rare exceptions).

 _The data showed an increase of only 0.28% of normal levels, which was, on average, a tripling of dystrophin levels for these boys._

I have no idea what to make of this. "Increase of only 0.28%" == "on average, a tripling"?

 _Unger was worried about the unknown risks of the drug, as well as the risks to boys who might get injection ports while they were also on immune- suppressing steroids, raising the risk of_ injection (sic).

<comment><USER4> I don't see an issue with the statements that were made.

An increase in dystrophin levels to .28% of normal levels, indicates that without the drug, the dystrophin level was at .093% of normal levels. (.093% * 3 = .28%) The drug succeeds at raising dystrophin levels, but the levels are still well below those of a healthy individual. (Which is the center-point of the whole debate)

A recessive x-linked trait will be more likely in men than women, because men have one x chromosome, and women have two. This means that a male with an x-chromosome with the Duchenne gene will always have Duchenne, whereas a woman would need a copy on each of her x chromosomes, assuming the woman isn't XY with androgen insensitivity, or XXY, etc.

A common example of an x-recessive trait would be colorblindness. Genetic colorblindness affects about 8% of men and .5% of women. Since men always have colorblindness if they carry the gene, we can use the prevalence in men as the population prevalence, assuming colorblindness doesn't play a role in sexual selection (doubtful, given the fact that fashion sense probably plays a role). .08 _.08 = .0064, which is pretty close to the prevalence of colorblindness in women.

Duchenne affects 1/3600 men. Assuming 1/3600 is the prevalence of the gene in the population, we'd expect 1/3600 _ 1/3600 = 1/12960000, or 1 in thirteen million women to have it. So we'd expect about 25 female cases in the US, and about 83000 male cases.

Edit: This is assuming having two Duchenne genes will still lead to Duchenne. If 3x normal levels of dystrophin leads to clinical relief, it's possible that having two genes allows for enough dystrophin production to prevent Duchenne. I don't know enough about the pathways involved, but it's possible.</comment></comment>


America's Monopoly Problem - <AUTHOR> http://www.theatlantic.com/magazine/archive/2016/10/americas-monopoly-problem/497549/?single_page=true <comment><USER2> Good to know... but how does this compare to similar foreign economies? Are we the exception, or is this a more general trend? Unfortunately, the Atlantic only looks inward and we're left to wonder.</comment>


Ask HN: Is Hackernews still using the file system as database? - <AUTHOR> I'd love to learn a bit about what the current hosting/tech stack setup looks like. IIRC the posts were stored in files on the file system and scaled quite well vertically. Is that still the case? <comment><USER2> While on the "how HN works" topic, I have a question:

I'm working on an HN app reader idea that I think is unique (more to come) that requires regularly getting the front page posts and comments. I wrote a script to do that through the Firebase API but holy shit it ended up needing thousands of requests to get all the data for the current state of the front page.

So instead, I wrote a scraper script to produce the same thing, just with 30 requests (1 per front page post). (Trying to turn these damn <tr>'s into a recursive comment tree was a quite the mindfuck btw)

...is it ok to scrape HN? I see nothing in the robots.txt to say it shouldn't be allowed, and I actually feel better making 30 scraping requests rather than XXXX API requests.

EDIT: Also happy to receive suggestions. I'm coding in Ruby and don't see an obvious way to access the Firebase DB directly (quite frankly I don't even know if what I just said makes sense) so any help is appreciated.

<comment><USER3> The algolia api has more bulk operations, so you can get more than one response per api request. Also if you look around, there are some datasets where people have downloaded all the HN API json responses, and have made them available, so you could use those for local development and testing.

As far as ethics go, it's probably better to use the API than scraping. Sure you're making more requests, but I think the firebase servers and cdn have far more ability to handle a lot of requests than the HN server does. If having that many requests would put firebase out in any significant way, then I'm sure they would put a query load limit on the HN api.

<comment><USER4> HN is behind Cloudflare; if you're not logged in the pages can be cached so I wouldn't expect small-scale scrapping to have any effect. Cloudflare might stop you though.

</comment></comment></comment><comment><USER5> The source code used to be open, and it isn't clear what happened to it. Maybe ask over on the arclanguage.org forum as HN is written in arc.

The closest thing to a source repo is this from many years ago: [https://github.com/wting/hackernews](https://github.com/wting/hackernews)

<comment><USER6> The official sources are at [http://arclanguage.org/install](http://arclanguage.org/install)

Community-supported version: [http://arclanguage.github.io](http://arclanguage.github.io)

<comment><USER7> [https://news.ycombinator.com/item?id=11176894](https://news.ycombinator.com/item?id=11176894)

> _A version of HN 's source code is included with the public release of Arc, > but HN's algorithm has many extensions that aren't public._

[https://news.ycombinator.com/item?id=13456306](https://news.ycombinator.com/item?id=13456306)

> _We 're unlikely to publish all that because doing so would increase two bad > things: attempts to game the site, and meta nitpicking._

Including at least at one time a Chrome extension for moderation:

[https://news.ycombinator.com/item?id=11670071#11670562](https://news.ycombinator.com/item?id=11670071#11670562)

<comment><USER8> > We're unlikely to publish all that because doing so would increase two bad > things: attempts to game the site, and meta nitpicking.

So security by obscurity helps at times.

<comment><USER9> Passwords are security through obscurity too. Things would be a lot less secure if all passwords were publicly available.

<comment><USER10> Security by obscurity is precisely defined as security that relies on the algorithm/implementation itself being private to be able to function. Key material being private does not qualify for this. The alternative is that security through obscurity becomes such an all-encompassing term as to become meaningless

</comment></comment></comment></comment></comment></comment><comment><USER11> We at [https://hubski.com](https://hubski.com) use a very hacked version of news.arc. Going on seven years. We moved data to postgres though.

The app is top notch, but it does not scale very nicely. Too much to hold in memory.

</comment><comment><USER12> I'm working on a tool that maintains an up to date archive of Hacker News and a companion tool to browse and search it. I'm not ready to release yet, but here is a dump of the stories, comments, and users from the Firebase API as a SQLite database with a full text search index: [https://archive.org/details/hackernews-2017-05-18.db](https://archive.org/details/hackernews-2017-05-18.db)

</comment><comment><USER13> The source of lobste.rs is actually a good second alternative to HN [https://github.com/jcs/lobsters](https://github.com/jcs/lobsters)

It has been in active development for several years, and is actually quite popular in its own site.

It also incorporates several interesting concepts like an invite-tree,etc.

I don't mind HN close-sourcing its code...But I'm definitely curious keeping Arc alive. IMHO, the rest of YC stack is pretty vanilla.

I wonder if Arc is the "burning candle that must never be allowed to die out"

</comment><comment><USER14> The database is now on Google Firebase, is it not?

At least their API is sourcing all data from Firebase last time I looked at it...

<comment><USER15> That's just for the API.

<comment><USER14> Thanks for clarifying... I'd be interested to hear about how they sweep the data from the existing database into Firebase. Must be in close to real time, because the Firebase API update feeds trigger every 10 to 15 seconds with updated data.

<comment><USER16> I wrote the original integration, which required updating some Racket libraries to support HTTP pipelining, chunked transfer encoding, and streaming JSON serialization :)

My version was near real time, with a 30-60 second batching delay IIRC, and could still be what powers it for all I know.

</comment></comment></comment></comment><comment><USER17> I would also like to know about how they have optimized arc and it's underlying runtime from racket . They seem to have proven that arc can scale .

<comment><USER18> It mostly proves that cloudflare can scale. Not that long ago Dan asked us to log out due to server load so that pages would be served from the cloudflare cache rather than that they had to be generated on the fly.

<comment><USER7> Thanks to DanG and others for their work keeping HN running! (for the past 3+ years: [https://news.ycombinator.com/item?id=7493856](https://news.ycombinator.com/item?id=7493856) )

\--

HN Scalability

[https://news.ycombinator.com/item?id=13755673#13756819](https://news.ycombinator.com/item?id=13755673#13756819) (3 months ago, on _Ask HN: Is S3 down?_ )

> _If you don 't plan to post anything, would you mind logging out? Then we > can serve you from cache._

[https://news.ycombinator.com/item?id=12909752#12911870](https://news.ycombinator.com/item?id=12909752#12911870) (6 months ago, on _Donald Trump Is Elected President_ )

> _please log out unless you intend to comment, so we can serve you from > cache_

> _We buried the previous thread on this because, at almost 2000 comments, it > was pegging the server._ | > [https://news.ycombinator.com/item?id=12911042](https://news.ycombinator.com/item?id=12911042)

[https://news.ycombinator.com/item?id=9174869#9175457](https://news.ycombinator.com/item?id=9174869#9175457) (1 year ago, on _Ask HN: Why the HN Nginx error for articles below 9M?_ )

> _We turned old articles off a few hours ago as an emergency measure because > the site was being crawled aggressively and our poor single-core Racket > process couldn 't keep up. Previous solutions that have served us well until > now, such as IP-address rate limiting, appear no longer to suffice._

\--

Source: [https://hn.algolia.com/?query=author:dang%20cache&type=comme...](https://hn.algolia.com/?query=author:dang%20cache&type=comment)

</comment></comment></comment><comment><USER19> Don't use a filesystem as a database. We did that back in 1998 mostly just because we were using 1998-era computing power, and the more static content you could serve the less likely your box would get hosed by Slashdot. Now my watch has more computing power than those old servers. Use a real database, even a file-backed one like BerkeleyDB (or god forbid, SQLite)

<comment><USER20> "God forbid, SQLite"? It's honestly a great product and a wonderful stepping- stone. I wish more new projects used it rather than half-measures like Mongo.

<comment><USER21> 100% agreed, it can hand a few hundred thousand hits a day. Obviously not okay for HN scale - but it's what my personal site and a bunch of side projects use.

<comment><USER22> Do you have problems with locking if two requests are made in the same time?

<comment><USER23> AFAIU it's only locked for concurrent writes, so a largely read-only workload should be okay.

<comment><USER22> Reading notes at [0] it seems it takes a little bit of configuration to get good setup.

[0]: [https://secure.php.net/manual/en/sqlite3.exec.php#usernotes](https://secure.php.net/manual/en/sqlite3.exec.php#usernotes)</comment></comment></comment></comment></comment></comment>


Decap of a Cell Phone SIM card [video] - <AUTHOR> https://www.youtube.com/watch?v=l_BfjEF513k <comment><USER2> If you want to understand more about what the glitching protection is about Scanlime recently made a very good video where she grabs firmware from a drawing tablet using such an attack.

[https://www.youtube.com/watch?v=TeCQatNcF20](https://www.youtube.com/watch?v=TeCQatNcF20)

</comment><comment><USER3> sim cards are as capable as 80s computers and run the JVM.

[http://www.extremetech.com/computing/161870-the-humble- sim-c...](http://www.extremetech.com/computing/161870-the-humble-sim-card-has- finally-been-hacked-billions-of-phones-at-risk-of-data-theft-premium-rate- scams)

<comment><USER4> If so, could obsolete SIM cards not be reused as embedded systems for free?

<comment><USER3> see defcon talk about them. it was hell to buy sim cards to do the shaddytel for torcamp.

basically, telcos make sure only them can update the code there. it's a computer running 24h, with full network and sensor access, that you carry everywhere, and you have zero control or visibility

</comment></comment></comment><comment><USER5> Hmm so even if you can ever buy a secure phone you trust, you need to put another, completely opaque, computer into it to make it function (on cell networks anyway.)

<comment><USER6> I wonder if there's any easy way to make a private GSM network so that we can see what data is actually passed between the SIM and the tower?

<comment><USER7> There is, Osmocom/OpenBSC. IIRC they got 3G working in a fully FOSS stack.

<comment><USER8> [http://osmocom.org/projects/openbsc](http://osmocom.org/projects/openbsc)

</comment></comment></comment></comment><comment><USER9> There was a talk on blackhat or defcon about the abilities of these controllers, running mini java applications and other cool things. I remember it being said finding necessary SDK's were very difficult and sometimes secretive. Makes me wonder.

<comment><USER10> > mini java applications

They say Java but it's basically C++. It's Java without strings and I'm not even sure it has normal 32-bit signed integers (heard somewhere that they didn't, but I can't find it right now). It's called Java Card by the way.

</comment></comment><comment><USER11> That is a very well-made video.

Like the video creator, I'm very surprised the humble SIM card has been made so capable!

</comment><comment><USER12> Honest question, why there is a need for a ARM based processor on that SIM card? AFAIK, the role of the SIM is to securely store all kind of IDs and PINs and contacts.

I am quite sure that my first SIM card, 20 years ago, didn't have such setup and worked, quite the same.

<comment><USER13> It sure did have such setup.

SIM cards can have micro applications that get exposed via the "SIM Services" menu entry or something similar named.

That is how before the WAP days, the carriers used to offer SMS based applications.</comment></comment>


Chat Bubble Blindness - <AUTHOR> https://www.atlistmaps.com/in-good-company/chat-bubble-blindness <comment><USER3> I too always find the Intercom or similar bubbles incredibly annoying, intrusive and distracting, especially when a bot pops up saying something that I absolutely don't need to focus on.

I think the purpose of this bubble is to promote Intercom. It's for sales/marketing people who see it elsewhere first (but never use themselves) and think it's a great idea. Next thing you know, there's an order coming from the C-level to engage with Intercom and integrate it.

The effect of the integration is that some people _will_ talk and it will seem like an improvement - compared to zero interactions previously, no doubt it is. As a consequence of this, nobody bothers to check if it's the best way to integrate the chat functionality.

(Although this is only one of the 1000 things that are wrong on the web today. All sorts of unnecessary things that you need to ignore, neutralize, refuse, dismiss etc. before you can get to the content.)

<comment> munk-a There was an article yesterday about the lack of increased web page performance to match the constantly growing infrastructure - one item that came up was the fact that many sites have unnecessary features that no one asked for. This quote:

> We first added Intercom when we launched Atlist— it just seemed like the > startup thing to do.

Seems like a perfect example of a feature that wasn't born out of demand or a perceived demand - it was just one of the bells and whistles available as freebies in frameworks so why not throw it in there.

<comment><USER4> Isn't it weird that we have entire companies like Intercom or Rasa whose value add is pushing automated, AI-driven "assistants" onto websites, and then the companies that buy into that entire value add and codebases hacked on by ML experts find that none of it even works better than how it was in 1998?

I could understand the thought process at first glance. NLP is revolutionizing things. Startups want the lastest technology if it's backed by Arxiv papers underneath enough layers. There is the image of people working dead-end jobs in call centers answering questions that nobody wants to spend time on but need to get answered anyways. So why not just make all of that go away, by inserting this ready-made digital replacement into your package.json?

I feel like the point is being missed though. Because even a call center employee working at Comcast is _still a human_. I think people are getting way too ahead of themselves in thinking code and "neural" networks (what a misleading term) can suddenly automate all of these things that could never be automated before, like dynamic conversations. And even before that the article proves that a single HTML form element is 62% more effective than a VC-backed GPT2-powered "solution" to a problem nobody had. Why require a dynamic conversation at all if e-mail works better?

Maybe there's some property about human interaction with founders or call center workers that's fundamentally impossible to replicate with technology, at the philosophical level, and we'd be better off investing into human resources instead of spending all of this engineering effort trying to work around that fact. Or maybe there theoretically isn't such a limitation, but I have serious doubts that 2020 is the year we'd finally be ready to declare that we've gotten past them.

<comment><USER5> While intercom's advertising may like to tout bots and AI chat, I can't imagine that is why most companies use it. They have a plethora of other features, sort of like proactive marketing automation that make it useful.

If it were just about the chat we would have dropped them for a cheaper solution long ago.

They continue to roll out new features and charge more and more for them though, so they will eventually run us off. Just not yet.

</comment></comment></comment></comment><comment><USER6> Am I the only one that finds the automatic popup chat really intrusive? I generally leave the website shortly after.

Give me the option, if I need it I will use it. Force it on me I will leave

<comment><USER7> As retail stores started to die, some smart cookie said:

"Hey, remember those people who follow you around the store and intruded as you were just browsing? Those sucked! Let's bring that to the web". And lo, chatbots were born.

The worst ones are the embeds that open a small window within the page. Hope you didn't want that content in the bottom-right.

<comment><USER8> This is an American disease. In most European countries staff might great you, but they will try to discreetly shadow you and once you act like you need a help they will step in to assist.

I absolutely hated the US diners with multiple 'are you ok? do you need anything?' interruptions during single meal. I know why it happens, I just find it really annoying.

<comment><USER9> _> In most European countries staff might great you, but they will try to discreetly shadow you and once you act like you need a help they will step in to assist._

This is even worse (to me) because most of the time there's nothing _discreet_ about it.

I'd rather have someone ask up front if I need help - at which point I can politely say no - than just follow me around the store, silently watching, like a hyena stalking a wounded gazelle.

<comment><USER10> He's being a bit hyperbolic. As I'm also European, I can tell you employees are off doing their own thing. If you do need help, you have to go find one.

<comment><USER9> In my experience, this is a global phenomenon that's probably driven more by management philosophy than by geography. It's endemic to e.g. luxury retailers and trickles down from there.

</comment></comment></comment></comment></comment></comment><comment><USER11> In my experience chat buttons are close to useless. First of all that's usually not a chat at all at first, there is some bot with a series of dumb checks and links. And if you manage to get a human on the other side they usually have a timer and if you dare to type your reply longer than 30 seconds they will just drop your session at any arbitrary moment with a comment "user didn't respond".

My internal ranking of contact options is like this:

Email - completely useless, won't even try. Nobody ever replied me on my email sent to the contact listed on a company website. (my address is my first plus last name at gmail, so nothing "weird" there)

Chat bubble - almost useless, will try only when desperate, like when other channels don't respond or gated (e.g. phone number gated by ten layer deep voice menu)

Telegram/Whatsapp chat associated with work number - almost always work and quick, but rarely made available for customers. Or available only after you've already contacted them some other way.

Phone - always works but needs a lot of attention and interruption from both parties

<comment><USER12> > _Email - completely useless, won 't even try_

Huh? That's how I always reach out about something, and probably get a response 90% of the time?

I don't think you're giving e-mail enough credit here. Not sure why you think it's useless.

I detest all the synchronous waiting associated with chat/IM/phone hold, I much prefer to send an e-mail and get a reply the next day whenever possible.

<comment><USER11> I would much prefer using emails too. My mails are usually to some reseller about specific unmentioned details of the product - like what is the type of power socket, is there a certain feature present in that specific model etc. Anything that is not easily searchable. I assume that regular call operators who probably also read emails don't have an easy answer and just ignore mails altogether.

</comment></comment></comment><comment><USER13> In the full blog post, they show that the numbers went from:

* Chat bubble: 34 conversations from 8,004 visitors (0.42%)

* Nav link: 45 conversations from 6,622 visitors (0.68%)

It probably performs at least on par with the chat bubble, but it doesn't seem like enough data to say confidently that the navbar outperforms the chat bubble.

I agree that it's a net win to remove the intrusive chat bubble if they're not sacrificing conversations, but the title is overstating the evidence.

<comment> steve-benjamins Hey I'm the original author!

How much data would you need for confidence? According to my calculations this is 98% confidence. That feels like 'enough' to make a decision for my small startup.

I'm pretty explicit in the blog post that this isn't meant to be universally applicable— it's just what happened to us.

<comment><USER14> First of all, kudos for quantifying your results instead of hand waving them. Yes, your results look like a ~60% improvement in conversion rate from the A to the B test, with a p value of 0.02 and a statistical power of around 80% for a two-tailed test. So that's good.

However context is important - at this level of significance you'd expect to see a similarly strong, but ultimately spurious, effect going from the A to the B test about 1 in 50 times.

Since you're not working on something safety critical, that's probably an acceptable false positive rate for you. But generally speaking, and in particular here since the absolute numbers and changes are quite small, I would be wary of trusting such a result. It seems promising but inconclusive. Maybe run a few more tests with disjoint (or nearly so) samples of visitors?

There are a few other things that could possibly confound the result - off the top of my head, your screenshots look like different pages between the A and B test. I'm not sure if that's how you ran the experiment or if you just happened to use two different page screenshots, but that would typically disqualify the result and require another test.

<comment><USER15> The way I'm seeing it is sure the error bars are huge. But it's very unlikely to be a regression. And team likes it better.

</comment></comment></comment></comment><comment><USER16> Just as a note: if I ran an A/B test and one bucket had 8,004 visitors and the other had 6,622 visitors, I'd be concerned about the way the test was setup.

There are some valid possibilities to explain the discrepancy, but there are also a lot of possibilities in which the test wasn't setup properly and it isn't measuring the right thing.

When you get a test result that confirms your suspicion, the first thing you should do is challenge those results.

<comment><USER17> It doesn't look like they A/B tested. More likely just switched to it 100%

<comment><USER18> Even more of a problem for interpretation of these results...

</comment></comment></comment><comment><USER19> Personally i dont mind the in web chat features (intercom being one) but what I really HATE is when you click on it and it says 'Were not available pls send us an email'. So program the damn thing with your working hours and dont display a chat bubble if there is no one to chat with at that moment in time. If I want to email I will find your 'Contact Us' page. Sorry for the rant perhaps I am alone in this...

<comment><USER20> That's a bit like when a site scolds you for putting spaces in your credit number, when it could just strip them automatically.

</comment></comment><comment><USER21> I really hate those live chat bubbles, and many people feel the same way. When I see one of those I instantly go looking for alternatives to your site. Sure, you might win more clicks or whatever, but it's also worth considering the goodwill lost.

It's kind of like hiring a used car salesman to do your sales. Sure, perhaps he'll sell more in the short run, but if that's what you're after there are plenty of unsavory tactics out there you can use. Wouldn't you rather customers just had a good, clean experience they're happy to share with others?

<comment><USER22> > When I see one of those I instantly go looking for alternatives to your > site. Sure, you might win more clicks or whatever, but it's also worth > considering the goodwill lost.

I think we should keep in mind that HN readers are far from the typical internet user.

</comment></comment><comment><USER23> I think somewhere along the line the expectations for what people get when they click on a floating chat box changed.

When we started Olark ([http://www.olark.com](http://www.olark.com)) in 2009 it was novel to show status before you clicked on the chat button, and to float on the screen.

I both were super important, because like you I hated clicking on a box, button, icon, for instant help and having it say "leave us a message" \- and I didn't like having to dig around to find some way to contact a business.

Now-a-days many chat/messaging products occlude presence in favor of collecting as much contact information as possible without letting you know if someone is actually there to answer the question --- or if you will be funneled to a bot.

There are a couple of reasons for this: \- the relatively high cost of having a person talk to you. \- the allure of getting leads for low cost (i.e. chatbot < $$ than person) \- it's far easier technically to ignore presence as boot-time or never even implement it. \- the growing lack of user expectation for an immediate response.

I think good human-to-human conversations are essential whether you are starting out or scaling a business. The trick is consumer behavior is changing due to dark patterns.

</comment><comment><USER24> I think the execution of the the chat bubble (a circle with a strange looking chat icon plus happy face) is not great.

Perhaps "blindness" is an issue. But also the chat bubble execution itself. Do visitors understand what it is?

I wonder what would happen if there were a persistent chat bubble with the text "live chat" or "chat now" "questions?"

In contrast, "Live Chat & Support" in the primary navigation is extremely visible and clear.

<comment><USER25> I see that circle thing all the time but I had no idea what it did until this moment.

<comment><USER26> It reminds me of the Lolwut pear more than anything else.

</comment></comment></comment><comment><USER27> This makes sense to me. I usually assume that there isn't a real person at the other end of those bubbles, and that if there is they'll only be trying to sell me something.

We use Intercom in our product at work. But it's for customer support, and is always initiated by paying customers.

<comment><USER28> You'd be surprised how many chats start with "Can I talk to a real person please?" and get shocked when I answer swiftly (during working hours of course)

<comment><USER29> You can write a script: "I'm a real person" and save time

</comment></comment></comment><comment><USER30> It is really great to see this researched and documented. While not every A/B test is proof, it is compelling in this case - and confirms my long-head view that these chat bubbles are just bad UX. There is a lot of other recent "UX best practices" that are actually worst practices, but because every unicorn is doing it, teams often feel they should do the same - without asking whether this kind of "engagement" is really serving the core product management objectives.

</comment><comment><USER31> I don't have chat bubble "blindness" — I have configured my ad blocker to hide them, as well as any other clutter I find distracting. I wouldn't be surprised if other users, rather than being blind to it, never even know the damn thing is there.

</comment><comment><USER32> I use the intercom style buttons to chat with sales reps all the time. It's the most convenient method for a few reasons,

1\. I never know what forms of contact a website will provide until I navigate around the site to find it.

2\. Finding the contact information can be difficult in the first place.

3\. With the instant chat I either a) know I will get a quick response, or b) it tells me how long a general response takes.

4\. I don't like having to use email for quick questions.

5\. I DO like that the chat falls back to email as a response mechanism if the response time is longer than I am willing to wait for.

<comment><USER29> I do too, I think it's nice to know where to look (I mean, in the corner). But it's annoying when there's no real person, just a nonsense script

Or if the chat widget asks for an email address directly. But then I can just type random letters incl an @

<comment><USER32> Yeah, there are certainly antipatterns that are in use for customer support.

</comment></comment></comment><comment><USER33> I find them to be intrusive, and end up ignoring them in the same way I end up not noticing things that look like ads. And sites that overdo it lose my attention altogether - I just move on to greener pastures.

Mind you, I used to mute the television when ads came on. The constant repetition drove me nuts. Radio is even worse, which is why I haven't used one since I was a kid.

Its not that I try to avoid any and all advertising. I think Twitter actually does a good job of it - at least in my feed, as the ads I get are directly related to my interests, and have actually resulted in purchases from me.

It's the constant unwanted disruption. Remember when people hated the <blink> tag in the 90's, or those ads that jiggled? They are all different variations on how to annoy people into thinking about one (generally unwanted) thing all day.

You know who does chat very poorly? LinkedIn. Once those notifications show up on the screen, it's so hard to get rid of them! You have to save messaging to the last on that site if you don't want to see the name list covering the right-edge of the screen, or a chunk out of the bottom when you minimize it.

</comment><comment><USER34> Their A/B testing results are most likely due to chance.

Using the technique from [https://www.evanmiller.org/sequential-ab- testing.html](https://www.evanmiller.org/sequential-ab-testing.html)

N = 14626, sqrt(N) = 121

T = 45

C = 34

T - C = 11, far below the 121 required to declare the change a success with 95% probability.

T + C = 79, also below the 121 required to declare no winner.

</comment><comment><USER35> I must confess I did not recognize that icon from _anywhere._ Seems like a much simpler root cause is that many others also did not recognize this as a chat icon, particularly those outside of webdev or design circles.

<comment><USER36> I was coming here to say something similar. There is plenty of data out there that labels are often preferable to icons because people understand words, but don't necessarily understand your icon.

So it may be that there isn't blindness to the icon, just less confusion with the text. You could further test this perhaps by reinstating the icon and labeling it. That might convert even better.

</comment></comment><comment><USER37> There's not really a lot of context here, why was the bubble annoying him? Were they pushing messages upon the user landing on the page? The real value of the bubble is to engage prospects on marketing sites and landing pages.

<comment> steve-benjamins Hey I'm the OP— we give more of the story here: [https://www.atlistmaps.com/in-good-company/chat-bubble- blind...](https://www.atlistmaps.com/in-good-company/chat-bubble-blindness)

Sorry. I didn't submit this!

</comment></comment><comment><USER38> Some perspective from the other side.

We've found intercom to be a massive improvement in our user conversations, up from when we have a live chat button as well as when we had a direct WhatsApp/chat functionality through a page.

I'm with everyone here in how annoying chat plugins are, in fact I was so against it that I fought tooth and nail to keep it out of our product. I eventually decided i should back it up with data and decided to give it a shot.

Speaking to the users, they love it. We're short circuiting a lot of conversations about usability and "how do I do this" little things that wouldnt have even reached us otherwise.

I rub with all of this is that I think we operate in a different user space than most of the commenters here (but I may be wrong). I say this because I still have chat popups, and it actively turns me off using a product and makes the decision for me in whether I buy something (to a no).

The difference might be that our users skew older - 40s to 50s and up, and are from a very old industry with little tech penetration. Email is still the preferred mode of communication, And they don't escalate anything except complete system failures. Intercom has somehow jumped that gap and makes them talk to us more. I'm as surprised as anyone.

</comment><comment><USER39> Someone has finally got to it. I hated the bubble for such a long time because I don't think you need chat everywhere and in your face. It reminds me of credit card agents chasing me in malls.

</comment><comment><USER40> If they were to actually test "chat bubble blindness", you have to replace the bubble icon with the text at roughly the same position and compare. They didn't do that. Rather, they moved the chat to more prominent position - <USER2> the top of the page near the center, with bigger space, etc. So of course it increases the engagement.

More important question is if they are getting spurious extra engagement (i.e. more chat traffic eventually costs more money), or they are getting the engagement they want.

<comment> steve-benjamins Good luck AB testing things in the real world :p

<comment><USER40> Indeed. I'm not saying they should do a perfect A/B test. Rather, in this case, their conclusion (of "chat bubble blindness") is not a logical conclusion you can draw from the experiment they performed.

<comment><USER41> No, that's not the conclusion. That's the portion that is explicitly speculation. There are tells in the language:

> _The navigation link led to a 62% increase in live chat conversations...This > felt like a win— I like live chat but I hate chat bubbles...In any case, don > 't expect to see a chat bubble on Atlist ever again. _

That's the conclusion: "For this page, losing the bubble is okay. We're going to lose the bubble."

> _And if I hate them, maybe users hate them too? Maybe users have developed a > chat bubble blindness— just like banner ad blindness._

That's the speculation. You can tell because of the 'maybe's in there.

This is actually quite well-written. The conclusions are written in clear language indicating fact. There is a potential "future work" description that speculates on possibilities while expressing the uncertainty.

IMHO very well expressed, good text. But what you've said has made me think about why people sometimes read scientific papers the way they do. They'll say "They claim that this will make fusion power possible" or something like that when those parts are in the "Future Work" section and are therefore unsubstantiated and intended to be a "here are some distributaries of this river of fact".

Interesting. I think the cues are adequate, but clearly they are not for many. I still think they should write the way they do and just accept that some in the audience aren't going to make it there.

</comment></comment></comment></comment><comment><USER42> I must say that I too have come to absolutely hate the chat popups. They're almost always bots and have some pre programmed rubbish responses. They are not "AI" and dont come close to helping me with anything. In fact, I've come to hate it so much that I have a chome extension that makes sure I never see another one! May I add that since that moment, I've had as much help on websites and I dont feel irate with one less thing popping up as I navigate!

</comment><comment><USER43> An interesting question is whether they got .24% _more_ chatters or not. Would leaving the chat bubble _and_ having the link lead to .75% of visitors chatting?

They had good reasons for getting rid of it (the "designing around it" issue is a real "tail wags dog" problem), so I'm not suggesting they made any sort of mistake! Just curious if one vs the other appeals to different kinds of ppl in a meaningful way.

</comment><comment><USER44> Good on them for testing assumptions.

Less than judicious use of any modal/floating element is a problem and adds to user frustration.

It's worth doing an external test with a fresh browser (no {cookies, localStorage, cache, logged out, default browser API permissions for the website} every month or so. With the basically-useless-now cookie disclosure banners, chat banners, adblock warnings, "returning? why don't you log in" modals, special offer modals, location permissions prompt, notification permissions prompt, occasional survey prompts, etc. These make me want to leave your site as fast as I can, sometimes before I bother staying for my purpose.

I think developers tend to be responsible for adding features but few if any people are responsible for removing features when they don't work well together or are no longer as relevant as they could have been.

Also, modals and floating elements are simply overused. They are the equivalent of _blink_ ed elements in the early web days. Use sparingly.

</comment><comment><USER45> Just want to say, I'm happy to hear this. That chat bubble is beyond annoying and many people abuse it to the point where it simply drives me away from the website. It can block screen elements and sometimes it bounces and/or updates the page title so that your browser tab titles changes in an incredibly frustrating way.

</comment><comment><USER46> between chat popup, cookie notice, notification confirmation, email subscription box... there is very little left to imagination then that this website is obviously on drugs

<comment><USER47> And with segment, hotjar, google analytics, heap analytic. Jesus, how much analytics do y'all need.

<comment><USER46> but MORE DATA

</comment></comment></comment><comment><USER48> I wonder if the conversions went up included a disproportionate amount of mobile users.

The chat bubbles are far too big on mobile devices.

I go out of my way to instantly close them and it does not delight me in the slightest to have to deal with them.

Having a chat button to press that is unobtrusive is a far better layout.

</comment><comment><USER49> Some pretty lazy maths concerning the statistics:

Assuming there was a 0.545% chance for visitors to start a conversation, and the chance didn't actually change between the tests.

In that case you'd have an ~8% chance to get 34 conversations _or less_ for 8004 visitors, and also an ~8% chance to get 45 conversations _or more_ for 6622 visitors, and there would be just a ~0.66% chance you'd actually see a gap _that large_ (or more accurately: gap between any x<=34 and x=>45) by random chance, suggesting there was probably some change in p in between the tests.

Mind I just played with a specific p of 0.545% here, and also halfassed it.

</comment><comment><USER50> I suspect that any effect is due to moving from an icon to text. We are inundated with icons. They become hard to see. They are harder to interpret. The meaning of text is much simpler to infer, in fact, it is the main purpose of text.

</comment><comment><USER51> Intercom was one of the first thing I put on my pihole blocklist. Since them I've had to add quite a few more. The one thing I absolutely cannot stand is making my tab blink like I have a notification.

</comment><comment><USER52> I leave the site immediately if there's intercom integration or similar.

<comment><USER37> This seems like quite an overreaction.

<comment><USER52> Rip it out of your site. I hate it.

<comment><USER37> Regardless of whether or not you hate it, for marketing and making sales it is clearly effective. If you hate it SO much and have technical knowledge, just block the Intercom domain or something.

<comment><USER52> The whole premise of this thread is that it is not effective. Your tone is like it's my problem. It's their problem.

</comment></comment></comment></comment></comment><comment><USER53> The main problem of those bubbles to me is that they break the UI. Sticking to your own UI gets more engagement as the article pointed out.

</comment><comment><USER54> I think the lift in conversions is because Intercom bubbles aren't live chat typically, they are a "Contact Us" form or an automated bot (not true "live chat"), so people are starting to ignore them. They changed their message at the top to say "Live Chat" so people click on that more because they are expecting real live chat.

</comment><comment><USER55> I have noticed some marketers have tried to fix chat bubble blindness by also adding animated popup messaging and sound effects to your chat bubbles. You can tell that consumers like it, because it drives up the interaction rate.

I'd be interested in seeing an A/B/C test between a basic bubble, a noisy bubble, and the text link demonstrated in the linked article.

</comment><comment><USER56> I get this, I usually ignore this bubble on website and I ignore it even more when it pops up and pretends to chat with me. So, I'm glad someone tried to remove it and got even better results.

</comment><comment><USER57> god I fucking hate chat widgets that sit there drawing attention to themselves and opening up without any interaction, sometimes I go so far as to use [an ancient page-editing bookmarklet]([http://mrclay.org/2006/04/23/mypage- bookmarklet/](http://mrclay.org/2006/04/23/mypage-bookmarklet/)) to get rid of particularly annoying ones.

</comment><comment><USER58> I don't think the issue is chat bubble blindness. I think the issue is material design floating action button blindness.

</comment><comment><USER59> Alright I guess — but as an aside, disabling keyboard scrolling on the page is a really weird design decision.

<comment><USER28> Can you elaborate?

Edit: I see now, that is not intentional.

</comment></comment><comment><USER60> On the other hand its a standard UI element that everyone knows. Of course, its often abused.

</comment><comment><USER61> Chat buttons allow us to troll websites that have no idea what they are doing- most of them.

</comment><comment><USER62> > it just seemed like the startup thing to do

This motivation sounds like a recipe for disaster.

</comment><comment><USER63> uBlock Origin and/or Privacy Badger block Intercom. I suspect it's not chat bubble blindness, it's chat bubble blocklist.

</comment><comment><USER64> I would never have known what this button did.

</comment><comment><USER65> This relative percentage stuff is silly.

</comment><comment> steve-benjamins Hey I'm the author.

This links to an Indie Hackers post which doesn't have the full context. You can find the original post here: [https://news.ycombinator.com/item?id=24060501](https://news.ycombinator.com/item?id=24060501)

Sorry OP— just thought I'd point that out :)

<comment><USER66> Normally dang (HN moderator) is willing to fix links to make things better. Should probably work here as well.

<comment> steve-benjamins Is there any way to get his attention for this? That would be amazing...

<comment><USER67> email him. contact link is in the footer.

<comment> steve-benjamins Awesome. Just did. Thank you!!

</comment></comment></comment></comment></comment><comment><USER68> I doubt this worked because it is less annoying but because it is now easier to locate the button in the first place and it tells you exactly what you need to know about its function.</comment>


Getting Real About Distributed System Reliability (2012) - <AUTHOR> https://blog.empathybox.com/post/19574936361/getting-real-about-distributed-system-reliability <comment><USER2> While it is a bit dated in terms of the 'types' of applications it is still quite relevant in terms of understanding "web 2.0[1]" distributed system reliability versus other systems.

One of the coolest things Greg Lindhal built at Blekko was a system that processed the error logs of every single system in the cluster all at the same time. It then reduced by string matching error messages that varied only by their variables into a single error (so 'error reading sector 10 on disk sda3' and 'error reading sector 20 on disk sde2' would reduce to 'error reading sector X on disk Y' x={...}, y={...}') really fun stuff. And you could easily pull out of that errors in the software (happens on many nodes) errors in hardware (happens on unrelated nodes) and errors in infrastructure (happens on nodes correlated by a switch or PDU or Rack)

[1] Distributed systems made of many many identical nodes both from a software and hardware perspective, managed by an orchestration service with software that assumes unreliable platforms.

</comment><comment><USER3> While this post is from 2012, much of it applies very much, especially when building your own systems/(micro)services:

>The actual reliability of your system depends largely on how bug free it is, how good you are at monitoring it, and how well you have protected against the myriad issues and problems it has. This isn't any different from traditional systems, except that the new software is far less mature. I don't mean this disparagingly, I work in this area, it is just a fact. Maturity comes with time and usage and effort.

Working at Uber on larger systems, it's surprised me how much more effort we're putting in operating the system reliably, vs the upfront planning/design (and we spent a lot of time on planning/design). I wrote in-depth about those practices and here's the relevant HN discussion: [https://news.ycombinator.com/item?id=20462349](https://news.ycombinator.com/item?id=20462349)

</comment><comment><USER4> Small thread from back then: [https://news.ycombinator.com/item?id=3724508](https://news.ycombinator.com/item?id=3724508)

</comment><comment><USER5> I wonder if anyone ever measured failure rates for real world distributed systems outside of FANG vs simple non-distributed alternatives.

</comment><comment><USER6> Probably requires "(2012)" in title.

<comment><USER4> Added. Thanks!</comment></comment>


The Dawn of Trustworthy Computing - <AUTHOR> http://unenumerated.blogspot.com/2014/12/the-dawn-of-trustworthy-computing.html

<comment><USER2> This is why Plan 9 regards "root" as a design error.

A centralised Authentication Service, preferably behind a securely locked door with armed guards, protects the Authentication Service. Everyone else needs to provide credentials to verify their identity in order to read and write files on other machines, which could be serving files from disk or synthetically.</comment>


Thermal Testing Raspberry Pi 4 - <AUTHOR> https://www.raspberrypi.org/blog/thermal-testing-raspberry-pi-4/ <comment><USER2> Single point of anecdata: I found that using the flirc passive cooling case + putting the whole thing on its side dropped my average temp a full 20°C (from about 65° to about 45°) while running a Plex server.

<comment><USER3> Have you used a heat sink in your tests or did you used a barebones board?

<comment><USER2> previously I was using the heat sync that came with the 4B, inside the ABS plastic case that came with it (both in the canakit set)</comment></comment></comment>


Show HN: HTTP Request Translator - <AUTHOR> https://ryan.govost.es/http-translator/ <comment><AUTHOR> Background: I wanted to use Python to automate some web requests, replicating some XHRs made by a site. Safari Web Inspector will show you the details of the request, and even lets you "Copy as cURL," but it was time consuming to write the corresponding Python code.

So I took the opportunity to learn a little bit of React, and wrote a simple tool to convert curl command line arguments to Python Requests. It covers only a small subset of curl options at this point, but the code should allow new ones to be added easily. The structure lets me to independently work on the parsing frontend or code generation backend, so I was able to easily add the ability to parse raw HTTP requests as well.

Someone pointed out that there is also the curl-to-Go tool which does the same thing for translating curl commands to Go code. [https://mholt.github.io/curl- to-go/](https://mholt.github.io/curl-to-go/)

</comment><comment><USER2> Python request doesn't appear on iOS 10.2, only JSON works. Unfortunately I can't have a look on the console now...</comment>


The Righthaven model as seen by a pro-IP-monetization lawyer - <AUTHOR> http://gametimeip.com/2011/05/09/righthaven-engages-bad-ass-litigatrix-copyright-superstar-dale-cendali/

<comment><USER2> Hilarious. About flaws in Righthaven's court filings: _I glossed over much of this criticism because these represent errors in execution, which can fowl up even the most well thought-out business models._

Proofread much? The idiom is "foul up". A "fowl up" is when ducks are in the air.

Second, this is the kind of worthless piffle that makes lawyers into bad jokes. It's based exclusively on the "how many angels can dance on the head of pin" reasoning that's divorced from ethics, morality or any empirical concern. It's all law, all the time. Fortunately, not all of morality is contained within the law.

Third: here's an "IP" lawyer being imprecise about "IP". He/she/it/them muddle copyright, patent, trademark and trade secret into a giant, ill-defined mess called "intellectual property". Only a lawyer could love "IP" because nobody else can afford to devote enough time to teasing out the meaning of the different usages of the phrase "IP".</comment>