Terminal and shell performance - <AUTHOR> https://danluu.com/term-latency/ <comment><USER2> iTerm2 author here.

I'll spend some time looking into iTerm2's latency. I'm sure there are some low-hanging fruit here. But there have also been a handful of complaints that latency was too low—when you hit return at the shell prompt, the next frame drawn should include the next shell prompt, not the cursor on the next line before the new shell prompt has been read. So it's tricky to get right, especially considering how slow macOS's text drawing is.

If I could draw a whole frame in a reasonable amount of time, this problem would be much easier! But I can't. Using Core Text, it can easily take over 150ms to draw a single frame for a 4k display on a 2015 macbook pro. The deprecated core graphics API is significantly faster, but it does a not-so- great job at anything but ASCII text, doesn't support ligatures, etc.

Using layers helps on some machines and hurts on others. You also lose the ability to blur the contents behind the window, which is very popular. It also introduces a lot of bugs—layers on macOS are not as fully baked as they are on iOS. So this doesn't seem like a productive avenue.

How is Terminal.app as fast as it is? I don't know for sure. I do know that they ditched NSScrollView. They glued some NSScrollers onto a custom NSView subclass and (presumably) copy-pasted a bunch of scrolling inertia logic into their own code. AFAICT that's the main difference between Terminal and iTerm2, but it's just not feasible for a third-party developer to do.

<comment><USER3> > If I could draw a whole frame in a reasonable amount of time, this problem > would be much easier! But I can't. Using Core Text, it can easily take over > 150ms to draw a single frame for a 4k display on a 2015 macbook pro.

Holy cow! I wonder if iTerm2 would benefit from using something like pathfinder[1] for text rendering. I mean, web browsers are able to render huge quantities of (complex, non-ASCII, with weird fonts) text in much less than 150ms on OS X somehow; how do they manage it? Pathfinder is part of the answer for how Servo does it, apparently.

[1]: [https://github.com/pcwalton/pathfinder](https://github.com/pcwalton/pathfinder)

<comment><USER4> > Using Core Text, it can easily take over 150ms to draw a single frame for a > 4k display on a 2015 macbook pro.

I'm guessing that you're somehow preventing Core Text from taking advantage of its caching. Apple's APIs can be a bit fussy about their internal caches; for example, the glyph cache is (or at least used to be) a global lock, so if you tried to rasterize glyphs on multiple threads you would get stalls. Try to reuse Core Text and font objects as much as possible. Also check to make sure you aren't copying bitmaps around needlessly; copying around 3840x2160 RGBA buffers on the CPU is not fast. :)

For a terminal, all you really need to do for fast performance is to cache glyphs and make sure you don't get bogged down in the shaper. Pathfinder would help when the cache is cold, but on a terminal the cache hit rate will be 99% unless you're dealing with CJK or similar. There's a lot lower hanging fruit than adopting Pathfinder, which is undergoing a major rewrite anyway.

(I'm the author of Pathfinder.)

<comment><USER5> "iTerm2 author here"

...

"(I'm the author of Pathfinder.)"

I keep thinking that sometime soon the magic of HN will wear off, or we will hit "peak hackernews" or something like that.

Not today!

<comment><USER6> Same reason I love HN. We don't have to guess what the creator was thinking. They'll tell us :-)

</comment></comment></comment></comment></comment><comment><USER7> I think one thing that this really points out is just how much care Apple has poured into Terminal.app. It's _very_ good, and every time I have to use another terminal application (ugh conhost.exe) I am reminded of this. It's got a bunch of really thoughtful little features (showing what processes are attached to the current pty, option-clicking to move the cursor rapidly, full mouse support for apps like vim, good linewrap detection, and recently support for rich-text copy/paste which is useful for showing coloured terminal output, etc. etc.), and it remains really fast and snappy despite these features.

On a related note, I am big into latency analysis and driving down latency in interactive systems. I'm quite familiar with the touchscreen work cited at the top, and having played with the system I can attest that <1ms latency feels actually magical. At that level, it really doesn't feel like any touchscreen you've ever used - it genuinely feels like a physical object you're dragging around (the first demo of the system only let you drag a little rectangle around a projected screen). It's amazing what they had to do to get the latency down - a custom DLP projector with hacked firmware that could only display a little square at a specified position at thousands of FPS, a custom touchscreen controller, and a direct line between the two. No OS, no windowing system, nada. After seeing that demo, I can't help but believe that latency is the one thing that will make or break virtual reality - the one thing that separates "virtual" from "reality". I want to build a demo someday that does the latency trick in VR - a custom rig that displays ultra-simple geometry that has sub-millisecond latency to human head movement. I will bet that even simple geometry will feel more realistic than the most complex scene at 90 FPS.

<comment><USER5> "It's got a bunch of really thoughtful little features (showing what processes are attached to the current pty, option-clicking to move the cursor rapidly, full mouse support for apps like vim, good linewrap detection, and recently support for rich-text copy/paste which is useful for showing coloured terminal output, etc. etc.), and it remains really fast and snappy despite these features."

On the one hand, I have to agree that Terminal.app is quite good and very impressive. I don't bother with a third party terminal application and _I do everything in the terminal_.

However, one of the very valuable things about working in the terminal is the safety and immunity that it provides. No matter what bizarro virus attachment you send me, I can text edit it in a terminal without risk. There's nothing you can paste to me in irc that will infect or crash my computer.

Or at least, that's how it should be.

But the trickier we get with the terminal - the more things it does "out of band" and the more it "understands" the content of the text that it is rendering, the more of this safety we give up.

Frankly, it bothers me greatly that the terminal would have _any idea whatsoever_ what text editor I am running or that I am running a text editor at all. It bothers me even more to think that I could copy or paste text and get results that were anything other than those characters ...

Make terminals fancy at your peril ...

<comment><USER8> > Frankly, it bothers me greatly that the terminal would have any idea > whatsoever what text editor I am running or that I am running a text editor > at all.

I'm not sure what you mean by this. Terminal.app doesn't know that you're running a text editor. It does know that you're running a process called 'vim', which is kind of magic, but not too much (ps has always been able to show what processes are attached to a given tty, for example). If you're referring to the parent comment's "full mouse support for apps like vim", they just mean it supports "mouse reporting" control sequences, which date back to xterm. If anything, Terminal.app is late to support this (only in the latest release, whereas alternatives like iTerm have supported it for ages).

> It bothers me even more to think that I could copy or paste text and get > results that were anything other than those characters ...

Well, the terminal has to interpret escape sequences for colors and such in order to display them, so why shouldn't it also preserve that metadata when copying and pasting? Like any other rich text copy+paste, it will only be kept if you paste into a rich text field; discarded if you paste into a plaintext field.

That said, there are a few 'odd' things Terminal.app supports: e.g. printf '\e[2t' to minimize the window. (This also comes from xterm.)

</comment></comment></comment><comment><USER9> We can do better in Alacritty. For those interested, I've filed a bug on our issue tracker about where this latency is coming from and what can be done: [https://github.com/jwilm/alacritty/issues/673](https://github.com/jwilm/alacritty/issues/673)

At the end of the day, there is a trade off to be made. Terminals (or any program, really) can have 1-frame input latency (typically 1/60sec) and give up v-sync and tearing results, or they can have a worst-case 2-frame input latency with v-sync, and then you're looking at 2/60sec or ~32ms.

<comment><USER10> Triple buffering solves the latency issue of vsync latency by trading off buring additional cpu/gpu time, bringing the worst-case latency back to 1 frame.

<comment><USER11> I don't think that's correct.

The way I understand it tripple buffering _adds_ latency in exchange for higher sub display hz framerates.

Double buffering renders the next frame while displaying the current. That results in latency of 1 frame since input. Triple buffering adds another frame to the queue, resulting in a 2 frame lag.

With double buffering the framerate gets cut in half if it cannot meet vsync, with triple buffering it can also get cut in thirds. So double buffering is 60 -> 30, where the frame lasts 2 refreshes. Triple is 60 -> 40, where one frame is displayed for 1 refresh and another is displayed for 2.

Nowadays it's probably better to use adaptive vsync, which simply disables vsync when the framerate drops. This will reintroduce tearing, which might be preferable in fast action games.

<comment><USER12> Triple buffering confusingly means different things. Both you and the OP are correct. The OP meant something in line with what Wikipedia says: [https://en.wikipedia.org/wiki/Multiple_buffering#Triple_buff...](https://en.wikipedia.org/wiki/Multiple_buffering#Triple_buffering)

"In triple buffering the program has two back buffers and can immediately start drawing in the one that is not involved in such copying. The third buffer, the front buffer, is read by the graphics card to display the image on the monitor. Once the image has been sent to the monitor, the front buffer is flipped with (or copied from) the back buffer holding the most recent complete image. Since one of the back buffers is always complete, the graphics card never has to wait for the software to complete. Consequently, the software and the graphics card are completely independent and can run at their own pace. Finally, the displayed image was started without waiting for synchronization and thus with minimum lag.[1]

Due to the software algorithm not having to poll the graphics hardware for monitor refresh events, the algorithm is free to run as fast as possible. This can mean that several drawings that are never displayed are written to the back buffers. Nvidia has implemented this method under the name "Fast sync"."

<comment><USER13> It annoys me that most games don't seem to offer this option - if they have this level of control, they just offer vsync on or off. The AMD gpu control panel can force triple buffering, but only for OpenGL, and I think the vast majority of games on Windows use DirectX.

Triple buffering uses more display buffer memory, but roughly the same cpu/gpu load as vsync-off. It's great for latency. It makes a whole lot of sense. It's been around for like 20 years. But you rarely see it used ...

</comment></comment></comment></comment></comment><comment><USER14> Interesting results. I have loved XTerm for a long time because it "felt snappy". On MacOS I've always preferred Terminal.app to the often recommended iTerm2 for similar reasons.

I think it's funny to have the suckless project page for st go on and on about how XTerm is clunky and old and unmaintainable, but the result of this small and clean minimalist terminal is a closer loser in terminal performance, which subconsciously and consciously detracts from the experience.

<comment><USER15> I've really wondered that too: why is everyone recommending iterm? I'm glad it's just a matter of taste -- I'm perfectly happy with Terminal (and running a shell inside Emacs in my terminal :-).

<comment><USER16> It's a matter of taste in that both have features that the other doesn't have (terminal being faster being it's main, but significant one).

As far as I know, in terminal you can't use cmd as meta key, which immediately kills it for me as an emacs user (furthermore in iterm2 you can set it up so that left cmd = meta, right cmd = cmd, which I find very useful).

<comment><USER15> You sure can (I assume you meant opt), and have been able to since the NeXT days. There are many Emacs users at Apple (look at the Emacs key bindings in the text widgets)

</comment></comment></comment></comment><comment><USER17> _the most common terminal benchmark I see cited (by at least two orders of magnitude) is the rate at which a terminal can display output, often measured by running cat on a large file. This is pretty much as useless a benchmark as I can think of_

It's a really helpful benchmark, IMO, as it's the main problem I see with different terminals. On a chromebook, most SSH clients are effectively useless because if you accidentally run a command that prints a lot of output (even just 'dmesg'), the terminal locks up for a huge amount of time, seconds or even minutes. You can't even interrupt the output quickly.

I appreciate that it's a different problem to the latency that the OP is trying to measure, but as a benchmark, it's actually very useful.

<comment><USER18> Later in the same paragraph he addresses your exact problem, and points out that the speed of quitting a command that prints too much output is poorly correlated with the speed of spewing output.

Given how easy it is to accidentally spew something that I don't want to wait for, even if it is spewing quickly, I'm squarely with him in not caring about the speed of display. Slow it down to just faster than my eyes can make sense of it, and make ^C fast, and my life will be better.

<comment><USER19> I looked into this quite a bit when I was optimizing xterm.js.

Based on my findings, ^C is highly related to the speed of the output because the process running in the shell may be way ahead of the terminal's parsing/rendering. Imagine you run `cat foo`, the shell could take around 1s to send the output over to the terminal, the terminal might then take 10 seconds to parse and render the output. So after 1 second a ^C will actually do nothing because the cat call has finished. This is the case with Hyper, it hangs due to slow parsing and too much DOM interaction (as hterm was not designed for this sort of thing).

There's actually a mechanism for telling the process to pause and resume (sending XOFF/XON signals), which allows the terminal and shell to stay completely in sync (^C very responsive). However, these only really work well in bash as oh-my-zsh for example overrides the signal with a custom keybinding. Related links:

Original PR: [https://github.com/sourcelair/xterm.js/pull/447](https://github.com/sourcelair/xterm.js/pull/447) Post-PR bug: [https://github.com/sourcelair/xterm.js/issues/511](https://github.com/sourcelair/xterm.js/issues/511)

<comment><USER20> If the sending process blocks after filling the pipe (on Linux, 512 bytes), hitting ctrl-C should be effective—any of these terminals should be able to sink that in negligible time.

If it instead takes a long time, there are probably large buffers between:

* If you're talking about ssh to a faraway machine, the TCP layer is probably responsible. (I'm not even sure if there's anything you can do about this; the buffer (aka "window size" in TCP terminology, plus the send and receive buffers on their respective ends) is meant to be at least the bandwidth-delay product, and as far as I know, the OS doesn't provide an interface to tell it you don't need a lot of bandwidth for this connection. It'd be nice if you could limit the TCP connection's bandwidth to what the terminal could sink.)

* If you're talking about something running on the machine itself, it's probably an over-large buffer inside the terminal program itself.

</comment></comment></comment></comment><comment> def- Was curious and tried this out a bit on Linux+X11 on an i7 6700k with the igpu:

                 stdout [MB/s]  idle 50 [ms]
        urxvt             34.9          19.8
        xterm              2.2           1.9
        rxvt               4.3           7.0
        aterm              6.0           7.0
        konsole           13.1          13.0  note: stops moving when printing large file
        terminator         9.1          29.4  note: stops moving when printing large file
        st                23.0          11.2
        alacritty         45.5          15.5

<comment> sim- Can you paste or describe your measuring technique?

<comment> def- For the throughput I created a file with 1 GB of random source code and ran `time cat file`

Latency for input: [https://github.com/pavelfatin/typometer](https://github.com/pavelfatin/typometer)

</comment></comment></comment><comment><USER21> If you're sensitive to latency and run Linux, try hitting Ctrl-Alt-F1, and do a little work in console mode at the terminal. (Ctrl-Alt-F7 to get back.)

For me this is a great illustration of how much latency there is in the GUI. Not sure if everyone can feel it, but to me console mode is much more immediate and less "stuffy".

<comment> ezequiel-garzon I wonder how many folks use linux virtual consoles for day-to-day work. I guess it's not practical for web developers given that you probably have to switch to X to check results with a graphical browser, no?

<comment><USER22> I do :)

Specifically I run kmscon on one virtual terminal + tmux for scrolling/tabs, then an X server running chromium on another. I'm not a web dev, but I am continuously having to switch between the two for tracking merge requests, ticket status, testing via our frontends etc.

It's not bad, but kmscon is pretty much abandoned at this point, and I don't know of anyway to have a setup like this run nicely on multiple monitors. It was meant to be just an experiment at an über-minimal setup, I was planning to switch back to my previous i3 + st based setup after a month or so, but now it's been most of a year and I'm still using it for some reason.

I think the big thing I really enjoy about this setup is the complete lack of window management. Even the minimal window management I had to do with i3 (1-2 terminals running tmux + 1-2 browser instances) is gone. It feels like that's removed a small unnoticed stress point from my work day. If I ever get round to setting up a window manager again I think I'm going to try and keep it limited to 1 terminal + 1 browser instance and rely entirely on their internal tab support.

<comment><USER21> That sounds interesting. I'm having trouble visualizing what you are doing though.

Are you running a distro or did you build this setup yourself?

What role does kmscon play? I think you could just run raw tmux in one VT and X in another? Although to me it seems slow to switch between the two.

It seems like Linux should support the multi-monitor setup as I said in a sibling comment -- maybe I will take some time to investigate it.

Was latency once of your considerations, or was it mainly lack of window management?

If you have time a screenshot would be helpful :)

<comment><USER22> I'm running Arch linux, I just installed kmscon and enabled the firstname.lastname@example.com systemd service to have kmscon take over the second VT. It includes its own login manager that replaces getty. I then also login to the linux console on VT 3 and manually launch X from there (with just chromium in my .xinitrc).

kmscon has better handling for colors, fonts, etc. than the linux console; that's the only real reason I'm using it. On my laptop's builtin display I have no delay switching between any of the linux console/kmscon/X; when I plug in an external monitor I do get ~1-2 second delay switching from the linux console/kmscon -> X, no delay the other way.

There does appear to be some bug with switching from X -> kmscon, it just shows a black screen, but I've gotten used to switching X (VT 3) -> linux console (VT 1) -> kmscon (VT 2) which seems to work around that. There's also another bug where the Ctrl key seems to get stuck down in kmscon when switching to it sometimes, has only happened ~4 times in the last 8 months and I can fix it by just running `systemctl restart kmsconvt@tty2` and attaching to my tmux session again.

Since I'm not doing any frontend changes I don't ever really need to look at both my terminal and browser at the same time so haven't taken the time into seeing if I can have different VT displaying on different monitors. I prefer the portability of a laptop over having the most productive single location setup.

Latency was not at all a consideration, it was purely an exercise in how minimal a setup I could have. I spent a couple of weeks without having X installed and using command line browsers when I needed to, but using GitLab and JIRA through command line browsers was a real pain (and if I recall correctly some stuff was impossible).

A screenshot is difficult since it's multiple VTs and I don't think kmscon has any kind of builtin screenshot support. Just imagine a full-screen terminal with tmux, you hit Ctrl-Alt-F2, now it's a full screen browser; that's basically it.

One other thing I do have setup to make life a little easier are some little aliases syncing the X clipboard and tmux paste buffer so I can copy-paste between X and my terminal. And I have DISPLAY setup in the terminal so things like xdg-open and urlview can open stuff in my web browser.

</comment></comment></comment></comment></comment><comment><USER19> I work on the VS Code terminal/xterm.js[1].

Hyper which currently uses a fork of hterm, is in the process of moving over to xterm.js due to the feature/performance improvements we've made over the past 12 months. Hyper's 100% CPU/crash issue[2] for example should be fixed through some clever management of the buffer and minimizing changing the DOM when the viewport will completely change on the next frame.

I'd love to see the same set of tests on Hyper after they adopt xterm.js and/or on VS Code's terminal.

Related: I'm currently in the process of reducing xterm.js' memory consumption[3] in order to support truecolor without a big memory hit.

[1]: [https://github.com/sourcelair/xterm.js](https://github.com/sourcelair/xterm.js) [2]: [https://github.com/zeit/hyper/issues/94](https://github.com/zeit/hyper/issues/94) [3]: [https://github.com/sourcelair/xterm.js/issues/791](https://github.com/sourcelair/xterm.js/issues/791)

</comment><comment><USER23> > even the three year old hand-me-down laptop I'm using has 16GB of RAM

> on my old and now quite low-end laptop

Trust me, that's not a low-end laptop. Either that has the shittiest cpu ever and a terribly mismatched amount of memory, or the author's view of that is high-end or low-end is skewed; in either case, what's low-end nowadays would be ≤4GB RAM. 16GB is LOTS, useful for developers that run large builds and/or VM's regularly.

I very much like the rest of the article though, would love to see some latency improvements here and there!

<comment><USER24> It may very well be your view that is skewed.

For user-facing terminals/workstations, I would consider ≤8GB low-end ("unusable"). 16GB would be mid-low ("usable"), 32GB would be mid-high ("comfortable"), ≥64GB would be high ("good").

For servers, ≤64GB is low-end, ≥512GB being high-end.

That 8GB would be considered low-end does not mean that no one uses it, though. Some people might still rock 2GB laptops, or use the original 256MB Pi 1 as a light desktop.

<comment><USER23> Apparently that's true then. I'm in the Netherlands, so maybe the US is different in this regard. Thanks for clearing up!

<comment><USER25> No, you're completely right in that 16 GB RAM is not considered low-end. You only have to browse for laptops for about 2 minutes to discover that 16 isn't at all "low-end".

<comment><USER24> I think you missed the point I was trying to make entirely (that people's definitions of "low end" differ, although "low-end" steadily moves up), but from that perspective:

\- 8GB is easily available and what most cheap laptops sport,

\- 16GB is either default or an addon for cheaper laptops,

\- 32GB is a premium that is not always available, and

\- 64GB is usually only available in _huge_ workstation or gamer "laptops", although there are some decent-size Dells with it.

While those are not my choice of metrics, they do seem to support the "low", "mid-low", "mid-high" and "high" labels I personally added. At most, you could argue that ≤8GB is low, 16GB is mid and ≥32GB is high. 10 years ago, 16GB would have been high-end for a laptop, but no more.

</comment></comment></comment></comment></comment><comment><USER26> I love the analysis of terminal latencies! And I'm in full agreement with the overall goal of less latency everywhere. But, of course, I feel like picking a few nits.

> And it turns out that when extra latency is A/B tested, people can and do > notice latency in the range we're discussing here.

Yes, this is true. But the methodology is important, and the test used doesn't really apply to typing in terminals. The test isn't a "type and see if you can tell it's slow" test, it's a hit the mark hand-eye coordination test, something you don't do when typing text. Latency when playing Guitar Hero is super duper important, way more important than most other games, which is why they have a latency calibrator right in the game. Latency when playing a Zelda game is a _lot_ less important, but they still try very hard to reduce latency.

The same people who can distinguish between 2ms of difference in a drum beat also can't distinguish between an extra 30ms of response time when they click a button in a dialog box.

I'd like to see a stronger justification for why lower latency in a terminal is just as important as it is for hand-eye coordination tasks in games.

      ~2 msec (mouse)
      8 msec (average time we wait for the input to be processed
      by the game)
      16.6 (game simulation)
      16.6 (rendering code)
      16.6 (GPU is rendering the previous frame, current frame
      is cached)
      16.6 (GPU rendering)
      8 (average for missing the vsync)
      16.6 (frame caching inside of the display)
      16.6 (redrawing the frame)
      5 (pixel switching)

I find this list pretty strange. It's generally right - there are a bunch of sources of latency. But having done optimization for game consoles for a decade, this explanation of game latency feels kinda weird.

Games that actually run at 60fps usually do not have greater than 100ms latency. They also don't miss vsync every other frame on average, that 8ms thrown in there looks bizarre to me. Render code and GPU rendering are normally the same thing. Both current and previous frame GPU rendering is listed, huh? Sim & render code run in parallel, not serially. The author even said that in his article, but lists them separately... ?

Consumer TVs come with like 50ms of latency by default. That's often half of it right there. Games are often triple-buffered too, that accounts for some of it. The ~2ms right at the top belongs in the 8ms wait, it disappears completely.

I just get the feeling the author of this list was trying hard to pad the numbers to make his point, it feels a like a very hand-wavy analysis masquerading as a proper accounting of latency.

<comment><USER27> I had a similar discussion with a colleague a few weeks ago. They were double counting latency in different pipeline stages when a delay in one stage causes high latency in the other stage.

My analysis would be something like this (assuming 60 fps): 1\. Input latency - variable from hardware 2\. Input event queuing - avg 8 ms (input event arrives in frame X but is handled in frame X+1) 3\. Animation, game simulation, submitting GPU commands, other CPU work - 16.7 ms 4\. GPU work - 16.7 ms 5\. Display controller scanning out frame - 16.7 ms 6\. Display panel latency - < 10 ms?

Steps 3 (specifically the submitting GPU commands part) and 4 can happen within one frame interval so you can be double buffered in that case. Otherwise you need triple buffering to maintain 60 fps. If you have a window compositor, then you need to account for when that latches and releases the buffer you submitted and if it performs copies or steals your buffer. This is the part that's especially murky. Games have it easy with fullscreen non compositing mode.

<comment><USER26> An interesting aspect of this is that for hand-eye coordination tasks, 60Hz provides a perceptual smoothness advantage over 30Hz. There's some benefit to having a higher frame rate even if it costs some latency. I don't know where the trade off point is, but it's not always better to reduce latency if it means a lower frame rate. It probably is always better to have both low latency and high frame rate, but that's hard.

As far as the latency analysis, I'd probably just lump display latency all in one and not try to break it down further. There's some number of ms it takes from the GPU output until pixels are on-screen. Consumer TVs are worse than computer LCDs because the TVs now are doing all kinds of filtering. OTOH, consumer TVs are also trending toward 240Hz refresh, so there is a counter- force causing latency to go down too.

Someone else here mentioned there are two kinds of triple buffering, which I didn't know, but FWIW I was talking about the 3-frames of latency kind. There's also a 2 frames of latency kind, and it sounds like that's the kind you're talking about?

Anyway, I tend to think about game latency as simply 1, 2 or 3 frames of latency, depending solely on how much buffering is going on. That explains almost all game latency, and the games are bad because they don't run at 60Hz, tons of games run at 15-30Hz with double or triple buffering, so the latency is automatically bad. No other sources of latency are needed to explain why. There is the 1/2 frame of latency between input and when the system polls the input and recognizes it, that's fair. So a typical double-buffered game has 2.5 frames of latency, and then add on the display device latency, and that's the sum total. It doesn't need to be made to look more complicated than that, IMO.

</comment></comment></comment><comment><USER21> If you care about end-to-end latency, I highly recommend this talk by John Carmack:

[https://www.youtube.com/watch?v=lHLpKzUxjGk](https://www.youtube.com/watch?v=lHLpKzUxjGk)

This talk blew my mind and made me feel like a terrible engineer. He's talking about end-to-end latency in VR, which actually has a commercial motivation because VR products with high latency will make you sick. (this obviously doesn't happen with shells and terminals!)

He's talking about the buffering and filtering at every step of the way. And it's not just software -- sensors have their own controllers to do filtering, which requires buffering, before your OS kernel can even SEE the the first byte, let alone user space. On the other side, display devices and audio output devices also do nontrivial processing after you've sent them your data.

It's an integrated hardware/software problem and Carmack is great at explaining it! It's a dense, long-ish talk, but worth it.

</comment><comment><USER28> One other thing to note is that compositors seem to add a fairly large amount of latency. I ran the app linked in the "Typing with Pleasure" post and I saw a roughly ~20ms improvement across various text editors with the compositor turned off (I'm using Termite with Compton as my compositor).

[http://imgur.com/0G3qbpr](http://imgur.com/0G3qbpr)

</comment><comment><USER29> Fun test to check how fast the terminal can handle loads of text, run "time head -c 1000000 /dev/urandom"

On a MacBookPro11,1 - 2,8 GHz this shows:

iTerm2: 2.182 total

iTerm2 with tmux: 0.860 total

terminal.app: 0.135 total

terminal.app with tmux: 0.910 total

Surprisingly, iTerm2 is faster with tmux than terminal.app is with tmux. But terminal.app without tmux is the fastest.

Anyone knows why the performance with tmux is so different between the both terminals?

<comment><USER30> My iTerm2 is taking 17s vs. 0.3s for Terminal.app. Any idea why yours is an order of magnitude faster?

<comment><USER29> Hm, have the same CPU? Was tested with build 3.0.15

</comment></comment></comment><comment><USER31> It'd be cool if someone would replicate this on Linux under X11 and a few Wayland compositors, and throw urxvt, rxvt, xterm, aterm, and some others in the mix.

<comment><USER32> And alacritty ([https://github.com/jwilm/alacritty](https://github.com/jwilm/alacritty)) would be interesting to see as well.

</comment></comment><comment><USER33> The common LibVTE-based terminals have problems with latency because they're deliberately capped at 40fps. Xterm doesn't have this problem.

Gedit gained this problem when it switched to GTK3. The forced animation for scrolling doesn't help. Mousepad still has low latency, at least in the version included with Debian Unstable, but I worry that port of XFCE to GTK3 will make it as bad as GNOME.

</comment><comment><USER34> So, on the other side, anyone want to build a true 'terminal emulator' that has baud-speed emulation?

top just doesn't look the same without the changes trickling down the screen, matrix like..

Thankfully I can run GlassTTY font connected to the KVM serial console for a near approximation.. but it's still too fast :)

Grew up in the VC/GUI transition era, but buying a vt220 and running it at 19200 on a used Vax taught me a Zen of command line that nothing else could... Not only did you have to think about what the command would do, but also how much text it would display, and whether you'd need to reboot the terminal after it got hosed up...

<comment><USER35> > So, on the other side, anyone want to build a true 'terminal emulator' that > has baud-speed emulation?

So this isn't exactly the same, but I improved my Vim muscle memory considerably by running the MS-DOS version of Vim inside of DOSBox, reducing its speed of emulation to the lowest option, and finding the most efficient ways to edit files at a decent pace by making use of the best command sequences possible at any given point in time.

</comment></comment><comment><USER36> Interesting article, but I don't quite get it.

I'd imagine that terminal users will often be looking at the last bit of output as they type, and hardly looking at the thing they're typing at all (one glance before hitting Return). They aren't going to notice a bit of latency. And terminals are often used to communicate over networks that introduce a lot more latency than any of the measurements here.

I think, for me, this is a bit like the sales pitch for Alacritty -- fastest terminal ever as long as you don't mind that it doesn't scroll. Someone is using their terminal very differently from the way I use mine.

<comment><USER37> It depends how you use a terminal. If you type a lot of commands, and you use emacs keys to jump around within the current line you're typing (like ctrl-p to go up to the previously entered command, ctrl-a to jump to the beginning of the line, replace the first word, etc.), those latencies add up.

Basically we want the stuff in our brains to be able to manifest itself as fast as we're thinking it. Already having to drive human hands to make this happen is a big penalty; we don't need unnecessary extra latency if it can be helped.

Still waiting for that neural interface... plug me in please.

</comment></comment><comment><USER38> I found this note in the appendix interesting with respect to why we don't seem to notice this latency

> Terminals were fullscreened before running tests. This affects test results, > and resizing the terminal windows can and does significantly change > performance (e.g., it's possible to get hyper to be slower than iterm2 by > changing the window size while holding everything else constant).

Perhaps we don't notice because it's so much lower at window sizes much less than fullscreen?

</comment><comment><USER39> Slightly off-topic, but this reminded me of a mystery: Does anyone else experience that bash takes a long and variable time to start, on many systems and without any fancy setup of any kind? What can a shell be doing that it takes three or four seconds to start?

<comment><USER40> Is part of this an experience of cygwin? Forking is slow under cygwin. If you can get to a recent Windows 10 with local admin rights, you can install windows-services for linux, and bash should perform well even on ten-year-old hardware.

If you getting unusual slowness in linux/BSD, some things to check: (1) any significant overhead on the system that could be making forking slow; (2) some option in your $HOME/.bashrc that is adding latency, e.g., indexing options or dynamic stuff in your PS1; (3) unusual filesystem stuff, e.g., your home directory is mounted over NFS.

If you're in linux and it is reproducible, run strace against a bash launch, and ctrl+c it when it gets stuck. Have a look at the recent system calls.

<comment><USER39> This is on Mac OS. I might try the tracing idea though.

</comment></comment></comment><comment><USER21> How did he actually measure the latency in this article?

Doesn't measuring keypresses to display latency require special hardware?

 _All tests were done on a dual core 2.6GHz 13" Mid-2014 Macbook pro. The machine has 16GB of RAM and a 2560x1600 screen. The OS X version was 10.12.5. Some tests were done in Linux (Lubuntu 16.04) to get a comparison between macOS and Linux. 10k keypresses were for each latency measurements._

 _Latency measurements were done with the . key and throughput was done with default base32 output, which is all plain ASCII text. This is significant. For example, terminal.app appears to slow down when outputting non-latin unicode characters._

</comment><comment><USER41> It would also be great to see how these scale with terminal size. I personally use iterm2 but after switching to a new Macbook pro this year with two 5k displays it's noticeably slower. I'm assuming some O(n^2) scaling behind the scenes but I haven't measured anything myself. Still, @gnachman I love your term especially with terminalplots.jl and drawing Julia repl images inline.

</comment><comment><USER42> I want to try Alacritty on OSX - but the big turnoff for me is the lack of binaries.

However, I don't know if that's intentional, because they don't think it's ready yet for people who won't install/compile the whole Rust stack from scratch?

<comment><USER43> > Precompiled binaries will eventually be made available on supported > platforms. This is minimally blocked on a stable config format. For now, > Alacritty must be built from source.

[https://github.com/jwilm/alacritty#about](https://github.com/jwilm/alacritty#about)

</comment></comment><comment><USER44> I guess this isn't measuring end-to-end latency which would be discretized in units of frames and would have a constant overhead from the display pipeline. I wonder if the differences between terminals would look much smaller if measured that way.

</comment><comment><USER45> I'm surprised Hyper did as well as it did on the latency test, after all it does run in Electron, which I'd expect to add a lot of overhead between keypress and text display

<comment><USER19> You should expect latency similar to a web app in Chrome, just that this has a backing pty processing the input and output.

</comment></comment><comment><USER46> > even the three year old hand-me-down laptop I'm using has 16GB of RAM

Oh my.. I wish that was the case. Even current development machines tend to have just 8 around here..

<comment><USER36> I have a (three-year-old) laptop with 16GB at work and another (three-year- old) laptop with 8GB here, and speaking as a C++-etc developer, I never notice any difference. Maybe if you use a substantial number of VM environments.

</comment></comment><comment><USER47> Interesting to see that both alacritty and Terminal.app are very fast but running tmux inside them kills performance.

<comment><USER48> I've never once worried or felt bothered by performance issues while running iTerm2, but I have in Terminal.app and I'm a heavy/complex tmux user.

</comment></comment><comment><USER49> I wonder how much performance hit font antialiasing in iterm2 causes, or if it was turned on during these tests.

<comment><USER50> font antialiasing on the original OS X completely destroyed performance in the terminal and gave the impression that that the whole system was slow.

</comment></comment><comment><USER51> How can st be so slow? It's tiny and hardly does anything!

<comment><USER44> Maybe it's doing hardly anything one character at a time.

</comment></comment><comment><USER52> Latency is why I could never use a WYSIWYG word processor. While I don't like VI that much, it's latency is low enough that it's not a problem. i.e. I press a key, and miracle of miracles, _the character appears on the screen_.

Using a WYSIWYG word processor, there's enough latency between keypress and visual update that I find them impossible to use.

When Apple came out with Pages, it was apparent that they paid strong attention to latency. That means the latency is small enough that (for me) using it isn't an exercise in frustration.

<comment><USER53> > I press a key, and miracle of miracles, the character appears on the screen.

As long as you press "i" first! :)</comment></comment>


Meshroom – Open-source photogrammetry software - <AUTHOR> https://alicevision.org/#meshroom <comment><USER2> The singer from UK indie pop (?) band Everything Everything self-studied over three weeks this app along with Blender and some othe stuff to make a very stylised, glitchy music video the other week.

[https://youtu.be/mcWwGBHa24g](https://youtu.be/mcWwGBHa24g)

He posted "Thanks for the help, guys!" messages in several Reddit forums. It was weird. "Wait, that's your band?" "Yeah, I'm the singer." "Oh, you sound great keep going!" These guys have sold out rather large venues in the UK. Anyway, this seems to have turned in to an endorsement of the band. Ok, yes it is. But I swear this isn't the singer again.

</comment><comment><USER3> For those with laptops or who don't have Nvidia GPUs, Meshroom can be used for free in the cloud using Google colab [https://github.com/alicevision/meshroom/wiki/Meshroom-in- Goo...](https://github.com/alicevision/meshroom/wiki/Meshroom-in-Google- Colab-\(cloud\)). I created this scan using this technique and I only have an integrated Intel GPU: [https://sketchfab.com/3d-models/parvati-cantor-arts- center-a...](https://sketchfab.com/3d-models/parvati-cantor-arts-center-at- stanford-ab1407479bee4259a267030350473b1b)

<comment><USER4> It looks great, but weighing in at almost _400K triangles and 300K vertices_ though, so it's going to be difficult to do anything practical with a model that size.

<comment><USER3> Blender's decimate modifier can reduce the polys. Infact the model I got from Meshroom had more vertices than the one I uploaded to sketchfab. [https://docs.blender.org/manual/en/latest/modeling/modifiers...](https://docs.blender.org/manual/en/latest/modeling/modifiers/generate/decimate.html)

</comment></comment></comment><comment><USER5> If anyone would like to see a Meshroom scan, here's a fire hydrant I 3d scanned in Downtown Portland, Oregon: [https://socialhome.network/media/uploads/2020/04/30/94efe3d3...](https://socialhome.network/media/uploads/2020/04/30/94efe3d3-41fe-434a-873d-488bebe8e1c7.gif)

I photographed it on my iPhone 8+ and used Meshroom to generate the mesh. The turntable animation was done in Blender.

</comment><comment><USER6> I'd be very interested in a comparison Meshroom vs. RealityCapture (which seems to become the standard in non-free photogrammetry).

<comment><USER7> Agisoft Metashape is pretty good as well. Results are comparable to RealityCapture (depending a bit on what you want) but personal license is only 179$.

I tried all sorts of free photogrammetry softwares way back but after I tried Metashape I decided to buy a license. With 32 GB of memory it's great. I'm taking photogrammetry models as a hobby - here's a recent one made with Metashape [https://skfb.ly/6S6NR](https://skfb.ly/6S6NR)

<comment><USER8> Impressive. How much time did you invest for that model?

<comment> ur-whale Don't want to speak for the OP, but that's typically the kind of quality I get directly out of Meshroom with no work other than throwing the pics in it and waiting ~ 4 hours (less if you have a fast, modern GPU)

<comment><USER7> "with no work other than throwing the pics in it and waiting ~ 4 hours (less if you have a fast, modern GPU)"

Personally I find taking the photos (~1000) is more work than actually waiting the software to complete the processing. Sure, the processing takes longer, but it's not as if you are turning the crank on the PC the whole time :)

</comment></comment></comment></comment></comment><comment><USER9> Was this always open source? Somehow in my photogrammetry journey I've heard of this and not tried it. I guess I assumed it was proprietary. Anyone have any comments on how this compares to COLMAP? Looks like better UI at least.

Also anyone interested in the subject check out [http://reddit.com/r/photogrammetry](http://reddit.com/r/photogrammetry) and [http://reddit.com/r/3dscanning](http://reddit.com/r/3dscanning)

<comment><USER10> The install process for Alicevision and Meshroom is a pain in the ass, I basically gave up at some point, would love to hear some input from others though. I'm currently using OpenSFM and it works great, just very slow.

<comment><USER9> Huh. I downloaded the binaries from their website and they run directly on my Ubuntu 18.04 machine with Cuda 10.2. Which is rare.

<comment><USER10> They have binaries? I must have missed that, the only install instructions I saw were for building from source.

<comment><USER9> Yep and it's using the GPU i just noticed.

Binaries are linked here: [https://alicevision.org/#meshroom](https://alicevision.org/#meshroom)

It's really nice! I read more about the project and it's apparently always been open source. It looks like a high quality project run by some researchers with university and EU backing. I've been trying to reconstruct large outdoor environments for robotics simulation and this might be just what I need!

<comment><USER10> Nice, I'm working on something similar but mostly indoor. Are you using SLAM for the robot itself?

<comment><USER9> Nice. I don't have a clear plan for localization yet. Right now I am working on deep net based trail following. [1] Eventually I will probably do some kind of vision based localization. [2]

If you can share info on your project I'd love to see it!

[1] [https://reboot.love/t/new-cameras-on- rover/277/4](https://reboot.love/t/new-cameras-on-rover/277/4)

[2] [https://github.com/ethz-asl/hfnet](https://github.com/ethz-asl/hfnet)

<comment><USER10> I can't unfortunately, work stuff, but definitely check out this if you haven't already [https://github.com/xdspacelab/openvslam](https://github.com/xdspacelab/openvslam) Also that ethz-asl repo contains a library called libpointmatcher, its pretty awesome if you ever need to do point cloud alignment

<comment><USER9> Thank you! I am concerned that traditional vslam techniques will fail outdoors in off road areas, but that does look nice.

<comment><USER10> I actually have had really good success with it outdoors, much better than indoor. Worth trying out at least, very easy to get started with if you can use docker

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> Fun software, easy to use. Was able to scan my kid's stuffed animal and reproduce a small 3d-printed version of it.

</comment><comment><USER12> I'm interested to try it.

I've been watching people do photogrammetry for a while now, but the results have seemed really bad. When you remove the texture, the underlying model always seems off. This is important for doing object->mesh->modify->3D-print workflows.

This implementation looks a bit better than the rest.

<comment><USER13> I have an iirc ~5M vertices mesh with vertex colors. In case you're interested.

</comment></comment><comment><USER14> They ever get around to HIPifying the CUDA bits?

<comment><USER15> Looks like they're still on it, but there was some progress on it in november 2019, and it looks like it works, although not the default yet, not even at compile-time: [https://github.com/alicevision/meshroom/issues/595#issuecomm...](https://github.com/alicevision/meshroom/issues/595#issuecomment-554623654)

I very much second this, as I hate relying on proprietary software of any sort for stuff that's essential, has a steep learning curve, or that I am going to use on a regular basis (plus other criteria). GPU drivers match the first and last point.

</comment></comment><comment><USER16> There's a free photogrammetry app in the iOS App Store called turn3d. It's a bit rough around the edges but does the job

</comment><comment><USER17> It seems like this has good promise, but scale is random on the output.

I've been looking for a good way to make a customized face mask using photogrammetry and my 3d printer, so I'm going to give this a shot. The only issue will be scaling the model so that my head is the right size for the modeling process.

<comment><USER10> The scale issue is a problem with any software like this unfortunately, it makes doing this type of processing in chunks very difficult as you are getting different scale outputs and nothing really relating the chunks together. OpenSFM does some neat stuff where they use location data and just search for matches between images near that location.

<comment><USER18> Dumb question: could you not simply include an object in the scene that is of known dimensions and use that to calibrate scale?

<comment><USER19> Yes, you definitely could? Not sure for this piece of software specifically though. But for land-based photogrammetry in general, they usually use known control-points that have fixed 3d coordinates attached to them in order to calibrate their models. They also use that to effectively "tie" the resulting model to an actual location on the map.

</comment></comment></comment></comment><comment><USER20> If you don't want to mess with setup etc check out [https://get.display.land/](https://get.display.land/) which will let you download and do whatever you want with the resulting models.

</comment><comment><USER21> CUDA-only :(

<comment><USER22> It seems that there is no photogrammetry software that supports Linux and Radeon at same time.

<comment><USER9> COLMAP can do CPU-only calculation. It takes a big longer but it's not horrible.

<comment><USER23> COLMAP is generally underrated IMO. It's what I primarily use because it's very flexible for different pipelines, it produces one of the best SFM results as a starting place, and it's open source with direct access to the database and decent documentation.

</comment></comment></comment></comment><comment><USER24> Has anyone tried it? I've been trying for a while to find a way to 3D scans stuff to 3D print but the quality of the final file has always been so crappy that I didn't even want to waste time and filament printing it

<comment><USER25> What have you tried that's giving you the crappy file?

I've used Scandy Pro[1] a few times and was surprised by it's resolution and ease-of-use. Only downsides are it's iPhone only and you get 1 free scan/24 hrs.

1- [https://www.scandy.co/apps/scandy-pro](https://www.scandy.co/apps/scandy- pro)

<comment><USER24> I've gotten ok results with large objects like a chair. But for small trinkets like a figurine or something, I get almost no depth detail, almost like I smoothed the surface.

</comment></comment></comment><comment><USER26> Great software.

Now if only they can start to catch up with Reality Capture in terms of speed...

</comment><comment><USER27> Related from 2019: [https://news.ycombinator.com/item?id=19684342](https://news.ycombinator.com/item?id=19684342)

</comment><comment><USER28> Is it possible to do this programatically?

</comment><comment><USER26> Odd that the title didn't use the word "photogrammetry" which is what most people call this thing.

<comment><AUTHOR> It's what they call it on their homepage. I didn't want to override that, even though I would've also called it photogrammetry.

<comment><USER29> I see "OPEN SOURCE PHOTOGRAMMETRY SOFTWARE" above the fold when I open up the site.

<comment><AUTHOR> I copied the title that was right in the middle of the "hero". Moderators feel free to add "photogrammetry" to title.

<comment><USER27> Added.

</comment></comment></comment></comment></comment><comment><USER30> No Mac = No go for me.

<comment> ur-whale vmware player

<comment><USER30> not gonna work, look at the requirements.</comment></comment></comment>


How a Russian revolutionary foresaw the rise of Japan - <AUTHOR> https://rbth.com/blogs/continental_drift/2016/05/10/how-a-russian-revolutionary-foresaw-the-rise-of-japan_591713 <comment><USER2> He was in that respect is a representative of his movement (Narodniks). Russian revolutionary movement in the mid 19th century consisted of 2 main parts - Zapadniks (Westernists) who was emphasizing technical aspects of government/society/economy organization and transition and Narodniks [https://en.wikipedia.org/wiki/Narodniks](https://en.wikipedia.org/wiki/Narodniks) who was more about organic transition through build-up from the bottom, through enlightenment of the regular, poor, people. Narodniks almost wanted to see what was happening in Japan as a kind of such revolution.

The Narodniks were also pretty close or overlapped on many issues with Slavophils - where ideas were of building up upon supposedly great intrinsic national qualities of Russian/Slavic people (again in many cases it was in opposition to the idea of following Western civilization). Such principle - only replacing Russian/Slavic with Japanese - was also very close to the corresponding political movement in Japan then.

</comment><comment><USER3> Funny how both Russians and Americans sent out gunboats to "open up" a can of whoop ass on themselves and bystanders.

Hindsight, blah blah. The story of an improbable timeline where Japan was left alone would make a fine alternate history book.

</comment><comment><USER4> What often goes unmentioned is that Japan had endured famine and economic upheaval during the 1830s. Domestic reforms had already been enacted ten years prior to the black ships in an effort to control a nascent market economy and the novel problem of high inflation. Perry's actions, while less than noble, helped provide a path out of their predicament by fostering external trade without the restrictions of the preceding 250 years.

<comment><USER5> The Geotaxers also credit Henry George style land taxation for much of the explosive growth during the Meiji period.

</comment></comment><comment><USER6> The question is how the Japanese "mutual aid" societal philosophy came about and actually lead to societal changes and development. Was it just part of the east asian Confucian ethics or something else? Figuring this out could be useful for many parts of the world wishing to progress from their current state.

</comment><comment><USER7> If you're not my Caste, you're non-human to me; [https://en.wikipedia.org/wiki/Nanking_Massacre](https://en.wikipedia.org/wiki/Nanking_Massacre)

</comment><comment><USER8> While this article seems to be rather accurate, there is too much revisionism around Japanese history and the international community is heavily biased when it comes to them.

</comment><comment><USER9> Anarchists seem to be pretty good at predicting geopolitical events in general</comment>


CyanogenMod domain situation has been resolved - <AUTHOR> http://www.cyanogenmod.org/blog/domain-situation-has-been-resolved

<comment><USER2> The CyanogenMod team has shown some amazing gentlemanly honor, mixed with a worthy dash of can-do pragmatism, and I hoist a glass o' the pure in their direction.

</comment><comment><USER3> apparently from the guy who managed the domain:

[https://plus.google.com/116028512018932696380/posts/4gDX3Hyg...](https://plus.google.com/116028512018932696380/posts/4gDX3HygM3E)

<comment><USER4> > "I purchased the CyanogenMod.com domain name 3 years ago and have been > responsible for the renewal fees ever since"

What's not clear is whether he purchased it (through a broker or private sale) or whether he "purchased" it in the sense he just registered it with a registrar. If the latter, he could not have been up for more than $60 ($30 is more likely).

The way he writes it makes it seem like the renewal fees ("... ever since") were a big deal and that he was doing them a huge favor by paying for it.

</comment></comment><comment><USER5> Fantastic to hear that this was resolved so quickly. I'm never left unimpressed by what a simple blog post and it's audience can accomplish.

<comment><USER6> And if we are to believe the dude who created the problem, the swarm of hate on his personal twitter & email accounts.

</comment></comment><comment><USER7> Lesson learned: Don't trust anybody with your domain name registration, it's your brand's front door.

</comment><comment><USER8> The comments on that post are horrible.

</comment><comment><USER9> There is so much wrong with the other guy's story [1] that it's hard to even know where to begin.

1\. No mention of the 10K extortion attempt.

2\. No mention of the "big deal" (fraud) perpetrated by impersonating Steve and or the CM team to negotiate deals that netted the guy (a guesstimated) $8K.

3\. The Registrar changing the NSs of a locked domain is either baloney, or a good reason to never use that registrar again. That would make it impossible to do a zero downtime domain xfer.

4\. Even on twitter his story was inconsistent or half full of him lashing out at people.

I've been told that the team is not pursuing anything, though PayPal, N2A and... what was the other big one... all have been contacted and told what happened (see #2).

Were there ever "the days" when people would say "I'm sorry" without caveats or excuses and mean it 100%? At best, he apologizes to anyone he "offended". I guess he offended my sense of decency, and offended my sensibilities given how to handle cases where you screwed up. Between his Twitter and this "summary", I would say the shovel is still in his hands. I wonder if he paused for breath or realized how big the hole was.

[1] [https://plus.google.com/116028512018932696380/posts/4gDX3Hyg...](https://plus.google.com/116028512018932696380/posts/4gDX3HygM3E)

<comment><USER10> According to ArsTechnica:

"No money has ever been extorted from CM nor will it ever be," Deveci told Ars. Ahmet claims that Kondik once asked him how much he would be willing to sell the domain for. "I replied as a joke and said $10K. He didn't even reply after that, he went offline, and simply started changing all my logins access and passwords."

[http://arstechnica.com/gadgets/2012/11/owner-of- cyanogenmod-...](http://arstechnica.com/gadgets/2012/11/owner-of-cyanogenmod- com-hijacks-domain-demands-10000/)

<comment><USER11> Either Deveci is trying to hide the fact that he attempted extortion, or he has no sense of proportion. $10,000 is exactly the wrong amount for a joke. $1 would be silly. $100 would be reasonable. $1 million would be a joke. $10,000 is extortion.</comment></comment></comment>


Thunderbird 38 Release Notes - <AUTHOR> https://www.mozilla.org/en-US/thunderbird/38.0.1/releasenotes/

<comment><USER2> Big news in this release is that the "lightning" calendar extension is now bundled together. I've been using it for a while now, since my department uses a CalDav calendar server. Like the rest of Thunderbird, I can make it do what I want.

I use gmail's web interface for personal mail but I'll never give up Thunderbird for work. It's mature and sane.

<comment><USER3> I really thought that this shipped as standard for a long time now. But then again, it's been too long since I've used my Thunderbird install. I'm mainly using webmail and my phone mail app.

</comment></comment><comment><USER4> I thought Thunderbird development was announced [1] to have stopped near 3 years go.

Am I mistaken?

[1] [http://techcrunch.com/2012/07/06/so-thats-it-for- thunderbird...](http://techcrunch.com/2012/07/06/so-thats-it-for-thunderbird/)

<comment><USER5> Thunderbird has been maintained with security fixes since then. And the underlying engine, Gecko, stays pace with Firefox ESR, so that it can stay up to date with security fixes as well. Firefox ESR just moved to 38.0 so Thunderbird does now as well.

There are no new major features or anything coming to Thunderbird, just bug fixes and security updates. So, it's still the best email client.

</comment></comment><comment><USER6> I hope they've fixed the missing menu bar problem on linux, I've been hobbling around without the proper menus for a month or two now. The simplified one is missing a few things I use often, like message box search.

I've been relying on these Moz products and their ancestors for twenty years now and not a big fan of their recent regressions, e.g. where they decide to remove widgets to compete with Chrome. With this menu bar problem, it's not yet clear to me if it is a bug, or by design.

I could go on. :/

</comment><comment><USER7> Other big news in this release is the shift (at least for a few releases) to match Firefox release numbers, so it's jumping now from 31.7.0 up to 38.0.1. They note on the Release Notes that "There was no Thunderbird 38.0 release." but neglect to include that there was also no 37, 36, 35, 34, 33 or 32 release.

I'm just waiting for the naming convention to change over to randomly-chosen members of Canidae so we can be more like Apple.

<comment><USER8> > Other big news in this release is the shift ... to match Firefox release > numbers, so it's jumping now from 31.7.0 up to 38.0.1.

Apparently you've been out of touch for the last several years. Thunderbird has used Firefox-like ESR branch numbering for shipping releases since version 17.

<comment><USER7> Clearly I need to watch this better.

</comment></comment></comment><comment><USER9> I'm excited to try out the maildir support. I've been looking forward to this coming out.

I don't really understand the Thunderbird to IceDove flow, I wonder when these features will make it to Debian unstable/testing?

</comment><comment><USER10> The OS X .dmg seems to be damaged. I get a 'no mountable file systems' error.

<comment><USER10> "The release is still in process", according to Kent James (via IRC), which may explain why the .dmg doesn't work.

</comment></comment><comment> f-hack Hope this release got the SPECIAL-USE implemented, but could not find anywhere.</comment>


Ask HN: How can a skilless person make money online? - <AUTHOR> Given a person with very few marketable skills, how can this person make money online?<p>There are services such as Amazon's mechanical turk that fit the bill, though the given example usually nets less than $8 an hour. <comment><USER2> Why not develop at least one skill to a more marketable level? The amount you earn is generally related to the value of your skill. Developing a rare skill is generally more lucrative than developing a more broadly available one. For example a food delivery person earns less than a high-rise building window washer. Good sales people often out earn programmers. NB: the accent is on "good" as opposed to average, which applies to many fields of work.</comment>


Betterment adds charitable giving - <AUTHOR> https://www.betterment.com/resources/inside-betterment/product-news/charitable-stock-donation/ <comment><AUTHOR> Coinbase should follow suit and allow people to donate those massive long term unrealized capital gains they have accrued.</comment>


The road to my first sale - <AUTHOR> http://propagandaposterstore.com/the-road-to-my-first-sale/

<comment><USER2> Yes, really interesting! And Good Luck!!

</comment><comment><AUTHOR> Thanks Joey, I appreciate the comment. I hope this can be of use to you in the future.

</comment><comment><USER3> Interesting read, thank you for sharing. Some really useful information in there.</comment>


Amazon says browser extension Honey is a security risk, now that PayPal owns it - <AUTHOR> https://www.theverge.com/2020/1/9/21059083/amazon-honey-browser-extension-security-risk-paypal-acquisition-competition <comment><USER2> In other Amazon press releases:

1\. Disabling Amazon's deep integration into Ubuntu's desktop search, is a security risk.

2\. Android tablets that are not Amazon Fire are security risks

3\. The reall MongoDb, if self hosted or hosted by Mongo/Atlas is a security risk, now that AWS provides it's own managed version

4\. Using the "one-click" patented workflow on any other site than Amazon, is a security risk

</comment><comment><USER3> As a sometimes paranoid person, the huge list of permissions that Honey asked for as a browser extension has always smelled to me like a privacy/security risk, regardless of owner.

 _Maybe_ the timing is suspicious on Amazon's part, but it does seem like a useful PSA as worded.

<comment><USER4> They make their money from commissions, not ads or personal data.

They've been audited by at least one security firm per the article, and their privacy policy [https://www.joinhoney.com/privacy](https://www.joinhoney.com/privacy) says "We do not sell your personal information. Ever."

Anyway they only have one permission on firefox and the usage for that permission Mozilla mentions is exactly what they do. [https://support.mozilla.org/en-US/kb/permission-request- mess...](https://support.mozilla.org/en-US/kb/permission-request-messages- firefox-extensions?as=u&utm_source=inproduct#w_access-your-data-for-all- websites)

<comment><USER3> I only saw the Chrome permissions because Edge shares the same permission model (even pre-Edgmium Edge / "Edge Classic"). I also may have seen it early in its history as well, as I recall looking at it on a "Is this spyware?" ask from a presumably very early adopter.

That said, even if it is the example use case for that lone Firefox permission, that's a hugely broad permission and I'd be hesitant with any extension that asked for it.

As for security audits and privacy policies, I'd be concerned if they didn't do their diligence on that front. It doesn't impact my paranoid skepticism of a startup one bad/dumb pivot away from changing their minds and injecting ads or selling personal data because their business model wasn't working. At least on that side of the equation, PayPal buying them does possibly increase some trust measures with the company as it should be less likely that PayPal would allow such a pivot. (Though PayPal themselves don't have a history of being the best stewards of their ancillary products, and healthy skepticism there abounds as well.)</comment></comment></comment>


Investors poured millions into a storage network that doesn't exist yet - <AUTHOR> https://arstechnica.com/information-technology/2017/08/investors-poured-millions-into-a-storage-network-that-doesnt-exist/ <comment><AUTHOR> The article is a little tough on Protocol Labs. I'm pretty sure it's the same team behind IPFS so they have a pretty good track record of delivery. The idea of Filecoin has been floating around for years and the dev team is a serious one. Of all the ICO's, this is one I'm not too skeptical of.

<comment><USER2> So serious but the code that was posted on github hasn't been updated in 4 years. I don't see this going well, at least for the community.</comment></comment>


Moore's Law 2.0 - <AUTHOR> http://gigaom.com/2007/03/08/moores-law-20/ <comment><USER2> Moore's law seems to die and rise from the ashes every 18 months or so.

<comment><USER3> Yeah, the times that Moore's Law has "become irrelevant" are usually the same times that it has become as "applicable as ever." People just love to announce!</comment></comment>


Walking and rolling of crystals induced thermally by phase transition - <AUTHOR> https://www.nature.com/articles/s41467-017-02549-2 <comment><USER2> For whatever reason, this reminded me of the Cheela in Bob Forward's Dragon's Egg. [https://en.wikipedia.org/wiki/Dragon%27s_Egg](https://en.wikipedia.org/wiki/Dragon%27s_Egg)

</comment><comment><USER3> Seems like you need temperatures of about ~140C (280F) to make this work.

Unfortunately, I think their hope of using these in soft robots may be complicated by the high operating temperature required. Wonder if they can make crystals which exhibit these effects closer to room temperature.

</comment><comment><USER4> This site could use a rule: if the original paper is open, don't post the phys.org pie-eyed futurist clickbait digest of it.

[https://www.nature.com/articles/s41467-017-02549-2](https://www.nature.com/articles/s41467-017-02549-2)

 _Walking and rolling of crystals induced thermally by phase transition_

<comment><USER5> OK, we've updated the link from [https://phys.org/news/2018-02-crystals-field- crystal-robotic...](https://phys.org/news/2018-02-crystals-field-crystal- robotics.html).

<comment><USER6> Well, it made a lot more sense why the original article was exciting. The updated article, with all due respect, is hard to parse for a non-expert.</comment></comment></comment>


A nonpartisan, independent review concludes U.S. Engaged in Torture After 9/11 - <AUTHOR> http://detaineetaskforce.org/

<comment><USER2> From the "no shit, Sherlock" files?

I mean, I don't mean to be flippant - or perhaps I do, it's late - but I genuinely didn't think there was still that much debate on this.

It's not like the Bush administration denies that they engaged in torture, they merely play semantic games by redefining the torture they engaged in as not-torture. Now we have some distance and are able to remind ourselves that this not-torture is actually, ya know, torture, then there's not really much left to debate is there?

That is actually a question, because I'd love to hear responses from folks who haven't internalized that the US did indeed engage in torture - and I am not attacking you for it if that's the case. I've possibly been laboring under a wildly inaccurate assumption.

<comment><USER3> "I genuinely didn't think there was still that much debate on this"

You'd be surprised. Many people will deny anything happens until CNN/NBC/Fox/Insert-Big-Media-Company-Here tells them otherwise. Because of that, we need more things like this, more sources of information we can point to and say "do you believe us now?"

Also, any time an official refuses to look into the past, in this case because it's not productive to "look backward", that's a pretty sure sign that somebody is hiding something (and this report proves it). No educated person on the planet truthfully believes that knowing and understanding the past is unproductive. Not one.

</comment></comment><comment><USER4> As Obama said, it is not productive to look back at our behavior. I mean, how horrible, what if we look back at what we did and analyze ourselves. "God" forbid, we might feel guilty or responsible. Heck, we might even learn not to repeat our mistakes, or worse prevent them in the future and who would want that to happen.

<comment><USER5> > "God" forbid, we might feel guilty or responsible.

If it's anything like previous immoral incidents give it 30-50 years. After all guilty parties have died off and any claims of restitution undeliverable. Look at slavery, the gov't only recently apologised for that act.

</comment></comment><comment><USER6> Some further reading:

"Report on the alleged use of European countries by the CIA for the transportation and illegal detention of prisoners" prepared by the European Parliament (2009) [1].

"Globalizing Torture: CIA Secret Detention and Extraordinary Rendition" by the Open Society Justice Initiative (Feb 2013) [2]

[1] [http://www.europarl.europa.eu/comparl/tempcom/tdip/final_rep...](http://www.europarl.europa.eu/comparl/tempcom/tdip/final_report_en.pdf) [PDF]

[2] [http://www.opensocietyfoundations.org/reports/globalizing- to...](http://www.opensocietyfoundations.org/reports/globalizing-torture-cia- secret-detention-and-extraordinary-rendition)

</comment><comment><USER7> Quite impressive and disturbing. I am glad to see the report. What I'd really like to see, though, are some indictments.

<comment><USER3> It's been over 10 years since we've had an impeachment. I'd say we're overdue.

</comment></comment><comment><AUTHOR> You can sign up for a free .epub or .mobi version of this report to be available at the end of April.

<http://detaineetaskforce.org/report/ebook/>

</comment><comment><USER8> The report is interesting, but what will be even more interesting is whether or not the US will have the strength to draw conclusions from this report and to hold those responsible accountable.

<comment><USER9> It won't be interesting, as the answer is clear already: _absolutely not_.

Unless you mean in like a hundred years, when all participants are dead.

</comment></comment><comment><USER10> _Much of the torture that occurred in Guantánamo, Afghanistan and Iraq was never explicitly authorized. But the authorization of the CIA's techniques depended on setting aside the traditional legal rules that protected captives. And as retired Marine generals Charles Krulak and Joseph Hoar have said, "any degree of 'flexibility' about torture at the top drops down the chain of command like a stone — the rare exception fast becoming the rule."_ (p12, abridged version)

What are these conversations normally like? Sort of a "Do what you can to get any information out of the suspect as I turn around and plug my ears." ?

I hate how they say this stuff isn't authorized from some high level.

 _"...Lawyers provided novel, if not acrobatic interpretations to allow the mistreatment of prisoners."_ (p13, abridged version)

<comment><USER11> > Sort of a "Do what you can to get any information out of the suspect as I > turn around and plug my ears." ?

I would imagine so. There has to be a non-verbal or symbolic channel of communicating that then provides official deniability.

As an example take discrimination and at-will employment. Let's say the owners want to fire a gay guy. They can't fire him officially because of various discriminatory laws so they maybe invent a non-verbal or a code word 'restructuring' so they lay him off due to restructuring.

In general part of operating in large bureaucratic institution is to learn:

* what rules can be broken (break them if you like but don't force the management to make an official statement on it and don't ask permission on paper)

* what rules should be broken ("you need to torture this guy and it is illegal so we'll just use a euphemism -- 'interrogation' you know what I mean, I know you know what I mean, but we can never write that down in an email")

* what rules cannot be broken, official and un-official (some things are never tolerated what these are have to be discovered carefully as these might not all be written down, say who can disagree with the boss during a requirement meeting... it is not written down but if you are part of the wrong group, you won't fare well)

<comment><USER12> A frequent method of torture by US forces is to threaten to turn a captive over to allied domestic Police or Army forces. It is useful to look at the relevant UN conventions to understand what the legal definition of torture actually is.

</comment></comment></comment><comment><AUTHOR> For people questioning "Why" this investigation was conducted, here's a statement from the task force

        The events examined in this report are unprecedented in U.S. history. In the course of the
        nation's many previous conflicts, there is little doubt that some U.S. personnel committed brutal
        acts against captives, as have armies and governments throughout history.
        But there is no evidence there had ever before been the kind of considered and detailed
        discussions that occurred after September 11, directly involving a president and his top
        advisers on the wisdom, propriety and legality of inflicting pain and torment on some
        detainees in our custody.

</comment><comment><USER13> Where is the big surprise? Is called Guantanamo... and Egypt/Libia/Siria's secret jails. Everybody knows that.

</comment><comment><USER14> Not only did it happen, but it has been made official policy, relabeled as enhanced interrogation. There are very few in Congress or the White House willing to denounce it at this point.</comment>


Ask HN: How often do you Google for answers while coding? - <AUTHOR> <comment><USER2> All the time. Many times per day. I've been programming for fifteen years, and I think my willingness to Google things has only gone up over that time. There are a lot more things that I know off the top of my head now than when I started, but I've got way less tolerance for wasting time trying to figure out weird problems when there's already an answer online.

<comment><USER3> 15+ years here too, I have exactly the same situation. 90% of the things I worked on years ago, I wouldn't bother with now; I would just google. There's only so many times you can make a library to handle 'common stuff' :) Even though Google didn't exist back in 'the day', I would say I look up stuff more and more as time goes on - I always take a look at what's out there before making a decision to get coding.

As cliched as it is: The greater your experience, the more you know how little you know.

OT: Interestingly I just came across my first 'expert sex change .com' link in years a few days ago.. for some ugly and stupid legacy server error, naturally I swore at the site again, and told my colleagues about how much we 'hated on that' in the day.

</comment></comment><comment><USER4> Constantly. Even when I know exactly what to code I like to view alternative examples to see if there is a better way to do it. Even if I know my code is rock solid I like to also filter search results for the past year just to make sure there isn't a newer+better way to do something that used to be best.

</comment><comment><USER5> For me, depends on the level of experience I have with the language/platform/library, ranging from once a month to a dozen times in a day.

</comment><comment><USER6> I've about 7 years experience coding various software in Python (data intensive, web apps, network app, etc). Today when coding in Python I mostly go to the official Python and Django doc directly, that I have installed locally. I open it about 2 to 8 times a day (4-6 hours span) depending on how well I'm familiar with the library I'm dealing with.

For more general questions not covered in the doc I Google about twice to 4 times a day in average. I usually go to stackoverflow.com directly to search.

When coding HTML/CSS/Javascript I clearly lookup caniuse.com about 20 times a day to ensure consistent browser support and learn about corner cases.

</comment><comment><USER7> A lot. Some of my searches from yesterday:

        suppress STDOUT print ruby backticks

        mechanize user agent alias list

        git gc garbage collect source

There is no shame in googling. While I would heavily recommend reading and learning documentation, once you get a passing familiarity (not memorize it) you at least understand what you are using and googling it simply can save time. If you need specifics on the implementation the docs will always be there.

There is a reason we have SO and google is so great at finding things on it. Have no shame :) It doesn't make you a poor programmer to be thinking things out enough you are googling alternatives, how the innards of something works, etc.

Obviously this is all just my opinion, though. I could be a moron and not know it.

<comment><USER8> > While I would heavily recommend reading and learning documentation, once you > get a passing familiarity (not memorize it) you at least understand what you > are using and googling it simply can save time. If you need specifics on the > implementation the docs will always be there.

And these days, the docs are usually online, and usually don't have a great dedicated search engine, so often googling is the best way to find what you are looking for in the docs, anyway.

</comment></comment><comment><USER9> Honestly, I've been programming for almost 5 years now and been learning a whole lot of different technologies, languages, frameworks, etc. since I started so its safe to say I'm barely an intermediate in alot of things. Because 5 years isn't alot of time, I google answers quite often for different reasons. Most of the time, its to figure out how to code something up instead of trying to figure it out myself (I have a general idea how to do it) so that I can get stuff done. Frankly, I'm too lazy to make mistakes. This is pretty bad since I don't really understand why things go wrong most of the time.

</comment><comment><USER10> For me recently, it's constant because I'm working with Spring Boot and ExtJS for the first time. So I'd say how often you're doing web searches is a function of your familiarity with the techs you're using.

</comment><comment><USER11> I use google quite often for searching $LANGUAGE, $PLATFORM, $LIBRARY docs... it's so much faster than using the provided search.

</comment><comment><USER12> For JavaScript and CSS it's a lot of googling. Mostly because they have no standards you can rely on.

For a language like Java, I mostly google to see if what I want to do already exists in an Apache library or is now part of the latest JRE. For well- documented Java frameworks which have JavaDocs attached, googling isn't often necessary.

</comment><comment><USER13> Basically when I don't have an example in my own codebase that I can use as prior art. I always want to find out what obstacles other people have found, as a way of making the process go more smoothly.

Nine times out of ten I end up at either Stack Exchange or W3Schools.

</comment><comment><USER14> I agree with the other answers that it is a function of how familiar I am with the language/framework. But I also like to see what other solutions people have come up with since I know there might always be an edge case I didn't think of.

</comment><comment><USER15> 9 times yesterday, according to my browser history. It was a slow day, though.

</comment><comment><USER16> Dozens of times a day.</comment>


Ask HN: What are your motivations to open source a major project? - <AUTHOR>

Intrigued by individuals' decision trees on deciding whether to commercialize something or open source it.
An example would be Firebase vs Hoodie or Firehose. <comment><USER2> _whether to commercialize something or open source it._

Those are not mutually exclusive positions. Software can be open source (or even free software) and still be commercialized.

In our case, we made all of our products open source for a number of reasons, some ideological and some pragmatic. Whether or not that proves to be a good decision from a financial standpoint remains TBD... but the important thing is, it doesn't matter. If we never make a dime from what we're doing, we'll all have benefited from the process in other ways. And we'll have contributed something to the world that somebody may gain some benefit from one day.

<comment><AUTHOR> True, I misspoke. I meant commercialize it as closed source / proprietary software. But cheers on your work!

Beyond your team, does the community contribute to your product in meaningful ways?

<comment><USER3> There's also the option of a shared source licence. This means providing the source code, but only allowing free non-commercial use. Commercial use would then come under a different licence and would have to be paid for.

Third party contributors whose work is included in commercial sales would then be given a portion of the revenue from those sales.

<comment><USER4> Only allowing non-commercial use is a serious inhibitor of the popularity of the project. More common is to have a AGPL/GPLv3 license that some companies will pay to get rid of. Alternatively use a business friendly license (MIT Expat) and have open core business model where some features are proprietary (like we at GitLab use).

<comment><USER3> A shared source licence still allows commercial use of the software - it just has to be paid for. If it is given away, with or without strings attached, there is no direct way for the developers to be remunerated for their effort. Open source might encourage a few companies to contribute to projects, but that only pays the developers of that software who are employed by those companies.

Private individuals, or non-profit institutions, can still use shared source software. A few might be put off if they think it won't help them to find a job, but even they will think twice if companies start paying to use it.

Having popular software is nice, but it doesn't pay the bills if you can't charge for it.</comment></comment></comment></comment></comment>


Ask HN: Can you help me not confuse the Googlebot? - <AUTHOR>

I have implemented the RSH library for my website www.trailbehind.com.
This lets me change the hash of the URL and store data when someone surfs around on my maps, so that I can provide links within the map and back button functionality.
However, I'm worried that Google and the other search engines aren't going to take kindly to these links.
I realized that Google wouldn't do things like index my page titles and other content, because they are generated when the DOM is ready via javascript.
So, for now what I've done is maintain the hard links. For example, these two URLs generate the same content:
1. http://cabin.trailbehind.com/#get_map_from_name/Yosemite_National_Park
2. http://cabin.trailbehind.com/park_map/Yosemite_National_Park/
I think the problem with this though is that if people link my site, they will end up linking the hash URLs, while Google will only check out the hard links. And thus, people blogging about me won't help my pagerank.
Has anyone ever dealt with anything similar or have thoughts on this issue? Google rankings are very important to TrailBehind, and part of our strategy (which is already working) is getting listed for the long-tail of parks and forests. <comment><USER2> How about redirecting the first URL to the second? Updating the fragment part via JavaScript shouldn't hit the server so that wouldn't be affected by the redirect. If you did get into a situation where it inadvertantly redirects then you could just check the referrer to prevent that.

<comment><USER3> I don't understand what you mean. If you update the location with location.href, the page will reload. I think the poster wanted to avoid reloads, hence the whole point of using rsh.

<comment><USER2> The page doesn't reload when ony the fragment part is changed.</comment></comment></comment>