Change careers like Tarzan - <AUTHOR> http://sivers.org/tarzan

<comment><USER2> Over at his blog, the ever-interesting, never-boring Derek Sivers has compared transitioning between careers with Tarzan, swinging between vines. I wanted to put that under a bit of pressure.

Derek says rather than quitting one career abruptly and moving onto the next, there should be an overlap period. In his own words:

Remember how Tarzan swings through the jungle? He doesn't let go of the previous vine until the next vine is supporting his weight.

Of course, that is great advice. But, at the same time, I worry the Tarzan image might be might be too limiting; I worry the picture it paints of people's working lives might be too simplistic.

In the new online economy, the very concept of the traditional 'career' is possibly dead.

In the 1950s, people used to make the vast majority, if not all, of their annual income from a single job. Just like Tarzan in the animation, you would only ever hold onto one vine at any one time. Moments when you holding onto two were exceptions to the norm, exceptions just like when you were transitioning between careers.

Is this still true today?

Not obviously. These days people are much happier (and able) to juggle different sources of incomes, to wear different hats at different times. Or, put in terms of the Tarzan analogy, people are much happier (and able) to hold onto multiple vines — many, many, many more than two — at any one time.

At one point you might be programming, writing and consulting; at another you might just be writing and consulting; at another you might be running a coffee delivery business, consulting for an e-reading start-up while making a few quid from blogging.

The Tarzan analogy is limiting because it encourages you to think of careers as those things that you have one of, or, at a stretch, two.

When it comes to careers, a better analogy might be a garden, or, even better, a window box.

Sources of income are like plants in this window box. At different times, some plants are smaller, some are bigger; some are growing, some are whithering; some flourish in this weather, and others flourish in that.

But, how can we make sense of career transitions in this new language?

Using this image, transitioning careers is like planting a new seed, alongside all the other plants you have growing, watering that one carefully — but not ignoring the others, maintaining a window box of flourishing flowers that will help you navigate changes in externalities like shifts in the economy and disruptive new technologies.

With this new image we can see another important lesson: You must adjust your skills and expertises to the changing, shifting economy; just like you should put different plants in your window box at different times of the year and in different climates and weathers.

<comment><USER3> Related: Told mother about a friend the other day. "Where does she work?" she asked. "Uh," I explained, "most of her income at the moment is in Patreon donations." Then I had to explain Patreon. "And then she also has some commissioned music and writing work, and one or two other things."

Said friend isn't convinced that this is a stable situation either, but it's the position she's working with. Mother, of course, was left dumbfounded by this bombshell, among others regarding friend.

Mother describes her own employment history as "I took whatever gigs I could get," and this time I asked where she expected that to lead, at the time. Said she thought there would be stable employment but the rise of recorded music everywhere killed most of the market for live gigs. Which, in hindsight, seems so, so obvious. But also speaks to a common ideal that the market discourages now, when expectations are that every job that's already there will be automated away someday.

</comment></comment><comment><USER4> I have always imagined Tarzan letting go of the previous vine and flying forward in free-fall for a second before grasping the next one, sort of like Spider-man.

What's more, if he did hold onto the old vine, and missed the new vine, he's going to swing backward, helplessly out of control and unable to either proceed or retreat.

All in all, not the best visual metaphor. That said, certainly it's good advice and common sense that you carefully plan out your next career more rather than plunge blindly ahead, burning bridges behind you.

But when you know something's right, you just feel it in your bones. The catch is that the money might not be there at the outset. But if you have planned ahead and stashed a year or two's living expenses, then you will have probably enough time to determine whether you have made a good decision or whether you need to go back to what you were doing.

<comment><USER5> _" What's more, if he did hold onto the old vine, and missed the new vine, he's going to swing backward, helplessly out of control and unable to either proceed or retreat."_

Still way better than if he let go of the old vine and missed the new one, although [http://www.tcm.com/mediaroom/video/501600/Tarzan-s-New- York-...](http://www.tcm.com/mediaroom/video/501600/Tarzan-s-New-York- Adventure-Movie-Clip-Boy-Gone.html) shows that is survivable.

<comment><USER4> Interesting point. Kids, don't try that at home!

</comment></comment></comment><comment><USER6> This is good advice.

Employers are increasingly unforgiving of people with gaps in their resumés. There are fewer second chances in this world and it's getting harder to recover from early failures.

<comment><USER7> _Employers are increasingly unforgiving of people with gaps in their resumés. There are fewer second chances in this world and it 's getting harder to recover from early failures._

I can only give this advice because I haven't had to do it (I wouldn't publicly endorse it if I were out doing it) and don't think I ever will: people need to start lying.

As far as I'm concerned, employers who hold gaps or unemployment or "job hopping" or low previous salaries or past political adversity against people-- basically, the dumbfucks who evaluate people based on social-status shit that has nothing to do with getting the job done-- deserve to be lied to. They won't do the right thing unless deceived, so deceive away. I don't lie on my resume (I don't need to, and I'm too much in the public) but I don't think it's unethical to inflate titles, play with dates, misrepresent performance reviews, or otherwise inflate one's status. As long as you're not claiming to be able to do work that you actually can't perform (i.e. actual job fraud as opposed to status inflation) you're OK as far as I'm concerned.

Of course, misrepresenting objective skills and training is different. If you claim to be a doctor and you've never set foot in a medical school, you're putting lives at risk and should be sent to jail. Job fraud is wrong, social status inflation is OK.

Great minds discuss ideas, middling minds discuss events, and small minds discuss people. The problem in business, especially with resume culture, is that everything comes down to discussion of people, because there are so many small-minded people in positions of power. Those who are unemployed for more than 6 months become "dirty" even if they're more than capable. As far as I'm concerned, that's proof of the current system's ethical bankruptcy and a bit of lubrication (harmless deception) to overcome its pernicious inefficiencies is right in order.

<comment> FLUX-YOU >I don't lie on my resume (I don't need to, and I'm too much in the public) but I don't think it's unethical to inflate titles, play with dates, misrepresent performance reviews, or otherwise inflate one's status. As long as you're not claiming to be able to do work that you actually can't perform (i.e. actual job fraud as opposed to status inflation) you're OK as far as I'm concerned.

Especially relevant to programming. Though I do struggle with convincing myself to put some things on my resume.

Example: I spent a year messing with front-end web development. Some days I'd just write a few Javascript functions and try to fix/break stuff, other days I would bang out a small project start-to-finish. Total maximum time per week would probably average to 15 hours.

If my prospects ask for 1 year of experience, it's really unclear what level of time spent and knowledge they're looking for. At 780 hours for that year, I've put in 3/8ths the time of someone who works full-time at (supposedly) 2080 hours per year and wasn't also distracted by a completely unrelated job.

Ethical Me wants to be responsible and not misrepresent. Practical Me says I need the money.

Unlike the doctor example, programmers have a lot of time to research and tinker without making someone bleed. It's difficult to be sure of what I can actually do before I've done it whereas doctors go through many, many tests to verify their skills.

<comment><USER8> Well, think of it this way: 90% chance if they like your inflated resume, they will ask you for a technical interview. If you don't pass that, ok, no harm done, if you do, you might get a job, and they think you will be able to do the job at hand.

I've advocated what Michael is saying before here. I was lucky to never have to do it, but I have been on the recruiting side and didn't care much for some resume inflation on candidates (specially in companies where HR are involved in the process because no one has 5+ years of mongo experience but if they hear you need an experience mongo guy that is what they are going to put in the ad)

</comment></comment></comment></comment><comment><USER9> I think that this is probably good advice, and it's communicated clearly and effectively with a great metaphor.

But... I think it fails because it's ineffective. You have to talk about the problem, not the solution ([http://katgleason.tumblr.com/post/47257463324/talk-about- the...](http://katgleason.tumblr.com/post/47257463324/talk-about-the-problem- not-the-solution)). People don't trust a solution. They have to understand the problem itself, the alternatives, and finally, why the solution is the best amongst the alternatives.

</comment><comment><USER10> I am really amazed at the amount of things that make it to the front of Hacker News with such little substance. I mean, that was a gif and a rather short few paragraphs of obvious advice.

</comment><comment><USER11> Sometimes the best thing to do is to just take a leap and grab the next vine-- vines that are too close won't take you very far.

</comment><comment><USER12> I was slightly disappointed that this wasn't about Tarzan switching from ape- man to British aristocrat.</comment>


The Path to Rust - <AUTHOR> https://thesquareplanet.com/blog/the-path-to-rust/? <comment><USER3> In my current job, I was given a task to write a service which should handle billions of events eventually in the fastest possible way. The language choice was given for me to decide, and I was thinking that maybe I'll do it with C++14, Scala, Go or Rust. C++ has it's quirks and I'm not really enjoying it's build tool of choice, cmake. Scala I can write fast, but scaling the app in Mesos would consume lots of memory; every task would take a huge slice of RAM because of the additional JVM. Go I just don't like as a language (personal taste) and I think the GC adds a bit too much of pausing for this app, so I gave Rust a shot.

The first week was madness. I'm fairly experienced developer and the Rust compiler hit me to my fingers constantly. Usually the only way out was to write the whole part again with a different architecture. On the second week I was done with my tasks, very comfortable with the language and just enjoying the tooling. Also I was relying on tests way less because of the compiler, even less than with Scala. If it compiles, it has a big chance of working. Cargo is awesome. Let me repeat: Cargo is awesome. Also I like how I can write code with Vim again and even though for some things I need to read the Rust source, it is pretty easy to get answers if you're in trouble. In the end I wrote some integration tests with Python and I'm quite happy with them.

Now I want to write more Rust.

<comment><USER4> > _Go I just don 't like as a language (personal taste) and I think the GC > adds a bit too much of pausing for this app_

Not saying you have to like Go, we all have different preferences. But FYI, the gc has gotten _much_ better in the latest releases. I don't think you would be bothered by gc pauses. (my understanding is that it' both faster and more "spread out")

<comment><USER3> We're doing real time stuff where these pauses matter. Just that we don't really need GC for anything and adding it to certain services cause more harm than it causes good.

<comment><USER5> How do you handle the reference counting pauses in Rust then; rely on them being deterministic? Or do you completely avoid the reference counting boxes?

<comment><USER6> Not the OP, but in my experience it's fairly rare to encounter someone using the `Rc` type in Rust. It's nowhere near as prevalent as `shared_ptr` seems to be in C++, for example.

<comment><USER3> There is also the `Arc` type, which is an atomic reference counter. You still need those, especially if you need to share stuff between multiple threads.

<comment><USER7> You can often get away with using scoped threads.

</comment></comment></comment></comment></comment></comment></comment><comment> loup-vaillant > _Rust is not the most beginner-friendly language out there — the compiler is > not as lenient and forgiving as that of most other languages […], and will > regularly reject your code […]. This creates a relatively high barrier to > entry […]. In particular, Rust's "catch bugs at compile time" mentality > means that you often do not see partial progress — either your program > doesn't compile, or it runs and does the right thing. […] it can make it > harder to learn by doing than in other, less strict languages._

I don't see how making the type system stricter makes the language harder to learn. Maybe that's because I know another relatively paranoid type system (Ocaml), but still.

A type system that rejects your code is like a teacher looking at a proof you just wrote, and tells you "this doesn't even make sense, and here's why". It may be frustrating, but this kind of feedback loop is tighter than what you would get from a REPL.

And you _do_ see partial progress: the type errors change and occur further in the source code as you correct your program. Each error is an opportunity to fix a typo or a misconception. The distinction between a broken prototype that doesn't even compile and a working program isn't binary: when you correct a type error, your program is _less_ broken, even though it doesn't compile yet.

<comment><USER8> > A type system that rejects your code is like a teacher looking at a proof > you just wrote, and tells you "this doesn't even make sense, and here's > why". It may be frustrating, but this kind of feedback loop is tighter than > what you would get from a REPL.

I've been trying to port a personal project to Rust recently.

Sometimes Rust's type system is too limited to understand why something is safe. Here are a couple examples:

* in C++ code, you can have a class in which one field has a reference/pointer into another which came earlier in the declaration order (and thus will be constructed first and destructed last). This is often a useful thing to do (one example: [https://users.rust-lang.org/t/struct-containing-reference-to...](https://users.rust-lang.org/t/struct-containing-reference-to-own-field/1894)) You can't do that in Rust. They have to be separate instance variables on some thread. If there's no thread running to own it, you have to use referencing counting (Arc or Rc) or unsafe blocks. Or maybe instead of keeping a reference, have all calls take the outer struct as a context argument and use some sort of struct/lambda which knows how to find the thing you're referencing given that (this is what I'm trying in my code). Someone proposed a language change for "self-borrowing structs" ([https://mail.mozilla.org/pipermail/rust-dev/2014-February/00...](https://mail.mozilla.org/pipermail/rust-dev/2014-February/008658.html)) but it didn't go anywhere as far as I can tell.

* in C++ code, you can loop over one instance variable and then call a private method on self which mutates a different instance variable. In rust, you'll get errors about self being partially borrowed. I think you have to restructure the other method to not take self, which probably means grouping things into child structs. Basically Rust doesn't look across functions boundaries to decide if something is safe so it has to consider this an error even if it isn't for the particular method you're calling.

Fundamentally, I think the choice is between these three options:

* use a garbage-collected language and not have to be explicit about these details. There's no possibility of buffer overflows or use-after-free errors but you have to pay the runtime overhead of the garbage collector. Go is clear about the costs involved: pauses up to 10 ms, 25% of all CPU cycles, and 50% of RAM (see [http://golang.org/s/go14gc](http://golang.org/s/go14gc)). Other GCed languages likely have similar costs even if they aren't stated as clearly.

* use an unsafe language like C/C++, enforce these things manually in your head and with comments, and occasionally have security problems when you screw up.

* use a safe-but-explicit language like Rust/Swift and have to "show your work" to the compiler quite a bit more, finding a different way if safety is too hard to prove.

<comment><USER9> For self-referential datastructures (your first point above), using an Rc or Arc shouldn't have any overhead. I agree that it would be nice to be able to express this, but it's not really that big of a problem.

For partial borrows, there has been some work on it ([https://github.com/rust- lang/rfcs/issues/1215](https://github.com/rust-lang/rfcs/issues/1215)), but I agree that this is something that's missing. That said, I _very_ rarely run into this, and there's usually some fairly obvious restructuring I can do to make it work out.

<comment><USER8> > For self-referential datastructures (your first point above), using an Rc or > Arc shouldn't have any overhead. I agree that it would be nice to be able to > express this, but it's not really that big of a problem.

I don't see how that could be true. It means using a separate heap allocation for each referenced piece (although the owning_ref thing steveklabnik mentioned might minimize that), a bit of extra RAM for the counter, and a bit of bookkeeping. I'm not saying the overhead is huge, but how could it be zero?

fwiw, I finished this section of my code, and the context approach I mentioned worked out well for me. It was just a bit of a puzzle to find a way Rust would like.

I'm pretty happy with Rust so far even though I've had to restructure parts of an apparently-working program to fit its model. My program is now more obviously correct, benchmarks are pretty good so far (though I wish profile- driven optimization were supported/mature: [https://unhandledexpression.com/2016/04/14/using-llvm-pgo- in...](https://unhandledexpression.com/2016/04/14/using-llvm-pgo-in-rust/)), and the open source library situation seems better than C/C++ for what I'm doing and actively improving where C/C++ is stagnant.

</comment></comment></comment></comment><comment><USER10> Well, the functional crowd won. An example expression from the parent article:

        let idx = args
        // iterate over our arguments
        .iter()
        // open each file
        .map(|fname| (fname.as_str(), fs::File::open(fname.as_str())))
        // check for errors
        .map(|(fname, f)| {
          f.and_then(|f| Ok((fname, f)))
            .expect(&format!("input file {} could not be opened", fname))
        })
        // make a buffered reader
        .map(|(fname, f)| (fname, io::BufReader::new(f)))
        // for each file
        .flat_map(|(f, file)| {
          file
            // read the lines
            .lines()
            // split into words
            .flat_map(|line| {
              line.unwrap().split_whitespace()
                .map(|w| w.to_string()).collect::<Vec<_>>().into_iter()
            })
          // prune duplicates
          .collect::<HashSet<_>>()
            .into_iter()
            // and emit inverted index entry
            .map(move |word| (word, f))
        })
      .fold(HashMap::new(), |mut idx, (word, f)| {
        // absorb all entries into a vector of file names per word
        idx.entry(word)
          .or_insert(Vec::new())
          .push(f);

Is there editor support for indenting this stuff?

<comment><USER11> More readable, and IMHO more idiomatic version: (Disclaimer: never tested)

        let mut idx = HashMap::new();
        for fname in &args {
          let f = match fs::File::open(fname) {
            Ok(f) => f,
            Err(e) => panic!("input file {} could not be opened: {}", fname, e),
          };
          let f = io::BufReader::new(f);
          let mut words = HashSet::new();
          for line in f.lines() {
            for w in line.unwrap().split_whitespace() {
              if words.insert(w.to_string()) { // new word seen
                idx.entry(w.to_string()).or_insert(Vec::new()).push(fname);
              }
            }
          }
        }

People who was introduced to the functional approach for the first time seems to enjoy it so much that everything becomes a hard-to-read mess of functions. I had similar experiences with Python list comprehension and C# LINQ.

<comment><USER12> Sorry, but I disagree. Although more verbose (as in more characters of source code), I could easily understand what the original code did. Your code has a high cyclomatic complexity and I have to keep all those nested for loops in mind when trying to figure out what you are doing.

I write C++ for most of my day job, and this would not pass code review because of readability problems in my team.

<comment><USER11> I know there are obviously lots of rooms for improvements. Indeed, if the code would become more complex I would immediately refactor. I can list some problems with my example:

\- The error handling should really have been refactored. `try!` would make this easier. (I haven't used it since it is in the `main` function and the code was a direct replacement.)

\- Error during reading words is not accurately handled. Again, `try!` would make this easier.

\- `words` and `idx` look coupled to each other, which ideally shouldn't have been.

\- `words.insert` is quite an opaque method; not everyone is sure if it returns true on a duplicate key or not. I've added a comment but frankly I'm not satisfied of that. <deleted> One alternative is to use `words.entry` instead, which gives a named enum variant. </deleted> Oops, `HashSet` does not have `entry`...

\- Ultimately, the body of the outermost loop should go to a function.

That said, do you really think that `line.unwrap().split_whitespace().map(|w| w.to_string()).collect::<Vec<_>>().into_iter()` is a chunk of code which can be read at a glance? It at least has to be named (like T-R's example). Functional approach means that you can split functions and _individually_ review them; the original code, IMHO, didn't.

<comment> T-R I think the issue with procedural loops (not to be critical, just in general) is that there's no abstraction - it's not clear what's getting mutated, or what the result is (or its type), it's harder to look at the individual steps, and it's harder to refactor.

The nice thing about functions like "map", "filter", "fold", "flatmap", etc., is that they describe intent - when you see a "map", you know it's not aggregating things, just applying a function: "map" has a distinct purpose from "fold" (and depending on how pure things are, you may not even be _allowed_ to do anything crazy).

Aside from that, the higher-order functions have pretty intuitive algebraic laws for refactoring (like with "compose" mentioned in my other comment) - It's not clear how you'd factor code out of a nested loop without understanding the whole thing, whereas "flatmap" (a.k.a. "bind" for the list monad) has laws for refactoring it.

<comment><USER11> I agree to you with the general sentiment. That said, Rust is not a functional language per se; the degree of functional decomposition is thus fundamentally limited. This is partly because it tries to be efficient, and as arielb1 pointed out, ownership sometimes makes it worse.

> It's not clear how you'd factor code out of a nested loop without > understanding the whole thing, [...]

You are right, loops are particularly hard to refactor. I still argue that my code is better (barring any future expansion) because it fits within handful lines; you cannot easily understand 100 lines of code with the cyclomatic complexity of 1, but you can often easily understand 10 lines of code with the cyclomatic complexity of 8 (e.g. triple loops). The size of code, syntax and contextual information matters as much as algebraic laws.

</comment></comment></comment></comment></comment></comment><comment><USER13> > This latter point is particularly interesting; the Rust compiler will not > compile a program that has a potential race condition in it.

I feel obliged to point out that this is false. Rust prevents _data races_, but not _race conditions_. You can read more in the Rustonomicon here: [https://doc.rust-lang.org/nomicon/races.html](https://doc.rust- lang.org/nomicon/races.html)

<comment><USER9> In my defense, the next sentence is "Unless you explicitly mark your code as `unsafe`, your code simply cannot have data races." That said, I've updated the article text to now say data races in both places.

</comment></comment><comment><USER14> Good to see such articles that provide an insight into various aspects of a programming language.

A couple of other beginner-friendly resources would include:

\- An alternative introduction to Rust [1]

\- 24 days of Rust [2]

\- CIS 198: Rust Programming [3]

[1] [http://words.steveklabnik.com/a-new-introduction-to- rust](http://words.steveklabnik.com/a-new-introduction-to-rust)

[2] [http://zsiciarz.github.io/24daysofrust/](http://zsiciarz.github.io/24daysofrust/)

[3] [http://cis198-2016s.github.io/](http://cis198-2016s.github.io/)

</comment><comment><USER15> I'd like to know what the author considers "systems work"; I don't consider garbage-collected languages (Go, Python) "systems" languages.

<comment><USER16> "Systems" has come to include things like networked services (eg. HTTP and DNS servers), which you can certainly do successfully in GC'd languages in many circumstances. Personally, I'd just like to see a widely agreed upon definition, so that we can all stop being confused when this comes up.

<comment><USER17> Naming confusion is completely normal. Even in fields like learning human anatomy you'll learn different words for some things depending on who your teacher is, even though one would think that there should have been sufficient time for such issues to settle and the professionals agree on one word. But naming always is an accident of space and time - where did it happen, when did it happen, and then the path from there. Language is dynamic and a little fuzzy - in the sciences as well. You can define all you want, the problem always is _other people_ , who choose their own definitions. It's fun - <USER2> keeps you on your toes :-)

In this context, for example, I don't see a reason for a world government to impose _one_ definition of "systems programming" worldwide by military force (this is what it would take!). Just like in almost everything in human language, you get it from the context. You should try to teach a computer to recognize human speech _including the meaning_ , then you'll realize that almost all of it relies on context and pre-existing knowledge. "Precision" comes from the interpretation - human language communication is the nightmare of people who love functional programming, it's full of hidden state and context and active interpretation.

Which is why it's so easy for this to happen: [http://dilbert.com/strip/2015-06-07](http://dilbert.com/strip/2015-06-07) If someone _wants_ to argue, there is no way to provide a water-tight human- language text that "Dick from the Internet" can't attack. There always is a way to mis-interpret human language.

<comment><USER16> Thanks, this is a great response. I shall henceforth embrace the fun and swear to never again be lured by the superficial temptations of naming hegemony!

</comment></comment></comment></comment><comment><USER9> Author of the post here. Curious that this got posted again. Was originally posted as [https://news.ycombinator.com/item?id=11773332](https://news.ycombinator.com/item?id=11773332). Can the posts be merged by a mod somehow?

</comment><comment><USER18> I really like my experience with rust so far also, but a few caveats:

* try!() Is pretty annoying

* Working effectively with C in non-lexical ways seems to involve some unstable libraries and still requires nightly rust

* Macros are safer, but can't do some things that C macros can. For instance, they are hygienic, which means you can't conjure up new identifiers. For that, you need a syntax plugin, which is very powerful but the APIs aren't stable yet. This goes to the previous point.

* A few annoyances, like warning when you don't use a struct field as "dead code". If I'm interfacing with C I probably need that struct field whether the rust compiler sees it or not, but I don't want to disable all dead code warnings for that.

<comment><USER19> You can attach the allow(dead_code) attribute to the struct - or even the individual field - which will scope it to that declaration only. Still annoying when you have a lot of them, but better in my opinion than not having dead code warnings on fields, since the use case you describe is the minority of Rust structs that are declared.

<comment><USER18> Maybe there should be an annotation saying that the layout of a struct is important for ABI compatibility, and it would silence warnings related to the layout.

<comment><USER19> C FFI structs have to be marked `#[repr(C)]` to guarantee they'll be represented how C structs would be. It might be a reasonable change to implicitly allow dead fields on any struct with a C repr (since using the struct in C effectively makes all of its fields public).

</comment></comment></comment></comment><comment><USER20> Is rust ever going to re-introduce the N:M model again ? For services which need to handle 1M connections the system threads are too expensive and mio brings back the callback hell.

<comment><USER11> Have you tried mioco [1]? It looks fine enough to avoid the callback hell.

[1] [https://github.com/dpc/mioco](https://github.com/dpc/mioco)

<comment><USER20> Thanks. So no clear winner yet. Does libraries have to be re-written to support the coroutines ?

</comment></comment></comment><comment><USER21> If I'm not hacking on something super low-level, like hardware or an OS, then should I still try Rust? Why not stay within super high-level/expressive programming languages like Haskell/clojure?

I ask because a lot of extremely smart people I know like Rust.

<comment><USER22> Rust is good.

Great tooling: Libraries can be written, published and used quickly. Code can be easily rebuilt on upcoming Rust versions without uninstalling anything. Integrated testing and documentation generation. Cross-compilation.

Truly cross-platform. As an example, Rust even supports both GNU and MSVC targets on Windows, that means Rust libraries can be linked into the C++ projects compiled with MINGW or MSVC. All the standard library features are cross-platform, unless namespaced otherwise.

Rust linear type system requires variables be either uniquely mutable or immutable, but not both. This solves whole class of resource management problems (memory being the most important) in the simplest possible way. This compile-time guarantee also works for all references to data, which become thin pointers at runtime. This, together with Send-able and Sync-able types, also ensures no data races.

Language is up-to-date. No-nulls, pattern matching, closures, generics, attributes will feel familiar to programmers coming from various languages. Language had many iterations and changes, and every bit of it was designed with care. Right now it may be a bit explicit, but sugar can always be added later if it appears worth it.

Stable release cycle, stability guarantee. We know when new beta will be renamed to stable and we know what's in it! That means there is time to check all ecosystem for possible breakage, even though historically Rust introduced hardly any breaking changes since 1.0 a year ago.

</comment></comment><comment><USER23> One thing I am missing is dependency injection/ioc. How does one effectively unit test without it?

<comment><USER9> You can still do dependency injection in Rust..? There even exists a create for it: [https://github.com/Nercury/di-rs](https://github.com/Nercury/di-rs). There's also some further discussion here: [https://users.rust-lang.org/t/how- do-you-implement-dependenc...](https://users.rust-lang.org/t/how-do-you- implement-dependency-injection-in-rust/213)

<comment><USER23> I saw that but it seems really complicated for something that should be straightforward. Or it just looks so? (coming from c#)

<comment><USER24> If I understand it right, Rust has "zero-cost abstractions" meaning that they don't exist at run-time, just when compiling. e.g. [http://blog.rust- lang.org/2015/05/11/traits.html](http://blog.rust- lang.org/2015/05/11/traits.html)

This is the right choice if you want the language to be "a better C" but it means that there is none of the Run-Time Type Information that we are used to in C# (or java).

The RTTI plays a small but key role in e.g. how a DI container works: The container works out what parameters a constructor needs at runtime and instantiates those recursively; and in how a mocking framework works: Generating a class on the fly with the right interface declaration. Even XUnit/NUnit work by using RTTI: The testing framework looks for all classes/methods with the right attribute metadata, and runs them.

It's easily fast enough for how we use C# and Java, but it's not zero-cost.

There are unit testing tools in Rust, but they cannot work the same way - IIRC, there is compiler support for them. So, While Rust as a good chance at being "A better C", don't expect it to do all the things that C# does involving types at runtime. A different goal meant a different language design.

There are other interesting things in Rust like the Macro system that might help do similar things, but it's really not going to work like in C#

</comment></comment></comment></comment><comment><USER25> > Once your code compiles, you'll find (at least I have) that it is much more > likely to be correct (i.e., do the right thing) than if you tried to write > similar C, C++, or Go code.

What issues in Go do this sentence refer to?

<comment><USER26> I don't think that Go can detect race condition errors at compile time. So it could compile and sometimes fails mysteriously. In Rust, as long as you don't use unsafe block, the compiler should detect it. This is not really a Go issue, rather that Rust is much more stringent on what it "correct" code compared to C, C++ and Go.

<comment><USER4> I think you're right. Go has a race detector, but it work on run time: [https://golang.org/doc/articles/race_detector.html](https://golang.org/doc/articles/race_detector.html)

Guess you would normally run your tests with it, and maybe compile a dev / beta / RC version with race detection enabled?

Never used it in prod, but it did find a problem for me.

<comment><USER26> I believe this is one of the main promise of Rust, even if it is not stated like that, if it compiles it works and it is safe.

But obviously for any other language (C,Go,etc) you can use a compiler + static analysis + dynamic analysis to have the same confidence.

What I like with Rust is that implicitly you know that every one has to run the equivalent of analysis tool to release/deploy the code.

<comment> vertex-four > I believe this is one of the main promise of Rust, even if it is not stated > like that, if it compiles it works and it is safe.

Not quite. There's certain types of issue which its type system won't detect - you can, in fact, easily get into a race condition, just not what's known as a data race - and there's no promise that what you've written actually does what you expect it to do, just that it'll do something without use-after-free and other memory-related bugs, and if you've written your program well it's more likely than e.g. Python that it does what you intended (in my personal experience).

Getting closer to "if it compiles, it works", but still not entirely there, there'd be the likes of Idris and other systems with proofs embedded into their type systems - but even then you still have to verify that the proof matches what you're actually trying to do in the real world.

<comment><USER26> I agree that "it works" is an overstatement. It is more akin to if it compiles it won't segfault. (Arguably this is not unique to Rust, but there is no overhead in runtime to achieve that).

</comment></comment></comment></comment></comment></comment><comment><USER27> This is also a pending issue: [https://github.com/rust- lang/rfcs/issues/349](https://github.com/rust-lang/rfcs/issues/349)</comment>


A short article on contemporary society from a personal perspective - <AUTHOR> https://firstname.lastname@example.com/now-we-have-done-it-74b9545d4642 <comment><USER2> Not intending to attack you or anything and legitimately trying to understand this: what did the officer mean by her statement? I think I'm missing something that I need to understand why you were offended.

<comment><AUTHOR> Now, I did not have a chance to ask her what she meant - but here is my understanding. She had been sent there due to someone reporting me as a suspicious individual standing near said location. I can see this because of the hoodie I was wearing. So I am assuming she was referring to me when she responded "it's already coming off" in her walkie-talkie, meaning the nuisance who had roused the attention of campus police (me) was thankfully walking away.</comment></comment>


The Bet-the-Company Pivot That Led to Nimble Storage's IPO - <AUTHOR> http://sequoiacapital.tumblr.com/post/69884048439/the-bet-the-company-pivot-that-led-to-nimble-storages

<comment><AUTHOR> Nimble Storage went public today. The company's Series A pitch is at the bottom of this post. It didn't set out to make a storage system first.</comment>


Huawei in contact with European technology providers of privacy - <AUTHOR> https://www.ctocio.com/tech/software/%e5%bc%80%e6%ba%90/29385.html <comment><USER2> Huawei can't patch basic security holes (namely ancient OpenSSL versions sprinkled through their LTE Basestation codebase): [https://www.theregister.co.uk/2019/03/28/hcsec_huawei_oversi...](https://www.theregister.co.uk/2019/03/28/hcsec_huawei_oversight_board_savaging_annual_report/)

HCSEC asked Huawei to update these vulnerable libraries last year when Huawei promised a $2 billion investment in security, yet this really basic firmware maintenance task has yet to be performed.

Huawei also did the same thing with a Windows driver: [https://arstechnica.com/gadgets/2019/03/how-microsoft- found-...](https://arstechnica.com/gadgets/2019/03/how-microsoft-found-a- huawei-driver-that-opened-systems-up-to-attack/)

This company does not give a fuck about security, even when they are under scrutiny. It isn't the "Chinese hackers" I'm afraid of, but rather normal blackhats and greyhats exploiting this cornucopia of vulnerable software to our detriment.</comment>


Ask HN: Which Web host is the best? - <AUTHOR>

I am deciding between NetworkSolutions and SiteGround. What do you think are the best hosts to start a website on? <comment><USER2> I've been using hawkhost and is very good, excellent customer service too.</comment>


Show HN: Awtk open source xplatform GUI toolkit for win, Linux, iOS, Android - <AUTHOR> https://github.com/zlgopen <comment><AUTHOR> Google translate: [https://translate.googleusercontent.com/translate_c?depth=1&...](https://translate.googleusercontent.com/translate_c?depth=1&pto=nl&rurl=translate.google.ca&sl=zh- CN&sp=nmt4&tl=en&u=https://github.com/zlgopen/awtk)

There's a designer you can download to graphically design your GUI.

Hopefully an english version is in the works.</comment>


Ask HN: How microservices works? - <AUTHOR> Stupid question: how exactly works microservices ? how they comunicate between different apps/languages ? they use rest api or is better use something like zeromq/rabbitmq ?
Sorry to post this here, but honestly I often hear about microservices but I don't really understand what they are.
Any good link/book/slide is welcome.
PS: I will appreciate some examples in real world by big companies(example: how they do in airbnb/uber/whatever ?) <comment><USER2> Very good read that gives a pretty high level description of services:

[https://www.amazon.com/Building-Microservices-Designing- Fine...](https://www.amazon.com/Building-Microservices-Designing-Fine-Grained- Systems/dp/1491950358/ref=sr_1_1?ie=UTF8&qid=1488268723&sr=8-1&keywords=building+microservices)

</comment><comment><USER3> There are two related questions:

1\. What is a micro-service?

2\. How is a micro-service architecture implemented?

The first is subject to some debate, and it's not a debate I'm particularly interested in so I'll leave it as an exercise.

Micro-service architectures are generally implemented in two ways: push and pull. A collection of REST API's will tend to produce a push architecture of explicit requests and responses. A log to which all requests and responses are written with each individual service responsible for finding applicable messages is a way of implementing a pull architecture. Queues can play a role in either but are at an implementation detail at a lower level of abstraction.

Chat bots [1] can be seen as an example of a pull architecture with a log. Bots scan the continuous stream of messages and ignore what does not apply and act upon what does apply, typically they return results to the message stream. A chat bot might have a queue to manage requests and responses (input/output) or not, but that's an implementation detail not a high level architectural one at the level of a chat system.

A chat system will tend to grow by adding more bots rather than expanding existing bots and each bot will tend to have little or no knowledge of other bots. Each bot will be independent: one will run Jenkins another will make coffee.

The big point is that services are a way of organizing a system architecture and micro-services are one granularity at which the system may be decomposed. It's probably arguable that what constitutes a micro-service versus a service is relative to the size of the overall system. In practice, what is and isn't a micro-service has some correlation to the practicality of delegating responsibility to a team <= two pizzas.

YMMV.

[1]: A chat bot might be an example of something that is small enough to be 'micro'\-- or deci, nano, atto, yocto etc. -- (or not).

</comment><comment><USER4> Lets say you have an ecommerce firm. When a user is browsing an item(diapers?), at a mimimum he/she expects to see images, description, reviews, cost, tax, delivery dates, promotions, related items. In addition to what you see on frontend, in the backend, there may be jobs run for analytics, revenue reconciliation, warehouse management optimization, carrier management, fraud checks, item data setup and a host of other activities. Each of these tasks has its own complex business logic and typically has its own team. Each team runs a service, possibly in the form of a REST API call. You can call them microservices.

</comment><comment><USER5> > how [microservices] comunicate between different apps/languages ? they use > rest api or is better use something like zeromq/rabbitmq ?

Everything you mentioned, and then several other mechanisms. Whatever happens to be fashionable where they are being implemented.

> Sorry to post this here, but honestly I often hear about microservices but I > don't really understand what they are.

A bunch of small applications running separately that talk to each other using different protocols (sometimes one of the RPC protocols, sometimes raw HTTP that is today called REST, sometimes ZeroMQ or AMQP or some other ESB). The idea has little more in it than this, though operationally there are still some things to figure out.

<comment><AUTHOR> Ok, you know any good talk/slide with real examples ?

For example I don't understand why companies like airbnb (I understand for uber instead) need microservices...it's just a frontend app + backend + api. This can(could) be an single app.

<comment><USER6> The backend does many different things, hence could be split into microservices

</comment></comment></comment><comment><USER7> Here's a good talk that, while it mentions how Amazon use micro-services, it's very broad, and will probably leave you with more questions then answers: [https://youtu.be/6K4ljFZWgW8?list=LL9RQYUX_WIjk8SAcyuarJEw](https://youtu.be/6K4ljFZWgW8?list=LL9RQYUX_WIjk8SAcyuarJEw) (The Future of Software Engineering • Mary Poppendieck)

</comment><comment><USER8> I would strongly suggest to read the book "Building Microservices" by Sam Newman. He gave fair good description of what it is and what are the considerations you need to think when build it.

</comment><comment><USER7> for example. a calculator as a service. where users can add substrct divide and multply. devision turns out to be most in demand and is thus turned into its own service with separare hardware and team.

</comment><comment><USER9> Take a look at [http://learnmicroservices.io](http://learnmicroservices.io) That has links to numerous resources including some introductory articles.

</comment><comment><USER7> they can be compleatly isolated. and are often used as a scale out strategy. or an optimization where a botleneck is taken out from a monolith so it can scale independenly.</comment>


Silent Data Corruption Is Real - <AUTHOR> http://changelog.complete.org/archives/9769-silent-data-corruption-is-real <comment><USER3> It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places)

(...sure, you could call zfs or btrfs "mainstream", I suppose, but when I say "mainstream" I mean something along the lines of "officially supported by RedHat". zfs isn't, and RH considers btrfs to still be "experimental".)

<comment><USER4> btrfs only uses CRC32c which is weakish. ZFS is great but not exactly portable. I started to use Borg now for archiving purposes as well, not just backup. For me (low access concurrency, i.e. single or at most "a few" users) that works very well. Portable + strong checksumming + strong crypto + mountable + reasonable speed (with prospect of more) is a good package. It doesn't solve error correction, though.

<comment><USER5> crc32c is not weakish, and was chosen for a reason: crc32c has widespread hardware acceleration support that remains faster than any hash, and crc32c can be computed in parallel (unlike a hash, it has no hidden state, so you can sum independently computed block checksums to get the overall blob checksum). Bitrot detection doesn't need a cryptographic hash. You may want a hash for other purposes (like if you somehow trust your metadata volume more than your data volume), but that's a separate and slower use case.

<comment><USER6> _Bitrot detection doesn 't need a cryptographic hash._

Not only is a cryptographic hash unnecessary, under certain circumstances it will actually do a _worse_ job.

Cryptographic hashes operate under a different set of constraints than error detecting code. With an error detecting code, it's desirable to _guarantee_ a different checksum in the event of a bitflip.

With a cryptographic random oracle, this is not the case: we want all outcomes to have equal probability, even potentially producing the same digest in the event of a bitflip. As an example of a system which failed in this way: Enigma was specifically designed so the ciphertext of a given letter was always different from its plaintext. This left a statistical signature on the ciphertext which in turn was exploited by cryptanalysts to recover plaintexts. (Note: a comparison to block ciphers is apt as most hash functions are, at their core, similar to block ciphers)

Though the digests of cryptographic hash functions are so large it's statistically improbable for a single bitflip to result in the same digest as the original, it is not a guarantee the same way it is with the CRC family.

Cryptographic hash functions are not designed to be error detecting codes. They are designed to be random oracles. Outside a security context, using a CRC family function will not only be faster, but will actually provide guarantees cryptographic hash functions _can 't_.

<comment><USER7> " _Not only is a cryptographic hash unnecessary, under certain circumstances it will actually do a worse job._ "

A cryptographic hash is unnecessary, but as you point out in the next to last paragraph, it is statistically improbable that it will do a worse job. Because collisions are statistically improbable.

<comment><USER8> Specifically, I wrote a program to search for single-bit-flip collisions in sha1 truncated to 16 bits. The program didn't need to search for long before finding two messages with the same 16-truncated sha1 with a single bit flip at bit 1 of byte 171 of a 256-byte message. 376 1 171 be44b935e7ecfc81d1fe2cddcd7c1d7e04338fd83fa994cd6a877732ca5d8db83346bd9ccbfc4c8770682bd307c782421a512a80a106be87825d5c13f3156e23ffaacdfc1651f88f775507d1175542def2ccf084271ebd4ead175c8a448be0d50b26f59d970301ebc5a7f672d3ea870d9a1e02f8f5fd01c38297b8aa264a3f07fec32f9a91aa359784d2d9ce0e4649465c705f50feed23dcbefc0a726cfadb5e47ee577ed45203f90d6e2e650d42ddb10cba49d06bd4cdad4e6eaf5cfcb062de2539fc847ce0c104f2e667369080eaaab5934ae5f7f1ba733c3d1bfbda87bfa72ef12475b9ff0edc4deb99e6a5cf387c7f6b9c71ea62b4db4bb67c92d36460dd be44b935e7ecfc81d1fe2cddcd7c1d7e04338fd83fa994cd6a877732ca5d8db83346bd9ccbfc4c8770682bd307c782421a512a80a106be87825d5c13f3156e23ffaacdfc1651f88f775507d1175542def2ccf084271ebd4ead175c8a448be0d50b26f59d970301ebc5a7f672d3ea870d9a1e02f8f5fd01c38297b8aa264a3f07fec32f9a91aa359784d2d9ce0e4649465c705f50feed23dcbefc0a726cfadb5e47ee577ed45203f90d6e2e640d42ddb10cba49d06bd4cdad4e6eaf5cfcb062de2539fc847ce0c104f2e667369080eaaab5934ae5f7f1ba733c3d1bfbda87bfa72ef12475b9ff0edc4deb99e6a5cf387c7f6b9c71ea62b4db4bb67c92d36460dd

[https://gist.github.com/jepler/96d1e779dc95b8941b208887e10a8...](https://gist.github.com/jepler/96d1e779dc95b8941b208887e10a8084)

On the other hand, any CRC will detect all such errors; a well-chosen one such as CRC32C will detect all messages with up to 5 bits flipped at this message size.

This is quite appropriate for the error model in data transmission, of uncorrelated bit errors. [https://users.ece.cmu.edu/~koopman/networks/dsn02/dsn02_koop...](https://users.ece.cmu.edu/~koopman/networks/dsn02/dsn02_koopman.pdf) is a pretty good paper, though there are probably better ones for readers without an existing background in how CRC works.

<comment><USER7> You are not testing a crypto hash. "Crypto hash" means it is cryptographically strong, not truncated to 16 bits. For example ZFS with checksum=sha256 will use the full 256-bit hash for detecting data corruption.

<comment><USER8> Yup, you're right. if you use a full size cryptographic hash then the number of undetected errors can be treated as 0 regardless of hamming distance. On the other hand, it has 8x the storage overhead of a 32-bit CRC.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> Oh, yes. Silent bit errors are tons of fun to track down.

I spent a day chasing what turned out to be a bad bit in the cache of a disk drive; bits would get set to zero in random sectors, but always at a specific sector offset. The drive firmware didn't bother doing any kind of memory test; even a simple stuck-at test would have found this and preserved the customer's data.

In another case, we had Merkle-tree integrity checking in a file system, to prevent attackers from tampering with data. The unasked-for feature was that it was a memory test, too, and we found a bunch of systems with bad RAM. ECC would have made this a non-issue, but this was consumer-level hardware with very small cost margins.

It's fun (well maybe "fun" isn't the right word) to watch the different ways that large populations of systems fail. Crash reports from 50M machines will shake your trust in anything more powerful than a pocket calculator.

<comment><USER10> Enterprise disks do have ECC cache as opposed to consumer drives. Was it a consumer drive?

</comment></comment><comment><USER11> ZFS is also crazy good on surviving disks with bad sectors (as long as they still respond fast). Check out this paper: [https://research.cs.wisc.edu/wind/Publications/zfs- corruptio...](https://research.cs.wisc.edu/wind/Publications/zfs-corruption- fast10.pdf)

They even spread the metadata across the disk by default. I'm running on some old WD-Greens with 1500+ of bad sectors and it's cruising along with RAIDZ just fine.

There is also failmode=continue where ZFS doesn't hang when it can't read something. If you have a distributed layer above ZFS that also checksums (like HDFS) you can go pretty far even without RAID and quite broken disks. There is also copies=n. When ZFS broke, the disk usally stopped talking or died a few days later. btrs, ext4 just choke and remount ro quite fast (probably the best and correct course of action) but you can tell ZFS to just carry on! Great piece of engineering!

<comment><USER12> Pretty fascinating. But just based on this comment, I reckon that these 1500+ bad sectors drives aren't worth your time. So, why? Is it just that you wanted to play with all these options and don't really care about the data on these drives, or do you actually believe it's reasonable bang for the buck?

<comment><USER11> I forget the disclaimer that you should not do this, ever :)

We had a cluster for Hadoop experiments at uni and no ressources to replace all the faulty disks at that time (20-30% were faulty to some degree from the SMART values - more than 150 disks). So this was kind of an experiment. All used data was available and backup up ouside of that cluster. The problem was that with ext4 after running a job certain disks always switched to readonly and this was a major hassle as this node had to be touched by hand. HDFS ist 3x replicated and checksummed and the disks usally worked fine for quite a time after the first bad sector. So we switched to ZFS, ran weekly scrubs - only replaced disks that didn't survived the srub in reasonable time or with reasonable failure rates and bumped up the HDFS checksum reads that everything is control read once a week. The working directory for the layer above (MapReduce and stuff like that) got a dataset with copies=2 so that intermediate data is still fine within reasonable amounts. This was for learning or research purposes where top speed or 100% integrity didn't matter and uptime and usability was more important. Basically the metadata on disk had to be sound and the data on a single disk didn't matter that much. This was quite a ride and it's long been replaced since then.

Just thought it's interesting how far you can push that. In the end it worked but turned out there is no magic, disks die sooner or later and sometimes take the whole node with them.

Don't go to ebay and buy broken disks out of believing with ZFS these will work. Some survive a while, most die fast, some exhibit strange behavoir.

That RAIDZ is more or less for "let's see where this goes" purposes, backups are in place it's not a production system.

<comment><USER12> Hah, thanks for the story.

It seems that limited resources often lead to some interesting solutions (and learning new things). A factor that is not very common in VC backed companies.

</comment></comment></comment></comment><comment><USER13> It's articles like this that re-enforce my disappointment that Apple is choosing to NOT implement checksums in their new file system, APFS.

[https://news.ycombinator.com/item?id=11934457](https://news.ycombinator.com/item?id=11934457)

<comment><USER14> Can someone explain why one would checksum metadata but not user data? Is the assumption everything's backed up on iCloud? If so, are system files checksummed?

<comment><USER15> Metadata writes can be considered atomic, so an integrated checksum is written at the time the metadata block is written, or overwritten. Whereas with data, you can't do overwrites of either data or checksum, it's not atomic and any kind of crash or powerfailure will result in mismatching data to checksum. So unless you have something really clever to work around this, you need a copy on write file system to do data checksums.

</comment></comment></comment><comment><USER16> "Data tends to corrupt. Absolute data tends to corrupt absolutely."

In both sense of the word.

Many moons ago, in one of my first professional assignments, I was tasked with what was, for the organisation, myself, and the provisioned equipment, a stupidly large data processing task. One of the problems encountered was a failure of a critical hard drive -- this on a system with no concept of a filesystem integrity check (think a particularly culpable damned operating system, and yes, I said that everything about this was stupid). The process of both tracking down, and then demonstrating convincingly to management (I _said_ ...) the nature of the problem was infuriating.

And that was with hardware which was reliably and replicably bad. Transient data corruption ... because cosmic rays ... gets to be one of those particularly annoying failure modes.

Yes, checksums and redundancy, please.

</comment><comment><USER17> If I were to run ZFS on my laptop with a single disk and copies=1, and a file becomes corrupted, can I recover it (partially)?

My assumption is the read will fail and the error logged but there is no redundancy so it will stay unreadable.

Will ZFS attempt to read the file again, in case the error is transient? If not, can I make ZFS retry reading? Can I "unlock" the file and read it even though it is corrupted, or get a copy of the file? If I restore the file from backup, can ZFS make sure the backup is good using the checksum it expects the file to have?

Single disk users seem to be unusual so it's not obvious how to do this, all documentation assumes a highly available installation rather than laptop, but I think there's value in ZFS even with a single disk - if only I understood exactly how it fails and how to scavenge for pieces when it does.

<comment><USER7> It depends. Metadata is always redundant with 2 copies (even when using copies=1). So if the file's metadata is corrupted, yes ZFS will fully recover and rewrite a 2nd good copy of the metadata. But if the data is corrupted, then ZFS can do nothing to recover (you may be able to partially read the file, but the rest of it will return I/O errors.)

<comment><USER18> Ok, so zfs for single drive users doesn't fix single data corruption.

Definitely I'm going to use my solution so. All the next generation FS stuff is cool (btrfs also indeed) but for the _simplest_ use case people just need safe data and fix it it the disk goes bad.

<comment><USER7> Why don't you use copies=2 with a single disk?

</comment></comment></comment></comment><comment><USER7> The exact same silent data corruption issues just happened to my 6 x 5TB ZFS FreeBSD fileserver. But unlike what the poster concluded, mine were caused by bad (ECC!) RAM. I kept meticulous notes, so here is my story...

I scrub on a weekly basis. One day ZFS started reporting silent errors on disk ada3, just 4kB:

        pool: tank
       state: ONLINE
      status: One or more devices has experienced an unrecoverable error.  An
              attempt was made to correct the error.  Applications are unaffected.
      action: Determine if the device needs to be replaced, and clear the errors
              using 'zpool clear' or replace the device with 'zpool replace'.
         see: http://illumos.org/msg/ZFS-8000-9P
        scan: scrub repaired 4K in 21h05m with 0 errors on Mon Aug 29 20:52:45 2016
      config:
              NAME        STATE     READ WRITE CKSUM
              tank        ONLINE       0     0     0
                raidz2-0  ONLINE       0     0     0
                  ada3    ONLINE       0     0     2  <---
                  ada4    ONLINE       0     0     0
                  ada6    ONLINE       0     0     0
                  ada1    ONLINE       0     0     0
                  ada2    ONLINE       0     0     0
                  ada5    ONLINE       0     0     0

I monitored the situation. But every week, subsequent scrubs would continue to find errors on ada3, and on more data (100-5000kB):

      2016-09-05: 1.7MB silently corrupted on ada3 (ST5000DM000-1FK178)
      2016-09-12: 5.2MB silently corrupted on ada3 (ST5000DM000-1FK178)
      2016-09-19: 300kB silently corrupted on ada3 (ST5000DM000-1FK178)
      2016-09-26: 1.8MB silently corrupted on ada3 (ST5000DM000-1FK178)
      2016-10-03: 3.1MB silently corrupted on ada3 (ST5000DM000-1FK178)
      2016-10-10: 84kB silently corrupted on ada3 (ST5000DM000-1FK178)
      2016-10-17: 204kB silently corrupted on ada3 (ST5000DM000-1FK178)
      2016-10-24: 388kB silently corrupted on ada3 (ST5000DM000-1FK178)
      2016-11-07: 3.9MB silently corrupted on ada3 (ST5000DM000-1FK178)

The next week. The server became unreachable during a scrub. I attempted to access the console over IPMI but it just showed a blank screen and was unresponsive. I rebooted it.

The next week the server again became unreachable during a scrub. I could access the console over IPMI but the network seemed non-working even though the link was up. I checked the IPMI event logs and saw multiple _correctable_ memory ECC errors:

      Correctable Memory ECC @ DIMM1A(CPU1) - <USER2>

The kernel logs reported muliple Machine Check Architecture errors:

      MCA: Bank 4, Status 0xdc00400080080813
      MCA: Global Cap 0x0000000000000106, Status 0x0000000000000000
      MCA: Vendor "AuthenticAMD", ID 0x100f80, APIC ID 0
      MCA: CPU 0 COR OVER BUSLG Source RD Memory
      MCA: Address 0x5462930
      MCA: Misc 0xe00c0f2b01000000

At this point I could not even reboot remotely the server via IPMI. Also, I theorized that in addition to _correctable_ memory ECC errors, maybe the DIMM experienced _uncorrectable /undetected_ ones that were really messing up the OS but also IPMI. So I physically removed the module in "DIMM1A", and the server has been working perfectly well since then.

The reason these memory errors always happened on ada3 is not because of a bad drive or bad cables, but likely due to the way FreeBSD allocates buffer memory to cache drive data: the data for ada3 was probably located right on defective physical memory page(s), and the kernel never moves that data around. So it's always ada3 data that seems corrupted.

PS: the really nice combinatorial property of raidz2 with 6 drives is that when silent corruption occurs, the kernel has 15 different ways to attempt to rebuild the data ("6 choose 4 = 15").

<comment><USER10> When a double-parity error is detected, the operating system should halt. Maybe that didn't happen properly. Tripple parity errors may go undetected, but how likely is that.

I wonder what 'really' happened.

<comment><USER7> I saw dozens of ECC errors in the IPMI log, and dozens of MCAs in dmesg. So the memory was failing so badly that it is probably there were 3+ bit errors.

</comment></comment></comment><comment><USER19> I know for sure that btrfs scrub found 8 correctable errors on my home server filesystem last July. This is obviously great news for me. Contrary to a lot of people here I've personally found btrfs to be really stable (as long as you don't use raid5/6 though).

</comment><comment><USER20> People grossly under-intuit the channel error rate of the SATA. At datacenter scale it's alarmingly high [http://www.enterprisestorageforum.com/imagesvr_ce/8069/sas-s...](http://www.enterprisestorageforum.com/imagesvr_ce/8069/sas- sata-table2.jpg)

</comment><comment><USER21> I'm not a database expert, but this seems like something I should worry about, at least a bit. Is this a problem if you store all your persistent data in a database like MySQL?

<comment><USER22> If your database does ZFS-like checksumming on all of its data, including the structures that it uses to find the data that you put in, and has the ability to correct errors, then no.

Realistically though, I don't know if MySQL has this. You'd probably be better off using a filesystem that gives these kind of guarantees and running your database on that.

<comment><USER23> Every ACID compliant database is supposed to calculate checksums.

<comment><USER24> Good to know.

</comment></comment></comment></comment><comment><USER25> Of course, my ZFS NAS backup is sound until a file that got bitrotted on my non-ZFS computer is touched and then backed up to it :/

It's kind of (literally?) like immutability. If you allow even a little mutability, it ruins it.

I think all filesystems should be able to add error-correction data to ensure data integrity.

</comment><comment><USER26> Shouldn't RAID 1,5,6 protect against data corruption because of disk errors?

<comment><USER27> Some disk errors, yes, but something as simple as a power failure can easily corrupt your data:

[http://www.raid-recovery-guide.com/raid5-write-hole.aspx](http://www.raid- recovery-guide.com/raid5-write-hole.aspx)

[https://blogs.oracle.com/bonwick/entry/raid_z](https://blogs.oracle.com/bonwick/entry/raid_z)

<comment><USER28> Hardware RAID does not suffer from the write-hole like MD-RAID does (thanks to on-board supercap-backed non-volatile memory).

I can't remember if it was merged upstream, but some folks from Facebook worked on a write-back cache for MD-RAID (4/5/6 personalities) in Linux which essentially closes the write-hole too. It allows one to stage dirty RAID stripe data in a non-volatile medium (NVDIMMs/flash) before submitting block requests to the underlying array. On recovery, the cache is scanned for dirty stripes, which are restored before actually using the P/Q parity to rebuild user data. I worked on something similar in a prior project where we cached dirty stripes in an NVDIMM, and also mirrored it on its controller-pair (in a dual-controller server architecture) using NTB. It was a fun project, when neither the PMEM nor the NTB driver subsystems were in the mainline kernel.

<comment><USER29> RAID journaling.

[https://lwn.net/Articles/665299/](https://lwn.net/Articles/665299/)

Haven't tried it but it seems to already be merged, at least write-through. They now work on implementing writeback and this IIRC isn't merged yet.

</comment></comment></comment></comment><comment><USER10> The story here is not how Silent Data Corruption is real. The story is that somebody did a bad home brew server build and fucked up.

So ZFS protects against end-user mistakes.

I was really hoping about a story on some large-scale study on silent data corruption, but no, just an ankedote.

Sad!

:D

<comment><USER30> I didn't go into detail in the article, but the server in question is running a Supermicro X10SLH-F-O motherboard, ECC RAM, and a Haswell CPU, in a Rosewill RSV-L4411 4U chassis. Is there a hardware problem here? For sure. But you can't write this off as being some dusty overclock mess bought at someone's garage sale.

I have, incidentally, seen this in corporate environments on traditionally- engineered sever-class hardware as well. This is just a much more easily- discussed case.

</comment></comment><comment><USER31> interesting find! I wonder what would be a good safeguard to this. I feel like just backing up your data would offer something - but a file could silently change and become corrupted in the backup too.

<comment><USER32> Hm. You could make two backups and checksum each file and safe the checksum. Then you could compare regularly file contents with the initial checksum, if there is a mismatch copy from the other backup.

<comment><USER33> Git-annex is nice for this; the fsck command will check the file against a checksum and request a new copy from other node automatically if the check fails.

</comment></comment></comment><comment><USER34> Yes, it is VERY real. Because no one gives a damn. Most people ( consumers ) would just ignore that corrupted Jpeg.

I am in the minority group that gets very frustrated and paranoid when my Video or Photos gets corrupted.

Synology has Btrfs on some range of their NAS. But most of them are expensive.

I really want a Consumer NAS, or preferably even Time Capsule ( with two 2.5" HDD instead of one drive ) with built in ZFS and ECC Memory, by default weekly scrub drive. And alert you when there is problem.

And lastly, do any of those consumer Cloud Storage, OneDrive, DropBox, Amazon, iCloud have these protection in place? Because I would much rather Data Corruption be someone else problem then complexity at my end.

</comment><comment><USER35> That gives me more reasons to experiment with DragonFly BSD by building a NAS using HAMMER file system.

</comment><comment><USER36> Uh. My ZFS-backed BSD NAS also has the hostname 'alexandria'.

</comment><comment><USER37> Closed source firmware on drives contain bugs that corrupt data. Are there any drives, available anywhere, that have open source firmware?

<comment><USER38> There is a corresponding project: [http://www.openssd- project.org](http://www.openssd-project.org)

<comment><USER39> Note that it's essentially a dead project. The "NEW!!!" platform mentioned on their homepage is from 2014.

</comment></comment></comment><comment><USER40> Just like data black markets.

</comment><comment><USER18> I started a really simple and effective project the last month to be able to fix from bitrot in linux(MacOs/Unix?). It's "almost done" just need more real testing and make the systemd service. I've been pretty busy the last weeks so I've only been able to improve the bitrot performance.

[https://github.com/liloman/heal-bitrots](https://github.com/liloman/heal- bitrots)

Unfortunatly, btrfs is not stable and zfs needs a "super computer" or at least as much GBs of ECC RAM as you can buy. This solution is designed to any machine and any FS.

<comment><USER41> Please stop spreading this misinformed statement. I assume you are referring to the ZFS ARC (Adaptive Replacement Cache). It works in much the same way as a regular Linux page cache. It does not take much more memory (if you disable prefetch) and will only use what is available/idle. We use Linux with ZFS on production systems with as low as 1GB memory. We stopped counting the times it has saved the day. :-)

ECC is a nice to have, but ZFS does not have special requirement over say a regular page cache. The only difference is that ZFS will discovery bit-flips instead of just ignoring them as ext4 or xfs would do.

<comment><USER42> > ECC is nice to have.

Actually it seems ECC is important for ZFS filesystems see:

[http://louwrentius.com/please-use-zfs-with-ecc- memory.html](http://louwrentius.com/please-use-zfs-with-ecc-memory.html)

<comment><USER43> To be clear, it is not ZFS that requires or even mandates ECC. Since ZFS uses data as present in memory and has checks for everything post that, it is prudent to have memory checks at the hardware level.

Thus, if one is using ZFS for data reliability, one ought to use ECC memory as well.</comment></comment></comment></comment>


Building a Mobile Intelligent Journal in Python - <AUTHOR> http://www.kpkaiser.com/entrepreneurship/building-a-mobile-intelligent-journaling-system-in-python/

<comment><USER2> Wow, this is really cool. I've been working on something similar for a little while now but it isn't nearly as developed or interesting. The only issue I have is the pricing model, which is fair but takes a little bit away from the product for me (monthly fees– cut the cloud and I'd put $100 down just to use the software however I want).

<comment><AUTHOR> Hmm, the monthly fees are simply to pay for the costs of Twilio. The project will be completely Open Sourced at the end, if I end up raising the money.

I'm also trying to raise money to get this into the hands of people in the developing world as quickly as possible. Each donation level pays my Twilio costs for a certain number of people.

My goal is 10,000 people in developing countries using my beta within 3 months.

Thanks for the encouragement, though!

<comment><USER2> Ahh, okay. I dropped off a modest donation so I can see the updates. Small note: at first I wasn't sure if by 'developing world' you meant developing nations or the world of programmers and developers... ;)

Good luck!

<comment><AUTHOR> Lol, that's pretty bad with the clarity of my message. Donations sponsor people in developing countries to get access, and pays for the text message fees from Twilio.

Looking to specifically target the entrepreneurs in the "informal economy" explained here:

[http://www.npr.org/2011/10/26/141503411/the-informal- economy...](http://www.npr.org/2011/10/26/141503411/the-informal-economy- driving-world-business)

I definitely need to improve my message and the clarity behind it.

Thanks for your support, appreciate it 100%!</comment></comment></comment></comment>


iPhone Notches on Android Phones - <AUTHOR> https://www.theverge.com/2018/3/4/17077458/iphone-design-clones-mwc-2018 <comment><USER2> I don't really see the problem with the iphone-like design, there are three conflicting design problems... People want things that aren't gargantuan to fit in their pocket, but they also want the largest possible screen. And they also want a high quality front facing camera. The camera has to go somewhere and if you put a big bezel at the top it looks like an old phone from two years ago.

<comment><USER3> Here is the problem.

 _But the problem with these notched screens on Android phones is that they're purely cosmetic.....the cutouts at the top are self-evidently motivated by the desire to just look — not function, look — like an iPhone X._

<comment><USER4> The Verge's argument is that because they have a chin at the bottom, the notch is purely cosmetic since they didn't utilize the space at the bottom. I find this argument to be poor reasoning. Moving to a notch at the top seems like an inevitable future for most smartphones - we need an earpiece as well as front facing cameras and other sensors, either we at a top chin, or we notch. Those are the options. The notch just means we have additional usable screen space and creates a more full screen experience. By moving to a notch now it means that more android phones can start practicing for that future design, rather than pushing it off to avoid looking like they're copying apple (despite android doing it first with the essential phone). Eventually that chin at the bottom will be screen as well, but that won't be a change to usability like having the notch is. I think Asus is smart moving forward with this despite the strange publicity that they probably expected it would get (such as this article). This seems like less a design issue or an aspect of one company copying another, and more like just The Verge trying to write something that will generate clicks.

<comment><USER5> > The Verge trying to write something that will generate clicks

Is it ever not the case? The economic incentives drive institutions like The Verge to promulgate clickbait and sensationalized titles

<comment><USER6> Presumably you should also be interested in writing content that generates credibility and a long term following.

</comment></comment></comment></comment></comment><comment><USER7> Any opinion of the iPhone's design decision should be reserved until you have had a chance to use the device for at least an hour.

It has become increasingly clear that the experience of the phone in-person and the experience of looking at product screenshots and marketing material are wildly different.

Anecdotally, every single people I know who has the iPhone X has indicated that they don't even notice that there's a notch after the first hour or so of owning the phone.

<comment><USER8> This is one of those things where Android folks made fun of the notch, to the extent that Samsung even made an ad making fun of it, and within a year, Android folks will have decided their notches are great, and in fact, better than Apple's notches for this reason or the other. (And of course, the "Essential did it first" thing has to crop up as well, even though they were likely developed without inspiration from each other at around the same time.)

Make fun of it right up until you copy it, is basically the industry standard now. Headphone jack going away was the same thing, pretty much.

<comment><USER9> > Headphone jack going away was the same thing, pretty much.

Except most manufacturers are still making phones with headphone jacks. Those who want to sell phones at least.

Even Apple has a headphone-jack option for those who has it as an absolute demand: The iPhone SE.

<comment><USER10> Yep. Case in point: S9. Headphone jack. Still waterproof, and they throw in a memory card slot just for further mockery. Samsung knows their stuff.

</comment></comment></comment></comment><comment><USER11> I'm pretty sure Andy Rubin's Essential Phone predates the iPhone notch. Odd that the Verge didn't mention it in the article.

[https://www.essential.com/](https://www.essential.com/)

…also, regardless of who invented it, it seems like this design feature is unavoidable. If you want to kill the bezel but still want to have things like front-facing cameras, you're pretty much stuck with a notch.

<comment><USER10> I don't feel that my S8 has any bezels worth worrying about. The iPhone X and its notch is a design monstrosity compared to the clean lines of Samsung's work.

I find the whole notch design an abomination which should just die and be put to rest: By adding a notch you effectively create a _software_ bezel for renderable content; web site developers have to deal with this. You can't show fullscreen video without obstruction, etc etc. Horrid insanity.

<comment><USER12> Have you even used the iPhone X? Myself and every person I've talked to do not even notice the notch after a short period of use. And I was initially very skeptical of the notch too. A better mental model is that the iPhone has 2 additional 'ears' of usable screen.

Compare this to the S8 which not only has physical bezels on top and bottom, but also software status bar at the top and software home bar at the bottom cutting further into usable screen space.

Not to mention Apple have nailed the gesture based UX. It feels totally natural and is a pleasure to use. Whereas any time I have to use an android phone it feels clunky and poorly executed.

Call me a fanboy, whatever. But I have never found the same quality experience on an android phone that I get with my iPhone.

<comment><USER10> I've seen it a fair bit and the more I see it, the less I like it. I have also read the web design guidelines for how to avoid putting important content in the dead zones, and (AYFKM) is my one and only response.

The bottom bar is switchable on the S8, and the top one disappears when it needs to. The physical bezels are _tiny_ , on the sides there are none; the S8+ is not lacking for (vertical) real estate.

</comment></comment></comment></comment><comment><USER13> A lot of the arguments complaining about the iphone's notch are missing two points:

1\. The notch is necessary to house faceid sensors that no current android phone with a notch has

2\. The notch isn't cutting into the screen, the ears of the notch are actually cutting into the bezel.

A lot of hate towards Apple (justified or not) has been towards their preference of form over functionality. It's ironic to see Android handset makers taking a functional compromise from the iphone X and making it a completely cosmetic feature.

<comment><USER14> This is especially ridiculous when you consider the Galaxy S9 has a screen to body ratio that exceeds that of the iPhone X. In the words of one of the most egregious iPhone X copycats - Asus:

 _Well, it's quite simple. Some people will say it's copying Apple, but we cannot get away from what users want. You have to follow the trends._

I'm going to guess those users don't reside in North America or Europe.

<comment><USER10> Yep, the S8 / S9 has really nailed the design as far as I'm concerned: Just one massive screen with no intrusions.

And they retain the headphone jacks as well, with a fairly decent DAC, so you can keep on using your high end wired headphones if you want.

</comment></comment></comment><comment><USER15> As someone with large fingers, I despise this trend of removing/reducing the bezels --- making it harder to hold the phone without either blocking part of the screen with fingers or activating the touchscreen. The screen is plenty big enough already; another few mm is not going to make much difference either way.

Notching the display is merely another development in this trend, and what's worse is that it turns what usually is a reasonably standard LCD panel into an even more proprietary part with its associated manufacturing difficulties, since the dimensions of the notch then become another variable in the design.

(More ironically, web and UI design trends are going toward _wider_ margins and blank space, almost as if to compensate.)

</comment><comment><USER16> Yet again, Apple manages to make their decisions standards. Good on them, bad on others. Should've waited some time to see if users want that damn notch.

Edit: My only problem with this is that there is copying going on even without general/mass acceptance.

<comment><USER14> The Android phone OEM's implementing the notch aren't really big players. When Samsung does it then it'll become the standard. And since the S9 already has a better screen to body ratio than the iPhone X I don't see them ever doing it.

<comment><USER2> The OnePlus 6 is rumored to have a notch almost like the Asus. By some measurements Oppo/OnePlus are now the 4th largest Android platform phone maker in the world.

</comment></comment></comment><comment><USER17> I know it looks bad in photos, but using the iPhone X I honestly do not even notice it at all

<comment><USER18> Yep, I had the same experience. I also thought getting rid of the home button would drive me crazy, by I don't miss it at all.

<comment><USER19> I've never used an iPhone X - how does one get back to the home screen from an app?

<comment><USER7> Swipe up from the bottom of the screen (similar to how you used to activate control center to toggle wifi).

The iPhone X's interface is more gesture based than ever before.

<comment><USER19> On iPhone X how would I get to control centre if Swipe Up has been remapped for home screen?

(I still swipe up for Control Center on the 6S)

<comment><USER7> You swipe down from the right of the notch at the top to get control center.

Swiping down from the left of the notch at the top (or from the notch itself directly - yes it recognizes touch) you get your notifications/lock screen.

<comment><USER20> Kinda irrelevant to the discussion, but the 'swipe down from top right' for Control Center is pretty universally hated AFAICT. The best solution I've seen suggested is to make it part of the multitasking interface: you swipe up and right from the bottom to go to the previous app, CC should be swipe up and left.

</comment></comment></comment></comment></comment></comment></comment><comment><USER21> The moment I saw an iPhone X in a commercial for something else, the notch really clicked for me. The notch does so much more than just extend the screen a little bit more. It makes the phone instantly identifiable, which is a marketing technique that Apple has always been fantastic at. I was watching an ad for some unrelated thing/service and my mind went "Oh look. An iPhone X." It's interesting seeing that they may have started yet another design trend.

</comment><comment><USER22> This reminds me an old jewish joke about chasidim.

A person visited a chassidic town and saw that all the chasidim when they entered the walked through doorways would dip their heads. He didn't understand why, perhaps it's a form of piety? So he asked them and was told "we do it, because 'the rebbe' does it". So intrigued, he asked the rebbe, why he does it. And he was told, "Can't you tell how tall I am? I'd bump by head otherwise".

</comment><comment><USER23> So the inter-company envy is coming back, like when iOS copied flat interface ideas from Windows Phone. I am wondering why we always end up with unified bland stuff instead of many possible variations so that everyone can choose what they like? It seems like nobody wants to take risks anymore. I remember time when every single app had completely different interface; these days everything looks alike. Even if it is more economical, one size doesn't fit all.

</comment><comment><USER8> Android P is supposed to add support for notches, so I assume software customization right now is an afterthought: They'll adapt to whatever Google gives them in the next couple months.

<comment><AUTHOR> Yeah I would think this would cause issues with apps until they update for the notch. Is there a default "notch less" mode that only uses the lower part of the screen? and developers opt in to using the space adjacent to the notch?

I can also see all the different sized notches being a headache for app developers in the coming years.

<comment><USER8> Well, MWC is where you show off all your new hardware that's coming out... later in the year. So everyone has their notchified hardware, but they likely won't release them until Android P is available.

<comment><AUTHOR> I guess I'm mostly thinking of apps that aren't well supported. If the developer doesn't notchify their app and someone opens it on their new notched phone a year from now I'm wondering how android P will handle that.

</comment></comment></comment></comment><comment><USER24> The first phone with a notch was the Essential phone but once apple made the iPhone X you knew it was going to be mainstream. It's unfortunate.

<comment><USER25> The Essential Phone and iPhone X were both in development simultaneously, unlike the current situation where ASUS, for example, has been very clear that it is intentionally copying Apple.

>"Some people will say it's copying Apple," Asus' global head of marketing Marcel Campos told The Verge, "but we cannot get away from what users want. You have to follow the trends."

[https://www.theverge.com/circuitbreaker/2018/2/28/17062030/i...](https://www.theverge.com/circuitbreaker/2018/2/28/17062030/iphone- x-clones-android-mwc-2018)</comment></comment>


Ask HN: PR and Project Post Mortems? - <AUTHOR>

I am interested in hearing personal experiences with deciding to end a project. How did other people react to the news? How did that make you feel? What did you do? What do you think you should have done differently?
I am getting pity emails, like people think I am ending my project for emotional reasons. This seems to be a pretty standard reaction to me personally, one which both baffles and offends me. I routinely inspire the "gosh, I feel so sorry for you" type reaction. It is usually not accompanied by any genuine attempt to be helpful. It is generally delivered in the most insulting way possible, like any personal problem I have ever had is evidence of stupidity and incompetence on my part, which is not remotely true.
Further, people who valued the information I was trying to share so little that they could not remember the url and had to ask me for it are announcing how cherished my wisdom is and they want to copy it while it is still available. I know this is actually pretty standard human behavior. If you announce something is banned, being discontinued, or otherwise of very limited availability, people who never had any interest will suddenly want it. I have had a class on social psychology. I know this is a well studied, well established phenomenon. But it is really making me grumpy.
So I am trying to figure out what to do that is more constructive than giving a piece of my mind to some total stranger who has sent me a pity party email. I have other reasons to be grumpy this morning, on top of this. So my usual objectivity and relative aplomb are escaping me.
I am hoping someone else who has been through something similar can clue me.
Thanks. <comment><AUTHOR> Another:

 _Shutting down a side project, my experience_

<http://news.ycombinator.com/item?id=5314735>

</comment><comment><AUTHOR> So I can find them later, cuz where were y'all when I was asking about this? Hello!

<http://news.ycombinator.com/item?id=5280860>

<http://news.ycombinator.com/item?id=5282143>

</comment><comment><AUTHOR> Just commenting so I can more readily find myself and check for comments when I get back from lunch.</comment>