Show HN: My Portfolio Website - <AUTHOR> https://port.greyface.now.sh/ <comment><USER2> I think this site would be perfect if you toned down the existential dreariness and replaced it with small sprinkles of humor, but mix in some more technical stuff (what are your benefits).

Being able to endlessly think about life and death isnt a benefit for someone who wants to hire you, and you won't get extra points for being "deep".

If that is who you are, you can leave a tiny single sprinkle of it in there somewhere, but make sure you also cover a lot of other areas too.

Very nice creative idea, to set yourself apart. That's always what I look for

<comment><USER3> I couldn't have said it better or nicer. Also the todo list example looks a lot like that, an example that could be simply git cloned. If the web/ CSS stuff is original you have more to show!

<comment><AUTHOR> I'll add the GitHub links to the respective pages, thanks for bringing attention to that.

<comment><USER2> It appears this section might be incomplete:

Branch Prediction pipeline flush, predict taken, predict not taken, branch delay slots

<comment><AUTHOR> Wut

<comment><USER2> Ugh sorry, I thought this was another one of my comments

</comment></comment></comment></comment></comment></comment><comment><USER4> Most people have everything else covered, but . . .

I'm guessing, but English is probably not your first language.

Use something like Grammarly for free (or even Word or Libre for *nix), put all of your text into it and use the suggested grammar edits. The things I noticed were simple like capitalization, no spaces after punctation, some misused words.

<comment><USER5> Absolutely, heed this. Nothing will make some people stop reading like the laziness (in my opinion) of incorrect spelling and grammar.

You're not always publishing in your (or someone else's) native tongue, but there are services (free and paid) and software (free and paid) for this.

</comment></comment><comment><USER6> Points for creativity on the boxes, but I think you could clean it up a bit. I also can't help that notice the todolist in your portfolio has a bit of UI issue. The vertical padding between items in the list seems odd, and they should be left indented much more IMO.

I don't want to discourage you, just want to give you some helpful feedback from the perspective of a prospective client.

</comment><comment><USER7> Thanks for sharing, full of character and personality, I like it!

</comment><comment><USER8> Suggest reducing line-height and horizontal padding - the proportions don't make sense for mobile browsers.

Your custom font takes a while to load - consider CDN? Or maybe my phone just slow.

</comment><comment><USER9> Love that 'Download your Art' at the end!

</comment><comment><USER10> Lovely! It definitely shows off your curious and experimental nature.

Had a fun time drawing some flowers :P

</comment><comment><USER11> Typo: "an adrenaline" not "a adrenaline"

</comment><comment><USER12> Super awesome! Points for creativity.

</comment><comment><USER13> The screenshot of the To-Do List is pretty bad

<comment><USER4> Lol, it works well. What else could they do better?

<comment><USER10> @2x maybe?

</comment></comment></comment><comment><USER14> scrolling the first card flickers on Firefox/android when the page height changes</comment>


We must have the right to blaspheme against Islam - <AUTHOR> https://www.spiked-online.com/2019/05/06/we-must-have-the-right-to-blaspheme-against-islam/#.XNMe6XfCCLo.hackernews <comment><USER2> It is important to have a right to blaspheme against any religion. People have fought hard against Christianity to gain this right and that helped to come to the modern secular state, and modern non-zealous religion when saying that bible is a lie won't get anyone killed.

We must help the people from remaining highly religious countries to do the same.</comment>


Why Google Still Desperately Needs Firefox - <AUTHOR> http://www.extremetech.com/internet/92558-how-browsers-make-money-or-why-google-needs-firefox

<comment><USER2> interesting. Not sure how serious the potential 'shift' to Bing would be, as I think alot of Ffox users would still change it to Google as the default.

Do you think Microsoft Bing is actively courting Moz on this issue ahead of the contract expiry?

<comment><USER2> ohhh... that's what all the release cakes sent to the Mozilla HQ from the IE team are for!

</comment></comment><comment><USER3> Firefox's Mozilla Foundation is probably the most valuable non-profit in the world right now.

Boy they have leverage.</comment>


Best Books on Health – A modern approach to a healthy lifestyle - <AUTHOR> https://bestbooksonhealth.com/ <comment><USER2> Just another collection of Amazon affiliate links (tag svprogramming-20) and descriptions ripped wholesale from other sites.

<comment><AUTHOR> Have you already read these books?</comment></comment>


Apple employees can't stop walking into the glass doors at Apple Park campus - <AUTHOR> https://www.theverge.com/2018/2/16/17020246/apple-park-headquarters-employees-injury-glass-doors-design <comment><USER2> That's from over a year ago. (Eg, posted [https://news.ycombinator.com/item?id=16398594](https://news.ycombinator.com/item?id=16398594) ). Is it still true?</comment>


101 Features Android Phones Had Before iPhones - <AUTHOR> https://android.gadgethacks.com/news/101-features-android-phones-had-before-iphones-0184066/ <comment><USER2> Is 3rd party bloatware number 1?</comment>


Anti-Piracy lawyers sentenced to prison for defrauding copyright holder clients - <AUTHOR> https://torrentfreak.com/anti-piracy-lawyers-sentenced-to-years-in-prison-for-defrauding-copyright-holders-180612/ <comment><USER2> We can also attribute the quote "Child pornography is great" to Mr Schlüter. Overall nasty guy.

[https://falkvinge.net/2012/05/23/cynicism-redefined-why- the-...](https://falkvinge.net/2012/05/23/cynicism-redefined-why-the- copyright-lobby-loves-child-porn/)

 _"My friends," Schlüter said. "We must filter the Internet to win over online file sharing. But politicians don't understand that file sharing is bad, and this is a problem for us. Therefore, we must associate file sharing with child pornography. Because that's something the politicians understand, and something they want to filter off the Internet."_

<comment><USER3> Misusing a sensitive topic to push your agenda is never fair.

And not to defend the guy, but the quote is a bit misleading and guilty of the same sin: using compassion for children as a leverage.

</comment></comment><comment><USER4> I remember one of these guys from back in the early computing days. He would go after young kids for pirating c64 games. What a prick.

<comment><USER5> They still do, pretty much.

</comment></comment><comment><USER6> A similar thing happened in Iceland a few years ago. Article is in Icelandic: [https://www.mbl.is/frettir/innlent/2017/05/05/snaebjorn_daem...](https://www.mbl.is/frettir/innlent/2017/05/05/snaebjorn_daemdur_i_16_manada_fangelsi/)

</comment><comment><USER7> The only crime here is that they were allowed to collect money in the first place from innocent victims. They should have gotten life and no one would be sorry for them. But then again, so should their clients.

</comment><comment><USER8> So the record companies got the same treatment as they gave their artists? Delicious.</comment>


The HipHop Virtual Machine - <AUTHOR> https://www.facebook.com/notes/facebook-engineering/the-hiphop-virtual-machine/10150415177928920

<comment><USER2> Sounds like they are some experienced software engineers:

>The first 90% of the hhvm project is done; now we're on to the second 90% as we make it really shine.

<comment><USER3> Really experienced mention the third 90% [http://morepypy.blogspot.com/2011/11/gothenburg-sprint- repor...](http://morepypy.blogspot.com/2011/11/gothenburg-sprint-report.html) (search 90%)

</comment></comment><comment><USER4> Couldn't help but chuckle at the code example

      return $hit

<comment><USER5> Yes its amazing how easy it sneaks in! Was looking at a gist of mine recently (<https://gist.github.com/1222875>) and it suddenly jumped out at me!

      my $hits = ...

      for my $hit (@$hits) {
          my $user = $hit->{_source};
          ...
      }

:)

</comment></comment><comment><USER6> The state of the art in JITs is LuaJIT2, despite almost no fanfare (and lots of fanfare by other JIT projects). I haven't had a look at hhvm yet, but if anyone wants a cool project, make hiphop emit Lua instead of C++, and run it under LuaJIT2.

LuaJIT2 really is magic, and Mike Pall is a humble but talented magician.

</comment><comment><USER7> This is a good reminder not to build anything other than a prototype in a language that you're going to regret later.

<comment><USER8> If they started with Python (or anything else) they would have ended up in the same place. They would end up writing their own compiler and VM because they want and they can. It's actually amazing that they didn't extend the language itself yet.

<comment><USER9> They would be sitting on JIT-ed pypy 1.7 at this point, which can yield over 10 times speedup in some cases- I've tested some of my own code.

So having superior language with better out of the box speed in default implementation - I'd hardly call same place.

<comment><USER8> ... or they would start their own project similar to Unladen Swallow.

I think picking simple language like PHP makes it much easier to create your own fully custom stack once you need one.

Ofcourse Facebook problems will never be your problems so every one should pick what is right for him at any given moment and stop dreming about (im)possible future.

<comment><USER9> Actually I think it would be different case - they would probably not need to reimplement language in their own way - because speed benefits are already there (when it comes to pypy). Instead they would patch other libs to work with pypy on a need basis. Which seems way less work than start from scratch and implement whole lang - and community would benefit a lot from this too. But since it will not ever happen it's just speculation at this point.

</comment></comment></comment></comment></comment><comment><USER10> Excuse my ignorance of LLVM but couldn't someone build a PHP LLVM JIT compiler and stand of the shoulders of that project?

I am sure I am trivializing the effort that it takes to make JIT for PHP.

<comment><USER11> LLVM isn't the worlds greatest target for a dynamic language, as Rubinius and Unladen Swallow found out. It does a lot of low level optimization, but a basic prerequisite to that is being able to do some sort of optimistic type- based optimizations, and LLVM has no infrastructure for that.

It looks like HipHopVM uses tracing which is probably a better way to go.

<comment><USER12> Are they at any risk for VM patent issues?

</comment></comment></comment><comment><USER13> This is the kind of things that make me want to work in a company like Facebook.

<comment><USER14> This is the kind of things that make me _not_ want to work in a company like Facebook.

</comment></comment><comment><USER15> The hhvm is some nice piece of software, but it also takes many resources. This will always remind me to be very careful when making a design decision, like what language to use for project.

</comment><comment><USER16> We are looking into running our API on HipHop, but not until the load justifies the effort to migrate. Currently running on lighttpd with PHP-FPM.

</comment><comment><USER17> Looks like an impressive, cool project.

The justification for why it matters seems a bit "off" to me:

      For perspective on why this matters, consider that many Facebook engineers
      spend their days developing PHP code in an endless edit-reload-debug cycle.
      The difference between 8-second and 5-second reloads due to switching from
      hphpi to the hhvm interpreter makes a big difference to productivity,
      and this improvement will be even more dramatic once we enable the translator.

Big leap of intuition follows, bear with me:

Clearly there are some very talented engineers working at Facebook, as evident by this project. On the other hand, apparently a large number of Facebook engineers are spending all their time in a run-debug cycle, trying to "make this darn thing work," and the engineers with talent are being used to incrementally improve the mediocre coders' lackluster productivity.

Guys, if three seconds in compile overhead makes such a difference in your productivity, maybe you should think for a few seconds about code correctness before you hit the compile button.

<comment><USER18> All engineers, those who work on HHVM like me included, spend their lives in a run-debug cycle trying to "make this darned thing work," whether "this darned thing" is a language runtime or the new photo uploader. The tighter you can make that loop, the more productive the engineer is, because she has less time to keep all the items she's mentally juggling pinned in volatile short-term memory.

<comment><USER17> Sure, we all go through life via a form of trial and error, learning from our mistakes.

Personally, I think the fewer times you have to go around that wheel, the better.

I don't want to diminish the technical excellence of the achievements touted, and as a coder of primarily compiled languages, I would welcome any such improvements in the compilers I use.

However, my larger point stands-- which is that if a few seconds shaved on the run-debug loop is really a big deal for your total productivity, it means you're looping too much.

Trial and error is a fine way to learn a language, or to debug truly mysterious issues, such as those that exist outside of the abstraction layer you're working in. But in my opinion it's a poor way to work. It means that you don't understand the code you're writing.

<comment><USER19> I disagree. My anecdotal evidence as a person who prefers to meditate on code is that many times the person who does quick iterations even beats people that understand the language. A deep understanding of the language, in fact, can sometimes be a hinderance to getting things done. I realize thats anecdotal, but if you have an organization (say 1000 people or whatever facebook has) and they are doing these iterations all day and you can shave 5 minutes more out of each persons day, that 5000 mins per day :)

<comment><USER20> At many companies, those 5000 minutes "saved" will be later squandered on Facebook. :)

</comment></comment></comment></comment></comment><comment><USER21> "The x64 machine code that the translator generates consumes approximately ten times as much memory as the corresponding HHBC. CPU instruction cache misses are a limiting factor for the large PHP applications that Facebook runs, so a hybrid between interpretation and translation may outperform pure translation."

This tells me that the engineers on the hhvm project are at least smarter than the engineers on the Java language.

<comment><USER22> what did the java engineers get wrong? Hotspot performs fast interpretations then compiles after multiple run of a piece of code.</comment></comment>


A form of offline payments is being developed for Bitcoin's Lightning Network - <AUTHOR> https://decrypt.co/10655/lightning-rod-offline-payments-bitcoin-lightning-network <comment><USER2> > Sheinfeld said he would not describe the functionality as "offline Lightning > payments" but as the "ability to pay someone and go offline right after > you've sent the payment"—so the payee is offline during the payment.

So not a fully offline method.</comment>


Why We Make Bad Decisions as Groups and How to Fix It - <AUTHOR> https://thetoolsweneed.com/why-we-make-bad-decisions-as-groups/ <comment><USER2> To paraphrase this article - 'if we trust Sarah to vote as we wish on the environment, and Frank to vote as we intend on Finance', what I don't understand is how would delegative democracy work when it comes to working out the environmental budgets for that year?

edit: additional clarifications

<comment><AUTHOR> That's a good point. I think it's hard to definitively know the best way to resolve issues like this without an actual platform to test it on.

One point that I wasn't able to fit into the article is that this idea isn't a homerun. There are lots of difficult things to figure out. I just think that it's hard to find a problem more deserving of solving, and right now we have our best minds solving the problem of how to spy on people better instead.</comment></comment>


Natural Language in Python Using SpaCy - <AUTHOR> https://blog.dominodatalab.com/natural-language-in-python-using-spacy/ <comment><USER2> This is a good tutorial on using spacy with Python. In my application we are leveraging voice recognition in chrome browser.

Probably will explore use of spacy for NLU and NLG to built a natural language interface for our application.

So far been using Google NLP API's for it powered by its pre-trained models.</comment>


Programming Can Be Fun Again - <AUTHOR> http://blog.polyglotprogramming.com/2011/9/5/programming-can-be-fun-again <comment><USER2> Dean Wampler keeps repeating this idea that ORMs have been an epic wrong turn for coders and that SQL-wrangling functional tools like Scala Play's Anorm represent a much more efficient way forward. I agree that being able to map/filter/fold over record collections is very cool, and definitely part of the solution, but it's crazy to think that going back to embedding brittle, non-type-safe, non-DRY, database-specific SQL calls into a host functional or OOP language is the best way to do this. We're using Squeryl for Scala and very happy with it - it's fully type-safe, uses very easy to define case classes for each db record type, and crucially allows all of that FP fold/map/filter goodness over the data models which are exposed.

Joyless heavyweight frameworks and flabby design patterns are definitely the problem, but Dean's survivalist-style head back to basics and raw SQL isn't going to let us build rich complex software in 2011.

</comment><comment><USER3> As someone who only does functional programming I do not think their power lies in their erm functionality. I have a hypothesis.

I think the key lies in how easy is it to look at your code and trigger the pathways in your brain associated with elegance. No offence to Java, it gets the job done but due to boilerplate, the likelihood of your code resulting in such triggers is much less than say clojure or scala. Consider the popularity of CofeeScript, node.js, python, ruby. These are not idiomatically functional languages but due to reduced boilerplate they too also make it easier for your code to be appreciated on an aesthetic level. What is the gain in this?

I have a belief that an appreciation for elegance is the natural outcome in a universe with a principle of least action. Prefer the explanation which takes less energy to unfold. So anways what happens is that your brain rewards you for writing code that tries to be elegant and turns aesthetic knobs. Pushing you for more and more of a hit. And you start to become more motivated to write code that rewards you thusly and a side-effect being programming is more enjoyable and you are more likely or encouraged to write code that is appreciable. Since code is meant for humans, as a start, optimizing in this way is ideal. Train yourself to appreciate algorithmic and mathematical elegance and you reap the rewards there too.

<comment><USER4> I don't know functional programming but I agree with your sentiment. I even think this is the reason why languages like C++ and Java are abused so much. Over the years of use, people start learning little tricks and hacks, and they want to deploy those. They find them elegant, while everyone else who has to read them...actually thinks they are horribly overcomplicated. C++ template programming is awful, but some people love it because they love how elegant their code can become. They love that they can immerse themselves in the language features. Unfortunately this provides a net detriment to the code you are writing. No matter how good your code is, someone will have to read it again, even if it's just you. Code should be simple. I believe code should be difficult to write bad. In the common languages though it's very easy to write undecipherable code. In Javascript for example, it's hard to write code that needs more than a few moments to understand. It's even hard to write "class" hierarchies!

</comment></comment><comment><USER5> Functional programming is admittedly fun, at least for those who are mathematically inclined. The problem is that it's hard, at least at this stage, to justify using a functional programming language as your _main_ tool.

Want to develop web applications? Nothing from the functional programming camp comes even close to the polished experience of using Ruby on Rails. Yes, I know about Play and other web frameworks for Scala, Clojure, and Haskell, but - and I repeat - they aren't even remotely close to the level of support, libraries, plugins, and documentation that you can get from the Rails world.

Want to develop mobile apps for iOS or Android? That's typically Objective-C or Java territory. Yes, you can develop Android in Scala, but you are a pioneer then (and you are virtually on your own if you encounter any issues with the stack, as very little documentation and community support is available for it).

Want to develop desktop applications? Again, functional programming languages are rarely the best supported option. C#, Objective-C, C++, or even Object Pascal (Delphi) are all fine choices. Opt for Scala, Clojure, or Haskell and you'll find little documentation and support, and in the case of the first two, you'll have GUIs that look anything but native.

So what are these functional programming languages good for? So far I've only found them to be good at doing heavy lifting in the back-end or in GUI-less scripts. An important spot for sure, but not the most prominent one to justify adopting either as my _main_ tool.

I love functional programming languages and I wish I could drop everything and write web apps in Clojure or Haskell. But the hustler in me wants to get stuff done and cannot justify the world of pain and waste of time I'd experience, especially when there are already tools that are simply better suited for the task at hand.

Should a project of mine scale to the point of needing a fast functional programming language for core functionality in the back-end, I'll be happy to employ it there.

<comment><USER6> Completely disagree about Ruby on Rails, the experience of using happstack ( <http://happstack.com/index.html> ) blows it out of the water.

The RAM Cloud database is really natural to use; it eliminates problems like object-relational mismatch and SQL injection by letting you do everything in idiomatic Haskell.

Plus, the type-system, monads, and algebraic datatypes make it much easier to write clean, correct code. Albeit, only once you've passed the learning curve.

<comment><USER7> That learning curve is the problem. Building a business ultimately requires people to be cogs at some level.

</comment></comment></comment><comment><USER8> "programming did become fun again when I started using functional programming as my main paradigm"

That's exactly what happened to me.

Even though I mostly program in C# (at work) and Ruby (for personal projects) these days, my code and my team's is full of functional idioms, mostly thanks to LINQ, lambda support in C# and lazy evaluation (IEnumerable generators).

Now our code is much shorter, way more readable and much safer compared to what it used to be. We avoid mutability and state changes (meaning, we don't use mutable variables or flags), most functions are testable by themselves, code reuse is much better, objects are much thinner, and even 17 old interns are fluent on such idioms by now. In fact, most new hires already come with LINQ experience, which is cool.

I don't know why the "fun" factor was increased so much, but it did for me. I believe it is because we're forced to think in a straight, linear way that produces the cleanest, safest code possible. It is like some kind of puzzle. It's just like working in Haskell: you have to think before you code.

Well, also it helps a lot that mundane tasks such as calculating the largest number in a list is reduced to list.Max()...

</comment><comment><USER9> Some of us never thought it stopped being fun.

</comment><comment><USER10> I've been working on a Cloure project for a few months now and, as the article says, it just felt natural to use the objects that came directly from the (relational) database without doing any ORM-type manipulation. My years of non-functional experience meant that I've had this niggling doubt that I've been doing the right thing. It's nice to have some validation...

</comment><comment><USER11> I find C and C++ fun to _write_. The difference is that functional languages make (50th-percentile) code fun to _read_. In C++ and Java, even 90th- percentile is a bitch to read, hence people don't. At best, those responsible for maintaining it will skim it. This is also why the "broken windows" pattern (once things get ugly, people "hack" them and they get uglier) sets in so rapidly in C++/Java projects.

Functional programming encourages beautiful code (although it certainly _can_ be ugly.) A consequence of this is that people, even if they're not skilled enough to write pretty code, actually read code-- not because they need to, because they want to-- and become better programmers as a result.

Functional programming, for a passionate programmer, refines the aesthetic sense and 2-4 years of it will give most people the skill to be a passable software architect (whether they are passable or great depends on talent IMO). Not so with C++ or Java. You can spend 10+ years in those languages and get really, really good at optimization but still have no aesthetic sense.

You also learn faster when using FP languages because you can _do_ more in them. You get to see the consequences of your architectural decisions long before the project goes multi-developer and is handed off to someone else or a team.

Functional programming is not a great career move. It makes you a difficult fit for 90% of jobs that are out there, and it makes a lot of people think you're a hipster jackass. But it's rocket fuel for your skill as a programmer.

<comment><USER7> I think this is one of the commentaries I've read on FP. I was a FP junkie and DSL addict until I realized my business was going to fail because I couldn't hire the right people at the budget I had.

It is why I'm an alcoholic now. Java makes me drink as I wait for Ocaml to be main stream

</comment></comment><comment><USER12> from the article: " as a Scala advocate, many developers complain to me that the immature support for Scala in IDEs is a real blocker for introducing the language in their Java environments. I understand the practical issue of meeting this now-typical expectation of Java shops, but I actually don't really care. I seldom use IDEs anymore. My Scala code is so compact that I use Emacs for it."

sUnless you happen to be doing greenfield development, this is a dealbreaker -- I like using scala, but it has to call / manipulate the enormous api surface from 1.5mm lines of existing java. And for that, I need tool support.

</comment><comment><USER13> I've recently started having fun (for the first time in years) tinkering with Project Euler problems, writing C#, and playing with Arduino. Android looks like it's going to be fun, but so far it's a lot of mucking around the huge infrastructure they created, trying to figure out how to do what I want.

C++ is no longer fun for me.</comment>


Complaints I'm Seeing About Java - <AUTHOR> http://dlweinreb.wordpress.com/2007/12/08/complaints-im-seeing-about-java/

<comment><USER2> One complaint I've always had that no one else seems to bring up: I can't take a flat array of bytes and map a struct over it. You have to fake that out in the most lamentable way...

I guess most people don't need to do that anymore.

<comment><USER3> I guess you are right, because I don't even understand what you mean?

<comment><USER2> struct MyStruct { int a; } ;

char bytes[]={0,1,0,0};

MyStruct *myStruct=((MyStruct [asterisk] )bytes);

...myStruct->a is now 256.

How do you format code here?

<comment><USER3> I see - I am glad that Java does not support that ;-)

Presumably what you want can be achieved with ByteArrayInputStream and DataInputStream.

        byte[] bytes = {0,1,0,0};
        DataInputStream in = new DataInputStream(new ByteArrayInputStream(bytes));
        int a = in.readInt();
        in.close();

untested, but something like that... There is also object serialization. I never had a use case for what you describe.

(format code with 4 leading spaces)

<comment><USER2> Guess how many CPU cycles my C code requires.

<comment><USER3> What time sensitive operation requires you to transform byte arrays into objects? I assumed something like reading from disk or network, and then CPU cycles are usually not the bottleneck.

Anyway, no doubt that you can write faster code in C, but I am glad that I did not have to look at pointer arithmetic for the last 8 years. Using your array->struct technique would be a major headache if the struct get's more complex.

<comment><USER4> There are plenty of other practical use cases for this kind of control. For example, casting an array of byte[]s to an array of int[]s. It's not hard to convert using a ByteBuffer, but its still annoying. You and up duplicating a ton of memory for no good reason.

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> I have been programming with Java for several years now, I have professional experience with C, C++, Smalltalk and Perl.

From this list I would like to have this: \- A short cut syntax to declare getters and setters. \- A shorter syntax to use "functions as objects". \- Easier to use stream classes.

<comment><USER6> I also get tired of auto-generating getter/setters all the time. I think annotations to designate getters/setters would be easy and fit in nicely. No need to invent new class "properties" to complicate things.

</comment></comment><comment><USER7> How about code blocks and yield? I recently started learning Ruby, and this has been one of the first major transitions in how I write code (after programming in Java for several years).

<comment><USER8> Any pratical application for yield that you couldn't do just as simply with a for loop? I'm not criticizing, I'm just curious.

<comment><USER7> Well, I can't guarantee that this is was a good use of block and yield, or even if it was advisable, but here's what I did one time...

I needed to construct a bunch of polynomials from a set of 20 odd coordinates (x,y) coordinates. I decided to use langrange interpolating polynomials, and I wanted the polynomial to match three of the twenty points (otherwise they bounce all over the place). The idea was then to select the polynomial that minimized the least squares difference with the other points that weren't used to create the curve(I'm not sure if this counts as practical, since there are other approaches to polynomial curve-fitting that would have been more appropriate). Anyway, I had a method that evaluated the least squares, and then I constructed the polynomial as a block of code and yielded to it in the iteration that evaulated all the point for the least-squares fit...

There are other ways to do it (always are) - I could have broken the polynomials into terms and reconstructed them in the method, for instance - but block and yield were very convenient in this case.

</comment></comment></comment><comment><USER9> Woa, I just had a year 2000 flashback!

</comment><comment><USER10> Way to stick your head deeper into the sand!

<comment><USER11> Considering how Dan Weinreb is one of the designers of Common Lisp, perhaps it is not he who is narrow minded.

<comment><USER10> Uhm, so what? It's not like he's infallible. His comments on why Java's misunderstood are silly! Judge the man by what he says, not by his past reputation.</comment></comment></comment>


Anatomy of a Haskell-Based Application, Revisited - <AUTHOR> https://tech-blog.capital-match.com/posts/3-anatomy-of-haskell-web-app.html <comment><AUTHOR> Love this bit: "...we are happy that we chose Haskell, a wonderful language that enables large refactoring with ease, perhaps even turning what could be unthinkable architectural changes in other languages into pure tedium."</comment>


Why do we use the Linux kernel's TCP stack? - <AUTHOR> http://jvns.ca/blog/2016/06/30/why-do-we-use-the-linux-kernels-tcp-stack/ <comment><USER2> Please don't rewrite your network stack unless you can afford to dedicate a team to support it full time.

Twice in my career I have been on teams where we decided to rewrite IP or TCP stacks. The justifications were different each time, though never perf.

The projects were filled with lots of early confidence and successes. "So much faster" and "wow, my code is a lot simpler than the kernel equivalent, I am smart!" We shipped versions that worked, with high confidence and enthusiasm. It was fun. We were smart. We could rewrite core Internet protocol implementations and be better!

Then the bug reports started to roll in. Our clean implementations started to get cluttered with nuances in the spec we didn't appreciate. We wasted weeks chasing implementation bugs in other network stack that were defacto but undocumented parts of the internet's "real" spec. Accommodating these cluttered that pretty code further. Performance decreased.

In both cases, after about a year, we found ourselves wishing we had not rewritten the network stack. We started making plans to eliminate the dependency, now much more complicated because we had to transition active deployments away.

I have not made that mistake a 3d time.

If you are Google, Facebook or another internet behemoth that is optimizing for efficiently at scale and can afford to dedicate a team to the problem, do it. But if you are a startup trying to get a product off the ground, this is Premature optimization. Stay far, far away.

<comment><USER3> The original article claims that having the TCP stack in the kernel causes performance problems because it needs to do excessive locking.

I can't judge, but if really that is true, then in principle, a user-space library could be written to take care of all those corner cases you mention, and still be faster than the kernel stack.

Of course that wouldn't be everyone rolling their own.

<comment><USER4> I've only poked at the FreeBSD TCP stack and not the Linux stack, but it seems like if the problem is locking, you should be able to get good results from working on the locking (finer grained locks / tweaking parameters) in less time than building a full tcp stack.

What kind of limitations are people seeing with the Linux kernel? If I'm interpreting Netflix's paper[1] correctly, they could push at least 20 Gbps of unencrypted content with a single socket E5-2650L (document isn't super clear though, it says they were designed for 40Gbps). My servers usually run out of application CPU before they run out of network -- but I've run some of them up to 10Gbps without a lot of tuning.

[1] [https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf](https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf) Context is accelerating https downloads, but some decent numbers anyway.

<comment><USER5> Gbps is not created equal. Traffic with many small packets takes a lot more resources compared to traffic with fewer but bigger. Netflix packages would be as big as they come.

<comment><USER6> Yes, in fact for things like Juniper/Cisco firewalls they will always quote PPS in full MTU packets. If you want to bring that shiny new firewall to its knees try sending it traffic with the minimum MTU of 68 bytes at line rate for the NIC.

</comment></comment></comment></comment></comment><comment><USER7> Years ago I worked for a company who was building a piece of networking hardware, and we had bought some code that implemented various well-known routing protocols. This included the TCP/IP layer, but it was just the BSD stack.

Our dev hardware had a boot loader that would use FTP to download the most recent images to finish the boot. After we cut over to this code, we started having an occasional problem where this download would fail -- the box would send a spurious RST and kill the connection. I was one of the protocols people, and it ended up on my plate.

I spent 3 days staring at tcpdump, adding enormous piles of debug traces, and finally going through the code line-by-line with a copy of the Stevens book open on my lap. Eventually I found the problem - they had made one single change to the BSD code, and in so doing introduced the bug. It was compiler- dependent.

Writing this code correctly is hard.

<comment><USER8> > It was compiler-dependent.

Reminds me of a bug i was wrangling in GTK, now hopefully long gone, that only showed up if the lib was compiled with a certain level of optimization set in GCC.

</comment></comment><comment><USER9> A few quick notes:

\- The Linux network stack has been getting faster. Workarounds that may have made sense 5 years ago, vs 2.6.something, may make less sense today vs 4.7. It makes it hard to research the topic, as you'll find statements of fact that are no longer true today.

\- The kernel stack is complex, but then it also handles a lot of TCP/IP behaviors. Eg, buffer bloat. You might code a lightweight stack that works great in microbenchmarking (factors faster), but falls apart when connected to real devices. I've worked on differing TCP/IP implementations that had this behavior, and the stack that was slower in the lab was faster for the customer's real world messy workload.

\- What is Facebook doing? eXtreme Data Path: [https://github.com/iovisor/bpf- docs/raw/master/Express_Data_...](https://github.com/iovisor/bpf- docs/raw/master/Express_Data_Path.pdf) ... this is really new and exciting stuff.

\- Unikernels :-)

<comment><USER9> Another technology worth mentioning is the Linux network parallelism features (which I think came from Google):

[https://www.kernel.org/doc/Documentation/networking/scaling....](https://www.kernel.org/doc/Documentation/networking/scaling.txt)

My colleague at Netflix, Amer Ather, wrote about this tuning and more in a post detailing some network tuning from a Xen guest, where he was reaching 2 million packets/sec:

[http://techblog.cloudperf.net/2016/05/](http://techblog.cloudperf.net/2016/05/)

</comment></comment><comment><USER10> I worked on ICQ project as a backend developer (C/C++) after Mail.Ru bought ICQ from AOL.

We had legacy from AOL - AOL's proprietary TCP stack in user space.

ICQ used AOL's TCP stack for external connections with outer world (i.e. with clients).

I asked AOL engineers why they needed TCP stack in user space. They said that back in old days (90s) there was no good scalable TCP implementation which could handle many simultaneous connections properly.

Each process which used this TCP stack had to fully own network interface.

Of course, we gradually replaced AOL's TCP stack with native stack because nowadays Linux TCP stack is good enough.

May be these days someone still need proprietary TCP stack in user land to meet specific needs which aren't available in native Linux implementation.

Also if you have TCP stack in user land, you have more control over it.

<comment><USER11> AOL had a bespoke webserver too.

[https://en.wikipedia.org/wiki/AOLserver](https://en.wikipedia.org/wiki/AOLserver)

<comment><USER12> I really liked AOLServer! My current company's first website ran on OpenACS, which was written in Tcl and ran on aolserver. It was really quite a nice system (and Tcl is a quite nice language).

But, the web server worked with the standard Linux network stack; at least, it did by the time I saw it.

<comment><USER13> Tcl may be the most under appreciated language.

</comment></comment></comment></comment><comment><USER14> The core functionality required is a "zero copy" networking lib : dpdk, netmap.

Normally, when the kernel recieves data from the network, it allocates a block in the kernel and copy the data into it. Then your read operation copies that data in your user space. The "zero copy" networking stacks avoids the data copy. The way it works, as I was explained, is that they use a shared memory mapped zone. This zone is organized as a pool of blocks managed with non blocking lists. Blocks have a fixed size big enough to hold the ~1500 IP blocks. I never used it so I don't know the details.

When data arrives, it is directly written in place in a block of the memory mapped zone. In the user space you use select/epoll/kqueue or polling if the waiting time is very small. Once you have a block you can process it. This block contains raw network data received from the network card. So it's up to you to code and decode the TCP/IP headers or use an existing lib like mTCP that does that for you. I was told that it can work with dpdk and netmap.

My colleague is currently using netmap for a high performance data acquisition application in a LAN and plan to test dpdk with mTCP this summer. mTCP should simplify programming. At CERN they are now testing data acquisition setups using dpdk to be able to use commodity component hardware.

My colleague told me that Netmap is available in the BSD kernel so that you can use it right away. It is not included in the Linux kernel and you then need to patch it in. Zero copy is the future of network programming. Linux is late on this one. Then there is dpdk on which I don't have much info yet except that it is made by Intel, it is open source and compatible with AMD processors. It is apparenly not easy to install.

Since dpdk and netmap communicates directly with the network card, it only works with supported network cards.

The gain in performance is significant, but I have no numbers at hand to give.

<comment><USER15> FreeBSD Netmap user here. You actually have to recompile the kernel with "device netmap" added in your kernconf. Piece of cake, after 20' you are good to go. But you need a real network card and the FreeBSD driver must be ready for netmap. Using intel 10Gbps adapters (~200euros) is a safe avenue (FreeBSD ixgbe driver). Even in VMWare, you can pass-through the PCI address of the adapter port to your virtual machine and have it talk to the card directly. Everything works very good.

The gain in performance is mind boggling! Trying to sniff approx. 2+Gbps traffic with Suricata using the "normal" avenue of libpcap ends up dropping a small percentage of the packets. And the machine will waste incredible CPU. Using Suricata with netmap (no need to recompile, Suricata pkgng binary build from FreeBSD comes ready) uses exactly one capture thread and drops ZERO packets. This behavior is stable for days!

Netmap is hands down awesome.

<comment><USER16> I was looking at Chelsio NICs and there were mentions of Netmap support. Do you know what it means for a NIC to support Netmap vs one that doesnt? Is it an extra optimization/fast-path?

<comment><USER15> Can't give a good technical answer to that. But I suspect that it should be a matter of driver mostly. When you mmap /dev/netmap from userland, the OS TCP/IP stack is disconnected and you get access to the card tx/rx rings. Obviously the driver has to facilitate this.

</comment></comment></comment></comment><comment><USER17> People who are interested in this should take a look at DragonflyBSD: [https://www.dragonflybsd.org/](https://www.dragonflybsd.org/)

The things this quite claims are crucial to high performance networking, that need to be done outside of the linux kernel, are already done in the DragonflyBSD kernel:

        The key to better networking scalability, says Van,
        is to get rid of locking and shared data as much as
        possible, and to make sure that as much processing work
        as possible is done on the CPU where the application is
        running. It is, he says, simply the end-to-end
        principle in action yet again. This principle, which
        says that all of the intelligence in the network
        belongs at the ends of the connections, doesn't stop at the kernel."

Dragonflybsd solves the locking and shared data problem by running a network stack per cpu core. If you've got 4 cores, you've got 4 kernel threads handling TCP. I don't understand the details of scheduling processes to the cpu handling its network connections, but Matt Dillon, the head of the project claims it's accounted for in the scheduler:

        <dillon>	we already do all of that
        <dillon>	its a fairly sophisticated scheduling algorithm.
        How processes get grouped together and vs the network
        protocol stacks depends on the cpu topology and the load on the machine

```

edit: If you want to try it out on a VPS, vultr is the best one I know that allows you to upload custom ISOs ([https://www.vultr.com/coupons/](https://www.vultr.com/coupons/)). There was a bug fixed recently in the virtio drivers that should make it pretty stable running on a VPS. You'll have to wait for 4.5 (a few weeks from now) or just upgrade from master (very easy).

<comment><USER18> Is it feasible to use dragonfly bsd for small-medium sized sites? I am currently using netbsd for my servers.

<comment><USER17> I don't see why it wouldn't be feasible for a site of any size. My only concern is running into issues with hammer. I was unable to mount my hammer partition after I filled up the drive. Not sure if it's a virtio bug or something that would affect actual disks as well, but that's something to be careful with.

</comment></comment></comment><comment><USER19> We don't use the kernel TCP implementation because we decided to rethink the entire networking stack (from DNS to TCP/HTTP/TLS) as it applies to native mobile apps. So we wrote this whole stack but using UDP instead of TCP, and of course implemented entirely in userspace both on client side and server side

Today's mobile app ecosystem has the amazing advantage of rapidly updating binary apps running on a billion devices. This allows us to ship new client side code (with app updates) to millions of devices every few days. It's the first time in computing history that such rapid iteration on network protocols has been possible and the only way to take advantage is to ditch the kernel and ship your code with the app.

EDIT: I wrote a thing about it in much more detail than the space here will afford : [http://www.infoworld.com/article/3016733/application- develop...](http://www.infoworld.com/article/3016733/application- development/in-search-of-a-cure-for-slow-mobile-downloads.html)

<comment><USER20> Questions: (Disclaimer, I know only the surface of networking)

"For example, PZP is able to identify the device, rather than its IP address, as the endpoint for data packets, allowing it to accommodate the intermittent nature of mobile connections in a fault-tolerant manner":

In what way is this different from NAT? Is this somehow aware of the phone switching towers (topology changes)? Or is there another advantage? How do they get that information?

"The PacketZoom protocol recovers from dropped packets "gracefully," with minimal overhead above the amount of data lost in the dropped packets."

I don't know signal processing well enough to know how this works, I'm guessing you don't need to do things in order like TCP, so you can ask to resend and then stich it in?

Where is the beef? Is it in following devices rather than IPs? What does the interface look like?

<comment><USER21> > ... I'm guessing you don't need to do things in order like TCP ...

kind of tangentially relevant: the problem with tcp is that it conflates congestion loss with error loss. this causes backoff, ergo poor performance under conditions such as lossy links, handovers etc. etc.

udp doesn't really suffer from any of this because, well you do everything yourself. another advantage with udp, imho, is that server side can be scaled quite easily :) e.g. you can have multiple processes terminating a udp control protocol, and demuxing connections onto session tasks which can be spread around.

i am kind of _surprised_ that ip addresses for devices end up changing. this might _only_ be true if you are handing over from/to different radio access networks wifi -> lte -> wifi etc. etc. roaming within a network would (must ?) not end up changing your addresses at all. for example, in an LTE network, PGW being an ip anchor, assigns addresses to the devices, and there is only one such 'thing' per epc core (slightly convoluted when moving from one operator to another, but the basic idea is still there)

from the article linked by GP, it seems that each device needs to register with a cloud based server, via which the device traffic is routed. this initial registration can be used to exchange things like IMEI or somesuch for device identification, which then remains the same as the device moves around.

<comment><USER19> _" this initial registration can be used to exchange things like IMEI or somesuch for device identificatio"_

PacketZoom stack does create our own randomly generated identifier per app install (so two different apps on the same device will have different ids). Nothing at all that would positively identify a device or a user (IMIE, UDID etc.). That's a strict no-no by app/play store policies.

<comment><USER21> > Nothing at all that would positively identify a device or a user (IMIE, UDID > etc.) ...

if we ignore (for a moment) that play-store/app-store policies forbid imei use by applications, i am just curious why you think IMEI cannot be used for _device_ identification ? thanks !

<comment><USER19> _" forbid imei use by applications, i am just curious why you think IMEI cannot be used for _device_ identification"_

I may have phrased it unclearly. I was trying to say that things like UDID or IMEI that can be used to identify individual users and/or devices are strictly verboten by app store policies (and PacketZoom doesn't read or use any of those identifiers)

<comment><USER21> > I may have phrased it unclearly.

cool ! thanks for clarifying the whole thing.

</comment></comment></comment></comment></comment></comment></comment><comment><USER22> I've done low level network programming for most of my career, and I've done a bit in the Linux network stack, as well as writing two partial userspace TCP stacks for different applications, but both were for middlebox type appliances where the goal was primarily to replicate enough of what the TCP state machines would look like on both ends of the transaction to do make some decision (in one case for firewalling, and the other was doing analysis to try to make inferences about user behavior on one end of the connection when they could be pivoting through multiple devices in the network.

It feels like networking isn't really a special case at all, just like most things the OS networking stacks are well understood, mature, and probably not worth trying to replace if you need a flexible general purpose networking stack that balances latency, throughput, correctness, security, memory and cpu efficiency, etc. But because they are trying to balance a lot of difference concerns across a wide swath of use-cases it's not hard to implement a better version if you have a very specific need.

<comment><USER23> Very well said.

</comment></comment><comment><USER24> Well I think the main reason is that you don't want to have hardware dependencies in your software. You instead depend on the kernel socket interface, and the kernel abstracts over hardware.

Most of the situations mentioned are where you explicitly have control over the hardware:

        - embedded devices
        - high frequency trading -- obviously they control their own hardware
        - Google -- ditto, data centers have very specific hardware

My understanding is that you can rely on Intel network cards having a similar programming interface, but there is still a fair bit of diversity in other hardware (correct me if I'm wrong)

I think there's no open source user space TCP stack because then you would have to recreate all the portability of the Linux kernel in user space... although I could be wrong about this.

<comment><USER25> There are DPDK drivers for a large number of NICs, not just Intel ones. Either the Kernel or DPDK needs drivers anyhow, just depends how they are written.

</comment></comment><comment><USER26> And when there is a bug in a string function, your program sends data to the wrong IP because OOPS! No protected memory! And when your user space process crashes, nobody is there to close your TCP connections for you. That's a reasonable trade off if you really need the raw request rate.

But if your web server spends most of it's time waiting for databases and taking to caches and network file systems, the common hardware abstraction of the kernel starts being worth it. Or if your binary needs to run on heterogenous hardware. Or if your single server runs multiple separate server processes.

Hybrid memory mapped stacks may end up being the best of both worlds, though. Time will tell!

<comment><USER27> What is a "Hybrid memory mapped stack"?

Edit: Oh, you must be referring to the network stack. I thought you meant the program stack (you know, the place where automatic variables in C are allocated).

(I'm still not sure what "hybrid" is supposed to mean here though)

</comment></comment><comment><USER28> "The TCP standard is evolving, and if you have to always use your kernel's TCP stack, that means you can NEVER EVOLVE." -> This is untrue, the Linux kernel is open source, you could "easily" write your own and replace the current. Either in a branch or trying to get it in the official tree.

<comment><USER29> Context: "Google can't force Android vendors to rebase kernels but requires new TCP functionality such as TCP fast open."

I don't think they're actually doing that though.

</comment></comment><comment><USER30> Dropping in to plug SPDK ([http://spdk.io](http://spdk.io)), which is like DPDK but for storage devices. This will become increasingly relevant as SSDs become much faster with next generation media, and storage has the added benefit that all SSDs have one of a couple standardized interfaces and can share drivers.

The equivalent layer of a TCP/IP stack for storage is probably a filesystem, and the kernel filesystems and block layers are at least as inefficient as the network stack for similar reasons.

<comment><USER21> > ... which is like DPDK but for storage devices.

honest question, with so many 'things' taking matter into their own hands, how do you ensure that they play 'nice' with each other ?

any and all insights are greatly appreciated !

<comment><USER31> Two answers:

1) Modern hardware is getting much better at multiplexing resources (aka "making the things play nice with each other"). See for example sr-iov.

2) More and more applications are distributed across many machines, with the resources of each machine entirely consumed by that one application. In that case there is only one "thing" running so the problem of multiplexing goes away. This is the premise of technologies like unikernel.

<comment><USER21> > 1) Modern hardware is getting much better at multiplexing resources (aka > "making the things play nice with each other"). See for example sr-iov.

sorry, but with sr-iov, you still have one single 'control' application muxing resources for applications sitting above it. for example, with dpdk, you more or less take over the complete network card.

the 'control' might be even human in some cases :) which/who carefully lays out the resource mapping.

another example, intel's cmt/cat techniques ([https://github.com/01org/intel- cmt-cat/wiki](https://github.com/01org/intel-cmt-cat/wiki)) map nic's rx/tx rings to processors l3 cache etc. this is ofcourse assuming that userland applications have complete control over pci-e lanes etc.

now, if you have two such control applications e.g. one doing packet-io and the other doing disk-io, how do you ensure that these control applications don't starve each other out ?

in canonical settings, kernel would be democratizing (is that even a word ?) access to the underlying h/w. but since that is bypassed, we are in a strange new world...

<comment><USER30> Sharing CPU resources between user space drivers is certainly a challenge. The best way to view this is that the kernel provides a general purpose solution for sharing resources with some associated overhead. Tools like DPDK and SPDK let you opt out of that, but now you are responsible for intelligently sharing the hardware.

You, as the application developer, have a distinct advantage though - you only need to solve the problem for your application, and using that knowledge can often lead to more efficient solutions. This may mean dedicating cores to the network or disk, or it may mean working in fixed sized batches, etc.

<comment><USER21> > ... you only need to solve the problem for your application, and using that > knowledge can often lead to more efficient solutions...

this ! _exactly_ this :) imho, the fundamental re-architecture of I/O subsystem for x86 machines has kind of relegated this playing field now to mostly solving _only_ s/w problem, rather than a combination of h/w and s/w.

for example, earlier if you wanted to write a very high performance node in, say the epc-core e.g. SGW/PGW/MME etc. you would assemble a bunch of folks with very diverse set of expertise. right from h/w i/o subsystem designers who could do npu's, switch-fabrics etc. to driver dudes, to 'infrastructure' folks to application programmers etc. etc.

in the current incarnation, a vanilla off the shelf x86 machine is more than sufficient. and if your s/w architecture is _right_, you can scale quite easily.

</comment></comment></comment></comment></comment></comment><comment><USER32> Something I would really like to see in modern operating systems is the ability to do TCP like UDP, i.e. you bind a socket and then send and receive TCP packets (everything after the ports in the TCP header) as datagrams.

The problem that solves is this. Right now if you don't want to use the OS TCP implementation your choices are a) raw sockets or tun/tap or kernel drivers or something equally heavy-handed that all require privileges, or b) encapsulate in UDP.

Which makes "encapsulate in UDP" a great choice until you see this:

      $ cat /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established
      432000
      $ cat /proc/sys/net/netfilter/nf_conntrack_udp_timeout_stream
      180

The reason for that is because UDP is connectionless. With TCP a long timeout is reasonable because the state can be discarded before that when the middlebox sees the connection close.

So if you need long-lived sessions, UDP requires you to send a lot of keepalives. But "TCP as datagrams" would let you actually look like TCP to the middlebox and get the long timeout and the ability to notify it when you're done, without requiring CAP_NET_ADMIN or similar from the OS.

</comment><comment><USER33> I blame this mostly on the monolithic kernel design of gnu/Linux. I wish someone would make an effort at a hybrid or pure microkernel gnu/linux. I'm keeping an eye on minix which looks interesting as well, but the only other interesting network stack I'm aware of is dragonfly bsd.

<comment><USER16> Exactly, was going to say the same thing. If you have a microkernel, your TCP stack will be in user space and can be more of a framework where applications can be a lot more flexible in how they use it. With the right library structure you could even have custom OSI layers while reusing the rest. This would certainly be less buggy and of higher quality than everyone rewriting it wholly. In that sense, I'm glad we have efforts to reuse seL4 for general purpose operating systems, because it also has a capability system.

</comment></comment><comment><USER34> > libuinet is a library version of FreeBSD's TCP stack. I guess there's a > theme here.

That is excellent, considering FreeBSD's TCP stack is supposed to be faster and better designed than that of Linux.

</comment><comment><USER35> TL;DR: we use it because then you don't have to write your own. The real question is why it's slow. The answer to that, according to someone they quoted, is that if you write your own TCP stack, it's inside your application and less stuff has to be copied and switched around (no overhead from receiving in the kernel and then having to pass it on). So it's not that the kernel's implementation is shitty, it's that you'd have to write your own for every application if you wanted to improve it substantially.

<comment><USER36> Given your comment and the one about having written a stack from scratch twice and regretting it wouldn't in then be a good idea to use just the kernel's battle hardened code but in user space? At most try to upstream some patches so the same code and be used in userspace. All the benefits (well used code) and none of the drawbacks (context switching).

Or am I'm hopelessly naive?

<comment><USER13> I think you can do the with the NetBSD rump kernel

</comment></comment></comment><comment><USER37> I guess using a unikernel also falls under this. Do they address some of the problems of trying this on a regular OS with a user space IP stack?

<comment><USER38> Well, yeah, I guess. When context switches are the problem, you either move the application into the kernel space (which is what unikernels do) or move the network stack into the userspace (which is what netmap, etc. do)

</comment></comment><comment><USER39> One reason to write a user space stack or, at least, a custom kernel module is to avoid the plethora of copying that can occur. Routing VXLAN interfaces to some kind of layer 2 encryption bridge? The kernel is going to copy that shit like 3 times before it hits user space. Writing your own stack can often save you all that copying around between bridges and interfaces.

Edit:

I'm going to plug my own zero-copy lib for Linux in golang. It has general purpose tcp/ip suppport (though no logic for handling tcp communication itself). It's lock-free and thread safe: [https://github.com/nathanjsweet/zsocket](https://github.com/nathanjsweet/zsocket)

</comment><comment><USER40> Reminds me of khttpd [http://lxr.free- electrons.com/source/net/khttpd/?v=2.4.37](http://lxr.free- electrons.com/source/net/khttpd/?v=2.4.37)

<comment><USER16> From what I recall, while Linux doesn't have that in the kernel anymore, Microsoft's IIS still uses an httpd driver, but I don't know if it's optional. And to be clear, ktthpd never got popular. It was written during the times of 10k battles and eventually syscalls were added for Apache and friends to achieve it without an httpd in the kernel.

Edit: khttpd (real name: tux) has actually only existed in Red Hat's and SuSE's kernel and was never mainlined.

<comment><USER38> IIRC Microsoft IIS got pwned really hard because of a bug in that driver…

<comment><USER41> Not as far as I know.

<comment><USER16> [https://technet.microsoft.com/en- us/library/security/ms15-03...](https://technet.microsoft.com/en- us/library/security/ms15-034.aspx)

[https://github.com/xPaw/HTTPsys](https://github.com/xPaw/HTTPsys)

[https://ma.ttias.be/remote-code-execution-via-http- request-i...](https://ma.ttias.be/remote-code-execution-via-http-request-in- iis-on-windows/)

</comment></comment></comment></comment></comment><comment><USER42> >it can do, in some benchmarks, 2 million requests per second.

Is there a standard specification for machines that benchmark network software? Or, when someone quotes a number like this, do they mean "in our environment"?

</comment><comment><USER43> Based on the article's title, I was expecting reasons to use the Linux kernel's TCP stack - i.e. to actually answer the question. I guess it was rhetorical, though, since the whole article was about reasons why one _shouldn 't_ use Linux's TCP stack.

All those reasons, however, reek rather strongly of premature optimization, and _that_ is the best reason why one should and does use the Linux kernel's TCP stack. 99.995% of the time, there are far worse bottlenecks in one's setup than one's TCP implementation.

</comment><comment><USER44> What would be more interesting, and useful, compared to all these network stacks, would be a machine readable specification of TCP/IP from which a correct implementation could be engineered.

However, the realist in me concedes, the specification itself, given the present state of the art, would probably fix, unsatisfactorily, many implementation details (in order for the implementation to pass the spec).

I believe we need a network protocol with a solid, simple semantics. IP, that is not.

<comment><USER45> You have seen the network semantics research project [https://www.cl.cam.ac.uk/~pes20/Netsem/index.html](https://www.cl.cam.ac.uk/~pes20/Netsem/index.html)? It is a formal model of TCP/IP validated with Linnux 2.4.20/FreeBSD-4.6/Windows XP (yes, that was ~10 years ago).

It is nowadays BSD licensed on GitHub [https://github.com/PeterSewell/netsem](https://github.com/PeterSewell/netsem) (and I'm currently reviving it [https://www.cl.cam.ac.uk/~pes20/HuginnTCP/](https://www.cl.cam.ac.uk/~pes20/HuginnTCP/))...

<comment><USER44> No, I wasn't aware. Thank you for pointing this work out. This is _exactly_ the kind of thing I was hoping for. Wonderful!

I can't wait to see what you have planned.

I was thinking after I posted my comment, that it'd be cool if someone could produce a fuzz tester that used both the specification, and the fact that you can turn the Linux and NetBSD network stacks into libraries (libOS and rumpkernel respectively) and co-engineer/evolve the spec whilst also finding and fixing bugs in both the network stacks.

Excited by what you'll be up to!

<comment><USER45> hmm, my other OS is MirageOS ([https://mirage.io](https://mirage.io)) -- also see [https://nqsb.io](https://nqsb.io) contains my previous two years of work ;)

I'd rather call it extensive exploration than fuzz testing what is in my mind...

</comment></comment></comment></comment><comment><USER46> > As far as I can tell, there aren't any available general purpose open source > userspace TCP/IP stacks available. There are a few specialized ones

"specialised" is probably key. If you have a hard problem and also time, money and skills you can probably make something that handles your special case very quickly, and doesn't do the rest of the general case at all. That is an expected trade-off.

</comment><comment><USER47> The kernel TCP stack has two important constraints that alternatives usually don't have:

1\. Ability to handle packets to/from multiple applications

2\. Expose the BSD socket API

</comment><comment><USER48> Did a UDP/IP/Ethernet "stack" (relied on a packet driver shim / smth to send / recv the Ethernet frames) in NASM for DOS while having couple months of experience and using Tannenbaum's book and RFCs.

Worked great (was sitting in a TSR, was stress tested via sending a storm of packets over a short netwrok cable directly to the machine (otherwise network printers would halt :)) while having Elite:Frontiers running its Cobra ship animation on the screen).

Were the times :)

[ humble point is, with things as simple as UDP, special-cased for no packet fragmentation, etc and when the scenario is LAN, you can do it easily and over couple days even, if you really have no other options ]

</comment><comment><USER49> Have I missed something where you provided an alternative?

<comment><USER50> The article mentions more than over alternative.</comment></comment>


USC doctoral student unravels 'tin whisker' mystery - <AUTHOR> http://www.sc.edu/news/newsarticle.php?nid=5371#.UMQOcOTAdnE

<comment><USER2> Congratulations, now can you actually link to the paper instead of summarizing it in half a sentence as is "caused by high-strain gradient built up inside the device"? This is useless reporting.

<comment><USER3> It is a bit hard to find one that isn't behind a paywall, but here's the actual paper: [http://www.osti.gov/bridge/servlets/purl/1013351-BXJitV/1013...](http://www.osti.gov/bridge/servlets/purl/1013351-BXJitV/1013351.pdf)

</comment></comment><comment><USER4> I assume whiskers cause a short circuit. Why can't the circuit board be coated with plastic, epoxy, non-conductive paint, or something to prevent the whiskers from causing a short?

<comment><USER5> Conformal coating is often used for protection in harsh environments.

<http://en.m.wikipedia.org/wiki/Conformal_coating>

<comment><USER6> Tin whiskers can break through conformal coating in some situations and the coating process does not always cover the entire surface evenly leaving thin areas or even entirely uncovered places in which whiskers can still form and escape. [http://www.freelists.org/post/tinwhiskers/NASA- experiments-c...](http://www.freelists.org/post/tinwhiskers/NASA-experiments- conformal-coat-vs-tin-whiskers)

</comment></comment></comment><comment><USER7> This is the University of South Carolina, not Southern California which most people probably associate with "USC".

</comment><comment><USER8> "Historically lead (Pb) has been added to the solders to prevent whisker growth. Due to the harmful effects on human health and environments, it is no longer a valid solution to prevent whisker growth with lead. With the shift to whisker- prone "Pb-free" solders, controlling whisker growth from low melting point metals has again become technically challenging."

</comment><comment><USER9> > While manufactures had been able to control some whiskers by mixing small > amounts of lead into tin solder, the 2006 European Union ban on lead in most > electronic equipment had ignited a debate among scientists about whether > whiskers would remain a perpetual problem.

Oh Europe. Yes, let's ban the use of minuscule amounts of lead so that people will have to replace entire devices when they fail.

<comment><USER10> that's not necessarily true.

As a mechanical engineer I used to work in an electronics factory. there is still a lot of soldering done by hand. not to mention that the operators of the automated soldering machines. during this soldering process there is a considerable amount of fumes, even though you're using blowers and filters, that the workers are exposed to. Lead is known to be hazardous for long exposes. So I guess keeping the manufacture-workers safety is as important as the end user's one.

</comment></comment><comment><USER11> This is an amazing discovery. I'm surprised it isn't more prevalent in the news. This whisker issue has been a thorn in the side of many since ROHS was implemented. The biggest impact with this I see is safer, more reliable planes. How exciting.

</comment><comment><USER12> If the functionality loss is caused by these whiskers shorting circuits, why can't one simply restore functionality by lightly scraping the soldering to knock off the whiskers?

<comment><USER13> The whiskers are often far too short for that to work, and that assumes the whiskers are even exposed such that you could "scrape" them. Furthermore, there'd be a risk you'd end up depositing the whiskers elsewhere on the circuit...</comment></comment>


Ceylon language - <AUTHOR> http://ceylon-lang.org <comment><USER3> Ceylon has showed up on HN a few times. Here's the link to recent comment thread.

[https://news.ycombinator.com/item?id=10463464](https://news.ycombinator.com/item?id=10463464)

</comment><comment><USER4> I like to skim through their gitter channel from time to time, to see what they are up to. But lately, it's not clear.

Does anybody here have more data on this? What's next for ceylon? What's the status with the intellij plugin? Or with the android support in general?

</comment><comment><USER5> Ceylon is such an under-rated language. It is what Java should have been - <USER2> it been curated better. RedHat should really spend more time/money on spreading the word - a lot of people have not even heard of it.

</comment><comment><USER6> Why elephant again? Don't we have too many elephants in logos already in popular OSS projects? Hadoop, PostgreSQL and now Ceylon...</comment>


California, the Wi-Fi state - <AUTHOR> http://gigaom.com/2007/03/02/cali-wi-fi/ <comment><USER2> Good article. Here's a related article to that that was linked in that story:

http://gigaom.com/2006/12/20/earthlink-sf-milpitas/

Milpitas is my hometown but since I'm away for school, I haven't gotten much exposure to the city-wide WiFi. I do remember receiving WiFi signal from Earthlink in surprising places the last time I was in Milpitas. They were offering free access for a while but I think the subscription requirement has kicked in by now.

I love and fully support the idea of mass WiFi access. I'm unsure about municipal funding, however. Either way, it's interesting to observe where this might be going.</comment>


Operation Scherhorn - <AUTHOR> https://en.wikipedia.org/wiki/Operation_Scherhorn <comment><USER2> Interesting. So Gehlen was not the Nazi super spy he made the CIA believe. East Germans and Russians fooled him as they liked.</comment>