Convert curl commands to Go code - <AUTHOR> https://mholt.github.io/curl-to-go/ <comment><USER2> Hey HN, I wrote this to scratch a perpetual itch after reading enough API documentations and re-writing basically the same code over and over. Right now it only supports the most common flags that I saw. As far as I'm concerned the project is finished, but PRs are welcome to cover any glaring omissions or fix bugs.

Someone also showed me this tool which does the same thing but supports more languages and options: [https://shibukawa.github.io/curl_as_dsl/](https://shibukawa.github.io/curl_as_dsl/)

<comment><USER3> Very cool. Flags I commonly see/implement (curl to Python or Ruby usually) include -o (download) and -k (ignore ssl warnings -- you might want an warning comment if you implement that one).

</comment></comment><comment><USER4> Great work, mholt!

He also wrote json-to-go[0]--copy/paste JSON and it generates a matching Go struct. This ends up being extremely useful and saves a lot of time for extensive JSON responses.

[0]: [https://mholt.github.io/json-to-go/](https://mholt.github.io/json-to- go/)

<comment><USER5> I use this all the time! Thanks for your work op

<comment><USER2> Thank you :) Glad to know it helps more than just me.

</comment></comment></comment><comment><USER6> This is awesome. Combined with Chrome Developer Tools' "Copy request as cURL" feature, you can reverse engineer some service and turn it into Go code so easily.

<comment><USER7> So how about a variation that takes a curl command and writes Go http server code that would _accept_ that request?

<comment><USER8> Speaking generally, it's more difficult to do much useful with that, because it's hard to do much more than route on the URL. You could prototype code that merely looks at the mentioned headers, but it's getting kinda silly at that point.

More specific to JSON, you may be looking for something like [https://mholt.github.io/json-to-go/](https://mholt.github.io/json-to-go/) (by the same author) or [https://github.com/ChimeraCoder/gojson](https://github.com/ChimeraCoder/gojson) (which I've used myself).

(I find that what I pay in using a static language like Go to manipulate JSON and having to use a tool like that often comes back to me pretty quickly when I take the tool output as a template and start turning all the structs into Real Objects (TM). The dynamic languages are pretty good at taking JSON and yielding a simple pile of dict/map/array/strings/numbers/etc., but if you want to get real objects back out of them the advantage over Go evaporates. Not because the dynamic languages make it "hard", but just because with one of these tools, both Go and the dynamic solutions are roughly as easy.)

</comment></comment></comment><comment><USER9> Given the connection with Chrome Developer Tools' "Copy request as cURL" I can see the usefulness of this in a general sense; however, I was personally somewhat sad to see that it doesn't actually convert to code that makes use of libcurl.

As such, the resulting project is very limited in exactly which cURL commandlines it is capable of converting, and, particularly, it is only able to handle HTTP connections (unless I missed something).

A generator that could generate libcurl compatible code would have the full power of cURL!

</comment><comment><USER10> This is really neat. There's, I don't know if it's a niggle or a bug or an interface issue, but well, let's call it a usability bug.

If your copy to curl request in chrome accepts gzip and/or deflate, then your tool will add an "Accept-Encoding" Header. So far so good.

In go 1.5.1 at least, from my cursory testing, the runtime will not auto- decompress a bytestream if you've manually specified an accepted encoding.

On the other hand, if you leave it off, it will offer gzip to a server and auto uncompress on receipt.

I'm still not sure this is a bug, but I definitely commented out my accept encoding line in my sample code because I don't want to manually decompress every response.

At any rate, thank you for making this; I'm using it with a slack integration right now.

</comment><comment><USER11> Another alternative is Postman [0]. Supports C, Java, Go, Ruby, Swift, etc. Available as a Chrome app. Not really the same (doesn't convert from one to another AFAIK), but it gives you a common interface.

[0]: [https://www.getpostman.com](https://www.getpostman.com)

<comment><USER12> Thanks for the mention. Postman can import curl commands too and then of course convert to all these languages. We are opening this up so that converters can be added/coded dynamically.

</comment></comment><comment><USER13> Note that by default, curl itself can also output C code version of the CLI command as well (yay)

</comment><comment><USER14> It's not that easy to parse this curl command. He needed 400 lines ([https://github.com/mholt/curl-to-go/blob/gh- pages/resources/...](https://github.com/mholt/curl-to-go/blob/gh- pages/resources/js/curl-to-go.js)). Most libraries out there do this job, but are focused on parsing from environment variables instead of a string argument, because the programs that needs this usually are CLI programs. That's what I found when doing the parse in ruby which basically has only OptParse. There is an opportunity here. Projects like these that receives a command input are very useful.

<comment><USER8> It turns out options parsing is fundamentally hard. It looks simple at first, but then it turns out that if you want to support everything anybody could possible want, you end up with some too complicated to use. Nor is it obvious which "middle ground" is correct. It's a perfect environment to produce exactly what we get, which is a bajillion options parsers for every halfway mature language in the world.

Also, that's not 400 lines of parsing code, that intermixes "parsing" and generation all together.

Mind you Go is not a good language for writing parsers in. (It isn't _especially_ bad, like, say, C, but it lacks almost every conceivable feature that could make parsing code anything other than very, very verbose.)

<comment><USER15> Just to expand on this: options can be position-dependent, and treat options as an algebra (a --and b --or c); or, just executed in order to eg change directory (-d dir1 a b c -d dir2 x y z).

Basically, by treating the arguments as a stream of tokens, it can be parsed with the same power used for parsing the source of programming language.

Yet for many uses there are natural complexity-levels for option parser tools (like position-independent boolean options, and options taking one argument), hence the many tools for many niches.

And they are easy to write, so everyone writes one.

</comment></comment></comment><comment><USER15> \tangent Wouldn't it make sense for this to be a library? Though most of the py looks so simple, it already has the libraries.

Or, to parse the curl command string at run time, so it can easily be human- read, as a curl command?

But I guess the idea is to generate a start-point template, which can then be customised for whatever is needed.

It would be nice for a tool to combine both - so new code is attached to the curl command string... but not very readable... and not very flexible (I bet the next thing you wanted to add would be something that hadn't been considered...)

</comment><comment><USER16> A couple of these tools have sprung up:

JSON: [https://mholt.github.io/json-to-go/](https://mholt.github.io/json-to- go/)

CURL: [https://mholt.github.io/curl-to-go/](https://mholt.github.io/curl-to- go/)

SQL: [https://github.com/knq/xo](https://github.com/knq/xo)

The JSON tool especially has saved me a lot of time.

</comment><comment><USER17> great idea - i wish this existed for every language !

</comment><comment><USER18> Would love the same thing for python!

<comment><USER19> [http://curl.trillworks.com/](http://curl.trillworks.com/)

<comment><USER20> I'm working with the PayPal API right now (send your condolences) and all their documentation is in the form of CURL requests. Just finished converting it to python requests commands and now this gets posted.

TIL'd.

</comment></comment></comment><comment><USER21> Looks like backtics aren't sanitized

        curl -XPOST httpbin.org/post  -d '{"sneaky backtick": "`"}'

produces

        body := strings.NewReader(`{"sneaky backtick": "`"}`)

</comment><comment><USER15> Super minor point: according to the curl manpage, _-d data_ implies POST, so you don't need _-X POST_ in the examples

</comment><comment><USER22> Go question. Is:

        if err != nil {
            // handle err
        }

Required, or can you also do:

        if err {
            // handle err
        }

<comment><USER23> There's no implicit comparison (if that's the CompSci name) in Go. You have to explicitly compare (!= is required)

</comment></comment><comment><USER24> Nice! Small nitpick: "-A" doesn't seem to be supported.

<comment><USER7> Yeah I don't see it in the list at the top of the page. But the author says that he'd be happy see a pull request for common options that aren't supported yet.

</comment></comment><comment><USER25> Go is really verbose but this is really great.

<comment><USER26> It's interesting that all the error handling branches are left empty. What does production Golang code do in that case as a common idiom?

<comment><USER27> One of the (possibly contentious) principles in Golang is that each error needs to be addressed individually with an appropriate response to the particular case in question.

In practice _most_ Golang errors are handled either:

1) Ignored

2) Logged and swallowed

3) Returned at the point it happened.

4) Fatally exits the program.

One of the "features" of the errors as return codes is that you don't get surprising results that come from the goto style jumps that happen with something like exceptions (unrolling state, etc).

<comment><USER28> That's not really true. It's certainly an _idiom_ in Golang to handle errors individually, but it's not a principle.

The principal is that errors are values like everything else, and you can write code to handle them however you'd like. See, for instance, the "write" example in this post:

[https://blog.golang.org/errors-are-values](https://blog.golang.org/errors- are-values)

There are libraries that people use (sqlx might be an example) that are more like that example than the if-error-return stuff here.

<comment><USER27> >The language's design and conventions encourage you to explicitly check for errors where they occur (as distinct from the convention in other languages of throwing exceptions and sometimes catching them)

[http://blog.golang.org/error-handling-and-go](http://blog.golang.org/error- handling-and-go)</comment></comment></comment></comment></comment>


Launch HN: Templarbit (YC S17) – Protect Your Web Apps from XSS Attacks - <AUTHOR> Hi everyone! I am Bjoern from Templarbit (https://www.templarbit.com). Me and my co-founder Matthias are part of the current YC batch and are excited to launch our product. Templarbit protects web applications from XSS attacks and other malicious activity.
Previously Matthias and I worked together at a cyber security firm where we saw many vulnerability reports. After spending some time running engineering at another startup we realized there is a big need for a security solution that can easily be understood and deployed. Something that helps software teams protect what they are building. We reached out to friends and strangers at other software startups to see how they handle the security of their applications. Surprisingly to us, not many teams felt like they did a good job in that area, mostly due to lack of tools available to them.
With the advent of browser support for Content Security Policies, there are new ways to protect against these attacks. Setting a CSP header is a great way to mitigate XSS attacks, but managing changes to the policy and having a reporting endpoint that gives you insights into what is being violated is still difficult. Templarbit helps with this. Our reporting dashboard can help you discover and fix violations in real-time and shows you in most cases exactly where in your app the issue exists. <comment><USER2> The most dangerous part of XSS is what developers _don 't_ _know_ is possible. There are lots of sinks and sources and the more features browsers support, the more surface area is exposed.

Lots of developers intuitively notice reflective XSS, but fewer notice persistent XSS and even fewer know about DOM-based XSS. Each of these has several, even dozens, of sinks and sources to check + be aware of.

I think your offerings would be vastly more valuable if you had a CSP policy generator that defaults to the strictest possible settings, allowing the user to opt out / loosen some rules. Perusing your docs, you only describe a small subset of what CSP is capable of. Your average user is short on time and will likely copy-paste your example policies as their first implementation iteration.

It's important to explain that _every_ page on their domain should be protected by a CSP policy. Protecting just a subset of the domain means that there is still vulnerable surface area.

Iframe embeds. Injected forms. Injected IMG tags. Injected meta tags. Data URIs. SVG DOM events. LocalStorage. Cookie overflow attacks. Charset sniff attacks. Charset attacks against specific databases. IE CSS expressions. Image/HTML + JS Polyglots. etc.

A developer that isn't familiar with all of the possible attacks is likely to not make the CSP as restrictive as needed. I _highly_ recommend if you are going to tackle XSS, try and aim to provide value for _all_ XSS attacks, not just the easiest to defend against.

Also, you should provide resources to explain why XSS is dangerous, what is potentially at risk, how much companies pay for XSS on bug bounties, and resources for the developer to know how to craft a successful CSP policy. Without these, you aren't selling your value proposition.

<comment><USER3> Thank you for your feedback! More resources and education material for developers will be coming very soon.

The default policy we provide is report-only. When we started we provided a very strict policy, but it broke too many apps. It was a major turn down for a lot of adopters. We are constantly evolving and are currently working on a feature that automatically generates tailored (stricter) policies depending on the reports we receive.

</comment></comment><comment><USER4> Thanks for doing this, I'm definitely going to consider it since we're implementing token based authentication and my friends have told me XSS is what I should be concerned about there.

How did you choose to go with a trial period instead of a freemium model like slack?

The issue my startup has with trial periods is that it's like "you have 14 days to start generating revenue". Freemium is better for us because it's like "You can learn about our service, and use our service while your staging your release, and when you launch if you're viable (traffic+revenue), we'll be your partner (ie. charge you)".

That's how slack kept us, and Salesforce lost us.

We shouldn't have to worry about paying while we're still learning how to use your service. If I haven't on-boarded it by the time the trial runs out, I'm going to cancel it.

Then again, I don't know your costs, so forgive me if I'm wrong.

<comment><AUTHOR> You are absolutely right, trials are difficult when there is a integration step that might take a week. We consider the trial to start once you are fully integrated, not when you sign up.

A Freemium model is something we are interested in! We wanna support smaller projects with a free tier in the future.

<comment><USER4> Yeah, that sounds good to me. Being a bootstrapped startup, "pricing" is the first page I go to.

<comment><AUTHOR> Yep. I do the same :)

</comment></comment></comment></comment><comment><USER5> Please add pricing. And tip; to get startup founders, don't subscribe to the silly moniker that startup's price themselves too low. I look at the low, mid and high end price by volume to estimate any new costs now and 6 months down the line. Way too many overpriced services cause it's in vogue right now based on VC's saying it is. Certainly what you are doing is interesting. Nice work.

</comment><comment><USER6> Small suggestion: your docs could use a little work. The language switching is broken, and the history state is broken, so when you click around the docs, you quickly end up with 20+ entries in your history.

Also, the site could definitely use more copy: how do CSPs work? How does the app work in general? Right now the only real details are in the documentation.

Also also, what information ends up back on your servers? Is any user data relayed through an API?

It's a very cool idea, just non-obvious at first glance what exactly it does :)

<comment><AUTHOR> That is great feedback! Some of this was on our "immediately post-launch" todo list. Expect us to implement everything you just mentioned over the coming days.

</comment></comment><comment><USER7> > managing changes to the policy and having a reporting endpoint that gives > you insights into what is being violated is still difficult.

Is this targeting management at companies with multiple products? As a developer I just use Django Middleware to add this line to all our responses and call it a day:

response['Content-Security-Policy'] = "default-src 'none'"

(Well, we still sanitize all our inputs and have the headers to block XSS reflection, but there's still not much complexity.)

<comment><AUTHOR> Setting a CSP header can be simple but managing changes to it can get difficult. Especially as team members add or remove services from the app (Intercom, Google Analytics, etc.)

We also provide a reporting endpoint that captures all violations. Do you currently use a CSP header with a report-uri setting?

<comment><USER8> Have you actually found much value from these reports? Most people I talk to tell me they're full of garbage.

<comment><AUTHOR> Noise is a problem and we are in this fight to improve exactly that.

<comment><USER8> There is clearly noise, but is there any signal at all? I wouldn't be surprised if there was no signal to be found.

<comment><AUTHOR> There is a lot more signal than noise, especially they way we aggregate the security reports. CSP can be deployed in report only mode, which provides a great way to detect how your policy has to come together.

<comment><USER8> I guess in my mind signal == attacks, but I guess there's also value in having reports when your policy broke things if you don't have other methods for detecting that, or if your integration tests disable CSP.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> I think Google Webmaster tools report some XSS vulnerabilities. I'm not suggesting their detection is in any way as sophisticated as your solution - but do you see internet search giants offering and improving this sort of service as threat to you in the long term?

Congratulations on the launch though, don't think this product is targeted towards us though so probably wont get a chance to try it but love the objective.

<comment><AUTHOR> Thank you!

Google Webmaster tools has made an effort to help fight XSS vulnerabilities by providing some educational tools to devs (among other things). It's not a solution in the same realm as ours, for example they don't collect any violations and show you detailed reports on where something bad happened.

I don't see Google as a threat to us at all. We actually sat down with them at their SF office last week to work closely together on how they can quickly provide a better Content Security Policy feature set to their cloud customers.

</comment></comment><comment><USER10> For PHP integration, have you considered hooking into CSP-Builder? (It's MIT licensed, so you can just make a private fork for internal use forever, but if you upstream your changes, I'd greatly appreciate it.)

[https://github.com/paragonie/csp-builder](https://github.com/paragonie/csp- builder)

<comment><AUTHOR> Yes we are looking at this. We will release a more integrated solution for PHP soon + docs.

</comment></comment><comment><USER11> Are you in the same space as Tcell.io? I think they have been doing this for a few years now.

<comment><AUTHOR> I actually have never heard/seen Tcell.io so I can't speak to them and what they do. Will check it out though!

</comment></comment><comment><USER12> Watch out you got a typo on the front page: "A central dashboard shows all secuirty events and directs your attention to the exact part of your application that has issues" \- security is spelled wrong

<comment><USER3> Thanks for finding this one. Just fixed it.

</comment></comment><comment><USER13> Offtopic - wow, so the links in submission text are clickable now, or is it a special kind of submission?

<comment><USER14> We did this for Launch HN and Show HN to lessen the penalty for making a text submission. There's a quality of density, and I would even say intimacy, in the text launches that fits well with HN.

</comment></comment><comment><USER15> I don't want to play with my API headers, if you can verify with TXT record on my domain, that would be great.

<comment><USER15> I thought that this was related with the verification but it looks like a feature of the browser.

<comment><USER3> No domain verification is required at this point. Content-Security-Policies are ideally delivered through the HTTP response headers and then enforced by the browser. We are working on making the integration as easy and smooth as possible.

</comment></comment></comment><comment><USER16> What does your service cost?

</comment><comment><USER17> looking for this for months. can't wait to test it.

<comment><USER3> This is awesome! Let me know if I can help you getting started.</comment></comment>


Information Civics: Deconstructing Power Structures of Social Computing Networks - <AUTHOR> https://infocivics.com/ <comment><USER2> Moxie Marlinspike's counter-argument about the perils of federation: [https://signal.org/blog/the-ecosystem-is- moving/](https://signal.org/blog/the-ecosystem-is-moving/)

 _" On some level, federation is appealing precisely because it does freeze protocols in time. It's great when centralized clients and servers roll out features that benefit us, but they could just as easily roll out features that don't. Federation gives us more collective control over what changes we accept, but that comes with an unacceptable inability to adapt.

Given that federated services always seem to coalesce around a provider that the bulk of people use, federation becomes a sort of implicit threat. Nobody really wants to run their own servers, but they know that it might be possible if their current host does something egregious enough to make it worth the effort.

However, over the past six years, we've also seen the user cost of switching between centralized communication services reduced substantially, particularly given the tendency towards addressing with user-owned identifiers like phone numbers. The device's address book is now the social network, so using phone numbers as an identifier has reduced switching costs by putting a user's social network under their control. In a way, the notification center on a mobile device has become the federation point for all communication apps, similar to how older desktop IM clients unified communication across multiple IM networks.

The effect has been visible in the messaging space, where market leaders have come and gone, new popular apps come out of nowhere, and even the most successful players seem compelled to continue iterating and improving their services as quickly as possible.

This reduced user friction has begun to extend the implicit threat that used to come with federated services into centralized services as well. Where as before you could switch hosts, or even decide to run your own server, now users are simply switching entire networks. In many cases that cost is now much lower than the federated switching cost of changing your email address to use a different email provider.

An open source infrastructure for a centralized network now provides almost the same level of control as federated protocols, without giving up the ability to adapt. If a centralized provider with an open source infrastructure ever makes horrible changes, those that disagree have the software they need to run their own alternative instead. It may not be as beautiful as federation, but at this point it seems that it will have to do."_</comment>


Feature Request in Chrome: Check SSL sites for HeartBleed (WontFix) - <AUTHOR> https://code.google.com/p/chromium/issues/detail?id=361352

<comment><USER2> This is bad engineering. Should browsers run a full suite of tests? How would you detect a version which has an old number but had just the patch applied? Why not test for other vulnerabilities whilst we are at it?

A browsers job is to browse the Internet using agreed protocols. If you want to create an addon that checks websites for vulnerability of the month because you are more security conscious then fine. Just lets not push the problem to someone else.

</comment><comment><USER3> TL;DR "It would take us a couple of months to get this feature out. I think it would be mostly moot at that point."

<comment><USER4> >"I think it would be mostly moot at that point."

I think that's stunningly naive.</comment></comment>


Web 2.0 Expo (held in NewYork sep16-19, video) - <AUTHOR> http://www.web2expo.blip.tv/#1283514

<comment><AUTHOR> This includes the Tim O'Reilly Keynote that we all discussed about earlier:

"Tim O'Reilly: Stop throwing sheep, do something worthy " <http://news.ycombinator.com/item?id=308555>

the video is named: "Web 2.0 Expo NY: Tim O'Reilly (O'Reilly Media, Inc.), Enterprise Radar"

Enjoy.</comment>


The ggplot flipbook – building charts slowly - <AUTHOR> https://evamaerey.github.io/ggplot_flipbook/ggplot_flipbook_xaringan.html <comment><USER2> Wonderful demonstration. I can't help but wonder--as someone who has flirted with R but certainly not committed--how much time and practice it takes to acquire decent fluency with these half-magical incantations to become productive.

<comment> black-tea This is ggplot. It's not a good example of what it's like to use R in general. I sorely wish there was a well-maintained ggplot for Python because the library is excellent but the language is terrible.

<comment><USER3> The tidyverse and data.table universes are two very comprehensive sets of packages that kind of offer a replacement to the base R syntax, and they are pretty nice languages.

</comment></comment></comment><comment><USER3> Really great examples. Anyone got any updates on what the best python equivalent is? Bokeh? Plotly?

<comment> kot-behemoth R's ggplot2 follows The Grammar of Graphics. The closest and best implementation of it in Python is Altair ([https://altair- viz.github.io](https://altair-viz.github.io)). It's a fairly young library, but is very simple to use, and can do really powerful things (see the last chart here [https://altair-viz.github.io/case_studies/exploring- weather....](https://altair-viz.github.io/case_studies/exploring- weather.html))

</comment></comment><comment> black-tea This is a great demonstration and shows the bit that finally clicked with me. When you learn to wield these tools you'll be considered a wizard by many when you're able to make beautiful plots in seconds.

</comment><comment><USER4> Is there a flipbook format that doesn't break content zooming?

</comment><comment><USER5> Very nice. I would love some kind of plugin for Rstudio that let you "play" through an arbitrary ggplot like this to help understand how charts are built.

<comment><USER6> The people of RStudio often produce lessons for DataCamp, which has interactive browser-based lessons on how to use ggplot like [https://www.datacamp.com/courses/data-visualization-with- ggp...](https://www.datacamp.com/courses/data-visualization-with-ggplot2-1) (first chapter free). Is that helpful?

<comment><USER5> I was thinking more like an automatic way to step through each subsequent "+"'d function rather than having to comment/uncomment repeatedly to isolate what each layer does.

</comment></comment></comment><comment><USER7> ggplot2's ultra terse, beautiful charts really beat out the competition. I do wish the api's were _less compact_ and more naturally explorable right out of the ide. R as a whole suffers these traits imho. I hope ggplot is ported to the RoW, ideally a typed language or runtime

</comment><comment><USER8> This is exactly what I was looking for all weekend, great! Thanks!

</comment><comment><USER9> tangential but what software were slides created in?

<comment><USER10> Xaringan: [https://github.com/yihui/xaringan](https://github.com/yihui/xaringan).

It's mentioned on slide 6.

</comment></comment><comment><USER11> super cool, well done!

love to see this for other language and plotting</comment>


CodeNameIze – Generate Human-Readables Hashes - <AUTHOR> https://github.com/jjmontesl/codenamize <comment><USER2> How similar is this to auto allocated Docker container names?

Could it serve as a better alternative as you can hash some useful state attribute(s) for easy identification?

<comment><AUTHOR> Well. This is not only for docker containers .. :)</comment></comment>


Show HN: I wrote a script to turn a webpage into bitmap HTML. (Caution: huge) - <AUTHOR> http://elliottkember.com/awesome/hn.html

<comment><USER2> Perhaps this could be a good start for another coding contest: The most inefficient webpage.

The object: take a copy of the Google search page and represent it in the most inefficient, bloated way possible. There would be two divisions: time-bloat and space-bloat.

Rules: Every piece of code must be involved in displaying something visibly on the page. Every statement in the Javascript must be making forward progress toward displaying the page.

The functionality of the page must be exactly the same as the original.

Any other rules?

(There wouldn't be any loading and waiting on external resources, of course.)

<comment><USER3> Calculate `h=SHA256(original_page)` and `n=len(original_page)`.

        do
          page = gen_random_bytes(n)
        while (h != SHA256(page))
        print page

It can be even generated client-side!

<comment><USER4> Can't the same hash mean different data?

<comment><USER3> Well, sure. But if you add another check for likelihood that `page` is fairly- correct html in utf8, there should be around one correct result really :) If you get 2, the other one is probably a <form> you can use to sell your soul to satan - do not hit submit if you get it.

</comment></comment></comment></comment><comment><USER5> This is cool, but when I first clicked it I thought it was transforming my current webpage into HTML/bitmap somehow. I was amazed. Then I realized it's a statically generated page. If you could make this into a bookmarklet that transformed the current page, then I'd be REALLY impressed. :)

</comment><comment><USER6> Then turn it back into HTML with an experiment of mine: <https://github.com/andershaig/img-con> \- It takes an image, writes a span with a background-color for each pixel. Turns a 100x100 image into a 7 MB HTML file.

<comment><AUTHOR> Uh - that's exactly what this script does!

<comment><USER6> You're right and I should have paid more attention. I had thought you were generating just the image from a web page. Having actually worked on something similar, I can say I'm very very impressed with the load time.

<comment><AUTHOR> Probably mostly the Gzip - the code is very repetitive. The RLE helps too.

</comment></comment></comment></comment><comment><USER7> Bonus points for apparently using RLE for same-colour runs of pixels. :)

<comment><AUTHOR> Thanks! I did it without first, but it was way too big and crashed the browser.

</comment></comment><comment><USER8> Oh, the humanity. Why would you do this (apart from "Because I can")?

<comment><USER3> Finally a pixel-perfect display on any browser! With no-license font embedding.

<comment><USER9> I laughed.

But, there's another way - use a damn scripted gif or png</obvious>

...or for monochrome pages, is an xbm smaller?

update: oh yeah, i forgot, xbm files are C, though I suppose you could meta script them with more C.

people who really dont want their text stolen are going to see the massive bandwidth we have now and start just using images. protects it from anything but ocr.

<comment><AUTHOR> Wolfram Alpha does this. I think it helps with writing formulae.

</comment></comment></comment></comment><comment><USER10> Source <https://github.com/elliottkember/img2html.rb>

</comment><comment><USER11> and all of the sudden, the new years clock done in sql seems useful.

hope you don't go over your bandwidth quota ;)

</comment><comment><USER12> I think it's to early to call this a startup... you need to think about your potential revenue sources. ;~)

<comment><USER13> Bandwidth deals? :)

</comment></comment><comment><USER14> If you are just looking to make a self-contained file that has the page as image, I would suggest you simply embed the image as base64.

I made a bitmap -> <table> once, that was pure hell like this. ;-)

</comment><comment><USER15> Why didn't we think of doing this before we had Canvas!

...oh that's why.

</comment><comment><USER16> Nice! this reminds of a script I wrote that turned images into svg: <https://gist.github.com/761332>

</comment><comment><USER17> how?

<comment><USER18> I would assume he takes a screenshot of the page (xbrow.se is one utility for it) and then picks the colour from each point and displays it... quite simple to do, _why_ is probably a better question :p

<comment><USER17> I thought he said he does it with javascript!

<comment><USER18> I just looked via Twitter and found he has posted it on Github, seems I was right: <https://github.com/elliottkember/img2html.rb>

</comment></comment></comment></comment><comment><USER19> And you'd be amazed to know that khtml (KDE's rendering engine, used by Konqueror) renders it instantly!

</comment><comment><USER20> Now if only links worked. Then you would have created a modern image mapping service. I like it.

</comment><comment><USER21> this week, i've been converting HTML output into PNG images. i used the preview feature of an HTML to PDF dll we have lying around.

</comment><comment><USER9> basically doesn't get rendered at all in ie9. big surprise.</comment>


Scanning for mobile devices through Wi-Fi using Pi Zero W - <AUTHOR> https://medium.com/swlh/scanning-for-mobile-devices-through-wi-fi-using-pi-zero-w-8099be08cc1e <comment><USER2> i believe you can achieve the same thing with a pwnagotchi if you fine tune it. The os already has everything you need to get started, its mainly used to capture handshakes but since it uses bettercap, you can tweak it to do other things. [https://pwnagotchi.ai/intro/](https://pwnagotchi.ai/intro/)</comment>


Man v Marathon: One scientist's quest to propel a runner past the 2-hour barrier - <AUTHOR> http://www.nytimes.com/2016/05/15/sports/two-hour-marathon-yannis-pitsiladis.html? <comment><USER2> As a runner and lover of endurance activities, this is an exciting prospect. Seeing what is possible in the easiest (or most extreme) possible environment will help us get closer to understanding the optimal altitude difference or even an optimal training, living, and competing environment.

<comment><AUTHOR> Agreed!</comment></comment>


How to become the sole owner of your PC [pdf] - <AUTHOR> https://github.com/ptresearch/me-disablement/blob/master/How%20to%20become%20the%20sole%20owner%20of%20your%20PC.pdf <comment><USER2> 30 years ago you could buy an IBM PC/AT and it would come with schematics, detailed programming information, and the complete source code listing for the BIOS[1]. There wasn't really anything "hidden", and you really felt like "the sole owner of your PC". The PC became the dominant platform because of this. Now, because of corporate interests[2][3], much of it is officially under a thick layer of red-tape NDAs, and what little else interesting out there comes mainly from leaks (be thankful for East-Asian companies' insecurity...), and there are still plenty of things hidden away. On the the other hand, all of this stuff _is_ documented somewhere; whether that documentation will ever see the light of day is the real question (for anyone reading this who does possess such information: please do the right thing ;-) Especially when it's far easier now than before to distribute lots of data, putting the complete ME programming documentation and maybe even source code online as a download or on a disc to be supplied is entirely feasible. But Intel decided against, leading me to think they really intended it to be a backdoor-ish sort of thing, and that makes me really sad.

[1] [https://archive.org/details/bitsavers_ibmpcat150ferenceMar84...](https://archive.org/details/bitsavers_ibmpcat150ferenceMar84_26847525)

[2] [http://boingboing.net/2012/01/10/lockdown.html](http://boingboing.net/2012/01/10/lockdown.html)

[3] [http://boingboing.net/2012/08/23/civilwar.html](http://boingboing.net/2012/08/23/civilwar.html)

<comment><USER3> > they really intended it to be a backdoor-ish sort of thing

Occam's razor says they really just sell a ton of CPUs to huge datacenters and wanted to solve the problem of having to have some tech go out and physically reset a machine when it misbehaves. If you've ever accidentally powered down a machine that's sitting in some lights-out facility in Northern Alabraska, this kind of technology is a godsend. Now I'm not saying I'd bet my hat that it's 100% secure, or that Intel couldn't be thumb-screwed by the feds to use it as a backdoor in some scenario, but it wasn't built from the ground up to be one.

<comment><USER4> No, Occam razor says that if all you you want to do is LOM, you put an embedded computer running Linux and ssh, connected only to to power and to the console, and nothing else, not a SPARC (?!?) core running proprietary code that does _way, way_ more than just implement support for LOM, and which has access to all memory and CPU state.

<comment><USER5> It's not a SPARC, it's an ARC. It's a very different processor.

<comment><USER6> ARC not was a simplified SPARC for CPU architecture teaching ? I had to build it on VHDL on my first year.

<comment><USER4> Well I have no idea how ARC came into existence, but looking at the ISA, it has nothing in common with SPARC (I write SPARC compilers for a living, so I know the ISA pretty well).

<comment><USER7> I don't know the full story it was a spin-off of the work that Argonaut did for the SuperFX chip in Star Fox on the SNES -> [https://en.wikipedia.org/wiki/ARC_%28processor%29](https://en.wikipedia.org/wiki/ARC_%28processor%29)

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> Relevant, related talk from 32C3 by Joanna Rutkowska

[https://media.ccc.de/v/32c3-7352-towards_reasonably_trustwor...](https://media.ccc.de/v/32c3-7352-towards_reasonably_trustworthy_x86_laptops#video&t=0)

</comment><comment><USER9> I've never heard of this before. This feels like science-fiction. You'd think this would have blown over the internet 50 times over? What can 'they' do with this?

<comment><USER10> It is like cell phone radios with DMA access. Anything they want, in practice. If you are buying any computer commercially it always has backdoored hardware with a wholly proprietary coprocessor with networking and memory access.

<comment><USER11> Can you please share links of evidence of this? Thank you

<comment><USER12> See [http://www.osnews.com/story/27416/The_second_operating_syste...](http://www.osnews.com/story/27416/The_second_operating_system_hiding_in_every_mobile_phone)

Every brand of smartphone has this secondary firmware, entirely separate from the primary OS. Without it, you can't connect to the cellular network.

</comment></comment></comment></comment><comment><USER13> Just more evidence that modern CPU designs need to die, and that a libre alternative is required.

<comment><USER14> The problem to be solved here is not just software. How do you beat the economies of scale that standard designs have achieved?

For most applications, the end user gets _vastly_ more computing power per dollar spent when they buy an off the shelf design that's in large scale mass production.

You could port BSD or Linux to your new architecture for a reasonable amount of money (though that's already an uphill battle selling that to a consumer). How are you going to even come close to the unit economies of scale that Intel and other large established players enjoy?

<comment><USER15> do you need 40ghz with 32 cores or 2 cores at 1.5ghz and the garantee of full control?

for personal computing, I'd take the later

<comment><USER16> By far the latter. My computation needs have been satisfied for years now; paternalistic, insecure platform lock-in crap irritates me far more than access to cutting-edge technology would benefit me.

</comment></comment></comment></comment><comment><USER17> While I wait patiently for more open hardware designs to become more popular / actually used (Be it SPARC, RISC-V or something else), I wonder what AMD's equivalent functionality is like in comparison to this - I strongly doubt that no equivalent exists for AMD CPUs.

<comment><USER18> From [https://libreboot.org/faq/#amd](https://libreboot.org/faq/#amd)

# Why is the latest AMD hardware unsupported in libreboot?

It is extremely unlikely that any post-2013 AMD hardware will ever be supported in libreboot, due to severe security and freedom issues; so severe, that the libreboot project recommends avoiding all modern AMD hardware. If you have an AMD based system affected by the problems described below, then you should get rid of it as soon as possible. The main issues are as follows:

# AMD Platform Security Processor (PSP)

This is basically AMD's own version of the Intel Management Engine. It has all of the same basic security and freedom issues, although the implementation is wildly different.

<comment><USER2> _The PSP is an ARM core with TrustZone technology, built onto the main CPU die._

That sounds even worse than ME:

 _Intel Management Engine (ME) is a separate computing environment physically located in the (G)MCH chip._

Theoretically, if a third-party can figure out how to make a compatible MCH they can use Intel CPUs without ME, but that is impossible with AMD's design.

Then again, developing a compatible MCH would be nontrivial too --- the last truly "open" x86 bus interface was probably Socket 370 (still in use by VIA and others), and the later bus interfaces are such high speed that they require some _very_ expensive signal analysers to even see the communications properly.

</comment></comment></comment><comment><USER19> There's always the non x86 path.

I have recently purchased a pi-top which is basically a 3d-printed laptop case + laptop battery + keyboard + display + raspberry pi 3. The keyboard could be better and I'll swap out the pi for a beagle bone black (pi comes with a binary blob) but it's a surprisingly useful package. I only bought it to experiment a bit with ARM assembly on the go but it's actually powerful enough for a lot of day to day stuff (mail, surfing, libre office, programming). Not great but good enough.

Of course you have to be willing to trade speed and availability of some programs for that control but in theory a beagle bone + input and output devices is a nice little open machine for a lot of day to day stuff.

<comment><USER20> ARM isn't a lot better. It helps that the (relatively simple) boards have the schematics available, but most platforms require binary blobs and there are few of them that don't have a bunch of useful information tucked away behind a gazillion NDAs.

The various warts in the implementation (e.g. no BIOS, so there's quite some effort going just into making something boot on a new ARM board) and the non- standard, or just closed source-dependent, augmentations that are required in order to make an ARM CPU do anything breathtaking (PowerVR and Mali, TI's EVEs) also mean that, at least if you're using Linux, you're often living outside the mainline kernel. I have very few kind words to say about some of the code that I've seen in manufacturers' kernel trees, _especially_ on let's-pump-two-more-cores-before-next-years-mobile-world-so-that-we-can-play- a-demo-that-looks-exactly-like-last-year-except-salespeople-are-gasping-for- some-reason, the ancient Indian name by which ARM is also known in some places. I would rather not have useful/important data reside on devices which run that stuff.

<comment><USER19> I have admittedly not investigated it very deeply but the Beagle Bone Black seems good enough with compromises. AFAIK it will boot without a blob but contains one for the GPU (PowerVR). The device will run non-OpenGL stuff fine and I believe there is some progress on reverse engineering an open driver. There's armv7 (am335x) from OpenBSD which is my base indicator that it's at least somehow usable without any blobs.

I'll yield to more informed people but searching a reasonably useful and completely open (I guess I can live without HDL for everything but it would be nice) machine has been a quest I go on every now and then. ARM seems to be the best (and most affordable) bet. I'll gladly take other suggestions as the whole ARM licensing model doesn't really sit well with me.

<comment><USER20> I go on that quest every once in a while, too, but without too much luck.

I haven't ran OpenBSD on the BeagleBone Black, but I imagine it has no graphical output of any kind. I see the X packages but the only on-board devices that are listed as supported are:

        BeagleBone, BeagleBone Black
    	Supported on-board devices:
    	  standard serial port (com)
    	  watchdog controller (omdog)
    	  ethernet controller (cpsw)
    	  GPIO controller (omgpio)

so it boots but it seems a little unlikely that you can do much post-1980s work on it.

I run into various ARM platforms at $work. The platform, as a whole, is _probably_ a step backwards from e.g. PowerPC; it's very relevant today because its power consumption is very hard to beat, and between mobile phones, tablets, IoT and in-car infotainment, this is an important topic. But unless you _need_ something that's super low power, the only thing ARM CPUs have to show for themselves is that they aren't x86. This is more than made up for by the headaches involved into getting stuff that runs on a company's Cortex A7 to run on another company's Cortex A7.

This isn't to say that it's a _bad_ thing. Pre-64-bit ARMs were designed (with the exception of some really old stuff in the 80s) as a platform for appliances, not for computers. They're excellent for designing phones and tablets and smart TVs and whatnot. It's trying to bolt a general-purpose environment on top of them that gives people headaches.

<comment><USER21> The display controller on ARM SoCs is typically separate from the Mali or PowerVR GPU and is often documented. If there are multiple ARM cores on the SoC then the display may still feel fast enough for basic stuff even though the CPU is doing all the work. Another factor is that some modern GPUs are 3D only, the graphics stack gets a lot more complicated if it has to translate 2D requests from an application into OpenGL calls to the GPU.

NetBSD has a framebuffer driver for TI OMAP CPUs so it could be possible to port it to OpenBSD.

<comment><USER20> Ah -- yes, on many (most?) devices, you can get framebuffer output without the GPU. I just don't know if OpenBSD has that.

</comment></comment></comment></comment></comment></comment><comment><USER22> Has anyone found a remote attack via the Management Engine yet? It's basically a backdoor; there must be some way to make it do things.

<comment><USER23> perhaps everyone who finds something disappears? </sarcasm>

seriously, AMT is stuff of nightmares. runs even when machine is powered off.

<comment><USER24> AMT is the reason I mock anyone here that talks about concerns of backdoors in Intel's RNG's being ridiculous. The whole CPU is backdoored with a backdoor that can run when it's _off_. Including probably the RNG given debug wires are probably connected to it. Let's not worry about one feature when we have a subversion this big.

Biggest irony: they advertise it as a feature for IT management. ;)

<comment><USER25> My team spent a bunch of time looking at AMT as a management tool -- it adds lots of complexity and accomplishes almost nothing that cannot be done easier a half dozen other ways. The only functionality we saw any real use for was an always on VPN and remote wake up where networks block WoL.

The other thing that's even stranger is Absolute Software, a little company nobody has ever heard of who somehow got every OEM to bundle their code in system BIOS for theft recovery since the 90s.

<comment><USER24> re Absolute

It seriously reads like a botnet description. I'd have believed them if they said "From the innovators behind Storm comes new endpoint protection..."

[https://www.absolute.com/en/about/persistence](https://www.absolute.com/en/about/persistence)

<comment><USER25> It is pretty disturbing. We were lobbied pretty heavily to evaluate it and did some field tests as a result.

You can do some wacky shit with it. There is one mode where you can configure it to delete certain directories or files to screw around with a thief. That can be tweaked to delete files that the system needs. Those modes are persistent, and even after reformatting or replacing the boot media, the agent will reinstall and retrieve its policy file.

If you use it with Intel AMT, it can also brick the device permanently. ([http://www.intel.com/content/dam/doc/product-brief/mobile- co...](http://www.intel.com/content/dam/doc/product-brief/mobile-computing- protect-laptops-and-data-with-intel-anti-theft-technology-brief.pdf))

Everyone who tested the product had two questions: "Who is buying this?" and "How do I get this dormant code off my devices?"

Laptops aren't cheap, but they aren't expensive enough as an asset to care that much about recovery. And there are arguably better ways to safeguard data assets.

</comment></comment></comment></comment></comment></comment><comment><USER26> I'm looking forward to this being complete: [https://www.olimex.com/Products/DIY%20Laptop/](https://www.olimex.com/Products/DIY%20Laptop/)

Any ideas what a Chinese ARM chip might have for backdoors?

<comment><USER27> Anything.

</comment></comment><comment><USER28> So ... is my macbook or Dell server pre-pwned by Intel and re-pwned by the Chinese? What, exactly, am I supposed to be afraid of here?

<comment><USER29> Nothing, as long as you have nothing to hide. :-)

<comment><USER30> So if you have business secrets, you will have worries... :-)

</comment></comment></comment><comment><USER31> Does anyone know of a tool or even sequence of actual commands to disable the Intel ME AMT functionality?

Did I miss it in the slides?

I see explanations and low-level temporary/soft-disablement references, but nothing immediately actionable.

<comment><USER32> There is no prescribed way to disable it, and it looks like the slides only provide a glimpse of the workarounds/hacks needed to do so. I imagine there's a lot we're missing from this presentation, though, because it looks like there was a demo.

<comment><USER33> The demo is also available in the repository. [https://github.com/ptresearch/me- disablement/blob/master/Int...](https://github.com/ptresearch/me- disablement/blob/master/Intel%20ME%20disabling%2C%20PHDays%20VI%20Demo.m4v)

</comment></comment></comment><comment><USER34> Direct link to the PDF: [https://raw.githubusercontent.com/ptresearch/me- disablement/...](https://raw.githubusercontent.com/ptresearch/me- disablement/e089b0e583070bc9a87b2d8e27ad7a9b0275f374/How%20to%20become%20the%20sole%20owner%20of%20your%20PC.pdf)

</comment><comment> King-Aaron So, forgive me for this probably being an oversimplification, but is this the same sort of thing as the old Clipper Chips?

[https://en.wikipedia.org/wiki/Clipper_chip](https://en.wikipedia.org/wiki/Clipper_chip)

<comment><USER35> As I understand it, the clipper chip was a combined encryption accelerator and key escrow device, with the escrow basically working as a government backdoor.

The Intel Management Engine is another computer in your computer, running its own OS, and able to see everything your computer is doing, and modify it, including things like network transfers.

</comment></comment><comment><USER36> How about firewalling whatever ports the Intel code may use? If it can't communicate with the Internet, presumably it's not likely to do any harm.

<comment><USER8> That only works for very simple surveillance. It has _complete_ control over your hardware, and it can encode information that it wants to get off of the system in a variety of different ways.

If you want to firewall ports or IP addresses on the machine itself, obviously that doesn't do anything, so what you'd need to do is do it on your router (that you hope doesn't have a similar backdoor that cooperates with ME), first you'd need to _know_ what to block, which is difficult enough, and then you'd have to trust that that information doesn't change.

But event then all it takes is for AWS or CloudFlare or $Foo to collude with Intel to get at your juicy data again, so you really would need to work on a blocked-by-default basis, which is possible, but not really practical, depending on what you're doing.

It really depends on what your threat model is. If your're a high value target to someone with a lot of resources, you're essentially screwed.

It can broadcast information via your speakers, and maybe even your microphone. It can encode data in the timing of your packets as they leave your system. It can encode data in it's power consumption, it can encode data in what it sends to the screen, it can send data out via bluetooth or wifi. There are probably more ways, that I didn't think of off the top of my head.

We have Free Software all the way down to the firmware level. Not widely available, but the potential is there. That is good.

But for computers that we can really trust, we need to go deeper.

 _looks up Czochralski process_

<comment><USER37> And it cooperates with Intel NICs - and when the ME is half-disabled, the NIC starts showing erratic behaviour.

I'm back to using a realtek NIC from 2006 now.

</comment></comment></comment><comment><USER38> The PDF won't open for me on a Nexus 5, doesn't show thru githubs mobile site either.

Edit: It worked when I requested the desktop version of github though.

<comment><USER8> _Sigh_... chrashes on Chromium on Linux as well.

<tinfoil_hat>maybe ME is detecting it and wants to prevent us from knowing about it?!?</tinfoil_hat>

<comment><USER39> Thrashes a bit in Chrome on Windows for me.

</comment></comment></comment><comment><USER40> The best way to disable intel ME is to reflash you system with ME FW in Manufacture mode (if your system isn't not already in the mode) and than use ME runtime disable command (can be sent by me-tools [https://github.com/skochinsky/me-tools](https://github.com/skochinsky/me- tools)) at each boot

</comment><comment><USER41> Seems like if I really want to know what's going on at that level, I should really use something more simple, like a raspberry pi...

<comment><USER42> Why would you switch to a system that relies on NDA-protected chips when there are free ones available like the BeagleBone?

<comment><USER41> Is a BeagleBone like a raspberry pi? :)

</comment></comment></comment><comment><USER43> It's only a matter of time. Once computer technology stabilizes, it will become commodity, well documented etc. you can already make completely open breadboards, that are more powerful than 50 year old computers. Might be several decades though.

</comment><comment><USER44> Maybe we should all flash our Intel cpu, claim warranty, and keep the next cpu. Initial support will have to RMA it under "The computer is broken it shuts down after 40 secs!".

<comment><USER4> How would that solve anything? (Not that you could claim warranty anyway.)

</comment></comment><comment><USER45> Link to the talk?

<comment><USER46> [http://www.phdays.com/broadcast/](http://www.phdays.com/broadcast/) > Scroll to bottom > Third last > How to Become the Sole Owner of Your PC

Talk is ~3.5minutes transcribed live to English. Sadly you're not missing much by just looking at the PDF.</comment></comment>


Show HN: Workadee–measure employee satisfaction at your company - <AUTHOR> https://www.workadee.com/ <comment><AUTHOR> Hey HN. I've been building Workadee for the last month and a half or so–it's a neat tool for measuring employee satisfaction at your company. It uses the Net Promoter System, so if you've worked in a marketing department or at a larger company that has used NPS, you're probably familiar with it.

Instead of the normal NPS methodology for surveying your customers, Workadee surveys your employees. It's a simple system with easy results - a single score that you can track over time, to see how your company is doing.

It's in beta right now, just gathering info about how people are using it a bit, before doing a bigger launch at the end of December/early January.

Give it a shot and let me know what you think!</comment>


Ask HN: Where to go from here?  Or, I need out of govet contracting - <AUTHOR>

Hello HN community,
I apologize if this has been addressed before on HN, I'm still scowering the posts via google but I desperately needed to get this up to start the conversation.  Apologies for the long post..
About 3 or so months ago I woke up and decided I needed to get out of my current career sector (government contracting - I'm in the Wash DC and surrounding metro areas).  For me, the day to day of government contracting has become unbearable.  Now I know there are a lot of people in the US hurting right now as far as jobs are concerned, and I don't want to come off as whiny so apologies and I will try to stick to my specific issues.  But I think that is part of my disatisfaction as well.
I've been working full time (more or less) as a programmer since 1997.  Part of that time was while I was a full time college student, then later during the .com run up I bailed on college for a higher salary.  I have since returned and received my BSc and I will complete my MS in comp sci this fall.  For most of that time I have been working as a government contractor for various agencies.  I also did about 4 years at two private companies - one a product company, and one a services-based company.  Well, when I say two private companies, what I really mean is non-government contracting.  I currently and always have worked for private companies, not as a government employee.
The allure of government contracting can be strong.  I was able to obtain a sought after security clearance.  While "even Wayne Gretzky was traded," I have no doubt I could quit my job tomorrow and have multiple offers in a few days.  I make a great salary and have great benefits (at least I think so).  I never work overtime and my hours are flexible.  But the work is killing me.
The emphasis on the "billable hour" is relentless (see: http://news.ycombinator.com/item?id=650691).  Not to mention that probably about 50% of my coworkers are apathetic and are simply putting in the motions day to day - a strong job market (for us) coupled with very little accountability from their parent companies ("..as long as you are billable!") contributes to this situation.  It is not uncommon for me to see coworkers doing sudoku puzzles, talking for all hours while doing no real work, or even clipping their finger nails in their cubicles (!).  That last one may not be specific to govt contracting..but you get the idea.  I know people who will be in that arena for life.  Good for them, they are taking care of their families, etc, but I don't want to work a job simply because it is damn near recession proof!
I've decided I want out.  I think I have a lot to offer the private sector - I'm intelligent, I understand the theory of computing (some of it, I think!), and I have an aptitude for picking up new tech and thinking through problems.  But I'm not sure where or how to start the transition.
I just turned 30, and while I have been somewhat following the non-Microsoft tech scene, I do not claim to be an expert (most of my career has centered around msft technologies).  I look at job postings for companies I have identified as being "in" the area I want to be involved with (I call this the "HN arena") and I see things like: Rails, nginx, NoSQL, memcached, etc as some examples.  I have no doubt I can learn these new technologies (new for me!) with some competency, and some I have been following for a while but never got around to working with (Rails, for example).  And I know that tools don't necessarily make the man.  But that is just a small list of some cool stuff I'd like to work with.
I know I learn best by doing, not reading a ton of books.  My initial plan is to build up some self-made projects, learn the tech, and begin doing more complex things on my own.  Posting code where I can, showing what I can do via github (actually I use mercurial, but you get the idea).  I'm not going to up and quit my job right now, but I'd like to begin a learning plan.
So, where do I go from here?  Any and all advice is appreciated.  Some postings I plan on reading today at lunch:
http://news.ycombinator.com/item?id=1830872 http://news.ycombinator.com/item?id=650437 (I'm not 40, but the title got me)
Thanks for your time.  If you know of any gems posted here on HN, I'd love a link.
Thanks. <comment><USER2> I'd suggest you go on with your initial plan to build projects of your own. Projects that reflect your passion and at the same time provides you with a way to learn the technologies which you've never used before. This will challenge you a lot. The reason I think you wanted out is because of the notion that government work seems too bureaucratic and limiting. It is as if there is no environment that would cater to the people's drive for innovation and motivation.

As I have said, learn something first by doing something you have the passion outside your comfort zone. Challenge yourself. Once you feel confident that you got the chops for that technology, try to move on to a new zone where you think you could be challenged a lot.

I don't have an idea of your current financial status, but considering today's economy, I think you should weigh your options between leaving your current job right away or sticking with it and doing your side projects on your spare time.

<comment><AUTHOR> Thanks for the advice. My plan is to stay with my current position at the moment and try to open up avenues on the side. Like you said, the economy isn't the greatest right now.

</comment></comment><comment><USER3> If you are so sure that you can get a good job in the private sector, put out some feelers before you leave. Worst case scenario, no one interesting responds,you can work on an idea and foundation for a startup at work.

Personally, the people surrounding you should not be the reason why you are frustrated or enthused, it should be about the work. If the work is getting you down, talk to your manager about what you just said to us. I think you will find that he will be most interested in maintaining a capable individual and find many opportunities for you to do fascinating work, on a higher level.

<comment><AUTHOR> Good point. I have a meeting with my manager this week to talk about various things, and I hope to bring up the subject.

As for the people - we spend a considerable amount of time in the workplace, and for me, respecting my coworkers is important to me. So I'm not sure I can totally focus on the tech side of things. On the other hand, if the people were great, but I was stuck doing vb6 development, I'd also be looking to leave. Then again, maybe it is me and not my coworkers. I don't know.

Hopefully I will be posting some links on HN in the future to some apps I built!

</comment></comment><comment><AUTHOR> Again, apologies for the long post folks.</comment>


Show HN: Minerva – Better Email. Offshore Mailpile and secure managed server - <AUTHOR> https://www.minervafabric.com/ <comment><USER2> I clicked through all the way, the last step being pretty much: "get a server now".

I was met with this:

 _Got it. What happens next? We currently develop and test the Minerva Fabric with a very limited set of friends & specialists.

Once we feel confident in the current iteration, we'll make new servers available on a rolling basis. Rest assured that you'll be among the first who can reclaim their privacy._

MVP and all, I get that. But why are you submitting it to HN when there's nothing there?

It's not done. This is just an email form to collect interest. This dilutes the value of "Show HN" posts.

<comment><AUTHOR> > It's not done. This is just an email form to collect interest. This dilutes > the value of "Show HN" posts.

I honestly didn't mean to abuse the Show HN. We trying to gather any interest and feedback from the beginning. I filed it under "show hn" to show that the people behind this are here and actively following the comments for feedback and questions.

In case this is considered show-hn spam and/or and a misinterpretation of the Show-HN Etiquette, i hope the mods will strip off the "Show HN" from the title.

</comment></comment><comment><USER3> It says it's built on Mailpile[1,2], but Mailpile says on it's own github project page that it is "still experimental and isn't very useful yet."[3]

Slightly confused how this can then be something to charge for at this time if it's UI is built atop something that is self-professed experimental and not useful.

But pricing is there, and there's nothing in the language to indicate I can't jump on this ship right now, including a _Download for free_ option.

Click _Configure $39_ button. Okay, asking for email address & where I'd like to have Minerva setup. Hong Kong sounds fun. And I want to use my own domain. But wait. _Request server_? Odd word choice for proceeding with configuration.

Let's backup a step and click download instead. Wanna check out the code and whatnot so I can see what all is going on that I'd have to support if I do it myself--and I sometimes like doing it myself for certain things. Trusting a third-party for email security & privacy has, let's say, taken a big hit recently.

Click _Download for free_ and ... there's no actual download. It's an email signup form to request a server. Odd wording for a free download, isn't it? _Request a server_? I can't really request a server to download. I'm supposed to be doing setup myself, so I need software, right?

Decided to drop an email address in anyway to 'request a server' for the configure-your-own option, and it appears it's an invite list--with no prior copy indicating it was an invite-only thing.

Not exactly a good approach here, Minerva.

[1]: [http://cl.ly/image/030u0i200a1N](http://cl.ly/image/030u0i200a1N)

[2]: [http://mailpile.is](http://mailpile.is)

[3]: [https://github.com/pagekite/Mailpile#welcome-to- mailpile](https://github.com/pagekite/Mailpile#welcome-to-mailpile)

<comment><AUTHOR> We're still at a very early stage. We tried to approach it in a way to see if people would show any interest in case there's the outlook of a finished product, not only a "This Could Be". We're sorry for the confusion this caused and hope to make up for it when we launch.

Re: free download, we will release most of our technological stack that will be running on your mail-stack if you pay us. So eventually you actually could (and hopefully will, if you can) setup the same thing we're offering yourself. But in case you don't want to have the hassle seeting it up AND keeping it up2date and managed, you will have the option to pay us for that part. It will still run on a dedicated VM (or maybe physical HW) that's dedicated for you and that you can take and move anytime you want (we will not ever lock you in, i can promise that right here and now)

<comment><USER3> That is not a good enough excuse.

If you are at an early stage and using other software that is at an early stage, then say so completely up front the way Mailpile does.

As a sibling comment stated, you've completely diluted the value of Show HN by throwing up a link to something you cannot even show.

If you want to gauge interest, then do it the way everyone else does--put up clear, unmistakable language that says you're at an early stage, in development, not ready to really take anybody at all on, and have a single email signup for receiving future info and/or early invite access.

That is how you make things clear. Here, you just farmed an email address in what, for me, amounts to a dishonest transaction by not having anything at all to provide me as part of the exchange at this time.

Don't apologize for "confusion" that was intentionally created. There is practically no chance you could have developed this site and not discussed how it was going to leave people with all these call-to-actions that had zero action behind them.

Potential users shouldn't have to figure this out for themselves.

EDIT:

Seriously, "in case there's an outlook of a finished product"? Good grief. If you're just launching trial balloons, respect interested users and HN enough to state that explicitly.

This is a joke.

You received a signup for an address from @restorethe4th.com. Please remove it from your email list.

<comment><AUTHOR> We removed your email from the list and changed the wording of the buttons to make it clearer that this is WIP.

<comment><USER4> How is it any clearer? The first button says "Download for free" and the second button after entering your email says "Request Server".

The first button should say "tell me when this becomes available" and the second should say "register my interest" or something similar.

</comment></comment></comment></comment></comment><comment><USER5> A one-sentence pitch isn't enough to get me to click a button which looks and feels like taking an action. I'd rather see at least a short paragraph go into a bit more detail, first.

As it is, the homepage is basically a splash screen. That might even be okay, but making the only interaction a button that implies it takes an action makes me feel like I need to make a yes/no decision on your product before I even have a good feeling for what it is.

<comment><AUTHOR> You're right. We changed the button-wording to make it more obvious there's more information hidden behind it instead of only a signup link. (Waiting for the cache to invalidate/expire)

</comment></comment><comment><USER6> Unless I misread entirely, your price is $39/mo. That's just not happening for personal email. I value my time, but I already have a vps and you're charging $500/year. And I'm even willing to pay for email; I already pay $2.50/mo via tuffmail.

<comment><AUTHOR> The quoted price is not for a single email account, but for the whole setup on a dedicated machine/vm. You're free to use up all the resources your machine has available, we will not introduce any arbitrary limits.

If you know 15 friends/co-workers that you want to share your private, secure mail-setup with, you're already below the 2,5/mon and have the assurance that your email and userdata will never be mined or even looked at by a provider. You will be able to move your mails (or even the whole VM dump) somewhere else anytime.

You basically pay for managed hosting of your own complete mail-stack, without any vendor lock-in.

<comment><USER6> oh, cool

</comment></comment></comment><comment><USER7> Why is there a large Philips logo on the metal block on the splash page?</comment>


Ask HN: Getting to onsites, but consistently failing - <AUTHOR> I've been on the job hunt for a while now, and I'm wondering what I'm doing wrong.  I'm getting attention from recruiters, passed every phone screen, and getting to onsites, but then consistently failing onsite.  It is not because I can't write code; I've written running code in a majority of these interviews (the rest were design/nontechnical, or whiteboard interviews where the code may or may not run).
Feedback I keep getting is more of the nature "we didn't feel like you had enough experience with X."  That doesn't really jibe to me because that should be obvious from my resume.  It seems like if "experience" is the problem, then I shouldn't have been invited to the interview in the first place, based on the resume.  A couple of times, I have gotten "It was close, but...," however, I'm not sure if I should believe that.
Assuming I can actually write code, and can do a somewhat credible job in system design interviews, what might be the problem?
Background: I have 5 years experience in software engineering, mostly web programming.  For the past 3.5 years, I've worked at companies here in the Bay Area. I'm interviewing for positions that match these requirements.
I am currently 0/6 in onsite interviews, and 7/7 in phone screens. <comment><USER3> Interviewing is a crapshoot. Often your performance is irrelevant unless it's an outlier (good or bad). What you describe is why coding portions of the interview are terrible to use. Ultimately it comes down to whether your panel believes your experience is similar enough to theirs to make them feel comfortable. It's simply unreasonable to expect to be able to adequately judge the results of years of experience over the course of an hour or three or four.

Also, companies _think_ they can afford to be selective. This tells me that companies don't actually have a development talent shortage to face, or else that they believe any such shortage is only applicable to other companies, or else they vastly overestimate the appeal they have to the labor market.

<comment><USER4> > Also, companies think they can afford to be selective. This tells me that > companies don't actually have a development talent shortage to face, or else > that they believe any such shortage is only applicable to other companies, > or else they vastly overestimate the appeal they have to the labor market.

This. I keep my eye on the market, deal with recruiters all the time and I've seen too many companies that are always looking. I've done interviews in such companies only to get contacted by another recruiter with exactly the same role and exact copy of job spec some 2 years later. I'd like to know reasons why, but I can only speculate. Could be that management is pushing for more developers, but incumbent developers don't want any, so they sabotage all the candidates. I got my previous 2 jobs because the companies were desperate - single interview and an offer on the same day. As far as I'm concerned there are just too many companies out there that are just wasting time.

</comment></comment><comment><USER5> A mistake I used to make: not preparing for the onsite interview.

Things like...

* Researching your interviewers beforehand. I start with LinkedIn. It can be useful to know if they have just joined the company, or have been there for years.

* Making sure you have some good stories to tell about your most recent role, what things _you_ did/influenced/changed. Practicing them with a friend or spouse.

* Being physically prepared for the interview, getting a good night's sleep beforehand, arriving at the office in plenty of time so you're not flustered. I aim for >30 minutes early. Don't actually go into the office until 5-10 minutes before the interview is scheduled to start though.

* Finding out about the dress code at the company, and dressing accordingly or, if in doubt, slightly (but not too much!) smarter than the dress code you expect.

* Being ready to give a good handshake - warm, dry, and politely firm. If you notice your hand is cold and clammy beforehand, sit on it for a bit!

* Making sure you are polite and courteous to _everyone_ you meet in the office. Maintain your posture. If you normally chew gum, don't. Assume that the interview starts the moment you set food inside the building.

* Be ready to appear fascinated and interested in the job role. Have some smart questions to ask at the end when they say "any questions?"

<comment><USER6> >good handshake - warm, dry, and politely firm

Anyone else suffer from seasonal (autumn, spring) palm sweat? This is like my single biggest issue during these periods.

<comment><USER7> Buy highly concentrated antiperspirant. The stuff you are supposed to apply once a week to your underarms. Use it on your hands nightly until you stop sweating.

</comment></comment></comment><comment><USER8> Interviewing is a game of numbers and luck. I know it is slim consolation. But I did 15 onsites, before landing on my dream job. Was rejected from all these onsites, even though I thought I did fairly well in most of them. It was depressing and I lost a lot of confidence during the process. But eventually it worked out. So, I guess, my advice would be to keep going at it. Keep improving incrementally with each interview, but I think you are doing all the right things and will eventually get an offer.

</comment><comment><USER9> All else being equal, you want to come up with ways to sound excited about the position without coming across as a kiss-ass. If I really want the gig (or if I'm desperate) I try to ask questions and subtly guide the conversation in ways that show I've done my homework and have thought about the company's / department's unique challenges etc. and how my interests, experience, and career goals align. It's delicate, but beyond a certain baseline of competence it's about making them like you. That gets tricky because it's so easy to show desperation which will sink you. People like you to be interested in _them_ and then they magically like you back. Ask _lots of questions_ and get them to talk about themselves.

I'm going to skip the whole speech about how you want to find a good fit or else you'll be unhappy because … uh … money.

<comment><USER9> By the way, I have over _twenty_ years experience. I had maybe 4 onsites last cycle, and yeah rejection is a real drag on your confidence.

</comment></comment><comment><USER10> Based on your being in the Bay Area, I feel your problem may actually be competition, and you should apply to employers that don't look as trendy, don't offer as many perks, or are earlier stage startups and that are more likely to be a little more desperate.

For onsites employers will invite often a bunch of people (give themselves opportunities) and they always, always have the secret hope to get someone who is really overqualified for the job. In a hot market it's likelier they do.

Otherwise, often in onsites the questions asked will be more free-form (including "do you have any questions") and it's expected that the candidate will show curiosity and knowledge outside their narrow range of expertise. For example, if asked something you don't know, say "I don't know, but if I had to do it at work, I would find out through doing X". (Something that has also worked for me is straight up asking the interviewer. "I don't remember if a left join or inner join is better in that case though. For my personal education, which do you think?") You need to look like a self-starter for most onsites, and that behavioral test is more important than the answers to the exact questions.

Interviewers tend to telegraph (especially in person) the answers they want. So if you're asked how you would design a component, and the interviewer looks impatient and says "hmmm, what about X though?" or is constantly asking you follow up questions, maybe fidgeting, it means they have a specific answer in mind you haven't given. And in my experience in that case, you can't just stick to your previous line of thought when that happens. You can either "throw things out and see what sticks", where you start detailing the suggestion that generates the best nonverbal feedback, or you can refuse to play the game and say look, I'm feeling I would do X because of my experience with Y. It looks like you have a specific answer in mind that was maybe informed by a different experience. Do you want to share it and we can talk about my thought process?" (This only works for open ended questions)

</comment><comment><USER11> Don't forget, you only need one to land.

Remember someone is paying you for your time and skill right now! If you can let the rejections go, a hard thing, you'll end up somewhere that's a better fit.

I recently changed roles and it took a handful of onsites to catch my stride - more than six for sure! I only ended up with two offers but they were from my top two companies.

If you'd like feedback you're welcome to reach out (username at gmail) and I'd also suggest doing a phone screen with Triplebyte, I was really surprised by how much constructive feedback they gave although that didn't lead to my next job.

</comment><comment><USER12> Anything to do with your appearance or demeanour?

</comment><comment><USER13> Not sure if that exists, but you may want to do a fake on-site with a coach; you're probably doing a few things wrong and the companies aren't telling you for HR reasons.

</comment><comment><USER14> Just get them to like you as a person. Interviewing is really fucked up and not meritocratic.

</comment><comment><USER15> If you're getting feedback about not enough experience in the same thing from multiple interviews? It means you may want to study it more or deemphasize it on your resume.

If you're getting that feedback on multiple areas, you might try reporting less experience -- only showing your bay area jobs and saying recent experience instead of experience. (The downside here is for many companies years of experience = level = pay)

<comment><AUTHOR> No, it's been twice I've gotten the "not enough experience" (non) answer. Once, the recruiter literally told me they were looking for someone with more "API experience." I don't even know WTF "API experience" is if not back end web development, and you can clearly see the number of years' experience I have in that on my resume.

<comment><USER16> All I could venture to offer to explain that feedback would be: did you use proper http codes? Post vs get vs put vs delete? Headers? Auth methods? Did you sign the requests? Maybe something else I'm forgetting.

</comment></comment></comment><comment><USER17> Founder of Pramp.com here. Coding and interviewing are orthogonal skills. Being good at the former doesn't mean you're good at the latter. My recommendation to you is to challenge your own assumptions about your interviewing skills. Do you know for sure that you can do a credible job at system design? Is the process you use when tackling a problem aligns with interviewers' expectations? How are your communication skills? The point is that practicing coding interviews with other engineers will allow you to get unbiased feedback about many of the aspects above.

</comment><comment><USER18> You might want to think about how you're selling yourself in these interviews. Imagine them sitting in a conference room going over candidates they have interviewed and talking about who they want to bring back for another interview and why. What would you want them to say about you in that discussion? Make sure to drive those points home through the whole hiring process (resume/cv/interview/etc).

</comment><comment><USER19> Interviews are hit and miss.

That being said, if you're applying for jobs you're qualified for, you should hit 50%, not 0%.

I wouldn't expect useful feedback from companies that interview you.

In your position I would look for a (paid) service that does a mock interview and gives you honest feedback. With minimum googling I found [http://programminginterviewprep.com/](http://programminginterviewprep.com/). I'm sure there are others.

Alternatively, you can ask trusted people you know to do it for free, but then that might bias the feedback (people you know usually don't want to hurt your feelings; you need honest feedback).

Also, if people tell you that you don't enough experience in X (e.g. JavaScript) then it's most likely not "you've been doing it for 3.5 years and we really need 4" but "we want someone who knows JavaScript in-depth and based on our interview we don't believe you're on that level".

So to remedy that, fill your JavaScript gaps and go deep enough so that you can present yourself as know JavaScript in-depth.

Since perception > reality, you don't have to be 100% expert. Sometimes it's enough to do 1 in-deep investigation and then talk about that one thing (e.g. micro benchmark es6 construct vs. its es5 transpiled version in Chrome, write a blog post about it with methodology and nice graphs and try to inject that interview conversation).

<comment><USER20> > That being said, if you're applying for jobs you're qualified for, you > should hit 50%, not 0%.

Most interview panels I've been on interview at least 3 candidates, sometimes 6 or 7. So the expected success rate has to be less than that.

And from my experiences on both sides of an interview, it's pretty close to being random. On more than one occasion I've seen carefully crafted interview scores thrown out in favour of who the most senior interviewer 'felt' was better.

</comment></comment><comment><USER21> It's a lottery (slowly becoming in literal sense with the same magnitude of probabilities), but I could vote for at least one of the two reasons - <USER2> salary requirements, or you are >>30 years old applying to a workplace full of people in their 20s.

<comment><AUTHOR> If it's salary requirements, then they're psychic, because I'm dodging that question left and right. I've literally not answered that question with a number yet. But I am in my mid 40's.

</comment></comment><comment><USER22> I haven't tried it, but I've heard good things about interviewing.io. You do a practice interview with someone for free and they give you feedback.

[https://interviewing.io/](https://interviewing.io/)

<comment><AUTHOR> I've actually signed up for this, and looked at some of their interview replays. They only do phone screen type coding interviews. Those are the kind I consistently do very well in anyway.

</comment></comment><comment><USER23> A lot of the times, the companies will pick the guys who have the best portfolio. You must be lacking in either finished projects or open source ones. It will be wise to invest your time in building it up to be more impressive and eye catchy.

<comment><USER24> This sucks for people who work in house. Most backend work seems to be like that.

</comment></comment><comment><USER25> What kind of numbers are we actually talking here? Three interviews? Five? Ten? Fifty? Its really hard to tell if its just an expected rate of failure or if you are missing a Hard/Soft skill...

<comment><AUTHOR> Good question. I am currently 0/6 in onsites, and 7/7 in technical phone screens. I anticipate having more interviews in the next couple of weeks, as I have no shortage of opportunities in my inbox.

<comment><USER25> I'm a little late to the party on my follow up... but the numbers should be an indicator of what your next steps are. If you 7/7 on phone screens that means you're doing just fine there, and that the issue is something in person (more than likely its an easy fix too! most of the screening for applicants happens in the early stages)

I would start taking detailed notes of the types of questions that you are being asked in person and examine the initial reactions of the people that are performing the interviews. There's something hidden somewhere in the interview and you can find it if you read the situation carefully.

Last note quick => I would REALLY REALLY dig in and make sure these are even companies that you would want to work for... just because they want to hire you doesn't mean that its a good fit for you. And despite what it may seem (the way companies run around pretending that you should feel fortunate to work for them) the consequences of taking a job that is a bad fit are significantly worse to you than for them to make a bad hire.

</comment></comment></comment><comment><USER26> What do you look like? Being attractive helps. Dress well, carry yourself well.

<comment><AUTHOR> I'm a 6' tall white male, probably average to slightly above average attractiveness.

</comment></comment><comment><USER27> Age? Sex? Race? In the US all these seem to matter, age only in the EU

<comment><USER24> Interesting that you say that. I am based in Europe and always felt that the sexism / racism thing was over blown (to be fair I don't hear a great deal about racism in tech).

</comment></comment><comment><USER28> Easiest way is, tell them that you are willing to work for half the wage required for this role and see what they've to say :)</comment>


Seasonal gene expression reveals annual differences in human physiology - <AUTHOR> http://www.nature.com/ncomms/2015/150512/ncomms8000/full/ncomms8000.html

<comment><USER2> This is really cool. It could be pretty fundamental biology that people just haven't looked at.

</comment><comment><USER3> Hey there, what's your seasonal gene expression?</comment>


Ask HN: Book recommendations for Linux 2.6 tcp/ip - <AUTHOR>

Sorry if "book review" is an annoying theme that often comes up, but I respect the opinions here, and I've found the reviews on Amazon to be a bit confusing.<p>I'm a long-time C programmer that has dabbled in  Linux programming, OS development, and networking. I'm very interested in doing some serious hacking of the TCP/IP (ipv4 and ipv6) stack in Linux 2.6 kernels. So although I'm not new to the whole scene, I need some serious help before I can understand the source code enough to do what I'd like to do. Can anyone here recommend a good book that you think matches these interests and my level?<p>"Linux TCP/IP Networking for Embedded Systems" (http://www.amazon.com/Linux-TCP-Networking-Embedded-Systems/dp/1584504811/ref=sr_1_1?ie=UTF8&s=books&qid=1249270908&sr=1-1) is probably my best lead, so far. It has some poor reviews, but despite the "Embedded Systems" in the title, it looks like a good general primer that's about at my level. Any thoughts, especially from owners of the book? <comment><USER2> Do you know there is "Understanding Linux Network Internals"? It's a fat beast but covers the subject in great detail. I had to write a device driver and needed to use a netlink socket for high speed chunked data transfer; for that particular purpose, I found ULNI to be too verbose and learned everything I needed from the Rubini book, the Love book, and various Linux Journal and LWN articles along with the requisite snooping into various sources.

Robert Love's book is a joy to read, very compact and quick, piecemeal introduction to the higher levels aspects of the kernel. Corbet, Rubini and Kroah-Hartman is good for some quick hands on. Supplment them with various other stuff online and linux/Documentation; nothing will force you to learn FAST like a buggy driver or a piece of unsupported hardware. I find casual academic hacking boring, and none of what I read makes sense until I am found by a very interesting problem.

Good luck.

<comment><AUTHOR> >> Rubini

I dismissed that book just because I thought a book on "Device Drivers" wouldn't cover the networking modules specifically, and that's really all I'm interested in at this point. Was I wrong? Do they cover it in detail?

>> Robert Love

Ooh yes - I've looked at that book before and almost bought it. You say it's a "compact and quick piecemeal introduction to the higher level aspects of the kernel". Would you say that book is at least detailed enough for me to start confidently modifying the source code of the networking code once I'm done? It would be nice to understand the kernel as a whole form a high-level, but what I really need is a detailed look at the TCP/IP stack.

>> Good luck.

Thanks!

<comment><USER2> You said you were "new to the whole scene"; you will at least need basic kernel hacking to work on the networking subsystem; at least enough to be able to use the appropariate memory allocation strategies; the options to kmalloc are not optional, you will have to grok what each strategy entails; same for slab buffers, synchronization primitives, error handling and clean up, etc.

The book you REALLY want is ULNI; the driver model is essential for understanding the kernel because everything in the kernel is "pluggable", you almost use the same mechanisms to register a device driver, a file system, a network protocol, or something as basic as creating a new sysfs entry.

If you're feeling too macho you can probably ignore Rubini and Love and just get ULNI and ctags ;-)

<comment><AUTHOR> No I said I WASN'T new to the whole scene. I've written kernel code before, and I'm very familiar with all those mechanisms. However the networking code is extremely optimized, and a good walk-through of the code is necessary before one is able to associate each instruction with a particular action as it fits into the whole algorithm - and that's the kind of book I'm looking for. Thanks, though.

</comment></comment></comment></comment><comment><USER3> The Linux Networking Architecture: Design and Implementation of Network Protocols in the Linux Kernel by Klaus Wehrle, Frank Pählke, Hartmut Ritter, Daniel Müller, Marc Bechler.

Here's an excerpt from the preface:

This book deals with the architecture of the network subsystem in the Linux kernel. The idea for this book was born at the Institute of Telematics at the University of Karlsruhe, Germany, where the Linux kernel has been used in many research projects and its network functionality is modified or enhanced, respectively, in a targeted way. For instance, new services and protocols were developed for the next-generation Internet, and their behavior was studied. In addition, existing protocols, such as the TCP transport protocol, were modified to improve their behavior and adapt them to the new situation in the Internet.

In the course of these research projects, it has been found that the Linux kernel is very suitable for studying new network functionalities, because it features a stable and extensive implementation of the TCP/IP protocol family. The freely available source code allows us to modify and enhance the functionality of protocol instances easily. In addition, the enhancement of the kernel functionality is very elegantly supported by the principle of the kernel modules. However, many studies and theses in this field showed that familiarization with the Linux network architecture, which is required before you can modify the behavior of a protocol instance, demands considerable work and time. Unfortunately, this is mainly due to the facts that the network subsystem of the Linux kernel is poorly documented and that there is no material that would explain and summarize the basic concepts.

Although there are a few books that deal with the Linux kernel architecture and introduce its basic concepts, none of these books includes a full discussion of the network implementation. This situation may be due to the following two reasons:

The network subsystem in the Linux kernel is very complex. As mentioned above, it implements a large number of protocols, which is probably one good reason for the enormous success of Linux. Both [BoCe00] and [BBDK+01] mention that the description of all these protocols and their concepts would actually fill an entire book. Well, you are reading such a book now, and, as you can see, it has eventually turned out to be quite a large volume, although it describes only part of the network functionality, in addition to the basic concepts of the Linux network architecture.

Operating-system developers normally deal with the classical topics of system architecture—for example, the management of memories, processes, and devices, or the synchronization of parallel activities in a system—rather than with the handling of network packets. As you go along in this book, you will surely notice that it has been written not by system developers, but by computer- science specialists and communication engineers.

While considering the facts that there was little documentation covering the Linux network architecture and that students had to familiarize themselves with it over and over again, we had the idea of creating a simple documentation of the Linux network architecture ourselves. Another wish that eventually led to the more extensive concept of this book was a stronger discussion of important communication issues: design and implementation of network protocols in real-world systems. Networking courses teach students the most important concepts and standards in the field of telecommunication, but the design and implementation of network functionality (mainly of network protocols) by use of computer-science concepts has enjoyed little attention in teaching efforts, despite the fact that this knowledge could have been used often within the scope of studies and theses. The authors consider the description of the implementation of the Linux network architecture and its structure, interfaces, and applied concepts a step towards strengthening the informatics component in networking classes.

The authors hope that this book will help to make the processes and structures of the Linux network architecture easier to understand, and, above all, that our readers will have fun dealing with it and perhaps learn a few things about the networking concept and its practical implementation.

The content of this book corresponds to our knowledge of the Linux network architecture. This knowledge is neither comprehensive nor exhaustive. Nevertheless, we have tried to represent the processes and structures of the Linux network architecture in a fashion as easily understandable and detailed as possible. We are thankful for all hints, suggestions for improvement, ideas, and comments, and we will try to consider them in later editions. Updated information about the Linux network architecture and this book is available online at <http://www.Linux-netzwerkarchitektur.de>.

~blacky

</comment><comment><AUTHOR> Just as an update if anyone cares, I decided to go with "Understanding Linux Network Internals". It doesn't cover TCP or IPv6, but it looks like the perfect level of depth that I'm looking for, so hopefully what I learn from this book will help me figure out the rest of the TCP part by myself.

The Linux Networking Architecture: Design and Implementation of Network Protocols in the Linux Kernel looked perfect, but it doesn't cover 2.6 kernels, only 2.2-2.4. Although it did cover more network protocols, I'm more interested in the current kernel than in a wide range of details.</comment>