Malware Analysis Communities? - <AUTHOR>

Since my senior year of college, I have been interested in IT Security, specifically Malware analysis and defense. However, I have not been able to find any sort of website or forum where I can ask questions and pursue discussions relating to malware analysis. Does anybody know of any communities that have sections for various malware-related topics including analysis, Anti-forensics, capture, etc? <comment><USER2> You could try the SecurityFocus forums:

<http://www.securityfocus.com/>

and Wilders Security forums:

<http://www.wilderssecurity.com/>

<comment><AUTHOR> Thanks for the links, Gnosis! Unfortunately I have not had any amount of success with anti-malware forums, mainly because they only deal with getting malware off of a computer, not analyzing it. The knowledge gap is just too great. I've encountered this with both Symantec and Malwarebytes. They are great anti-malware products and their staff are helpful (to a degree) but they do not have the knowledge that will aid me in figuring out how malware works. In fact, during my conversations with Symantec Suppost (as a business professional via the company I work for, which has a standing contract with Symantec) it actually took 15-20 minutes until I was able to communicate with the rep that I didn't want help getting malware off of a system, I wanted help figuring out how the malware worked. Needless to say, they couldn't help me in the slightest.

But thanks again, even though these sites don't look like they will help with malware analysis they look like they have a good deal of other information.</comment></comment>


Amazon is replicating its Video business model in Gaming - <AUTHOR> http://www.reflectionsofthevoid.com/2016/02/is-amazon-using-lumberyard-to-replicate.html <comment><USER2> Why not replicate something they do successful? Like their books business.

<comment><AUTHOR> Too some extent the book business model is similar to the video one. By commoditizing the publishing industry (free self publish with high margin) they generated an explosion of new authors on their kindle platform. However, their book business model is more advanced than the video as it cover both business model the service and the complimentary business model. This is reflected by the existence of the prime and kindle unlimited offer. Ultimately, it is possible that the video and possibly gaming business will evolve to something similar to the book business. This is a decade long strategic play by amazon, it won't happen overnight.</comment></comment>


H1B It's a lottery this time too - <AUTHOR> http://immigrationgirl.com/uscis-confirms-h-1b-lottery-completed/ <comment><AUTHOR> Looks like the outsourcing companies have defied Trump administration's paper tigers. No positive changes for the American workers from the H1B side. Fake job descriptions and unskilled workers are going to rule the roost this FY too.

If you read the comments at the bottom of the page, it shows how this visa is being abused.</comment>


Genetic Drawing - <AUTHOR> https://github.com/anopara/genetic-drawing <comment><USER2> I've done some things like this before, with lines, ellipses, etc., but the brush strokes are neat, and look much better. I wondered what fancy brush engine it was using, but it turns out just four images of brush strokes do quite a good job!

It would probably be possible to use a brushstroke engine like say Krita's to do better, but I don't know how accessible that is outside the GUI, and the CPU implications might end up challenging (often many strokes need to be randomly generated before you get one that's an improvement worth keeping).

<comment><USER3> Generating images is interesting because you can use the difference between the original and the last generation to bias the search, which can cut down the number of strokes you need to test.

On the other hand, what's happening here is also slightly different to what people usually mean by genetic optimisation in that it looks like it's only the most recent brush stroke that's being randomised. Normally you'd keep the entire stack of strokes around as a genome and allow any of them to mutate, which means you'd need to do a full render at every generation, as opposed to just blitting a single stroke per generation.

<comment><USER3> On reading the code, I think the latter is actually what it's doing. Might have a bit of a play with this, it looks fun.

</comment></comment></comment><comment><USER4> Very nice! The sampling mask makes a big difference. I played around some with creating shaders in Unity for this type of thing last year. I liked how it looked on still frames, but I was never quite satisfied with the results for movement.

<comment><USER2> Is this generating each frame from scratch, or are they aware of the frame before?

With the genetic drawing method linked, you could start with the current frame, and add brush strokes to the regions that are different to make it more like the next frame. That way things that aren't changing would stay still and not flicker.

<comment><USER4> Yes, it's aware of the previous frame. I believe I used some kind of motion "heat map" that would indicate where the changes were needed based on accumulated deltas that would gradually decay. However, I still had to keep the brush strokes on a reasonably regular-ish grid to get enough paint brush coverage for real-time rendering. Also, if they were too randomly placed, the image jumped around too much.

</comment></comment></comment><comment><USER5> This is so cool. Does anyone have other examples on GitHub of procedurally generated art like this?

<comment><USER6> [https://github.com/fogleman/primitive](https://github.com/fogleman/primitive) is a favorite of mine.

</comment></comment><comment><USER7> How is this performance-wise? Would be cool to apply to a video, where you'd drop ~4/5 frames and then start drawing, but instead of starting with a blank for each frame start with the previous frame.</comment>


Google's origin partly lies in CIA and NSA research grants for mass surveillance - <AUTHOR> https://qz.com/1145669/googles-true-origin-partly-lies-in-cia-and-nsa-research-grants-for-mass-surveillance <comment><USER2> Most breakthrough tech can trace it's origins to federal grants, many from defense agencies.

</comment><comment><USER3> Government Owned Omniscient Global Law Enforcement

Anyone else to take a stab for G.O.O.G.L.E. ?

</comment><comment><USER4> This is giving their grants far too much credit.

</comment><comment><USER5> Ha! ...and Facebook's?

</comment><comment><USER6> duh.</comment>


Ruby on Rails Description of a CSRF Attack - <AUTHOR> http://ianloic.com/2007/05/18/insecurity_is_ruby_on_rails_best_practice/ <comment><USER2> Please keep in mind this is from 18 may 2007... CSRF protection is now enabled by default.</comment>


Compare cost of living between 2 cities - <AUTHOR> http://www.expatistan.com/cost-of-living

<comment><USER2> This fails to take into account state and local taxes, which are often seem the single largest driver of difference in cost-of-living. Seattle, no income tax. CA, different story.</comment>


The Over-Engaged Knowledge Worker - <AUTHOR> http://www.ianbicking.org/blog/2019/01/overengaged-knowledge-worker.html <comment><USER2> _> This is the reality of knowledge work that none of these conceptualizations address: it's hard (in very specific ways), some of it we don't want to do, and the work we don't want to do piles up and becomes dominant simply because it remains undone._

Let me rephrase this for you. A lot of so-called "knowledge work" is just tedious, mind-numbing bullshit. Not only that, but you have to plow through it while fighting information overflow and your own overly complex tools, while acting completely alone. There are no librarians on the Internet.

No wonder everyone craves distractions.

 _> In the context of the browser, how do we contextualize pages and interactions inside some abstract task?_

This train has sailed off its tracks long time ago.

Browsers aren't real tools. They are designed to make for smooth information consumption so that Google and co can harvest your money and attention as efficiently as possible.

I can list _several hundred_ things a web browser should do if it aims at being "knowledge worker's tool". Let me just give some categories:

1\. Features for pausing and restarting work in different mental contexts. 2\. Local personalization. Shit like Timelines should be local and user- controller. "This is too complicated, so let's let Facebook/Twitter/Microsoft do it" is a lame excuse. 3\. Did you notice that browsers still don't have any good authoring tools? Every websites reinvents their own WYSIWYG. "Developer tools" have everything for debugging, but almost nothing for creations of stuff. 4\. User-driven integration between pages. (For example, something as simple as "when I open this page, go to that other page, find all the entries with word X and paste it here".)

Bah, that's enough. I can continue, but I think the point was made, unless the reader doesn't want to see it on purpose.

<comment><USER3> _Browsers aren 't real tools. They are designed to make for smooth information consumption_

Well yeah. They're browsers, that's what they are. "To browse: an act of casual looking or reading."

<comment><USER4> They're not really browsers - they're web page viewers.

The views are organised by the needs of the source technology - "pages" generated by a web server - not by the needs of the user.

A lot of useful features - cross-referencing, comparisons, task-based content searches, dynamic content update notifications - are either impossible or poorly implemented on the server side.

There's never been an active browser that tries to integrate information instead of being a dumb page viewer with tabs and some form filling options.

<comment><USER5> Early on, the Web was imagined to have a plethora of "agents" (bots) which act on behalf of their users, extracting information from pages, following links, aggregating the relevant data, etc.

This worked _really_ well for search engines, but certain economic incentives crept in which pretty much killed off this idea. In particular:

\- Many sites gained revenue by advertising

\- Advertising only really works if a human is viewing it (it's possible to integrate advertising into the data, with product placement, "native advertising", etc. but that's more expensive than slapping some Javascript in some iframes)

\- Exposing data publically in easily parsed formats will mostly stop humans from viewing the page. Hence so much data is now in silos.

\- Exposing services, like search, to bots can get expensive. Hence the rise of API keys.

There are other reasons too, like scraping being hard and fragile due to styling having too much influence over markup (CSS helped a little; a more radical attempt was sending raw XML and creating the page via XSLT). The rise of Javascript-only sites, and the difficulty of implementing all of the many JS APIs that a site might try to use, has also hindered alternative user agents.

The end result is the Web being dominated by a handful of user agents: a few search crawlers and a few user-facing browsers :(

</comment></comment></comment></comment><comment><USER6> Could someone more familiar with the subject and vernacular provide some sort of summary of what this is trying to say? I've read it two or three times and I _still_ can't work out the point being made.

If any.

<comment><USER7> When people discuss "knowledge work", they often slip into imagining an "idealized" knowledge worker: one who is very engaged, and doing "big" meaningful things one at a time.

In reality, (1) we are all distracted, (2) our work usually consists of a large number of small chunks that are spread over time (3) we all have a large pile of undone work (both "job" and "personal") that we're always feeling guilty about.

What do we do about this difference between the ideal and reality? We could just give up on the ideal and resign to the reality — or even make it worse. Or we could try to fight it superficially, and fail (there's a reason things are the way they are). Or we could start from accepting the reality (instead of ignoring it), but try to move towards the ideal, e.g: (1) reduce whatever it is that makes us seek distractions ( _very different_ from blocking distractions), (2) make it easier to get back on track (e.g. keep track of the task being done).

In coming up with this problem statement, the author believes he's diverged from the usual problems being imagined: things like trying to block distractions, trying to maximize efficiency, "collaboration".

[That just restated the post, so I don't know if it helps :)]

<comment><USER8> > who is very engaged, and doing "big" meaningful things one at a time.

And one who knows exactly what steps are going to be involved and how long each one is going to take, weeks or even months in advance. The fact that I approach every new programming problem with: "ok, how the hell do I figure this one out?" made me feel really nervous early on in my career, since there seemed to be this default expectation that not just good, not even just adequate, but even mediocre programmers could blurt out a list of the tasks involved in solving a problem, "estimate" (meaning predict with +/\- 5% accuracy) the hours associated with each and then robotically plow through each of them. This was back in the 90's, before somebody named this "agile" and built a set of Orwellian tools and "coaching strategies" around it, but after a few years I couldn't help but notice that, in spite of being an abject failure at accurately predicting how long each bug fix was going to take before I even really knew what it entailed, I was also one of the most sought after developers in every organization I worked at.

</comment></comment></comment><comment><USER9> Most people who uovoted would know but others may not. Ian Bicking is creator of virtualenv (and others) and one of the people doing meta thinking of tools and envs in the python world. Interesting to see something from him after some years.

</comment><comment><USER10> I'm so connected yet so disconnected with my inner self. Knowledge workers are under appreciated in the regular day job. If you also happen to be creative then you'll be challenged by some other ego trying to take credit for your work. In the struggle and dilemma: "should I care about this guy?" you forget about it and start to build resent. Then burnout is then a fact. Your challenges in college were way more benefitting and purpose-driven instead of your boss's wallet. That's why I say CS degree was awesome, working as dev is draining.

What's this comment about? Just need a rant...

<comment><USER11> I guess it also depends on where you work. You can do some pretty amazing and creative work if you find the right niche to work in.

<comment><USER10> true that! yet to find...maybe I move into hardware, something tangible is inspiring

</comment></comment></comment><comment><USER12> The problem statement is eloquently stated. Yet the blame is nonsensical. The developers of _Solitaire_ are to blame? Solitaire was a non-trivial application written for the express purpose of selling an operating system.

The reality is that the problem statement, that knowledge workers are constantly engaged with unproductive, nonsensical asks, is true. Knowledge workers must overcome this by being exactly the worker they've been advised _not_ to be: ignoring emails and chats, not being present at their open-plan desks, declining any meeting that won't immediately please their manager.

No sinister force is the cause of these problems; simple entropy of organizations is to blame: failing to allocate resources to document and streamline processes is to blame.

</comment><comment><USER13> In order to minimize context-switching and mantain browser "sessions" I recommend OneTab[0], really simple idea but is really useful.

[https://chrome.google.com/webstore/detail/onetab/chphlpgkkbo...](https://chrome.google.com/webstore/detail/onetab/chphlpgkkbolifaimnlloiipkdnihall?hl=en)

</comment><comment><USER14> > _So how might we approach idea generation around knowledge work without > idealizing the knowledge worker?_

Editing tip: you can delete sentences like this without loosing much. The following text should do a good enough job of making a point without an introduction helping it along. The trick to editing is to backspace as many words as you can.

<comment><USER15> What's wrong with an introductory question?

Also, losing not loosing.

<comment><USER16> It's conversational, and adds nothing: the reader who was following along already asked themselves that question, and the reader who's hasn't latched onto the thesis doesn't understand why this is a good question to ask.

</comment></comment></comment><comment><USER17> Claudius : Can you advise me?

Laertes : I am lost in it, my lord.

</comment><comment><USER18> a friend sending you a mortgage refinance proposal isn't work though, unless that friend is giving you money for looking at it. And I guess the only people getting sent mortgage refinance proposals are people who might have some expertise in that subject.

</comment><comment><USER19> This website is beautiful.</comment>


Why your site sucks ? - <AUTHOR> http://ifohdesigns.com/blog/resources/why-your-site-sucks

<comment><USER2> Everyone's an expert</comment>


xkcd Phone 4 - chris-at http://xkcd.com/1707/ <comment><AUTHOR> The hardest feature to implement is "Turing Completeness", given those so called "Walled Garden"...</comment>


Peter Thiel and the Ice Storm - <AUTHOR> https://medium.com/@jasongreenlowe/peter-thiel-and-the-ice-storm-744a9de7a136#.k4jfpej44 <comment><USER2> There's no pleasing you. Peter Thiel had the courage to get on stage at the RNC and proclaim he was gay, thereby opening a dialog in the republican party. This is how you affect change. Through debate and discussion. It turns out more people will listen when you stop calling them bigots at the first controversial point of view.

<comment><AUTHOR> Hi, Blackflame. I didn't call Peter Thiel a bigot, and I don't think Peter Thiel is a bigot. I _do_ think he's remarkably unconcerned about other people's problems.

I would never criticize anyone for having a debate or a discussion, but Peter Thiel just heartily endorsed Donald Trump for President, even though Trump's platform explicitly and repeatedly calls for treating gay people (and lots of other people) as second-class citizens. There's a big difference between a debate and an endorsement.

<comment><USER3> > I do think he's remarkably unconcerned about other people's problems.

With respect to this and the blog post on poverty I cannot agree.

Thiel has been complaining that outside of Silicon Valley people are not getting wealthier for at least a decade and a half. There's even a video in 2008 where he angrily accuses Eric Schmidt of telling people to eat iPhones. If anything he is remarkably concerned with other people's problems compared to his contemporaries.

The common belief in many wealthy circles is that everything is progressing onwards and upwards (look at any blog posts by Peter Diamandis). Unfortunately this is utterly afactual when you subtract computation and its effects. The average person's intuition is actually correct about the economy and the communities of the wealthy are basically self constructed Potemkin villages.</comment></comment></comment>


Hardcoded secrets, unverified tokens, and other common JWT mistakes - <AUTHOR> https://r2c.dev/blog/2020/hardcoded-secrets-unverified-tokens-and-other-common-jwt-mistakes/ <comment><USER2> Well the fact that you need to 'choose' a cryptographic protocol out of the suite of ciphers reminds me of why the creator of WireGuard opted for no cryptographic agility and instead chose versioned protocols, unlike OpenVPN.

From the Wireguard paper [0]:

> 'Finally, WireGuard is cryptographically opinionated. It intentionally lacks > cipher and protocol agility. If holes are found in the underlying > primitives, all endpoints will be required to update. As shown by the > continuing torrent of SSL/TLS vulnerabilities, cipher agility increases > complexity monumentally.'

[0] [https://www.wireguard.com/papers/wireguard.pdf](https://www.wireguard.com/papers/wireguard.pdf)

<comment><USER3> If someone wanted to utilize JWT in a new system, and therefore could freely choose any of the JWT options, what would the "most secure" be?

It would probably save a lot people headaches if jwt.io published a chart of "use case" and "algo/cipher selection". From what I've researched, all the JWT code libraries give you a menu of selections, and it's on you to research which algo/cipher to use, and given the volume of selections, that's a substantial reading list.

Is there a 'versioned' JWT that picks sane defaults as they are developed + improved?

<comment><USER4> Here's your recommendations: [https://security.stackexchange.com/a/233863/111020](https://security.stackexchange.com/a/233863/111020)

And yes I just spent 2 hours writing that.

There can't be a great authentication/crypto that can improve over time. In theory, it must block old things to be state of the art. In practice, it's used in client-server communications, that would break all communications if nothing could talk unless they're always on the exact same version.

<comment><USER5> Good recommendation. I should add that RSA is faster on verification but slower on signing than ECDSA. For most application, the amount of traffic is small enough that the difference doesn't matter but for large enough traffic, you should consider the difference. The performance of course depends on a lot of factors but

[https://connect2id.com/blog/nimbus-jose- jwt-6](https://connect2id.com/blog/nimbus-jose-jwt-6)

would give some reasonable idea on the practical difference on modern hardware.

</comment></comment></comment></comment><comment><USER6> These failure modes have been used to criticise JWT over the years. But all of these errors are of the same level as "don't concatenate input strings into SQL strings" or "don't store passwords in plain text". That these errors are made says much, but not about the technology, IMHO.

<comment><USER7> An authentication protocol that has this many footguns and insecure defaults is probably just not a very good authentication protocol.

<comment><USER4> Irrelevant. If the application doesn't call the function to verify the user token, the application is insecure, it doesn't matter what protocol or token format is used.

<comment><USER8> My response to that would be "Parse, don't validate" ( [https://lexi- lambda.github.io/blog/2019/11/05/parse-don-t-va...](https://lexi- lambda.github.io/blog/2019/11/05/parse-don-t-validate) ).

In other words "the function to verify the user token" should _also_ return the payload data (e.g. the user ID, or whatever), and it should ideally be the _only_ way to get that payload data. That way there's very little that can go wrong:

\- If the verification function doesn't get called, then the application doesn't get its payload data (e.g. user ID), and hence can't be misusing that data.

\- If the application _is_ using the payload data, then it must have come from the verification function, and hence has been validated.

I'm adding JWTs to a system right now, and forcing this interface by encrypting all of the payloads. The only system with access to the decryption key will refuse to decrypt tokens which aren't signed, valid, non-expired, etc.

</comment></comment></comment></comment><comment><USER9> Thanks, for the article. Always good to refresh the common pitfalls.

There is a RFC which also details the best practices for JWT: \- JSON Web Token Best Current Practices: [https://tools.ietf.org/html/rfc8725](https://tools.ietf.org/html/rfc8725)

On the similar topic, some more interesting RFCs / Drafts from IETF on OAuth: \- OAuth 2.0 Threat Model and Security Considerations [https://tools.ietf.org/html/rfc6819](https://tools.ietf.org/html/rfc6819)

\- OAuth 2.0 for Browser-Based Apps - [https://tools.ietf.org/html/draft-ietf- oauth-browser-based-a...](https://tools.ietf.org/html/draft-ietf-oauth- browser-based-apps-06)

</comment><comment><USER10> Reminder: If your jwt only contains a session_id from your framework, which you then look up in a database every request, you absolutely should not be using jwts.

<comment><USER11> I'm using it for cross server auth

</comment></comment><comment><USER12> Regarding leaving the secret in your codebase during development... always make sure the secret is not in your git history. Change it when you move it out of the repo. I know this seems obvious, but I always do a git history on files that might have tokens or secrets in history and you'd be surprised how often I discover them.

<comment><USER13> > you'd be surprised how often I discover them

Oh yes, now defunct but we had a search engine just for that :| [http://archive.is/0GNLl](http://archive.is/0GNLl)

</comment></comment><comment><USER14> I evaluated JWT recently and found it so overcomplex and full of footguns that it was unacceptable for a security-critical component. It boggles the mind that this was designed as recently as 2015.

I plan on going for a simple bespoke solution which just uses an off-the-shelf algorithm to sign a structured payload (likely BARE), which in addition to being less full of shit will make for smaller and more managable tokens.

<comment><USER15> Can you elaborate on this?

> ... full of footguns that it was unacceptable for a security-critical > component.

What it not secure about it?

<comment><USER14> See TFA for a subset of the problems. Security should be designed so that it's easy to do it right and hard to do it wrong, but JWT offers many opportunities to do it wrong. Maybe you can figure it out with enough reading of TFA and other articles online, but there's no reason to take a risk on such a shoddy security standard in the first place.

</comment></comment></comment><comment><USER16> Interesting article. On this note, recently as part of small group of people we needed to implement API -> API authentication. Unfortunately, we don't have anyone who has solely implemented this before, so we took to searching to try and pick something resembling industry standards. Of course Oauth 2 got brought up, however the spec and articles on it seem bizarrely out of sync with out needs. Notably on two points:

1\. It seems any piece of content talking about Oauth 2 is referring to 3rd party, external authentication. Ie i want a client API to talk to my API but with an identity managed by a third party entirely. This is not our use case, but seems to be the primary focus for Oauth 2 articles.

2\. The rest of the spec _feels_ very loose for our use case. There's loose definitions _(it feels like)_ on some basic patterns of tokens and renewal tokens, but it all seems so wildly flexible that we're almost lost on what to actually write, what technologies to use to generate tokens, etc.

Any tips for narrowing the field, to implement the right thing, in the right way? This article struck a chord with me.

<comment><USER17> most of my knowledge about oauth is about using it at my dayjob for years, so don't take my answer with any authority. its more a users perspective of the technology then how it was actually meant to be used, which i do not know.

first off: the whole idea of using oauth in your systems is to externalize authentication. its main usecase is if you have several services which share the same users... usually microservices, but everything else which supports oauth works as well, obviously

if you do not actually want to externalize your authentication, then oauth is likely not what you actually want. (keep in mind this only applies to the process of authentication. you will probably still create a `users` table to put a foreign key on your domain entities. said users table just won't have a `password` field or similar)

under this perspective it makes sense that you basically only find blog posts to integrate 3rd party tools, right? you're just setting out to implement a new oauth2 server otherwise after all, and... i'd strongly suggest you don't. there are so many pitfalls you can fall into. its not a trivial subject.

thankfully, there are readily available FOSS oauth2 servers like keycloak around if thats what you want to use...

but to come back to my initial point: if you do not want to externalize your authentication, then do not use oauth! almost all frameworks already have tools available you can use to authenticate with api tokens. it keeps you within the bounds of your frameworks strength and you can automatically fetch a token on your frontend after the user authenticated with your usual flow.

but i'm already off topic considering your first question. if you _do_ use a solution like keycloak, there are `oauth clients`. these behave exactly as you want.

<comment><USER16> > first off: the whole idea of using oauth in your systems is to externalize > authentication. its main usecase is if you have several services which share > the same users... usually microservices, but everything else which supports > oauth works as well, obviously

We don't want to externalize it. For clarity _(because i think i implied internal API->API)_, our primary concern right now is a client API interfacing with our API. We want to choose some implementation that clients would expect, and easily reason about.

There may be a future where those clients have to authenticate to multiple APIs / microservices of ours, but currently it is one API setup for this explicit purpose.

> but to come back to my initial point: if you do not want to externalize your > authentication, then do not use oauth! almost all frameworks already have > tools available you can use to authenticate with api tokens.

Yea, not using Oauth was my thought was well. My concern however, was trying to pick something clients would expect. I don't want it to feel custom, arbitrary, hodgepodge.

Our very early impression / plan is to use a token renewal, and a shortlived token, similar to oauth. However this loose, and custom, and i don't want clients feeling like we're making things up. Hell, i don't want to make things up.

My current thought is that I need to research JWT more, as it may fit what we need, and be more standardized.

 _edit_ : And PASETO

<comment><USER17> You should probably just try out a foss oauth server to get a feel for it.

Just start a keycloak server with docker and write a small Webservice in your language of choice which only let authenticated users open the website, which just prints the user name for example.

After you did that, you could write a second service which accesses that api using a client (not user) and prints which users have accessed it since it was started.

That should be doable within a few hours at most and give you a feel for the technology.

If you're fluent in python, I'd personally suggest just starting a hello world flask project, connect it to a keycloak and write a second cli script which simulates the non-user access. (If you use Java, keep in mind that springboot2 uses Springsecurity5, which was incompatible with the official keycloak Java sdk the last time I checked. Either use springboot1 or expect to have a slightly harder time figuring out what to write in the application.properties)

Reimplementing advanced authentication systems like jwt on your own is extremely error prone and frankly unnecessary if you're not in the business of authenticating users. I'd suggest to either use whatever your framework prefers (which is usually just a static and very long, manually rotated token) or try to externalize it by using readily available foss solutions

</comment></comment></comment></comment><comment><USER18> Simply not hard coding your secrets is a good first step. Even better is to think about how you're going to rotate secrets. So much easier if you build it in from the start than if you need to tack it on later. Once you have secret rotation supported, you can set up mechanisms to do it regularly and you can rest a bit easier at night.

<comment><USER19> This is why looking into JWKS might be a decent first step.

</comment></comment><comment><USER20> We'd recently moved our jwt code into rust (using the jsonwebtoken crate), and it's wonderful to have this article to read in that context. The pitfalls listed in the article were mostly handled by design, so we had to correct mistakes in the PHP implementation that were impossible to make now.

Eg we'd been using the `iss` field, a claim, to determine which key to use, when we should've used the `kid` in the header. All claims are now completely inaccessible until verified to the point where there's no way to access them short of manually deserializing the jwt. We'd also allowed some invalid tokens where the `exp` was in the past, only if you were refreshing the token of an active session. This could've caused problems if we ever wanted to issue a token that expired very quickly, since you could have a token that's theoretically 10-times longer lived than you intended.

</comment><comment><USER21> I only see a list if four "Don'ts". Would have been nice to have some "Do instead's". Of course I can try to think about "good" ways to access my secret other way than hard coded, but then I may not be the audience. If I already made the mistake of storing keys hard coded, then how should I know that my new way is better than this?! For articles like "X mistakes in topic Y" I expect some solutions to solve the problems. The other way is only pointing at fools for me.

</comment><comment><USER22> This otherwise good list omits the most pervasive mistake, which is not to use stateless JWTs for authentication.

<comment><USER23> Could you elaborate on the why here?

<comment><USER8> From my understanding, the use case for JWTs is where:

1) One system produces a JWT which makes a bunch of "claims" about the bearer; e.g. 'their email address is email@example.com'.

2) Another system which trusts the first system can consume these tokens and use the "claims" to authenticate the bearer, e.g. 'the email address email@example.com is associated with user ID 12345, so the bearer must be user 12345'.

3) Now that the user is authenticated, the second system can initiate a session for them (e.g. storing a cookie in their browser, or whatever).

4) From now on that JWT should be rejected; we can approximate this by making it short-lived, but we can do better by the second system update some state. For example, updating the "last seen" time of a user, and rejecting any tokens issued before the this time.

The idea of 'using stateless JWTs for authentication' is presumably the common practice of avoiding steps 3 and 4: rather than having the second system initiate a session for the user, the JWT is treated as if it were a session token; rather than invalidating the JWT after its first use, it is accepted over and over again as part of the user's subsequent requests. This also requires the JWTs to have a longer expiry time than necessary, since they have to survive until the user's _last_ interaction, rather than their _first_ (in fact 'refresh tokens' can be used as well, but the principle is the same).

As for why it's a bad idea, I recently came across PASETO (a simplified alternative to JWT) which specifically calls this out as an insecure practice due to the possibility of replay attacks ( e.g. [https://github.com/paragonie/paseto/tree/master/docs#was- sta...](https://github.com/paragonie/paseto/tree/master/docs#was-stateless- session-tokens-one-of-pasetos-design-goals) )

<comment><USER24> JWT (specifically JWS) is essentially a simplified modern alternative to X.509 certificates and I think it should be perceived in that way.

Do we use certificates for authentication? Yes, and it works well.

Do we use certificates for authorization? Not likely. Instead, usually we ensure ID of entity which we communicate with (as result of authentication) and lookup it's permissions in database.

Do we recognize any certificate body as immediate permission requisite? No! Instead, we have signed claim with public key which entity should present in order to prove it's identity (and prove possession of private key later). I. e. we issue end-entity certificate which describes and verifies public key of that entity.

Do we NEED to revoke these end-entity certificates or have them expiring really fast for normal operations like logout? No. There may be something similar to CRL and OCSP for handling emergencies, but this is optional. Instead, we are not trying to handle authorization tasks with authentication mechanism. It's different things. We may just change status for authenticated entity in authorization database. Like: "user:admin;device:XXXXX;status:terminated".

What semantical meaning has revocation/expiration of token issued to firstname.lastname@example.com? She is no longer alice? I doubt that. She is no longer allowed here? Or her specific device is no longer allowed here? Or her service subscription is just expired? Either way it's not a question for part which recognizes known entities and must work in a "yes" or "no" fashion.

It's not wrong to use JWT for authentication. It's wrong to use ANY authentication for ANY authorization.

</comment></comment></comment></comment><comment><USER25> Weak secrets is also a common issue. HS256 and similar algorithms allow for offline bruteforce attack.

<comment><USER26> I would guess that secrets are also often reused over different environments like test and prod.

</comment></comment><comment><USER15> Anything when not used as it was designed to be used is dangerous.

JWT is a very useful technology, one that increases UX by decreasing overall latency.

Any technology, can be "abused" or used by people that do not understand how it works, and as a result you will have an insecure system.

<comment><USER27> The technology here is digital signatures.

JWT is a standard for using that technology, and one that makes several design decisions which make it fragile.

This is the difference between, say, the idea of a car (and the benefits thereof) and a Ford Pinto.

</comment></comment><comment><USER28> I remember JWT coming on my radar around 10 years ago during the height of RESTful bikeshedding. It was a "must have" because sessions, being stateful, are not pure REST. Now that we've come off the height of the pure REST fanaticism and have learned about a bunch of issues that JWT causes, is it really better than just having a session id and a session stored on the server? Does it actually provide any technical benefit? Honest question.

<comment> brown9-2 These are orthogonal concepts. A session ID is just a token that the server uses to lookup some state about the request that presented the token. A JWT is a token that can be used to present a claim of who the requestor is (and the server can verify it). A session ID token doesn't help my request prove I am who I say I am when I call your API for the first time, unless you've implemented some sort of state store that all of your API services and server share.

<comment><USER29> > unless you've implemented some sort of state store that all of your API > services and server share.

Yes, it's called session storage, and it used to be incredibly common. These issues are not "orthogonal", because a primary promise of JWTs were the ability to get rid of that shared session storage and just put that identifying info into the signed token.</comment></comment></comment>


Coronavirus May 'Reactivate' in Cured Patients, Korean CDC Says - <AUTHOR> https://www.bloomberg.com/news/articles/2020-04-09/coronavirus-may-reactivate-in-cured-patients-korean-cdc-says <comment> nil-sec This came up since the beginning in January. The number of people that were reported to reinfect like this seem to lie well within the expected rate of false positives. I don't understand how this claim is made repeatedly, without a thorough statistical analysis of what is expected.

<comment><USER2> It's not just tests. All symptoms disappear - we had health workers coming back to hospital and days later symptoms reappeared

EDIT: to clarify: in my hospital there were colleagues who tested positive, isolated, had no symptoms, went back to work, still no symptoms. After a few days symptoms are back and they tested positive again

<comment><USER3> It's flu season, it's quite possible they have had both flu and Coronavirus. My wife had the flu in Feb & early March and the symptoms are identical to the Coronavirus.

They're working in a hospital so they'll catch everything.

<comment><USER4> Stupid question does being infected with some virus helps if a second pathogen infects you ? As in 'immune system globally more active' or not at all ?

<comment><USER5> Not a stupid question, but pretty much not at all. In fact, you are slightly immune compromised by the first pathogen against the second pathogen because your adaptive immune system is highly active against the first pathogen. The body has two systems, an innate immune system that has mechanisms to innately attack bacterial and viral pathogens in general and an adaptive immune system that adapts to each specific pathogen. The innate system is a first line of defense but is like a form of combined suicide bomber and cannon fodder throwing cell after cell to overwhelm the pathogen with death and there is significant collateral damage. The adaptive immune system takes time to ramp up to each new pathogen and if it's working on one pathogen doesn't really switch to working on the other. Rather older cells are/have adapted to the first pathogen and newly born cells begin to work on adapting a defense to the new pathogen. This takes time and there are, at the beginning, very few cells adapting to the new pathogen.

</comment></comment></comment></comment></comment><comment><USER6> I wouldn't read too much into this yet. Coronavirus tests are notoriously flaky and Korea has thousands of recovered people, so it's entirely possible that people either get false negatives twice (so they appear cured but aren't), or get true negatives followed by a false positive later (so they appear sick but aren't).

If there are patients who test negative for several weeks but _then_ relapse and get clinically sick with COVID-19, we'll have cause to worry. On the other hand, if this was a common phenomenon, it'd be happening in China already.

</comment><comment><USER7> Whether the tests are flaky or not, we shouldn't really discard the fact that virologists don't generally say that infected people do not get reinfected, but rather that infected people will have milder symptoms and therefore less lethal diseases. There was a really good episode on the immune system on TWIV recently.

[https://www.microbe.tv/twiv/twiv-597/](https://www.microbe.tv/twiv/twiv-597/)

<comment><USER8> Will COVID-19 become another cold? Were the 4 other coronaviruses that cause colds just as deadly when they first appeared? Interesting to ponder.

<comment><USER9> Actually yes. The virus that caused the 1918 "Spanish flu" is now one of the seasonal flu viruses that afflict us every year.

</comment></comment></comment><comment><AUTHOR> I'm hoping it's just bad tests in the first place.

[https://covid19usa.io/](https://covid19usa.io/)

Edit: Adding my website for tracking coronavirus cases over time in USA state and counties. Original Show HN here: [https://news.ycombinator.com/item?id=22778152](https://news.ycombinator.com/item?id=22778152)

<comment><USER10> Most likely. You need to develop antibodies to defeat a viral infection and, once you do, they don't just disappear overnight.

<comment><USER11> Well, as far as I know, the antibodies do disappear at some point, but now your immune system knows the virus and can quickly produce new antibodies again, if it finds the virus again. But I think over time the immune system also forgets.

But to the patients that got sick again. "Bad test" would be the best outcome yes, but I know, that with herpes virus for example, you can never really get rid of them. Your immune system defeats them, but some virus cells remain in stasis somewhere sleeping - and resurface, if your stress level is high enough. So it would not surprise me, if this coronavirus might have similar traits.

[https://en.wikipedia.org/wiki/Herpes_simplex](https://en.wikipedia.org/wiki/Herpes_simplex)

<comment><USER12> Something I read recently is that the immune system is complex containing both an innate immune system an adaptive immune system. It's possible that in mild cases of Covid-19, it may be successfully handled by the innate immune system, which doesn't confer additional immunity. A mild case my occur if the initial infection is by a relatively small number of virus particles.

<comment><USER11> Also possible yes. The killer cells (do not know the proper english name) are general purpose defense cells against anything that the body does not know. Only if there are too many unkown cells of similar type, they get categorized by other cells in the lymphs of the immune system and then other cells finally produce antibodies. But that process takes time.

<comment><USER13> >The big "eating" cells (do not know the proper english name)

macrophage

<comment><USER14> >macrophage

That's arguably Greek ;-)

<comment><USER15> _The problem with defending the purity of the English language is that English is about as pure as a cribhouse whore. We don 't just borrow words; on occasion, English has pursued other languages down alleyways to beat them unconscious and rifle their pockets for new vocabulary._

— James Davis Nicoll

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER16> > _About 51 patients classed as having been cured in South Korea have tested > positive again_

So does this mean they're symptomatic again, or just testing positive?

<comment><USER17> A friend of mine fell ill with COVID-like symptoms; bad enough to be unable to work remotely, but not bad enough to go to the hospital and be tested (in the UK). He was sick for 5-6 days, then felt better, and thought he'd completely recovered. One week later, his symptoms returned for 2 days, and now he's feeling better again.

Was it COVID the first time? Was it COVID the second time? Was it stress, or his body adjusting to being in the same environment 24/7 (rather than spending half his day outside the home)? We may never know. An antibody test would show if he'd had it _at least_ once, but that could well have been 2 back-to-back colds with asymptomatic SARS-CoV-2.

<comment><USER4> I Wonder if a virus could create waves inside one's body

<comment><USER18> It is a totally unrelated virus, but if you get chickenpox you may get many years later shingles, the virus may remain inactive in nerve cells fo a long time. [https://en.wikipedia.org/wiki/Shingles](https://en.wikipedia.org/wiki/Shingles)

<comment><USER4> What a strange game of life

</comment></comment></comment></comment></comment><comment><USER19> Is it me or are Bloomberg articles in general panic stricken? I've been observing this over a number of articles lately, almost all of them are adjusted to sound like they want us to be more afraid and tensed, maybe my subconscious bias dunno.

<comment><USER20> I remember a thread about their writers being rewarded when moving the market.

</comment></comment><comment><USER6> [https://outline.com/n8cTuW](https://outline.com/n8cTuW)

</comment><comment><USER21> IMHO I'm fairly skeptical of these claims. There have been a lot of scrambling to get testing online worldwide, so I'm fairly certain that Pr(False Positive) and Pr(False Negative) are reasonably high given circumstances. I don't have the numbers on me right now, but I _think_ sensitivity is only 70%... So it's not hard to imagine a situation where a combination.

The thing that in my mind sorta kills this "reinfection" argument is our previous experience with SARS virus [1]. Figure 1 of the paper shows IgG reactivity for up to 3 years. Anecdotally, I've heard from my immunity bioinformatics colleagues that the spike protein that is recognized by the immune system is very well conserved.

I haven't seen a ton of _great_ evidence that have really pushed my priors against the conclusion that these handful of cases are better explained somehow by faulty testing/some other combination of complication. As always, willing to adjust these priors if given compelling evidence. :)

[1] [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2851497/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2851497/)

</comment><comment><USER22> It is unclear whether these patients are shedding viral RNA particles from their prior infection or actual infectious virus from an ongoing infection. The way to tell the difference would be to perform viral cultures instead of the PCR-based test.

<comment><USER5> Your post makes little sense to me. viral RNA particles are virus no? You distinguish between viral RNA particles and "actual infectious virus". What is the difference between the two?

<comment><USER23> PCR tests will still detect signature bits of RNA even when the virus is physically broken and no danger at all. Virions are not alive like bacteria are, but they are still more than the pure RNA ("data"). To be infective they need the whole, intact package of RNA and a set of tooling molecules that the parent cell created created alongside the RNA copy (or that were taken from regular cell stocks).

In year 2000 computer terms, the virus doesn't just need the bytes of the VBS file, it also needs the email header that says "ILOVEYOU".

<comment><USER5> Not really though. Naked RNA is incredibly unstable when just out in the environment or floating around in blood, mucus, etc. Sure, if you dump a pot of naked RNA on the floor, mop it up and analyze it that second your PCR test will pick it up, but you would have to be right quick about it.

</comment></comment></comment></comment><comment><USER24> I am not sure whether this has to do with testing accuracy, however we had a similar case [0] recently in India.

> A 60-year-old woman from Pune who tested negative for coronavirus died three > days after being discharged.

> However, on April 4, she was rushed to Sassoon Hospital in Pune, where she > died even before admission. When her sample was tested again, it turned out > to be positive.

[0] [https://economictimes.indiatimes.com/news/politics-and- natio...](https://economictimes.indiatimes.com/news/politics-and-nation/pune- womans-death-due-to-covid-19-puzzles-authorities/articleshow/75003417.cms)

<comment><USER24> Correction: This isn't a case of reinfection/reactivation as the woman in context was never tested positive when she was alive.</comment></comment>


Microsoft Responds to Abusive Labor Conditions at Chinese Manufacturer - <AUTHOR> http://www.fastcompany.com/1617116/microsoft-chinese-suppliers-ncl-kernaghan-kye

<comment><USER2> I wonder, are the reporters here genuinely concerned about improving workers lives, or are they mostly enjoying having some leverage over a mighty, powerful company?

<comment><USER3> They are interested in generating controversy, which generates buzz, which generates ad revenue.

<comment><USER4> On this site a large chunk of people think about life and world and people _just_ in an economic way. Maybe _maybe_ , there are others who just live their life following different patterns.

<comment><USER3> I guess in my experience with this site, the vast majority of folks are capable of viewing life and the world around them from many different perspectives, which is why the conversations are so engaging, and why I keep coming back frequently.

</comment></comment></comment></comment><comment><USER5> If you're interested in this sort of thing, may I encourage you to watch a 3 part BBC series "Blood, Sweat and T-Shirts".

It follows 6 young fashionable Brits as they travel to India to spend 3 weeks living and working (almost) as the Indian people who make the clothes they wear. It was made in 2008, but is showing now in Australia on ABC2

<comment><USER2> What is a good way to watch it, if you don't live in the UK? The web site only seems to have a trailer?

<comment><USER5> Sorry, couldn't find a link, but it's on TV here (Aus) at the moment.

</comment></comment></comment><comment><USER6> Do you want to pay your mouse 200 € ?

I submit you do not.

That's why companies like Microsoft work with companies like KYE.

<comment><USER7> Well, there's the crux of it. I have a friend, an Irishman, who is livid that Dell is closing their operation in Ireland and moving it to Poland, putting lots of Irish out of work. But when he wanted a PC, he shopped around for the cheapest one, that's how he ended up with Dell in the first place. It's the same with a lot of people I notice. All about the worker's rights but they buy their clothes at Primark, their food at Lidl, their MP3 players are brands you've never heard of from China, everything as cheap as they can.

If you want to see what people believe, look at how they spend their own money.

<comment><USER8> _"If you want to see what people believe, look at how they spend their own money."_

More along the lines of:

"If you want to see what people are blissfully ignorant of, look at how they spend their money"

<comment><USER7> c/blissfully/wilfully/

</comment></comment></comment></comment><comment><USER9> Companies can automatize build process of chips with nanoscale precision, but somehow mice build+assembly can't be automatized or at least semi-automatized? Same goes for other users of chinese manufacturers like Apple/Foxconn. They consciously knew where they were getting their stuff from because it was cheaper and they knew why it was cheaper.

I don't know the economy behind it - but I can guess that robotization of, at least, most of the process would calculate itself in nicely over time. Come on, it's 2010 - Robots, where art thou?</comment>


Ask HN: What do you do about reading if you have hard time reading as web dev? - <AUTHOR> You have dyslexia, literacy issues, visual impairment, etc. What do you do about it? <comment><USER2> Look into supplement that enhance Alpha waves (Acetyl L-Carnitine, Piracetam, N-Acetyl Semax + N-Acetyl Selank, etc), listen to brainwave entrainment audio sessions (Neuro-Programmer 3), search for "dyslexia treatments" online (and on Reddit) to see what others have come up with, etc.

</comment><comment><USER3> Make sure you have headphones and some good text to speech tools.

Keep working with specialists in learning disabilities to develop strategies for working around and overcoming your particular set of challenges.</comment>


Show HN: Goconf – Dead simple Go configuration file library - Noah-Huppert https://github.com/Noah-Huppert/goconf <comment> Noah-Huppert I made this small little library because the alternatives like Viper were too complex and did not provide a clear simple getting started example.

My goal was to make a library which does 1 thing well. Hopefully this library will help anyone who needs to load configuration files.</comment>


Look Who's Talking About Educational Equity - <AUTHOR> https://quillette.com/2020/08/12/look-whos-talking-about-educational-equity/ <comment><USER2> I wonder if the author went to private school for K-12. A quick google search shows that he got his BA from Vanderbilt. Everything about his argument reeks of academic pretension and a bias rooted in a dislike for social justice in education. I don't know if I buy the idea that institutions who are training educators are the problem...it's too one dimensional and doesn't capture the nuances and realities of public schools.</comment>


Ask HN: Can you use Amazon gift certificates for AWS? - <AUTHOR>

I was just wondering...in case I get a gift cert. for christmas this year :D <comment><USER2> LMGTFY: [https://www.google.com/search?client=safari&rls=en&q=can+i+u...](https://www.google.com/search?client=safari&rls=en&q=can+i+use+amazon+gift+cards+on+aws&ie=UTF-8&oe=UTF-8)

3rd result, from Quora: "NO"</comment>


Ask HN: When did you take greater control over your life? - <AUTHOR>

Share your story about the time you took greater control over your life. <comment><USER2> One of my own 'AHA' moments was when I discovered that you can't. Yes you can never take control over your life. You need to relax and go with the flow. We all think that we are controlling our lives, but consider this, suppose time is a river and your life can be represented by you sitting on a small boat. You can puddle a bit and control a little bit of where you are (akin for example to choosing a career or job).

However, the flow of the river determines the overall direction! Now, for the interesting part. Now and then other people get on your boat. A girlfriend, a child a wife! As you go along the river, people come and go.

Another interesting part is what I call 'crossroads', they also appear almost randomly. Situations where a 'do this' or 'do that' can influence the journey. You will probably get about four in life. Most of the times when this happens you will not know is happening. For example you went on holiday to Costa Rica and ended up meeting a girl in the Airplane, should you eventually marry this girl it can influence the rest of your life.

Money and success can give you a temporary dejavu of control. You can have a great business, you control your time, your health is tops and out of nowhere your boat hits a rock, or somebody came in the boat and drilled a hole.

This is advice from a 'baby boom hacker' they exist on HN too! :). Married same day as Charles and Diana! So far the score is on my side! Had a lot of bumpy rides, married three times, 5 kids, a number of businesses, went bust twice, lived through two wars (Cyprus and Gulf), lived through the bad years in South Africa, worked in five countries.

Control? We can't even determine the outcome of a program with certainty!

<comment><AUTHOR> Thanks for the feedback. I understand where you're coming from.

If I could simplify your view: Stuff happens, deal with it. I don't mean to ignore the nuance. But to generalize it further: the universe is stuff happening that you can't control. You can _think_ you're controlling it but you aren't.

That's a metaphysical view I can't rule out. But I was thinking about control in the sense that, I have more control over a car once a learned stick shift. Or humans have greater control of over the Earth resources than since our hunter-gatherers days; we know how take sand to silicon.

But I never meant to suggest (with the original car metaphor either) that I'm completely in control of life. I did mean, and I do think, that I can get _greater_ control over life. In the same way I can chop down I tree and turn it into a table. I just need the skill. Though, it's hard to learn.

<comment><USER2> Yes it is a bit of a metaphysical view and by no means I am advocating a fatalistic view, where one should just sit and wait for things to happen. You can 'control' your immediate environment to an extend but not much more. To 'control your life' in the real sense it is impossible. Life is mostly a 'random walk' a bit like the stock market going in a certain direction but to find sense out of the noise is very hard. Are you here to-day on HN because your DNA makes you? Are you interested in CS because you had a teacher (parent etc) that influenced you (even subtly) at school? What would have happened if he was not there? So what is my suggestion? Just drive your boat, enjoy the trip and don't let it rust in port!

'Or humans have greater control of over the Earth resources than since our hunter-gatherers days; we know how to take sand to silicon'... this is for another post :), just watch the videos on the Taiwan typhoons! We can only control a very tiny part of that fractal, but maybe you right as people we may be able to give it a bit of colour! Just in case you wondering I call myself an Agnostic Atheist Christian Buddhist!

</comment></comment></comment><comment><USER3> Five years ago. Incidentally, almost exactly five years before "Ask HN random crap" started getting old.

<comment><USER4> Incidentally HN is less than 3 years old, meaning that it was never not old :-P.

<comment><USER5> uh, no.

</comment></comment></comment><comment><USER6> Ask HN: Have you stopped beating your wife yet? ;-)

FWIW, I do drive a stick shift, and I've always driven a stick shift (so, close to 10 years now...) I learned to drive on one. I think my stall-count was 5 on the first day and 37 on the second day, and I only got the car moving about twice on the second day...

<comment><AUTHOR> EDIT: The original thread title was: When did you switch to stick shift?

The title was metaphorical. I meant to get people to share their coming of age stories; when they decided to take more control over their life: switch from automatic to stick shift, so to speak.

Well, when did you?

<comment><USER6> Even metaphorically, I'm not sure the question makes a whole lot of sense. My whole life has been a series of progressively more independent and progressively more important choices. I chose to go to a charter school instead of my public high school when I was 14. I chose not to go to college when I got out at 19, instead working at a tech startup. A year later, I reversed my choice. I chose to study abroad for a semester in New Zealand my junior year. I chose to switch my major from physics to CS in my last semester. I chose not to immediately get a job upon graduation, instead finishing up a volunteer project I'd been working on all through college. I chose to work in a financial software startup for a couple years, and then chose to quit and found my own company when the opportunity came up with a friend. I chose to abandon it when it became apparent we weren't going to succeed, and get a job at a big company.

There's no single point where I "switched on" and said "Okay, now I'm going to take charge of my life." That's the stuff of movies - most of the time when people try it in real life, they go right back to the routines they had before.

<comment><AUTHOR> Thanks for the feedback. I should've been more specific. But I felt asking a specific question while easier to understand, excludes people by definition. Like if I asked, why did you decided to startup? That excludes employees. So I asked "take greater control". But I never said "get complete control".

I meant share a story about a time when you were prompted to become more independent; like why you became an entrepreneur, consultant or employee for instance. Or if you left home, why? What happened? Why the choice of more independence? Frustration? Drive to do good? Forced by circumstances?

Maybe I'll ask again but differently next time. I didn't think asking for the full story was trivial.

</comment></comment></comment></comment><comment><USER7> When my kid was born. That was a pretty good wake up call. I was 28 at the time and up to that point just taking whatever gig that I could get. Lots of money one part of the year, dirt poor another, it didn't really matter.

The extra responsibility helped me to focus quite nicely.

</comment><comment><USER8> I'm alive twenty five but I've still got no control.

</comment><comment><USER9> I painted myself into a black hole of frustration. The situation was bound to fall apart, and it did about a year ago. Unfortunately, not at my initiative.

I am hoping that this is the time when I take control. Better late than never.

(In some ways, I've always taken control. In never being satisfied with the -- usually inefficient and boring and mistake-laden -- status quo. However, trying to introduce changes from the bottom up, or even just follow your own road, has limited effectiveness in some organizations.

That's just the way they are. You could say it was my fault for not getting out sooner, but then I was dealing with a lot of stuff on multiple fronts.)</comment>


You Don't Need Blockchain - <AUTHOR> https://www.offerzen.com/blog/heres-why-you-dont-need-blockchain <comment><USER2> >The crypto-sphere is interesting because it has grabbed everyone's attention despite being so poorly understood.

Could agree more. A distributed ledger of anonymous transactions does not meet the needs of a well regulated financial system (anti-money laundering, fraud repayments, etc..). I'm sure some of these things can be layered on top of the Blockchain (i.e. Libra) but why tear down everything we have and build upon a system that designed to vague and untraceable and horribly inefficient</comment>


Mexico's Most Sorrowful Spirit Became a Cultural Phenomenon - <AUTHOR> https://www.atlasobscura.com/articles/the-weeping-woman-in-mexico <comment><USER2> > Off the screen, La Llorona is a well-known and pervasive legend who serves > as a cautionary tale for multiple generations in Latinx households

Latinx households, really? There is a minuscule but vocal movement who pushes to change the Spanish language so gendered words that end with 'a' or 'o' end instead with 'x'. For example, 'niño' and 'niña' would become 'niñx', and would be pronounced 'niñe'. I would just let this slide if the article was in Spanish, but ffs, the English word 'Latin' doesn't have a gender and shouldn't be written as 'Latinx'.

You make the world more inclusive by being more inclusive with others, not by speaking politically correct.</comment>


Show HN: An opinionated approach to have type safety in native JavaScript - <AUTHOR> http://www.rafelsberger.at/blog/2015/04/an-opinionated-approach-to-have-type-safety-in-native-javascript/

<comment><USER2> Interesting take on runtime type checking. Have you seen tcomb ([http://gcanti.github.io/tcomb/](http://gcanti.github.io/tcomb/))? I've been using it myself and I like this idea of using it for domain driven design. Check it out if you haven't, maybe you can find something to differentiate as avocado is still in its early stages.

</comment><comment><USER3> I agree with the reasoning behind this and the approach seems nice. As it happens, I've also written a library in this space (shameless plug):

[https://github.com/philbooth/check- types.js](https://github.com/philbooth/check-types.js)</comment>


Show HN: Gmail Thread Splitting - <AUTHOR> https://www.streak.com/split-gmail-conversation-threads-break

<comment><USER2> I have never understood gmails features. They consitantly do not work for me, maybe the way i use email is too old school or something.

Threads are consistently broken, the messages are in the wrong order and are almost never grouped logically with seemingly random emails getting placed in same thread.

The new tab system seems to only be right about 30% of the time, I constantly have extreamly important emails going to the promotions and updates tab even after I have seemingly said all further emails of this type should go to primary.

I'm genuinely curious do these features work well for others? Am I alone in finding that these things just don't work?

<comment><USER3> Works near-perfect for me. The tabs work, the threads work. Happy gmailer right here! :)

I do everything from gmail, it's pretty much my life. I receive about 50 mails a day. I have important logs and reports from my servers come into gmail. I hardly ever have to re-organize.

Now, how do we find out the difference between how you and I use gmail?

</comment></comment><comment><USER4> For someone who runs into this issue quite often; this is a great add-on. You'd be surprised how many issues and embarrassing emails have arisen due to Gmail incorrectly threading emails. Great job Streak :)

p.s. Is it just me, but I can't fathom why the Gmail team doesn't scoop these guys up ASAP :)

<comment><USER5> Yes, it's so obvious it's hard to see why Google hasn't implemented it already. I wonder what will happen with this project, unlikely that Google would acquire something rather than assign a small team to fix this within a week?

</comment></comment><comment><USER6> Bravo. Ignore the haters who haven't run into thread issues and thus believe no one else can possibly run into these issues. "Just change the subject!" Clearly these people have never sent a bulk email, or considered the fact that others may use email differently from them. Really tired of this mentality of "I don't have complete information but I think I'm great thus I'm just going to assume you are clueless".

<comment><USER7> Apparently I am one of those haters. I guess "Is this really a thing people have trouble with?" is not a valid question, but I really don't understand when/where/why this thread-splitter feature would be useful, let alone useful enough to enough people to warrant extended time on the HN front page.

Can somebody (who won't presuppose my motivation or my opinion) answer my question?

<comment><AUTHOR> Our users have asked for this for 2 use cases:

1) You send an email bcc'ing a bunch of people. Those people start responding, Gmail threads it into 1 thread. Sometimes you want this and sometimes you don't

2) The topic of a thread changes and you want to split the thread into two logical threads.

<comment><USER4> alooPotato: Is there way to split it off to another thread while keeping the same subject?

<comment><AUTHOR> Currently we have this limitation where the new thread has to have a different subject but we're working on removing that restriction. Should be a few more days before thats ready.

</comment></comment></comment></comment></comment><comment><AUTHOR> We started playing with the new Gmail API ([https://developers.google.com/gmail/api/](https://developers.google.com/gmail/api/)) and created this fun little feature.

Ever notice Gmail sometimes gmail threads things it shouldn't? (i.e. when you bcc a lot of people). We built a way for you to break of a subset of the messages into their own thread.

</comment><comment><USER8> It's been renamed from "Thread Breaker", hasn't it? Because it's still called that in the screenshot of the 'More' menu, alongside "Split the Thread".

(Thread Splitter's a much better name :)

</comment><comment><USER9> I don't see anything in your privacy policy about email handling, but maybe I'm just skimming too fast.

When I click on install on the extension, it says that it requires access to *.mailfoogae.appspot.com; do you route my mail through your app in order to add this feature to gmail, or is it done completely locally within the sandbox of my account/browser?

<comment><AUTHOR> In order to make this particular feature work, we hit the Gmail API with your Oauth credentials (which you can revoke at anytime). The actual process of splitting up the thread requires us to download some of your inbox content, modify it, then upload it. We do not store a copy of your mail ever.

</comment></comment><comment><USER10> I have very rarely run into misbehaved threading. Even so, for me personally it's not enough to warrant the effort of doing a manual split. Threads are ephemeral and it's fairly trivial to change the subject and fork the thread.

However, I would really love to see thread merging or some kind of grouping into a master thread which can track all the sub-threads. I am fairly certain this would have very high utility for a lot of other people here especially if its dead simple to do (group > search thread > select thread > ok). I could easily use this to merge a bug report email thread with the internal bug tracking thread, etc. Or a candidate's job application with his interview feedback, etc.

<comment><AUTHOR> This is essentially our main product - Streak.com.

We have a blog post on how we use Streak for bug tracking internally: [http://blog.streak.com/2013/07/how-streak-uses-streak-to- tra...](http://blog.streak.com/2013/07/how-streak-uses-streak-to-track- bugs.html)

<comment><USER10> You'll have to excuse me for not having noticed this, it's the first I have heard of Streak. This is indeed quite impressive and comprehensive.

</comment></comment></comment><comment><USER11> I haven't tried it yet, but oh my I've been waiting ages for you guys to add this feature! I'm sure it wasn't easy.

For context, the merging of emails threads in Gmail would at times make Streak unusable. This should be a big boost to Streak usability!

I also notice you guys added a "reminder" feature - effectively putting you head-on with Boomerang (which I currently pay for).

I have Streak, YesWare and Boomerang installed in my Gmail. It's too many add- ons for my liking. Boomerang, although the only one I currently pay for, seems to be the one that will have to go.

</comment><comment><USER12> Nice.

I'm personally waiting for a tool to group and split mails in a mailing-group- compatibile tree view. Or at least something respecting the usual headers mailing groups use to mark which mails belong to which thread (so that when you change the topic the e-mail doesn't go to a different thread in GMail).

Not that I care about it for 99% of my conversations, but one Hackerspace I subscribe to seems to be stuck in 1980s Unix Land, and get terribly annoyed at people using GMail for mailing ;).

</comment><comment><USER13> Another amazing feature from the Streak team. My whole company is thoroughly addicted.

</comment><comment><USER14> AWESOME feature.

</comment><comment><USER7> My favorite thread splitting plugin is "editing the subject field when replying". Is this really a thing that people have trouble with?</comment>


Microsoft Wants to Buy Social News Reader Prismatic - <AUTHOR> http://techcrunch.com/2015/03/01/prismatic-2/

<comment><USER2> Good for the founders. BTW, although I used to really enjoy Prismatic, I stopped using it because I did not like the Twitter permissions it required on signup (ability to post using your account, etc.). I have been systematically reviewing mobile apps and web apps I use and dropping those that require privileges that don't seem reasonable to me.

</comment><comment><USER3> Prismatic has some very talented people and they have poured an incredible amount of work into their product. Good for them.

This is another success story for Clojure (Prismatic has released some nice Clojure libs)</comment>


BBEdit 13.0 Release Notes - <AUTHOR> https://www.barebones.com/support/bbedit/notes-13.0.html <comment><USER2> My daily workhorse editor since 1992. Thanks, Rich, for 27 great years.</comment>


Stargate Physics 101: A comedy about the importance of software testing - <AUTHOR> https://archiveofourown.org/works/3673335 <comment><USER2> _We eventually tracked this down to some of the stabilisation algorithms – they work fine within the intended time frames, but there 's an inefficiency in them that gets worse the longer the wormhole is open and eventually they can't keep up with the shifts._

An allusion to the accumulated rounding bug that caused the Patriot missile incident in 1991? That system was likewise intended to operate for only short periods.

From [https://autarkaw.wordpress.com/2008/06/02/round-off- errors-a...](https://autarkaw.wordpress.com/2008/06/02/round-off-errors-and- the-patriot-missile/)

In the Patriot missile, time was saved in a fixed point register that had a length of 24 bits. Since the internal clock of the system is measured every one-tenth of a second, 1/10 expressed in a 24 bit fixed point register is 0.0001100110011001100110011 (the exact value is 209715/2097152).

On the day of the mishap, the battery on the Patriot missile was left on for 100 consecutive hours, hence causing an inaccuracy of 9.5E-8x10x60x60x100=0.34 seconds.

The shift calculated in the range gate due to the error of 0.342 seconds was calculated as 687m. The shift of larger than 137m resulted in the Scud not being targeted and hence killing 28 Americans in the barracks of Saudi Arabia.

<comment><USER3> Nice comparison!

But, more likely, it's just a play on a reason a Stargate (when unaided by some hostile energy source or time dilation field) just shuts off after 38 minutes with no explanation.

<comment><USER4> I agree that it is a play on that...

However in Stargate the actual 38 minutes thing isn't a computer limitation it is a physical one. In the original show (i.e. SG1) Carter said it is "impossible" due to physics for it to stay open longer, and they also established that with enough raw power (e.g. blackhole, ancient device, etc) it could be kept open longer or near indefinitely, it just wasn't possible with the normal power the Stargate needs to operate.

Keep in mind they did exceed 38 minutes many times with different explanations each time[0] none of which were computer related. It is also worth noting that earth in the show built their own DHD which presumably would have different computer limitations than the ancient built DHD (and as they established when all DHDs in the universe died except theirs due to the malware).

[0] [http://stargate.wikia.com/wiki/Stargate#Exceptions](http://stargate.wikia.com/wiki/Stargate#Exceptions)

<comment><USER5> > Carter said it is "impossible" due to physics for it to stay open longer

At the beginning of the story it's implied that things like that are why Carter would be upset. Because the physics they had thought they had learned from the stargate hardware turned out to be software limitations instead. It's not like humans are ever capable of creating a wormhole to test for themselves.

<comment><USER6> Later on, when Carter gives a lecture on wormhole physics at an Air Force cadet academy, the teenage protagonist of that episode reacts in the same way, and Carter gives an "I thought that way once, but then I saw enough compelling evidence to change my mind about what's possible" speech.

</comment></comment></comment></comment></comment><comment><USER7> That's hilarious. And damn, exploding planets sure do emphasize the sheer random, uh, "creativity" of users. If there's a way, they'll find it. And if there isn't, they'll make it. I wonder if Temit was an ascended being, because surely near-omniscience is the only explanation for his ability to predict some of the crazy things gate users would try.

Anyhow, one minor issue: the gates were thought of by a guy named Amelius the night before they left the Alteran Home Galaxy, long before they ever reached Earth. :)

<comment><USER8> > Temit was an ascended being, because surely near-omniscience is the only > explanation

Any sufficiently advanced QA department is indistinguishable from ascension.

> for his ability

Stop misgendering my werewolves. >:(

> Anyhow, one minor issue: the gates were thought of by a guy named Amelius > the night before they left the Alteran Home Galaxy, long before they ever > reached Earth. :)

Yeah, I had the timeline slightly wrong for how long it took them to get to earth. In my defence there's no evidence of them making Stargates pre Milky Way (he _came up_ with the idea in the Ori home galaxy but they then spent a few thousand years looking for somewhere to settle), and we don't know how long the Alterrans lived, so it's _just_ about plausible, but it's a bit of a reach.

<comment><USER9> The show Stargate Universe is set on a alterran Galaxy seeding ship. The stargates in that show are definitely an earlier design than the milky way ones, demonstrated by their ability to only dial other gates within a few dozen light years instead of across the galaxy

<comment><USER8> Technically they're an earlier design than the ones found in the Milky Way by the time of the show.

But, um, honestly I hated Universe and didn't watch much of it, so it's not very surprising that this doesn't manage to be compliant with it.

</comment></comment></comment></comment><comment><USER10> I read this and do not consider it to be about "the importance of software testing"... it reads more like a demonstration of the hubris that one can fix a fundamentally flawed system by trying to dream up test cases and then fixing them one by one as if they were in isolation of each other. One can imagine someone writing something like this about a sorting algorithm, slowly discovering corner cases of large numbers, negative numbers, 2s compliment wraparound situations, custom comparators that violate invariants, values that are being mutated while sorting... the more of these issues you find the more you should question that you really have defined the problem correctly, not patting yourself on the back for having good testing.

</comment><comment><USER11> Haven't read the article in full but as a Stargate SG-1 fanatic, I can tell you there were numerous episodes where the plot played upon the number of bugs and design flaws in the reverse-engineered software that powered the gate on Earth. Almost every season Dr Carter would speak of the improvements she had made to the "dialling computer" and other such things. There was an episode called "Time", my favourite, where they inadvertently dialled a planet next to a black hole and the worm hole could not shut down. It was subsequently learnt that this would never have been possible had it not been for design flaws in the dialling computer program.

<comment><USER12> Ah. "Error code 23, extreme gravity present"

It's OK, we don't know what this error means. We'll override it.

</comment></comment><comment><USER13> Each item in the list of untested edge-cases at the end corresponds to an _excellent_ episode.

Hilarious stuff.

<comment> Tuna-Fish What episode does the "2\. Rotating gate" refer to? I don't remember seeing that one...

<comment><USER8> About a third of them aren't from any episode, including that one.

</comment></comment></comment><comment> david-given It's funny because it's true. Every word.

...well, not the bits about planets exploding. I've never written code which did that. Yet. I did once write a program which summoned demons, but never made it work.

<comment><USER14> Do you work in the High Energy Magic building of the Unseen University? Giving Hex[1] the keys to summon demons sounds like a bad idea.

[1] [https://en.wikipedia.org/wiki/Hex_(Discworld)](https://en.wikipedia.org/wiki/Hex_\(Discworld\))

<comment><USER15> Or possibly at the Laundry[0]?

[0]: [https://en.wikipedia.org/wiki/The_Atrocity_Archives](https://en.wikipedia.org/wiki/The_Atrocity_Archives)

</comment></comment></comment><comment><USER16> > The coordinate calculations were pretty hairy, so we just hard-coded each > location into the Stargates for now. It shouldn't be hard to extend properly > later.

Ha! Gold.

<comment><USER17> Yeah, this is the most plausible explanation I've ever heard of the utter insanity of DHD design.

<comment><USER18> Some friends and I were chatting about this last night while watching the heinous MacGuffins and Deus ex Machina moments in the final episode of Atlantis, oddly enough, backpedalling furiously trying to come up with alternative excuses for the coordinate system.

Our best stab was handwaving frantically and uttering something about a Hilbert curve [1][2].

But that still doesn't really explain anything about why the ancients would pull the equivalent of going from IPv6 to _v4_ by dropping the number of coordinates in the Milky Way gates.... or, anything else, really.

[1] [https://en.wikipedia.org/wiki/Space- filling_curve](https://en.wikipedia.org/wiki/Space-filling_curve) [2] a la [https://xkcd.com/195/](https://xkcd.com/195/)

</comment></comment></comment><comment><USER19> Kudos for the language section. We use strange words like "bug" and "error" when we really mean more complex ideas. Where are the words for "flawed by virtue of being untested", "acceptable flaw because the chain of causality is too long in the physical world", "failure to map possible external states (as opposed to internal states)". Language design for spoken technical language should be a 'thing'.

</comment><comment><USER20> Ahh, Stargate.

Never gets old, and I wish we'd find out how Universe ends already.

<comment><USER21> I hope there will be a new series someday.

Looking forward for the reboot triology of the original movie though :)

<comment><USER22> I just hope O'Neill will be spelled with two L's.

<comment><USER13> Indeed.

(Apologies in advance for the shitpost)

</comment></comment></comment></comment><comment><USER17> That kind of explains the attitude of ascendent beings towards our universe: they treat it all as one big pile of legacy code - not to touch under any circumstances!

</comment><comment><USER23> If you think about it the entire show was actually "Testing in Production"

<comment><USER24> That was even mentioned in some of the episodes, I vaguely recall some scientist been horrified because they'd only decoded 97 of the over 200 error codes/status codes.

Probably McKay.

There was a strong technical slant in that show.

EDIT:

> For instance, he knew that when Carter got the Stargate working with three > supercomputers, they ignored 220 of the 400 feedback signals the Stargate > can emit during any given dialing sequence.

[http://stargate.wikia.com/wiki/Meredith_Rodney_McKay](http://stargate.wikia.com/wiki/Meredith_Rodney_McKay)

I miss stargate so much.

<comment><USER25> McKay becomes a really nerd genius on Atlantis... I hope that at least write some books about what happens on Destiny before they got to sleep.

<comment><USER26> I always wondered why he suddenly becomes allergic to citrus.

In one of his first appearances on SG1, he invites Carter to the cafeteria, mentioning that they have lemon chicken, with excitement. If I'm not mistaken, it's not until the SGA fork that he seems to be allergic to citrus.

<comment><USER8> I think you're confusing episodes. In his first appearance he mentions that he's allergic to citrus. The "I love lemon chicken" incident comes from Carter's visit to a parallel universe when she's experimenting with the merlin device in a later season.

...I'm really embarrassed that I know that from memory.

</comment></comment></comment></comment></comment><comment><USER27> Oh, Stargate- man i remember that- i got that idea for that city-traint wriggling itself through the gate from one wormhole to another from this. Great Inspiration.

[https://springrts.com/phpbb/download/file.php?id=9695](https://springrts.com/phpbb/download/file.php?id=9695)

Brilliant show. Strong science attitude of anything goes, if you put your mind to it. Also actual c++ code when "Hacking" was done.

</comment><comment><USER28> Half-expected the ending to be "So, anyway, we've been reassigned to this Ascension project, because the brass want that working _yesterday_ and they've decided to add every available engineer to speed it up."

</comment><comment><USER29> I enjoyed the part where the author implied that the asteroid belt was the result of a stargate crashing into Tellus.

</comment><comment><USER30> > We shipped one of them off to the nearest inhabited planet in another star > system – horrible hellhole of a place called Abydos.

That is only going to be correct if you disregard the whole "Universe" series. Otherwise, the chronology is wrong. And can't be just about the movie, otherwise there are no "ancients".

All in all, an excellent read. This was almost certainly written by a software engineer, or two.

</comment><comment><USER31> Wow, never thought I'd ever see an AoO link on HN. I usually mentally associate that domain with slash fic.

<comment><USER32> It used to be on someone's blog. It got moved to AoOO for posterity, I guess.

</comment></comment><comment><USER33> He forgot the part about how you get cold when you go through the stargate...

I need to watch this show again.

<comment><USER26> They resolve this in the show. Carter mentions the rough ride and cold was due to their home-grown dialing program and instabilities in the wormhole. Further tweaks result in the improved travel comfort.

</comment></comment><comment> guard-of-terra The kind of people who design things like this tend to think about such issues defensively, from the start.

Much of our fundamental problems with software seciruty stems from the fact that people don't think of software as of something particularily dangerous.

<comment><USER8> One would certainly hope this to be the case, but it's hard to watch SG1 and get the impression that the ancients had much of a concept of defensive design.

<comment><USER34> On the contrary, my view is that the Ancients knew defensive design is a /hard/ problem, for higher level languages. The lower level parts (like the actual Stargates) had smart companion interfaces where the safety features exist. The fact that you can over-ride a safety, presumably if you know what it exists for and have a use case for ignoring it, is a good engineering feature for a device that might be used in more extreme circumstances. (Such as evacuating from an Ori invasion.)</comment></comment></comment>


How to save multiple numbers into one byte - <AUTHOR> http://www.allinsight.de/how-to-save-multiple-numbers-into-one-byte-int/ <comment><USER2> I'm a bit shocked to see this on the front page. That's possibly a comment on me as much as anything.

But this is just really simple bit packing. Is this something many engineers (maybe frontend/JS) don't know about now?

<comment><USER3> I guess we never do bit shifting in higher level languages at all. E.g. in Python, there is no point to do this, there is only one type of int (64 bit) and it still creates a new int object, which is not very fast.

<comment><USER4> You certainly can do this in python and often will need to when dealing with binary formats. The struct library is a great help for this. ([https://docs.python.org/2/library/struct.html](https://docs.python.org/2/library/struct.html))

<comment><USER3> > This module performs conversions between Python values and C structs > represented as Python strings.

You can, but probably it's even slower than doing regular bit shifting operations (<<, >>, &, |, ^, ~)

<comment><USER5> Where you can, you should use struct. For example, decoding a 32-bit integer in struct is faster than doing it manually:

      In [11]: %timeit hex(b[0] | (b[1] << 8) | (b[2] << 16) | (b[3] << 24))
      The slowest run took 6.89 times longer than the fastest. This could mean that an intermediate result is being cached.
      1000000 loops, best of 3: 715 ns per loop

      In [12]: %timeit hex(struct.unpack('<I', b)[0])
      The slowest run took 8.77 times longer than the fastest. This could mean that an intermediate result is being cached.
      1000000 loops, best of 3: 587 ns per loop

The resulting code is also going to be cleaner, especially if you end up needing to unpack multiple members at once, which is common when parsing a binary file.

The struct module is, I believe, written in C. (Pedantically, the struct module isn't; the C module _struct is hidden/private, but if you open the source for struct.py, you'll see it referenced.)

</comment></comment></comment></comment></comment><comment><USER6> This is simple binary packing approach to storing small-type data in wider types. Delta compression is a natural next step from here. I once published a blog post on that topic: [http://bitcharmer.blogspot.co.uk/2013/12/how-to- serialise-ar...](http://bitcharmer.blogspot.co.uk/2013/12/how-to-serialise- array-of-doubles-with.html)

Mind you, the reference implementation from the blog is really poor in terms of code quality and performance because I was not allowed to open-source what I actually developed for my client, but the ideas still stand. Delta compression can make a huge difference in some cases; because cpu cycles are cheap and bitwise operations are very fast it has the potential to bring serious benefits in some scenarios, ie. lower latency of transmitting (numeric) data over networks.

</comment><comment><USER7> Since 3⁵ (243) is less than 256, you can actually fit 5 "trinary" numbers into a byte.

The code gets more complex and slow, but not really by much.

<comment><USER8> Just table lookup.

The word is "ternary", by the way.

A useful intermediate representation would be "binary coded ternary" (BCT?) whereby 121 gets represented as 011001. The code 11 isn't used, just like 1010 through 1111 are not used in binary-coded decimal (BCD).

The 10 bit BCT representation of a five digit ternary number could index into an array of 1024 entries, each yielding its numeric value, that fits into a byte.

Likewise, a reverse table of 243 values can map to a 10 bit BCT.

Conversions between BCT and textual digits are trivial.

<comment><USER7> I don't understand this fully, but it seems complicated.

The way I have in mind is the same way you'd separate out the digits in a decimal number, except you'd use 3 instead of 10 in the math.

<comment><USER8> Suppose you have the ternary number "12102". We can encode each digit in two bits: 01,10,01,00,10 very easily. That gives us 0110010010, a ten bit number: a "binary coded ternary" whose value, in decimal, is 402. Now, we use that as an index into a table bct_value[402]. That table just looks up the numeric value of this code as an integer value in the range 0 to 242, which is also an 8 bit binary number: 146. That's our encoding of the five ternary digits in one byte. With the precomputed table, we just look up. We can also have a reverse table that takes 146 to 402, which we can break into groups of two bits and map to digits.

<comment><USER7> OK. That's the super optimized way to read the encoded data. Makes sense if performance is important.

Writing is the harder part. I suppose you could just make 15 such tables, one for each possible value you could write.

</comment></comment></comment></comment></comment><comment><USER9> Alternatively, just store it naively and gzip. Or use a sparse array. Or, best, don't do anything until storage size is an actual, measured bottleneck

</comment><comment><USER10> See also bitfields, which make the compiler generate the shifting and masking code for you:

[https://en.wikipedia.org/wiki/Bit_field](https://en.wikipedia.org/wiki/Bit_field)

<comment><USER11> Erlang also has pretty nice syntax for handling bits & bytes.

[http://erlang.org/doc/programming_examples/bit_syntax.html](http://erlang.org/doc/programming_examples/bit_syntax.html)

<comment><USER10> Whoa, that's some practical syntax. Suddenly I'm filled with envy that none of the C-family languages have that.

</comment></comment></comment><comment><USER3> Either I don't understand something or the Resulting bytes after c should be '11100100' and after d should be '11100110'.

<comment><USER12> I agree with you. His Resulting Byte is incorrect in both partC and partD. Otherwise it was a good article.

</comment></comment><comment><USER13> And if you need to store values that are not powers of 2, you can use the following kind of encoding/decoding. In python3:

        from math import *

        def number_to_trits(n):
            # little-endian encoding
            # (i.e. smallest precision trit first)
            trits = []
            v = n
            for i in range(1,9):
                v, d = divmod(v, 3**i)
                trits.append(d)
            return trits

        print(number_to_trits(8))
        # [2, 2, 0, 0, 0, 0, 0, 0]

        def trits_to_number(trits):
            n = 0
            for i, trit in enumerate(trits):
                n += trit*3**i
            return n

        print(trits_to_number([2,2]))
        # 8

</comment><comment><USER14> I also really liked this page on bitpacking, which also offers a technique to implement them using only math operations.

[http://number-none.com/product/Packing%20Integers/index.html](http://number- none.com/product/Packing%20Integers/index.html)

I recently implemented a bitpacker in a game VM that didn't have bitwise operations, and the math-based version worked perfectly.

</comment><comment><USER15> Isn't it possible to use SQLite or something for something like this? That would greatly simplify most of these tasks, making your dataset reachable through a fast declarative language.

As many games of Go as 100000 is, it's not so big you couldn't use a regular, untuned database.

<comment><USER16> Bit fields certainly have lots of uses that can't be replaced by databases.

<comment><USER15> I'm not questioning the usefulness of bitfields. But in this case, it would be silly not to use a database instead.

The method that the article describes (storing two-dimensional arrays of "empty", "black", and "white" in 2 bits) is also not the best one to use to store Go games. Chopping up games and creating deltas from them, or storing the moves rather than the sheets themselves and recreating those sheets on the fly, are best for data compression.

Bitfields are great, but not for this.

</comment></comment></comment><comment><USER17> This method is only optimal if your numbers have ranges that fit nicely into powers of two. See here for optimal bit packing: [https://codeplea.com/optimal-bit-packing](https://codeplea.com/optimal-bit- packing)

</comment><comment><USER18> Neat :)

I wrote some very slow code to set/get the Nth bit in an array of bytes.

I was planning on using it for Chess bitboards - [https://en.wikipedia.org/wiki/Bitboard](https://en.wikipedia.org/wiki/Bitboard), but it could be used for say 8 booleans to the byte.

unsigned char getbit(unsigned char * bits,unsigned long n){

        return (bits[n/8] & (unsigned char)pow(2,n%8)) >> n%8;

}

void setbit(unsigned char * bits,unsigned long n, unsigned char val){

        bits[n/8] = (bits[n/8] & ~(unsigned char)pow(2,n%8)) | ((unsigned char)pow(2,n%8) * val);

    }

<comment><USER10> You should be able to replace the pow(2, x) part with (1 << x), so e.g. getbit() becomes

        return (bits[n/8] & (1 << (n%8))) >> n%8;

The compiler will turn the division and modulo into shifting and masking.

<comment><USER18> Yeah that's a good point! Don't know why I didn't think about that.

</comment></comment></comment><comment><USER19> Great read, but you should really move the motivation up to the top before you start the explanation.

</comment><comment><USER20> "a lot" is two words.

<comment><USER21> I don't know what alot of numbers looks like, but this is what a regular alot looks like:

[http://hyperboleandahalf.wikia.com/wiki/Alot](http://hyperboleandahalf.wikia.com/wiki/Alot)</comment></comment>