Support for U2F security keys - <AUTHOR> https://blog.1password.com/introducing-support-for-u2f-security-keys/ <comment><USER2> I have long debated getting a Yubikey but have held off because I don't want to have to carry around several dongles at all times to be able to send an email.

Surely other people are in the situation of:

\- iPhone, iPad

\- Macbook with only USB-C ports

\- Windows/Linux workstation with only USB-A ports

Is there currently a non-cumbersome solution that will work on all of these?

<comment><USER3> A bluetooth capable U2F device like the Titan.

<comment><USER4> [https://solokeys.com](https://solokeys.com) are an option as well if you like open hardware. [https://github.com/solokeys/solo](https://github.com/solokeys/solo)

I think the NFC ones are shipping after they worked out some kinks.

<comment><USER5> Just got mine last week (Solo, Solo Tap, and the DigiPass SecureClick). All work great for their respective uses.

</comment></comment></comment></comment><comment><USER6> Got a free YubiKey from Wired. Then I read that mobile is a pain, and that I really need two ... it's sat in my bag now for months, unused. I already use 2FA, and it works good enough - I'm not sold on how this will make my life better, especially on mobile.

<comment><USER7> The primary purpose of U2F/WebAuthn is to break phishing attacks. Code-based TOTP 2FA, the kind you're probably using now, is already adequate to the task of making sure you're not credential-stuffed.

<comment><USER8> Autofill of a password manager is a working countermeasure against phishing too: If autofill does not work there is something wrong and you should look closer...

<comment><USER9> Until the next exploit that can steal passwords from autofill...

<comment><USER10> I have to click a button in the extension to get it to fill. Not quite Auto fill, but pretty close.

</comment></comment></comment></comment></comment><comment><USER11> I switched recently to Bitwarden. 1Passwords pricing/subscription changes was the the push I needed.

Bitwarden has been fantastic, I highly recommend it.

[https://bitwarden.com](https://bitwarden.com)

<comment><USER12> $2.99 per month is too steep of a price to pay for your personal security? Really? I'd dump my Spotify/Apple Music/Netflix/whatever in a heartbeat, if I had to choose between paid subscriptions in my life

<comment><USER13> I'm selling water bottles at $100/gallon. Considering water is literally essential to life, how many can I get you?

<comment><USER12> You must have had better ones than _that_, when you were thinking up a witty reply. I hope that wasn't you bringing your best.

<comment><USER13> What immodesty; to presume oneself to be worthy of the best.

<comment><USER12> Rather: why half-ass anything in life?

</comment></comment></comment></comment></comment></comment><comment><USER14> So this works with 1Password the website? I've been using 1Password for years, and never go to the website? How does this make my life better? Can I unlock 1Password the desktop app with the U2F key?

<comment><USER15> Our implementation of 2FA only happens when adding your account to a new device. Subsequent unlocks do not require any sort of 2FA except for certain conditions.

Our apps do not currently support U2F/WebAuthn when signing in, so they'll default back to TOTP based until we implement support for U2F. We aren't making any promises as to when this will arrive but at least two of our apps now have some form of support for it internally. It's far from complete and not ready for users but it is being worked on.

Note that U2F in this case is only about authentication, not decryption of data. This is why it's only used on initial setup of your account on a new device. The cryptography side for unlocking 1Password is entirely independent of U2F/MFA.

Hope that helps but let me know if you have any questions.

Kyle

1Password

</comment></comment><comment><USER16> If this is the sort of thing where I can just tell 1password this device is OK with 2 factor, once per device, I could use this. If this the sort of thing that whenever I wanted to lookup a password, forget it. Even if I had to two it once a week or once a month I wouldn't bother. Maybe if I were paranoid about being a target I would, but I'm not.

<comment><USER17> Literally the first paragraph:

> Last year we added two-factor authentication to provide another layer of > protection for your 1Password account. When this is enabled, you are > prompted to enter your second factor any time you sign in from a new device.

If you read it carefully, you'll have the answer to your question.

</comment></comment><comment><USER18> Switched from lastpass to gnu pass and storing my private subkeys on a couple of yubikeys has actually worked really well for me (took a few days to get my head around gpg and smartcard setup, but that was worth doing even if I didn't setup pass). That said, I am running android, windows/WSL and ubuntu; other platforms may not be so painless.

</comment><comment><USER19> It appears via the screenshot that you can have multiple 2FA devices, which is great. I love my Yubikey in theory, but in practice I'm only using it for services where I can have a TOTP or SMS 2FA backup method, because I'm not convinced it will always work or be available. Even if having SMS 2FA enabled negates any security benefits of the Yubikey.

Thus far it's just Dropbox and Gitlab that I use it for, since they're among the few services that allow multiple 2FA methods to be used at the same time.

<comment><USER20> All the services that I have used with U2F support have supported multiple keys. Google, Gitlab, Github, and some others which I forget.

They have all worked with Yubico U2F keys and with the Google Titan keys. Pretty convenient way to have two factor authentication. I like the Yubikey 5 Nano as you can leave it plugged into a port in your laptop all the time.

<comment><USER21> > All the services that I have used with U2F support have supported multiple > keys. Google, Gitlab, Github, and some others which I forget.

I've run into a number of services that only allow a single U2F key (it's been a while, so I don't remember the exact ones). Even if they do support multiple U2F keys, how do you handle enrolling both? I keep my backup key offsite, so ideally I could enroll it without physically possessing the device. If I have both in my possession at all times (or even sometimes), I'm at risk of losing both of them.

</comment></comment></comment><comment><USER22> I'm seeing several links to different physical keys in the comments. Is there somewhere/someone that verifies these keys? Like a 3rd party testing/standards body?

I've always had it drilled into me that doing crypto yourself is fraught with peril. It seems that doing hardware would be doubly dangerous. I'd want more verification that the implementation is correct and "strong".

<comment><USER23> I think verification that the implementation is correct is easy enough, which implies that it is "strong", because these devices simply implement a spec.

What you might want to look at is things like hardware hardening or side channels. (Whether or not you consider this a matter of "correctness" can be argued, but here I would consider correct = implements correct algorithm.)

I think attacks against U2F devices are fairly difficult because you can't really use them as any kind of oracle, just due to the way the user interface works. But I am not a crypto expert, I just know how U2F works.

</comment></comment><comment><USER24> I've had an OnlyKey for a couple of years now. It offer 12 slots per profile and 2 profiles, and the slots supporting TOTP, U2F, Yubikey, plaintext and a whole lot of other tools. With the configuration app and firmware open source, I'm really surprised I've never seen anyone else with one.

<comment><USER25> [https://onlykey.io/](https://onlykey.io/) I assume you mean this - it looks great!

</comment></comment><comment><USER26> This looks awesome and I wish I could use it. Could you guys also consider bringing back the completely-offline mode that doesn't make my password manager depend on a 3rd party service? I'm prohibited by company policy from using my favorite password manager because of this.

<comment><USER15> You mean the licensed version? It never went away.

When you launch you'll be prompted to purchase. On the screens near the bottom there is a line of text about purchasing a license, go that route instead of signing up for the 1Password.com service.

Kyle

1Password

<comment><USER26> Interesting. If this is the case, I think you have a communications problem. I was under the impression that after 6.0, the only way to get a license was to have your older one grandfathered. I can't find any information about this on your website. __All __of the options on your product info pages other than "enterprise (email us for a quote)" show monthly subscriptions only.

Where can I see product info about the licensed version? Can you provide a link?

<comment><USER27> This was so difficult to figure out, and responses from AgileBits support so disingenuous, that I wrote up how to do this on iOS: [https://www.davidschlachter.com/misc/1password-ios- standalon...](https://www.davidschlachter.com/misc/1password-ios-standalone)

<comment><USER26> "Disingenuous" is really how it feels to me too, unfortunately. Doesn't give me a warm fuzzy feeling about them being committed to supporting it. But I have to admit, the clear answer from Kyle in this very thread gives me hope. I will definitely be upgrading my 6.0 license to 8.0. AgileBits: please keep the licensed version forever, and try not to make it seem like such a step-child. :D

</comment></comment></comment></comment></comment><comment><USER28> Just thought I'd mention that _PayPal_ recently quietly added TOTP second factor support. It's not u2f but it's better than SMS. Perhaps u2f is in the cards now since they're listening to customer feedback on the issue.

</comment><comment><USER29> For those of you who have the desktop version - it looks like U2F support is a work in progress:

"So while it works great as your second factor in those browsers, for now you'll still need an authenticator app set up to use with the 1Password desktop and mobile apps (and any unsupported browsers)"

</comment><comment><USER30> only tangential, but I've wanted to carry my Yubikey on my keyring, but have always been nervous about making it unreadable by sullying the contacts. Should I be concerned about this? Where do you all carry them?

<comment><USER31> That's how I've been carrying mine for the last four years and it's fine. I think you're more likely to lose it than to damage it. I have a second one that's identical (registered to the same services) that I keep in a safe place.

</comment></comment><comment><USER32> I'm using this with a contactless smartcard, works just fine on Android [https://github.com/tsenger/CCU2F](https://github.com/tsenger/CCU2F)

</comment><comment><USER33> Hasn't this been in Bitwarden and Lastpass for a while now?

<comment><USER34> Bitwarden might (not sure), but Lastpass only supports Yubikey's TOTP implementation, not U2F, and they seem to have no real motivation to support U2F based on their support ticket responses

</comment></comment><comment><USER35> So this only applies to 1password.com not the desktop app synced through cloudprovider/filetransfer/whatnot?

</comment><comment><USER36> Can you use Yubikey NFC to do 2FA on most/all iOS apps? That's my biggest barrier to getting a Yubikey.

</comment><comment><USER37> Finally! Been waiting for this

</comment><comment><USER38> I have never used 1password.com.

Adding 2FA to it is great but I think the best security is likely still just to sync and use local apps for this data, to avoid being exposed to any JavaScript vulnerabilities or if 1password.com were ever hacked.

<comment><USER39> 1Password has Wifi Sync option too.

<comment><USER38> Yep! 1password is great. I do use their cloud sync service with all the apps, I just don't ever use the website or the browser extensions to limit my exposure.

<comment><USER15> Note that the browser extension actually does limit your exposure quite a bit. You make trade offs here.

For instance, if you aren't using the browser extensions, how are you getting your password to the browser to sign in? Copying and pasting? It's possible for any app on your system to read the clipboard.

Drag and drop should be a better alternative there, as we now support that in 1Password 7.

The extension though uses either Safari App Extension (for Safari, obviously) or Native Messaging Host (Firefox and Chrome browsers) and aren't susceptible to clipboard type snooping.

The browser extensions also only present items that match the website you're on. This helps a lot in phishing attempts.

So, yea, you could not use the browser extensions but you're going to have to trust that YOU always do the right thing.

Note again that 1Password does not "auto fill" like other password managers, where simply visiting the site fills the data in. You always have to explicitly ask 1Password to fill into the page.

Just some insight anyway.

Kyle

1Password

<comment><USER38> Thanks Kyle. I do appreciate your response! And the tip about drag-and-drop, I may make use of that :)</comment></comment></comment></comment></comment>


In Defense of Bad Ideas - <AUTHOR> https://surjan.substack.com/p/12-in-defense-of-bad-ideas <comment><USER2> Bad ideas are magical when they come in the form of long call options (bounded cost, unlimited upside).

Bad ideas destroy companies/nations/civilizations when they come in the form of short put options (bounded benefit, unlimited downside).

<comment><USER3> > long call options

> short put options

I tried to think of what a "short call option" or a "long put option" would be, and I don't think the concepts exist. Do they?

<comment><USER4> If you buy a call option, someone has to be on the other side of the contract. They are the short position.

<comment><USER3> Sure, but they don't have a call option. All they have is an obligation.

<comment><USER5> Short just means they have a negative amount of that, in this case a negative amount of options.

<comment><USER3> No, short means your position is predicated on the idea that the future price of something will be lower than the current price.

Owning a positive number of put options makes you short the underlying stock, which is presumably why the comment upthread described put options as "short put options". Selling put options gives you a long position, not a short position.

I wanted to analyze this in terms of the original comment, in which "short put options" have limited upside and unlimited downside, but put options do not have unlimited downside for either the buyer or the seller. The buyer can't lose more than the price of the option, and the seller can't lose more than the strike price. Put options are a case of "limited upside, limited downside" from any perspective.

Call options are the ones with unlimited upside (for the buyer) and unlimited downside (for the seller).

<comment><USER6> You have to think of the futures definition of long and short, not the equities definition of short. In every futures contract, the long is paying money and the short is receiving it for the contract. Every options contract has a short, the writer, and a long, the buyer.

<comment><USER7> In this case, the original commenter explicitly (erroneously) stated "options". Options cannot have unlimited downside; you can always choose not to exercise the option.

<comment><USER6> If you write a put option, it has downside limited only by the underlying zero price of the security minus the premium. If you write a call option, your downside is theoretically unlimited. If you buy a put or a call your downside is limited to the option premium.

edit: i should say writing a put option (short put) your potential downside is the difference between the strike price of the option and the premium, if the underlying security goes to zero.

<comment><USER3> If the underlying security goes below zero, that doesn't increase your potential downside. The option you sold will force you to buy the security, but it won't force you to sell it.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> The article assumes that SpaceShipOne's design was actually a good design. It wasn't.

To be precise, it was a good design to satisfy the artificial requirements of the prize, but nothing else. The design doesn't scale. The hybrid rocket motor has been the source of countless headaches, lousy performance, and more than a decade of delays, the feathering mechanism has already cost the life of a test pilot, and the need for a pilot plus co-pilot severely reduces the payload capacity.

Now, the article's thesis – that good and bad ideas are sometimes hard to distinguish before they've been tried – is still true, but the author chose a bad example.

<comment><USER9> It's a great example. The only reason these factors clearly outweigh the positives is because the benefit of hindsight. Or do you imagine the brightest minds of NASA just weren't thinking clearly for decades?

The killer feature of the shuttle was supposed to be the cost savings from being able to make multiple trips using the same equipment (sound familiar?). It turned out not to be a feasible solution from a technical cost standpoint, but if people can't clearly estimate the cost of adding a new feature to a website, I totally believe would have been indistinguishable from a good idea at the time.

<comment><USER10> _> It turned out not to be a feasible solution from a technical cost standpoint_

The shuttle turned out not to be a feasible solution because it was loaded down with other requirements that had nothing to do with the killer feature you describe. A key one was the requirement from the US Air Force that the shuttle have enough cross range capability to be launched into a polar orbit and then return to the launch site after one orbit [1]. That was what drove costs up to the point where the shuttle wasn't feasible any more in terms of the killer feature.

[1] [https://history.nasa.gov/sts1/pages/scota.html](https://history.nasa.gov/sts1/pages/scota.html)

</comment></comment></comment><comment><USER11> I'd forgotten about that stuff, but it seems like _the_ bad idea was the prize itself.

I mean, the winner did just enough to get the prize, and stopped, and nothing further ever came of it, right?

Just because of how SpaceX overshadows everything in retrospect.

<comment><AUTHOR> No, they're still going at it. Partnered with Virgin to make further iterations of the Spaceship One concept under The Spaceship Company. They're shooting to do private spaceflights under the name Virgin Galactic.

<comment><USER11> Orbit though?

<comment><USER12> Sub-orbital. Think getting the craft up to altitude of 100+ km, hang out for a few minutes, and descend back to Earth. SpaceShip One, the prototype inspiring the Virgin Galactic fleet, was capable of reaching speeds of 0.9 km/s. [1] Escape velocity to get into orbit from Earth's surface is 11.2 km/s. [2]

[1] [https://en.wikipedia.org/wiki/SpaceShipOne](https://en.wikipedia.org/wiki/SpaceShipOne)

[2] [https://en.wikipedia.org/wiki/Escape_velocity](https://en.wikipedia.org/wiki/Escape_velocity)

<comment><USER13> Escape velocity is a velocity to escape gravitational field. Orbit velocity is a velocity to orbit a thing inside its gravitational field.

For example, ISS has orbit speed of about 7.7 km/s at 350 km above the ground. Orbit speed "at the surface" of Earth is about 8km/s, I believe.

<comment><USER12> Thanks for the clarification.

<comment><USER11> My point, not to be overly disrespectful, is that suborbital flights are so different from what SpaceX has done that they seem like party tricks, not stepping stones.

In the big picture, it's not a huge amount wasted, I was just remarking on it being less significant in retrospect than one might have hoped.

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> Reminds me of the strange aircraft designs of Burt Rutan, the closest thing the 80-90's had to an Elon Musk: [https://www.flyingmag.com/photo- gallery/photos/awesome-airpl...](https://www.flyingmag.com/photo- gallery/photos/awesome-airplanes-burt-rutan/)

<comment><USER15> If you're referring to SpaceShipOne, that's because Rutan's company built it: [https://en.wikipedia.org/wiki/SpaceShipOne](https://en.wikipedia.org/wiki/SpaceShipOne)

</comment></comment><comment><USER16> I've always wondered how terrible automotive designs make it to production. Engines or transmissions that quickly fail, cars that are just garbage in quality...nobody along the years of development stopped to say "maybe we shouldn't build this..", or perhaps someone did?

Did the company building this get their moneys worth when it comes to lessons learned?

<comment><USER17> When it comes to mechanical failures like that, it's never due to incompetence; it's due to cost cutting concerns (e.g. using less material, cheaper / inferior material) and / or planned obsolescence. Automotive engineers are highly competent, and most will have had a lot of education when it comes to materials, strengths, stresses, wear and tear, etcetera. And car companies build up that knowledge over the decades.

Definitely not a problem with all cars, but the better built cars are more expensive.

I want to say "you get what you pay for", but at the same time I know nowadays there are a lot of products (e.g. electrical tools) where the brand is bought up and the new owner makes big cost cutting measures to maximize their profit margin, the money coming from "brand inertia", that is, a brand is known for being good for a couple years after they start reducing quality and durability.

<comment><USER18> Nissan's CVT seem to be a counterexample of everything in your first paragraph.

I could also bring a lot of counterexamples to your second paragrah (e.g.: many german cars).

<comment><USER19> "better built cars are more expensive" is not the same as "expensive cars are better built".

</comment></comment></comment></comment><comment><USER20> Would have loved to read a longer writeup exploring that premise (which I agree with). The difference between Bad ideas and Good ideas, is determined by the outcome. Something that seems bad but ends up working (milliondollarhomepage, pet-rock, etc.) ends up being brilliant, in the eyes of the public.

<comment><USER21> I would argue that Walter Benjamin's "History is written by the winners" (rephrased) is someone a longer write-up of the same premise, albeit not in a technical area.

</comment></comment><comment><USER22> That rotary rocket looks like a design that could work, tbh

Instead of rocket propulsion, first stages could in theory deploy helicopter wings and get a decent slow down even before spinning them up to full speed (with autorotation the blades act like a parachute/wings).

Seems like it would be a harder implementation than the high precision control with a rocket, though.

</comment><comment><USER23> Some ideas are just bad Example: Should I take this unmarked pill a stranger just handed me? No you probably shouldn't Example 2: Should I jump off this cliff with rocks at the bottom? Is it higher than 2-5 feet. Probably not.

Example 3: Should I eat this raw meat product that has been sitting out on my counter for more than a day. Probably no. I understand the point here, but some ideas are just bad ideas.

</comment><comment><USER24> Does anybody know, why the rotary rocket body did not start spinning itself once it lifted off the ground? There's only one propeller.</comment>


The Tragedy of the New Coronavirus Spikes - <AUTHOR> https://www.newyorker.com/magazine/2020/07/06/the-tragedy-of-the-new-coronavirus-spikes <comment><USER2> RealClearPolitics links to TFA, and just below links to this story as well, "Where are all the deaths?"

[https://spectator.us/where-deaths-coronavirus- wave/](https://spectator.us/where-deaths-coronavirus-wave/)

They both overstate their respective cases.

I also would toss this one into the mix, "CDC Now Estimates there have been over 20 million Coronavirus cases";

[https://outkick.com/cdc-now-estimates-there-have-been- over-2...](https://outkick.com/cdc-now-estimates-there-have-been- over-20-million-coronavirus-cases/)

The upshot is that this article could more accurately be titled _Where's the tragedy of the new Coronavirus spikes?_. Strip away the politics and we have a huge uptick in case with in some cases the lowest death rates we've seen since March.

The media keeps loosing the message in the midst of the disaster porn and political finger-pointing. We need the economy to be open to the absolute greatest extent possible while hospital capacity remains near maximum. The long-term damage (in dollars and lives) in over-regulating the extent of the lockdown could exceed the damage of COVID by an order magnitude.

Needless to say, the more cases we can shift forward into summer and fall, versus having to deal with those cases in winter concomitant with flu season is how we prevent the disaster of a second wave. Hospitals in winter can be overwhelmed by a bad flu season alone, and COVID didn't go exponential until after flu had tapered this year. We do not want to see them peaking together in vulnerable populations.

This drumbeat of articles like this just totally miss the point entirely, as if we could just keep handing out trillions of dollars a month indefinitely until a half billion doses of vaccine are ready to go, and as if even if we did that we're not inadvertently causing hundreds of thousands of death due to side effects of quarantine and things like canceled elective procedures.

A lot has changed over the last couple months, but the media is still "reporting" like it's February.

<comment><USER3> Is it possible that Covid has already evolved to be less virulent?

Many a new (to humans) disease has started off unusually deadly, only to evolve to keep its host alive longer to help further its spread.

<comment><USER2> I doubt it? I think changing demographics of the people being infected is a more obvious culprit. Hundreds of thousands of young adults (20-30) went out in very large groups over the last month. At the same time we've finally gotten better at actually protecting the elderly. This is the result;

[https://img.apmcdn.org/6b486323329122674a9b5fbfdcf2e51f9ad83...](https://img.apmcdn.org/6b486323329122674a9b5fbfdcf2e51f9ad83f8d/uncropped/8e1875-20200626-covid- cases-age.png)

The younger demographic happens to be vanishingly unlikely to die, and highly unlikely to even be hospitalized, from a case of SARS-CoV-2.</comment></comment></comment>


Practical scaling techniques for web sites - <AUTHOR> https://medium.com/@derwiki/practical-scaling-techniques-for-web-sites-554a38dbd492#.8i5i73u3z <comment><USER2> I operate a website that receives moderate-to-occasionally-enormous traffic, but it only brings in a modest income. Consequently I have taken measures over the years to squeeze the most out of our cheap little server (4GB RAM):

1) I cache HTML aggressively on a RAMDisk.

2) Everyone (apart from the visitor who triggers the page generation) gets the cached page.

3) Portions of the page that are user-specific--hence skipping the cache--load dynamically when they are almost scrolled into view (sorry JS haters). Same with many images.

4) Requests are piped through CloudFlare for some free CMS.

5) Remaining static files are served via a low-cost CMS.

This setup has managed to remain online with as many as 6,800 concurrent visitors, though at that point it started barfing '500 Internal Server Error's into some viewports.

I avoid stale cache files by just dumping the entire cache any time any content-altering event occurs (e.g., new comment, edited post, etc). Since it's on a RAMDisk, it's quick to delete and rebuild. When our traffic is huge, it's always concentrated on one or two pages, so the important part of the cache is quickly restored.

<comment><USER3> What do you mean by CMS? Maybe a CDN, or am I mistaken?

<comment><USER2> Oh, whoopsie doodle. I did mean CDN. Sorry about that.</comment></comment></comment>


Abusing hash kernels for wildly unprincipled machine learning - <AUTHOR> http://jeremydhoon.github.com/2013/03/19/abusing-hash-kernels-for-wildly-unprincipled-machine-learning/ <comment><USER2> Essentially random projection, which has good theoretical justification and comes in handy quite often, for instance, in SVMs [1]. I'd be concerned about using something as naive as CRC32 though, ideally they'd be using a cryptographic-strength hash fn.

There's also an entire area of research around "semantic hashing" and local embedding, that starts with such a random projection, and tries to improve the mapping to be better at a certain task while still remaining low-dimensional.

[1] <http://arxiv.org/pdf/1211.6085v2.pdf>

<comment><USER3> I think this is actually quite different from random projection, which creates linear combinations of several continuous variables. RP is much more similar to PCA in that respect. OP's method works on a single categorical variable and essentially shuffles the possible values, then 'folds' them down to an arbitrary size (the size of your hashing function).

>ideally they'd be using a cryptographic-strength hash fn

Cryptographic adds nothing; there is no danger from someone reversing the hash (and if there is you may have bigger problems). Any hash function that is suitably random in its redistribution of the variables should suffice.

<comment><USER2> It's a sparse random matrix, but definitely in the same family of techniques.

I was concerned more about distributional properties of different hash functions than reversibility. Checksums-as-hashes often work, but it's not that much work to swap in something a little more robust. If you want a good output distribution, pick something where that's a goal of the algorithm.

</comment></comment></comment><comment><USER3> This is pretty similar to the approach that many predictive modelers already use: compress data down into 'dummy variables', where each variable represents some attribute. For example, you could convert the variable state into 51 dummy variables, one for california, one for DC, etc. Hashing makes the programming a little easier and helps avoid throwing out data in the long-tail of the distribution when the number of possible values is very high. It's common in NLP exactly because of this.

Hashing comes with costs though. It's much harder to interpret a model with hashed variables because the variables lose meaning. Also, some information may get thrown out; as OP mentions integers may have continuous meaning and categorizing them can really damage the model. If you have a very large dataset it probably won't matter because most modeling methods asymptotically converge (i.e. as you have more and more data the model can learn that 333 and 334 are similar just by seeing enough examples that this is the case), but if you don't you could be throwing out valuable information. In a case like this I suppose the modeler could manually go in and convert his integers to floats to 'tell' the algorithm the data is continuous.

<comment><AUTHOR> Your point regarding lost meaning is quite salient. It can be useful and enlightening when a learner reports a measure feature importance (such as in random forest models, [http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home....](http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#varimp)).

When you say "hashing makes the programming a little easier," I think you hit the nail on the head. I'm not trying to improve classification accuracy -- my goal was just to make it as easy as possible to learn on arbitrary structured data.

<comment><USER3> >my goal was just to make it as easy as possible to learn on arbitrary structured data

I'd be very careful about throwing arbitrary data at your learner, at least if you don't understand your data well. Oftentimes the predictors and response are not properly separated in the same way they will be during real-world usage (for example, in time); this leads to target leaks, where your model is effectively cheating by using data it won't have in production.

Target leaks are obvious when the classifier performs suspiciously well on in- sample test data, but sometimes the repercussions are more subtle but still very damaging in a production environment.

<comment><USER4> Hm, couldn't a hybrid approach deal with this? Eg hash all the data except a few dimensions you think are vital, and add those to the resulting hashed array?

<comment><USER5> Or location-aware hashing?

</comment></comment></comment></comment></comment><comment><USER6> This is very similar to the hashing trick used in vowpal wabbit (<http://hunch.net/~vw/>). I haven't used this python module, but vw is absolutely incredible to use.

<comment><AUTHOR> Great point! VW was actually an inspiration for this blog post. I don't believe it handles structured data quite the same way as hashkernel, but it does a great job of accepting both categorical and continuous features. We use VW models to classify all sorts of badness on my team at Facebook ([http://www.newscientist.com/article/dn21095-inside- facebooks...](http://www.newscientist.com/article/dn21095-inside-facebooks- massive-cybersecurity-system.html)).

</comment></comment><comment><USER7> I love this entry. Two months ago, I wrote a perhaps more-pedantically- technical entry about the hash trick and its use in building a simple classifier, even making it play nice with Scipy's sparse linear algebra (eventually writing my own CSR times CSC multiplication method in Cython): [http://blog.newsle.com/2013/02/01/text-classification-and- fe...](http://blog.newsle.com/2013/02/01/text-classification-and-feature- hashing-sparse-matrix-vector-multiplication-in-cython/)

</comment><comment><USER8> Nice writeup, jhoon!

In case anyone else was searching his/her eyes trying to figure out where the "likes" field is actually sanitized out from the the training data set, it's the item.pop("likes") inside the function "is_liked": [https://github.com/jeremydhoon/hashkernel/blob/master/fblear...](https://github.com/jeremydhoon/hashkernel/blob/master/fblearn.py#L53)

... Maybe I like long names too much, but my vote would be something longer like "check_if_liked_and_remove_likes" since "is_liked" sounds like a predicate that won't be modifying its input.

</comment><comment><USER9> Meh. Diving a little bit more into the results on how well the hash kernels algorithm did with UCI Adult Names data set is mildly disappointing. Take a look at the results ([http://archive.ics.uci.edu/ml/machine-learning- databases/adu...](http://archive.ics.uci.edu/ml/machine-learning- databases/adult/adult.names)) and you'll see that hash kernels rank 14 of 17.

To be fair, I would definitely like to see how this algorithm does with other data sets as well.

All I see this algorithm doing is basically a projection of a highly dimension feature set onto a random n-d projection via a hashing function. Another words, it's not clear to me how an optimal classification boundary can be constructed using this random projection. I feel comfortable with understanding the performance characteristics of techniques such as SVM or vector quantization since they both focus on implementing algorithms that optimally reduce the dimensionality of the feature space. However, random projects are, in my humble opinion, overrated.

Also, there are two additional sources of parameterization that might make it difficult to use this tool: the selection of a hash function (and accounting for the information loss via a random projection) along with the arbitrary hash kernel array size. That makes it somewhat difficult to train and validate a model.

However! I really do like the ideas presented here and look forward into diving into this and other variants. I really appreciate the OP for taking the time to piece together this cool tool and compare it's performance with other algorithms.

</comment><comment><USER10> Anyone interested in the article should give VW a try. You can get good baseline performance in a fraction of time compared to other traditional approaches like liblinear et. al.

<https://github.com/JohnLangford/vowpal_wabbit/wiki>

</comment><comment><USER11> You are one of my favorite online authors merely for this post. Do you write often?

I'm going to explore using this to fill parts of a vector in an otherwise normal regression algorithm and see how it does against only linear regression and only hash kernel on my dataset.

</comment><comment><USER12> For someone (me) not particularly knowledgeable on the subject. I'm curious to understand the contrast between this and vowpal wabbit. Could someone offer an insight into when it would be useful to use this project as opposed to vowpal?

<comment><AUTHOR> First, please don't use hashkernel in production. It's a toy designed to showcase how hash kernels can help us (easily) learn on structured data.

VW is faster, smarter, and more featureful. However, it doesn't know how to learn on structured data.

</comment></comment><comment><USER13> I don't know anything about this space, but I have some real-world datasets and love throwing algorithms I don't understand at them. Right now I'm writing classifiers by hand, it'd be awesome to outsource my job to some AI. Does anyone have a good starting point where I can figure out which modern classification algorithms are relevant or even possible given my dataset and computer power available?

<comment><USER3> This is a pretty decent flowchart to finding appropriate machine learning algorithms for your problem: <http://i.imgur.com/iVo4AAw.png>

</comment></comment><comment><USER14> I don't understand why this is "wildly unprincipled." Doesn't it have all the guarantees that Weinberger and Shi et al papers describe?

<comment><AUTHOR> This is a great question. The "unprincipled" part comes in the application of hashkernel. For example, any integer-valued fields encountered in structured data are treated as categorical features (whereas floating-point numbers are treated as continuous features). Of course, it's possible that some of these integer-valued fields should be treated as continuous features. When using techniques in ways they were never intended to be used, like treating continuous features as categorical, many of the assumptions that make something "principled" no longer apply.

<comment><USER14> eh, it's just binning :)

</comment></comment></comment><comment><USER15> Writes one blog post - front page of HN</comment>


A ToC of the 20-part linker essay (2008) - <AUTHOR> http://lwn.net/Articles/276782/ <comment><USER3> As someone who has worked far more with dynamic linking in Windows the ELF system has always seemed needlessly complex; the extra indirections of the GOT/PLT mechanism and PIC are avoided by simply linking DLLs with different load addresses (so they don't always need to be relocated, but can be if necessary), and the only thing that needs to be associated with a symbol is an address. Makes it especially convenient when writing things like in-memory executable compression and shared data between processes. The ability to "import X from Y" easily is also nice.

Anyone with experience from the other side (working with ELFs and then moving into Windows) want to share their views?

<comment><USER4> I don't have much experience with the Windows model, but if I'm understanding you correctly . . .

ELF does support load-time relocation, as appears to be done under Windows if two base addresses conflict. That just isn't the standard way of doing things, for several reasons.

First, load-time relocation imposes a cost at startup (whereas PIC imposes a smaller cost throughout the lifetime of the process). I'm given to understand that, back in the day, the startup delay for programs using large shared libraries could be quite noticeable.

Second, load-time relocation reduces in-memory text section sharing in the case where relocation is performed. If your goal in using shared libraries is saving RAM, this is a problem.

Third, it's a matter of inertia. Originally, UNIX shared libraries (a.out format) were built with a static, non-relocatable base address. Library authors had to coordinate via a central authority to ensure compatibility. PIC seemed like a good way to get as far away from that problem as possible - <USER2> so I understand.

Indirection through the GOT/PLT also serves another purpose, even in load-time relocation code - it enables replacing a symbol in one shared library with a symbol from another (eg via LD_PRELOAD). Though that's more of a side benefit than a justification.

<comment><USER5> Keep in mind that you don't need text relocations for code sequences that are naturally PC-relative, like jumps on x86, or pretty much anything on ARM and amd64. Windows DLLs don't need text relocations for references to other modules: the IAT (Import Address Table) does pretty much the same thing as the GOT and in pretty much the same way.

<comment><USER6> Um, ARM by default is very not PIC; loads have only a 4k offset so references to the data section are stored as absolute addresses in the text section, unless you explicitly specify PIC which adds another 'add rN, pc' instruction to each reference. (and actually I believe Windows doesn't even support these PIC references in ARM code)

</comment></comment></comment></comment><comment><USER7> See also: [http://www.iecc.com/linkers/](http://www.iecc.com/linkers/)

</comment><comment><USER8> If anybody prefers a .mobi: [http://www.filedropper.com/linkers- ianlancetaylor](http://www.filedropper.com/linkers-ianlancetaylor)

(If anyone has a suggestion for a good place to upload that, let me know. I don't know whether the site I found on Google is sketchy.)</comment>


Some REST best practices - <AUTHOR> https://bourgeois.me/rest/ <comment><USER2> Most of this has little to do with REST at all, e.g. URLs don't actually matter in REST (because of HATEOAS) and should be considered opaque. This is more HTTP API best practises (best practise in the author's opinion). REST has become a meaningless term.

<comment><USER3> I must disagree. REST is defined by four interface constraints: [http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch...](http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm#sec_5_1_5) and identification of resources is the first principle. This functionality is implemented by URLs in HTTP and is therefore critical. HATEOAS (the 4th principle) also seems to be gravely misunderstood. It only means that the API should have a version that is fully useable in a web browser.

<comment><USER3> My statement that HATEOAS is misunderstood is corroborated over and over by the replies to my comment. HATEOAS being misunderstood is so weird because it is an acronym of its own definition so I have no idea where all these other definitions are coming from. Let's examine it closer. HATEOAS stands for:

Hypermedia as the engine of application state

 _Hypermedia_ means HTML, period. Putting a list of URLs in a text or JSON response does not magically make it hypermedia.

 _Engine of application state_ means that all representations of the resources must be possible.

Combine those two and it means that all functionality must be accessible to and from the text/html content type. It must be able to handle all supported HTTP verbs and no fancy request headers that are not supported by HTML forms or hyperlinks.

<comment><USER4> "Hypermedia means HTML, period."

Um, no. XHTML and SVG are hypermedia.

JSON isn't hypermedia, but you can define hypermedia formats that use JSON (just as SVG uses XML).

<comment><USER3> True, but with so much confusion over a blatantly clear definition I figured I needed to keep it simple.

</comment></comment></comment></comment></comment><comment><USER5> As an experienced developer who as had to implement dozens of API's over my career, therre are two things that I care about:

1\. Documentation - If I have to spend the next month doing trial+error to get your API working, thats no good. The best APIs have a page that describes how get up and running quickly.

2\. Good error messages - When something goes wrong I want to know what went wrong. The error message being in a consistent format isn't that important, as long as there is a description of what happened. "Oops, an error occurred, try again later" is not a good error message.

Most of the stuff in this article is superficial stuff that I don't think matters.

<comment><USER6> Better documentation is a non-trivial example application that, one, exercises all "normal" uses of the API, and, two, shows what the API developers consider to be best practices, all the way through.

Let me into your minds. Show me how you think I think. If you don't know how the people consuming your API think, your API is going to be pretty bogus. If your way of thinking is inimical to me, fine and dandy. Just give me a reasonable way to find out which isn't reading hundreds of pages of Javadoc or similar.

And if the example contains copy-and-pastable code, or reusable functions, so much the better. Make the licenses align, even if the code which implements the API is fully proprietary, and things will work out just fine.

Oh, and if you can't keep your example code working, that says something I need to know, too. Something nasty.

</comment></comment><comment><USER7> > For a clean and conscice structure, you should always use nouns.

Well, yeah, it's what everyone says, but not so easy to do in practice. Especially in complex apps there's a lot of stuff that isn't actually creating, updating or deleting an object, it is semantically an action. Say, action "send email" (and not some _random_ email, but related or even defined by some business object you are referring to). Or in bookkeeping apps you often need something like "recalculate", which you probably wouldn't like to think of as "updating" because it isn't "take these values and apply to that object", but rather "hey, it's time to make some decision: please run some process (probably, with side effects) and tell us what the result is". Or it might be an action like "moveToQueue<Name>". Or better yet, you explicitly are telling your system to make some interaction with third-party service which is essentially imperative (like GDS, or some external trading API or whatever that isn't an _object_ for your system, but really more like a service for which your app is just a friendly gateway). It is a little bit hard to think of good example right away, because if there is action "purchaseTicket" and there's no object "ticket" in your system you might ask _WHY there 's no object ticket in your system_, but in all specific enough systems there always are situations like this, and mostly for some good reason. And yeah, there's also decisions made for performance sake or simplicity of client-app (like some frontend-side "action queue" so that you can use reactive programming on front-end in more natural way).

So what do I do then? All URLs like "/user/123/order[s]"(PUT,GET,...), "/product/321/comment[s]"(PUT,GET,...) and, suddenly "/sale/copyToAnotherService"? Now _that_ kind of inconsistency is something I _really_ don't like.

…And while I was writing the last one I though about: how should look url for something like "getRecommendedProducts" which depends on both user and product?

Now writing nouns/action names is a bit messy and so not-RESTful, but you _always_ are able to write exactly what you mean by that request.

<comment><USER8> You know that the verb mechanism is extensible, right?

If your API could benefit from a particular verb (SEARCH, MOVE, CHECKOUT), just use it. You don't need (more) permission from (another) RFC - RFC 2616 and 7231 [1] already gave you permission to add verbs.

Most libraries and tools support arbitrary verbs including `XMLHttpRequest` and `curl`. Yes, there are a few libraries and frameworks that don't easily support arbitrary verbs, but the ones that have added support for PATCH generally have opened up.

Look, there are two Internets. One for people and one for machines. Our RESTful APIs are intended for consumption by machines, not people. You don't have to wedge everything into GET and POST and abuse query parameters to convey what you _really_ mean.

[1] [http://tools.ietf.org/html/rfc7231#section-4.1](http://tools.ietf.org/html/rfc7231#section-4.1)

<comment><USER7> Is it really safe and reliable? Are there any situations when I should NOT add my custom verb whenever I please?

<comment><USER9> That sounds like a recipe for heartburn later, when you run into some WAF or whatever that only allows PUT/GET/DELETE/POST.

<comment><USER8> The cockroach that you see is usually not the only one around.

A framework that doesn't support extensible verbs in 2014 is bound to have more problems than just a failure to implement one RFC correctly.

<comment><USER10> He's not talking about a framework, he's talking about web application firewalls that only support the default verbs. It's quite common and has nothing to do with which framework you use.

<comment><USER8> Thanks for clearing that up, yes, as I've said, you should not use custom verbs if you have to interop with firewalls or proxy caches that perform DPI unless you've tested them.

<comment><USER11> Problem is that you may test the system now and it all works fine. But the day after you deploy, somebody named Murphy adds or updates a proxy somewhere in your topology that breaks things. Or your organization gets a new customer or partner with a differently configured network.

<comment><USER8> That's an argument for never doing anything.

<comment><USER12> Its an argument for being conservative in your design philosophy.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER3> I'd say maybe half of these are actual good practices. In my experience, these are not:

I agree that keeping resource names consistent is good, but using plurals is not the only, and probably not the superior, solution.

Versioning is an idea that gets talked about a lot but can easily create more problems than it solves. Small improvements in APIs should be backward compatible and major ones should be new APIs. Whatever you do, putting "v1" in the middle of your URL is the wrong way to do it. (And so is putting "api" in your URL. It's better to keep your URLs clean of ambiguous and redundant information.)

Nesting resources is an anti-pattern that quickly creates a mess. I've learned this the hard way so please don't repeat my mistakes.

The 3rd principle of REST interface constraints is "descriptive" error messages. It is better to be clear and thorough than consistent. Consistency gets you nothing, especially if it is consistently bad.

<comment><USER13> The article gives some motivation for its best practices, you do not. Would you care to provide some?

> using plurals is not the only, and probably not the superior, solution.

Why?

> Whatever you do, putting "v1" in the middle of your URL is the wrong way to > do it.

Why?

> Nesting resources is an anti-pattern that quickly creates a mess.

Why?

Edit: not sure why the downvotes. I'm genuinely interested, but the parent doesn't contain much useful content.

<comment><USER3> I suppose one must first define the qualities between a good and a poor API. I would say that it is the same as any code: The design should be readable, maintainable and modular.

The plurals or not thing isn't a big deal. I really just object to the author picking one and claiming it's better with justification.

Putting "vi" in your URL decreases both readability and modularity. All things being equal, concise URLs are more readable. "v1" are extra characters without significant benefit. It also creates huge maintenance overhead. How many APIs do you think you can simultaneously maintain well? I am happy with one good one myself. Every extra new version you create (as opposed to gracefully improving the existing one) is more technical debt.

Let's take the nesting example: /artists/8/albums. What if you want to access an album by itself? You also need an /albums resource so you've just created two resources when you need only one. The /artists/8 response should include a list of albums that hyperlink to the /albums resources. If you absolutely have to have a clean list (without the artist header information, just filter it with a query string /albums/?artist=8. This accomplishes the same thing in a more predictable and consistent way without magic URL params. The major benefit of this type of design is it makes tracing execution from the interface to a code location much clearer by avoiding spaghetti URL routing.

<comment><USER14> I've actually seen people use versioning in their URLs for _deprecated APIs only_ , with the current version not using a version number.

<comment><USER15> That seems pointless, since you're still shifting the ground under your users' feet.

<comment><USER16> What's the alternative?

<comment><USER17> Some say that if HATEOAS is followed, there should never be a reason for a client to "know" an API. The assumption is that every link in a resource can be characterized precisely (via link relations, profiles, media types, etc.) enough that automated clients know what they can do and how they should do it. In practice, we're not really to that point yet, but also in practice there is often a human around somewhere to help a client navigate a changed API even if it can't figure that out automatically.

<comment><USER16> I think I'd agree that automated clients have a ways to go before that works. But sometimes you don't even want anything automated: sometimes the API just needs to change because e.g. the world around it has changed, requirements have changed. A version string in the API seems like a reasonable way to do that.

<comment><USER17> I think we can agree that there is some level of change at which we should _not_ consider "the world" to have changed. I.e. if previously one could frobnobulate Widgets A, B, and C, and now Widget A has been replaced by Widgets Y and Z, that probably isn't a change to "the world". Only a client ill-conceived enough to maintain its own Widget Catalog would get screwed up by that change. (Don't laugh; this has been done _lots_ of times.) Now suppose that before frobnobulating Widget B, we must first fizzgiggle it. There is a non-empty set of clients that survived the widget update but that will DIAF with the widget-specific-dependency-of-transitive-widget-verb update. So, don't write a Transitive-Widget-Verbs-and-Their-Widget-Specific-Dependencies Catalog into your client either. _Et cetera..._

Eventually we'll all get to a point at which the generalization must stop. REST advocates think that point is much farther away than is commonly believed. In the case of my dumb example, giving the client developer a version number wouldn't really have helped her avoid a rewrite, because Widget A is no longer available and Widget B must be fizzgiggled before it is frobnobulated. You could imagine that she has enough pull to insist on treating her Widgets exactly as she pleases, but that's not the hypothesis here. (After all in that case she could just have her own entry URI.) It's much better to use media types, link relations, etc. to let the client implicitly learn the information it needs to know every time it needs to know it.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> There is a draft RFC for returning errors in APIs based on JSON.

[http://datatracker.ietf.org/doc/draft-ietf-appsawg-http- prob...](http://datatracker.ietf.org/doc/draft-ietf-appsawg-http- problem/?include_text=1)

Their example:

HTTP/1.1 403 Forbidden Content-Type: application/problem+json Content- Language: en

       {
        "type": "http://example.com/probs/out-of-credit",
        "title": "You do not have enough credit.",
        "detail": "Your current balance is 30, but that costs 50.",
        "instance": "http://example.net/account/12345/msgs/abc",
        "balance": 30,
        "accounts": ["http://example.net/account/12345",
                     "http://example.net/account/67890"]
       }

type, title, detail and instance are attributes defined by the spec. balance and accounts are extra attributes added for this error. I suppose that if I want to return validation errors to decorate an input form I'll have to add many custom attributes like those.

</comment><comment><USER19> The use of 401 Unauthorized is an annoying one because of a little thing in the spec: The response MUST include a WWW-Authenticate header field (section 14.47) containing a challenge applicable to the requested resource.

AFAICT, This makes it makes it inapplicable to REST APIs that do not use basic or digest authentication, at least without violating the spec for the 401 response code.

Is there something that I've missed?

<comment><USER20> Why not pass a dummy value to satisfy the spec?

<comment><USER21> Because a dummy value satisfies the syntax but not semantics of the spec.

</comment></comment></comment><comment><USER22> Here are a couple more lists of Best Practices for REST APIs.

[http://blog.mwaysolutions.com/2014/06/05/10-best- practices-f...](http://blog.mwaysolutions.com/2014/06/05/10-best-practices- for-better-restful-api/)

[http://www.vinaysahni.com/best-practices-for-a-pragmatic- res...](http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api)

The Github team does a pretty good job of incorporating modern best practices as well, so I often use their API as a reference: [https://developer.github.com/v3/](https://developer.github.com/v3/)

</comment><comment><USER23> One very bad practice that is quite common is using status codes in application specific ways and _not_ including a standard error payload.

One way to think of it is that status codes are part of the transport, and your own application logic (validation, application error conditions, warnings, etc.) should never use HTTP error codes.

A dumb proxy server should be able to cache any response (including its status code) on a GET or HEAD without needing to know about any application logic.

</comment><comment><USER24> One minor nuance I use is adding a trailing slash to indicate resource vs custom verb.

GET /products/ : will return the list of all products

POST /products/ : will add a product to the collection

GET /products/4/ : will retreive product #4

PATCH/PUT /products/4/ : will update product #4

POST /products/4/someCustomVerb : custom verb

</comment><comment><USER25> There can never be enough articles and books supporting API best practices.

Here are two of my favorite REST resources (pun intended).

Richardson Maturity Model: steps toward the glory of REST

[http://martinfowler.com/articles/richardsonMaturityModel.htm...](http://martinfowler.com/articles/richardsonMaturityModel.html)

RESTful Web Services Cookbook: Solutions for Improving Scalability and Simplicity

[http://www.amazon.com/gp/product/0596801688](http://www.amazon.com/gp/product/0596801688)

And I am looking forward to widespread adoption of the REST problem standard posted here a few weeks ago.

[http://www.ietf.org/id/draft-ietf-appsawg-http- problem-00.tx...](http://www.ietf.org/id/draft-ietf-appsawg-http- problem-00.txt)

</comment><comment><USER14> > _use nested routing for a cleaner design_

I think this is actually quite debatable and I've seen a lot of people argue that nested resources are an anti-pattern. I only ever use nested resources myself when the child resource is existentially dependent on the parent.

<comment><USER26> I think what he's describing, or at least what I do, is when you get all albums belonging to an artist.

        GET /artist/8/albums
            { albums: [89, 93] }

        GET /albums/89
            {
                album: {
                    id: 89,
                    title: 'Blah'
                }
            }

<comment><USER27> Agreed. Nested routing is fine, as long as it doesn't become redundant.

Do:

/artist/8/albums

/albums/9

Don't:

/artist/8/albums/9

Using query filters versus nesting are virtually synonyms. I would say there is a minor semantic distinction in that the first one is a definite relationship, and the second one is a possible relationship. That is, "get the albums for artist 8" versus "get the albums searching for artist 8". The search could include other parameters as well, making it not as definite.

/artist/8/albums

/albums?artist=8

<comment><USER14> Chances are you're (well at least with me) going to do filtering anyway so /artist/8/albums becomes redundant ;)

<comment><USER27> I see what you're saying, but it's not redundant, it's just a synonym. By redundant I meant that there are keys that are not necessary.

/artist/{artist}/albums

/albums?artist={artist}

AlbumsController#index(artist)

</comment></comment></comment></comment></comment><comment><USER28> Stopped reading at "Version your API". Clearly this guy has no idea what he's talking about [0].

[0] [https://twitter.com/fielding/status/376835835670167552](https://twitter.com/fielding/status/376835835670167552)

<comment><USER29> This is is a genuine question: if you don't version your API _somehow_ , then how do your clients handle breaking changes?

<comment><USER28> Version numbers simply represent a way of managing change over time. They certainly aren't the _only_ way, nor are they the most ideal in all situations, but they certainly are the most simple, accessible, and explicit, and they don't require a lot of planning ahead.

By contrast, the reason why designing real REST services is hard is because you actually have to _design_ them. This requires long, hard thinking about the domain of the problem at hand, and as such, doesn't square well with our "agile" methodologies of week-in and week-out iterative hacking.

Much as I typically loathe self-promotion, I did give a talk on this recently that enumerates some design strategies: [https://speakerdeck.com/nateabele/designing-hypermedia- apis](https://speakerdeck.com/nateabele/designing-hypermedia-apis) \-- check out slides 8-20.

</comment></comment></comment><comment><USER30> What status code should one return when the server receives incorrect credentials for an authentication API ?

401 - Unauthorized looks like the correct answer, but if you read the Wikipedia article 401 is specifically intended for HTTP Authentication using usernames and passwords set in the headers.

I recently ran into a situation where a third party library on receiving a 401 proceeded to then ask for HTTP credentials and try to resubmit the request when actually the server indicated that we had just submitted the wrong username/password for our login API (which doesn't even use HTTP Auth). We don't use WWW-Authenticate headers but the third party library seems to be following the standard.

So should we be returning 400 instead ?

<comment><USER21> > So should we be returning 400 instead ?

403 Forbidden (or 404 Not Found if you don't want to leak information about the existence of a resource to unauthorized users) is probably the right one to use when the access is unauthorized in the general sense, but not the specific kind of HTTP Authentication issue that 401 addresses.

<comment><USER3> The spec for 403 states that "Authorization will not help and the request SHOULD NOT be repeated" so I don't think that is appropriate either. It's a good question though. I've always used 401 and haven't run into problems but I can see why it's probably not correct. I am sure that 400 (with a proper response body explaining the authentication requirements) would not be wrong, but I am uncertain if it is best.

<comment><USER21> > The spec for 403 states that "Authorization will not help and the request > SHOULD NOT be repeated" so I don't think that is appropriate either.

I think that in context "authorization" in 403 can only be understood to mean the same thing as is authorization is implied to mean by the use of "Unauthorized" with the specific definition in 401 -- that is, reauthentication via the HTTP authentication methods. Under that view, 403 (and, to avoid leaking information, 404) fits.

400 does not seem to fit: it is not a generic code that fits the whole class of things in the 4xx series. Its definition is specifically "The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications." But the problem that is being identified is _not_ malformed syntax.

<comment><USER30> Yup this is definitely confusing. That's why I asked it here. Hoping that someone knowledgeable might see this .

</comment></comment></comment></comment></comment><comment><USER31>
      > Here are some bad examples :
      > /retreiveClientByOrder?orderId=1

I note the author didn't go on to demonstrate a better way of doing this. What would he recommend, /clients?by_order_id=1 ?

<comment><USER32> Using sub-resources, you get something like this: GET /orders/1/clients

<comment><USER33> So mapping this request onto a SQL database would be something like...

        SELECT * FROM orders WHERE client_id = 1

And for GET /clients/1/orders it seems like in practice it would be the same relation since it would be wasteful to store two tables. Is this roughly correct?

<comment><USER34> No. Think of it as "for order 1 list all clients", so it would be:

        SELECT * FROM clients WHERE client_id IN (SELECT client_id FROM orders WHERE order_id = @order_id)

The GET /clients/1/orders is what would result in your query (for client 1 list all orders).

</comment></comment></comment></comment><comment><USER35> is it just me or no one can agree on what the best practices for a REST API are?

<comment><USER36> No wonder.

REST manipulates resources but not every software service maps to resources, or (very) painfully so. This is easy to overlook when you have not had to implement a substantial service that is not a simple storage interface in this fashion. It might be tempting to write an article not trying to solve or even address this but it wrongly makes things appear more simple than they are.

REST often tries to map a graph of resources on a tree (the URL space). This problem is more related to the web and naming in general yet there are many possible solutions and no clear rule to apply, leaving plenty of room to blind alleys and time lost pondering which option is the least ugly. I'm not blaming REST here but this still is a major problem seldom addressed in articles.

REST, HATEOAS and friends lean a lot on the theory side seducing you into thinking machines will use your API without human intervention and make you lose time building for a non-existant use case (and your boss doesn't want to make automated access to his data _that_ easy by the way). I suspect they will succeed at the exact same time the semantic web succeeds.

REST tries to convince you that everything will map cleanly to the HTTP standard, and it's pretty close, but most of times it falls short in a way or another.

In pratice most blog posts will tell you to use nouns, to rely on status codes, etc, yet every other popular API will derail in a way or another when faced with the cases that don't fit.

To sum it up, theory is cute, reality is less. Having a clean scheme where everything could theoretically fit is very seducing to any software engineer. In a way it feels like the holy grail, and it could be, but maybe it's not. And if it's not, trying to make everything fit in the wrong model could just be an awful waste of time. I still like the fact that it forces my brain to think in a non-intuitive way about some problems, but well. Our profession is plagued with micro-cults, holy grail models, silver bullets, magic blueprints we try to cast everything we can get our hands on in. MVC comes to mind. I think REST is becoming my second favorite.

</comment></comment><comment><USER37> Well, if you want to never write anything that scales yes, those are in fact the correct practices for writing bone-headed APIs for applications that would blow up with 20k concurrent users doing simple GETs.

If for some unknown reason you want to write APIs that don't go down faster than a 15 year old girl faints upon stumbling onto a boy from 1D in a local McDonalds you should:

a) use verbs

b) separate heavy end point from light end points

c) make URL easy to parse by load balancers and proxies

</comment><comment><USER38> I found "Build APIs you won't hate" to be an excellent resource. Probably somewhat of a beginners text, but I found it to be extremely clear and concise, and very readable for the amount of information it covered. It was also very pragmatic about when to adhere to REST dogma and when to deviate from it.

[https://leanpub.com/build-apis-you-wont-hate](https://leanpub.com/build-apis- you-wont-hate)

</comment><comment><USER39> What is the best practice regarding filtering? Suppose you'd like to filter the artist's albums to only show albums from a range of years (2005-2014).

<comment><USER14> Query parameters on the collection. Dealing with ranges is always a tricky one though and I've seen different REST libraries use different querying.

</comment></comment><comment><USER31> > One way is to do is to pass the API version in the URL (/api/v1/...)

If you do it like this, how do you then deal with 1.0.1? Especially in a framework like rails, where paths basically map to a folder structure?

I agree it's much nicer to be able to test manually when you have the version in the path but it significantly reduces flexibility later on.

<comment><USER40> What we did for this case was to "not break the API during a minor version". For instance, only return more data and not less. Do not change the conditions/meaning of errors.

If you need to, then it's the right moment to switch to v2.

<comment><USER15> Exactly. All minor/revision API changes should be backwards compatible. You can add new optional parameters and new methods/objects, but you can't change the semantics of anything existing.

If you need a breaking change, that's time to increment the major number. Besides that, who'd want to support that many versions running live?

</comment></comment></comment><comment><USER41> Post is missing advice on what to do to notify the client that the request is completely serviceable but should be retried later (i.e. Twitter's 420, or the actual RFC standard 429 [[http://tools.ietf.org/html/rfc6585]](http://tools.ietf.org/html/rfc6585\])).

This is key for modern services.

<comment><USER42> > Twitter's 420

...really Twitter?

<comment><USER41> "Enhance your calm." ;) [https://dev.twitter.com/overview/api/response- codes](https://dev.twitter.com/overview/api/response-codes)

Was superseded by the RFC standard 429 in API version 1.1.

</comment></comment></comment><comment><USER43> I personally think that POST should hit an existing resource and create a new, related one. While PUT should hit an existing resource and modify it (if permissions allow). GET and DELETE are pretty obvious. Those are the conventions I use when I create REST interfaces.

</comment><comment><USER44> This is a much (much :-)) better post about "REST"/"HTTP" API best practices:

[http://www.vinaysahni.com/best-practices-for-a-pragmatic- res...](http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api)

</comment><comment><USER45> If you're coming from a Rails background, then you know, most of this stuff is actually done (and also documented) for you by the framework itself, if you use its scaffolding feature. One more reason why I love Rails and the people behind it.

</comment><comment><USER46> This is a great article, I wrote a similar blog post recently [1]

[1] [http://scotch.io/bar-talk/designing-a-restful-web- api](http://scotch.io/bar-talk/designing-a-restful-web-api)

</comment><comment><USER47> We use jsend[1] for our responses.

1: [http://labs.omniti.com/labs/jsend](http://labs.omniti.com/labs/jsend)

<comment><USER28> As the author of JSend, I no longer recommend this. Browsers' capabilities and my own understanding of HTTP's semantics have evolved beyond there being any further use for it.

<comment><USER47> Can you elaborate on this? Can you give some examples?

</comment></comment></comment><comment><USER48> One thing I've noticed that will become increasingly important in the future is for JavaScript APIs it would be really nice if you had a per user path, for example:

api.com/steakejjs/v1/

This way, user's of your API can implement Content-Security-Policy in a secure manner (where an attacker can't use your 3rd party API to exfiltrate data). It's not like there aren't other ways to exfiltrate data, but this will definitely help.

</comment><comment><USER49> I'd accept both singular and plural. At least redirect one to the other.

</comment><comment><USER50> there are lots of problems (particularly with versioning/backwards compatibility) that using better protocols (i.e., protobufs) can solve.</comment>


What I Meant to Say Was Semantic Web - <AUTHOR> http://bits.blogs.nytimes.com/2007/10/19/what-i-meant-to-say-was-semantic-web/index.html?ex=1350446400&en=532437c4047c9867&ei=5088&partner=rssnyt&emc=rss

<comment><USER2> The Web 3.0 label was a bad idea. The people that have the most to gain from the Semantic Web right now are technical types, and I think most of us are suspicious of the Web 2.0 label.

The Semantic Web has an (undeserved) reputation for being vapor or hype, "Web 3.0" just exacerbates that.

</comment><comment><USER3> An interesting article, but could they please have someone edit it before posting! Thats the most English errors I've ever read in a NY Times publication - in fact in any major newspaper.

<comment><USER4> That those errors cannot be automatically detected and eliminated is actually relevant to the overhyped semantic web, no?

Maybe it is some sort of Andy Kaufman level joke.

<comment><USER2> > That those errors cannot be automatically detected and eliminated is > actually relevant to the overhyped semantic web, no?

No. That would require your computer to understand sentences meant for humans; the whole point of the Semantic Web is that it's machine-readable.

<comment><USER5> If you could make it understand sentences meant for humans, it _would_ be machine-readable!

This was the topic of my degree thesis, but I gave up on it (far too complex to parse everything, and finding useful applications is difficult).

In fact, I think the Semantic Web has application problems. They say things like auctions, but I don't see what's wrong with SQL for those.

And if you used Google to search the web for millions of currently-listed auctions on webpages in XML format, you'd have a difficult time trying to find what you want.

<comment><USER2> > They say things like auctions, but I don't see what's wrong with SQL for > those.

For local data sets, SQL's great. For remote, centralized data sets you can use something similar.

But once you want to combine data sets spread out over several sites (say, combining IMDB with local movie listings) you need to express that data in a format with globally unique identifiers and distributed extensibility. That's RDF. You can still have a nice query language, see SPARQL (which is much nicer than SQL, IMO).

> And if you used Google to search the web for millions of currently-listed > auctions on webpages in XML format, you'd have a difficult time trying to > find what you want.

Sure, if you're just doing full-text search like Google does. Of course, the point of having machine-readable data is that you're not limited to full-text search.

</comment></comment></comment></comment></comment><comment><USER6> Submarine?

<comment><USER7> If it is, its not very subtle ;) So probably not.</comment></comment>


Genome Laser: broadcasting the human genome via laser beam at Burning Man - <AUTHOR> http://www.indiegogo.com/projects/genome-laser/x/3683724 Genome Laser is an an audacious Burning Man project in collaboration with Control Tower and friends that spans science, technology and art. For the first time in history, at Burning Man 2013, the data from individual human genomes will be broadcast in public via laser beam. <comment><USER2> What a mental project - awesome.

</comment><comment><USER3> awesome</comment>


The secret truth about holiday travel, a letter to the traveling public - <AUTHOR> http://blog.flightcaster.com/the-secret-truth-about-holiday-travel-a-lette

<comment><USER2> I must confess I don't get this post. The argument seems to boil down to "things could be worse, so stop complaining".

<comment><USER3> There's a cynical proverb in Brazil, used to pretend to justify conformism: "If the rape is unavoidable, just relax and enjoy".

This post seems to get too close to it.

</comment></comment><comment><USER4> Telling everyone to just sit back and be amazed at their luck for living at this time and place in history is, IMO, sad. We can do better and we will do better. Progress is our purpose.

</comment><comment><USER5> Well said. I always cherish things that make me step back from my normal viewpoints, and remind me of how lucky we are.

It's almost certain that anyone reading this site is in the luckiest top 10% of the world, having access to food and internet and enough money to fly as to be bothered by it.

If you have a job that you like (which a lot of us do), and is well paying, that probably narrows you down into the top 5%. Congrats, and sit back and enjoy it.

Happy Holidays!

<comment><USER6> It's great to remember how lucky we are.

However, don't let this distract from our shortcomings (as a society). And be especially careful when things are moving in the wrong direction.

When things get worse, do not settle for less. That's not gratitude; it's submission.

</comment></comment><comment><USER7> The fix for flying in the US is to finally, really invest in high speed rail all over the country. When you have the option of a roomy, relaxed seat on a train that doesn't take much longer (once the time spent in the airport is factored in), the air travel industry will shape up. And they will use their ill-gotten influence in Congress to bring TSA along for the ride.

</comment><comment><USER8> Yes, 30 minutes of "security" checks and body scanners are a problem.

It is a matter of PRINCIPLE.</comment>


InstaThreat.com: Demand letters as a service - <AUTHOR> I'd love to hear your feedback on my side project InstaThreat.com - we send physical demand letters to people who owe you money. Basically, if someone owes you $500 - $5,000, your options have previously been limited. We sit in the space between doing nothing and hiring a lawyer.
We'll send 3 legal-sound letters that escalate in seriousness over the course of 6 weeks for $49 encouraging people to pay up. Our service has been used to get back rent deposits, collect payment for consultants and freelancers, get money owed from friends, etc.
I'd love to hear your thoughts. Would you use the service? Why or why not? What services should I add or change? Thanks. <comment><USER2> I'm not in your target market, I just wanted to mention that the name doesn't seem to fit the seriousness of what you do. Expect a lot of people asking "What does this have to do with instagram?"

Edit: forgot to mention it sounds like a good opportunity and I wish you the best seizing it. Also your landing page does a good job explaining what you do, except it doesn't mention the price and the "tell us about your situation" line seems to contradict the "...in 2 minutes" claim in the main headline.

<comment><AUTHOR> Thanks for the feedback. Yea, the name could probably use some work. I'm pretty small, so I could definitely change the name down the road.

Good observation about the price. I purposely don't mention price on the home page because I don't (yet) want that to deter people from filling out the form - I want that contact info ;0) I use Stripe to ask for payment after they submit the form. In my ads, I'm testing mentioning price (to set expectations) as well as not mentioning price to see what performs better.

What would you suggest instead of "tell us about your situation". I figured I need to gather at least some info about the issue for customizing the demand letter.

<comment><USER3> I'd be interested to hear how the show-price-versus-not-show-price test works out regarding conversion. Also, have you thought about offering multiple price points for different levels of customization? That might muddy the waters or it might let you capture more purchases ¯\\_(ツ)_/¯

<comment><AUTHOR> Yea, I need to collect some more data on price vs. no price in ad. It seems like a price point of $25 might work a lot better than $49 based on percentage of ads served and CTR - [https://cloudup.com/c-Jy2nwWjQC](https://cloudup.com/c-Jy2nwWjQC)

<comment><USER3> You'd be undercutting the competition! [http://justtelljulie.com/](http://justtelljulie.com/)

</comment></comment></comment></comment></comment><comment><USER3> This is an awesome idea! Particularly useful to freelancers. I agree with namenotrequired that InstaThreat might not be the best name, but it's certainly serviceable.

<comment><AUTHOR> Thanks! The idea was a direct result of a friend who was hired to do a 3 month dev contract and got stiffed for the last month ~$5k for no good reason.

<comment><USER3> I almost wish that I had been cheated recently so that I could try it out myself =P

<comment><USER2> I can help you with that if you like :)

</comment></comment></comment></comment><comment><USER4> The service might work, but the name is horrible and, well, threatening.

Like, physically, creepily so.

Please don't use it.

<comment><AUTHOR> Thanks for the feedback. It was a quick way to get going...copying things like Instacart and Instawork. Any suggestions?

<comment><USER4> Some variant on Insta(nt)(Demand|Letter), PayMeNow, etc. should do fine.

The idea is the tone should be coldly neutral, but not... threatening.

<comment><AUTHOR> It started out as a joke, hence the ridiculous name =0)

<comment><USER4> Understood, but most people, when they get communications with the word "threat" in the letterhead or domain name, definitely would not take it as a joke.</comment></comment></comment></comment></comment>