SSH gets protection against side-channel attacks - <AUTHOR> https://www.undeadly.org/cgi?action=article;sid=20190621081455 <comment><USER2> RAM encryption for sensitive data is overlooked in so many applications, even "highly secure" applications like veracrypt [0] only recently started adding it.

In my opinion server-applications of all sorts should encrypt their private keys by default; this makes cold-boot attacks and other memory-escape attacks so much harder, since now two totally unrelated memory chunks have to be combined in order to retrieve the private key (in fact one could argue that, the higher the quantity of memory chunks is, the harder it is to correctly decrypt the original private key; although to me this has the security through obscurity smell).

In practice however, this is rarely done. OpenSSL for example has a whole lot of memory-management functions [1], but none of them seem to include RAM encryption (AFAIK it is not present in the source code elsewhere either, but it's a large one, so I am not familiar with all code. Related: in fact some people even argue it's way too large for it's own good [2]).

Even better still, would be to use the CPU Cache for private keys, since this memory is even more difficult to access through any sort of cold boot. An interesting paper with much more information about this subject can be found here [3], to whom it may concern.

[0] [https://github.com/veracrypt/VeraCrypt/commit/321715202aed04...](https://github.com/veracrypt/VeraCrypt/commit/321715202aed04dd9892d1c0686d080763ab212d)

[1] [https://www.openssl.org/docs/man1.1.0/man3/OPENSSL_zalloc.ht...](https://www.openssl.org/docs/man1.1.0/man3/OPENSSL_zalloc.html)

[2] [https://queue.acm.org/detail.cfm?id=2602816](https://queue.acm.org/detail.cfm?id=2602816)

[3] [https://www.ieee-security.org/TC/SP2015/papers- archived/6949...](https://www.ieee-security.org/TC/SP2015/papers- archived/6949a003.pdf)

<comment><USER3> Yup, we added this feature to Varnish Cache a few years ago, random key encryption. It generates a random key at startup and encrypts all memory with it. Since this kind of memory is only resident for the lifetime of the process, it works. We stored the random key in the Linux kernel using the crypto API [0] just because its not safe storing any kind of keys in a memory space used for caching (Cloudbleed [1]). We then use the key to generate a per object HMAC, so each piece of data ends up with its own key, which further prevents something like Cloudbleed. Since we used kernel crypto, overhead was about 50%. If you stay completely in user space, its probably much lower.

[0] [https://www.kernel.org/doc/html/v4.17/crypto/userspace- if.ht...](https://www.kernel.org/doc/html/v4.17/crypto/userspace-if.html)

[1] [https://en.wikipedia.org/wiki/Cloudbleed](https://en.wikipedia.org/wiki/Cloudbleed)

<comment><USER4> Could this be implemented at the OS level, i.e. whenever a proces launches, the OS generates a key that it will keep to itself and use to transparently encrypt all memory allocated by that process?

<comment><USER3> Possibly, but memory is accessed using plain CPU instructions, so it would be hard to transparently encrypt all memory for an application at the kernel level. You do have virtual memory, but I dont think that could be leveraged for this. But who knows whats possible there, maybe if you align and address each memory value at the page boundaries and always force a page fault you could have a really poor implementation :)

Transparent disk encryption, not a problem since devices have filesystems which can implement encryption at that layer.

<comment><USER5> Modern Intel chips can encrypt memory on the fly without performance loss (SGX does this). However I think it's not exposed for non-enclave use. Perhaps it should be.

Note: inside the enclave there _is_ a performance loss but that's due to MAC checks. If you just want encryption without integrity against tampering you don't need that.

</comment></comment></comment></comment></comment><comment><USER6> When I first saw this post, I got excited that someone had finally addressed the side channel from 17 years ago, which gathered inter-packet timing from ssh packets (and the fact that passwords are not echo'd) to a) detect when passwords were being typed, and b) use Hidden Markov Models to extract keystroke timing to try to extract the likely passwords.

There have been defenses proposed over the years, but none were accepted by OpenSSH AFAICT.

[https://users.ece.cmu.edu/~vsekar/Teaching/Spring19/18731/re...](https://users.ece.cmu.edu/~vsekar/Teaching/Spring19/18731/reading/SideChannel_Song.pdf)

[https://www.researchgate.net/publication/2907598_Timing_Anal...](https://www.researchgate.net/publication/2907598_Timing_Analysis_of_Keystrokes_and_Timing_Attacks_on_SSH)

<comment><USER7> The proper defense is to not use passwords. It's 2019, nobody should be using passwords on SSH-enabled shell accounts.

Heck, I use secure tokens for authentication so this memory encryption hack is useless to me--even _I_ don't know what my private key(s) are, nor does OpenSSH or any other software on my computers. But I appreciate that we're a long way from ubiquitous hardware-based authentication.

<comment><USER8> After you are connected you might still want to enter a password (for sudo perhaps).

<comment><USER9> The solution there is to have the app use line buffering rather than a "raw" term mode that exposes inter-char timing on the network. How widely that's followed in practice, I do not know, but one would certainly hope that sudo does it.

</comment></comment></comment></comment><comment> yjftsjthsd-h Unfortunate that there's no commentary on performance impact. It's symmetric encryption on a few kB, so probably fast, but I'd like to have numbers.

<comment><USER10> It's not mentioned because it's irrelevant. There is no security / performance trade off for a secure program. It's secure or its not.

<comment><USER11> There are a lot of knee-jerking in this thread. I don't think anybody is arguing that this is a bad patch, but if it _does_ have a notable performance impact (which it probably does not) it's still worth knowing. If only to be able to answer questions like "why do our backups suddenly take 10% more time to complete? Do we have a problem with our network infrastructure?"

It's not about tradeoffs, it's about understanding what's going on and anticipating potential problems.

By your logic we shouldn't have to discuss the performance regressions on CPUs who implement spectre/meltdown mitigations because "it's irrelevant". Obviously these patches are necessary but the performance impact is very relevant for many users.

<comment><USER12> By the parent's logic, there's no concept of improving security, because if there remains attacks, then it is still not secure. You can't take a position of "it's either secure or it isn't."

</comment></comment></comment></comment><comment><USER13> Could password managers like 1Password (AgileBits Inc.) employ similar techniques? They seem to make very little effort in this regards. E.g. on 1Password, [https://discussions.agilebits.com/discussion/101551/article-...](https://discussions.agilebits.com/discussion/101551/article- just-published-in-washington-post-is-saying-1password-and-others-have- security-flaws).

<comment><USER14> This kind of mitigation really only makes sense on shared machines (such as servers). On a desktop OS, if an attacker is in a position to read memory from other processes, it's pretty much game over already.

Browsers implement Spectre/Meltdown mitigation on desktop OSes because without that, JS could read secrets from other JS contexts executing in the same process. One of the mitigations is in fact to just segregate JS contexts into different processes depending on the domain they belong to. But most apps don't execute untrusted code so most apps don't have this sort of in-process attacker to worry about.

<comment><USER15> > On a desktop OS, if an attacker is in a position to read memory from other > processes, it's pretty much game over already.

Not really. Password managers have access information to your other accounts in the cloud. Your desktop may not have your most valuable data.

</comment></comment></comment><comment><USER16> I've looked at it. There's still a spectre window of opportunity to get the shielded private host keys.

sshkey_shield_private => explicit_bzero() openbsd-compat/freezero.c

[https://github.com/openssh/openssh- portable/commit/4f7a56d5e...](https://github.com/openssh/openssh- portable/commit/4f7a56d5e02e3d04ab69eac1213817a7536d0562)

It's only using the insecure freezero, which is using the insecure explicit_bzero. A simple compiler barrier only, no memory barrier. so it's unsafe against the advertised spectre/meltdown sidechannel attacks, the secrets are still in the caches.

</comment><comment><USER17> > Attackers must recover the entire prekey with high accuracy before they can > attempt to decrypt the shielded private key, but the current generation of > attacks have bit error rates that, when applied cumulatively to the entire > prekey, make this unlikely.

It seems the real mitigation isn't the prekey size, but the temporal sparseness of the symmetric key -- since I would've imagined attackers would just try to obtain the symmetric key rather than the prekey. Weird to see they they didn't even mention this... I imagine attackers would try to find a way to get the symmetric key to stay in memory for a while.

<comment><USER18> The prekey is hashed into the symmetric key. Both the hash function and the symmetric cipher have avalanche effects that mean that N bit errors require the attacker to bruteforce 2^N combinations.

unprotected RSA keys on the other hand have structure and are dense in memory. That means fewer bit-errors and and the ability to guess the missing bits faster than O(2^N).

<comment><USER17> Yeah but my concern was about direct leakage not brute force. That's what Spectre is about.

<comment><USER18> Oh yeah, but the temporal sparseness doesn't just apply to the symmetric memory encryption key. The most important part is that it also applies to the asymmetric host keys, which are the actual thing one wants to have protected.

prekey -(hash)-> memory key -(decrypt)-> host private key

</comment></comment></comment></comment><comment><USER19> Is this only for OpenBSD's implementation of SSH?

<comment><USER20> If we wait long enough, it will go all the places that the portable release at openssh.com pushes it.

It will take a few service packs, but even Windows will get it.

        C:\>ssh -V
        OpenSSH_for_Windows_7.6p1, LibreSSL 2.6.4

</comment></comment><comment><USER21> Does this only protect the SSH host key?

What about the session keys for the sysadmins who have logged in, and maybe keep their session up 24/7?

<comment><AUTHOR> Session keys are rotated. See "RekeyLimit" in sshd_config(5).

</comment></comment><comment><USER22> Take my money!

<comment><USER23> Great!

You can donate to OpenSSH[0], whose "funding is generally done via the same donation framework" as the rest of OpenBSD, to which you can donate either directly[1] or via the OpenBSD Foundation[2]. If you're serious about donating obviously please check that these links are legitimate and I'm not a scammer. (I'm not affiliated with OpenBSD in any way.)

[0] [https://www.openssh.com/donations.html](https://www.openssh.com/donations.html)

[1] [https://www.openbsd.org/donations.html](https://www.openbsd.org/donations.html)

[2] [https://www.openbsdfoundation.org/donations.html](https://www.openbsdfoundation.org/donations.html)

<comment><USER22> 50$ donated! I hope that's good for some beers :)

On a sidenote paypal donation took way too long. If anyone is interested in making open source donations simpler feel free to ping me.

<comment><USER20> For your donation, you also received preliminary code for quantum-resistant key exchange.

        $ sed -n '/NTRU/,/enabled/p' ChangeLog
        firstname.lastname@example.com using the Streamlined NTRU Prime
        4591^761 implementation from SUPERCOP coupled with X25519 as a stop-loss. Not
        enabled by default.

</comment></comment></comment></comment><comment><USER24> Krypton Authenticator app anyone? [https://krypt.co/docs/security/privacy- policy.html](https://krypt.co/docs/security/privacy-policy.html)

</comment><comment><USER25> It could use the x87 floating point register stack to store the encryption secret. These registers are not used unless there would be some assembly in the SSH libs that accesses them.

<comment><USER26> Wouldn't they be saved to memory anyway on a context switch?

<comment><USER25> Yes, but I thought this is done in kernel space and a userspace attack has no access to that while the syscall is served.

<comment><USER26> That is true - it's probably more secure than storing them in the process, but several of these side-channel attacks can apply to kernel space (depending on hardware and security patches applied).

It's also not portable - OpenSSH runs on non-x86 architectures, and they might not have spare basically unused registers lying around.

Finally, I'm not sure the x87 registers have enough space to fit these keys. You have 8 80-bit registers, for a total of 640 bits. Your typical SSH private key might be 2048 bits or more.

So it's a fun and creative line of thinking, but probably not practical in this case.

</comment></comment></comment></comment><comment><USER15> > Hopefully we can remove this in a few years time when computer architecture > has become less unsafe.

Narrator: It hasn't.

</comment><comment><USER27> Couldn't they also move the keys around in memory to every second or keep the bytes of the key separated (this seems like it would be similar to encrypting)?

These side channel attacks even under ideal conditions take a very long time and part of the problem is they basically need to guess at memory addresses. Even when data is in a known location, it is sketchy. Anything that slows down locating data would help immensely.

</comment><comment><USER28> Does this support all Private Keys across all operations?

Trying to understand the coverage.

<comment><AUTHOR> Host keys only, as those are the only ones that are long-lived.

Session and cipher keys are ephemeral for forward security, and are rekeyed on a regular basis: see "RekeyLimit" in sshd_config(5).

</comment></comment><comment><USER29> Anyone with the actual link to the specific commit?

<comment><USER30> Sure:

[https://github.com/openbsd/src/commit/707316f931b35ef67f1390...](https://github.com/openbsd/src/commit/707316f931b35ef67f1390b2a00386bdd0863568)

<comment><USER31>
       -  /* $OpenBSD: authfd.c,v 1.113 2018/12/27 23:02:11 djm Exp $ */
       +  /* $OpenBSD: authfd.c,v 1.114 2019/06/21 04:21:04 djm Exp $ */

what weird tool/infrastructure do these guys use, that they maintain CVS/SVN string identifiers in the files but nevertheless use git?

I actually liked these identifiers. There were tools like ident ([http://manpages.org/ident/1](http://manpages.org/ident/1)) to deal with them.

<comment><USER32> They use CVS, not git. The repository that's on GitHub is a mirror of the CVS repository; I don't know what tool is used to do the mirroring.

Here is the original change in authfd.c in cvsweb:

[https://cvsweb.openbsd.org/cgi- bin/cvsweb/src/usr.bin/ssh/au...](https://cvsweb.openbsd.org/cgi- bin/cvsweb/src/usr.bin/ssh/authfd.c?rev=1.114&content-type=text/x-cvsweb- markup)

</comment></comment></comment></comment><comment><USER33> > Hopefully we can remove this in a few years time when computer architecture > has become less unsafe.

I think the author misspelled 'decades' here :)

<comment><USER34> I believe all consumer AMD Ryzen CPUs support Secure Encrypted Memory (SEM), but for some reason it's not enabled by default on most Ryzen devices/motherboards. It's a shame.

I'd also love to see AMD bring an updated and patched version of Secure Encrypted Virtualization (there have been some attacks against it, although still fewer than against Intel's SGX) to consumer Ryzen in the near future. With so many cores available in consumer AMD CPUs (up to 16 now), people will start to use VMs more. Even Windows 10 has the easy-to-use Windows Sandbox now, as well as the App Guard sandbox for Edge.

Not to mention they could use this as yet another "killer app" of their many- core CPUs, because otherwise people will eventually start to wonder why even get CPUs with so many cores over CPUs with fewer cores but higher singlethread performance. No different than say Verizon promoting high-quality 4k streaming on its new 5G network.

I would've already preferred to see this in Zen 2, but at least Zen 3, which will otherwise bring few performance improvements and remain on the 7nm node, should come with these as some sort of "security-focused generation of Zen".

<comment><USER20> Isn't AMD's encrypted memory meant more for the case of someone with physical access aggressively cooling a running system, then cutting power and removing the chilled memory for analysis (which will preserve contents much longer when cold than at normal temperatures)?

The case of an attack on the SSH Agent would take place within the CPU, perhaps even in the same core on a separate HT/SMT thread, where the memory would be cleartext.</comment></comment></comment>


Ask HN: Why is Enterprise software UX so bad compared to Consumer Software? - <AUTHOR> In my experience working for various companies, I've found the the software we're required to use for various tasks always has clunky, inefficient, and sometimes just bizarre UX.<p>How did software that is so hard to use become so prolific in the B2B space? <comment><USER2> It's not. It's great. Most people forget that UX is like a language. You can't go about changing a language that people are accustomed to. It's a very expensive thing. Enterprise users pay a lot and are use to it. Changing the UX/UI means retraining tons of people, breaking integration that other businesses have built on their system. Most companies don't provide API, so businesses build on UI. What do you think happens when Amazon or Google or Facebook changes their UI? lots of businesses that screenscrape break. Now these customer are usually not paying, but imagine that they were paying. You now have a swarm of angry paying customers. I work in enterprise. There are lots of integration we have with other companies UI/UX. Over http, over terminal/mainframe. If anything changes, impact is millions. Of course to our shock and utter horror. NOT! Other businesses have built logic on our UI/UX. If we change some things, their integration breaks, hundreds of thousands or possibly millions of dollars at stake too. You see clunky, I see money, I see happy users, I see users familiar with the way it is and no retraining needed for 10,000 users.

<comment><USER3> Doesn't explain why the UX is bad. There are great user experiences from 40+ years ago that hold up.

<comment><USER2> Additions are added without breaking previous UX even when a better flow is possible. Imagine you have F and it's natural to go from E->F but it's also very nice to go from A->E->F, well if you already built a path from A->B->C->D->E, with Enterprise, they will have A->B->C-D->E->F, where modern consumer software will just change it from A->E->F and maybe move B to the main menu.

</comment></comment></comment><comment><USER4> Having worked on Enterprise software, the guys that are paying for X feature to be added usually have no UX sense themselves and, unless we are explicitly given the freedom to put it in as we see fit, we're left to the customer to tell us where it goes.

Customer: I want this new button right there.

Developer: This menu is already quite cluttered with other features we've put here. Could we not add it to a new menu?

Customer: No, I spend like 95% of my day in this menu, so I want it right here.

Developer: This might be confusing to some of our other customers, though.

Customer: Yeah well they're not paying for it. Do you want my money or not?

</comment><comment><USER3> The people who design/build consumer software often eat their own cooking. Feedback cycles are rapid and short.

The people who design/build enterprise software rarely eat their own cooking. Feedback cycles are infrequent and long or non-existent.

</comment><comment><USER5> Good UX costs money and there's no incentive to improve it since the people who buy it don't use it.

</comment><comment><USER6> Because whoever makes decisions on purchase usually don't use them.

<comment><USER7> This exactly the problem. Similar to the third-party payer anti-pattern in healthcare.

</comment></comment><comment><USER8> In consumer, the customer are typically advertisers and the users are typically the product. Facebook ad products are just as strangely designed, complex and such, because no one approving Facebook Ad spend in a big deal is the one using it. So the same company can actually end up with both, a "product" that serves decision makers in ad agencies, and a "product" that retains consumer eyeballs.

When you say clunky, inefficient and bizarre UX, you mean the user UX, which doesn't matter in enterprise. If it didn't save a VP or better yet the CEO of a large company mad time, make them more money, save them money or mitigate risk, it would not be there in the first place. Because systems like SAP are incredibly good at both doing that in complex use cases and making the case for it, they prevail.

Startups often try to go enterprise and say their UX is better. Notion may be great, but based on pricing it's creating <$10/user/month in enterprise value. There are products in specialized industries (e.g. Bloomberg in finance) that charge $2000/user/month. They serve the real customer, which is the VP or CEO who signs, very well.

<comment><USER9> If you've ever seen someone good at a Bloomberg terminal it is incredible what they can do. Think of it like vim for finance.

</comment></comment><comment><USER10> Companies have processes and changing those processes is very difficult and risky for the managers involved. A new software provider really needs to fit round the existing processes if it wants to make a sale.

Buying one person, or one team, a tool like Adobe CS doesn't really change any processes so it's easy. But dropping a new system in that changes a process and everything changes.

"Helen from risk wants a new fraud detection system" is effectively the same as "Helen wants to review all our fraud detection policies and if anything changes and subsequently goes wrong it won't just be Helen that gets fired". There's a huge pressure on the software provider to make their system support whatever unique, crazy or arcane processes the risk team were previously using, and support all existing interfaces into and out of the risk team, because nobody wants to sign off on any changes. So enterprise software has to be crazy flexible. And flexibility makes for bad UX.

</comment><comment><USER11> > _How did software that is so hard to use become so prolific in the B2B > space?_

If the software relies on sales people who answer "Yes, absolutely!" to every question from prospects, you end up in a bad place.

Generally speaking, User != Purchaser. You push to include User in the loop and development process. Sometimes, you are not allowed, and sometimes the contract itself is signed contingent on this. It can get carder if you're not allowed to sollicit users' feedback, _but_ are given carte blanche to develop: you're developing based on what you think will be right for the user. You will have hard conversations with the users at the end of the project when they receive the software and you're allowed to interact and "fix a few things here and there", at which point you'll try to cram in as much user feedback as possible to "make it right".

Now, if you're not building for a particular client and you are working on _your_ product that's targeting the enterprise, there are other distribution methods where you design for the _user_ who works for an enterprise so there's a "de facto" adoption, which then can lead to an enterprise software that is good.

A quick litmus test: if you are on a website and want to use the software, and there's a button that reads "Schedule Demo", or "Book Appointment with Sales", or "Contact Sales", chances are the software in question is the kind you don't like.

</comment><comment><USER12> Lack of competition. Lack of people who understand or care about UX. For example, a typical enterprise software company will hire programmers who are both skilled in programming and have domain experience, i.e. a financial programmer for a financial industry app. This is because they believe, probably correctly, that understanding finance is more important for their developers than understanding how to make beautiful animations play at 60fps.

There's a lot of other problems. "Enterprise" software often comes from companies that aren't actually software companies, in the sense that they aren't run by the guy who wrote the first version of the product. They're run by non-tech people who hired some developers to build their idea. These people typically create large and bloated product and project management organisations that try to convert programmers into replaceable cog wheels that convert JIRA tickets into code commits. They often dream of out-sourcing their devs entirely and would love to do nothing but sit in the middle collecting fees without needing to write software. They certainly cannot judge the quality of the work produced, even by looking at the user interface. Maybe they have some vague feeling it could be better but they aren't going to dive into the detail and figure out why or how.

Developers know this and have minimal loyalty to the product or organisation, so whatever work is done is always minimal effort work. If the ticket can be marked as done they move onto the next one.

Successful consumer facing software firms are often quite different. They're run by someone who uses their own product, at least sometimes, and who probably made the first version so they care about the details and know what's possible. They hire developers who are NOT domain experts, on the assumption that if you need expert knowledge to use the app you're not making consumer software, so they hire people who have a proven track record of creating beautiful and easy to use UI. What the UI exactly does is not so important. They tend to trust developers more, in my experience, because they know a small performance or UI edge can be sufficient to differentiate them in the market.

</comment><comment><USER13> Primary agent problem. The people who purchase Enterprise software (accounting, IT) aren't the same people who use the software, therefore they don't care about things like UX when making purchasing decisions.

</comment><comment><USER14> [https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...](https://en.wikipedia.org/wiki/Principal%E2%80%93agent_problem)

</comment><comment><USER15> In my opinion, the reason is that Amazon, Facebook et al know their users will not receive training or persevere with poor UX or an unintuitive UI. Therefore, they have to design good software in order to be competitive.

On the other hand enterprise software implementations always include a business change element which involves lots of communication and training, especially for key users. Therefore there is less obligation for vendors to build intuitive solutions.

</comment><comment><USER16> One reason could be that consumer software is somehow forced to have a good UX because they need more users. More users more profit.

Enterprise users are already all there. Once a solution is choosen users are forced to use the software as is, no matter how bad or good it is. There is just no more incentives to make it better for corporate users. Think SAP, it make me think of this every time I see it.

</comment><comment><USER17> I wonder how much enterprise is driven by the customer and customers are often terrible at considering the impact of requests.

If they will pay for button or feature right there... often there it goes.

If they like the layout or design of something they're familiar with...there it is.</comment>


Ask HN: Prgmr is unreliable. Please advise on VPS hosting. - <AUTHOR> I've been using prgmr.com to host a web application in beta mode. I didn't want to spend a lot of money before I get any serious users, so I went with what seemed the cheapest option but still gave value for money.
My experience has been OK, except for one thing - they are completely unresponsive. I know their motto is "We don't assume you are stupid." but charging people money for a service and not providing any support is just rubbish. I've emailed them with some questions in the past and they never get back. A while ago they started issuing ticket numbers and I thought they must be improving the service, but they still never get back.
Last couple of days, my server has gone down a few times. It went down again this morning. Now I can access the root domain, but not the subdomains. I have written to prgmr support and I got my ticket number. But I'm not expecting any real help.
I know this is a bit desperate but I'm hoping HN user lsc (who runs prgmr) will take note of this.
Meanwhile, can I ask you to advise me on choosing another VPS host? I know Linode and Slicehost are the most recommended ones here. Given the prices, I might go in for Linode but I'm wondering if any of you have advice on running a beta cheaply until launching a product.
Sorry for the rant. This is my first time running a VPS server. I'd appreciate any help.
EDIT: Thanks a lot for your responses everyone. In addition to Linode, a few other good alternatives have been suggested which I will look into.
I have managed to get my server up again now and also got a response from prgmr. It's a shame that I had to do this publicly, I don't mean to defame prgmr. But I think it's worth pointing out that this is possibly one bad feature of competing on price. <comment><USER2> we got a ticket from you at Tue Jun 15 03:51:50 2010

we sent a response at Tue Jun 15 06:25:58 2010

that's what, a little more than two and a half hours in what is the middle of the night for me?

This is a /whole lot/ better than I would expect my response time to be in the middle of the night. If you are disappointed with this response time, I have utterly failed to set expectations. email me and I'll refund your last month and you can go over to Slicehost or what have you.

Slicehost and linode both pride themselves on support. Slicehost has phone support, and from what I hear, they might help you with your apache problem, too. I met some of the slicehost support people at a xen conference a few months back, and it sounds like they really go the extra mile... so for you, the extra they charge might be more than worth it.

email me and I'll get you a refund of your last month as per policy.

(Note, you don't need to post a HN story to get a refund of your last month, I'll do that for anyone who asks.)

<comment><AUTHOR> Luke, thanks for your response. I don't want a refund. I will email you.

</comment></comment><comment><USER3> Linode is pretty solid. They have had about 2(?) outages in the past year or so that I can remember. I've been using Linode since 2008 and am happy.

<comment><USER4> I have a linode for my totally inactive blog and a small project I am working on. We also just migrated all our servers for SWIX to linode VPS's. All in all linode has been a pretty great experience.

The linode library is pretty helpful too, especially if you are fairly new to linux or systems administration.

</comment></comment><comment><USER5> I'd recommend ChunkHost. You can sign up for their free beta. I'm a paying customer of their's and have had only minor problems. To clarify that, they had a scheduled maintenance a while back that involved a reboot of the instances and I had forgotten to set nginx to start at boot, but they'd communicated the maintenance window long in advance of the maintenance and so I knew to check to make sure it was up and it was a good thing to fix in general.

Their prices are very reasonable, they have a nice, simple online management thing that lets you deal with DNS, re-imaging chunks, etc.

I have used/do use: EC2, Slicehost/Rackspace Cloud, Linode, and prgmr.

I found prgmr to be a pain. There are just some small things that make administration so much easier.

Slicehost/Rackspace has gotten expensive compared to the competition, but they're pretty rock solid.

EC2 has some wonderful features, but it doesn't go down to the cheap levels I want for my personal hosting.

 _Linode is always a great performer_ and the only reason I'm not with them is that I'm getting a better price at ChunkHost. While I haven't had problems with ChunkHost, Linode is a larger player if you're looking for business hosting you can sell to your boss. As others will tell you, it's easy to recommend Linode.

<comment><USER6> >I found prgmr to be a pain. There are just some small things that make administration so much easier.

I bet lsc would be interested in your feedback if you sent him an e-mail.

Maybe not, but maybe.

<comment><USER2> I would certainly be interested in hearing it; I can't promise I'd implement, though. I am most interested in implementing things that make it /possible/ for a user to deal with problems without my help. I am less interested in things that make it /easy/ for a user to deal with problems without my help. Also, there is a long queue of improvements I currently need, some of which are quite critical.

</comment></comment></comment><comment><USER7> I'm a happy customer of Slicehost. Their customer service is top notch. The few times when I needed help, I went to their chatroom and people were there to help immediately. They really do man the chatrooms. Very responsive and helpful. It's not easy to maintain such high standards for years.

Their articles is another plus point. It's constantly updated. It covers most of the things you need. No doubt they are not the cheapest. But their customer service more than makes up for it.

</comment><comment><USER8> I've had a great experience with prgmr.com, even when asking them possibly- stupid questions about my configuration. If you can get to one domain but not others on a VPS account, it's likely not the VPS provider's fault...

<comment><USER9> >If you can get to one domain but not others on a VPS account, it's likely not the VPS provider's fault...

Yeah, was gonna say - a VPS provider's job is to make sure you have a running box and a good pipe to it. This is almost surely a configuration issue on the VPS and not the provider's fault.

<comment><AUTHOR> I agree but what happened is that the server did go down and initially I could not access any of my sites, including root domains. Then the server seemed to have come back to life, but the subdomains still weren't working. That turned out to be some Apache issue, which I resolved by restarting it.

<comment><USER9> Which confirms what I said -- the issue you were complaining about was a problem with _your configuration_ , not with the VPS provider. No provider guarantees 100% uptime, and you have to be able to handle things when there is an outage. The root domain going down _was_ their problem, your subdomains not coming back afterward wasn't.

<comment><AUTHOR> I know well that one can't expect 100% uptime. But I have had a few outages since yesterday. Even so, my main problem with prgmr is not the hardware but their almost complete lack of support.

<comment><USER9> I think the point is the fact that you're conflating what definitely appear to be configuration issues with provider issues seems to indicate that your expectations for support may be a bit unrealistic, especially for a provider like prgmr that specifically advertise themselves as being for users who _will_ be able to identify the difference between the two.

I certainly understand your frustration, but I think the heart of the matter is that prgmr may be inappropriate for your needs rather than "prgmr is unreliable." I feel like they're quite clear about the fact that they're not for everyone.

</comment></comment></comment></comment></comment></comment><comment><USER10> Try rackspacecloud servers (<http://www.rackspacecloud.com>) - uses pretty much the same backend as slicehost (slicehost owned by rackspace). I find them cheaper as it's not a fixed cost, but pay as you go.

<comment><USER11> This. I've setup several client servers on slicehost in the past 6 months and they have been solid. More recently, I've put a couple clients on rackspace cloud because the SLA will make the client feel more comfortable. It's been equally solid.

</comment></comment><comment><USER12> I've had good luck with webfaction. I've only been using them for a couple months, but the one time I had an issue they were very quick to respond.

They're big on Django and I guess Ruby (don't use it). No problems getting a Django site up and running. No issues installing packages or other software in my user account.

<comment><USER13> Indeed. Webfaction is great for a budget plan. But it isn't an actual VPS. You can do most of the stuff you'd want to do with a VPS, but you don't get root access. Plus, their higher-level plans aren't that great.

<comment><USER12> That's right. I'm in the habit of conflating VPS with shared hosting.

</comment></comment></comment><comment><USER14> > This is my first time running a VPS server.

No offense, but it sounds like prgmr isn't for you. When they say "we don't assume you're stupid" they're really trying to say "we assume you won't call us for help, so we have low prices".

<comment><USER15> There is a huge difference between calling for help with setting up your VPS and calling to ask why their servers are down and when one can expect them to be up again. I can understand that they won't handle the first type of call, but I damn well expect them to handle the second.

<comment><USER16> Yes, except in this case, he claims the root domain works, but not his subdomains. If he only has 1x VPS, this means he has messed up his own configuration. Prgmr/Linode are paid to offer a service, but if you want them to administer your server, you should probably use shared hosting, or pay for a qualified server administrator

<comment><USER17> There are a lot of cheap VPS providers out there. You might be interested in trying burst.net : <https://www.burst.net/linvps.shtml>

</comment></comment></comment></comment><comment><USER18> Gandi is quite good (but based in France). I'm using their VPS hosting service since March and it's flawless (12€/month, and you can take advantage of the cheap euro if you're in the US). 3 months is too short to have a solid opinion, but I've been using their DNS service for years without any trouble.

Their motto is: "No bullshit"

<http://en.gandi.net/hebergement/>

<comment><USER19> I use Gandi for domains (not VPS) and while it works ok, support is not very good. I've had tickets going unreplied for a couple of weeks which IMHO is absolutely unacceptable from a big company like Gandi (actually, I believe it's unacceptable from anyone). So from my experience with them I can assume they might provide you with a good service as long as you don't need to get in touch with their support team.

I've been a Slicehost user in the past and really liked their service.

</comment></comment><comment><USER20> Your post is incredibly unfair to Prgmr. You are obviously not great at server administration and you seem to imply that you are requesting server administration support.

All Prmgr promises is that they'll keep your VPS running most of the time. It's on you to maintain it. Not only that but you know lsc is on the west coast and you posted this article while he was still in bed. If your slice is running, lsc is doing his job.

Apache configuration issues are on you.

<comment><AUTHOR> The server actually went down. I could not even ssh into it. I know I am not great at server admin but if it were just Apache config issues I wouldn't make such a big deal out of it.

Like I've said elsewhere, my beef with prgmr is not the servers or uptime, but rather their poor customer support. That's what I mean by "unreliable". It may seem quite desperate, but complaining on a public forum was my last resort to get any attention on the issue.

I appreciate it must be hard to run a single-man company (I know I run one too), but at least one needs to communicate with the customers.

<comment><USER20> Were your support requests related to the server going down? Was that happening that often? Or were they for server admin support? Because you make it seem that only recently did the machine start dying.

From what I've read you seem to be a nightmare customer. Not only expecting a service you aren't paying for, but complaining about it in a public forum when things don't go your way.

>I know their motto is "We don't assume you are stupid." but charging people money for a service and not providing any support is just rubbish. I've emailed them with some questions in the past and they never get back. A while ago they started issuing ticket numbers and I thought they must be improving the service, but they still never get back.

Are you saying here you are sending support requests because your server is malfunctioning/down or are you requesting server administration support?

Honestly, you signed up for a service that openly states that you are expected to run your own machine and then you complain on a professional forum that the company owner is hand holding you.

<comment><AUTHOR> _From what I've read you seem to be a nightmare customer. Not only expecting a service you aren't paying for, but complaining about it in a public forum when things don't go your way._

Service I'm not paying for? Come on, now that's not true.

I've sent support requests for a number of issues in the past and today for the server being down. Are you saying I should expect to be ignored?

 _Honestly, you signed up for a service that openly states that you are expected to run your own machine_

I have been running my own machine, even though I am no pro server admin. But surely I have the right to ask about downtime and other issues which are not in my hand.

<comment><USER20> >Service I'm not paying for? Come on, now that's not true.

You are paying for a hosted VPS. Not a managed server. If you need help managing a server you should pay for that.

>But surely I have the right to ask about downtime and other issues which are not in my hand.

Other than downtime (Which of course is perfectly acceptable to complain about) you still are alluding to trying to get server admin support from a VPS provider. What other kinds of VPS hosting issues did you run into? I'm curious. Of course you could just apologize for trash talking a VPS provider for not providing you managed server support.

</comment></comment></comment></comment></comment><comment><USER21> Not had any issues with pgrmr support or reliability myself, but can't recomend linode/slicehost enough - personally slightly prefer slicehost, but use linode as well since slicehost don't have UK servers.

edit: oh, and possibly consider www.ovh.co.uk, a French company. Never used them myself, and don't know how good their VPS servers are, but have heard many endorsements for their dedi servers (and their prices are very cheap)

<comment><AUTHOR> Thanks, I hadn't heard of ovh. My concern with going with someone like that is I don't know how much better the support will be than prgmr. I think I should just stop acting cheap and go with Linode.

<comment><USER21> What kind of support are you going to be wanting?

<comment><AUTHOR> Well, basically if something goes wrong (e.g. server goes down a few times in 2 days), then at least someone I can talk to, ask what's happening instead of a wall of silence? I don't need phone support, even email is fine as long as I am not ignored.

<comment><USER21> Slicehost/Linode are both fine for that, will ask a friend about OVH

</comment></comment></comment></comment></comment><comment><USER22> Since I wrote my article (<http://journal.uggedal.com/vps-performance- comparison>) comparing VPS performance last fall Linode has had two network outages in their Newark data center which have affected <http://wasitup.com>. Other than the network glitches their hardware have been 100% stable (I'm currently running on 4 different physical machines).

</comment><comment><USER23> My experience with linode:

* Customer for ~4 years

* At one point my server catastrophically failed

* No backups. Or, now it's an add-on for an extra fee. See point above.

* They don't offer support for much... officially. But they make their internal IRC channel public so if you go in there and ask they'll just help you anyway.

</comment><comment><USER24> Have you emailed Luke about your discontent?

<http://book.xen.prgmr.com/mediawiki/index.php/Complaints>

<comment><AUTHOR> I haven't emailed Luke this time, but I have in the past. I have never been rude to him or his staff, but I should email him now regarding my discontent.

</comment></comment><comment><USER16> Firstly, if you can access your root domain, but not the subdomains, you have configured something wrong (unless you are using multiple IP's). That's YOUR fault, and neither Prgmr or Linode is likely to help much there (although in both cases, people in the channels are more than happy to provide some suggestions).

Now, in regards to stability, I find both to be equal. The difference is, that for the cost of 1x Linode, you could buy 2x Prgmr nodes in different location and run them simultaneously to provide much better stability.

Maybe you should jump onto IRC, but from the sounds of things, you may not have the experience to administer a Prgmr Node. Linode is easier to learn with, but is much more expensive

Just onto Freenode IRC / #prgmr, and people will help. But honestly, there was a short server outage a month ago, but your current server issues are likely configuration faults on your side.

</comment><comment><USER25> I was about to write something similar about prgmr. They just don't care.

In our case, we just got bored of _all_ VPS providers and took a dedicated server. Take the time to compare a VPS to a dedicated server and you may have a few surprises. My current server is 80$/month and a similar one on Slicehost would be 250$/month.

<comment><AUTHOR> What dedicated server are you using? Care to give more details?

<comment><USER25> We're on <http://www.iweb.com>

</comment></comment></comment><comment><USER26> I know that's not quite what you're looking for, but I use <http://www.lowendbox.com/> to look for good deals on VPS (they have coupons) and <http://www.lowenddedi.net/> when I want a dedicated server... At lowendbox, I found <http://www.quickweb.co.nz> (they're based in New Zealand but my VPS is in the US). I'm paying $4.95 for a 256M ram/20GB disk. Quickweb is very good and have awesome support.

Also, because they're in New Zealand, you can ask for stuff late at night (in the US) and they'll answer promptly.

Depending on what you want, <http://prq.se/> might also be a good choice.

<comment><USER2> Lowendbox is a great place to look for /cheap/ providers; I'd suggest that the OP should look at something a bit more up market.

<comment><USER26> Yes, I totally agree with you. That's what I meant with "that's not quite what you're looking for". I just hope someone who searches for "vps site:ycombinator.com" finds this...

</comment></comment></comment><comment><USER27> I'm quite satisfied with proPlay (<https://www.proplay.biz/>).

Advantages: \- Rock-stable (didn't have any issues so far) \- Cheap (e.g., for 3 Euros a month you get a vServer with 384 MB RAM and 8 GB harddisk) \- Non- Enterprisy: Support is available via IRC instead of an useless callcenter

Disadvantages: \- It seems to be a one-man company. However, it has been in business since 2003 \- They are based in Germany. So from the US the latency to their servers is somewhat higher than a US-based provider and their websites are primarily in German

If you check them out have a look at their "vps premium" offers. they are even cheaper then the offers that you find directly on the mainpage.

</comment><comment><USER28> I have personally had a great experience with Linode. Though depending on how much bandwidth you use it might be cheaper to go with Rackspace Cloud over Slicehost/Linode.

<comment><AUTHOR> Bandwidth and memory usage is something I don't know how to find out myself. I have asked prgmr support about my bandwidth usage, but again no response. How do you estimate yours?

<comment><USER29> I like vnstat: <http://humdi.net/vnstat/>. It's easy to set up and has almost zero overhead. I have it send me an email from cron estimating monthly usage once a week. It won't break things down per-process, per-port, or per-remote- host, but I haven't found a nice tool to do that yet.

<comment><AUTHOR> Thanks, that looks useful. I will give it a go.

</comment></comment></comment></comment><comment><USER30> In europe: ovh, gandi, or hetzner (they will also offer VPS soon). They are cheaper than rackspace, linode, etc... , and it's not a 2 man shop.

</comment><comment><USER31> I use rimuhosting.com and rackspacecloud.com, happy with both.

<comment><USER32> I have to add my support for Rimuhosting. I have accumulated nine VPS instances with them over the last three years, and I have never, not even once, experienced an outage. We use an independent monitoring service to keep track of all of our services, so I receive alerts when a service goes down. I know this all to well because of some lingering services we had with MediaTemple for too long.

The upsides of Rimuhosting are:

* Stellar support. No really, _stellar_ support. Their techs know what they're doing, and they respond quickly * Flexible plans - choose your own distro, VPS size, even semi-dedicated plans in between shared VPS and dedicated servers. * Did I mention the support rocks.

A lot of what I know about system administration, I learned from Rimuhosting techs. Not only do they fix what needs to be fixed, but they send you a great summary of the steps they took, complete with copy/paste from the command line. It's a classic "teach a man to fish" approach to support. They've spoiled me so badly that I get a little upset when I receive terse "problem resolved" responses from other companies.

I can't say enough good things about them. They're worth every penny they charge.

<comment><USER33> I'm also a Rimuhosting customer, with a single VPS I've been gradually scaling up -- I've had very smooth sailing in terms of reliability over the past few years, they offer a discount if you participate in any OSS development, and the techs have been a great help in resolving the questions I've had while getting everything set up how I want it.

I've also noticed that periodically when I'm researching server setup-related questions, Google points me to their resource pages, which seems like a good sign.

</comment></comment></comment><comment><USER34> Try chunkhost.com , I currently pay $9.50 for a 512MB/20GB VPS

<comment><USER35> I'm seeing $19.00/month on the site, how'd you manage that?

<comment><USER13> If you sign up during beta, they give you a lifetime discount (chosen randomly) when you start paying. When I did it, the max discount was 30%, but they might have increased the discount since then.

</comment></comment></comment><comment><USER24> _I have managed to get my server up again now and also got a response from prgmr._

What was the issue?

<comment><AUTHOR> Only thing I know is that the server went down and initially I could not access any of my sites, including root domains. Then the server seemed to have come back to life. I could ssh in and see my primary domain, but the subdomains still weren't working. That turned out to be some Apache issue, which I resolved by restarting it.

I got a reply from support at prgmr but haven't been able to trace down what exactly happened.

<comment><USER24> It seems you have over-estimated your system administration skills and expect prgmr to compensate, even though that's clearly not the business they're in. Frankly, this thread reflects pretty badly on your ability to take personal responsibility for your business.

</comment></comment></comment><comment><USER36> I have used RimuHosting, VPS.net, Slicehost and EC2.

Slicehost and RimuHosting by far have the best customer-service experience. I really liked the _idea_ of slicehost, but unfortunately in practice trying to scale up a 4GB instance to an 8GB instance to handle load isn't fast-enough to actually be a great production strategy -- your VM contents are effectively copied to another machine and booted in it's place, so if you have a ton of small files, that copy operation will take a long time, especially if the server is getting hammered.

Also, you cannot scale beyond the 15.5GB instance because it is 1 physical server. So if you have potential massive growth down the road, you'll have to look into load-balancing.

As for cost, I disagree with the few that said Slicehost was expensive -- in our experience it was a lot cheaper than EC2 -- go ahead and price out comparable VM's on EC2 _and_ don't forget the bandwidth, that is where they get you.

Unfortunately we didn't stay with Slicehost because of 2 VM host failures in 2 months. About 3-4hours of complete outages both times and 1 time our VM failed for no reason... it was really frustrating to have all these mysterious issues and it seemed there was no "fix" for these things... they were just god's will or something, who knows.

We moved to VPS.net next and it was great, for 12hrs... in the following week we had 2 mysterious complete failures of our VPS (tried, 4, 8 and 12-node configurations IIRC) and each time the 1st tier support would respond with "Our server admins are looking into it and will get back to your shortly", and then another 3 hours of down time before a useless "Ok, your VM has been restarted" response... that's it... no identification as to why the failure occurred so often or what was going on.

Again, another great idea "in theory" that just ended up sucking for us. However, if you need massive scalability, VPS.net will let you scale your individual VMs up to like 64 "nodes" -- which comes out to some insanely large machine.

To their credit, I got tweets and emails from the CEO and head of CS to help me after I blogged about the experience... but it is one of those "thanks but no thanks" situations...I didn't want to keep doing that dance with failing servers.

I eventually ended up BACK on RimuHosting -- 2 years prior we had left to try and find a more easily scalable VPS platform after no downtime on Rimu. We got dazzled by these other AJAX-enabled management sites and so on... Rimu has a very simple/ugly web interface, but an incredibly responsive team of very very smart people all dedicated to server stuff... and in the end we just couldn't come close to replacing that with _real world_ experience on high-load sites (Dugg, slashdotted, etc.)

That being said, there are a lot of options Rimu doesn't advertise on their site well that make it one of the cheapest hosting solutions out there... like going over 2TB of bandwidth a month in their Dallas center only costs $0.10/GB -- Amazon is $0.15/GB

They can also scale you up to an 8-core, 72GB monster dedicated server on the high end for $1k/month -- price that same thing out on EC2 with the same allocated bandwidth and it's like $1700 even with the reserved instances.

Then you throw the really responsive customer support that is willing to do almost anything reasonable for you for free (including configing/installing software, etc.) and even though they have no sexy AJAX on their site, my life is a lot easier hosting with them.

We do all our serving through them, even stuff we have to scale over time with growing VPS accounts -- if you need custom setups that aren't on their site, just ask. They'll likely toss it together for you.

\--- Hope that helps, I know you have a lot of feedback to read.

<comment><USER37> I completely agree with your point about Slicehost resizing not being a great production strategy.

Our app servers (on Slicehost) were getting hammered yesterday, so I issue a resize (1GB -> 2GB). After about 20 minutes it gets to about 97% and then stalls. In a panic I log into the server and kill all the apache2 instances. I look at the Slicehost console again and it says 40%. It actually started over. By the time it actually restarted with the resized slice it was over an hour later. The traffic was already gone by the time the resize finished.

</comment></comment><comment><USER38> I've used linode for a short while and am still using slicehost. If your apps are disk I/O heavy, go for slicehost, if you're looking for more RAM, linode may be a good choice. My stuff was not CPU heavy at the time, so I have no comments on that, but linode seemed pretty fast.

I left linode because I kept getting emails from linode support about disk I/O, and that was for an initial import of a 50GB postgres db.

After that, I kept getting emails from them once in a while, when my app was getting disk I/O busy. I can't run a business with these worries in my mind.

Slicehost is and has been working great for me, and they have great support.

<comment><USER39> Is this the email that states:

 _This is not meant as a warning or a representation that you are misusing your resources. We encourage you to modify the thresholds based on your own individual needs._

You bailed out because of that?

<comment><USER38> No. Its emails from human beings. Perhaps I got bad nodes... but I had just short of 10 slices if my memory serves me right

</comment></comment></comment><comment><USER40> Storm on Demand (by LiquidWeb) and GigeNet are pretty good. Storm starts at $50/mo for dedicated CPU core and 2GB, GigeNet starts at $10/mo for 384MB and shared CPU

</comment><comment><USER41> I have been using Fivebean (<http://fivebean.com/>) and had a great experience so far.

</comment><comment><USER42> My vote is for ARP Networks.

<http://www.arpnetworks.com/>

They officially support FreeBSD and OpenBSD as well as various Linux distributions, and since their VPSs are KVM rather than Xen-based you don't have to worry about building or running a Xen kernel inside your VM. In my experience they're very reliable.

</comment><comment><USER43> Zerigo is another decent host. You can go with their hourly plan for 2 cents per hour for a VPS with 256 MB of RAM.

</comment><comment><USER44> I've had a Linode 360 since Jan 2010 and love it! rock solid performance, create customer service and community.

<comment><USER16> I used to use Linode, and I agree, it has a great community. But for the cost of 1x Linode, I can purchase 2x Prgmr nodes

</comment></comment><comment><USER45> I had a good experience with Prgmr for the most part (dirt cheap memory for the price), and was reliable while I was using it. My only gripe was that IO performance was terrible, to the point that using ls in bash was annoying.

I switched to Linode and have been very happy with it.

</comment><comment><USER46> We are using QuickWeb VPS, cheap yet excellent uptime and support. i'm running my site with them for 6 months now and going great.

site is www.quickweb.co.nz/wht for promos packages! they have server at softlayer so very stable network.

</comment><comment><USER47> I had a great experience with DynDNS VPS. Reliable and fast.

</comment><comment><USER48> I have had good experience with dreamhost: <http://www.dreamhost.com/hosting- vps.html>

<comment><USER49> I've had a few small sites (sub-50k uniques) hosted on Dreamhost for the past year. They go down. Alot. And the support is very slow to respond.

<comment><USER50> I've been a dreamhost customer since 2004 and receive near-instantaneous support (my e-mails are responded to within 15 minutes). I don't host anything mission critical there, but their shared hosting is pretty decent for a few PHP + Ruby sites I serve up.

</comment></comment></comment><comment><USER51> Linode</comment>


LinkedIn Neutered Rapportive Today - <AUTHOR> http://ahamlett.com/blog.html#linkedin-neutered-rapportive-today

<comment><USER2> I have another theory... It's served its purpose and people are onto them...

Let me explain. A while back 'friends' started popping up in my suggestions list. I noticed that many of these were 'private' email addresses, ie email addresses i've used to sign up for a service, and were used only there, and nowhere else. I do not use these addresses to send email, they're only used to receive emails (newsletters etc).

When i approached linkedin about this, they claimed i'd given them permission to access my gmail contacts on the mobile app. When i asked them for an exact date, they could not provide me with one, but responded with inStructions on how to remove these contacts.

I'm 99% sure that i have never done so, i'm very very careful about this (manage a list of 15k+ members of an organization i run). The answer also did not make sense, because many of these suggested contacts should not show up in my gmail address book (receive-only, no interaction).

My theory is that they've used the rapportive lookups to build a database of email addresses checked by my rapportive addon whenever i happen to click on an address in my inbox.

If you search online, you'll find many people complaining about linkedin 'inviting' contacts without the initiative being taken by the user, and others who wonder how linkedin knew that they were acquainted with some of the suggested contacts.

I must admit i have not bothered to gather conclusive proof of this, but like i said, many of the suggested contacts were single-use (generated on the fly on my own domain, anyone not found is forwarded to my admin account).

I have disabled the rapportive addon after their response because i no longer trust it. Did miss it for a while, but with the changes outlined in this article, that's no longer an issue either.

Ps i was one of the first 150k members of linkedin. Used to think they were the only decent social network in town. This trust has been lost forever. Guess that's hat happens when you go public. Growth required at all costs.

<comment><USER3> I deleted contacts from the site that I apparently imported in 2006, first, it was nearly impossible to find the page that allows you to delete them, and second, they are still showing up as suggestions with the reason of "you imported this contact."

They really are gobbling up everything they can, and even if they say they are removing stuff, they never will.

</comment></comment><comment><USER4> I bet one could make a successful business by just noticing aqcuisitions as they happen and immediately starting work to clone the acquired product exactly. This pattern of shutting down the acquired service and leaving all their users hanging is so common that it seems like it should be easy to capitalize on.

In this case, you would have had two years to see this coming (and if you check out the comment thread from when the acquisition happened, you'll notice that everybody did see it coming) and do the rather straightforward development work to get a new version up and running. Then all you'd need do is drop a link in this thread right here today to your feature-wise Rapportive clone that actually still works as advertised, and you'd have 326,000 new customers tomorrow.

<comment><USER5> You could do much the same thing looking at startups which are 2 years old. The post-acquisition survival rate of a product is not that much better than the pre-acquisition survival rate - getting bought by a larger company does not mean that they won't eventually decide the same thing you would have found on your own: the product does not bring in enough money to pay for itself.

Acquisitions are a risky investment. You buy a company and it might turn out to be worthless.

</comment></comment><comment><USER6> The post offers Full Contact ([http://www.fullcontact.com/](http://www.fullcontact.com/)) and Vibe ([http://vibeapp.co/](http://vibeapp.co/)) as alternatives. Does anyone have any other recommendations?

<comment><AUTHOR> [https://ark.com/mail](https://ark.com/mail)

[http://www.connect6.com/Home/ChromeExtensionDetail](http://www.connect6.com/Home/ChromeExtensionDetail)

[http://falcon.io/](http://falcon.io/) ([https://chrome.google.com/webstore/detail/falcon/fcbcnboheai...](https://chrome.google.com/webstore/detail/falcon/fcbcnboheaijdfnkchlbeilgmaebdogd))

<comment><USER7> Are these plugins sending every visited web page to their servers, or is this done when the user clicks/hovers on a name/profile?

<comment><USER8> I can't talk about others. I'm the founder of Vibe. We take privacy very seriously. We don't read anything more than we should. All we do is 'mark' the emails right in your inbox/page. When you hover/click an email, we send just that email to the server and returns the data,

</comment></comment></comment></comment><comment><USER9> Curious, is there an API that I can send an email string to and get all of their social profiles listed? For example, I put in email@example.com and get facebook.com/mike, twitter.com/mikeguy1, etc

EDIT: Looks like FullContact does this: [http://www.fullcontact.com/developer/docs/person/](http://www.fullcontact.com/developer/docs/person/)

<comment><USER10> Every one of our customers who has access to our API demonstrates that the Ark (YC W12) API has much more data than FullContact's API. Send me an email and I'll hook you up.

<comment><USER11> Kipp from FullContact here. Interestingly, we've had several customers put us head-to-head against Ark in match tests using the same source data sets, and all have wound up going with us due to much higher match rates & better data returned.

We always encourage potential customers to try other services. With over 500 paying customers of the FullContact API from some of the internet's largest SaaS providers, we're pretty confident in the quality and quantity of our data.

Bottom line is, none of us will have the same amount of data as Rapportive, because Rapportive had direct access to LinkedIn's data warehouse. (And as we all know, LinkedIn has never been shy about hoovering up data from anywhere it can get it.)

</comment></comment></comment><comment><USER12> Obviously, there are many reasons behind why LinkedIn took this decision (and indeed made the original purchase decision) and they may be using the technology internally at LinkedIn in other ways. And, I am a fan of LinkedIn and use it everyday.

Having said that, I am baffled by this. While I appreciate that there is a battle own an individual's primary social network and spend more time/content on LinkedIn vs say Facebook. But, IMHO, wouldn't it have been more prudent to build up the functionality of the LinkedIn component while still offering the functionality that 300k plus users obviously wanted anyway? While this will probably be re-offered (it now looks that it is) as a LinkedIn only plugin, LinkedIn are limiting their reach to only LinkedIn users.

Had they continued to build the multi-network capabilities and brought the marketing/reach of LinkedIn to the product, I dare say they could have owned a significant portion of all GMail users and likely been no.1 in the space. Controlling that (and offering preferential treatment to LinkedIn) what definitely add value.

Who knows the end result, but it looks like a missed opportunity that someone else will come in and take.

<comment><USER13> The original purchase decision I find fantastically brilliant -- and I hate that word.

When LI bought Rapportive they bought access to my entire actual social/contact graph, whether I wanted them to have it or not. I've never done the "upload your contacts so we can spam them" on LinkedIn, but because of Rapportive: every one I email, and everyone who emails me, LI knows about. They know the frequency of my interaction, all in exchange for giving me a bit of information about the people who write me.

They must have reached a point where either they have mined it sufficiently (and new installs are low) to not worry anymore about crippling it, and/or they realize that the bulk of the data that users care about is the LI data anyway (true for me), and they'd like to try and continue productizing it along that vector -- rather than simply living off the data feed it produces.

</comment></comment><comment><USER14> Founder of Streak here - we've actually thought of adding this functionality to Streak but as others have mentioned the hard part is finding a good data source. Give our current infrastructure, we could build a rapportive replacement pretty quick if we could find a good partner on the data side....

</comment><comment><USER15> Gmail has ~500mn users. Rapportive is possibly the most used Gmail extension out there for Chrome & Firefox. By deduction, Rapportive is the largest business/product built on top of Gmail. With only 326,000 active users isn't it a worry that products on top of Gmail are difficult to market?

Is this a reason of fear for companies like Streak and Yesware who bet heavily on the Gmail extension? (Technically their market size would have been 500mn but since the largest player only has 326k users, the total addressable market becomes really small). Something seems off since YC itself has many companies that are following this model.

Anything I'm missing out here?

<comment><USER14> Founder of Streak here, we aren't too worried that Rapportive _only_ had 326K users. First that number is a pretty low estimate. Thats their current _active_ weekly users. This is after they let their product die slowly after a few years. This also doesn't count all the installs that didn't use to be from the chrome webstore nor does it count safari and IE.

Even if those numbers were accurate, I don't think products like Rapportive/Streak/Yesware are limited at all by Gmail. If anything being in Gmail helps a ton with engagement. The reason the install numbers seem low is the same reason any startup may have low install numbers - their small, they need to perfect their product/distribution/sales/etc. I don't think it has anything to do with the fact that they are in gmail.

<comment><USER16> The fact a user might have installed from somewhere other than the webstore, or if they're on Safari or IE, is entirely irrelevant - the 326k is _active_ users, eg people who have accessed the Rapportive API in the last week. That's _everyone_ using Rapportive regardless of where they installed from or which browser they're in.

Whether or not Rapportive is _limited_ by GMail is a good question. Getting 0.065% market penetration after 4 years would worry me, but doesn't mean they weren't making giant piles of cash. You could easily run a profitable business with 326k users even if only a _tiny_ proportion are paying something. The important thing to take away from Rapportive's numbers is that a huge market doesn't automatically translate in to a huge market _of available customers for your product_. There might only be a tiny proportion who want what you're offering. Fortunately that doesn't actually matter if you can run your business profitably anyway.

</comment></comment></comment><comment><USER17> It looks like only the front end has changed. I have a Python library (github.com/jordan-wright/rapportive) that directly calls the backend API, and it still works fine.

</comment><comment><USER18> Its beyond me why anyone would want to sign up for Linked In, let alone have them trawl your contacts etc.

I don't understand the pressure people feel they're under to sign up. Much like Google, you're the product.

</comment><comment><USER15> It still shows all the user profiles to me. Has this been rolled out only to selective groups?

</comment><comment><USER19> 326k users? that's... shockingly low

<comment><USER3> Think about it this way, 326k users who've handed over their address books and every incoming/outgoing email address they've used. Their power/value isn't in their users, but their corpus of identities.

</comment></comment><comment><USER20> I dunno... I've literally never used any of those buttons, and the new deeper LinkedIn stuff they expose seems useful.

</comment><comment><USER21> Still works for me...

<comment><USER22> me too, but I'm in Kenya. Maybe they just did US users first?</comment></comment>


How the algorithm for coloring the iTunes song list works - <AUTHOR> http://stackoverflow.com/questions/13637892/how-does-the-algorithm-to-color-the-song-list-in-itunes-11-work

<comment><USER2> You may find this post by Wade Cosgrove of Panic interesting

[http://www.panic.com/blog/2012/12/itunes-11-and- colors/](http://www.panic.com/blog/2012/12/itunes-11-and-colors/)

<comment><USER3> I've taken their algorithm and ported it over to JavaScript a while ago: [https://github.com/lukashed/itunes- colors](https://github.com/lukashed/itunes-colors)

</comment></comment><comment><USER4> How I'd do it:

1\. pick 500 random points from the album art

2\. cluster them in 10 classes (k-means is an old, well understood algorithm)

3\. pick the median point of dominant class for the bg

4\. pick a contrasting color from a different cluster for the text

<comment><USER5> I wouldn't pick the median of the dominant class for the BG, but rather from the borders (even though this could work)

for 4 you want that, picking from one of the colors of the album or a contrasting color if this is not available from the album

</comment></comment><comment><USER6> How I'd do it: Use Embedly! Our extract API returns the dominant colors of all the images it finds: [http://embed.ly/docs/extract/api/response#extract- images](http://embed.ly/docs/extract/api/response#extract-images)

For those that are curious, what happens under the covers is a random sampling of the image followed by a conversion to LAB color space (even better than YUV in terms of human perception) and finally some K-means clustering.

To see this in action, check out [http://tightro.pe](http://tightro.pe).

</comment><comment><USER7> They also use a similar trick based on the icon colors for the iTunes ios app update feature (for those of us who still like plugging our phone or tablet into the USB cable). In the latest iTunes, when you click on an app in icon mode to find out what's new in the update, the background reflects the colors of the icon, which can make for some odd combinations...

</comment><comment><USER8> I've done something similar in one of my older iOS apps however I used a 8x8 sample. Still an interesting read.

</comment><comment><USER9> So iTunes changes colour for every album you click on? This sounds awful..

<comment><USER10> Not really. It changes the background colour of the album view , see [http://thomaspark.me/wordpress/wp- content/uploads/2012/12/it...](http://thomaspark.me/wordpress/wp- content/uploads/2012/12/itunes11.png)</comment></comment>


Prenda Saga Update: John Steele Pleads Guilty, Admits Entire Scheme - <AUTHOR> https://www.popehat.com/2017/03/06/prenda-saga-update-john-steele-pleads-guilty-admits-entire-scheme/ <comment><USER3> From Wikipedia: Prenda Law, also known as Steele | Hansmeier PLLP and Anti- Piracy Law Group, was a Chicago-based law firm that ostensibly operated by undertaking litigation against copyright infringement, but was later characterized by the United States District Court for Central California in a May 2013 ruling as a "porno-trolling collective" whose business model "relies on deception", and which resembled most closely a conspiracy and racketeering enterprise,referring in the judgment to RICO, the United States Federal anti- racketeering law.

<comment><USER4> Previously at Popehat: [https://duckduckgo.com/?q=prenda+site%3Apopehat.com&t=ftas&i...](https://duckduckgo.com/?q=prenda+site%3Apopehat.com&t=ftas&ia=web)

HN: [https://hn.algolia.com/?q=prenda](https://hn.algolia.com/?q=prenda) (much of it Popehat)

<comment><USER5> The first link in TFA is to popehat's prenda-law case which is 4 pages of the sordid history of these garbage people back to early March 2013: [https://www.popehat.com/tag/prenda-law/](https://www.popehat.com/tag/prenda- law/)

</comment></comment></comment><comment><USER6> So nice to hear they are finally getting down to the time behind bars stage. I hope (although don't particularly expect) that this will have a chilling effect on other unscrupulous lawyers who shake down people using fraudulent constructs.

<comment><USER7> Why were they (Prenda) caught? Did they try shake down someone powerful?

<comment><USER8> IIRC, a John Doe defendant got a young aggressive lawyer hoping to make a name for himself to defend him pro bono. The lawyer started digging hard, refused to just accept a dismissal, and went on a counter-attack that eventually dug up enough for other parties to join in.

Huh. Phrased like that, it sounds like a John Grisham novel.

<comment><USER9> Which is a great story but it's a little terrifying if it's "System working as intended"

It seem to have only come this far because they doubled down on some already outlandishly scumbag behaviour. If they'd just stopped this might have all blown over for them.

<comment><USER10> It does kind of make you wonder how frequent tactics like "strawman defendant" or "made up hacking claims to get discovery" are used by other firms - <USER2> it sure seems like these guys got caught because they kept up these sorts of tactics relentlessly.

<comment><USER11> From that perspective, all the publicity surrounding this firm might actually make the problems associated with them worse. Other unscrupulous lawyers might study this and learn which specific mistakes not to make, rather than learning to avoid this model altogether.

<comment><USER12> Their primary error was arrogance, assuming that they were smarter than anyone challenging them even in the face of evidence to the contrary. I don't think there is a cure for that, short of US Marshals at your door with a warrant and handcuffs.

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> I paid up years ago because I had an open AP and didn't want to get dragged into court. Thoughts on how to get any of the ~$2000 back?

<comment><USER14> It's unlikely, but you could do it. After going through the trouble of bringing a suit -- once you have the judgment against them, you'd have to get in line behind the other creditors. :(

<comment><USER15> Unless they are in bankruptcy it's a free for all once judgement is granted. There is no line and can be collected on by any legal means. I used to work for a lawyer who primarily did collections of business accounts, some of which we personally garunteed, depending on the age/credit of the business.

<comment><USER14> I guess I leaped ahead and assumed that everyone else with judgments would quickly force them into bankruptcy.

</comment></comment></comment></comment><comment><USER16> _the deal also contemplates that Steele will cooperate — become a rat, if you want to be unkind_

A real-life prisoner's dilemma unfolding.

Couldn't have happened to a couple of nicer guys. /s

<comment><USER17> I know. I thought this meant that this Schadenfreude Fountain had dried up. Looks like instead the story is moving into Act II.

</comment></comment><comment><USER18> _> Upon entry of judgment after his sentencing, John Steele will be a convicted felon with a federal fraud conviction. His career as a lawyer — or, more generally, as a gainfully employed person — is over._

While I share everyone's hatred of the tactics these people have employed over the years, I also hate that felons in the US are given such a hard time _after_ they have served their sentences. People make mistakes and should be allowed to move on from them. A couple of years in federal prison shouldn't carry with it a lifetime of financial despair.

<comment><USER19> I recognise that what I'm going to say next will be very unpopular on HN. Harsh punishments work exceedingly well as a deterrent. Harsh punishments make other prospective law breakers think twice before harming innocent people. I can agree with the notion that petty criminals deserve second chances, but what this guy did isn't petty at all. If even one innocent person is spared the ordeal of victimization, then harsh punishments are well worth the costs.

<comment><USER20> > Harsh punishments work exceedingly well as a deterrent.

That is an _enormous_ claim and there are reams of research on this topic - your assertion is _not_ supported. It doesn't even stand up to the most cursory scrutiny - if it were true, then societies and states with harsh punishment would have lower crime - and this is very, very, very obviously not the case.

[https://www.nij.gov/five- things/pages/deterrence.aspx](https://www.nij.gov/five- things/pages/deterrence.aspx)

[https://www.psychologytoday.com/blog/ethics-in- question/2015...](https://www.psychologytoday.com/blog/ethics-in- question/201509/harsh-justice)

[http://www.sentencingproject.org/wp- content/uploads/2016/01/...](http://www.sentencingproject.org/wp- content/uploads/2016/01/Deterrence-in-Criminal-Justice.pdf)

[https://undark.org/article/deterrence-punishments-dont- reduc...](https://undark.org/article/deterrence-punishments-dont-reduce- crime/)

tl;dr Social norms, accepted membership in 'legal' society and education are the best ways to deter crime, followed by _certainty_ of punishment - not its _severity._ The reason people commit crimes is, in rough order, usually out of desperation, social norms that lead them to believe the prohibited act _should_ be acceptable or ignorance of the knowledge that the prohibited act is illegal, for greed, and, lastly, then for the sheer thrill of it. Severity of punishment doesn't act as a good deterrent for _any_ of those categories - _certainty_ of punishment acts as a deterrent for the latter two, but for the first two, desperation and ignorance, are fairly well inoculated against rational counsel.

Before you start arguing the philosophy regarding merits of punishment v. deterrence, I am going to put it bluntly: the claim at issue - that severity of punishment is a deterrent for crime - is an _evidentiary_ claim and it _cannot_ be proven from first principle. It _must_ be supported evidence, and as far as I can tell, the numbers are in, and they do not appear to be on your side. Unless you have studies that support your conclusion, the philosophy is totally irrelevant.

<comment><USER19> You've clearly spent some time looking up links to support your cause, so it's unfortunate that you've cherry picked only the ones that support your beliefs. Here are some others that say the opposite.

"Sentence Enhancements Reduce Crime"

[http://www.nber.org/digest/oct98/w6484.html](http://www.nber.org/digest/oct98/w6484.html)

"Longer prison sentences deter re-offending, study shows"

[http://www.telegraph.co.uk/news/uknews/law-and- order/8504923...](http://www.telegraph.co.uk/news/uknews/law-and- order/8504923/Longer-prison-sentences-deter-re-offending-study-shows.html)

Longer sentences deter crime up to a point

[http://www.economist.com/blogs/freeexchange/2016/03/criminal...](http://www.economist.com/blogs/freeexchange/2016/03/criminal- justice)

"Longer prison terms really do cut crime, study shows"

[https://www.theguardian.com/law/2012/jul/07/longer-prison- se...](https://www.theguardian.com/law/2012/jul/07/longer-prison-sentences- cut-crime)

"the average add-on gun law results in a roughly 5 percent decline in gun robberies within the first three years"

[https://www.law.upenn.edu/cf/faculty/dabrams/workingpapers/D...](https://www.law.upenn.edu/cf/faculty/dabrams/workingpapers/Deterrence12312011.pdf)

Note also that 2 of the 4 links you provided are comparing certainty-of- punishment vs harshness-of-punishment. This is a false dichotomy. There's no reason you can't have both. Having harsh punishment doesn't reduce the certainty of punishment in any way.

Is there a lot of contentious debate on this topic, and a lack of consensus on this topic? Of course. But given the lack of clear answers, I'd much rather err on the side of preventing victimization.

<comment><USER20> > But given the lack of clear answers, I'd much rather err on the side of > preventing victimization.

In America, we have historically low crime rates and historically low violent crime rates. Simultaneously, we have an insane crisis of prison overpopulation that leads into multi-generational cycles of poverty.

I too would rather err on the side of preventing victimization.

Right now, from where I sit, over-criminalization is a bigger problem than violent crime.

</comment></comment></comment></comment></comment><comment><USER21> How did these guys think they could keep the con running for so long? Being lawyers, you'd think they'd have some common sense.

<comment><USER22> Being lawyers they have enough common sense to do the math and figure out that the cost of being caught would never outweigh the profits. At that point the only incentive is to make as much money as possible for as long as you can.

<comment><USER23> I don't know about that - is ~10 years in federal prison worth a few million dollars (after splitting it between Hansmeier and other co-conspirators)? Also, they're not going to get to keep the money.

As lawyers, I'm betting they could make the same amount of money in the same timeframe practicing legitimately (or, knowing how these idiots operate, "semi-legitimately"), but without the federal prison time.

<comment><USER24> Nobody ever thinks they will get caught. "I'm too clever for that" is basic human reasoning. Some are, some aren't. The real question is, are you willing to risk it?

</comment></comment></comment></comment><comment><USER25> There's something that I don't understand in this case(I'm not American): to my knowledge in the US the Police can trap you by proposing to sell you drugs or bomb making ingredients and then arrest you.

The Prenda modus operandi doesn't seems much different than what the US Police is doing, so why do they face prison ? Don't get me wrong I think they deserve to go behind bars, but I would have thought what they did was legal in the US.

<comment><USER26> If a private citizen tries to sell drugs or bomb ingredients, then extorts the buyer by threatening to tell the police, I think that would probably be illegal as well.

A lot of the trouble here is _how_ they went about this scheme, not merely the basic idea of what they did. They repeatedly lied to defendants and courts about their operations. Some of the materials they sued over were things they didn't actually have copyright for, but they used false documents to claim that they did. Those false documents involved forged signatures and even invented a person who didn't exist. They even filed defamation lawsuits against a person whose signatures they forged, for daring to say that they hadn't signed this stuff.

Wikipedia has a huge list of the crazy stuff they've done: [https://en.wikipedia.org/wiki/Prenda_Law](https://en.wikipedia.org/wiki/Prenda_Law)

So, it's not just about trying to entrap people by "pirating" their own content. It's about massive fraud and abuse along the way.

</comment></comment><comment><USER27> By coincidence, I was viewing this today:

[http://bestfunny- videos.com/video.php?vid=ObZDipKRH0c&13-558...](http://bestfunny- videos.com/video.php?vid=ObZDipKRH0c&13-55859-Paul-Hansmeier,-Esq.-v.-John- Doe)

My understanding of court procedure is limited, but it is pretty clear that the appellants are making some pretty incredible decisions that surprise the judges, including insisting that the proceedings are a criminal case with the possible outcome that the appellants would face life imprisonment for criminal contempt (17:11 - 17:45). I gather this is high comedy for lawyers.

<comment><USER28> That's a great clip

Fun fact: Tallman, the judge in the middle with the great stories - is one of three judges who sit on the FISA court

<comment><USER23> The judge in the middle is Judge Pregerson.

<comment><USER29> So is he the one on the FISA court, or is Tallman? I mean I'm impressed with the whole panel and it's a hilarious proceeding, but I'd be interested to see this point cleared up, and Wikipedia's list [0] doesn't mention either. (I don't really know where else to look.)

[0] [https://en.m.wikipedia.org/wiki/United_States_Foreign_Intell...](https://en.m.wikipedia.org/wiki/United_States_Foreign_Intelligence_Surveillance_Court)

<comment><USER23> Tallman is on the appellate level of the FISA court [1], which is why his name doesn't appear on that Wikipedia page. Pregerson isn't on the FISA court at any level.

[1] [http://www.fisc.uscourts.gov/fiscr_membership](http://www.fisc.uscourts.gov/fiscr_membership)

</comment></comment></comment></comment></comment><comment><USER30> they are truly scummy esp with their ADA shakedowns.

ADA was a great way for people with disabilities to get equal access but now I fear its future due to the constant abuse by scumbags like Prenda and others.

<comment><USER31> Could you clarify how these shakedowns are performed? Are the victims non compliant with some law?

<comment><USER11> Typically the victims are low-margin businesses operating in old buildings that were designed long before ADA existed. These businesses may never have served a wheelchair-bound customer before, or if they did it was an _ad hoc_ process. In other words, physically lifting a wheelchair over some stairs or just bringing the customer's purchases outside. That's fine for reasonable customers, but the wheelchair people involved in these lawsuits do not behave in reasonable fashion.

<comment><USER32> > Typically the victims are low-margin businesses operating in old buildings > that were designed long before ADA exited

That's not really true. Under the ADA, buildings built before 1992 are generally not required to perform renovations for compliance unless they are already performing renovations, at which point they are required to adherence to the requirements. And the amount that they are required to spend on compliance is a function of the amount that they are spending on the other renovations.

It's actually more complicated than that, because there are two sets of requirements (1993 and 2010), with different safe harbor provisions for each, and renovations to old buildings may or may not invalidate the 1993 safe harbor, but that's the general idea.

</comment></comment></comment></comment><comment><USER33> These types of operations abound in Germany, too. Arguably, adult movie producers make more money via "Abmahnanwälte" than via regular distribution.

</comment><comment><USER34> So what would be the Nash equilibrium in this instance?

</comment><comment><USER35> Anyone who has watched Law & Order knows making a deal first is best. I also bet once he is out of jail he will write a book and get hired on some TV channel as an expert and probably make more money than he did as a crook.

<comment><USER36> Did I miss the part in Law & Order where it says this is an accurate representation of US law, the court system etc?</comment></comment>


Unpatched Exploit Lets You Clone Key Fobs and Open Subaru Cars - <AUTHOR> https://www.bleepingcomputer.com/news/security/unpatched-exploit-lets-you-clone-key-fobs-and-open-subaru-cars/ <comment><USER2> > Bleeping Computer is not sharing the link in this article.

Security through ( _trivial_ no less) obscurity is no security at all as they say, so...

[https://github.com/tomwimmenhove/subarufobrob](https://github.com/tomwimmenhove/subarufobrob)

<comment><USER3> Yep, and BTW that link is on the Youtube page Bleeping Computer DID link to...

[https://www.youtube.com/watch?v=ewMZCxi8l8A](https://www.youtube.com/watch?v=ewMZCxi8l8A)

As a side note, it seems from the video that the "rolling code" is 4 digits, so maybe the device could be used also for "brute-forcing" in a "reasonable" time even if you haven't recorded "last code".</comment></comment>