Sharding Pinterest: How we scaled our MySQL fleet - <AUTHOR> https://engineering.pinterest.com/blog/sharding-pinterest-how-we-scaled-our-mysql-fleet/ <comment><USER2> As I read on their blog, Instagram uses similar logic with PostgreSQL. If anyone interested, it's accessible on [http://instagram- engineering.tumblr.com/post/10853187575/sha...](http://instagram- engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram)

<comment><USER3> That is an excellent compare & contrast. The both use 64 bit IDs.

Instagram: 13 bit shard ID, 51 bit "local" ID consisting of 41 bit timestamp in milliseconds and 10 bit sub-millisecond ID. So this scheme supports 1024 IDs per millisecond per shard for 41 years, and 8192 shards.

Pinterest: 16 bit shard ID, 10 bit type ID(?), 36 bit local ID, 2 bits reserved. This supports 68 billion objects per shard and 65K shards, but does not represent time. So you need another field / more storage for that. Also notable is the large 10 bit type ID field which seems to be only actually used for a handful of values, leading to a large chunk of bits that don't change across IDs.

In short, Instagram's scheme is more efficient largely due to the leverage of timestamps in the ID instead of type information.

</comment></comment><comment><USER4> It seems mysql (and hopefully postgresql sometime soon) with custom sharding logic in the app layer still hits the sweet spot for scaling to the order of 100M users. With some thoughts going into designing an appropriate data model and sharding logic, certain join queries can be delegated to the databases too.

<comment><USER5> Hi, at Zalando, we are scaling all of our core businesses with PostgreSQL. Depending on your dataset, it can be fairly easy to shard your data for a horizontal scale-out (think of independent customer datasets). We have lots of databases that we scale horizontally to much bigger numbers. But, we also developed several tools that makes working with shards mostly transparent. Did not find a better source but one way we use PostgreSQL can be seen in detail in the following slides: [http://gotocon.com/berlin-2013/presentation/Why%20Zalando%20...](http://gotocon.com/berlin-2013/presentation/Why%20Zalando%20trusts%20in%20PostgreSQL)

<comment><USER6> 5 TB easily fits on a single Oracle instance on a single host.

<comment><USER7> It also easily fits in a Postgres instance on a single host, if you only look at "can store X amount of data".

And even if Postgres is slower, for the money you save in license costs you can buy a few beefy nodes extra.

<comment><USER6> And your engineers work for free.

<comment><USER8> Free, no. Less money than Oracle DBAs? (Yes.

</comment></comment></comment></comment></comment></comment><comment><USER9> > To edit a Pin, we read-modify-write the JSON under a MySQL transaction:

I've seen a different approach where you keep a version number on the row, do your read, modify in memory on the app server, then do your write like this

        UPDATE db03429.pins SET blob='<modified blob>' WHERE local_id=7075733 AND version=53

then look at the result and make sure that it modified one row. If it returned zero rows, you retry (or show a failure to the user, whichever is appropriate for your use case).

The reason you'd do this is so you can't ever have the row locked for a long period of time. A lot of people don't think about database scalability so _even if they know_ that the code they're writing runs while a transaction is held open, they don't care that that transaction is blocking anything else that is trying to read the row they're working on.

This can lead to row lock bloat over time, which can cause scalability / availability issues as app servers wait longer and longer to get read (or write) locks on all the rows they care about for their current request. This is mitigated a bit if you're requiring / encouraging people to read from slaves instead of master, though.

</comment><comment><USER9> > We only interact with the master in production. You never want to read/write > to a slave in production. Slaves lag, which causes strange bugs.

These can be worked through, with discipline. You probably only need a fully consistent few on a surprisingly small number of pages.

Generally you'll want to read from the master right after a user changes something (if you have a model where user clicks a button -> you go to a different page). User changes setting on a pin, clicks 'save', you render a new page showing their updated pin. This page view should probably come from the master, or else you risk the user's change not showing up, causing confusion.

Reads from slaves are fine, as long as you're not using something that was read out of a slave as an input to a database write somewhere else (which you shouldn't be doing anyway!). If you render a page that lets a user change their profile (say site.com/edit_profile), the user data can come from a slave, but if you take _all_ the field values and blindly write those into the master, that's where you run into "time travel" bugs. You just need to find out what the user changed and only make those changes in the master.

</comment><comment><USER10> Can someone explain to my why people are still recommending mysql over postgres? This is a serious question, it just seems that Postgres has more features and I cant think of any good reasons mysql would scale any differently other than it has been along a little longer (so there are more blog posts + experienced engineers) ?

"MySQL is mature, stable and it just works. Not only do we use it, but it's also used by plenty of other companies pushing even bigger scale. MySQL supports our need for ordering data requests, selecting certain ranges of data and row-level transactions. It has a hell of a lot more features, but we don't need or use them. But, MySQL is a single box solution, hence the need to shard our data. Here's our solution:"

what about that paragraph is not true of postgres also ?

UPDATE:

This schemaless json reminds me of this friendfeed blog post from years ago:

[https://backchannel.org/blog/friendfeed-schemaless- mysql](https://backchannel.org/blog/friendfeed-schemaless-mysql)

<comment><USER11> There are features on both sides which the other database doesn't have.

With this specific workload, I think MySQL will work pretty well. Two features in particular: innodb clustered index and compression.

(I work on the MySQL team.)

<comment><USER10> thank you!

</comment></comment></comment><comment><USER12> Is it surprising that Pinterest is on MySQL? Has Postgres finally usurped the position of default DB for startups?

<comment><USER13> Came here to say the same. It makes sense that older companies are locked into their database as migrating would be too difficult like Facebook. I though Pinterest was a relatively young company and I'm surprised it chose mysql. Instagram is build on top of postgres and was founded in 2010, pinterest was founded in 2009.

<comment><USER14> Other than Oracle being a part of the equation MySQL is still an open source database and too quote the article:

"Aside: I still recommend startups avoid the fancy new stuff — try really hard to just use MySQL. Trust me. I have the scars to prove it."

In many ways, Postgres is cutting edge with it's features and capabilities. I see lots of updates from Postgres that include new SQL features (json objects) - conversely I see lots of updates from MySQL that are about durability and scalability e.g. (galera cluster, percona updates etc...)

From the use case Pinterest exposed in the article, I'm not sure what the additional features of Postgres would actually buy them?

<comment><USER15> JSON object storage for one? I'm a little confused as to why their schema relies on storing JSON data as a TEXT field in MySQL. I don't believe this is technically incorrect, but seems to negate many of the features of relational database design. Why not just store each entry as either a traditional column or use a solution that enables native JSON storage.

<comment><USER16> _> Why not just store each entry as either a traditional column_

Slow ALTER

 _> or use a solution that enables native JSON storage_

This would help if you needed to select or join on the individual columns, but if Pinterest don't need to do that, then this falls under the _" avoid the fancy new stuff"_ quote from the article.

<comment><USER15> > Slow ALTER

In practice, how often do schema migrations take place though?

<comment><USER16> Looking over the source history at $DAYJOB, we run a migration containing DDL statements on average every two days. Mostly for adding new features on small(er) metadata tables where the ALTER doesn't hurt - but occasionally on our bigger XX-GB log tables, where it takes all morning.

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> Nice article and good explanations.

But I still wonder why some people store blobs in a database rather then simple files on a file-system!?

<comment><USER18> There are a lot of things your FS is not going to give you (eg. transactions), and if you're going to have to query it anyway to get that, you might as well have your blob there too.

For "small" blobs, say <256K, your database is probably faster too. I can only recall one such study (and of only NTFS vs MSSQL), and NTFS was only advantageous at >1MB blobs: [http://research.microsoft.com/pubs/64525/tr-2006-45.pdf](http://research.microsoft.com/pubs/64525/tr-2006-45.pdf)

<comment><USER14> also operationally it's easier... you can rely on replication to backup those files instead of separate scripts that would need to work in-conjunction with the database replication.

If say you have the database with blobs - you can restore the db and everything is back up and running. If instead, you need to also restore the individual files... well - not only do you have lots of individual disk writes, you also have to make sure you still have all the files...

</comment></comment></comment><comment><USER19> Is this the guy that's going to reddit?

<comment><USER20> Very nice guy. He used to work at Azul with Cliff Click.

</comment></comment><comment><USER21> How are they going to solve the problem of querying into the data that's stored as json? E.g. trying to find all pins whose "link" is from, say, reddit. Just pull out all data and filter them through in client side? That's not gonna scale. Or having a sort of cron job that periodically picking out interested fields in new json data and store them in a separate table? <\-- this is essentially what we do in one of our projects, but curious to see how they do it, or alternatives.

<comment><USER11> In MySQL 5.7 this will be possible because there is a native JSON data type + indexing available via computed columns.

<comment><USER21> That's good to know. 5.7 seems a bit new though. Before 5.7, what are the common practice to query into json?

<comment><USER21> Not sure why I'm down voted, that was a genuine question...

</comment></comment></comment></comment><comment><USER22> PINTEREST: publish your public outgoing IPs so we can whitelist you and block the rest of amazon ec2

It is the responsible thing to do, otherwise other website bots can spoof you.

<comment><USER23> Hi there, Pinterest traffic engineer here. I've added a task to take care of this. Thanks for the suggestion!

<comment><USER22> I'm embarrassed I didn't say "please". But thank you!

</comment></comment></comment><comment><USER24> Am I missing the obvious? They're using text columns as blob stores for JSON data? How on earth do you query that in MySQL? How do you run a secondary index, on say the user_id? Is it just one gigantic instance of Elasticsearch or Lucene?

<comment><USER25> HBase? no idea, I would also like to know that one.

</comment></comment><comment><USER26> This reminds me of how FriendFeed was using MySQL to store unstructured data.

[http://backchannel.org/blog/friendfeed-schemaless- mysql](http://backchannel.org/blog/friendfeed-schemaless-mysql)

<comment><USER27> This is exactly what I was thinking

</comment></comment><comment><USER28> I'm curious to know how Pinterest or Instagram decide about what goes inside each shard? Do they shard by user ids or something else? Secondly, would like to know if a shard gets more data than other shard, how do they load balance?

</comment><comment><USER25> a minor con of this approach is that you have to add an extra layer to your application to do these operations, i.e. to abstract them. But probably the speed & other gains surpass the cons, and in such a big team, i'm sure they will easily handle such an abstraction layer.

One good idea is to open source it, so if other people can take advantage of it, they will also help you maintain it and find bugs for it.

A question: If you started this now, would you consider using Postgres-XL AFAIK it supports similar shardings, in a more transparent manner for the developers. Any thoughts on this?

</comment><comment><USER29> Can somebody help me understand this conversion?

Shard ID = (241294492511762325 >> 46) & 0xFFFF = 3429

Type ID = (241294492511762325 >> 36) & 0x3FF = 1

Local ID = (241294492511762325 >> 0) & 0xFFFFFFFFF = 7075733

<comment><USER30> bitwise operations.

241294492511762325 >> 46 means shift the binary form of number which is

      0000001101011001010000000001000000000000011010111111011110010101

to the right 46 spaces while replacing with 0s

      0000000000000000000000000000000000000000000000000000110101100101

0xFFFF is hex version of 1111111111111111 (note two missing bits he reserved for gold :D )

& 0xFFFF is doing a AND operation of the bits.

      0000110101100101
      1111111111111111
      </comment></comment><comment></comment><comment>----
      0000110101100101

so 0000110101100101 in decimal is 3429

</comment><comment><USER31> I know nothing about scaling databases but this reminds of Amazon's Dynamo DB object store but without the consistent hashing trick .

<comment><USER32> Dynamo is very complex, beyond consistent hashing. It also uses (or used) gossip and lots of tricks (hacks) to gracefully handle adding and removing nodes and distributing data to new nodes, etc. This uses a simple partitioning scheme albeit it does have its own consistency issues.

</comment></comment><comment><USER33> Isn't it unsafe to expose internal database IDs to external clients? I would have generated a second GUID for public view.

<comment><USER34> But how do you ensure that it is unique?

<comment><USER33> Twitter's Snowflake, though it's gone now. The principle is the same. [https://github.com/twitter/snowflake](https://github.com/twitter/snowflake).

</comment></comment></comment><comment><USER35> Too many configs and maintenance.

NoSQL is really better than SQL to scale your databases. It is just sad there is no one open source NoSQL db as good as Google BitTable.

<comment><USER36> _NoSQL is really better than SQL to scale your databases._

Downvoted for what is essentially your opinion presented as a fact. If you want to make such a statement, you'll need to give more context - either in the form of proof, experience, or "other". Just something would suffice really - anything. You'll find a lot of people agreeing with you if you add such context.

Without context, it's nothing more than "Vanilla is better than chocolate - hands down."

<comment><USER35> NO context, it is just absolutely.

</comment></comment></comment><comment><USER37> > Shard. Or shard not. There is no try.

sorry, that was bugging me...

</comment><comment><USER38> How embarrassing for them, conquering yesterday's tech today!

</comment><comment><USER39> This looks like a big hack to compensate for using the wrong tool. Cassandra would have been a better solution IMO.

With Cassandra, you can set replication factors, speed up the writes, and automatically shard the data without having to manage your own "mapping tables".

<comment><USER32> Cassandra was immature when this work was started (late 2011). The team was also much more familiar with the ins and outs of operating mysql.

<comment><USER39> Technical debt is not an excuse to use the wrong tool. This is how you end up having a hammer.

Cassandra was mature by late 2011 and we were using it in production back then (billions of records).

<comment><USER32> The mapping tables are used to store relationships between objects, not to define shard locations. The shard mapping is encoded in the object id.

In any case, I would have a knee jerk reaction to not trust any database system that hit 1.0 a month before I wanted to start using it, especially for mission critical core data.

<comment><USER40> Doubly so when everything is catching fire around you from how overloaded it all is.</comment></comment></comment></comment></comment>


Why I Still Use Windows 95 - <AUTHOR> http://www.andrew-turnbull.net/tech/windows95.html <comment><USER2> There's a definite pleasure in using old, well-designed pieces of technology that were state-of-the-art in their time and have managed to remain competitive with newer equipment. I love my record player, I still use a high- quality CRT, I ride a steel bike, and I print documents with a mid-90s HP LaserJet. Many of you would cite LISP, a 50-year old language, as a perfect example of this phenomenon.

Windows 95, however, really doesn't strike me as one of those things.

<comment><USER3> Well, there _is_ a reason why it sold six kazillion copies back in the day.

Unless you think the reason was that it was hyped to high heaven in a gigantastic advertising avalanche.

</comment></comment><comment><USER4> I assume he also drives a model T, writes with a fountain pen, and uses a cell phone from 1995 as well?

Seriously, though, the advances from Windows 95 to Vista are less about improving the "tool" and more about improving the experience of using the computer. I don't think you can really call a computer a "tool" anymore, anyway. It is the communication, entertainment, and productivity center of many people's lives. It serves far more value than a hammer or a pen. It has the ability to morph into practically anything. Back in 1995, when the real value of a computer was still somewhat ambiguous, people called them tools because that was the closest metaphor to a a familiar paradigm. Today, the computer is the paradigm.

The argument this guy uses is kind of stupid and reminds me of the short- sighted, extremely stubborn attitude of the most geeky people I know. He doesn't like Internet Explorer 4, so he refuses to use an operating system based on IE. While IE 4 was a terrible internet browser, it served Windows pretty well; using the idea of back and forward buttons for browsing files/folders is still used today in Linux, Mac OS X, and Windows. Sure, Windows 98 sucked at it, but that's exactly what Windows XP improved on. (Arguably, Windows Vista improved on this as well.) That's why people upgraded.

In the end, by using Windows 95, he is just torturing himself. He might gain pleasure from the novelty of writing stories like this one and getting strange reactions from people, but in the end, his experience of using his computer is just... terrible.

<comment><USER5> There are still people who swear to a fountain pen. Not all old tech is useless.

<comment><USER4> I love fountain pens. The idea is that a ball point pen is far more practical for every day use.

<comment><USER6> I think the advantages of a ball point pen were more to do with having the ink reservoir inside the pen, rather than the ball point itself. There are plenty of pens with internal reservoirs on the market, whether ball, fountain or fibre, I've been using one of these[1] almost continuously for the past 6-7 years.

[1] [http://www.amazon.co.uk/stainless-fountain-cartridges- conver...](http://www.amazon.co.uk/stainless-fountain-cartridges-convertible- bottled/dp/B000JTPEAS/)

</comment></comment></comment></comment><comment><USER7> This article reminds me of my father. He still uses what we call "lefebvre books" to do his business's payroll by pen. When I was younger, I thought he was an idiot. Eventually he rationalized it to me for long enough that I thought it was charming and really interesting that his business is like a snapshot of how things were run in the past.

Right now, his company grew to more than 20 employees and he's selling it because his tools and methods don't work in his changing environment and he refuses to adapt to new tools.

Don't get me wrong, he's by no means unhappy, but his business is no longer running at it's full potential. I think a comparison can be drawn here. I believe that your output is a function of your methods AND tools.. I don't care who you are, you can't build facebook in DOS.

If your environment is changing and you still want to excel in that environment, I believe that you have to adapt.

</comment><comment><USER8> I wonder why he doesn't use Linux, seems he could build whatever he needs and forget the bloatware of windows. Specially with a distribution like Gentoo he could build a custom system and not have to have the technology drawbacks of Windows 95.

<comment><USER9> That was my first thought as well: "Hasn't he ever heard of Linux?"

But then again, if he's got all the apps he needs, running as well as he needs them, and isn't bothered by being unable to run newer applications (which he pretty clearly states at the end of the essay): well, more power to him, I guess...

<comment><USER10> Exactly. This is definitely one of those cases of use whatever works for you. I really can't fathom using windows 95, but reading over his reasons for doing so it seems like those are valid reasons for him. Still makes for an interesting read on a os no one ever really talks about anymore.

</comment></comment></comment><comment><USER11> I admire and salute this guy. He's one of those that keep our whole geek society interesting. He's a bit like a writer who still uses a manual typewriter rather than a word processor. It's very charming.

(On further browsing of his site, he seems an interesting chap. These mid-late 90s style sites chock full of various types of content are far more interesting than most of the vapid personal sites nowadays that have nothing besides blog entries.)

</comment><comment><USER12> Every time a new (microsoft) OS is realeased, you find tons of people who say that it isn't any better and there's no need to upgrade. Then a few years later everyone says, "you're still using previous_version?". (Vista actually seems like it may be the exception...)

It seems to me that this is an extreme case of the above situation. This guy has something that works for him. I would say works well, but until you've used something for a while and understand how it enhances your productivity, you really don't see the error of your old ways.

Some are asking why he doesn't install Linux. My thought is that if one of your decisions when choosing an OS is maintaining compatibility with your DOS games, you're probably not an ideal candidate for linux. Note that it doesn't matter if you Can play DOS games on linux; it's just a completely different mindset.

</comment><comment><USER13> At first, I was looking for the publishing date hoping I would find April 1st somewhere, but I guess not...

</comment><comment><USER14> Win95 c'mon, lets help this guy with other cooler ways to getting attention and being "different".

</comment><comment><USER15> andrew turnbull is completely right. i used windows 95 too, as long as possible (2006).

windows 95 has many good things that the modern version of windows have lost or never had:

\- lightweight (Doesnt even use memory <128 MB)

\- very fast

\- total cost of ownership = 0

\- functional (office, opera, etc: everything works somehow)

\- easily replaceable (ebay)

\- robust (works even with damaged hardware - somehow)

\- security (through obscurity)

\- i have an idea how it works (in modern version of windows i have somehow lost the overview)

\- no need to update (just accept how the system is)

\- free of distractions (no widgets, sidebars, audoupdates)

:)

<comment><USER16> The changes that MS makes now seem to be just for selling new versions. I use Windows 2000, it is handy, no Internet registration scheme where they track your usage, and so on. When I was younger I wanted to run the newst thing out there and would change often. Now I stick with whatever works best until it breaks.

</comment></comment><comment><USER17> Assuming he is using it, how exactly is he running Firefox 3 (he has an entry about it's release on his blog) on Win95? I haven't tried it but the OS isn't showing a message and then denies the possibility to install FF3? And what kind of productivity do you get with, let's say, 6 or 7 year's old software? I wonder, really, what kind of development/usage is he attributing to that machine.

<comment><USER2> Pre-Firefox Mozilla (what became Seamonkey) could be made to work on Windows 95... I was unfortunately forced to do so one time. You had to download a couple extra drivers and install some updates from Microsoft, but it worked.

</comment></comment><comment><USER18> I still have NT 4.0 install media around, and have even tossed it on an older box a few times to remember what it was like back in the days when Windows could be a decent developer OS.

Unfortunately, anything older than Win2k leaves you pretty much out of luck for mobile computing (power management, WiFi, and display switching) as well as most modern peripherals (USB support being notably lacking).

For even more retro fun, though, I recommend firing up a copy of Mac OS System 7 inside Basilisk II or SheepShaver -- even under emulation, it absolutely _screams_ through day-to-day tasks.

</comment><comment><USER19> I think the best reason for him to switch from Win95 to WinXP is Office 2007.

</comment><comment><USER20> Security Updates?

<comment><USER11> Windows 95 is so old that I'd doubt (though I might be wrong!) that its vulnerabilities are being actively exploited anymore. WinNuke ( <http://en.wikipedia.org/wiki/WinNuke> ) was the most annoying, and in the late 90s you were guaranteed to be nuked within a few minutes of getting online without a firewall.</comment></comment>


Working on own startup instead of an internship. How to make sure I'm learning? - <AUTHOR> I'm a junior in college majoring in computer science. This summer I decided to work on my startup instead of getting an internship. But I see all of my friends spending 8 hours a day and learning how to work in large teams and learning new technologies. How do I make sure that there's no stagnation in what I learn or is this something to even be concerned about? Would love your feedback/personal experience! Thanks (In my startup I'll be working mostly on iOS and React for dev stuff) <comment><USER2> I'm in the works on a start up as well. I think the best way to avoid stagnation is to challenge yourself. Back when I was looking for a job I had the option of two. One with a small company basically being the whole IT department and one with a big name company doing menial work. I picked the big company and kind of regret not going with the small to learn as much as I could have from the challenge.

<comment><AUTHOR> Thanks! What are some things that you did to challenge yourself? or do you have any suggestions about how I can challenge myself?

<comment><USER2> In relation to my start-up now I've been challenged by learning the finical side of things. I know the tech and love the tech but all that money stuff has not come as easily. So taking the time to learn it all has been a bit of a challenge.</comment></comment></comment>


Domain without dot (http://uz) - <AUTHOR>

http://uz is a domain without dot.<p>(HN doesn't allow to post url without dot. that's why I add as a text) <comment><USER2> Thanks, this is a good URL to use for a quick test if you have an internet connection, e.g. "traceroute uz" or "ping uz"

(On my Mac, you can type just "uz" in Firefox but in Safari you need "uz/")

<comment><USER3> Better is to "ping uz.". Otherwise you'll trigger a lookup in the local domain, which might take some time to fail.

</comment></comment><comment><USER4> Can try [http://188.8.131.52](http://184.108.40.206) if uz/ is not working for you.

dig output: ;; ANSWER SECTION: uz. 14182 IN A 220.127.116.11

</comment><comment><USER5> Can't get it to work on my phone. Safari coming up with cannot be opened as the server cannot be found. Will be interesting to see if I can get it to resolve on another device or connection.

<comment><USER5> Have tried on my Mac with the same wifi connection, using Chrome and [http://uz/](http://uz/) loaded up fine (Safari worked too).

An iOS issue?

<comment><USER6> Works on my iPhone using Safari over 3G but not on Wifi.

</comment></comment></comment><comment><USER7> my company proxy doesn't allow me to see the page, maybe they use the same regex as HN :-/

</comment><comment><USER8> Can anyone explain how this works?

<comment><USER9> There is nothing magical in the TLD (top level domains). They're domain names just like the others.

It just happens that in general, no A record is created for them. An A record has been created for the uz. TLD. That's all there is to it.

<comment><USER10> yup, this ^ and I have a feelling a lot of these new gTLD's are going to be doing this

</comment></comment></comment><comment><USER11> [http://uz](http://uz). works for me (Windows)</comment>


Unpredictable Behavior with C# 6 Exception Filters - <AUTHOR> http://www.volatileread.com/Wiki/Index?id=2095

<comment><USER2> Good Spot! This seems like a bug and should be reported to the Roslyn team.</comment>


gRPC in Production - <AUTHOR> https://about.sourcegraph.com/go/grpc-in-production-alan-shreve <comment><USER2> Another downside in gRPC is that (de)serialization can be relatively slow. One of my pet projects is a tagging server that takes a list of tags and returns a list of objects that match, retrieved from RocksDB. I tested gRPC (proto and FlatBuffers) and Cap'n'Proto. For all of them, the process looked like:

Insert: Object comes in (built in RPC system for both), object is assigned ID (which is added before serialization), object is written to RocksDB, object is serialized and indexed.

Query: List of strings come in, strings are looked up in tag index, intersection of results is done, objects are retrieved from RocksDB, objects are deserialized, objects are added to list, objects are returned to client.

FlatBuffers was unfortunately a no-go since it doesn't permit modification of fields that haven't yet been set and it didn't have a sane way of making a copy (which would also be slow).

Protocol Buffers worked but the pure Python driver is incredibly slow and the C++ driver for Python was non-default and marked "experimental". I eventually adapted to it but it was still quite slow even on the server side. From memory a response with C++ client and server took ~9ms, 4-5ms of which were spent on serialization/deserialization.

Cap'n'Proto eventually won for me. The Python driver is unstable and memory usage soars like an eagle until Linux shoots it down with an OOM but the server was much faster. Typical response times were closer to 3ms, most of which was spent in RocksDB or the actual index lookup. A downside though was that Cap'n'Proto has its own built in and odd library for async stuff and doesn't really support threading.

<comment><USER3> gRPC is slow with Python, but most people don't pick Python if they are serious about performance. gRPC has continuous benchmarks running[1] to track and improve perf. I agree the Python one has some serious problems, but don't let it be the whole story. The C++ implementation is 100x faster.

[1] [https://performance-dot-grpc- testing.appspot.com/explore?das...](https://performance-dot-grpc- testing.appspot.com/explore?dashboard=5652536396611584)

<comment><USER4> I'm aware and I did try out a C++ client but there was still a definite upper bound to its performance and latency was still ~2x Cap'n'Proto.

<comment><USER3> If the perf of gRPC is not good enough, you should file an issue on Github. I'm kind of surprised that gRPC loist by 2x. That's about how much better netperf is.

When you say you used Cap'n'proto, did you mean the RPC and serializer, or just the serializer? I ask because gRPC's serialization is pluggable, and doesn't have a dependency on Protobuf.

</comment></comment></comment></comment><comment><USER5> The only thing holding back gRPC is JS web support. If it had that it'd be time to drop swagger completely. As it is you need to go protobuf -> swagger -> js lib, but it's cumbersome and doesn't work 100% (e.g adding auth keys, etc.).

Will there be any progress on JS web, or can it just not be done at all with HTTP1? Even a subset of features for basic GET/POST would be fine...

<comment><USER6> I'd say load balancing is still a big gRPC hurdle. See [https://github.com/grpc/grpc/blob/master/doc/load- balancing....](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md) for the current official stance on load balancing. It's pretty convoluted IMO.

<comment><USER7> I agree. You kind of have to roll your own for every language you use which is odd. And retries. Hopefully some more standard way of handling this will emerge.

EDIT: I enjoyed the GopherCon talk... I don't think the video is up yet but when it is I recommend watching...

</comment></comment></comment><comment><USER8> Decent overview, but, remember that when evaluating something, projects change over time. Even better, you can be the change you want to see! Thus far, the grpc project has been fairly responsive in making solid changes, either through PRs or filing issues. ;)

Regarding the complaint about errors, there is already protocol support for structured error handling: [https://godoc.org/google.golang.org/grpc/status](https://godoc.org/google.golang.org/grpc/status). [https://github.com/grpc/grpc-go/pull/1358](https://github.com/grpc/grpc- go/pull/1358) should make this easier to use. In practice, the provided error code set is very good, so give them a try before making things more complex.

Worst case, you can just stuff things in a header or trailer.

</comment><comment><USER9> You can add Expedia to the list of production grpc users. Our entire hotel pricing backend is grpc microservices written in Scala. Some of these services take > 100k TPS. We have found grpc to be extremely scaleable.

If that sounds interesting, I'm hiring engineers, shoot me a note at joholland at Expedia.com

<comment><USER3> Do you use the Java generated stubs or generate your own? There are several people using gRPC with Scala that could benefit from more idiomatic stubs.

<comment><USER9> We use the java stubs. We briefly looked at some scala proto generators, but most times you dump the proto class into a rich scala type right away anyways.

</comment></comment></comment><comment><USER10> The point about operations is a very, very valid constraint of REST. Easy and common stuff like, "run this thing in the background", "send of this one, ephemeral, message" are very unnatural. Maybe a hybrid / bastard child of REST and gRPC would be a good marriage of resource and operations modelling.

<comment><USER11> The Google Cloud standard for async operations via the google.longrunning.Operation service: [https://github.com/googleapis/googleapis/blob/master/google/...](https://github.com/googleapis/googleapis/blob/master/google/longrunning/operations.proto). This should be usable either via REST/JSON or gRPC.

It's designed to be generic for any kind of async operation, and to be "mixed in" with your APIs. There are utilities for waiting for an operation to finish (via polling); in theory it should be possible to use some type of server-push to avoid polling, but I'm not sure if anybody's doing this.

(I'm an Alphabet employee who's working on APIs delivered via gRPC / REST/JSON.)

</comment></comment><comment><USER12> Thrift is Facebook's version of gRPC right? If so, I don't quite understand the comparison, and how gRPC succeeds where thrift "fails". Wouldn't all language implementations of gRPC have to be well documented, reliable, highly performant and easy to install?

<comment><USER13> My understanding is that Thrift is Facebook's analog to Google's Stubby. Or gRPC [which is the next generation of Stubby].

It's not implementing the gRPC interface. I don't think gRPC was open-sourced [2015] early enough for Thrift [2007] to be built to its API.

Disclosure: Google employee who uses Stubby. [but is not on the team]

<comment><USER14> Thrift was written at Facebook by an ex Google intern trying to recreate something close to protobuf+Stubby.

</comment></comment></comment><comment><USER15> > Inefficient (textual representations aren't optimal for networks)

REST APIs don't _have_ to be text-based, AFAIK. Why not just send/receive binary?

<comment><USER16> Sure, but then you'd have to write a serializer/deserializer for your messages. You could also use an existing one like Bson but then why not just use Protobuf?

<comment><USER17> Because bson, msgpack etc are self describing? It means a way lower barrier to entry.

<comment><USER18> I'm not sure I see how self-describing is a lower barrier to entry. It seems like generally the main reason why you'd want a self-describing format is if you're writing client (and server) language bindings by hand. If you have a tool to auto-generate those language bindings, you can skip that step, and there's no need to make a step easier if it's done for you automatically.

I can see where self-describing is better when it comes to side issues like debugging, exploration, or the hassle of configuring your build to generate code from the IDL files, though. But if I had to choose only one, those are lower priority to me.</comment></comment></comment></comment>


You Don't Need A Co-Founder - <AUTHOR> http://publicbeta.co/you-dont-need-a-cofounder/ <comment><USER3> So, the takeaway from the article is: don't co-found with someone technical, hire/outsource them. Well, I could say: good luck! You either get some half- assed solution or something ( if you oursource ) you don't even asked for, because they didn't understand your business requirements ( because of the language, distance, culture, etc).

Don't fall for this. Really, get yourself a technical co-founder. Amaze him. You can call him in the middle of the night, telling your new idea. Next day, he will already had done dissecting it and have a technical spec ready. Or any other crazy stuff that you want and he will say simply "No". You have no idea how useful is a "No" in a start-up. And so on, and so on...you get the picture.

PS. I am using a neutral "he" here.

<comment><AUTHOR> I've done both and I can definitely attest that outsourcing the technical bits of your new startup is beyond tricky. It is however not impossible. The point here is that taking a technical co-founder onboard isn't your only option; it's totally possible for you to get to V1 (by outsourcing technical tasks), generate initial revenue and hiring a developer (with some equity) instead (as an example).

<comment><USER3> Of course. There is also the third solution. With the vast amount of internet resources (think stackoverflow, etc) it possible for a founder to get to a alpha version 100% on his own. It is cheaper and sometimes more effective, because of missing managing overhead and lack of communication errors.

But in the end, yes, you have to deal with the dillema: hire or take someone on board, share the responsability and the earnings.

</comment></comment></comment><comment><USER4> As a solo technical founder _, I more or less agree with these points. I can color them a little bit from my own experience.

1\. Add in maturity. Building a product and selling it is hard. You can be harder, though. this may get easier as you get older? While it's nice to have someone to support you, it is not necessary. You can be a lone wolf and absolutely kill it. It's a mindset.

2 & 3 together. External validation can/should come from outside the day-to- day. 1 person's head is an echo chamber. 2 or 3 people can turn into an even more convincing one. A small council of real customers and a regular small round-table meeting with other entrepreneurs takes care of it.

4\. I believe in acquiring the skills yourself. Business and tech problems collapse into a single idea when you have all of the skills yourself (sometimes even just at an "intern" level). For capacity, this is a problem. You usually need more horsepower for bigger problems. I'd adapt this point to focus more on local outsourcing. You should be still be the architect and foreman though. So you should know how to code and mock things up perfectly.

If the tech people don't get it, it is always your fault. You have not explained it correctly or you have the wrong people. The first is more likely if you don't know how to build a web app at all or can't mock it up to the actual UI element detail. Of course, sometimes it is the latter. Just not usually. :)

_founder = solo business owner in a smaller niche area. Points also apply to a much larger business I'm starting now.

</comment><comment><USER5> This writer misses one of the most important issue, which would be under his heading "Validation". Validation isn't (just) internal! It's not that _you_ (necessarily) need convincing your 'idea' is any good.

Validation is external. The fact that two people are _both_ working on something makes 'something' 1000x more real than if some guy is working on it.

There's always going to be information asymmetry between internal and external: the fact is, if there are two people on the inside, that is one of the strongest signals (to people on the outside, who have limited exposure to the low-level progress) that there is something there.

If it's just some guy saying, "Hey I've been building x for the past y" that is a very weak signal.

I might even go as far as saying, "The only time you can afford not to have a cofounder is when you can make it all the way to completed product + substantial revenue without having to convince anyone of anything about the future of your business."

Anyone includes but is not limited to: employees, partners, your parents, your spouse, and, of course - Investors!

If you never need to convince anyone of the future of your project, until it's incorporated, profitable, making payroll - then you might do it alone. But for 99% of us, this does not apply, and a cofounder is one of the best signals around for sharing internal state of validation.

<comment><AUTHOR> I don't agree. Whether you're a solo founder or have 10 co-founders, you won't succeed if you're building something that no one wants. You can get customer validation regardless of whether you have co-founders or not.

</comment></comment><comment><USER6> A very familiar scenario for me. As a solo non-technical founder I've so far build my startup using funding via Seedrs (crowd equity funding in the UK) and contractors in the UK and Russia. I got some decent VC interest, but they all say they will only invest if I have a technical co-founder. Hence, I go off spending 50% of my time looking for a co-founder. It's like deciding that within 2 months you need to find your life-partner and get married - <USER2> level of commitment and just as difficult to find - but probably more expensive if you break up!

So I now have someone who I think is good lined up, I just need to persuade him to finally decide to join me now. But then I think - is it right to get a co-founder and give away loads of equity, just to satisfy some VCs? He seems like a really good guy, and I think I would like to work with him, but do I _really_ need a co-founder or is it just for the VCs? The development has worked very well with my contractor so far, managed by myself. Why not just raise some more non-VC money (via Seedrs again) and continue this way?

Dilemma.

<comment><USER7> Why not go with the contractor? You already have a decent working relationship.

<comment><USER6> You mean get him as a co-founder or continue with him as a contractor? He's based in Russia which makes him becoming a co-founder a bit more tricky.

</comment></comment></comment><comment><USER8> Please stop posting sales pitches on HN that masquerade for the first 75% as a blog post giving unbiased advice before coming up with the line "and that's where my company comes in".

The article is clearly biased based on the product and this should be called out immediately.

<comment><USER9> Please stop creating throwaway account for the sole purpose of outing someone. If you want people to take you seriously - put your karma where your mouth is.

p.s: I don't mean to shoot down your claim by the way

</comment></comment><comment><USER10> The YC people deal with enough startups to see probabilities. They have data, and they say the odds improve dramatically with cofounders.

This article (mostly) doesn't dispute that, and explores ways to perhaps compensate for the lack of a cofounder and redress those odds a bit. That seems useful as a source of ideas, as long as we keep it in the right category.

It is in a different category from the YC statement, which was based on numbers. YC didn't draw conclusions about cause/effect, they just correlated. That's pretty solid ground. This article puts up several specific possible causes of the cofounder advantage and then possible solutions, which might work, if the cause/effect inference is right. Two layers of inference, based on anecdotal evidence.

The article made me think about the issue in a different way. I thought it was valuable in that way. I'd be really interested to know if people with data have correlated solo startup success with any other factors. Does anyone know of any such attempt?

</comment><comment><USER11> I've been objectively successful as a single founder over the past few years, so its definitely possible. I've hired, fired, pivoted, and made more money than I made mistakes but I always felt the lack of a cofounder has been holding me back.

There is only so much emotional weight I can offload onto my amazing wife, and although some of the people I've worked with (contractors, clients) have made amazing contributions - technical, business, ideas, and emotional - at the end of the day I can't shake the fact that I'm alone. I bear all the responsibly for my mistakes and all the rewards of my successes, and even though I've been lucky to have more of the latter the emotional toll of the rollercoaster has been tremendous. When your wife or parent tells you you're doing great or that everything will be OK it helps, but the impact is diminished because of course they are going to support you.

My most successful (although not my most profitable) projects have been collaborations with other talented people both technical and nontechnical, and while the author is certainly correct that a cofounder isn't necessary, I would think long and hard doing anything alone.

<comment><USER12> Do you talk to other founders much? I am a solo founder, but in a co-working space. I can bounce ideas off of or ask for help from many of the other founders there. That makes it feel like it's no big deal that I'm a solo founder.

<comment><USER11> I love Coworking and even I help run a space here in Madison. I should've mentioned that as one of the major things that keeps me sane.

</comment></comment></comment><comment><USER13> "But do you really need a co-founder? Even if you're not technical and you think you need a technical co-founder?"

I relish the day when non-technical co-founders and single founders fall by the wayside and technogeek types learn how to sell themselves. It's largely an introvert/extrovert personality thing, where technologists are convinced they should keep to interacting with computers and not other people -- but it's just not true. There's no reason why they shouldn't do both aspects of work as the technology-side of the business is the one that generates the value and it doesn't make good business sense to share profits with someone who only sells the value you create when you can just do that part yourself.

</comment><comment><USER14> I have a ton of respect for Adii, but from my experience it is SIGNIFICANTLY harder to do something great without a cofounder.

Sure, if your a cereal entrepreneur like Adii or captain crunch, it's doable, but for most of us who are still trying for our first big breakthrough, we need all the help we can get.

Almost any startup/product needs two things. A good quality product and users. When speed is important it's extremely difficult for one person to do these very different tasks exceptionally well. As a CTO (who isn't overly technical, just enough to build prototypes, PM, and lead the team) I feel completely overwhelmed just thinking about having to do all that our CEO does on the business and growth side.

</comment><comment><USER15> The typical case is to not do this. If you're capable of doing the development work for what you want to build on your own, it's exhilarating though. As long as you have external validation (paying customers, ensure you talk to people smarter than you on a regular basis,...) it's nice being able to make your own decisions without worrying about friction.It's a double edged sword with the amount of work you need to do, but if you enjoy what you do, for many who would be crazy enough to do this in the first place, it's mostly a non issue.

</comment><comment><USER16> Solo founders are rare for a reason. If you just hire people to do a job, why would the most talented people work for you? Doesn't make much sense in most cases if the founder has not already a lot of capital. I would say increasing the odds of success is much more valuable than getting a higher payout in the success case (expected value = chance * payout).

On the other hand, it is hard to find the people who have very similar goals.

<comment><AUTHOR> Talented people are also attracted to working on something they're passionate about and having the freedom to execute on that passion. :)

And not all businesses needs to be set up purely as a vehicle to an exit. Many businesses can instead operate as "lifestyle businesses" and earn great revenues / profits for years to come.

</comment></comment><comment><USER17> This is a pretty good article, I share a lot of the same sentients, but I'm on the other side of the fence. I"m on the technical side, although I've started making friends who are business minded in the area which has helped out a lot. I think that the right co-founder makes your team progress at more than 2x your normal speed.

</comment><comment><USER18> My opinion is that it really depends on the individual, especially their motivation and focus. Some people are cut out to be solo founders, most aren't. When I tried working on projects alone, I tended to get distracted easily. Partnering with a finisher has helped me get things done.

</comment><comment><USER19> Why wife/gf/bf isn't equivalent to an actual full-time co-founder [http://www.youtube.com/watch?feature=player_detailpage&v=9R2...](http://www.youtube.com/watch?feature=player_detailpage&v=9R2xgM- pu18#t=668) (max levchin)

</comment><comment><USER20> I think that this article is "meh," so let's look at it step-by-step:

1\. Emotional Support

Author gets emotional support from wife and admits that you _will_ need someone to help you through the shit-storm that is founding a start-up. But most young entrepreneurs don't have spouses. Seems like a co-founder fits the bill here.

2\. External Validation

A couple of points: you _should_ listen to your customers, but your customers are a very bad barometer for gauging big company decisions -- sometimes you need to make changes that they won't like. Changes that are good for the company (or changes that help you pivot) but piss off significant portions of your customer base; the customer is _not_ always right. So if customers are a bad source of validation, what about other entrepreneurs? Let's face it: they are busy with their own projects to give you any meaningful feedback. The author's last point is to "write that down" \-- seriously? Just get a co- founder.

3\. Ideas / Brainstorming

Brainstorming is not merely spit-balling. Author says that "you want to prevent brainstorming ideas and possible solutions that are inherently not viable or even true" \-- but who else knows your company as well as a co- founder? I.e. who else could gauge whether or not solutions are viable (or true)?

Let's ignore the shameless plug here (and the fact that using a website for brainstorming seems _prima facie_ silly); I just want to point out that brainstorming outside of your company is worthless. You, quite obviously, need to brainstorm with someone _in_ your company. Might as well make it a co- founder.

4\. Skills / Capacity

It seems pretty obvious that you want to do something in the tech arena, you need to be pretty tech-savvy. If you're not tech-savvy, you need to associate yourself with someone that is. There's a good reason investors don't invest in business people. I remember listening to a talk by some of the people at Mint. One of their anecdotes went something like so: "When you go and pitch to a VC, every programmer is +$50mil in valuation, and every suit & tie "business guy" is -$100mil in valuation."

Apart from the intuitive attractiveness of this train of thought, there also has not (ever) been a successful "business guy" with no tech expertise and no technical co-founder doing much of anything in the start-up world.

Finally, let me just say that the people at YC know what they are doing. They rarely accept single-founder start-ups for a very good (and obvious) reason. The lack of technical expertise is also a big red flag. After all, if I've never seen the inside of an engine why would a bank fund an auto-shop I want to build.</comment>


Google j2objc, a Java to iOS Objective-C translation tool and runtime - <AUTHOR> https://github.com/google/j2objc

<comment><USER2> This was apparently used for Google's new Inbox and the new Sheets App.

[https://plus.google.com/110412141990454266397/posts/PH9Yz9un...](https://plus.google.com/110412141990454266397/posts/PH9Yz9un2LL)</comment>


Candy Japan hit with credit card fraud - <AUTHOR> http://www.candyjapan.com/candy-japan-hit-with-credit-card-fraud <comment><USER2> I commented this on yesterday's jsbin article, and I'll write it again.

Don't implement the payment processing code yourself. (And using Stripe is _still_ implementing it yourself - they supply only one part of the process.)

Writing this code will take time that you are not using to develop and market your product. (cf opportunity cost). Your code will be buggy. Your code will be weak. Your code will not support the various alternative payment options popular in, say, Holland. (Yes, this is a thing, and it is called iDEAL.) Your code will sooner or later be exposed to a campaign of fraudulent transactions.

Use FastSpring, or your alternative of choice. Yes, it costs more, if you don't put value on your weeks of time to create and support an alternative, and if are comfortable having a weak fraud detection, if any.

When your business becomes successful, when the fees to FastSpring start to dwarf the costs of implementing a payment system yourself, then - perhaps - consider re-doing it yourself.

There are indeed weaknesses in using FastSpring, which after seven years are becoming a pain for me. I'll be writing about that soon. But this is not an issue for a one-person endeavor getting started.

<comment><USER3> Living in Holland, where everyone does online transactions with iDeal, I find it hard to understand why the rest of the world is using credit card payments at all, for anything. It is massively insecure, it's expensive for the merchant, and theft is ignored (when millions of stored creditcard records are stolen, those cards are not invalidated and replaced?). This is all at the cost of customers and other merchants.

I'll keep trying to avoid using a creditcard as much as possible and use other payment methods when I can, both online and offline.

And yes, when implementing payments on a website, use a service that does all of the complex things for you, and never store sensitive data yourself.

<comment><USER4> Having visited NL, my card was practically useless. Any other country no problem, but Netherlands? So I don't understand you really.

That being said, direct payment gateways are common, but they require cooperation with banks. There's probably hundreds of them and only make sense on local market where you can have on or two implemented to satisfy 80%+ of customers (and others will just use regular card).

<comment><USER5> That's the whole point, in the Netherlands we use debit cards (known as 'PIN cards' to the locals) instead of credit cards. Usually these are Maestro cards. I don't know the exact number, but as a Dutch resident I guess that >99% of all POS transactions are done using such card, not a credit card.

Cashiers in most common shops probably don't even know how to accept a payment by credit card, so the'll just refuse.

Meastro cards are usually protected by a PIN and require a POS system to be used. This was a problem for e-commerce, so the Dutch banks participated in iDEAL. It's basically a redirect from the merchants website to the bank site of the customer, where they use their own cardreader and debit card to validate the transaction before being redirected back to the site of the merchant.

<comment><USER6> In many countries (such as where I'm from, Sweden, or where I live now, Japan), debit cards and credit cards are interchangeable, and you can't even tell by looking at the card if it's debit or credit. They're both VISA or MasterCard branded, and what's backing it is only the business of the cardholder, not the merchant.

I remember Maestro cards in Sweden as being for under-18's, and then when you become an adult you get a VISA/MasterCard debit card instead, but I don't know if that's true anymore.

<comment><USER7> In Canada, a distinction is made because the fees on debit are much lower (comparable to cash handling costs) and the banks have put tremendous marketing efforts in pushing for debit cards and branding them "Interac."

That said, the cards themselves use exactly the same technology, look the same and debit cards are usually Maestro/Cirrus or Visa/Plus compatible so we can do debit transactions in europe.

The online version of debit however has yet to catch on.

<comment><USER8> Canadian here, and I find that most of my peers (30s) use their credit cards for almost everything and their debit cards collect dust. The reasons are several:

1) CCs offer rewards (travel, cash back), usually in the range of 2% of purchases

2) CCs offer insurance and extended warranties on some purchases.

3) CCs help you build a good credit rating, which is important for someone who intends to apply for a mortgage at some point.

4) CC offer superior protection to debit cards. Yes, CC fraud may be easy, but as the card holder you will almost never be on the hook.

I've had my CC compromised, the bank detected it immediately and issued me a new one. Never was there any question who would bear the cost of the 5-figures that the fraudster spent. My brother had his debit card compromised once and the story was completely different. The bank wanted him to _prove_ that the fraudulent transactions were not his and he had to go to great lengths to do so.

My personal preferred order of payment is:

CC

Cash

Debit (last resort)

<comment><USER9> On the other side of the fence, also Canadian, my debit card has been compromised many times. Bank always shuts it down immediately and lets me know, then returns the money. I have never been asked to prove anything. Usually they ask me to double check and see if any other charges than the ones they caught were fraudulent.

This has been as simple as someone using a copy of my card to buy candy and video games, something I could have done but didn't in this case.

Really impressed with the services of my banks when it's come to debit fraud.

<comment><USER8> To be fair, my brother's incident was quite a few years ago by now, so it's quite possible that the response has improved.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> We ran an online store selling merchandise for a very large mobile-game vendor, so naturally we experienced these types of fraud attempts.

For us, there's one killer technique. Detect fraud however you want to / can (this can be a mix of heuristic data from your stats, third party, or whatever you want) - but when you detect it - don't decline the user.

Send them to a fake purchase confirmation page.

Suddenly they'll be getting 100% success from your site, and they'll drop you immediately.

On the backend, put the transaction to manual approval, so if it IS a legitimate client, when they email you, you can manually approve the order.

Over the years, as others have pointed out, detection methods change, but using the above technique invalidates their reasons test with your site. (very similar to Mailinator's technique for people scraping their site.. [http://mailinator.blogspot.fr/2011/05/how-to-get-gmailcom- ba...](http://mailinator.blogspot.fr/2011/05/how-to-get-gmailcom-banned-not- that-i.html) )

<comment><USER11> I think Amazon does this. Regardless of whether the payment goes through or not, you'll always reach an 'order confirmed' page. Looks like they've adopted the same technique for curbing fraudulent purchases.

</comment></comment><comment><USER12> Interestingly enough I found this subthread from an HN Post earlier today to be very relevant: [https://news.ycombinator.com/item?id=10234561](https://news.ycombinator.com/item?id=10234561)

> Regarding his fraud issue, I found that my website was being used in the > same way when I added a credit card payment form. I implemented a system > that first does an "Auth". If that passes, then I pass details to MaxMind > and get back a response with a "riskScore". If the score is too high, I void > the auth and decline the transaction. This has saved me a lot of chargeback > fees, though it's still not perfect. I prefer PayPal because a "not > authorized" just reverses the transaction; there is no chargeback fee.

\- Osiris (HN user)

IMO, there's more worth reading in that subthread. It's an interesting topic, at least to me. I feel sympathy for any merchant that has to go through this kind of pain.

<comment><USER13> This issue is so costly and prevalent that I feel its a huge disservice for companies that offer credit card services to merchants to not either 1) mention this issue and recommend a fraud check service, or 2) include fraud protection in their service.

I actually ran into an issue a little while ago in that I allowed my MaxMind account to run out of queries. Not realizing this, I saw a few days of higher than normal sales but just assumed it was a good day. When the first chargeback came through, I immediately noticed the problem, bought more credits, then began to go through each transaction one and a time and noticed a pattern in the email addresses (all two words followed by 3 numbers @hotmail.com) and individually refunded each and every one of them.

I still got chargebacks but I was able to dispute them by showing that the transaction had already been refunded.

My ideal credit card would be on where the physical card has e-paper on it with a 6-digit PIN that changes periodically, like 2-factor auth, and that PIN could be required for purchases to be authenticated. The new smart-chip cards don't help at all with online purchases, only point of sale.

On a side note, I found that nearly all of my fraudulent purchases came from Vietnam to the point that at one time I put in an IPTABLES rule to block the entire country.

<comment><USER14> Nobody seem to know or mention : [https://en.wikipedia.org/wiki/3-D_Secure](https://en.wikipedia.org/wiki/3-D_Secure)

I have to use it with most online shops here in Switzerland.

<comment><USER15> 3D Secure was mentioned in the other thread. Folks recommended avoiding 3D Secure / Verified By Visa because so many banks implement it insecurely, and the redirect model is easy for phishing scams to imitate:

[https://news.ycombinator.com/item?id=10235328](https://news.ycombinator.com/item?id=10235328)

That redirect will kill conversion rates too, being redirected to a site you didn't expect claiming to be your bank but not matching its URL... of course it will freak some people out. Consider that some banks make the Verified By Visa password the customer's birthday (ie something easily researched) and you quickly realize it's a horrible system.

<comment><USER16> Re: Conversion rates, as a consumer I have got used to it and it does not affect conversion for me at all.

Everyone uses it now in the UK and you always get redirected to the exact same page. I expect it, it doesn't put me off buying.

So it's a bad objection to the system, because once everyone's using it, it becomes the norm. Yes, there will be a dip in conversions to begin with as consumers are scared by the new page, but it will recover. Some people's implementation sucks, but that's a different matter.

<comment><USER17> I loathe verified by visa. I'll still buy from that site, but only in unusual circumstances. Verified by visa means I'd rather buy from somewhere else.

</comment></comment></comment></comment></comment></comment><comment><USER18> I also had something like this happen on a site I built for my wife's work's site, a Boys & Girls Club[0]. I had a donation button that let people make an open donation to the club. It's such a tiny site with little traffic, but apparently the SEO must be decent because somehow it got targeted by people appearing to come through Brazil and Poland.

Suddenly one day, hundreds of donation attempts. Checking the failed transactions, the script was trying $1, then $2, etc up to $22 then starting over, each time with a different number. About $200 in successful donations came through before I caught it.

So I just shut it all down for now. I'm not really sure how best to fix it. The club has no money and already complain about the transaction fees they get charged per swipe (and those are all pretty standard rates in the U.S.). Being a non-profit, they expect a lot of stuff for free, and unfortunately won't consider paying for an anti-fraud service. Even PayPal takes too big a cut.

Realizing that this wasn't a helpful comment, just a "me too." It sucks. Oh well.

[0]: [https://salembgc.org](https://salembgc.org)

<comment><USER19> How can they not be willing to pay a transaction fee from even PayPal for donations? Seems their options right now is to take donations via a service like Paypal with a transaction fee, or just take no donation.

<comment><USER20> They may actually not have a choice. My wife is a lawyer who does a lot of work with non-profits, and maintaining non-profit status can require jumping through a lot of legal hoops. A quick google suggests GBCA is a 501(c)3 non- profit, so it's very possible that adding a gateway in front of a donation that is tax deductible may create some legal risk (real or perceived), or be outright not allowed by current laws.

<comment><USER21> What the heck are you talking about? Fundraising expenses and overhead are totally ordinary and expected for non-profits. Obviously if it would be very high, it's problematic (as percentage of donations going to program expenses would low--think hiring a fundraiser, and giving them 50% of every dollar they bring in.), but a ~2% credit card processing is unexceptional.

</comment></comment></comment></comment><comment><USER22> We at Sift Science ([http://siftscience.com](http://siftscience.com)) might be able to help. Feel free to email me at jason at siftscience dot com

Even if you don't use us, we published some articles to help merchants new to dealing with fraud: * [https://siftscience.com/sift-edu/fraud- basics](https://siftscience.com/sift-edu/fraud-basics) * [https://siftscience.com/sift-edu/prevent-fraud](https://siftscience.com/sift- edu/prevent-fraud)

<comment><USER23> Sift Science is a great service, I would highly recommend testing it out.

<comment><USER22> Thanks for the kind words.

</comment></comment></comment><comment><USER24> One other tip: Block all TOR exit node IP's. You'll find mostly fraud and spam coming from them.

Deep down, I would love to support TOR in principle. I know there are people living in oppressive regimes that need access to information. I want to support that side of TOR. In reality it's still the transport tunnel of choice for scammers and criminals. The costs just don't outweigh the benefits. Considering the CDNs that block TOR (Akamai, Cloudflare, Incapsula, etc), you wouldn't be the only one blocking TOR. I'd also throw in EC2, GCE, Azure, and Rackspace IP's too.

<comment><USER25> Hopefully nobody is living in a regime so strict they can't order candy delivery without fearing for their life.

<comment><USER24> Never underestimate how far San Francisco will go to impose its will.

</comment></comment></comment><comment><USER26> Currently we're integrating Sift Science to avoid this otherwise serious and annoying issue. I think it happens to everyone who's directly accepting credit cards online. Does anyone have experience with Sift Science or similar services? (I know MaxMind has one but that, to me, seems inferior to SS's.)

<comment><USER27> Came here to recommend a sift science type solution. I've previously integrated fraud systems like Cybersource (clunky interface, wouldn't recommend)

<comment><USER22> Thank you for the recommendation!

</comment></comment></comment><comment><USER28> Second article in one day mentioning how bad credit card fraud can be to handle, especially for smaller sites.

I wonder if there might be an opportunity there? Or if the solutions have to be so custom that it'd be impossible to work out.

<comment><USER29> There is a huge opportunity there for companies like Stripe to buy eg MaxMind and integrate a quality risk scoring system directly into their payment solution. It's an obvious product expansion for someone like Stripe. Kind of hard to believe they haven't already gone after this.

<comment><USER30> PayPal does to some extent. And then people complain about how expensive PayPal is. In a commodity, race to the bottom world of cc processing I am not sure it would be a good business practise to bundle fraud prevention and processing. Let people be bitten and then willing to pay to avoid it is probably a better practise than charging for it up front. But they do buy them. ACI bought ReD last year for instance [http://www.aciworldwide.com/news- and-events/press-releases/a...](http://www.aciworldwide.com/news-and- events/press-releases/aci-completes-acquisition-of-red.aspx)

</comment></comment></comment><comment><USER31> I had something similar happen with one of my services. I have an "Update your Payment Details" page for paid subscribers that lets them enter new card details when their old card expires or they just want to switch the card they're using with us. It normally gets used a few times a month, and anywhere from zero to one time over the lifetime of a customer.

But then the bad guys found it. And individual users started updating their card details dozens of times each day.

This went on for several days before I noticed it in the logs. It was easy enough to fix: users now get one update per year, unless they email me and ask what's wrong with that card update page. The bad guys moved on to greener pastures and life went back to normal.

</comment><comment><USER32> This sucks and can often be a crippling blow for a small business if it's not caught early.

I remember we used to deal with a lot of this at Twilio, except the carders would then also try to cash out the credit card into Twilio credit. The pattern was try to spend $10 on a card, then one minute later try to spend $1000 once the first charge went through.

But it's a lot less costly than toll fraud, which I learned the hard way working on our auth system at [https://charge.co](https://charge.co).

</comment><comment><USER33> Did you considered to accept Bitcoin payments? This way you wouldn't have to worry about chargebacks.

<comment><USER34> Bitcoin is not the best thing for recurring payment. But it would be great to pay a year in advance with Bitcoin - and yes that solves chargebacks or fraud issues.

</comment></comment><comment><USER35> Clearly, card companies are going to have to adjust their policies for failed transactions. It's not like it costs real money to decline a fraud attempt; stop punishing merchants.

<comment><USER35> My experience with card company fraud detection systems is that they were bafflingly useless. Maybe they've gotten better recently. But even Stripe seems to let through things that are obvious fraud, stuff that wouldn't pass a basic spam filter.

It's almost like they want to charge you those fees!

</comment></comment><comment><USER36> Another interesting problem is that recent breaches have put _so many_ cards in the carding market that clearing them seems to have become a bottleneck. If we could some how work with the banks to perhaps create honeypots for these people it might help us clean up the mess.

</comment><comment><USER37> As I can see all solutions listed in this thread is to protect yourself better and let fraud happens for less protected merchants. The real solution is to fight back against fraud. All fraud attempts must be recorded in the public space, for example [http://fraudrecord.com/](http://fraudrecord.com/) . The best solution is to have something like open blockchain Grossbuch for fraud. What are current best solution for FIGHTING BACK AGAINST FRAUD?

</comment><comment><USER38> This is something we (www.smyte.com) can help out with -- send me an email at pete at smyte dot com if interested.

Also happy to answer any questions about general mitigation techniques.

<comment><USER39> I tried to go to your site but am getting a 500 error.

Edit: Interestingly enough, www.smyte.com is fine, but smyte.com yields the 500 error.

<comment><USER38> Yes, odd issue with our dns, link corrected.

</comment></comment></comment><comment><USER40> We at Ravelin ([https://www.ravelin.com](https://www.ravelin.com)) are building solutions to stop this. Still surprises me how many people think banks/schemes are responsible for CC fraud, when it's actually the merchant that gets punished. And yeah, fraudsters operate like locusts.

<comment><USER41> I don't understand why does the merchant have to pay the 15 fee. It sounds to me the entity that should handle the loss of fraud is the bank.

<comment><USER40> Blame the lack of competition in the international payments system I guess.

</comment></comment></comment><comment><USER24> The one-time card numbers generated by the old AMEX Blue system were great for shopping on sites you didn't trust. Too bad that they dumped that and the card reader system around 2002-2003? I really liked it and felt much more confident using it. Seems like it would have cut down on fraud a lot more than many systems in use.

That said, I really don't know why anyone would want to try and write their own payment integration gateways nowadays. There are so many good alternatives out there. Why not use them? You should focus on what you love, not payment processing. If you're just selling simple goods, you can easily setup a store on Shopify, Weebly, WIX, etc. Let them handle the fraud stuff. Sure, you might be on the hook for a few bad charges here and there, but at least you didn't waste hours writing payment code.

<comment><USER42> I just started using Blur by Abine, which allows something similar. (Basically, you can fund prepaid cards to use on sites rather than giving out your real card) not an outright endorsement. I haven't used them for long enough to give a definitive answer on if they're good, but it costs $40/year plus $2 for each "masked card".

So far it's worked for me.

I definitely agree no Dev should write their own gateway. There are great ones out there who are developed by people who are paid to do nothing but that.

</comment></comment><comment><USER43> I wonder would that still be the case if Candy Japan was using something like Stripe or Balanced, what happens in that situation? Would you be still responsible for 15 EUR chargeback fee? What did Recurly do in this case?

<comment><USER44> With Stripe you are still responsible for the chargeback fee, but they do not charge you for transactions that don't go through. They also have _some_ fraud protection, but it did pass some obvious fraud in our case.

</comment></comment><comment><USER45> Shouldn't the credit card companies be the ones to handle the loss?

<comment><USER30> You would wish so:). But no. Merchants are on the hook for all costs, including penalties for incoveniencing the cc companies (either a set fee per fraudulent transaction, or high fees if exceeding thresholds of fraud if you are a larger guy).

</comment></comment><comment><USER46> Braintree offers "advanced fraud protection" if you pass device information along with the transaction request [1]. Has anyone tried using this, and if so, is it effective at stopping this problem?

[1] [https://articles.braintreepayments.com/guides/fraud- tools/ov...](https://articles.braintreepayments.com/guides/fraud- tools/overview#fraud-tool-comparison)

</comment><comment><USER47> What about using the Twitter "buy" button now? Is it vulnerable to the same type of fraud? See this example [https://twitter.com/mperham/status/643480319870369792](https://twitter.com/mperham/status/643480319870369792)

</comment><comment><USER48> Damn shame to hear about this Bemmu -- would've felt like a massive punch to your guts!

Maybe you should set yourself up some email alerts when things seem 'off'. i.e. no referral, and the user/bot spends no time filling out the form and hitting submit. What's your glue code like?

I'm rootin' for ya. :-)

<comment><AUTHOR> Hey veb,

This fraud would have been easy to detect if there had been any kind of detection system in place. But I had no fraud checking, captcha or 3-D secure, as I hadn't expected there to such a "fraud wave".

Those payments didn't feel like necessarily coming from a bot, as there were time delays and for many purchases they even filled in the questionnaire. It felt more like someone had a pile of numbers they were entering manually.

My form made that easy to do, as I had tried to eliminate any field which wasn't absolutely necessary.

<comment><USER49> > _...they were entering manually_

Definitely wouldn't discount that possibility. Unrelated to CC processing, but we used a service that gave us a device ID on signups to detect people signing up multiple times (a tipoff that fraud was likely to come).

We later found fraudsters were using Amazon's Mechanical Turk to get real people to register manually, thus getting around our device detection.

</comment></comment></comment><comment><USER50> i simple solution to cut down on this is never allow more than 2 or 3 failed card attempt from a single ip address. You have to link it by ip because they will just make a new account if you do it by account. Carder will move on to another site if running cards is not simple and fast

</comment><comment><USER51> I don't understand the search conversion statistic. If a guy is testing a bunch of cards, wouldn't he just find your site once and try them all sequentially?

If a bunch of people had bought these packs of stolen cards, wouldn't they be spreading out over a bunch of different sites?

<comment><USER52> The automated scripts that the criminals use will try to simulate a real customer signup and transaction, otherwise it's too trivial to filter them out.

This is also why fraud detection is a big industry, you basically need full time staff to analyze the current types of fraudulent transactions and update your filters as fast as the criminals update their systems.

</comment></comment><comment><USER53> Been working in e-commerce fraud for ten years...

The credit card security model is broken as designed with the exception of chip and pin and chip and signature.

Uptake for 3D Secure is uneven around the globe. Some places it is mandatory and others unheard of. For example not using it will negatively impact close rates in India but using it in the U.S. will negatively impact close rates.

With 3D Secure each issuing bank was responsible for how they implemented the process and it varies widely in both ease of use and security between issuing banks. Some banks may use simple passwords that can be fished while others may use one time pins sent by SMS to your mobile.

When a merchant uses 3D Secure fraud liability shifts to the issuing bank and the merchant gets a discount on the interchange. This is also true when the bank hasn't implemented 3D Secure or has done so poorly. Because of this when 3D Secure was launched some banks put out systems that were non-functional simply to avoid the liability shift (i.e. it would prompt for a password but there was no functionality to create one). Issuing banks have gotten their act together in the last decade but it's still uneven.

Even when a merchant implement 3D Secure the merchant still needs to manage fraud levels to card association rules even though the merchant does not have liability for the fraud. This is the "we have to assume the risk but we don't have to do business with you clause." Individual issuing banks may chose to decline all of a merchants transactions if loss rates are unacceptable and the merchant may still lose their merchant account if overall loss rates are too high. However because the charge back shifts to the banks the merchant may not even know which transactions are fraudulent.

Recently rules have been changed so that banks aren't required to prompt for authentication in all cases. This is called risk-based-3D secure. So even if Candy Japan had implemented it he still may have seen the attack because the banks may not prompt for that low of an amount. He wouldn't be out the money but still may be in trouble with the association if he didn't do something to stop the attack.

When looking at automation there are two fundamental risk that any solution needs to address. The first one is class separation which in the industry is known as scoring or out-sorting. In this area machine learning algorithms are well understood and there are a few companies on the market that will rent merchants a model such as Sift Science or MaxMind.

The second one is anomaly detection and in the industry this is know as velocity control. The problem is similar to detecting a hacker once they have penetrated the network, detecting a bio-terror attack based on hospitalization data or detecting insiders who have flipped. These types of things have generated a lot of military interest and a lot of DARPA funding but little of it has made it into the commercial market due to the high rate of false positives.

Card testing is one example of a 'high velocity' attack and are usually the work of professionals. High velocity fraud has been more common with digital goods but more recently we've seen more high velocity attacks where an attacker will upload a merchants products on a site like Amazon or eBay. When the attacker gets a sales they will use stolen credit cards to get the trageted merchant to fulfill the order. Since these are professionals they generally will have tested the merchants fraud systems to understand what is considered low risk by the merchants fraud models and unless the merchant has some monitoring losses can be very high by the time the charge backs start coming in.

Currently there is no automated, fool-proof method for managing these high velocity attacks that I'm aware of. Merchants initially started putting limits around the number of times that a single card could be used and the attackers responded with SQL injection attacks to get card in mass. Merchants responded with limits on email addresses or other data points and the attackers responded by mass registering emails on free email providers and registering their own domains. Merchants responded with limits on IP addresses and used of geo-ip location services and attackers responded with open proxies and VPNs. Merchants responded with limits on orders from co-location facilities and VPN providers and attackers responded by compromising end-user machines en-masse. Merchants responded with proxy detection and device fingerprinting and attackers responded with using RDP to end-user machines to launch attacks. And it continues.

</comment><comment><USER54> Unjustified credit card fees and the financial/psychological burden of chargebacks (resulting from uncontrollable credit card fraud) are the main reasons why I want the Bitcoin experiment to succeed. Lets give Bitcoin some love where it deserves.

</comment><comment><USER55> But why? Ordering periodic shipments of candy from Japan? Most credit card fraud involves something that can be resold. This service gets you a small envelope of candy. It's not like getting cases of Pocky.

<comment><USER56> " The stolen cards originate from credit card security breaches, resulting in a big list of card numbers. These are later sold online in packs filtered to working card numbers only, which can be purchased for about $10 per valid card.

To be able to compile and sell these packs, the carders need to know which ones are valid. To do this, they will use an online store or service to place an order for the sole purpose of seeing if the charge goes through or not.

</comment></comment><comment><USER57> TIL Stripe does not come up with built in fraud protection.

</comment><comment><AUTHOR> I just added in an anecdote that I forgot to mention at first. Encounter with a police officer. Starts from "After starting to deal with this..."

</comment><comment><USER47> 1\. How many sales would you lose if you required everyone to use PayPal everytime? 2. (What about Google cash? or Square cash?)

<comment><AUTHOR> It's difficult to test, because by not having credit cards a lot of fake orders are now excluded, lowering the conversion ratio that way.

So if I did a test of showing only PayPal to half of customers and PayPal + credit cards to half, the one with CC could win just because of the fake sales.

<comment><USER47> I guess what I mean is, "do people really need the credit card option? If you just demand they use PayPal and they want your unique service, then they'll find a way to pay".

</comment></comment></comment><comment><USER58> Couldn't you do a preauth on the cards and then completion when you are ready to ship?

At least then you don't get hit with chargebacks.

<comment><USER59> Last time I talked with a credit card company, they wouldn't even let me file a chargeback until 30 days after the transaction. How would this stop chargebacks?

<comment><USER58> A preauth on a card reserves the credit without actually completing the transaction. The credit is still there but not available for use.

You can complete the transaction by sending a 'completion' request, at which point the credit is gone.

The preauth also has an expiry. If the preauth expires, the funds are released with no transaction taking place.

So when a user signs up, take a preauth. If you have any reporting on your sales, like this guy, you can check for any anomalies before fulfilling orders. Anything that looks odd, don't complete the order.

</comment></comment></comment><comment><USER60> From my experience running eCommerce sites, the author is right, beside fraud 'noise', fraud happens in waves.

<comment><AUTHOR> That was my mistake, assuming that since I was OK with the noise, everything is fine. Didn't know about the fraud tsunamis.

<comment><USER60> Was stunned two when hit the first time by such a tsunami. Your blog post is great, don't think this is common knowledge.

</comment></comment></comment><comment><USER61> Even PayPal or Google faced up to fraud. You can try Maxmind or manual check every order. Contact me to more solutions.

</comment><comment><USER62> What about services like maxmind-fraud and the like that authenticate buyers?

Are they too expensive per purchase?

</comment><comment><USER63> Unless there's some bug in your transaction processing system causing it to leak excessive amounts of data on why the transaction was denied it makes no sense for checkers to use it over their own merchant accounts.

(And typical checker traffic is more like thousands of transactions rather than tens)

It seems much much more likely that someone was just trying to card some candy.

</comment><comment><USER64> Would verifying zip codes mitigate this issue, or do the carders have access to that too?

<comment><USER65> ZIP codes seem to be a US-specific thing.

<comment><USER66> ZIP codes are just a funny name for post codes, and most places have postcodes.

<comment><USER67> Damn you Ireland and your "no postcodes" hell.

<comment><USER30> We have postcodes now! Or well, eircodes. Which are totally not postcodes, but rather unique house identifiers that happen to look as postcodes. And that noone will use.

[http://www.eircode.ie/](http://www.eircode.ie/)

</comment></comment></comment></comment></comment><comment><USER68> Sorry that you have to deal with fraud and loose money!

Could you start shipping biscuits or chocolate candy again, I am a bit disappointed with the stuff you currently include in your packages :-(

<comment><AUTHOR> Thanks for the feedback. Chocolates are not so good to send in this weather (melt risk), will start including those again during colder months.

</comment></comment><comment><USER69> That's why the whole credit card system is insane.

1) It's a pull instead of a push. Anybody with your CC details can attempt to charge it. Bitcoin model is much more sane.

2) There's not even a "approve the charge" option.

3) The cartels controlling the credit card systems charge insane amounts of money for transactions, attempted or real.

I really want Bitcoin or some other alternative to succeed.

<comment><USER54> Exactly. I hope we will see a world soon, where Bitcoin (or another cryptocurrency) is as ubiquitous as credit cards now.

Transaction fees for a failed credit card payment? Its more than insane.

</comment></comment><comment><USER70> One thing you can do is make the account creation process more difficult, maybe require phone verification, or social auth. It would tie a fixed value to each bad attempt for the fraudster.</comment>


How To Live Like James Bond For A Weekend - <AUTHOR> http://nerdfitness.com/blog/2011/09/12/live-like-james-bond/

<comment><USER2> Somewhat tangential, but:

 _It was at this point that the guy at the end took a hit when he shouldn't have and I almost jumped over the table and punched him.

Public service announcement – if you're going to gamble in a casino where other people depend on your play (like in Blacjack), spend an hour or two beforehand learning how to play properly, or ask the others at the table for advice so you don't screw them over. We gamblers thank you. The more you know..._

The actions of other players don't affect you statistically. Assuming the deck is fairly shuffled, it makes no difference to your expectation whether you get dealt the 10th or 11th card.

<comment><USER3> This person could also be counting the cards, and adjusting their play according to the cards remaining in the deck, i.e. they're a better player than you!</comment></comment>


Incredible CG Video of Asteroid Impact (Watch in HD) - <AUTHOR> http://jp.youtube.com/watch?v=-zvCUmeoHpw

<comment><USER2> This is by far the most disturbing (and beautiful) animations I have ever seen in my life.</comment>


Myth: HR is on your side. Reality: The Uber sexual harassment scandal - <AUTHOR> https://www.vox.com/first-person/2017/6/19/15830408/uber-sexual-harassment-scandal-hr <comment><USER2> One has to be incredibly naive to think HR is on the employee's side. Always follow the money:

Who pay's HR? Company; ergo - if HR is against company, HR gets replaced.

Same for binding arbitration: Who is the arbiter's repeat customer, company or client? company. ergo, if arbiter is not good for them, arbiter gets replaced.

Also, when you buy real estate, some people believe that _their_ agent is looking out for them. However, if you look at the money, one party (buyer) brings money to the table, and three parties (buyer agent, seller agent, seller) take money proportional to the deal amount; Therefore, everyone except buyer wants deal amount to be maximized.

<comment><USER3> Given that a lot of HR is deliberately duplicitous about this, and that apparently -- still -- other "authority" figures in many young peoples' lives don't address this (every college "career counseling" center in the U.S.: I'm looking at you), I think one bout of naiveté can be excused -- although still preferably avoided.

HR IS NOT YOUR FRIEND!

Hell, HR often is not even the hiring manager's friend. "Best practices" are often a poor fit to the real world and individual circumstances. Hiring managers often have to jump through hoops to try to get HR to act on the candidates they actually want.

The way to work HR, is to know that -- if they're competent -- they are all about risk management.

So, make what you want also the way they minimize their risk, to the extent you can.

When I left one of my jobs, some of my PTO time had lapsed per a newer "use it or lose it" policy -- this "loss" probably aided by another prior action, lumping "vacation" time and "sick" time into a single pool.

But, I'd done well by my team. And HR didn't want any unnecessary waves. So, they paid it to me, regardless.

It was also a time of enough organizational transition, that this individual action would most likely "get lost in the wash" and not become the basis for another employee claiming unequal treatment.

THAT's how HR works.

They help you, only when it helps them.

The rest is all empty rhetoric. If you buy it, they just gained (gamed) some cheap compliance from you -- that's all.

P.S. As always, individual circumstances _may_ happen to vary. But regardless, go in with your eyes open and your guard up.

</comment></comment><comment><USER4> At least this article gets it right in identifying this as an industry-wide problem instead of narrowing responsibility to a single company. I can only attribute this to the fact that this article was written by an HR professional instead of a journalist, who often confuse cause and effect according to their own personal biases and agendas.

[http://www.patheos.com/blogs/geneveith/2011/08/the-murray- ge...](http://www.patheos.com/blogs/geneveith/2011/08/the-murray-gell-mann- amnesia-effect/)

Correctly identifying this as an industry-wide problem means that all HR departments get appropriate scrutiny instead of focusing scrutiny on one specific HR department, which has the unfortunate effect of causing society to ignore problems that exist in practically all HR departments.

</comment><comment><USER5> Yes, yes, it's become a cliche to point this out. (See: every comment posted as of this moment.)

But that's not the real lesson here -- we now see that this stonewalling/gaslighting _didn 't even help the company_, since HR were _so_ hardcore about being _so_ kafkaesque that Uber got stuck with a major PR scandal that cost numerous managers they were ostensibly protecting!

That's a particularly nasty kind of frustration -- not simply when someone opposes your goals, but does it so confusedly that they're hurting themselves in the process and yet you're still powerless to stop it! (Anyone can relate to Fowler's experience if they've had to deal with that kafkaesque, self- hurting bureaucracy "that will make you look bad if anyone actually takes an interest in the case".)

So no, it's not enough to dismiss this case as "HR protects the company,"; that logic wouldn't explain why, in this case, they "protected" the company by doing extremely risky things that could blow up badly.

</comment><comment><USER6> Does anyone actually think that HR is on their side? :p

<comment><USER7> When I was just starting my career, I did. I think a lot of people assume that they are there to help and protect you.

<comment><USER8> Sniff. I remember when I was young and naive too. Actually, it may have been mostly true thirty years ago, before the workplace became a minefield.

<comment><USER7> There's nowhere that teaches you this lesson except hard experience. I found out when I was accused of theft by security at my office.

</comment></comment></comment></comment><comment><USER9> HR is always on the side of the company. Think Cattle Resources in a farm.

I remember a post that compared it to secret police. First rule in any country with secret police: stay clear of secret police. I can't find it but there are plenty of them. Examples

[https://theartsmechanical.wordpress.com/2016/02/02/the-hr- se...](https://theartsmechanical.wordpress.com/2016/02/02/the-hr-secret- police/)

[http://www.cmo.com/opinion/articles/2015/11/25/is-hr-the- new...](http://www.cmo.com/opinion/articles/2015/11/25/is-hr-the-new- corporate-.html)</comment>


Why are there two tides a day? - <AUTHOR> https://cosmosmagazine.com/geoscience/why-are-there-two-tides-day <comment><USER2> Apparently, the author is wrong. See "Does Earth really have two high-tide bulges on opposite sides?" ([https://physics.stackexchange.com/q/121830](https://physics.stackexchange.com/q/121830)) on Physics StackExchange.</comment>


Moving Content to a new Domain and Keeping SEO - <AUTHOR> http://www.renownedmedia.com/blog/moving-content-to-a-new-domain-and-keeping-seo/

<comment><USER2> You don't need to group your regexes if you're not using the captured values for anything.

Don't use .htaccess if your site actually gets traffic. It requires a stat/file read on every request you receive. Put it in your VirtualHost in the Apache configuration.</comment>


David Byrne (Talking Heads) on Creativity, Writing and Collaborations - <AUTHOR> http://journal.davidbyrne.com/2010/03/031810-collaborations-updated.html

<comment><AUTHOR> "To have one's work responded to by another mind, or to have to stretch one's own creative muscles to accommodate someone else's muse, is a satisfying exercise. It gets us outside of our self-created boxes."

It's one thing to get feedback - but to try and actually _create_ something with a person that you're not used to working with every day seems much more challenging.</comment>