Police detectors to warn mobile phone-using drivers - <AUTHOR> https://www.bbc.co.uk/news/uk-england-47896472 <comment><USER3> This seems like a silly idea - as the article says, there's no way for it to tell whether it's the driver or a passenger, but even if there's only one person in the car, there's no way to tell if the person is actively using a mobile phone or if it's just sitting in the car and there's some app running on it that happens to be transmitting data...

Then add in the likelihood that it'll pick up all sorts of random interference and have all sorts false-positives... It'd probably be just as accurate if it just had a sensor to detect cars, and flash the warning at a randomly chosen 30% or so of cars...

<comment><USER4> Considering the failure mode is fatal, it's not that unreasonable to expect people to turn off their phones in their cars in my opinion.

If you are driving, you can't respond anyway, so just wait until you get to your destination to reconnect.

Phone use is now killing more people than drunk driving so I don't feel like it's an over-reaction. Drivers had their chance to be responsible and they killed a bunch of people instead.

<comment><USER5> Phones have legitimate uses in cars. Satnav, music, or even just an indication that you just pull over and answer a phone. Given that I've never used one illegally, I think it's fair for me not to want to go through a power cycle for every journey.

How could Uber work without one?

<comment><USER4> The utility for devices in a car is obvious, no one is arguing that they aren't useful. But that's just a shame, because using them kills people every day and we haven't found a way to police it yet. Power cycling may just be a caveat of the policing methods.

Driving is a privilege that is constantly abused. Perhaps we just couldn't handle unfettered use of devices in cars the same way we couldn't handle drinking and driving?

I am all for some kind of controlled way of safe phone use in cars, but it doesn't exist. Better to remove the temptation until it does.

<comment><USER6> That sentiment can be made about a lot of things including cars themselves:

"The utility for cars are obvious, using them kills people every day. We just couldn't handle unfettered use of cars. I am all for some kind of controlled way of safe use of cars, but it doesn't exist. Better to remove the temptation until it does."

<comment><USER4> Sure, but not about the downsides of removing them. It's a balance of pros and cons. Having people stop using their phones at all in a car wouldn't be a huge deal. It is only a recent problem after all. Not having the automobile would be devastating to the economy and people's lifestyles.

The pros of phone use in a car are real, but shallow. The cons are that they cause accidents. It's heavily weighted toward cons.

</comment></comment></comment></comment></comment></comment><comment><USER7> For context, across the UK we have similar systems for speed limits.

They are not used for enforcement or prosecution. They are informational boards. Actual speed cameras are different (usually a yellow or blue box that doesn't give any feedback).

It's basically a sort of 'big brother is watching' reminder.

Some of them show your actual speed on a big LED board, leading to individuals attempting to get a "high score".

It sounds from the article like this would be the same. A board that flashes a mobile phone if you might be using one.

As far as I'm concerned, it's a complete waste of time (because I know these boards are meaningless), but the other side of the coin is that a decent number of people actually are afraid of enforcement action due to the signs and slow down.

<comment><USER8> It's not fear of enforcement that motivates people to act on these signs, it is the moral reminder that their behaviour is not socially acceptable (not to mention illegal). For a large group of people this works. Speeding (and being a smombie behind the wheel) is something people often do unintentionally, or with a rationalization mindset of 'everybody does it' (or something along those lines). For both these cases a reminder without further consequences can guide them to better behaviour.

For the rest who just make their own rules enforcement remains necessary of course, but it helps if you can get a good number of people to cooperate intrinsically.

<comment><USER7> I suppose it depends a lot on your local area.

Where I live there are tons of these 'flashy speed limit' signs.

I don't think I've ever seen anyone obey them. 35-40mph in the new London '20 zones' is completely normal behaviour if the road permits; everyone does indeed do it, including buses and off-duty police/ambulances.

It is, in fact, empirically socially acceptable (by the drivers if not necessarily by people who live there).

In areas with speed bumps you see much lower speeds because people don't want to damage their vehicles. Enforcement, basically. The same is true for real speed cameras, people tend to speed up and slow down.

That said, mobile phone use is different. I think most people intuitively recognise that using a phone at the wheel, particularly in an urban area, is really quite dangerous, and maybe a sign would work.

<comment><USER9> Don't want to sound condescending about drivers but on my street I see plenty of people who don't alter their behaviour with the speed bumps - <USER2> because they are not observant or it's not their vehicle (the council rubbish vans get thrown over them at high speed by their angry careless drivers!)

</comment></comment></comment></comment><comment><AUTHOR> The article suggests that the detectors will detect 2G, 3G and 4G usage, but will recognise if a bluetooth hands-free kit is in use. This sounds like complete snake oil to me, please could someone explain if this is possible?

<comment><USER10> And what if you're using your phone for navigation? It will still transmit data to google maps or whichever maps service you use, but its usage would probably considered legal.

<comment><USER11> Heck, your phone could be on silent in your pocket and a push notification might set this off. Your phone still has to transmit the acknowledgment.

</comment></comment></comment><comment><USER12> I'm in Houston right now, no car, so we walk around here (it is very rare seeing someone not in a car here, no public transport, but that's another topic) what really baffled me was the amount of people on there phone while driving. Its insane. I read about that problem. But I thought it was exaggerated. But hooollyy Molly. It is a lot. There was even an add on tv for a collision detector along the line of:while distracted by your phone, it warns you about an upcoming collision!

</comment><comment><USER13> I use my phone every day for Google Maps, surely that is going to be a false positive?

</comment><comment><USER14> "The technology will not be used as an "enforcement tool", the forces said, but was instead aimed at educating motorists..."

I hope there will be enforcement tools in the near future and I believe the fines and other forms of punishment should be very high. First offence could mean no driving for a month along with a huge fine that is based on salary, second offence something along the lines of losing your license completely and having to go through the complete process of examination, etc after a year of reflection without driving, again with a hefty fine. Apparently you weren't that interesting in driving to begin with, in the bus you can look at your phone all you want. Third offence means you don't get a license for 20 years. Killing/injuring someone while using your phone in the car should result into some form of prison sentence.

In my eyes there is 0 excuse for being engaged with your phone in the car in any other form than passively while using it for navigation. The worst are people who are looking at messages and writing.

A few weeks ago I was almost hit by a driver who sped over the crossing I was walking on. I was paying attention enough to see him with his phone in his hand, looking at something on the screen, never looking up and I was just able to stop walking. I don't know if I'd been killed if I weren't paying attention, but it wouldn't have been pretty.

I don't even care that much about myself though, I already have little faith in people's ability to drive properly so I pay attention and hope that keeps saving me. What I worry about are my children, who are bound to make some mistakes when crossing streets, etc, them being kids after all. The idea of someone killing them because they had to make a lame joke in a group chat in order to earn some emoji-based kudos is insane yet somehow not impossible.

</comment><comment><USER15> Using a phone wile driving is one of my pet peeves, but this doesn't seem like an effective approach. If a problem person is already distracted (not a false- positive), will they even pay attention to the flashing sign?

Just last night I had to knock on the hood of a woman using her phone while driving in the parking lot after she nearly ran over my wife and daughter walking to a restaurant. She was completely oblivious to the goings on around her and got extremely offended when I made the "hang up" gesture and told her to put down her phone while driving. ¯\\_(ツ)_/¯

</comment><comment><USER9> It should just turn on when there's a car as that will be equally effective as the system (with its flaws) and then they could save the £6,000

</comment><comment><USER16> Anyone using Carplay or Android Auto is going to get flagged by this. This is stupid.</comment>


Nifty Modals - <AUTHOR> http://tympanus.net/Development/ModalWindowEffects/

<comment><USER2> Try the modals on this site by clicking "Sign Up," submitting without filling in the form to see the error, and closing by clicking the X.

[http://canarykno.ws](http://canarykno.ws)

Should I open source it?</comment>


Show HN: Send Stylish Messages on WhatsApp,Messenger and More - <AUTHOR> https://github.com/bauripalash/lwing <comment><USER2> An example in the README would be nice.

<comment><AUTHOR> Thanks for suggesting, adding immediately</comment></comment>


Investing in domain names gives higher returns than investing in NASDAQ - <AUTHOR> http://www.eurodns.com/eurodns-news/article/domain-name-investing-buy-buy-buy-#.T7nxcL8tiHj

<comment><USER2> Selecting your data can make your numbers look better than numbers that are generated in a more authoritative fashion. If you're in the business of developing a market of suckers, this may be a profitable practice.

</comment><comment><USER3> Investing in domain names also makes you a horrible person who is helping to ruin the internet. Then again, the same could be said for those investing in Facebook stock.</comment>


The Master of Money: "The Snowball: Warren Buffett and the Business of Life" review - <AUTHOR> http://www.tnr.com/story_print.html?id=12ef5554-1023-4be9-ad93-681003b280ef

<comment><USER2> _His share price is still off more than 40 percent from its highs, underperforming even the S &P 500._

Oh, wow. I thought Lewis was in finance, but apparently he doesn't realize that the peak-to-trough performance of one member of an average is very likely to be higher than the peak-to-trough performance of that average. This is true, even with a two-stock index:

        Year  Stock A  Stock B  Avg.
        0     10       12       11
        1     12       10       11
        2     5        5        5

Note that A and B both had a larger peak to trough loss than the average. And yet, the average consists of A and B.

 _e railed against derivatives as weapons of mass destruction, and now turns out to have been sitting on a $68 billion pile of credit default swaps and exotic put options on various stock market indexes._

He complains specifically about counterparty risk. Someone who read what he said would know that; someone who heard thirdhand quotes might not. The derivatives he holds don't have any counterparty risk for him, because he's getting all the cash up front and paying out based on future events.

</comment><comment><USER3> I read it recently and liked it quite a bit. I think one of the things that stuck out for me is that he and Graham were taking this sort of 'quant' (using the term very loosely, as some people here do that, and I don't really know about it in detail) approach back in the 50ies - they absolutely poured over the stats in an era when that wasn't as easy as it is now. Now, Buffet is big enough to be able to do stuff ordinary investors can't, like buy entire companies.

In other words, perhaps part of his success belongs to a bygone era that won't come around again.

<comment><USER4> Graham and Buffet engaged in"fundamental analysis", which attemptes to determin an implicit value of a company based on business (accounting/financial) information. The traditional source of this information is from balance sheets, income statements, sales figures, and talking with management.

This is drastically different from "quant analysis", which tries to determine a stock price, or engage in arbitrage, from previous movements, relations to other stocks, and a host of other data largely irrelevant to computing the implicit value of a company. This information comes from real-time data streams from god-knows-where.

<comment><USER5> that and buffett has been outspoken against quants. he has also been accused of 'being in the right place at the right time' or 'lucky' quite a few times, but this is not the case.

another good book on buffett which has some sections dedicated to buffett vs quants and buffett vs 'its all luck' ppl is "The making of an American Capitalist" ([http://www.amazon.com/Buffett-American-Capitalist-Roger- Lowe...](http://www.amazon.com/Buffett-American-Capitalist-Roger- Lowenstein/dp/0385484917)) -- first book I got on him, perhaps the best (got snowball, haven't read it yet).

<comment><USER2> The books go well together. "Making of" is way more hagiographic, and less detailed.

What would be really great would be a wikified list of his transactions, linked to the financial statements and news stories that were current at the time.

</comment></comment></comment></comment><comment><USER6> Great article worth the long read.</comment>


The Idiot's Guide to High Frequency Trading - <AUTHOR> http://blogmaverick.com/2014/04/03/the-idiots-guide-to-high-frequency-trading/ <comment><USER2> The NYSE has always required market makers continuously buy and sell the stocks they specialise in. They must do this in all market conditions, even if it means running a loss. In exchange, they get privileged access to order flow information. This is why those seats are valuable. NASDAQ challenged that model by removing the physical trading floor. Instead of humans standing in a pit there were humans sitting behind screens. But in both cases there is an institution guarding access to the market's nerve centres. Barriers to entry were raised and the incumbents protected.

HFT takes the abstraction one step further. Democratised is that access to privileged order flow - anyone who can pay can get it. Lost is the role of a designated market maker - HFTs can pull out when markets get rough. This is a valid debate.

But don't confuse yourself. The order flow privileges being criticised have always accrued to institutional market makers. It is no surprise that those incumbents are the ones driving the present lobby.

<comment><USER3> > The NYSE requires designated market makers to buy and sell the securities > they specialise in. They must do this even in violent market conditions

This "mandate" tended to break down when tested by actual market panics. Bids dried up in 1987, and histories of the 1929 crash often mention clerks and floor runners who entered stink bids at $0.01 for stocks that had recently traded at $30 or $40, and got filled because the specialists had abandoned their posts.

Whether human or computer, market makers know better than to stand in front of a freight train.

<comment><USER2> Market makers can declare "self help" in the event of technical difficulty. Exchange rules prohibit declaring self help to avoid adverse market conditions. In practice, self help declarations have a peculiar habit of cropping up in rough markets. Nevertheless, "guarantors of market liquidity" remains a rallying cry for traditional market makers.

</comment></comment></comment><comment><USER4> If HFT GUARANTEES profits, why are the profits to HFTs declining so sharply, and why do HFT firms make such a small fraction of what the buy-side firms make?

Much more discussion here:

[https://news.ycombinator.com/item?id=7531429](https://news.ycombinator.com/item?id=7531429)

<comment><USER5> Declining profitabilities caused by increased competition. What market risk are the scalp-style of hft strategies (not all hft's are scalping) taking ? If they can cancel orders at abandon and make pennies if they win the race (against other hft's), but can simply x out of their order if the price doesnt go their way is as close to a riskless profit as it gets.

virtu's prospectus as a case in point. [https://www.sec.gov/Archives/edgar/data/1592386/000104746914...](https://www.sec.gov/Archives/edgar/data/1592386/000104746914002070/a2218589zs-1.htm)

<comment><USER6> Manoj Narang of Tradeworx has stated their average holding time is up to 10 minutes. Is he taking risk?

[http://washpost.bloomberg.com/Story?docId=1376-N2CB0F6TTDTQ0...](http://washpost.bloomberg.com/Story?docId=1376-N2CB0F6TTDTQ01-3SAG95I8F8SD907LL6IA4CLK0P)

<comment><USER5> 'Average' holding times don't help understand the issue. You could have one position that was a long-term bet

edit: I did not imply that none of the hft strategies were taking market risk. The ones that scalp certainly seem to.

<comment><USER4> This is the second time you've used the word "scalp", as if all liquidity on the public markets for the last century weren't funded by "scalping".

In the absence of "scalping", trading in stocks works like trading in houses. There are lots of buyers. There are lots of sellers. In the majority of cases, they disagree materially on the correct price. Therefore, it (a) takes forever to enter or exit a position, and (b) often forces people to accept terribly unfavorable pricing.

The "scalp" market makers take is the market price for always having a counterparty willing to trade with you at a price near the true market value of the trading instrument.

If you want the markets to work more like the real estate market, you can do that: place limit orders. The fact that market orders carry a premium price isn't a subtle detail of the market; it's trading 101.

All things being equal, you want the "scalp" to be as thin as possible. The wider the spread, the closer the scalping blade comes to the skull. Liquidity has a price, and investors want that price to be as low as possible.

So now, an exercise for you: at the height of HFT profit-taking, was the price of liquidity (a) lower or (b) higher than it was during the 1990s?

<comment><USER7> Though houses also have the additional complication that they're not (at all) fungible.

</comment></comment></comment></comment></comment></comment><comment><USER8> Complete idiot's guide is right.

Exchanges offer price-time priority. A 'better' price (higher bid, lower offer) gives you priority over a 'worse' price. Given two orders at the same price, an earlier order gives you priority over a later order.

If you spend millions of dollars on computers, data feeds, and salaries to skilled personnel, to predict the motion of markets _correctly_ and work within the system to make money, then you've earned that money fair and square. Companies that don't operate within the rules get shut down fast and hard by the SEC and/or FINRA. This is a _highly_ regulated and policed industry.

HFT is hardly different from Warren Buffett identifying a company with a 10-100 year time horizon. Should we penalize Buffett because he's smarter and better at that game than everyone else in the world? Is he scamming the person from whom he bought shares in 1984?

As Knight Capital showed us two years ago, you can spend those millions, and _still_ screw up and lose $10M per minute for the better part of an hour until your firm is bankrupt. No one is guaranteed to make money.

Posting an order and canceling it is fair game. When the order is in the market, it's a live intent to trade. When it's canceled, it's no longer a live intent to trade. Why is this "moral" when it's a person doing this on a time scale of seconds or minutes, but "fraud on the market" when it's a computer doing it on a time scale of micro or milliseconds? If someone hits that HFT computer's bid, there's an _obligation_ to fill that order - nothing is different about such an order just because it's placed and canceled quickly. The only objection is fear of the unknown. And the only difference here is that it's being done based on an automated strategy, faster than people can react. Your broker has access to HFT tools. Your 401(k) manager has access to HFT tools. If you're a day trader, your order is probably being routed through HFT tools.

<comment><USER9> It's more like taking a peek at the other players' cards, when only you have the ability to do that.

We have a bunch of laws that establish the pretense that everybody in the market is equal. I suspect we should get rid of most of the laws and drop the pretense. Folks should participate with the understanding that there are unfairly advantaged operators at all levels. But until that happens it's hard to morally square aspects of HFT.

<comment><USER6> No one can see the other players cards in trading unless the player shows them. Brokers sending IOIs to dark pools releases info to the market. Hitting venue A and then venue B in a serial fashion will release info the market, letting some traders cancel before you get to B.

There is no mechanism on the lit markets for anyone to see an order BEFORE it interacts by either posting to the book or being crossed with another order and generating a trade. Anyone who says otherwise is unfortunately misinformed.

<comment><USER10> Flash orders

<comment><USER6> No longer available on lit US equities. A failed experiment by exchanges to compete with internalizers by providing the opportunity for price improvement to those orders who elect to be shown to participants prior to posting or taking.

Can you think of one active in today's market?

<comment><USER10> On which instruments are flash orders still operational?

<comment><USER6> In US equities? None on lit venues. I'm sure some dark pools still handle IOIs.

<comment><USER10> Outside of US equities.

<comment><USER6> No idea. My game is US equities.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> If you want to know what the actual effects of HFT are on individual investors rather than hypotheticals, a study was done in Canada last year on that exact question. Spoiler alert: HFT is good for the little guys.

[http://qed.econ.queensu.ca/pub/faculty/milne/322/IIROC_FeeCh...](http://qed.econ.queensu.ca/pub/faculty/milne/322/IIROC_FeeChange_submission_KM_AP3.pdf)

<comment><USER12> 1\. That paper hasn't been published or peer reviewed.

2\. One of the authors used to work for HSBC as a derivatives trader.

3\. That study analyzes the impact of a fee-per-order regulatory regime. It is not a study, as you claim, of a world with high frequency traders vs a counter factual world without them.

</comment></comment><comment><USER13> "With these changes the fastest players were now able to make money simply because they were the fastest traders."

Directly contradicts "speed is not a problem".

Speed directly translates into being able to "front of trades from slower market participants". If they did not have the speed, they could not algorithmically predict the actions of (or react to activity on one exchange before it reaches other exchanges) and jump in front of slower participants.

There's other issues on top of this, such as paying for early or privileged access. But, if you are faster, you can successfully arbitrage across time or distance(really same thing as time)

</comment><comment><USER14> Correct me if I'm wrong but couldn't this problem be solved by some simple rules or act of legislation ?

For example, make it a rule that when you buy a stock, you have to hold it for at least some time - say a minimum of 5 minutes.

Or you could just impose a fee/penalty on traders that buy sell the same stock within one day of trading.

(note - I'm merely speculating here and have no specialized knowledge)

</comment><comment><USER15> My (non-expert) take: HFT is a tool and it can be used for good or bad. _Potential_ good uses may include: reduction of bid/ask spreads, improved liquidity and faster arbitrage of things like ETFs.

However, it can also be used for bad: [http://www.sec.gov/News/PressRelease/Detail/PressRelease/136...](http://www.sec.gov/News/PressRelease/Detail/PressRelease/1365171484972#.Uz75MfldXm4)

In that link, the SEC charges that a certain firm employed a trading strategy using non-bona fide orders to attract interest on another (bona fide) order. This involved submitting orders on the other side[1] to simulate interest.

HFT makes such strategies viable, or at least more viable.

1\. [http://www.nanex.net/aqck2/3598.html](http://www.nanex.net/aqck2/3598.html)

EDIT: Corrected to remove inaccurate information.

<comment><USER6> The action you link to was not perpetuated by HFT. The vast majority of Lightspeed's business is good ole fashioned screen traders, and that is likely what this was. Previous actions for spoofing and layering were also the result of manual traders using regular day trading tools.

<comment><USER15> My mistake. I assumed HFT was involved because the timescale of the "layering" orders and subsequent cancellation (following the bona fide order execution) was on the order of several hundred milliseconds.

</comment></comment></comment><comment><USER16> John Stewart had a really interesting interview with Michael Lewis this week about HFT:

[http://thedailyshow.cc.com/guests/michael- lewis](http://thedailyshow.cc.com/guests/michael-lewis)

</comment><comment><USER17> Just jump to the second comment for the actual details. Welcome to the internet, where if you are popular you can write any shit you like and people will read it, even if you don't actually know what you are talking about. (I wrote the software that runs some of the major stock exchanges. I did that for 13 years).

</comment><comment><USER18> Is "registering algorithms" really a good solution? It seems incredibly hard to analyze or simulate such a complex system.

</comment><comment><USER19> I really wish the "Idiot's guide to high frequency trading" would start out by explaining what high frequency trading is to me like I'm an idiot...

</comment><comment><USER20> Step 1) Have deep pockets, big balls and the right skills

</comment><comment><USER21> read a great article on this the other night

[http://www.nytimes.com/2014/04/06/magazine/flash-boys- michae...](http://www.nytimes.com/2014/04/06/magazine/flash-boys-michael- lewis.html?_r=1)

</comment><comment><USER22> Note that this: [https://news.ycombinator.com/item?id=7524315](https://news.ycombinator.com/item?id=7524315) is a better place to start than the linked post.

</comment><comment><USER23> The title of the article sounds oxymoronic.

</comment><comment><USER24> The solution is very simple: a Tobin Tax.

</comment><comment><USER25> I have a lot of experience with HFT. Lets clear all of this up.

How exchanges work for dummies.

All of the modern US stock exchanges work as a limit order book. Orders are filled on a price-time basis. In other words, you put in an order, if it isn't filled immediately it gets put in the limit order book with your order going to the end of the line ( lowest priority ) at that price. That's it, that is how it works. Lets look at an example.

I want to buy 250 shares of ABC. Right now there are orders in the book. 5 orders to buy at 9.95, each one for 100 shares. There are also 5 orders to sell at 9.96 also each order for 100 shares. I have several options.

(1) I can put in a market order to buy 250, when this order gets to the order book sell orders will be filled starting with the oldest ones, the ones that have been in the book longest, first. My order will be matched, this is why they call it a matching engine, with two of the sell orders and then a partial on the third one. As orders are filled in the order book messages are sent out to all subscribers of the data that those orders were filled so everyone can keep track of the state of the book. It is important to note that when my order gets to the matching engine/order book there is no chance for the orders in the order book to get cancelled. Those orders aren't told "you are about to get filled do you want to cancel". (2) I can put in a limit order to buy at 9.96 in which case the order will be filled as described in 1 unless the sell orders at 9.96 are cancelled before my order gets to the matching engine. (3) I can put in a limit order to buy at 9.95 in which case my order goes into the order book at 9.95 and I have to wait for the orders ahead of me to either get filled or cancelled and then I have to wait for someone to send in a sell order, either market or limit, that will get matched with my buy order.

That's it! Repeat, all day, every day. OK, that is a lie, there are special order types and that is one way exchanges try to differentiate themselves but that is a simplified way of how this works.

So where is the problem that Brad Katsuyama saw? I don't know him all I know is what I've seen and read. One thing to note is that he talks about buying large blocks of stock. Very large. From [http://www.nytimes.com/2014/04/06/magazine/flash-boys- michae...](http://www.nytimes.com/2014/04/06/magazine/flash-boys-michael- lewis.html?_r=0) "It used to be that when his trading screens showed 10,000 shares of Intel offered at $22 a share, it meant that he could buy 10,000 shares of Intel for $22 a share." 10k shares at $22k/share, that is $220k in value. So he isn't talking about some small retail order, this is, as he points out, institutional flow.

So lets think about what happens in this kind of case. There are 13 exchanges, all running limit order books. They are geographically disperse and internally they are implemented differently and have different latencies and different message protocols etc. At some instance in time all 13 exchanges have some orders to sell at $22.00. Think about that. In some sense that is awesome. Look how efficient the market is. If you went to 13 gas stations what are the chances that a gallon of regular gas will cost exactly the same at every station? Anyway, back to our problem. You want to buy 10,000 shares. You fire off your order to exchange A to buy 10,000 shares. This hits the matching engine and you get filled on 10,000 shares ( THERE IS NO WAY FOR THOSE SELL ORDERS THAT ARE IN THE BOOK TO BE CANCELLED!!!! The orders in the book aren't sent a message "hey, a 10,000 share order is coming in and about to hit your order do you want to cancel your order first?". THAT DOES NOT HAPPEN!!! ). The thing is that those 10,000 shares that got matched, lets say each order in the book was 200 shares. Your 10,000 share order just took out 50 orders. The fact that these orders were executed is sent out to every data subscriber who is listening for that data. All messages are timestamped, sometimes down to the nanosecond. From that the recipients can tell "wow, all of these 50 orders for 10,000 shares were executed at the same time so that was one order."

Now, if you had orders in at other exchanges to sell at $22 what would your response be? For a second lets think about something more mundane, like TVs. Suppose you work at Target and you have a friend who works at Walmart. You're working one night and a guy shows up driving a semi-trailer. He walks in and says "How many 50 inch flat screens do you have? I'll buy them all, as advertised at $400." You sell all 20 that you have to him. As he's walking out the door he says "Now I'm going to Walmart." Just then your buddy at Walmart texts you and says "what's up? slow night right." and you go "Hell no, just sold 20 50 inch TVs to a guy, he bought everything I had and says he is headed for your Walmart to buy more." Now what does your friend do? In a world where prices are set by supply and demand Walmart could look at that and go "wow, there is a big buyer of 50 inch TVs out there, lets increase our price from $400 to $410." And so that is what Walmart does and sure enough the guy comes in and buys all the TVs from Walmart for $410. Does the TV buyer say the market is "rigged"?

In any case that is very similar to what happens in this case. All of the HFT and market makers see this big order rip through a lot of sell orders at exchange A and they go "wow, there is a big buyer, I better change my sell order from selling at $22.00 to selling at $22.01 or $22.10 or $25 or whatever". Brad Katsuyama looks at that and goes "I've been front run, this market is rigged." And then he has a piece of software written so that all orders arrive at the different markets at the same time, as opposed to leaving his trading workstation at the same time, and suddenly there isn't any front running. There isn't any front running at all, what he is seeing is the reaction to supply and demand.

HFT and market makers live in fear of something called "adverse selection". Market makers are required to post a two sided market, they are supposed to put up a price and quantity that they are willing to both buy and sell at right now. They publish this and they can't back away from it, they have to live up to it if another order shows up to buy or sell at a price that matches they have to execute the trade. The risk to them is that they are trading with someone who has information that they don't have. Just like playing poker or any game, you're going to lose if you play with people who are better than you. In this game though sometimes just having a big order is enough to move the market even if the person behind the order doesn't actually have any new information. Market makers and HFT either want to buy on the bid and sell on the offer and make the spread or they want to do arbitrage. That is how they make money. They lose money if they trade with people who are smarter or faster than they are. As soon as they see big orders come in they get really skittish and so they immediately move their orders to reduce their risk.

I've written a lot and bored almost everyone. Ask away and I'll do my best to answer your questions.

A couple other comments:

1\. I'm not saying HFT is good or bad, I don't know. 2\. Lots of people mention costs. Just to ballpark the whole thing. For the technology you need to Arista switches, that is $500/port, get a 48 port switch, that is $24k. For servers you need something decent, not crazy, dual CPU, 64 GB, running linux, maybe solarflare for kernel bypass networking, custom software. The hardware will cost you $6k/server. Then you need a connection to an exchange, varies but lets say $10k/month, then you need market data, $5k/month, then you need an order entry port, that is maybe $1k/month. That's it. Well, that is at one exchange, then you need low latency 10 Gbps cross connects between exchanges and you need to do that at each exchange. Then you need to find a clearing firm and you need to become a broker dealer ( exchanges and clearing firms only talk to BDs ). This is a highly regulated environment. You must have a compliance person, you will get audited yearly by SEC, your exchange, FINRA, all kinds of stuff. You will need a serious accountant. You have to have written procedures. 3\. And how in the world do you avoid the Knight type of meltdown? Well, when you send in an order the order goes to the exchange. But the exchange also sends the order out via a "drop copy". This might go to your clearing firm so they know you aren't losing tons of money. But you can also get the drop copies and then you can constantly do your own risk checks that your orders you think are open are actually open but the ones you cancelled are actually cancelled. Most exchanges also have "cancel on disconnect" so when you really don't know what is going on you shut off your connection to the exchange and any open orders in the order book are cancelled for you.

</comment><comment><USER9> Some aspects of HFT seem highly analogous to insider trading to me, and insider trading is currently illegal. Abusing the ability to cancel orders fast in order to ping for non-public data about other orders sure seems like an insider advantage. As I understand it this "pinging" is central to HFT strategies. Am I missing something?

<comment><USER6> There is nothing about HFT that is analgous to insider trading. Unfortunately the people describing it in public like Cuban simply don't understand what's going on. There is no non-public information at work here.

<comment><USER26> The information that the HFT traders pay millions of dollars to acquire before the rest of the market does is not public at the time that they acquire it by any definition of public that does not include having to pay millions of dollars to acquire the information.

<comment><USER6> That's fundamentally wrong. Direct feeds are available to anyone who wishes to purchase them. Do you have a source for a piece of data purchasable by a firm that isn't on the public side of the information pipe?

<comment><USER27> > Direct feeds are available to anyone who wishes to purchase them.

Out of curiosity, how much does a direct feed cost, and is there a practical limit to the number of simultaneous buyers (for example, physical server space in a data center)?

<comment><USER6> Price lists are all public. Look at nasdaqtrader.com or batstrading.com.

</comment></comment></comment></comment></comment></comment><comment><USER28> If HFT adds no value, rigs the game, adds cost to everyday investors and traders looking to make an honest fair buck from the markets, will it be allowed in the future?

I almost feel like SEC will come down on them hard.

Basically, I'm bummed at the idea that as individuals, we can't innovate in this field without ponying up several million dollars in startup cost.

<comment><USER6> It doesn't cost several million dollars. It does require some capital, likely less than your average valley angel deal. It is not easy and you'll probably fail, but you can certainly try. Phone up Nasdaq, they'll happily give you all you need to know along with pricing information on their website.</comment></comment>


Bone Lisp – Lisp Without Garbage Collection - <AUTHOR> https://github.com/wolfgangj/bone-lisp/ <comment><USER2> Since ulisp (lisp for Arduino) came up on HN the other day, I've been trying to implement a lisp in C with garbage collection that will run on my x86 machine (your usual macbook).

It being my first ever "real" c program, I'm not quite there yet. Very curious to read more source code of small languages, preferably things that are implemented in one file only.

<comment><USER3> If you just want a GC, here's a tiny one I wrote:

[https://github.com/munificent/mark-sweep](https://github.com/munificent/mark- sweep)

And an article I wrote about it:

[http://journal.stuffwithstuff.com/2013/12/08/babys-first- gar...](http://journal.stuffwithstuff.com/2013/12/08/babys-first-garbage- collector/)

This was the inspiration for uLisp's GC ([http://www.ulisp.com/show?1AWG](http://www.ulisp.com/show?1AWG)). :)

If you want a more full-featured but still small language in C, you could take a look at Wren:

[http://wren.io/](http://wren.io/)

[https://github.com/munificent/wren](https://github.com/munificent/wren)

<comment><USER2> I am using your blog post about the mark and sweep-GC for my program! Thank you for publishing it! Very informative, entertaining and to-the-point. Thank you!

<comment><USER3> You're welcome! I'm glad you enjoyed it. It was a lot of fun to write.

</comment></comment></comment></comment><comment><USER4> This makes me wonder whether it would be worthwhile to design languages with no garbage collection or manual deallocation. Non-stack memory can be allocated but never freed, and memory exhaustion would be fatal.

We might be at the point where this is an acceptable simplification for very fine grained processes and services. Has anyone explored this model?

<comment><USER5> Just because memory can be freed doesn't mean it has to; the programmers can choose not to.

The mode of not freeing has been explored in programs that are short-lived and run on top of an OS that reliably cleans up memory. For instance, system utilities and compilers and such.

Got a C program that crashes due to premature allocation, according to Valgrind, and it is hard to figure out why? It it just a short-lived utility that does some job and exits? Then just #define free(p) ((void) 0) and rebuild.

<comment><USER6> Or because it's considerably faster—if you don't have to deallocate, you can just reserve a bunch of pages on startup and use a stupid simple bump allocator.

Actually, D's compiler more or less works this way: [http://www.drdobbs.com/cpp/increasing-compiler-speed-by- over...](http://www.drdobbs.com/cpp/increasing-compiler-speed-by- over-75/240158941)

<comment><USER5> I have seen a program, which meticulously freed everything on shutdown, take over a minute to actually do so under a large and complex test case involving lots of data. In production use, this would be a complete waste of CPU cycles.

<comment><USER7> Over a minute? At least it didn't take a few days...

[http://lists.gnu.org/archive/html/coreutils/2014-08/msg00012...](http://lists.gnu.org/archive/html/coreutils/2014-08/msg00012.html)

[https://news.ycombinator.com/item?id=8305283](https://news.ycombinator.com/item?id=8305283)

</comment></comment></comment></comment></comment><comment><USER8> As someone who hasn't dealt with Lisp since the 1980's, help me out here. Immutable structures and Lisp seems to be incompatible, as Lisp's lists are about the most mutable things you'll ever see.

If, for example, I want to put a value into the middle of a list, would I be doing something akin to. . .

MyList = MyList.firstHalf + NewValue + MyList.secondHalf?

And how doesn't this become a horrible bottleneck?

<comment><USER9> You can represent the list as a zipper data structure[1]. The idea is to represent a "cursor" pointing to some point in the list as a triple: (list of points before, point at current position, list of points after). So a zipper for an arbitrary list "xs" pointing to the beginning would be (() (car xs) (cdr xs)). There are some things you can do to handle the case with empty lists, which I'll leave to the reader :).

Given a zipper (as x bs) you can move the cursor right (rather, compute the zipper with the cursor one step further to the right, since we're purely functional) as ((cons x as) (car bs) (cdr bs)). You can insert a new element "y" into you current position by computing (as y (cons x bs)). If you discard the old version of the zipper every time you move or insert, this will use the same amount of memory as just storing the list, and you get insertions in O(N) time and O(1) memory. There's a whole field devoted to making data structures like this for purely functional languages, and this zipper concept extends itself rather naturally to trees.

[1][https://en.wikipedia.org/wiki/Zipper_%28data_structure%29](https://en.wikipedia.org/wiki/Zipper_%28data_structure%29)

</comment></comment><comment><USER10> Garbage collection should be in the hardware.

<comment><USER11> Intel tried this in the early 80s: [https://en.wikipedia.org/wiki/Intel_iAPX_432](https://en.wikipedia.org/wiki/Intel_iAPX_432)

<comment><USER12> Wow, the iAPX 432 sounds really cool. It contains a lot of ideas that I wish had taken off. Pity it failed.

That failure might prove that those ideas were bad. But I think it just proves that MS-DOS compatibility is really, really important.

Weep for humanity.

<comment><USER13> I always see those failures more as political than technical.

How mankind would be if there weren't a few lunatics that kept on trying to fly and fail?

Just to cite one example from many.

<comment><USER10> You are so right. One cannot succeed in technology outside of politics.

</comment></comment></comment></comment></comment><comment><USER14> It would be cool if the garbage collector could be written in the language itself.

<comment><USER9> In this case a garbage collector is unnecessary. In this Lisp, list structures are immutable, which means it is impossible to create a list with a cycle. Thus, the interpreter can use basic reference counting without fear that anything will leak.

<comment><USER15> In Haskell, data are immutable and yet cyclic structures are possible.

      allOnes = 1 : allOnes

<comment><USER9> This is possible in Haskell because it allows for lazy evaluation. In a strict language, immutability means you can't form cycles.

<comment><USER15> While non-strict evaluation helps with forming cyclic structures in general, in this specific example it is not required. You just need a language that allows names to appear in their bindings, such as in C:

      struct node { int x; struct node *next; } allOnesList = { 1, &allOnesList }, *allOnes = &allOnesList;

<comment><USER9> Haha, I hadn't thought of that! That ability to create self-referential data is the main factor. That isn't really possible in lisp without some sort of letrec function, which this lisp shouldn't support. You would also have to watch out for recursive function definitions. Usually, the function body will just contain the symbol that refers to the function, which conceptually is similar to a straight pointer to the function, and someone might try to optimize it that way.

In fact, with more thought, I realized the lazy evaluation is actually orthogonal to forming cyclic data. The "allOnes = 1 : allOnes" line doesn't necessarily form a cycle (depending on the interpreter/compiler, I imagine ghc and any other sensible implementation will form a cycle). Naively, this will just create a thunk which will get repeatedly evaluated, generating a long list of ones.

EDIT: Nevermind, you're right.

<comment><USER15> Thunks don't get repeatedly evaluated, sorry. Just once at most.</comment></comment></comment></comment></comment></comment></comment>


PostgreSQL vs. MS SQL - <AUTHOR> http://www.pg-versus-ms.com <comment><USER2> I scrolled to a random page. It was totally absurd, stopped right there.

"Crucially, because open-source software tends to be written by people who care deeply about its quality (often because they have a direct personal stake in ensuring that the software works as well as possible), it is often of the very highest standard (PostgreSQL, Linux, MySQL, XBMC, Hadoop, Android, VLC, Neo4JS, Redis, 7Zip, FreeBSD, golang, PHP, Python, R, Nginx, Apache, node.js, Chrome, Firefox...). On the other hand, commercial software is often designed by committee, written in cube farms and developed without proper guidance or inspiration (Microsoft BOB, RealPlayer, Internet Explorer 6, iOS Maps, Lotus Notes, Windows ME, Windows Vista, QuickTime, SharePoint..."

<comment><USER3> I know numerous guys on the SQL Server team and it actually causes me a bit of anger when I read a clueless statement like that, for the author I feel sorry for his ignorance; the anger comes from the fact that lots of clueless people will actually read this and believe it to be authoritarian because of the verbosity of it alone.

<comment><USER4> The SQL Server group (and their product) is one of the very few things in Microsoft that has earned my deepest respect. Not Windows, not Exchange and, certainly, not ShamePoint.

I frequently rank Microsoft's product line as follows:

1 - Natural keyboard - the one there is no reasonable replacement for (the kinesis keyboards are 5 times as expensive). If there is a reason my none of my machines is 100% Microsoft free, this is it.

2 - their mice - simple, precise, comfortable, inexpensive

3 - SQL Server - it almost makes having a Windows server around worth the pain.

Having said that, I have enormous respect for PostgreSQL. It's a very solid RDBMS and it's my database of choice most of the time. It is more comfortable to use from a command line than SQL Server will ever be (which is OK, because a CLI is not a high priority over there).

<comment><USER5> I just can't replace the keyboard.

</comment></comment></comment></comment><comment><USER6> As a DBA, that has years of experience with both. I disagree with you.

I stopped reading after the very first bullet, and just skimmed through most of the titles.

First, you can import/export csv's. Hell all you need to do to export, is right click the view and hit export, then select .csv.

Second, you don't want such a simple installation, MSSQL has several options/packages/configurations. If you don't need it you turn it off, otherwise you will be charged a fee.

That being said, if you have chocolatey installed in powershell. It's also a one line command, and you can configure your own custom installation command.

choco install MsSqlServer2012Express

Third, I don't feel like correcting the rest of your mistakes.

I'm not shitting on Postgres, it's pretty awesome, but out of the box MSSQL is better. Especially when you have hundreds of severs running it.

<comment><USER7> The author spends quite some time explaining that MS SQL Server's CSV is mangled and broken, and what you've said is that it exports CSV. So what does your comment contribute, except to say that you didn't read the article?

<comment><USER3> QFA "MS SQL Server can neither import nor export CSV"

<comment><USER8> I took it to mean "can neither import nor export conformant CSV"

<comment><USER9> Yeah, me too. The author points this out painstakingly too. I'm not familiar with MS SQL CSV exporting so I can't verify the truth of that statement though.

</comment></comment></comment></comment></comment><comment><USER10> This article is a _fantastic_ tour of some of Postgres' programmer-friendly features.

If you're interested in databases but aren't familiar with Postgres, it's a good read if you skip the criticisms of MSSQL. Some of those criticisms are spot-on, some are iffy, and some are rah-rah-yay-yay-open-source exhortations that seem to be lifted from Slashdot threads fifteen years ago.

(For whatever it's worth, I love MSSQL. Been using it for well over a decade. That said, I certainly don't mind reading criticisms of it.)

<comment><USER9> This is exactly the feeling I got from the article. While the author did state that he wasn't as familiar with MSSQL as with PostgreSQL, he didn't do very well in enumerating on the facts surrounding his criticisms of MSSQL. I think this greatly hurt his good points.

But yeah, an awesome resource for people dipping their toes in the Postgres pool.

</comment></comment><comment><USER3> There is so much here, it is so verbose...so religious.

There is the appearance of facts, but mostly it is bias by omission. Take for example the CSV complaints..."MS SQL Server can neither import nor export CSV. Most people don't believe me when I tell them this". This is just false, patently and obviously false. BCP and SSIS, both part of SQL Server have excellent support for importing and exporting flat files. I did keyword searches through the whole post, no instances of BCP, SSIS or Integration Services (the long term for SSIS).

I am sure others will break this thing apart section for section but I wish people would be fair and open about stuff like this - I mean, seriously if you are going to put the time and effort into a long document like this, don't be so blatantly bias.

<comment><USER11> They likely should have said "standards compliant CSV." I've worked a lot with CSV (too much!) and let me tell you Microsoft's tools are simply awful at following the CSV standard.

Excel in particular drives me up the will. Plus the automatic inferences in Excel break all kind of things. Have a six digit number? Randomly a date. Have a UPC? Going to corrupt it in numerous ways (leading zero stripping, converting to scientific notation, etc).

I can definitely sympathize with anyone who criticises Microsoft's CSV support. They really just want you to use an Excel format (either the binary format or the zip-XML one). However Excel formats are overly complicated for the type of data migration work CSV is often utilised for (XML is too large, it adds up, and requires custom code to import/export (even if that is XSL from CSV), JSON might be better but isn't "yet" popular).

<comment><USER12> _> let me tell you Microsoft's tools are simply awful at following the CSV standard._

There's no such thing. CSV is a convention that people mostly agree on but there's no written CSV standard and no CSV standards body.

<comment><USER13> RFC 4180 is close enough for me.

</comment></comment></comment></comment><comment><USER14> The author is pretty misinformed about how enterprises work.

a) Security is critical. It routinely trumps common sense and evidence. And telling people "not to fall for it" is advice that will get you nowhere.

b) Vendor support is critical. And no your local mom+pop consulting shop doesn't count. PostgreSQL could really do with solid, global companies like Microsoft, Datastax, Mongo etc who offer training and proper SLAs.

<comment><USER15> What vendor support do you get exactly from Microsoft? Their EULA typically has a SHOUTY CAPS NO WARRANTY CLAUSE, so what does the SLA cover? Have you successfully resorted to the SLA to handle problems you've had?

I'm honestly curious. I just don't know how SLAs work, and whom can you blame if you have problems with MS SQL.

<comment><USER16> Vendor support from Microsoft is about requesting help with issues. You get x number of support calls, depending on what you paid. Microsoft has fixed service levels for that, and you've little option other than paying more money if you want a quicker turn-around.

SQL Server itself doesn't ship with an SLA.

You'd get that from your service provider or systems integrator, because the customer chooses her own service levels (determined by business continuity objectives). Her SP or SI (or in-house IT department) would then deploy SQL Server in a way intended to meet the required service level. Specifically, the customer might opt for five nines (99.999%) availability. Such a deployment doesn't look anything like one that must only be available from 8am to 5pm on weekdays - even though both SLAs are met using the same SQL Server code base from Microsoft (but on vastly different hardware and network configurations).

The availability associated with an SLA usually goes hand-in-hand with disaster recovery (recovery time objective, and recovery point objective[1]), but can also apply to support turn-around (a support request is triaged, and based on severity is resolved within the amount of time specified for that severity by the SLA). As mentioned, if I'm not mistaken Microsoft has fixed service levels for that. IIRC an MSDN subscription gets you a small number of free requests. An Enterprise Agreement gets you a whole bunch more.

There are many other quality objectives you can specify with an SLA, including efficiency (capacity), integrity (security), and robustness (stability) [2].

So unless you're hosted by Azure or have an Enterprise Agreement, Microsoft is rather unlikely to provide you with an SLA. And even then _you 'd_ be the one telling _them_ what your service level requirement is. If Microsoft can't deliver on your service level requirements, you'd do it yourself, or get a systems integrator to do it for you.

[1] [http://www.druva.com/blog/understanding-rpo-and- rto/](http://www.druva.com/blog/understanding-rpo-and-rto/)

[2] [https://www.wittenburg.co.uk/Entry.aspx?id=d8c54975-bd0a-410...](https://www.wittenburg.co.uk/Entry.aspx?id=d8c54975-bd0a-410b-b266-d73c1f45c641)

</comment></comment></comment><comment><USER17> Three things I really enjoy about MSSQL are:

1) SQL Server Data Tools for complex schema management.

2) Graphical execution plans.

3) It's the database best supported by Entity Framework. If you know what IEnumerable is, and you know what IQueryable is, and you know what a leaky abstraction is, you can use EF to quickly start getting data in and out of SQL Server with very reasonable performance.

<comment><USER18> Just on your #2: I'm not entirely sure how the graphical executions plans help in the end. I've seen too many plans that wouldn't fit on 30" display. The postgresql explain text format is reasonable, and can even be automatically logged when queries are slow [1].

[http://www.postgresql.org/docs/9.3/interactive/auto- explain....](http://www.postgresql.org/docs/9.3/interactive/auto-explain.html)

<comment><USER17> If an execution plan doesn't fit on your 30" monitor then you've probably got a more sophisticated query that requires more analysis than simply eyeballing execution plans for clustered index scans or places where predicate pushing hasn't worked as effectively as you hoped.

External tools aren't really in the scope of this discussion, but while SSMS does a pretty good job of visualizing execution plans SQL Sentry Plan Explorer does a better job (a more compact view with a color gradient for the most expensive operators) and makes my job even easier.

[http://www.sqlsentry.com/products/plan-explorer/sql- server-q...](http://www.sqlsentry.com/products/plan-explorer/sql-server-query- view)

</comment></comment></comment><comment><USER19> I develop against MSSQL most of the time and have tinkered with Postgres a bit was agreeing with most of the points, until I got to:

> "but MS SQL Server does have a bizarre failure mode which I have witnessed > more than once: its transaction logs become enormous and prevent the > database from working. In theory the logs can be truncated or deleted but > the documentation is full of dire warnings against such action."

This belies what appears to be a fundamental ignorance of SQL Server and is not at all bizarre. If a database has been deployed in a non-simple recovery model, then the transaction log needs to be truncated as part of a backup procedure. If this isn't the case then you should have hired a/better DBA.

</comment><comment><USER20> > PostgreSQL: the docs actively encourage you to simply use the TEXT type. > This is a high-performance, UTF-8 validated text storage type _which has no > length limit_.

The emphasised part is incorrect, Postgres's text storage has a 1GB limit (that stands for CHAR, VARCHAR and TEXT which all use the same underlying mechanism): [http://www.postgresql.org/docs/9.4/static/datatype- character...](http://www.postgresql.org/docs/9.4/static/datatype- character.html)

> In any case, the longest possible character string that can be stored is > about 1 GB.

Although note that this is 1GB, you have to take in account both multibyte characters and compression (which by default is enabled on text values of more than 2k, using LZ)

<comment><USER21> That being said, if your relational database has 1GB records, you really need to reevaluate whether you are using the right tool for the job.

<comment><USER11> While I kind of agree, what exactly is the "right tool" for storing large blobs?

You can use the filesystem but then you have to manage data migration/sync by "hand" which is kind of a bitch for distributed systems.

Honestly storing large blobs with meta-data telling you about the blobs is extremely common. I'm yet to hear of how you're meant to do it correctly.

<comment><USER22> commonly, s3 or zfs nas for storage and pg for metadata

<comment><USER14> Commonly ? No not even close. The most common solution is that people just dump it as a BLOB in whatever database they are currently using. Which based on numbers is likely to be MySQL.

</comment></comment></comment></comment></comment><comment><USER23> I too am biased towards PG. At one of my previous gigs, I had to develop a HTML5 mobile app around a MS SQL database (though eventually we migrated to PostgreSQL). Among the problems we ran into:

A) Indexes disappeared without rhyme or reason. (This might have been a "too many cooks in the kitchen" thing, where someone restored from a backup and didn't tell anyone.)

B) In the default configuration provided by the php5-mssql package in Ubuntu, text fields were chopped at 255 characters.

C) Our Windows Server did updates, then came back up with a "you need to renew your license" lock that prevented any I/O and cost us half a day of productivity (on top of the licensing costs).

And of course, D) it screwed up the file encoding when we did the final export during the "migrate to PostgreSQL" step.

Incidentally, migrating our platform to PostgreSQL and kicking the tires was one of the last projects I completed at that job, and everyone was much happier for it.

<comment><USER24> I don't think your experiences with SQL Server are representative. I'm not a fanboy, I've run into more than my fair share of issues with MS SQL, but none of your problems are a result of any SQL Server version I've heard of. Your response screams FUD, to me.

Now, you say that you possibly had too many cooks in the kitchen. So perhaps your complaints are due to the other cooks, or maybe some of your complaints came to you second-hand. But here is my analysis of your complaints, having run SQL Server for a number of years:

A) Just doesn't happen. Index-related statements are transactional and ACID- compliant. (i.e.: you can, in a transaction, add, drop, and update triggers, indexes, entire tables, etc. along with data.) This has been the case since at least I've started using SQL Server with the 2005 release.

B) I don't know why this is the case with php5-mssql, but I would blame the authors of the package. varchar, nvarchar, text, etc fields all support > 255 characters.

C) I don't know of any version of SQL Server or Windows Server that does this. No Windows Server version "locks IO" on a license check failure (this would obviously cause huge problems with filesystem consistency). As far as I know, no SQL Server version does live license checking or phones home. I have no idea what your symptom was, but I suspect someone else broke something. A running SQL Server installation does not "stop", nor does it need the license to be renewed at any point. Cumulative and service pack updates don't do license checks either, as far as I know. In fact, an expired license doesn't affect anything, and probably even reflects a misunderstanding of how licensing works. If you purchased a license, even if you paid yearly for 2-3 years, your SQL Server license is good forever. Forever! New versions require software assurance, but you have to choose to install those manually, it's opt-in, not opt-out. Automatic updates should never break SQL Server.

D) Is it at all possible this was the migrate to PostgreSQL tool's fault? Because I've run weekly backups and restores of SQL Server databases for the past several versions for many years, and had no problem with a .bak file being corrupt.

<comment><USER23> D) Nope. I wrote the tool and tested it multiple times. The data kept coming out malformed, and playing with mb_convert_encoding() didn't really help much.

I appreciate the feedback though. This was my only time ever using MS SQL.

(Re: the 255 char thing, I had to edit the /etc/tsql/tsql.conf to change the sybase version to make the problem go away. Not a MSSQL problem, but definitely a problem with using it from a PHP + nginx on Linux env.)

</comment></comment></comment><comment><USER25> The author mentions no understanding N'V' etc, early on, then rants about MS's support for Unicode...

Also, IMHO one of the single biggest shortcomings from PostgreSQL is that there's still no baked in solution for replication with any kind of either multi-master or hot failover. If PostgreSQL supported this (in the box) as well as MongoDB was, along with having PL/v8 support easier to install, it'd be my db of choice.

MS-SQL does have a lot of shortcomings compared to PostgreSQL from a developer perspective, but from being able to easily install and administrate one over the other, I think MS-SQL has a significant lead here.

As long as replication/sharding are bolt-on solutions for PostgreSQL, it really isn't an option for a lot of projects.

<comment><USER26> Fortunately Postgres is getting a lot more replication love. You can already do a lot with the warm-standby and FDWs to do basic sharding, and they're rolling in a lot more to 9.4 and 9.5. EnterpriseDB are the place to look for that to be made in to a full product.

Someday soon Postgres will have materialised views that auto update :)

However, as someone who works with MSSQL daily they're still not comparable. I'd say Postgres is much nicer in development and I'd use it for my projects, but MSSQL has much more scalability without developer intervention.

</comment></comment><comment><USER27> Alright so somebody with experience in a wide variety of DBs help me out here. Are the enterprise solutions really any more scalable than something like postgresql or is it more that they're more accommodating an enterprise mentality where you'll tend to run into more skill level diversity on the extremes and team A often doesn't necessarily have team B's expertise at stuff? By accommodating of course I mean willingness to take a buttload of money to help teams out with problems when needed and/or to blame when their mistakes explode in their faces and to never ever try to tell anybody that they're doing it wrong once VIPs with enough hit dice are invested in that sort of thing.

I do gather that a lot of DB admins that aren't very political about it do in fact respect MS SQL for some of its more competitive features. I'm just wondering if there's any reason to start with it for a low-cost startup scenario that could ultimately result in a non-trivial but fairly straightforward DB schema with potential for being used by very large institutions (universities at the largest I'd imagine).

I'm ignorant enough (primary experience is in web UI) that I'm leaning towards postgresql because I like the way the Django guys think and they seem to dig it. Also the no-nonsense license and yes, the not-profit-motivated thing is nice when backed by a strong core group which I gather postgres has.

But do the enterprise DB solutions handle severely massive amounts of scale better for some reason? Or is it more that they're culture-friendly to the sorts of companies that typically handle DBs of this nature?

</comment><comment><USER28> For all its faults, this article does mention one thing about PostgreSQL that I have always been happy with: ease of installation.

Having been through many, many installs of Oracle and MS-SQL in my career, the speed at which you can get a PostgreSQL server up and running with data is simply awesome.

<comment><USER25> Now setup replication and a solution for hot/fast failover to a new master...

Not having an in the box solution for this is the single biggest thing keeping me off of PostgreSQL...

</comment></comment><comment><USER29> Hm. I love postgres and find many of its features to be intriguing and exciting. I work with MS SQL Server professionally and find many of the things it does to be excellent as well. The unfortunate thing about this piece is that much of it is actually Op-Ed and is based on the author's preference (e.g. cascading drop). The second section is nearly unreadable as it's full of FUD and smear.

Yes, there are things that PG does better than MSSQL. Yes, there are things that MSSQL does better than PG. It sucks that we still have to debate it based on a smear piece from someone with an agenda.

</comment><comment><USER30> MSSQL has some problems, some of which the post talks about. However, I tried pg out, planned to use it instead of MSSQL.

1\. HA, via replication, clustering, etc. are all fantastic as well as easy and trivial to setup. Once PG can offer a simple little wizard and setup replication, or shared-nothing clustering with automatic fail over, awesome. Or even tx log shipping with a few clicks.

Instead, last I tried (9.0 I think), pg drops you off with some weird system that makes you run shell as a specific user, where you run generic sounding commands. Oh and some default config that seemingly has some bad defaults you probably should change.

2\. Overall, MSSQL makes it easy to run a DB. I was doing a billion transactions a day (each which wrote to a few tables and also included a real ACID balance update), and I didn't need a full time DBA or have to have particularly awesome experience beforehand. With pg, I sorta got somewhere, but I had little confidence

3\. Development is far superior on MSSQL. Supporting other languages is a null point, because they aren't running as part of the query execution engine. That is, there's no real difference in using Python inside PG versus an external client, as far as I could tell. You still had to submit queries and make a transition. So TSQL seemed far nicer to work with than plpgsql. Although, the record types in pg were much nicer I'll admit. Another annoyance: pg didn't offer multiple returnsets in many situations. This made it awkward to run a sub function that needed to return results from different tables as separate queries. And the perf tools and UI was just so, so much better.

4\. There'd be strange "little" features you'd just expect to be there, like materialized views, which pg simply does not have. Pg still lacks materialized views. The current implementation is essentially pointless, as it doesn't update the view automatically.

I'm very much for PG, and believe it's an important project and am trying to use it for future development. Much because Microsoft went back on its word that it wouldn't move to an Oracle-style licensing where you pay for CPU power instead of just sockets. They've also made questionable decisions with Enterprise vs Standard, putting extra cumbersome limitations. They aren't adding features like JSON or arrays quickly. And also, I think it's important to run open source and try to make sure free systems stay viable.

But MSSQL has a lot going for it, and the ease of use and built in HA options don't even seem like goals for PG. If licensing weren't in the way, and I just wanted an easy system that was capable but didn't require lots of time, MSSQL makes a strong choice.

<comment><USER31> Interesting that you mention HA/replication, reading the article I was swept along with the pg evangelism but I fully expected at least a grudging concession to mssql on replication.

We use both pg and mssql, I was surprised the article didn't mention replication. While one might be justified in describing pg as more "developer friendly", mssql is more "enterprise friendly". As a developer, I prefer pg, but as a stack designer, I still lean more toward mssql. Put another way, if I'm joining a project as a developer, I hope they're using pg, but if I'm consulting and designing a stack for a company I'll probably recommend mssql, almost definitely if they're doing something that requires replication, and/or the team isn't going to have lots of db expertise.

I guess this is quite self-contradictory on one level but I think it's a fairly common view (having checked with some colleagues).

I know that pg has support for replication, but it certainly isn't as easy to set up as mssql, and you need to do a LOT of reading to figure out what flavor you should use, and it's not entirely obvious what features you get with each, or which ones are really recommended and which are semi-deprecated.

CTO: Do you support replication? mssql: yes, and it's pretty easy to set up. pg: well, we've got 3 or 4 ways of doing that, depending on what you want exactly, method A isn't really maintained any more, and method D looks like it's going to be great in a release or two, so maybe that's the one to bet on, but in the meantime you have to put up with a bit of pain.

(dramatisation, probably inaccurate paraphrase, but that's what the cto heard)

<comment><USER32> I've actually found MSSQL to be easier to use as a developer than PG. Setting up local instances, quickly making backups and what not, designing databases, are all much easier with MSSQL because of their incredibly easy to use tools.

I keep wanting to switch to PG every few months because it's open source, but keep being stumped by the lack of good tooling. pgAdmin is a joke, rarely have I ever used a more unintuitive piece of software.

</comment></comment></comment><comment> ely-s Microsoft has a thing for very long names though – possibly its greatest achievement ever is

        Microsoft® WinFX™ Software Development Kit for Microsoft® Pre-Release Windows Operating System Code-Named "Longhorn", Beta 1 Web Setup

</comment><comment><USER33> Tangentially, can anyone recommend a good book for learning PL/PGSQL and other ways to procedurally program on psql?

</comment><comment><USER34> Is there something like this for MYSQL? pg-versus-my.com?

</comment><comment><USER35> What about the lack of upsert in PostgreSQL?

<comment><USER36> Well, the same as the lack of upsert in SQL Server. Bear in mind that UPSERT != MERGE.

[http://www.postgresql.org/message- id/CAM3SWZRP0c3g6+aJ=YYDGY...](http://www.postgresql.org/message- firstname.lastname@example.com)

</comment></comment><comment><USER37> Pg is SOLID but I'm sure mssql have its merits.</comment>


Saving the Earth by tackling a local issue like shutting down a coal plant. - <AUTHOR> http://deregulatetheatom.com/2012/08/how-the-focus-on-climate-change-allows-complacency-macro-vs-micro/

<comment><AUTHOR> The point is that shooting too high like, for example, stopping global warming, is maybe much harder to achieve than a smaller local issue in which you can be more of a player and have a better chance at affecting change.</comment>


The state of Android updates: Who's fast, who's slow, and why - <AUTHOR> http://arstechnica.com/gadgets/2014/08/the-state-of-android-updates-whos-fast-whos-slow-and-why/ <comment><USER2> So basically, the vendors who maintain Android forks that diverge significantly from upstream take longer to ship updates, the carriers don't care about updates, and even slightly-aged phones get left out in the cold. It seems like all of that could be significantly helped by Android and the vendors' forks being operated like open-source projects, with continuous pulls and rebases as necessary to keep the forks in sync.

Unfortunately, the entire OS and all its feature set of stuff that's got nothing to do with cellular is still held back by the carriers "concerns" about controlling the devices on their network: the carrier-locking and attempts to prevent the device owner from having root access. We need to separate the connectivity from the user-facing OS, the way we have a dichotomy for our wired internet connections with ISP-controlled modems and user-owned and controlled routers. The hardware's pretty much set up that way already, though the carriers might want a bit more compute power available to them in the cellular baseband if they don't get to have access to the application processor.

<comment> k-mcgrady Why do US carriers care so much about modifying device software and branding etc. In the UK I don't think any carriers mess with the device in anyway. At least not that I've seen. Competition is almost completely based on coverage, price, data etc.

<comment><USER3> It actually costs money to compete on those things. Branding is comparatively inexpensive.

</comment></comment></comment><comment><USER4> It would be nice if OEM performance on delivering updates was tracked for all first-tier OEMs and more devices. This is great guidance for buyers of Android devices, but there needs to be more of it.

</comment><comment><USER5> The article doesn't mention the _huge_ number of unbranded/clone/not-well- known Android phones/tablets from China, which basically won't get any version newer than the one it came with (unless unofficial ROMs come out).

I have one myself, it's running a 4.2.x (rooted) and does what I need it to, so I'm not really concerned. It probably doesn't affect the average user all that much either.

<comment><USER6> I think for the user, the fragmentation of Android at versions 2.x was an issue, since major, fundamental features were coming in at each release. The features for 4.x have been less fundamental - my own phone gets typical use, and has gone 4.1 -> 4.2 -> 4.3 through the carrier's own slow process, but I haven't noticed a difference in the day-to-day functioning of the device (it does run a little smoother, and the battery is 2 years old and showing it, but nothing in terms of features)

<comment><USER7> I don't know by heart about the versions you quoted, but it might just be that some of the user visible changes have been hidden from you for the sake of not irritating users.

</comment></comment></comment><comment><USER8> Wasn't the whole point of Android to leave the HW abstraction to the kernel?

What's then preventing to leave the old kernel and just dump a new userland on it?

<comment><USER9> No.

</comment></comment><comment><USER10> > As for everyone else, how quickly they update seems to depend on how > complicated their skin is and how much they take advantage of the update > mechanisms Google has created.

Honestly, as someone who worked at an OEM, all that matters is how many engineers are assigned. There are usually hardly any assigned to port a new Android version to an old product, and almost all are assigned to the next product release. The skin often isn't even a huge issue. At the company I was at there were two different software divisions, and the one that ported a new version of Android to platforms (ripping out all the software drivers and getting the hardware ones working, mostly) was completely different from the one that worked on getting the skin framework many levels above running on the vanilla version of Android then produced for the target.

</comment><comment><USER11> LG's Optimus G is supposedly getting the Kit Kat update in August/September/October. I'm not sure why they're even bothering given how late they are. Personally, I've switched to Cyanogen on my Optimus G despite some issues with charging and battery life.

</comment><comment><USER12> I really wish Android was split into a kernel which just needed to support the device and _everything else_ which was an app that was updated regularly or else you couldn't market your phone as an Android.

<comment><USER13> That's exactly what Google has been doing for the past years. It took some time but now : -Google Apps are entirely decoupled from the OS. They are preinstalled and live in the system partition but are updated by the PlayStore. That way pretty much everyone (minus some apps that don't support 2.x) always has the latest version of these apps. -Google Play Services embeds many features that are considered OS level : security, job batching, location, chromecast, ... It is updated every 6 weeks.

Not to mention that from a third party dev perspective, Google provides compatibility libraries that allows to manage apps that run on a dozen of APIs levels, with either backports or 'gracious fallbacks'.

There are some features that can't be decoupled from an OS update though, and so far it does not look like it is going to change.

I am curious to see how the OEMs are going to handle Android L. The 4.x updates were very smooth from an user point of view with only added features here and there. L is revamping many things, which might confuse some users.

<comment><USER14> That is still far away from just a device specific kernel (although it does cover most everything a user would care about).

I recently looked into porting cyanogenmod to my tablet (Dell Venue 7). It has an interesting scheme that seems more in line with what tomjen wants. There is an unlocked bootloader that allows you to flash the "/system" partition. Within a days worth of porting, I was able to Intel's fork of Android and successfully flash the system partition. [1] What is interesting in their setup is that the root parition (and bootloader) live in a separate part of the storage, and I did not find a way of writing to them (I think that requires a key). This still isn't creating just a device specific kernel, but the root filesystem is pretty low-level stuff that even most tinkerers wouldn't care about updating, and you can work around it (run an updated version from /system) if you want to.

[1] Unfourtuantly, Intel's changes appeared to not have propigated back to cyanogenmod yet, so I abandoned trying to port it to avoid duplicating a lot of effort.</comment></comment></comment>


Grace Hopper, mother of COBOL - <AUTHOR> https://en.wikipedia.org/wiki/Grace_Hopper <comment><USER2> I find it really hard not to do a BSG reference here.

"And Zeus warned the leaders of the twelve tribes that any return to COBOL would exact a price in blood". -- Book of Pythia

<comment><AUTHOR> So say we all.

I'll nerd out all day over some BSG.</comment></comment>


Old Reddit - <AUTHOR> https://old.reddit.com/ <comment><USER2> Literally a link to Reddit...

<comment><USER3> With a specific subdomain, that is.

<comment><USER2> I think most people know that exists... It says at the top of the page that it's still available.</comment></comment></comment>


Retired lab chimps see the sky for the first time - <AUTHOR> http://www.hlntv.com/article/2013/03/06/chimp-haven-lab-animals-go-free

<comment><USER2> I think that as a society we under-estimate the enormous gratitude we own these animals. We subject them to our tests to make our lives "better" (among useful things, to test make-up and crap like that). In a way we should be ashamed of ourselves that we're even doing this but I guess humans will always be humans regardless of how great we think we are.

<comment><USER3> Gratitude is the least of it. Deepest apologies, corrections, and reparations are more like it.

More specifically, you can join a petition, deadline March 22: [https://secure.humanesociety.org/site/Advocacy?cmd=display&#...
More general: http://www.humanesociety.org/issues/biomedical_research/help...

</comment></comment><comment><USER4> Lived in a lab cage for 50+ years? That's disgraceful.</comment>