Ask HN: How do you find an honest and truthful workplace? - <AUTHOR> I'm looking for a job where people highly value honesty and truthfulness in all areas of the company.<p>Where do I start? <comment><USER2> They do exist, but they are rare because dishonesty is far more profitable financially, even though it comes at the expense of "moral profit" and "social profit" for the world.

And it is a bottom up as much as it is a top down problem. Employees self- servingly turn a blind eye if not actively participate in the dishonesty to benefit their careers and salary. Check out _The Banality of Systemic Evil_.[1]

You are far more likely to find honesty in small orgs than large.

I wish the post about Kickstarter choosing social profit over financial[2] profit lasted longer on the front page today. That it didn't says something sad I think about the priorities of HN readers and the tech community as a whole.

Don't give up! The most important thing is to work against the banality of evil not just in your workplace but in yourself. It takes a lot of courage and sacrifice.

-

[1] [http://opinionator.blogs.nytimes.com/2013/09/15/the- banality...](http://opinionator.blogs.nytimes.com/2013/09/15/the-banality-of- systemic-evil) (HN discussion: [https://news.ycombinator.com/item?id=6436719](https://news.ycombinator.com/item?id=6436719))

[2] [https://news.ycombinator.com/item?id=10250120](https://news.ycombinator.com/item?id=10250120)

</comment><comment><USER3> You will have a hard time finding a place like this, because honesty and truthfulness do not perform well in large groups where there is not a pre- rational basis (e.g. religion or blood-ties) for them.

You should look for smaller companies for more honesty, but also recognize that there is another side of the coin: there will be a stronger ingroup tendency and it may be harder to gain trust, and there will be less stability in this economic environment, which can add tensions.

I worked for a company that started as a fairly small, high-trust startup of about 50 people and that grew into a huge public company. Despite the leadership remaining relatively constant, once a certain size was hit it became apparent that political management and power relations became more important than productivity. Honesty regarding mistakes, in particular, almost completely vanished.

I'm convinced that group size is the most important factor in trust formation.

</comment><comment><USER4> Workplaces are made up of people. People vary in their levels of honesty and transparency. An individuals level of honesty can vary over time as circumstances change. Two people might not even agree on what constitutes the truth.

So to expect any workplace to be completely honest and transparent is not only impossible but naïve.

</comment><comment><USER5> I remember the point in my career when I started having to routinely sign non- disclosure agreements, put "proprietary" labels on my writing, and meet with attorneys regarding agreements and contracts. I gradually realized that all real managers and executives routinely work with information which can't be shared - much of it specifically being hidden from coworkers. Combine that with all the competing interests (maximize profit & minimize costs for all the major stockholders and investors vs. all the employees). And so, eventually, I finally realized that "honesty" and "integrity" in the corporate workplace are largely an intentional illusion. If someone believes in that their workplace is "moral", it tells me that he is either someone insignificant in the company or someone who has, like most executives, adopted a "business value system" which is usually very "Ann Rand-ish" and "free market" usually with some "personal responsibility" thrown in (it helps to ease the cognitive dissonance when justified by "they deserved it" and dehumanized by "markets")

Addendum: I've spent more than a decade in open source, so anyone who tells me that I should try that is just drinking the kool-aid.

<comment><USER6> There is a spectrum. Obviously the evil turds at companies (mostly) don't go around murdering people like common gangsters. But the way some of them act I don't understand why they just don't quit their job and start robbing and shooting people. There are quite a few pieces of shit out there that will make anyone wonder why they are even try to pretend to follow any sort of rules or laws at all. Why go through the effort of coming to an office. Perhaps they are playing a game to see how well they can fool people.

</comment></comment><comment><USER7> Be very careful what you wish for. The sword of truth is a hard thing to live with. You might consider tempering the high value you place on these things.

Having said that, if this matters to you a whole lot, I recommend growing your own. Create a business that fits your high values. If you can create something that both fits your high values and also pays the bills, you will have added something of real value to the world.

</comment><comment><USER8> _" How do you find an honest and truthful workplace?"_

Make it yourself. Another way is find people where the intersection of technology and these attributes collide. Hacker spaces are a good place where technology is made unburdened by commerce. When money is introduced though, that's where you'll need to pay extra attention.

Honesty, truth and money mix like oil and water.

<comment><USER9> > When money is introduced though, that's where you'll need to pay extra > attention.

It should be pointed out that while money is a prime example here, money is not the only system of value in society. Any system of value can be abused. Sometimes prestige and fame are abused similarly to money. Look at all the weightlifters and athletes who have been found using performance enhancers, and not all these people were doing it merely for the money. Politicians and Priests are further examples where occasionally money is secondary to the real drive. Then there are Police Officers and Soldiers, not that I am trying to paint any stereotypes here since each individual is unique.

</comment></comment><comment><USER10> You start by interviewing and asking your interviewers questions. Important questions! Ask about their backgrounds, work process, the types of projects they work on, turnover rate, etc.

I made the mistake of not asking the right questions when I accepted my first job out of college. I left in less than a year.

You'll find that there are a lot of different ways companies run, and you won't know much about them until you've worked at a few, and interviewed at 20-30 of them.

Know this: Every company has problems to a certain extent. It's your job to make sure you can balance the pros and cons in your favor. Decide what's the most important thing for you.

Go out there and interview as much as you can. It'll give you a sense of the companies and people who run them.

</comment><comment><USER11> Reading between the lines, it sounds like something bad has happened to you at work. Rather than ask for the impossible (because as PhilWright says in this thread, companies are made up of people), you might be better served to explain the incident that has prompted this question.

</comment><comment><USER12> Truth!? People can't handle the truth. If people valued truth and honesty there wouldn't be "the company". If you speak the truth you can't tow the line! Tell people that shadows are shadows and they'll probably kill you.

</comment><comment><USER13> Carry a lantern and live in a barrel.

<comment><USER11> Diogenes - is this you? [[https://en.wikipedia.org/wiki/Diogenes_of_Sinope](https://en.wikipedia.org/wiki/Diogenes_of_Sinope)]</comment></comment>


Guess which country has the highest Twitter penetration rate? Saudi Arabia - <AUTHOR> http://arabcrunch.com/2013/11/saudi-arabia-records-highest-twitter-penetration-in-the-world.html

<comment><AUTHOR> 41% of Internet users in Saudi Arabia are on Twitter. the population there like to talk and express their opinion which makes twitter ideal for them</comment>


Apple IPad's 'Buzz Saw' Success Cuts PC Sales at HP, Dell - <AUTHOR> http://www.businessweek.com/news/2011-05-18/apple-ipad-s-buzz-saw-success-cuts-pc-sales-at-hp-dell.html <comment><USER2> > _"The tablet is going to replace at least the home computer."_

I don't know about that. But what I think is still a huge deal, and far more likely, is that families will trend back toward one shared PC in the den and use multiple secondary devices for individual use.

While I don't think you can get away _without_ a home PC, you can certainly share one more easily when all but a handful of tasks are doable (if not more enjoyable) on a tablet.

And that's going to hurt a lot of home PC purchasing, with the notable exception of when the kids go off to college.

<comment><USER3> > While I don't think you can get away without a home PC

Are you saying this about today, May 18 2011? If you are, I suppose you might be right. But by, say, the end of the year, or maybe next year, I think you'll be absolutely wrong. The tablet market moves fast and soon there won't be a single task that isn't only doable but indeed more enjoyable on the tablet.

Right now, my Xoom outperforms (measured in my happiness levels, not CPU speed) all my home PCs for every single task except software development. I see no reason at all to have a home PC if you own a Xoom or similar tablet. I just bought the nice bluetooth keyboard which means that now even composing blog entries or writing emails is nicer on the tablet.

Everything's changing, and fast. It would be shortsighted to suggest that everyone needs an i7, 4GB of RAM, 2TB of storage and a DVD drive in 2011. Desktops will become more and more of a niche device, used only by nerds, while tablets and their descendants will take over the whole consumer market.

<comment><USER4> Tablets can't do serious gaming.

<comment><USER5> 99% of people don't give a toss about serious gaming.

The iPad has become the primary gaming device for me - I rarely ever play video games elsewhere.

<comment><USER6> Also, gaming consoles exist. A PC was never necessary for serious gaming.

<comment><USER7> untrue, but it was more than ten years ago...

<comment><USER6> Why is that? You will have to explain your odd view a bit more, I fear.

I can understand that some people prefer PC gaming for one reason or another but I think I'm quite correct in guessing that those people are a tiny minority – it's a niche.

<comment><USER7> there was a brief window when online 3d gaming was first taking off and none of the major consoles supported it in a good way -- the only game in town for hardcore gaming was the PC. Sure, the dreamcast might have been _able_ to do it, but everybody was using gamespy to connect to their favorite servers.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> The numbers don't add up...

PC sales are about 85-90M per quarter. iPad was selling at 4.7M that quarter. If every person who was going to buy a PC bought an iPad it would be a 5.5% reduction. Consumers are about half the market, so maybe you say 11% of the consumer market. And that again is assuming EVERY iPad sale is a lost consumer PC sale. HP dropping 20%+ is bigger than the iPad.

<comment><USER9> You forgot about Mac :)

</comment></comment><comment><USER10> > Tablet owners are actually more likely than US online consumers in general > to have recently bought a PC

[http://blogs.forrester.com/sarah_rotman_epps/11-05-17-hps_ea...](http://blogs.forrester.com/sarah_rotman_epps/11-05-17-hps_earnings_and_the_post_pc_era)

</comment><comment><USER11> The price point is similar. Both browse the internet just fine. But one has Angry Birds and FaceTime. One is new and cool. One is dusty, has viruses and keeps reminding you that your software license may be non-genuine.

</comment><comment><USER12> Buying an iPad is buying a second device of a kind different than anything you currently have.

Buying another computer is done to replace one that is obsolete (something that is less frequent over the last several years).

In other words, computer sales were destined to "fall off" a bit anyway. Once tablet devices reach a saturation point with limited benefit for upgrading, they will suffer a similar fate.

<comment><USER13> You are correct, but the event you're taking about (the "everyone who can afford one has a computer" event) happened years ago. We're already there. Destiny is history.

That said, things never really decline, because prices are going down, the need for computing is going up, and so the market is always expanding and sales just steadily grow now and forever.

You only ever see these kinds of false "declines" where people are really just shifting to a different kind of computer.

</comment></comment><comment><USER14> It seems like a pretty big jump to determine causality here - nowhere in the article do they mention consumer studies indicating consumers have purchased the IPad in lieu of a PC. Until they do that, this seems like speculation at best.

For instance, maybe people realized that a new PC doesn't buy them anything in terms of performance for their web browsing and document editing needs.

<comment><USER5> Anecdotally, I know several people who have replaced their computers completely with iPads. They won't be buying new laptops any time soon (and yes, they use their iPads for work).

<comment><USER15> Anecdotally, I didn't bother to set up two monitors when I started my current job (since I was the sole IT person) and after about six months I added one since there are a few lying around and was blown away by how much my productivity improved.

Granted, I'm doing development, but most tasks involving a computer are positively impacted by increases in screen real estate. I've had the bookkeeping office say they won't work with less than two monitors, and one of them needs to be largish. Given the spreadsheets they work with, I see the need.

There are plenty of tasks where the lack of real estate might seem to be unimportant, but what it comes down to is that when you're trying to fit anything of nontrivial complexity on a screen the size of the iPad, you use more and more mental energy on context-switching as the complexity increases.

<comment><USER16> This comment is interesting, because when the iPad first was announced, I had a lot of people give me the "oversized iPod" line and I tried to explain that even just significantly increasing screen real estate was enough to open the device to massively increased functionality. Glad to see some analogous, anecdotal support.

</comment></comment></comment></comment><comment><USER17> If tablets really are the future of computing, then developing countries can bridge the digital gap quickly with inexpensive tablets. [http://www.engadget.com/2010/07/23/35-tablet-from-india- look...](http://www.engadget.com/2010/07/23/35-tablet-from-india-looks-to-be- worth-every-paisa-video/)

Imagine 500 millions like these in every Indian's pocket.

<comment><USER18> Those Indians, they sure have big pockets! ;^) Jokes aside, that's a good point. When you have your own computing device, you have the chance to play around with it instead of just following a script ("click the blue e, type facebook, press enter..."). These low-cost models lack 3G and have a horrible battery life, but if they become popular, I'm sure we will see enterprising individuals opening hole-in-the-wall cafés offering cheap wi-fi and charging cradles.

</comment></comment><comment><USER19> I worry a lot about this, because users have a lot less freedom with the iPad than with PCs; they're a lot more vulnerable to Apple, which can abuse its power in the same way that Facebook can.

If the iPad or something like it wins, disruptive innovation in computers is dead, because incumbents will control the market, and incumbents don't like disruptive innovation.

<comment><USER20> I worry too, but the good news is that Honeycomb looks like it may well escape the near-death experience of launching too early with the Xoom. If you gloss over the small glitches nearly everyone who uses it loves it.

Fingers crossed we're heading to at least two viable tablet OSes, if not three (number three being a total wildcard, but I would probably pick windows 8 if only because eventually business adoption will give the momentum).

<comment><USER19> Honeycomb is still proprietary, isn't it? And Motorola is notorious for locking down its devices so users can't control them, even if the users _do_ have a version of software that they want to install. (Edit: glad to hear they aren't doing that on this one.)

<comment><USER20> > Honeycomb is still proprietary, isn't it?

It is proprietary but users have enormously more freedom than they do with the iPad. Just the ability to side load apps makes a universe of difference. Further I'd be very surprised if most of the current crop of tablets don't get the upgrade to (open source) ice cream sandwich at the end of the year.

> And Motorola is notorious for locking down its devices

The Xoom bootloader is unlocked, so presuming ICS gets released it should be possible for people to flash it on there even if Motorola doesn't want to do it.

</comment></comment></comment></comment><comment><USER8> Tablets are purely a very short-lived stop gap (think a few years).

The future is phone sized devices with foldable screens. Use it small when you're on the go. But when you want to sit on the couch or on the bus, you open it up.

The dedicated tablet becomes a niche product for those people that need the full power of a dedidicated large machine.

[http://www.slashgear.com/samsung-develops-sweet-foldable- amo...](http://www.slashgear.com/samsung-develops-sweet-foldable-amoled- screen-with-no-seam-13151802/)

<comment><USER21> Is it prognostication day already? I dissent.

I call the future is: Your world in your phone sized device backed by the cloud, with the option to use anonymous, mindless larger displays when in close proximity. (Bonus good future if I can also leap to a processor in an anonymous proximate device for speed.)

EOM

Reasoning: Producing enough light for a larger display requires power, requires mass, interferes with the light weight of the phone. Decouple the display and the phone. Ditto for CPU power and storage.

<comment><USER8> _I call the future is: Your world in your phone sized device backed by the cloud, with the option to use anonymous, mindless larger displays when in close proximity._

I've seen the future, and that's not it. :-)

Too expensive, and just can't be ubiquitous. How many large displays can you put in a bus or in the stands of a football game or at McDonalds?

With that said, such devices will exist, but they'll be your TVs in the house.

And on battery life. Even today people can get some decent efficiencies, and it will get better. Here's what Engadget had to say about the Kyocera Echo (a dual screen phone, with a 1370mAh battery, which is smaller than the iPhone 4s):

"We took a different approach. We charged the phone fully and then went about using it as we normally would for an entire day -- checking emails, browsing through TweetDeck, zoning out on a conference call for an hour or so, watching a bit of Sprint TV and playing entirely too much full-screen Pac-Man. We'd say 70 percent of our usage had both panels open, which is likely far greater than what you'd see after the novelty wears off. That said, we managed around 15 hours of use before it petered out, and on a second try -- one that involved far less dual-screen action -- we squeezed out 22 total hours. "

15 hours isn't ideal, but with some work that we're seeing out of Samsung, I don't think 24 hours is out of the question in a year or two at all. And that's with a relatively small battery.

EDIT: You have to love cowards that downvote, but don't dare say a peep.

</comment></comment></comment><comment><USER9> The losts are in revenues, not units.

It's entirely possible that people opt for lower priced computers instead of iPads.</comment>


Show HN: I'm building an open-source, high-frequency trading system - <AUTHOR> http://scarcecapital.com/hft <comment><USER2> This is a nice little research project, and I hope you learn a lot from it. Having written algorithmic trading systems I think you are missing on a couple of central points: 1\. There is no such thing as the "best" algorithmic trading platform, because algorithmic trading is such a broad term. Architectures that make sense for one class of trades does not make sense for other class of trades. 2\. Contrary to popular opinion on this forum, algorithmic trading is not necessarily only low-latency and high-frequency. Frequency, latency, and algorithmic levels are really dials on the specification system and along with cost dictate how you will build the system. Do some work figuring out the trades you are targeting before setting those dials. 3\. The actual execution system part of an algo trading system is usually the easiest part. If you've written one before, writing new ones is usually trivial. Finding the trade, building appropriate risk systems/practices, building back testing frameworks and exchange reference data systems are all much more challenging and take the majority of the time in these systems. 4\. Finally, if you are truly targeting low-latency and high frequency events concurrency is your enemy, not your friend.

To everyone saying that the cost in these systems is the cost of co-locating servers or fpga cards, you're wrong. You can get hosting deals/leases on those kinds of things for the same cost as high-end web hosting. The cost of running these systems are 2 fold: 1 paying the employees (because your competitors can pay them a lot) and 2 having deep enough pockets to survive the bad days/weeks/months. These are the same costs that have always existed in the trading space and have nothing to do with electronic trading. In fact, electronic trading has lowered the information asymmetry and made it easier for new participants to be involved in the markets.

<comment> bink-lynch > ...truly targeting low-latency and high frequency events concurrency is your > enemy, not your friend.

This reminds me of the LMAX financial trading platform where they started with a concurrent model but ended up using a single-thread "...that will process 6 million orders per second..."

<http://martinfowler.com/articles/lmax.html>

<comment><AUTHOR> Yes, but the concurrency has been shifted to the two disruptors at either end of the business process logic. And I suspect the business logic is fairly simple and lends itself to the straight through solution. If you want to react to a news event (say), you have to somehow send a signal to the feed requesting recent history (last 20 minutes say), have something waiting to look at it, crunch the numbers when they arrive to see if somethings up, then interupt the existing trade decision/risk management schedule with a potentially better trade. Complexity increases quickly and concurrency helps do all of that at high speed.

<comment><USER2> As your complexity curve increases, your latency expectations must go down. That is what I meant when I said there is no single trading system that can be the "best".

If what you are interested in is complex decision making then it may make sense to use a different sort of messaging technology than LMAX, but you won't be getting into anything remotely "low latency". Nothing wrong with that, just needs to be a known expectation.

<comment><AUTHOR> Yep, I've poured over the technology and concepts there and its awesome. So awesome I think you can increase complexity without too much of a latency penalty. Just behind the fast guys but way ahead of the straight algo guys is where I think the opportunities lie (I'm kind of answering your previous question about where the dials sit in my mind). I might be wrong but, and would like to test out all dial positionings at once before locking in. And that's where a flexible haskell version could shine.

<comment><USER2> I have no opinion about the choice of haskell. That said, you are already making decisions about the dials if you go in with an architecture that relies on massive concurrency, functional languages, etc. You cannot for instance get the latency dial very low with that central architecture decision.

Again there is nothing wrong with that. It's just better to state it up front as an expectation and/or a goal than to assume you are going to be able to pivot easily once you have an architecture in place.

</comment></comment></comment></comment></comment></comment><comment><USER3> I am curious about this project because I love this subject.

However, it is impossible to build an open source HFT system especially because it will be open source. The other sharks will know how you trade and they will only need to play against you by sending opposite orders every time. That is what happens when the source code of a big US bank were stolen. So it should be more appropriate to say it will be a fast trading system. The other reason is you can't build a perfect HFT trading system because it's a moving target. It is the war in the micro seconde market. The sharks will send you wrong orders just to disrupt your system or they will manipulate the market. And finally there is a hardware problem. To do a HFT trading system you need to rent your own severs inside the market, use FPGA cards, etc.

In conclusion, I only say there is a wrong goal to say "I will build the best open source HFT trading system" rather than "I will build the best open source fast trading system". This project is interesting (and there is Haskell cool!) so I will stay connected to see how it evolves.

<comment><USER4> Couldn't the infrastructure be open source while the specific trading algorithms kept closed source?

<comment><USER5> This could be a lot more fun than classic codewars.

For 99% of the "system" there's no particular reason you'd have to connect to the NYSE and trade real stocks for real money. Looks like they're feeding from iqfeed, but I think it would be huge fun to create an imaginary competition exchange, convince about 100 algo writers to compete, and shove them up against each other purely for the fun of it, see who's a better algo writer.

With modern virtualization, it should be pretty easy to distribute a pack of images, including practice data, to replicate an entire financial system on a small scale in your basement, then once you think you have a decent algo, upload your image to the competition league. If your hft codewars league wants limits for storage or cpu cycles, virtualization is a pretty easy way to implement it.

Superficially the most obvious thing to do is totally make everything up, from top to bottom, but it would probably work just as well to use real live market data and overlay on top of it. Just as air traffic control simulation has very slowly moved over the decades from purely random to sorta realistic to actual airport data.

This could be quite a bit of fun.

<comment><USER6> We did this in collaboration with UofC, using our Algo trading platform ([http://www.optionscity.com/event/uchicago-midwest-trading- co...](http://www.optionscity.com/event/uchicago-midwest-trading-competition)) and it was really fun.

Hm, I wonder if there is an interest to make this a more open event :)

<comment><USER7> >Hm, I wonder if there is an interest to make this a more open event :)

Yes. I bet there is.

</comment></comment></comment></comment></comment><comment><USER8> Have you considered python for the research/algo side of things?

I work in a hedge fund in London at the moment and there's a massive shift away from R towards python in the algorithmic shops that I know about here. There's also the great work that quantopian are doing on their backtesting framework zipline [1]

[1] <https://github.com/quantopian/zipline>

<comment><AUTHOR> Yes, I am aware of that trend and considering a switch. I'm personally more used to R is all, but if I found collaborators I'd jump to where the code base goes.

The only other factor is I find R pretty aligned with haskell having a somewhat functional pedigree, so that code translates pretty nicely between a rapid hack at the problem to the more robust approach.

<comment><USER9> zygomega, I'm one of the zipline maintainers and we'd love to collaborate with you. There are a few people doing HFT research with zipline, and there's a lot of work to do. At quantopian (my day job), we focus on longer hold periods, so there is room in the zipline ecosystem for you to do HFT.

The main benefit we've found with python as the algo language is that it allows for stat programming with pandas, but also OO or functional programming for the algo logic. This smoothes the transition from research to production, just as you're describing with R -> haskell, but you can stay in one language.

I think one of the biggest potential wins with parallelization is if you can assume all positions are closed overnight, most often true for HFT. That way, you can simulate all the trading days in a test range in parallel. This is quite similar to the parallel processing we do to handle the large number of concurrent backtests running at quantopian. We did all of that with python, but I'd be fascinated to see it done with haskell.

<comment><USER8> I would see haskell very much as the plumbing side of things. The tools available for handling and reasoning about streams are streets ahead of anything I've seen elsewhere. With zeromq and protocol buffers (that's what we use in our stack) you could very nicely separate the plumbing of the data from its consumption. I'd love to see something like this as well!

How would you handle the position sizing part of the algo if you're testing all days in parallel? Wouldn't the trade size depend on the all of the previous day's PNL?

</comment></comment></comment></comment><comment><USER10> Are you trying to get hired by the financial industry, or is there some other reason for doing this?

As you may well be aware, HFT is a scourge on the world's economy, and it's a game only the biggest and best-connected players benefit from.

<comment><AUTHOR> No, I'm already in the finance industry. There are two main reasons I'm doing this: \- I think the finance industry is very closed when it comes to intellectual property development, and an open source approach can be seriously competitive. An open approach may well be the future when it comes to being 'connected' \- HFT is an interesting multi-disciplinary problem and the shear breadth of expertise required - modern chip design, low-latency software/hardware interaction, lock-free concurrency, fault-tolerant system design, adaptive learning algorithms, k-means clustering - means I'm learning heaps every day.

I just don't agree that HFT is a scourge. It's an ecological shift (neither good or evil) and longer-horizon investors need to evolve.

<comment><USER10> > an open source approach can be seriously competitive

Do you really think any Joe Schmoe off the street could just grab your open source HFT and start making money with it? If not, how does your project benefit anyone?

Are you working in the industry, but not in HFT? Maybe this is project is just practise for getting a job in HFT? I imagine that's where programmers get the fattest paychecks in the world. Only a fraction of what the sociopaths running the show get, of course, but good money nevertheless.

> I just don't agree that HFT is a scourge.

Good thing you're not at all biased.

<comment><AUTHOR> Good points. To be more precise, I think that an open source approach to system research and design can be hyper-competitive versus closed-door secret- squirrel development run by a committee looking for short-term wins. This project benefits me firstly because I get feedback on my initial scratchings and maybe even collaboration on areas outside my comfort zone. And it could well enable me and others to avoid having to work for the sociopaths.

I am biased in thinking that monitoring the market and trading at low latency isn't fundamentally a bad thing, and shouldn't be taxed because others are too lazy to do the same thing. I certainly think that hft has been used by evil people to front-run unknowing third-parties, often in collaboration with middle men and women who turn a blind eye to the morality of their business models.

<comment><USER11> Others are two lazy? It's my understanding that the average Joe has no access to the hardware and low latency network connections to be able to do this on his own.

<comment><AUTHOR> There is a rather large economic rent attached to HFT at the moment. If my project or others like it can eat into that rent a touch, it might not stay that way.

<comment><USER7> Exactly, if you take the position that gains from HFT are basically "stolen" profits from otherwise more productive market participants, then any that can be "stolen back" is a win.

<comment><USER12> There have always been "market-makers" why do you care if they are algorithmic or human ?

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> I'm curious about the "best trading platform" claim - what is going to make it better than any other? Especially as the real High Frequency Traders are busy spending fortunes on placing their hardware as close to the exchange as physically possible.

<comment><AUTHOR> Speed is only one issue with autonomous algorithm design. Yes, the speed thing grabs the headlines but the boiler-plate objective is to front-run the slower players. There's a wealth of opportunity in processing the event stream in a more robust way than others and faster too. Think semi-HFT, semi-autonomous.

Most trading platforms are primarily loss leaders for the 'professional' version and otherwise attached to a non-open source business plan.

<comment><USER13> But doesn't that mean you're at the mercy of the professional HFTs who can "front run" anyone running on your platform?

<comment><USER2> You've hit a pet peeve of mine so sorry about the tirade. People use the term "front run" incorrectly and it is important to point this out. Front running is a very specific activity and is illegal. If there is evidence of front running people need to be prosecuted.

That said, I've worked on several systems and never seen any of them that would allow a faster market participant to see your order flow without your permission. This is what front running is.

People who don't understand the term seem to think seeing market data before other participants is front running it's not. There have always been and will always be some traders that see (and can react to) market data faster than everyone else. In the era of electronic trading this information latency cost is going down, not going up.

</comment></comment></comment></comment><comment><USER7> FYI: This isn't an HFT bot but is a decent, recent Python + ncurses trading client framework for Bitcoin on MtGox. <http://prof7bit.github.io/goxtool/>

Even if you aren't interested in Bitcoin, it might be useful as a real-ish and cheap place to test with low barriers to entry. I've found that backtesting on historical data is usually not realistic enough since most people fail to consider liquidity.

I'm interested because I like to trade, would like to learn more, and I have had some small success. But I have trouble sticking to my plan. I let emotion get the best of me and wind up losing my gains.

<comment><AUTHOR> Isn't MtGox over as a bitcoin trading platform? I agree more generally that bitcoin could be a useful market to test out.

And your story fits exactly with a hands-off automated approach to trading. We wont ever be charting a price series in the production environment because computers don't get charts.

<comment><USER7> >Isn't MtGox over as a bitcoin trading platform?

I wish. I keep waiting for some more interesting news from Coinlab [1][2], but so far, Mtgox are still the largest exchange by volume. Here is a chart showing volume by (selected) exchanges. <http://bitcoincharts.com/markets/currency/USD.html>

The recent events seem to only have got them more customers (though it remains to be seen how many of them will stick). In any case, some of the other exchanges have API's that are similar to mtgox.

[1] <https://mtgox.com/press_release_20130228.html>

[2] [http://www.forbes.com/sites/jonmatonis/2012/04/24/coinlab- at...](http://www.forbes.com/sites/jonmatonis/2012/04/24/coinlab- attracts-500000-in-venture-capital-for-bitcoin-projects/)

</comment></comment></comment><comment><USER14> Isn't the barrier to HFT the fact that you need enough capital so that your profits cover the cost of co-located servers and FPGAs in the exchange datacenter, without which you have a latency handicap? (in addition to, of course, coming up with good algorithms)

<comment><AUTHOR> Yes, if the only edge you have is speed. But a good algorithm may well be the next battleground as the benefits of ultra-low latency reduce.

</comment></comment><comment><USER15> Cool! Very interesting.

I think given the languages you have selected you are coming more from a quant background? These languages are great for heuristics and analysis but you would really want all 'static' components such as connectivity built in assembly/C/C++. For 'algo' components I like Java as you can still pull microsecond order latencies when crunching numbers but more importantly it gives you a huge time to market advantage than the C/C++ for almost the same speeds. I'm also not clear on if you are connecting directly to the market for market data or using aggregation (like reuters). The latter would be too slow. I'm also not clear on what middleware you are using which is probably the biggest decision you will have to make. Most either use inhouse tech or 29west LBM (everyone still calls it that even though they were bought out).

An overlooked part of HFT in my experience OS optimisation and even things like TCP bypass (for some components) which can lead to huge speed advantages and end to end latency reductions. I agree with those about FPGA.. in my experience they really don't come into the equation except for components that rarely change.

Anyway a few guys including Martin Thompson have felt similarly to you and initiated the lodestone project (<http://lodestonefoundation.wordpress.com/>). If you are keen to learn more their architectures for low latency, distributed and componentisation then I feel that you could join forces and contribute to that initiative. FYI: Martin built most of the technology behind the disruptor (<http://lmax-exchange.github.io/disruptor/>).

Great to see interest in making this knowledge more widely available :)

<comment><AUTHOR> thanks mangrish,

quant. How could you tell? I'm conecting with an aggregator cause the direct market feeds are monopolistic price gougers. It's an easy switch if we make it up that curve.

I'm not sure if I even understand what middleware is (I'm a quant), but I think the answer is the disruptor!

<comment><USER15> From the languages.. I've worked with lots of quants and they all rave about R!

Yeah..the aggregator is where you get done in both cost and latency. The prop shops and hedge funds pay through the nose for that stuff so unless you come packing a little capital, true HFT is an issue.

On the middleware, not really.. so you would have a market data component that will be pushing stuff to various components (real time risk, the pricer, and the trading engine). The disruptor sits on the 'in' queue to those components.. the middleware is what pushes messages between instances running on the same/different machines.

Hope that helps :)

</comment></comment></comment><comment><USER16> In another month or two the analytical tooling for doing stats and numerics in Haskell land are going to have a huge leap forward in capabilities. Might be worth considering going full Haskell then :-)

<comment><USER17> Whats happening then?

<comment><USER18> From my talks with Carter previously he's working on a haskell based platform for analytic tooling in Haskell. Basically the core primitives for making large scale data analysis apps in Haskell.

Or that's what he was up to in august. Hopefully he can chime in here and update everyone on his progress. I honestly hope he succeeds in his plans.

<comment><USER16> Still working on it!

Been taking a bit longer to get the core worked out that I'd have liked, but life happens (eg my mom had cancer for a month this winter, though she's fine now, which is awesome. She didn't even need chemo or rad!!).

Also, I was original planning to _NOT_ write my own linear algebra substrate, but I quickly realized all the current tools suck, and that I needed to come up with a better numerical substrate if I wanted to do better.

What do I mean by this? With all the numerical tools out there presently, there are none that address the following falsehood that many folks believe is true: "you can have high level tools that aren't extensible but are fast, or you can have low level tools that are extensible and fast.".

I want high level tools that are fast. I want high level tools that are fast _AND_ extensible. I want it to be easy for the end user to add new matrix layouts (dense and structure, structured sparse, or general sparse) and have generic machinery for giving you _all_ the general linear algebra machinery with only a handful of new lines of code per new fancy layout. I want to make it idiomatic and natural to write all your algorithms in a manner that gives you "level 3" quality memory locality. I want to make sure that for all but the most exotic of performance needs, you can write all your code in haskell. (and by exotic I mean, maybe adding some specialized code for certain fixed sized matrix blocks that fit l2 or l1, but really thats not most peoples _real problems_ ).

Heres the key point in that ramble thats _kinda_ a big deal: getting "level 3" quality memory locality for both sparse and dense linear algebra. I think I've "solved" that, though ultimately the reality of benchmarks will tell me over the coming few weeks if I have or not.

Likewise, I think I have a cute way of using all of this machinery to give a sane performance story for larger than ram on a single machine linear algebra! Theres going to be some inherent overhead to it, but it _will work_ , and doing a cache oblivious optimal dense matrix multiply of 2 square 4gb+ ish sized matrices on a macbook air with 4gb of ram is going to be be a cute benchmark where no other lib will be able to do out of the box. Likewise, any sparse linear algebra will have lower flops throughput than its dense equivalent, but thats kinda the price you pay for sparse.

What I find very very interesting is that no ones really done a good job of providing sparse linear algebra with any semblance of memory locality. I kinda think that I have a nice story for that, but again, at the end of the day the benchmarks will say.

I at the very least hope the basic tech validates, because there needs to be a good not gpl lin alg suite with good perf for haskell. Hmatrix being gpl has cock blocked the growth of a nice numerics ecosystem on hackage /in haskell for years, and its about time someone puts on some pants and fixes that.

Assuming the tech validates, I really hope the biz validates too (despite me likely making various pieces open source in a BSD3 style way to enrich the community / get hobbyist adoption / other libs written on top, people in haskell land try to avoid using libs that use licenses that arent BSD/MIT/Apache styles), because theres so much more that needs to be done to really have a compelling toolchain for data analysis / numerical computation / machine learning / etc, and I really really like spending my time building better tools. Building the rest of that stack will be outlandishly tractable assuming my linear algebra tech validates having the right regimes of performance on large matrices. (amusingly, no one ever benchmarks linear algebra tools in the 1+gb regime, and i suspect thats because at that point, vectorization means nothing, its all about memory locality memory locality, and a dash of cache aware parallelism).

thats the vague version :)

And thats also not even touching my thoughts on the analytics / data vis tools that go on top. (or the horrifying fact that everyone is eager for better data vis tools, even though most data vis work is about as valuable as designing pretty desktop wall papers to background your power point presentations.... so even if i get everything working... I have a horrifying suspicion that if i allowed unsophisticated folks to use the tools, most of the revenue / interest would be around data vis tooling! Which would mostly be used to provide their customers/end users with pretty pictures that make them feel good but don't help them!)

Point being: i want to be able to say "you understand math, you understand your problem domain, and you can learn stuff. Spend 2-3 weeks playing with haskell and my tools, and you'll be able to focus on applying the math to your problem domain like never before, because you didn't even realize just how terrible most of the current tools out there you were wrestling with are!"

<comment><USER16> I really really really hope the biz+tech combo validates... because then I could occasionally stop and think "holy fuck, I'm bootstrapping my fantasy job / company, the likes of which I imagined / dreamed of as way back as middle school and high school!"

Realistically theres 3 different outcomes:

the tech doesnt validate (and thus the biz doesnt either) --- then i'm looking for a day job ... (and I'm pretty darn egomaniacal and loud, finding a good fitting dayjob would take a bit of work!)

the tech works yet the business doesnt --- Not sure how that would happen esp since no investors means enough income to support myself would still be a successful business, though I guess i'd have some compelling portfolio work if I went job hunting

the tech and biz both validate, and earning enough to move out of my parents --- magic pony fantasy land of awesome. what more could anyone want? MORE AWESOME PROBLEM DOMAINS THAT NEED BETTER TOOLS (i mean, that would really be sort of the ideal, but remains to be seen if that can happen.)

<comment><USER18> Thanks for the update Carter. I'll root for you.

one point to make. The interface and the performance don't have to appear at once. The interface will be the longer lived portion. So sort that out, and you can focus on performance as problems crop up.

I know it's horribly boring to say, but getting those first few customers gets you into a virtuous cycle. Given that you're bootstrapped, even a few customers will get you in a very good place, where you can spend on development.

I remember us (or rather I) talking about Mathematica. When it first came out it was _horrible_ for numerics. Truly terrible. But it was easy to transfer technical papers to. You simply wrote down what was on the paper, and you were done.

So people used it, and eventually performance got better over time as they invested in it.

<comment><USER16> Agree with everything you say. Hence why I'm just going to be releasing the Lin alg soon. It actually turns out for the linear algebra code done right, the API has an intimate relationship with the possible performance! (This will be more apparent once I get things out the door).

There's a lot ill not be even trying to do in the first release : eg parallelism, distributed computation, sse/avx intrinsics.

Fret not, things are moving apace, and basic tech validation and thence conditioned upon that, public release, are approaching scary fast! :-)

</comment></comment></comment></comment></comment></comment></comment><comment><USER19> 1\. Timestamps in your seed data might benefit from nanoseconds if you really talking about "high" frequency.

2\. I agree with you comment that it is easier to think about concurrency in Haskell than in something like C++, however you can't really compete with C/C++ in Haskell. Not even with cgo (Go packages that call C code), not with OCaml or any other higher level beasts that promise the speed. Fortran would be the only one faster for the "algo" part of your initiative. But again, if this is just an exercise, Haskell and others (I prefer Clojure for example :) will do just fine.

3\. Would make sense to split the "platform" in two (very different) parts: "Quantitative Analysis" (a collection of tools and rules) and "Technical Glue to Read and Stream". Each can/should be divided further of course, but the two above are essential yet very different for a true "HTF Platform".

<comment><AUTHOR> I wish I could get down to nano units. iqfeed (a good value feed) just got millisecs in so will settle for that.

I'm preparing some speed tests between C++ and haskell on an identical block of processing so stay tuned! You might be surprised - haskell is way ahead of clojure on compiler smarts.

The split you suggest is exactly what I think is wrong with the way things get done right now. I'd like to integrate the quant inside the read and stream - now that's potentially a large speed up that might compensate a tight budget.

<comment><USER20> Keep in mind that if you're not an experienced C/C++ coder, you're going to be (largely) benchmarking your relative ability in either language rather than the intrinsic speed of each language.

<comment><USER2> This is true of any language/coder. It's what makes cross platform benchmarking so difficult for non-trivial problems.

</comment></comment></comment></comment><comment><USER21> I'm building an open source matching engine. I would love to pair up the two systems for a stress test. I'll keep tracking your project and ping you again when I have a system up a running if you're interested.

<https://github.com/fmstephe/matching_engine>

<comment><AUTHOR> We have some plans on the drawing board for market maker simulators that could do with some fast matching so I'll keep an ear open for that ping.

<comment><USER21> What sort of performance baselines would you require?

<comment><AUTHOR> I have no idea yet. I see you're geeking out in go and there's been a lot of advocacy coming through to look at go as an alternative.

<comment><USER21> Yeah, Go is a great language. However, I am trying to develop some low level queues that should be faster, and less generic, than the ones that Go provides. But I am having some difficulty putting memory fences into my code.

But if I can get a reasonable level of performance out of the standard Go queues then I will just use those in the meantime.

</comment></comment></comment></comment></comment><comment><USER22> I worked on a framework for managing stock and options positions based on feedback methods developed by Stafford Beer. It doesn't do the algorithmic/transaction part, but rather focuses on providing reporting and alerting for large hierarchies of institutions, accounts, stock and options positions. The proprietary algorithmic code could be added by subclassing the code that I wrote. It's called the Viable System Agent. It's written in Smalltalk, tested under both Squeak and Pharo. It's licensed under the BSD license. It can be found at:

<http://home.rbcarleton.com/rbc/software/smalltalk/VSA/>

Look at the RBC-VSA-Portfolio category. That's where all the stock/option code is.

</comment><comment><USER23> The industry is doing the vast majority of its HFT development in C++ or Java. Data analysis is mostly done using Python although I still see a lot of R.

If you are doing this to break into the industry, I suspect the languages you used should have been the above. Also the above languages would probably have been better to attract open source developers who are also hoping to use their code and experience from this project to break into the industry.

<comment><AUTHOR> I've been in the industry for too long (though not in low-latency) - I'm doing this to break out of the industry! And I think the finance game is long overdue for a good disruptive technologic event (not implying my humble project is it). R is a personal choice for data analytics (I love ggplot2). But the haskell thing is more than that. It's been done before (<http://www.starling-software.com/misc/icfp-2009-cjs.pdf>) and even goldman's has a thriving erlang/OCaml hacker ethic ([http://www.zerohedge.com/article/aleynikov-code-dump- uncover...](http://www.zerohedge.com/article/aleynikov-code-dump-uncovered)).

</comment></comment><comment><USER24> I have no experience in the financial industry, but HFT has always fascinated me as a potential source for very high rates of "events". Could you share your general insight about the sheer volume of data that commonly gets pushed through an HFT system? I'd also be terribly interested in a multi- megabyte/gigabyte "recording" of HFT trade data.

<http://www.nyxdata.com/capacity>

<comment><USER17> This project is not HFT. It does not use a full book data feed. Really even the feeds that come from the exchanges are not precisely timestamped enough and HFT firms stamp their own . The data is not huge although trading system need to be able to deal with large spikes in the rate of events. Usually now people are using 10G/40G/Infinibad as a connect to the matching engine.

</comment></comment><comment><USER25> This could be an interesting course for people wanting to know more about 'computational investing' and the algorithms behind stock (market) analysis and trading: <https://www.coursera.org/course/compinvesting1>. Not specially targeted at HFT, but interesting basic background information nevertheless.

</comment><comment><USER26> Sorry to burst your bubble but high-frequency trading is only viable and necessary for market makers. The end result of any type of algorithm you think of is always a curve-fitting function of existing data.

<comment><AUTHOR> No need to apologise - curve fitting is a big issue. But I would think that everything that happens on kaggle would be curve fitting by your definition right? And how is the stuff that happens in brains something other than curve fitting? It's how you connect the data dots and what you choose to focus on that makes the difference.

<comment><USER26> You are asking how the human brain works, that's a whole different issue and I don't know what Kaggle is/does.

</comment></comment></comment><comment><USER27> Check out IBrokers: <http://cran.r-project.org/web/packages/IBrokers/index.html>

It'll get you up and running with IB in no time.

<comment><USER28> On a similar note, a while back I wrote a bi-directional, fan-out adapter between the Interactive Brokers API and ZeroMQ.

<https://github.com/brotchie/ib-zmq>

The annoying thing about the IB API is that there's no framing; that is, you can't simply consume the message types you are interested in without parsing the entirety of every variable length messages.

ib-zmq resolves this annoyance by parsing incoming messages and placing them individually into ZeroMQ message frames.

I also wrote an alternative to the IBrokers R package with a much nicer interface using this ZeroMQ adapter. It parses most IB API messages, but hasn't been used in production yet.

<https://github.com/brotchie/r-zerotws>

</comment></comment><comment><USER29> I assume you've checked out Marketcetera ( <http://www.marketcetera.com> ). What will be your platform's primary differentiators?

<comment><AUTHOR> Not having to register and login is one immediate differentiator. Marketcetera is a big, big code base - I think our ambitions are more focused.

</comment></comment><comment><USER30> Good to see Haskell. I must give you a complement for being part of pushing Haskell to the mainstream!

BTW, many trading firms do use haskell in Trading, but thats all privately held.

</comment><comment><USER31> On a possibly related note, Josh Levine, creator of the Island trading engine, released the Foxpro source a few years ago.. Interesting stuff..

<http://josh.com/notes/island-ecn-10th-birthday/default.htm>

[http://josh.com/notes/island-ecn-10th- birthday/ISLAND.PRG.TX...](http://josh.com/notes/island-ecn-10th- birthday/ISLAND.PRG.TXT)

<comment><USER32> Jerk boy!

</comment></comment><comment><USER33> I would be much more interested in an open source solution to _fight against_ HFT, or at least make it more difficult.

I see the financial world as a necessary evil, and I think some of its parts have benefits for the rest of the world. HTF is not one of those parts; I see it as a pure burden.

You could have made an open source Patent Trolling system and I would feel the same.

<comment><AUTHOR> Wow, that's a pretty low ranking being shoved into the patent troll category of evilness. When someone goes and patents an obvious algorithm at some point in the future and my little project shows up as prior art would I be redeemed :)

</comment></comment><comment><USER34> I thought HFT was now based on FPGAs and ASIC circuits ... how could this compete in terms of latency/processing ?

<comment><AUTHOR> By putting it on FPGAs maybe. Stage one is to get a robust process for measuring and researching latency - I'm in Australia and US data is bouncing into my mac via a lousy connection and a tcp port in 250 millsecs on average. There's a long way to go. I suspect that if you can keep within a bulls roar of the low-latency crowd the big gains will be on the algorithm processing side. Algorithms still add up a few million numbers every time they recalculate a moving average - there's better ways to do it.

</comment></comment><comment> nezza-_- You need to do <i class="icon-iconname"></i> instead of <i>icon- iconname</i>... The links all have icon names in them.

Edit: Ah. It now works only with JS on when putting the icon-name into <i> and not in the class. Sorry for that, seems to be new in Bootstrap.

<comment><USER35> It's Twitter bootstrap. <http://twitter.github.io/bootstrap/base- css.html#icons> Bootstrap V3 has tweaked the way icons are implemented slightly.

<comment><AUTHOR> thanks heaps for that!

</comment></comment></comment><comment><USER36> Aside from all the other points raised against this, there's also the fact that the timing on HFT is so close that it requires close colocation of the physical hardware to the trading systems. Only big boys get to have access to those racks.

</comment><comment><USER37> I have access to servers in all the major trading rooms if you need to start deploying this to a production environment to start trading. Contact me at dasickis [at] gmail.com (I'll reply back from my non-filtered e-mail address)

</comment><comment><USER38> > using haskell,R and emacs org-mode.

That last is so very, very, cool.

I'll have to dive in, take a look, see if I can find out _how_ you're using it. Been wanting to do a project with org-mode for a while, now.

<comment><AUTHOR> It is - I can't imagine working without it. But I'm not sure I'm the poster child for how to use org-mode properly. Every piece of code in the repo actually resides in the org file which is a big monolithic journal of where I'm at.

o-blog is a great emacs package that lets you publish sites straight from org.

</comment></comment><comment><USER39> +1 for using Haskell.

</comment><comment><USER40> How can someone who has absolutely no idea about the financial industry and HFT learn about it?

Can anyone recommend any resources (books, tutorials, etc..)?

<comment><USER31> mikevm, check out Dark Pools by Scott Patterson. Great book about the evolution of HFT:

[http://www.amazon.com/Dark-Pools-High-Speed-I- Financial/dp/0...](http://www.amazon.com/Dark-Pools-High-Speed-I- Financial/dp/0307887170)

</comment></comment><comment><USER41> Which software you used to build the diagram on the main page?

<comment><AUTHOR> The graphviz dot language: <http://www.graphviz.org/Documentation.php>

It has good support in org-mode and a there's a nice haskell package that takes the dot code and turns it into an internal graph representation.

</comment></comment><comment><USER42> Have you actually used this in a production setting?

<comment><AUTHOR> No, and I am making no claims. I have a market event feed coming in, a good idea of what the event processing looks like and a rough idea of how to send an order to a broker. I think the project needs to get to production fast though.

</comment></comment><comment><USER43> you're doing it wrong: haskell is too slow.

<comment><AUTHOR> I thought so too for a long while, then I tried to do a touch of concurrent code in c++ and had to gouge my eyes out. I'm excited about the speed up haskell brings to development. You can plan things in haskell you can't imagine in other languages.

<comment><USER44> +1 for doing it in Go. If concurrency is your objective, Go makes it easy... it's also very fast both to program in and to run.

<comment><USER45> > +1 for doing it in Go

Conspicuously you don't address Haskell at all. Then again, who cares what might actually be best for OP? There's an advocacy bingo card in play!</comment></comment></comment></comment>


Help with day job: Algorithmic staff scheduling solutions? - <AUTHOR>

At a gross level, the problem is simple: schedule an army of staff for one-person-per-day coverage, at any given day the staff is split into 3 pools, each staff has a vacation requirement, each staff has at most 2 shifts per week, etc.
I'd hate to do this manually as it has been done at my organization for centuries. I'd love to do something cool like genetic algorithms (eg http://www.sersc.org/journals/IJAST/vol14/1.pdf).
What has everyone here found useful? Open source / free alternatives are a big plus. This also sounds like an optimization problem, can I fire up C++, R, etc to plug into some optimization library for this? <comment><USER2> I'm making a service that does this. Its not public but works OK. I made it for a summer camp where the campers (~250) sign up for different stuff each day (Archery, Swimming, Sailing, + 30 others) so the demand for appropriately trained staff in each activity can fluctuate wildly on a day to day basis. The staff (~70) have days off but the campers don't, so the available talent pool also varies significantly. Let me know if you want to try it. I'm definitely going to charge for it when its publicly available.

</comment><comment><USER3> Isn't this essentially the stable-marriage problem matching staff to shifts where shifts have no preference who staffs them?

</comment><comment><USER4> This is what prolog was made for. You might have good luck with googling "constraint-based programming"</comment>


Allan Lichtman's has correctly predicted Presidential elections since 1984 [YT] - <AUTHOR> https://www.youtube.com/watch?v=mp_Uuz9k7Os <comment><USER2> Obviously very impressive, but I've been wondering how likely it is that someone would get this by chance. If we assume there's a 60% chance of getting it right on average (this could be lower, but a 10% increase in accuracy seems like a reasonable amount considering the amount of polling data available),there's a 1% chance someone could get that record by chance. Given the amount of individuals in politics, is it too far outside the realm of possibility that this is largely chance?</comment>


Why iGoogle is About to Change the Web... Drastically - <AUTHOR> http://ryanspoon.com/blog/2008/05/29/why-igoogle-is-about-to-be-game-changing-for-google-and-for-us/ <comment><USER2> Old news, but still true.</comment>


Ask HN: what is the most inspiring entrepreneur story ever? - <AUTHOR>

Hi HN,<p>I am wondering what do you think is most inspiring entrepreneur story ever? <comment><USER2> Elon Musk, Steve Jobs, and Jay Z are all pretty inspiring.

EM: [http://www.bloomberg.com/video/73460184-elon-musk- profiled-b...](http://www.bloomberg.com/video/73460184-elon-musk-profiled- bloomberg-risk-takers.html)

SJ: <http://www.youtube.com/watch?v=-vrM7AvmxTA>

Jay Z: <http://www.youtube.com/watch?v=T_O7ra9LDis>

<comment><AUTHOR> Thanks

</comment></comment><comment><USER3> Tony Montana - lots and lots of people have been inspired to try and do what he did, often with similar results.

</comment><comment><USER4> Soichiro Honda

Gary Vaynerchuck

</comment><comment><USER5> It didn't end well and he made some poor business decisions along the way, but Tesla still deserves consideration. <http://en.wikipedia.org/wiki/Nikola_Tesla>

</comment><comment><USER5> Lord Kelvin has to be a contender [http://en.wikipedia.org/wiki/William_Thomson,_1st_Baron_Kelv...](http://en.wikipedia.org/wiki/William_Thomson,_1st_Baron_Kelvin)</comment>


Should the police be able to investigate your genetic family tree for any crime? - <AUTHOR> https://www.nytimes.com/2019/06/11/opinion/police-dna-warrant.html <comment><USER2> I'm reminded of parallels between cell phone location privacy and genetic tracking. The average person does not understand that they are creating a record of everywhere they go when they have their cell phone on. The same is true with your DNA - while it's true that you do in fact leave a literal trace of your genes, you don't expect that you could be identified later on simply by the fact that you had lunch.

What we really need is a legislative solution, but unfortunately I have low expectations for Congress, seeing that Carpenter v. United States even had to happen.

<comment><USER3> What would you legislate?

Your location is not private information and never has been. Not sure how you could even legislate to make your DNA protected private information- under HIPPA for instance, considering you leave it everywhere you go. Your fingerprint is not private, your face is not private, and your DNA is not private. I'm not sure how they could be.

<comment> village-idiot Your location isn't private if you're observed in a location where you have no expectation of privacy, i.e. outside. But if I'm at a friends house, I absolutely have an expectation that my presence in their home is private to the world at large.

The problem is that dragnet surveillance does not:

1\. Handle the subtleties of how we expect privacy to work.

2\. Require direct observation of someone in a context where there is no expectation of privacy.

It just works all the time, and contains the ability to unmask private events in the past.

<comment><USER4> your expectation of privacy is sort of inconsistent here. if you don't have an expectation of privacy when your out in the world, and it's fair game for people to observe you traveling to your friend's house, how can you expect that your presence is private?

<comment><USER5> My problem is the scale of the issue, at least when it comes to phones. If I am at my friens the people involved are apt to forget almost everything in a few days. The cellphone tracking system, in theory, will studiously remember exactly where you are till the end of time by a company wit history of give or selling said data to the government in secret.

<comment><USER6> Well, they _have_ to give your location information to the government. I think it's a bit naive to expect a company to keep your data secret from the Feds. They're just as subject to the law as we are.

If we use electronic devices at all, we're just best advised to operate as if the government has access to anything we say, write, or record on them. That includes our locations. If you're going to see your weed man, or your mistress, you probably want to leave it at home. And no, don't bother buying a "burner" because they can track that too.

All that said, companies should try to keep your location information secret from anyone _other_ than the government. One of the issues right now is that they don't. They make good money using location to target ads for instance.

<comment> village-idiot > Well, they have to give your location information to the government.

It's not like this requirement was an inevitable outcome of governments. We can change laws.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> That's the case in France where DNA samples are taken on any person arrested, whether they committed a crime or not, including demonstrators (or bystanders) in a political march.

The national DNA database now includes over 5 percent of the population and it is a great tool for the police since the authors of serious crimes typically have almost always committed petty crimes before. It also means that you can solve small crimes like burglaries with DNA.

It is a bit worrying from a privacy protection point of view though. And I barely see this database ever questioned by the French press.

<comment><USER8> > That's the case in France where DNA samples are taken on any person > arrested, whether they committed a crime or not, including demonstrators (or > bystanders) in a political march.

FWIW ~20 US states do the same, this was ruled legal in 2013 (Maryland v. King, DNA collection is part of police booking procedure).

CODIS (the FBI's DNA database) contains more than 17 million non-forensic profiles[0], that's also >5% of the population.

But wait, there's more!

The Department of Defense's own DNA database (DoDSR) has more than 50 million records (collection started in the 80s and every applicant to a uniformed service gets included), and since the 2003 National Defense Authorization Act can be accessed by federal or military investigations for "the purpose of an investigation or prosecution of a felony, or any sexual offense, for which no other source of DNA information is reasonably available". So it can't currently be searched / "wild matched" against an unknown sample but if there's a suspect, a sample and no other DNA source then it's an option.

[0] [https://www.fbi.gov/services/laboratory/biometric- analysis/c...](https://www.fbi.gov/services/laboratory/biometric- analysis/codis/ndis-statistics)

<comment><USER9> >The Department of Defense's own DNA database (DoDSR)

The reality of this is that it's just a big ass warehouse deep in the farmland of Virginia which houses unprocessed tubes of DNA. Its like the end of Raiders of the Lost Ark.

Sounds nefarious but it is actually a goldmine for genetics research.

<comment> village-idiot Turns out even genetic databases need an index.

<comment><USER8> As noted in the original comment, currently DoDSR can only be used for nominative sampling aka you have a name and want a known generic sample (and you have a federal or military judge allowing it). The limitations of the medium / storage means it can't practically be used beyond that.

</comment></comment></comment></comment></comment><comment><USER10> DNA should only be used to prove innocence and to corroborate other evidence. Just because someone's DNA is there, doesn't mean they did something. It is too easy to start planting DNA. If there is other evidence and their DNA is there, then yeah. Book 'em Danno!

<comment><USER11> Would you feel differently if your child's murerer was on trial?

Not trying to be gross, just trying to shine light on the idea that judges are in a difficult situation. They have to consider fairness for all parties involved.

<comment><USER12> That question is one of the pit falls we have to avoid to preserve any chance of avoiding a police surveillance state. "What would you allow if you were maximally emotionally impacted by a crime" is basically a blank check for 100% surveillance because inevitably any restriction on methods will eventually allow some crime to go unsolved.

<comment><USER13> Exactly this. At worst, you need to at least taper the emotionally charged question with the counter-point; "Would you feel differently if your child was falsely accused of murder because of genetic comtamination?"

<comment><USER12> Yep, to see how badly things can go when 'scientific' evidence goes unquestioned look at the mess that is stuff like bite pattern analysis which gets presented as scientific but has really bad actual statistics.

Justice can't just be about catching people it has to also weigh the cost of false positives and the effect that the methods have on society.

It's a constant issue with things like TOR, encryption, and alt currencies like BTC. They can do a lot of good for repressed people but they're also inevitably going to be used by people most people would agree are awful people to protect themselves from justice. If you're developing those tools you have to be ok with that just like we have to decide as a people where we're ok with the balance between preventing crime/catching criminals and personal liberty and privacy.

</comment></comment></comment></comment></comment><comment><USER14> I built a prototype of a Private Set Intersetion using elliptic curve homomorphic encryption method for this very use case: The scenario goes 1) FBI indexes the dna using MASH algorithm (minhash like locality sensitive hashing of genomic sequences) of the person they are searching for. They then use homomorphic encryption and post this encrypted data online with a bounty for anyone to claim. 2)Now, individuals and corporations can try to claim the bounty; they also run MASH on their dna databases and encrypt each mash index using the same homomorphic strategy and submit it back to the FBI. 3)The FBI can now complete the private set intersection algorithm on the encrypted data and will only reveal information iff an assailants genealogy is found. The remaining sequences appear cryptographically random and thus protect the privacy of those persons.

I am currently working in blockchain space. If you want to know more feel free to message me.

<comment><USER15> Please stop making the world a worse place to live in.

</comment></comment><comment><USER16> I've heard it argued that the police and other authorities have the right to know who you are.

Is this really the case? Does a representative of a government have the right to your identity?

If so... is there an extent to which they can know who are you are? The difference here between a vague answer, a trust based assertion, or even a deep proof.

<comment><USER17> There is a distinction between your identity(who you are) and your legal identity(who the governments papers say you are). Your legal identity is created with your birth certificate and ends with your death certificate.

Legaly you agreed to act under this legal identity when your birth certificate was created but since you where too young to give consent its not legaly binding.

Thats the argument but it would need to go through a judge who wont care about technicality of law as the law itself is a technicality to justify government.

<comment><USER18> This is direct from Freeman on the land conspiracy theory, and it's a legal nonsense.

<comment><USER17> If this has been debunked I would like to know and stop repeating it, can you provide links?

</comment></comment></comment></comment><comment><USER19> Genetic data affects not just an individual but their family tree as well. Maybe a warrant should have to be issued for all of them to get access to this data.

<comment><USER20> But wouldn't this conflict to the (natural?) right of an individual to publish their own data? Do I need a permission from all my family tree to be allowed to publish the contents of my DNA?

<comment><USER21> If, outside of our control, your password and mine matched except for one character and that near match was public knowledge, would you object to me publishing my password here in cleartext?

There's a big difference between publishing your own thoughts - which may have nothing to do with what your family thinks - and publishing your DNA, which is inextricably tied to that of your family. It's my data, which I have a "natural" right to share as I see fit, but it's also very similar to yours, which you have a right to keep private. These rights are incompatible, one has to give.

<comment><USER4> > If, outside of our control, your password and mine matched except for one > character and that near match was public knowledge, would you object to me > publishing my password here in cleartext?

I would certainly prefer that you not do that, but I don't think I should be able to to _compel_ you not to. this is sort of a weird example because there's no clear benefit to posting your password.

a similar example: you and I both live in the same apartment building. you really hate my best friend and don't want them to know where you live. my best friend already knows that you and I live in the same apartment building. should you be able to prevent me from giving my address to my friend so we can hang out?

<comment><USER22> Ok how about imagine that friend was stalker. Someone potentially dangerous to me and those around me. They know we live in the same building so yes in that case it would make perfectly sense that I could compel you not to give away your address to that person.

<comment><USER4> imo that's only reasonable if we're actually roommates. if not, you should pursue a restraining order against this person, not try to restrict me from releasing my own address.

</comment></comment></comment></comment></comment></comment><comment><USER23> Data scientist here. I'm surprised nobody here has yet mentioned the false- positive issue with trawling a large dataset: even a _tiny_ false positive rate leads to a very high probability of convicting an innocent person if you mine a large enough database.

<comment><USER24> But the only reason for such a database _is_ to get a conviction. Case solved.

</comment></comment><comment><USER25> Many of the comments here seem to assume that the use of DNA tests without the accused's consent is a bad thing. I fail to see why. We use facial features and eyewitness-testimony to ID criminal dependents all the time. No one argues that consent/warrant is needed in order to ID someone using their facial features. More generally speaking, I fail to see the harm that can arise from using DNA to identify suspects, whereas the benefits are so much more compelling.

<comment><USER26> DNA is only circumstantial. Being a suspect is harmful. Hence DNA testing could make innocent people suspects and thus harm them.

Since harming innocents is bad, there is a downside to wide usage of DNA tests. Whether tgat downside weighs up against the upside is a debate. But we need to acknowledge there is a debate to be had.

<comment><USER25> Sure, but I think it's also worth acknowledging that suspects get identified all the time on the basis of circumstantial evidence far flimsier than DNA traces. The "burden of proof" needed for the police to investigate someone as a suspect is extremely low, for good reasons.

</comment></comment></comment><comment><USER27> Interestingly enough, I recently received a bulk email from GEDMatch.com about exactly this subject.

They changed their TOS so that now users need to explicitly opt in to sharing their data with law enforcement. That's good for user privacy, but it does make me wonder what they were doing before this change.

<comment><USER28> The Daily did a podcast interviewing the founder of GEDmatch.

Part1: [https://www.nytimes.com/2019/06/06/podcasts/the-daily/dna- ge...](https://www.nytimes.com/2019/06/06/podcasts/the-daily/dna-genealogy- crime.html)

Part2: [https://www.nytimes.com/2019/06/07/podcasts/the- daily/geneal...](https://www.nytimes.com/2019/06/07/podcasts/the- daily/genealogy-dna-crime-privacy.html)

</comment></comment><comment><USER29> I think whether you find this useful or dangerous hinges entirely on your perspective on governments and history.

If you are sure governments are mostly a force for good and democracy is suited to keep it that way indefinitly, overpowered permissions like these have no real downside.

As engineers we certainly also ask ourselves how systems fail when something goes wrong, even if all we do is to prevent this. This kind of power is not only a possibility for a state to become more powerful, it is also the perfect ethnic genocide toolkit and therefore something that makes it substancially more dangerous for a state to fail.

Separation of powers, the oversight of the press, limited power and information for the state over the individual without good reason – all these mechanisms are there _for a systemic reason._ These reasons are sometimes only to make it harder for governments to drift into unlawful and unjust behaviour. Any change to the balance of power between executive, juridicative, legislative ans public should be carefully considered before executed.

So this must _really_ pay off in order to be useful enough to justify the risk that comes with it, including the (in?)effectiveness of punishments and the cost of prevention through other means than harsh punishment.

</comment><comment><USER30> Sure, but we need to publish the DNA and contact info of anyone who becomes a police officer. What is sauce for the goose is sauce for the gander.

</comment><comment><USER31> In broad sense all investigation techniques are privacy invasion: surveillance, search, detention or arrest. I think we should let law enforcement use DNA sources. But it should be heavily regulated, and under heavy control.

Let's say anonymous inquiry that just return yes/no and then you need court to reveal identity.

</comment><comment><USER32> All those DNA tests with ancestry.com are also free fodder for the five eyes and law enforcement

</comment><comment><USER33> Seems to me the legal question is whether it's a reasonable search. If a court rules in the affirmative that poses a second question: search what, and does that imply that right for police to collect DNA evidence in order to conduct a thorough search?

</comment><comment><USER34> I'm glad the Original Night Stalker was caught, but I found it very disturbing he was caught based on another family member doing one of those ancestory DNA things.

Also, offtopic, but that dendrogram in the first picture is a very incestual family tree.

</comment><comment><USER35> > If the police felt free to use it in an assault case, why not shoplifting, > trespassing or littering?

For the same reason dna tests are not already used for shoplifting, trespassing or littering?

<comment><USER36> And that reason is...?

<comment><USER37> DNA has almost no probative value in those cases, since they almost never involve a sample which can be genotyped that has a high probability of being linked to a particular crime.

</comment></comment></comment><comment><USER38> I think the justice system needs to be reformed to a rehabilitation modal and before it's able to investigate one's genetic family tree for any crime. I've studied neuro science and the philosophy of determinism to realize people don't have any real control for their actions. So right now I think we're failing when it comes to humanity.</comment>


China produces 95% of world's gallium, used in 5G base station chipsets - <AUTHOR> https://www.fierceelectronics.com/electronics/china-produces-95-world-s-gallium-used-5g-base-station-chipsets <comment><USER2> Used in a lot more than 5G chipsets, gobs and gobs of things use GaAs (gallium arsenide). IR LEDs, tons of different ICs, PV cells/panels etc. It's also just fun to play with, the Slow Mo guys have a great video playing with gallium [https://www.youtube.com/watch?v=A9J1gkw9BI0](https://www.youtube.com/watch?v=A9J1gkw9BI0)</comment>


Distributed systems for fun and profit (2013) - <AUTHOR> http://book.mixu.net/distsys/single-page.html <comment><USER2> Distributed and highly available systems are pretty interesting but something I've noticed is that it's really surprising how much you can get done with a centralized system (say a single redis instance) and how much easier it is. Some of the cloud VMs can run for years with only transient issues (GCE live migration is neat).

High availability is really cool as well, but I recently learned that world of Warcraft has apparently had <97% uptime for years due to weekly maintenance. That leads me to reconsider just how necessary those additional nines are. It seems like a lot of services would be better off optimizing for, say, operational efficiency than uptime. Doing server upgrades and rolling out patches when you have a multi hour maintenance window sounds waaay easier than live updates to your servers.

Of course it depends entirely on the service you are running, but I'm still impressed that a service as large as world of Warcraft can work fine with that level of uptime.

<comment><USER3> 97% is not impressive. Nine nine's reliability is - like Ericsson's AXD301, built using Erlang: [http://stackoverflow.com/questions/8426897/erlangs-99-999999...](http://stackoverflow.com/questions/8426897/erlangs-99-9999999-nine- nines-reliability)

<comment><USER4> I think nulltype's point is that yes, 97% is considered "low reliability" by software engineers, but Blizzard and the WoW players don't seem to mind this low reliability.

</comment></comment></comment><comment><USER5> I've been reading this, and I think it's a great intro to the subject that isn't inscrutable and doesn't go into excessive detail. I also like: [http://christophermeiklejohn.com/distributed/systems/2013/07...](http://christophermeiklejohn.com/distributed/systems/2013/07/12/readings- in-distributed-systems.html) as a source for more in depth readings in specific subjects. Many of these are seminal papers

</comment><comment><USER6> [http://nil.csail.mit.edu/6.824/2015/schedule.html](http://nil.csail.mit.edu/6.824/2015/schedule.html)

Someone posted this last time I asked about distributed systems.</comment>


Switching from C# to Java - <AUTHOR> http://lpeer.blogspot.com/2010/04/switching-from-c-to-java.html

<comment><USER2> Totally reactionary, prejudiced statement.

But my first reaction on headline before reading the article was "Ugh. I'm sorry". (I will read it in a second, I felt this obdurate urge to rant BEFORE I read it).

I coded in Java for work (After being hired to do Java) for years, then went over to Python, and worked with some C# for about a year. Now I'm back working with Java, and Scala (for which I have lots of love - it has a lot of what I love about Python but in many cases better).

There are little things in C# that just really give it an edge over Java. Among other things relating to Generics (Which are awful to begin with in that the JVM currently erases type information at compile time [Scala has a few clever hacks to get around it]) - as I understand it, C#/CLR automatically generates multiple specialized paths for all possible primitive types with generic collections.

And being a fan of closures, anonymous functions, etc. Java is just painful. C# has these, and more.

Now Off to read the article :)

<comment><USER3> I keep seeing people bash Java's Generics because of type erasure - but what exactly is so awful about it? All the questions on stackoverflow where the answer is "impossible because of type erasure, or use this workaround" don't seem to be about terribly important or useful things.

<comment><USER4> The case that I've bumped up against several times is when trying to get the class of the generic object. Something like:

      T obj;
      obj = getObjFromSomewhere();
      log(T.class);

This type of operation is most useful if writing framework type of some sort. I've come across it when trying to write a generic JPA base DAO type of class.

<comment><USER2> Scala has Manifests to emulate reified types for just this reason.

[http://www.scala-blogs.org/2008/10/manifests-reified- types.h...](http://www.scala-blogs.org/2008/10/manifests-reified-types.html)

It's improved a bit in 2.8 - in porting some code I discovered a nice little hack that lets you declare your variance like this:

[A <: SomeBaseClass <: Manifest]

Which tells the compiler in an oh so gentle manner to reify for you.

</comment></comment></comment></comment><comment><USER5> I can understand porting C# code to Java code, but what I don't understand and what the article doesn't explain is how they handled their dependencies, i.e. System.* libraries.

They clearly didn't port the CLR libraries to the JVM, so they must have modified their code to work with Java libraries, e.g. Swing, etc. To me, it seems like this would be the bulk of the work, not language translation from Java to C#.

<comment><USER6> Good point - its not the language translation per-se but the ecosystem surrounding it unless there's is one of those projects where they hand-coded everything and used no external libraries at all. I doubt it very much though.

</comment></comment><comment><USER7> How does the converted code look, though? Is it viable to continue working on that, rather than fixing the original C# code? (Ie comments would be carried over in the conversion).

<comment><USER8> That would be my question. In my experience code that's been transformed programmatically is not meant for the eyes of mortal man.

</comment></comment><comment><USER9> It sounds like they rejected Grasshopper, the .NET to Java converter that works at the bytecode level, without doing a test to see if it would have worked, on the grounds that they wanted to be open source.

They probably should have given it a try, because if it had worked, it would have opened up a possibility that they seem to have completely overlooked. Use Grasshopper to convert at the bytecode level, then use a Java decompiler to go from bytecode to Java source.

You'd lose comments in the source code that way, but I bet it would not be hard to write a tool to transfer the function level comments from the C# source to the Java source. Subsequent hand editing to translate any C#-isms to Java-isms in the comments, and restore any crucial sub-function level comments, would probably have only taken a few days.

</comment><comment><USER10> As with many HN articles, the headline is misleading. Really, the language switch is a symptom, and not the driver of the decision.

From the article, their requirements included:

 _one of the first tasks we got was to make the management application cross platform, well this was expected considering the fact that the acquisition was done by RedHat._

And:

 _Sticking with C# requires technologies to help us run it on Linux. We found Mono ..., and the second option we found was Grasshopper which is a project of Mainsoft to compile MSIL to Java Bytecode. ... These 2 solutions were taken off the table ... we wanted to use a technology supported by Red Hat._

So it appears to me that the decision had nothing at all to do with technical strengths of the language. Moreover, it had nothing to do with the platform as such -- whether its technical strengths or any end-user demographics -- but purely politics driven by Red Hat.

<comment><USER11> No, they tried Mono and found it wasn't ready for primetime. Read the article.

</comment></comment><comment><USER12> I wonder why they couldn't use Mono. I can't really grok how they managed library usage, and how difficult it will be to maintain the java one when on platform updates etc.</comment>


Boomer outsmarts hackers: "Kiss your cash goodbye" - <AUTHOR> https://cybernews.com/security/boomer-outsmarts-hackers-kiss-your-cash-goodbye/ <comment><USER2> Metro-bank saved him by repeatedly refusing to let him send his cash to the scammer. He couldn't outsmart his own shoes.

<comment><AUTHOR> [Author of the article] I understand your point. But the headline is always more of an editorial choice. Nonetheless, Mr. Richards was able to come out on top, and that includes his picking up on the scam throughout the conversation with the hacker.

Also, I'd like not to lose sight of the other main points of the article: -PayPal's inability to stop obvious fraud -Modulr Finance's position and reputation for assisting scammers, especially in its ability to create new accounts in seconds -Ms. Matthews, obviously a victim of the scam, not getting proper assistance from her bank

<comment><USER3> >But the headline is always more of an editorial choice

I think the word you're looking for is "clickbait"

<comment><AUTHOR> > I think the word you're looking for is "clickbait"

We're a young site, and we're still testing and learning, like many startups and companies and businesses. If you don't test, you'll never know, right?

Did you read the article though? I'd love to hear what you thought about it</comment></comment></comment></comment>