MIT computer scientists can predict the price of Bitcoin - <AUTHOR> http://newsoffice.mit.edu/2014/mit-computer-scientists-can-predict-price-bitcoin#.VEadQI474jk <comment><USER2> The problem with the paper is not overfit. They claim to have run their simulation with out of band ("live") data. The _actual_ problem with the paper is that we have no idea if their simulator is any good, which means that their result (89% return in 50 days) could be totally bogus. In other words, we don't know if the actual bitcoin exchange would fill their orders at the same prices (if at all) as their simulator does. A decent simulator for a high- frequecy strategy like this is not trivial because you have to incorporate all the exchange behavior (documented or not) into your simulator, and then you have to validate your results by comparing your simulated results to the results of some actual trading. The fact that they spent none of the paper on the details of the simulator makes me extremely skeptical.

<comment><USER3> People so often overlook the role that execution plays in trading. As time scales shrink the impact of execution increases, and indeed many HFT strategies are not profitable without top tier execution (both in terms of fee schedules and technology). This is also one of the places that most academics fail when analysing a trading strategy. They make typical "assume a frictionless surface" types of assumptions that break down in a real market.

To your point, high-frequency simulation is damn hard and it is highly likely that they failed here and have tainted their results with bad simulations. Most of these papers would be well served to avoid dollars and cents and simply analyse the statistical qualities of their signal to their target over time. That is the first step in this business, anyway. Simulations only happen once the signal has been through the wringer.

<comment><USER4> I see no mention of the spread, costs or even ability to put on a short position, exchange lag variability (huge issue when simulating even on modern exchanges, let alone fly-by-night bitcoin markets). Additionally, it is very easy to find trend-following signals that "work" but break down when conditions change very quickly. That seems to make up most of the prediction, along with an order book imbalance signal that might be more stable.

I think a better approach would be looking at order book features and lags vs. other markets. The costs are high on BTC markets so it would be pretty tough to overcome those though. Anyone with experience to do this is probably doing it somewhere more lucrative. I think BTC markets only trade a few million USD a day.

<comment><USER5> About 35 million USD in trades on the public bitcoin-paired markets today. About 40 million USD today in total volume for all pairs on these exchanges.

[http://www.cryptocoincharts.info/coins/info](http://www.cryptocoincharts.info/coins/info)

<comment><USER4> More than when I last looked, but to put that in perspective, you could trade $40mm in a few price ticks in major currency pairs and CME avg daily turnover alone is > $100bn: [http://cmegroup.mediaroom.com/2013-06-07-CME-Group-Sees- Reco...](http://cmegroup.mediaroom.com/2013-06-07-CME-Group-Sees-Record- Trading-in-its-Global-FX-Complex-Transacted-239-billion-in-Notional-Volume- Across-FX-Futures-and- Options?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+mediaroom%2FCMsF+%28CME+Group+Media+Room%29)

Public spot FX markets and OTC trading do huge volumes too.

Some BTC exchanges charge like 60 bps per trade. Finding a signal to overcome that cost and do enough trading to make it worthwhile would be quite difficult.

</comment></comment></comment></comment></comment><comment><USER6> This is short-term trading. "Every two seconds they predicted the average price movement (on OKcoin) over the following 10 seconds. If the price movement was higher than a certain threshold, they bought a Bitcoin; if it was lower than the opposite threshold, they sold one; and if it was in-between, they did nothing." I don't see them allowing for commissions and fees.

OKcoin, at peak, had a trading volume so high that it's generally considered to be fake - the exchange operators manipulating the price. What this group at MIT may have done is reverse-engineered the fake trade generation algorithm.

<comment><USER7> > What this group at MIT may have done is reverse-engineered the fake trade > generation algorithm.

Just to be clear, there is nothing wrong with this. Infact, sitting around and reverse engineering what other traders are doing is what many funds do. I'm in this group so I"m happy to answer questions if anyone has any.

> Every two seconds they predicted the average price movement (on OKcoin) over > the following 10 seconds. If the price movement was higher than a certain > threshold, they bought a Bitcoin; if it was lower than the opposite > threshold, they sold one; and if it was in-between, they did nothing.

To be clear, this is the core of what most HFT systems do these days. Consume many different factors, give each factor a decay factor to tell the system when the signal goes stale and distill them all into a value that says, but or sell or stay.

It's worked well for Renaissance Technologies:)

[http://en.wikipedia.org/wiki/Renaissance_Technologies](http://en.wikipedia.org/wiki/Renaissance_Technologies)

<comment><USER8> Where can I read about the actual strategies used? Perhaps some that aren't used any more. I've read an absolute ton about how to set up a trading system with hadoop and cassandra and blah blah but that's all pretty trivial. I'd be more interested in the strategies... this decay factor is interesting. Where did you find out about this?

<comment><USER4> A simple signal would be more shares bid than offered at the inside market. Imagine a market with 1000000 shares bid at $4 and 100 shares offered at $4.01, it's more likely to tick up than down in the very near-term. This isn't tradable though since the predictions aren't large enough to overcome the spread and you can't expect to join the 1000000 share bid and trade when it's good (the 1000000 shares that made you think it was going to move up all have to trade or cancel for you to trade, not looking so great anymore).

Real HFTs use other features of the order book, movements in related products (maybe a move in oil futures in the last minute impacts the price of an airline stock in the next 10 seconds, etc.), and so on. The signals are not really that complicated, but competitors eventually converge on knowing the same signals and compete them out of existence/profitability. Also, how you execute around your prediction matters just as much if not more.

[http://queue.acm.org/detail.cfm?id=2534976](http://queue.acm.org/detail.cfm?id=2534976) and [http://queue.acm.org/detail.cfm?id=2536492](http://queue.acm.org/detail.cfm?id=2536492) have some more examples. [http://www.decal.org/file/2945](http://www.decal.org/file/2945) also has some ideas for old signals that probably won't work anymore.

</comment></comment></comment></comment><comment><USER9> This seems like massive historical overfit, which can lead to arbitrarily precise fit, but no predictive capability.

Any model, if given enough parameters, can be made to match historical data to an arbitrary degree.

I also run several Bitcoin bots. I can tell you that slippage is not insignificant. If you make transactions every ~10 seconds and incur 0.1% fees each time, this is an extremely significant effect in aggregate. Also bid-ask spreads, while usually small, often aren't in periods of high volume.

<comment><USER10> Agreed. According to the paper: They trained the data, once, with 3-4 months of data from Feb to May. Then they tested the data, once, with ~6 weeks of data from May 6 - June 24.

There was no cross validation involved in assessing the performance of this model. Using one subset for training, and one subset for testing and calling that conclusive is naive.

No interesting conclusions can be taken away from this paper due to flawed methodology and failure to take into account real world variables like bid-ask spread, commissions, how quickly trades can be executed, whether orders will even be filled, etc.

<comment><USER11> Cross validation too uses training sets and test sets. This sort of time- ordered data will not be independent, so the prequential approach seems to be a more suitable approach to measuring forecast accuracy. (I haven't read the paper.)

<comment><USER10> Cross validation for time series, goes something like this:

[http://robjhyndman.com/hyndsight/crossvalidation/](http://robjhyndman.com/hyndsight/crossvalidation/)

(near bottom of the article)

Anyways, as somebody who has spent GIANT amounts of time experimenting with machine learning using market data (mainly stocks), I can give you TONS algorithms I've created that would show similar, even much higher, gains when you only test on 6 weeks of data. Been there done that. For example 6 weeks you make a 100% return, then in the next 3 weeks you suffer a 50% loss. Reality sets in...

</comment></comment></comment></comment><comment><USER12> Sometime in 2013, before the bitcoin prices exploded, I downloaded some bitcoin historical price data and ran symbolic regression on it with Eureqa. It came up with a formula that fit the observed data fairly well, and wasn't very complicated.

But when I extrapolated it forward a few months, it predicted the price would explode to unreasonable levels. I was disappointed and threw it away, assuming that it must be wrong.

<comment><USER13> Thank you for sharing the Eureqa software. Wasn't aware of it. +1

</comment></comment><comment><USER14> I'm late to comment, but something which I'd like to point out is that this is done by the same team behind the Twitter trending topic prediction technique from a few years back, as mentioned also in the article [1].

When their Twitter technique was released, I spend a few weeks reading through Nikolov's PhD thesis (the advisor gets most of the the fame in the press articles but Nikolov's thesis has all the details) and trying to implement it in R. My observations at the time: extremely simple algorithm which would be shot down by most peer reviewers for being not very novel (the affiliation helps a lot here). That said, I believe greatly in pragmatism, and the approach was actually working well. What I did find out however is that their was a great deal of data selection and pre-processing involved making the approach hard to implement in a real-life, real-time setup. I get similar feelings from this work.

[1]: [http://newsoffice.mit.edu/2012/predicting-twitter- trending-t...](http://newsoffice.mit.edu/2012/predicting-twitter-trending- topics-1101)

<comment><USER15> Could you specify what kind of data selection and pre-processing was required and why exactly it was hard to implement it in a real-life setup?

</comment></comment><comment><USER16> The paper states that the strategy was simulated with live data and makes no mention of slippage. I've never traded bitcoin so I'm not sure how difficult it is to get fills, but that along with spreads are non-trivial components of real trading.

<comment><USER17> it would have been better if they set up an wallet and put bitcoin in it and demonstrated the trades that were automatically executed by this wallet. Since bitcoin is a ledger history, we could be 100% certain that SOME algorithm 'did the right thing', although we couldn't be certain that they spun up more than one algorithm and then just showed us the best one.

<comment><USER18> ...That's not how Bitcoin trading works. In order to trade, your bitcoins need to be deposited into an exchange. Trades are not recorded on bitcoin's public ledger because there isn't a transaction for every trade. Bitcoin cannot currently handle that many transactions.

</comment></comment></comment><comment><USER19> They didn't include any discussion of:

1) execution (are they expecting to buy on the bid and sell the offer?).

2) commissions. They only made 3,362 yuan on 2,872 trades. A yuan is about 12 cents, so they are making 15 cents USD per trade.

A .1% commission would cost them roughly 5 yuan per trade, but they are only making 1.17 yuan/trade.

<comment><USER20> OKcoin (the exchange they relied on in the paper for data) claims 0% commissions and a bid-ask spread of just 0.04 RMB. how real is that, I have no idea. but if you relied on this information you could believe that you can still take a few additional spreads worth of slippage (in addition to crossing) to compensate for execution latency and still come out very far ahead

</comment></comment><comment><USER21> Success in predicting markets is measured in profit.

This research team should start a company that offers a service that allows users to deposit bitcoins, which the company then invests according to their alleged predictions, and then pay interest on deposits, and keep a part of the profit for themselves.

Doubling the initial investment one time is one thing, but this hypothetical company being able to double its investment every 50 days for years is something else. I doubt they can. A doubling every 50 days is x160 every year.

I think claims of being able to predict market prices should be met with great skepticism. Especially prices of easily traded commodities, including bitcoins.

The only proper measure of an ability to predict market prices is profit, because profit also measures the extent of the predictions: how much can you move the market (by trading according to your predictions) until you can no longer predict what will happen? Obviously, there's a limit. No one can extract unlimited profit from any market. So there definitely _is_ a limit to how much you can earn from your algorithm. If you can earn 10% p.a. on an investment of maximum $5000, your algorithm isn't really worth much. If you can earn 1000% p.a. on an investment of up to $100M, your algorithm is _great_. But without knowing these figures we really only have a claim, seemingly a claim of them being able to make _a lot_ of money, but choosing not to do so.

</comment><comment><USER22> I fixed the article's headline image: [http://i.imgur.com/QVgcgNI.png](http://i.imgur.com/QVgcgNI.png)

If everyone began using the paper's strategy, would the strategy still work?

Also, the strategy seems less effective than portrayed in the news article. If you look at the "results" section, it seems like the profit flatlined shortly after starting, then had success due to some major trading event, then eventually flatlined again: [http://i.imgur.com/CBjEjgo.png](http://i.imgur.com/CBjEjgo.png)

Wouldn't it be more accurate to say "this strategy is effective under some very specific circumstances"?

Also, does anyone know how equation 4 was derived? [http://i.imgur.com/vkx8ZEC.png](http://i.imgur.com/vkx8ZEC.png)

It seems like the key insight of the paper, but there's no mention of where it originated from. Is it a common equation in statistical modeling? I'd like to learn more about it. Does anyone have any suggested reading or coursework I should study?

<comment><USER23> I think equation 4 is just derived from equation 3. The top bit counts every time before that y_i has taken a particular value y, and multiplies it by the distance of the x value then, x_i, from the value of x now. (Squared and exponentiated because this is the pdf of the normal distribution.)

Here's an interpretation: if there were no noise, you might just count the number of times x took on a particular value and y took on a particular value, and divide that by the total number of times x took on that value. This would give you an empirical estimate of the prob of y given x.

Because there's noise, they weight the counts by the pdf of the normal distribution of x - x_i. So, whenever x_i was close to current x, and y_i was a given y, that increases the probability of y occuring now.

</comment></comment><comment><USER24> How many of the people who are bashing the paper in this discussion are machine learning experts? Especially the overfitting crowd - looks like emotional attachment to one's favorite topics is not really impacted by said person's overall education and expertise.

<comment><USER12> Here is the discussion on /r/MachineLearning: [https://www.reddit.com/r/MachineLearning/comments/2jwpv1/mit...](https://www.reddit.com/r/MachineLearning/comments/2jwpv1/mit_computer_scientists_can_predict_the_price_of/)

</comment></comment><comment><USER25> ... for the next 15 minutes.

When you predict the future of a market, you change the future of that market. People start investing on the basis of your predictions and whatever opportunity for profit you found is closed. This is why HFT people iterate constantly and also why they put their servers as physically close to the market as possible.

<comment><USER26> > This is why HFT people iterate constantly and also why they put their > servers as physically close to the market as possible.

Do you happen to have read this :-

[http://www.nytimes.com/2014/04/06/magazine/flash-boys- michae...](http://www.nytimes.com/2014/04/06/magazine/flash-boys-michael- lewis.html?_r=4)

</comment></comment><comment><USER27> No they can't. If they could they wouldn't tell anyone, and they would make millions (billions?) of dollars.

</comment><comment><USER28> This kind of innovation is cool, but it's a zero sum game. The bitcoin markets are already driven by competing bots. Their profits will be reduced as other bots iterate on their algorithms.

<comment><USER29> Pleas don't spout out 'zero sum' like you think it means something insightful here. Any marketplace is 'zero sum' if you think about it. There's a buyer and seller. So what?

<comment><USER30> Most marketplaces are not zero-sum.

For example, futures and options are zero-sum because every dollar of profit that one trader makes is offset by a dollar of loss from another trader.

But stock markets are not zero-sum. Prices can be bid up without a single share changing hands, creating new wealth out of thin air, and everyone wins. Conversely, prices can go to zero, destroying wealth and making everyone a loser.

Same thing for most other markets you can think of, from real estate to used cars to your local flea market. As with bitcoin exchanges, none of these are zero-sum since there doesn't have to be a loser for every winner.

<comment><USER3> You're mixing the concepts of realized and un-realized profit. A market price change marks your open position and gives you an unrealized (loss)gain. But to realize that you have to trade, plain and simple. And someone must take the other side of that.

A more compelling argument against zero-sum is that traders and investors have different time frames and objectives. And, indeed, the classic argument in the futures market is that a farmer who buys a hedge from a speculator represents a potential positive outcome for both.

<comment><USER30> I start a company to make cheap autonomous flying cars. I invest $1,000 of my own money and build a working prototype. It looks promising, so you buy 50% of my shares for $10,000.

I just made a 20x return on my investment, and you own 50% of a great opportunity. Who's the loser here that makes this a zero-sum market?

Our autonomous flying cars go into production, and now your 50% is worth $100 billion, and you sell your shares to Elon Musk. Where's the zero sum?

<comment><USER3> First, I never claimed the markets were or were not zero-sum, only that your attempt to explain was wrong as it mixed realized and unrealized profit.

Second, zero sum requires a definition of utility for each participant. If those definitions vary (and they almost assuredly do) then a reasonable argument can be made that to define whether the activity is a zero-sum game or not requires a commonality amongst those utilities (dollars, for example). This is often not the case in trading (for example, the farmer hedging his crop with a speculator who seems to profit short term).

</comment></comment></comment></comment></comment></comment><comment><USER31> They should have made more money rather than publishing more quickly. It used to be possible to do these sorts of things to the stock market but when these sorts of regularities are discovered the process of exploiting them also eliminates them once enough money is being made. Heck, a major trading firm got started by noticing that stocks went down on the weekend (and of course they don't any more).

</comment><comment><USER32> 89% over 2 months. Not bad. My algorithm did 8,000% over 6 months using real money on real exchanges and I have the trade history to back it up.

<comment><USER33> What exchanges do you recommend for bot trading? ... I've dabbled a bit trading manually on bitfinex but that's all.

</comment></comment><comment><USER34> Can a HFT-knowledgeable commenter chime in on the viability of the Sharpe ratio here?

From a physics perspective, it appears that the Sharpe ratio of 4.1 is roughly equivalent to a 4.1-sigma claim that their algorithm is better than random trading. I can't check easily, but I'd guess that the movement of Bitcoin prices isn't normally-distributed (looking at the paper's time series suggests that there's more low-frequency power there). If so, I'd guess that a more robust measure of the claim's significance would show it to be less significant.

Put differently, I'd guess more than 1 in 15,000 random sets of 2872 trades (their number of trades) would yield comparable profit. Furthermore, a simple buy-and-hold would've yielded a 20+% return over the same period.

<comment><USER3> A sharpe of 4 is ok. Most "HFT" type strategies have sharpes so high they don't talk about sharpe anymore as it is meaningless. It is simply a different type of trading. See Virtu's pnl distribution in their S1. As a point of reference, Blair Hull is on record saying they are interested in nothing below a sharpe of 10 (r-finance talk, should be googleable).

<comment><USER35> Thanks for sharing.

Link: [http://www.nasdaq.com/markets/ipos/filing.ashx?filingid=9443...](http://www.nasdaq.com/markets/ipos/filing.ashx?filingid=9443224)

</comment></comment></comment><comment><USER36> The fact is anyone can *Predict the price of bitcoin, but will you actually put your own money to trade?

Like what we did here, [http://financeai.com/forex/btc](http://financeai.com/forex/btc) it does show some degree of correlation between sentiment and the price.

</comment><comment><USER37> Ever thought how our big data overlords (Google, Facebook, MS, Twitter, etc) just need to check for correlations between their users' data input and stock exchange movement?

They own the stock "matrix".

<comment><USER36> You mean something like this?

[http://financeai.com/stock/nyse/twtr](http://financeai.com/stock/nyse/twtr)

</comment></comment><comment><USER38> Those interested in automatic trading of bitcoin using algorithms should check out [https://cryptotrader.org](https://cryptotrader.org)

</comment><comment><USER39> By publishing the paper you essentially invalidate it, as people take advantage of it. Happens time and time again in any market.

</comment><comment><USER40> doesn't predicting the price change the price? Just like how knowing the future changes the future.

</comment><comment><USER41> He says "Give me your money and I'd be happy to invest it for you." Alright, I'll do it... just tell me how.

</comment><comment><USER42> SLIPPAGE SLIPPAGE SLIPPAGE!!!

</comment><comment><USER43> Can they predict how to get real money back in exchange for their Bitcoins?</comment>


Edward Tufte on Background Color - <AUTHOR> http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0000M0

<comment><USER2> Good points, there are also biological concerns where your pupils shrink with brightness making brighter displays clearer.

Another semi-bio related issue is contrast. The net aggregate brightness of the display should be roughly similar to the surroundings. So a "white background" needs to be microscopically brighter than the room with black letters, but a "black background" needs the few illuminated glyphs to be incredibly bright to meet the same average brightness level.

I grew up on terminals and CRTs where for visual quality (and tube lifetime) you needed white text / black background but having the luxury of choosing for a couple decades now, I am happier with black letters on white-ish background.

I would be very interested in seemingly obvious neuroscience study if your whole visual system works better with light fine details on a dark background or the opposite colorscheme. There doesn't seem to be an obvious answer. For example a wolves eyes on a pitch black forest background should be highly noticeable, which sounds good, but it should also be highly stressful for obvious reasons.</comment>


3rd party report shows anomalous heat production - The Rossi Effect - <AUTHOR> http://ecat.com/news/3rd-party-report-shows-anomalous-heat-production-the-rossi-effect <comment><USER2> here is what I want to see. A double-blind experiment, where the suppliers provide the E-cat, sealed if need be. The testers will randomly supply the E-cat with either hydrogen gas (H2) or deuterium gas (D2), or deuterium hydride (DH). The magnitude of the measurable difference should suggest the likelihood of the process being nuclear.</comment>


Texas leads the US in wind power – and now it's ramping up solar, too - <AUTHOR> https://electrek.co/2020/02/21/texas-leads-the-us-in-wind-power-and-now-its-ramping-up-solar-too/ <comment><AUTHOR> TLDR Texas is adding 9GW of solar generation capacity in the next two years.

> Solar currently provides just 2.2% of energy to the Texas power grid. But > according to the state grid manager, the Electric Reliability Council of > Texas (ERCOT), solar developers are expected to add 3.5GW of capacity from > new utility-scale solar projects this year and 5.5GW in 2021 (via the > Houston Chronicle). One gigawatt can power around 700,000 homes.

> That includes projects that have signed interconnection agreements with > ERCOT, which means they have a high likelihood of completion. ERCOT > represents 90% of the state's electric load.

> Most of the planned solar projects are expected to come online before summer > 2021.</comment>


I.M.F. Hit by Sophisticated Cyberattack - <AUTHOR> http://www.nytimes.com/2011/06/12/world/12imf.html?hp <comment><USER2> > officials declined to say where they believe the attack originated

Which means they know. Which means it's China since if it was some backwater country they'd just go ahead and say it, and if it was a western one they'd assume it was teenagers. But coming from China, it is the government doing it, so that's the only country that's taboo to name after being hacked.

<comment><USER3> This is a very strong accusation based on that quote ...

<comment><USER4> Based solely on that quote, yes. Taking into account other indicators, such as the purported origin of other recently reported 'sophisticated attacks' on high profile targets, it doesn't seem like a stretch. To say China or their government is _definitely_ involved isn't reasonable, but to say they'd be a strong suspect is reasonable.

</comment></comment></comment><comment><USER5> >> communications with national leaders as they >> negotiate, often behind the scenes, ...

Looking forward to reading the stuff on Wikileaks.

Only absolutely necessary things should be classified. Not every time a goverment f*cks up, the should just print a "secret" onto it to get away with things.

Its not a democracy if we don't have enough information to make our election choices.

<comment><USER6> I respectfully disagree.

There are a lot of things, especially when you're dealing with market sensitive information, that's best done behind the scenes. If, for instance, the Prime Minister of Spain calls the IMF and asks what kind of deal they would be able to make in a bailout of the country it's in all parties best interest to keep it secret. If this hypothetical phonecall was public everyone in the financial markets would short Spain, thus making it a selffulfilling prophecy.

I'm fairly certain that a contingency plan exists if Greece defaults. I'm also fairly certain that there's a report somewhere in the IMF stating that the chances of a Greek bankruptcy are greater than 30 or 40%. If these plans were to be made public Greece would default within a week, and wreak as much if not more havoc on the world economy than Lehmann brothers.

Some things are best negotiated out of the public view.

In a few years these records should be made public though.

<comment><USER7> That's the problem with secrets. Who knows what is best kept from the public and what is being hidden at a cost to them?

<comment><USER8> When the problem with secrets is the reversed equivalent of the problem with no secrets, perhaps you should stop thinking of it as a problem and start considering it a fundamental limitation of the ability to withhold information.

</comment></comment></comment></comment><comment><USER9> No matter how sophisticated or amateurish this attack is, or was, you can be sure that it will be used by politicians as ammunition to help pass laws which most people reading this will probably disagree with.

</comment><comment><USER10> Since when did phishing count as a sophisticated cyber attack?

<comment><USER11> They actually used the phrase "spear phishing", but then failed to describe it properly. Spear phishing is much more targeted, sometimes as finely honed to one or two people.

Of course, the rhetoric is always hyped. I've seen over the years where individual connections in a port scan was counted as a separate attack. Without details, this could be similarly "simple".

</comment></comment><comment><USER12> Thank you, Anonymous. Somebody has to take on those evil bastards.

<comment><USER13> Where in the article did it say anything about anon?</comment></comment>


Ask HN: What percentage to expect for salary increase between jobs? - <AUTHOR> Let's assume that cost-of-living and other factors are mostly ignored. If I'm fielding new offers in the tech industry (software engineering), what percent increase compared to my current salary should I reasonably expect? <comment><USER2> You can get salary expectations per titles from places like Stack Overflow and Glassdoor.

Most hiring managers shouldn't offer "X% over your current salary" \-- and you shouldn't look for that either. You should look at your experience, your capabilities and compare that with market rates for those.

If your current employer is paying you 20% below market rate, then this is an opportunity for you to reset to (or above) the market rate. But in this example, if you were to use your current salary as an input to determining your new one, you'd probably still lose out.

The trickiest bit is mapping titles among different companies. The public salary data is usually indexed by title. You can probably come up with good estimates for which title you think makes the best sense for you.

<comment><AUTHOR> This is great advice, thanks!

My question stems from the fact that I make about median market rate for my experience and location. I'm having trouble making a plan for how my salary should increase as my experience and career evolve. If I'm currently making median market rate, I'm unsure if it's reasonable to expect a jump into the 75th percentile for my next position during the negotiation process.

</comment></comment><comment><USER3> In India, you are required to mention your current salary and the expected salary on the job application to even be considered.

The expected salary is usually a 30% bump if you are between junior-mid levels.

</comment><comment><USER4> It will depend on your location, job title, the company, and your education background.

The good news is that you can plug all of that into a salary calculator on Glassdoor or Indeed.

I've also found talking to recruiters helpful in this one case. They have a good idea of what salary you can command on the job market based on your skills and work experience.

These articles I wrote walk you through finding what salary you can get and negotiating during the interview process:

[http://www.climbuptheladder.com/how-to-know-if-youre-paid- wh...](http://www.climbuptheladder.com/how-to-know-if-youre-paid-what-youre- worth/)

[http://www.climbuptheladder.com/they-asked-me-what- salary-i-...](http://www.climbuptheladder.com/they-asked-me-what-salary-i- want-in-an-interview-what-should-i-say/)

</comment><comment><USER5> You can get ANY salary increase % depending on the following:

\- You should do very well in the interview

\- It should be dependent on how much they need your skills

\- It should be dependent on how much they can pay for a role like yours, or one notch above [Please take Glassdoor data with a pinch of salt - some of the data is averaged out, or outdated]

Negotiation skills are key.

</comment><comment><USER6> One thing that (most) people ignore is the risk involved in switching jobs.

Let's say you make 100K today if someone offers you 120K that's a 20% increase. BUT, what happens if you get fired or quit after 2 months. How long will it take you to find another job and how likely are you to match that 120K or 100K?

If the answer to that question is not "a week", the job move should make sure this risk is smoothed out. Usually but a signing bonus on top of the pay increase or something like that.

Not saying that every job should have a signing bonus or that you should hard require it, just saying that pay increase is not the only thing to look for.

</comment><comment><USER7> Honestly I don't see how anyone can answer this without knowing what you are already making and what your local market is offering (assuming you also want to stay in the same place with the same title). If you're underpaid (e.g. making $65k in the Bay) you could more than double your salary quite easily. However if you're already making $200k as a senior engineer in say Seattle or Austin, well I would expect anywhere from 0 to a decrease, with a very rare possibility of an increase at all.

<comment><AUTHOR> That's fair. I expanded on the details further in another comment, which I'll copy below.

> My question stems from the fact that I make about median market rate for my > experience and location. I'm having trouble making a plan for how my salary > should increase as my experience and career evolve. If I'm currently making > median market rate, I'm unsure if it's reasonable to expect a jump into the > 75th percentile for my next position during the negotiation process.

Median market rate in my location (Pittsburgh) for my skills is around $75k. The 90th percentile for someone of equal skill is about $100k. To clarify my question, using the details from your comment, what can I reasonably expect for the growth rate between jobs as I approach the 90th percentile of pay?

<comment><USER7> Thank you, given what you've just added I'd defer to other's comments about relying on negotiation skills. I wouldn't accept anything less than a 10% increase (mixed between equity and cash) unless the title bump was meaningful and could actually have an impact on your career growth (i.e. _not_ being VP of Engineering for a 1-person engineering team). If you're right in the middle, it shouldn't be hard to find salaries for 10% more.

<comment><AUTHOR> Thanks for the advice! I was initially concerned that negotiating for a 20-25% increase would make me unreasonable for my currently median-level skills. It seems that if 10% is a fair, but low-end, increase, then I can continue to target ~ 20%, so long as I practice negotiating.

<comment> le-mark It depends on the employer as well. For example we have local companies here with horrible reputations that sometimes pay well above market rate for developers.

Makes negotiating much easier, also more risk in working there, as another comment says.

</comment></comment></comment></comment></comment><comment> le-mark It depends on the local market. For example in my local market, with 5-10 years exp you max out at around $85k. The ceiling for me seems to be around $120k (15 years Java enterprise). I judge this on constantly interviewing the past few years.

The idea being, if you're at market rate for your skill set, you may not see much increase at all.</comment>


Prison Without Punishment - <AUTHOR> https://www.themarshallproject.org/2015/09/25/prison-without-punishment <comment><USER2> I violated 3 years felony probation late last year here in Florida and was picked up on the warrant on May 18th.

The original charges were possession charges from about 5 years ago, and I only had 6 months to go when I was violated for working out-of-county (I had to go where the contract was). It was a simple "technical" violation so I figured I would be ok.

I was not. I was sentenced to 15 months _in prison_ for some reason that no one could explain. I didn't even "score" prison time, yet my crap public defender just sacrificed me to the state prosecutors.

Thank goodness I had 13 months credit time served, so I only had to do like 51 days "up the road". Florida D.O.C decided to make it hard on me for whatever reason, and sent me back to Gulf C.I. Annex in the panhandle, some 9 hours from my home.

I was first assigned to a "T-dorm" where approx 80% of my fellow inmates were serving life sentences. Yes, you heard correctly...they made me, with my little 51 days, live in that particularly dangerous environment.

Needless to say, I avoided talking about my release date.

There are absolutely no programs at Gulf C.I. I am 100% convinced that the entire system is designed to _create_ recidivism, not prevent it. The C.O's smuggled in K2 and it seemed everyone was smoking that shit constantly.

It was easier to get drugs then it was to get a book to read.

I was released on the 7th with no phone, money, and prison issued clothes. I had to stay in shelters for 2 nights until I finally made it back to central Florida.

I am in the process of writing a blog post about the entire incredible experience, but the tl;dr version of it is that the system is a sledge hammer and every problem is a block of concrete to be smashed to dust and swept away.

<comment><USER3> Just curious, possession of what? And how much?

<comment><USER2> Possession of heroin, Possession of over 20g of weed...both felonies.

You can read my previous posts about it. I'm just glad it's all finally over and I'm free to move and work as I need.

<comment><USER3> That's ridiculous. I hope we smarten up as a society soon.

</comment></comment></comment></comment><comment><USER4> German here, living in the US (and liking it here in the US).

I can confirm that generally there is a different viewpoint. In Germany prison is somewhat of the last resort mostly to be avoided, and even then the aspect of rehabilitation (and deterrence) is most important.

In the US I find there's often a notion of revenge, as in "This person must suffer for what for (s)he did!"; and more severe sentences usually to be considered as "more justice".

Edit: Some spelling for clarity

<comment><USER5> >In the US I find there's often a notion of revenge

US is puritans. Revenge is one of the cornerstones of the puritan moral.

<comment><USER6> That's as simplistic as saying Australia is a bunch of criminals.

The USA spans a huge continent which was settled by a huge number of very different groups all with different moral and philosophical foundations.

<comment><USER5> No. There is no national party in Australia with a base consisting mainly of criminals. Whereis the US Republican party's base display all the worst characteristics of puritanism, including their "holier than thou" main driving attitude which directly feeds their views on crime and punishment.

<comment><USER7> I can't speak to Republicans' ideas of crime and punishment, but I will say that the US is a place which encourages and fosters individualism strongly as opposed to collectivism (right term?). In a more collectivist / society focused model as you might find in Europe, it makes sense that the focus of prison would be rehabilitation and programs focused on improving the greater good. In a model emphasizing individualism, it makes sense that the focus of prison would be punishment: The offender made certain choices as an individual and has harmed the victim's ability to operate as an individual. The victim therefore has the right to seek justice.

<comment><USER5> while it sounds logical, the counter-example would be Russia. It has a very collectivist, hive, mentality - completely opposite to US, yet the same focus on punishment (in hellish prisons) instead of rehabilitation. The common though is the "you've sinned -> go to hell" religion. The countries like German are much more secular (as of today), and that naturally results in more humane society, including the criminal punishment policies.

It is also noticeable in social economical policies - the approach "you're poor because you're a lazy bum" can be seen as a form of punishment which is a natural view in the US (especially to the religious part of the society) while obviously less so for Germany. Note that such "punishment" is not explicitly inflicted by people, instead it viewed along the lines like "God loves hard working good people (and hates the opposite ones)".

</comment></comment></comment></comment></comment></comment><comment><USER8> Could it work? Of course, the data is abundantly clear here. If you treat prisoners like shit, then their chances of recidivism are much higher. If you want to reduce crime, then you have to treat prisoners well. There's really very little controversy here from an academic perspective.

The barrier is cultural. The public in the U.S., by and large, expects prisoners to be punished harshly. Retribution and deterrence rank way higher than rehabilitation in terms of the desired outcome of incarceration. As far as the average American is concerned, anything that happens inside of the prison walls, including but not limited to rape, murder, torture, is the price you pay for breaking the law.

If we want to have better prisons, then we'll need people to develop some degree of empathy for prisoners, and that's a tough battle in the U.S.

<comment><USER9> _If you want to reduce crime, then you have to treat prisoners well._

There are no prisoners without there first being criminals.

<comment><USER10> Nelson Mandela, Allan Turing, and Oscar Wilde might disagree with you on that.

To be fair on Mandela the modern US stance is that he should have been assassinated, rather than imprisoned.

[https://wikileaks.org/cia-hvt-counterinsurgency/press- releas...](https://wikileaks.org/cia-hvt-counterinsurgency/press-release.html)

<comment><USER11> Mandela was at war with the govt of SA, so that's a complicated case.

<comment><USER10> He was imprisoned for inciting workers' strikes and leaving the country without permission... it doesn't seem that complicated to me... His charges of overthrowing the government were so baseless they had to be thrown out.

</comment></comment></comment></comment></comment><comment><USER12> There are no absolutes when it comes to criminal punishment. There is always an exception to everything.

That said, I've long thought if you stop treating people like animals, they'll stop acting like it. I've never studied prisons, or psychology, or anything remotely related. But I genuinely believe this philosophy is investigating.

<comment><USER13> Well there's countries like Norways which make prison look like vacations to people in countries with less resources (except the not going out whenever you please of course)

Foucault argued that the point of prison is to create more crime though.

<comment><USER14> > Foucault argued that the point of prison is to create more crime though.

Add profits generated by the prison industrial complex and it actually starts making real economic sense to keep them coming back.

</comment></comment></comment><comment><USER15> Note that in the US, prison is a multibillion dollar industry. It cost around $30,000 (on the low end) to incarcerate someone yearly. Even if they spend a day or two, someone is making money. Nothing to be said of the calling minutes. All in all this may be a $50-$100B industry in the US.

That is not going anyway.

<comment><USER16> When I first read your comment it immediately brought to mind privately run prisons, but after a few moments though I realised you're comment calls attention to the ongoing wealth transfer from the general tax-payer to the prison associated workforce, who then go on to spend that money in the general economy. Prison generates economic activity, which generates tax revenue. Governments have a vested interest in increasing tax revenue.

<comment><USER17> Digging holes generate tax revenue as well.

<comment><USER16> That's what the highway department is for.

</comment></comment></comment></comment><comment><USER18> We have pretty relaxed prison system in Finland as well and it seems to be working pretty well[1]. They doesn't have the stigma as it does here the US and actually my sister lives next to a closed prison with her small kids and it's a no biggie.

[1] [http://www.theatlantic.com/international/archive/2013/09/why...](http://www.theatlantic.com/international/archive/2013/09/why- scandinavian-prisons-are-superior/279949/)

<comment><USER19> Difficult to compare the two, when Finland has a very homogeneous culture, and is tiny.

<comment><USER20> I often see this given as an excuse for the U.S.'s prison system, but it seems a sort of non-sequitur along the lines of "difficult to compare the two, because Finland is named Finland".

Is it just a codeword for the sentiment that the U.S. has black people, and they're inherently evil, so cannot be rehabilitated or treated nicely like Finland? That seems to be the only explanation that fits. If that's what you think, why not say it?

If that isn't what you meant, maybe you can explain further. For instance, you could try to finish these sentences:

"The USA is a larger, richer, country, and therefore it cannot build a prison facility that holds 300 people and is rehabilitative rather than punitive because......"

or for homogeneity:

"The USA has a somewhat more diverse population than Finland, and therefore it cannot build a prison facility that holds 300 people and is rehabilitative rather than punitive because......"

I'm curious to see what the explanation is. It certainly isn't anything obvious to me.

</comment></comment></comment><comment><USER21> In prison (US federal), when a guard was a dick to inmates, our saying was "We're here AS punishment, not FOR punishment".

</comment><comment><USER22> The title derives from a single quote of someone that claims prisoners in Germany are not put in prison as punishment, but only to protect the public. But that is simply not true: punishment most certainly is a goal of accusers and most prisoners certainly feel punished by being locked up, so that goal is at least partially achieved. The major difference is that the government does not consider it the main goal and other goals, such as rehabilitation, are given more weight than in the US.

<comment><USER23> The claim is entirely true (§2 StVollzG [1]), but you may be misunderstanding something here. The rehabilitation of offenders and the protection of the public are indeed the sole two permissible objectives of incarceration in Germany. But there is a difference between (1) the imposition of prison sentences and (2) how they are being carried out. Judges may take other factors into account to determine the severity of a sentence, but once imposed, rehabilitation of offenders and protection of the public are the only two goals that the process of carrying out the sentence may serve.

[1] [http://www.gesetze-im- internet.de/englisch_stvollzg/englisch...](http://www.gesetze-im- internet.de/englisch_stvollzg/englisch_stvollzg.html#p0013) (and yes, the translation of the section title is ... imperfect).

<comment><USER22> Punishment is not an official goal of the government. Nevertheless, in practice it certainly is a goal, as required by the people the government represents.

Punishment is a goal of many victims and of people who believe punishment discourages crime. When the perpetrator is brought to justice, the victims- turned-accusers will demand punishment. It is the goal of the government to represent their interests. It partially does this by setting up the prison system as it is and imposing prison sentences.

If dealing out punishment, as wished by the victims, was not an indirect goal contributing to the way things are, prisons would be more comfortable. The circumstances in prisons, their austerity and severity, cannot be explained except by wanting to punish the people in there. They are not optimally suited for only rehabilitation and protection of the public.

This is of course entirely understandable.

</comment></comment></comment><comment><USER24> I think the treatment of prisoners in the US is a big part of the problem. European countries don't force prisoners to wear orange prison uniforms, or transport them in waist chains and leg irons.

Sometimes respect and dignity for the prisoners by the prison authorities can go a long way towards reforming the prisoner.

</comment><comment><USER25> Sounds like a good idea to me. In America, we do too much imprisonment, and not enough rehabilitation. I understand has a lot to do with costs and politics, but I really think we should stop warehousing people for years and years, with no rehabilitation.

</comment><comment><USER26> In the Federal system inmates work towards something similar through their behavior. Look up Federal Prison Camps.

<comment><USER27> As someone who was just visiting a prison today, I can assure you that prison camps are _nothing_ like this. Just a glorified version of home ankle monitoring. No psychologically training. No saving money.

Read the description by the CEO of y-combinator's Pigeonly. They look for any possible infraction to send you back to prison. He had to illegally keep a cell phone in order to found his business.</comment></comment>


EFF on the Lodsys patent claims - <AUTHOR> https://www.eff.org/deeplinks/2011/06/apple-steps-lodsys-litigation

<comment><USER2> It's fascinating how our legal system, and patent law, have apparently created what is little more than an (apparently) legal extortion scheme.

IV Purchases all these (typically low-quality, frequently obvious, very seldom "Advancing the Arts" class) patents, and then approaches companies to license their various patent portfolios.

The brilliant part of all this, is that if you don't license directly from IV, instead of IV suing you for patent infringement, they _sell_ the patent to a third party, who then attempts to license them to you, or, if they are unsuccessful, sues you.

It feels an awful lot like "Buy some of my insurance, or something bad might happen to your warehouse." And it's all totally legal!

What's strange here, is that Apple (A "Mall") purchased the License (some "Insurance") - and now the people who sold that insurance are going around burning down stores inside that mall, after the little storekeepers didn't buy "insurance."

Apple is going to make the claim that because they bought the "Insurance" that all of the store inside should be protected as well.

And, at no part in this conversation, is Apple even really challenging whether it should be having to purchase "Insurance" - because they do the same thing to other companies, so they have some interest in protecting the status quo.

Now, yes, I realize that there are useful patents, particularly in the drug industry, where it may take 100s of millions of dollars and many years to invent something, and it's almost certain that most of those drugs would never, could never have been invented without patent protection (or massive government subsidies).

In my view of the world, I thought that Patents were supposed to be about "Advancing the arts" - not lining the pockets of the already very rich, to make them even richer.

I have to wonder when the this system will (finally) be overthrown and replaced with something half sane.

<comment><USER3> I'm not a fan of software patents, but one of those linked in the article is the first software patent that gave me pause (patent 7620565). Most software patents I read consist of "apply <canned algorithm x> to <situation y>". This one doesn't read like that. I think it's very important that this patent is pure fluff - it does not actually keep you from producing a good product in any given field.

The patent was filed in 2006. From my recollection, I can't think of any websites or devices that used this sort of feature before 2006. Today it seems fairly common, but in tech people are so used to blindly copying features (because its so easy to do after someone else has already done all the research) that I can't rightly say this innovation didn't originate with the inventor. All the devices used by potential infringers involving this patent existed for years prior to the filing of this patent, which to me says its not as easy to dismiss as obvious.

The patent was also created by someone who seems to be focused on inventing things. People rail against software trolls because they don't produce an actual product, but if the inventor could not license or sell this patent, this fairly common feature might not exist today.

Because of all the above, this isn't a patent I would point to as an example of why we need to abolish software patents. That said, I'm still not a fan of software patents, and think abolishing them would be a net good.

<comment><USER4> _if the inventor could not license or sell this patent, this fairly common feature might not exist today._

Bzzzzt, I strongly disagree with this supposition. If the inventor approached Apple, was rebuffed, and then Apple copied the invention in the creation of its iOS ecosystem, I might agree with you. However, it appears that what has happened here is that the invention was independently created subsequent to the patent filing.

Which suggests to me that it would have been invented any ways and thus the ability to patent the idea has no bearing on whether the invention would or would not have been discovered.

p.s. By :"bzzzzt," I mean that I am "buzzing in," not making a rude noise intended as an insult!!!

<comment><USER3> True, assuming the first iPhone had this feature, when comparing the dates of the patent filing to the date of the first iPhone release it's likely that it was independently invented.

</comment></comment></comment></comment><comment><USER5> It's never discussed in these cases where the original inventor has gotten to. Apple licenses from Lodsys who bought from Intellectual Ventures who owns 30,000 patents, apparently none of which they have invented themselves.

Where are the original inventors?

What I have seen in industry is companies write contracts so that people who invent things automatically transfer ownership to the company, often even if the invention was made off hours and has nothing to do with the company, and in some cases when the invention was completed before or started after their term of employment.

Now the whole argument is that these patents protect the rights of inventors and give them an incentive to invent.

Yeah? Well let's see how much the inventor is getting paid out of these millions. I bet the amount the inventor, the dude that actually did the work, gets is absolutely nothing, or maybe he got a $1 bonus for the invention, which is a typical clause I've seen. $1 awarded for patents.

<comment><USER6> I'm not sure what you're looking for here. Actually when this story broke, there was an assumption that Lodsys was owned by Daniel Abelow the original inventor of the patents. This was proven to be incorrect. Abelow had sold the patents to IV (I think) which were later purchased by Lodsys. Once he sold the patent rights he doesn't have much to do with anything. I can't find the article now but some news or blog outlet tracked him down and I believe the statement they got was along the lines of "I sold the rights to those patents a while back, and I've got nothing to do with this anymore"

His website is here <http://www.abelow.com/>

<comment><USER5> Thanks, I also found Dan's other site here: <http://computing2.com/>

So in this case the inventor did make some money off it at some point.

Abelow has a bunch of other patents that he seems to have never used himself but just sells to these various interlinked companies that then sue people. Another patent, 5251294 covers anyone using rollover menus in javascript. If you've got those and aren't paying WebVention a license fee (which is a shell company of IntellectualVentures, the company that sold to Lodsys) you might be in trouble, well except it seems to have expired in 2010, but they had a good ride up to then.

discussion: [http://stackoverflow.com/questions/2609936/javascript-css- ro...](http://stackoverflow.com/questions/2609936/javascript-css-rollover- menus-are-patented-and-subject-to-licensing)

analysis: [http://www.law.com/jsp/lawtechnologynews/PubArticleLTN.jsp?i...](http://www.law.com/jsp/lawtechnologynews/PubArticleLTN.jsp?id=1202436141846&slreturn=1&hbxlogin=1)

litigation: [http://news.priorsmart.com/webvention-v-abercrombie-fitch- lX...](http://news.priorsmart.com/webvention-v-abercrombie-fitch-lX5/)

<comment><USER6> Agreed I have to say its not to say that Abelow is not part of the problem, from looking at his site I have to say he's part of the systemic patent troll system. From what I can tell he just ideates and then patents without ever building anything.

</comment></comment></comment></comment><comment><USER7> Typo, if any EFF people are here:

> (...) _in many instances, will come out to than the cost of defending a > lawsuit._

I assume it should read "will come out to be more than the cost".

<comment><USER8> Less than. The troll would prefer to not litigate, but that you just hand over the money.

<comment><USER7> Er, right, I had a bit of a mental lapse there.</comment></comment></comment>


Top bugs of C++ projects found in 2018 - <AUTHOR> https://habr.com/en/company/pvs-studio/blog/444568/ <comment><USER2> I don't know why this website has copied the blog text from another site, but if you want to read the article on the original site, go to [https://www.viva64.com/en/b/0619/](https://www.viva64.com/en/b/0619/)

<comment><AUTHOR> Thanks, I cannot update the link anylonger.</comment></comment>


What makes a great mobile developer? - <AUTHOR> http://blog.hackerearth.com/2014/01/great-mobile-developer.html

<comment><USER2> I think the last point is the most important:

6) Know the need. Don't build things which users won't find useful.

However all points (except maybe 5) apply to all forms of development, not explicitly for mobile development.

Can anybody offer additional points that only apply to mobile development?

<comment><AUTHOR> Hi,

Thanks for your feedback. I think we've also reached a point where mobile development isn't new anymore, and as platforms mature, it's looking like the essentials are the same as the other.

On a broad level, I think being mindful of design (UX actually), battery and data are the main differentiating factors from mainstream development, which I think we've covered here.

Look forward to other differentiators too :)</comment></comment>


Is try..catch really useful? - <AUTHOR> http://robertjwozniak.com/article/is-try-catch-useful <comment><USER2> I feel like this assumes that the errors that might be thrown by a section of code are knowable. For some high level projects with poorly documented dependencies interacting with third party services that are in constant flux it's nearly impossible to know everything that's going to happen and try..catch allows you to program defensibly when an assumption the programmer made is invalidated.

</comment><comment><USER3> I ended up making an account for this one. Exceptions are a section in one of my favourite blog entries:

[http://blog.ploeh.dk/2015/04/13/less-is-more-language- featur...](http://blog.ploeh.dk/2015/04/13/less-is-more-language-features/)

Their argument is that Exceptions are GOTOs in disguise and point to the use of sum types.

I have found this to be good advice when using Java 8+. Composing Optional<?>, Either<?, Exception>, or CompletableFuture<?> has provided sufficient tooling for handling failure scenarios.

I also find that Exceptions slow down my reasoning about code, because they are side effects - see functional programming.

<comment><USER4> The Try pattern is good but if you are using a whole bunch of libraries/existing code that doesn't use it already it's just going to be confusing to introduce it.

</comment></comment><comment><USER5> Should we have error handling at all ?

Software engineers: no, too much work to think about every possible error case

Sysadmins (looking at above statement): NOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO

Now in theory you could do error handling without try...catch. Golang especially espouses that vision. However, like C before it, in practice Go programs ignore pretty much every error case.

To that practical situation, I will say, try...catch is far, far superior. Why ? Because it terminates your application instead of running the program in a way that erases the database. As an additional bonus try...catch gives you exactly what happened, and the entire stacktrace that mostly specifies what all the parts of the application was doing (and if you know what you're doing in Java/C++, including all values of all internal data structures. In Python ... well a little bit less but still substantial information).

<comment><USER6> > try...catch is far, far superior. Why ? Because it terminates your > application

Nope, that's try _without_ a catch. Every time you catch an exception you prevent that (though of course you can turn around and terminate anyway).

> try...catch gives you exactly what happened

Depends on the implementation. Python's is actually one of the best this way. For C++ what you say is simply not true. It's possible for an extremely diligent programmer to define exceptions that capture more useful state, but in practice almost no exception-intensive codebases provide as much information as in a typical error-return based system.

<comment><USER7> I find Java stack traces to be among the most readable. Exactly one line per stack frame, and just enough information to hunt the source file/library down the whole call stack.

Python is good but slightly too verbose because it needs to dump whole file paths.

</comment></comment></comment><comment><USER8> .then().catch() is _basically_ just a fancy wrapper around try..catch, so I'm not entirely sure what the example is supposed to show?

With async that code would be equivalent to something like:

      try {
        const response = await fetch(...);
        const data = await response.json();

        if (data.errors) {
          handleErrors(data.errors.code);
        } else {
          handleResponse(data.response);
        }
      } catch (error) {
        error; // handle error
      }

<comment><USER9> There's a difference: .then().catch() is worse because you can forget the .catch() part, and your error will be discarded unless you set up a global error handler (ugly because now the error will not bubble up properly). Not so with try ... catch.

<comment><USER10> Isn't forgetting to chain a .catch() just the equivalent of completely forgetting the 'try' block in async/await code though?

Seems like in both cases, the cause is the same: you weren't aware or forgot that the async logic might throw, and so you don't write code to handle it.

<comment><USER11> Sure, but the result is different. Missing out a try/catch will make your program crash/log an error/whatnot when the error happens. missing out a .catch() _might_ put an error in the console sometimes.

Fail fast is an important principle, but in my opinion fail hard is fairly important too. If your program explodes every time a bug happens you're more likely to fix them. If it merely doesn't do the thing it's supposed to it's easy to miss and easier to ignore.

<comment><USER10> What you say is true and I don't disagree with it, but we're talking about different levels of abstraction. Yes, at a lower level there is a technical difference, but if you zoom out far enough, the result is not different, i.e. in both cases the feature does not work.

I understand and agree with what you are saying about failing fast and hard being more convenient to catch bugs, as an engineer, but then we get into what is a bug and what can be ignored?

If the result is a mission critical feature breaking, you can't ignore that. Inversely, if you can ignore it, then it isn't mission critical.

So my point was, at a higher level of abstraction they _are_ the same thing. The cause is forgetting to handle an Error in your code, the effect is a feature breaking.

Regarding that cause then, the OP was saying that it's easier to forget chaining a .catch(), but that is only assuming that you'd actually remembered to use a try in your async/await code, i.e. you are aware of the fact it might throw. I can't think of many times where I have been aware something might throw and forgotten a .catch(), but for some reason would have remembered to wrap the call with a try, were it async/await code. I guess I am saying it's unclear to me how try/catch helps one remember to handle Errors, over .catch(). Seems like the syntax is not the issue, the issue is not being aware or ignoring that the code might throw in the first place.

Obviously, this is all dependent on context and application and both perspectives are 'right' for different scenarios.

<comment><USER11> They are not the same, not even at a higher level. The default action is different, ignoring errors by default is dangerous. Sure in many cases you'd like to catch the error and ignore it, but that has to be a conscious decision someone should have at some point thought about.

</comment></comment></comment></comment></comment></comment><comment><USER12> First of all, the way of handling the errors really shouldn't depend on the size of the project. Nobody is ever going to rewrite their error handling just because the project grew up.

Second, and likely more important fact is that the server returning the non-4xx status code is not quite an exception. It is just one step away from your happy-path. The mechanism of exceptions should be used for things that you don't expect. Mostly because of the cost they imply on the runtime (at least for C++ and similar languages which do stack unwinding and not so much for Swift, which treats the exceptions as just a different returned value with almost zero runtime cost).

</comment><comment><USER6> Every complaint about a goto is valid for try..catch, except it's worse because it's a _non local_ goto for which the destination is unknown at compile time. Ugh. Yesterday I had to deal with a piece of code that relied on a certain exception to do a particular kind of repair, but someone had added a _different_ exception twelve layers down (and across an RPC boundary). This was much harder to debug than it would have been with error returns, because nothing in those dozen intermediate layers had a chance to see and log the new exception. It just flew right by them, and when it arrived at its destination there was no information about its provenance. Thus, I had to go through a tedious process of instrumenting each layer to see how far it got before the flow of control was yanked away from it, which took hours instead of the minutes it would have taken to step through the same thing with error returns.

I know there are some techniques that could have made that experience better, but this kind of crap seems to happen _every time_ I have to interact with a codebase that relies on exceptions as a primary error-handling mechanism. That means practically all C++ and Java where it's strongly idiomatic, and some Python where it's more weakly so. Error returns might be more verbose, but they're ultimately better for maintainability.

<comment><USER13> From the CFG perspective, try..catch is strictly weaker than goto (you can't build irreducible control flow with it for example). And, I mean, any return is basically a non-local goto for which the destination is unknown at compile- time.

Your story seems strange since I think one of the best features of exceptions is that if you don't catch them, you always get a full stack trace which is a great boon for debugging. If you bubble up error returns, the only contextual information you get about where the error occurred is what every caller attaches to the return. But I have had terrible trouble getting stack traces (or types) from Python exceptions when they cross a process boundary so maybe that's it.

(I have no strong opinion about whether exceptions are good or bad myself.)

<comment><USER6> > you always get a full stack trace

You get a trace from where it was _caught_. Where was it _thrown_? Nobody knows. To be fair, that's the same as you'd get with error returns, but at least those dozen layers in between had a _chance_ to see and log the error. Or in a callback-based system you do get a trace from where the error originated.

The problem is that the act of throwing an exception destroys practically all information other than what you put in the exception yourself, and very few programmers implement exceptions in a way that provides as much information as is already available with error returns. Exceptions should be _exceptional_ \- rare events that don't get thrown or caught as part of normal operation. When every possible outcome of calling a function is reflected as a different exception, even if it's a fairly common case, that's when even simple control flows turn into big piles of random jumps.

<comment><USER13> No, you get a trace to where it was thrown.

<comment><USER14> In C++?

<comment><USER13> Yes? Is there a difference in terminology here? Obviously you get a trace to the point of the throw.

        ~ $ cat a.cxx
        int a() { throw "a"; }
        int b() { return a(); }
        int main() {
            b();
        }
        ~ $ clang++ -g a.cxx && gdb -ex r -ex bt a.out
        < ...snipped... >
        #6  0x0000555555554779 in a () at a.cxx:1
        #7  0x0000555555554789 in b () at a.cxx:2
        #8  0x000055555555479d in main () at a.cxx:4

<comment><USER14> There is no catch in your example. I thought the point was that in C++, preserving the stack trace when having caught an exception is rather non- trivial.

Exceptions that don't get caught usually allow you to get the stack trace through a crash report/core dump/whatever, but then there's not much difference between an uncaught exception and a plain abort().

<comment><USER13> Yes, I said "if you don't catch them you always get a full stack trace".

The comparison was between returning error values and exceptions, not between exceptions and abort. The difference is with abort is of course that once you are alerted to the error by seeing it be uncaught, you can insert an appropriate handler for it. A file-open function can throw on file not found but I will not very happy if it aborts.

<comment><USER14> Ah, after reading the full thread, that is actually what you said, and I don't really know what the point of the commenter you replied to was, now...

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> I would argue that reasoning about promise chains are difficult enough to where an abstraction is desirable.

If you asked me to identify the most maintainable and easy to read of these two pieces of code, I would say the answer is easy.

Promises:

      const p1, p2 = undefined;

      somePromise()
        .then((res) => {
          p1 = somePromise(res);
          return anotherPromiseFuncThatNeedsP1(p1);
        })
        .then((anotherPromiseRes) => {
          p2 = anotherPromiseRes;
          return p2;
        })
        .catch(e => {
          // log, throw etc.
        });
      // done

vs

try/catch:

      try {
        const p1 = await somePromise();
        const p2 = await anotherPromiseFuncThatNeedsP1(p1);
        // done
      } catch(e) {
        // log, throw, etc.
      }

<comment><USER16> Put another way, monadic error handling _really_ requires `do` notation.

<comment><USER15> In general, I would agree.

I'm sure there are cases where Promises are still superior, but I think our brains work easier in the x = y sort of notation.

</comment></comment></comment><comment><USER17> I generally am picky on my try/catch implementations... I use it way more often on the backend (PHP/laravel) than front. Though I will tap into then/catch on async functions which I guess is built-in try/catches.

On the backend I mostly do try/catch when I have a bunch of things tied together in a db transaction... If any part of the commit fails or anything errors at all the whole thing will rollback. Esp. handy when doing multiple relationship attaching in one go.

Also it comes in handy when dealing with API's and things, but for most scenarios it's not needed too often. But I also don't test every little thing, I believe moderation in all things including try, catch, testing. lol.

</comment><comment><USER18> Unfortunately that web page cannot be fixed via "reader" mode in Safari.

But as I read it I felt there was an implicit "yes, but" running through it. To step up a level:

\- unexpected problems occur \- you have to manage them. \- managing error return codes is tedious, and consumes a lot of lines of code, which is an opportunity for errors in and of itself.

Try/catch is a sign that something went wrong, but it allows you to manage sequence points (via catch) to deal with them. With unwind-protect (typically via automatic destructors in contemporary languages) most of the required boilerplate is automated. Yes, there are corner cases (e.g. the reason for C++'s make_shared) but there are _many_ more when you're tracking error returns.

Now where I disagree right up front is that try/catch is _not_ a way to "handle errors". It's a way to _deal with them_. We already have experience of using signalling to _handle_ errors: the Common Lisp condition system allowed you to catch an error and _restart_ the computation. It was a wonderful, general mechanism (even more general than just this case described here) and indeed, it was so general and so powerful that it also turned out to be an opportunity for incomprehensible code. We all learned from that, which is why nobody else has a facility for handling errors.

(BTW another great example of this kind of generality was method combinators. Sounds great but in practice: no thanks!)

</comment><comment><USER4> Well, I have to say I am completely unpersuaded.

</comment><comment><USER19> I think try catch is fine with some documentation or commenting around why the developer chose to do so instead of specific handling for all the different possible exceptions.

</comment><comment><USER20> try..catch means you don't have to have perfect knowledge of what errors may be raised - which you simply do not have when dealing with external code. It also means that you can guarantee that no exceptional case has occurred that you have not handled. Either you explicitly deal with the issue or you have the option to fail & log.

</comment><comment><USER21> Yes, error handling is really useful.</comment>


JQuery Gantt editor. Open source, SVG based. - <AUTHOR> http://roberto.open-lab.com/2014/04/10/jquery-gantt-editor-svg-and-critical-path/

<comment><USER2> This post was upvoted by sockpuppets. I banned the sockpuppets and buried this post. I've also banned the site, at least until we get a promise that this will never happen again.

Please don't ever do this on Hacker News.

</comment><comment><USER3> nice...if you are interested in making the view tree-based you might find this HN post useful: [https://news.ycombinator.com/item?id=7569063](https://news.ycombinator.com/item?id=7569063)

<comment><AUTHOR> that is really nice too :-)

</comment></comment><comment><USER4> Just testing it out. A great online tool. However, when you click on delete with no items, a JS error "Uncaught TypeError: Cannot read property 'canWrite' of undefined" is thrown. Cheers!

<comment><AUTHOR> Thanks for the feedback. Fixed!

</comment></comment><comment><USER5> I had to build one of these once in Flash, how did you handle leap year

<comment><AUTHOR> javascript Date object handle leap year by default

</comment></comment><comment><USER6> I'm not allowed to work on weekends? Thoughtful, alas.

<comment><AUTHOR> Yes, you can. It is configurable, see i18nJs.js file and more in detail the function isHoliday(date)

<comment><USER6> Ah, ok great, thanks. I was just using the demo as is.

</comment></comment></comment><comment><USER7> Great piece of code!!! Thanks

<comment><AUTHOR> welcome

</comment></comment><comment><USER8> Nice work. What kind of open source license?

<comment><AUTHOR> MIT

<comment><USER9> It's great that you chose this license, it is truly selfless to pick the MIT license. Kudos for that. More projects should be on MIT.</comment></comment></comment>


Ask HN: How do I find a few customers for validating a niche product? - <AUTHOR> YC teaches that startups should find a few customers in real life that really love your product. My ideal customer is someone who travels somewhat frequently overseas for business.<p>How/where can I find said people since I don't know anyone personally who would fit that profile? <comment><USER2> Go to your nearest airport and start talking to people?

I'm curious how you arrived at your idea if you don't know people who would actually use it.</comment>


Pilosa: An open source, distributed index - <AUTHOR> https://www.pilosa.com/docs/latest/data-model/ <comment><USER2> Technically, it is very interesting -- it uses Roaring Bitmaps under the hood and builds a query engine on it. So an easy way to think about it is that it maps categorical data into a giant compressible distributed bitmap.

I've been planning to see if I can (mis)use it as an OLAP replacement but I haven't had time to get to it.

<comment><USER3> You _definitely_ can... the feature set keeps growing. We have multi-field filtered GROUP BY now. It's amazing to see how flexible Roaring Bitmaps can be!

</comment></comment><comment><USER3> Source code:

[https://github.com/pilosa/pilosa](https://github.com/pilosa/pilosa)

</comment><comment><USER4> >Pilosa is a standalone index for big data. Its goal is to help big data storage solutions support real time, complex queries without resorting to pre- computation or approximation. Pilosa achieves this goal by implementing a distributed bitmap index which provides a compact representation not of the data itself, but of the relationships present in the data.

[https://www.pilosa.com/pdf/PILOSA%20-%20Technical%20White%20...](https://www.pilosa.com/pdf/PILOSA%20-%20Technical%20White%20Paper.pdf)

</comment><comment><USER5> Not sure who this document is aimed to. It's not technical enough to appeal to programmers that are working closely with Pilosa. And it's not written in a way to make it easy to understand for people that don't know anything about Pilosa (such as myself). I mean a subtitle called "Time Quantum" is enough to make me confused. Would appreciate a more generic "what is this" intro if possible.

<comment><USER6> I found the use cases section the most informative. You can click on a use case to get a write-up. Here's a few excerpts from transportation:

> Pilosa is a distributed bitmap index that sits on top of a data store. The > key to understanding and then using Pilosa is converting data such that it > is represented in ones and zeros. This dramatically reduces the size as well > as accelerates query times.

> For example, timestamps are important information, but we tend to be > interested in individual components of a timestamp, especially when > analyzing data with cyclic trends. Timestamp components are stored as groups > of bitmaps, known as "frames". We create one frame for the day of the week, > as illustrated in the following table. Along with similar frames for year, > month, and time of day, this accelerates queries that ask questions about > rides belonging to any logical combination of these time groups.

> [...]

> Because each data point includes pickup/dropoff times and total distance > travelled, it's easy to determine the average speed of the trip. As an > example, we use this as a first order approximation of congestion. We > created a frame representing average speed, with a spacing of 1 mph.

> In order to answer questions about congestion, we needed to first determine > what speeds constitute slow traffic. One of the basic queries in Pilosa is > the TopN function, and we used that to get a list of all the different > average speeds. By performing a count on each we built a histogram of how > many rides fall into each speed bucket, and decided from there which buckets > deviate enough from the norm to constitute congestion.

<comment><USER7> >converting data such that it is represented in ones and zeros

er, what? Isn't it all?

>This dramatically reduces the size

huh? There is no symbolic encoding less efficient for length than binary.

<comment><USER8> Well, you could decide that one axis is monkeyIndex, and the other is amountOfBananasOwned, and have a quite compact representation of which monkey owns what number of bananas.

I.e., decide on a symbolic meaning for the axes rather than converting data wholesale.

<comment><USER7> That doesn't sound compact at all! Every monkey's banana count uses a fixed number n of bits, where n = max(amountOfBananasOwned). That's horribly inefficient, when an ordinary binary counter uses n = log2(amountOfBananasOwned).

Which is not a criticism of Pilosa - I'm sure it's doing something very clever - I just don't understand what.

<comment><USER3> Bit-sliced indexing is the clever magic here. This post goes very deep on it [https://www.pilosa.com/blog/range-encoded- bitmaps/](https://www.pilosa.com/blog/range-encoded-bitmaps/)

But really, you use one bitmap for each binary bit of an integer, and it turns out you can generate arbitrary range queries on your dataset by doing various combinations of boolean operations on those bitmaps.

<comment><USER9> This page perpetrates an apparent contradiction. Starts off with:

> _Pilosa is built around the concept of representing everything as bitmaps._

What this literally means is that not a single datum is stored that is not in the bitmap.

But then the diagrammed examples look like this:

                    manatee loris sea_horse
      Vertebrate    1       1     1
      Invertebrate  0       0     0
      Breathes Air  1       1     0

Pardon my ignorance, but I see here a bitmap of 9 bits plus six character strings that are obviously not inside that bitmap.

If those strings are removed, the bitmap means jack squat.

Are they understood to be in another bitmap?

<comment><USER3> One does have to maintain some understanding of the how integer row and column ids are linked to what they actually represent.

Sometimes this is a function which might map (for example) row 3 to the letter 'd', 4 to 'e', and so on. Sometimes it has to be a lookup table which can be kept within Pilosa, or externally. Sometimes the IDs map directly to what they represent (day-of-month, year, passenger count, etc.)

So strictly speaking, not _everything_ is a bitmap, but the bulk of the heavy lifting in terms of serving queries is computation on bitmaps.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> How do you handle race conditions? E.g. an app updated the persistent store but crashed before it could update Pilosa?

<comment><USER3> Pilosa is best used in conjunction with something like Kafka with (e.g.) separate consumers for Pilosa and a persistent data store.

</comment></comment><comment><USER11> "Continuous Analysis on Really Big Data - Pilosa is an open source, distributed bitmap index" [https://www.pilosa.com/](https://www.pilosa.com/)

</comment><comment><USER12> Any chance there'll be some built-in support to perform collaborative filtering? Seems like a database of relations like this would be awesome for user-based collaborative filtering.

<comment><USER3> Great question! We have actually done some experiments with this in the past and will likely be rolling out features like this on top of Pilosa as part of Molecula [https://www.molecula.com/is-your-data-ai- ready/](https://www.molecula.com/is-your-data-ai-ready/)</comment></comment>


21 Surprising Facts About Digital Advertising - <AUTHOR> http://condorly.com/21-surprising-facts-digital-advertising/ <comment><USER2> > The future of advertising hinges on the combination of nuanced methods, > creativity, better data collection, and delivering a personalized > experience.

Curious, any agencies/advertisers doing this now?

</comment><comment><USER3> Interrupted by a huge pop-up after a few seconds of reading.</comment>