Ask HN: Has anyone used machine learning to trade? - <AUTHOR> I'm thinking of trading on either the FX or Stock markets using ML algorithms (primarily neural networks) on an intraday timescale. Does anyone have any experience in doing this? Any tips/stories/recommendations on brokers, data, etc? Is it a complete waste of time to try without being a big fish (I've only got about 10K to venture on this)? <comment><USER3> I don't know much about stock markets, but I did a consulting gig for a large currency trading company a while back where I was doing a bit of work on their algorithmic trading platform. I learned that unless you are trading at least a million dollars a trade it's very unlikely that you will make money.

The reason for this is that the large companies won't talk to you unless you're trading at those volumes since they charge pretty high transaction fees. The smaller companies that will talk to you all have accounts with the larger companies and make money by increasing the spread between the bid and ask price they receive from the large companies. The expected return on a trade is usually somewhere in the region of 1-10 pips (where a pip is one ten thousandth of a dollar), with 10 pips being a really good return and 3-4 being more realistic. The smaller companies usually increase the spread by up to 3 pips in each direction on the price they get from the larger company which swallows up your return.

While millions of dollars per trade might sound like a lot the leverage in this market is also very high at 50-100 times meaning that with your 10k you could trade 500k-1M though on a sum that low you might get less leverage. However if you lose money on a trade you could eat substantially into your capital. So if your $1M trade goes against you by 10 pips you'd lose 1k.

I'm not ruling out making money on that market, but I'd say it's very unlikely unless you have a lot more money to start with. I am not an expert in this area though. This is just what I learned from the small amount of time I spent working with the company.

</comment><comment><USER4> Yes, I did work on that. I used a neural network to trade FX. I wrote my own neutral network framework using code generators for x86 and CUDA.

The neural network was a fairly normal feed-forward network with GA-inspired, but heavily modified learning process. It combined layers of neurons with memory and memory-less neurons. It is essential to keep the network size small, because a big network will just memorize the data you use to train it.

Also, I used several other inputs, such as the frequency of trades per minute, overnight rates for each currency, and stock market indexes for the relevant countries.

Finding historic data was the biggest problem. I used Bloomberg data from my university, but that was a pain in so many ways. There is NO free data for most of the inputs I needed. Good data from Thomson Reuters or something like starts at $5k/mo.

Overall, the model worked fairly well, generating about 10% annual returns even with a few handicaps I put to be on the safe side. This was about an year ago and the only reason I stopped working on it was that my partner in the financial startup quit. I'm sorry if some of my terminology is wrong - <USER2> haven't been working in that field since then. If you have any questions, I'd be happy to talk about this. My email is in my profile.

</comment><comment><USER5> I have friends who do this, and they told me, anecdotally, that for every 10 trades they do, 8 or 9 lose money.

While they do make it up on the rest, you have to ask yourself if you can afford the costs (brokerage commissions and fees) of executing your strategy in that context (they all work for banks and hedge funds).

If you really want to get into this, you might be better off applying to a quant-based hedge fund.

A firm like that will have the capital and brokerage connections to make the game profitable.

Also, take a look at Ed Thorp's web site (<http://edwardothorp.com/index.html>).

He was one of the original quants and has many interesting articles on the industry.

<comment><USER6> I do this and I make money in 1 out 2 trades. Do you know what the heck you are talking about ?

<http://www.brettsteenbarger.com/articles.htm>

<comment><USER5> Half your momentum trades are profitable? Is that before or after trading costs? How often and at what volumes are you trading?

<comment><USER6> All I can say is that 8/10 are losers is highly improbable. Given no other information, if you enter a trade there is a 50/50 chance that the next tick is going to be up or down. But notice I said given no other information, there is much more information from the behavior of the charts. If you are a small player the charts will help you be right more than 50% of the time. The rest is how your algorithms and trading methodology take advantage of the information.

<comment><USER5> That is a gross oversimplification.

It also seems you've backed off your claim that 50% of _your_ trades are profitable (if you actually trade).

<comment><USER6> Here is something a little bit more complicated. 70-80 % win ratio. Please stop with the 8/10 losses misinformation.

<http://www.tradingtheodds.com/strategy-performance/>

</comment></comment></comment></comment></comment></comment><comment><USER7> In general, machine learning would probably not be a great approach to day trading. There are several problems:

1) Limited market data: As an individual, you have limited access to market data. At best, you probably have real time top of book data but without paying for it, you almost certainly aren't getting a real time depth feed. Going off of only last trade data gives you even less to work with. In most cases free quotes are delayed making them worthless to day traders (although not longer term investors).

2) Limitations of market data: Even if you have a lot of historical data with a full order book, that is still of limited utility for simulations. The problem is that everything you do affects the market so any trade you simulate needs to be doing small enough quantity to not make your simulation unreasonable. How to deal with simulating market-affecting trades is a complex problem and the only way to truly know is to run your trades in the market.

3) For day trading, there is a lot of profit to be captured simply from the microstructure of the market (its moves back and forth, etc.). To capture much of this it helps to be fast but you won't be able to colo boxes at an exchange with only 10K.

Machine learning seems more suited to longer term investments and trading on a slower platform than making a quick profit arbing or scalping. You may want to consider this rather than day trading since not only does day trading require much larger amounts of capital, but its also extremely risky.

Also, if you plan to trade equities, enjoy all of the ridiculous SEC regulation (order marking, etc.).

</comment><comment><USER8> _Is it a complete waste of time to try without being a big fish (I've only got about 10K to venture on this)?_

Yes. The intraday movement of stocks is small enough that you'd need to be using a large fraction of your capital on every single trade in order to overcome trading costs; and based on the Kelly criterion you'd need to have unrealistically high odds of betting correctly in order to avoid going broke.

<comment><AUTHOR> Do you think that interday would be more suitable then?

<comment><USER9> A strategy: run a LOT of simulations, keep running them every day. When they are too good to ignore, go raise some capital.

<comment><USER10> people run MC all the time, not really a new strategy -- expect no breakthrough on that...

<comment><USER9> I'm suggesting that you run all of your trades with play money, and see how it goes. I'm not talking about Monte Carlo simulations.

</comment></comment></comment></comment></comment><comment><USER11> DON'T. All money that can be pumped out of the market by standard ML algorithms has been pumped out by standard ML algorithms.

</comment><comment><USER12> haliax,

If you want more information on this, there was an IAMA on Reddit about 2-3 months ago, on a guy who claimed to do exactly this. It was incredibly informative, and went into a lot of the details that you ask for here.

[http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automat...](http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automated_independent_retail_trader_i/)

</comment><comment><USER13> My understanding is this is a pretty huge field : <http://en.wikipedia.org/wiki/Computational_finance> and that if you try to roll your own, you are computing against a variety of math and CS phds that big trading firms hire to do analytics.

</comment><comment><USER14> If you're interested in reading more, take a look at the message board <http://www.nuclearphynance.com>.

And $10K will almost certainly not be enough. You'll spend all your money paying commissions or bid/ask.

</comment><comment><USER15> Can't you just download a historical set and use that to test whatever algorithms you think you would use? Then when you are confident enough in their performance, you can take the risk? I wouldn't say it would be a _complete_ waste of time because you would be sure to learn something. If you think you can realistically make money--well then, that's a different question that I wouldn't be fit to answer.

<comment><USER16> > Can't you just download a historical set and use that to test whatever > algorithms you think you would use?

Be careful - much of the historical data has been pruned to eliminate some data from companies that didn't survive.

I've heard that one of the major quant houses (Shaw?) can often guess what data set you used to test your strategy by looking at the strategy.

Plus, there's a selection or fitting problem. Your strategy may have worked well on average from 1972-1999, but that doesn't mean that it will work today.

And there's a velocity problem. There are pricing errors in old data to exploit. And, if you had today's computers then, you might have been able to find them fast enough because your systems are faster than what folks had then. However, today you're competing with folks who have faster systems than you do.

<comment><USER15> You make some good points, but isn't this always going to be a problem when training your algorithms? You can't really train in real time so by the nature of the industry you're always going to be put at a disadvantage by using older data.

</comment></comment></comment><comment><USER17> The only tip is: do it on paper first, with the exact setup, lag times, etc. that you're going to have when doing it live. Do it for a _while_.

And realize that this is a fanciful pursuit which is likely to lose you money, because the guys you're going up against have $ millions, ns-scale lag times, free or near-free transaction costs, and PhDs in theoretical physics.

</comment><comment><USER6> 10K is too little. Statistically speaking, a small draw down can wipe out your acount. Most professionals have stop losses to limited their loss to 1-2% of their account. Google "trading money-management". You probably want to paper trade for a while before risking real money. If you are really interested in markets you can follow it as a hobby while working as a programmer. I used Interactive Brokers for trading futures, fx, options and stocks. They are low- cost and provides an API for automation. I write java code for my own personal "proprietary desk trading".

</comment><comment><USER18> Being a "small fish" is actually an advantage since you have less capital you need to put to work and thus can more easily get fills and catch the bigger waves generated by the bigger fish institutions like mutual and hedge funds.

The institutions have to hold on for longer periods and can't really "scalp" trades per se given that the bid/ask volume would collapse if they didn't distribute and "average in" slowly overtime to accumulate a position vs. buying or dumping their entire position all at once.

<comment><USER19> Being a small fish means you get killed by brokerage fees, which don't get cheap until the value of your trades is in the millions.

<comment><USER18> True. Each side has their own set of advantages/disadvantages. And so you'd have to adapt and take that into consideration in addition to bid/ask spread. It's not impossible (nothing is impossible), and you'd only get killed by brokerage fees if you're overtrading.

If you can find an algorithm that can trade only a few times a day and ride stocks that has a volatility of 1% or more then even at retail commission levels you can still do pretty well as you slowly build up your account to trade strategies that require more churn.

</comment></comment></comment><comment><USER10> intraday NN is irrelevant because NN are prone to overlearning, so the stuff becomes even more horrible - actually in reality you don't care which way it moves, as long as it moves: this is why the "big fish". It is like owning the place in Vegas: you don't care about the bets (much less about their substance or "direction"), as long as there are bets.

if you really want intraday, think about news.google.com and <http://rs2007.limsi.fr/index.php/TLP:Page_12> to automatically tweak the few k base parameters (SVM, associative memory etc).

<http://www.bbc.co.uk/science/horizon/1999/midas.shtml>

Options, Futures and Other Derivatives, J. C. Hull, Prentice Hall Int'l

Financial Calculus, M. Baxter and A. Rennie, Cambridge University Press

but the ML part is almost irrelevant: it is not the lever in the leverage (think CDO trade).

<comment><USER20> _intraday NN is irrelevant because NN are prone to overlearning_

I call bullshit. A lot of the most successful applications in machine learning are modeled using neural networks.

To regularize a neural network, you use standard machinery that controls the capacity of the machine. The scale of the weights cannot be too large (this is a famous result from IIRC Haussler in 1992) so penalize the weights using an l2 (Gaussian) prior. This reduces the model complexity and the potential for overfitting.

</comment></comment><comment><USER21> You might want to start our by reading "Drunkards Walk", and then walk away and find something a bit less chaotic to throw your money at.

</comment><comment><USER22> How much do you know about trading now? If you know little, I would recommend the textbook Trading & Exchanges by Larry Harris. It's somewhat out of date but still very good. I work at a high-frequency trading shop and that is the first book they shove at you when you walk in the door.

Other than that, I'd like to second what several other people have already said: getting a job at a firm is a good way to learn, you're outgunned in terms of brains and money, and 10k is probably too little. Now, you can do a lot of learning for 0k while you save money, and every startup is outgunned by goog, msft, but that doesn't mean there isn't room in the ecosystem. Like a startup against an established player, you could find a niche that is going unexploited. What makes trading harder is the established players don't tell you much about what they're doing, so you don't know if you are doing something new or walking into a trading strategy perfected 10 years ago by DE Shaw. With a software startup, you have more information about what your competitors are selling, what their products are, how you might differentiate yourself, what's missing from the internet, etc. Software companies promote their products and try to get public attention; trading shops try to keep as many secrets as possible.

</comment><comment><USER23> The majority of people who have responded to you with various pieces of opinion/information forgot to answer this question:

 _Does anyone have any experience in doing this?_

So take everything with a grain of salt-- there seems to be a lot of advice going around by people who don't have skin in the game-- especially those who say it can't be done. Unless they've put money down to test this, then their opinion should be discounted.

For those saying commissions are to high for a shorter term timeframe: trading has been commoditized. Lightspeed offers .40 per lot, and IBKR offers around that given a certain amount of trading volume. Forex brokers make money off the spread, and so the commission that you pay is only from the bid ask. Since you are trading small amounts, you will get hit on the comission structure, but if you pay per share rather than per trade it won't hurt as bad. You also might want to look into equity futures as well-- very liquid, good tax advantages, and good commission structures.

Biggest thing about small sizes is that you're subject to the pattern daytrader rule-- you can't make >3 intraday trades if your portfolio is <25k. SEC rules.

I'm a discretionary trader, and not a system trader. I am currently developing some systems to go live but they are not ready as of yet.

I also know of many who are quants, and it's possible and profitable.

</comment><comment><USER24> You might find it interesting to look at this site, which gives you a lot of the infrastructure for testing your FX strategies with real time data (although not direct access to the data).

<http://preview.collective2.com/>

Strategies are independently monitored and ranked on the site, and can be switched from simulations to real accounts when you're ready. If your strategy performs well, it will be noticed and you can get paid commissions on it (potentially helpful for raising capital).

For FX and equity data, you might as well use Yahoo finance. I run a Slicehost server that scrapes it once every minute with no problem. It's not ultra high frequency but you can't hope to compete in that space anyway, as others have pointed out. You should start building a historical database now before you do anything else, so that you'll have something ready when you need it. You probably don't need years and years of it.

I worked as a quant for a while and would support the other comments that standard ML techniques are unlikely to outperform, but you might be the exception.

</comment><comment><USER25> I just want to add that I read some of the Turtle Trader books and one more on Trend Following. The last book finally got across to me the point that the trend followers used computers and math not to try and predict where the markets were headed but rather to try and classify markets as rising, falling, or side to side. Their theory is that once a market has a direction it is likely to keep that direction - until it changes.

This is very different from most of the stuff I read about with neural nets. In most of the other books the idea to try and predict when markets are over sold or about to change so that you can be in on the ground floor.

I'm doing a webapp about card counting at black jack and the trend following approach seems a lot more like the reality of card counting. When the deck is hot or in your favor you bet more. Most people think that card counting is a super complicated exercise in combinatorics where the player is recalculating basic strategy in real time and able to prove that certain plays are the best.

Just something to consider.

</comment><comment><USER26> A strange game. The only way to win is not to play.

<comment><USER27> Reality begs to differ.

<comment><USER28> If by reality you mean Buffett or Soros, consider the thousands (hundreds of thousands?) of extremely smart and hardworking people who fail to replicate their success.

<comment><AUTHOR> By reality one could consider any of a number of small prop trading shops which do rather well.

</comment></comment></comment></comment><comment><USER29> If you're interested in this, why not join a firm that does it? They will have a large capital base to start with and eventually start your own venture. Unfortunately 10k won't get you very far in this game

<comment><USER30> Practice some with historical data, get a job with a firm that does this, and eventually strike out on your own. If you prove yourself, people will hand you millions to manage. Starting with your own money seems like a bit of a fools game.

</comment></comment><comment><USER31> One of Ray Kurzweil's companies, FatKat, does quant finance based on pattern recognition. <http://www.fatkat.com/overview.html>

</comment><comment><USER32> This is not what you are asking but a market with more inefficiencies and less transaction costs is Betfair. There is a simple api and free historical data.

I spent a bunch of time with Neural Nets and SVM and that data. Sadly I didn't find a profitable strategy. I would love to hear about anyone else who is looking in that area.

<comment><AUTHOR> What type of analysis did you look into? i.e. what were your input vectors / output?

<comment><USER32> I would generate a back or lay output signal for 'Correct score' bets for soccer matches before the game started. Input would be everything you can pull out of the the data set especially including the odds on other more standard bets like the match odds and the over / under odds.

There are also arb opportunities between the various arcane bets, but that involves speed more than ML.

<comment><USER33> [http://www.nytimes.com/2009/11/23/sports/soccer/23iht- SOCCER...](http://www.nytimes.com/2009/11/23/sports/soccer/23iht- SOCCER.html?ref=global-home)

</comment></comment></comment></comment><comment><USER34> You'll need at least 25k in order to be a day trader in stocks. You can get around that by trading futures. [http://www.finra.org/Investors/SmartInvesting/AdvancedInvest...](http://www.finra.org/Investors/SmartInvesting/AdvancedInvesting/DayTrading/P005906)

</comment><comment><USER35> If you know .NET, Do have a look at NinjaTrader and their forums. You will find quite a few folks who are in to automated trading.

I ventured into retail trading (manual) and lost around 5$. Email me if you are interested, I can give you more details (email address available in my profile).

</comment><comment><USER36> Noticing comments on the high degree of proficiency of the competition(or so is the perception.)

Anyone know of any particular markets where a small timer might have a better go at it due to insufficient volume to attract the bigger players, or for any other factors?

<comment><USER37> If anyone knew, they would be crazy to tell you.

Besides, the bigger players could easily afford to risk a few hundred grand finding out whether they can plug the existing system into the small market profitably.

</comment></comment><comment><USER38> UCSC Professor Helmbold used machine learning to discover optimal portfolio allocation: <http://users.soe.ucsc.edu/~dph/mypubs/OnLinePortfolio.pdf>

</comment><comment><USER39> of course, get MetaTrader4, open up a demo account on FXCM, Alpari, dbFX or whatever and write your own software and backtest it without losing any money. Boston Technologies has a free expert advisor, maybe you can build up on it - <http://www.bostontechnologies.com/expertadvisors.html> . I am not keen on automated trading though, since I do a bit of TA and FA, no software I know can do this - and where is the fun in it.

</comment><comment><USER40> Please read <http://en.wikipedia.org/wiki/Fooled_by_Randomness> before you take a plunge.

</comment><comment><USER36> On a related note, can anyone point to any writings/information on Kurzweil's work in this area?

</comment><comment><USER41> Don't bother.

D.E. Shaw, Renaissance Tech, Goldman Sachs, etc. have already hired hundreds (if not thousands) of people smarter than you to do exactly this. They have more Math, Physics, Econ, CS PhDs than you can count/

<comment><USER23> _They have more Math, Physics, Econ, CS PhDs than you can count/_

So did BSC, AIG, LEH, and MER

<comment><USER42> More to the point! The _big_ disadvantage ML methods is that they are just fancy statistics. The main faulty assumption here is that they believe the future looks a lot like the past, so they try to find even more complex patterns in historical data to extrapolate future trends slightly faster than everybody else.

In order for this approach to work, you need very precise market conditions. When markets are stable, your edge is zero and you would be better off investing in some sort hedged index (minimum effort, decent result). When markets are not that stable, it is when you can anticipate trends and get a decent profit for your effort. However, when markets suffer big unexpected turns is when fortunes change hands... and it is when your algorithms give you _negative_ advantage! While you are warrantied to keep following the numbers wherever they take you,a savvy finance human agent has a chance to smell the blood and discretely head for the exit before everyone else. So, it is your call guys. In my opinion, this will not be a viable approach until we get some type of semantic-web-ly mine-able news feeder.</comment></comment></comment>


Months 3 to 17 of my baby's sleep and breastfeeding schedule - <AUTHOR> https://www.reddit.com/r/dataisbeautiful/comments/6s0ba9/months_3_to_17_of_my_babys_sleep_and/ <comment><AUTHOR> I first saw this on this site: [https://flowingdata.com/2017/08/08/14-months- of-sleep-and-br...](https://flowingdata.com/2017/08/08/14-months-of-sleep-and- breast-feeding/)</comment>


Show HN: Alert me when word appears on HN - <AUTHOR> http://hnalert.tk/ <comment><USER2> You've got a bug. Your thing sent me the following email:

Output of command described as:

xyzzy

has changed.

COMMAND:

wget -qO- [https://news.ycombinator.com/](https://news.ycombinator.com/) | html2text | grep -i xanadu | wc -l

WORD DIFF:

{+0+}

DIFF:

0a1 > 0 \ No newline at end of file'

<comment><AUTHOR> That's not a bug, it's a 'feature'. But it's a good call to change it anyway :)

You get the email because the output differs from 'empty' (default) to 0 (it's not on the front page right now). I'll change the default output to 0, as the corner case is irrelevant.

Thanks!</comment></comment>


Erlang dev offers bounty to improve his networking code - <AUTHOR> http://erlanginside.com/netflix-watch-out-erlang-dev-offers-1000-bounty-to-improve-his-multicast-code-106

<comment><USER2> Link to the actual offer: [http://groups.google.com/group/erlang- programming/browse_thr...](http://groups.google.com/group/erlang- programming/browse_thread/thread/1931368998000836)

<comment><USER3> Please don't post to the Google group, though. Subscribe to the Erlang Questions mailing list at <http://www.erlang.org> instead.

Also, freel free to follow wagerlabs on Twitter, email me or come to #erlang on irc.freenode.net

<comment><USER3> The ultimate goal here is to broadcast to hundreds of thousands of clients with minimal latency and on a minimal number of EC2 instances.

</comment></comment></comment><comment><USER4> Speculative work?

I'll pass.</comment>


Y Combinator's non-profit mission - <AUTHOR> http://www.ft.com/intl/cms/s/0/72d90f7a-3c20-11e5-8613-07d16aad2152.html

<comment><USER2> Paywalled article (in the UK at least).

<comment><USER3> elsewhere too</comment></comment>


Ask HN: alternatives to Flippa? - <AUTHOR> I'm considering buying and/or selling web sites in the next few months.<p>I already know of flippa.com for that, but would like to know if you tried alternatives that would be less known or more interesting. <comment><USER3> I made a website whose sole purpose is to show you similar and alternative sites. This should act as a nice starting off point.

<http://www.moreofit.com/similar-to/flippa.com/>

Top results are: WebsiteBroker, SitePoint MarketPlace, AfterNIC, and DealASite.

<comment><USER4> Xmarks also has a similar site feature based on our bookmark data:

<http://www.xmarks.com/site/flippa.com>

It also categorizes sites so you can look up the top rated sites for an individual topic:

<http://www.xmarks.com/topic/web_sites_for_sale>

<comment><USER5> I think <http://www.similarsitesearch.com/> is the best search engine for finding similar websites. It's hard to distinguish the differences in search engines for finding similar sites for popular websites like flippa.com. I saw all of them returning the similar results except that similarsitesearch.com gives more results than others. similarsitesearch.com generates the best result for smaller websites like moreofit.com.

BTW I think moreofit.com has very good UI. Better than any other similar ones I have seen.

</comment></comment></comment><comment><USER6> The forums at digitalpoint are also known to have some decent volume - <USER2> necessarily high-end sites but sufficient for some small-scale website flipping. If you know what you are looking for, contacting website owners directly is a method I recommend as an alternative to online marketplaces.

Browsing through flippa and the likes is like searching for a needle in a haystack for me and only once in a while a few golden needles emerge.

<comment><AUTHOR> Thanks - this makes a lot of sense and match my impression when browsing flippa, as well :)

</comment></comment><comment><USER7> Here's what I don't really understand: if a website is actually, for real, bringing in decent passive income, WHY would you ever sell it? Result: only failed passive-income websites are for sale. So why not start from scratch then, especially with the risk involved (site may be banned by Google etc)

<comment><USER8> a. there's no such thing as "passive income" - there's always some effort.

b. even if there was such a thing as passive income, it has to produce a better ROI than alternative investments.

<comment><USER7> a. There is such a thing as passive income, sites where the _only_ effort you put in (for years) is counting the money.

<comment><USER8> I've yet to see an example.

EDIT (adding an appropriate - though ancient - quote and reference):

"Internet commerce appeals deceptives (sic) to a particularly male fantasy. Guys like the idea that after a short initial period of programming, a computer will tirelessly slave away for them, making them money 24 hours a day. Set up the site, walk away, and watch the money pile up in your bank account.

You can feed this fantasy by reading articles in the business press about <http://amazon.com>, the perennial poster children for Internet commerce. They set up what is essentially a front-end to a wholesale book distributor's database, and now they are selling books every few seconds. It sounds like they are rolling in money.

Well, it turns out that I know some people who work at amazon.com. The customers don't always fill out the forms exactly right. The books aren't always in stock like they should be. The customers send e-mail asking when their books are going to be shipped. So instead of one Unix box and a big vault for the cash, the company has 200 employees sucking all the money out of it. And remember, this is the best that anyone has really done: high expenses and high sales. More typical is an Internet store with high expenses and low or no sales."

<http://philip.greenspun.com/wtr/dead-trees/53002.htm>

<comment><USER9> The only example I have seen is plentyoffish.com. According to this: [http://www.inc.com/magazine/20090101/and-the-money-comes- rol...](http://www.inc.com/magazine/20090101/and-the-money-comes-rolling- in_pagen_5.html) the guy who owns this works barely an hour a day and earns more than 4M$ in revenue of it.

<comment><USER8> Having seen first hand the pathetic joke that is Inc's journalistic ability (and integrity), I doubt every word in that article.

They're just slightly above 419 scammers.

<comment><USER9> This was the link I was looking for when I posted that, a slightly more "authoritative" source. <http://www.nytimes.com/2008/01/13/business/13digi.html>

<comment><USER8> I wasn't really fair to Inc, having heard all the plentyoffish hype, I just don't buy it. And its still not actually passive.

I've been involved with a friend's site that pretty well fits the category. After some initial programming, SEO, etc, it did very well for 2+ years (> $100k revenue), but user support grew and grew, and then clones popped up, and started ranking ahead of him in Google results, then the revenue dropped to next to nothing. He could invest more time to get back to the revenue level he was at, but I just think that's the natural cycle for "passive" web sites.

Things go wrong, competition appears, users never quite fit the mold you want them to fit in, bad things happen externally...

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER5> I think <http://www.similarsitesearch.com/> is the best search engine for finding similar websites. It's hard to distinguish the differences in search engines for finding similar sites for popular websites like flippa.com. I saw all of them returning the similar results except that similarsitesearch.com gives more results than others. <http://www.similarsitesearch.com/> generates the best result for smaller websites like moreofit.com.

BTW I think moreofit.com has very good UI. Better than any other similar ones I have seen.

</comment><comment><USER10> WebsiteBroker.com is less known, but still gets decent traffic:

[http://siteanalytics.compete.com/websitebroker.com+flippa.co...](http://siteanalytics.compete.com/websitebroker.com+flippa.com/)

</comment><comment><USER11> It seems like it may be easier to find sites in the market that you'd be interested in buying and then contact the owner(s). Has anyone ever had any success in doing this?

</comment><comment><USER12> Surprisingly ebay's businesses for sale > websites for sale can be a decent place to search

</comment><comment><USER13> Can you tell us more about your plans? I've always wanted to buy a site for a passive income source but I'm always worried I'll get scammed.

Btw if you have a business that makes money from adsense can you deduct buying a new site as a business expense?

<comment><AUTHOR> I'm more on the buying side for the moment. The thing is our consultancy (we're 2) is quite beneficial this year and I expect (but might be wrong) we will earn less next year.

So I'm looking for sites I could maintain, promote and develop.

I understand your worries about getting scammed: I think it's a whole new job, with very specific points to check etc :)

I just asked my accountant to see if the buy could be deducted (at least here in France). I assume it will be doable.

<comment><USER13> Anyone know if it's deductible in the US?

Can you shed any light on the type of sites you're looking at? How do you evaluate them?

</comment></comment></comment><comment><USER14> Hello,

So you know, we have just started listing sites for sale at <http://marketplace.searchforecast.com/>

We typically have been dealing with sites that turnover more than $50,000 a month and most of these sales are done offline. We own and operate the CRS Conference (<http://www.crsconferences.com>) sponsored by Google, AOL, eBay which over 1000 publishers attend as well as many website buyers and sellers. Many of the transaction/negotiations happen at/around the Conference. In fact, our next conference is in Los Angeles in January 2011!

We also large interactive Nasdaq listed companies + other large networks actively buying domains. These large networks buy 20-30 websites a month. We know these firms personally and hence offer our clients premium access to these very reputable buyers.

Unlike many website listing websites, SearchForecast is a business which operates a large technical staff optimizing some of the largest websites in the world (BBC, Harrahs Casino, Deloitte). We also have over 20,000 visitors per month to our AdSense Directory at <http://www.searchforecast.com/adsense_directory_index.php> from all over the world which is why we are now listing websites for sale online.

We would certainly like to speak with you about your requirements and prefer that we post your site as a free listing on the Marketplace. We will waive the $29 listing fee by using the promo code 2740835 (we have the fee to keep away spammers!) Follow this link to <http://marketplace.searchforecast.com/selling- business.php>

We do take a commission upon completion of the transaction, which is 10% but we offer a unique "Sellers Guarantee" - you only pay us when you have been paid by the buyer. This gives our clients peace of mind. For larger clients that have over $1M, we have lower success fees.

We would be happy to match the sellers commission you have in place with other agents.

I would be happy to answer any questions and look forward to hearing from you,

Best Regards, ________________________________ Marc Phillips CEO, SearchForecast Telephone: 650.530.0895 Cell: 415.606.0900 Email: email@example.com Website: <http://www.searchforecast.com> Skype: marczphillips Twitter:@SearchForecast ________________________________

<comment><USER13> Thank you for your message. I am currently out of the office, with limited access to e-mail.

I will be returning on 9/28/2010.

If you need assistance before then, you may reach me at 301-912-9001.

Jon Bezos</comment></comment>


Show HN: Orchestra – Deploy machine learning models as APIs in minutes - <AUTHOR> https://orchestrahq.com <comment><AUTHOR> Partially scratching my own itch but fundamentally, we do believe that data scientists and data engineers have other valuable activities to handle. Building APIs and thinking about scaling it shouldn't be one of them.

Lots more features to come such as AB model testing, performance tracking, version control, testing, etc. but we're starting off with a simple model to API use case.

Currently, Python and R will be supported with the intention to introduce other data science friendly languages.

We're close to launching our initial solution and would like to extend 1 year free on any subscription plan to folks on HN.

Happy to answer any questions.</comment>


A Zero-Math Introduction to Markov Chain Monte Carlo Methods - <AUTHOR> https://medium.com/@benpshaver/a-zero-math-introduction-to-markov-chain-monte-carlo-methods-dcba889e0c50 <comment><USER2> I really wanted to like this article, because I came accross the term MCMC quite a bit, but never knew what it was exactly.

I'm slightly disappointed, though. Almost all of it is dedicated to introducing prerequisite terminology (prior, posterior, likelihood, markov chains) which probably a lot of readers will already be familiar with, and then the actual explanation of MCMC is just this:

> To begin, MCMC methods pick a random parameter value to consider. The > simulation will continue to generate random values (this is the Monte Carlo > part), but subject to some rule for determining what makes a good parameter > value. The trick is that, for a pair of parameter values, it is possible to > compute which is a better parameter value, by computing how likely each > value is to explain the data, given our prior beliefs. If a randomly > generated parameter value is better than the last one, it is added to the > chain of parameter values with a certain probability determined by how much > better it is (this is the Markov chain part).

"subject to some rule", "it is possible to"... I feel like this is really glossing over the actual explanation of how this works. Actually, I still have no idea why there is a Markov chain. What is the structure of the chain? Where does it come from and why can't we just sample the parameter without using a chain?

Anyway, I appreciated most of the article and it started out really promising. It would be great if the author could try to expand on that still-baffling part ;)

<comment><USER3> I humbly submit my article as an alternative. I clearly explain why there is no alternative, and give a concrete description of the markov chain. Though after the initial description of what problem MCMC is solving, I do dip into the math.

[https://jeremykun.com/2015/04/06/markov-chain-monte-carlo- wi...](https://jeremykun.com/2015/04/06/markov-chain-monte-carlo-without-all- the-bullshit/)

<comment><USER4> Your article is interesting. But I see a problem in your exposition, in MCMC we don't know the probabilities of individual events. For example let p_n proportional to 1/n^3. We know that 1/n^3 converges but we don't know the sum. The key factor here is that we can compute the transition probabilities without computing the sum of the series. In the continuous variable setting we can compute the transition probabilities without computing difficult integrals. I hope you can incorporate this key factor to improve the quality of your article.

<comment><USER3> I'm sorry but I can't make any sense of what you've written.

<comment><USER4> Let X be the random variable that takes values in natural numbers n>1 with p(X=k)=A/k^3 with constant A such that sum p(X=k)=1, you can't apply MCMC with ypur explanation since you don't have an oracle for p(x=k). To apply MCMC you only need an oracle for the quotient of probabilies, here the transition probabilities are k^3/j^3 and you don't need to compute the unknown constant A.

<comment><USER3> I don't think this distinction is key. You're talking about a detail of the Metropolis-Hastings algorithm, while I'm defining a problem that Metropolis- Hastings is one of many algorithms to solve. Sure, Metropolis-Hastings might not need an oracle, but providing an oracle doesn't make the sampling problem any easier, and the way I wrote it makes it clearer why the sampling problem is hard. "Avoiding a difficult integral" is an engineering detail.

<comment><USER4> On one hand, mathematically speaking, the problem of sampling a finite state system when the probabilities of each state are known is a trivial one, as you know very well since you have shown that method in your posts (computing cumulative probabilities ...) On the other side, your blog is about computation and math, and in the realm of computation the complexity of algorithms is a key factor. Mixing time and burning time of markov chains are also computationally important.

The curse of dimensionality is a cause of Monte Carlo methods to be used instead of other numerical methods in high dimensional problems.

You should communicate to your audience that the advances in stochastic sampling and its application to science has revolutionize the field not because a breathrough in maths but because great key insights that allow us to explore multidimensional problems effectively with the help of computers.

Another important factor is that MCMC is used extensively in Bayesian statistics, but to explain that main application of MCMC in your blog it is necessary to introduce concepts like Bayer's theorem, likelihood, posterior density function, and prior probabilities.

There are some blogs in which all those concepts are illustrated and code in R or python is provided. Perhaps MCMC requires more than one post to be adequately explained.

To sum up, I am a follower of your posts and I appreciate the effort you take to show the main ingredients and the python code. My only desire is for you to continue improving the content of your blog for us to follow and enjoy.

Merry Christmas to you and your family from a follower in Spain.

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> It's silly to say something like "A non mathematical introduction to this mathematical concept". What the author really means is that he's not using equations in his introduction. Far too many people think equations are what mathematics is solely about, and titles like this perpetuate that stereotype.

<comment><USER6> This post would have been better if it actually used equations. A few short equations are easier to understand that a bunch of lengthy paragraphs. In fact P(A|B) = P(B|A)P(A)/P(B) is compact and not hard to understand, but there are entire books written about it!

[https://www.goodreads.com/book/show/10672848-the-theory- that...](https://www.goodreads.com/book/show/10672848-the-theory-that-would- not-die)

<comment><USER7> I agree that equations should have been introduced, but I disagree that your example is not hard to understand, which is the point. I'd rather read about the concepts than look at esoteric equations.

</comment></comment></comment><comment><USER8> Why call it a "zero math introduction" to a math topic? Kind of twists my guts a bit to read that. Seems to suggest some of a wannabe or love-hate relationship with math that you don't want to know any math but you find this math topic interesting?

<comment><USER9> One of my gripes with the current education system is that it trains the pupil to simply regurgitate knowledge without ensuring that the pupil truly understands the topic - in this case MCMC methods.

Articles such as this one are a great resource to help what would be a mindless regurgitator actually understand what the whole point of an MCMC method is for. The 'why' to the 'what' essentially. And you don't need math to explain that.

<comment><USER10> > And you don't need math to explain that.

There's an argument that you need math to explain it because it is math -- just not what we're taught to think of as "math".

One of my gripes with the current education system is that it makes it hard to recognize when math appears in situations not explicitly involving numbers, equations, and matrices.

<comment><USER11> It reminds me of a text that Feynman wrote about when he gave classes in Brazil, the students knew all the theory and the math behind it, but they could not recognize the phenomenon when it happened in the real world.

So I guess that is not only in math, but in the entire corpus of knowledge, our system makes us prepared for grading tests, not for applying the knowledge.

<comment><USER12> He told a room full of students that a French curve (that extremely curvy shape used in technical drawing) has the remarkable property that the tangent of the lowest point is always horizontal.. and they all believed him.

<comment><USER13> Well it's true, isn't it? I think that point was that they were astounded that the curve had that feature because they didn't understand why it was always horizontal. They thought it was magic.

<comment><USER12> Uh the point seemed that they knew the theory (of differentiation, tangents, trig etc) but were absolutely helpless in practically applying it. Couldn't relate the theory they had learnt to what they knew of the world, at all. Didn't think 'But the lowest point of ANY curve is horizontal!'

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> Who exactly is the target audience for this? A lot of developers I've met who hated math had the skills for it. If you can master a new programming paradigm in a few weeks than you can do math, it's not the monster all theses writers make it out to be.

<comment><USER15> _> A lot of developers I've met who hated math had the skills for it. If you can master a new programming paradigm in a few weeks than you can do math, it's not the monster all theses writers make it out to be._

Let me start off by saying that, as someone whose research is applied mathematics, I agree with you that math does not need to be "scary" if the pedagogy is tailored to the right level and style.

That being said...I think you're being a bit optimistic (or cavalier?), and maybe swinging too far to the other end of the spectrum. I don't believe programming has much overlap with mathematics at all. It's very close to applied logic, but I think even then it's specifically much more like an engineering discipline than a mathematical one. At the higher levels of computer science (like complexity theory) I see a much broader overlap with mathematics, but you use very little of that in typical programming.

I think the skills that make someone a good programmer and the skills that make someone a good mathematician are essentially orthogonal: I've met mathematicians who are excellent programmers (and vice versa), but I personally think that's more because they have learned to keep one discipline "out of the way" of the other one. I don't think most computer scientists or mathematicians actually make very good programmers (and vice versa) because there is little actual overlap in their skillsets.

Once you get past calculus (and maybe linear algebra), math becomes primarily about proofs, not computation. Proving abstractions in math and programming software do not have a significant overlap. I think most people can reasonably learn an undergraduate math curriculum (say, up to abstract algebra and something like elementary number theory), and I think most people can reasonably learn programming. But I don't see any realistic skill transference between these two, i.e. learning one won't make you learn the other any faster.

I'm not trying to claim one subject is obviously harder than the other, I'm just saying that it might give someone false confidence to have them dive into complex math just because they have experienced success in learning new programming frameworks or paradigms every year for their career. That is setting them up for failure - it takes significant mathematical maturity to read through a textbook and understand it without a teacher, just as it takes some domain-specific maturity to optimize learning a new programming language from a textbook or documentation.

People who want to learn math should be prepared to spend 5 - 10 minutes reading each page of a math textbook to really understand the material, followed by a few hours on each chapter's problems. _This is absolutely doable for an autodidact_ , but I think my framing is a bit more realistic than yours. Even if they're innately capable of learning the math, they should be prepared for it to be essentially as difficult as learning programming for the very first time.

<comment><USER16> It's more the style. Often the technical jargon is arcane and there's usually not a clear distinction between when the author is using a very exacting mathematical term or just a generalized english term unless you are knees deep and spend a large number of your waking hours in a very specific field.

Then there's the naming ... P(X) and we're in probability and X is a random variable but if we are using just "p" then we're in physics and talking about momentum. If it's "P" then it's power, unless it's P(x) and then we're talking about geometry. And then you can italicize it, bold it, put a hat or a dot on it or under it, make the braces square, straight, or curly, and you get something else entirely; I mean completely different fields. Sometimes the same thing is used in multiple fields and then you need substitution syntax when using the two together. Great system!

So if you have a job where you are juggling 6 or so disciplines and you see a jumble of stylized letters in an equation along with a description, it's a fairly absurd system especially when the author assumes that all the readers know what they mean when they say i+p(j)/k or that when they use common everyday words which, in that particular branch of mathematics are actually very specific technical terms.

I often read things and think "what on earth is this person saying?" and then have to go back and decrypt this terribly designed mathematical language everyone uses that we are all supposed to say is a glorious and perfect interface. It's not, it's god awful and horrendous. The vast majority of humanity run screaming from it and can't interface with it to work through even simple concepts that they probably already know.

It's the modern version of ancient latin and it's become equally heretical to insist that we must create a better, more humane, more consistent, more discoverable, more flexible interface to describe the world that isn't just vestigial symbols from far-flung authors spanning 3000 years thrown together in a huge dumpster fire.

<comment><USER17> Every discipline has the problem that it's jargon is impenetrable to outsiders. Try showing someone with zero programming knowledge a text talking about "string search" and "pointer chasing" and they will be just as confused.

Mathematics is only different because it is useful in many other fields, so you get lots of non-mathematicians trying to make use of some result in isolation. When they don't understand the explanation, they blame it on the unfamiliar words and notation, but often that's just a symptom of not understanding the concepts. Anecdote: I have been taking a course taught in Chinese, which I don't speak very well, so all mathematical jargon is new to me. But just seeing how they were used let me recognize the words for familiar mathematical concepts.

Of course some mathematical writing is just bad, but usually mathematicians will then agree that it's bad. But if some formula comes with an explanation using words you don't understand, there is no way around looking up definitions until you understand the prerequisites. There is no language you could translate mathematics into to make it magically more understandable, unless the translation always prepends an introductory textbook.

I don't think it's heretical to long for better notation, I just don't think it's going to help much. But everyone is free to make up their own symbols, so feel free to go ahead. If it's really better, people will adopt it, just as they adopted superscripts for powers instead of one-off symbols, Leibniz notation instead of Newton's, and so on.

</comment></comment></comment></comment><comment> happy-go-lucky I have not read the article, but a title like _Zero-math introduction to something_ may attract those who hate math, do not want to work on something involving math, or are too lazy to re-familiarize themselves with it. What they gain from reading such articles is short-lived. They cannot put it to use. This leads to disappointment and a disturbing trend towards math anxiety.

As a kid, I tried to keep away from math not because I hated it but because my math teacher did not know how to teach it. It was my conclusion then that they had not learned the subject thoroughly. As an adult, I decided to take matters into my own hands and started learning it by myself.

<comment><USER18> Though I agree with the theme of what you said (mainly that you shouldn't run away from the math), I don't think it's as mutually exclusive as you paint it. Its always good to get different view points of the same topic, and a lot of the times not having it be do math heavy lets you have nice intuition behind the topic. A good example is 3blue1brown's videos. He gives great intuition without sacrificing rigour in his presentation, while not filling his videos with the daunting formulas and such.

</comment></comment><comment><USER4> I recommend the example in the book Doing Bayesian Data Analysis, in this book to motivate MCMC : A politicean want to visit all of US states and each with frequency proportional to the total population of the state. He takes decisions to visit a new state comparing the population of the current and the proposed state, He don't know in advance the total population of every state, but applying MCMC he archieves his goal. Here you may think that computing the total population of the US is a simple detail, but in other problem the computation involves a very difficult multivariable integral with thousand of variables, that in practice is impossible to calculate.

</comment><comment><USER19> It has occurred to me that the discussion has shifted towards programming vs mathematics, instead of the actual discussion proposed by the OP to focus on MCMC.

This happens a lot here, focus is lost very easily.

</comment><comment><USER20> The first thing I saw when I loaded the page was a chart with numbers. If charts are no longer math, then someone has certainly redefined the term math.

</comment><comment><USER21> mcmc treatment in hopcroft, kannan is really really very good. highly recommended. there are a large number of articles on the web e.g. at math- intersection-programming which are based on that. but, i would really be wary of any 'zero-math' thingy for that.</comment>


Why Wall Street Trading Technology Needs to Enter the Crypto Market - <AUTHOR> https://www.forbes.com/sites/shermanlee/2018/02/27/why-wall-street-trading-technology-needs-to-enter-the-crypto-market/ <comment><AUTHOR> Has anyone worked on an high frequency trading stuff here? I would be keen to learn from you!</comment>


Django's goals are probably not our goals for our web application - <AUTHOR> https://utcc.utoronto.ca/~cks/space/blog/python/DjangoGoalsNotOurGoals <comment><USER2> The author essentially states that he just wants to build a website and "be done with it" and never have to touch it again. He seems to acquiesce that security updates are necessary, and I suppose his greatest complaint is that old versions of software (eg python 2, django 1.11) are no longer receiving security updates, which necessitates him to update to python3 and django 2.

I feel that. But on the other hand, without trying to sound too mean: what exactly does he expect? Is he also complaining that "Windows's goals are not our goals" because Microsoft stopped supporting Windows XP? Does he complain that Ubuntu version 11 is no longer getting updates? Java SE 9?

For better or for worse, this is simply how software works. Things have to be updated. Unless you want to eschew security updates (which is not smart), there is never such a thing as a fully "finished" product. Software maintenance is part of the job responsibilities.

This post is particularly targeted at Django, but I feel that it applies to almost every web framework or piece of software, and I'm afraid this is unfairly casting Django in a bad light. I'm also afraid that we're going to see a _lot_ more posts like this in the second half of 2019 as we move towards python 2's EOL date.

<comment><USER3> The author acknowledges that some maintenance is necessary, especially for security updates. My reading of it was that his complaint is that in keeping up with the security upgrades, Django breaks other things.

This is true of a number of frameworks I've worked with. If you want security, you have to constantly stay on top of the other changes that the framework makers implement, often "fixing" things that aren't broken, removing features, or changing the way things work.

I don't know how practical it is, but it would be nice to see a framework where security updates and feature updates were separate things. One could keep one's site secure through updates without having to debug formerly working things every few months.

<comment><USER4> Unfortunately it's not practical because the test matrix explodes. They would have to develop and test every security update on every previous version which is quite costly, especially if the codebase has diverged.

The best strategy is probably what many large distros like Ubuntu have adopted: designate only a few long term support (LTS) versions, and commit to supporting only those versions.

<comment><USER5> The only matrix i support now is latest release and master, not even LTS anymore because it quickly gets too old. I consider not updating my software to be tech debt: something I must do before adding a new feature. In return, my software is always up to date and ready for new features, except once in a year when I do the said upgrades.

</comment></comment></comment></comment><comment><USER6> On the frontend, our company uses Backbone.js.

My favorite part of Backbone.js is that its development as a framework is essentially "complete" (it works as designed, functions reliably / predictably, and the maintainers aren't running in circles trying to reinvent the framework with breaking changes)

That said, I feel like I'm losing "street cred" for admitting that we still use Backbone.

It would be great, at least in my opinion, to see more frameworks like Backbone.js (with respect to reaching a "framework is complete" state).

Backbone.js Changelog: [https://backbonejs.org/#changelog](https://backbonejs.org/#changelog)

Edit: All of the above also applies to Underscore.js ([https://underscorejs.org/#changelog](https://underscorejs.org/#changelog))

<comment><USER7> We did some client work up until about 2 years ago with Backbone, too. We loved it. The entire team knew it very well and it got the job done. At least a few developers at the client would also be familiar with it every time, so it was a win win.

</comment></comment><comment><USER8> The author writes that they have a particularly big piece of maintenance coming up.

> The latest change we need is an especially large amount of work, as we will > have to move from Python 2 to Python 3.

This sounds like the transition from Django 1.11 to (probably) Django 2.2, if they're moving from LTS to LTS, and it definitely is a big piece of work if you also have to change python versions.

Django defined a release schedule DEP[0] a few years back to try to preference stability and reduce the amount of maintenance when moving between LTS versions, which is approximately every 3 years.

If your application runs on an LTS version without any deprecation warnings, then you __should __be able to move to the next LTS version without __any __changes. Which is nice in theory, but 3rd party packages in particular aren 't always aligned in such a nice way. If the stars align though, it is 6 years (LTS + LTS) of no changes.

This is all to say, I think the author has a point, but that Django has recognised the maintenance burden as an issue, and has tried to help developers with that as best they can, without just shutting down all progress. I think the balance is mostly right.

Moving from 1.11 to 2.2 should be mostly painless. Unfortunately, moving from Python 2 to Python 3 is going to make that more painful. From 2.2 the maintenance burden should be much less than in previous years. Stick with Django if the features meet your needs.

[0] [https://github.com/django/deps/blob/master/final/0004-releas...](https://github.com/django/deps/blob/master/final/0004-release- schedule.rst)

<comment><USER5> Every once in a while someone opens an issue on a repo i maintain asking for "Django 1.11" support: 2 days ago, 20 days ago ... For me, "LTS" is a mistake indeed and and goes against any Continuous Delivery / Agile philosophy that I know of, it serves as an excuse for users to not update their software which in reality accumulates tech debt. I mean, if you're going to do very little customizations to a system then LTS might be fine, but if it serves as a base to something that's custom development from scratch, like Django, it really doesn't look like a choice that's going to pay off in the long term.

<comment><USER8> As a library author that's well within your rights to determine what you're willing to support or not.

Django has a very big user base. There are many users working on "very large sites" where the cost of "churn" is dramatic. There are threads on Django- developers right now complaining that the current model is too aggressive. Striking that balance is difficult for a heavily used framework.

</comment></comment></comment><comment><USER3> I feel their pain.

One of the big sites that I maintain was built upon a framework that was fine for its purposes five years ago. But the developers of that framework have decided to take it in a direction that is incompatible with the site's goals.

Once I'm done with my current project, my next task is to rebuild that site without its current framework, and there's a 95% chance I'll go without any framework at all. Just plain HTML5 + PHP + MySQL + CSS3 + a _very_ small amount of javascript.

I know it sounds primitive for a 700+ page site with interactive elements supporting English, Spanish, and Chinese; but earlier this year my boss gave me permission to de-framework a similarly-sized site, and it worked out really well.

Frameworks are great for many projects, but not all. It's important to weigh the risks of being at the mercy of someone else, and that organization may eventually adopt a different vision of the future. Their pivot will not make your life easier.

<comment><USER9> Would a microframework (no idea what's the equivalent of Flask or Sinatra is on PHP) help? I still have nightmares from my beginner days of doing everything in plain PHP.

<comment><USER3> _Would a microframework (no idea what 's the equivalent of Flask or Sinatra is on PHP) help?_

I have no idea. Maybe. I've just been coding straight PHP. With the way IDEs are these days, once your environment is set up the way you like and you have your favorite macros bound, it's not a problem.

I'll look into Flask and Sinatra before I start the next site. Any other ones you recommend?

<comment><USER9> Flask would be my personal preference but I have a Python background (you mentioned PHP so I assumed that was your main language).

Sinatra is in Ruby. The "hello world" looked very similar to Flask but I haven't explored it deeper so no idea.

For PHP maybe also check out Phalcon ? It's not as "light" as the aforementioned ones but maybe they're more conservative in terms of breaking changes?

<comment><USER10> And probably the nearest equivalent in server-side JavaScript is Express.

</comment></comment></comment></comment></comment><comment><USER11> This is endemic to the tech world. So much of it is invested in a reflexive forward momentum that is inimical to at least some of the products it facilitates. This is very much true for example of many pretty slow-moving small-scale line of business mobile & web apps. So many of them end up being a waste of money after initial launch as the platform outruns them, and further work becomes, as the author here suggests, pure overhead.

It's among the reasons I tend to be deflationary when being consulted with about extending web or mobile presence by small business. Often I just suggest they don't bother, or do something far more minimal than their first impulse.

</comment><comment><USER12> This is a need I'd like to see addressed more broadly in software. The computing needs of many in my non-technical entourage have not changed meaningfully in the past 10 years - they crop photos, write letters, put together videos of family holidays, do basic spreadsheets, etc - and yet every few months I field support requests because their software has updated and some feature has gone missing, or the UI has been redesigned, or it's not compatible with the new OS update, etc.

Bug/security fixes and new format support (eg a photo editor probably wants to support HEIC in the 202x) are fine, but we don't really need the user facing part to change. This is a niche that seems open source could fill nicely.

(The big edge case in all this is the web browser)

<comment><USER13> Open source software has always had a major problem with user experience (as well as aesthetics).

Most people working in those projects come from a strong technical background, and in many cases users come from that background as well - so there's not much of a push for improvements in areas like ux, nor a business need to improve to boost sales.

</comment></comment><comment><USER14> There's a business opportunity here that Rails people already caught onto: you can start a 3rd-party "Django LTS" service that backports security fixes, and charge money for that.

<comment><USER8> I've previously suggested this as something the DSF could do as a money maker, but I don't think the operational complexity makes sense for a non profit foundation.

I'm surprised no one else has picked this up as a business idea though. The actual technical work likely wouldn't be very hard.

</comment></comment><comment><USER15> I think complexity here is migration to Python 3. Seems like the author picked Django and Python 2 in 2011. Python 3 was released in 2008. So by 2011 should have been clear that you might be facing a migration project sometime in future. Having made that choice getting 8 years out of Py 2 is pretty good. (Not saying he made the wrong choice. Would have made the same choice myself.)

</comment><comment><USER16> I'm aware of the well-established movement to static websites for (mostly) static sites, but what work is happening right now to counter this problem in rich web applications? Are there frameworks that take a stance that favors stability and maintenance?

<comment><USER17> Frameworks aren't the design choice that favours stability. You take the inversion of control to get all the convention/plug-and-play/just-works features that come with it. Those are focused on developer productivity, which is a constantly shifting goal.

Avoid that design and using a "libraries over frameworks" approach solves this problem but has other tradeoffs obviously, although a good ecosystem of libraries that have designed for interop solves most of them. These library ecosystems aren't the norm though, and the lure of "I'm going to package these together in a framework, that does things the way I like, and then brand it and help my career" for the library devs is always there (see Flask in Python for example, it was the "unDjango" until it got popular and now it's just another framework)

</comment></comment><comment><USER18> Different frameworks and language run-times move at different paces. It's very often the case that projects don't match their product pace and life-cycle to their toolset pace and life-cycle. "Standards" be it ABIs, APIs, data formats or markup languages all have different paces and life-cycles. Then there are the paces and life-cycles of the stacks underneath, the OS, the hardware, networks and data sources.

Then there's the security question. If this is a concern at all, ultimately the system may need to be air-gapped (e.g. like many systems depending on Windows XP are these days) to safely keep it running in lieu of fixes.

Another approach is common in the gaming scene with painstaking work on timing and bug-accurate emulation to keep old software running.

Ultimatley every system has a cost per annum, not a static capital cost. The end game being to maintain the hardware and "firewall" or update the software.

</comment><comment><USER19> The author isn't considering an alternative strategy, which is to address security issues in-place (i.e., without upgrading Django). This means keeping up on them and substantively addressing them through configuration or possibly custom code, but for something they want to freeze in time with respect to a platform, it's a reasonable choice.

Every time I've had security issues to address, there's almost always a short term fix (close a port, disable a particular eval method). In the cost/benefit analysis of maintaining a legacy app, it's a legitimate option to simply rely on those.

</comment><comment><USER5> Take a day or two to do your upgrades once in a year and you'll be just fine. Start upgrading your own plugins as soon as possible, so that you can learn how to practice the upgrade steps. Python 2 to 3 was hard for me for the first few codebases, but then piece of cake for the other dozens of codebases I have upgraded.

</comment><comment><USER20> I don't really understand the point being made here. Can someone reinterpret for me please?

<comment><USER21> The author has opted to use Django and would like to get to a state where their app is essentially "completed" \-- where it will work fine without having to touch it again or make major changes.

The conflict arises because Django has a stated mission to always be improving (like a perfectionist). This causes issues for the author when they have to re-write parts of their application in order to keep up with updates to the mainline framework.

The author would like be able to get things like security fixes and performance improvements without breaking changes -- whereas the Django team is willing to require breaking changes in pursuit of improvements.

Ultimately, the author may be better served by using tools/frameworks that are considered "completed" (examples might be things like jQuery) or have a maintenance philosophy that prioritizes backporting and compatibility over improving the features/experience.

<comment><USER20> Oh I see.

Software is a living thing - it needs to keep up with the technologies it is built on for a whole bunch of reasons.

Some software of course can be "complete" but that tends to be pretty specific stuff with a very defined scope.

</comment></comment></comment><comment><USER22> That's why you develop in Common Lisp. The language specification is 20 years old and is never getting updated. Your software will work forever!

</comment><comment><USER23> Vanilla PHP is your friend then. Don't use frameworks.

<comment><USER24> Vanilla PHP isn't any "safer" in this regard. There's lots of old PHP code out there that stopped working when PHP 7 removed a bunch of old, long-deprecated legacy cruft. That was good for the platform overall, but if you wrote a vanilla PHP app in 2008 and haven't touched it since, odds are it complicated your life.

TFA is asking for a platform that _never_ moves. The problem is that the only platforms that never move are ones that have died.

<comment><USER23> That's a good point. You need a language that doesn't move and always remains backwards compatiable. But I presume C is not a great choice as it's too low level.

I think JS will have this property for a long time, because of it's browser ties. And Haskell is another language I think won't change dramatically if you use the fundimentals. However in both cases you will want to roll your own web framework or at least pin to one you like, ideally something minimal.

However it does raise the problem of security patching, or lack of if you go down such a route.

The other mitigation, which you can do in any language is Inversion of Control. Have your code specify what sort of things it needs rather than use hard dependencies. E.g. "I need something that takes a string and a function and routes it" rather than "I need Express.js", with a layer that maps this up to actual implementations.

This way if frameworks change the mapping layer should be all that needs to change. Again pick a language like JS, Haskell, probably C#/Java etc. that is likely to be backwards compatiable for a few years to come.</comment></comment></comment>