The Sinister Scientist Behind the CIA's Mind-Control Mayhem - <AUTHOR> https://www.thedailybeast.com/the-sinister-scientist-behind-the-cias-mind-control-mayhem <comment><USER2> > when "Artichoke scientists came up with a new drug or other technique they > wished to test… they asked the CIA station in South Korea to supply a batch > [of] 'expendable' subjects...

While I'd like to believe the CIA has since found its moral compass and abandoned these kinds of experiments, I have a suspicion that it might still be going on and they just found ways to hide it better.

There had to be other technology since then that they looked at and said "we should try this thing on humans".

> One of Pfeiffer's subjects was James "Whitey" Bulger, who later became a > notorious Boston gangland killer. Bulger said that as a young inmate, he was > given LSD daily for more than a year.

That is bizarre. Wasn't another killer, the unabomber also tied to some psychological experiments when he was in college...

</comment><comment><USER3> The CIA were just _obsessed_ with LSD for all sorts of HUMINT purposes. Part of this was that they were operating under the faulty assumption that the USSR and other Cold War powers were ordering massive amounts of the stuff, so the whole "Russians are gonna do it faster, stronger, and better than us UNLESS we prepare" complex kicked in.

There was, of course, a strong incentive to test anti-interrogation techniques, should enemy intelligence use LSD against captured spies/our guys. From my own teenage experimentation, I can posit that, in the right set and setting, it'd be damn hard for me to keep any sort of secret. But it would probably be the combined fear from being tortured/interrogated while definitely having a bad trip that would give me up.</comment>


Experimenting with a Neural Network-based Poker Bot  - <AUTHOR> http://www.mattmazur.com/2009/10/experimenting-with-a-neural-network-based-poker-bot/ <comment><USER3> Certainly a fun thing to ruminate over, but the model Matt uses here is way too simplistic to draw any conclusions from about the sufficiency of using a NN.

A short stack raising >11% when he shouldn't be is a pretty severe leak ;)

Just a thought: I don't see the implementation of the algorithm here, but if you're using the values of individual cards scaled to 1, then you are saying that v(A) - v(K) = v(8) - v(7) which seems absurd. Imagine that there were (fantasize for the sake of argument) a few shoves in front of you, then should be that v(AK) - v(KQ) >> v(87) - v(76).

I'd need to give more thought to this problem in general, but I have to ask Matt...why did you choose short stacking full ring NLHE? It seems mechanical for someone very familiar with the game, but it's so situational. The first area I'd think you'd be most effective would be LHE-HU-SNG's.

<comment><USER4> _Just a thought: I don't see the implementation of the algorithm here, but if you're using the values of individual cards scaled to 1, then you are saying that v(A) - v(K) = v(8) - v(7) which seems absurd_

Instantiating the features correctly is crucial to getting good neural network accuracy.

It might be the case that a thermometer representation for the card value is more appropriate. The thermometer representation is that there are thirteen bit-values, and the first n values are on for the value n.

Using this sort of input representation, the network can model the non-linear relationship between different card values, instead of assuming a strictly linear correlation.

<comment><USER3> It's a neat idea, but I doubt it would work well because the hand value in this context is so qualitative (whereas in HU LHE it would be more quantitative.)

</comment></comment></comment><comment><USER5> His tests are so simple that he's not really testing anything. For some real analysis of how neural nets stand up to other player prediction models, check this out:

<http://spaz.ca/poker/>

In particular, read the paper titled "Opponent Modeling in Poker: Learning and Acting in a Hostile Environment". Section 4 outlines several prediction methods, one of which is a neural net.

</comment><comment><USER6> Does anybody know about these 2 ideas? 1) a site that lets people upload their bots and play bot vs. bot 24/7. The idea is to try and find the best algorithm.

2) A site that a human player plays at against the computer. The computer also tries to model the player using a NN and tell the player how "Big" of a NN was needed to model their play. Perhaps how quickly the computer was able to start predicting the player's play? The idea being that you want your play to be very hard to model with a NN so that you are more unpredictable in real life.

<comment><USER7> Based on personal experiece, I feel like (2) would not work very well (nor would using a similar idea to test your bot). When real money is on the line, my entire state of play changes. There is a large psychological difference between playing a bot for fun, and playing a bot for money. Playing for fun, I am more likely to place large bets, bluff, test its logic, and in general play loosely. For real money? No thanks.

Note that this is all based on personal experiece, and may not be the common case, but I feel like the average hacker will play to test the bot, not to win.

</comment></comment><comment><USER8> You really don't need anything that complicated to beat low-stakes no-limit poker short stacking. I used to know a few people who made decent hourly rates just mechanically following a very small spreadsheet. I did it myself for awhile and it seemed like an easy win, though not as profitable as simply being good at the limit games I was playing and nowhere near as challenging/fun. My personal sample size was small, but I still suspect a small set of if/then statements would have been a winner.

The real challenge is finding enough tables to play. The bad thing about short stack is that sometimes someone calls you and you win, and now you've got at least a medium stack. Most sites don't let you come back to a no-limit table with less money than you left with until some period of time (for instance 2 hours) has passed. Party Poker did not have this feature (maybe still doesn't) so it would have been easy back then, but for an American now I don't know of any site of substantial volume where you wouldn't quickly burn through all available tables and find yourself waiting.

Even for a bot, there might not be much profit in it.

<comment><AUTHOR> It seems easy, right? That's what I thought too.

I purchased 7M+ NL200 hand histories from a guy who mined them all day every day. I filtered that down to hands where someone was playing with 15 - <USER2> stacks and then for every person that played more than 5,000 hands in that range, calculated their profits and winrates.

Here's the breakdown: <http://www.mattmazur.com/images/ShortstackingNL200.png>

It's important to remember when looking at these numbers that the vast majority of people who try shortstacking never make it to 5,000 hands so this is heavily skewed toward the survivors. It would be like analyzing the revenue of startups that are over five years only and then using that average for the average revenue of all startups. There's only 80 people in the full ring group and 19 in the 6-max group. Just looking at these folks, only about 1 in 5 win more than 2bb/100 and in my experience that's not easy to do. The people that are profitable shortstackers would likely win a lot more playing with a normal stack.

I don't think there are many long term profitable shortstackers following rules. Again, I think the majority of the profits come from a few strategic plays that would be impossible for a spreadsheet, or a bot, to emulate.

<comment><USER8> Idk, those numbers seem pretty reasonable to me and don't indicate any real level of difficulty. 58% of people in it are even or worse, which means if there is any sample bias at all, it's very small. The fact that losers tend to drop out over time is probably mitigated at that level by the fact that winners will quickly rise from low stakes to at least middle ones.

2-4 bb/100 is great even for a human since you can do huge volume that way. The spreadsheet method can easily be applied to 8 tables at once, maybe more. 2 bb/100 (thats $4 as calculate by PT right?) could be $30/hr. 4bb is obv $60, which is more in line with what people I knew were making.

Also when short stacking, opponents at higher level games don't necessarily respond any more appropriately. In fact a lot of times they play worse against it because they have more idea what to do vs. normal opponents. Low level players like to call big bets too much, which makes them accidentally better vs. a short stacker. My friends claimed they were getting higher bb/100 at the $3/$6 and $5/$10 games than the $1/$2 because of that.

<comment><AUTHOR> I just realized that I have the hand histories for all cash No Limit hands played on PokerStars from Dec 1, 2007 to Mar 12, 2008. I've got all the tools built to do the analysis, so I can pick apart the results from the lower and higher stakes as well. Any recommendations on how to do it?

Not PTBB/100, just BB/100. PokerTracker couldn't handle all the data.

<comment><USER8> Wow, how many hands is that? That's really cool.

I suppose you could possibly control for play style, thinks like what Poker Tracker tracks. VPIP, etc., and of course buy-ins if you do some technomagic to figure them out. (I'm assuming someone who short stacks all the time plays differently than a normal player who just lost half their stack.) Not sure how good your findings would be though.

</comment></comment></comment></comment></comment><comment><USER9> Have you checked out "The Mathematics of Poker" by Bill Chen and Jerrod Ankenman? This books does some neat game theory based analysis of poker and comes up with some surprising results. I've pondered what some sharp hacker type might do given that text and some time to grok on a bot.

I think the one result most applicable in this scenario is the section with jam or fold tables based on hand and stacksize. Basically, they show that there are completely defined hand ranges when you should jam or fold when stacked at a certain number of BB at no-limit.

I've pondered doing a neural network, but for limit rather than no-limit. I still may, mainly because I think working on a bunch of inputs that are hard to quantify on the fly when playing would be fun to experiment with via the neural net.

Thanks for your posts on poker bots - although I have to admit I've basically quit playing online over the last year because of the proliferation of HUDs, data mining automation, and bots.

Oh, and for anybody interested in game theory and/or poker, you simply have to grab the book. It's a bunch of fun and not scary on the math for those on this forum.

<comment><AUTHOR> I've read The Mathematics of Poker; the binding is broken from keeping it propped open too much. That being said, it was not very helpful actually building a bot. They do a lot of nice little game theory examples, but most of it is not applicable to real life game play.

The push/fold tables are another story, and I'll write about that soon.

</comment></comment><comment><USER10> For those interested in the PokerBot aspect, check out: <http://pokerai.org/openholdem.html>

It allows for bots in Perl and has a DLL interface. I've been meaning to make a V8 based DLL to write bots in Javascript for a while, but it's not a high priority for me.

P.S. Did anyone notice this guy has a Tetrinet bot?! That game made high school tolerable.

<comment><AUTHOR> The TetriNET bot was a blast. You can download the full source code on the site as well.

</comment></comment><comment><USER11> One of the issues with NN is that there is no way to find out the relative significance of the input variables and how they affect the output response. Atleast I couldn't find a way when I tried using them a couple of years back. Secondly, I guess there isn't a way to cluster similar cases together to analyze what rules bind different decisions together. Also, one cannot find the partial dependence of the output on a given input variable. All these problems are solved by the random forest classifier which I have been working with and really like.

<comment><USER12> I agree that analysis of neural networks is not easy, but there are certainly ways to do it:

 _there is no way to find out the relative significance of the input variables and how they affect the output response_

Calculate partial derivatives of inputs w.r.t. outputs.

 _there isn't a way to cluster similar cases together to analyze what rules bind different decisions together_

PCA analysis of activations of the hidden layer. Could also do k-means clustering of activations of hidden layer.

The random forest classifier looks interesting, I'll have to investigate that. Any suggestions for papers/tutorials?

<comment><USER4> _there is no way to find out the relative significance of the input variables and how they affect the output response_

"Calculate partial derivatives of inputs w.r.t. outputs."

Yes. There are other ways to interpret the output. See, for example, "Visualizing Higher Layer Features of a Deep Network." by Erhan et al 2009: [http://www.iro.umontreal.ca/~lisa/publications/?page=publica...](http://www.iro.umontreal.ca/~lisa/publications/?page=publication&kind=single&ID=247)

It is true, though, that more powerful models are less explainable. But in return you have more compact modeling and training via gradient descent. This is faar faster than the combinatorial optimization involved in ensembles of trees. (Trust me, I've implemented both.)

 _there isn't a way to cluster similar cases together to analyze what rules bind different decisions together_

"PCA analysis of activations of the hidden layer. Could also do k-means clustering of activations of hidden layer."

I concur. However, for 2-d or 3-d visualization, you should use the more recently developed t-SNE algorithm instead of PCA or other alternatives. t-SNE does waay better. Software (in Matlab and Python) is available at: <http://ict.ewi.tudelft.nl/~lvandermaaten/t-SNE.html>

Also, you should look at the JMLR paper ([http://ict.ewi.tudelft.nl/~lvandermaaten/t-SNE_files/vanderm...](http://ict.ewi.tudelft.nl/~lvandermaaten/t-SNE_files/vandermaaten08a.pdf)) and supplemental material ([http://ict.ewi.tudelft.nl/~lvandermaaten/t-SNE_files/Supplem...](http://ict.ewi.tudelft.nl/~lvandermaaten/t-SNE_files/SupplementalMaterial.pdf)), to see the visualizations produced by t-SNE and competing methods. This qualitative evaluation by looking at pictures speaks for itself.

 _Also, one cannot find the partial dependence of the output on a given input variable._

If your model assumes that the output is a non-linear combination of inputs, then yes, it is hard to express the output in terms of a linear decomposition. But that was your choice of modelling assumption, presumably because linear models are insufficiently powerful to fit the underlying variations.

"The random forest classifier looks interesting, I'll have to investigate that. Any suggestions for papers/tutorials?"

Random forests were developed by Leo Breiman (RIP).

"Ensemble methods in machine learning" by Diettrich (2000) compares different tree ensemble methods. He concludes that boosting an ensemble of decision trees is better, except when the data are very noisy, in which randomized trees are better. (Boosting is when you focus on the examples that the model is currently doing the worst. Randomized instead works on random subsets of examples.) The main reason boosting is worse than randomized trees in the noisy case is because the AdaBoost exponential loss is sensitive to outliers. Which is to say, AdaBoost boosts the wrong loss function. Boosting an appropriate choice of loss function (perhaps a regularized log-loss) is probably superior to randomized trees in most circumstances.

"Improved boosting algorithms using confidence-rated predictions" by Schapire and Singer (2000) is a great introduction to boosting.

Around the same time Llew Mason and Jerome Friedman independently demonstrated that boosting is essentially fitting an additive model using gradient-based methods to select the features that have the steepest loss gradient. So you should follow up by looking at their work.

</comment></comment></comment><comment><USER13> This bot attacked Texas Hold'em. Presumably most of the others do as well, given that it is far and away the most commonly played game online (disclaimer: complete assumption).

What about attacking 7-card stud, stud8, omaha, and o8? These games have more information to analyze than hold'em because more cards are visible. It seems to me that would provide more of an edge to a computer, which could calculate drawing odds faster and more accurately than a person.</comment>


How Trees Calm Us Down - <AUTHOR> http://www.newyorker.com/tech/elements/what-is-a-tree-worth <comment><USER2> I was born and raised in Brooklyn, NY, but moved out to rural NJ in my thirties. I bought a house on a lake with no motorboats, plenty of black bears and raccoons and lots of trees. I now live in the rice fields of East Java, Indonesia, so I guess you can say I love the outdoors.

I do question the science or numbers in the study as much as I believe the basic premise to be true, however, correlation does not automatically imply cause. People suffering more after trees are removed can also mean that urbanization or development brought factories, or unhealthier air, rodents or any number of other negative factors with it.

I do intuitively relax more, and take great solace in my surroundings, and I do believe it is better for people. I would like to see more research on this; there have been a lot of debacles in the past two years in the social sciences and psychology with statistics and peer review. Some of the studies were taken for granted and are now under the microscope for being inconclusive or just wrong.

Yea for trees! And plants, animals and all that entails!

<comment><USER3> As someone who suffers from hay fever being around trees can be weeks of pain for me every year

<comment><USER4> There is a massive, recent hayfever problem in Japan because cedar trees planted to be used for timber after WWII weren't as construction methods changed.

[https://en.wikipedia.org/wiki/Hay_fever_in_Japan](https://en.wikipedia.org/wiki/Hay_fever_in_Japan)

<comment><USER5> The thing that always gets me is that when I spent summers in canada I experienced not an iota of hayfever, but I get pretty significant symptoms in the UK. I think it's about the only strong draw to emmigration at some point in the future!

<comment><USER6> I'm the same way. I'm from the US and have never had any allergies ever, though I've spent a lot of time outdoors. I spent a summer in the UK one year, and the whole time I had horrific allergies. I was basically hopped up on Benadryl the whole time.

<comment><USER7> Opposite for me. I grew up in rural England surrounded by flowers and forests. Never had a hint of allergies. Lived in Germany on the edge of the Black Forest for ten years... no problems.

Moved to New England and I had no idea what is wrong with me. It took me three years to figure out it was tree pollen and not a spring cold... haha

In California now so we will see how it goes.

</comment></comment></comment></comment></comment></comment><comment><USER8> I opted for a lower quality apartment this year because all it's windows face vast green fields or trees. The effects are undeniable - some of my favourite times this year have been spent just sitting on my balcony admiring the greenery.

<comment><USER9> Went on my first cruise this year. I'm so glad we got a balcony -- I think I spent most of the trip just sitting on the balcony enjoying the sound of the ocean, or watching people at port. The balcony was my favorite part of the whole vacation...

<comment><USER10> I agree. My favorite time was eating breakfast sitting on the balcony admiring the ocean.

</comment></comment></comment><comment><USER11> Having trees in the city is nice, and Berlin has an ok level (at least where I live). But I recently started to do long weekend day hikes in the area around the city, and the effect is even better. The constant change of natural forms while moving really frees up my mind and floods it with new impressions that I don't have on my work days. I used to have a meditative effect from running, but it has become a bit too much routine in that regard.

There must be something about "natural forms" (as in varying, not changing, non-rectangular) that creates that feeling.

<comment><USER12> I fully agree. Trees (and plants in general, and rocks) are more fractal, and contain interesting detail on many levels... There's something pacifying about being around them vs flat, rectangular urban surfaces.

<comment><USER13> Personally, I've always felt nigh claustrophobic when surrounded by trees. Give me the flat, wide-open desert, though, and I feel sublime. There's something pacifying about seeing as far as my eyes can see. Now, I do love some national parks, such as the Grand Canyon, or Zion, in which you can become surrounded by trees and mountains and feel utterly awestruck at the beauty. But I feel my spirits tick up as soon as I hit the vast expanse of desert again upon exiting the trees.

</comment></comment></comment><comment><USER14> Read more about The Biophilia Effect: [https://en.wikipedia.org/wiki/Biophilia_hypothesis](https://en.wikipedia.org/wiki/Biophilia_hypothesis)

</comment><comment><USER15> "Are trees alive?" is the question to ask yourself. They can seem very un- alive to us humans. But when the wind blows and their leaves move you can see it. They are literally WAVING at you. Think back to when you were 8 on the playground and a friendly kid waved at you. Trees just wanna play. But wait you say, that's just the wind. The tree isn't deciding to move like the 8 year old kid decided to wave his/her arms. OR DID the tree purposely make its leaves in a shape to catch the wind and that movement is 100% intentional. When you see it that way you can stare at trees for hours. Also, every single one of those trees is naked. When you are bored/depressed/lonely just stare at trees and giggle.

<comment><USER16> "Trees show the bodily form of the wind" \- 禪林句集

</comment></comment><comment><USER17> From what I've seen living in Toronto, NYC and Montreal, streets with nicer houses/apartments tend to have more trees. Those neighborhoods also tend to be quieter.

Take the example of NYC. The Upper East Side and Clinton Hill are two neighborhoods with a relatively large number of trees. Both of those neighborhoods are two of the most expensive and quiet neighborhoods in Manhattan/Brooklyn.

So it could just be that quiet streets and nice houses calm us down. But then again, maybe having more trees is what causes neighborhoods to be nice and quiet. As far as I can tell, it could go either way.

<comment><USER18> In cases like this, trees are a luxury for the rich. Not being gripped by poverty might have a calming effect.

<comment><USER19> The study controls for income & area income, so that calming effect should already be accounted for.

</comment></comment></comment><comment><USER20> One hypothesis is that we've also evolved to associate greenery with healthy land and lifestyle. I can see why these signals from millions of years can have the 1% quoted effect.

<comment><USER21> I think its obvious, we like rivers, mountains, flowers, greenery as they all represent fertillity of the land! (Mountains mean that there will be water even in draughts)

<comment><USER22> Only if you consider Coors Light, the mountain beer, to be water (which it is).

Draught is the british (and Canada, other places as well) spelling for draft as in draft beer. Not trying to be grammar nazi, just thought it was funny cause the typo was somewhat well placed.

<comment><USER23> > if you consider Coors Light, the mountain beer, to be water

What did water ever do to you?

</comment></comment></comment></comment><comment><USER24> All of those saying correlation is not causation did not read the article. The study detected an immediate and neasurable effect from just walking among trees.

<comment><USER25> Now I think this probably _is_ causation, but pointing out that the correlation is really strong is not an argument against "correlation is not causation".

<comment><USER26> Reflexively trotting out "correlation is not causation" is a thought terminating cliche. It short circuits our thinking. People often stop at the statement, feeling satisfied in their scientific abilities.

</comment></comment></comment><comment><USER27> An office enriched with plants makes staff happier and boosts productivity by 15 per cent.......

[https://www.uq.edu.au/news/article/2014/09/leafy-green- bette...](https://www.uq.edu.au/news/article/2014/09/leafy-green-better-lean)

<comment><USER28> This study was limited to "employees from two large commercial offices", and probably suffers deeply from a Hawthorne Effect[0]. As usual, the rigor is doubtful.

[0] [https://en.m.wikipedia.org/wiki/Hawthorne_effect](https://en.m.wikipedia.org/wiki/Hawthorne_effect)

<comment><USER27> Yep. I don't believe the 15% but I think it has value.

Random fact the Hawthorne Effect was BS. Perhaps it's true but the original 'lighting' effect experiment was rubbish.

</comment></comment></comment><comment><USER29> Interesting finding, but the article at least leaves one with more questions than answers (I haven't looked up the original research).

My own personal experience tends to confirm the main point put forth. Indeed, when we moved to the US Southwest several years ago, I thought I would miss oceans the most (having always lived on a coast). But no, I really miss seeing green -- my first time back east after moving here, the impact of seeing all those trees was really tremendous (& positive).

Having said that, the effect mentioned in the study can also be due to the amount of attention that a city street demands, and a lot of other factors. (Walking down Broadway in NYC in the middle of day just isn't the same as strolling through West Village on a Sunday morning!) Not to mention what other commenters have pointed out, e.g., correlation != causality. Quite likely the researchers have thought about this; I would be interested in what they found.

</comment><comment><USER30> I remember an explanation for the calming effect of nature by David Allen of Getting Things Done fame. He claims that the environment is too complex so your mind "lets go" he repeats some of that here [https://www.linkedin.com/pulse/20121027044918-402563-david-a...](https://www.linkedin.com/pulse/20121027044918-402563-david- allen-explains-it-all) I'm not sure what the basis of that analysis is, but he could be right. I contrast that with the jail from THX 1138 which doesn't seem like it would be relaxing [http://nightflight.com/wp- content/uploads/THX-1138-6.jpg](http://nightflight.com/wp- content/uploads/THX-1138-6.jpg)

</comment><comment> et-al National Geographic had a similar article earlier this year if you'd like to read more:

"This Is Your Brain on Nature"

[http://ngm.nationalgeographic.com/2016/01/call-to-wild- text](http://ngm.nationalgeographic.com/2016/01/call-to-wild-text)

</comment><comment><USER31> Anyone notice that the sound and atmosphere contributes to their well-being?

</comment><comment><USER32> I recently visited an abbey in Killarney, Kerry in Ireland. The monks build an enclosed walkway around a very old yew tree, it was fascinating: [http://www.killarney.co/muckross-abbey- killarney.html](http://www.killarney.co/muckross-abbey-killarney.html)

Perhaps inspired by a similar line of thinking.

<comment><USER33> Skipton Castle, Yorkshire, Conduit Court yew tree, planted in 1659.

[https://en.wikipedia.org/wiki/Skipton_Castle](https://en.wikipedia.org/wiki/Skipton_Castle)

</comment></comment><comment><USER34> _Some of the walks took place in June, whereas others took place in January; most people didn't particularly enjoy trudging through the harsh Michigan winter, but their scores jumped just as much as in the summer trials._

I found this the most interesting point in the article. I would have assumed that any psychological effect of viewing trees would be largely due to their greenness, since that is their dominant visual aspect. But, assuming a largely deciduous environment, naked trees in winter would seem to have the same effect. So the effect must be stimulated by something deeper than just raw color.

</comment><comment><USER35> "Berman and his colleagues have zeroed in on the "low-level" visual characteristics that distinguish natural from built environments. To do this, they broke down images into their visual components: the proportion of straight to curved edges, the hue and saturation of the colors, the entropy (a statistical measure of randomness in pixel intensity), and so on."

I wonder whether these principles could be incorporated into architecture and interior design, so we feel like we're in a natural setting even when indoors.

(Even better with trees visible through the windows, of course.)

</comment><comment><USER36> Plant fruit and nut trees. This is vastly more useful than trees for the sake of calm.

See permaculture, food security, urban farming, distributed production, decentralization.

Trees for some zen or aesthetic cause is an elitist and ignorant perspective. Land use in suburban environments is extremely poor. Food sustainability is very poor.

Trees are a good starting point to start researching. But there are much more serious reasons than a warm fuzzy feeling.

</comment><comment><USER37> I can never show this to my mother, she was right all along!

<comment><USER26> Quite the opposite, she would be thrilled!

</comment></comment><comment><USER38> You mean I don't have some mystical connection to the trees and that it's simply burned into my synapses from eons of evolution?!

</comment><comment><USER39> Interesting. I love walking among the southern live oaks around where I live. They're just amazing trees. There was one that got hit by lightening a couple weeks back and split down the middle. I actually felt empathy for it. These trees are usually hundreds of years old.

</comment><comment><USER40> A tree on a street is incredibly expensive to maintain. The tree itself need maintaince from a gardner and the surrounding road and sidewalk needs extra maintenence too.

<comment><USER41> The price is worth paying, and planting more trees will bring the cost per tree down.

<comment><USER40> > The price is worth paying

I looked up prices as I was initially in doubt if this was a good cost- benefit. This article claims that New York Cty trees have a lifetime cost of ~3,500 USD (in gardening). I was under the impression of a much higher _annual_ cost, so I was way off in my initial reasoning.

This report puts annual gardening costs of Portland trees at around 40usd [1]

These numbers do not include added road, curb and sidewalk maintenance incurred by the tree, as far as I can tell.

[0][http://scienceline.org/2013/10/the-high-cost-of-carbon-in- ci...](http://scienceline.org/2013/10/the-high-cost-of-carbon-in-city-trees/) [1] [https://www.portlandoregon.gov/parks/article/514095](https://www.portlandoregon.gov/parks/article/514095)

</comment></comment></comment><comment><USER42> > twenty per cent better ... on tests of memory and attention

> five times bigger in people who have been diagnosed with clinical depression

<comment><USER43> In this case the ellipsis is extremely misleading.

> Those who had taken the nature walk performed about twenty per cent better > _than their counterparts_ on tests of memory and attention

<comment><USER42> In a randomised trial, how is that misleading?

<comment><USER43> Ah, you're correct, I'm wrong

</comment></comment></comment></comment><comment><USER44> Claude Monet clearly new something about it.

</comment><comment><USER45> I can see a market for a VR movie/game which allows the user to walk/drive through (or just sit in) a forest :)

<comment><USER46> Such a typical tech reaction/sentiment: what's better than real trees? A VR simulation of trees! :)

<comment><USER47> For those people where getting out and about to real woodland is impractical, then yes I see a good market for VR solutions such as this.

Housebound or otherwise infirm people would benefit from VR simulations of peaceful locations.

Likewise so would all us shut-in techies that live in our basements...

<comment><USER48> It could work quite well as the new tab screen in a browser. Slowly move the camera through the woodland so that the view changes. That would keep the view incidental whilst keeping some novelty.

<comment><USER49> Sounds like the trend of slow television: [http://www.wsj.com/articles/SB100014241278873249040045785391...](http://www.wsj.com/articles/SB10001424127887324904004578539110228634592)

</comment></comment></comment></comment></comment><comment><USER50> Oxygen?

</comment><comment><USER51> Correlation is not causation.

Houses on streets with trees are more expensive. People that can afford to live there are healthier for obvious reasons.

Likewise, people that are put into better hospital rooms are probably just patients the hospital is willing to expend more energy on, because they have deeper pockets/are the right ethnicity/are "respectable people" etc..

Is there any non-depressing source of science journalism left in the world?

<comment><USER19> While the study doesn't seem to have taken into account the houses' values, it has taken into account income (as the article states) and area income (the article doesn't state this). I do not know if that's enough to correct for housing costs. But I do assume that accounting for area income should go a long way to counter your example regarding affording houses. That said, of course correlation isn't causation.

<comment><USER51> Most of us don't buy houses on our income, but on our credit. This speaks to class, not to wealth; someone with good credit, from a background that grants them that status, is more likely to have a job with good benefits, health insurance, a family that can recommend good doctors, etc.. They're also less likely to live in a food desert, more likely to be able to afford fresh fruits and vegetables, etc..

Income speaks only partially to this. Systematic racism is still very strong in housing; people that live in these houses are more likely to have white privilege and to not have the chronic stressors of racial inequality nagging at their health either.

But I'm sure we're "wired" to "enjoy green spaces." The dryads no doubt protect us when we can afford to be near them.

<comment><USER19> You seem quite certain that the research is invalid and quite certain of your systemic racism hypothesis. I would invite you to find evidence for your hypothesis in the actual study results.

For your hypothesis that the study results are skewed by white privilege at least the following things would have to be true: 1\. Neighboorhoods with more citizens from lower classes would need to be less green than neighboorhoods with less citizens from lower classes. (At least where I live, in Amsterdam, that would presumably not be the case as the Bijlmer, one of the largest and poorest neighboorhoods is very probably the greenest one. As an aside, the amount of trees is sometimes seen as a problem for that neighboorhood due to the combination with high crime rates, as foilage provides cover for people with criminal motives, makes the area darker (less light getting through) and makes it feel less safe when combined with the sight of huge high-rise flats). This only adds to problems for the people living there and the stereotyping by those not living there :/).

2\. The correlation between house prices and area income would need to be lower than expected but there would need to be a significant correlation between house prices and class (f.ex. credit score)

Contrary to your statement, I do think there's a large correlation between the first two, but I haven't checked with housing data for Toronto. For one, there's also the option to rent a house so the correlation between income and housing quality will be higher, as people with higher incomes can afford higher rent. If I read this report correctly, around 30% of Toronto apartments are rental ones [http://www.trebhome.com/market_news/rental_reports/pdf/renta...](http://www.trebhome.com/market_news/rental_reports/pdf/rental_report_Q1-2016.pdf). With rental homes, surely there's a large correlation between income and housing cost. Secondly, while I do agree that income and class won't necessarily be correlated at the individual level, I would assume it evens out on the neighboorhood level. Neighboorhoods where people have good credit, income stability, good status, a good job, good medical contacts and access to good food, will surely have a higher average income. And actually, income does seem to matter when buying a house in Canada ([http://globalnews.ca/news/985258/to-rent-or-to- buy-8-questio...](http://globalnews.ca/news/985258/to-rent-or-to- buy-8-questions-canadians-should-ask-before-taking-the-plunge/))

So I am sympathetic to your argument that systemic racism makes it more likely that one lives in bad housing, but don't buy that this will explain the study's results. I do wish they would have worked with housing costs/rent.

You did make me investigate Toronto a bit. It has quite a lot of food deserts [http://www.ehatlas.ca/built-environments/food- deserts](http://www.ehatlas.ca/built-environments/food-deserts), and apparently its child poverty rate is quite high. I'm often confounded by how cities and economies are built in that part of the world. In the Netherlands, cities are littered with supermarkets and the overall availability of bikes and scooters makes it even easier to get to one. Credit scores are much less of a thing here, too. Whether you can buy a house is based on income and income stability.

Btw, you do seem to be assuming the american(?) situation to some extent. Canada has universal health insurance and putting patients with deeper pockets in better rooms seems to be an odd explanation then, too.</comment></comment></comment></comment>


How to keep settings in ASP.NET Core? - <AUTHOR> http://foreverframe.net/how-to-keep-settings-in-asp-net-core/ <comment><USER2> Just... no[1]. For most of the example settings either environment variables, a secure service, or enough environment variables to connect to a service, or other environment pki in place for such communications.

Stop putting your settings in config files for service applications... if it's something that runs on a desktop, sure. If it's a server app, don't do it.

[1] [https://12factor.net/](https://12factor.net/)</comment>


The Ars guide to building a Linux router from scratch - <AUTHOR> http://arstechnica.com/gadgets/2016/04/the-ars-guide-to-building-a-linux-router-from-scratch/ <comment><USER2> If you don't want to do all that configuration, PFSense is a good alternative. [https://www.pfsense.org/download/](https://www.pfsense.org/download/)

PFSense is the same thing below the hood, but with a web front-end and plugins.

Most off the shelf wireless routers work fine as an access point, but are quite bad as a router. So you can just plug your old wireless router into this thing (with DHCP etc turned off), and your whole setup will be much better.

<comment><USER3> I'll echo the pfSense recommendation. One correction, though: pfSense is built upon FreeBSD, not linux.

<comment><USER4> IPFire is my choice if you prefer Linux.

</comment></comment></comment><comment><USER5> Ubiquiti's EdgeRouter Lite is a popular, fast, cheap (<$100), solution in this space.

People are running FreeBSD and Linux on it:

[http://www.daemonology.net/blog/2016-01-10-FreeBSD- EdgeRoute...](http://www.daemonology.net/blog/2016-01-10-FreeBSD-EdgeRouter- Lite.html)

[https://wiki.gentoo.org/wiki/MIPS/ERLite-3](https://wiki.gentoo.org/wiki/MIPS/ERLite-3)

<comment><USER6> One note is that it's my understanding that running Linux or even FreeBSD will mean that you can't hit 1Mpps if that's important to what you're doing.

<comment><USER7> The driver for the network offload is proprietary, however the pre-installed OS is Debian Linux with a fork of Vyatta.

<comment><USER6> Excellent point. I should have been more specific.

</comment></comment></comment></comment><comment><USER8> Here's a tutorial for doing the same thing with OpenBSD from a few years ago: [http://www.bsdnow.tv/tutorials/openbsd- router](http://www.bsdnow.tv/tutorials/openbsd-router)

<comment><USER9> The OpenBSD site has an example as well.

[http://www.openbsd.org/faq/pf/example1.html](http://www.openbsd.org/faq/pf/example1.html)

As a point of comparison, the iptables syntax as shown in the Ars article is far harder to grok at first glance than either of the pf examples. For example:

        pass in on egress inet proto tcp from any to (egress) port { 80 443 } rdr-to <ipaddr>

vs:

        -A PREROUTING -p tcp -m tcp -i p4p1 --dport 80 -j DNAT --to-destination <ipaddr>:80
        -A FORWARD -p tcp -d <ipaddr> --dport 80 -j ACCEPT

<comment><USER8> Yeah, I've always found iptables to be a little hard to understand.

It also sounds like it's getting replaced[0]. My guess is that we will see iptables around for a long time after it's been deprecated. ifconfig, for example, is deprecated[1] yet it's still around and being used.

[0] [https://lwn.net/Articles/564095/](https://lwn.net/Articles/564095/) [1] [https://lists.debian.org/debian- devel/2009/03/msg00780.html](https://lists.debian.org/debian- devel/2009/03/msg00780.html)

<comment><USER10> ifconfig is still used on BSD and, most importantly, MacOS X. Also, the Linux replacement for ifconfig is ip from iproute2, and that was made by the guy who did Linux QoS. He sort of sucks at usability.

For example, `ifconfig eth0 down` becomes `ip link set dev eth0 down` and `ifconfig eth0 192.168.1.2` becomes `ip add add 192.168.1.2/24 brd + dev eth0`; I had to look these up to make sure I had the correct syntax.

Okay, the actual command is `ip address add` and not `ip add add`, but it allows extreme abbreviation at the expense of discoverability.

</comment></comment></comment></comment><comment><USER11> I'm sad they don't use dnsmasq in the article, it's a ton more easier to setup than bind and even commercial routers use it.

<comment><USER12> Yeah, plus it can also serve as DHCP server, one less package to install/maintain.

</comment></comment><comment><USER13> I wonder how well the Turris Omnia[1] will compare to a homebrew solution like this.

[1][https://omnia.turris.cz/en/](https://omnia.turris.cz/en/)

<comment><USER14> I cant wait for mine to arrive :)

I like those automatic security updates and network traffic analysis...

</comment></comment><comment><USER15> I had trouble getting a hold of one of those C1037U boxes from China. The seller would 'run out of stock' frequently if I found one for a decent price.

I ended up going with the APU2B4 board (an upgrade from the APU1D mentioned in the article.) I put pfSense on it, and it's been running perfect for a few weeks now.

Even that board is probably massive overkill for most people. I have 50/50 internet, and with full bandwidth used by torrents, a VPN and ssh session open to the router, and the web interface open, I'm still only getting about 10-15% CPU.

[http://pcengines.ch/apu2b4.htm](http://pcengines.ch/apu2b4.htm)

<comment><USER16> Interesting. I have been using such PCs in my Chinese office for about a year. They are made in a factory about 30 minutes from me. I was considering designing a better looking case and bundling a more reliable power supply to export these but I got busy with bigger business.

These Shenzhen factories are somehow getting these Intel CPUs for next to nothing. Factory price for the i5 model was about $100.

<comment><USER17> At least in the past, I've heard of people getting lots of low binned / questionable QC'd intel chips for dollars per chip. They basically go for auction to the highest bidder. If you're willing to deal with a high defect rate (Either extensive QC yourself or just don't give a shit), it is a pretty good deal.

</comment></comment></comment><comment><USER18> Seems to be missing on of the most crucial parts: Keeping the software up to date to avoid being a victim to security issues.

<comment><USER19> During the "Linux-Setup" part of the article:

""" ...and whether you want automatic security upgrades. (Spoiler: Yes, you do.) """

</comment></comment><comment><USER20> This is also a nice simple and cheap device running OpenWRT ($25) with Wireless N, 2 100 mbit lan and USB: [https://revspace.nl/GL- iNet](https://revspace.nl/GL-iNet) \- [http://www.gl-inet.com/](http://www.gl- inet.com/).

I've got a about 50 deployed, managing them with Ansible, super nice and cheap. USB powered as well.

<comment><USER21> If you like that, check out the Nexx WT3020H. Very similar specs but you can get them from China for about $13 USD.

Best of all, they're based around a MediaTek CPU, which doesn't have the same USB quirks as the Atheros AR9330 used in the GL-iNet.

I've personally upgraded my 3020H units from 8MB SPI to 16MB, but I've also heard that you can order them directly from the factory with 16MB if your order is large enough, or they're willing to customize.

</comment></comment><comment><USER22> At least in my setup, a small SSD means that a complete reboot for the router takes 22-24 seconds, and so TCP sessions will not drop.

<comment><USER23> Be careful when using a very small SSD or something like a CF card for a router. Enabling logging to disk can wear the flash memory out in a matter of weeks

<comment><USER22> I have a 64GB SSD and a little over 4.5GB used in the last year. I don't think that flash exhaustion is likely in the near future... but I also have a 32GB USB stick plugged in, which gets a backup copy once a week. The great thing about building your own router is that you can easily replace parts.

</comment></comment></comment><comment><USER24> 1400 - 2400 usd for that small box? Is that worth it??

<comment><USER25> It was $250 before, a lot of sellers on aliexpress just jack the price when they're out of stock instead of delisting the item.

</comment></comment><comment><USER10> It is almost 4 years since World IPv6 Launch. I'm very disappointed that, other than a few randomly timed rants from Iljitsch van Beijnum, Ars Technica has made no visible movement to IPv6. No AAAA record for Arstechnica.com, no guides to installing IPv6, and now a tutorial for setting up routers spreading FUD about how difficult it is to install IPv6.

</comment><comment><USER26> I have been running Linux boxes for 20 years as my home router, but just recently bought a Cisco RV325. Sort of got tired of maintaining it, and it took allot more power.

How will these smaller, embedded motherboards handle 1G Ethernet? Will be getting google fiber within next year.

<comment><USER11> Been using an APU board since they came out and I have a 1Gbps fibre connection. Unfortunately my measured speed comes to about 500/700Mbit but I belive that's due to either shitty equipment in the city wide fibre grid or my own switches/cables. I'm not really a network tech.

Either way the APU handles it fine for a home network and generates no noticeable heat.

</comment></comment><comment><USER27> Curious if the 120 GB SSD could be doubled as network storage acessible through WiFi?

<comment><USER28> Yep. Just install the samba/nfs/iscsi/other networking daemon and off you go. About the only thing to keep in mind is that you are going to need to configure the daemon before you use it to ensure that it's not listening on your WAN address, because you probably don't wanna have your files visible to all and sundry.

</comment></comment><comment><USER29> I'd like to play with Linux on this multi-nic board when I get some time and money [http://www.banana-pi.org/r1.html](http://www.banana-pi.org/r1.html)

</comment><comment><USER30> I use MikroTik for the nice hardware, low power, and RouterOS.

<comment><USER31> I bought MikroTik for the netflow feature, which can reveal active malware via hardware packet counters by endpoint.[0] The next cheapest router with netflow is in the thousands of dollars. MikroTik is $180 at Amazon.

[0] [http://www.irongeek.com/i.php?page=videos/houseccon2015/t302...](http://www.irongeek.com/i.php?page=videos/houseccon2015/t302-the- fox-is-in-the-henhouse-detecting-a-breach-before-the-damage-is-done-josh- sokol)

<comment><USER32> Which one do you have? Mine is an RB751U-2HnD and it is plagued with problems, to the point that I set-up an automatic reboot every other day and I'm thinking about switching to Ubiquiti gear...

<comment><USER33> A client of mine has a RouterBOARD 1100 X2 AH that I also decided to reboot daily for stability reasons. A RouterBOARD 951G 2HnD used in a branch office lost all IPSec configuration and hangs when viewing the settings on the command line. Certified MikroTik technician says the thing has to be reformatted and set up from scratch.

My first experience with MikroTik products. Not good.

<comment><USER34> Just to chime in with another anecdote, I've got several MikroTik RB493G's acting as PPPoE access concentrators that are stable as hell and have been in service for a couple of years with no issues.

In general, I've had better luck with the lower end of their product line.

</comment></comment></comment></comment></comment><comment><USER35> Looks like he hasn't hit the ip_conntrack_tcp_be_liberal problem/setting yet. Good luck with streaming Netflix with that router...

<comment><USER36> Care to explain?

<comment><USER35> Here we go: [https://www.pitt- pladdy.com/blog/_20091125-185551_0000_Linux...](https://www.pitt- pladdy.com/blog/_20091125-185551_0000_Linux_Netfilter_and_Window_Scaling/)

<comment><USER36> Anything recent? That is from 2009.

<comment><USER35> ip_conntrack_tcp_be_liberal is still in kernel sources and it is enabled in distributions like openwrt. The author does not mention this, so very likely that his/her custom router will drop traffic. Recently I ran into the issue with Netflix traffic which seemed to use window scaling. In my case I did not disable scaling, but had to enable this option on arch Linux.

</comment></comment></comment></comment></comment><comment><USER37> I have a lot of respect for those who know iptables well enough to make things like this. It looks so fascinating, but so complex.

<comment><USER38> Me too! As others in the thread have mentioned, OpenBSD and pf make for a (IMHO) much easier configuration. Not sure what kind of difference in performance one might expect. I suspect both Linux and OpenBSD are more than capable of keeping up with any traffic one might throw at such a router.

<comment><USER39> BSD is at a huge disadvantage in QoS capability, to the point that it really shouldn't be recommended as the OS for the gateway on a typical low-speed bufferbloated residential ADSL or DOCSIS connection.

<comment><USER40> I don't see any disadvantages of BSD for QoS.

I believe the syntax for writing QoS on FreeBSD and OpenBSD provides very good expressive capability [1]. By using tagging [2], one can assign QoS priority to anything that a firewall rule can define.

Having used FreeBSD QoS on dial-up, ISDN, DSL and cable over the years, it is this expressiveness that is one of the reasons I prefer the pf packet filter and thus BSD.

Here's an example for bandwidth limited wan. Interactive ssh sessions get a queue with a minimum bandwidth; scp and sftp bulk transfers go to a separate queue.

      queue rootq on em0 bandwidth 100M max 100M
      queue ssh parent rootq bandwidth 20M
      queue  ssh_interactive parent ssh bandwidth 10M min 5M
      queue  ssh_bulk parent ssh bandwidth 10M
      queue std parent rootq bandwidth 20M default

      block return out on em0 inet all set queue std
      pass out on em0 inet proto tcp from any to any port 22 set queue(ssh_bulk, ssh_interactive)

[1] PF - Packet Queueing and Prioritization [http://www.openbsd.org/faq/pf/queueing.html](http://www.openbsd.org/faq/pf/queueing.html)

[2] PF - Packet Tagging (Policy Filtering) [http://www.openbsd.org/faq/pf/tagging.html](http://www.openbsd.org/faq/pf/tagging.html)

<comment><USER39> OpenBSD doesn't support any form of Active Queue Management or fair queuing. It _used to_ support an old and mostly useless AQM (RED) but dropped support in 5.6. Now there isn't even a way to apply ECN marking.

A hierarchy of dumb FIFO queues does not make a real QoS system. It can produce reasonable-looking benchmark numbers when the prioritization rules and benchmark are contrived to match, but in the face of real-world traffic that uses HTTP to carry vastly different kinds of traffic over links that don't have constant bandwidth and latency, OpenBSD is hopeless. Even the rate- limiting capability that OpenBSD has is rudimentary and lacks the ability to account for per-packet overhead and framing overhead, which is necessary for accurate traffic shaping on common service types like ADSL.

If you say you don't see any disadvantages for OpenBSD on QoS, then your idea of QoS is twenty years out of date.

<comment><USER40> FreeBSD and PFSense do currently have ECN. I should have limited my comments to them.

You're right, OpenBSD no longer has ECN, and my comments regarding OpenBSD were out of date.

And yes, ECN is far more effective because it provides feedback to the sender.

<comment><USER39> Yes, they've got ECN capability, if you're satisfied with RED as your AQM. pfSense apparently also has an implementation of plain CoDel. Work is also underway to bring fair queuing to the BSD world so that networks with more than one simultaneous traffic flow can get some good QoS: [http://caia.swin.edu.au/freebsd/aqm/](http://caia.swin.edu.au/freebsd/aqm/)

All of the BSDs are still way behind; the best by only a few years, while others are stuck in the 1990s.

<comment><USER40> I believe you can use ECN with CoDel. None of this is restricted to a single flow.

<comment><USER39> CoDel doesn't do fair queuing so its performance on a mixed stream of traffic is much worse than fq_codel and other AQMs that have a fair queuing component. Please, read _some_ of the literature on what's been developed in recent years. You don't seem to know anything about this subject other than having seen some of these terms show up in pfSense configuration pages and thinking it means it's at parity with Linux.

<comment><USER40> Perhaps it would be helpful for you to cite some of this literature.

<comment><USER39> I doubt it would help to point you to anything specific, since you've managed to not find for yourself any of the wealth of relevant information that's been written over the past several years, from draft RFCs to LWN articles and Wikipedia articles to blog and mailing list discussions by internet luminaries and even some of their comments in recent HN discussions. At some point you have to actually put in the minimal effort of googling a topic and skimming at least one of the results, instead of assuming that the people around you are making stupid and obvious mistakes.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER41> Going back to the first article: it was around $300. But as a project for my home, the lack of wifi is more frustrating.

<comment><USER42> realistically, what is the best option for adding wireless networking to this or any other setup based on a generic box? i assume one can build an access point with a typical wifi dongle but i am not aware of any of the software means required.

<comment><USER10> Realistically, WiFi driver code sucks, and there is inherent tension between routing, which benefits from wires, and WiFi, which benefits from open space. See, for example, how the Google OnHubs have the minimum of just 1 LAN port.

You can add WiFi to a generic box, but the Ars Technica staff are promoting a consensus that it's better to use a separate access point that is designed to be good as an access point.

[http://arstechnica.com/gadgets/2015/10/review-ubiquiti- unifi...](http://arstechnica.com/gadgets/2015/10/review-ubiquiti-unifi-made- me-realize-how-terrible-consumer-wi-fi-gear-is/)</comment></comment></comment>


Trump and Brexit: why it's again Not the economy, stupid - <AUTHOR> http://blogs.lse.ac.uk/politicsandpolicy/trump-and-brexit-why-its-again-not-the-economy-stupid/ <comment><USER2> I think the link he shows between supporting Trump and favouring authoritarianism is tenuous but I can't spot any obvious flow in the main thesis - most Trump supporters are doing well economically but feel alienated culture-wise. Also, important to note but not pointed out by the author is that the support for Brexit was highest in areas that had seen a large percentage change (1% -> 2%) in immigrant population, not necessarily a large change (10% -> 14%).</comment>


Is there a shortage of engineers or a glut? - <AUTHOR> http://wadhwa.com/blog/2010/11/22/is-there-a-shortage-of-engineers-or-a-glut-please-comment-i-may-write-an-article-about-this/ <comment><USER2> Easy...

There's lots of people with IT degrees, and lots of people claiming to know how to program.

The number of those you'd actually want to hire is an order of magnitude smaller.

</comment><comment><USER3> One of the companies I work with has been trying to staff up so I have been doing some interviewing. The company's needs are relatively modest: demonstrable skill in a couple of production programming languages (C++ and Java, say), demonstrable ability to design programs and data structures when given a specification, a basic knowledge of algorithms and performance analysis, an understanding of production operating systems that goes beyond having used an editor running on a Unix system editing, compiling, and then running a "hello world" program, demonstrable skills in one or more real-world application domains, some basic knowledge of hardware, an reasonable understanding of parallelism and concurrency, some basic knowledge of networking, and so forth. It is very hard to find candidates who really can demonstrate a level of skill commensurate with the _great_ credentials they provide in their resume.

</comment><comment><USER4> Pro baseball scouts might claim there to be a shortage of promising recruits, but there are many young, ball-playing men in America who would say that an entry-level baseball "job" is hard to come by. Who's right?

</comment><comment><USER5> Lots of people don't practice in the areas they were educated in.</comment>


Impending kOS - <AUTHOR> http://archive.vector.org.uk/art10501320 <comment><USER2> >Kdb+ has sharp elbows.

No shit. I used to work as a quant, and while I was an okay quant and mediocre trader at best, I survived for three years in the industry because of my kdb+ proficiency: the firm I was at spent a couple of million dollars on kdb+ only to find out that most people could not wrap their heads around kdb+ let alone debug it effectively.

My (former) colleagues were definitely smart people. In many ways, they were way smarter than myself. But I somehow could get a much better handle of kdb+'s idiosyncrasies, and my ability to stare at dense k/q code (usually no more than a dozen lines) and figure out what's wrong with it earned me the reputation as the "q guy" \- and some level of job security.

The firm eventually phased out kdb+ completely after my boss and I left (the two proponents of kdb+).

<comment><USER3> I've worked in many shops that use kdb+ and the ones who really benefit are the ones who bothered to get some training on it rather than those who just assume they'll wing it somehow. Kx themselves have been running great intro workshops for a couple of years now. Some guys at the next desk attended one and came back buzzing with excitement at how they now saw through the noise. So the take away is - if you didn't bother to learn it, stop complaining that you don't understand it. It's not difficult to get when it is explained well, but you're not gonna get it just by staring at it.

<comment><USER2> >I've worked in many shops that use kdb+ and the ones who really benefit are the ones who bothered to get some training on it rather than those who just assume they'll wing it somehow.

Yea, I know all about the training and First Derivative. My employer also hired them.

In their defense, every First Derivative KDB+ consultant that I worked with was very sharp and an excellent teacher. They really knew their stuff, and First Derivative is no small part of what has made KDB+ so successful. However, even with their excellent pedagogy, most of my co-workers were totally lost/weren't willing to apply themselves to learn q/kdb+ well.

Here is another way to think about it: many people can't ever get their heads around certain conceptually difficult topics, say, measure theory or quantum physics. I don't think kdb+ is nearly as hard, but it seemed that way looking at my peers who were no slowpokes.

</comment></comment></comment><comment><USER4> The last line of TFA reads like the beginning of some sort of movie in the drama/thriller category. "kOS is coming. Nothing will be the same afterwards."

That's what irritated me most.

What I'd like to understand is - what led the author to this particular conclusion? Is it the fact that this language is super expressive and concise? Is it that it routinely [1] outperforms its C counterparts even if it ultimately translates to C? Is the Z graphical interface so superior that it'll blow the pants off Cocoa and Quartz and X.org or Wayland or what have you? Why would one rewrite emacs or vim on it? I don't want some basic 4 line text editor - I would like to be _productive_. Why would Mozilla spend energy porting firefox to it? Or Google, chrome? Or bash?

Simply talking about the history of K/kdb+ and how brilliant its creator is _simply_ doesn't help the reader understand why they should be excited about it. If that was the intention of this article, then the real points to make should've started _after_ that line.

That would've been much more interesting.

[1] - No pun intended, of course

<comment><USER5> What you're actually seeing is testimony: people saying they are seeing something amazing, and they aren't very good at explaining what they saw.

Btw: k doesn't translate to C. It's actually a quite simple interpreter. The fact that it outperforms other languages so easily _should_ be saying more about those languages than it should be saying anything about k.

<comment><USER6> So would k be even faster if it were compiled to machine code rather than being interpreted? Would that involve an unacceptable speed versus space trade-off? Or am I missing some crucial reason why k code has to be interpreted?

<comment><USER7> According to wikipedia's uncited 'performance characteristics' section of their page on K (programming language):

> The small size of the interpreter and compact syntax of the language makes > it possible for K applications to fit entirely within the level 1 cache of > the processor.

Sounds like the overhead's acceptable?

.

Edit: the following page (2002) says

> Even though K is an interpreted language, the source code is somewhat > compiled internally, but not into abstract machine code like Java or Python. > The interpreter can be invoked interactively or non-interactively, and you > can also compile a source file down to a binary file for product > distribution, if you wish.

[http://www.kuro5hin.org/story/2002/11/14/22741/791](http://www.kuro5hin.org/story/2002/11/14/22741/791)

<comment><USER8> Thanks for the link to kuro5hin btw - it's a nice intro to k but also I've not been on k5 for years, I had no idea it was still going :)

</comment></comment></comment></comment></comment><comment><USER9> The same story told again -- it's aim is to generate this magical atmosphere around a fast db engine and language that's deliberately obfuscated to make people who work in it feel smart, so they try to spread that it's the best. KDB/Q is a nice tool, not the holy grail how they put it. And it's not fast because they know something better - it's because it lacks almost any safety measure.

<comment><USER2> I totally agree. That's why I wrote a parser for the language to help me and my coworkers debug =) [https://github.com/kiyoto/ungod](https://github.com/kiyoto/ungod)

As I commented above, the syntax is horrible. But I still think it is an effective tool for certain problems.

</comment></comment><comment><USER10> Here's the text editor they're talking about: [http://www.kparc.com/edit.k](http://www.kparc.com/edit.k)

The code is, well, not the easiest to understand.

<comment><USER11> That's an understatement. If we are allowed code like this, I can write an editor in ONE line of JavaScript. :)

<comment><USER12> Note that his lines are < 60 chars.

Can you write this editor in one line of less than 6000 chars of javascript (without a "textarea")?

<comment><USER13> It does not matter. This is not a programming language; languages are designed for human comprehension. Human comprehension is an irreducible part of the whole paradigm of programming, being necessary to understand the _nearly always_ complex logic required to describe anything of value.

This is gibberish. It fails at the one task of a programming language: to translate something a human can understand into something a computer can understand, with as little confusion as possible.

<comment><USER12> No, it is just not a language for the masses who have gotten used to all programming language syntax being minor variations on the Algol/Pascal/C theme.

Much like math and physics, this is an extremely terse notation that is immensely useful to those who practice it, and looks like gibberish to those who have not attempted to study it. You could give the same comment about an article in any area of math whose notation you are unfamiliar with -- and you would be just as equally wrong.

</comment></comment></comment></comment></comment><comment><USER14> The APL family was developed to think about math and linear algebra in particular. Iverson's "Notation as a tool of thought" f'rinstance: [http://www.jsoftware.com/papers/tot.htm](http://www.jsoftware.com/papers/tot.htm)

If you've worked with such things for your day job, exposure to an APL language is mind blowing in the same way as exposure to Lisp is. You'll rapidly find out that an awful lot of the numerics world is an ad-hoc reinvention of an APL language. Leading thinkers in the numerics world have noticed. Have a look at the Tensor type inside Torch7, or -idx- class in Lush (the same thing): they are, in fact, a sort of APL with a more conventional, aka painfully wordy, notation.

Writing an editor in an array language seems crazy, but then, writing a parallel processing system in a language that was designed to run applications in your web browser also seems crazy. If people had stuck with APL style languages, well, databases, particularly distributed databases (Kx and 1010, both K based systems, scale to Pentascale, and have for a long time), would suck less, as would CUDA programming. Their revival could make life easier in these problem domains.

</comment><comment><USER15> Googled around, Kuro5hin (that's a name I haven't seen for some time) has a tutorial for K from 2002: [http://www.kuro5hin.org/story/2002/11/14/22741/791](http://www.kuro5hin.org/story/2002/11/14/22741/791)

The download link at [http://www.kparc.com/](http://www.kparc.com/) asks for password, so I'm not sure whats going on with that.

<comment><USER16> You have to have a personal invite to download the code atm. The whole article is telling you it is not available yet, but "if coming".

<comment><USER17> I may be mistaken but I believe zokier is referring to a download link for k whereas you are referring to a download for kOS. No?

<comment><USER18> Right. The part of the page that says "download (then $chmod +x k)" and linking to [http://kparc.com/download/k](http://kparc.com/download/k) is password-protected. I just want to follow along some of this tutorial material. I'll try to do this with Kona.

</comment></comment></comment></comment><comment><USER19> The language looks fascinating. Check out Kona, an open source implementation: [https://github.com/kevinlawler/kona](https://github.com/kevinlawler/kona)

</comment><comment><USER20> Interesting article. Found this ([http://queue.acm.org/detail.cfm?id=1531242](http://queue.acm.org/detail.cfm?id=1531242), submitted here [https://news.ycombinator.com/item?id=8476120](https://news.ycombinator.com/item?id=8476120)) interview with Arthur Whitney (from 2009) which is also really interesting.

</comment><comment><USER21> > "It is a lot easier to find your errors in four lines of code than in four > hundred."

Looking at his code on [http://www.kparc.com/edit.k](http://www.kparc.com/edit.k) I'd like to disagree with that statement

<comment><USER22> If I posted four lines of Chinese or Sanskrit, it's likely that native English speakers would disagree that they had much meaning either. However, this doesn't mean that those lines are inherently devoid of meaning or difficult to parse.

<comment><USER23> That is a fun experiment. I studied linguistics in college, and I do not think anyone ever discussed textual density of different languages with the "same" content (the latter part would be its own terrifying chestnut; if you have not studied machine translation and semantic eval and good luck ever confirming such a statement).

I studied Arabic a lot, and Chinese about a year. I cannot speak to Chinese with only one hazy year under my belt, but I can speak to Arabic.

Because Arabic has lots of syntax realized at the morpholgical level, you can encode a whole sentence (subject (with declension inherent and gender variable, verb conjugated (to passive/active, past/present/future, standard/subjunctive) and direct object (declension inherent and gender variable) _all in one word_ as we know the in English.

أضربه (A-dr-b-u; a (I) dr-b (hit) u (him/it): I hit him (present tense)

And that is a super simple example. I have seen much more compicated setences in one word, and even better in two or three. So, I hypothesized Arabic is very, very dense. I think and Russian and others could be considered similar.

However, with this level of density (maybe we argue "compression" from a CS perspective) I noticed books and their translation were routinely about the same length in pages. Never identical mind you, but never something crazy like 50 pages more (I am guessing; it has been a long time since I made such an experiment and would have trouble agreeing with someone on what is significant).

Now, one could hypothesize a shitload about what this means, but computation is realized as the same "stuff" (machine code instructions) in programming languages, where no parallel exists in human language for mapping human language to computaion, as far as I know from my between minor and major courseload in linguistics, specifically computational linguistics. If someone can contradict me, I would LOVE to read about measured cognition and language constructs.

<comment><USER24> It's important to separate spoken information density from written information density. Some languages win at one while losing at the other. Your arabic example was shorter than the equivalent english on paper, but longer when spoken (4 syllables vs 3).

In terms of information density per syllable, mandarin wins, with english coming in a close second. When speaking, english usually has more syllables per unit time than mandarin, so english has the highest spoken information density of any language. Japanese is the on the opposite end of the spectrum. Despite having the highest syllabic rate, it has the lowest information density.[1]

For written information density, logographic languages win. This is pretty obvious if you've seen a Chinese or Japanese translation of something familiar, such as a Harry Potter book. They're ludicrously thin.

1\. See the figures at the end of this paper: [http://www.ddl.ish- lyon.cnrs.fr/fulltext/pellegrino/Pellegri...](http://www.ddl.ish- lyon.cnrs.fr/fulltext/pellegrino/Pellegrino_to%20appear_Language.pdf)

<comment><USER25> > When speaking, english usually has more syllables per unit time than > mandarin, so english has the highest spoken information density of any > language.

Of the seven languages in the study, using 20 specific short texts, that were originally written in English then translated (well?) in other languages.

<comment><USER24> They recognized this issue and accounted for it. From the paper:

 _Since the texts were not explicitly designed for detailed cross-language comparison, they exhibit a rather large variation in length. For instance, the lengths of the 20 English texts range from 62 to 104 syllables. To deal with this variation, each text was matched with its translation in an eighth language, Vietnamese (VI), different from the seven languages of the corpus. This external point of reference was used to normalize the parameters for each text in each language and consequently to facilitate the interpretation by comparison with a mostly isolating language (see below)._

It shouldn't be particularly surprising that english comes out ahead. It has a huge vocabulary, tons of phonemes, and makes many parts of speech optional. It lacks tones, but would probably have to sacrifice some phonemes to stay comprehensible.

<comment><USER25> That just deals with the variation in length of the texts, not the effect of translation quality or other possible problems with the experiment, like written -> spoken conversion.

</comment></comment></comment></comment></comment></comment></comment><comment><USER26> There's a contradiction I always see in these pieces: they talk a lot about the importance of using the right data structure. But these languages get their incredible conciseness by not giving you any choice about your data structures; their array type is hardcoded into the language, and if you want to use something else then your code balloons.

<comment><USER12> K basically has 3 data shapes:

atom (int, float, char, date, symbol, ...)

list (one dimensional array of atoms, dicts, flips or lists)

dict (a map from one list to another)

There's also a flip, which exchanges the first two indexes applied to an item (so, e.g., it effectively transposes a list of lists) but it is just sugar (both syntactic and semantic).

You can trust Whitney that all of these are properly implemented, including appends.

It's not often that you actually need more. I've discovered this after using K for a while, and going back to python.

Back in my pre-K (ha!) C++ and Python day, I had an awful lot of classes everywhere. After using K for a while, my Python and C both have much much fewer (structs in C more often than python, as C is missing python's dict). And the code has gotten much shorter and more efficient. Arguably, more readable as well. And I've essentially dropped C++ for C, because the extra complexity is just not worth it.

</comment></comment><comment> ah- k/q really doesn't have to be this unreadable, that's just Arthurs style. Here's some code in C by him for comparison: [http://kx.com/q/cs107/a.c](http://kx.com/q/cs107/a.c)

<comment><USER27> I tried cleaning it up a bit: [https://gist.github.com/lukechampine/f54fce8fd756254cefb2](https://gist.github.com/lukechampine/f54fce8fd756254cefb2)

But the actual meaning of the program is still lost on me. I can only guess it has something to do with parsing files (note the checks for curly braces). Feeding it its own source code produces some output, but I have no idea what it actually modified.

<comment><USER28> It's a solution to CS107 assignment 1 : [http://web.stanford.edu/class/cs107/assign1.html](http://web.stanford.edu/class/cs107/assign1.html)

<comment><USER27> Guess I could have just looked at the URL, huh. Well, it was fun trying to reverse Whitney's code anyhow.

Humorously, it looks like this code would have received a poor grade. It meets just about every standard for low quality outlined here: [http://web.stanford.edu/class/cs107/landmarks.html](http://web.stanford.edu/class/cs107/landmarks.html), particularly "Fast code which doesn't work quite right." (Due to an off-by-one error, this code fails to properly reconstruct the example text.)

<comment><USER29> > Due to an off-by-one error, this code fails to properly reconstruct the > example text

What's your example? It seems to work fine to me on "{all is well}{ell that en}{hat end}{t ends well}"

</comment></comment></comment></comment></comment><comment><USER30> I love K/Q and is using it in my startup. Thanks a lot for Kx's recent freeing up the 32bit version. To use APL-like languages I have to really shift the way of designing/modeling things. Most importantly K may not be best for lots of developers working on the same thing. Object oriented languages will fit better in that case. K projects often only involve one or two developers who model things in vector thinking (column based thinking in data domain), know exactly what to do and how to do it. Vector thinking is not suitable for all problems, but works really well if it does. Btw, new 3.2 version of kdb appears to be even faster than before. It also improves websocket integration and JSON data conversion. Very nice to integrate with nodejs/Qt.

We also use Forth which I think is a really another way of shifting mind.

<comment><USER31> What kind of computation is done at your startup? I've used k/q for trading, graph analytics and computer vision (scene reconstruction).

</comment></comment><comment><USER32> I seriously thought that the article, and some of the code examples people are posting [0] was a joke! I need to go rethink my career.

[0] [http://www.kparc.com/$/edit.k](http://www.kparc.com/$/edit.k)

<comment><USER33> That's certainly good - we should learn languages which are different enough to justify learning.

Your code in whatever language you work will benefit from that rethinking.

</comment></comment><comment><USER34> So K is a general purpose programming language? If the claims are true, why don't they submit some entries to the Computer Language Benchmarks Game?

<comment><USER35> It's proprietary.

<comment><USER36> <sarcasm>Yeah, that's why.</sarcasm>

<comment><USER37> Perhaps they asked if their K programs were wanted:

[http://benchmarksgame.alioth.debian.org/play.html#languagex](http://benchmarksgame.alioth.debian.org/play.html#languagex)

</comment></comment></comment></comment><comment><USER38> Does a good formal introduction exist for K, or Q, or APL, or J, or any other languages in this family? Something with, you know, a syntax definition at least, and any kind of formal definition of the semantics.

The closest I could find is this [1] but "The model is expressed in SHARP APL", so from the start, it's circular.

[1] [http://www.jsoftware.com/papers/APLSyntaxSemantics.htm](http://www.jsoftware.com/papers/APLSyntaxSemantics.htm)

<comment><USER39> It's not formal, but it is helpful to watch these series of videos by Martin Saurer on J:

[https://www.youtube.com/watch?v=VSJpJt3c11c](https://www.youtube.com/watch?v=VSJpJt3c11c)

It goes form solving some Euler problems to a full-blown web app in J.

</comment></comment><comment><USER15> I wonder what K would look like with bit reader-friendly syntax. I tried running a program that supposedly "will take a K expression and produce its English translation" ([http://kx.com/a/k/examples/read.k](http://kx.com/a/k/examples/read.k)), but either it doesn't work with kdb+/q or I can't figure out how to use it. Does anyone have some example output, or advise?

</comment><comment><USER40> A language/kernel/db/ui to watch. Reading the comments here, it seems the language separates the wheat from the chaff; the average brogrammer won't he able to handle this, but many of us are very interested in exploring.

An OS this small is incredibly exciting to me.

</comment><comment><USER41> Worked at Morgan Stanley's fixed income desk many years ago as my first job after grad school, programming A+. It was a decidedly acquired taste. The language and style were utterly alien to anyone coming from a background in C (and at that time an early version of Java.) While A+ was fast for manipulating arrays (much of what the trading floor needed), it seemed that doing GUI development with A+ was really pointless. The speed of the language didn't matter in GUI interactions, and it was very hard do understand other people's code.

</comment><comment><USER42> Um, so what happened?

<comment><USER5> It's not done yet.

This summer, Pierre and I got kOS to boot directly into g (the graphical interface; formally called z) with ISR, keymap, modesetting, basic filesystem, etc weighing in around 100 lines of C. That was pretty exciting. Could probably be done with less with some deeper changes to Arthur's code, but it's still very useful to run k under Linux. Oleg made a silly little game in kOS.

Arthur and Oleg did some performance benchmarks staging k against q (current kdb+), Postgres, some "popular RDBMS" (that I can't name), and MongoDB. It was impressive that k is so much faster than q, but it also really underscores the cost of the wrong data structure (and how hard it is to get the right one with SQL or MongoDB).

<comment><USER43> How much are you planning to opensource?

I realize you have a thriving commercial software company and that's cool. But...wow. This is exactly the sort of thing Alan Kay's team has been working on for the past five years, and you guys seem to be beating them to it, with a completely different approach. It would be pretty amazing to be able to dig into it, find out how the whole system works, and contribute.

<comment><USER5> I'm not actually employed by kx, so I can't say for certain.

I don't think Arthur's opposed to open sourcing bits though.

<comment><USER44> Would it run under Kona? [1]

1\. [https://github.com/kevinlawler/kona](https://github.com/kevinlawler/kona)

<comment><USER5> No. A lot of the verbs have changed in k.

<comment><USER45> so would it work if Kona changed?

<comment><USER14> Kona is K3. K5, the subject of the article, is being invented now. I'm quite certain there is no plan to make Kona K5 compatible.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER46> So, I read about the fast interpreter and small language. How do something like this?

"Whitney's strategy was to implement a core of the language – including the bits everyone thought most difficult, the operators and nested arrays – and use that to implement the rest of the language. The core was to be written in self-expanding C. As far as I know, the kdb+ interpreter is built the same way.

Unlike the tall skinny C programs in the textbooks, the code for this interpreter spills sideways across the page. It certainly doesn't look like C."

This mean that the code is very unreadable C? Like a kind of code-golf?

How replicate it for built a speedy interpreter? And what if I use lua or python instead?

<comment><USER47> There are two opposite schools of readable C.

The mainstream says: readable C has function and variable and type names that express meaning, so a function is read like a narrative with verbs, adjectives and nouns. The fact that this narrative scrolls over pages, is unimportant.

The APL/K/J school says: readable C has functions, variables, and types named with single letters, so that the totality of a function is short enough to fit in _one glance_ \- preferably, on one line that does not need a scrollbar. The function does exactly what it says, no more and no less; its intent is thus completely clear. To name it descriptively would ruin the ability to grasp the whole thing as a gestalt.

<comment><USER46> Ok, so APL/K/J is for people that read compressed code..

But that how relate to how build a fast interpreter? Faster than C?

</comment></comment></comment><comment><USER48> I hate to love KDB because its a very expensive closed platform. But if you understand some of the concepts and how easy it is to achieve those concepts with a few lines of q code you can do some brilliant things. Yes, KDB is a great data store and provides very quick methods for crunching that data with its vector-based approach.

However, whats really impressed me with KDB is that you can do so much more with it. In some banks it has effectively become the messaging middleware for connecting hundreds of disparate data sources. In addition to passing messages you get the data storage and analytics tools for free...

</comment><comment><USER49> This seems to be a very fun language to write in, in the same way optimizing assembly code in the ancient times was fun; the age of skilled artisans. Unfortunately (for programmers - great for the rest) now is the time of a factory worker. Due to that, I don't think this language and its platform has a long future.

Its continued survival is rather a testament to excellency of marketing (as evidenced by this article!) rather than actual merits.

<comment><USER33> I think the opposite :) - marketing of R is light years ahead. Language is very obscure, but so good that from time to time you're going to hear wonderful stories from new enlightened.

And with the rise of parallel programming we're going to actually switch more to more vector way of describing and solving problems. Just like Lisp ideas are spreading everywhere in modern languages, APL ideas are also fruitful.

</comment></comment><comment><USER50> I'm interested in a kOS with a "god says ..." program built in ...

<comment><USER51> get Terry Davis on the job. Is he opposed to porting his holy code to other esoteric platforms?

</comment></comment><comment><USER52> > kOS is coming. Nothing will be the same afterwards.

There seems to be this strange idea going around that if we just get the right tool, everything else is going to change forever. I see this a lot with people trying to create IDEs that let non-programmers create programs without really knowing how to code.

But the thing is, most people just don't have anything worth coding. The problem isn't that the tools don't exist. They do, even if they aren't perfect. It's that making something that matters isn't an easy thing to do. And no tool can change that.

<comment><USER5> If I am from another planet, and say I don't know why programs are so big and slow and buggy, and the most complicated program you see I've produced is a glorified calculator, it's too easy to be patronising and say _well, that 's because you haven't done anything complicated_.

However if I then show you a programming language, a database engine (similar in capability to SQL but around 1000x faster), a graphical desktop with icons, mouse, editors, filesystems, ISR, and so on- and it's still under 400 lines of C code, then maybe it's easier to have this conversation that I think we need to have: That maybe _all you programmers_ have simply been doing it wrong, and there's something fundamentally wrong with the way you and _everyone else_ programs computers.

I think most programmers take so much offence from _the thesis_ : that everyone programs wrongly (or badly) and that a fundamental shift in the way we program could make programming much better, that it's been difficult to actually work on this problem. Just look at all the people who are complaining about the text editor being difficult to read without saying _I can 't read this, and I want to get better_.

And I think it should be obvious: You've built bridges for thousands of years, so you expect you're pretty good at it now, and yet there are still improvements in bridgemaking today; why do you expect programming to be any different? Why do you have "best practices" for programming if you've only been doing it for a few decades and don't really know how to do it yet?

Maybe it's much easier to think we're working on giving non-programmers the ability to program, but we're not: we're trying to make programming suck less.

<comment><USER52> >And I think it should be obvious: You've built bridges for thousands of years, so you expect you're pretty good at it now, and yet there are still improvements in bridgemaking today

Right, and I'm not saying programming won't improve. It obviously has. No one wants to write a web app in C++ or assembly. What I'm specifically taking issue with is the idea that there is some sort of monumental change out there that is going to enable us to do... what exactly?... well no one can really answer that, but the claim is that it is big and exciting and will change everything.

<comment><USER5> When I can fit my program on the screen, I don't make any mistakes.

I think this is true of most programmers.

While "hello world" type programs tend to be the pedagogical example, kOS demonstrates that the complexity of such a one-screen low-defect program is much higher than people previously thought.

<comment><USER53> Not being facetious, but does this extend to using larger screens, multiple screens, wider columns or multiple windows, and smaller fonts? Or was one screen just meant as an estimate for 200 lines or so?

<comment><USER5> I think it is the act of scrolling or window-switching or head-moving that is the cost. To that end, I find using a MBA screen to be perfectly adequate for programming because it fits entirely in my field of view.

My screen is about 55 lines tall and maybe 170 characters wide. I sometimes shrink or increase the font slightly to make programs fit on screen.

</comment></comment></comment></comment></comment></comment><comment><USER54> Sounds great! Where's the documentation on IPC / mutexes / threading?

> todo

> files, procs, tcp/ip, usb, ..

Oh.

<comment><USER31> E.g. - (k4/q) [http://code.kx.com/wiki/Reference/hopen](http://code.kx.com/wiki/Reference/hopen) shows how to open a file/proc.

The rest of the wiki is quite useful including a references and tutorials for q.

k5 uses operators instead of words like q.

Btw, being able map/reduce w/ 1000 procs on a 8GB Linux vm (using k5) is both useful and fun.

</comment></comment><comment><USER55> Kind of curious to play around with that text editor. Any chance of K running on OSX?

<comment> ah- You can get kdb+ for free from here: [http://kx.com/software- download.php](http://kx.com/software-download.php)

However, they've been working on a new version of k that's not publicly available yet and I suppose the kparc stuff requires that.

</comment></comment><comment><USER56> Hey Geo, if you read this, congrats :)

Tristan

</comment><comment><USER57> joke: search jqk on google images

</comment><comment><USER58> I don't buy this story. Maybe kdb is fast and great, but the article attempts to describe it as a work of a genius, better than anything else because it's 100 lines of code, doing its own memory management and running on bare metal. But in fact many other programs do their own memory management and can run on bare metal. JVM or .Net do their own mem management, all database servers too, and many, many others. So what's left on the table? 500 lines of C code that's meant to change the world? I really doubt it (or, these guys arent using line breaks). We've read so many similar stories about Lisp and how it's one language that can do everything in 5 lines of code or how you can build an entire OS and all applications in Lisp, but where's that OS now?

<comment><USER31> What's left on the table is the efficient use of resources.

When I mentioned to a friend that the current version of K5 was a binary < 100KB, his response was "I don't believe it. I can't write HelloWorld in less than 100 KB!".

Access to more resources does not mean that one should be wasteful.

K benefits from a dedication to avoiding waste and duplication and efficient use of mathematical concepts. The Fundamental New Computing Technologies at VPRI has similar goals (an entire end-user system including "Office" apps in 40 KLOC or less).

Being able to prototype a multi-proc map/reduce algorithm in k with 1000 procs on a laptop with 8 GB RAM is quite nice.

> 500 lines of C code that's meant to change the world? I really doubt it (or, > these guys arent using line breaks). There are line breaks, undoubtedly. > That said, I'm sure the code is concise - much like k code.

K3 was 1200 lines of code and included the language, windows(GUI), database, IPC, REPL (w/ simple debugger), FFI and OS interaction. The Windows executable was 320 KB.

</comment></comment><comment><USER59> Every ten years, some company comes around and claims that their functional/data flow/columnar/meta shortish is orders of magnitude better. Usually, the one thing they have going for them is that they focus on only a small subproblem. That lets them be small. Every demo is one of no edge cases, no exceptions, and no I/O errors. (Or they cram all that into some "standard" library.)

The real challenge is that, 99% of the time, requirements and integration is what kills you, not raw performance. For the cases where performance (or formal correctness, or whatever) matters, the main challenge is usually to convince the market that it's worth paying for, and then finding the right developer project match.

<comment><USER33> This is all false for APL. First, APL is right there in the history with Fortran and Lisp - and its "every ten years" became irrelevant already when it was used to analyze IBM 360 hardware and found some bugs which were fixed in time for shipping. Second, these languages are truly for very wide ranges of problems - it's the industry curse that APL isn't used more widely; I guess the reason is it's harder to learn. But you can do can use it for really everything. Who'd thing JavaScript would be the language to write Linux emulator? So it's less wonder that k is used for OS writing.</comment></comment>


Yosemite's Visual Design - <AUTHOR> http://ignorethecode.net/blog/2014/10/25/yosemites_visual_design/

<comment><USER2> > _Yosemite's Finder is still the same basic Finder Apple shipped in the first > Mac. The world has changed around the Mac, but the Mac has remained the > same._

Well, it now also has tags, tabs, 2 new view modes and new sorting options, thumbnail sizing, Airdrop, Quicklook, and several other options besides.

If it's "stationary", that's compared to what? Windows Explorer? Nautilus? Where are those mythical updated and modern file managers that leave the Finder in the dust? Except if you mean just accretion of stuff and features, like the everything-and-the-kitchen-sink ware such as PathFinder...

And what exactly has 'changed' in the world of file management? The Cloud? Well, Apple has added that too, in the file open/save dialogs, and now with the cloud drive.

<comment><USER3> Yes, you're right, Apple has added more stuff to the Finder, but as you imply when you mention Path Finder, accretion of stuff isn't really progress.

Here are some specific things that I'm thinking about when I say that "the world has changed":

When the Mac came out, the idea was that you'd have a small number of locally stored files from a small number of applications. You'd have some text files you created in your text editor, some image files you created in your image editor, and so on.

Today, though, neither the "few files" assumption, nor the "locally stored" assumption, nor even the "files" assumption holds true.

"Few files": people have a lot of files. They have thousands of images, downloaded mp3s, a decade of documents and letters, possibly hundreds of thousands of emails, electronic bills, applications, and so on.

"Locally stored": files aren't just local anymore. People may have images on Flickr, spreadsheet documents on Google, letters on their Office account, and so on. And files should be accessible from their phones, their tablets, their computers, the web...

"Files": These things aren't even necessarily files. From the user's point of view, is an mp3 a file? Should it be shown in file system? How should something like a piece of music exist in something like the Finder? What about a friend on Facebook, or somebody from your email client's address book, would it make sense to show something like that inside the application that appears when you start your computer? If somebody sends you a file via email, would it make sense for your file browser to understand that relationship, and allow you to, say, browse files based on people?

I think all of these changes in how we use computers warrant a rethinking of how we should interact with our data.</comment></comment>


Scalable Work Queues with Beanstalk - <AUTHOR> http://www.igvita.com/2010/05/20/scalable-work-queues-with-beanstalk/

<comment><USER2> There are some tasks, such as asynchronously-generated web content, where a message queue is the obvious solution. But there are a class of problems where either a message queue or batch processing system (e.g., Hadoop) could be used. Consider the summation of user mention counts from the Twitter streaming API for an example; either a job is put on a queue for each tweet containing a mention, or each of the mentioning tweets are thrown into a bucket that is used as the source for a batch job that recurrently executes.

From what's mentioned in the article, it sounds like some of the tasks that Beanstalk is being used for at PostRank are the same type of tasks that other companies, such as FlightCaster, are doing with Hadoop/Cascading.

The trade-off seems to be that message queues are more flexible and can offer lower latency of job completion but batch processing systems provide better support for admin concerns like adding worker nodes, debugging, and reporting.

<comment><AUTHOR> "From what's mentioned in the article, it sounds like some of the tasks that Beanstalk is being used for at PostRank are the same type of tasks that other companies, such as FlightCaster, are doing with Hadoop/Cascading."

Hmm, not at all. A message queue and a job queue are not necessarily one and the same. What we need is real-time scheduling, with up to the second resolution for each job. We don't run a batch "go fetch all of these pages" jobs, rather our system is always running, always fetching content. For that, a heap/work queue is required.

<comment><USER2> I've enjoyed reading your posts over the years, thanks for them.

You're correct that a message queue isn't necessarily a job queue, I was referring to job queues.</comment></comment></comment>