Ask HN: What would you use to query large (2-25TB) of read-only data? - <AUTHOR>

The data set is written once a month in bulk, and read many times by different users.  The last month of data (~2TB) is the hotspot.  A year's worth is ~25TB.
The attributes of one record are as follows:

  * 1-4 character string
  * float
  * float
  * integer
  * integer
  * integer
  * integer
  * integer
  * 1 character
 For each 1-4 character string, there are many records -- sometimes several per second.  As an example, in the span of a month, one of these strings can be associated with 18 million records. There are about 10,000 unique 1-4 character strings, but not all as active as the previous example.  The data is queried by two attributes: 1-4 character string and timestamp.
Potential solutions I've come up with (feel free to debate any of these):

  * Put everything (or just the hotspot) in a MyISAM compressed database.
  * Put everything (or just the hotspot) in an InnoDB database with a proper clustered index.
  * Put everything (or just the hotspot) into CouchDB with proper views.
  * Put everything (or just the hotspot) into MongoDB with proper indexes.
  * Put everything (or just the hotspot) into Redis ZSETs with timestamp as SCORE and distribute across nodes.
  * Load all of the data into a long-running Hadoop job.
 Feel free to ask any questions too. <comment><USER2> It's kinda lower level, but look into Fastbit

<https://sdm.lbl.gov/fastbit/></comment>


Reflecting on my Summer at 500Startups - <AUTHOR> http://www.tawheedkader.com/2011/09/reflecting-on-my-summer-at-500startups/ <comment><USER2> I echo a lot of TK's thoughts here. Despite minor issues (turns out 500startups is still a startup too), overall it was a massively useful summer for Tinfoil Security. 500startups really helped us focus on those things that we weren't already good at, by surrounding us with people who _were_ good at those things; for us, mostly design and distribution.

<comment><USER3> ...and they also surrounded you with us beer swilling Aussies (whether you liked it or not).

<comment><USER4> The Aussies are the best part of 500!

</comment></comment></comment><comment><USER5> We're just disappointed to lose you to the West coast!

</comment><comment><USER6> +1 on the importance/usefulness of the 500 family. TK nailed it with the N:N model.

A complaint that I've heard is that 500 is too big and that Dave has spread himself too thin. I couldn't disagree more... Dave has made himself and his connections available to us far more often than I would've expected, given the fact that there are some 165 startups (I've shared a meal and/or drink with Dave 3 times in the past month, despite living in Chicago)

FeeFighters wasn't incubated at 500startups, but Dave/500 invested in us and we're a part of the family despite being based in Chicago. The email list and camaraderie are super useful to all, and getting more useful all the time... Need beta customers who will put up with a rough product? No problem. Need an expert in ___? I bet there is someone on the list who fits the need. I know there are plans afoot to even further increase the usefulness of the family.

If you are lucky enough to get an offer of investment from 500, take it. Not for the money, but for the family.

<comment><USER7> i'm not in the club, but my friends are, and everyone has been extremely gracious by association. it's a wonderful community.</comment></comment>


Frances Allen has died - <AUTHOR> https://www.ibm.com/blogs/research/2020/08/remembering-frances-allen/ <comment><USER2> Frances Allen wrote "A Catalogue of Optimizing Transformations" in 1971. 50 years(!!!) later, they are still the backbone of optimizing compilers. I think the only major thing missing is autovectorization.

She was in her 30s when she wrote it.

<comment><USER2> I think you should read it if you work on compilers, even now. But the reason is a bit different.

It was the survey of the state of the art at the time, but obviously it is not the state of the art now. Then why should you read it? Because it is written in two layers.

The first layer goes, we tried many optimization ideas, but only these were effective in practice: inlining, register allocation, etc. Others were not. Surprisingly, this layer is still mostly true today! This is both happy and sad depending on your view. Personally I think it testifies that compiler is a mature field, and it matured by 1970. (And that Frances Allen did lion's share of work maturing it.)

The second layer is, so here is how you should do inlining, register allocation, etc. While this layer is also full of gems, it is necessarily badly outdated. The paper predates graph coloring register allocation, for example. On the other hand, ironically, the state of the art 1970s algorithms are often a good choice today when you want an algorithm that is fast and low memory. (Ironic, because they were slow and high memory at the time!) This doesn't apply when there is an important new concern, for example cache locality, but happily it mostly doesn't affect compiler.

I think there should be a project to write the-state-of-the-art-in-1970s compiler. It would be a great simple and minimal alternative to GCC and LLVM, and it would also work great as a JIT code generator. We probably should name it Fran.

<comment><USER3> Maybe I should call the backend for Objective-S Fran :-)

> graph coloring register allocation

I knew some people working on linear optimization, at the time one of the most performance-intensive applications around (with real money/competitiveness in many industries riding on it). The compiler that produced the best code, by quite a bit, was the IBM FORTRAN compiler (3090 was the preferred target at the time), which also didn't do graph coloring. It just allocated the registers by loops from innermost to outermost.

¯\\_(ツ)_/¯\

For a simple compiler, we should also look at Wirth's works, and Wirth's rule: an optimization has to pay for itself, that is, the compiler compiling itself with the optimization both in the executable and in the source code has to be faster than without it in both places.

<comment><USER4> > For a simple compiler, we should also look at Wirth's works, and Wirth's > rule: an optimization has to pay for itself, that is, the compiler compiling > itself with the optimization both in the executable and in the source code > has to be faster than without it in both places.

That will result in a compiler best at optimizing programs that behave like compilers. Which may be fine for simple compilers, but it's worth keeping in mind that will not be a good compiler for domains where the programs don't behave like compilers (such as scientific computing, or even ML these days) that also want high performance.

</comment></comment></comment></comment><comment><USER5> I worked at IBM research from 1997 until 2007, after finishing my PhD in compilers and programming languages.

About two weeks after I started working there, she showed up in my office, and introduced herself. She'd heard about a new PL person joining, and she'd gone and gotten my dissertation and read it, so that she could come talk to me about it. Not that my dissertation was anything special: that's just the way that Fran was.

She was an amazing person. Brilliant, and kind, and generous. The world needs more people like her.

</comment><comment><USER6> Fran and I were on the same floor at IBM's Watson lab. I was in an AI project doing applied math, e.g., some optimization, and mathematical statistics (for the AI work we were doing, system monitoring, i.e., anomaly detection, better than our AI work!). She was regarded as a major expert in compiling and numerical codes for scientific computing.

I heard that she was working on a software product that among many other things would do fast matrix multiplication using some parallelism.

So, just for the heck of it, I wrote and ran a little routine in PL/I that used PL/I's feature of multi-tasking to get some parallelism and showed my code to her. She was a little surprised I'd written the code, had a smile, and explained why her work closer to some hardware features (I don't recall the details) would be faster!

I wasn't surprised or disappointed that my little PL/I tasking code would be slower than what she was doing, but at least I got her to explain the hardware she was using and how she was exploiting it!

As I recall, she was married to Jack Schwartz at Courant Institute of NYU and as in

Nelson Dunford and Jacob T. Schwartz, _Linear Operators_.

<comment><USER7> I'm sure you are just trying to share a memory of interacting with an illustrious colleague here - and my sympathies go to you as it's always a shock to learn of the passing of someone you worked with.

But I think you maybe need to work on how you present this anecdote - as it is, it reads like you tried to mansplain her own research to a Turing Award winner. I hope you approached with more humility than this telling suggests?

Also, you should be aware that contextualizing professional women in terms of who their husband is or what his credentials are has long been used to underplay women's individual achievements. Again, I don't think that's your intent, but you could consider whether, in the case of talking about Jacob Schwartz, you would have been moved to drop in the detail of who he was married to?

<comment><USER8> Why is there an assumption that graycat is male? Was there an indication of gender?

<comment><USER9> Not who you're responding to, but: My assumption was that the user is _the_ greycat, which is the well-known alias of the programmer Greg Wooledge.

Now that you call it out, I realize that it's probably a mistaken assumption, because Greg spells it "greycat" not "graycat". But it wasn't an unreasonable assumption for HN.

</comment></comment></comment></comment><comment><USER10> :(

Optimizing Compilers for Parallel Computers, lecture by Frances E. Allen [https://www.youtube.com/watch?v=Qv- wXcUxrmE](https://www.youtube.com/watch?v=Qv-wXcUxrmE)

Frances Allen, 2006, ACM A.M. Turing Award Lecture, "Compiling for Performance: A Personal Tour" [https://www.youtube.com/watch?v=NjoU- MjCws4](https://www.youtube.com/watch?v=NjoU-MjCws4)

<comment><USER11> She also spoke at the acm turing100 symposium in 2012.

[https://www.youtube.com/watch?v=xLTUvFboveM](https://www.youtube.com/watch?v=xLTUvFboveM)

</comment></comment><comment><USER12> Fran gave a talk when I interned at TJ Watson one summer. Her stories of the early days of compilers were beyond fascinating and made it clear how much we all now were just building on ideas they established decades ago.

Later, my wife was the first to receive the IBM PhD fellowship established in Fran's honor. Fran awarded it to her at a conference (Grace Hopper I think) and of course was gracious, offering to help as your career moved forward. Thankful for that investment in our future.

</comment><comment><USER13> I'd recommend reading her interview in Coders At Work. I never realized that compilers were already a flourishing field by the time C came around, and that C actually ended up having some negative effects in compiler dev.

<comment><USER14> It was more that C hindered optimization research. The interview was awesome as you said.

<comment><USER10> C and Unix hindered a whole lot. It was the PHP of its day.

<comment><USER15> Nice analogy. Although it's very unlikely that PHP will ever be as highly regarded as are C and Unix today.

<comment><USER16> I wanted to say that C is not really very highly regarded nowadays, but that only adds extra php-directed snark to your post, which is not wholly a bad thing.

</comment></comment></comment></comment></comment><comment><USER17> First female IBM fellow.

First female winner of the Turing Award.

Lots of other notable stuff.

Sadly, this is the first I've heard of her. Hopefully all that means is I'm not a real programmer.

Edit: To be clear, I really meant "I hope other people here are familiar with her work, even though I am not because I'm not a real programmer." I'm happy to see that some people are, in fact, familiar with her and her work.

<comment><USER18> > Hopefully all that means is I'm not a real programmer.

Sadly, no, it's not just that. Most my immediate colleagues, for example, don't know of her or her work, either. It's one heck of a field.

She had expressed some dismay, in interviews, at being the first woman to win the Turing Award. Not the Turing Award part, of course, the "first woman" part. She was far from being the first deserving candidate who didn't happen to be a dude. So I hope she wouldn't mind linking this here, even today: [https://www.hillelwayne.com/important-women-in- cs/](https://www.hillelwayne.com/important-women-in-cs/)

<comment><USER19> Yeah. If anything, I would say it means they're not a "real computer scientist," whatever that means.

Just to put the second part of your comment in perspective, there have only been 3 female winners of the Turing award, total, by my count.

<comment><USER20> _> a "real computer scientist," whatever that means_

At university we would say that computer science was the art, where programming was the craft[1]. It is probably fairer to state that programming is an application of the science.

[1] Though with less of a sneery tone than used when people said "mathematics is the art where computer science is the craft"

<comment><USER18> I think that this might be exploring the analogy's breaking point.

It can be quite easy to draw a clean line between theoreticians and implementors in other fields. Biologists and veterinarians, physicists and engineers, etc. But who was the last major computer science theoretician who wasn't also a skilled programmer or system architect (or both)? Alonzo Church?

<comment><USER21> After looking through all the Turing Award winners, my answer is Stephen Cook - a CS mathematician who did nothing on engineering, discovered the concept of NP-completeness.

</comment></comment></comment></comment></comment></comment><comment><USER22> This is absolutely worthy of a black bar.

I read one of Frances' papers on compiler optimization, and while some of it went over my head, it was still valuable information; the world is a sadder place without her.

</comment><comment><USER23> My dad should have married her. Same story except Spartans instead of Wolverines and he went into Defense instead of IBM, and he's older and still rockin' Siri on his AppleWatch.

</comment><comment><USER24> Wish we could get the black bar up.

</comment><comment><USER25> RIP

</comment><comment><USER26> I'd love to see a black banner / bar in her honor.

<comment><USER27> As an aside, when there is a black bar, I would also love it to link to the reason it is there.

<comment><USER28> That's a nice idea, like how Google doodles link to some background information.

</comment></comment></comment><comment><USER29> Going on a tangent, I think most of us missed Bill English's passing a few days ago since I think because people only noticed it on the weekend when there are less HN users active.

<comment><USER30> It was mentioned in this article: [https://news.ycombinator.com/item?id=24059231](https://news.ycombinator.com/item?id=24059231), but not in an obvious way.

</comment></comment><comment><USER31> @dang can we get a black bar in her honor?

<comment><USER32> signed.

</comment></comment><comment><USER33> Why doesn't the black bar link to the related submission?

<comment><USER34> That's a great idea

</comment></comment><comment><USER35> I'm glad we got a black bar for her.

</comment><comment><USER36> My brain read the title as "France Had Aliens" which I was really curious to read. Thanks brain.</comment>


Moxie, a Social Robot That Helps Kids with Social-Emotional Learning - <AUTHOR> https://spectrum.ieee.org/automaton/robotics/home-robots/moxie-a-social-robot-for-childhood-development <comment><USER2> Shouldn't parents be doing this? If they're not, then why did they have a kid? The robot seems well designed, inoffensive, and the company's focus is well intentioned, but I can't help thinking that this represents a complete societal failure on so many levels.

<comment><USER3> Having a robot companion as an "impartial" third party may help kids work through issues they (feel like they) can't talk with their parents about.

For better or worse, parents are human beings, and (even unconsciously) may stigmatize behaviors or feelings that a child may experience, making it difficult for the child to bring up these issues with the parents.

There is also the case where children may be neurologically atypical in a way that makes interacting with other humans difficult. This is another opportunity for a robot to shine.

You're not wrong -- helping children develop social and soft skills does fall under the job description of being a good parent. But even parents with the best intentions may need help from time to time, and I would argue that a parent who can recognize the value in something like this (as a supplement, not a replacement for parenting) has the child's best interests in mind.

<comment><USER4> Not to snap at you but the neurologically atypical suggestion brings to mind the one offensive dehumanizing trope of autistic people of thinking of them as something inhuman to be forced to act "normal" instead of taking the step one of actually trying to understand them. I have such a sense of deja-vu that it certainly doesn't sound like the first daft attempts to try to use robots to "fix" them with no real thought how the hell that would work.

<comment><USER3> Thanks for your comment, it is a nice reminder to receive.

When I wrote the comment, I was remembering a coworker whose nephew had ASD, and could not process spoken word from his family well. They discovered that using the "say" command on an OSX Terminal to make the computer "talk" was something that did work for communicating, and the boy would listen to the computer's advice for when it was time for bed or time to brush his teeth.

It was not my intention to perpetuate dehumanizing stereotypes, so I apologize if it came across this way. Rather, I was trying to emphasize the positive impact of having more parenting tools in your metaphorical tool belt.

At the risk of sounding ignorant again, I would argue that good hygiene and sleep are habits that everybody should build, regardless of your neurological disposition :)

</comment></comment></comment></comment><comment><USER5> Is it just me, or are we building technology to replace the messiness of a well-rounded, social life with a very clinical approach?

Is this a problem that needs a robot to solve?

<comment><USER6> Was thinking same. Like facebook trying to connect people which they did to an extend but also gave many spite and anxiety.

The robot feels like a bandaid solution. If a robot is required then there are deeper issues that need to be addressed here.

</comment></comment><comment><USER7> This feels very similar to the Black Mirror concept of "Ashley Too", which comes with a warning of "Ashley Too may perpetuate loneliness".

[https://www.youtube.com/watch?v=JQyzn3CWJuM](https://www.youtube.com/watch?v=JQyzn3CWJuM)

</comment><comment> tom-thistime This application space seems kind of terrifying. [https://en.wikipedia.org/wiki/Harry_Harlow](https://en.wikipedia.org/wiki/Harry_Harlow)

<comment><USER8> Holy !@#$ that's the most disturbing wikipedia entry I've ever seen. Harry Harlow is the worst

<comment> tom-thistime Yeah I was thinking let's not continue his work, especially not on human children.

</comment></comment></comment><comment><USER9> Why are we gamifying relationships?

Making friends is a mission? C'mon

I can't imagine where will this lead us.

<comment><USER10> First step towards AI/robot rights heh

Yeah I can't see kids "helped" by robots grow up that well, but I hope I'm wrong and they'd actually be better humans :/

</comment></comment><comment><USER11> Why is this technology not also offered as a mobile app? Seems like they could make it more affordable and available? Instead of a $1500 upfront cost, it could be a no-contract subscription service. It might be helpful for adults as well.

</comment><comment><USER12> I watched the video in the article. The father appears to not really engage with the child at all.

Either he is an actor for the movie, or this could be part of the problem with the child's emotional issues.

</comment><comment><USER13> I think just the marketing of this will have a small impact as well. Soooooo many people don't know how to meaningfully converse. Especially when it comes to their kids. Outside myself, I've never heard another parent add affirmation and gratitude to their language when talking to their kids. "Thank you for telling me about your day" is often "oh, that's great honey" at 1/2 attention commitment and the difference is strikingly impactful over time for the kiddo.

I suppose I'm saying they need to have these as trainers for grown ups too.

</comment><comment><USER14> Parents should educate their kids, regardless of any disorder. The disorder is actually allowing these solutions to replace the only thing that no one can give: your attention and love.

But it's a nice engineering project!

</comment><comment><USER15> Reminds me of the story behind that odd Robot appearance in Rocky movies... [https://www.telegraph.co.uk/film/creed/rocky-iv-robot- butler...](https://www.telegraph.co.uk/film/creed/rocky-iv-robot-butler- stallone/)

It was initially developed to help children with Autism... Stallone has a child on the spectrum and wanted to check it out... it's a pretty neat story.

</comment><comment><USER16> It's like using taste to teach a person to hear.

</comment><comment><USER17> This is deeply dystopic.

<comment><USER18> Yes. But from what I'm reading here, the idea of a "Social Robot" doesn't seem to faze anybody -- not to mention that it will be "helping" children learn "social skills". OMFG.

<comment> tom-thistime It fazes the living daylights out of me.

<comment><USER19> This fazes the living daylights out of every intelligent human being except those tasked with spreading this morbid narrative as wide as they can for the buck.

</comment></comment></comment></comment><comment><USER20> Suggested title fix.

Moxie, a Social Robot (err, invasive mobile app) that helps kids (err, anyone with money) with Social-Emotional Learning (gather data on your children).

</comment><comment><USER21> I'm short this

</comment><comment><USER19> Way to go, MOCKsie, mock their inability to stay humans as hard as you can!

</comment><comment><USER22> Probably it comes with all Virtue Signaling rhetoric pre-loaded.

</comment><comment><USER23> This may not seem like it right away, but I believe it is a really big data point on a very important curve of the importance of computing devices in our lives. This topic is a bigger deal than it seems to be perceived as. The responses here are exactly what they should be to any huge development:

"why should this exist?"

"isn't this our responsibility?"

"this is the parents' fault"

"people should get this from their friends"

"think of the children!"

It sounds like how people concerned about radio, television, videogames, computers, and smart phones attacked those technological advances that carried with them huge societal impacts. While those may even be correct assessments and valid critiques, they are not effective arguments to stop use.

Conversational interfaces are not just a way to access information; they're a (perhaps horribly stilted) way to introspect and fulfill our basic need to socialize. We as a society often lament our lack of socialization and our increased isolation from each other, and the right answer here might be "prevent that isolation", but the easy answer may be "put a band-aid on it". This is a very powerful band-aid and a tough genie to put back into the bottle.

Imagine a world where you aren't lonely. You can talk to a variety of personalities about problems you have, about things you want to learn, about ideas and thoughts and fears. You can talk to something that's receptive, something that's combative, something that strives to motivate you, something that strives to soothe you. You could talk to a crowd of them, talk to them while playing games together, ask them about the news of the day, or anything. You can talk to someone new everyday, or one great friend for your entire life. They're always there and willing to listen, willing to engage. They never get angry at you or sick of you.

That kind of capability might save a lot of lonely people from taking a lot of horrible actions that are driven primarily by loneliness. It also sounds like a horrific dystopian rejection of our real, basic social bonds. But, maybe fixing those for real is a bridge too far!

Of course this specific product/idea is not "the thing" that brings those capabilities to the masses (it has pretty strong limitations), but it looks surprisingly polished and capable, and it's a very specific wedge into the space. It might get rejected today for being creepy (see: google glass), or it might get rejected for just not really being ready (see: google glass), and kids might just hate it. But we are clearly closer to that future world than we were six months ago.

Also, there are just _so many_ consequential fallouts of this in the long run: consequence-free abuse of robot conversors may lead to pathological behaviour directed at "real" people, the goalposts for the turing test may shift widely in either direction, surveillance will mean something else entirely when everyone talks to robots all day, propaganda delivered via these could have unfathomably large effects, the addictive nature of these and the predatory behaviours this opens up to the makers of these agents, the value of socialization overall being regarded very differently by our children, the arguments parents will have with their kids over whether or not their "virtual friends" are real or matter or valid or valuable, what the value of human life itself is to us... it's a whopper!

<comment><USER24> I don't even worry about propaganda, but about advertising. Just imagine your best friend telling reminding you every morning to only buy Kellog's, because they have the taste of REAL CINNAMON.

<comment><USER19> Advertisement did not ruin lives and cities in WWII, propaganda did.

</comment></comment></comment><comment><USER22> sad to live in a world where this is needed.

</comment><comment><USER25> No. This is stupid bordering on evil.

The robots do the _scut_ work, freeing up humans to do the important _human_ things: Parenting, teaching, etc.

This is "a complete societal failure on so many levels." as spacecity1971 says in a sib comment.

Remember Sony's dancing robot? When I saw that I was like, "WTF!? Make a robot to do the dishes! _I like_ to dance!"</comment>


HN as a city - <AUTHOR>

Intelligence floats in big cities, we breath it, and for those like me that dwell abroad, HN can be that breath of intelligence.
Has any of your post help you with a synergistic approach, new connections, give you a new outlook, fund your business or suggest a successful alternative? I'd like to know any post in HN that has made a profound effect on you.
For example, one post about learning and teaching maths linked to Thurston "On proof and Progress in Mathematics" that I consider can impact many young mathematicians. Kremer's model of innovation as a function of density of population means that tools as HN can expand the power of our minds to create and innovate like never before.
Going into a meta post, I can imagine that HN may evolve into some kind of network that allows us to get powerful insight. But, before reaching that goal, is necessary to filter and reward those whose creative mind we all want to share.
At the minimum, those creative minds that we all want show off should be allowed to move to any country, permanent visa granted, if they are going to share their insight to help boost the economy.
Related: http://en.wikipedia.org/wiki/Spatial_intelligence_of_cities
http://en.wikipedia.org/wiki/Intelligent_city
Edit: added link to intellengent city and Spatial intelligence. <comment><USER2> Not life changing, but in a recent post on HN entitled "How do you get to write so well?" I found many usefull suggestions for getting better at writing, becomming good at communication is one of the most important skills to aim for, specially if English is not your mother language.</comment>


Show HN: Talk tech, get hired. Our DRY approach to job hunting - <AUTHOR> http://techresidents.com

<comment><AUTHOR> Note: We strive to make our chats interesting to developers and allow them to take chats in whatever direction they choose. We're constantly adding new chats, but here are a few examples:

War Story: Bug - Tell us your favorite war story about a memorable triumph over a trying bug.

Slow Process Speculation - A linux process is reportedly running slowly. Speculate on the potential causes of the slowness, and the steps you would take to prove or disprove each theory. Creativity is encouraged.

Unicode for Grandmas - Unicode is frequently misunderstood. Think you can explain unicode to an audience of grandmothers?

For or Against ORM - Describe your stance on Object Relational Mapping (ORM) tools.</comment>


Bebo.com is back and there wont be penises [video] - <AUTHOR> http://bebo.com/

<comment><USER2> According to news site cir.ca, "the video shows several lewd drawing that people had posted on the original incarnation of Bebo". I haven't viewed the video, but it may be NSFW.</comment>


Experts pledge to rein in AI research - <AUTHOR> http://www.bbc.com/news/technology-30777834 <comment><USER2> "In the short term, this could mean research into the economic effects of AI to stop smart systems putting millions of people out of work."

This seems unfortunate and somewhat challenging to do. Our current economic model encourages improving efficiency of systems. This seems like a good thing. Its really too bad that people "need" jobs. Jobs should be creating value or they shouldn't exist. Artificially "creating" jobs to prop up the systems feels like fighting against reality and a bad long term plan.

<comment><USER3> How about a tax on AI that goes towards unemployment benefits of some kind?

<comment><USER4> I don't understand why AI eliminating jobs is considered a problem. If any job can be done by a computer program it simply means that humanity has outgrown this kind of work and that nobody should do that mindnumbing crap anymore. Let's forget that the word "computer" used to mean "guy who sits in an office of an accounting firm, and adds numbers all day long".

<comment><USER5> That's fantastic, as long as there are alternative jobs for those people to go to that they can feasibly perform. This has historically been the case with technology replacing work, but it's not clear whether it's the case this time.

If there's no (or insufficient) jobs for people to go to, in a society that predicates your ability to live a pleasant life on having one, you have a huge social problem. Our current system is not set up to cope with 40% of the population being out of work.

</comment></comment></comment></comment><comment><USER6> Why not do the same with all of technology?

Why do shareholders of big corporations profit from science in a grossly non- proportional way, while more than 50% of world's population has to live on under $2 a day?

It is time for the world's greatest minds to start thinking about how to fix capitalism, because it seems to be seriously broken.

And we need it fixed more than we need e.g. iPhone 7.0, or Google Adwords 2.0.

<comment><USER7> Coming from a formerly communist country, I have a severe distrust towards "great minds" with some radical fix for capitalism.

Actually, we should start treating social fixes like we do technical rollouts. Prove it on a small scale somewhere first, and then carefully expand.

This seems like common sense. Yet for some reason changes in policies tend to be sweeping, national or even international.

<comment><USER8> Clinical medicine is a better analogy than technical rollouts. But you're right, modern development economics argues that social fixes (policies) should be empirically driven using evidence-based randomized control trials.

Esther Duflo has pushing this approach at the MIT Poverty Action Lab. The main insight is to reject grand generalizations and broad theorizing. Sitting in an armchair pondering how to "fix capitalism" is unlikely to lead to useful lines of thought. Society is too complex a system, cause & effect relationships can be highly localized and context-dependent; formulaic thinking just leads one down ideological rabbit holes.

</comment></comment></comment><comment><USER9> I wonder: Why should we care about what Musk and Hawking mean about AI? This article doesn't mention too much bad stuff, but earlier they have said that we should be afraid of AI/singularity.

With my thesis now in AI, I probably know far more than those two about this. And we're sooo far away from AI being a superforce destroying mankind.

<comment><USER10> You can probably assume they are in contact with the leading experts in the field. From the letter:

        The initial version of this document was drafted by
        Stuart Russell, Daniel Dewey & Max Tegmark, with major
        input from Janos Kramar & Richard Mallah, and reflects
        valuable feedback from Anthony Aguirre, Erik
        Brynjolfsson, Ryan Calo, Tom Dietterich, Dileep George,
        Bill Hibbard, Demis Hassabis, Eric Horvitz, Leslie Pack
        Kaelbling, James Manyika, Luke Muehlhauser, Michael
        Osborne, David Parkes, Heather Roff Perkins, Francesca
        Rossi, Bart Selman, Murray Shanahan, and many others.

</comment></comment><comment><USER11> AI would, and should, succeed human intelligence properly implemented. We are nowhere near even understanding it as a problem, much less solving it. I attended an AGI conference a couple of years ago (summer of 2012 iirc). The general feeling was that we are still a lifetime away from a solution.

</comment><comment><USER10> This was discussed two days ago: [https://news.ycombinator.com/item?id=8870456](https://news.ycombinator.com/item?id=8870456)

</comment><comment><USER12> I don't know much about the philosophy of AI and I'm only familiar at a basic level with modern AI algorithms. From what I have been exposed to I don't see any reason to think AI is any more than a set of statistical frameworks. Is there any reason to believe that these statistical frameworks are comparable to biological intelligence?

Am I thinking about this the wrong way?

<comment><USER13> It sounds like you're talking about the kinds of AI in use today? That's not what the cautions are about, since current "AI" however good at reading individual words or flying drones is not yet capable of human-level thought. The cautions are about trans-sapient AI, which doesn't exist yet. Even if it's simply a beefed up "set of statistical frameworks" linked in the right way to get a computer behave like a human, humans develop and use nuclear weapons, humans go on shooting sprees, humans decide to go to war...

<comment><USER12> I see, I didn't think about that. So you mean AI that are not necessarily intelligent or conscious but are just more integrated in our lives, for example in the military or law enforcement?

I can see how a bug in those kinds of systems could cause things like that without necessarily being conscious or intelligent.

<comment><USER14> The implication is that humans are just statistical algorithms. We have so far not found any evidence to the contrary.

</comment></comment></comment></comment><comment><USER15> > _Research into AI, using a variety of approaches, had brought about great > progress on speech recognition, image analysis, driverless cars, translation > and robot motion, it said._

How much of this progress required training data generated by working humans? What would feed future statistical algorithms if this source of training data was greatly reduced?

</comment><comment><USER16> So long as we can pull the plug or disconnect the interfaces we'll be ok with AI. Once we can't, then we have a problem.

In effect the scariest AI is distributed, self propogating and can't be unpowered. Effectively a virus. I have yet to see a meaningful distributed AI, even in concept.

<comment><USER17> But it's not inconceivable that in the near-future an AI could train itself to mutate (even if this means interfacing with mechanical turk or freelance websites and paying humans to do it).

</comment></comment><comment><USER18> I'm glad someone is finally addressing this, what with a major advance in artificial intelligence being only five or ten years away.

</comment><comment><USER19> People make mistakes. That is all.

</comment><comment><USER20> This sounds like a publicity stunt...

My real concern with all of this is always the uncontrolled ecosystem of steadily evolving viruses and malware. We will never have control of that... and there is no telling what it can become in the future.

I think it will be a simple error induced by some random mutation in one of these malicious progams, not some vast artificial intelligence, that causes us problems in this arena first.

<comment><USER21> We need more AI working in the domain of computer security. Stuff that learns under specific guidelines to restrict computational behavior, given specifications of expected behavior.

<comment><USER22> You'll have AI working the other side of the arms race just the same. Tools aren't just used by the good guys.

<comment><USER21> That won't stop the bad guys from making those tools while the good guys don't, because they are afraid their tools will be used by the bad guys.

Ignorance is not a good defense strategy.

<comment><USER15> Inadvertent bad guys (good intentions with unpredictable errors) can be more damaging than intentional attacks.

<comment><USER21> Can you cite an example?

<comment><USER15> Any well-known vulnerability, e.g. heartbleed, gotofail, etc.

<comment><USER21> I am genuinely interested in hearing the 'good intention' explanation of these bugs.</comment></comment></comment></comment></comment></comment></comment></comment>


Ask HN: What to do with 1 hour per day? - <AUTHOR>

Hi, after reading some productivity blogs I realize how much I can improve myself spending 1 hour per day in an activity.
In a normal day, I usually have the following activities: programming(Python/Django), sports(tennis, running), studying.
I was thinking about the best activity in which I would invest my extra time, the activity must be something that I could catch up right away and learn/make something useful.
I want to increase my problem solving skills and I think some course like SICP could be a good idea but I don't know if I would like to consider another options.
Thank you for your time. <comment><USER2> > I want to increase my problem solving skills

Try these if you haven't <http://projecteuler.net/> <http://www.pythonchallenge.com/>

<comment><AUTHOR> Thanks for your answer, I'll take a look at them.</comment></comment>


Spoofing the Samsung Smart TV Internet Check - <AUTHOR> http://www.sodnpoo.com/posts.xml/spoofing_the_samsung_smart_tv_internet_check.xml <comment><USER2> "Smart" TVs are dumb, by design, in their very conception. TVs get upgraded perhaps once a decade or so. They don't need to have software in them, and shouldn't. Leave that to people and companies who actually know how to produce software. TVs should display an image. That's it. Nothing else.

There are no exceptions.

<comment><USER3> >There are no exceptions.

Wow, well I guess the issue is closed. I wish they had contacted you from the start, otherwise this multinational, multi-billion dollar corporation could've avoided the entire market. /s

I mean are you intentionally being dense? It has been a LONG time coming to supplant the multiple devices I have to keep around my TV just to use cable, streaming services, etc. Now I don't need to fire up the Wii if I want to watch Hulu or Netflix, movies off a media drive or USB. If only I could integrate the cable box into the TV, then it would really be a game changer.

I don't know about you, but I consider myself an above-average tech consumer and I value simplicity and if companies can manage to keep my TV functional for the next 5-7 years with all of my services, then it is money well spent.

<comment><USER4> I agree with your well-placed sarcasm -- this is not a black and white issue.

However, I _disagree_ that there is fundamental value in built-in TV software supplanting third-party devices. That is, unless a _software company_ is making the _software_.

I have a Roku, and built-in software from Sony on a blu-ray player (on separate TVs). The Roku is so much better that it's _painful_ to use the TV without it:

\- The buffering is much worse on the Sony. When it switches between ads and the video it takes several seconds to rebuffer.

\- The delay between menus is measured in seconds, while the Roku is almost instantaneous

\- It's almost like Sony didn't even have one UX designer working on their project. It will do idiotic things like when Hulu opens, create a modal dialog that says "The internet content is ready" and I must click OK to proceed.

\- The remote and other menus are slow and hard to decipher just like you'd expect from built-in software.

The point is that building the user experience of a video client is not trivial. I can't even blame Sony, really -- a non-software company will never do a good job at that kind of thing.

The Roku is a world of difference and until a company like that is building the built-in software, the mainstream of the market will be in third-party devices.

<comment><USER5> >The point is that building the user experience of a video client is not trivial.

No, but "Sony" have done it before and it's been fine. PS3. (I've not tried the PS4's media player.)

>I can't even blame Sony, really -- a non-software company

Sony Vegas? ACID? Sound Forge? Sony is not short of software production, "Sony Creative Software" makes a few software products most people here would have heard of. Not to mention SCE, they've made quite a bit of software and no- doubt have some UI engineers.

<comment><USER6> "Sony Vegas? ACID? Sound Forge?"

They bought every single one of those, from Vegas and from Sonic Foundry, Sony was not responsible for their design or creation. The UIs were AFAIK fully in place before Sony ever owned them, though I am sure some changes have been made.

</comment></comment></comment></comment></comment><comment><USER7> Devices that infer that connectivity to one site implies connectivity to the Internet are flawed. Having connectivity to one site means only that a route exists between you and the remote host. That's it. It says nothing about any other site. It's generally not good to do "Internet connectivity" checks. Just check to see if the site you want to connect to is responding. In the case of Samsung, who cares if samsung.com is down if I want YouTube. Just connect to YouTube and if that's down, then tell me.

The only useful exception to the above that I've seen is for Wi-Fi hotspot detection (e.g., checking a known response from Microsoft NCSI). The key difference here is that if the NCSI server is down or not responding, the user is not blocked from trying to connect.

<comment><USER8> So, yes, they are broken when you think about them like that. But, they cover the 95% use case... That case where most people say "The Internet is down" ;)

Giving the 95% a good experience should always be prioritised above giving 100% of users a mediocre experience.

<comment><USER9> Browsing samsung.com on ones TV's close to a 0% use case, not a 95% use case for users. Therefore checking to make sure samsung.com is up helps about 0% of their users.

</comment></comment></comment><comment><USER10> I really don't like how Samsung handles this.. I couldn't use my TV all morning to watch Youtube (I know, problems..), just because they think "If Samsung is down, don't bother opening the Youtube app". Seriously, what the heck...

Also, this is not the first time this happens.. I mean their data center was on fire (www.engadget.com/2014/04/20/samsung-com-outage-sds-fire/), but isn't there some failover?

<comment><USER11> Apple devices also had a similar problem a while back [http://www.zdnet.com/what-went-wrong-with-ios-6-wi- fi-700000...](http://www.zdnet.com/what-went-wrong-with-ios-6-wi- fi-7000004598/)

When iOS or Mac connects to wifi, it hits a similar URL to Apple to check if the device is stuck behind a captive wifi network. If the check failed, it attempts to throw up the networks login page.

But Apple's check server went down, and therefore peoples iPod touches, Macs and iPads got stuck in a nasty cycle of trying to throw up a non-existant login page.

<comment><USER12> Actually it wasn't that the server was down, it was that it was misconfigured and returned unexpected 404 responses which made it look like a captive portal.

</comment></comment></comment><comment><USER13> If we don't start building (and pushing for) more distributed/decentralised systems then this is the future we can expect more of. Aside from this, why should my TV need to dial home at all?

<comment><USER14> Why? To confirm connectivity.

Welcome to HN, where answering a question gets downvoted.

<comment><USER13> My browser doesn't do that so there's really no _need_ for a TV to do that.

<comment><USER15> Are you sure your browser doesn't do that? There's usually some code that checks for 'captive portals' \- i.e. wifi providers that want you to login before giving you a real internet connection.

<comment><USER13> What I mean is that my browser (Safari) doesn't dial home (Apple) to decide whether or not it can try serving me content (HackerNews). NB That doesn't mean it never connects to Apple without my knowledge (or elsewhere), but rather that it's not a point of failure.

Edit: I'm making a distinction between what my _browser_ does vs the _OS and wifi_. In any case, your home TV doesn't really have to deal with captive portals.

<comment><USER16> Your Mac does exactly that actually.

[http://blog.erratasec.com/2010/09/apples-secret-wispr- reques...](http://blog.erratasec.com/2010/09/apples-secret-wispr-request.html)

Apple broke the page once and everybody's wifi broke in response. Single point of failure.

<comment><USER17> How did everybody's wifi broke? I would have imagined that the only thing that would have happened was that, for wifis that required a login portal, you would have to launch your browser manually and let the portal hijack any pagerequest.

How did they manage to ensure that this didn't work if there wasn't such a portal or if you did it manually?

<comment><USER18> It's not safari that does it but the "OS" itself, and by that I mean the wireless networking tool. I would imagine that there's some code that runs upon successful connection to an AP and checks to see if you're in a captive portal or not.

The resulting UI looks like a plain window with no UI elements except a close button and a browser frame so you can auth with the captive portal system, and goes away once connectivity is "restored".

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER19> This just reaffirms my impression of Samsung as a decent hardware company and a poor software company.

I was thrilled to switch from my Samsung Galaxy S3 to my Nexus 4. The S3 had horrible Samsung-equivalent apps for everything: email, chat, a browser, and a lot of system services that I couldn't uninstall. It also had some specially built update mechanism, separate from Google Play, that would drain my battery, until it was fixed.

I like Samsung though, as long as they stick to producing hardware. My ARM Chromebook is a lovely device: Samsung hardware with Google software on it.

</comment><comment><USER20> I build "smart" TV apps for a living. Samsung's connectivity check for their TVs from 2010 has a list of five domains that it checks against. Discovered this during some debugging a couple of years ago. The list, in the order pings are made, is:

      icq.com (not kidding)
      yahoo.com
      samsung.com
      google.com
      Some other domain I can't remember

<comment><USER21> And it requires all five to connect, rather than just one?

</comment></comment><comment><USER22> Single points of failure everywhere. This is not the future we were promised. At least the fixes are in our hands.

<comment><USER23> Custom and open operating systems are the only way forward against this sort of bullshit. It's not just crappy engineering, it's also TVs doing analytics on what you watch, always-on cameras, etc..

As long as we have the power to root these devices and install our own software on it, we will be fine. But for how long is that going to keep up?

<comment><USER24> > TVs doing analytics on what you watch, always-on cameras

 _Who_ 's doing the watching now? That sounds ridiculously Orwellian...

> As long as we have the power to root these devices and install our own > software on it, we will be fine. But for how long is that going to keep up?

The most interesting part of this is that "rooting" often relies on finding and exploiting a _vulnerability_ , something that would be considered detrimental to security and normally thought of as a bug. In other words, this power is coming from having not-so-secure devices. Imagine if this TV was more secure; it used DNSSEC and HTTPS to authenticate/encrypt communications, and was designed to be resistant to tampering via hardware (secure processor, encrypted memory, etc.) -- ostensibly for things like DRM. Do suggestions like "all Internet traffic should be encrypted" start to look less appealing now? To say it plainly, in this case "insecurity is freedom."

<comment><USER25> "Users should be able to install certificates" isn't super helpful for most users, but it is sensible and also a nice clear message.

</comment></comment></comment></comment><comment><USER26> This is exactly the sort of problem that made me decide to buy a regular television after having previously bought a "smart" television. It does not anger me quite so much as Microsoft trying to get me to pay them a subscription fee to access my own network hardware, but it does drive me in the direction of wanting an open-source set-top HTPC to sit at the interface between dumb viewscreen and Internet.

</comment><comment><USER27> So from that XML, you seem to control the URLs the TV will grab. Just a few buffer overflows away from root on the TV, and of course you don't really own the hardware you bought until you have root, right?

</comment><comment><USER28> A while back I "hacked" my Samsung TV. I spoofed a DNS entry to point to my laptop, and put a tiny webserver on it. Then I could hit the "News" button on my remote and get obscene fortune cookies on my TV!

Then I turned it off. Internet on TV is silly, as far as I can tell.

My girlfriend's wonderful Panasonic has tons of internet features -- web, youtube. Web is nearly useless because of lack of keyboard; youtube is awkward; vimeo isn't bad. Using an Android app, selecting your youtube video, then doing "send to TV" is great when we bother to do it.

The "smart" Panasonic is a dumb screen in front of Roku. The only other non- dumb features we use is the "play videos from SD card" feature, and the "play from DLNA (my laptop)" feature -- both get used quite often, and work well.

</comment><comment><USER29> testing a connection for various features is not as robust as actually trying the operation you want to complete and determining if it worked.

drop all the extra work and let code fail gracefully while in action.

</comment><comment><USER4> It's a good thing they didn't use HTTPS. And a good thing that something else doesn't actually rely on samsung.com (listing services, API endpoints, whatever).

This is a cool hack for now but if I had to make a prediction I'd bet there's a future coming where this isn't possible, where the two are even more intertwined -- even though most of us know they don't need to be and wouldn't design our own systems that way.

</comment><comment><USER30> Oh boyo.

We have samsung smart tvs at home, which Plex runs on top of... looks like I'm going to have to implement something like this with openWRT in future..

</comment><comment><USER31> I assume Samsung is using Linux as the underlying OS here. If so, does Linux have a "Network Awareness" API? I understand that Windows has one. More info: [http://technet.microsoft.com/en- us/library/cc766017%28WS.10%...](http://technet.microsoft.com/en- us/library/cc766017%28WS.10%29.aspx)

<comment><USER19> As far as I can see, this issue exists because, fundamentally, there is no such thing as "Internet access". If a device can access only 50% of hosts on the Internet, does it have Internet access? What about 1% or 99%?

Consequently, it makes no sense to make "Internet access" a binary variable that a software system uses to determine whether it has access to _all_ Internet sites.

One would think the YouTube app would have a built-in mechanism for handling a failure when connecting to youtube.com. It's entirely possible that samsung.com could be up, but youtube.com down. I assume the YouTube app gracefully handles this, and this would render Samsung's binary state "Internet present?" variable useless at best.

 _TL;DR_ : Having a system-wide "Internet available?" variable is meaningless because individual apps need to be able to handle not being able to connect to a particular host anyway (eg. youtube.com, netflix.com).

<comment><USER32> Well, the system-wide "internet available?" variable isn't completely useless from a user perspective.

If youtube.com happens to be down for whatever reason, there isn't much the user can do about it except wait, so there isn't much for the YouTube to do besides display a standard error message and call it a day.

But in the more common case where the user's internet connection is unavailable or misconfigured, there are troubleshooting steps the user can take (make sure it's plugged in, try turning the router off and on again, check network settings, etc...).

I suspect the design was to help users get their TVs connected to the internet by providing more useful diagnostics when they are offline. To do that, you need some kind of system-level check for connectivity. Now, there are certainly better ways they could have implemented the feature, ways that didn't involve a single point of failure where a fire at one building means that millions of TVs can't access Netflix or YouTube for no good reason, but the intent wasn't completely meaningless.

</comment></comment></comment><comment><USER33> Very helpful. I have the LG smart tv and i would like to know if in my case (lg) could happen the same... Just in case to know if it happens. my opinion about these smart tv's is very positive. I like to browse or watch films over netflix or others. Hope that in a couple of years it gets even smarter :)

</comment><comment><USER34> Nice work! I ran into the same issue with my samsung "smart" TV, and I was starting to set up a similar hack when my wife beat me to the punch: hit the "factory reset" option in the menu, and then it loses the wifi settings and stops trying to check for updates.

</comment><comment><USER35> I wouldn't be suprised if it's vulnerable to XXE too ( [http://projects.webappsec.org/w/page/13247002/XML%20Entity%2...](http://projects.webappsec.org/w/page/13247002/XML%20Entity%20Expansion) )

</comment><comment><USER36> It amazes me that most of these "Smart" TVs don't use SSL or any sort of encryption in first place.

<comment><USER37> I know first hand some other brands do.

</comment></comment><comment><USER38> > I overrode the DNS and pointed it to 127.0.0.1 and tried again.

Is there an easy way to spoof that on your local home network?

<comment><USER39> Run your own local DNS server. Virtually every lightweight router-oriented Linux distro includes dnsmasq if you don't want to run a full-blown DNS server.

<comment><USER40> Moreover, your router is very likely to already run dnsmasq under the hood.

</comment></comment></comment><comment><USER41> I have a Vizio "smart tv" which is about as "smart"

</comment><comment><USER42> I guess this explains why my Fios TV app isn't working.

</comment><comment><USER43> Wow now I want to buy a DRM... I mean "Smart" TV!</comment>


Linus Tech Tips critize Nest Cams, video does not appear in subscriptions - <AUTHOR> https://www.youtube.com/watch?v=H7p5YEOrlSc <comment><USER2> I cant see it too, but this video was just stupid. Comparing Nest Cam vs home made camera, of course its gonna be cheaper. Its litteraly comparing apple vs oranges.

<comment><USER3> Even if the video is stupid, videos being selectively shown in the subscription tab is worrying. These are people I want to hear from and if I now have to go to each channel to see if there is something new I am going to go somewhere else.

</comment></comment><comment><USER4> "Linus Tech Tips1 hour ago We messed up when we published the video. Unchecked the notify subs checkbox."

</comment><comment><USER3> I saw it in my subscriptions but that was probably around an hour after it was posted.

It is gone now.

</comment><comment><USER5> Related: [https://m.youtube.com/watch?v=D1NX2dr3voU](https://m.youtube.com/watch?v=D1NX2dr3voU)</comment>


Samsung Kills Headphone Jack in Latest Smartphone After Years of Mocking Apple - <AUTHOR> https://www.macrumors.com/2018/12/10/samsung-removes-headphone-jack-galaxy-a8s/ <comment><USER2> So, they're both wrong? (Samsung and Apple)

I like headphones

And enough edges to hold the thing without pressing on the screen. Having dead spots at the edges of the screen is not a solution.

<comment><USER3> You'll have to pry my LG G6 from my cold, dead hands....

Unless you trade it for a G7

</comment></comment><comment><USER4> Same mocking occurs every time Apple removes a soon-to-be-obsolete piece of hardware (serial/parallel ports, disk drives, CD drives, USB2/3 ports, etc). So far I think the only big mistake they've made is removing the Esc key.

<comment><USER5> Indeed I can't imagine the collective pain of all the VI users.

I have BT external keyboards both at work and at home. Never use the MBP keyboard.

<comment><USER6> You can use CTRL [ in vi to do the same function as the ESC key.

<comment><USER5> Oh good to know. I am not really a Vi user. Even as an Emacs user, I feel the loss of physical ESC key acutely.

Anything that makes you look down on keyboard is sub optimal.

</comment></comment></comment></comment><comment><USER7> Personally, I'm quite happy that the headphone jack is gone. It was the number one flaw is most of my phones. Either, the jack would get loose and start to crackle or I would get a spark of static electricity that would blow out headphones. AirPods have been amazing. No cords, no static electricity, and I can easily jump between all of my devices. If I really need a photo jack, I can use the dongle. Best part? If the dongle fries from static electricity, it's a $9 part.

</comment><comment><USER8> Clickbait title. It's a budget phone, the A line, not the flagship S or Note line. They are probably testing out the feasibility of no headphone jack.

<comment><USER9> Why would they need to test that? Apple has already done it for them.

<comment><USER10> Exactly. Samsung's strategy is to turn the headphone jack into a premium feature by removing it from the cheaper phones. It's not a test, it's a marketing tactic.

</comment></comment></comment><comment><USER11> Is there a conclusive opinion on the negative effect that bluetooth signals have on our bodies or is it just conspiracy talk?

<comment><USER12> Where did you read about it?

</comment></comment><comment><USER13> Imagine how much real estate they get after removing it, right?</comment>


Woot's take on affiliate marketing - <AUTHOR> http://www.woot.com/minions <comment><USER3> Affiliate marketing is, to reuse a quote from the guys behind Spreedly, a "star search.". Many will enter, few will win. You keep the barriers to entry low in the hopes that you find one of the comparatively few professional affiliates who know what they are doing, as opposed to more numerous people who either intend to put a single link on their blog or, worse, have recently bought a Make Money Online ebook.

What successful affiliates are doing is very frequently some shade of gray, which is one of the reasons to have affiliates in the first place - <USER2> deniability. [Edited to add: a well-designed affiliate program also generates organic SEO benefits, and this facet profits even from a single link on an unremarkable blog, when that gets duplicated at scale.]

<comment><USER4> I've worked for an affiliate marketing company, and some of the highest earners do truly earn impressive amounts (tens of thousands of dollars a month). You are also incredibly right about the shady business that goes on in this industry. In my opinion, the most profitable and highest conversion campaigns are always the most gray. The fact that many advertisers usually require a landing page (rather than a direct link) is really fishy by itself.

<comment><USER5> I used to make a healthy living designing landing pages for affiliates. Several of my clients were making several million a year (all made a "healthy" living) and I was bringing in several hundred a day from just designing landing pages. I left because I couldn't handle that "gray area" people talk about, but I swear those are some of the most intelligent and hardest working people I've met. Shame many are missing that deep entrepreneurial spirit many here on HN have.

<comment><USER6> I'm not too familiar with this field, can you explain what the "gray area" is?

Is it people linking to the affiliate without disclosing they are getting something from it? Or is it more blackhat SEO and spam?

<comment><USER3> I could talk for hours on this. Some things which amused even a jaded SEO:

1) Rebill scams. (Theft, basically.)

2) As a reward for winning this trivia quiz, here is a one month free subscription to $OFFER.

3) Review site: stars awarded based solely on affiliate commission.

4) Brand arbitrage. Many ways to do it. The easiest is "$BRAND is a scam. Don't buy until you read this!"

<comment><USER7> 5) "This offer is only available if you subscribe via this special link before 12pm on [script returning today's date]"

6) Essentially worthless "make money fast" products that are little more than product-based pyramid schemes shilling the opportunity to make money by signing up themselves to marketing the "product" to other suckers

7) Virtually anything related to casinos - especially the ones purporting to explain how to "beat" them (and amongst those, especially those providing information on how to churn casino bonuses playing blackjack that's actually accurate)

</comment></comment></comment></comment></comment></comment><comment><AUTHOR> How to manage expectations:

1\. Amazon Associates: "Make money advertising Amazon products"

2\. eBay Partner Network: "You can earn a lot of money"

3\. Woot: "...destroy your online reputation AND earn literally pennies through dull, thankless tedium"

<comment><USER8> The funny part - Woot is owned by Amazon.

<comment><USER9> But have clearly maintained their culture post-acquisition (as they said they would)

</comment></comment></comment><comment><USER10> I did very well myself with affiliate marketing. The trick (if you want to do affiliate marketing and still feel good about yourself) is to provide some value add.

So when I started iPadCaseFinder (now owned by someone else and changed into a storefront), I built a really cool 'finding' application so people could find the perfect iPad cases for them. It earnt pretty well!

<comment><USER11> I think the disagreement most people would have with this approach is that you're coming at it from the wrong side. It's money first and then "oh yeah, I'd better add some value."

<comment><USER10> You're wrong, I decided to found this site because my friend dropped and cracked my iPad, and I couldn't find any suitable cases. The site was inspired by a true life problem, and I know 100's of people that found it very useful.

</comment></comment></comment><comment><USER12> Pretty funny, and interesting that this comes from an Amazon company, Amazon having pioneered the online affiliate industry and finding much success in it. Still, It's a pretty accurate portrayal of the whole affiliate scheme, and I think it might have the effect of keeping away the low-value affiliates while not deterring those who could actually provide a mutually beneficial relationship.

</comment><comment><USER13> I wonder if anyone at woot had the groupon idea (before groupon came out) but was quashed because it just wasn't woot-like. It seems like woot lost a couple orders of magnitude of scale by (apparently) making themselves the fulfillment bottleneck.

<comment><USER14> They're both important. I'm never sure I'll use a groupon when I buy it, but I can always be talked into buying a gadget if it's cheap or a shirt if it strikes my fancy.

</comment></comment><comment><USER15> it's actually a 50/50 proposition...many affiliates who actually make sales(besides the few rare super affiliates), do so by simply ranking for the first page as "[brand] review, since that way they reach the customer just before they hit the buy button

So as an advertiser, you basically just end up paying money for sales that would have been yours anyways

<comment><USER16> Well, that depends on if the top review would have been good otherwise.

One thing affiliate programs appear to be awesome at is stuffing Google with positive reviews of a product.

(Which leads me to wonder how long it will be until the Big G starts downranking pages with easily-spotted affiliate links.)

<comment><USER15> they already do that, which is why affiliates link go to great lengths to hide the fact that they are linking to an affiliate site

</comment></comment></comment><comment><USER17> Well, if you're worried about scaling issues from a flood of affiliates, I guess this is one way of trying to manage the volume...

</comment><comment><USER18> Very well done! Looks like a full script of our experiences with AdWords back when it was new and shiny.

So, Woot is actually serious about this? Glad to see someone put things as they are.

</comment><comment><USER19> Of course they have that opinion. Affiliate marketing requires time and targeting, neither of which woot.com can offer affiliates.

</comment><comment><USER20> It's certainly more creating than adding a link to your commission junction page. It'll probably get more visibility too.

</comment><comment><USER21> Very creative. I laughed a little.

<comment><USER22> I didn't think it was funny at all, very creative indeed though.

</comment></comment><comment><USER23> The cool landing page really gets killed by the unfortunately drab commission junction page

<comment><USER24> I like the way your eyes get directed on that page. Read the whole pitch then follow the trail back to the top.</comment></comment>


Get Your eBook in the Apple iBookstore - <AUTHOR> http://lulublog.com/2010/04/03/lulu-on-the-ipad/

<comment><USER2> What kind of content restrictions does Apple have for media? The rules must be less onerous than the restrictions for apps. Social norms allow for a lot of sex in music, movies, and books. If Apple were to enforce app content standards for media, then hoards of famous works would be excluded, including most of rock n' roll.

<comment><USER3> disney

</comment></comment><comment><USER4> _"Pricing. All eBooks planned for iBookstore distribution must have retail pricing that complies with Apple's guidelines. We will automatically price Lulu eBooks submitted to the iBookstore to meet these guidelines. The creator revenue that you receive per sale will be 80 percent of the profit after deducting Apple's share. On a $9.99 book, for example, you will receive $5.60 ."_

My $9.99 book just became $14.99 then

</comment><comment><USER5> Given this, what is the cost for the services of a professional editor? Is the model going to be a flat-fee or percentage of book sales?

<comment><USER6> Probably flat-fee. A talented editor can make your manuscript better, but that doesn't guarantee sales.

<comment><USER3> I think a good model could also be for an author & editor to team up with a book project, where they both share in the revenue. I think this would incentivize both to do their best.

<comment><USER5> I like that model, the question is what is the cut?

<comment><USER3> I think that is up to them. I can imagine an author and editor producing a book and selling it on Fifobooks (my startup), and have an agreement between the two of them.

I see it similar to cofounders launching a startup; they each bring something to the table and it is up to them to agree on a valuation/profit share.</comment></comment></comment></comment></comment>


Oracle spins own Linux for mega hardware - "Red-Hat compatibility holds us back" - <AUTHOR> http://www.theregister.co.uk/2010/09/20/oracle_own_linux/

<comment><USER2> That article is the motherlode of acronyms and jargon.

Really, check out this paragraph:

> The Exalogic Elastic Cloud is founded on Oracle's Exadata server and a > version of Oracle's WebLogic Java application server with Oracle Coherence > running on Oracle's virtualization, 30 compute servers with 360 cores, 960GB > SSD and 40GB InfiniBand networking.</comment>


Winning the Ride-Sharing Startup Battle with Personalization: A Java How-to - <AUTHOR> http://blog.prediction.io/winning-ride-sharing-startup-battle-personalization-java/ <comment><AUTHOR> Is there other use case demo of personalization you guys want to see?</comment>


A crypto-currency primer: Bitcoin vs. Litecoin - <AUTHOR> http://www.zdnet.com/a-crypto-currency-primer-bitcoin-vs-litecoin-7000024301/

<comment><USER2> One key aspect is that the Litecoin blockchain loads into the app in about 20 mins (and not 2 days or more with Bitcoin... roughly.. I should time it) And that Litecoin appears to be desired by drug-site users. It's basically the same in usage... (perhaps faster, although it depends on the number of confirmations needed to deem as secure) ...but more chance that Litecoins will rise in value. There's also more US regulations applied to Bitcoin in terms of what can go into the code (as far as I know.) If someone asked what crypto to use.. I'd say Litecoin. I don't get hung up emotionally on it.</comment>


The Problem with Mindfulness - <AUTHOR> http://nautil.us/blog/the-problem-with-mindfulness <comment><USER2> For the first year of my mindfulness practice, I cheerily recommended meditation to acquaintances who were struggling with anxiety and a lack of joy. After a harrowing experience of dissolution that lasted for several weeks, I realized that mindfulness isn't just calm and pleasant experiences. I still continue to practice and it does bring me joy but I no longer recommend this particular practice to anyone who has a rocky past and a limited support system and just wants to feel 'good'. For that, I think diet, exercise, and sleep are safe bets.

<comment><USER3> Do you mind elaborating on your "harrowing experience of dissolution"? I've been practicing for less than a year now and I'm curious about your experience.

<comment><USER2> It happened while i was meditating for 1-2 hours a day sitting up and then lying down on my couch and practicing for another hour or two. At some point, I looked at my feet at the end of the couch and I didn't recognize them as 'mine'. I felt in to my body and no longer identified with the sensations. Not particularly unpleasant on its own, but it was coupled with a sensation of doom that was similar to coming up on a hallucinogen and hearing news that someone you knew died. The fear and sensation of pressure and stress lasted for about 2-3 weeks, all day, every day. It was an inescapable sense of doom. I did not feel any sense of compassion or joy that I had heard arise in conjunction with my other symptoms. Eventually, the feeling passed. Now I make sure to explicitly cultivate a sense of self-compassion and my 'lows' have been a lot less fearful. I do have a history of depression and have been diagnosed with ptsd so if you're relatively healthy and generally feel okay about life I wouldn't be too worried about such an experience in everyday practice. I also remembering that maybe the event was something meaningful and 'dove in'. I kept meditating in isolation. In hindsight I could have pulled back a little bit. While I have had one or two episodes similar to that since, I understand they're momentary and instead of meditating, I distract myself with tv or something similar.

<comment><USER4> I really think this, your incidents here are something you need to go and discuss with a nice psychologist. You shouldn't have to bear these kinds of things alone, and there are healthy solutions to this.

Take care.

</comment></comment></comment></comment><comment><USER5> This happens when the person is not calm/tranquil enough to handle the realization of no self (emptiness). One should also develop tranquility / calmness by doing samatha meditation (like anapana). With having developed tranquility the realization of no-self and emptiness are easier to handle.

</comment><comment><USER6> I lead a small group of people twice a week in mindfulness practice. I agree wholeheartedly with what this article is saying. Mindfulness has been sold as the latest snake oil, a cure for all ills. It is not, and it should not be taught that way. Mindfulness is the benefit of practice, not the salve for our wounds.

<comment><USER7> Yes. Mindfulness is not about just being present during pleasurable moments. A lot of the value, to me, is being present when my mind or expectations conflict with the world around us. That leads to an examination of ourselves, insight, and ultimately growth. This is very different than the withdrawl into feel good moments, which is what is often sold to us.</comment></comment>


Clojure Web Infrastructure - <AUTHOR> http://www.glenstampoultzis.net/blog/clojure-web-infrastructure/ <comment><USER2> Excellent breakdown.

This is an important point about Clojure - the culture seems to be leaning towards multiple small libraries, rather than whole frameworks or packages.

Which is actually kind of neat, since a properly written Clojure library is much more composeable than libraries in most other languages.

<comment><USER3> Isn't this sort of the same philosophy as the way Lisps are built? It seems like having proper namespacing and a overall functional style lend themselves to smaller, more self-contained libraries.

Similarly there isn't as much of a need to have a monolithic stack like Ruby has, because Ruby's style tends to step on other's work a bit leading to preferring a all-in-one style for reliability.

</comment></comment><comment><USER4> Holy Crap why oh why couldn't someone have posted this two weeks ago?! I just upgraded an internal application from Clojure 1.1.0 to Clojure 1.2.0 and had to upgrade my entire web-stack as a result! I was more than a little annoyed to find out that not only had Compojure outsourced nearly all of it's earlier functionality to other libraries, but that I could no longer just download the jars. I'm sure Leiningen solves all of your dependency needs when you are on Linux or a Mac, but I cannot get it to work on Windows and Clojure package maintainers seem to have forgotten that there are people who might want (or need) to install libraries manually without things like Leiningen or Maven. </rant>

I understand that Clojure is still young and that it's ecosystem is still evolving at a fast pace, and I understand why the changes being made are largely improvements, but for my particular circumstances it's a large headache.

EDIT: Here's a hint for people want to get a Clojure environment going but can't use one of the package managers for some reason or other: You can visit clojars.org and download all of the jars that you need. This was, you don't need a build system on you development machine. You still have to figure out what the dependencies are manually, but you aren't stuck if you can't get Leiningen or Maven working.

<comment><USER5> I know this sounds completely crazy, but you could always run a Linux VM on your windows machine. I find this more convenient than trying to figure out how to make things work on Windows some days.

Alternatively, if you're having trouble with the lein.bat, you could try lein.ps1. (<https://gist.github.com/395038>) I believe that lein.sh is meant to work pretty well on cygwin now, though.

<comment><USER6> At work, I'm even doing it the other way around: Windows runs as a guest in Linux. Nothing get's done in Windows unless it's more work to do it in Linux. Interestingly, for me that's only two things: VPNs and Citrix Metaframe to connect to clients. That's it.

I can't stress how much more productive you'll be. And we're a shop selling Windows software to Banks and bigger corporations.

</comment></comment></comment><comment><USER7> I'm currently using a bunch of these; I'm not using enlive for templating as such, just modifying existing markup after it's been generated. For classical templates I've found that freemarker, a Java templating system, works perfectly with the Clojure types.

clj-record shows promise but is still rather rough around the edges. I'm hoping to contribute some patches.

I'm still trying to get to grips with Leiningen: one thing I haven't figured out is what the best way to use local jar files for Java libs. Also, using locally modified versions of libraries which are already in the leiningen system doesn't seem an intended use case.

</comment><comment><USER8> as clojure runs on JVM so is there is any string attached to the language whose other end is with the Oracle in any crazy manner imaginable? after the google vs oracle patent case i am really afraid of java. what is the point of open source java if there are patent issue which can come and get you.

<comment><USER9> IANAL, but i believe that as long as you don't try to implement your own JVM, you're safe from lawsuits. Imagine that Microsoft sues some Linux company because of some windows patents(didn't they actually do that?), would that stop you from using either Linux or Windows?

</comment></comment><comment><USER10> I removed the routing layer (Compojure) and my code is now clearer and shorter, by resolving URIs directly to functions. At least for developing this is great. Lets see where this will take me.

</comment><comment><USER11> App Engine Magic looks cool, I hadn't seen that before, but trying it right now. I am most interested if AOT helps reduce loading request times.</comment>