Are You Vitamin D Deficient? (Infographic) - <AUTHOR> http://www.informationisbeautiful.net/2010/vitamin-d/ <comment><USER3> As a person living in Seattle (rains a lot!) who is of a darker skin color because of my race, I get told every now and then that I must be very lucky because I'm never going to be Vit D deficient.

I'm surprised that people think having darker skin color means I already have oodles of Vit D, when it's quite the opposite - people with dark skin color synthesize less Vitamin D on exposure to sunlight. I probably need lot more of that big ass star :)

</comment><comment><USER4> I take 4,000 UI of Vitamin D each day in gelcap form (Vitamin D is fat- soluble, so much better absorbed that way than with dry tablets -- this information is based on blood tests that the cardiologist at the Heart Scan Blog did <http://heartscanblog.blogspot.com/> ). Living in Canada and working inside most of the day, that seems to be a minimum for me. I could probably up the dose to 6,000 UI this winter...

Here's the kind I get: [http://www.vitaminlife.com/product- exec/PNAME/Vitamin_D_2000...](http://www.vitaminlife.com/product- exec/PNAME/Vitamin_D_2000_IU/product_id/35147)

I also take 3 grams of omega 3 (adding up to 1800 mg of EPA/DHA), 500 mg of Vitamin C, and a multivitamin.

<comment><USER5> You might consider switching to a D3 supplement that includes some K2; they play nicely together.

<comment><USER4> I'm not familiar with K2. I'll do some Googling around, but if you have good sources, I'd appreciate it if you shared them. Thanks.

<comment><USER5> As far as I understand it, higher levels of vitamin D3 greatly increases absorption of calcium from the gut, and the K2 is necessary to ensure that the calcium gets deposited in the right places.

Sorry, I don't have an existing set of references for this, but here are a couple things I found:

From [http://www.drjessicaseaton.com/Chiropractic_in_West_Los_Ange...](http://www.drjessicaseaton.com/Chiropractic_in_West_Los_Angeles/Vitamin_D3_%28and_K2%29.html)

"Increased levels of vitamin D require increased levels of vitamin K, especially of vitamin K2 (menaquinone). Without enough vitamin K2 there is an increased risk of hardening of the arteries and other soft tissues."

From <http://en.wikipedia.org/wiki/Vitamin_K>

"...high intakes of vitamin D but low intakes of vitamin K were suggested to pose an increased risk of hip fracture."

<comment><USER6> evlovingstuff - I think you'll like this article about K & D interaction:

[http://www.lef.org/magazine/mag2010/sep2010_Brittle-Bones- Ha...](http://www.lef.org/magazine/mag2010/sep2010_Brittle-Bones-Hardened- Arteries_01.htm)

Here is the summary: "The remarkable discovery that blood vessel cells can transform into bone-forming cells confirmed the little-known link between atherosclerosis and osteoporosis. At the core of this connection is insufficient D and K intake. These vital nutrients operate in synergy to optimize bone mineralization and prevent calcium deposits in vascular tissue. Low vitamin D is linked with arterial disease and bone loss, while vitamin K stimulates bone formation and modifies specific proteins (Gla) that help protect against arterial calcification.

Vitamin D taken in higher doses (5,000-10,000 IU/day) has become popular over the last two years based on findings showing that this potency is required to achieve optimal blood levels (over 50 ng/mL of 25-hydroxyvitamin D). Relatively few supplement takers, however, understand the critical need for aging humans to also take a daily vitamin K supplement. Fortunately, Life Extension members were informed in 1999 about the critical need of including vitamin K in their supplement regimen."

<comment><USER5> Thanks fubari, great information!

</comment></comment></comment></comment></comment></comment><comment><USER7> Man, this is a terrible infographic imo, though this sort of thing gets linkshared around the internet like wildfire, so I must be in the minority on that. Some parts of it are fine, like the map, but some parts would be a lot more straightforward as a few sentences or some normal bullet points, instead of that confusing mess of flow-chart-esque lines in the "Vitamin D - <USER2> FOR / INFLUENCES / SUSPECTED EFFECT" section. And the graphics in the "D-Day" section aren't really telling me much at all.

Also, why does the whole thing have to be one giant image?

<comment><USER8> The map seems pretty bad too: What about areas south of -42° latitude during their winter?

<comment><USER9> The population south of 42°S is pretty small. Also, they may have lower ozone levels that allow more UV through.

</comment></comment></comment><comment><USER10> This infographic suggests taking 2,000 IUs/day. Both the Mayo Clinic and WebMD say that more than 2,000 IUs/day can be toxic: [http://www.mayoclinic.com/health/vitamin-d/NS_patient- vitami...](http://www.mayoclinic.com/health/vitamin-d/NS_patient- vitamind/DSECTION=dosing) [http://www.webmd.com/vitamins-lifestyle- guide/supplement-gui...](http://www.webmd.com/vitamins-lifestyle- guide/supplement-guide-vitamin-d)

<comment><USER4> This is from memory, I don't have my sources right now, but most of what I've read is that the body can deal very well with too much Vitamin D. What it doesn't do well is a deficiency.

A few of my pharmacist friends told me about how some people are being prescribed doses of 50,000 UI, and I've read somewhere that below 14,000 UI a day over extended periods, no toxicity was found. So I kind of doubt that 2,000 UI is an upper bound, unless you have a very specific condition. Just spending a day on the beach probably gives you a MUCH higher dose than this.

But as I said, this is from memory. Take this with a grain of salt and do your own research. Personally, I'll keep taking my 4,000 UI/day, something that I've been doing for about 4-5 years (and while this is anecdotal, I've seen improvements in seasonal mood changes and the number of times I'm sick per year).

<comment><USER11> IU

<comment><USER4> I'm not sure if I wrote it like that because I used to writing the acronym for "user interface" or because in French, my native language, they write it as "UI" (unités internationales). But thanks for the correction.

</comment></comment></comment></comment><comment><USER12> Most guides to getting Vitamin D make blanket statements about how much ultraviolet light is available. The amount of sun varies throughout the winter and by time of day, as well as latitude. You can probably get more UVB at high noon in December than you can at 3pm in October.

I'd love to see a guide that shows you how long you need to be out at a specific time, date, latitude and skin color.

</comment><comment><USER13> I found the wikipedia entry useful in citing specific interesting studies (e.g., suggesting >50,000 IUs/day (orally) may be needed for toxicity). Plus a clarification of the 'how' of sunlight-produced D self-limiting:

<http://en.wikipedia.org/wiki/Vitamin_D>

</comment><comment><USER14> As tech people, we need to remember that our glorious LED backlit Macs just aren't as important as that big ass star holding us all together.

Here's my infographic: [go outside everyday!]

<comment><USER15> _our glorious LED backlit Macs just aren't as important as that big ass star holding us all together._

Hey, let's not go overboard! Can we agree that both Macs and the Sun are important? Hrm?

</comment></comment><comment><USER5> Shameless plug of my dad's website on vitamin D: <http://www.vitamindwiki.com/tiki-index.php>

</comment><comment><USER16> I feel like I should be offended about the Northern Hemisphere-centric nature of the plot. What about the lack of sunlight below 42 S from May to July?

(Admittedly that only includes half of New Zealand's South Island, the tip of Tasmania, and part of Patagonia. Plus all of Antarctica, but I assume if you're wintering in Antarctica you're probably well aware of Vitamin D deficiency.)

</comment><comment><USER17> 10 minutes outside in summer? Does that address our alledged reduced ability to make vitamin D with age? Does it address that vitamin D is made from cholesterol and lots of people are on an anti-cholesterol crusade? Does it address that UVA hinders the reaction which UVB encourages, and therefore sitting in the sun behind glass might undo some of the benefit? (hint: no).

Does the daily recommended amount for supplementation address which kinds of Vitamin D are useful and which aren't? Which kinds are more easily absorbed? That different people absorb different amounts? That it's possible to overdose? (hint: no).

These skirt around the real point which is to raise some particular blood level of a specific kind of Vitamin D. If you aren't getting that, specifically, measured then whatever else you do in terms of sunlight or supplementation is winging it and hoping for the best (and risking overdose) or merely assuaging some guilt that you should be healthier by doing "something".

<comment><USER18> Does this mean that vegans are more likely to be Vitamin D deficient (and vegetarians to a certain degree)?

<comment><USER19> Vegans are extremely likely to have many deficiencies anyway.

</comment></comment></comment><comment><USER9> I have a lot of physician friends who say that testing vitamin D levels is the latest "fad", so to speak.

I got tested out of curiosity and was in the intermediate range (30-40 ng/ml) (technically not "normal" (>40ng/ml), but not "low" (<30 ng/ml)).

Amazingly, I had higher levels than anyone else I talked to who had been tested. My friend said it was the highest level he could remember (of his patients). My doctor still suggested that I take 1 or 2 2000 IU D3 capsules per day.

If you're curious, get tested. I did have to convince my doctor that I wanted to know the number. He said, "trust me, it will be lower than it should be."

<comment><USER20> Fad or not, get tested for vit D and calcium absorption.

I was a rugby player at amateur level until 3 years ago, when I fractured twice my left wrist in 8 months. Seeing that the second fracture was due to a pretty light contact, I asked to get tested and I was diagnosed with osteoporosis. This is quite rare for a 36 years old male such as myself, I understand it may not be so rare in the next years.

</comment></comment><comment><USER21> I remember Dr. Weil saying that if you live north of, say, Atlanta, there isn't enough UV exposure for to get sufficient vitamin D from the sun during winter months. He recommended sardines as a natural source of vitamin D.

source - somewhere in his interview with Kevin Rose: <http://www.youtube.com/watch?v=t7wSp7Y4bFM>

</comment><comment><USER22> Nice infographic.

Needs to add symptoms of Vitamin D deficiency to fit the headline though.

</comment><comment><USER23> Quick note on the infographic. It is extremely important to _not take_ cod liver oil as your vitamin D source. The problem with cod liver oil is that it contains toxic levels of vitamin A, which acts as an antagonist against vitamin D. They use the same receptor and so when there is too much vitamin A in the blood, even moderate toxicity, the good effects of vitamin D will be greatly diminished.

Also, if you are going to take a multivitamin, make sure it is 100% beta carotene, as other forms of vitamin A (eg. retinol, the kind of vitamin A that cod liver oil contains) cannot be flushed out of the system and will lead to toxicity -- therefore you cancel out any benefit you would have gotten from the D. Below is a more thorough commentary on the latest research:

Dr. John Cannell, MD. via Vitamin D Council

 _"Although activated vitamin D and vitamin A signal through common cofactors, they compete for each others function. Retinoic acid antagonizes the action of vitamin D and its active metabolite. In humans, even the vitamin A in a single serving of liver impairs vitamin D's rapid intestinal calcium response. In a dietary intake study, Oh, et al, found that a high retinol intake completely thwarted vitamin D's otherwise protective effect on distal colorectal adenoma, and they found a clear relationship between vitamin D and vitamin A intakes, as the women in the highest quintile of vitamin D intake also ingested around 10,000 IU/d of retinol." [1a, 1b, 1c]

"Furthermore, the consumption of preformed retinol even in amounts consumed by many Americans in both multivitamins and cod liver oil may cause bone toxicity in individuals with inadequate vitamin D status. Women in the highest quintile of total vitamin A intake have a 1.5-times elevated risk of hip fracture." [2]

"Indeed, a recent Cochrane Review found that vitamin A supplements increased total mortality rate by 16%, perhaps through antagonism of vitamin D." [3]

"Another recent Cochrane meta-analysis concluded that although vitamin A significantly reduced the incidence of acute lower respiratory tract infections in children with low intake of retinol, as occurs in the Third World, it appears to increase the risk and/or worsen the clinical course in children in developed countries." [4]

"As early as 1933, Alfred Hess, who discovered that sunlight both prevented and cured rickets warned about vitamin A consumption, concluding, ...as to a requirement of thousands of units of vitamin A daily, the unquestionable answer is that this constitutes therapeutic absurdity, which, happily, will prove to be only a passing fad (p 662)."

"Unfortunately, Hess's prophecy of a passing fad proved premature and Americans continue consuming multivitamins and/or cod liver oil containing small amounts of vitamin D but undesirable quantities of vitamin A. For example, multivitamins, until recently, had small amounts of vitamin D (200 to 400 IU) but high amounts of preformed retinol (5,000 to 10,000 IU). This pales in comparison to a tablespoon of modern cod liver oil, most of which contains sub-physiological amounts of vitamin D (400 to 1200 IU) but supra- physiological amounts of completely preformed retinol (5,000 to 15,000 IU or, in some cases, 30,000 IU)." [5, 6]

"Clinical lore holds that Vitamin A is an anti-infective. We suggest that lore exists because of old cod liver oil studies. Semba reviewed early literature on vitamin A, usually given as cod or halibut liver oil, as a successful anti- infective. For reasons that are not entirely clear, fish liver oils of the time contained much higher amounts of vitamin D then does modern cod liver oil, perhaps because modern processing removes the vitamin D during distillation and then replace it at lower doses. Furthermore, a meta-analysis concluded that vitamin A, when given alone, increases the incidence of respiratory tract infections. If vitamin A increases the risk of respiratory infections, its high content in modern cod liver oils will only mask the full benefit of adequate vitamin D nutrition."

"As the prevalence of vitamin A deficiency in the United States is apparently much lower than the prevalence of subclinical vitamin A toxicity, we cannot recommend cod liver oil for either adults or children. (We exclude fish body oil from our warning, as it contains no vitamin A or vitamin D but is a very important source of omega-3 fatty acids.) For example, in a recent assessment of serum retinyl esters in a group of obese individuals, four percent had levels >10% of total retinol which usually indicates hypervitaminosis A. Instead, a diet rich in carrots, sweet potatoes, cantaloupe, and other orange fruits and vegetables will supply all the carotenoids the body needs to make retinol without the potential for hypervitaminosis A, especially when additional preformed retinol already exists in dairy products, eggs, and fortified cereal. We wish our diet were as rich in vitamin D. With the exception of infants on formula or toddlers drinking large amounts of milk or fortified juice, adequate amounts of vitamin D are virtually impossible to obtain from diet." [7,8,9] _

Sources (Gathered by Vitamin D Council [http://www.vitamindcouncil.org/newsletter/2008-december.shtm...](http://www.vitamindcouncil.org/newsletter/2008-december.shtml)):

[1a] Calcium and vitamin D intakes in relation to risk of distal colorectal adenoma in women. <http://www.ncbi.nlm.nih.gov/pubmed/17379616>

[1b] All-trans retinoic acid antagonizes the action of calciferol and its active metabolite, 1,25-dihydroxycholecalciferol, in rats. <http://www.ncbi.nlm.nih.gov/pubmed/15987844>

[1c] Vitamin A antagonizes calcium response to vitamin D in man. <http://www.ncbi.nlm.nih.gov/pubmed/11585356>

[2] Vitamin A intake and hip fractures among postmenopausal women. <http://www.ncbi.nlm.nih.gov/pubmed/11754708>

[3] Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases. <http://www.ncbi.nlm.nih.gov/pubmed/18425980>

[4] Vitamin A for preventing acute lower respiratory tract infections in children up to seven years of age. <http://www.ncbi.nlm.nih.gov/pubmed/18254093>

[5] The acute and chronic toxic effects of vitamin A <http://www.ncbi.nlm.nih.gov/pubmed/16469975>

[6] Vitamin A as "anti-infective" therapy, 1920-1940. <http://www.ncbi.nlm.nih.gov/pubmed/10203551>

[7] Vitamin A supplementation and childhood morbidity from diarrhea and respiratory infections: a meta-analysis. <http://www.ncbi.nlm.nih.gov/pubmed/12640379>

[8] Evaluation of vitamin A toxicity <http://www.ncbi.nlm.nih.gov/pubmed/2197848>

[9] Retinol to retinol-binding protein (RBP) is low in obese adults due to elevated apo-RBP. <http://www.ncbi.nlm.nih.gov/pubmed/18641048>

</comment><comment><USER24> What a horribly unscientific infographic! UVA "ages" the skin and UVB burns your skin but also creates vitamin D? The truth is that UVA is a longer wavelength radiation that for the most part lacks the power to penetrate the skin, while UVB _does_ do damage to the DNA. It's exactly that sort of genetic damage that "ages" the skin, in the terminology of the author. It can also lead to cancer.

Eat pretty much any fatty fish as is common in arctic cultures anyway, and you'll be fine. Ethical vegans can eat lots of mushrooms. <http://en.wikipedia.org/wiki/Vitamin_d#Natural_sources>

</comment><comment><USER25> From the map it looks like all of Canada is Vitamin D deficient. Interesting, and a bit concerning. Maybe I should be taking more supplements.

</comment><comment><USER26> Seems like it's time to buy me some cod liver oil... I hope it's not as nasty as I'm thinking it will be.

</comment><comment><USER18> Living in Seattle was brutal for me, it left me incredibly depressed and I ultimate fled back to San Francisco for my sanity. Even SF is pretty rough, so I only live in the corporate apartment complexes in SOMA where I can lay in the sun an hour a day after swimming, it keeps me quite happy (and tan!)

<comment><USER27> It would seem quite a stretch to contribute this to vitamin D, though. What with the environment, amount of sunlight, culture, etc etc.

</comment></comment><comment><USER28> Probably not, because we've lived up here in this "gray and cloudy" environment for thousands of thousands of years, and we're doing just fine; we're adapted. But we are aware that living "up here" is a problem for people who comes from "down there".</comment>


Microsculptor's Incredible Hulk Fits in Eye of Needle - <AUTHOR> http://www.wired.com/underwire/2009/08/microsculptors-incredible-hulk-fits-in-eye-of-needle/ <comment><USER2> I saw an exhibit with things like this at "The Museum of Jurassic Technology" in Culver City (LA area).

<http://www.mjt.org/exhibits/hagop/hagop2.html>

</comment><comment><AUTHOR> FTA: "I started making houses for ants because I thought they needed somewhere to live," he says. "I made them shoes and hats."</comment>


Ask HN: Is Rust worth it compared to Golang? - <AUTHOR> I've been studying Rust rigorously for about two months now and I'm almost done with the community book on Rust. However, I still feel like I don't have a strong grasp on Rust. Rust seems to have many edge cases and I like that Rust is being transparent about them, but at the same time it's very demotivating and burdensome. I chose Rust over Golang for a variety of reasons, primarily it's support for WASM and lack of a GC, but with each passing day Golang is looking better and better, ha.<p>For those whom have experience in both Rust and Golang: how long did it take to become proficient in Rust compared to Golang and which did you learn to appreciate over time over the other, if at all? <comment><USER3> I used to write everything in Golang, but recently ported many things to Rust, so I feel like I have a good idea here. I'd consider myself quite proficient in both. I see Go and Rust as almost opposites.

Go is "easy" and makes you immediately productive. Like, productive on the first day. It's a great language for that reason. Compilers don't complain, no complicated generics, easy concurrency, loose interfaces, GC, etc. But this comes at a cost - it's easy to write sloppy code. Because of its flexibility, my code often suffered from concurrency bugs (don't get me started on deadlock debugging), bad abstractions, or performance issues. I felt like I was productive, but I often had to re-factor my codebases multiple times because of these issues.

Rust is the opposite. I didn't feel immediate productive, and the fight with the compiler was tough at times. But most of the time, the compiler was right and I was wrong. This typically resulted in "good" code once it compiled and I rarely had to go back and refactor my code. And of course there were no pesky concurrency bugs due to the compiler checks.

These days, I would choose Rust over go for my own projects. If I consider the time spent on debugging bugs in Go, I consider myself more productive in Rust, even if the initial code takes longer to write. However, if I was leading an engineering team, I would consider Go instead. It's much easier to pick up, and it's easier to read someone else's Go code. Reading someone else's Rust code can be hard.

</comment><comment><USER4> I have little experience in Golang apart from reading the Tour of Go and writing toy programs, but I have more experience in Rust. In my opinion, Rust is definitely worth it, even if at the end of the day I work with other languages professionaly, its principles (ownership, borrowing, safe concurrency) really helped me to improve myself. I started to be productive with it by actually writing code, I have read the book during 3 weeks but I learned much more quickly by confronting myself directly to the compiler after. I also became productive with the right tools (vscode + rust analyzer, rust cookbook, rust by examples), the right crates (regex, lazy_static, anyhow, actix, reqwest ...) and pragmatic principles (sometimes boxing, cloning or using refcell when necessary, in non critical code). As always, use the right tool for the right job, there is a learning curve and more cognitive overhead but I really appreciate working in Rust.

<comment><AUTHOR> Thanks for your answer! I'll definitely check out Rust's cookbook. I know many hate PHP and that sentiment is slowly changing, but it's my native tongue (that I learned back in 2008 before Rust and Golang), and the cookbook I bought on it introduced me to many new concepts not exclusive to just PHP.

I would say the one aspect about Rust I have reservations about is its reference counting aspects. I feel that subject should have been baked into the language like it is for Swift with syntax but still remain as an optional or advanced feature set. But then again Rust's macro system is more than capable of smoothing out its blemishes. I just wish there was a curated list of Rust macros I could find. Would you by chance know of say a Github repo with a curated list of macros?

</comment></comment><comment><USER5> The inclusion of algebraic data types, coming from someone who uses TypeScript on the frontend and functional languages on the backend, already makes Rust the best choice for me, and conversely makes Go a non-starter. Rust definitely takes longer but I've been learning it through the Book and also by doing the exercises on Exercism [0]. The exercises really help in understanding, as for example, I didn't understand borrowing too well until doing the prime number generator problem, using a sieve of Eratosthenes which can use a mutable vector of previous primes.

[0] [https://exercism.io/tracks/rust](https://exercism.io/tracks/rust)

</comment><comment><USER6> 1) How long did it take to become proficient in Rust compared to Golang?

A long time, probably a couple months to get productive, and a couple more to learn the foundation pretty well. And this is with a steady supply of side projects to help me solidify my learnings.

2) Which did you learn to appreciate over time over the other, if at all?

I chose Rust over Go because for my side projects (at the time) Rust was faster and had a better package manager. Since most of my side projects are command line tools, Rust fit in quite well in this domain.

A side note. I think it's more helpful to choose a language based on how well it solves your problems rather than its amount of shortcomings. If Go has the right packages and tools, go with that. If Rust does, choose that one instead. For me, the ability to quickly pick up the best language for the job is IMO more valuable than picking a language with less 'edge cases'.

<comment><AUTHOR> Solid advance. I chose to learn Rust in depth first because I read (and have seen) that it's the language of choice for working with WASM, which is what I'll be doing. Cargo also seemed to be universally well received from what I read about it compared to Golang's package management. Ultimately I see myself using both Rust and Golang, but perhaps just Rust until Golang 2 is stable, since I won't even need it for another year perhaps.

</comment></comment><comment><USER7> If I were to read a book about rebuilding a car engine, I wouldn't feel comfortable with actually doing so. Experience remains necessary to reach that comfort level.

Define concrete goals with learning any language and that will help you measure progress. Rust is a massive language with many features that aren't required learning in order to accomplish many types of development. Over time, you'll probably learn them out of curiosity and continued personal development. You'll also return to code that you seek to optimize.

</comment><comment><USER8> I am probably at a similar stage to you, investing seriously in learning Rust in the past month or so. Coming from having used Go as my "main" language for almost a couple of years, already I can tell it is going to be a longer journey to become as proficient with Rust.

The biggest thing I am missing right now from Go is the fast compile times, something I rarely had to think about before. In some of the small Rust projects I've tried out, compilation takes minutes on a clean run or after having updated dependencies. When you're checking out a lot of different Github projects, or experimenting with different crates, it really adds up.

I've tried out Rust's equivalent of channels (the `std::sync::mpsc` module - [https://doc.rust-lang.org/std/sync/mpsc/](https://doc.rust- lang.org/std/sync/mpsc/)) and it feels clunky by comparison to Go. Spawning threads in Rust via a `thread::spawn()` call feels archaic compared to `go func` (putting aside that threads aren't exactly equivalent to goroutines, you might generally use them in the same way). I'd actually hope Rust could introduce first class language constructs for concurrency like Go has, at some point.

There are some other Go things like implicit interfaces I am learning to get used to without, but I am really liking Rust's concept of ownership and having proper enum types/pattern matching is a breath of fresh air. I probably appreciate Go more now for what it is - a straightforward, performant language with very good devex.

</comment><comment><USER9> Disclaimer: I am neither proficient in Rust nor Golang - rather a hobbyist using them in non-professional contexts. I have a bit more programming experience with Golang than Rust (just finished the Rust book).

I think you should identify which ecosystem and toolset you like working with. I'm leaning towards Golang for web development and web/system protocols. I chose to explore Rust as an intro to embedded development and audio/dsp programming - an alternative to learning C++. Rust seems great as a well- absracted, yet powerful and performant low-level programming language. I feel much more comfortable writing code with the Golang toolchain, but I feel much more confident with my end-result thanks to the strictness of the Rust compiler. I appreciate the specifics Golang conventions and opinions as well as the openness/comprehensiveness of the Rust language and community - <USER2> saying there aren't conventions and idioms, rather there are so many options and approaches to implementation within the language. YMMV, figure out what feels more natural to your process, style, and application context.

<comment><AUTHOR> Thanks for answering. I probably see myself concluding the same. I asked IH the same question and I ended up concluding that in my case where I'll be doing lower level programming for the WASM, the end users would probably benefit more if they used Golang. Using both is definitely an option, I'm just too unfamiliar with either language to have known the right call. I just wish Golang 2 would move a little faster, but I don't blame the community for being prudent.

</comment></comment><comment><USER10> I am actually doing the opposite transition now, I have been learning Go and using it a bit at work and on hobby projects for a while. I recently started working through the Rust book and I find it much more appealing at this point, and I am seriously considering converting my personal project to Rust to get some hands-on experience.</comment>


Site Search Could Kill Amazon - <AUTHOR> https://www.mytotalretail.com/article/how-site-search-could-kill-amazon/ <comment><USER2> While I think that they're overdramatizing the problem, perhaps Amazon should worry about this since I feel like the search is the only way to actually interact with Amazon. The quality of the metadata they use for browsing by category and filtering on features is so poor that any way of finding things on amazon besides search seems just unusable. I regularly "showroom" on other websites with better product browsing before searching for a specific product on Amazon to check their pricing.

<comment><USER3> Yeah, me too actually. Especially when searching for things like PC parts, Newegg does way better (though PC Part Picker is even better). I feel bad not buying from the interface that worked best... but Newegg's order process is pretty evil, so I don't feel that bad. ("Pay $9 to have your order shipped possibly today, instead of waiting an unspecified amount of time." Do they still do that? I saw that once and never shopped there again.)

<comment><USER4> I call this the "massive heterogeneous catalog" problem.

Generally e-commerce retailers have grown from a fairly narrow set of product categories (e.g. books for Amazon) to adding more and more diverse categories. This has a dramatic impact on site search quality.

If you consider a simple example, shoes. You only need a couple of facets to filter products to a reasonable set to browse through: gender and size. Now start adding accessories, athletic clothing, and so on, and the results end up getting harder to navigate with generic search terms like "shoe" giving less relevant results (not having the context of the user's intent hurts here).

I tried "shoe" on Amazon, got over 400,000 results with the first item being a shoehorn. It takes a bunch of clicks to deal with that.

This search problem gets worse as catalog sizes grow even bigger. Personalized results help a lot and Amazon seem to fail me with this, they don't do a good job bubbling up the products I buy to the top.

It's a hard problem to solve but it's not going to kill Amazon.

<comment><USER5> Absolutely - this challenge is especially acute for all ecommerce retailers with broad catalogs. Target, macy's, walmart, jet, etc all face this challenge.

Systems like Solr, elastic search and endeca (out of the box) all assume relevance means keyword frequency in a product page, with some weighting depending of title, description, tag, etc. Delivering relevant results that users might want to purchase requires taking these systems, adding or customizing their NLP techniques, operationalizing historical user search & purchase data to determine intent, personalizing by shopper history, etc.

The challenges of massive heterogenous catalog affect other areas... Chief among them search result personalization… an individual's gaming purchase history might cause 'button down' to return gaming keyboards, rather than oxford shirts, while a pet products purchase history could lead to a search for turkey returning turkey dog food.

The fact that Amazon fails to personalize search results is evidence of the difficulty & opportunity here. The sort of pervasive personalization found in AirBnb, facebook, google are simply out of reach of most ecommerce retailers…

<comment><USER4> > Target, macy's, walmart, jet, etc all face this challenge.

I've been buying more household items from Jet recently because their smaller category is easier to navigate. Plus there's no third-party sellers and no pricing confusion like there is with Amazon vs. Amazon Fresh vs. Amazon Pantry.

> among them search result personalization

Agree 100%. Retailers need to consider the sometimes overlapping contexts of browsing history, purchases, and importantly the current browsing session (with weighting given to cart contents). Somebody currently browsing for food items should see food items when searching for "turkey", not dog food.

My favorite search example that fails without context is "dress". Does it mean "dress", "dress socks", or "dress shirts"? Even if it means "dress", are we talking about women's or girl's dresses?

I did an experiment a few years ago and found that it was possible to improve search relevancy dramatically by keeping track of items looked at and purchased, bucketing by category/sub-category, with an exponential decay and using this to boost popular categories in results. It's terribly low tech, but it gives a lot better results than no personalization.

There's a bunch of retailers that I visit frequently (and purchase from) that force me to search, the filter by men's, and do this for _every_ search. It would be great if they could just learn this coarse-grained level of personalization.

> The fact that Amazon fails to personalize search results is evidence of the > difficulty & opportunity here

The opportunity for Amazon is massive. They don't seem to consider my purchase history at all when ranking products, for example if I search for "olive oil", the 31st item is the one that I've purchased three times in the last couple of years and the _only_ olive oil I've purchased.

I've spent a big chunk of the last decade trying to improve ecommerce search and it's a very neglected area across the board.

</comment></comment></comment></comment></comment><comment><USER6> I didn't read the whole article. When I got to the page I was presented with a modal for something I didn't care about. Read the first paragraph scrolled a tiny bit and another modal popped up so I just closed the page. You know what kills sites, an annoying user experience.

</comment><comment><USER7> My dream for an online store is the following:

A magic list that I can add _any_ item to, select a delivery date, and have the backend automatically fulfill each item, at the cheapest price, on time. I want to completely abstract away the concept of a store. I want a to-do list that automatically delivers stuff to me. Additionally, I want a standardized return policy no matter who fulfills it.

I want to be able to say:

\- (3) Organic Roma tomatoes

\- (6) Sonicare replacement brush heads

\- (1) Patagonia Down Sweater, Black, Medium

\- (1) Bookshelf assembly service, in-home, 3/16/19

and magically have it fulfilled. One could have a slider to make the tradeoff between speed of fulfillment and cost.

Amazon already sort of does this, but not at the best prices. One would hope that in the magic to-do list model, local retailers would be able to outbid non-local retailers, as their shipping costs would be inherently lower.

<comment><USER8> Wait so, you want shoppers to present a purchasing list, and have retailers compete to fulfill that order? Essentially like an auction with a ceiling?

<comment><USER7> Yes.

</comment></comment></comment><comment><USER9> I feel like the title of this article is way, WAY overblown.

</comment><comment><USER10> Not better managing 3rd party sellers is what's going to kill Amazon. It's pretty common to see orders sit two weeks without shipping, or marked as "shipped" with no tracking number, or counterfeit items, items different than what's pictured/described, etc.

Banning the crap sellers would also make search easier. Less duplicate listings.

<comment><USER11> Yes, the biggest threat to Amazon is Amazon. If buying something from Amazon becomes as much of a crapshoot as buying from eBay, then why bother? St least if I buy from say, B&H or Elderly, I know I'm going to get the thing I paid for.

<comment><USER12> This is where I'm at. If it doesn't say ships from and sold by Amazon I don't buy it. This is true on other sites that have integrated third party sellers as well. It's not worth the shipping delays, possible counterfeits etc.

</comment></comment></comment><comment><USER13> I'm not sure why I'm replying to click-bait PR, but here it is anyway:

Maybe I'm alone here, but I have an extreme distaste for personalized search results. I know what I want to see; it's pretty rare search knows what I want to see when it's trying.

Take the "chips" example from the interview--maybe I'm tired of eating the same fucking chips every day. Maybe I just want to know what the most popular options out there are without having to scroll a lot. Not to mention that I have a preference for Pringles, I'm going to type the brand name into the search when I want to find it. (If I'm a dog lover, I don't expect Google to rank dog results first when I type in "animals." I expect information about animals in general.)

<comment><AUTHOR> Founder here. It's a probabilistic problem from the perspective of solving personalization. If you tend to want something different after trying the same thing a bunch of times, the algorithm should learn that as well-- that's also personalization.

There's also been a ton of research done that most consumers do want personalized experiences, will pay more for them, and will be more likely to churn if they don't have them. There's a pretty good list at [https://venturebeat.com/2017/08/18/hyper-personalization- mar...](https://venturebeat.com/2017/08/18/hyper-personalization-marketing-to- a-segment-of-one-vb-live/).

Some examples from that article:

>>> Forrester uncovered the fact that 77 percent of consumers have chosen, recommended, or even paid more for a brand that provides a personalized service or experience.

Accenture found that 75 percent of consumers are more likely to buy when you show you recognize them as an individual and provide recommendations based on their unique wants and needs.

</comment></comment><comment><USER14> The best way to currently search Amazon is through Google. But if we're being honest, how many sites can really say that ISN'T the case? Search is hard. It's a problem that a company with the resources of Amazon should have fixed but to say it's a potential killer is more than a bit overblown. All the subpar 3rd party sellers probably have a more significant impact on bottom line.

</comment><comment><USER15> Eli & his team are very smart -- personalized search is incredibly difficult, and very few people actually do it. But realistically speaking, there's not a product out there that wouldn't benefit (as long as the price point is ok).

Looking forward to see AI search grow in popularity and become democratized for developers.

</comment><comment><USER16> If site search is something that Amazon does that badly (which they do), then why would you think that doing it well is the key to beating them? If it's not _necessary_ to be a sales giant, then why assume it's _sufficient_?

<comment><AUTHOR> This is a really fair question. Amazon doesn't do it well, but just about no one currently does it better. It's like looking at Yahoo or AltaVista before Google came out and saying they're plenty successful without having incredible search.

Of course, it's not the same, and retailers need to do a hundred other things correctly as well, but there is a lot of data to show that people go where the good search is.

</comment></comment><comment><USER17> The first company to hit viral status with "centaur" shopping models will kill Amazon the way Amazon killed retail.

<comment><USER18> What are centaur shopping models?

<comment><USER17> I guess I as unclear — "centaur" by analogy with centaur chess, which is when you have an AI and human play together.

A company which faithfully executed an AI for me, eg looking at _every_ pair of jeans and then using NLP or whatever on the reviews to rank them based on my shopping history and stated preferences, measurements, etc.

In many ways, sites are already stumbling in this direction: they do analyze purchases and so forth, but the level of granularity is poor and the control that the customer has over it is basically non-existent.

But whoever takes the (ironically Amazonian) approach of serving customers with AI (as opposed to using the same tech for marketing at them) is going to win, because they'll have solved the discoverability problem — which plagues online shopping (and online things generally).</comment></comment></comment>


Paving the way for human-level sentence corrections - <AUTHOR> https://tech.grammarly.com/blog/paving-the-way-for-human-level-sentence-corrections <comment><USER2> The problem that the authors are trying to tackle is an interesting and difficult one.

I have noticed that when dealing with natural language using artificial intelligence / machine learning techniques, the work being done by computer scientists very often would have greatly benefitted from collaboration with a linguist or other sort of language expert, especially in the design phase of an experiment. This work is a good example of what I mean.

People trained in CS or similar precise fields develop, over time, a tendency of thinking in terms of "getting the right result" (I say this as one of these people). When dealing with natural language, however, sometimes there simply is no single correct result.

Consider the topic of fluency that the authors work on: is there a rigorous, objective definition of "fluent"? The answer, as any linguist would tell you, is "no". There are idiomatic expressions, grammatical structures, contractions, slang, and so on that vary from city to city within a country, let alone globally. What may sound "fluent" to one native speaker of a language may sounds strange to another. It is impossible to objectively generally evaluate "fluency". In particular, any practicing linguist will be able to give examples, likely off the top of their head, of English sentences that would be rated as "fluent" by someone from one geographical area and "awkward" by someone from another.

Furthermore, using Mechanical Turk to find humans to rate the fluency of a particular sentence makes for an unclean dataset and evaluation benchmark. The linked post says that, in the end, 50 people found via Mechanical Turk rated sentences for fluency; since any one language is used significantly differently around the globe, there will be an unpredictable range of fluency ratings for at least some sentences across just 50 people around the world. Choosing a different 50 people to rate the same sentences would most likely result in different fluency ratings.

I do not mean to detract from the authors' work; this is a difficult problem to tackle, with no clear path to a general solution. However, I am forced to wonder why the authors, who, based on their biographies linked to in the article, seem to have a range of experience, did not comment on the considerations I've mentioned here.

<comment><USER3> Maybe they went to ten linguists, and all they got as an answer is "there is no objective definition of 'fluent'. You are trying to find a single correct result that doesn't exist!"

Then, armed with the naiveté of thinking that if there is something like 'fluency' it must be possible to measure it, they just threw a bit of money at the problem. Note that asking a representative group of people is the closest you can get to exactly what you want to measure (apart from asking everyone). It doesn't matter that there's no agreed-upon method to measure the quality of pizza: if I maximise the subjective impression, I'll get exactly what I wanted.

</comment></comment><comment><USER3> I don't quite get this... According to the bar graph, the automated systems fail to correct around half of even orthographic (spelling) mistakes. One example is "advertissment", which macOS is now trying really hard to correct against my will in this text area.

Another example is "From this scope, social media has shorten our distance", where "scope" is supposed to be "perspective". That seems to be something that machine learning should easily pick up on, and indeed, when I just tried it on google translate, I couldn't get it to make this mistake without my original (german) sentence also encroaching awkwardness.

So I'm unsure how much value it is to win against systems that fail rather spectacularly. I also don't quite understand why you would need manually- created data for this task, instead of just buying everything ever written in TOEFL essay questions and pitting it against the New York Time's archive.

It's obviously quite likely that there are good reasons for all this. They may have thought a bit longer about it than I just did.

</comment><comment><USER4> Hmm I'm sensing a bit of garbage-in garbage-out here. For starters their original sentences contain unlikely typos instead of homonyms which would be much more commons. (it complicates the learning as well I'm sure since some changes were made to correct similarly spelled terms which could really change a sentence's meaning once applied) Second, the human corrections aren't that good. We really need to stop creating data sets using anonymous exploited labor that is paid pennies. (they did screen the Amazon Turk users, but if you live in America or work at a university, is there really a shortage of fluent English speakers around you?) Overall I'd say the fluency-editing approach shows promise and would be a boon to ESL-learners, but the training data needs to be improved.

</comment><comment><USER5> This is a great project -- in Phase One, the algorithm will correct sentences written by people who didn't learn basic literacy in school and who subsequently endeavor to avoid reading or writing any text, preferring video. In Phase Two, the algorithm will do away with the poorly written source and create something entirely on its own. Based on my sampling of contemporary human-crafted sentences, Phase Two will take place just in time.

Apropos, my all-time favorite malapropism took place 50 years ago when I was a teenage TV repairman. I visited a household, spied a record turntable, and asked, "Is that a stereo turntable?" "No," replied the customer, "It's _monorail_."

I was able to avoid blurting out, "I think you mean _monaural_ , yes?" \-- for three reasons. One, it's regarded as bad form to correct the grammar of customers, who are always right. Two, technically, the turntable was in fact monorail (i.e. able to follow only one recorded track). Three, I was too busy trying not to laugh.

<comment><USER6> > In Phase Two, the algorithm will do away with the poorly written source and > create something entirely on its own. Based on my sampling of contemporary > human-crafted sentences,

A problem with this is that there will be a tendency for it to become normative. This is what happened to the OED. Originally it was an etymological dictionary of the usage of English. Now it is regarded as an arbiter of 'correct' English.

<comment><USER5> > A problem with this is that there will be a tendency for it to become > normative.

Yes, true. It would turn description into prescription, but we're already approaching that point. I'm not advocating this, only mentioning it.

> This is what happened to the OED. Originally it was an etymological > dictionary of the usage of English. Now it is regarded as an arbiter of > 'correct' English.

I suspect those behind the OED would deny that as a goal, while acknowledging it as an outcome.

I have a little fun with people who think dictionaries prescribe correct usage, by pointing out that, according to current dictionaries, "literally" and "figuratively" mean the same thing. This is true because that's how people use the words, and a dictionary's purpose is to dispassionately record how people use words, without judgment or rancor.

This is why "reign it in" (now seen regularly) will become an accepted substitute for "rein it in" \-- people want to say it that way, so be it. _Reigning_ is what a monarch does to a kingdom, _reining_ is what a cowboy does to a horse, but people are free to say what they want.

</comment></comment></comment><comment><USER7> This was an interesting read for someone unacquainted with the field–it appears to be very difficult to fix "awkwardness" in sentences; none of the methods were able to reduce it significantly. It looks to me that awkwardness is more based on common usage than on actual grammar, perhaps this could be improved with a solution similar to Google Translate's, which looks at real world usage instead of syntax?

<comment><USER8> Real world usage would have to be curated though, awkward sentence constructions or word choices do happen in real world usage. Or as the article shows, there can be multiple, very different ways of fixing awkwardness. I'm not sure what it would look like to find a solution that's "fitted" to several of these.

</comment></comment><comment><USER9> Fluency is nice, but semantics matter the most.

</comment><comment><USER10> s/are comprised of/are composed of/

<comment><USER11> Are you saying that "are comprised of" is not grammatically correct in its context in the article? Why?

<comment><USER10> [https://en.wikipedia.org/wiki/Comprised_of#Evaluation](https://en.wikipedia.org/wiki/Comprised_of#Evaluation)

</comment></comment></comment><comment><USER12> I have a feeling this is one of those places where ML will not be useful until we have strong AI.

Certain grammatical errors are impossible to fix unless you understand the overall meaning of the text. Sometimes this meaning is embedded over many paragraphs. Errors involving incorrect word usage are unsolvable when words have more than one meaning and you don't comprehend the subject at hand.

<comment><USER13> Once we have strong AI, whatever that buzz word means, what then would be the usefulness of understanding slang?

Personaly I think the usefulness is already to be able to enterpret a concept encoded in slang as the same as the concept derived from a message encoded in a different dialect (or language).

I would never assume a machine spoke this language, only that they understood it. Machines should evolve into speaking succinctly as to not include unneccessary complexity in their messages as they would strive to be well- understood like all other persons do. I fail to see why we would want to produce slang-encoded messages, unless we want to mask the fact we are a machine.

<comment><USER12> Ambiguous messages do not imply slang. Plenty of words have multiple meanings in normal and formal English. It's a much worse problem in tonal languages like Chinese. Tell me how you could grammatically correct this without understanding meaning [https://en.m.wikipedia.org/wiki/Lion- Eating_Poet_in_the_Ston...](https://en.m.wikipedia.org/wiki/Lion- Eating_Poet_in_the_Stone_Den)

Strong AI isn't a buzzword either, it's been in use for as long as I can remember. Maybe you would be able to understand my Grammer better if I said super human general intelligence and wasted a bunch of space in the process.

I don't think you read my comment? You seem to imply that the corrections would be unambiguous while my point was that some errors are uncorrectable without understanding meaning.

<comment><USER11> > _Plenty of words have multiple meanings in normal and formal English._

There are some stats from Wordnet on polysemy in English. Obviously this depends on the granularity of a set of senses in a dictionary, but regardless English has many polysemous words (26,000+ according to Wordnet). And more importantly, these polysemous words also tend to be the most common words, hence words like "set" having around 120 definitions in the Oxford English dictionary.

[https://wordnet.princeton.edu/wordnet/man/wnstats.7WN.html#s...](https://wordnet.princeton.edu/wordnet/man/wnstats.7WN.html#sect3)</comment></comment></comment></comment>


It's scary to change careers - <AUTHOR> http://karenx.com/blog/its-scary-to-change-careers/

<comment><USER2> Scary and exciting!

Seven months ago, I became a full-time pastor. Quite different from web development using Java and programming in general. Twenty three years working as a geek is not the most helpful background for pastoring. On the other hand, I love the pastoring and I made sure that the church has a decent web page. :-)

The biggest thing for me to get used to was (still is) managing my time, being a 24x7 people person (I'm a type B personality) and juggling the needs of a congregation. Do I visit A, B or C today? And don't forget to phone D, pop an email to E and be sure to remember that you have lunch with F today and breakfast with G on Saturday. And must have a sermon ready for Sunday. And be available for people to phone or text or Facebook chat with. It's all good, but can get complicated because of the people issues ... and it's all people issues!</comment>


WP Secure Downloads, An Ecommerce Solution For Selling Digital Products - <AUTHOR> http://tomuse.com/wordpress-secure-download-plugin-ecommerce-sell-digital-products/ <comment><USER2> Looks quite useful and is something I'll be exploring to use for a project I've got in mind. It is a bit more expensive than I would have expected. I understand it's meant for commercial customers who can afford the cost, but had you described this to me I would have guessed half those prices as an upper limit.</comment>


We are the owner and one of the professors from UnemployedProfessors.com AMA - <AUTHOR> https://www.reddit.com/r/IAmA/comments/5v7wvg/we_are_the_owner_and_one_of_the_professors_from/ <comment><USER2> The owner apparently decided to withdraw their IAMA but you can see a cached version of the post here:

[http://archive.is/EdvDf](http://archive.is/EdvDf)

> I'm the owner of UnemployedProfessors.com, and I'm joined here today by > /u/IamDeleuzienne and /u/IamProfessorRogue, two of our star writers. You > might have seen an article about us on reddit's front page last week (TIL), > which led to several requests for an AMA. We're here to answer your > questions and share our stories. Proof We get some pretty crazy clients and > a lot of hatemail, which we understand. We help people cheat. Sometimes we > are infiltrated by journalists or other academics who try to dox us. > Sometimes people try to 'out' us as a fake paper mill. For example, this > Daily Dot journalist who planned to expose us, only to receive an A- paper > written by /u/IamDeleuzienne As far as we know, we're the only major site > that writes original papers, guaranteed to pass TurnItIn. Edit: 9pm EST - > We're still answering questions, and will be answering them throughout the > evening, but there are around 1500 comments, I need to pee and my professors > are doing work for the site in between responding, so there may be a delay.

Edit: Apparently UnemployedProfessors.com serves up everything, including account login, via insecure HTTP: [https://www.reddit.com/r/IAmA/comments/5v7wvg/we_are_the_own...](https://www.reddit.com/r/IAmA/comments/5v7wvg/we_are_the_owner_and_one_of_the_professors_from/de02xhb/)</comment>


Straight White Male: The Lowest Difficulty Setting There Is - <AUTHOR> http://whatever.scalzi.com/2012/05/15/straight-white-male-the-lowest-difficulty-setting-there-is/

<comment><USER2> Very relevant to a number of recent HN discussions on sexism. The opening paragraph rings a bell:

 _I've been thinking of a way to explain to straight white men how life works for them, without invoking the dreaded word "privilege," to which they react like vampires being fed a garlic tart at high noon. It's not that the word "privilege" is incorrect, it's that it's not their word. When confronted with "privilege," they fiddle with the word itself, and haul out the dictionaries and find every possible way to talk about the word but not any of the things the word signifies._

</comment><comment><USER3> Is this really tech news? I'm exposed to this argument daily on Reddit, and it hasn't changed how I feel about anything, except I'm more tired of hearing it. What else do people intend to accomplish? I already treat my fellow humans as equals. Do they want me to pity them?

</comment><comment><USER4> Best explanation I've ever read for what "majority privilege" means.</comment>


Slickmap CSS file to map out sites during design - <AUTHOR> http://astuteo.com/slickmap

<comment><USER2> Neat. This reminds of a prerequisite charts for university courses[1,2].

_______

[1] <http://www.texample.net/tikz/examples/labs-schema/> [2] <http://www.texample.net/tikz/examples/prerequisite-chart/>

<comment><AUTHOR> You've just reminded me of the time I spent parsing through university course books -- what a nightmare; I have to assume these days they've improved on the process of 'look up course #, turn to the back of a separate book, dig through an array of numbers and hope for the best..'

Somehow through it all I ended up in three calculus courses simultaneously and decided university wasn't for me.

</comment></comment><comment><USER3> This is actually from July 2009. Not that it matters all that much as I'm sure it is still useful but I thought I should point it out.

<comment><AUTHOR> Sure is! :) Just found out about it earlier today and it'll come in handy for a site redesign I'm doing, thought I'd spread the word.

</comment></comment><comment><USER4> That's beautiful, also the fact that you think about restful url's beforehand is a big plus. It's also good to think about the language/terms that you want to use throughout your application.

</comment><comment><USER5> Are site maps still a thing? I don't remember the last time I actually saw one. Still pretty nice though, lots of other users for this type of chart.

<comment><USER6> I'm about to use it for documentation and reference purposes.

It feels like a great tool for that

</comment></comment><comment><USER7> Not just "sites;" I can see this being great for some API's topology.</comment>


Why I'm finally dumping Ubuntu and switching to a Mac - <AUTHOR> http://mariusandra.com/blog/2012/07/why-im-finally-dumping-ubuntu-and-switching-to-a-mac/

<comment><USER2> That is why I still beleive open-sourced, non-profit organizations will never be able to compete with the major powerhouses. Apple and Microsoft have the resources gained from charging high prices for their products to be able to constantly innovate and perfect their products. Ubuntu is an operating system built on the free time of developers, that does not receive the complete attention needed of a full fledged product. Ubuntu will have to change it's philosophy to be able to compete with the big powers.

<comment><USER3> There are numerous counter examples of open source software that successfully competes or dominates the competition. While Ubuntu may be a victim of its philosophy, the little you say does not convince me when Mozilla, Apache, WordPress, and even the linux derivative Android are if not #1 in their fields, at least impressive competitors. Of these I think only Android is run by a non-profit organization.

</comment></comment><comment><USER4> Until you run into another small annoyance on your mac (or system of choice)

I switched to linux because when something annoys me, it is much more likely I will be able to fix it.</comment>


Top 50,000 paying Adsense keywords (purchased from a shady looking website) - <AUTHOR> https://docs.google.com/leaf?id=0B2QExXr67OcTZjNlNDYwMTktNmE4YS00MTYyLTg1NGUtMmJhMzg0YmIzZjBi&hl=en <comment><AUTHOR> So, last March, after two startup ideas bombed, I was doing some research on where online advertising revenue was being spent. I figured that if I was going to even partially monetize with ad revenue, it made sense to figure out what topics and niches were paying.

As part of that research, I ended up purchasing a list of the top 50k paying keywords off a shady looking website. The site only took PayPal, and they were only charging 50 bucks, so I figured it was worth it, and I wasn't risking too much aside from exposing my PayPal account, so I bought it.

I was pretty skeptical of the data at first, but I randomly spot checked around 50 search terms, with my Adsense account, and they seemed to correlate pretty closely.

I then wrote a script to cross reference how much people were paying for each set of keywords with how many results were returned for those keywords. That gave me a ratio of High cost keywords::number of results, which was enlightening.

Mind you, this data is suspect. Google doesn't charge a fixed rate for their advertising rates, from what I understand, each time an ad is served, the price essentially depends on an auction and the price varies accordingly. Also, I bought this data off a shady looking website during a really low point in one of the worst economies in decades. _Caveat Emptor_

That being said, I thought you might find it interesting.

<comment><USER2> Do you have the rights to re-distribute this data?

<comment><USER3> Facts can't be copyrighted. There's no problem with redistributing the data.

<comment><USER4> Facts and ideas can't be copyrighted, but their expression and structure can.

<http://www.templetons.com/brad/copymyths.html>

This is definitely a copyright violation, and I am shocked and enraged that this was voted so highly on HN and not flagged.

[Edits]

1\. The standard "Recipes and collection of recipes" argument. This is a collection of recipes.

2\. I am not shocked that there is a copyright violation. I am shocked that such an egregious violation has been voted so highly on HN. We are all digital workers on HN, and such disregard for digital work is (still) shocking.

<comment><USER5> Enraged? Your reaction doesn't fit the crime, as is so often the case with copyright.

</comment></comment></comment></comment></comment><comment><AUTHOR> I throw myself on the court of community opinion. Please feel free to flag this post if you think it's unethical or wrong. And, the mods and/or the HN algorithm will mark it dead.

I'm really not trying to be a ass. I thought it was interesting, and the recent post where the author was driving traffic to his blog by posting the top 50 keywords reminded me of it. I did this project over a year ago in March of '09. I don't think I'm giving away any trade secrets at this point.

I thought I'd share the whole list I'm sure he bought in the spirit of openness. But, as I said, it's really no skin off my nose. Feel free to flag this if you think it's inappropriate. I'm really not up for getting flamed or arguing.

<comment><USER6> Well, you certainly convinced me to download with this comment :]

</comment></comment><comment><USER7> These surface on the SEO underground every once in a while. I personally don't think the data is that valuable except for getting less-than-savvy folks to pay scads of money for a CSV file (present company excepted, most folks who buy this are going to be the DigitalPoint "How do I make $1 a day with AdSense!?" crowd that buy most Make Money Online products).

That said, the existence of lists like this always makes me shoot a baleful glare in the general direction of Google's "transparency." They are very happy to give away all _the rest of_ the world's information, but anything Big G thinks is commercially sensitive _for Google_ is guarded like Fort Knox. (Did you know AdSense users can't even tell you the CPC rates they get without violating the TOS?)

<comment><AUTHOR> You've hit the nail on the head. I didn't really want to bring up that issue, but I'm glad that you did. Google also makes it against their TOS to automatically crawl their search content: [http://www.google.com/support/webmasters/bin/answer.py?answe...](http://www.google.com/support/webmasters/bin/answer.py?answer=66357)

IMHO, that's patently absurd considering the business they are in.

re: transparency:

In what other business would it be remotely okay to not talk about the purchase price, or the sales price of an item? That's just patently silly.

And, the transparency really only gets worse the closer you get to Google headquarters. I can't tell you how many Googlers that I end up having brief chat's with, that can't talk about what they do or what they are working on in more than very vague terms. It's creepy.

re: _I personally don't think the data is that valuable except for getting less-than-savvy folks to pay scads of money for a CSV file_

You're absolutely correct. The list is really not that valuable, but you see where people take that information and run with it putting out hundreds of cheap SEO sites that target mesothelioma. Where I did find value in it, was in categorizing sectors where people pay a log of money for customer lead generation. That's Legal and Financial.

So, I mulled that idea around for a while, and I decided I would try and focus my startup, <http://Newsley.com> on the Ecnomic and Financial News space. One quarter of all online ad spend is devoted to Financial services marketing. It's a 3.5 Billion a year industry. And, I dont' really see that many financial services ad that much, so they must be paying a lot.

I've been taking Newsley in a different direction now that I'm part way into the business, I'm talking to a lot of people that manage large sums of money for a living, and I'm picking their brains about their news finding and filtering problems. It's been really valuable. My current business has really nothing to with the original list, but it at least pointed me in a certain direction, and I'm quite happy I did it.

</comment></comment><comment><USER8> I'm flagging you. I was one of the people paying 50 bucks for it, and this just starts some kind of stupid DRM battle. The guys did the work to accumulate the data, they're not asking for your firstborn. Giving it away is just screwing over somebody else's business model simply because you can.

</comment><comment><USER9> So somebody spent the time to figure this data out, is selling it, and you're now giving away their product for free? Is there more to this story?

</comment><comment><USER10> It's forbidden right now, is there any other url?

<comment><AUTHOR> [http://ran.dom.crap.s3.amazonaws.com/Top%20Google%20Key%20Wo...](http://ran.dom.crap.s3.amazonaws.com/Top%20Google%20Key%20Words.csv)

</comment></comment><comment><USER11> Likely this data comes from google's keyword tool, which has a paid API.

semisland.com/semrush.com has alot of good data of this type. Also, they scrape the ads and organic rankings for a ton of keywords.

</comment><comment><USER12> For those that don't want to download, I'll give you a major theme in this after a quick perusal.

1\. It's very hard to bid on trademarked names. For example "Disney" may take issue with your "Orlando on $20 a day" clickbank ebook ad.

2\. It's a little less competitive if you use misspells. So bidding on "Dinsey" could provide higher CR. This is a fairly well-known theme.

Do note, my example was not included in the file.

<comment><USER13> _Caveat_ : a trademark protects the owner from you attempting to confuse the public into buying your product because they think it comes from the trademark owner. Using typos is attempting confusion. Thus simply relabelling things as "dinsey" doesn't avoid a potential trademark suit.

</comment></comment><comment><USER14> Has anyone actually looked at these? Most of them have typos (ie: ameratrade instead of ameritrade).

I don't think this is the top 50k search terms, maybe it's the top 50k when there was a google suggest or something?

I refuse to believe "visaa" ranks higher than "visa" and that "ameratrade" was #1.

<comment><USER15> It's not a list of search volume, it's a list of the top keywords ordered by cpc price in adsense. Some (most?) likely have little to no search volume at all.

<comment><USER14> Ah my mistake, thanks. Makes sense then that one offs are the most valuable (lowest competition, latches on to a real one)

<comment><USER16> Also it could be that people who misspell these terms are easier to monetize.

<comment><USER17> Now that would be a fascinating study indeed! The correlation of misspelling and impulse shopping :)

</comment></comment></comment></comment></comment><comment><USER18> 10 Big Myths about copyright explained <http://www.templetons.com/brad/copymyths.html>

<comment><USER19> And the relevant tidbit: "Facts and ideas can't be copyrighted, but their expression and structure can. You can always write the facts in your own words though."

<comment><USER20> It's not a fact, it's a database and databases are protected by law.

<comment><USER13> This depends on your jurisdiction.

</comment></comment></comment></comment><comment><USER21> Has anyone downloaded this yet?</comment>


Google Wave Versus the Rest, Feature by Feature - <AUTHOR> http://lifehacker.com/5451174/google-wave-versus-the-rest-feature-by-feature

<comment><USER2> I'm amazed that people are still maintaining the breathless hype about Google Wave. Look, if you have to make a comparison table, or write a book with the most frequently asked questions about it, or actually _convince me_ that a piece of new technology is absolutely fantastic, then that technology isn't.

If it was fantastic, if it was disruptive, if it was the best thing since sliced bread, you wouldn't have to do this. All my friends would already be telling me about how fantastic it was. With Google Wave, all my friends said "meh" or "is this it?".

It might become awesome with a better client or a different client or different service sthat plug into it, but as long as it remains the way it is now, it is completely uninteresting. It's not news.

Another funny thing, this article compares Google Wave to services it explicitly distinguishes itself from. How about comparing it to other online collaboration tools that are actually in the same niche? Or would that turn out too negatively for Google Wave and spoil the hype?

<comment><USER3> The ironic thing about Wave is that despite Google's positioning it to get everyone working on the internet this is a product that will prove its value on corporate intranets.

Enable authentication via Active Directory and Wave can be a SharePoint killer, not to mention how it could cut the volume of internal email in the corporate world. One big thing I would like to see is how hard a company- hosted Wave becomes to manage once you get a lot of content which is a big pain with systems like SharePoint.

As Wave is/will be open-source I see an opportunity for those who want to bring it inside corporate environments. I don't think Google can do that, the model needed to sell to and support the corporate world is not in its DNA.

</comment></comment><comment><USER4> Often the "I have more features charts" ignore the fact that more features often means worse user experience. It's the elegance, and flow mapped to how we visualize, think and communicate that counts. Not easy to chart.

</comment><comment><USER5> One of my friends is behind a corporate firewall that blocks wave.google.com. Our solution? A shared "pseudo-wave" in google docs. Turns out to work remarkably well for us with near RT chat features (his firewall also blocks IM).

</comment><comment><USER6> Google Docs allows users to compare revisions and link to other documents, despite the chart saying it doesn't.

</comment><comment><USER7> this chart clearly shows that everything you can do with Google wave can already be done with other tools.

</comment><comment><USER8> Does anyone know of a single living soul that actually sat through the google wave introduction video?</comment>


InfluxDB v0.10 GA with hundreds of thousands writes/sec, 98% better compression - <AUTHOR> https://influxdata.com/blog/announcing-influxdb-v0-10-100000s-writes-per-second-better-compression/ <comment><USER2> We are using 0.10 GA in production. We have seen lot of disk space reduction from 22GB to 700MB and performance improvement order of magnitude. Influxdb is the modern timeseries database.

Keep Rocking !!

<comment><USER3> I wonder what sort of data you're storing with that reduction? I'm using it as a drop-in replacement for Graphite (with Grafana as a front-end, loving both BTW) and just had disk space reduced from 140GB to 250MB :D

</comment></comment><comment><USER4> Be _very_ skeptic of this, I wasted weeks trying to get git master versions of this not fall over including tsm1 storage engine. They are very good at marketing and made nice query and API entry, but data storage has been a total shit show.

Looking at less than a month of issues:

* [https://github.com/influxdata/influxdb/issues/5440](https://github.com/influxdata/influxdb/issues/5440)

* [https://github.com/influxdata/influxdb/issues/5482](https://github.com/influxdata/influxdb/issues/5482)

* [https://github.com/influxdata/influxdb/issues/5534](https://github.com/influxdata/influxdb/issues/5534)

* [https://github.com/influxdata/influxdb/issues/5540](https://github.com/influxdata/influxdb/issues/5540)

If you need something that won't fall over, it's not glamorous and a bear to setup, but OpenTSDB will sail with massive load once you've gotten it running.

<comment><AUTHOR> Not sure what issues you're seeing, but we have people running this in production at significant scale.

One of the issues you linked to was for 0.9.6. Others were there because they had super high tag cardinality and not enough memory to actually run.

When people post comments like "I have 50 million series and it crashes on my box with 2GB of memory!!!", they're not relevant. You shouldn't expect miracles...

<comment><USER4> No, and you're countering rhetoric with rhetoric so here are the specs: bare metal 96G RAM, 16 cores, 36 disks (also tried 36 SSDs). Nothing exotic, and something similar you should be using to test before releasing another ill- fated storage engine based on "It works on my Macbook Pro with an artificial load test, ship it!". On the same box carbon is usable, and TSDB can scale linearly across a cluster of these machines. Influx burns down as soon as more than a few GB of metrics are persisted even after backing off collection.

<comment><AUTHOR> I'm not quite sure what you're doing. If you have that level of hardware I don't know why you're running out of memory. Our tests with 100B points split across 1M series ran on much more modest hardware than that and didn't have a problem.

There's something different with what you're doing than with what we're testing. If you can give more detail about what your actual data looks like, I may be able to help.

Are you doing this on v0.10 (beta1 or greater) and you're seeing this problem?

<comment><USER5> > If you have that level of hardware I don't know why you're running out of > memory.

Did you folks fix the "oops, I accidentally selected too much and OOMed the machine" crash that was supposed to be resolved by the new query planner in .10? That was my immediate showstopper for InfluxDB for a very high-volume infrastructure, because I didn't feel like proxying InfluxDB just to enforce chunking, and the mere existence of the bug gave me pause. I hit that in 24 hours and shelved the system in 48.

Even pre-1.0, I'm not encouraged when I have to "work around" pretty obvious oversights in reliability, so I put you back on the back burner to give you time to mature and transitioned back to a Kappa-style architecture for my needs. I might still use InfluxDB for low-volume aggregations out of my stream processing, but it's for sure out of the hot path for the foreseeable future.

<comment><AUTHOR> The current version still has the problem that if you do a massive SELECT then it will fill up memory until the process gets killed. In the future we'll give controls to limit how much memory a query can take.

I think it's common in databases that if you throw a massive query at it that the server doesn't have the resources to handle, things will go wrong. It'll thrash, or crash, or generally have poor performance. We'll be working on improving the failure conditions, but if you put a query to a database that is too big for the server to handle, some sort of failure scenario will be encountered. Just a question of how it's handled.

Also, there's no new query planner in 0.10. We have a bunch of work getting merged in for the query engine for 0.11. But if you throw a huge query at the DB that the server can't handle, you'll still have great sadness.

Which DB ended up solving your big query problems? Maybe we're just a poor fit? Or what kinds of queries should we be working on optimizing?

<comment><USER6> _The current version still has the problem that if you do a massive SELECT then it will fill up memory until the process gets killed._

That means any ad-hoc query can trivially crash your server. That's pretty serious. It's even worse if the ingest is using statsd-influxdb-backend with udp since during the crash you'll lose data.

Do you really think it's ok for a novice user to be able to crash the server from your web gui if they make a minor mistake (e.g something like

      SELECT value FROM /series.*/

not realizing how many series their regex actually applies to until it's too late and the has gui stopped responding)?

 _I think it 's common in databases that if you throw a massive query at it that the server doesn't have the resources to handle, things will go wrong._

This thinking is worse then wrong. Not only is it wrong - a simple unqualified select won't crash any "common" database - it makes anyone with any "common" database experience doubt your other claims.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> I have been forwarding Riemann metrics to it and visualizing with Grafana, pretty fun so far. Love that it's written in Go and fairly lightweight on resources compared to JVM-based technologies.

</comment><comment><USER8> I just love the ability to GROUP BY queries, they work so fast and saves me from writing a stupid map-reduce style job for that basic thing. I am building a time series nice analysis product with it and this change would work very fine.

</comment><comment> chaotic-good _It depends greatly on the shape of your data, but with regularly spaced timestamps at second level precision and float64s, we've seen compression reduce each point down to around 2.2 bytes per point._

AFAIK you're using float64 compression scheme from "Gorilla" paper. 2.2 bytes per data point is possible with it but only on data that doesn't utilize full double precision (example: small integers converted to float64). You should compare compression algorithm used by TSM storage engine with zlib or any other general purpose compression algorithm, otherwise this number will be meaningless.

<comment><USER9> Honestly if you have a series of increasing unix timestamps you should be able to do much better than 2.2 bytes each.

<comment> chaotic-good Even simple delta+RLE can squeeze increasing unix timestamps up to few bytes. I'm talking about floating point data compression. Even the specialized state of the art algorithms can't guarantee you 2.2 bytes per float in general.

</comment></comment></comment><comment><USER10> I had another go at this new release after failing to get any of the 0.9.x releases to suit my use case of massively high volume writes. Now that I managed to get some of my data in it, aggregation queries seem to be lagging in speed, I would like to know more about what is actually being done currently to improve the query speed rather than waiting out for the v0.11 release as I am deciding whether InfluxDB is the way forward for my use case.

<comment><AUTHOR> The new query engine work should get merged into master early next week. You'll be able to test against a nightly build then to see what kind of performance improvement will be part of 0.11.

What kinds of queries are you seeing poor performance with? That would help us troubleshoot and improve, thanks.

<comment><USER10> Thanks for the update. Most of the queries I am testing now are 'mean' and 'group by' aggregation for downsampling and visualization in Grafana. Looks like the problem is CPU bound as I see all 8 cores got maxed out when processing the queries. Currently there are about 50 series and 5B data points in total.

</comment></comment></comment><comment><USER11> Been using 0.9 for a few months now and been pretty happy with it. Though I'm not really looking forward to figuring out how to move from telegraf 0.2 to 0.10, since that had some breaking changes.

I do love the simplicity of influx+telegraf, kapacitor also looks cool. Chronograf seems like a bad version of grafana still... maybe in the future if its FOSS and somehow manages to be better than grafana I'd use it.

</comment><comment><USER12> I don't get all the hate. Reading the comments here yesterday and then today again, made me really question what's going on in this community.

They (InfluxDB) made huge progress, they work hard on an open source DB (and ecosystem) and people insult them and question Paul Dix's >30 minute response time to your support questions in a HN thread.

</comment><comment><USER13> I would love to see some good (and up to date) documentation on replacing graphite with influxdb (retention, rollups, etc).

Last time I tried it out I vaguely recall that configuring rollups was kind of painful -- lots of nearly duplicate CS queries, even for a relatively small number of series.

</comment><comment><USER14> Is the text ingest engine usable with TCP yet? UDP is a bit weird for this when I may care greatly about whether my server is successfully feeding a stream of lines too influxdb.

<comment><AUTHOR> We don't have the TCP listener wired up yet, but it's on the roadmap. Thanks for bringing it up, I'll try to get it prioritized for sometime in the next few months.

</comment></comment><comment><USER15> They did the mongodb approach. Build a shitty db and then build a normal one and claim 100X faster.

And then add synchronized disk commits (like postgresql has always done) and performance goes puff (meaning lower than pg) (example: elasticsearch, mongodb).

<comment><USER16> This is not the first time I have seen an anti-InfluxDB post that also subtly pushes PostgreSQL in its place. Now I totally agree and I am not happy with InfluxDB's performance, and maybe I'm just being paranoid, but it's interesting I've seen this a handful of times.

<comment><AUTHOR> Have a look at this release. The write performance is massively better than it was. We did tests writing up to 100B points against a single server.

The next release will focus on improving query performance, but this one still works for many cases.

<comment><USER17> "Massively better" doesn't mean deployable to production either. Especially not if you've only tested with a single point of failure with a single sized payload. But I'm willing to give it a third (and final) trial:

1) Does "DELETE * FROM foo" still cause the system to lockup, freeze, and require a restart to free memory? Or are there other conditions/queries that cause the system to become unstable?

2) There's no README/CHANGELOG/dependencies info on the download page. Which is the preferred version of Go to install on my servers for Influxdb - 1.4 or 1.5?

<comment><USER18> I'd guess go version doesn't matter, you should install the binary which is statically linked, no?

</comment></comment></comment></comment></comment><comment><USER19> can you efficiently store logs in InfluxDB? would love to read if somebody have tried that before... (just wondering)

<comment><USER20> We store some logs at a small scale (5k/s) and queries are fine 0.9.5. You just have to be smart how you do tags still. Hopefully 0.10 is the same.

</comment></comment><comment><USER21> A lot of comments have mentioned Time Series use. How does it compare to Cassandra?

</comment><comment><USER22> But does it work? It wins the most buggy database servers I've ever used in 20 years award.

Is writing less than 20MB/sec of data something to brag about?

<comment><USER23> We're using it with GitLab.com and we like it. 0.9 didn't work at all due to the volume but with 0.10 everything is functioning OK

<comment><USER24> A full cluster or a single node?

<comment><USER25> We currently run InfluxDB 0.10.0-nightly-614a37c (I have yet to upgrade it to the stable release) on a single DigitalOcean instance with 8GB of RAM with 30-something GB of storage. The previous stable release (0.9 something) didn't fare very well, even after we significantly reduced the amount of data we were sending (we were sending a lot of data we didn't really need).

Switching to 0.10.0-nightly-614a37c in combination with switching to the TSM engine resulted in a very stable InfluxDB instance. So far my only gripe has been that some queries can get pretty slow (e.g. counting a value in a large measurement can take ages) but work is being done on improving the query engine ([https://github.com/influxdb/influxdb/pull/5196](https://github.com/influxdb/influxdb/pull/5196)).

To give you an idea of the data:

* Our default retention policy is currently 30 days

* 24 measurements, 11975 series. Our largest measurement (which tracks the number of Rails/Rack requests) has a total of 28 539 279 points

* Roughly 2.3 out of the 8 GB of RAM is being used

* Roughly 4 GB of data is stored on disk

This whole setup is used to monitor GitLab.com as well as aid in making things faster (see [https://gitlab.com/gitlab- com/operations/issues/42](https://gitlab.com/gitlab-com/operations/issues/42) for more info on the ongoing work).

<comment><USER24> Thanks for the information. :)

Unfortunately, I need 2+ instances with Active/Active or failover to seriously consider anything for production which is why I've not touched InfluxDB beyond some light testing.

</comment></comment></comment></comment></comment><comment><USER26> I don't like the fact that old data will continue to use the old engine after the upgrade.

<comment><USER27> There's a migration script that can be used for moving old data to the tsm engine.

</comment></comment><comment><USER28> I have nothing to do with influx and I probably won't in the future.

There are way too many haters in HN. You venomous minority who shit on every bit of good news that isn't yours -- keep your negativity to yourself.

You fucking monkeys infected with rage.

<comment><USER29> On the one hand I agree with your sentiments, on the other hand I disagree with your approach.

<comment><USER26> Looks like all the hordes of oldies from slashdot is now coming to HN</comment></comment></comment>


Levandowski started an AI religion non-profit in 2015 - <AUTHOR> https://www.theguardian.com/technology/2017/sep/28/artificial-intelligence-god-anthony-levandowski <comment><USER2> Alternate headline: Levandowski started a tax shelter non-profit in 2015.

</comment><comment><USER3> Hmm a god created by liar and a thief. Not a god I want.

</comment><comment><USER4> oh man. i want in.

</comment><comment> throw-away-8 Sounds like a page from L. Ron Hubbard's playbook. The guy doesn't miss a step.</comment>


F-35 fighter jet fails to impress pilots who prefer old F-22 model - <AUTHOR> https://www.express.co.uk/news/science/1121988/US-Air-Force-F-35-fighter-jets-USA-f-22-lockheed-martin <comment><USER2> The F-22 is a world-beating air superiority fighter. It's designed to be an utter terror in the air, to the extent that it isn't even allowed to be sold to other nations. I imagine it's quite the trip to fly.

The F-35 is both less expensive per unit (though not by much) and far more flexible in terms of mission profile. It's designed to kill targets far, far beyond line of sight. However, if you're pulling dogfighting maneuvers in the F-35, something has probably gone quite wrong.

</comment><comment><USER3> F35 has been an incredibly expensive platform to develop and deploy.

Possibly the most expensive military system program in history.

However, it is beginning to reach its extraordinary potential of sensor fusion, electronic attack, force multiplication, and hybrid teaming.

We heard very similar things about F15 and F16 40-50 years ago, the same period of time they have completely dominated the air domain.

Odds are, F35 will be an extremely capable aviation platform.

Having said that, and from the perspective of an aviation geek, I would rather see F35 funds NOT be spent to deter adversaries, but to be invested in other areas such as infrastructure, education, health insurance experimentation, and affordable housing solutions.</comment>


6 Damn Good Reasons to Startup in Buenos Aires - <AUTHOR> http://tech.co/reasons-startup-buenos-aires-2014-08

<comment><USER2> there may be six good ones, but i can think of one bad one: the argentine peso.</comment>


Teenager's memory resets every two hours after traumatic kick to the head - <AUTHOR> https://www.news.com.au/lifestyle/real-life/teenagers-memory-resets-every-two-hours-after-traumatic-kick-to-the-head/news-story/71f93a4a2d8a1c5def2ba48089f44130 <comment><USER2> This reminds me of a book by Yoko Ogawa called "The Housekeeper and the Professor." It's a really beautiful story of the awkward friendship between a former math professor, whose memory "resets" every six hours or so, and his caretaker.

<comment><USER3> I believe it's been adapted to a movie too, if you'd fancy that.</comment></comment>


Is the lack of computer security due to programmer laziness and/or incompetence? - <AUTHOR>

The exploits of lulzsec in the past several weeks, and the relative ease with which they infiltrated so many websites reminded me of a question I have had for a while.
Basically, is the current state of affairs, where so many websites are vulnerable, where your computer can get a virus just by reading a PDF or playing an MP3, etc, due to programmers' laziness and/or incompetence, or is this a fundamental problem than can never be solved no matter what languages are used and no matter what computing paradigm is implemented?
Things like SQL injections and buffer overflow attacks seem eminently solvable. For example, to address buffer overflow attacks, simply do a bounds check before writing to any array.
I'm not a CS major or security expert, so I may be mistaken here, but in theory at least, it seems reasonable to have an OS where you can say something like "this action is a read-only action, and therefore no part of the hard disk of the computer will be modified upon executing this read-only action"
For example, playing an mp3 song, or viewing a jpg or pdf file, are all read-only actions, and so any program which attempts to change anything on the computer while performing any of these read-only actions will be intercepted by the OS and stopped. To me, it has always seemed ridiculous that the way computers and languages are designed today allow a computer to be infected by simply listening to a song or viewing an image.
Is the above feasible in theory, if not with current OS's and programming languages?
If it is not feasible, can you explain why not? If it is feasible, can you explain why there are so many compromised websites and computers that have been taken over by hackers? <comment><USER2> The short answer: Security is HARD.

The long answer:

When even OpenSSH, a product dedicated to security, with highly trained, security focused minds, has new vulnerabilities coming to light on a regular basis, chances are this isn't an easily solved problem.

There can be any number of reasons why a security hole exists, covering laziness, incompetence, ignorance, or simple mistakes. Unless you can mathematically prove the correct functioning of your program (no small feat), there are bugs in it, and chances are some of those bugs are security vulnerabilities.

There have been numerous attempts at protecting so-called "data-only" memory, but thus far they have not stopped buffer overrun exploits completely.

The only way to be more sure that your program is not vulnerable to memory pointer exploits is to use a language that doesn't expose pointers to the programmer. But then you sacrifice speed and executable code size. And even then you can't be 100% sure because the compiler and runtime have bugs in them.

The more safety you bake into a language, the slower programs written in it will run. And even then, you're only safe until the next generation of exploits come out that take advantage of something nobody thought would be a security hole. And round and round we go.

</comment><comment><USER3> Ok, boss, we've got the prototype going, it should only take about two more weeks to test it and get it secure enough to... What, now? Seriously? It's not... Ok, you're the boss.

<comment><AUTHOR> Can there be an OS and/or language where, making something secure comes by default when you write anything, thereby rendering obsolete the need to make something secure after you make a prototype?

e.g. a language that automatically does array bounds checks, whether the programmer asks for them or not.

Such a system would protect against managers and bosses' demands as well as against lazy and security-incompetent programmers.

<comment><USER4> The problem is that "secure" isn't a well-founded concept. For instance, Java does the array bounds checks you're talking about (and throws an ArrayIndexOutOfBoundsException if they fail). It checks memory accesses and won't let you access memory that's been freed or is otherwise invalid. It does

However, I don't think anybody could seriously say that Java is totally "secure." There are plenty of security-related things that could go wrong (e.g. a man-in-the-middle attack) that Java can't handle unless the developer takes steps to prevent them. By definition, that isn't "by default."

If you want to prevent security problems "by default", you're going to need to specify each problem you want handled, and you're going to need to recognize that there are a lot of security problems that a language or OS simply can't stop by themselves.

</comment></comment></comment><comment><USER5> Getting companies to develop software for your OS would be very hard. Your OS would have to have hooks into the outside software to know what the program wanted to do with a file handle. Create a new mp3? Change the info of the mp3? Add album art? Play the Mp3? Delete it? What about future features that haven't been thought of yet?

The outside company just wants the ability to access a file. What it does with it is not up to the OS.

</comment><comment><USER6> Implementing security measures usually provides little short-term return on time invested. Feasibility is not the issue.

</comment><comment><USER7> Managers.</comment>


YA sci-fi novel with tons of curious comp-sci references - <AUTHOR> http://0x23.xyz/ <comment><AUTHOR> Just downloaded this for my kid. Full of "in joke" comp-sci references. Even uses Random Forest to crack a problem - LOL! And has self-driving car chase in hacked Tesla ʘ‿ʘ

Implies Cicada3301 references, though probably fake.

Blurb from website:

This lengthy book was written for young adults, especially those of a more "gifted" and scientifically curious persuasion. It is a sci-fi adventure set against a backdrop of exotic physics and internet hacker mythologies. It also explores a number of contemporary sub-themes, for those who care to think. These include: scientific ethics, philosophy of mind, game theory, probability, privacy and the nature of morality.</comment>