Documents shed light on U.S. surveillance programs - Suraj-Sun http://www.cnn.com/2013/08/09/politics/nsa-documents-scope/index.html <comment><USER61> Ah, yes. CNN. I noticed that when embassies in Yemen and elsewhere were recently shut down, CNN was careful to _repeatedly_ note in its on-air and online coverage that it was thanks to the "intercepted" messages that we were aware of the threat. Ex: [1]

In fact, in at least one story, they went so far as to even specifically mention that the "controversial NSA program" was the source--apparently for those too slow to get the hint.

Such ridiculously transparent PR. And those brilliant and determined terrorists seem so easily thwarted, don't they? "Aww, rats. Foiled again! I knew I shouldn't have sent that email from my Hotmail account, with the subject 'Evil Terrorist Master Plan'. They closed the embassy. Now what do we do with these suicide belts?"

Really, who believes this stuff?

[1] [http://www.cnn.com/2013/08/04/politics/us-embassies- close](http://www.cnn.com/2013/08/04/politics/us-embassies-close)

<comment><USER62> I believe it actually. Evil and effective are not mutually exclusive.

The real question though is if prevented attacks would have been prevented anyway without this evidence, or in general if the price is worth paying.

<comment><USER61> > _Evil and effective are not mutually exclusive._

So, we have instituted this massive security apparatus/surveillance state for groups of people who are largely incompetent, ineffective boobs?

But, we don't know that they are incompetent boobs?

Any way you slice it, it smells and begs the question: why are we doing this? And, in this particular case, it's a little too convenient that they are taking such public and hyper-publicized action on "threats" that do appear to be easily foiled. Because a.) the plotters allowed themselves to be intercepted even with so much recent publicity around our programs (btw, so much for the number one argument that whistleblowers like Snowden help the enemy) and b.) simply closing a few embassies nullified the threat. And, oh, BTW, let's make sure that our PR machine (read CNN and others) repeatedly mention that we all have the good ol' NSA to thank.

Sorry, I just don't see how this goes down easily for anyone.

> _The real question though is if prevented attacks would have been prevented > anyway without this evidence, or in general if the price is worth paying._

In this case, I don't believe there was an imminent threat. I think it was propaganda. But, on a broader level, I think your questions are valid for those inclined to believe we should trade freedom/privacy/etc for security (i.e. most Americans). But, surely the answer is in large part a function of whether and how many "effective" threats are really out there vs. just evil people.

So, your "real question" is unanswerable at this point on a general level. We don't know who the threats are and, in fact, they are intent on keeping this secret for "security reasons". It is the quintessential bogeyman. We must simply trust that these threats exist and are of a scale that warrant the massive security apparatus/surveillance state that we have erected.

</comment></comment></comment><comment><USER63> "monitor" ... "internet traffic" ...

Great, you don't comb through every bit sent via YouTube, Netflix, etc. The smoke and mirrors are strong with this one.

It was YouTube's Robert Kyncl that was quoted in 2012 that "90% of all web traffic will be video." Add in your image traffic, and there's a lot of room to still "monitor" EVERYTHING of real value.

<comment><USER64> So true. I don't have the numbers but I guess 1.6% could be enough to monitor all email/messaging traffic.

<comment><USER65> Did some quick back of the napkin math.

Daily internet traffic (from the NSA white paper) is 1,826 Petabytes. 1.6% (* .016) is 29 Petabytes that the NSA "touches."

If the average email or chat is 75kb, then the NSA just admitted it touches 389 BILLION messages DAILY.

Also, the NSA says it only "reviews" .025% (* .00025) of the 1.6% it "touches." Small right? Well that means that the NSA "reviews...."

97 MILLION messages DAILY.

Assuming they probably don't check solicitations, I'm assuming they "touch" all emails, chats, etc. Right? And they "review" a non-trivial amount of them by quantity.

Math check? Please someone tell me I'm wrong.

Edited — I was off a decimal point, still a lot of messages daily.

<comment><USER66> 96 Million. Not that that changes your point at all. You're not wrong.

<comment><USER67> 97M [https://www.wolframalpha.com/input/?i=.025%25+of+1.6%25+of+1...](https://www.wolframalpha.com/input/?i=.025%25+of+1.6%25+of+1826+PB+%2F+75+kB)

Of course 1.6% could be about everything in terms of information that is out there, excluding raw non-meta data like bit torrent, mpeg, etc. One way of 'hiding' data for the NSA would be to embed it in these raw data streams with incorrect metadata.. would definitely harder to find that.

<comment><USER65> Edited.

Nm, I missed a demical point in a different spot. #fml, you are right. 97M.

</comment></comment></comment></comment></comment></comment><comment><USER65> This is a game of semantics. They only "monitor" communications that they are "lawfully authorized" to monitor. Every government official has been playing this game for 2 months now.

The real questions are:

1\. HOW MUCH DOMESTIC INTERNET TRAFFIC IS COLLECTED?

2\. HOW IS THAT INFORMATION ORGANIZED?

3\. HOW LONG DOES THE GOVERNMENT KEEP THE COLLECTED INFORMATION?

<comment><USER68> The NSA are semantic sociopaths. You can never expect them to answer a question honestly because they'll redefine each term to some unexpected yet valid meaning.

<comment><USER69> ... or invalid. They're not that picky.

</comment></comment></comment><comment><USER70> Can't wait to learn how, like "collect" now means inspect, "monitor" now means permanently store.

Assuming storage, it's simple: metadata, deduplication, compression, selective monitoring, keeping only the logs but not the content (e.g. for YouTube videos), exclusion of certain bandwidth-intensive protocols, as 'thezilch mentioned...

<comment><USER71> Aaaah, the confusing and deceitful language that is doublespeak.

</comment></comment><comment><USER72> Where is the white paper?

edit: [http://www.scribd.com/doc/159222203/NSA-White-Paper-on- PRISM](http://www.scribd.com/doc/159222203/NSA-White-Paper-on-PRISM)

edit_2: [http://www.scribd.com/doc/159218355/Obama-Administration- Whi...](http://www.scribd.com/doc/159218355/Obama-Administration-White-Paper- on-NSA-Phone-Metadata-Surveillance-Program)

<comment><USER73> [http://i2.cdn.turner.com/cnn/2013/images/08/09/2013_08_09_th...](http://i2.cdn.turner.com/cnn/2013/images/08/09/2013_08_09_the_nsa_story1.pdf)

CNN's link for those who dislike Scribd because it hates us all. :)

</comment></comment><comment><USER74> > "In any case, the Justice memo says "any arguable privacy intrusion arising > from the collection of telephony metadata would be outweighed by the public > interest in identifying suspected terrorist operatives and thwarting > terrorist plots."

A little footnote at the bottom is all this is worth? How do they know this? How do they know that "any arguable privacy intrusion" is outweighed by the public interest in identifying "terrorist" plots, when no one was given the choice and no debate has occurred? That was (allegedly) Snowdens' entire point of releasing the information that he did!

Not to mention that the premise isn't as simple as "privacy intrusion vs thousands of people getting blown up".

<comment><USER75> For the record, if I ever get blown up, I don't want to be used as an excuse for stripping citizens of their liberties. My death is inevitable; freedom isn't. We thought it worth paying in blood to get that freedom, and I think it even more valuable today.

And honestly, as a citizen, I see myself as partly responsible for a lot of the things that drive terrorist actions against the US. I know that if somebody blew up my family's wedding, I'd want revenge. That's not to say that I'd be happy to be killed in retaliation for some US government idiocy. But if the cycle of revenge violence has to stop somewhere, let it be with me.

</comment></comment><comment><USER76> So, what's the interesting traffic left which can be passively monitored?

We absolutely need OTR and then something like OTR for multiparty chat for chat. And, ideally, traffic analysis resistant. It needs to be the default, and painless.

Email is mostly a lost cause for TA, but TLS covers a lot of transport. Sometime in 2014, TLS should be the default, and "required TLS" should be considered an acceptable way to configure your MTA.

Standard web traffic is increasingly moving to HTTPS for a variety of reasons unrelated to NSA.

The really hard part is protection from metadata collection, and making big targets resistant to demands for 1) disclosure and 2) modification to accommodate monitoring.

</comment><comment><USER77> Considering how much of the internet traffic is youtube and netflix (almost 50% according to [http://www.reelseo.com/netflix-youtube-us-internet- traffic/](http://www.reelseo.com/netflix-youtube-us-internet-traffic/)) and how much is just duplicated (specially in the big things like very shared p2p files) and things they already had, that number don't take out the possibility that they are taking all the interesting/private stuff that goes around.

Also, that don't includes what is not internet traffic, like intrusion into private networks and servers.

</comment><comment><USER78> So that place in Utah is gathering only 1.6% of ... whatever it is gathering.

CNN and a government is a kool-aid for the masses.

</comment><comment><USER79> What percentage of the internet isn't passive content, like Netflix, other streaming media? Then subtract out garden variety Web content. Then subtract spam.

I suspect 1.6% is pretty close to the amount of interpersonal e-mail and real- time voice and text that traverses the Internet.

</comment><comment><USER80> Reddit x-post from /u/Jou_ma_se_Poes in this thread:

[https://pay.reddit.com/r/worldnews/comments/1k0qwn/two_encry...](https://pay.reddit.com/r/worldnews/comments/1k0qwn/two_encrypted_email_services_have_closed_down_for/)

Everyone should read this, if you're interested in this story at all. Great succint capture of the currently known info:

\---

Knowledge is power. The NSA haven't been to careful with their information. With 15000 documents classified documents out there I am sure that eventually my "unknowns" for the programs below will be known. I hope someone at the NSA chokes on their coffee when they read what we know and what we will still know. I am sorry the list is so long, the NSA have been busy. If you know what any of the unknowns are do let us all know.

Dial Number Recognition (DNR) Digital Network Intelligence (DNI) Geospatial Metadata Analysis (GMA) SIGINT Geospatial Analysis (SGA) SIGINT Terminal Guidance (STG) Each SIGAD is basically a collection site, physical or virtual; the SIGAD alphanumerics are used to indicate the source of intelligence FOR a particular report.

The NSA listening post at Osan in Korea has the SIGAD USA-31. Clark Air Force Base is USA-57. US-987LA is with Bad Aibling Germany US-987LB are telecommunications surveillance in Afghanistan

XKEYSCORE processes most of the SIGINT traffic that comes into the NSA by way of various SIGADs, and compartmentalizes it by selector. A selector might be "RUSFOR," which would stand for Russian foreign ministry intercepts.

PRISM is US-984XN analyze information collected and deposited in a database - 117,675 active surveillance targets in PRISM's counterterrorism database

Analysts working on a problem can request that a particular collection site be tasked, or used. The form they fill out is known as an SP0200. Additionally, when they wish to discontinue using a SIGAD for a mission, they send in another SP0200.

TURBULANCE - Consists of nine sub programs including Traffic Thief, Turmoil and Tutelage. Other subprograms unknown.

TRAFFIC THIEF - a database of "Meta-data from a subset of tasked strong- selectors" An example of a strong selector is an email address. In other words, it would be a database of the metadata associated with names, phone numbers, email addresses, that the intelligence services are specifically targeting

AGILEVIEW: DNI tool

AGILITY: DNI tool

SKYWRITER: DNI reporting tool

TREASUREMAP: DNI visualization tool

DISHFIRE: DNI

ETHEREAL: DNI

TUNINGFORK/SEEKER: DNI

OCTAVE: DNI/DNR tool for tasking

UTT - DNR tool for tasking

MAUI: a database for finished NSA intelligence products.

PROTON: SIGINT database

CULTWEAVE: SIGINT database

HERCULES: CIA terrorism database

MAIN CORE: database which contains personal and financial data of millions of U.S. citizens believed to be threats to national security

PINWALE/UIS: archived foreign and domestic e-mails database (FBI access)

MARINA - Internet/Telephone metadata database Unclear

MAINWAY: Call records Database - contains metadata for hundreds of billions of telephone calls made through the four largest telephone carriers in the United States

FALLOUT: Filter for MARINA and MAINWAY

NUCLEON: Voice data Database (FBI access)

CONVEYANCE: Filter for NUCLEON

ARTEMIS: Geospatial analysis

AIRHANDLER Geolocation analysis

DRAGONFLY: Geolocation analysis

GJALLER: Geospatial analysis

GOSSAMER: Geospatial analysis

GROWLER: Geospatial analysis

KINGFISH: Geospatial analysis

STINGRAY: Geospatial analysis

TEMPTRESS: Geolocation analysis

WITCHHUNT: Geolocation analysis

CADENCE/GAMUT: Collection mission system for tasking

UTT (Unified Targetting Tool?)

COURIERSKILL: Collection mission system

TURMOIL: Collection mission system

METTLESOME: Collection mission system

WEALTHYCLUSTER: Collection mission system

Enhanced WEALTHYCLUSTER (EWC)

MASTERLINK: Tasking source

HIGHTIDE/SKYWRITER: Desktop dashboard

RENOIR: Visualization tool

SKOPE: SIGINT analytical toolkit

DROPMIRE - NSA surveillance programme aimed at surveillance of foreign embassies and diplomatic staff, including those of NATO allies.

BLARNEY - program to access internet data at key junctions and is facilitated by arrangements with commercial cable companies and internet servce providers. (part of Upstream)

FAIRVIEW - NSA mass surveillance programme aimed at collecting phone, internet and e-mail data in bulk from the computers and mobile telephones of foreign countries' citizens. (part of Upstream)

BOUNDLESS INFORMANT - is a big data analysis and data visualization system to give NSA managers summaries of NSA's world wide data collection activities - summarizes data records from 504 separate DNR and DNI collection sources (SIGADs).

CPE (Content Preparation Environment) is a tool for writing report based on the analysis.

ANCHORY: an intelligence community-wide database for intelligence reports.

Mira4 and VERAS: German programs obtained by NSA (purpose unknown)

Surveillance Detection Units (SDU) are organizations belonging to the US Government that have conducted secret surveillance that potentially broke national laws in various European countries.

SIMAS - SDU's data is collected in a system called Security Incident Management Analysis System

NSANET - network carrying information rated Top Secret or Sensitive Compartmented Information. (Snowden access)

TERRORIST SURVEILLANCE PROGRAM - forerunner of PRISM

STELLARWIND is the code name of a Sensitive Compartmented Information security compartment for information collected under the President's Surveillance Program

TRAILBLAZER - earlier NSA effort.

PIONEER GROUNDBREAKER - IBM, ATT and NSA...

TEMPORA - British program including recordings of telephone calls, the content of email messages, Facebook entries and the personal internet history of users. Shares this with the NSA.

CCDP - The Communications Capabilities Development Programme is a UK government initiative to create a ubiquitous mass surveillance scheme for the United Kingdom. It would involve the logging of every telephone call, email and text message between every inhabitant of the UK

DHS

FAST (Hostile Intent) - Future Attribute Screening Technology - <AUTHOR> screening

MATRIX - Multistate Anti-Terrorism Information Exchange Program closed down 2005

Trusted Traveller Program [https://www.dhs.gov/trusted-traveler- programs](https://www.dhs.gov/trusted-traveler-programs)

No Fly List - <USER2>

Terror Watch List - over a million

Disposition Matrix - number unknown

 __Programs unknown in the NSA: __

AIRGAP /COZEN - <USER60>

 __AMHS __\- unknown

ARCANAPUP - <USER60>

ARTEMIS - <USER60>

 __ASSOCIATION __\- unknown

AUTOSOURCE - <USER60>

 __BANYAN __\- unknown

BEAMER - <USER60>

BELLVIEW - <USER60>

BLACKPEARL - <USER60>

CADENCE - <USER60>

CHALKFUN - <USER60>

CINEPLEX - <USER60>

CLOUD - <USER60>

 __COASTLINE __\- unknown

COMMONVIEW - <USER60>

 __CONTRAOCTAVE __\- unknown

CONVERGENCE - <USER60>

CREEK - <USER60>

CREST - <USER60>

CROSSBONES - <USER60>

CYBERTRANS - <USER60>

DOUBLEARROW - <USER60>

 __FASCIA __\- unknown

 __FASTSCOPE __\- unknown

FOREMAN - <USER60>

GISTQUEUE - <USER60>

GLAVE - <USER60>

GLOBALREACH - <USER60>

GOLDMINER - <USER60>

GOLDPOINT - <USER60>

GOSSAMER - <USER60>

HIGHTIDE - <USER60>

HOMEBASE - <USER60>

INFOSHARE - <USER60>

 __INTELINK __\- unknown

JOLLYROGER - <USER60>

LIQUIDFIRE - <USER60>

MASTERSHAKE - <USER60>

MESSIAH - <USER60>

 __METRICS __\- unknown

 __MICHIGAN __\- unknown

NEWHORIZONS - <USER60>

NIGHTSURF - <USER60>

NORMALRUN/CHEWSTICK/FALLENORACLE - <USER60>

OAKSTAR - unknown (part of Upstream)

OCELOT - <USER60>

 __OCTSKYWARD __\- unknown

PATHMASTER/MAILORDER - <USER60>

PANOPTICON - <USER60>

 __PLUS __\- unknown

PRESENTER - <USER60>

PRINTAURA - <USER60>

RAVENWING - <USER60>

ROADBED - <USER60>

SCORPIOFORE/CPE - <USER60>

 __SIGNAV __\- unknown

SHARKFINN - <USER60>

SNAPE - <USER60>

SPOTBEAM - <USER60>

STORMBREW - unknown (part of Upstream)

STINGRAY - <USER60>

SURREY - <USER60>

TAPERLAY - <USER60>

TAROTCARD - <USER60>

TRACFIN - <USER60>

TRAILMAPPER - <USER60>

TREASUREMAP - <USER60>

TRICKLER - <USER60>

TUNINGFORK - <USER60>

TUSKATTIRE - <USER60>

TWISTEDPATH - <USER60>

 __WEBCANDID __\- unknown

YELLOWSTONE/SPLITGLASS - <USER60>

<comment><USER80> Here is the second part of the post as it was too long for HN:

\-----

Edit. WIRESHARK and ARCMAP removed. See Entropius' post. Both commercial/harmless programs. Don't want to create a paranoia sandwich. It is bad enough as it is.

Second edit ETHEREAL removed - thanks to PointyOintment and azaydius

Third edit: Someone wrote to me asking what my problem is:

Thank you for contacting me with your concerns. I will answer as honestly and faithfully as I can. I am not an American. I am an attorney in a far away land. Look up my user name if you really must.

ALL the information I have posted is publically available. ALL of it. __The NSA should be more careful. __

What has the information been used for? The one XKeyscore slide stated over 300 terrorist captured. The NSA claim that their internal spying has disrupted 58 plots in the USA. Where are all these supposed terrorists and when are their trials starting? We 're not talking about the Boston Bombing, because they weren't there for that. We're talking about 58 seperate terrorist attacks which Americans have never heard about? Oh wait I hear you say "they were kept secret for a reason..." No... How about this is all fiction. Where is the evidence? General Alexander should be able to sit back, smile and start listing the targets, preparations and everything regarding each of these "58 supposed plots", yet he can only point to a single guy sending $8500 to Al- Shabaab. WOW! $8500!!! Stop the frikken presses! $8500!! /s

The other uses of internationally intercepted communications is to mark people out for death based on who they have been talking to. You are aware that the US uses telephone metadata from Afghanistan to launch strikes. Meaning that the content of the phonecalls matters not, what matters is how many times you speak to someone considered to be few enough hops away from someone considered a terrorist? I think the continual drone strikes are a crime against humanity and you only have to consider what it is like [to live with the constant threat of death hanging over them.]([http://www.livingunderdrones.org/living- under-drones/](http://www.livingunderdrones.org/living-under-drones/))

I'm not really enjoying my schadenfreude and I told you so when it comes to US citizens losing their rights. FFS! I TOLD YOU SO! You know how many thousands of downvotes I have had to endure trying to warn of the dangers? I eventually just gave up. Americans just knew better. It fucking pains me that I got nowhere fast.

I believe that privacy is a basic human right. It is not permissable to trample it underfoot because "terrorists" and "terrorism". I have lived through a real armed campaign with bombings and landmines and I didn't buy it then and still not buying it now.

Which USG clandestine operations do you believe are of value? It all seems to have backfired rather spectacularly.

What about the abuses of these very same progams that we're hearing about? Do you think the internal surveillance should be automated and watch for people spending money they shouldn't have, can't explain and/or haven't declared to the IRS? So you think the internal surveillance should be used for petty drug possession busts?

How many people are there in Al-Qaeda? A 100 as the ISI reckon? A 1000? How many of those are in the USA? OK. Now WHY are there 117,675 active surveillance targets in PRISM's counterterrorism database? It's out by a factor of 117:1. There are over a million Americans on the Terror Watch List. What is THIS ALL ABOUT? Do you have any idea?

What is the FISA court record for and against the government? All those warrants to do all that spying.... how many cases have eventuated because of this.

So yes, I am against the NSA and everything they do and enable. I don't think the NSA represents the USA in any meaningful way. From the outside it looks like a rogue operation with no meaningful oversight.

</comment></comment><comment><USER81> In other news, 98.4% of the world's Internet traffic is Nigerian spam.

</comment><comment><USER82> If they had any clue they'd just watch traffic to Google Analytics and instantly have something like 75% coverage. Or maybe they already do that.

</comment><comment><USER83> And what percent of those 1.6% may involve an American citizen? And of that percent, how many are being collected under a lawfully issued warrant?

<comment><USER84> Yeah, it matters what they do to Americans, but what matters more is what is done to the rest of the world's stuff and people. We don't get to vote in the thing which gets a president in (and what an interesting process that is). We get no say, but its apparently ok to bomb us, drone us, spy on us, grope at us, render us, irradiate us to see us naked etc. I supposed it helps keep me from terrorising Americans though.

</comment></comment><comment><USER73> Also of note: NSA claims that analysts review 0.00004% of global Internet traffic (0.0025% of the data they monitor, by my math).

</comment><comment><USER85> so the NSA analysts actually review "only" 7.304 TB (1826 PB * 0.016 * 0.00025) of data daily...

<comment><USER85> ... and to "monitor" the 1.6% traffic, they must filter the data at a (combined) rate of 2.705 Tb/s to keep up.

</comment></comment><comment><USER86> I would never trust NSA again EVER.

</comment><comment><USER87> Wow, a 98.4% failure rate.

/sarcasm

</comment><comment><USER88> That's still huge amount, no matter what it actually means.

If they ignore most of bittorrent and streaming services that dominate Internet traffic, the percentage of mail, skype and http traffic they monitor is incredibly high.</comment>


An in-memory distributed Posix-compatible file system - <AUTHOR> http://bshafiee.github.io/BFS/

<comment><USER2> The standalone mode is boring -- it's a ramdisk, about the same as you get from tmpfs.

The standalone_swift mode is interesting -- it's a ramdisk with eventual commit to disk -- but I don't see performance figures or essential config details, like how often it flushes.

The distributed mode is surely cool for some use cases, but it doesn't seem to commit to disk and it neither replicates nor works in parallel.

<comment><AUTHOR> 1-Well if you don't want to use a backend storage what else do you expect?

2-It will flush as soon as you call flush or close your file. It's not necessarily disk and can be any configured backend (right now only swift is supported; however, it's easy to develop other backends as well)

3-Replication is done in the backend storage. Once your data is flushed it's replicated to your desire in Swift, Amazon or whatever backend you are using. What do you mean 'nor works in aralllel'?

<comment><USER3> If the back end is eventually consistent, how can you claim POSIX compliance on the front end? User writes and fsyncs, BFS node goes down, somebody tries to read. What will they get? Who the heck knows? The only in-memory copy is gone, and the back end you've chosen doesn't guarantee that you'll get the most recently written data on the next read.

It's perfectly OK if you decide on a different consistency model, but then you can't say you're POSIX compliant (actually you can't anyway for legal reasons but that's another topic).

<comment><AUTHOR> If you flush a file it will be in the backend; Therefore, even if a node goes down, another node will be responsible for that. There will be obviously a delay for the recovery. In POSIX if your buffer is not flushed there is no guarantee that you will get the latest version neither.

Yeah, maybe I should totally get rid of that "POSIX" compatible thing. I had that in mind because I expose my fs interface through fuse library and supposedly they cover POSIX.

<comment><USER3> > In POSIX if your buffer is not flushed there is no guarantee that you will > get the latest version neither.

That's not true. If it's not flushed there's no guarantee that it will be _durable_ (i.e. on stable storage, will survive a crash of the entire system). However, there is a guarantee that it will be _consistent_.

[http://pubs.opengroup.org/onlinepubs/009695399/functions/wri...](http://pubs.opengroup.org/onlinepubs/009695399/functions/write.html)

> After a write() to a regular file has successfully returned:

> Any successful read() from each byte position in the file that was modified > by that write shall return the data specified by the write()

Even more specifically, and relevantly to the current point...

> If a read() of file data can be proven (by any means) to occur after a > write() of the data, it must reflect that write(), even if the calls are > made by different processes.

Many POSIX requirements related to what the database folks would call ACID are hard to meet in a distributed file system, but they are nonetheless requirements. This particular requirement implies that if a node fails but the system as a whole remains up, then the last write must be _immediately_ available. It could be met, for example, by doing your own synchronous in- memory replication. However, if you rely on pinging the data through a backing store that only provides eventual consistency, then the requirement is not met.

</comment></comment></comment></comment></comment><comment><USER3> The obvious, comparison for standalone mode would be to a regular file system built on a ramdisk. The obvious comparison for network mode would have been Plain Old NFS with a ramdisk on the server. Both comparisons are missing. Also, this statement is flat-out untrue.

> IOZONE performs all of its operations on one file

Getting iozone to use multiple files and threads is trivial (-t option), and the results for that would be much more interesting than those for a single file/thread. Unfortunately, such benchmark errors are _extremely_ common. I don't fault the author so much as his advisor, who seems to have a background in computer networking (hence the strong focus in this work on alternative network interfaces) but none in storage. My advice to the author would be to find a more appropriate advisor for this project.

<comment><AUTHOR> 1- This is just a preliminary evaluation. I just made the repository public 3 hours ago. I am planning so many experiments including the ones you mentioned.

2- As I said more benchmark including multiple files with iozone, postmark are coming. This is just to give viewer a sense. P.S I am happy with my supervisor :) thanks for your recommendations though ;)

<comment><USER3> As you continue your testing, I strongly recommend that you use different numbers of threads/files and queue depths, as well as different file and I/O sizes. If you want to make scalability claims, you'll also need to test with multiple clients and servers. It's also good practice to show the _exact_ iozone commands used, or (even better) fio command line and profiles.

More generally, I think it would be very helpful if you could be more explicit about how your system works. For example...

* What are your consistency/durability guarantees? I mean _really_ , since they're clearly not POSIX.

* How do you detect and respond to faults?

* How would you describe the system in CAP terms?

* How do you reconstruct file system state from the backing-store object state after a node failure or cold restart?

* How do you decide which node should hold a file? How do you re-make that decision when servers are added or removed? Can they be?

These are the decisions that every distributed file system must make (and an explicit comparison to others sure wouldn't hurt). While it might seem like a lot, you did say this is for a master's thesis and it's stuff your advisor should already have required. It will certainly come up when you go to defend that thesis. If this truly advances the state of the art in some way, you should already know exactly how and be able to explain it.

Disclaimer: I work on GlusterFS. On my own, I've worked on two toy projects (CassFS and VoldFS) that explored ideas in how to build a distributed file system on top of another kind of data store. I might be able to help you, if I knew more about which issues you'd already addressed and which remain.

<comment><AUTHOR> Yes, I plan to run experiments with different numbers of threads/files and many other factors. Right know I only have access to a cluster of 7 old machines. That's the best I can do and for the hadoop case I used all of the machines. Sure thing I will provide all of the commands that I used. I just made this project public to get this type of comments. I have not written or explained anything yet. So there are A LOT to explain like the questions you raised. I will try to answer them here as well: Here is a big sketch of the system: Nodes use fuse to provide a traditional file system view to applications. Zookeeper is used for consensus and it keeps track of what file (name) each node is holding. Swift (or any other persistent storage) is used as the backend. Nodes communicate through TCP or Zero_networking to each other for remote IO operations. * When a file is flushed my consistency/durability guarantees are whatever the backend storage is. If swift, then it's swift's consistency/durability guarantees. While in memory (not flushed or closed) it's consistent because there is only one copy of file at the host node but not durable because if that node dies it's gone. _if a node fails there is a master (elected with zookeeper) which will assign the files that node was holding to other nodes (assuming that nodes has flushed it 's files to the backend). Failure in zookeeper and swift are out of my project scope. _ As soon as the system starts, it goes through a zookeeper master election and then master distributes files in the backends to the live nodes. * right now it's just a silly algorithm (the most free node) however, it can/should be changed to a more advanced mechanism which probably considers load and many other factors as well.

I am very happy to answer your questions and I really enjoyed answering them. I am sure this is the type of question I will get in my defence session as well. And that's why I post this in the news. :)

Awesome, it will be great if other people can contribute in my project. There are many things left to do and a tons of space for improvement. Maybe we can discuss more through email.

</comment></comment></comment></comment><comment><USER4> Are you planning to benchmark this on 10G/40G networks? looks like on first glance it is limited by the available network bandwidth.

<comment><AUTHOR> I wish if I could :) but right now in our lab I don't have access to such equipments. Honestly, that was the plan because using 10/40G networks will significantly reduce the delay in remote operations as well as increasing the throughput in remote operations.

<comment><USER3> Might I suggest a round of testing on Amazon or Rackspace? They both have 10Gb-equipped instance types with SSDs and everything, for a few bucks per cluster-hour. That shouldn't break even an academic budget. You might not be able to do your special ZERO networking in that environment, but you'd be able to explore other interesting corners of the performance space and your experiments would be reproducible by others.

<comment><AUTHOR> Yes, I will talk about that with my supervisor.</comment></comment></comment></comment>


How to Land Your Kid in Therapy - <AUTHOR> http://www.theatlantic.com/magazine/archive/2011/07/how-to-land-your-kid-in-therapy/8555/4/

<comment><USER2> And alone with the counterpoint:

[http://thelastpsychiatrist.com/2011/06/is_the_cult_of_self-e...](http://thelastpsychiatrist.com/2011/06/is_the_cult_of_self- esteem_rui.html)</comment>


Classification of the principal programming paradigms - <AUTHOR> https://www.info.ucl.ac.be/~pvr/paradigms.html <comment><USER2> The idea of "programming paradigms" is not a useful way to talk about programming language design.

> The trouble with programming paradigms is that it is rather difficult to say > what one is, or how we know we have a new one. Without precise definitions, > nothing precise can be said, and no conclusions can be drawn, either > retrospectively or prospectively, about language design. > > _What, if > anything, is a programming paradigm?_ [0]

Luckily, we've studied programming languages and their design rigorously for decades, and we already have a rich, common language for discussion the design space of programming languages: type theory.

It sounds intimidating: "theory" is often general abstract nonsense that's only useful in academic research, etc., etc. But that's not the case at all. "Type theory" is really just a really useful way to understand individual features of a programming language, and how they compose with one another.

[0]: [http://www.cambridgeblog.org/2017/05/what-if-anything-is- a-p...](http://www.cambridgeblog.org/2017/05/what-if-anything-is-a- programming-paradigm/)

<comment><USER3> Type theory is only useful up to a certain point; there are plenty of languages that appear identical from a type-theoretic viewpoint but that have deeper richer interpretations in category- or set-theoretic settings. Check out the actual diagram in the article; it shows off some distinctions that would have been totally washed out if only types were considered interesting.

<comment><USER4> If a language has syntax specifically for some paradigm then the paradigm is reflected in the type system. A paradigm is invisible in the type system if it is implemented as library only.

<comment><USER5> Isn't this why sometimes entire paradigms can be implemented as libraries in Lisp? Isn't the whole idea of DSLs dependent on this?

</comment></comment></comment></comment><comment><USER6> Lets just put Oz, Alice, and Curry in every box.

How "state" is classified seems quite ad-hoc.

</comment><comment><USER7> There _many_ important dimensions of difference in programming languages that matter to the programmer. Which of those are "fundamental"? What does that question even mean??

One fork in the road between academic/theory answers to that question and developer answers concerns the role of the standard libraries and available IDEs. The out-of-the-box work/contribution of those tools is super important to developers in practice, but irrelevant to academic discussions. Some academics focus on real theory and declare that their statements about PL are basically a kind of applied mathematics - use it if it is somehow relevant to you, and ignore it, like other piles of math, if it isn't. Others make claims that the distinctions capture important properties like preventing careless errors. In that case, one can also talk about which tools are helpful/important to prevent careless errors. And similar remarks can be made for other properties such as speed/brevity of coding/expression, availability of pre-existing parts and ease of integrating them, etc. IMO, there is room for theory that tries harder to abstract the dimensions that drive software practice.

</comment><comment><USER8> I'm working on a new flavor of the logic paradigm [https://github.com/still- dreaming-1/PurposefulPhp](https://github.com/still-dreaming-1/PurposefulPhp)

</comment><comment><USER9> Could someone explain to me why everyone is crazy about type theory? ATM i learn category theory by watching and reading the awesome material from Bartosz Milewski. Should i stop and learn type theory instead, if i want to know more about programming languages in an abstract / general sense?

</comment><comment><USER10> I think there's a form of misplaced attention when it comes to programming. There should be a focus on the UX of the programmer. The programming paradigm is just one component of this. It's missing the forest for the trees.

<comment><USER11> There is a phrase "developer experience" ("DX") that I've seen used occasionally. It would be great if that concept could be developed further, similar to how "UX design" has been maturing as a field over the years.

<comment><USER12> For all the maturing "UX design" has done over the years, using a computer still feels mostly like swimming in garbage. Sure, now it's pretty brand-name garbage. But I'd hardly call that an improvement.

</comment></comment></comment><comment><USER13> I don't think it's possible to create a satisfying diagram like this. It's already been said, but compartmentalizing languages into "paradigms" is generally pointless.

</comment><comment><USER14> Kinda surprised to see Erlang and Smalltalk in two different columns.

</comment><comment><USER15> 2008 diagram, missing a decade

<comment><USER16> What new major paradigms do you think it is missing?

<comment><USER17> Type systems, which the chart doesn't address at all, are now usually considered at least as important an aspect to language's 'paradigm' as the (increasingly-blurry) functional vs imperative distinction. And a lot of the examples are pretty dated, coming as it did at the tail end of the object- oriented and XML fads. And concurrency and especially parallelism are given short shrift.

<comment><USER18> Paradigms are quite orthogonal to type systems. Mozart/Oz and AliceML implement more or less the same set of paradigms. One is dynamic the other is statically typed.

<comment><USER17> Then they don't implement the same set of paradigms. Type systems (can) fundamentally shape how computations may be expressed in a language. For example, C++'s type system was famously proved Turing-complete.

<comment><USER19> So is Java's type system [1], and Rust's and Haskell's with some modest extensions, and Scala's. Many type systems have turned out to be Turing complete.

[1] [https://arxiv.org/abs/1605.05274](https://arxiv.org/abs/1605.05274)

</comment></comment></comment></comment></comment></comment><comment><USER20> JavaScript: Mistake-oriented paradigm.

<comment><USER21> Could you please start commenting civilly and substantively?

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

<comment><USER20> I have no other way to describe weak typing and scoping rules of the said language. I stand by my words.</comment></comment></comment>


Precise Garbage Collection for C [pdf] - <AUTHOR> http://www.cs.utah.edu/~regehr/papers/ismm15-rafkind.pdf

<comment><USER2> Very cool stuff, and it sounds like they nailed their original goal, which was to provide a more accurate replacement for the Boehm collector for their PLT Scheme code. But in case anyone is excited about this being a general-use solution, the first sentence hints that it won't work with most C code:

 _Magpie is a source-to-source transformation for C programs that enables precise garbage collection, where precise means that integers are not confused with pointers_

Later, to crush your hopes completely:

 _Precise GC requires that a C program obeys many restrictions - the same ones that apply to using conservative GC, and more - or the program can crash. For example, to use precise GC, a C program must never extract a pointer from a variable typed as long, at least not if a collection may have occurred since the value was stored in the variable._

Yeah, definitely not going to work with any large program that has been touched by an average C programmer.

 _These extra constraints are, however, satisfied by normal C programming practices._

-snort-

<comment><USER3> If you are starting a project from scratch, and need GC, I would imagine it would be less of a pain (on a non-trivial bit of code) to abide by the restrictions than to have to find and plug memory leaks by hand all the time.

<comment><USER2> Sure, assuming there's no more suitable language than C, and assuming all the C libraries you use abide by the restrictions. The article says, "Library functions that are opaque to the GC remain safe as long as they do not invoke callbacks or retain copies of pointers," which helps, but isn't a complete pass. (Plus, even though C got void* before I entered elementary school, I still occasionally see library APIs that use int or long instead of void*.)

Also, the article says, "Custom allocators, in contrast, typically break these invariants permanently, because they allocate objects of statically unspecified shape in a block of primitively allocated memory." I'm not entirely sure, but it sounds like this means you can't use any library that uses a custom allocator. Certainly you'll have to manage resources from such libraries by hand. At that point you might as well use any garbage-collected language with a good C interface.

<comment><AUTHOR> Unless, of course, you're _implementing_ a garbage-collected language.

I agree that its application is limited (although I'm surprised they were able to convert even parts of the Linux kernel). But if I ever found myself implementing the runtime for a language with GC, I'd probably reach for this solution.</comment></comment></comment></comment>


Ask HN: customer appreciation in start-ups? - <AUTHOR>

Hello, my name is Trista Myers, I am looking for a community management/ customer service internship this summer.  Ideally, I would like to be a part of a small company where I can make a difference. I am looking to help a company develop or build-up their customer service strategies, customer relations, and community engagement. I am interested in helping companies stand out because of the way they treat their customers, by creating crafts or projects and by adding an element of cuteness and selfless-ness in the form of custom ideas and handmade items. Carsonified (http://carsonified.com/projects/), the company that I interned with last summer does a great job coming up with unique things that stand out and surprise their followers and it's something I think could be worked into most web start-ups.  But I am looking for some feedback and perhaps a little guidance to see if this is something start-ups think about, if it's something your start-up would like to do or is already doing?
Is "make-customers-happy-by-making-cute-things" a position that you would consider creating within your organization? If not, why don't you think this could apply?
I think it's also something that can be used to develop or maintain relationships with other start-up founders or people of interest-- sending them some swag or a card when something worthy happens in their life is a nice reminder of your connection.
So, let me know what you think:  is this a topic that has been tossed around by your team? Are you already doing some things like this? How do you see this fitting in? Do you have any examples of other companies who do nice things for their customers that go above and beyond what you'd expect?
You can contact me at trmyers[at]clarku[dot]edu or message me on twitter @tristamyers to be in touch. <comment><USER2> I speak only for my startup, but we are honestly so busy and broke that it would be difficult for us to allocate resources (even if it's just mind share) for something like this. We've chosen to focus on the product experience, as we solve a major pain in the industry, so customers thank us w/o little pins that cost us money and get thrown away.

Rereading that sounded a little harsh. It's definitely a good idea for some organizations, just not my startup.

OTOH, when I was at engineering school, Microsoft was very popular on campus because they gave out a super sized super ball with a light up inside. Gasp! That made them very popular, even amongst open sourcers.</comment>


Aficionado Culture Is Where Fun Goes to Die - <AUTHOR> https://www.bloomberg.com/view/articles/2017-08-05/aficionado-culture-is-where-fun-goes-to-die <comment><USER2> Can't this apply to coding frameworks? I feel the entry is too high using Stack overflow and when answers are wrong they are promoted anyways.</comment>


Who Needs the Tech IPO? Open source and Facebook have changed the economics of Startups - <AUTHOR> http://www.slate.com/id/2196405/

<comment><USER2> This article brought up an interesting point on the cost of hosting servers through cloud computing. I don't think this writer was especially tech savvy, but she/he has the right idea. For example, scaling one of the top Facebook applications can cost upwards of $30,000 a month if you use single dedicated servers, but that cost can jump all the way down to <$5,000 if you're using multiple EC2 servers of various sizes to maximize the throughput on each instance.</comment>


Stack Overflow Unveils the Next Steps in Computer Security - <AUTHOR> https://stackoverflow.blog/2017/03/30/stack-overflow-unveils-next-steps-computer-security/?cb=1 <comment><USER2> That puts a new twist on security.</comment>


Why an F1 car is more energy efficient than an electric car - <AUTHOR> http://www.espn.in/f1/story/_/id/15152695/why-f1-car-more-energy-efficient-electric-car <comment><USER2> The article's reasoning only works if you assume you're extracting _gasoline_ already refined from the earth. You're not.

I'd bet the energetic cost of refining gasoline from crude oil is much larger than that for the coal or "powerplant oil", thus seriously offsetting that 33% vs 50%?

This is ignoring the more obvious fact that a powerplant can operate at nearly constant efficiency, while a real car driving on a read road will have a wildly varying efficiency, even with a hybrid engine...)

And even if electric is "less green" now, we are kind of sure that we can drive down the price of electric power very low in the future (solar, renewable, classic nuclear, fusion etc. ...only one breakthrough in only one of these will have _huge_ price lowering consequences), while the price of oil can only go up...

<comment><USER3> Even extracting crude has ferocious energy costs for some sources. Alberta's tarsands are a good example, with the Syncrude and Suncor projects being some of the largest co2 emitters in Canada.

</comment></comment><comment><USER4> I'm an avid F1 follower and not a fan of electric cars but this article is so poor it's unbelievable.

Let's compare worst case scenario for what we don't like and ignore all inconvenient facts for what we like.

Lazy

</comment><comment><USER5> What a disingenuous title. It should say "Why an F1 car is more energy efficient than an electric car running on coal and oil grid power".

I get all of my electricity from a nuclear power plant, so if I had an electric car the title statement would at least be false for me. For people on solar, the title doesn't make any practical sense, as less efficient solar panels aren't causing any extra pollution (except during their initial construction).

<comment><USER6> > all my electricity from a nuclear power plant

Yeah, and there's no unaccounted for externalities with that strategy.

</comment></comment><comment><USER7> One of the biggest things I see with the article, outside of the poor comparison, is this: Sure, gaining electricity from coal or oil isn't clean. We know that. But we can make that better by switching the sources for electricity. Wind, water, solar, and nuclear power are all cleaner ways to produce electricity.

A hybrid car - or an outright gasoline car - always must rely on oil of some sort to be able to run. We do not have a clean, reasonable alternative for this as we do with electricity.

<comment><USER8> Biodiesel exists, we can probably figure out similar processes for other fuel types.

<comment><USER7> Some cars can run on a mix of ethanol and gasoline now, and I think more found their cars would run (though with possible long-term damage). If I remember correctly from when I lived in Indiana, they blamed rising food costs on farmers switching out their food and feed crops for bio-fuels. In addition, drought conditions or crop diseases would mean we are right back on oil-based fuels and/or there would be a crisis because of lack of oil. I like the idea that it is clean (at least cleaner than oil, anyway), but I think there are better options than vegetable fuels.

</comment></comment></comment><comment><USER9> I'm a huge fan of f1 but I think this article is a bit disingenuous. f1 engines are hundreds of times more expensive than normal engines, and they are only designed to run for a few hours before needing to be replaced completely.

<comment> k-mcgrady >> "they are only designed to run for a few hours before needing to be replaced completely."

That's wrong. An F1 engine has to do about 800 miles/1200km based on current rules. In testing over 4 days Mercedes completed about 2000 miles and I believe that was without an engine change. Obviously these numbers are much less than a person would drive in a year but significantly more than a couple of hours. I'm sure they could be designed to go longer and be continue to be efficient of necessary too as rules have forced this change over the years.

<comment><USER10> Ooh, over a thousand kilometers you say? Yes, that's definitely much closer to the 300,000km a regular car engine achieves than the GP was suggesting.

<comment><USER11> I really fail to see how the planned lifetime of F1 engines is related to its potential fuel efficiency. If anything, I would expect the engines could be made even more efficient if they were designed to last 300.000 miles, at the expense of power output. I find it hard to believe that tuning an engine so far it will wear down quickly will improve it's efficiency.

<comment><USER9> An F1 engine doesn't even have a starter. It's made out of exotic material. It runs at 18,000+ rpm, and idles at what most car engines would redline at! The entire car can weigh no less than ~1500lb, including the driver!

Making the engine last longer would likely require a huge increase in weight, which would reduce efficiency.

<comment><USER12> The new combustion engines are limited to 15,000 rpm. They usually stay well below the limit because of limitations on fuel amount (100 kg per race). [https://en.m.wikipedia.org/wiki/Formula_One_engines](https://en.m.wikipedia.org/wiki/Formula_One_engines)

</comment></comment></comment></comment></comment></comment><comment><USER13> How much does it cost (in terms of money and carbon footprint) to prepare the fuel and engine for a F1 car compared to that of a modern power plant?

<comment><USER14> How much (in terms of money and carbon footprint) is required to build a modern power plant?

Your comparison is disingenuous.

</comment></comment><comment><USER15> This article doesn't explain it well, but converting fuel to energy with this sort of thermal efficiency (40-50%) is indeed much better than both normal ICE and coal power plants at around 30%.

Obviously racing engines are highly tuned and not suitable for general use, but there's no reason I can see that TJI technology could not be adapted for normal engines.

<comment><USER16> Doesn't sound as if it's really very efficient, though, when you measure the entire chain (sunlight, plants, oil fields, crude, refined fuel, rolling wheels) against the currently achievable chains from sunlight via electricity to rolling wheels.

Which I realise isn't your point but IMO it's what an F1/FE comparison should count.

<comment><USER15> OK, but most electric cars don't get their electricity from sunlight. So that's not a fair comparison either :)

Of course F1 cars are not an efficient means of transport. The point is that the new rules introduced constraints that have produced real innovation in efficiency, giving significant gains that could be utilized in passenger vehicles. This is the real news, not silly headlines like "F1 cars are more efficient than electric cars".

<comment><USER16> Are you suggesting that most cars get their electricity from nuclear-powered plants? Because gas, coal, oil and even most hydroelectric energy all are in some form powered by sunlight.

</comment></comment></comment></comment><comment><USER10> Apples and oranges. Dear author, please redo your argument with Formula One versus these race cars: [http://www.worldsolarchallenge.org/](http://www.worldsolarchallenge.org/)

</comment><comment><USER17> Can someone explain how they are getting near 50% thermal efficiency? All the articles mention it, but I can't find a reasonable explanation.

<comment><USER18> It's a turbocharged hybrid engine with two electric motor-generator units. One on the shaft between the turbine and the compressor and the other is on the crankshaft.

Excess heat energy is captured from the turbo with the generator instead of blowing the excess out of the waste gate. The turbo is pre-spooled by turning it with the electric motor so the internal combustion engine runs at or near stoichiometric optimal fuel-air mix at all times.

The other motor generator does regenerative braking and can drive the engine.

The Mercedes engine is rumored to develop around 950 horsepower from the 1.6l V6 engine (of which about 160 are from the electric motors).

I'm hopeful that similar technology will find its way to road cars. City buses in particular would benefit a lot due to frequent stops.

</comment></comment><comment><USER19> The comparison in the article is pretty odd, but it's still amazing to see how much effort goes into engineering F1 cars. I wonder what they could produce if they were allowed to run free and didn't have to deal with F1 regulations.

<comment><USER9> I think the sport would be destroyed if they were allowed to run free.

By now they would have removed the driver, and that would take most of the $$ out of F1 ;)

The Williams 1993 car was in many ways the high point of tech for F1. Since then, I think FIA have been trying to balance the rules for (1) safety (2) balancing the role of the constructor and the driver, and ensuring that driver skill was a major part of the equation.

<comment><USER12> They would have to remove the driver. Check the famous Red Bull concept car from 2010, only simulated, not built. [https://en.m.wikipedia.org/wiki/Red_Bull_X2010](https://en.m.wikipedia.org/wiki/Red_Bull_X2010) 500 km/h and 8 g lateral acceleration with the final design. I don't think you can drive a car like that for long.

The Williams car with active suspensions was the last technical improvement that was not a rules loophole up to maybe the Mercedes f-duct and wing stalling. F1 has become a spec formula with the only goal to keep lap times more or less constant and some common ground with production cars (the current hybrids, Le Mans too.)

</comment></comment></comment><comment><USER20> They also need an engine rebuild every how many miles?

</comment><comment><USER21> What about an electric F1 car?

<comment><USER22> Formula E? - [http://fiaformulae.com/](http://fiaformulae.com/)

</comment></comment><comment><USER23> This is ridiculous. The energy consumed by an average road car to move 1 human 1 mile is far less than that of an F1 car, by a huge margin.

<comment><USER12> Formula 1 cars are designed to run fast, so they need a lot of energy, but efficiency is about how much energy you extract from fuel, not about how much energy you need.

<comment><USER23> Efficiency can be defined in many different ways. For road cars the amount of fuel used is far more important than the energy extraction efficiency, in practice. Which was my whole point.

</comment></comment></comment><comment><USER24> Dupe: [https://news.ycombinator.com/item?id=11449314](https://news.ycombinator.com/item?id=11449314)</comment>


Pepper Spray Sales Soar on Amazon - <AUTHOR> https://www.bloomberg.com/news/articles/2020-06-01/amazon-shoppers-snatch-up-pepper-spray-as-u-s-protests-continue <comment><USER3> Also guns. Big jump in their stocks today.

[https://www.wsj.com/articles/gun-stocks-surge-past- broader-m...](https://www.wsj.com/articles/gun-stocks-surge-past-broader- market-11591038711)

<comment><USER4> When gun stocks are the market's biggest risers, that really should be telling you something about the prospects for the rest of it. You'd have thought.

<comment><USER5> Gun stocks usually rise during periods of unrest/violence/mass shootings.

</comment></comment></comment><comment><USER6> is it really that surprised? After the riots/lootings in my metropolitan and has spread to my near by suburban areas, I am seriously thinking about buying a few hundred rounds of ammunition now.

</comment><comment><USER7> Can you imagine Pepper Spray is illegal in NYC? It's better than a gun because it's non-lethal in the vast majority of cases, so you won't need to hesitate, plus you don't have to aim so much, can hit multiple assailants.

I advocate pepper spray for all instead of guns.

<comment><USER8> While you're right on those accounts, it does have some disadvantages - <USER2> tends to fill a room, and has a much higher chance of catching bystanders, so you can't use without any forethought. That being said, you also don't kill them when that happens.

<comment><USER9> There's also pepper gel, which more accurately disables a target rather than spray: [https://www.amazon.com/SABRE-Red-Pepper-Gel- Strength/dp/B004...](https://www.amazon.com/SABRE-Red-Pepper-Gel- Strength/dp/B004NKSPR8)

<comment><USER10> Yes. Pepper gel is much nicer to use. At the other extreme, bear spray covers a huge area (and comes in a huge can).</comment></comment></comment></comment>


Why Our Genome and Technology Are Both Riddled with "Crawling Horrors" - <AUTHOR> http://nautil.us/blog/why-our-genome-and-technology-are-both-riddled-with-crawling-horrors

<comment><USER2> Nothing's really jumped out to me yet, but tools need to raise flags about the complexity of code. A pre-commit hook that recognizes long functions, classes with more than ten member variables, classes that program to an implementation rather than interface? Could it check to see that classes that can be easily unit-tested are?

The world expects programmers to make clean code, but we've not really advanced our tools much outside of Java and C#. I'm always envious of what the refactoring tools can do in those languages.

<comment><USER3> I never understood many people's fixation with avoiding long functions/methods. I would much rather have that hook check long call stacks. I have never went "wtf" as I dove into a new code base because of long functions, but I do it all the time when five levels of inheritance is used to provide all the magic necessary to "shield" me from whatever the code is actually doing.

<comment><USER4> I definitely _have_ gone "wtf" when dealing with some long functions. A 380 line switch/case comes to mind. I have also had to deal with the 5 level deep inheritance that was equally difficult in its own way. Figuring out where to balance between length and depth is one of the more difficult parts of programming in my opinion.

After many years maintaining my and other people's code, I do tend toward shorter methods. On the other hand, splitting up a method into several methods just to keep under a hard line limit causes more problems than it solves.

<comment><USER3> Sure, of course I prefer shorter code. But I'd much rather read a hundred straightforward lines than wrap my head around yet another abstraction (of which there are thousands in any non trivial codebase).

I must disagree on the switch statement. A main loop with an rather large switch statement is a perfectly normal way to write a trivial interpreter. C provides a nice way of doing it, and as long as you declare your constants properly it's all perfectly readable. A lot of excellent code bases are structured that way. (Awk comes to mind, IIRC, and it's very easy to read and extend.)

</comment></comment></comment></comment><comment><USER5> I still believe in a future where programming-by-specification and advanced theorem proving systems result in the ability to make systems that are provably without a certain class of bugs. Then it would be possible to upgrade systems effortlessly by simply proving that they satisfy the same primary specifications.

Then of course writing the spec is the hard part, but I'd at least be interested in seeing what those challenges turn out to be like.

<comment><USER6> is there any difference between a sufficiently rigorous spec and actual programming?

<comment><USER7> Depends on what you mean by "sufficiently rigorous". There's a sense in which an algorithm description is just a more detailed spec, but if a spec is just "specify all properties of output", that can be described in predicate logic. The spec language doesn't necessarily have to be turing complete (I think).

</comment></comment></comment><comment><USER8> At the end of the article, the author wants us to treat artificial complex systems as living things. I believe this is a mistake. This mind set leads you to believing that parts of a program or process are Just Too Hard To Understand. You'll end up supplicating the system, instead of working with it.

</comment><comment><USER9> "If our brains were simple enough for us to understand them, we'd be so simple that we couldn't."

I guess this can be applied to any system. An advanced system and one that can be understood easily don't seem to be compatible.

</comment><comment><USER10> The road to the stars passes through a refactoring of the human genome. That will be possible before a re-implementation of life.

<comment><USER11> I'm not sure that refactoring our genome is going to be enough. We need bodies that can survive cosmic radiation as well as maintain structural integrity for the thousands of years required for most interstellar trips. We probably also need a psychology that doesn't recoil in horror at the thought of 10,000 years in space. And frankly, we probably aren't smart enough to do space travel well, even if we did have radiation-hardened bodies with million-year lifespans.

Even saying "we need a rewrite" doesn't cover it, as that still implies the same design goals. We need to rethink ourselves from first principles to have any chance. To draw upon the IRS example from the article, this isn't a matter of slapping a web front-end on software from the 60s. This is more like trying to adapt 50 year-old tax return software commissioned by the US government, to run the server for your new MMO (and, I suppose, reworking that web front-end to act as the client, heh). There is not a single thing about the human body adapted to space travel including our brains - why would there be?</comment></comment>


ADC (Analog to Digital Converter) Module for FPGA Design - <AUTHOR> http://dev.emcelettronica.com/adc-analog-digital-converter-module-fpga-design <comment><AUTHOR> Howto interface an Analog to Digital Converter (ADC) to the FPGA board. In this post I will give some ideas about interfacing an ADC to the FPGA design. One of the Digilent Inc.'s product is the ADC module codenamed Pmod-AD1.</comment>


What Makes Hackers Tick? A.k.a. Linus's Law - <AUTHOR> http://books.google.com/books?id=4SeIQZjpzCwC&pg=PT14#v=onepage&q&f=false

<comment><USER2> This is the _other_ "Linus's Law", the one by him and not the one named by Raymond. See [http://en.wikipedia.org/wiki/Linus%27s_Law#By_Linus_Torvalds](http://en.wikipedia.org/wiki/Linus%27s_Law#By_Linus_Torvalds) . (Torvalds described it in the prologue to the book.)

To p7g5 - the book was published in 2001. Has it or the prologue proved influential, or worth of review and reevaluation after 14 years? I don't get the sense it has.</comment>


AppAnnie users are getting their Apple ID's locked - <AUTHOR> http://www.appannie.com/blog/current-issue-affecting-analytics-users/

<comment><AUTHOR> As a satisfied user of AppAnnie I got an email today regarding Apple locking Apple ID's of users of AppAnnie. It just requires an email reset which is not a big deal but AppAnnie seems shut down by Apple. A shame since itunes connect data sucks.</comment>


Visual Turing Test - <AUTHOR> http://turing.deepart.io/ <comment><USER3> I got 9/10\. Chose the first one incorrectly.

For me, the giveaway for computer-generated images was that they don't focus on the "subject" and have too much detail on the background.

For example, look at the computer-generated boats - they blend into the background. It's hard to tell where the boat ends and the rocks begin, or where the sails end and the clouds begin. Or look at the grey house in front of the dude's face or the pipe behind the glasses lady's head, which bring in too much attention but add nothing to the painting.

<comment><USER4> Yesterday I was reading this blogpost ([http://maryrosecook.com/blog/post/scarface-prince-of-the- cit...](http://maryrosecook.com/blog/post/scarface-prince-of-the-city-and- pieter-de-hooch)) so as soon as I clicked this link I thought no way can a computer learn to do what I read yesterday on its own!

I had read headlines about computer-generated images but had never seen them, yet I managed 9/10, anticipating that an artist's technique is not just relation between color etc. but a philosophy.

Edit: Also, it is not a turing test in true sense, since it's one sided. It would be turing test if I tell them what to draw & then it's tough to distinguish between what an artist & computer paints.

</comment></comment><comment><USER5> This is not a Turing test. Please correct me if I am wrong, but I bet that none of this was drawn by a computer. Most likely, those that were supposedly "drawn by computers" were pictures drawn by humans but applied various type of filtering (e.g. those by the deep learning algorithms).

<comment><USER6> What about a photograph with filters applied to it? Does that count?

<comment><USER7> Think what would really count as a "visual turing test" is if the computer- generated images were really "generated" by a computer, not rendered from a photo. Still, it's an interesting exercise in how much the rendering can make the photo feel "opinionated"—as if you can sense the painter behind it; that's the criteria I used for choosing, but I got a 7/10.

<comment><USER8> > what would really count as a "visual turing test"

I disagree - nobody would say drawings/paintings (by humans) based on photos or observations by eye are less legitimate than scenes completely imagined.

<comment><USER9> How about the following: The examiner can provide any picture of their choosing to two subjects A and B, where one is a human and the other a machine. At the end of a reasonable period, both subjects respond to the examiner simultaneously with one drawing each, derived from that same picture. In creating each drawing, A and B are both trying to convince the examiner that they are human. The examiner then should identify which of the two subjects is a computer and which is human. To better reflect the traditional Turing test, the examiner should be able to repeat this process with different pictures (keeping the identities of A and B unknown, but fixed) before making their determination.

Interestingly, this is a Turing test that, when applied purely to humans, doesn't require said humans to share a common language.

<comment><USER8> That's a nice protocol.

I have to quibble though that the Turing test is meant to be a test of intelligence, and this sort of task seems pretty different, although it may actually be visual-AI-hard (to invent a new term). So may deserve a different name.

<comment><USER9> You can call it a test of human-level aesthetic sensibility if you'd think a highly intelligent alien species would be unlikely to share our aesthetics. Although, arguably, a highly intelligent alien would be able to understand our aesthetics to the point where they pass the test regardless...

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> I got 6/10\. I guessed that most of the things that looked "too real" were based on photos with filters applied to make it look like a painting, and that the abstract stuff was generated by humans. E.g. I was very surprised a computer produced this: [http://turing.deepart.io/f/0.png](http://turing.deepart.io/f/0.png) or this [http://turing.deepart.io/f/4.png](http://turing.deepart.io/f/4.png)

But the opposite heuristic doesn't work either, because not all the computer paintings are abstract. It mimics a lot of different styles and I am extremely impressed.

<comment><USER11> >I guessed that most of the things that looked "too real" were based on photos with filters applied to make it look like a painting, and that the abstract stuff was generated by humans.

I too used the same method and 5/10\. Looking at the other comments, it seems people have better idea what art is than I do.

</comment></comment><comment><USER12> 10/10\. Most of these computer "generated" pieces are not good art. This is painted by a human who knows what it is to be human in the place that is painted: [http://turing.deepart.io/t/7.jpg](http://turing.deepart.io/t/7.jpg). This is just eh, shit? [http://turing.deepart.io/f/7.png](http://turing.deepart.io/f/7.png).

I am convinced intelligence programming will tackle these issues some day. But apparently not yet.

<comment><USER13> I was only 3/10 - so I think it takes an eye for art to beat it. In my mind this is like a chess program getting to 1200. Not enough to beat an expert, but it's on to something.

<comment><USER14> I'm not an expert by any means and I got 10/10

<comment><USER13> Perhaps it's what we were looking for?

I suspect that we can be trained to spot fakes too to make it harder. Perhaps there will be some co-evolution?

</comment></comment></comment></comment><comment><USER15> Note that all of the computer generated pieces were created by this impressive deep net algorithm (paper prepublished in 8/2015): [http://arxiv.org/abs/1508.06576](http://arxiv.org/abs/1508.06576)

Somehow I cannot believe how fast they got the idea to create profit from it: [http://deepart.io/](http://deepart.io/)

<comment><AUTHOR> Yeah, things move fast there: Paper (August) -> Open source code (September) -> Deepart (November).

As a side note, for me the pace was one of the reasons to move from science to data science (shameless plug: [http://p.migdal.pl/2015/12/14/sci-to-data- sci.html](http://p.migdal.pl/2015/12/14/sci-to-data-sci.html)).

<comment><USER16> Were you part of the DeepSense Right Whale Kaggle team? That was some good shit.

<comment><AUTHOR> I work at DeepSense, and with these guys, but I didn't participate in this team. I was shocked as well; and now I try to learn from them as much as possible. :)

</comment></comment></comment></comment><comment><USER17> 8/10, but I think this is extremely impressive.

If I wouldn't know that some of them were generated by ANN, I wouldn't question that like 90-95% of them were painted by a human. And I consider myself pretty alright at painting.

These 2 are especially awesome:

[http://turing.deepart.io/f/0.png](http://turing.deepart.io/f/0.png)

[http://turing.deepart.io/t/9.jpg](http://turing.deepart.io/t/9.jpg)

<comment><USER18> That second one would make for an amazing canvas print.

<comment><USER19> The second one is actually painted by Francoise Nielly [http://www.galeries- bartoux.com/fr/artistes/francoise-nielly...](http://www.galeries- bartoux.com/fr/artistes/francoise-nielly.html#.VrZsz17L-DA)

</comment></comment></comment><comment><USER19> Here are the results so far [http://turing.deepart.io:3838/](http://turing.deepart.io:3838/)

<comment><USER20> As a 10/10er, I think the fact that people tend to score less goes to show that:

1\. The photos are low-res enough to hide the most ridiculous artifacts produced by neural nets

2\. The machine-generated images are trying to emulate the likes of van Gogh, not the likes of Leonardo (again hiding the extent of NNs complete inability to understand what they're doing)

3\. Most people simply neither paint nor appreciate visual art, so this is not as powerful as the Turing test! (One point of the Turing test is that a human easily passes it, because all humans speak; they don't all paint.)

<comment><USER21> 9/10\. The low res made it harder to tell which one was computer generated, but it was still pretty clear. Now if they could do it full size, I'd start to be interested.

<comment><USER19> [https://deepart-io.s3.amazonaws.com/hi-res/leon3.jpg](https://deepart- io.s3.amazonaws.com/hi-res/leon3.jpg) \- hi-res version of the algorithm is ready. Soon we will make the second edition of the test, stay tuned.

<comment><USER22> this is great work kidzik. looking forward to see more about it. hi-res are quite good, that even if in the uncanny they can be quite a success for deepart lovers.

<comment><USER19> Thanks! We teamed up with the authors of the original algorithm and Leon figured out quickly how to improve resolution.

</comment></comment></comment></comment></comment></comment><comment><AUTHOR> BTW: "An artist or an ape?" test by Mikhail Simkin [http://reverent.org/an_artist_or_an_ape.html](http://reverent.org/an_artist_or_an_ape.html)

<comment><USER23> I got 100% on this one. I looked at all of them before making my selections and found what seemed to be a pattern amongst the ape ones.

<comment><USER24> Yeah, that seems like the trick. The ape ones are really very similar to each other.

</comment></comment></comment><comment><USER25> Knowing upfront that one is computer generated does not really work nor showing it to an audience that has seen deep neural net pics for months on the frontpage of HN. Showing them on the street in a non tech neighborhood and ask which is the better/nicer looking painting without saying that one is done by a computer should get some better results.

Prepare for some people actually getting angry about it once you tell that the paintings they liked most are computer generated (from pictures but still).

<comment><USER26> > Prepare for some people actually getting angry about it once you te that the > paintings they liked most are computer generated (from pictures but still).

My girlfriend is a fine arts major and isn't speaking to me right now.

</comment></comment><comment><USER27> Note that it is much easier for a computer to create a human-like painting than to behave like a human. The Turing test requires a two-way interaction medium.

<comment><USER28> Exactly right - the Turing test as it was originally conceived is very hard to implement (some argue impossible). To use the popular phrase, this is "not even wrong".

</comment></comment><comment><USER29> 9/10

The computer does not duplicate selective detail. Artists will put detail in areas of focus, and intentionally obscure or abstract other areas. Also, there is often an interplay between medium and message. To that last point, I'm guessing there is still a human "artist" who chose the reference scene and pose and artist style, even if they didn't paint it directly. Towards that, I see this as a creative tool to be included in some future Photoshop, or similar.

Also... Share on Facebook? "App Not Setup: This app is still in development mode, and you don't have access to it. Switch to a registered test user or ask an app admin for permissions."

<comment><USER30> 8/10, also got "App not Setup": [http://imgh.us/not-set- up.png](http://imgh.us/not-set-up.png)

</comment></comment><comment><USER31> Got 8/10\. Of the choices, I was instantly sure that this one wasn't computer- generated from a photo because of the unnatural dark area on the top: [http://turing.deepart.io/t/6.jpg](http://turing.deepart.io/t/6.jpg). I think (even if photos weren't involved) the fact that the dark area is out-of-place but still aesthetically congruent may suggest something about human creativity that would be hard to replicate using a computer.

<comment><USER32> Actually I thought that was human because the nipple stood out so clearly.

</comment></comment><comment><USER33> It looks more like these images are procedurally made rather than actual AI created paintings.

</comment><comment><USER34> Knowing that it was a turing test actually made it easier for me. (8/10) While I did take into account the painting style (the random rainbow look was suspect), I mostly ignored images where the subject looked like a photograph. This one threw me off on that basis though: [http://turing.deepart.io/t/4.jpg](http://turing.deepart.io/t/4.jpg).

<comment><USER3> For me the giveaway was the motion effect of the hair following the tilt of her head, followed by the spiralling background.

</comment></comment><comment><USER35> I recently played around in this space ( [https://medium.com/data- engineering/artificial-startup-style...](https://medium.com/data- engineering/artificial-startup-style-437f6090b1f7) ). I was merely thinking through what it would take to turn this into a Turing test and not actually executing one.

These guys seem to be trying harder to actually make this a Turing test, but as the commenters above have noted, the problem with using source photographs makes this a fairly imperfect test. I have a suggested fix for that.

In my last effort in that post, I actually used one source photograph, and then compared the output by a human painter and the AI painter. My original results were terrible, but then quickly vastly improved by a better implementation of the algorithm, the Deep Forger implementation.

I think that this variant in the procedure, having both the human and the AI start from the same source photograph, could make for some interesting variants on Turing testing.

</comment><comment><USER36> I wasn't very successful, but I tried to use "uniformity" criteria - if it looked like there are portions of the picture that were being omitted or abstracted away, I assumed human and vice versa.

I wonder if the algorithm could be improved if it had a sense of what is "important" in the picture, and then choose a different algorithm to process important and unimportant portions.

</comment><comment><USER20> I got 10/10 (no Facebook or Twitter account to share this, unfortunately.) _Update:_ my wife got 10/10, too.

Right now neural nets do pretty poorly with "loose/messy" styles (you get noise instead of brushstrokes; if they posted high-resolution images there, I cannot believe anyone would score less than 10/10.) Neural nets do _exceptionally_ poorly with "neat" styles (try copying the style of the Mona Lisa into your photo and it'll paint an eye into your mouth.)

However, I'm pretty sure that an algorithm hand-crafted to replicate an artist's style might fool me (just like a hand-crafted chess algorithm wipes the floor with a neural net plus some tree search; not so for Go, I know - <USER2> don't even play Go so I don't have a firm opinion on that one.)

I must say that this whole "photo + style" thing really gets my goat - not the research as much as the sort of press coverage it gets. I project that a stock market crash will improve such coverage significantly.

</comment><comment><USER37> I'd be very interested to see the source images before processing, to see how the resulting images differ from the originals. FWIW, [http://turing.deepart.io/t/8.jpg](http://turing.deepart.io/t/8.jpg) has a URL visible in the middle.

One thing I did find interesting is that I did the test initially and got 4/10 (most felt like guesses to be honest). I repeated the test before looking at the answers (though obviously knowing the score introduces some bias), but zoomed the page so the images were about 2-3x bigger than the default. My score increased to 9/10, and I was much more confident of my answers whilst selecting them.

</comment><comment><USER38> 9/10, but I was already familiar with the algorithm. Useful features to find the nature of the "artist" for me included the following:

\- Is the style of the painting applied uniformly over the whole image? Humans will, for effect, leave parts of a painting less emphasized and developed, whereas this algorithm will generally apply a style to the entirety of an image.

\- If a painting has a surreal style, the subject is generally also surreal. Human painters distort shapes and forms. This is not done by this algorithm.

\- Humans will add contrast to make objects stand out, even if colours are similar to their background. This is something the computers haven't yet completely figured out.

Still, this is a very impressive algorithm.

<comment><USER39> 8/10, on a mobile device, just going on gut instinct

</comment></comment><comment><USER40> I somehow doubt that this is just a photo stylized by a computer. Aren't the eyes way out of proportion? [http://turing.deepart.io/f/0.png](http://turing.deepart.io/f/0.png)

<comment><USER41> Yes, exactly. While the background in the picture of the baseball player doesn't look photographic, at least the proportions are realistic. But the proportions of the female face are completely unrealistic. How could it be based on a photograph of a real human?

Are these computer-generated images based on photographs of real subjects, or on photographs of stylized paintings?

I don't think this test is very interesting until we can see the images the computer-generated ones were based on.

</comment></comment><comment><USER42> I got the 8/10\. Style is extremely important in understanding an artist and his work. Without a grasp on the original style, it's hard to see if a painting is genuinely of that style. So a more substantial visual Turing test would be a series of original works by an artist v.s. a series of simulated works targeting that style by AI.

</comment><comment><USER43> 9/10\. I could tell from some of the pixels and from having seen quite a lot of computer generated imagery in my time.

</comment><comment><USER44> 10/10\. My partner got 5/10, which he claims is because he's "more like a computer" (he's autistic). There were features in all of them that struck me as expressive in a very human way.

It's amazing to live in a time when questions about human identity and expression are more than just hypothetical.

</comment><comment><USER45> 9/10 I think you can somewhat tell by the visual flourishes that a real artist will add to their painting that a computer will not. Also the computer ones while not completely obvious I could tell were generated from a real picture without exaggerations a real artist may add.

</comment><comment><USER46> Got 10/10\. I think though if I would have only be looking at one picture at time and were to say yes or no. My score would have been much lower. Quite often in the computer-generated ones it looks like someone have gone nuts with the smudge tool in photoshop.

</comment><comment><USER47> 9/10, missed the last one. I feel like the main difference between the generated one and the drawn one are:

\- obvious artifacts in the neural networks

\- humans paint light differently than how a camera captures them, evident even after heavy processing.

</comment><comment><USER48> Apparently I'm a computer since I only got 3/10\. Am I the only one that seems to have a lack of artistic aesthetic?

I do try to appreciate art, but I'll readily admit that I just don't get a lot of it.

</comment><comment><USER49> Your result is 10/10!

I guess as of today T-1000 would get his bitch ass kicked.

</comment><comment><USER50> I got 8/10 right. I didn't really think about how I might figure it out, but just went with my gut feeling. Apparently that worked pretty well. I wonder why!

</comment><comment><USER51> 3/10\. I wouldn't call this a 'turing test' though, in a turing test I can ask questions, here it's totally static.

</comment><comment><USER52> Well, is it really considered a Turing Test? I thought in Turing Test, the interrogator is the one to ask (choosing from questions at his will).

</comment><comment><USER53> 5/10 – I'm not human?

</comment><comment><USER54> does anybody know a way to run an algorithm with similar results on my own pictures? or how kids are calling it: "is there an open source version of this?"

<comment><AUTHOR> The easiest way is... [https://deepart.io/](https://deepart.io/). Open source: [https://github.com/jcjohnson/neural- style](https://github.com/jcjohnson/neural-style) (not sure if exactly this one, there are a few implementations).

</comment></comment><comment><USER55> 9/10 first one incorrect

</comment><comment> Ono-Sendai Your result is 10/10!

</comment><comment><USER56> Well, this is really nonsense, unfortunately.

First, all images could've been painted by a human.

Second, if I always click the left one, it says _Your result is 5 /10_. A turing test is to test "a machine's ability to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human". If I – presumably a human being – score 5 out of 10, the test is not a Turing test after all, because humans aren't tested in a Turing test.

Edit: My observation was obviously wrong. Thanks for pointing it out.

<comment><USER57> You are not being tested, you are the judge. now, if all the judges do no better than chance, the system has passed. I got 8/10, and knowing the deepart pictures from before, I was pretty confident in some of the judgments, so I don't think they pass.

<comment><USER19> Indeed. We haven't pass yet - that's the first attempt though. Still only 66% of human accuracy is quite remarkable.</comment></comment></comment>


WTF is a CTO? - <AUTHOR> https://medium.com/@mattetti/wtf-is-a-cto-24b9ad4d6e50#.51kwiognx <comment><USER2> Probably depends on the company.

In most companies I worked for the CTO was the vision person. Since they all were technology companies and the tech was the product. The CEO always was the sales guy who knew how to talk to business people.

I also worked in companies whithout a CTO. Where the CEO was secretly also CTO and the person I would dub CEO normally was the CFO. Sometimes because of ego reasons sometimes because the business guy was good in bookkeeping but not so good in business development.</comment>


The Audacity of Shamelessly Asking - <AUTHOR> http://beanoriginal.net/the-audacity-of-shamelessly-asking/

<comment><USER2> So the lesson is not to be audacious in private - do it in front of a large audience where rejection won't be tolerated the same way.

<comment><USER3> The same principle applies in romance (proposals, confessions, etc.), where, if you win over the crowd, she'd be vilified for rejecting.

<comment><USER4> Or he.</comment></comment></comment>


The Necessity of Self-Help Lit - <AUTHOR> http://www.iasc-culture.org/THR/THR_article_2016_Fall_DavisNecessity.php <comment><USER2> Many people do not realize that one can get professional help that does have academic rigor and will be somewhat paid for (depending on insurance coverage).

If you can get over the still present social taboo of going to a therapist I highly recommend it both for relationships and personal growth (there is not just CBT but some newer forms that I have forgotten the names of). If you don't like a therapist you can easily switch and most are happy to refer and connect you with others (if they don't they are not a professional).

If you are willing to be slightly experimental (and if you are buying a book I would imagine you are) there are also Universities and other Institutions that are always looking for participants in studies.

Another extremely pro to a professional therapist is that they keep longitudinal records. Many people have a hard time tracking their progress and if they do it still might not be accurate compared to a third party observer.

<comment><USER3> How might someone approach/shop therapists (CBT or otherwise) with the goal being personal growth? I've given thought to how useful a professional opinion may be (most of my barriers are self-inflicted), but I'm unsure about how much I'd get out of it and how to present my needs to the therapist at first. Any advice?

PS: If you have any reading/links on the subject handy I'd appreciate it. My searchings have found little.

<comment><USER4> I can't say that I have any links, just personal experience.

I've seen the same therapist for 5+ years. He helped me get through my divorce and grow into a much better person. Recently, I've started trying to use him as an accountability mechanism for improving the still remaining ~5% of my life that I wish to continue improving. I've found it to be less effective that I had hoped b/c he's not really serving a function he's trained at nor are our meetings frequent enough to be just-in-time useful (~once/mo.).

I suspect a mastermind group of like-minded strangers that meet on a weekly basis would be more effective (and cheaper!).

<comment><USER5> >I suspect a mastermind group of like-minded strangers that meet on a weekly basis would be more effective (and cheaper!).

Lean In Circles[1] are a good example of this mechanism. I haven't looked too deeply into them, but I briefly participated at my school (they're all- inclusive, but focused on women in tech) and it was a very positive experience. They may have some useful data for you on how to effectively practice this idea.

[1]: [https://leanincircles.org/](https://leanincircles.org/)

<comment><USER6> Yes I have been in 2 masterminds and both have been great for support and advice. As a result I build a SaaS to help more people connect this way because I believe that the more we help each other the better for everyone. www.eccountability.io

</comment></comment></comment></comment></comment><comment><USER7> An anecdote, that I at least find kind of funny in my own life:

I deal with diagnosed anxiety and OCD, and thus have read dozens of self-help books on working with these issues. A funny thing happened when I started reading the genre, in that I found myself somewhat addicted to it after the first 3-4 books.

I brought this up to my therapist after about the 10th book in two months, and he joked but also seriously said, ok, time to take a break, pick a few unrelated and fun fiction books and go to town on those instead.

Not offering medical advice here as everyone is different, but I've noticed a gradual decline in symptoms when I'm able to focus less on the issues themselves.

<comment><USER8> Thanks for sharing that. I am a moderator of an addiction-related subreddit, and recently a commenter made a short list of things they had struggled with which included (unsurprisingly) alcohol, tobacco and drugs. But they also included an unhealthy dependence on 'self-help' which was my first encounter with this idea. So you're not alone!

<comment><USER7> I'm pretty sure people with addictive personalities can also get extremely addicted to their own thinking (stealing this idea from DFW's Infinite Jest, but I agree with it)

<comment><USER9> Funny you should bring up DFW, who was apparently quite into self-help books himself:

[https://theawl.com/inside-david-foster-wallaces-private- self...](https://theawl.com/inside-david-foster-wallaces-private-self-help- library-f84d5f56fccd#.32kd1v8vm)

</comment></comment></comment></comment><comment><USER10> Browsing the "self-help" shelf at my local charity shop, I came across _Tarzan of the Apes_ and _Breakfast at Tiffany 's_. I like to think that this is a clever deconstruction of the genre.

<comment><USER11> I brought my copy of 1984 from the non-fiction section of the bookstore.

<comment><USER12> I sometimes leave copies of the Bible in the fiction section.

</comment></comment></comment><comment><USER13> > _On the other hand, the self-sufficiency and self-mastery promoted in much > self-help are, as critics suggest, illusions. We do not make ourselves, and > we cannot validate ourselves. Imagining that we can will bring not > independence or confidence or trust, but, ironically, a lot of anxious > searching for the approval of others—or a return to the self-help literature > for the key to that elusive autonomy._

What?! How then _can_ we validate and make ourselves? Is the author a proponent of submission to a god? I don't understand how there is any other way to be validated, because one can never truly be validated from the outside anyways - there will always be external critics.

Edit: formatting

<comment><USER14> There's a step short of submission to a deity: peer-review.

I suspect the author meant something more like 'We do not solely make ourselves, and we cannot solely validate ourselves.'

<comment><USER13> That's still validation from the outside, from which one will never receive the clearest, most honest review, though I admit it can be helpful on the journey. The highest clarity comes from within, which is what all paths of truth teach (and in fact, what self-help books are often based on, inspired by, or try -- and sometimes succeed at -- teaching)

<comment><USER14> I have found my highest clarity to occasionally be obscured by clouds, hence my tendency to look outward once in awhile.

<comment><USER13> Yes, me too. Though I don't think it will be the ultimate return. I do believe that someone else who has reached the unreachable can point the way, but that at that point it will still come from within.

</comment></comment></comment></comment></comment><comment><USER15> All I can think about is "Little Miss Sunshine" where one of the characters wants to be a self-help star on the self-help circuit, but really needs the most help of them all.

</comment><comment><USER16> The content was great but I wanted to mention the narrated recording at the top by curio.io was excellent. Really a pleasure to listen to while working and I'd never heard of the site before.

</comment><comment><USER17> My problem with self-help books is: The key ideas they contain can be expressed in a single page.

<comment><USER18> The key idea, if there is only one to be had, still has to be "owned" by the reader and that requires knowing the why.

I took more issue with the notion that having an MD behind your name makes what you say more valid (Deepak Chopra) versus just being some dude who teaches "neural linguistic programming" (Anthony Robbins). Having read (3 books each) both Dr Chopra and Mr Robbins I can tell you that the former writes books that are so unscientific as to make you wonder if he forgot everything he learned in med school or is just making so much money that he doesn't care how much he metaphysically babbles about nothingness [sic]. (Pretty sure I know the answer.) No matter what you think about Tony Robbins, you have to admit some of his ideas have traction.

If you read enough of these books you'll find yourself making some improvements to your life. Malcolm Gladwell's "Blink", etc, were hugely impressive to me if only for his ability to tell the story of achievement and to make you believe genius-level ability is attainable, notwithstanding the allegations of plagiarism and the recycling of ideas.

The real issue today is that there are so many motivational speakers/writers that it's become impossible not to waste a lot of time trying to discern the steak from the bullshit.

<comment><USER19> > The real issue today is that there are so many motivational speakers/writers > that it's become impossible not to waste a lot of time trying to discern the > steak from the bullshit.

This sounds "ripe for disruption." Any startups trying to solve this?

<comment><USER18> You could...</comment></comment></comment></comment>


Automatic Regression Testing in JavaScript - <AUTHOR> http://ethanfast.com/2012/02/automatic-testing-in-javascript/

<comment><USER2> Please post to HN when you open source it. Very interesting stuff.</comment>