Australian software engineer got asked algorithm question when entering US - github-cat http://www.news.com.au/travel/travel-advice/travellers-stories/aussies-weird-immigration-interview-in-the-us/news-story/8222c65d2f12e6691ef27c9b1753e821 <comment><AUTHOR> People are quick to blame this on Trump, but reading through the comments on a previous article for this incident ([http://www.news.com.au/travel/travel- advice/travellers-stori...](http://www.news.com.au/travel/travel- advice/travellers-stories/aussies-weird-immigration-interview-in-the-us/news- story/8222c65d2f12e6691ef27c9b1753e821)), it looks like it's par for the course to test someone's knowledge on their claimed profession. Here's a comment from one of the users:

> Had the same experience 12 years ago: admitted I am developing linux device > drivers, and had to explain differences between kernel 2.4 and 2.6 APIs. The > guy actually understood it.

Testing occupational expertise is not a bad way to find people who are lying. To be able to explain the difference between kernel 2.4 and 2.6 APIs or how to balance a binary search tree are questions one could not possibly answer without actually being in that profession.

Let's say you're a hitman traveling to Malaysia to take out the estranged brother of a dictator. Before you go, you're given a fake passport and character sheet telling you your profession, your family situation, etc. It's pretty easy to answer questions like "how long have you been working as a photographer?" or "how did you meet your wife?" but a lot more difficult to prepare for questions like "What is the best lense for close ups?"

<comment><USER2> I've been asked my profession a number of times on entering the US (as a visitor - not on a work permit). I'm an electrical engineer, but my response is usually just "engineer". Electrical engineering is such an incredibly broad profession, including everything from chip designers to radio engineers to power systems. Just how many quizzes is CBP going to have design to cover every branch of my profession and not deny an antenna designer entry because he doesn't know anything about power transformers?

<comment><USER3> What is ohm's law? What's the difference between AC and DC?

Not very hard. It won't deter someone willing to learn EE 101 but will deter the guy who plans to overstay his visa and work as a dishwasher in his cousin's restaurant.

<comment><USER4> That assumes the border agents are familiar with all professions to come up with smart questions, or to understand answers that say the same thing with different terminology.

At best they will google "<profession> interview questions" and ask the first one, and do word-by-word matching on the answer.

For Electrical Engineering, the first one that comes up for me is "Why star delta starter is preferred with induction motor?".

</comment></comment></comment></comment><comment><USER5> Hey y'all, I'm the person they mentioned in the article.

Because this is a fairly technical forum (and some people were asking about language specifics or runtime characteristics) I'll let you in on the question even though I left it out of the journalist interviews - hopefully this information will help you make sense of the article/interaction for yourself:

Write a Python program to take two numbers as input and if the sun is bigger than 100, output "this is a large number" but if it is less than 100 output the sum of the numbers.

Lol!

<comment><USER6> What were you denied entry for previously?

<comment><USER5> Sorry? I wasn't denied entry ever... Ahh the news.com.au article isn't worded very clearly: the person in front of me was denied entry (not me!)

</comment></comment></comment><comment><USER7> One cannot but wonder what happens if you answer the question in a different manner than that suggested by Wikipedia or whichever resource the CBP draws their questions from.

I think the basic idea - questioning people about things they ought to know, given who they claim to be - is a good one.

However, it only makes sense if the one doing the questioning is able to judge the quality of the answer - or, for that matter, determine whether your inability to answer satisfactorily is because you bluffed - or if you're just a bit outside your professional comfort zone.

(I would probably have to answer 'software engineer', as that is what my business card says - however, I am much more of a hardware/systems guy in practice, only our HR department is completely unable to change my title to something a bit more descriptive. If speaking to another engineer, I wouldn't have any problems convincing him of my bona fides - however, if a CBP officer just looks up question #13 in the 'SW engineer' quiz - I may be in trouble.

<comment><USER8> > I think the basic idea - questioning people about things they ought to know, > given who they claim to be - is a good one.

It's an awful idea. False negatives. Discriminates against people with harder professions. Wouldn't keep out determined people

<comment><USER7> As to the last part, if people are sufficiently determined, you're in for a hard time, anyway. A 30-second screening at the border is pretty good at sorting out the Epsilon minus semi-morons, though. (Whose numbers, one would suspect, vastly outdoes the really determined ones.)

As for false negatives, I wouldn't be too concerned as long as you got to talk to someone reasonably well versed in your field. Also, as suggested by golergka and rconti - they probably use the questions as a way to elicit a response from you and judge that, more than your actual answers. That makes a lot of sense to me, now that I pause to think of it for a second.

</comment></comment></comment><comment><USER9> I think lots of people on this thread are missing the point. This won't necessarily detect a determined spy who's done their homework. It's not meant to. It's a quick and effective sanity check to make sure that a person hasn't fudged their visa credentials and they're well in their rights to ask and exercise their discretion. They've been doing it for years.

I doubt a perfect answer to their trivia question is required. If it's in your field you'll be able to say something halfway intelligent in response, if you've lied on your visa it will become obvious pretty quickly.

Outrage about confiscating phones and demanding passwords is justified. This, on the other hand, seems pretty reasonable.

</comment><comment> d--b It's actually fairly typical. When you enter on a H1B visa they ask you questions to see if your occupation (the one you got the visa for) is not made up.

I work in Finance, and when I was on a H1B visa (back in 2010), the immigration officer asked me some bond-related questions, just to see if I had any clue.

They also asked my wife what I was working on.

It's pretty annoying to feel that you're not being trusted, but yes, they don't trust you.

Also the fact that he's Australian is a reassuring proof they don't base these question on color of skin...

<comment><USER10> Is this true ? I have entered the US at least 30 times on H1 and never been asked anything like this and have never heard anything like this from my friends either

<comment> d--b it's nothing like interview questions, just "hey so what did the 30y bond was headed last week?". Or "So should I buy stocks?".

Probably just to check whether I would start crying and confess I didn't know a thing about finance.

They also asked my wife if she knew all my first names (I have 4!).

But yeah they definitely asked some occupation-related questions.

</comment></comment></comment><comment><USER11> Original, with a little bit more detail: [http://www.news.com.au/travel/travel-advice/travellers- stori...](http://www.news.com.au/travel/travel-advice/travellers- stories/aussies-weird-immigration-interview-in-the-us/news- story/8222c65d2f12e6691ef27c9b1753e821)

> "The vibe I got was weird. He asked me a question, then asked me a follow-up > question to prove I wasn't lying.

> "Do they not allow bad software engineers into the United States?" Thornton > joked.

See also, more recently: [http://money.cnn.com/2017/03/02/technology/andela- engineer-c...](http://money.cnn.com/2017/03/02/technology/andela-engineer- customs/index.html)

</comment><comment><USER12> _"'I've got a problem, I'm trying to write a computer program, can you help me?'"_

"My software consulting rate is normally $400/hr, plus an initial startup fee, but I'm not authorized to work in your country, so, no."

</comment><comment><USER13> I always found the cold, hard manner of border agents in the US quite off putting in general.

Last year I was fortunate enough to attend AWS Re:Invent, standing in the queue at McCarran airport, I saw the guard laughing and joking with the people in front of me so I thought ah, maybe they're all not like this!

As soon as it came to me the guy completely changed, he went cold, he spent the most amount of time asking why I'd ticked "business" on my customs form, as I didn't know what to tick, then proceeded to ask me what conference I was going to and what it was about and why I was attending. I know that's his job and he's just making sure everything lines up, but it just felt unwelcoming.

<comment><USER14> I've read somewhere that, if border controls were abolished, about a billion people from all across the world would move to the US. I'm guessing that, since people already want to come to US pretty badly, being welcoming is not a high priority for border personnel.

<comment><USER15> What possible source do you have for 1/7th of the world wanting to abruptly uproot their lives and move to the U.S.?

</comment></comment></comment><comment><USER16> I am not surprised by this. When I was at Newark the conversation with immigration guy was something like this:

Him: What do you do? Me: I am an entepreneur.

Him: Haha, ... we are all entepreneurs, now really, what do you do? Me: Websites

Him: I've heard there is a Sillicone Valley in Russia and they want to take over that ours. Me: Um, okay. (I am from Czech Republic)

</comment><comment><USER17> A few days ago a Nigerian software engineer was asked to balance a BST at JFK airport: [https://twitter.com/cyberomin/status/835888786462625792](https://twitter.com/cyberomin/status/835888786462625792)

<comment><USER18> I think he was asked to write a function that will return true if the tree is balanced, not to actually balances it, which in my opinion is an exponentially easier question.

</comment></comment><comment><USER19> Don't blame this on Trump; it goes on all over the world.

If the border police officer for any reason decides that you are suspicious then they will toy with you as they please. That includes asking arbitrary questions under the threat they can send you back to home country for any or no reason. Plus they can fingerprint you and go thru all your travel goods including phone and laptop.

As foreigner you have no rights standing at the border of the country you are entering.

And the border police operates without the checks that a normal police force operates under.

If you are a frequent traveller you better get used to being interrogated; answer short and precise without using any trigger words.

</comment><comment><USER20> One of my math professor (Canadian) once told us a story (~10 years ago) that the US border control asked him to "state and prove Rolle's theorem" after he had told them he was traveling to a math conference. Apparently he answered "it's something to do with the mean value theorem, right?" and that was good enough.

</comment><comment><USER21> I don't like the nervous atmosphere they make at the US border. It's the only country apart from Cuba where you get the feeling they might deny me entry for no reason. And it's the only country that I personally know people have been denied entry for no reason.

It's the way the questions come out, like a movie. The words come out slowly while the guy looks for something in my eyes.

I haven't been grilled on professional questions, but I wonder what I should say I do. Programmer or Trader are both legit. If I say programmer, I might get a technical question. If I say trader, maybe I just have give him a random stock tip.

<comment><USER22> ever travelled to Israel? their immigration interviews are legendary.

<comment><USER21> I have, and it was not memorable?

</comment></comment></comment><comment><USER23> This reminds me of a George Gamow's story (which can be an urban legend attributed to him) in Soviet Russia, I think when he was trying to flee. Allegedly, he was detained by some local warrior group in Caucasus(?). The local chieftain didn't believe he was a theoretical physicist, so he asked him to derive the n-th error term of the Taylor expansion. Fortunately, he was able to do that and save his life!

I take it as one of those lessons of usefulness of mathematics in daily life. :-)

</comment><comment><USER24> This makes me wish some of my colleagues were issued an airplane ticket to the US. Possibly one-way.

</comment><comment><USER25> I've also been asked for my occupation, and then my business card, and then my website – which they actually started browsing as I was standing there in the passport control. This was back in 2009.

</comment><comment><USER26> I'm in the US right now. Upon arriving in SFO (from Sydney) on a holiday (ESTA), I was asked the following non-standard questions:

"So you're a software engineer? What is your most proficient language?"

Response: "Javascript"

"What does the atob function do?"

I was quite taken back and asked to repeat, and the CBP agent spelled out "a t o b". I answered something to do with base64 decoding and was let through

<comment><USER27> Haha! Shit! The only time I've used atob in 20 years of doing JS was to read the ProtectionHeader of a SmoothStreaming manifest file recently. I'll bet little to none JS programmers know specifically what it does or if it even exists.

</comment></comment><comment><USER28> It's a little bit strange that even though Python was mentioned there's not a hint about the actual problem in question.

<comment><USER5> I didn't tell the journalists the question because it wouldn't mean anything to their target audience.

Most of these reposts (including this one in hacker news) have not reached out to me for additional or audience-specific details :)

</comment></comment><comment><USER29> The fact that this has started happenning in parallel with H-1B crackdown is not a coincedence. Someone somewhere issued a note that too many fake programmers are slipping in, so that's the result.

</comment><comment><USER30> A investment banker I know ( private equity, with a well know firm that has a AUM of $340 billion ) was asked :

"Why do you you hate the USA ?"

when he last flew to the US.

</comment><comment><USER31> Why not just strap them on a lie detector and ask if they have any intention to do anything illegal?

</comment><comment><USER32> I just don't believe this. It's fake news. This line made it clear to me:

"He said the officer appeared to be mid to senior level, and there's no chance the conversation would have been overheard by anyone else in line."

So basically there is no one who can corroborate with his story. This trash should not be allowed into any publication which calls itself 'news'.

I'm surprised that people actually believe that stuff by default. In this new world, the default should be to distrust the media and that line I quoted above gives a very good reason to distrust it.

<comment><USER5> I was the person in the story and I'm pretty sure this person could! [https://twitter.com/cyberomin/status/835888786462625792](https://twitter.com/cyberomin/status/835888786462625792)

<comment><USER32> I don't believe that either. It's such a specific question - I don't know if even Facebook or Google would ask such a question in their interviews.

This guy you just linked to has posted 11K tweets in the past; obviously an attention/status seeker. Also his follow-up tweet gave it away:

"So @Twitter I can now get the blue tick, yes?"

He almost didn't get any retweets before his 5 minutes of fame.

Wow people on HN can be so gullible.

<comment><USER5> Out of curiosity, what threshold of evidence would you accept that I provide to verify that I was indeed asked to write code at the US border?

I'm almost certainly not going to provide it, but I'm still curious if you're willing to elaborate.</comment></comment></comment></comment>


The "Windmill" Problem on the 2011 International Mathematical Olympiad [video] - <AUTHOR> https://www.youtube.com/watch?v=M64HUIJFTZM <comment><USER2> The approach to solving this problem looks very elegant to viewers with/without a mathematical background and the author's use of visual explanations towards solving it step-by-step helps untangle the ambiguities in this puzzle.

Correctly proving this without assistance is one thing, but explaining it to non-mathematicians via a YouTube video sounds so difficult that some I.M.O candidates may struggle with this. Even so, I think the author is perhaps a professional/skilled mathematician or both which greatly helps explain this proof in a concise fashion.

On the other hand, I find that problems like this may be (ab)used in the future for technical interviews at financial/asset/investment management institutions for software engineering roles. Over the top indeed, but I think it would very difficult to justify using mathematical proof questions in interviews.

<comment><USER3> The author is Grant Sanderson ([https://en.wikipedia.org/wiki/3Blue1Brown](https://en.wikipedia.org/wiki/3Blue1Brown)) who has an undergrad degree in math from Stanford and worked at Khan Academy before starting his YouTube channel 3Blue1Brown. Also, the student who is mentioned in the video (Lisa Sauermann) as having solved this problem at the 2011 IMO (and attaining the only perfect score) just recently started as a Prof. at Stanford ([http://web.stanford.edu/~lsauerma/](http://web.stanford.edu/~lsauerma/)) as a 27 year old.

<comment><USER4> Wow. She went straight from getting her PhD at Stanford to teaching there. That's almost unheard of. She must not only be a brilliant mathematician but an amazing teacher too.

<comment><USER5> Not only do teaching skills play no role in getting tenure at a place like Stanford, teaching is actually a threat to research productivity and research universities will hire professors who do as little teaching as possible, leaving most of it to older who professors who are no longer productive researchers or to post docs, teaching assistants or other staff.

To understand how this works, the researcher will apply for some grant, say $300,000 to study some question in geometry. Now, why does a mathematician need grant money when their only tools are a paper and pencil (maybe a laptop with Tex installed)? First, the university gets 1/3 of that money as "overhead", so the researcher is left with $200,000. Then, the researcher will pay to "buy out" his teaching load which is more money paid to the university, say $150,000 to not teach 2 classes for a year. With the remaining $50,000, he may spend money to fund a post doc to come and assist him for a semester. Again, that money goes to the university. So the researcher may get $300,000 but it all ends up in the pocket of the University, which in turn pays him a good salary with the assumption that he keeps the grants coming. A place like Stanford gets about 1/3 of its funding from these research grants, 1/3 from its endowment, and 1/3 from tuition. It hires researches to get the grants, grad students and adjuncts to teach, and the sports teams and other events help with endowment.

Thus research professors are hired on the basis of their ability to avoid teaching loads, not on their teaching skills.

<comment><USER4> > say $150,000 to not teach 2 classes for a year

Is that why she is teaching two classes in her first semester, including one which is lower division? Usually you don't put the crappy teachers in the lower division classes, you give them graduate seminars.

I appreciate your cynicism, but based on her teaching load, I'm going to guess that she is _also_ a good teacher.

<comment><USER6> > Usually you don't put the crappy teachers in the lower division classes

Since when? The hard-and-fast rule is junior faculty are assigned intro classes. We often give youthful teachers higher marks than crusty, doddering emeriti, perhaps for good reason, perhaps not.

> I appreciate your cynicism, but based on her teaching load, I'm going to > > guess that she is also a good teacher.

GP did not question her teaching ability, but your inference of said ability from her impressive ascent at Stanford. It's a bit like inferring LeBron James must really be mature since he entered the NBA straight from high school.

</comment></comment></comment></comment></comment></comment><comment><USER7> This phenomenon, whereby a person who knows the "trick" to solve a puzzle cannot accurate gauge its difficulty, seems to extend beyond mathematics. Adventure games (including text-based, parser-driven, and point-and-click) suffer badly from this problem. They are chock-full of puzzles that only make sense in hindsight (if at all). They can be really fun though!

<comment><USER8> Tanya Khovanova has a paper [https://arxiv.org/abs/1110.1556](https://arxiv.org/abs/1110.1556) with a list of some more of these problems with easy solutions that are hard to find. She calls them "Jewish Problems" because they were used by USSR universities to discriminate against Jewish applicants. The applicants would fail to solve them but the university would justifiably be able to claim they were easy.

<comment><USER9> Hey, thank you so much for posting this! I picked a random problem ( the one on logs) in that paper and solved it under 1 minute. I feel like a million bucks now!

</comment></comment></comment><comment><USER10> This can be formally solved by constructing the arrangement of lines passing through each set of two points, then computing the dual of the arrangement.

The cell with the maximum depth on the arrangement contains the points in the "middle", meaning they have as many points on one side, as they have on the other.

Then you can prove that a line starting on any such point will visit every other point an infinite number of times.

<comment><USER11> Just pick a point and rotate a line around that point continuously. Keep track of the number of points on the left. Since this count is essentially continuous (the jumps are of size 1), at some point during this rotation you will have an almost-balanced configuration (same number of points on left and right, up to parity error).

<comment><USER10> Yes, but it must be a point with certain properties, such that the same number of points are on either side of the line initially, otherwise, it would not work.

<comment><USER11> I am giving an alternate proof of your implicit statement "There exists a line which has the same number of points on each side". You did this by computing duals of cell arrangements. I am arguing that you don't need to do that.

The proof I outlined will work for any point. Initially it might have the wrong number of points on both sides but for some rotation it will have the correct number of points on both sides.

</comment></comment></comment></comment><comment><USER12> This guy makes excellent math visualization videos -- some of the best I've ever seen.

<comment><USER13> Yes! I really enjoy them as well-- and he has his animation software on github: [https://github.com/3b1b/manim](https://github.com/3b1b/manim)

<comment><USER14> Would be interested in how this stacks against mathbox2*

* [https://acko.net/blog/mathbox2/](https://acko.net/blog/mathbox2/)

</comment></comment></comment><comment><USER15> This is hella cool.

"Knowing when the math is hard is way harder than the math itself"

But then maybe the math is hard only because it's not being explained well? (I hope it's uncontroversial to suggest that our current methods of teaching math are not the best of all possible worlds.)

I get that this problem came up in the context of of a math puzzle contest, and that some people enjoy solving puzzles. I am questioning their utility as an educational device.

I kinda think that we should teach math as fast as we can so that we can concentrate on the stuff that's really hard, not just _apparently_ hard because someone is being coy with the easy routes.

<comment><USER11> Are you trying to say that math is hard because the teachers are withholding all the tricks?

> I am questioning their utility as an educational device.

Puzzles like this aren't found in mainstream math education contexts. As you acknowledged in your post, they are only found in math competitions. What do you mean?

<comment><USER15> > Are you trying to say that math is hard because the teachers are withholding > all the tricks?

Kind of, although I don't think they do it deliberately.

Things like teaching logarithms without a slide rule.

> Puzzles like this aren't found in mainstream math education contexts. As you > acknowledged in your post, they are only found in math competitions. What do > you mean?

You're right. Let me try again.

Check out William Bricken's "Iconic Math" [http://iconicmath.com/](http://iconicmath.com/) or "Proofs without Words" [https://en.wikipedia.org/wiki/Proof_without_words](https://en.wikipedia.org/wiki/Proof_without_words) or the other 3Blue1Brown videos for that matter.

I think that _most_ math seems hard to _most_ people _only_ because we are not creative in the ways that it is presented. We should use science to figure out how to present math so that people get it as fast as they can, in part so that we can find and concentrate on the _actually hard_ math problems.

E.g. Alan Kay using Smalltalk to teach calculus to little kids in the context of modelling falling objects, to me kinda proves that it shouldn't take a whole semester to teach calculus to teenagers.

<comment><USER11> > Check out William Bricken's "Iconic Math" > [http://iconicmath.com/](http://iconicmath.com/) or "Proofs without Words" > [https://en.wikipedia.org/wiki/Proof_without_words](https://en.wikipedia.org/wiki/Proof_without_words) > or the other 3Blue1Brown videos for that matter.

Yes, those references demonstrate that _some_ mathematical facts can be demonstrated easily, given the right presentation.

I agree that there is large gap between current mathematical presentations and the optimal presentation.

I am curious how effective the optimal presentation is. How easy can we make math? We have to be careful of the trap that the 3b1b video warns us against - when we understand something it is very difficult to put yourself in the shoes of a beginner. Something that looks like an elegant and clear presentation may seem like gibberish to the beginner (look at the YouTube comments on the 3b1b video).

I tried to google "Alan kay teaching calculus smalltalk" and didn't find anything. I am curious to see how much he actually taught the kids. It's clear that the typical college student, after taking a typical calculus class, doesn't really get the point of calculus. They may be able to follow some algorithms for differentiating and integrating but I don't think they understand when they can apply calculus.

<comment><USER15> I went and looked the Alan Kay thing up. It's not calculus, just acceleration. Sorry! It's described in "The Real Computer Revolution Hasn't Happened Yet" [http://www.vpri.org/pdf/m2007007a_revolution.pdf](http://www.vpri.org/pdf/m2007007a_revolution.pdf) in the section titled "Children Discover, Measure, and Mathematically Model Galilean Gravity"

I think an important piece of the puzzle (no pun intended) is customized presentations with feedback for individually-tailored education. But then it occurs to me that group activity is also crucial for learning, eh? I'm not an expert.

</comment></comment></comment></comment></comment><comment><USER16> Since this is a puzzle that will certainly nerd-snipe a number of us, could someone who already watched the video tell us if there is a "spoiler" moment in it or if we could watch it bit by bit in case that we get stuck?

<comment><USER17> The problem:

Let S be a finite set of at least two points in the plane. Assume that no three points of S are collinear. By a windmill we mean a process as follows. Start with a line l going through a point P ∈ S. Rotate l clockwise around the pivot P until the line contains another point Q of S. The point Q now takes over as the new pivot. This process continues indefinitely, with the pivot always being a point from S.

Show that for a suitable P ∈ S and a suitable starting line l containing P, the resulting windmill will visit each point of S as a pivot infinitely often.

<comment><USER18> if not all points in P are coplanar then what does it mean to rotate clockwise? perpendicular to which axis?

<comment><USER19> They are all coplanar, but no three points are colinear. The first sentence just means that there are at least 2 points in the set.

</comment></comment></comment></comment><comment><USER20> This is great, the visualization is so helpful. Even better I think would be if the point field was counter rotating and scrolling such that the windmill was constantly falling forwards and backwards either side of being vertical, keeping the two sets of points bisected and on either side of the screen.

</comment><comment> d--b Mmh intuitively, I would have thought that the proof would involve the enveloppe of points. If the line starts with a section that crosses inside the enveloppe of the set of points then it remains so, and hits all points, while a line that starts outside remains outside and so can avoid some points.

Any formal proof along those lines?

<comment><USER21> By envelope, you likely mean the convex hull. By "crosses inside the envelope", you probably mean it intersects the hull. That does not work. If you have a triangle inside a triangle and a central point inside that, that gives you seven points. If you start on the inner triangle, intersecting the outer triangle (convex hull), you can repeatedly visit the points of the triangles without touching the central point.

</comment></comment><comment><USER22> I enjoyed watching this similarly insightful video [1] on "The hardest problem on the hardest test" of the Putnam Competition.

[1] [https://www.youtube.com/watch?v=OkmNXy7er84](https://www.youtube.com/watch?v=OkmNXy7er84)

</comment><comment><USER23> The video gives excellent intuitions. But do try writing down a rigorous argument after watching it!

</comment><comment><USER24> From the same channel : [https://www.youtube.com/watch?v=jsYwFizhncE](https://www.youtube.com/watch?v=jsYwFizhncE) overview of very elegant connection between blocks collision and PI.

</comment><comment><USER25> Great presentation. I wonder whether all correct solutions submitted on the contest day had the same solution.

</comment><comment><USER26> Now I really want to know what question 6 was and an equally informative explanation what made it so hard!

<comment><USER27> Numberphile has a couple videos about it

[https://www.youtube.com/watch?v=Y30VF3cSIYQ](https://www.youtube.com/watch?v=Y30VF3cSIYQ)

[https://www.youtube.com/watch?v=L0Vj_7Y2-xY](https://www.youtube.com/watch?v=L0Vj_7Y2-xY)

<comment><USER28> The actual question is at [https://www.youtube.com/watch?v=Y30VF3cSIYQ#t=5m31s](https://www.youtube.com/watch?v=Y30VF3cSIYQ#t=5m31s) , if you don't want to hear a 5 minute rant about how hard it is.

</comment></comment></comment><comment><USER28> I'm going to have to watch this later, because the cat I've got on my lap appears to be deeply alarmed by the blinking eyes of the "pi" avatar.</comment>


Reconstructing Twitter's Firehose - <AUTHOR> https://docs.google.com/document/d/1xVrPoNutyqTdQ04DXBEZW4ZW4A5RAQW2he7qIpTmG-M/edit <comment><USER2> This is a really neat approach! A friend of mine actually did his thesis working with Netflix to analyze Twitter and determine if Netflix was down[1]. He took a much more brute force approach, taking advantage of the fact that his search space was limited to re-assemble a "firehose" for a given search term leveraging hitting the search API from multiple machines and then reconciling the stream and removing duplicates. This approach definitely wouldn't scale to the entirety of the firehose, but I think that, in general, interest in Twitter data revolves around more specific queries.

[1] [https://users.soe.ucsc.edu/~eaugusti/pages/papers/docs/SPOON...](https://users.soe.ucsc.edu/~eaugusti/pages/papers/docs/SPOONS:%20Netflix%20Outage%20Detection%20Using%20Microtext%20Classification%20-%202013.pdf)

</comment><comment><USER3> > _Twitter's firehose is a complete stream of all tweets made on their > platform and is only available to a few businesses at an extraordinary > price._

Anyone happen to know what that price is?

<comment><USER4> Hundreds of thousands per month. At least that was the price in 2012.

<comment><USER5> Not only that, but you need to _prove_ you can handle the traffic.

<comment><USER4> Not in my experience. Why would gnip / twitter care if you can handle it, as long as you pay and abide by the terms of the contract? The burden to fetch the data within the specified window is on the client, it's not a push system.

For context: Back in the day, I worked at Klout. We had to pay a crippling monthly sum just to get access to the @mentions stream.

<comment><USER6> At one point I also read that you had to prove you could handle the traffic. Whether that was a requirement of a technical nature or just to make sure you knew what you were doing so they wouldn't end up holding your hand all the time I'm not sure.

They don't specifically say you have to verify anything, but they do dance around being able to handle the volume in a couple of different ways in their streaming guides for us average mortals [1]. It mostly seems to come down to the fact that they are, at least to some degree, buffering on their end to make sure you don't miss anything if there's suddenly a spike that's too large for you to handle or if your connection degrades for a while, etc.

It also wouldn't surprise me if it were originally simply a limitation of their internal systems and they couldn't buffer or allow any kind of replay because they were literally just writing everything off to a stream as it came in and didn't have any mechanism for going back and reading from the actual database after the fact.

1: [https://developer.twitter.com/en/docs/tweets/filter- realtime...](https://developer.twitter.com/en/docs/tweets/filter- realtime/guides/disconnections-explained)

</comment></comment></comment></comment></comment><comment><USER7> I am the author of this document. If anyone has any questions, I'd be happy to answer them!

<comment><USER8> The timestamp is generated per server. The system time could differ across nodes in the cluster (even with NTP) by nano seconds. So, isn't the accuracy of the first tweet is approximation if there are multiple tweets containing same word "earthquake" at the same time at ns level? But I get the point.

<comment><USER9> Tweet object from the Twitter API has a "creation_time" that has 1s resolution, whereas the snowflake creation time has 1ms resolution. No doubt these could disagree, but if that happened then maybe both authors could get a prize?

</comment></comment></comment><comment><USER9> Wonder how it fits with developer terms.

eg. 5d

      Do not use, access or analyze the Twitter API to monitor or measure the availability, performance, functionality, usage statistics or results of Twitter Services or for any other benchmarking or competitive purposes, including without limitation, monitoring or measuring:

          the responsiveness of Twitter Services; or
          aggregate Twitter user metrics such as total number of active users, accounts, total number of Periscope Broadcast views, user engagements or account engagements.

[1] [https://developer.twitter.com/en/developer- terms/agreement-a...](https://developer.twitter.com/en/developer- terms/agreement-and-policy.html)

<comment><USER10> I reckon not very neatly.

</comment></comment><comment><USER11> This is proper "hacker news" right here

</comment><comment><USER12> The statistical claims in the article make the assumption that tweets are being sampled uniformly at random, which is most likely false.

The fact that 3 machines handle 20% of tweets suggests that tweets are not in fact assigned to machines in a uniformly random manner. I would guess that there is a geographic bias as to which machines handle which tweets.

<comment><USER7> When I did the analysis, I was puzzled why certain machines handle a higher percentage of tweets compared to others -- so you are most likely correct that there may be some geographic consideration to the distribution.

I'm rewriting the code to include a prescan of the time range to determine which server ids are in play at the time and which server ids are most active.

Figuring out how to deconstruct Snowflake was challenging and there is still a lot of analysis left to do.

<comment><USER13> > Figuring out how to deconstruct Snowflake was challenging and there is still > a lot of analysis left to do.

Why don't you just read the code?

<comment><USER14> How do you read the code of an implementation detail of Twitter's servers? There's no guarantee that the example code they released years ago still matches what they use.

</comment></comment></comment></comment><comment><USER15> Discord also utilizes Twitters Snowflake algorithm for the ridiculous amount of messages that are sent in chats.[1]

[1][https://discordapp.com/developers/docs/reference/](https://discordapp.com/developers/docs/reference/)

<comment><USER16> Link doesn't work for me, I just get a partially-loaded page with an eternally-spinning spinner (FF65 on OSX).

Gotta go to [https://discordapp.com/developers/docs](https://discordapp.com/developers/docs) and then click on the Reference link in the sidebar.

<comment><USER17> If you remove the trailing forward slash the link works fine.

</comment></comment></comment><comment><USER18> That's quite amazing!

One thing is that the data center ids are in the tweet ids, so it could be used to get a rough location of Twitter users.

<comment><USER18> I see there is interest in this observation so I used a little ruby (.to_s(2)[-22..-17].to_i(2)) to get the datacenter id.

Then ran it on a few Twitter accounts: [https://pastebin.com/w8Dnj5kM](https://pastebin.com/w8Dnj5kM)

It does work, and it's going to be hard to patch

edit: I realized you don't only get one location but the whole location history of a Twitter user. Also locating Twitter's data centers as it doesn't seem to be public information

<comment><USER7> This is really interesting. When I did the original analysis on datacenter / server ids, I didn't think about correlation with user accounts. Nice observation!

<comment><USER18> I was curious and just did that :) [https://gist.github.com/localhostdotdev/48ed13972c3e5391a47f...](https://gist.github.com/localhostdotdev/48ed13972c3e5391a47f8e3dd7b9e0dd) (small sample of ~1500 localized tweets)

</comment></comment></comment></comment><comment><USER19> That's excellent stuff. I had no idea that's how snowflake IDs were constructed.

Now makes me wonder how hard it would be to make a tweet which linked to itself...?

<comment><USER20> It's not that hard: [https://twitter.com/mauritscorneIis](https://twitter.com/mauritscorneIis)

Code is here: [https://github.com/pomber/escher- bot](https://github.com/pomber/escher-bot)

It's a pity that twitter UI doesn't embed the recursive tweet.

</comment></comment><comment><USER21> >Twitter does sell premium data services including their much coveted "firehose" stream. Twitter's firehose is a complete stream of all tweets made on their platform and is only available to a few businesses at an extraordinary price.

As a developer, I have no understanding of how is data monetized. How much is user data worth? What kind of user data is worth more than others? What is the data used for (why would anyone pay "extraordinary price" for tweets?). Who pays for user data - is there some public market that I am not aware of?

<comment><USER22> Firehose access is prohibitively expensive. Last I heard the cost was based on something like " 30% your company's annual revenue " (this is some sort of second hand rumour I heard when I asked).

Services like gnip have everything from the firehose (ever) and it can be retrieved by paying a monthly fee (hundreds, thousand of $ for 500k twwets)

</comment></comment><comment><USER23> To store such a firehose stream of data, you will need approximately 0.3gb of storage per one second of data.

This is if you only collect username, timestamp and tweet, excluding any additional metadata such as data center, likes and retweets, not to mention images and videos.

Full calculation here: [https://docs.google.com/spreadsheets/d/1BIAguT9Qvy0GK- dalpQf...](https://docs.google.com/spreadsheets/d/1BIAguT9Qvy0GK- dalpQfkbzqtCv0pvNuxU3zbMiR-iQ/edit?usp=sharing)

<comment><USER24> I believe your calculations are too high by a factor of 1000... 53 bytes/(typical tweet) * 6000 typical tweets/second is 318 KB (or 0.3 MB) per second, not 318 MB.

<comment><USER23> Yikes, you are totally right. Fixed

</comment></comment></comment><comment><USER25> Do the server and datacenter ID's effectively leak location data?

<comment><USER26> 1) The leaked location is just "datacenterlocally", e.g. westcoast USA or northern europe.

2) There is no guarantee that traffic IS NOT routed from another region to handle traffic peaks.

</comment></comment><comment><USER27> If you need access to Twitter data you should sign up for a developer account and contact them. It's expensive but not prohibitively so for a company that needs Twitter data to do business.

For academic research Twitter offers products which dramatically reduce the amount of data you need to consume via the full search API or historical powertrack.

<comment> m-p-3 I'm wondering if the Wayback Machine has access to it.

<comment><USER28> I would be _extremely interested_ in getting the Twitter corpus ingested into the Internet Archive.

To my knowledge, the Archive does not have access to the firehose.

<comment><USER29> The Hadoop archive they recently moved to Google is 300PB. Good luck.

</comment></comment></comment></comment><comment><USER30> This is really interesting, an unforeseen insight into ids that can be enumerated even partially. I suppose encrypting or decrypting them is too costly so any other ideas for getting the properties of Snowflake at scale without this sort of attack being possible?

<comment><USER31> If you wanted to "encrypt" them, you could increase the key space and add a random salt to each one.

</comment></comment><comment><USER32> (Using figures cited in the article) Assuming that most of the time only 20 machines are responsible for ID generation, and ~50% of tweets use the first available sequence number for a given millisecond (so half the time machines only process one new tweet per millisecond), can we estimate Twitter's new tweet QPS average to be 20,000?

Edit: actually this is an upper bound for the 50% because it assumes that every machine processes a tweet every millisecond, which may not be the case

<comment><USER7> It's variable based on what's going on at the time but I've seen upwards of 7k tweets a second for the sections of the timeline that I've ingested using this technique.

Someone suggested trying it when the New Year starts in Japan. Apparently there are tens of thousands of tweets per second then.

</comment></comment><comment><USER33> If Twitter is already storing the tweet timestamp, is there a reason to generate sequential IDs? It seems like they could move to a UUID scheme and protect their "firehose" from reverse-engineering.

<comment><USER34> They're snowflake IDs, not _exactly_ sequential.

Timestamp bits, shifted all the way left Data center id shifted left Server id shifted left Increment this millisecond

Since the left bits are the timestamp, they are date/time sortable and a much more useful index/primary key/whatever than a UUID.

<comment><USER10> You can prefix the uuid with a timestamp for the same effect without leakage. There are chronological uuid projects: [https://github.com/uucid- project/spec](https://github.com/uucid-project/spec)

<comment><USER34> You _can_ , but why would you? You're advocating slapping lipstick on a pig not really engineered for this use case vs. using something that was designed for it.

UInt64's are going to be all around more efficient.

<comment><USER10> You wouldn't, just showing that it's an easy problem to solve.

</comment></comment></comment></comment></comment><comment><USER35> Twitter will just randomize the machine IDs and datacenter IDs

<comment><USER36> Doing this for historical data would break their API. They'd also need a more complex scheme than snowflake currently provides

</comment></comment><comment><USER37> Interesting! Great work. I wonder why they didn't just use uuid and have a timestamp attribute to sort them in the UI?

</comment><comment><USER38> How can Twitter be a free speech platform if you cant receive the speech for free?

<comment><USER39> Hence the distinction between "free as in speech" and "free as in beer".

<comment><USER40> Just referring to the speech part... Is the speech free as in speech if it isn't free as in beer? If there is a gateway to prevent it from being freely distributed? With GNU the free as in speech is also free as in beer -- if all you want is the speech (aka code). If you want services around the code it will cost when companies are trying to make a business on it. Correct me if I'm wrong (and I probably am), but doesn't every additional dollar reduce free as in speech because of reduced access?

<comment><USER31> GNU/FSF supports selling code for money. Twitter does the same with tweets.

You can download your own tweets for free, or even tweets of someone else freely, just doing it for all someone elses costs money.

So, with GNU _usually_ you get both, but that's not necessary. And usually seen as completely meaningless, as if there would be a very useful GPL open source software that was not free to download and use, then a simple consortium of people could get together and buy one copy and then redistribute it for free. (Though they would need to do this for all future versions.) -- which is similar to what the author proposes with regards to forming a group of ~1 700 users/folks to brute force the firehose.

</comment></comment></comment></comment><comment><USER41> This looks to considerably lower the pain of mapping retweet networks, thanks!

</comment><comment><USER42> >Twitter's statuses lookup API endpoint allows for a total of 1,200 API calls every 15 minutes. Each call allows the user to pass 100 ids for a total of 120,000 id requests every 15 minutes using both APP auth [...]

Use the secret consumer keys from Twitter to bypass these limits: [https://gist.github.com/shobotch/5160017](https://gist.github.com/shobotch/5160017)

<comment><USER43> I can't imagine these will last long now that they've been posted in a comment on a front page HN story.

<comment><USER44> These are hardcoded in the apps. If they disable a key, they would need to release a new version of the app (ok), and all users would need to update (infeasible).

<comment><USER45> Wouldn't they just enforce the limit with those keys as well?

<comment><USER46> Sounds like a solid reason to not build around a platform that can arbitrarily kill your project or business. ActivityPub seems to be gaining traction, w/o the beholden to a single entity issue.

<comment><USER47> A platform that can arbitrarily take down your business like a mobile app on iPhone or Android?

</comment></comment></comment></comment></comment></comment><comment><USER48> Interesting technical write up but a little naive about its uses as it is probably against the Twitter Developer agreement to use the API as it circumvents the rate limits.

<comment><USER49> You can't circumvent the rate limit, that is the idea of a rate limit. The article is stating this all the time.

The genious idea is simply shrinking down the search space by probabilistic assumptions. Never thought about using the snowflake ids to get historical data, i am impressed.

</comment></comment><comment><USER50> If sequence ID means what I think it means... then if I reply to a twitter thread that already has 4095 replies... there could be the possibility of a tweet ID collision, possibly causing another tweet made simultaneously to mine to be unretrievable...

After years of waiting fruitlessly for Twitter to implement my feature request, it may finally be possible to delete someone else's tweet.

<comment><USER51> The id of tweets includes the time stamp. The sequence is just an incremental number per server per millisecond. As far as I can see it's not based on retweets.

<comment><USER50> Thanks for the quick explainer on what the sequence is. Reassuring to know it's not the ordered number of tweets in a thread.</comment></comment></comment>


The DOD had two "better-than-Hubble" space telescopes just sitting around. - <AUTHOR> http://www.theatlantic.com/technology/archive/12/06/hey-brother-can-you-spare-a-hubble-dod-sure-have-two/258061/ <comment><USER3> I worked on the Hubble (my dad was systems manager for Perkin-Elmer's bid), on the ball bearings. They are literally the ones that were rejected from the spy sats.

The spy sats bought a bunch of ball bearings (these might be a foot in diameter and are speced to be extremely low noise at low turn rates). They tested them all (using a phono needle resting on the outside of the bearing while it was slowly turned). The ones that made the least noise went in the sat while the others were sealed in a plastic bag and put on a shelf in the clean room.

I was told that when Hubble came along, the US no longer had the capability to make those (I'm not sure if that was true). In any event the ones that went in Hubble were the least noisy of the ones that had sat on the shelf. My summer job was (largely) testing to see which was the best. A cool job.

<comment><USER4> In a product design class, the prof asked us to draw the distribution of resistors in a bag of, say, 5-ohm resistors, bought from a local electronics shop.

It turned out that a normal distribution centered at 5 ohms was not quite right. It was a normal distribution, but with a deep notch taken out right at 5 ohms. All the resistors that tested to very close tolerance had been bagged separately, and sold at a higher price.

(The context was why you might want to put, say, four 5-ohn resistors in series, rather than just use one nominal 20-ohm resistor.)

<comment><USER5> I hope your product design class covered error propagation, because one 1% 20 ohm resistor would still be cheaper and more precise than 4 5% 5 ohm resistors.

<comment><USER6> But four 5%-tolerance 5-ohm resistors could be chosen to add up to almost exactly 20 ohms, but a 20-ohm resistor would never be 20 ohms due to the notch.

In the old days, it might have been cheaper than the more accurate resistors. These days, high precision resistors are relatively inexpensive.

</comment></comment></comment></comment><comment><USER7> "Better-than-Hubble" is flat out wrong here, the only correct interpretation is "with larger apertures than Hubble", but there is so much more to a telescope, especially one in Hubble's class. A spysat is optimized for taking only exposures lasting a fraction of a second, for example, whereas Hubble is optimized for taking exposures that last many minutes at a time, with guidance that keeps Hubble pointed in the same direction to sub-pixel precision.

Let's take a moment to walk down the aisle with the label "Current Hubble Science Instruments". First, we come upon WFC 3, a 4k x 4k high quantum efficiency CCD imager with a wide field of view, spectral coverage from the near-UV through visible spectrum and with 63 different narrow and wide-band filters. Oh, and with a set of grisms and prisms for taking spectra, and with a separate 1k x 1k IR sensor with 17 of its own filters. Next up is NICMOS, an IR imager and spectrometer with a spectral range of 0.8 to 2.4 microns, this instrument alone places Hubble into the rankings as one of the top 3 or 4 most productive and capable Infrared telescopes in history all on its lonesome. Then there is STIS, a 3 CCD sensor with coverage from far-UV through near-IR wavelengths which is capable of taking high resolution spectra for 500 separate points simultaneously while also taking a full frame image, this instrument alone is responsible for much of all of the astronomical observations in the UV-range throughout all of history. And there are several other equally impressive instruments I have not mentioned.

The point is, none of these instruments or capabilities are going to be on a spysat. And they make up the vast majority of the value of a space based observatory. If a space telescope was nothing more than a big mirror, some guidance, and an imager then we could save billions of dollars, but they are so much more than that.

<comment><USER8> I honestly don't know the answer to this, so perhaps somebody can provide some info: Does it cost more to produce the mirror or to produce the various CCD imagers and science instruments?

It would seem to me that the CCD's should be cheaper - we have a LOT more experience producing CCD's than we have producing 3 meter mirrors capable of being launched into orbit. Surely the huge amount of research being put into producing low noise CCD's for cameras and phones would have the effect of making it easier and cheaper to produce the science instruments for a space telescope?

<comment><USER9> NASA reps said the satellite itself and the optics are the most expensive and time-consuming parts of the project.

</comment></comment></comment><comment><USER10> NASA has a fully functional copy of Hubble "sitting around" at Goddard Space Flight Center as well. If something goes wrong in space, fabrication of replacement components and the training of the astronauts that will fix it does not occur in space. It is invaluable to have an exact duplicate on the ground for this reason.

Interestingly, the total 2010 US Space budget was $64.6B. The entire rest of the world combined spent only $22.5B. NASA's 2010 budget was $18.7B. Many programs that people think are NASA projects are actually defense projects. For example, the GPS system is not included in NASA's budget, it's spearheaded by the Air Force Space Command, and comes out of the Defense budget.

Chances are the main satellites that these are duplicates for have been decommissioned, so these are no longer needed. I would guess they are actually two distinct but similar designs, and not two copies of the same design. I would assume NASA already determined that the risk of these satellites failing and NASA being incapable of fixing them is outweighed by the desire to have higher powered telescopes in space.

My mother has worked in the thermal blanket lab at Goddard for years. Several years ago, she got one of the engineers working on the James Webb Space Telescope to take her and I on a tour of the clean room where they are fabricating one of the core components, the micro-shutter array. The micro- shutter array is an array of 65,536 shutters on an area about the size of a postage stamp. We got to go into the clean room and see the entire process. It is very similar to the process used to fabricate semiconductors, and I think they were operating at about the 60nm level. The idea of the micro-shutter array is that each shutter can be independently operated to shut out interfering light sources, so that the telescope can look much further back in space and time for deep fields. These should be spectacular. Instead of imaging the entire shutter area as the Hubble does, JWST will be able to close all but one micro-shutter which should allow very long exposure times, and the ability to see extremely distant objects. More on the array at <http://www.jwst.nasa.gov/microshutters.html>.

Edit: Also, the Hubble is huge. It is a cylinder with a diameter of perhaps 15ft and a height of roughly 40ft. Pictures really don't do it justice, I had no appreciation for the size until I saw it. I know my mother did some of the thermal blanket fabrication (think the tin-foil looking stuff on the outside of spacecraft) for Servicing Mission 4.

<comment> excuse-me There isn't a fully functional flight spare of Hubble sitting in bubble wrap waiting to fly. There is a mockup of the Hubble structure and there were flight spares for SOME optical components and some parts of instruments. The mockup is to train astronauts on instrument swap outs. Many of the instruments and the current solar panels aren't even Nasa builds

The "better than hubble" telescopes are old keyhole era spy sats, this is the design Hubble was originaly based on. It's also partly the reason for the original Hubble screw-up.

The contracts for Hubble went to companies that had experience building 2.4m mirror space telescopes (nod+wink). But security walls in the suppliers meant that the people who had build these spy sats didn't work on Hubble or pass on their knowledge - then oversight of the Hubble build process was hampered by security concerns that meant NASA QC engineers never got to see the mirror being built.

A few other points.

The cost of the Hubble 'airframe' was a small part of the space telescope cost, compared to the launch and service missions, the building of the STSci and the 1000 of professors, postdocs, researchers and students that it funded. Just the cost of storing it in a clean room for 3years after Challenger was estimated at more than raw commercial build cost.

The JWST isn't just a better Hubble it's a completely different concept - actually designed as a scientific instrument rather than compromised by being a spy sat design built to justify the Shuttle launch - as HST was.

<comment><USER11> I have to second this. I worked on two NASA space telescopes (both X-ray telescopes), and although we had on the ground something like the telescopes that were flown, the equipment on the ground was not flyable, and if the launch rocket had blown up, we would have been SOL.

In fact, some of the folks down the hall from me had a telescope they were working on fail to deploy properly after launch due to the canister containing the telescope not opening. There was no spare telescope--they had to start over from scratch.

<comment><USER12> So there is an unopened space telescope sitting in a canister in orbit?

Wouldn't a device to open. The canister be cheaper than starting from scratch?

<comment><USER11> As I understand it, the designated time arrived and the canister was supposed to be open, but wasn't. A latch had failed. One or more of the payloads inside the canister started deploying themselves anyway, wrecking themselves and everything else inside the canister.

The telescope in question wasn't worth terribly much, at least compared to a Shuttle launch, which at the time cost at least $500 million, not including the cost of any payload, so even if the telescope hadn't been ruined, trying to rescue it wouldn't have made any sense.

There were some other satellites in the canister, however, and IIRC, the total value of all the satellites was nothing to sneeze at.

</comment></comment></comment></comment></comment><comment><USER13> I've just worked it out.

Years ago I went to one of my first programming conferences , sat in the same room as GvR and Stroustrop and had one of those what am _I_ doing here moments. But it was an amazing revelation - hundreds of people way smarter than me all of whom chatting where the side conversations and corridor chats are always more interesting and informed than the set talks

HN is just like the corridor outside the worlds biggest tech conference. The door bangs open and someone is talking about space and suddenly you realise there are actual rocket scie fists standing in ear shot.

I used to want to write scintillating blog posts and bump my karma score. But that happens in the main hall. Not in the corridors and I think I am happier here, like a young guy open mouthed that brilliant people actually like explaining stuff and it's beginning to make some sense...

<comment><USER14> You're already wiser than Walter Bright.

<https://news.ycombinator.com/item?id=3061222>

</comment></comment><comment><USER15> From Contact:

> "First rule in government spending: why build one when you can have two at > twice the price?" - <USER2>

For once, art has under-imitated the hyperbole that is our life.

<comment><USER16> Building two is hardly twice the cost. The design and development work has surely got to be more costly than fabrication, assembly and installation.

Not that this nit-pick changes the nature of the discussion.

<comment><USER15> 1) My comment wasn't meant as a serious critique of the situation

2) Econ isn't my forte anyway

C) You have my permission to stop being pedantic and laugh

D) I switched from numbers to letters, lest anyone develop the impression I am at all serious

:) When you make bullet lists like this, you can throw in a smiley emoticon and it will confuse at least a handful of people

6) Hope this made your Monday a little brighter :)

<comment><USER17> Pure class. I love this guy.

</comment></comment></comment></comment><comment><USER18>
        Asked whether anyone at NASA was popping champagne,
        the agency's head of science, John Grunsfeld, answered,
        "We never pop champagne here; our budgets are too tight."

Austerity notwithstanding, that's really a sad sign of the times (and state of science funding in the US).

<comment><USER19> There's no austerity. I don't even know where you'd get the idea there's some sort of austerity in the US.

NASA's problems lie elsewhere, mostly in the form of having a boss (Congress) who demands that they spend their money in mind-bogglingly inefficient ways, and then adds further inefficiencies by continuously jerking them around. Giving them more money won't solve this problem.

<comment><USER20> You're half right. I worked for NASA on Hubble. I guarantee you we were not living high on the hog. Holiday inn for 6 weeks at a time, cheapest flights for travel, cheapest everything. They balked at a $12 meal once when It was within an hour or so of a flight home once. We had trouble buying $300 hard drives at times (to run $100k software on). My engineer's salary was modest at best.

But yes, Congress makes all that "sacrifice" irrelevant due to the absurdity of their whims.

Another project I workded on: Al Gore literally had a dream one night, resulting in a colossal clusterfuck called Triana - <http://en.wikipedia.org/wiki/Triana_(satellite)>. That's $100 million, wasted on a politician's fantasy of being able to see the whole sunlit side of the Earth. What would Musk have done with that money?

<comment><USER21> Even sadder, Triana costed only 1/3 of what is spent on the war in Afganistan each day. I would prefer to see people in NASA enjoying wide arrays of 15K drives, competitive salaries, and business class seats...

<comment><USER16> Perhaps we can even splurge on SSDs with that kind of money. Spinning disks are so obsolete when we're talking that much cash.

<comment><USER22> Not when you are talking about storing Petabytes, which NASA does routinely.

</comment></comment></comment></comment></comment></comment><comment><USER23> Before we get all excited about this, I want to point out that part of the costs of any space mission, which are included in the budget figures you usually see are both the launch and the maintenance and post-launch operations for commanding the satellite and analyzing the data.

Just having two satellites sitting about isn't really that shocking, nor is the fact that the DOD has technology that is more advanced than Hubble, Hubble was originally funded in the 70s, slated for a launch in 1983, and finally launched in 1990.

</comment><comment><USER24> That's what you now know about. They are probably quite primitive devices if they let you find out about them.

From real experience, the commercial and public space programmes are positively infantile compared to the black project space programmes.

<comment><USER25> Sadly, I'd imagine all those super toys are designed to point at Earth - different design requirements than for looking outwards.

<comment><USER26> That's what has me curious. The Hubble isn't designed to point at earth and would be of little use if we tried. Why does the US Military have telescopes that are on par with Hubble? Are they looking out into space as well? Why would they do that?

<comment> excuse-me As far as the telescope optics is concerned looking up or looking down is irrelevant- 500km or 500 million light years is the same focus setting!

Looking up has more stringent pointing and tracking requirements just because we want to sit on the same object for hours at a time, but we also have lots of bright stars in the field to track. Although our downward looking colleagues also use small star tracking telescopes on their toys.

The big difference was generally in the cameras. Astronomers use a 2D CCD (like your digital camera) to take a long (hours) exposure which is read out at the end. Spy satelites (used to) use a 1D sensor like a scanner or fax machine which was constantly read out as the earth passed underneath - producing a long continuous strip image across a target.

</comment></comment></comment></comment><comment><AUTHOR> Can someone explain how two telescopes meant to focus on relatively close targets, without any cameras inside, are "better" than a Hubble? What metric are they using?

<comment><USER9> The lenses are just as big but they have a moveable secondary, which gives them better focus and 100x field-of-view compared to Hubble. [http://www.washingtonpost.com/national/health- science/nasa-g...](http://www.washingtonpost.com/national/health-science/nasa- gets-two-military-spy-telescopes-for- astronomy/2012/06/04/gJQAsT6UDV_print.html)

<comment><USER27> I assume the NRO telescopes had their instrumentation _removed_ before gifting the optics to NASA, yes? Or were they really just "blank casings"? Either way, it's kinda freaky to think that there are such powerful monitoring devices in space...

<comment><USER9> Since they were never launched, and (according to NASA) the satellite itself is the part that takes the longest to build, it's plausible that the instrumentation was never installed before the mission was scrapped for whatever reason.

</comment></comment></comment></comment><comment><USER28> Not too long ago there was this diplomatic incident in my country where the US military forgot to mention they where bringing (or tried to smuggle in) a briefcase with some GPS equipment and other stuff. They where discovered when their plane was inspected and the briefcase was not in their customs declaration.

They where forced to open the briefcase, which was less than 1 meter (3 feet) wide. They agreed only if the briefcase was opened under a roof, alluding it was standard procedure because of spy satellites looking at its content.

That was when I realised that spy satellites currently have ridiculous amounts of optical resolution I had never though equipment could achieve in orbit.

<comment> excuse-me They can't. The diffraction limit of a 2.4m (HST/KH) mirror from their orbit is around 6cm, the practical limit with an atmosphere is around 10cm.

However the photo-recon interpreter guys are amazing. They can look at a 1m resolution image of a vehicle made up of half-a-dozen blurred pixels and say "that's the new mkII whatever - you can see the extended wheelbase from the length of the shadow"

<comment><USER28> Can't they be using bigger telescopes? The James Webb Space Telescope is declassified technology and has a self-assembling 6.5m mirror.

I'm not sure how atmosphere is optically taken into the equation, but maybe having a mathematical model of it can be used to increase the 10cm practical limit... sort of like with heat shimmer binoculars.

<comment> excuse-me You could use a bigger mirror, although a 6.5m mirror in low earth orbit is going to have a very short lifetime due to drag. The atmosphere has much less optical effect looking down, simply because all the phase tilts are close to the target and so don't have much angular effect. It's like laying frosted glass on top of a document and looking at - compared to putting the frosted glass upto your eye and looking at a distant object.

Really there is a limited return from higher and higher resolution imaging. Once you have vehicles, roads, missile batteries etc spotted there isn't much point being able to see which of the operators are bald.

With modern wars photo-recon is even more limited, you could have mm accurate imaging from a drone of an Afghan tribesman but it wouldn't tell you what he thinks of your politics.

</comment></comment></comment></comment><comment><USER29> _Are drones replacing space telescopes?_

This sounds like a reasonable speculation to me. Drones would be orders of magnitude cheaper, more manuverable, expendable, not as subject to being obscured by cloudcover, able to survey many places at once or swarm over a wide area, etc.

<comment><USER30> But we can't fly drones over China and Russia like we can Pakistan

<comment> excuse-me And satellite surveillance is less useful these days.

1960s - we have photographed Soviet SAM sites around Moscow. We know where to avoid to send bombers in.

2010 - we have photographed a semiconductor fab in china. We don't know what they are making but we can tell where the fab is to an inch.

<comment><USER30> You can tell a lot about what they're making based on its proximity to other resources, the shape of the buildings, the schedule of supply deliveries, etc. Intelligence analysts are a crafty bunch.

<comment><USER31> How'd that work out for the US with Iraq?

[http://articles.cnn.com/2004-04-03/us/powell.iraq_1_official...](http://articles.cnn.com/2004-04-03/us/powell.iraq_1_official- iraqi-organization-biological-weapons-labs-state-colin-powell?_s=PM:US)

<comment><USER32> Has anyone written a definitive account of the intelligence failure here? A step-by-step kind of thing, not a pointless "BuSh LiEd!!!" kind of thing?

<comment><USER31> Yes. A commission appointed by Bush took a look at the nation's intelligence community after the Iraq failure and found it to be severely lacking.

[http://articles.cnn.com/2005-03-31/politics/intel.report_1_m...](http://articles.cnn.com/2005-03-31/politics/intel.report_1_major- intelligence-failure-iraq-survey-group-national-intelligence- estimate?_s=PM:POLITICS)

<comment><USER32> Thank you, that lead me to the actual report here- <http://govinfo.library.unt.edu/wmd/report/index.html>

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> I think SpaceX should take one. Throw maybe $10k of off-the-shelf parts in there, add some comms, and you have the best proof-of-concept and advertising ever.

<comment><AUTHOR> I think Facebook should buy one. I'm tired of hearing about nebulous data- mining aggregate privacy concerns; I want to be afraid of Facebook literally watching me, damnit!

<comment><USER33> Google owns a satellite service, (<http://en.wikipedia.org/wiki/Keyhole,_Inc>) why shouldn't Facebook and Apple?

<comment><USER34> Keyhole wrote the predecessor to the Google Earth software, but while it's a great name for an imagery company, they don't run satellites themselves.

Google did pay for a half share in GeoEye-1 (<http://en.wikipedia.org/wiki/GeoEye-1>) and have exclusive commercial access.

</comment></comment></comment></comment><comment><USER35> It would be interesting to see the two telescopes (or whatever device) that replaced these two.

There's probably a very limited number of groups that would be able to use this technology in a way that was agreeable to the military.

</comment><comment><USER36> More proof that defense gets a lake of cash for every bucket that NASA gets.

Optically I'd think looking down at a relatively bright earth would be a much different task than looking up at a _very_ dark universe. Also field of view is much less important.

</comment><comment><USER37> It's interesting to think about how far advanced their equipment is now if this is their equivalent of "spare change" to throw at NASA

</comment><comment><USER38> Define "better".

If these telescopes have to be equipped with instruments, what is left? The optical system? Thats nothing compared to the instruments.

Better in terms of optical system would mean an order of magnitude at least. This is obsolete, more or less, because ground based instruments come closer to a proper correction of atmospheric turbulences. They are cheaper to build and cheaper to run. The next generation optical space telescope may be a very long baseline interferometer. This cannot be build on earth.

Infrared is also a space domain. Thats why Herschel, ISO, Spitzer and James W. Webb are space telescopes.

</comment><comment><USER39> If only the worlds combined military budgets could be directed towards a global space (/science) program. Less investment in death and more in the future of human kind.

<comment><USER40> It's more difficult to profit from scientific exploration than it is from weapons and the maintenance contracts that come with them. The military industrial complex has a stranglehold due to simple market forces.

</comment></comment><comment><USER41> I wouldn't complain too loudly. Otherwise, next time, they'll quietly dispose of such things, rather than risk public backlash.

</comment><comment><USER42> 'Second, if the DOD didn't need these two birds, which are both better than any civilian telescope, what _do_ they have?'

This line worries me the most, hubbles pictures of things thousands of light years away are spectacular. Imagine what a better scope could so pointed at earth 300 miles below!

</comment><comment><USER43> Now I see why the government is in so much debt. Building something they already had two of, saving potentially billions in research & development creating another telescope. This is ridiculous. What else have they got two of just "sitting around"

<comment><USER33> because the KH-12s didn't work as designed?

</comment></comment><comment><USER44> Would they be any use as ground-based telescopes?

<comment><USER45> Probably not. The design of telescopes face different environmental challenges between Earth-based and space. On Earth, we have the benefits of accessibility but the downside of peering through the atmosphere. In space, there's no atmosphere but you have to deal with inaccessibility, extreme cold, micrometeorites and space debris, and radiation.

Also take into consideration mirror size. On Earth, we have built (and are building) _giant_ mirrors many meters across. In relation, the mirrors on Hubble and other space telescopes are rather small in comparison. Its really the lack of atmosphere that provides most of the advantage to space-based installations.

<comment><USER33> well that, and the CCDs love cold.

</comment></comment></comment><comment><USER46> Yes, in case you didn't know, the DoD operates a secret space program completely independent of NASA and with an unknown budget.

</comment><comment><USER47> Good god.</comment>


Overachieving Ivy-League Grads and "Success" - <AUTHOR> https://qz.com/986966/overachieving-ivy-league-students-are-learning-the-wrong-lesson-about-what-it-takes-to-be-successful/ <comment><AUTHOR> I don't know how true it is, because I'm not traveled, but I've always heard the French being labeled "aloof/lazy." The French author says he eventually got caught up in [The US'] overachieving culture. It never occurred to me that Americans would be perceived that way... Are Americans labeled "overachievers" abroad?</comment>


Bitcoins the hard way: Using the raw Bitcoin protocol - <AUTHOR> http://www.righto.com/2014/02/bitcoins-hard-way-using-raw-bitcoin.html <comment><USER2> Do not, do not, DO NOT, try writing raw transactions at home. You will fail and you will lose a lot of money.

If you must: \- Use testnet! If you don't know what testnet is, you have no business writing live raw transactions. \- ALWAYS validate your transaction first by having bitcoind pretty-print it, and checking the output carefully, before sending.

<comment><USER3> What someone does with their money (virtual or not) is their choice, and not for you to decide...

<comment><USER4> It's solid advice, as it's very easy to lose money writing transactions manually.

</comment></comment></comment><comment><USER5> Be careful if you try this at home; a lot of people have lost money this way. (Not sending change is a common mistake.)

<comment><USER6> To elaborate on that. If not very careful, you can end up sending a big fee, since the fee is calculated in the following way:

sum of all inputs values - sum of all outputs values

"Change" is an output that you control to which you will send the remaining amount. Failing to do so will end up sending all remaining value as transaction fee, which then goes to the miners.

If you end up doing that (as I did few months ago) you still have a chance though - through blockchain.info you can get information about who mined the block in which your transaction was included. If you're lucky, it will be one of the big pools. You can then try to contact them and ask to send the amount back to the originating address. In my case BTC Guild was nice enough to do exactly this.

<comment><USER7> With pools moving towards paying miners the transaction fees it becomes more difficult to recover from mistakes like this. The pool would have to contact the miners that received the fees to recover them.

</comment></comment></comment><comment> aaron-lebo If you want something in a similar vein, Khan Academy has some good videos describing the protocol and not just a high-level overview (though there is that, too).

[https://www.khanacademy.org/economics-finance-domain/core- fi...](https://www.khanacademy.org/economics-finance-domain/core- finance/money-and-banking/bitcoin/v/bitcoin-overview)

</comment><comment><USER8> One of the best articles I've ever read about bitcoin.

<comment><USER9> Came in here just to say that. Thank you for whoever posted this and more importantly, whoever wrote it.

<comment><USER10> It looks like they're one and the same, so the karma is going to the right place :)

</comment></comment></comment><comment><USER11> I've always wanted to learn how to query a coin network for things like current difficulty and total hashrate without running a full blown wallet to get a console.

Is this documented anywhere perhaps? I cannot seem to find.

This helps a little [https://en.bitcoin.it/wiki/Protocol_specification](https://en.bitcoin.it/wiki/Protocol_specification)

but very low level and no mention of hashrate

<comment><AUTHOR> Try [https://blockchain.info/stats](https://blockchain.info/stats) for stats from the past 24 hours. That site also has a whole lot of historical charts.

</comment></comment><comment><USER12> Linking to a thread about a paranoid blog/rant, regarding:

"Bitcoin is a cryptographic botnet"

      Bitcoin, I believe after reviewing the protocol
      description here is a US gov't black op.

[https://news.ycombinator.com/item?id=7165893](https://news.ycombinator.com/item?id=7165893)

Possibly worth discussing, since it's reactionary slant was directly provoked by OP's topic.

Seems to have gotten buried. Possible deliberate forum sliding?

[http://lunaticoutpost.com/Topic-Technique-1-FORUM- SLIDING](http://lunaticoutpost.com/Topic-Technique-1-FORUM-SLIDING)

The only reason I might suggest considering this botnet premise is due to hardware mining.

ASICs are pretty much the norm for mining now, and near as I can tell, there hasn't been much discussion regarding all the custom hardware floating around lately.

<comment><USER5> Not familiar with the crackpot index, huh?

BTW, the behavior of ASICs is well understood and has been audited by the authors of mining software (who are independent from the ASIC vendors).</comment></comment>