"I was just asked to balance a Binary Search Tree by JFK's airport immigration" - <AUTHOR> https://twitter.com/cyberomin/status/835888786462625792 <comment><USER2> Once (also in JFK) I was quizzed a bit by the CBP officer. I told him I was a math grad student, he said "well, then tell me about the Euler problem". I explained that Euler was a fairly prolific guy, and asked if he could be more specific. He didn't relent, apparently he had seen some documentary that was all about the "Euler problem".

Eventually, we moved on to my background, and I mentioned I had done a masters in computational neuroscience. He said something like "oh neuroscience, my great aunt had that", I think he thought it was an illness? Was sort of expecting the reality TV cameras to be busted out at that point...

He let me through though! Usually I clear immigration in Ireland (one of the few places you can do the immigration before you leave), and those folks are always much more pleasant.

<comment><USER3> I was once traveling back from Italy after attending PKDD conference. I was really sick and had a really bad headache, so the following exchange really scared me:

        "Why did you go to Italy?"
        - To attend a conference.
        "Which conference did you attend?"
        - European Conference on Machine Learning
        "Machine Learning? (smile) How do you learn from machines?"

I forgot the exact answer I gave, but I was pretty afraid that he had a Wikipedia page on "Learning from Machines" and would not let me pass if I failed.

<comment><USER4> Are you sure he wasn't just joking around? I've had a TSA officer joke around asking me if I hacked into their devices (I was wearing a hackathon t-shirt)

<comment><USER5> > Are you sure he wasn't just joking around?

One rule of thumb is that you _Never_ joke with the airport security (1). So wouldn't that go both ways?

1) [https://www.theguardian.com/commentisfree/oliver- burkemans-b...](https://www.theguardian.com/commentisfree/oliver-burkemans- blog/2012/nov/27/dynamite-advice-bomb-jokes-airports)

<comment><USER6> So you're saying I should better not sport my new t-shirt in the airport?

[http://www.tshirthell.com/funny-shirts/if-im-not-on-a- watchl...](http://www.tshirthell.com/funny-shirts/if-im-not-on-a-watchlist- someone-isnt-doing-their-job/)

<comment><USER7> My then-teenage brother once went through US immigration wearing a t-shirt branded with the initials of the then-popular brand, French Connection UK.

Needless to say the US immigration officials took some offense and asked him a bunch of questions. I'm not sure they realised it didn't say what they thought it did, even after we explained about French Connection.

These were obviously simpler times as I also as a teenager once went through US immigration wearing a Threadless Community Party t-shirt: [https://www.threadless.com/product/383/the_communist_party](https://www.threadless.com/product/383/the_communist_party)

<comment><USER8> The dyslectic f*ck brand was and still is annoying. In Melbourne AU they had a 20 or 30 metre tall logo covering an entire wall of an industrial building. Very glad to see it gone.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> I wonder if P = NP would end up being proven by someone who didn't want to be sent back after 23 hours of traveling.

<comment><USER10> I can see it now. In 100 years we will learn about how a TSA clerk who had failed mathematics back in grade school went on to prove P = NP and become synonymous with "genius" for generations.

<comment><USER11> I think he meant by a traveler who is stopped by a TSA agent.

<comment><USER12> Sure, but they gave the TSA Agent a copy of their proof...

</comment></comment></comment></comment><comment><USER13> More I read about things like these, even more I love Germany (my new home). The officers in Frankfurt Airport are always nice and I never had any problems nor heard about one. I hear it's even better in Berlin.

I don't understand why host countries don't try to make themselves the most favorite country in the world for their guests. I believe, "not allowing terrorists/drug dealers/illegal workers etc. etc. inside" is just the marketing as I seriously doubt people who slip through the cracks of a saner screening process would be a significant problem. Therefore, I wonder the _real_ reason but can't think of any.

<comment><USER14> > "not allowing terrorists/drug dealers/illegal workers etc. etc. inside"

All three of those have an access to a very sophisticated immigration scam operatives. Terrorist get extensive training in faking their identities, drug dealers have huge network of contacts and insider moles and illegal workers know the trafficking cartels.

The real reason of American immigration fiasco is deep rooted racism and against primarily non-white people. You will barely hear anything from any politicians about white people who illegally come and stay in USA from Canada and Europe. If H1B beneficiaries were mostly British or Canadians US government would have increased the limit 5x by now.

<comment><USER15> > All three of those have an access to a very sophisticated immigration scam > operatives.

While that is true in many cases, in many others it is not. I lived on a border town for a few years and made friends with some DHS agents there. Listening to their stories re drug trade I got the impression that many of the people carrying drugs over are actually poor, uneducated, uninformed, etc. While some of the techniques may be sophisticated, I got the impression that the traffickers themselves frequently were not. I would imagine basic abrasive interrogation techniques would be reasonably effective.

<comment><USER16> Devil's advocate -- maybe the poorly educated, poorly trained traffickers were overrepresented in the sample of traffickers border agents encountered, because, by definition, a successful trafficker is not caught or observed trafficking.

</comment></comment></comment></comment><comment><USER17> In a away this actually makes sense. Instead of relying on automatic systems and algorithms, you rely on people, namely the agent performing the interview. A short chat about selected topics might be actually quite revealing. Point not being if you know the exact right answer for the BST question but more how you react and if there are inconsistencies in your story (most people are pretty bad at lying).

All of this could be backed by the personal data has been collected before hand.

The old (and often linked?) article about airport security in Israel[1] describes how the actually rely a lot on people being able to spot strange behavior.

[1] [http://www.huffingtonpost.com/daniel-wagner/what-israeli- air...](http://www.huffingtonpost.com/daniel-wagner/what-israeli-airport- secu_b_4978149.html)

<comment><USER18> This is exactly correct. When I returned from a weekend vacation in New York a few years ago, Canadian customs asked me about the tourist sites I visited. It's not that they thought attending a musical and visiting the 9/11 memorial was suspicious; it was just a convenient topic of conversation while they watched to see if I seemed nervous or otherwise suspicious.

<comment><USER19> I don't know about 'Correct'. When you enter Australia you never even talk to anyone anymore - immigration is all done via self-serve kiosks.

<comment><USER20> For the select few with the right citizenship. Everybody else queues up to talk to immigration. During busy time it can take 2h. The self serve kiosks are not the norm.

<comment><USER21> Travelling to Australia as a New Zealand citizen is about as easy as it gets. My last flight from Sydney to Auckland I was at home 45 minutes after the plane landed.

</comment></comment></comment></comment></comment><comment><USER22> This happened to an Australian guy recently. Was asked to code up some stuff in Python.

[http://www.news.com.au/travel/travel-advice/travellers- stori...](http://www.news.com.au/travel/travel-advice/travellers- stories/aussies-weird-immigration-interview-in-the-us/news- story/8222c65d2f12e6691ef27c9b1753e821)

<comment><USER23> I'm sorry, but after the stress of traveling though an airport, I'm not coding anything until after I get home (or to a hotel) and rest.

<comment><USER24> You are always allowed to get home if you prefer that as long as it's outside U.S.

<comment><USER23> Sure, defend the ridiculous.

<comment><USER25> Note the bonus assumption that legal immigrants don't matter and the US isn't "really" their home.

</comment></comment></comment></comment></comment><comment><USER26> Around 2002 we (a group of Russians) were returning with friends from a ski trip in Austria back to Germany. There happend to be a police control in the train. Then I've realized I had no passport with me. The best ID I had was my business card (of course, no photo on it).

So the policeman looks at it and says "Oh, the database department. Which databases do you guys use?" Basically I was able to identify myself with some Oracle and knowing who Scott Tiger was.

The policeman appeared to be an IT drop-out who switched to police.

</comment><comment><USER27> I'm going to wait for more information before believing this. There have been so many cases of fake twitter outrage that it makes sense to wait a day or two before taking out the pitchforks.

What's sad is that in this climate it's not completely unbelievable that something like this could happen.

<comment><USER28> I'm not sure it would be cause for pitchforks if true. Immigration officers in general tend to act aggressive and ask weird questions as some sort of attempt at behavioral screening. Canada only cares that tourists have enough money to leave and don't plan to work, but the last time I went, I got asked why there was mud on my car. Seriously.

</comment></comment><comment><USER29> Something similar happened to me around 5 years ago. I'm a white 20-something male and was returning from Colombia on my own, with no bags except a small backpack. I guess that seemed suspicious enough that I was asked specific, quiz-like questions about my profession, and then sent to secondary screening.

In other words, this does not seem to me like anything new.

</comment><comment><USER30> I think there's been quite a bit of overreaction here.

Part of their job is simply to determine that you are who you say you are. They do this every day - within reason I expect they can ask you any arbitrary question related to you and your life to observe how you respond. They just want to catch those out that aren't, for whatever reason, being genuine about their story. I think there's a false assumption here that somehow @cyberomin's ability to actually _solve_ a BST was in some way tied to his likelihood of entry. That's obviously ridiculous. I imagine his response was still genuine enough that they believed him. Frankly as another commenter joked, I think they could have asked him to prove P=NP. It's when you reply confidently with an 'obvious' solution that they might raise an eyebrow.

Yes it would be unreasonable if their ignorance actually lead to you being denied entry. But in not one of the experiences here, or with @cyberomin, was that actually the case. The thing is, of course, that they're aware of their ignorance and the absurdity of their questions. You see, they know that they're bluffing. That's kind of the point.

It is, perhaps rather unfortunately, part of their job to make you feel uncomfortable, ask probing questions and weed out inconsistencies in your responses. Isn't that part of professional questioning? I can understand the frustration at the abruptness of the questioning, and it's hardly a friendly welcome. I've been angered leaving immigration before and I have to remind myself that ruffling my feathers is all part of the act. When first complaining to a friend she asked, "what did they insult you or something?" Actually, no. They just asked me lots of questions.

Clearly I'd like to think some questions are off-limits - I know there have been justified concerns about this - especially recently. And I understand this is hot topic right now - precisely why I think we should be careful - more careful than ever - not to overreact. From what I observe: The guy arrived, he was asked questions about his profession, his answers were evidently sufficient and he was let in. It happens to thousands of people every day. Honestly? I don't think there's a story here.

<comment> joatmon-snoo The problem is that there are infinitely better questions to ask - like, "oh, what's your day-to-day like?" or "who do you work for? what do you work on?" \- rather than terrible technical questions.

=====

> I think there's a false assumption here that somehow @cyberomin's ability to > actually _solve_ a BST was in some way tied to his likelihood of entry. > That's obviously ridiculous.

You just got off a plane. You've been in the air for the past 23 hours. The agencies in charge of border security at your arrival airport have a reputation, and a well-deserved one at that, for (1) harassing foreigners, (2) having unclear policies, and (3) more recently, having no idea what the actual law is (or actually outright disobeying it).

Still think it's that ridiculous?

=====

On top of that, there's a long history in our industry of people reading questions and answers off pre-prepared lists and saying "oh, your nuanced, _correct_ answer doesn't match up with what I've got written down here - sorry, I'm trashing your application".

While it would absolutely disgust me to see the TSA/CBP use something like that, I would also be completely unsurprised to see it happen - especially if it happened to a non-US citizen, since they have next to no legal standing in the U.S.

<comment><USER30> I think this gets to the heart of how so much of this discussion misses the point. The question is not about solving CS problems. It's about determining the authenticity of the passenger's story. In this example, provided that the border agent is at least sufficiently aware that a BST question is a 'hard problem', the question may have simply been posed to observe if the passenger reacts appropriately to that. Even if the answer is "look, I've just been on a plane for 23 hours! you couldn't possibly expect me to..?", You're right, the agent couldn't. Tick! The response is consistent with the passenger's story.

It's a bluff. Which is to stress a more important point: it's not for us to second guess the subtleties of a border agent's questioning. That's part of their professional training. -- That is of course, unless we have cause to believe the questioning is in some way unethical. And you raise (1), (2) and (3) that are clearly examples of unethical behaviour. But with respect to this story - where is the evidence that any of this happened? [1]

This is _precisely_ why we should be careful not to generalize and be accurate in our criticism - if we are to be taken seriously in an area where there is understandably much cause for concern.

[1]: To be clear: A technical question about a passenger's stated profession is not unethical. Yes, it would be unethical if the quality of the _technical_ answer was somehow tied to admittance. But again there's no evidence that that happened.

</comment></comment></comment><comment><USER31> Returning from Canada back in the 90's, I was asked what I had been doing there.

"Attending a conference on computational geometry."

"What's that?"

Pause for thought -- several seconds.

"Applying computers to solving problems in geometry."

This got me waved in. Co-workers told of the encounter later on proposed an alternative reply: "You wouldn't understand." Given the tone of the agent, those might have been very costly words.

<comment><USER18> Could have been worse. If you had been at a conference on _algebraic_ geometry, you might have been the mathematician who started talking about blowing up points on the plane while in the security line up...

<comment><USER32> You can do algebra on a plane, but don't do algebra on an aircraft:

[https://www.washingtonpost.com/news/rampage/wp/2016/05/07/iv...](https://www.washingtonpost.com/news/rampage/wp/2016/05/07/ivy- league-economist-interrogated-for-doing-math-on-american-airlines-flight/)

[http://www.telegraph.co.uk/news/2016/05/07/italian- mathemati...](http://www.telegraph.co.uk/news/2016/05/07/italian- mathematician-taken-off-flight-after-fellow-passenger-al/)

</comment></comment></comment><comment><USER33> I want to joke that CBP might be looking for referral bonuses by referring great candidates to tech companies.

However, if it's really true, it is depressing. He was actually given an A4 sheet to balance a BST. Wow.

What if he failed the test? I assume that CBP cannot deny you entry because you didn't brush up on your algorithms and data-structures during your flight. But, they can still keep you detained for secondary screening.

<comment> nevi-me This is what happens when you have a populist president, people start disregarding the law and listen to the President even if he/she is wrong.

Imagine this happening at the time when companies are starting to accept that you can't interview candidates solely by asking them to solve algorithms on a whiteboard, then some inexperienced staff go to Wikipedia and expect people to be able to recite it.

The question's what happened as he said he was too exhausted, does he get flagged for secondary screening? The travel ban is suspended AFAIK.

We are witnessing self-division at sad levels. We had a flare up of xenophobic attacks in South Africa last week. In an earlier incarnation of such attacks in 2008, mostly Zulu people would ask you what certain things are in Zulu, and if they perceived that you don't know what they are, they would accuse you of being a foreigner, and anything could happen to you. An example that became popular was "what is an elbow or knee in Zulu?". I'm not making this up, and this happened in a country with 11 official languages.

The point I'm getting at is that this lawlessness started because a populist president and other leaders started shooting from the hip, making inflammatory statement and playing political football with our lives.

Trump is worse IMHO as he signs an Executive Order and things go crazy until the courts can act. You should worry when your first citizen is not well- acquainted with your Constitution like our dear Jacob Zuma and your Donald Trump are. The country ends up being run through the courts, and the courts don't act expediently.

<comment><USER34> > This is what happens when you have a populist president

Spare me. This stuff was happening pre-Trump, and it will happen post-Trump.

<comment><USER35> Care to provide a citation for people being asked hard algorithmic questions by CBP before?

To be clear, I don't think Trump is directly responsible for this. But I think he's sent a signal to fascists and racists around the country that America is for white people and it's okay to throw your weight around. He's a bully, so he's emboldened other bullies (which, unfortunately, dominate law enforcement).

<comment><USER36> > _But I think he 's sent a signal to fascists and racists around the country > that America is for white people and it's okay to throw your weight around_

You have that exactly backwards. He was voted in legitimately by the right and center because the left failed to do anything to address their grievances, justified or not, and circumstances.

Trump (and Brexit, etc.) is the signal that the public is sending to the left and may heaven help us all because we're sure going to need it.

<comment><USER35> I think the reasons Trump won and the message he's sending in office are, in fact, different things. There were a whole lot of unhappy Trump voters (people voting for him despite lots of reservations about his policies and demeanor) and a lot them were due to the left not addressing their grievances.

That doesn't change the fact that his rhetoric _has_ emboldened hard-right fascists and bullies. It's inarguable at this point.

</comment></comment></comment></comment></comment></comment><comment><USER37> Not as spectacular, but 4 years ago when I arrived at SF airport after a 11 hour trip the immigration officer asked why I was there. I told them I was visiting a software conference. To my surprise he then asked me to write 16 in hexadecimal. Luckily I didn't screw up.

I asked him how come he knew about that stuff and he told me that in a former career he was working in IT as well. I thought it was funny.

<comment><USER38> I was just overcome with anxiety...the answer is 10, right?

<comment><USER37> yes :)

</comment></comment></comment><comment><USER39> December 2016, I arrive in JFK from Moscow and a border officer asks me about the purpose of my visit. I explain that I attend the business meeting in NC. She wants to help and informs me that I have to pick up my luggage in NYC and then check it in for my next flight. I smile and reply "fortunately I don't have any luggage". Immediately she becomes suspicious of me and asks "But you are here for a business meeting, where is your business suit?" I explain that I am IT and everyone in my company is OK with a casual style". She nods and let me go. "Thanks God IT does not really care about my clothes" I think and successfully enter the land of the free.

<comment><USER40> Plot twist: she knew you didn't have any luggage all along. That information is available to her when she pulls your info up on her computer.

<comment><USER39> In JFK there 2 modes of going through the passport control : the old one involving an officer with a computer, fingerprints scanner and camera and the new, automated one that involves you interacting with a kiosk that prints some kind of voucher at the end. This voucher then is given to an officer who does not have a computer, but she usually asks a couple of extra questions. I used the new way, I like it. So perhaps she didn't know anything :-)

</comment></comment></comment><comment><USER41> I have a theory on how CBP is trained to scan people, that seems more plausible without assuming malicious intent: they're not expecting you to know exactly how to balance a BST, but just enough to hold a conversation about it while they look for telltale signs. My guess is that even a reply like, "Look, I'm a self-taught web developer and I don't deal with algorithms day-to-day, sorry. But you can ask me web dev questions if you wish." would work.

<comment><USER42> Agent: "wrong answer,you are not allowed entry, good bye".

<comment><USER43> What reason do you have to believe this would happen?

<comment><USER44> In many government jobs, the criteria for moving a task forward is the ability to check off a tick box. So you do not admit you do not know the answer; you give a plausible answer that lets the officer check the box. That's the case for most govt questions. Very little depth is required in the answers as not everything can be validated in the timespan of an interview. So admit you know what a binary tree is and talk about it in high level terms. Unless the officer was a previous programmer, the checkbox gets checked.

When you say you do not know or answer a different question, it makes it harder to check that box off. Most officials just want to check a box and move you (and themselves) on. Don't make it difficult for them.

<comment><USER45> You assume that the box says next to it "can answer questions about his career specialism correctly". It probably doesn't. More likely it says "does not appear nervous when asked about his career specialism, and appears to know topics he claims to know" or something more like that.

</comment></comment></comment></comment></comment><comment><USER46> Not even close, but similarly: travelling to Israel for work some years ago I was asked what I did for a job. Software engineer, I said. 'Where is your laptop?' I explained I didn't have one, that I worked only on desktops. It took some persuading the guy that you could be a software engineer without a laptop.

</comment><comment><USER47> I flew to the USA late last year, through London. When going to the desk for my connecting flight at LHR, the gentleman there did some sort of pre- authorization and asked me a number CS question. Nothing as specific as the above, but things like what languages I used for work, specific tooling, who my clients were etc. It felt quite strange to be asked these things, it's never happened to me before when traveling anywhere including stateside. What's more, this was a person hired by American Airlines, not some CPB officer – or, at least, it seemed that way.

<comment><USER43> Is it possible the person was just curious/interested?

<comment><USER47> I thought so at first as well, he seemed curious but kind of prying, so I asked whether the questions were necessary. He answered politely but sternly that yes, yes they were. After that, it was a bit awkward and stiff but still quite polite. It all just seemed very strange, I've never been questioned like that before.

Once I got to the states however, the only problem I had was that lines were long. The CBP officer handling me and letting me in to the country was very nice and polite, and after the usual "business/pleasure?" and "how long are you staying?" questions etc. I was let in to the country with no issue.

</comment></comment></comment><comment><USER48> At first I thought this was a joke, but after reading some of the comments here I'm truly scared of traveling abroad. I've been programming computers for 25 years and I don't think I could build a binary search tree, let alone balance one. :O

<comment><USER43> So what? Neither could the guy who tweeted this, and he was still allowed into the country.

</comment></comment><comment><USER49> Potential answer: "Call the balance method on the BST object"

<comment><USER23> While cheeky, that answer gets you at least 3 hours in the cage.

</comment></comment><comment><USER50> The best CS subject to talk about in such context would be "race conditions"!

<comment><USER51> Deadlock holiday

</comment></comment><comment><USER52> Google's recruiting tactics are getting ridiculous!

</comment><comment><USER53> I have been asked something similar about 7 years ago, when I went to renew my Green Card, few days ahead of my scheduled appearance. The agent, a math major with a sense of humor, asked me to do 12 factorial and then he woulde accommodate me. I did, and he held his side.

But I kindly doubt this guy's binary search claim.

</comment><comment><USER54> [https://trends.google.com/trends/explore?date=now%207-d&geo=...](https://trends.google.com/trends/explore?date=now%207-d&geo=US&q=binary%20search%20tree)

Looks like people are studying up?

</comment><comment><USER55> Are they doing this for everyone? "Oh, you work in a slaugherhouse? Disembowel this cow."

Absolutely absurd.

</comment><comment><USER56> This is a story I've read it somewhere.

"At the airport, When I said I am Japanese, the officer handed me a Japanese newspaper and asked me to read it aloud. The newspaper had really rare name of the place I couldn't read it. The officer suspected me."

</comment><comment><USER57> That's pretty impressive that the immigration officer knew what a binary tree is. People who can't code a FizzBuzz but faked their way through an interview should be concerned.

<comment><USER58> It seems they just had a sheet of questions to which you had to give the exact answer they had written down.

<comment><USER51> Like a real interview then.

</comment></comment></comment><comment><USER59> Some of the replies in the Twitter thread are hilarious

        whiteboarding, it's the new waterboarding? Furious /@AnnieTheObscure

</comment><comment><USER60> I really don't see the problem here. They likely need an anchor for enough conversation to "vet" a person's story. If someone is utterly clueless about their stated career, it should be a red flag enough for some further questioning.

Folks have a hair trigger about being offended by this sort of thing but it seems completely reasonable to me.

</comment><comment><USER61> Thankfully he didn't get shot.

[https://www.theguardian.com/us-news/2017/feb/24/killing- of-i...](https://www.theguardian.com/us-news/2017/feb/24/killing-of-indian- man-in-kansas-bar-investigated-possible-hate-crime)

</comment><comment><USER62> I wonder if they are that insistent on people whose occupation is porn actor. I imagine so many lines of questioning there could be legally considered sexual harassment that the interviewer would decide to let you go out of self- interest.

</comment><comment><USER63> What is the point of CBP? Can't their jobs just be automated away by immigration kiosks? Surely there's no human value they actually add. What even is the effectiveness of these routine "randomly selected" checks beyond sniffer dogs for cocaine mules.

</comment><comment><USER64> Border agent : "You are shrunk to the height of a nickel and your mass is proportionally reduced so as to maintain your original density. You are then thrown into an empty glass blender. The blades will start moving in 60 seconds. What do you do?"

</comment><comment><USER65> For those who wonder what the optimal algorithm is:

[https://en.wikipedia.org/wiki/Day%E2%80%93Stout%E2%80%93Warr...](https://en.wikipedia.org/wiki/Day%E2%80%93Stout%E2%80%93Warren_algorithm)

<comment><USER66> Yeah. I learned about the Day-Stout-Warren algorithm today, after reading this discussion, and it's beautiful.

</comment></comment><comment><USER67> ... This is odd considering that a disproportionate amount of terrorists are engineers.

<comment><USER68> "so you're an engineer? can you design me a bomb?"... either answer .. "haha, busted"

</comment></comment><comment><USER69> And looks like he didn't give answer: [https://twitter.com/cyberomin/status/835997085916872704](https://twitter.com/cyberomin/status/835997085916872704)

</comment><comment><USER70> I don't understand the problem with this. In Ben Gurion airport in Israel, if you look like a religious Jew you'll most likely get asked questions regarding the Torah portion of that week.

</comment><comment><USER71> In 2004, when I came to US for the first time as a student, I was asked by an immigration officer at JFK to make the XOR gate using OR and AND gates on a paper.

</comment><comment><USER72> Obviously the correct answer is "you type the question how to balance a binary search tree in stack overflow and wait for an answer"

</comment><comment><USER73> I would say I had just got off a flight and I was tired and needed a shower and I was not in the mood to work as I was on vacation.

</comment><comment><USER74> Back when the IRA was detonating bombs with regularity, one completely harmless redhead I know was detained and questioned for hours nearly every time she traveled from the US to London (which was often).

On the one hand, it's definitely a serious problem that needs solving, on the other hand it's neither completely irrational, nor a new problem.

</comment><comment><USER75> If you answer with your hourly rate they let you in?

</comment><comment> ma-nu if I wrote a program to draw a doughnut, will they be able to make out that I just pulled their leg :) ?

</comment><comment><USER76> Could be that they're hiring.

</comment><comment><USER77> Anyone want to take odds on this story being fabricated?

<comment><USER43> It sounds plausible. Guy claims to be working as a programmer, border guard googles "basic programmer interview questions" or similar and asks them to see how the guy reacts. If he is totally lost, the probability that he's lying about his profession increases.

<comment><USER78> Experienced devs and dev managers (me included) fuck up consistently in recruiting - employing people who "interview well" but drag the team down, or overlooking or excluding great potential team members 'cause they don't perform well "programming a whiteboard" \- and now we expect some minimum wage rent-a-bully to accurately assess a "programmer's" skills based on reading wikipedia entries?

I'm _reasonably_ certain my best front end dev probably doesn't even know what a binary search tree is. Same with half of my mobile devs, they probably vaguely recall them from college, but have never used them in 5 or 6 year mobile app dev careers. Should _they_ be denied entry at a border because they can't regurgitate some wikipedia page completely unrelated and mostly irrelevant to their specialisations?

<comment><USER79> CBP agents get good wages and great benefits. They do not make minimum wage.

<comment><USER78> Apologies...

I'd like to issue a correction:

" … and now we expect some reasonably well paid rent-a-bully with great benefits … "

</comment></comment></comment></comment></comment><comment><USER80> inb4 turns out to be complete bullshit

</comment><comment><USER68> "i'm a CS phd", "prove it" .. non-story.

but i think they should have asked him to spend a bit of time writing cool oss code (maybe before his flight)

</comment><comment><USER81> Still waiting on the impeachment proceedings. Any day now.

</comment><comment><USER82> Sounds like grounds for a suit of some kind.

<comment><USER43> Why?

<comment><USER83> Because in the US, frivolous lawsuits are as expected as frightening border control overreaches.</comment></comment></comment>


UK Architects Declare Climate and Biodiversity Emergency - <AUTHOR> https://www.architectsdeclare.com/ <comment><USER2> Is anybody aware of a similar initiative for software developers?</comment>


Please rate my startup: Sociaholic.com - <AUTHOR> http://sociaholic.com/

<comment><USER2> gosuri, I am doing a podcast about various ideas/projects/stuff that HN members are working on.

Interested in being on one of the episodes?

I put up the first episode and am pivoting based on the feedback I got from the HN community: <http://news.ycombinator.com/item?id=1379942>

If you are interested, email me: marc {at} ideatin {dot} com</comment>


Ask HN: Do you think Reddit will sue me for my URL shortener? - <AUTHOR>

I launched a URL shortening service yesterday for reddit geeks like myself:
http://rddd.it
I'm hoping the community embraces it like they did with imgur.com, but observing Reddit's fickle nature, I'm withholding my expectations. My real concern is that Reddit (or Condé Nast) will frown upon the domain similarity and come after me. Not that they'd get much from my asset-less student self, but I'm not looking for trouble.
I tried my best to give rddd.it a different theme, as to not infringe on Reddit's trademark.
So why did I do this?
In a saturated market of URL shorteners, I decided that the best course of action would be to start my own. This sort of logic can only come from somebody who was dropped on his head one too many times as a baby.
And since I'm secretly having an affair with HN, I thought I'd take this opportunity to share it with you guys first. I even gave HN it's own URL for easy access. I hope you like it:
http://rddd.it/hn
The site isn't complete yet, some minor tweaking/polishing still needs to be done. Do wish me luck! <comment><USER2> Excuse me but, how do you expect redditors to use your shortener ? Reddit is about sharing links and, for some people, your service may seem to get in the way between them and content.

<comment><AUTHOR> In the same way they would use bit.ly or tinyurl.

<comment><USER3> Which, if I'm not mistaken, is really frowned upon since bad things could be hiding behind those urls.

<comment><AUTHOR> Well, using URL shorteners to submit links to news aggregators is dumb, I agree. But I use URL shorteners to send friends/family to links and that's really the point of rddd.it.

I'm not saying use rddd.it ON reddit. I'm a little confused where that assumption was made.

<comment><USER4> Why would you do this? I only tolerate it on twitter because of the character limitations. Outside of such a restriction, URL shorteners just get in the way and I find their use quite obnoxious.

<comment><AUTHOR> I find them very useful when I want to send a quick email to somebody with a link. It looks neater (OCD) when you say: "Look what I found on the internet - rddd.it/xxxx"

Also, I link so many people to interesting reddit threads and I'd rather use a shortened link that allows me to email/tweet and yes, even text to somebody.

</comment></comment></comment></comment></comment></comment><comment><USER5> I think that's ambiguous enough that it's probably not infringing on their trademark. That doesn't prevent them from saying it does though.

 _Disclaimer: I'm not a lawyer!_

<comment><USER6> Last year I bought devvit.com/net/org for a site I'm working on (has a new name now) and after doing a lot of polling, enough people thought it was too close to risk. Only anecdotal, but devvit is far less similar than rddd.it, and it only takes a mere thought for a trademark holder to try and have a go..

<comment><AUTHOR> What was the intended purpose of devvit? Social aggregator for developers?</comment></comment></comment>


Hands on with Chromium-Based Edge - <AUTHOR> https://www.thurrott.com/microsoft/203601/hands-on-with-microsofts-new-edge-browser-built-on-chromium <comment><AUTHOR> I'm not a fan of this switch. Edge wasn't my favorite browser, but the web is worse off with it gone.</comment>


Ask HN: Do you have startup ideas to share? - <AUTHOR>

<comment><USER2> A few months ago, I had a BRAVO study done for acid reflux issue. Basically, a capsule with wireless transmitter is attached to esophagus wall. You carry around a bulky wireless receiver with antenna sticking out that records pH readings for 48 hours. You also record when you are lying down, eating and encounter acid reflux episodes on a piece of paper. After 48 hours, you need to take receiver and paper with recorded information back to the clinic so that they can retrieve the data from recorder.

Pain points:

1\. A Bulky receiver that patient need to carry around or keep within couple of feet of your body. (inconvenience).

2\. Patient need to remember to record on a paper when you are lying down and various other information.

3\. Patient need to take the receiver back to clinic.

4\. Clinic need to enter the information listed on paper and download data from receiver and upload to an analysis system.

5\. If receiver is lost, patient is charged $3,000 to $5,000 and clinic need to buy another receiver.

In this age of smartphones with gyros and apps, I thought these pain points can be easily solved by creating a Smartphone App and turning smartphone into receiver from wireless capsule in esophagus. Smartphone can be attached to the chest to detect when a patient lies down. Patient can manually select options in apps for various information that need to be captured. After 48 hours, smartphone automatically sends captured information and data to a central server over internet.

Benefits:

1\. Everyone has smartphone, no need for clinic to provide a bulky receiver that need to be returned to clinic after 48 hours and deal with lost receivers.

2\. Patient doesn't need to go back to clinic just to return receiver and completed forms (Travel time and cost).

3\. Clinic doesn't need to perform data entry and upload information to a system (Expensive medical labor).

Challenges:

1\. Potentially limited market. But I am sure there are a few other similar medical monitoring procedures to which this solution could be extended (small market).

2\. Potential FDA approval and clinical trials (expensive and long time to market). Depending on wireless protocol used between capsule and receiver, may need to redesign the capsule (much larger hurdle for a new startup instead of existing maker to modify).

3\. Bringing together people with medical imaging domain knowledge and mobile OS App developers.

Why I am not pursuing this idea myself?

I thought about pursuing this idea for a short while as I spent couple of years consulting in medical imaging IT. But thought of putting together a qualified team and raising financing stopped me in the tracks. I prefer ideas that I can bootstrap (result of previous scars raising OPM). Most of my medical imaging contacts are on East coast of Canada and I am now on West coast of US so difficult to put together a good team with people I know. No experience with Mobile. Already involved with another idea in a totally different space.

Hopefully, sharing this idea helps someone else.

<comment><USER3> Can you explain to me what the benefit of this study is supposed to be? (I think finding a way to eliminate the study would make more sense.)

<comment><USER2> The study measures the pH over 48 hour period near the entrance of stomach. This helps determine the frequency and strength of acid reflux into esophagus from stomach to assess the severity of gastroesophageal reflux disease (GERD). Typically, capsule is placed in esophagus during the endoscopy.

Based on my readings before the procedure, in my opinion, BRAVO capsule technique is currently the most advanced, accurate,and convenient method compared to previously used methods such as stents/catheter.

More info [http://my.clevelandclinic.org/services/esophageal_ph_test/hi...](http://my.clevelandclinic.org/services/esophageal_ph_test/hic_the_48-hour_bravo_esophageal_ph_test.aspx)

I am not sure what do you mean by eliminating the study. How else are you going to measure the pH to detect acid reflux in esophagus.

<comment><USER3> Thank you for replying. I am trying to understand what benefit there is to studying the frequency and strength. How is that information used? What good does it do you to know?

I am sorry if I sound obtuse. I am genuinely trying to understand this. I have a serious medical condition which causes extreme acidification. Reversing the acidity has reversed a lot of my symptoms. I did not need a fancy test to do that. So I am trying to understand ... why people do this sort of thing, I guess. It makes no sense to me but what I do makes no sense to other people. I don't know how to cross that communication gap right now. I am trying to figure it out.

<comment><USER2> The study helps figure out the severity and seriousness of the problem. There are wide range of treatments, from lifestyle changes to extended treatment with drugs, maintenance drugs and surgery. There is no one treatment that applies to everyone and study helps make a much better informed decision about treatment options. Too much acid reflux can result in lining deterioration, perforation or even cancer in esophagus. Based on study results, doctors can recommend the checkup frequency for esophagus too.

Hope this helps clarify why such studies are performed. I look at such studies as data-driven decision making.

<comment><USER3> Thank you for your patience. I am not sure it helps because I think what I do is data driven but it comes from another direction. I am not sure I understand any better or know what I need to be asking, really.

</comment></comment></comment></comment></comment></comment><comment><USER4> I'm working on a highly distributed cloud framework that implements a Bitcoin operated OpenStack deployment. Basically, you install OpenStack at your house or small data center, launch a small instance with some Open Source code that turns the instance into a virtual appliance controller, and then join a pool of other providers/users. The pool operator provides search functions (like geo) and Bitcoin addresses so others can launch a server by just paying coin to a given address. No usernames, no passwords. Secure, reliable payments for providers to get ROI on excess/hobby capacity. Fast, easy, and dirt cheap.

The initial pool run by myself will provide low trust, low cost, highly transient instances. Chaos Monkey model. Market prices will likely fluctuate based on instance launch demand.

The concept encourages multiple pools running the code, so there will be future opportunity in building up a brand that attracts a given set of infrastructure providers and/or service companies to band together and coop. This could be done to leverage features like geo location, excess capacity requirements, feature sets, or trust levels through accreditation. Think about a handful of Silicon Valley companies sharing infrastructure among themselves and being able to balance the costs through Bitcoin. They could also share cooperatively in the areas of operational competence and security, given they have a good relationship between the companies (which many do).

For me, it's been a long haul working on the concept, especially considering I had to get OpenStack accessible for the average user. Now I'm mostly done with my OpenStack scripts, I'm finally cranking out the code for the controller and have it launching instances by sending in Bitcoin to the instance addresses. It's pretty slick when you use it.

More work has to be done, so it'll be a few months before I do a general release of the first pool. If you or anyone else is interested in this concept, building a new pool brand, or building the services that need to be build around it, hit me up. My email is in my profile.

<comment><USER2> Like your concept. I am very interested in OpenStack myself, more look at it as a building block for private cloud. Sounds like you are trying to create a NOVA pool for virtual instanced from excess available CPU/memory capacity of the participant users. Not sure you really need to have OpenStack to achieve this. What will happen to all the virtual instances running on a NOVA node if a participant user turns off their NOVA node? Also, how are you going to handle shared storage side of OpenStack?

About 12 years ago, I was trying to do something similar pooling together excess storage capacity and tried to position as solution for SMB and consumer market (big mistake). A positioning and solution focusing on Disaster Recovery and enterprise most probably would have been more successful.

<comment><USER4> Here's the whitepaper: [https://github.com/StackMonkey/xovio- pool/blob/master/whitep...](https://github.com/StackMonkey/xovio- pool/blob/master/whitepaper.md).

Each provider (a user) installs their own OpenStack cluster, which includes a nova controller. That cluster runs a virtual appliance who's job is to watch a given Bitcoin address for payments. If it sees a payment it either starts an instance or resets a counter to keep an instance running. Payments by users are taken by the pool operator. The pool then starts micro drips to the instance, in most cases half a cent or so.

The pool I'm going to run will be at stackmonkey.com. If a provider shuts off their controller, something crashes, or there's a rig failure, only a small percentage of the total number of instances will be affected. Payments won't matter because the instance costs are so low, and the drip can be turned off when the appliance stops responding to the pool controller.

OpenStack makes this handy because it handles all the heavy lifting for starting and stoping instances, doing network setup, images, instance sizes, etc.

Initial audience for this low trust pool will be hackers, developers, testers, tor nodes, individuals, etc. Very low cost, highly ephemeral instances.

There's a video here: [https://vimeo.com/86717476](https://vimeo.com/86717476)

<comment><USER2> Thanks for the whitepaper and video. So, at high level the resource provider is accepting Bitcoin instead of dollars to provide a virtual instance to a customer. Does this sums up what you are trying to do?

I have hard time imagining who might be the customer that will be willing to run their virtual instance on low trust ephemeral pool and for what purpose. What are some use cases for your solution? Is there such a low trust ephemeral pool already in existence that accepts dollars? Why would someone use this solution over just launching a VPS or EC2 instance?

</comment></comment></comment></comment><comment><USER5> I have one but I think it pretty crazy and I am not really sure if it can work. I would love if you could find holes in my idea and help me to refine it or tell me if it can work at all.

So I have been thinking that the biggest roadblock to solar adoption is the high upfront cost of installation. If there could be an incentive to make money with solar a lot more people might be willing to "invest" in it.

So my idea is similar to carbon credits. It is a lot easier to measure the amount of energy generated by the solar panels. So the amount of clean energy generated by the households could be sold off the organisations with a bigger carbon footprint. Similar to how advertising networks sell page views to the advertisers.The unit of energy is well established. We could set the price of 1 watt of clean energy to a bidder.

So the question remains why the bidder would want to buy this clean energy? I don't have an answer to that besides that it could be a good marketing tactic. For very large cooperation it could be a chance to gain some good PR (if they really care about that).

The people participating in this program would have to install a chip that would monitor the generation of this clean energy. With networked chips like tessel and spark.io coming out soon it could become pretty easy for people installing solar panels to connect these panels to a local cloud where there could be an automatic transfer of funds to bank accounts of people depending upon the amount of clean energy that they have generated. Now for a give area this energy should be the same for a standard solar panel since the are will receive more or less the same sunlight. With some work standard pricing can be reached for a locality.

<comment><USER6> Great idea. Unfortunately, you just described New Jersey's SREC system ([http://en.wikipedia.org/wiki/Solar_renewable_energy_certific...](http://en.wikipedia.org/wiki/Solar_renewable_energy_certificate))

<comment><USER5> Ah I see. Well it is encouraging in that it is not totally crazy! Any way I have never heard of such a system in India, where I live. So maybe if I or someone else can make this work there is a lot of money to be made here.

</comment></comment></comment><comment><USER7> I read about this yesterday [http://www.google.com/atap/projecttango/](http://www.google.com/atap/projecttango/)

Last September I posted this on my blog: Imagine sitting in an airport, for example, and being able to join a First Person Shooter Game where all of your opponents are in the same airport – they could be other travellers, airport staff, pilots – anyone. But here is the main feature of this game: The virtual location you see in the game IS the airport and it is created using all relevant data (floorplans of the airport, wayfinding apps, and apps on your phone that are able to map rooms, measure direction, etc.). The game environment in which you travel seeking your opponents is generated using data being generated by their devices as well – sort of a micro crowd sourced battlefield with some help from publicly available data.

Where do I find a Tango Developer to work with to create this Massively Multiplayer Mobile game?

</comment><comment><USER8> Just one idea. Could anyone take these two stories from today:

Telecom firms mine for gold in big data despite privacy concerns [https://news.ycombinator.com/item?id=7286080](https://news.ycombinator.com/item?id=7286080)

Cryptocontracts Will Turn Contract Law Into a Programming Language [https://news.ycombinator.com/item?id=7287155](https://news.ycombinator.com/item?id=7287155)

... and create a bitcoin like wallet/api of a consumer's permission to trade data.

A Twitter User Is Worth $110; Facebook's $98; LinkedIn's $93. [http://www.forbes.com/sites/georgeanders/2013/11/07/a-twitte...](http://www.forbes.com/sites/georgeanders/2013/11/07/a-twitter- user-is-worth-110-facebooks-98-linkedins-93/)

<comment><USER5> The idea of bitcoin like wallet api sounds good. But can you explain the correlation with the stories that you linked?

<comment><USER8> The 'wallet' would be a verification/allowance mechanism for user-stored private usage data. Basically the user becomes the data broker. And an API/(exchange) could charge a fee on the other side of the business.

</comment></comment></comment><comment><USER9> Over time, I have come to realize that what usually constitutes a startup idea is very hard to "share". Sure, you can express it in words, but an idea in your head is usually far more subtle than the words you can string together to convey it.

Edit: grammar

<comment><USER10> That's why people make such a big deal about an elevator pitch.

If you can't express it as an elevator pitch, it's probably not a fully formed idea.

</comment></comment><comment><USER11> Couple ideas: time sharing marketplace/factory/warehouse. 3d printer, other maker tools, cnc, robots to move stuff around but virtual. delivering take'n'bake pizza.(costs are pretty low for a large delivery area). doorhangermarketplace.com - taskrabbit but focused on doorhangers.

</comment><comment><USER12> You can read the article of Paul Grahm in order to find some ideas by yourself: [http://paulgraham.com/startupideas.html](http://paulgraham.com/startupideas.html)

</comment><comment><USER13> We're publishing out ideas that can eventually be turned into a startup here: [http://www.blazingx.com](http://www.blazingx.com)

</comment><comment><USER14> "get me stuff from the store" among neighbors, friends and work team mates. "task rabbit" among social network with favors exchange

<comment><USER11> are you the only one working on it?

</comment></comment><comment><USER15> A group voting system that uses text messaging that isn't frightfully expensive for a small (~50-100) set of users.

</comment><comment><USER16> Recently i thought about making to-do list that don't suck :) Interested to know more about it?

<comment><USER17> While a noble pursuit, I think there are dozens of start-ups (completely free/open apps as well as for-profit businesses) trying to do exactly this. I think Trello counts as a big one.

<comment><USER16> Trello is great, i used that for few times. But it turned that i spent much of my time to organise things than make it done.

</comment></comment></comment><comment><USER18> Amazon for latin america</comment>


Web 2.0 Is Reminiscent Of Marx - <AUTHOR> http://methainternet.wordpress.com/2008/01/23/web-20-is-reminiscent-of-marx/ <comment><USER2> I've had a gutful of this reductionist tripe.

Yes, Web 2.0 allows uneducated fools to become movie directors, artists, musicians, writers, etc but let me tell you NOBODY is mistaking this stuff for high art or culture. Quality will still rise to the top, and people will still be willing to pay for quality. If the amateur is as good as the pro, then great, Web 2.0 has uncovered a new talent.

Put another way, the people using Web 2.0 technologies to create rubbish were NOT going to be reading Plato, Kafka and Nietszche had Web 2.0 not existed. These are two mutually exclusive sets. What Web 2.0 has done is given these amatuers a voice that will likely not even be heard by most. The real artists will still create great works irrespective of the noise, and they will continue to be recognised because people aren't stupid and can spot quality when they see it.

No one is suggesting that the regulat joe and the pro are equal in their skills to produce good work.

</comment><comment><USER3> This is better:

[http://www.comedycentral.com/motherload/index.jhtml?ml_video...](http://www.comedycentral.com/motherload/index.jhtml?ml_video=91639)

</comment><comment><USER4> Looks like a plagiarized article.

</comment><comment><USER5> What an ugly font.</comment>


Lessons Learned in the Startup World - <AUTHOR> http://www.bigstartups.com/matt/blog/51/Lessons-Learned-in-the-Startup-World <comment><USER2> Interesting, I'm going to try to make it to the one in Silicon Valley in November.</comment>


Ask HN: How do you manage to stay social? - <AUTHOR> Recently, I have noticed that I cannot remain social even though I want to. Most of my friends spend their time watching TV series, or in some other solitary activity. Thinking about it, it is like the present day world promotes if not help people stay solitary. Some articles suggested alarming consequences :https://www.marketwatch.com/story/america-has-a-big-loneliness-problem-2018-05-02 . But nobody even wants to talk about it. <comment><USER2> Whenever I get lunch with a group of coworkers, this is what everyone talks about and it's absolutely exhausting. One issue is that I'm 24, craving new, novel social interaction, while most of them are in their late 20s or in their 30s, settled down, and fully in the Netflix trap.

The only benefit to seeing this is that it makes me want to never become like it.

The most genuine conversations I have are with actual friends: People I'd hit up on the weekend to walk in a park, drive somewhere for a hike, or grab dinner with. I also have a circle of friends that I go out with, which I understand not everyone is interested in.

Get out of the house as much as possible, and embrace the relationships that you genuinely enjoy. Small talk about Netflix will still be a thing with people you're not close with, but you can help steer the conversation to be more interesting if you know the things that you do/don't want to talk about.

</comment><comment><USER3> One new tactic: I walk our dog at approximately 8:30 every night. I used to be mildly annoyed by this, treating it as a chore to do before I could get back home slink into my chair, and enjoy the rest of the evening.

I recently purchased a pair of Airpods, and having high-quality USB headphones on me at all times has changed the way I think about phone calls. I have a list of about 12 people (parents, brother, long-time friends that live in different cities now), that I'll call randomly during that walk. If they pick up, great. If not, I just go to the next one. At this point, most of them know that I may just call them while walking the dog, and that they're under no obligation to pick up.

The end result is 2-3 times a week, I spend 30+ minutes nurturing a relationship with someone who is important to me, but who I don't see face-to- face often. And my dog gets a longer walk because of it.

</comment><comment><USER4> Find new (additional) friends who like to do sociable things. Meetup is a good place to find people like this. If you like a place which has its own subreddit (eg r/losangeles etc), you may find people who regularly do sociable stuff.

Turn solitary activities into group activities. If your friends are into a particular TV show, throw a get together/party where you hang out and watch and discuss that show.

Invite your friends who are into solitary activities to come along to some of your new found more group oriented activities.

</comment><comment><USER5> I've done a poor job of it lately, but when I wasn't travelling as much and was single, I relied on Meetup and social mags to find things to do and people to meet. This has gotten harder to rely on after moving to Dallas. A lot of the events we've found on both sources are not to our liking, and a lot of the stuff that is (live music, mostly) are in bars, where people tend to go in groups and not interact much with each other. Most of the dope shit is on Facebook, but you need to know the right groups to be made aware of said dope shit.

Regardless, understand that most of the people you'll meet will be fleeting; finding and building deep relationships with other people is hard work!

That said, what you're feeling is normal. A lot of people's social circles extend to their workmates, old friends from school and folks they talk to a lot from church. (Churches do a really good job of bringing people together that otherwise wouldn't.) People tend to be very cliquey by default because it's comfortable. Making friends as an adult is a lot more difficult than as a kid.

Also, try to meet people outside of your industry. It will help you gain perspective and might even make you better at your own work. This is especially important in tech. Many of us make a lot of money and work in what most would consider paradise (if you get to wear whatever you want, have a ping pong table and beer/wine on tap, congrats, you qualify), but it's very easy to forget that and think that everything is terrible because there are wealthier folks out there. Venturing outside of the tech bubble helps with that.

</comment><comment><USER6> Put yourself in a situation where you see the same people regularly. I show up to a weekly tennis clinic and hold season tickets to one of the local sports teams, for example.

</comment><comment><USER7> Joiners know the answer, the rest of us have to grudgingly recognize that joining an organization (that isn't insane) which is doing something meaningful together is one of the few ways out of the house that doesn't lead to watching TV in bars.

Luckily, these are a lot easier to find (and scrutinize first) than they used to be.

</comment><comment><USER8> Sometimes it's fun to rent an Airbnb somewhere beautiful and invite friends to come hang and co work, hike, chill in the hot tub, invite their friends and hang out and talk as a group.

</comment><comment><USER9> We bought a house with a porch and eat dinner out there whenever possible. We probably know about half of the people on our block.

Also, when you have kids it's easier to bond over common ground with the kids. Otherwise I found myself in awkward conversations yakking about work.</comment>


Just launched Vloggo! Feedback? - <AUTHOR>

Hey guys,
I'm looking for feedback on Vloggo, the startup I just launched. Vloggo allows you to share your life through 60 second videos.
URL: http://vloggo.com
iPhone App: http://bit.ly/jq6Zft
Let me know what you think! Many thanks! <comment><USER2> I really liked the ideia, but I missed the option of logging in with facebook/google/twitter accounts. The possibility to login on your vlog with your facebook credentials will be great to your site. I believe users do not want to create more and more logins with different sites (unless it is totally necessary.)

</comment><comment><USER3> Cool idea!

I notice you're using Amazon SES for your transactional mail. How is your experience with them so far?

<comment><AUTHOR> No complaints for SES thus far :)

</comment></comment><comment><USER4> Great job on the layout. I agree that the scrolling feed on the front page lets me know immediately that it's an active community and actually makes me want to join up.

</comment><comment><USER5> I love the green! However, white text is a bit difficult to read on the light green background in your footer nav bar. I'd recommend going the darker green there.

<comment><AUTHOR> Thanks! I figured blue social networks are way over used. I'll have to check out the color scheme on a few other computers because the text is very easy to read on my screens. Thanks for the suggestion!

</comment></comment><comment><USER6> Looks pretty cool. I like the way you scroll videos at the right bottom of screen, that makes me want to use it. Good luck.

</comment><comment><USER7> This is actually pretty cool. When will the android app be ready?

<comment><AUTHOR> It'll probably be a couple of months before Vloggo gets to Android. I'm looking for a co-founder and funding as my first priorities.

</comment></comment><comment><USER8> Interesting idea. Any plans for an optional HTML5 player?

<comment><AUTHOR> HTML5 seems to be the way of the future, so I'm definitely exploring that as an option.

</comment></comment><comment><USER9> Very cool! Good luck with it.</comment>


Need to count things quickly on App Engine? Try "Fast counters" - <AUTHOR> https://github.com/pocketgems/gae-fastcounters

<comment><AUTHOR> From the author of gaesessions <https://github.com/dound/gae-sessions></comment>


Azure Is Down - <AUTHOR> https://status.azure.com/ <comment><USER3> "Multiple Services - <USER2>

Summary of Impact: Between 00:57 and 03:40 UTC on 20 Nov 2019, customers and services utilizing Azure Front Door (AFD) services were impacted by an infrastructure service failure. This resulted in a loss of connectivity to multiple services reliant on AFD.

Azure Front Door services provide Edge caching and network entry point services to the Microsoft global network. This issue impacted a large percentage of Microsoft Services, though not all services were impacted. Many impacted services were able to initiate fail over from the AFD platform, providing immediate mitigation to their customers.

Preliminary Root Cause: During a recent periodic deployment, initial safety checks did not detect the issue and prevent the roll out. Monitoring detected once service failure was experienced and alerted engineers.

Mitigation: Engineers immediately initiated deployment rollback procedures to correct the underlying Azure Front Door issue. This was completed at 02:40 UTC on 20 Nov 2019, at which point Impacted services began recovering.

A detailed root cause analysis will be published within 72 hours." [https://status.azure.com/en-us/status/history](https://status.azure.com/en- us/status/history)

</comment><comment><USER4> [https://twitter.com/AzureSupport/status/1196966470619500546](https://twitter.com/AzureSupport/status/1196966470619500546)

Funny to me that the cloud companies haven't put their status page on not- their-cloud.

</comment><comment><USER5> Azure is so down. lol. Nothing loads for us. Even the status page.

</comment><comment><USER6> Yeah, seems to be the case. Office 365 blipped and shortly after came back up. Other services like OneDrive are unavailable, as is the Azure portal.

</comment><comment><USER7> Our website stayed up hooray! hcaptcha.com

</comment><comment><USER8> I can't connect to tfs :-(</comment>


Ask HN: Docker, where do I start? What's the point? - <AUTHOR> I miss the container train. I'm not pro or against Docker but I'd like to understand the buzz around it. <comment><USER2> Here's my understanding of what docker is. Deploying software consistently can be difficult. Your code always depend on some dynamically loaded libraries on the operating system. Even if you statically link your software, you still have dependency on the operating system that you developed against.

The classic way to mitigate this is to use virtual machines. But virtual machines are big, and bulky. Docker combines several tricks that are only available on Linux so that a single process thinks it's running on a virtual machine. Your binaries are shipped in a diff format against a particular OS version. If your binary depends on some third party packages, you can apt-get install them and ship these together with your binary image.

Another example of docker use case is in software development. Your OS might be running a version of python, while you are developing against a different version. Your current project may have dependencies on particular third party libraries. By installing all these into the docker image, it will be a lot easier for you if you need to come back to make some changes in the future.

Furthermore, docker being free, there are now a range of solutions that have popped up around managing the lifetime of docker containers for scaling etc. These then to be interesting to people who are running internet-scale software, but less so if you are writing some enterprise applications.

</comment><comment><USER3> The best abstraction I have heard is that in terms of developer experience, containers/Docker are a way of versioning execution environments that's analogous to the way VCS/git versons source control.

When it comes to deploying an execution environment, containers/Docker have an advantage over configuration tools (such as Puppet/Chef) by being atomic. Containers cannot be partially deployed and therefore put a machine in an arbitrary state. Orchestration layer tools (Kubernetes/Swarm) handle deployment at the level of multiple containers.

Generally, containers reduce dependency conflicts between applications. On the other hand, there's another level of indirection that may not be appropriate to a particular business model or make a particular developer's life easier. That doesn't mean they're not worth checking out.

</comment><comment><USER4> I recommend this article, it will cure you from Docker.

[https://thehftguy.wordpress.com/2016/11/01/docker-in- product...](https://thehftguy.wordpress.com/2016/11/01/docker-in-production- an-history-of-failure/)</comment>


Google Reader's awesome loading animation - <AUTHOR> http://www.youtube.com/watch?v=-1lKz1HJQ6Q

<comment><USER2> Took me a few plays of the video to figure out what was going on. I finally noticed it in the bottom right corner. That said, it is pretty and unique. Haven't seen a loader like that before.

<comment><AUTHOR> Yeah, sorry about the quality of the video. I should've had an indicator for the loader.

</comment></comment><comment><USER3> Yeah, pretty easter egg. But basically, a waste of CPU cycles and pixel flips. As well as being visually distracting.

<comment><AUTHOR> I don't know if it takes up that many CPU cycles. Agree that it is in a useless spot. Could've used that extra space for more text (post content).</comment></comment>


Stop Marketing, Start Selling - <AUTHOR> http://www.georgesaines.com/?p=709

<comment><USER2> This is incorrect. Marketing isn't just "selling without accountability" - marketing is an omnipresent (to varying degrees) feature of business, from product development to launch and popularization.</comment>


Scrapy Tips from the Pros - <AUTHOR> http://blog.scrapinghub.com/2016/01/19/scrapy-tips-from-the-pros-part-1/ <comment><USER2> I love ScrapingHub (and use them) but these tips go completely against my own experience.

Whenever I've tried to extract data like that inside Spiders I would invariably (and 50,000 URLs later) come to the realization that my .parse() ing code did not cover some weird edge case on the scraped resource and that all data extracted was now basically untrustworthy and worthless. How to re- run all that with more robust logic? Re-start from URL #1

The only solution I've found is to completely de-couple scraping from parsing. parse() captures the url, the response body, request and response headers and then runs with the loot.

Once you've secured it though, these libraries look great.

PS: If you haven't used ScrapingHub you definitely should give it a try, they let you use their awesome & finely-tuned infrastructure completely for free. One of my first spiders ran for 180,000 pages and 50,000 items extracted for $0.

<comment><USER3> > code did not cover some weird edge case on the scraped resource and that all > data extracted was now basically untrustworthy and worthless.

Your data should not be worthless just because you dont catch some edge cases early. Sure there are always some edge cases but best way to handle them is to have proper validation logic in scrapy pipelines - if something is missing some required fields for example or you get some invalid values (e.g. prices as sequence of characters without digits) you should detect that immediately and not after 50k urls. Rule of thumb is: "never trust data from internet" and always validate it carefully.

If you have validation and encounter edge cases you will be sure that they are actual weird outliers that you can either choose to ignore or somehow try to force into your model of content.

<comment><USER2> Hmm, I'll have to investigate that, any tips for libraries to use for validation that tie well into scrapy?

What do you do if you discover that your parsing logic needs to be changed after you've scraped a few thousand items? Re-run your spiders on the URLs that raised errors?

<comment><USER4> Spider Contracts can help you: [http://doc.scrapy.org/en/latest/topics/contracts.html](http://doc.scrapy.org/en/latest/topics/contracts.html)

</comment></comment></comment></comment><comment><USER5> Extremely well designed framework. It can cover more than 90% of use cases in my opinion. I'm currently working on a project written in Scala that requires a lot of scraping, and I feel really guilty that I'm not using Scrapy :(

<comment><USER6> I think you can still combine the two. For example Scrapy can be behind service/server to which you'd send request (with same args as if you were running it as a script + callback url) and after items get collected Scrapy can call your callback url sending all items in json format to your Scala app. Or if you want to avoid memory issues for sure, you can send each item to Scala app as it gets collected. Basically, idea is to wrap Scrapy spiders with web service features - then you can use it in combination with any other technology. Or you can use Scrapy Cloud to run your spiders at [http://scrapinghub.com/](http://scrapinghub.com/).

<comment><USER7> There is ScrapyRT: [http://blog.scrapinghub.com/2015/01/22/introducing- scrapyrt-...](http://blog.scrapinghub.com/2015/01/22/introducing-scrapyrt-an- api-for-scrapy-spiders/)

In the project I work on we do have the usual periodic crawls and use ScrapyRT to let the frontend trigger realtime scrapes of specific items, all of this using the same spider code.

Edit: Worth nothing that we trigger the realtime scrapes via AMQP.

</comment></comment></comment><comment><USER8> I've played with Scrapy before to make a proof of concept and I was pleased with how easy it was (haven't yet had to use it for anything else yet).

That being said, had no idea how sophisticated it could get. This is super impressive, especially the JavaScript rendering.

</comment><comment><USER9> Yay! Another article on scrapy. I'm just getting started and my first goal is to scrape a tedious web-based management console that I can't get API access to, and automate some tasks.

Very glad to learn about this site Scraping Hub. Keep the war stories coming. It's technologies like these that brighten up our otherwise drab tech careers and help some of us make it through the day.

</comment><comment><USER10> I have been using this framework for more than three years and seen how it has evolved and made scrapping so easy. The portia project is also awesome. I have customised scrapy for almost all the cases like having single spiders for multiple sites and providing rules using JSON. I think it is highly scalable with bit of tweak and scrapy allows you to do very easily.

</comment><comment><USER4> Hey, author here! Feel free to ask any questions you have.

<comment><USER11> Here is a first one : What are the best ways to detect changes in html sources with scrapy, thus giving missing data in automatic systems that need to be fed ?

<comment><USER4> Hey, not sure if I understood what you mean. Did you mean:

1) detect pages that had changed since the last crawl, to avoid recrawling pages that hadn't changed? 2) detect pages that have changed their structure, breaking down the Spider that crawl it.

<comment><USER4> 1) detect pages that had changed since the last crawl, to avoid recrawling pages that hadn't changed?

You could use the deltafetch[1] middleware. It ignores requests to pages with items extracted in previous crawls.

2) detect pages that have changed their structure, breaking down the Spider that crawl it.

This is a tough one, since most of the spiders are heavily based on the HTML structure. You could use Spidermon [2] to monitor your spiders. It's available as an addon in the Scrapy Cloud platform [3], and there are plans to open source it in the near future. Also, dealing automatically with pages that change their structure is in the roadmap for Portia [4].

[1] [https://github.com/scrapinghub/scrapylib/blob/master/scrapyl...](https://github.com/scrapinghub/scrapylib/blob/master/scrapylib/deltafetch.py)

[2] [http://doc.scrapinghub.com/addons.html?highlight=monitoring#...](http://doc.scrapinghub.com/addons.html?highlight=monitoring#monitoring)

[3] [http://scrapinghub.com/scrapy-cloud/](http://scrapinghub.com/scrapy- cloud/)

[4] [http://scrapinghub.com/portia/](http://scrapinghub.com/portia/)

</comment></comment></comment></comment><comment><USER12> I tried using PHP to scrape 50,000 webpages for a couple of fields, got it done in 4 hours, with scrapy it took 12 minutes. Been using it ever since.

</comment><comment><USER13> Wow, never heard of Scrapy! Looks like I've reinvented it in Clojure: [https://github.com/nathell/skyscraper/](https://github.com/nathell/skyscraper/)

<comment><USER5> That looks pretty cool. I was planning on writing something similar in Scala, but I'm not sure if I have enough experience with the language to get it done.

<comment><USER14> If you're lazy (and if you're into FP you must be hehehe), just use that Clojure library. Calling Clojure code from Java is easy, and I'm sure it's not much harder from Scala.

</comment></comment></comment><comment><USER15> Man, I feel old. Does anyone remember learning web scraping from one section of Fravia's site? Ever try to move forward from that to write a fully fledged search engine? These memories are from 15 years ago... quite amusing how much hasn't changed. In hindsight it was probably easier back then due to the lack of JS-reliant pages, less awareness of automation and less scraper detection algorithms.

<comment><USER16> What is the state of web scraping? I've got a few thousand URLs that is like to build a search engine around:

[https://github.com/melling/SwiftResources/blob/master/swift_...](https://github.com/melling/SwiftResources/blob/master/swift_urls.tsv)

I'm using Swift to preprocess the data and I host my server on AppEngine using Go:

[http://www.h4labs.com/dev/ios/swift.html](http://www.h4labs.com/dev/ios/swift.html)

Since my engine is stitched together, I could use Python or Perl to scrape the sites and extract the words, ignoring JavaScript and css.

</comment></comment><comment><USER17> What I need is an API scraper, scrapy seems to be mostly for HTML. I know how to look at network requests in Chrome dev tools and JS function to understand the shape of REST API, so I need something to plan the exploration of the arguments space. For example if you want to scrap airbnb, you look at their API, find there is a REST call with a lat, long box, I need something to automatically explore an area, if the api only give the 50 first results and you hit this number of calls it should schedule 4 calls with half the size boxes and so on. If the request has cursors, you should be able to indicate to the scraper how to follow it. I don't know what is the best tool for that.

<comment><AUTHOR> It's not entirely clear what you're up to, but FYI Scrapy works fine when scraping JSON data:

[http://stackoverflow.com/a/18172776/417194](http://stackoverflow.com/a/18172776/417194)

</comment></comment><comment><USER18> I like this article but for its discussion of these libraries. On another note...

Am I the only one who dislikes Scrapy? I think it's basically the iOS of scraping tools: It's incredibly easy to setup and use, and then as soon as you need to do something even minutely non-standard it reveals itself to be frustratingly inflexible.

<comment><AUTHOR> Scrapy is about as flexible and extensible as you can get... Care to elaborate on "frustratingly inflexible"?

<comment><USER18> I do a lot of scraping specific pages and often have to auth, form-fill, refresh, recurse, use a custom SSL/TLS adapter, etc., in order to get what I'm after. I'm sure Scrapy would be great if I just had a giant queue of GET requests. Also, don't get me started on the Reactor.

</comment></comment></comment><comment><USER19> We use Scrapy for a few projects and it is really really good. They have a commercial-side to them, which is fine, but for anyone doing crawling/scraping I'd strongly recommend it. Good article also

<comment><USER20> Same, it's really nicely put together, lots of sensible defaults and it's easy to add your own bit of awkward logic when necessary.

</comment></comment><comment><USER21> Great to see Scrapy getting some love. It's really well done and it scales well (used it to scape ~2m job posts from ATS & government job banks in 2-3 hours).

<comment><USER22> Using it for the same use case, scraping a whole lot of job posts. Scrapy is love, scrapy is life.

</comment></comment><comment><USER23> Here's slides for a talk I gave about an interesting approach to scraping in Clojure [1]. This framework works really well when you have hierarchical data thats a few pages deep. Another highlight is the decoupling of parsing and downloading pages.

[1] - [http://slides.com/escherize/simple-structural-scraping- with-...](http://slides.com/escherize/simple-structural-scraping-with- skyscraper)

</comment><comment><USER24> I love scraping web and produce structured data from web pages. The only downside of using XPath or similar extracting approach is necessity of constant maintenance. If I have enough knowledge about machine learning, I would like to write a framework that analysis similar pages and finds structure of data without giving which parts of page should be extracted.

<comment><USER4> Maybe you should give a try at Portia ([http://scrapinghub.com/portia/](http://scrapinghub.com/portia/)). It does exactly what you mean.

You may also be interested in this library: [https://github.com/scrapy/scrapely](https://github.com/scrapy/scrapely)

</comment></comment><comment><USER19> Quick question. Could you use Scrapy to specific individual pages from thousands (or millions) of sites, or would you be better off using a search engine crawler like Nutch for this? I want to crawl the first page of a number of specific sites and was looking into the technologies for this.

<comment><USER25> Yes, if you use subclass CrawlSpider, then you will be able to set rules on your crawls [1]

[1] -[http://scrapy.readthedocs.org/en/latest/topics/spiders.html?...](http://scrapy.readthedocs.org/en/latest/topics/spiders.html?highlight=crawlspider#crawlspider- example)

</comment></comment><comment><USER26> Confession : I am guilty of using regex superpowers to extract data from urls. Will check out w3lib soon!

<comment><USER27> I've had to write some gnarly XPath expressions to extract data with Scrapy.

> //b[contains(.,'City')]/following-sibling::a[not(preceding- > sibling::b[contains(.,'Country')])]/text()

</comment></comment><comment><USER28> Anybody interested in browser automation, can try [http://api.phantomjscloud.com](http://api.phantomjscloud.com)

disclaimer, I wrote it. No crawler yet, though that's next after a new website.

</comment><comment><USER29> That reminds me, I was going to write a scraper to extract my HN comments.

<comment><USER30> HN API? [https://github.com/HackerNews/API](https://github.com/HackerNews/API) Haven't used it yet, so no comment on how well it works/how far back it goes)

</comment></comment><comment><USER31> What kinds of careers often deal with web scraping and doing these sorts of tasks? I am really interested in the field and some of you seem to be real experts in this field.

<comment><USER4> All sorts of careers, for example:

\- developers who want to develop some data-based product (a travel agency website, who finds the best deals from airline companies);

\- lawyers can use it to structure the data from Judgments and Laws (so that they are able to query the data for things like: which judges have interpreted this law in their judgments) (more on this: [http://blog.scrapinghub.com/2016/01/13/vizlegal-rise-of- mach...](http://blog.scrapinghub.com/2016/01/13/vizlegal-rise-of-machine- readable-laws-and-court-judgments/))

\- (data-)journalists who work on investigative data-based articles (they use it to gather the data to build visualizations, infographics, and also to support their arguments).

\- real state agencies can use it to grab the prices of their competitors, or to get a map of what people are selling, what are the areas where there is more demand.

\- large companies that want to track their online reputation can scrape forums, blogs, etc, for further analysis.

\- online retailers that want to keep their prices balanced with their competitors can scrape the competitors websites collecting prices from them.

More on Quora: [https://www.quora.com/What-are-examples-of-how-real- business...](https://www.quora.com/What-are-examples-of-how-real-businesses- use-web-scraping)

</comment></comment><comment><USER32> Shame Python 3 isn't there for Scrapy yet - although checking the repo looks like someone is actively (last few hours) porting it. Good to see.

</comment><comment><USER33> Rubyist here...how does Scrapy compare to Nokogiri?

<comment><AUTHOR> Much like apples to oranges.

Nokogiri is a tag soup parser. Scrapy is a web scraping framework.

In addition to tag soup parsing, Scrapy handles a slew of things such as text encoding problems, retrying urls that fail owing to network problems (if you wish), dispatching requests across multiple spiders with a shared crawl frontier (see Frontera), shared code between similar spiders using middlewares and pipelines, and what have you.

There's a Lisp joke that goes something like every sufficiently complex piece of software in C is a slow, buggy, poorly implemented version of Lisp. Very much the same could be said about Scrapy and web scraping projects. :-)

<comment><USER33> Makes sense, thanks for the clarification. To your knowledge, is there anything comparable to Scrapy in Ruby land?

<comment><AUTHOR> Not that we're aware of. Most rubyists use request and a tag soup parser, without benefiting from any type of parallelization that you get from Scrapy.</comment></comment></comment></comment>


Grafana raises $50M in Series B funding - <AUTHOR> https://news.crunchbase.com/news/grafana-labs-brings-in-50m-series-b-for-open-source-developer-platform/ <comment><USER2> Congrats to the team at Grafana! I've been working here for a little over a year and I'm really excited to see how far we can go with this.

</comment><comment><USER3> Here's a link to the official announcement: [https://grafana.com/about/press/2020-08-17-series-b- announce...](https://grafana.com/about/press/2020-08-17-series-b- announcement/)</comment>


Ask HN: Hacker News missing from Google results? - <AUTHOR>

When I search for "hacker news" on Google, this website is not in any of the results. Do others see the same thing? What is going on? <comment><USER2> This has been discussed before -- see [https://news.ycombinator.com/item?id=5955043](https://news.ycombinator.com/item?id=5955043) (for the discussion) and [https://news.ycombinator.com/item?id=5955374](https://news.ycombinator.com/item?id=5955374) (for Matt Cutts' comment).

<comment><AUTHOR> Thanks! I don't why they don't limit Google, Bing crawlers to the homepage via robots.txt. Then crawling should not overload an already busy website. Honestly, it seems like an overreaction to me.

<comment><USER2> For what it's worth, pg said that "crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory" (at [https://news.ycombinator.com/item?id=3277661](https://news.ycombinator.com/item?id=3277661)).

</comment></comment></comment><comment><USER3> For me it comes up as either the 4th or 6th result depending on whether you put it in quotes or not.

Interestingly the HN article it links to is the one the other day discussing how Google paid Adblock Plus to get its ads white listed.

<comment><AUTHOR> Interesting!

</comment></comment><comment><USER4> If HN were a popular result, the quality of this community would rapidly decline. Eternal September etc.

<comment><AUTHOR> How? It is pretty clear that this website is only interesting to "hackers", and there are already a lot of people who visit the site. I think what makes HN different than Reddit or any other similar website is the rules on posting and the subsequent consequences for not following those rules. That is what encourages interesting, useful comments and prevents stupid comments like "you suck, kill yourself". As far as I can tell, the Google ranking has little to do with it.

<comment><USER4> It's almost impossible to avoid a decline in quality as the quantity of users increases. Reddit being a prime example.

Every now and again there are threads posted here discussing the apparent 'HN decline' as the userbase grows.

<comment><AUTHOR> To me, those threads sound like old people complaining rather than an actual problem. "Back in my day..." You still did not answer my question: how?</comment></comment></comment></comment>