"Equation Group" ran the most advanced hacking operation ever uncovered - <AUTHOR> http://arstechnica.com/security/2015/02/how-omnipotent-hackers-tied-to-the-nsa-hid-for-14-years-and-were-found-at-last/ <comment><USER2> This is exactly what the NSA should be doing. Everyone (rightfully IMO) complains about overly broad data collection happening within the USA, but here (as with stuxnet) you have the exact opposite, a targeted foreign activity conducted with care and targeting. I know it's not for everyone (not least because not everyone is in the states, huh), and it could be considered a bad precedent, but it's not like Iran asked our permission before they launched their nuclear program, or other states are actually waiting on the US's example to have their own intelligence services do their jobs. If you hold that something like 9/11 should be prevented, and that (actual) WMD programs should be stalled, then it follows that this is a fine way to go about it.

<comment><USER3> I am from germany and I hope most of americans don't have this point of view.

1\. give up freedom for some stupid terrorist attacks? why should we? We've lost the keystone of freedom when we do this and they have already won.

2\. Do you really think these extensive intelligences do stop terror? (maybe they do sometimes, but terrorists will find new ways)

3\. IS and others are the result of the Iraq war which was a offensive war from the US justified by a lie (Sadam had no weapon of mass destruction). Do you still think that America is the world police? The reputation of the US has gone rapidly down in the last decade. For the most here in germany the US are not the good ones any more. Other countries have their own way of life. America has to accept that.

<comment><USER4> > "give up freedom for some stupid terrorist attacks?"

What level of freedom is lost here? We're NOT talking about mass surveillance generally but the targeted access described by the OP. It's the difference between the police parking a van outside a suspected gang hideout (hopefully after getting a warrant), and parking a van outside every home in America. Most Americans are OK with the former and NOT OK with the latter. And, IMHO, that's a valid trade-off to make in a democratic society.

You might object to a particular surveillance target (Merkel), which is understandable. But would you object to spying on Putin? I suspect even Germany (especially Germany) would be okay with the NSA conducting targeted surveillance of senior members of the Russian military with ties to Eastern Ukraine.

In an ideal world, there would be no surveillance or any surrender of liberty in any scenario. But the world is not ideal and compromises are made. Germany is not immune to this. Case in point: Hate speech is illegal in Germany, which is understandable given German history, but a violation of free speech rights in the U.S. The challenge is not to reject all infringements of freedom wholesale but to identify where lines can be drawn between what can be tolerated and what must not.

<comment><USER5> They excluded a few countries and two usernames. Medium-level infection reached 18 countries.

I think it's foolish to defend alleged NSA operations when the NSA won't even acknowledge such operations.

If the government wants to have a debate about the rules of digital warfare (or a particular war), then let's. If they don't, then why should we defend their secret tactics? We have no idea what the motives and objectives are of these operations. And we have little knowledge about how they've affected people, innocent or otherwise.

Your casual analogy to parking vans outside homes seems reasonable. Until you consider what that actually means in our real life. For instance, vans outside homes has been a large part of the war on drugs which has imprisoned a staggering number of black youth. What do you think is the analogous fallout of this malware? Drone strikes? Defend those.

<comment><USER6> The notation that a debate on the rules of digital warfare will do anything is questionable.

Arms control limitations (SALT,START), the hague convention, etc, work because there are means of verifying countries adhere to what they agree on (and ostensibly punishing those who don't).

Given the difficulty of attributing cyber attacks (e.g. Sony), much less cyber espionage, there's little reason to think this is possible in this case. And that's just for direct action.

If we're talking about tactics and capabilities, it's impossible. How are you going to make sure there aren't 30 people somewhere writing malware for a government? You can't, at least absent far more invasive spying or some kind of DRM that makes writing malware illegal.

<comment><USER5> My point was mostly that I will not defend secret war/aggression/cybercrime especially in light of recent history. I'm surprised so many people here defend this malware claiming that it's (1) justified and (2) targeted. Since, we have no idea what it's for and it's heavily infected 18 countries. I'd guess they also support targeted torture and rectal hydration too. Just as long as it's not citizens. Except for just the really bad citizens.

<comment><USER7> No one in this thread has said they support torture. This malware has nothing g to do with torture. What was the point of that fake argument? Supporting targeted malware is not supporting torture.

<comment><USER5> There was some sarcasm in my comment above and I didn't fully detail what I meant.

The point was that something doesn't become ok just because it's targeted at non-citizens or the targets are more limited than simply everyone. (Unless other context can justify it... but we're being kept in the dark). It's still dragnet surveillance. Similarly torture is wrong and no amount of "targeting" can change that. So I think there's a disconnect in people who oppose mass surveillance but approve of this. I presume many people ok with this malware are opposed to torture; it wasn't to be taken literally.

<comment><USER8> Surveillance is situationally justifiable. Torture is not. This isn't a complex problem.

<comment><USER5> I think that surveillance and dragnet surveillance are fundamentally different and not comparable.

<comment><USER8> If so, then it's hard to imagine how you could distinguish between those in such a way that labels the methods described in the article as dragnet surveillance, rather than non-dragnet.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> >It's just about impossible for an outsider to reverse engineer a hard drive, read the existing firmware, and create malicious versions.

It's hard, but it isn't like this hasn't been done before: [http://spritesmods.com/?art=hddhack&page=3](http://spritesmods.com/?art=hddhack&page=3)

And to be honest, I think what they call "about impossible" is just a question of investing enough time and having a somewhat decent understanding of microprocessors and reverse-engineering. I'm not trying to downplay Equation Group's achievement, but it feels like arstechnica is starting to exaggerate here.

<comment><USER10> The whole "embedded programming is scary and impossible" sentiment seems to be pretty common these days. I mean, it's impressive, but well within the reach of an interested nation-state or even a skilled hobbyist like Sprite_tm.

It seems like the prevalence of modified router firmwares (and botnets based on them) should make it clear that firmware modification is A Thing That Happens(TM).

<comment><USER11> Anyone who thinks this kind of work requires nation-state backing should look at the (hobbyist!) projects to jailbreak the Xbox, X360 and PS3. Low-level kernel- and hypervisor- programming in which obstacles are casually overcome by dropping zero-day memory corruption bugs in core libraries that would be worth tens of thousands of dollars today just from bounty programs, deploying crypto bugs that make "custom RC5" and "thousands of iterations of SHA1" look like the shoplifted- from- Schneier technique that it appears to actually be (TEA hash collision, ECDSA nonce repeats, &c).

And these were, more or less, student side projects.

<comment><USER12> That doesn't mean that it's easy, it just means that hacking a PS3 is _highly motivating_. Making some lame centerfuge explode half-way around the world is high on patriotism (I guess) but short on lols.

<comment><USER9> He didn't say that it's easy, just that it isn't so hard that you need nation- state backing to do it. Judging purely from their capabilities, Equation Group could be a small group of college graduates who make a living with hacking and don't settle for the low-hanging fruit.

Of course the other evidence strongly suggests that this group is NSA- employed, but the real reason why their malware is so much more sophisticated than anything else out there isn't that nation states have staggering amounts of resources, it's simply that normal malware doesn't need to be sophisticated.

<comment><USER13> You are saying that you think it is reasonable that a small group of "college graduates" have created and executed a global espionage campaign against airgapped military targets using software that makes previous nec plus ultra NSA cyberweapons look obsolete.

</comment></comment></comment></comment></comment></comment><comment><USER14> The update confirms it is the NSA.

It is incredible, even the most generous estimation of the NSA's capabilities before the Snowden disclosures now look conservative. This is the stuff conspiracy theories are made of.

<comment><USER15> UR = UNITEDRAKE ("Regin", basically?). And that'd probably be email@example.com: that's the format their addresses are in.

This does seem to be, broadly-speaking, NSA's top-dollar brand-new 0-day-laden (at the time) malware, that they use to launch their less shiny stuff, which is more awkward and a massive overfunded modular boondoggle. This does not seem to be as freely shared around with the "Five Eyes".

By the way, there are innocent machines _in the US_ infected with this thing, at this very moment. Anyone care to explain that?

The hard-drive component should be completely detectable, _if_ you don't boot from it, based on the (small, sadly incomplete) fragment of (Cortex-M0?) stuff I've seen. Power-cycle it, send an ATA reset, read the MBR _and following sectors_. Look out for the NIC "option rom" persistence module, too - you may be well-advised to do it from something really exotic that doesn't run x86, just in case! (Independent hackers are running (µ)Linux on hard disks now, so it's not surprising a huge agency able to spend billions of dollars of tax money funding contractors on tiny pieces of this project got something of a head start!) Not sure of a good way to detect it in software, but it's not perfect, so it probably can be redpilled somehow.

Watch for "CD-ROM"s that unexpectedly have ATIPs, I guess?

<comment><USER9> Detecting an infected hard drive in software would be the usual malware arms race: you find some characteristic of it, they improve the firmware.

But if we start to systematically check for it, it should be easy to discover via hardware debugging. Find the JTAG interface on the hard disk controller (or whatever debugging interface the specific processor uses), dump the firmware and compare it to firmware dumps from other hard drives of the same model. I don't see how they could fool that process (given that you have a clean machine to read out the firmware).

Of course to be thorough you would have to check pretty much the firmware of every component of the computer.

<comment><USER16> > dump the firmware and compare it to firmware dumps from other hard drives of > the same model

And then ponder the unstated assumption that said other hard drives may or may not have been exploited already. Dealing with a state level actor is hard, in the "trusting trust" sense.

<comment><USER17> Well update your threat model appropriately. What are the realities if they've somehow hit _every single hard disk_ in the US? What is the likelihood this level of subterfuge can be maintained? How many people are involved?

Just because you can imagine it doesn't suddenly make it _practical_ , and it certainly doesn't mean they're going to burn that capability outing some guys porn habits either.

<comment><USER16> > What are the realities if they've somehow hit every single hard disk in the > US?

Who knows. The problem is you're not so concerned with "every single" hard drive in the US, but you may well be concerned with the other one you wish to use as a benchmark.

When you're dealing with things like hardware being compromised on the way from the plant to the store, or (as mentioned) a burned CD being compromised in the mail, and other things that really only governments can do, it changes the whole nature of the threat model.

</comment></comment></comment></comment></comment></comment><comment><USER18> I think the lesson here is not so much that governments can do this, but that entities with extremely large budgets are implementing broad, interconnected initiatives like this.

I think realistically any entity with over $20M could participate meaningfully in these kinds of exploits. The key is that in order to be useful many overlapping initiatives need to exist.

It seems like exploiting things like firmware would be pretty easy: You just get a member of your team who is a bit overqualified to apply for a job at the target company. A relatively small team could accomplish this in a few years, aided by the scarcity of top tier engineering talent.

The hard part is the social aspect of the attacks, but a single clever individual can come up with many.

<comment><USER19> well, in TFA it's a little easier for the NSA than embedding a mole, NSA just tells WD they want to buy drives, but need to audit all the source code including firmware for security issues, then modifies firmware, then intercepts hardware orders from targets and replaces them with ones with compromised firmware.

<comment><USER18> True, but I think it makes more sense to look at these tactics in the most general way possible. Firms could be doing them, as could much smaller governments.

Most press about itesec is focused on individual vulnerabilities and the tactics used in specific exploits. What we're seeing is tremendous "long game" strategic sophistication. Stuxnet was one example of using many attack vectors to eventually achieve a goal and these revelations suggest that was just the beginning.

Sure, intercepting the mail could be difficult, but all it takes is a few compromised UPS drivers and warrants are no longer needed.

</comment></comment></comment><comment><USER20> We should all be aware of the fact that these people are the enemy. They don't do these things to "protect freedom", they do them to destroy freedom.

Most of the people on Hackernews are uniquely placed to resist. Secure your software. Refuse to cooperate unless legally compelled. Analyze and publish any evidence of government attacks.

If you work for the NSA, understand this: you aren't American. You are the Internet Daesh. You will lose.

<comment><USER21> In what way are people who work for the NSA not American? They are literally working to protect American citizens from violence. While you may not agree with their methods, saying you are somehow "more" American than they are, let alone that they aren't American at all is a huge stretch.

I think I'm too young and naive to really form a good opinion on what they are doing, but I do know that I definitely haven't done anything I can remember for the sake of this country. Maybe they're being harmful, maybe not, I don't know, but at the very least I like to think they are trying to be helpful.

<comment><USER22> So it's a good thing they're spying on that poster child of terrorist anarchy; Angela Merkel?

<comment><USER23> Why should terrorism be the only reason to spy on someone?

Traditionally you spied on your friends to see what they were really thinking vs what they were saying.

</comment></comment></comment></comment><comment><USER24> I would like to ask anyone reading this to hypothesize with me. What if the US government was broadly and knowingly corrupt? How comfortable would you feel knowing that they had such broad and powerful technical capabilities? How would you fight against such a machine, to uproot the corruption?

I'm in awe of these technical feats, but also cautious about the implications of an all-knowing, all-powerful government presence who can infringe on your basic rights at-will.

<comment><USER25> It's not a what-if actually, the US government - or at least the higher-ups, like the senate - are bought by those with money. And buying votes / power / votes is the core principle of corruption.

<comment><USER24> I didn't want to sound too presumptuous when I wrote the comment, but I would agree the current state of things is far from ideal.

</comment></comment></comment><comment><USER26> What is the NSA is doing here is more subtle and I do not think they fully realize it. They are heavily financing and accelerating the speed of damages that people on the dark side can do. They are "inspiring" a bunch of curious teenagers (in the best case) or a bunch of cyber criminals (in the worst case) to create another malware like that. They are putting out there first of all the ideas, second the conviction that it is possible to do so, and third sample code to study, improve and deploy. This race to being the "smarter" spy is unfortunately leading us in a very risky world to live in. A lose-lose preposition. The press and the antivirus researchers are also not being too smart here to make this public available.

<comment><USER26> By the way, would you put a bunch of secret technology (maybe a new bomb) partially hidden in a manhole in the middle of the street, just in case a spy passing by needs to use it? That is pretty much what they are doing here... code is easy to move around and to copy. Today (or in relatively short amount of time) who wants that code bad enough will be able to find it. That technology that took yeas and million of dollars to develop is now available to the bad guys. Face palm!

</comment></comment><comment><USER27> Based on the article, are we to presume that this only affects tech in hostile countries? Or are they doing this to US-based equipment as well?

Seems like this will backfire spectacularly when foreign countries and companies stop buying American made tech for fear of these hardware backdoors. Spectacularly irresponsible.

<comment><USER28> Wait, is Mexico a "hostile country"?

<comment><USER29> Their central government is not hostile to the United States; cartels and certain cartel-influenced local governments certainly are.

<comment><USER28> There are of course entities hostile to the United States in every country, including the United States.

</comment></comment></comment></comment><comment><USER30> This is pretty intense malware, but at some level it's reassuring how narrowly-targeted these attacks seem to be. Arguably, this is what the NSA is _supposed_ to be doing: targeted attacks against key systems in hostile nations, not mass dragnet-surveillance of everyone on the Internet.

<comment><USER31> Hostile nations now include Belgium and the UK apparently

</comment></comment><comment><USER32> I don't know what's more disturbing: the fact that US government does this with impunity or the fact that a sizable group of technically competent citizens defends it.

<comment><USER33> I think the "sizable group of technically competent citizens" realizes that espionage between nation-states is as old as nation-states themselves, and that because of the nature of humanity, it's not going away anytime soon.

<comment><USER32> It's also where the money is. Getting paid to do the wrong thing is also an old trait of humankind.

Remember Google's original motto? Whatever happened to that?

</comment></comment></comment><comment><USER34> do we really need to encourage people to go into this line of work with language like this:

>A long list of almost superhuman technical feats illustrate Equation Group's extraordinary skill, painstaking work, and unlimited resources.

with the effort these people spend contributing next to zero value in the world (strongly negative value if you add up all the energy wasted on all sides including their 'enemies'; if all these sides spent the same resources on positive constructive research instead, we would be ahead). This genius could be applied to making a compiler/interpreter that just does what you're trying to do, regardless of whether you're a programmer.

it's a binary choice: have these people perform 'superhuman' feats of self- destruction and obfuscated encryption and finding zero-days....

... or give humanity the tools that every one of the seven billion people on it can instantiate any idea in seconds and have it actually be correctly interpreted and done.

Resources are spent tricking people so they don't notice something. Where are the resources being spent helping people do what they're trying to?

The example I always go to is: since the creation of the United States, how much inter-state (Iowa and Massachusetts) spying, warmaking, border control, etc, is wasted?

Could this have something to do with its performance over the last couple of centuries?

I think of these types of programs as 'welfare for geniuses'. give them an office and something to do.

but for God's sake, spit them out again. The optimal amount of state spying is, let's face it, much closer to zero than its current levels.

Spend on enough universal education that everyone shares values and nobody is destructive; put the rest into fundamental research and development.

I don't mind that the government exists to do research, employ people, and keep the world safe. But put some limits on it, and please don't encourage this with language like I quoted at the top.

humanity has better things to do with its time. nobody wants to live in a prison.

</comment><comment><USER35> Might as well tattoo a UUID on our foreheads and be done with it at this rate. God I hate hearing the apologists for the NSA in these threads. Are you so devoid of empathy?

<comment><USER36> Empathy for whom? I know pacifists want to believe we live in a post-war world, and if only we would not make guns or do the kind of things the NSA is doing here we'll suddenly live in a perfect world—but by what fantastical mechanism do you actually think that will happen?

<comment><USER35> The old pacifist keyword, its a bit like the communist keyword.

The only war we have at the moment is the one you (assuming you're a shill) created and want to sustain. Because if you don't, it all falls down. Its about the US dollar, and oil.

</comment></comment></comment><comment><USER37> Anyone get the feeling that at this point, the NSA et al, have moved onto even more sophisticated attacks?

<comment><USER29> Given that the unearthed program is a decade and a half old, I definitely get that feeling.

<comment><USER38> I'm picturing the NSA engineers laughing at the "advanced hacking" Kaspersky just unearthed, meanwhile working on stuff that would blow our mind and that we'll only discover 20 years from now.

<comment><USER39> i can't imagine how my mind can be blown any more than it already has.

<comment><USER37> Skynet?

</comment></comment></comment></comment></comment><comment><USER40> I'm pretty sure intercepting and screwing with mail is a felony... isn't that a felony? How exactly can the US Government have it's employees committing felonies with no warrants or oversight?

<comment><USER30> There are likely secret warrants in FISA court.

<comment><USER15> That doesn't really explain the US researchers infected with this.

<comment><USER41> That's what the FISA court is for: to issue secret warrants for surveillance of US citizens' domestic communications.

<comment><USER42> FISA is for Foreigners outside of US soil. You literally are exactly incorrect.

<comment><USER43> Not exactly - and that's why FISA is so controversial.

The FISA Amendment Act permits the court to issue a warrant if _either_ the sender or recipient is a non-US citizen abroad (since the "target" can be either party involved in the communication). Foreign nationals who are acquired abroad as FISA targets can later enter the US and remain subject to the warrant. And if there is uncertainty about the location or nationality of the target, the dispute is frequently resolved in favor of the surveillance agency.

The ACLU has fought FISA since it was enacted, as they believe it creates a way to unconstitutionally surveil people in the United States.

</comment></comment></comment></comment></comment></comment><comment><USER44> Takeaway lesson #1: re-flash all HDDs/SSDs with verified vanilla firmware from original manufacturer.

Lesson #2: Beware any and all data media, CDs, DVDs (Netflix? Blockbuster? Hollywood screeners? ...?), USB drives.

<comment><USER45> #1 using Jtag, nothing else will matter, and even then it will only work until your drive is reflashed again in another attack. Best way to secure HDD would be modifying firmware to DISABLE firmware upgrades and external access to service area permanently. Afaik HDDs keep some executable modules stored on HDD platters, that means 'simply' cutting WR leg on eprom chip is not enough. We are talking quite significant firmware rewrite here.

</comment></comment><comment><USER46> One of the images [0] in the article identifies a C&C server used by the attackers, technology-revealed.com. The script appears to embed an invisible iframe pointing to a page on that domain, which probably infects the machine using some zero-day exploit on the browser or one of its plugins. The domain is still registered, but appears not to be running an HTTP server anymore. Might be interesting to investigate if someone wants to look into that.

[0] [http://cdn.arstechnica.net/wp- content/uploads/2015/02/malici...](http://cdn.arstechnica.net/wp- content/uploads/2015/02/malicious-php-script-1280x719.jpg)

<comment><USER47> The article also mentions fanny.bmp as an NSA exploit. Here a post from 2010 [1] from someone who encountered this virus on an USB stick and saved the virus: [https://forum.lowyat.net/topic/1488855/all](https://forum.lowyat.net/topic/1488855/all)

</comment></comment><comment><USER48> Wasn't there a demonstration of a similar HDD exploit posted on HN earlier?

edit: found it [https://news.ycombinator.com/item?id=8665865](https://news.ycombinator.com/item?id=8665865)

</comment><comment><USER49> Interesting to compare the sophistication of this malware and physical weapon systems such as the Tomahawk cruise missile. One video of cluster munitions shows how the munitions try to attack armor, then vehicles, then groups of people, and then land to become pop up landmines. [1]

One has to assume that the capabilities of the other actors is similar. Makes for a pretty scary picture when you think about it.

[1] [https://www.youtube.com/watch?v=CY9gojFu- _U](https://www.youtube.com/watch?v=CY9gojFu-_U)

<comment><USER35> Admittedly I've had a few beers, but what was going on there? It looked like a chaff device or something. I didn't see the massive destruction I was hoping for.

</comment></comment><comment><USER50> I feel bad for the people that work for groups like this. Sure, you get to hack every developed nation in the world and are privy to the most sensitive information in the world. You're also therefore the world's largest liability, and your life is probably very expendable compared to the data you're retrieving. They might have a unique identity just for picking up milk.

</comment><comment><USER51> I'm surprised the CPU microcode angle isn't mentioned.

<comment><USER41> Has there ever been a non-theoretical use of microcode in malware? I would expect it to be difficult for a lot of reasons (the maximum microcode size is probably quite limited; the format is extremely proprietary; and there's only so much overhead you can add at the instruction level before you start causing a noticeable slowdown).

Also, microcode updates aren't retained when the system is powered, so you still need to have a persistent backdoor installed at the OS level. Given that, it's not clear what additional benefit a microcode backdoor would provide.

<comment><USER52> If you are going to modify a BIOS, then the BIOS can apply the microcode update. No need to tamper with OS then.

</comment></comment></comment><comment><USER53> Just think, a member of the group is probably reading this thread right now...

<comment><USER54> hahahaha...

<comment><USER55> This made me LOL

</comment></comment></comment><comment><USER56> So there is this new shodan-like database that takes data from massive scans of the Internet (I think?) etc... Might be worth running all C&Cs through it to discover more domains , for example [https://rateip.com/ipv4/220.127.116.11](https://rateip.com/ipv4/220.127.116.11)

</comment><comment><USER57> Equation Group: Questions and Answers by Kaspersky [pdf] --> [https://cdn1.vox- cdn.com/uploads/chorus_asset/file/3415904/E...](https://cdn1.vox- cdn.com/uploads/chorus_asset/file/3415904/Equation_group_questions_and_answers.0.pdf)

</comment><comment><USER18> The point isn't whether we should be surprised this is the NSA, the interesting thing is that this layered strategy is what you do if you have access to lots of money and talent.

</comment><comment><USER58> How can one detect infection?

</comment><comment><USER59> Scalpel, not a sledgehammer.

</comment><comment><USER60> While this is undoubtedly scary, it looks very Windows-specific and would seem unlikely to affect a somewhat security-conscious Linux user.

Java and IE exploits, autorun files, NTFS...

> The malicious firmware created a secret storage vault that survive

> military-grade disk wiping and reformatting, making sensitive data

> stolen from victims available even after reformatting the drive and

> reinstalling the operating system.

And that's why it's possibly not the greatest idea to replace simple firmware and drivers with small operating systems (Intel AMT, microcode, SSDs, Smart TVs etc.).

<comment><USER9> I suspect that most of their targets use Windows, so we have more malware samples from Windows hosts. The article specifically mentions that they heavily suspect that a Mac malware exists, yet they haven't found it yet. Consequently we don't know which exploits they use to get on Macs. They also talk about a wide range of servers being infected, only attacking Windows servers would be very limiting.

My best guess is that if you are a target, using Linux makes their work harder but not impossible. They had over a decade to figure this out and vulnerabilities in Linux software are found all the time.

<comment><USER60> The site shows a PHP script targeting vbulletin. Obviously there are lots of ways of attacking servers, especially if they run PHP applications and such.

There was once a PDF exploit for iOS that made it possible to jailbreak the phone (or run any code) if the user opened a PDF, so the PHP program could have been redirecting to something like this.

Linux malware would probably have a much lower effectiveness / $, so it doesn't pay off for them.

</comment></comment></comment><comment><USER61> What's worse is the fact that this would be more terrifying if this weren't an action to the NSA's credit. Chinese, British, and Russian governments also have the capability of developing intrusive malware at such a scale.

I see that this is confirmed as the NSA, but the world has been awfully silent on the intrusiveness of Chinese malware practices. You can not travel safely in China without your domestic devices being breached, yet we don't hear about large-scale projects from their government, which also possesses "near limitless resources" at their disposal.

I just think it's interesting there's so much attention on the NSA.

<comment><USER62> People would be angry even if they didn't know that Equation Group was part of the NSA.

</comment></comment><comment><USER63> Absolutely! I live in Bulgaria, Eastern Europe, a poor, small country, which has been accepted in NATO and the European Union, but still is hosted by the past-Soviet`s secret intelligence services of Moscow, which are now illegal, but has agents and the absolute power over the legal secret services in Bulgaria. The surveillance is much larger, than in the USA, but all the info goes to Moscow.

So, what are you troubling about, when NSA is pure American, and has the purpose to protect you from the legions of Russian, Chinese hackers - officially military, or just playing and stilling criminals, and all the Islamic recruiters, who are trying to recruit American citizens for their Jihad against The Civilization...

I don`t understand, are you stupid, people, or what? NSA has the duty to protect you, and you are acting as children, whom parents has been installed a security program on your PC, to protect you from pedophiles... It`s the same case with the NSA surveillance!

</comment><comment><USER64> We definitely throw our weight around and interfere with Iran's domestic affairs. But this is definitely a case of "we think our morals are better than yours" (and we have bigger guns so we're gonna enforce those morals).

The Iranian government funds terrorist groups around the world, and has called for the destruction of certain other countries. So yeah, if we have the ability to stifle them (particularly their nuclear ambitions) I say we go for it.

I do feel bad for many of the Iranian people. Many of them do not like their government and are therefore held hostage by the events around them, which they have no control over, and may disagree with.

<comment> Ar-Curunir Hmm, and what about the Saudi government that actually does fund terrorist activities and opposes almost everything that America claims to stand for?

America is fine with propping up that regime, so why is Iran an issue, when from everything that I've read and experienced (12 years in the Middle East) there have been no implications of Iran in modern terrorist activities?

What moral have high ground can you take then?

<comment><USER65> "when from everything that I've read and experienced (12 years in the Middle East) there have been no implications of Iran in modern terrorist activities?"

\- Iran does not recognize Israel.

\- Iran supplies political support and weapons to Hamas,[14] an organization classified by Israel, the United States, Canada, the European Union and Japan as a terrorist organization. Mahmoud Abbas, President of the Palestinian National Authority, has said "Hamas is funded by Iran. It claims it is financed by donations, but the donations are nothing like what it receives from Iran".[15] From 2000 to 2004, Hamas was responsible for killing nearly 400 Israelis and wounding more than 2,000 in 425 attacks, according to the Israeli Ministry of Foreign Affairs. From 2001 through May 2008, Hamas launched more than 3,000 Qassam rockets and 2,500 mortar attacks into Israel.[16]

\- Iranian proxies killed an estimated 1,100 US troops in Iraq.[35] In addition, insurgents supported by Iran reportedly committed acts of terrorism.[34][36][37]

Lots more here: [http://en.wikipedia.org/wiki/Iran_and_state- sponsored_terror...](http://en.wikipedia.org/wiki/Iran_and_state- sponsored_terrorism)

I am not claiming these are all fact, but I think there is little doubt that they are at least "implications" as you put it.

(edited for formatting.)

<comment><USER66> > \- Iran does not recognize Israel.

Neither does Saudi Arabia! Why does it even matter, since 32 countries do not recognize Israel; including huge US beneficiaries like Afghanistan, Bangladesh and Pakistan?

> From 2000 to 2004, Hamas was responsible for killing nearly 400 Israelis and > wounding more than 2,000 ...

... and in turn Israelis killed 4907 Palestinians and wounded 8611? If the number of killed is a factor in determining one's "terroristness", then surely you can't claim that Israel is a victim here! Just last year, in Operation Protective Edge, 73 Israelis were killed and 664 wounded; whereas 2100 Palestinians were killed and 11000 wounded: [http://www.jewishvirtuallibrary.org/jsource/History/casualti...](http://www.jewishvirtuallibrary.org/jsource/History/casualtiestotal.html)

> Iranian proxies killed an estimated 1,100 US troops in Iraq.

... and Saudi proxies killed nearly 4000 on 9/11 !

<comment><USER67> I can't help but think that many of the people posting on these boards are either shills or have psychological disorders, because they ignore completely obvious facts like the ones you point out and continue to hold onto their views. That type of willful blind spot is not tolerated with other subjects here on Hacker News.

<comment><USER68> There are political differences to be expected, but let me try to explain why the USA is willing to rub elbows with the Saudis and not with Iran. My credentials are this: I spent 6 months working on strategy in the CENTCOM AOR bitching about every single country in the Persian/Arabian Gulf every single day. The choice is one of picking the lesser of two evils. The USA has a great interest in keeping and increasing its geopolitical influence in the Middle East (Israel is not included in this region even though we have to plan for their actions). The two main regional powers in the Middle East are Saudi Arabia and Iran. We used to be friendly with both countries back in the 1970s but Iran had a revolution and the Ayatollah didn't want anything to do with us, and it remains that way today. Although the current President of Iran (Rouhani) has mentioned that he would like to improve relations with the USA, he is not the boss regardless of how many people voted for him; the Ayatollah is the Supreme Leader and he still openly calls for the death of Israel and the USA. So we obviously can't be friends with Iran. That leaves us with Saudi Arabia, and while they also sponsor terrorism, they are all we have left. We overlook their actions, and in exchange we are allowed to keep a significant military force there, which props up their government and provides them with a pretty big stick when negotiating with the smaller countries in the region (the only other stick they really have is their gargantuan oil production). So our PATRIOT batteries protect the family of Saud from Iran, and we get to have a little say in their goings on. It's as simple as that. Just about everybody we deal with is a piece of scum that would like to see the USA brought down a peg, but Saudi Arabia doesn't do it openly and they depend on our money and protection, so we get along a little better than them. Iran openly calls for our destruction. The last several US presidents, both Republican and Democrat have decided that it's better for us to support Saudi Arabia than not have any influence in the Middle East at all. The only other option is to withdraw our forces (without Saudi Arabia's cooperation we would probably lose our basing agreements with all the other countries in the region) and have no say whatsoever and watch as the Middle East goes in a direction that isn't beneficial to the USA in the slightest.

\--And to get back to the topic of the parent post, yeah, this is exactly what the NSA is supposed to do. It is supposed to do two things: Secure the information of the USA and its citizens, and to undermine the security of everything else. A lot of US citizens were upset when they found out that the NSA vacuum had gotten their information as well, and rightfully so. But the NSA is still primarily focused on targeting external entities, and although those entities are well within their rights to complain, don't expect the NSA to stop just because someone didn't like it. Regardless of what a federal judge says about collecting on US citizens, no judge will EVER tell the NSA to stop its clandestine activities on foreign networks.

<comment><USER67> I appreciate your informed and reasonable response, but you didn't really address the two main points of the grandparent poster.

1\. Iran isn't really different in stance from several other countries. You say that Iran calls for the death of Israel and America, but this isn't really true. The one quote from Ahmadinejad that gets thrown around turned out to be a mistranslation. And even if Iran's leadership did say such things, they are not even close to being alone in such heavy handed rhetoric. And anyway words shouldn't be enough to start trillion dollar wars. Otherwise Iran should have the right to attack the US for McCain and other American politicians' equivalently inflammatory speech.

2\. Israel is as complicit in terror as other powers in the middle east, if not more so, but they never called out. The reasons are usually vague or fallacious, such as them being "friends" or "like us" or the only democracy in the middle east.

<comment><USER68> The question I was answering above was "why Saudi Arabia and not Iran? Aren't they both bad?" Regarding your other points, I'll answer them below (and I admit my attitude upfront on the first one, you just posted that you are frustrated with people with a willful blind spot and then you give me #1):

1\. Iran's calling for the death of Israel and the USA is just a single missed translation? Give me a break. A news article from five days ago: "As usual, Iranians chanted "Death to America" and also denounced Israel." [http://www.npr.org/blogs/thetwo- way/2015/02/11/385396449/on-...](http://www.npr.org/blogs/thetwo- way/2015/02/11/385396449/on-..). It's a slogan over there: [http://en.wikipedia.org/wiki/Death_to_America](http://en.wikipedia.org/wiki/Death_to_America). And they aren't alone in their rhetoric. North Korea says it too. And guess what? We've isolated them as much as we can as well. The countries that are willing to openly say "Death to America" are ones that haven't matured enough on the world stage to realize that you should say one thing and do another. The rhetoric alone isn't enough to start a war, but their actions aren't that innocent. Iran's first "blue water navy" excursion was planned to sail all the way to the coast of the USA and enter our territorial waters, just to say that they could (but their ship didn't make it). They built a mock US aircraft carrier and practiced shooting anti-ship missiles at it. They aren't just talking big, they are actively posturing as the opposition to the US influence in the region.

2\. Yeah, Israel does some stuff. The Jews have killed a lot of people (even going back to biblical times). They usually get a pass because they are seen as the "small fry" backed into a corner by all the Muslim countries surrounding them. It isn't true, but that is the story that is spun. Similar to how we treat the Saudis, we let Israel get away with a lot of things that we should probably put a stop to--seriously, pretty much anybody with sense can see that compromise is the best way to solve the Israel/Palestine issue, but when two peoples hate each other that much, sometimes it's easier to watch them punch it out. If you try to break up a fight, you risk getting punched yourself. And besides, we don't get along with the Israelis as much as most people think. Seriously, we share more intel with New Zealand than we do with Israel.

<comment> saiya-jin Subjective experience here - been in Iran (Tehran, Isfahan, Yazd and places between those, backpacking & mountaineering). I think we all know better than to judge whole country of 80 million based on few radicals that are conveniently showed in TV? I have met several hundred people, most were very fond of west, US too (that surprised me), few didn't care and there was exactly 1 taxi driver in Isfahan, who asked me if I knew Obama - because he is war criminal... arguably not very bright type :) Maybe my experience isn't representative of the whole country, but few people screaming Death to america aren't either.

We get it, you (US) want influence there, because of oil. And because of... oil. You don't like that there is actually somebody who doesn't care that much and isn't afraid. A bit egoistic, and I don't get what do you expect to get in long term, except for keeping whole region unstable, with weak rulers who bow in front of you... ah wait</comment></comment></comment></comment></comment></comment></comment></comment></comment>


Ask HN: How to get started with a very large code base? - <AUTHOR> I recently got started with a very large code base for a project, and I can't find my way through it, the documentation is very bad, do you have any specific advice for getting started with such code base ? Any tips will be helpful. <comment><USER2> Debuggers are great for this. Nothing will teach you like stepping through the code. Do whatever it takes to get one working.

I recommend replying to your question with at least a _bit_ more detail, particularly the language or languages in question and what the general environment is. I could be more detailed for a few environments but I don't want to just blindly guess.

</comment><comment><USER3> Write tests for every behavior the system currently exhibits.

Then systematically comment out everything -- one function/block/class/whatever at a time -- making note of which tests break.

That should give you a better idea of how everything hangs together -- and uncover the parts that aren't even used anymore.

Almost any software that you inherit, that has been around for a while, will be a complete mess.

If you have time for a book, Michael Feathers' Working Effectively with Legacy Code will likely help a lot.

<comment><USER2> "Write tests for every behavior the system currently exhibits."

That's often harder than just bashing your head against the problem. Retrofitting code bases with any sort of unit testing is almost always a challenge on its own terms... one much easier when one has at least an idea of how it's working already. And that's after I apply the principle of charity and downgrade "every behavior" to "many behaviors of interest".

It'd be great if it works, but I suspect it would only work when it's unnecessary anyhow because any code base amenable to that treatment wasn't that hard of a problem in the first place.

<comment><USER3> I am of course, making the assumption that OP knows what the software _does_ \-- but not how it goes about doing it.

</comment></comment></comment><comment><USER4> I found this [1] in a comment where someone asked a similar question. This really helps only if you are using a git repo though. There are some good gems in this [2] thread too.

[1]: [https://github.com/gilesbokett/rewind](https://github.com/gilesbokett/rewind)

[2]: [https://news.ycombinator.com/item?id=9784008](https://news.ycombinator.com/item?id=9784008)

<comment><USER5> I get -404 with [1]

</comment></comment><comment><USER5> _Take notes_ (external to the code) on everything you learn about a given module, whether code/flow analysis, insights on input / output / objects, visible interactions, and what it reveals about how project-specific libraries are used. Otherwise, it's so easy to have an "oh, _that 's_ what that does" insight and lose it later because you assumed you'd remember it.</comment>


The Pen Gets Mightier - <AUTHOR> http://www.theatlantic.com/magazine/print/2010/09/the-pen-gets-mightier/8184/ <comment><USER2> I've played with one of these and they are pretty neat. As others have mentioned, most of the functionality relies on using special paper which is printed with patterns of dots. The paper doesn't _look_ special though -- the dots are almost unnoticeable, and it's ordinary paper in texture and thickness etc.

The piano-playing functionality mentioned in the article really does work like magic -- draw keys, play keys, done.

In general though, I struggle to think of a reason I'd use it: I type much faster and more accurately than I write, and I seldom have any reason to be matching audio with text transcriptions.

<comment><USER3> Serious question:

Aside from school note taking and small meetings, is there any other purpose to this?

I'm trying to think of another scenario this would be useful.

It would be cool if this could do some basic Visio diagrams or something (even though I haven't used Visio in years).

This seems so darn cool, but I barely write anything by hand anymore. Just some basic design (layout, diagrams) and lists (groceries, supplies, to-do's, etc).

<comment><USER4> It has an API so people can develop new applications for it. I didn't find anything like that in my quick scan through it.

Link to their developer page:

[http://www.livescribe.com/cgi- bin/WebObjects/LDApp.woa/wa/De...](http://www.livescribe.com/cgi- bin/WebObjects/LDApp.woa/wa/DeveloperOverviewPage)

</comment></comment></comment><comment><USER5> Kind of lame that the article doesn't link back to the actual product: <http://www.livescribe.com/>

I also found this video of someone demoing the pen: <http://www.youtube.com/watch?v=OU_RKv5zemM>

</comment><comment><USER6> The business model of similar pens is selling the paper. Because the dots make each square inch globally unique, they are effectively selling - and _making_ \- real estate (a bit like ICANN selling domain names.)

I think this is unbelievably cool and clever - but I also really don't like it.

<http://www.wired.com/wired/archive/9.04/anoto_pr.html>

<comment><USER7> It is _possible_ to print the paper at home, if you have a color laser printer. (600+ dpi required, but I haven't seen any lately that aren't at least 600.)

Newegg has printers that fit the bill starting at $150-ish. No special toner required, although Anoto does make some sort of custom ink you can buy for the purpose.

So it is possible to avoid buying their paper, just possibly annoying.

</comment></comment><comment><USER8> The pen depends on this technology -- a special dot pattern on the required paper allowing the pen to know the page and position where it is pointed:

<http://en.wikipedia.org/wiki/Anoto>

The first 'C-Pen' using the technique was launched in Sweden in 1998, internationally in 1999. Jim Marggraff's LiveScribe was formed in 2007, to create other products based on the technology -- following the successful 'pentop' product Marggraff created at LeapFrog using the paper.

<comment><USER9> Anoto is pretty damned cool and I've been tracking the products for a while. LeapFrog has 4-5 products using the tech now (the Tag family) and has cost reduced it significantly.

I guess Microsoft even considered it a threat to the point where they announced their own product as one of their amazing R&D discoveries, but quickly dropped any discussion of it.

</comment></comment><comment><AUTHOR> Does this thing work as well as he makes out? If so, very cool.

</comment><comment><USER10> Look like the paper and the pen ain't dead just yet!</comment>


Qanon is more important than you think - <AUTHOR> https://www.theatlantic.com/magazine/archive/2020/06/qanon-nothing-can-stop-what-is-coming/610567/ <comment><USER2> Maybe if they didn't ban us everywhere we wouldn't be such conspiracy theorists.

</comment><comment><USER3> Qanon is a psyop. Change my mind</comment>


Ask HN: What do you guys think about Nokia now? - <AUTHOR>

Nokia has released few products recently and now its gaining momentum and its share price is seeing a steady increase. I was wondering if anyone has experienced Nokia Lumia series and what is the general feel.<p>You can also quickly submit the poll. <comment><USER2> Post a proper poll. The way you have done it means that

1\. we can't see how many votes each option gets; and

2\. you get lots of karma for not posting anything insightful.

<http://news.ycombinator.com/newpoll>

Although I see you don't have enough karma to post a poll. Best hope that someone else finds it interesting enough to post properly.

<comment><AUTHOR> You are correct. I am not very well versed with HN. I saw a poll recently which was done like this and I liked that. Although now I see that that was using the poll feature which I cannot use as of now.

I too hope someone thinks its an interesting idea to check up on Nokia's image among HNers given recent proceedings.

</comment></comment><comment><AUTHOR> I dislike Nokia

</comment><comment><AUTHOR> I like Nokia.</comment>


Backpack connects you with travelers so you can buy items in other countries - <AUTHOR> https://intl.backpackbang.com <comment><AUTHOR> I believe this is a YC backed startup and there was some press around the time of the initial Demo Day. Looks like they have a brand new website , and seems to have pivoted to just one geographical market. A lot of HN crowd said this would totally work in Europe as well. Has anyone used this service so far ?</comment>


Facebook Music sharing requires friends use the same music partner - <AUTHOR> http://gizmodo.com/5842983/facebook-fails-to-let-fans-share-music-across-platforms

<comment><AUTHOR> Key takeaway: "if Facebook friends want to become 'friends with (musical) benefits,' they'll both need to subscribe to the same music service... The way Facebook has set up music sharing will encourage a single winner to emerge among the existing music subscriptions. And that winner will be whichever music service first gains critical mass among Facebook users as the way to share music."

Talk about a land rush. This should be interesting.</comment>


Ask YC: What do you scrape? How do you scrape? - <AUTHOR>

Theory-wise, there's regular languages, context-free grammars, and combinatorial categorial grammars ( http://openccg.sf.net ). But regular + lists seems adequate for most tasks.
What sorts of scraping do you find yourself doing?
What are your biggest frustrations?
What's the coolest hack you've encountered while scraping?
My cofounder and I have been working on a domain-specific language to make scraping quick and easy, so that you can write, say, 100 different website scrapers in less time -- http://dartbanks.com/simplescrape . We'd love feedback on this approach. <comment><USER2> I do all my screen scraping with PHP, curl, and some regex. Previously I used plain PHP.

I use it to scrape television listing data (<http://ktyp.com/rss/tv/> was my old site, and <http://code.google.com/p/listocracy/>) and more recently to scrape resume data from job posting websites for a (YC-rejected :P ) side project I'm working on.

The hardest part I've encountered with scraping is odd login and form setups. For example Monster.com uses an outside script to attempt to fool scraping. A couple other sites use bizarre redirecting across pages. Also AJAX certainly has changed the way a lot of screen scraping is done.

Finally, the most useful tool I've used is LiveHTTPHeaders (<http://livehttpheaders.mozdev.org/>) which is great for following how a site operates.

Edit: For PHP, another interesting tool for scraping is htmlSQL (<http://www.jonasjohn.de/lab/htmlsql.htm>) which allows HTML to be searched using SQL like syntax.

<comment><USER3> "Also AJAX certainly has changed the way a lot of screen scraping is done."

I'd be interested in how you tackle this one. I've always used something like Perl/Curl/wget etc for scraping, but (like you say) JavaScript messes that up. I've had moderate success using GreaseMonkey and regexps in JavaScript code, but it's a bit fragile. I'm thinking of using GreaseMonkey + jQuery, since that should allow me to select DOM elements very easily. But if you have a better way, please share :)

<comment><USER4> Even though it's actually a testing tool, you might have some luck with Canoo Webtest + Groovy (<http://webtest.canoo.com>). Webtest uses HtmlUnit which has pretty good Javascript support, and means you don't have to mess with regexps to get around the document structure, and Groovy lets you use an actual programming language rather than the awkward Ant-based syntax of Webtest. It takes some getting used to, and I haven't used it for web scraping, but it's a pretty powerful combination.

<comment><USER3> Thanks, I'll give it a try. I'm collaborating on a project which involves getting info from online financial markets, btw, but it's getting held up because of this scraping problem. So new ideas might help get it moving again.

</comment></comment></comment></comment><comment><USER5> If you really want to go bonkers on scraping, there are books on this.

<http://nostarch.com/frameset.php?startat=webbots>

<http://www.oreilly.com/catalog/spiderhks/>

That probably covers the topic of scraping pretty exhaustively.

</comment><comment><USER6> We are working on a lot of scraping and analysis and here are a few links that you might be interested in if you are using Python:

<http://pyro.sourceforge.net/>

<http://pyprocessing.berlios.de/>

<http://www.sqlalchemy.org/>

<http://codespeak.net/lxml/>

<http://nltk.org/index.php/Main_Page>

The biggest hurdle is in understanding how to navigate through a complex site - such as a forum, real estate etc. We have created a visual tool for this however there are other methods. Look at dapper.net as this is useful.

I am wondering if there could be some collaborative effort from the minds on this site to create something unique and groundbreaking

</comment><comment><USER7> I've done my share of screen scraping, gathering all different kinds of data. Movies, sports, finance, you name it. Here are three things I can tell you:

1\. Take the time to get very familiar with regular expressions. If you think you know your regex pretty well, go to the docs or get a book and find three things you don't understand and understand them fully. Then find three more.

2\. The data doesn't have to be perfect. In most cases you can clean it after you've stored it. It's generally better to get more than you think you might need (in terms of data or html/formatting around the data) and then go back and clean it later

3\. Generally, my most successful data mining algorithms involve a lot of hacks. There are very few clean formulas...usually I have to play with the data for awhile and fix a lot of one offs and special cases and then it ends up coming out ok

</comment><comment><USER8> I use Ruby, with its nice regex support and libraries (open-uri, REXML) and the hpricot and mechanize rubygems.

Yahoo Pipes is also fun to play with; and Firebug is the scraper's best friend.

Right now I'm working on scraping _public_ LinkedIn data. In the past I've done Craigslist and Twitter. I haven't done anything really hard, though -- mostly things that can be read as XML.

Here's a few cool links if you're interested in scraping with Ruby: <http://del.icio.us/jeremyraines/scraping>

</comment><comment><USER9> I'm working on something similar. Turns out scrapping a small part of the problem. I don't use beautifulsoup. Turns out you can transform html of a page into a list, which can easily be scrapped.

Now that I have used it to extract data out of many different types of pages. I'm looking to turn it into a dsl. So that the code looks natural. Currently it's just functions which search for tags in html. You can then easily filter some or others. here is an example

(extract-all page [(and (tagp _ :a) (classp _ "jdtd4"))])

</comment><comment><USER10> The most powerful, general level scraping stuff I've come across lately has been ScRUBYt : <http://scrubyt.org/> .. although I admit I don't have much to do to use it often.

It lets you specify which items on an initial / prototype page you want to scrape, and then it builds up a set of rules than then work on future similar instances of that page. Good for scraping eBay, Google, stuff like that.

</comment><comment><USER11> I use BeautifulSoup when needed for simple scraping.

My biggest frustrations, right now, are really around getting data from lots of different websites in subtly varied forms. This is a tough problem to automate. I certainly haven't found any tools that make it simple.

I'd be happy with a 50% correctness rate, looking for very loose patterns. I just haven't found a tool and, while I have some ideas for how to do it, it's a major project in itself to produce something that can do this.

For example, imagine writing a scraper that would parse out every food recipe online. Whether it be in forums, blogs, etc, etc. That's the sort of scraping I'm looking for and the best I'd have is putting together a neural network or other system that I can train against human-provided data. Unfortunately getting such a system to partition the text to just the recipe would be difficult.

<comment><USER12> Getting just the recipe would be the hardest part, but it's still doable. Once you figure out that you're currently parsing a recipe (via keywords, close matching, whatever) you could fan out and look for common start/end tags like
,
, etc. If you use something like Beautiful Soup you could do this pre-parse instead of post-parse and eliminate a lot of extra stuff (no recipes in the  tag, etc.)

After that it just becomes an issue of removing the cruft around the recipe. I would start with common stuff: splitting things up by
 or inner
 since if someone is gonna have something before / after their recipe (say, on a forum) it'll be split up with blank lines somehow (well, usually). This will be another time to use things like close matching and teaching the algorithm what it gets right/wrong so it can weigh things as recipe/not better in the future.

If you do all this and add more specific edge cases as time goes on, I think you'd be able to maintain a 50% correctness rate pretty easily.

Edit: And it'd be much cheaper than a neural network ;)

</comment></comment><comment><USER13> I use curl, wget and links to retrieve data from sites and then I filter it with old sed and grep.

I created a mashup of AIM + Flicker.

If you use AIM 6 or AIM lite send a message to MyPictureBuddy

then send a message and enjoy.

basically You type a keyword and it gots to flicker and retrieves image information to display pictures right inside your AIM chat session.

I also have another Bot that parses HackerNews XML and then display it on the chat session. The bot name is

HackerNewsYC

</comment><comment><USER14> I used Mechanize and Hpricot on a project recently to create a sort of poor- man's API. My client is a performing arts organization that wanted a new website but they already had a (dreadful) internally-hosted site for selling tickets.

In order to keep website users from having 2 accounts I created an interface that scrapes the sign in, sign up, lost password, change password, and couple other screens of the internal system. So when users come to the website and "login" they're actually logging in to the internal system and I just record their session from the internal system so I can masquerade as them as they go about their business.

It's not going to support 100s of connections per second but it gets the job done for their traffic levels (36,000 views the first day of launch).

</comment><comment><USER15> Our startup, Feedity - <http://feedity.com> , generates/creates RSS web feeds from virtually any webpage, for the purpose of content tracking and mashup data reuse.

We scrape public webpages (with an option for content owners to restrict access), and we use the .NET Framework' in-built socket implementation (System.Net namespace) for fetching remote content.

Our biggest frustration was to deal with invalid charset/content encoding of the source webpages. But we resolved it using a custom module. Now everything we parse is unicode (utf-8)!

The collest hack we've encountered while scraping is utilizing the Conditional GET behavior using the HTTP If- Modified-Since header.

</comment><comment><USER16> Either Beautiful Soup, or Yahoo Pipes... I have a website that parses RSS feeds, and some sites don't have feeds yet! Or if they do, they aren't usable. So I use Pipes to scrape a page and turn it into a feed using their regexp operator, then my site uses that feed.

</comment><comment><USER17> I'm surprised nobody has mentioned dapper yet (www.dapper.net) it's a really nice approach at turning web-sites into structured content.

</comment><comment><USER18> HtmlPrag turns any HTML into nice s-expressions. It's a Scheme library. <http://www.neilvandyke.org/htmlprag/>

I've used it a lot - it's really great.

</comment><comment><USER19> I used Hpricot to scrape web.archive and reddit to make <http://reredd.com>.

Plan on scraping past billboard charts to let people listen to the radio back in time.

</comment><comment><USER20> I come from a Perl background so I've been using HTML::TreeBuilder and XML::TreeBuilder to do my parsing. It will basically load an HTML/XML file into it's own tree structure and give you an easy way to go through it. By knowing how each site names their divs/classes I am able to scrape.

I took a quick glimpse at beautiful soup and it seems to be doing something similar - someone let me know if this is correct.

<comment><USER21> Yes. You can even regex search through the tree. Weeeeee!

BeautifulSoup is nothing unique, but it can handle malformed data that saves you a ton of hassle.

</comment></comment><comment><USER22> Scraping EU public procurement contracts from the "Tenders Europa Daily" database (<http://ted.europa.eu/>). There's more than a million documents with each document requiring up to two requests. Been at it for several weeks with a multithreaded scraper and we're almost through. Using Solvent (simile.mit.edu/solvent/) to generate xpath expressions and HtmlAgilityPack (www.codeplex.com/htmlagilitypack) to run the xpath on the downloaded html with regexps as the topping. They're a match made in heaven (<http://www.itu.dk/~friism/blog/?p=40>).

The login procedure is gothic and took a lot of wiresharking to figure out. .Net has pretty good scraping-support in the WebClient and HttpWebRequest classes found in the System.Net namespace.

Will publish results soon... :-)

<comment><USER6> Be careful here. The content is actually copyrighted. Whilst you can scrape it their T&Cs expressly forbid it. They sell licenses to access this information - the license is NOT expensive and they provide direct access to all the data in XML.

<comment><USER22> [http://ted.europa.eu/Exec?DataFlow=ShowPage.dfl&Template...](http://ted.europa.eu/Exec?DataFlow=ShowPage.dfl&Template=TED/important_legal_notice)

Quote: "Reproduction is authorised provided the source is acknowledged. However, to prevent disruptions in service to our normal users from bulk downloads of TED data, we reserve the right to check for, and block, attempts to download excessive quantities of documents, particularly using automated or robot-like tools."

... they apparently chose not to exercise that right in this case, the scrape completed last night (all 18 GB of it).

</comment></comment></comment><comment><USER23> I've written a lot of this sort of program over the last 18 months. This is something that people are in need of all the time. I would say that there isn't yet a tool which does this to the level that customers want.

I use Mechanize, both in its Ruby and Python forms (I prefer Ruby) and plain old regular expressions to get the information that I want. Often times I will use a divide and conquer strategy by removing part of the web page (for example, the ) and successively paring it down to what I really want.

Javascript can be a problem. What I normally do is actually read the Javascript on the page, and then recreate that behavior in my Ruby code. Often times this means simply setting some form values (usually hidden) and then submitting the form.

</comment><comment><USER24> Scraping HN for fnid's to auto-post xkcd comics :-P. It was a hack so I just did string searches.

</comment><comment><USER25> web scrapping with beautiful soup (is this old school already?) - parsing the Sydney Future Exchange for data back in 2003 (still running)

</comment><comment><USER12> We're using a general-use multi-threaded crawler to get the pages and then using Beautiful Soup and a bit of regex to parse them. Though we are scraping multiple sites, they are all in the same "category" so to speak, so there are a lot of generic parsing methods that are simply overridden when necessary. PyParsing was played with for a while, but since data comes in so many slightly varied forms I was ending up with rules that were miles and miles long just to find a simple price or date/time on a page that would work for the largest number of sites possible.

</comment><comment><USER26> My startup, <http://www.FuseCal.com> (previously discussed at <http://news.ycombinator.com/item?id=146134>), scrapes calendar events out of web pages and into your personal calendar. In the general case, we don't know anything about the layout of the page before trying to extract the events, so there's something of a classification problem first.

</comment><comment><USER27> I'm a big fan of using Hpricot + Ruby. I'd say the sites I had been scraping but I doubt my old client wants it to come out :|

To get the most bang for my buck (developer time wise) I would visit each site with firebug in inspect mode, hover the data I want to extract. From there I figure out how I would style that element, and because Hpricot supports CSS selectors I've straight away got a method for pulling that data out of the page.

</comment><comment><USER28> This sounds redundant already, but I scrape using beautiful soup. Right now I'm scraping a lot of news sites and feeds for a project I am working on.

</comment><comment><USER29> This can be very useful. I use pyparsing with custom python code for scraping.

</comment><comment><USER21> I always do custom stuff in beautiful soup, but this looks somewhat cool.

Maybe have it so you can edit the sample text and language and see the results all on a web page?

</comment><comment><USER30> <http://gatherer.wizards.com> with BeautifulSoup, the only parser I've found that can deal with this @$%^! HTML.

</comment><comment><USER31> Do any of you who scrape fear retaliation from the sites you scrape? Maybe you are violating a ToS or scraping copyrighted text, and they cut off your IP. Thoughts?

<comment><USER6> I think you have to take into consideration the TOS, copyright and also robots.txt. If you ignore these then its well within the site owners rights to do something about it - blocking you or further. We always look at the robots.txt file first and use that as our benchmark in terms of what they (the site) wish robots/crawlers to look at

</comment></comment><comment><USER32> I have a similar DSL built in Ruby that can be run by by either Mechanize or Watir. I highly recommend Watir if you need to scrape ajax.

</comment><comment><USER33> I've been using C# with the HTMLAgilityPack. Probably not as fast as it could be, but C# is what I know best.

</comment><comment><USER34> Yum, declarative!

I can't say I'm crazy about the syntax, but I'll give this a try when I get home.

</comment><comment><USER35> Firebug can be helpful for finding elements you want to regex on!

</comment><comment><USER36> anyone that complains about HTML scraping is a pussy. Seriously its trivial compared to what we had to do in the past. I like hpricot for ruby.

</comment><comment><USER37> Longest Common Subsequences are quite useful as well.

<comment><AUTHOR> Do you mean for adding a little resilience to your rigid model, or something funkier?

</comment></comment><comment><USER38> anyone have success with emacs and w3? I haven't given it a shot yet, but seems like its interactive nature might be useful.

</comment><comment><USER39> What: Banks. With: lib-www-perl.</comment>


Google's Diversity Chief Responds to Controversial Leaked Anti-Diversity Memo - <AUTHOR> https://www.inc.com/emily-canal/google-responds-to-internal-memo-controversy.html?cid=hmside1 <comment><USER2> > "Distribution of preferences and abilities of men and women differ in part > due to biological causes and that these differences may explain why we don't > see equal representation of women in tech and leadership," the unnamed > engineer wrote in an essay that went viral within the company before leaking > online.

So, a random employee thinks that he has solved a problem that social sciences are trying to figure out. And that same random employee thinks that the rest of the world should hear his theory (or the team mates of the random employee ).

This is news because is about Google, otherwise, it will go unnoticed. The Internet is full of random's people opinions based on nothing, and here is mine for what is worth.</comment>


Ask HN: Review My App - Alchitect - <AUTHOR> http://alchitect.com

<comment><USER3> It looks like Alchitect is a PHP MVC-style web framework, but I had to click around quite a bit to figure that out.

Why not make the first sentence on the front page something like this:

"Alchitect is a MVC web framework written in PHP that does [ whatever it is that makes Alchitect special]"

<comment><USER4> Hold on mate, Alchitect does not seem very MVC from its Twitter timelines tutorial ( <http://alchitect.com/app/documentation/twitter_timeline> ) -- the HTML (V) is weaved deep inside the function (C).

Technicality aside, for a framework to be popular, it's necessary (though not sufficient) to have lots and lots of documentation/tutorials.

<comment><USER3> Well buddy, it is MVC according to the documentation, and the Twitter example itself uses the file AboutController.php. That's reason enough for me to give it that label.

It may not be _good_ mvc, or even a good framework. Of course, whether it's MVC or not has nothing to do with the point I was making: that websites should tell the visitor clearly what the website is about.

<comment><USER5> If it looks like a duck but wears a shirt that says "goose"...

</comment></comment></comment></comment><comment><USER6> I like the premise: To help non-coders make modern websites.

My project is similar in premise. I admit I haven't installed your framework but here are my thoughts.

I think you either are a coder or you aren't. I don't think its beneficial to have a middle road. Things like dreamweaver are said to allow anyone to "make a website". The kicker is you need to _learn_ how to work dreamweaver. All you are doing is trading one learning curve for another. It doesn't make sense to me. I'm not saying you are trying to be dreamweaver, the example is meant to illustrate the problem with targeting non-coders and telling them "hey you can make a website with my program".

The point of all this is that since my project involves the same premise, and after thinking long and hard about it, I realize that enabling non-coders to make websites is all about the UI. If you tell a guy that is just starting with code "hey you don't have to write any sql!!! Well he won't know what that means either way, so your proposed benefit flies right over his head.

I would say I am probably in your target userbase. I know how to code well enough to understand the benefits of your framework, but _since_ I know how to code well enough to understand your benefits, that basically means I'm already comfortable writing sql and enlisting MVC to make things easier on me. (I use kohana). So as much as I appreciate all the work you've put into your framework to help me, I am extremely hesitant to abandon kohana and learn alchitect.

I guess my point is about marketing. Figure out _exactly_ who you want to target, and make a product that fits their needs to the T.

In my case, my project targets non-coders. Therefore it must be 100% about the UI. Think of weebly.com That site is bleeding out of the nose with insane UI features.

I'm not saying in any way that this is your demographic, but rather to encourage you to pinpoint your customers!

Hope this helps.

</comment><comment><USER7> I personally think your installation demo would lead directly to a no sale. There are a few reasons. In your demo video 1) The first thing you show is an error. I understand why you do this, but saying to somebody 'here's the error that comes up when you first start' isn't a great experience. 2) I suspect that the installation looks overly complicated to anybody who is not already familiar with how to set-up a website and sql, and you even go on to show them how they can change their settings in text if they need to. If you made it so simple, why do they need that info. You also take the viewer to a command prompt, and look at their database tables. Non of this spells 'Unprecedented Ease' to me. 3) You don't show what making an actual web-page is like. I suspect that is going to be the most important thing for most people. They can probably get through the install (though I suspect you should be able to make it an better experience), but I want to know what it looks like to build a site.

Looking at your documentation on 'creating a page' 1) you say that a user entering a field in to a form must put the '_' character between words. If this is so simple, you should be doing that for the user. Don't let the user make a mistake, you're supposed to be taking care of them. 2) 'Now for the really interesting part' you say. Put this code in the form??? wait, if you're making this so easy for me, why do I need to know code? Plus this is now code that I would need to learn, so I am going to need to learn how you think it is best to implement.

Basically, similar to apsurd, I think you've gone middle of the road here. You've taken out some of the work for programmers who probably don't mind writing the code, and you haven't made it simple enough for non-coders, and you're forcing them to learn too much stuff.

</comment><comment><USER8> You may want to play with the name a bit, but it could be too late in your development for that - the first thing I thought of when I read "alchitect" was a site that I could tell all the kinds of booze I had in my closet, and it would give me some interesting mixed drinks (alcohol + architect). But that may just be my mind being somewhat recently out of college.

I would also change the video you have highlighted on the front page - <USER2> watching it, I have a good idea how to install _something_ , but I'm not 100% sure what I would have installed, or why I would have wanted to do that. Based on the description at the top, I think Alchitect is some kind of web framework, but I'm not clear on what makes it special, distinctive, or even what language/platform it's for.

As opposed to an install video, I'd make the main front-page video into something like one of the Rails videos - "Watch me go from nothing installed to a basic CRUD MySQL site in under two minutes" - and highlight along the way what makes your framework different from every other one.

</comment><comment><USER9> Great Job on the short installation manual video, it seems pretty straigforward to get started. A Demo of what Alchitect can be used for would be great. Also, I couldn't find the other Video Tutorials mentioned in the introduction. Good start...

</comment><comment><USER10> Zelphyr, make sure that you have showdead turned on in your HN profile, as there is a comment by JimmyL which may be helpful.

</comment><comment><AUTHOR> Great suggestions! Thanks all!</comment>


Zoncolan: How Facebook uses static analysis to detect, prevent security issues - <AUTHOR> https://engineering.fb.com/security/zoncolan/ <comment><USER2> Does this happen to use SPARTA ([https://github.com/facebookincubator/SPARTA](https://github.com/facebookincubator/SPARTA)) for its abstract interpretation?

<comment><USER3> Zoncolan and SPARTA both use abstract interpretation at their core, but as far as the code/implementation goes they are mostly unrelated. Zoncolan is built on top of the Hack type checker ([https://hacklang.org/](https://hacklang.org/)), which means that it's largely written in OCaml.</comment></comment>


Ask HN: What Setup You Use? - <AUTHOR>

<comment><USER2> threads exactly like this:

\- [https://news.ycombinator.com/item?id=6170156](https://news.ycombinator.com/item?id=6170156)

\- [https://news.ycombinator.com/item?id=1507695](https://news.ycombinator.com/item?id=1507695)

but you can also tell us what kind of setup you're looking for. vim setup? emacs setup? computer setup?

</comment><comment><USER3> Sublime Ubuntu Google Chrome (trying to phase into Firefox)

I use Python and Javascript

</comment><comment><USER4> Vim Ubuntu Google Chrome

I use C and Coffeescript.</comment>


UK: Snowden reporter's partner involved in 'espionage' and 'terrorism' - <AUTHOR> http://www.reuters.com/article/2013/11/01/us-uk-nsa-idUSBRE9A013O20131101

<comment><USER2> The LAX shooter, on the other hand, apparently was not.

<comment><USER3> He was a criminal that should be charged with crimes and be prosecuted in accordance with due process. Throwing around the term terrorist loosely gets us no where as a nation. It just fuels the mass hysteria that the war on terror has become and contributes further to the erosion of our civil liberties.

<comment><USER2> The fact is that all terrorist acts are criminal. The LAX shooter had a note on him that described his act as political, while Miranda did not kill anyone.</comment></comment></comment>


Try our app (Flowdock, the team messenger) - <AUTHOR> http://blog.nodeta.fi/2010/03/10/flowdock-launches-public-beta/

<comment><USER2> I started using it this morning. After my session auto-expired, I can no longer log back in. It seemed useful until that happened, but lack of reliability is a deal-breaker.

</comment><comment><USER3> This is beyond awesome! Perfect interface, perfect mix of Yammer and IRC and data importing, perfect across the board.

Are you guys going to have a Yammer import feature? If you could import all our past messages from Yammer into that right-hand column then our team would switch in a tenth of a second flat.

<comment><USER4> Hi! Thanks for the feedback. We're sure planning to make the conversion as easy as possible, and Yammer import is on top of our list. We will announce new features in our blog and on Twitter, stay tuned. :)

<comment><USER5> I'm also super impressed. You could have gone in any number of crazy directions but you kept it simple (dashboard, incoming, searches). I love it. It's like Campfire on steroids.

Personal feature request: preview youtube URLs, images, and tweets pasted into the chat area. Less important: let people select a range of chat messages and tag en-masse so they show up together in a search.

<comment><USER6> Those are both awesome suggestions that I enthusiastically second. Especially the paste-preview thing, that's practically a must-have.

</comment></comment></comment></comment><comment><USER7> "its what google wave should have been"?

i would be differentiating yourself from wave as much as possible... you don't want to be competing with google when wave gets released sooner or later.

<comment><AUTHOR> (Actually Wave was announced and released to beta while we were developing Flowdock so it's was really interesting for us.)

Yeah, it's true that we should differentiate ourselves from Wave as much as possible, but the comment "It's what Google Wave should have been", means that for some people Flowdock is what they expected or wanted from Wave.

The problem I see with Wave is that it's trying to be everything for everyone, which is a really ambitious goal. With Flowdock we just want to provide tools for teams that work constantly and intensively together.

<comment><USER7> don't underestimate google. it's probably safe to assume they know _exactly_ what they are doing.

</comment></comment></comment><comment><USER8> This looks really good. What would make it great is if the api allowed information to go in both directions. That is, it looks like right now I can have external information listed and then talked about with Flowdock. But!, can I then have the flowdock discussions sent back to the external source if said source accepts it? (For example, I have a workflow system where an external source lists current workflows, but I need all the discussion about those workflows sent back to the source system in some form)

</comment><comment><USER9> App icons looks a little aliased in Chrome running in Ubuntu: <http://dl.dropbox.com/u/121630/flowdock_raw.png>

edit: Oh, and how much time does it take for a "session to expire"? Mine expired in 5 minutes of innactivity or so...kind of annoying if you're waiting for a team mate that went to the bathroom for example.

<comment><USER4> Thanks, we're working on the icons.

Sessions do not actually expire anytime soon - we just deployed a bugfix release and had to restart the backend. We'll make the deployment experience a bit smoother soon. :)

<comment><USER10> I'm guessing that you are doing something right now since I just get a "500 Internal Server Error" every time I try to go to the flow.

</comment></comment></comment><comment><USER11> Looks good, playing with it with one of my teams. I set it up in fluid, but when I try and login via fluid, it seems to send me into my default browser to open the page that lets me select which flow I want to use.

One feature that would be nice is not matching search on twitter streams... ie +term1 -term2, would help with signal to noise ratio for what would be interesting for my teams.

<comment><USER4> This is a problem with Fluid's default settings. You can fix it by going to Preferences -> Advanced -> Add a new pattern: _flowdock.com_

We'll try to document this process a bit better.

</comment></comment><comment><USER12> Tried to signup .. Got 500 Internal Server Error. :(

<comment><USER13> I'll bet you have NoScript switched on?

<comment><USER12> I don't. I'm just using a vanilla Chrome install on Ubuntu. I do have Flash installed, don't see why that would matter in this case, though.

</comment></comment></comment><comment><USER14> Looks interesting, will certainly check it out. One of the weaknesses of other project tracking tools for Victus Media has been the speed at which we change priorities. Continual user feedback helps us manage where we want to go long term, versus what we hack out tonight.

Would appreciate any tools that change as fast as we do, while keeping th big picture highlighted

</comment><comment><USER15> Haven't used the app yet, but one thing I would suggest is re-doing your voiceover on your very prominent product demo video and making cuts between demoing key features. I can understand your accent just fine (although it couldn't hurt to have someone with a very neutral english accent do the voiceover) but the quality of the audio is really poor.

<comment><USER16> Absolutely. The audio is so quiet in your voiceover that even max volume on my laptop speakers and Youtube slider I can't understand a word you're saying. You need to normalize the volume level using any sound editor, and better yet, use a good mic.

If we can't understand what you're saying, how likely are we to use your product?

<comment><AUTHOR> Thanks for pointing this out. Apparently making screencasts is not really our strongest asset. We try to make a better one later.

</comment></comment></comment><comment><USER17> I recommend a test team that people can join.

You'll have to think of a way to prevent it from becoming a wasteland of "asdlkfjsakj" posts though.

Looks good!

<comment><USER18> I would really appreciate that, because I had no clue what organization to enter in the sign up form (will I be able to change it later? can I sign up again for another organization with the same email?). Moreover, I didn't understand how should one use this application within the 3 minutes I invested :)

</comment></comment><comment><USER19> Loved it when I tried it this morning, but the [apparent] overload of the servers led to my team dismissing it without a second thought.

</comment><comment><USER20> nice work guys...interesting to see a different take on the Flow as a team collaboration medium. Also, Love the flow based puns like "Flowser". At Shareflow we call the creator of the flow the "Flowner".

<http://zenbe.com/shareflow>

<comment><AUTHOR> Thanks, nice job with Shareflow as well.

After we decided with the name it was quite funny how many 'flow' products were out there :)

</comment></comment><comment><USER21> Is it me or does the nodeta image on the left look like a bong at first sight?

<comment><USER22> You've probably seen too many bongs. I didn't make the connection.</comment></comment>


Seems like YouTube stops supporting Firefox on Linux - <AUTHOR> https://www.reddit.com/r/linux/comments/h7tnfu/seems_like_youtube_stops_supporting_firefox_on/ <comment><USER2> If you read the comments, this appears to be due to people changing privacy.resistFingerprinting in about:config and/or Google breaking things.</comment>


Contractual: Design by contract in JavaScript - <AUTHOR> http://codemix.github.io/contractual/

<comment><USER2> I like the idea, but I am not a fan of the compile-step.

If there was an alternative implementation of JavaScript with labels used only in this manner (pre-defined meaning, no usage otherwise) they would actually be a good practice.

<comment><USER3> Sadly the compile step is unavoidable if you want the contractual guarantees, but it's possible to use Contractual JS directly, without compiling it, so you could potentially avoid compilation during actual development.

</comment></comment><comment><USER3> author here, this was my easter hack. You might find the online demo a bit more accessible: [http://codemix.github.io/contractual/try.html](http://codemix.github.io/contractual/try.html)

</comment><comment><USER4> interesting hack</comment>


Bootstrapping is thinking long, not small - <AUTHOR> http://projectidealism.com/posts/2014/1/27/bootstrapping-is-thinking-long-not-small

<comment><USER2> I couldn't agree more. I'm glad the OP took the time to write this up, as I've been thinking something very similar recently. In fact, as recently as earlier today, I saw some comment about bootstrapping that hinted at this "bootstrapping == small" thing.

Sorry, but that's just wrong. TFA here nails it... going the bootstrap route doesn't mean you don't intend to build a billion dollar business, it just means that, if you do, you plan to do it on your schedule, not a VC's. That's it.

FWIW, we're proudly bootstrapped / self-funded at Fogbeam Labs, and anybody who knows me can tell you that I don't do small dreams. As far as I'm concerned, we're on our way to being a public company with a multi-billion dollar market cap. I can't tell you when we'll get there, but that's the goal.

Now, that said... I don't think "bootstrapped" or "not bootstrapped" is something that's intrinsic and unchangeable. We haven't taken any outside money yet, but I'm not saying we _never_ will. We might do it tomorrow if the circumstances dictate that it's the right thing to do, relative to our goals. Or maybe next year. Maybe never. Outside money is just a tool, like a screwdriver or an Open Source library... we may need that particular tool, we may not. If we do, we'll use it. If not, we won't use it just for the sake of using it.

<comment><USER3> Thanks for sharing. I particularly like your comment, "We haven't taken any outside money yet, but I'm not saying we never will."

It's also important to really believe in what you're doing, funded or not. You don't get a special medal for taking money or by bootstrapping. You're still a business and you're all competing.</comment></comment>


Practical Type Inference Based on Success Typings (2006) [pdf] - <AUTHOR> http://www.it.uu.se/research/group/hipe/papers/succ_types.pdf <comment><USER2> _Changing the philosophy of a programming language, especially one with existing applications of considerable size, is not a task with a high likelihood of success. In this paper, rather than starting from a static type system and trying to squeeze Erlang into it, we followed a different approach._

I've always wondered why people still try to shoehorn type systems into dynamic languages. So, why isn't there a Dialyzer for Ruby or Python or JS?

<comment><AUTHOR> There are type inspection tools for JS - Webstorm has one and then there is the open source tern.js and facebook's flow. I imagine in erlang it's a bit easier to infer types in erlang because it's immutable and because functions don't just take whatever amount of arguments (like in JS). I'd say erlang is not as dynamic as JS.

The research around erlang is due to the fact that it's heavily backed by Ericsson and the university in Uppsala (it's common for Uppsala master theses to be erlang-related). The other languages have more "neutral" origins.

</comment></comment><comment><USER3> Dialyzer is a really nice tool for Erlang. I think Python might be getting something similar in the coming revisions but Erlang has had it for a while.

Here is a more concise and approachable intro on the topic:

[http://learnyousomeerlang.com/dialyzer](http://learnyousomeerlang.com/dialyzer)</comment>


IPad taken apart then re-assembled in under four minutes - <AUTHOR> http://www.youtube.com/watch?v=Fs79QEbWrLc

<comment><USER2> Just to clarify? It's not a speed-assembly video. Four minutes just happens to be the length of a video that contains stop-motion animation of an iPad being "taken apart and then re-assembled".

</comment><comment><USER3> Maybe it's me, but that video was painfully campy. Yet I couldn't look away.

<comment><USER4> The sound effects were over-the-top; I had to turn it off.

</comment></comment><comment><USER5> More accurate title:

IPad taken apart then, in under four minutes, re-assembled

Which shouldn't be too surprising - it was designed to be quick to assemble - AKA manufacture.

<comment><USER6> A more un-misleading title:

IPad taken apart and re-assembled.

Four minutes is the length of the video and has nothing to do with how long it took to do the actual work.

</comment></comment><comment><USER7> What's the point? People are doing this with assault rifles, as maintenance. But with iPad? Heh.</comment>