Blacklisted as Malware: a Downside of using Amazon EC2 - <AUTHOR> http://denis.papathanasiou.org/?p=165

<comment><USER2> I recently found <http://sendgrid.com/> which is designed to solve the email sending problem from EC2, and includes a REST API.

<comment><USER3> The founders Isaac and Jose were in Techstars with me this summer, and I highly recommend SendGrid. They've got an awesome plugin architecture too that's an immense time saver for things like adding and managing unsubscribe links.

</comment></comment><comment><USER4> This is a very large problem that Amazon needs to spend some time addressing.

I run a fairly large production site on EC2 (14 instances) and when we went into production we quickly found out we were unable to send email from our servers to anyone... just about every email provider has blacklisted EC2's address space.

I literally have to run an "off site" server from Amazon as a mail relay to work around this issue.

Word of warning, EC2 is awesome, but this is an important lesson learned if you are moving to the cloud.

</comment><comment><USER5> The blacklisting you experienced, could have happened with any other host. I've seen this happen with dedicated hosting companies, where a server was previously used by spammers and the ips allocated to a new client. When they try to send mail, guess what they're on all the major rbls. As far as sending mail out from the EC2 environment, that's an on going problem. Take a look at this <http://www.spamhaus.org/sbl/sbl.lasso?query=SBL79954> . I think AWS users would be willing to pay for clean ips that are rate limited. Maybe an EC2 SMTP Service. Your block is using a browser bl and I don't know how those are shared by http filters.

</comment><comment><USER6> I wish the issue was explained a bit more in the article. In my opinion if the blocking was due to host-names then it had nothing to do with EC2. If the blocking was due to re-use of IP addresses then maybe it had something to do with EC2 (but the article didn't explicitly claim that). As far as I can tell the effect was due to using a outsider server at all (instead of carrying one into the client's own private data center), which I think has nothing to do with having used EC2.

<comment><AUTHOR> It was based on IP address.

You're right in that all hosting services suffer this problem to some degree, but it's a particular problem on EC2 since it's so easy to request IP for an instance that does something bad, then release back into the pool for other instances to use, a short time later.

It only costs a few dollars to that on EC2, whereas most other hosting services require at least a month's commitment and higher fee(s) before you can defile one of their IP addresses.

BTW, the reason I posted was to find out what other blacklists might be out there, so if your company is blocking access, I'd appreciate letting me know which service they're using.

<comment><USER7> Are you sure it was based on IP address? This domain was used to host questionable links a few years ago, including suspicious antivirus software:

<http://web.archive.org/web/*/workstax.com>

[http://web.archive.org/web/20070127212047/http://workstax.co...](http://web.archive.org/web/20070127212047/http://workstax.com/)

<comment><AUTHOR> You're right about the prior activity for the domain, but both Blue Coat and McAfee complained based on the IP address we're using now (i.e., the EC2 address).

But my purpose in posting this is to ask: what other filtering lists are out there?

Getting ourselves off is easy now, since we are a legitimate site; the hard part is figuring out who else may be blocking us.

</comment></comment></comment></comment><comment><USER8> You can try something like <http://www.mxtoolbox.com/blacklists.aspx> , it checks a bunch of RBLs all at once.

</comment><comment><USER9> I think this speaks more to the low quality of these blacklists than to a problem at Amazon. Did anyone who noticed the offending site contact Amazon to have it removed, or are they simply content to keep adding more blocked IPs as the offender jumps from server to server on EC2?

</comment><comment><USER10> Amazon needs to check its IPs against the blacklists before releasing them back into the AWS pool.

<comment><USER11> Seems like this points to a longer term problem with blacklisting by IP address as they continue to become more transient with respect to the hosted service.

<comment><USER3> There's a similar problem with some API providers like Twitter throttling usage from App Engine and EC2 IP ranges: <http://news.ycombinator.com/item?id=793939>

</comment></comment></comment><comment><USER12> what if you get an amazon elastic IP which isn't blacklisted?

</comment><comment><USER13> define(`SMART_HOST', `some.host.with.a.clean.ip')dnl

<comment><USER14> When creating mail severs for testing purposes, we too have noticed that many of the IP addresses in the AWS pool are indeed on blaclists or private reputation lists. A smarthost is certainly recommended if you want mail from one of these IP addresses to make it past spam filters. In theory, once you've had the IP address for a while, you should be able to get it delisted and cleared up, but some of these lists and toosl will just block entire subnets that are known to frequently change hands. I think the only real solution would be for Amazon to set aside some IP space and call it premium, and only give it out to validated or longstanding customers, IMHO.

@MxToolbox

</comment></comment><comment><USER15> I am the founder of <http://critsend.com>, we make your email ends up into its appropriate inbox.

We have a special offer for EC2 customers, we have some servers directly at AWS so latency is low and bandwidth is free for them.

We are currently in closed beta but if you subscribe and add say you saw us on Hacker News, we will send you an invite asap. Our first 1,000 emails are free and after that we are pay per use.</comment>


Ask HN: who is still using Opera on their iPhone? - <AUTHOR>

It's been a little while now since it was released: who is still using it?
Just out of interest after the large amount of publicity it had.
For my part I left it installed to see if they updated/fixed all the issues (which, for me, were show stoppers) - and nothing new as yet... <comment><USER2> I use it to catch up on the news during my morning bus ride. There are bugs, but the small speed and interface differences compared to Mobile Safari keep it preferable for scanning lots of text in lots of small articles.</comment>


NASA researches windowless cockpits; future businessjets might employ - <AUTHOR> http://www.aviationweek.com/aw/generic/story_channel.jsp?channel=defense&id=news/FA18101008.xml&headline=Blind%20Aft-Cockpit%20F-18%20Points%20To%20Biz%20Jet

<comment><USER2> BSoD

<comment><AUTHOR> Indeed.

<http://images.google.com/images?q=aircraft+birdstrike>

</comment></comment><comment><AUTHOR> Article excerpt:

 _Testing is under way of an F-18 with a covered aft cockpit to help prove that a camera-equipped aircraft with no forward view windows could be a viable option for a future supersonic business jet.

The external vision system tests involve Gulfstream Aerospace and are taking place at NASA Dryden Flight Research Center at Edwards Air Force Base, Calif., using the agency's F-18 systems research aircraft (SRA). The aircraft's aft cockpit canopy is shielded with only small cut-outs to represent side windows, while the forward view is taken up with a a 22-inch high-definition (HD) display. Gulfstream is exploring supersonic business jet configurations affording little or no forward view.

The liquid crystal display, bolted in place over the glareshield, is being used to display an image from an HD camera mounted behind the head-up display in the front cockpit. A safety pilot flies in the front cockpit, ready to assume control if the situation warrants, while the mission is flown from the aft cockpit by a pilot using only instruments and the visual display on the external vision system. ...

Five flights have taken place since mid-September, and the work is expected to culminate with three night flights around the end of October._</comment>


Ask HN: My niece wants to learn about "computing". Where do I start? - <AUTHOR>

Hey all,
My niece asked if I could teach her about computing. She's about 8, but she's vehemently opposed to so-called "fluffy stuff" - her words. I was thinking I'd just bust out the Wizard book and work on it with her and her brother, but perhaps it may be too abstract?
She seems more interested in thinking about computing than programming games, etc; so any recommendations on where to start? <comment><USER2> take her to the bookstore and show her books on various topics, find out what she's interested in particular, then take it from there. i wouldn't avoid showing her any book or resource. if she gets it great, if not no harm done.</comment>


Silicon Valley Episode 1  - <AUTHOR> https://www.youtube.com/watch?v=VvkmsI54ss4

<comment><USER2> Not available in the UK. Region lock for online content is stupid, I get it all the time trying to watch Comedy Central clips - if anything it has made me less likely to ever give them money.

</comment><comment><USER3> I find it interesting that HBO is allowing YouTube to stream this for free - is this a strategy they normally employ?

<comment><USER4> It looks like this is on HBO's youtube channel so not only are they allowing it, but they are the ones streaming it for free. It makes sense to stream premier episodes for new series if it means they can hook people into the show and possibly bring in more HBO subscribers.

<comment><AUTHOR> They have done this before for new shows.

</comment></comment></comment><comment><USER5> surprisingly accurate portrayal of SV, and hilarious</comment>


Norway first with battery-ferry, now mandatory where possible. Hydrogen is next - <AUTHOR> https://translate.google.com/translate?sl=auto&tl=en&js=y&prev=_t&hl=no&ie=UTF-8&u=http%3A%2F%2Fwww.tu.no%2Fartikler%2Fi-2015-ble-norge-forst-ut-med-elferge-na-skal-ny-milepael-nas%2F358972&edit-text=&act=url <comment><USER2> Current ferry, Ampere, is 2 x 450kW (600HP) motors, 1000kWh of batteries, and makes 34 trips of 5.7km. Aluminum. NICE! [http://www.ship- technology.com/projects/norled-zerocat-elect...](http://www.ship- technology.com/projects/norled-zerocat-electric-powered-ferry/)

</comment><comment><USER3> Electric ferry- battery _powered._ They plan to develop hydrogen fueled ferries- but may start in passengerless freighters until the safety stuff is worked out.</comment>


Making Use of a Low PageRank Site - <AUTHOR> http://contentini.com/why-you-need-an-auxiliary-low-pagerank-site-content-strategy-tip/ <comment><USER2> I fail to see his point as to why the site would need to maintain a low PageRank. The only time he even addresses that point is with the following:

 _By purposefully maintaining a low ranking site, which has good on-page SEO (titles and headings), you can create a regular stream of small posts relevant to your subject matter, without having to create extensive, high-effort articles that your site/brand is known for._

But how does the PageRank of the site _change_ your intentions for the type of posts you write? I would have just played it off as trying to make the post more popular by giving it a counter-intuitive theme, but then at the end of the post, he writes:

 _...please don't link to it! It needs to maintain a low ranking to serve its original purpose._

So unless I'm misunderstanding, it sounds like he's saying, create a site full of trivial posts, but make sure that it maintains a low PageRank, so that it doesn't become popular and hurt your brand.

Why would it hurt your brand? Because you're not actually contributing anything useful. But if you're not contributing anything useful, then no one will link to it.*

The corollary is that if a lot of people link to it, you must actually be providing something useful. So you really have nothing to worry about.

* One caveat to that is if you write a meta-post about that site explaining its purpose and then a bunch of people link to it for that reason alone.

<comment><AUTHOR> Sorry, I guess I wasn't clear enough in the article.

The point of the low PageRank is that it identifies the low-competition keywords/key-phrases. i.e. If you have a HIGH PR site, then you can rank pretty well for any key phrases, but if you have a LOW PR site, then you'll only rank for the key phrases that nobody else ranks for.

So, as you regularly post a range of topics on a low PR site, you'll only see search referrer traffic coming through for the subjects that have a 'gap'.

It's not really about the brand, that's just an aside.

Also, of course, ranking on Google is made up from hundreds of factors; PR is just one of them, but it's indicative of lots of the other factors (number, quality and diversity of incoming links).

<comment><USER2> Thanks for the clarification, that makes more sense.

I think you overestimate the difficulty in ranking for keywords for which many others already rank, though. For instance, my relatively new consulting website ranks on the front page for common words/phrases, like "phusion passenger," and it only has a PR3.

Your article states that PR3 is low enough, but I would say that if you have good in-page SEO, etc, you really need a PR1 or PR2 to get the effect you're talking about.

</comment></comment></comment><comment><USER3> This is the concept behind "Being Yourself for a Living": [http://www.kadavy.net/blog/posts/be-yourself-for-a-living- th...](http://www.kadavy.net/blog/posts/be-yourself-for-a-living-the-vision/)

If you just blog about everything you know and experience, eventually, some surprises will pop up in your Analytics. By virtue of the fact that you know something about these topics, they are viable for you to pursue further.</comment>


Problem solving with Unix commands - <AUTHOR> http://vegardstikbakke.com/unix/ <comment><USER4> Gary Bernhardt[1] gave a great talk about practical problem solving with the unix shell: "The Unix Chainsaw"[2].

"Half-assed is OK when you only need half of an ass."

In the talk, he gives several demonstrations a key aspect of _why_ unix pipelines are so practically useful: you build them _interactively_. A complicated 4 line pipeline started as a single command that was gradually refined into something that actually solves a complicated problem. This talk demonstrates the part that isn't included in the the usual tutorials or "cool 1-line command" lists: the cycle of "Try something. Hit up to get the command back. Make one iterative change and try again."

[1] You might know him from his other hilarious talks like "The Birth & Death of JavaScript" or "Wat".

[2] [https://www.youtube.com/watch?v=sCZJblyT_XM](https://www.youtube.com/watch?v=sCZJblyT_XM)

<comment><USER5> > In the talk, he gives several demonstrations a key aspect of why unix > pipelines are so practically useful: you build them interactively.

The standard Unix interface might have been interactive in the '70s, back when hardware and peripherals were horribly non-interactive. But I don't know why so many so-called millenial programmers (people my age) get excited about the alleged interactivity of the Unix that most people are familiar with. It doesn't even have the cutting edge '90s interactivity of Plan 9, what with mouse(!) selection of arbitrary text that can be piped to commands and so on. And every time someone comes up with a Unix-hosted tool that uses some kind of fold-up menu that informs you about what key combination you can type next (you know, like what all GUI programs have with Alt+x and the file|edit|view|… toolbar), people hail it as some kind of UX innovation.

<comment><USER6> I think the interactivity you describe might be a different thing from what your parent is talking about.

From what I understand, your parent talks about how the commands are built iteratively, with some kind of trial-error loop, which is a strength that is supposedly not emphasized enough. And I agree by the way. Nothing to do with how things are input.

<comment><USER4> That's correct. Articles/tutorials or an evangelizing fan often show the end result: the cool command/pipeline that does something cool and useful. The obvious question when someone unfamiliar with unix upon seeing something like the pipeline in this article:

        comm -1 -3 <(ls -1 dataset-directory | \
                     grep '\d\d\d\d_A.csv'   | \
                     cut -c 1-4              | \
                     python3 parse.py        | \
                     uniq                      \
                     )                         \
                   <(seq 500)

is "Why would I want to write a complicated mess like that?" Just use ${FAVORITE_PROG_LANG:-Perl, Ruby, or whatever}". For many tasks, a short paragraph of code in a "normal" programming language is probably easier to write and is almost certainly a more robust, easier to maintain solution. However, this assumes that you knew what the problem was and that qualities like maintainability are a goal.

Bernhardt's (and my) point is that sometimes you don't know what the goal is yet. Sometimes you just need to do a small, one-off task where a half-assed solutions might be appropriate... _iff_ it's the right half of the ass. Unix shell gets that right for a _really useful_ set of tasks.

This works because you are free to utilize that powerful features incrementally, as needed. The interactive nature of the shell lets you explore the problem. The "better" version in a "proper" programming language _doesn 't exist_ when you don't yet know the exact nature of the problem. A half-assed bit of shell code that slowly evolved into something useful might _be_ the step between "I have some data" and a larger "real" programming project.

That said, there is also wisdom in learning to recognize when your needs have outgrown "small, half-assed" solutions. If the project is growing and adding layers of complexity, it's probably time to switch to a more appropriate tool.

<comment><USER5> I generalized interactivity to the Unix that most people seem familiar with.

"The interactive nature of the shell" isn't that impressive in this day and age. Certainly not shells like Bash (Fish is probably better, but then again that's very cutting edge shell ("for the '90s")).

Irrespective of the shell this just boils down to executing code, editing text, executing code, repeat. I suspect people started doing that once they got updating displays, if not sooner.

<comment><USER7> How is that not impressive for vast majority of developers?

For the past couple decades, the only other even remotely mainstream place where you could get a comparable experience was a Lisp REPL. And maaaybe Matlab, later on. Recently, projects like R, Jupyer, and (AFAIK) Julia have been introducing people to interactive development, but those are specific to scientific computing. For general programming, this approach is pretty much unknown outside of Lisp and Unix shell worlds.

<comment><USER5> The author is an MS student in statistics. Seems that Unix is well-represented in STEM university fields.

Old-timey Unix (as opposed to things like Plan 9) won. When does widespread '70s/'80s computing stop being impressive? You say "unknown" as if we were talking about some research software, or some old and largely forgotten software. Unix shell programming doesn't have hipster cred.

<comment><USER7> > _When does widespread '70s /'80s computing stop being impressive? You say > "unknown" as if we were talking about some research software, or some old > and largely forgotten software._

That's precisely what I'm talking about. The 70s/80s produced tons of insight into computer use in general, and programming in particular, that were mostly forgotten, and are slowly being rediscovered, or reinvented every couple years. Unix in fact was a step backwards in terms of capabilities exposed to users; it won because of economics.

<comment><USER8> Had Bell Labs been allowed to explore UNIX commercially, none of us would be having this discussion.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> This [0] the most complete post I've read on the topic. Lays out all the relevant tools. Spending some time going through each tool's documentation/options, pays off tremendously.

[0]: [https://www.ibm.com/developerworks/aix/library/au- unixtext/i...](https://www.ibm.com/developerworks/aix/library/au- unixtext/index.html)

<comment><USER10> Wow, great find. Sad how hard it seems these days to come across an easy-to- follow primer on a topic without narrative fluff and/or ads everywhere. For those interested in a standalone copy there is a PDF of the content available here [https://www.ibm.com/developerworks/aix/library/au- unixtext/a...](https://www.ibm.com/developerworks/aix/library/au-unixtext/au- unixtext-pdf.pdf)

<comment><USER11> Writing clear tutorials is a fair amount of effort, more than I originally thought when I first did it.

</comment></comment></comment><comment><USER12> The brilliant fun of working with the Unix CLI toolset is that there are millions of valid ways to solve a problem. I also thought of a "better" solution of my own that took an entirely different approach than most of the ones posted here. That's not really the point.

What's great about this article is that it follows the process of solving the problem step by step. I find that lots of programmers I work with struggle with CLI problem solving, which I find a little surprising. But I think it all depends on how you think about problems like this.

If you start from "how can I build a function to operate on this raw data?" or "what data structure would best express the relationship between these filenames?" then you will have a hard time. But if you think in terms of "how can I mutate this data to eliminate extraneous details?" and "what tools do I have handy that can solve problems on data like this given a bit of mungeing, and how can I accomplish that bit of mungeing?" and if you can accept taking several baby steps of small operations on every line of the full dataset rather than building and manipulating abstract logical structures, then you're well on your way to making efficient use of this remarkable toolset to solve ad hoc problems like this one in minutes instead of hours.

</comment><comment><USER13> If you bother to write a python script to parse the integers, why not use python to solve the whole problem?

<comment><USER14> This is one of the many reasons I think PowerShell did UNIX philosophy better: you don't need to parse text because the pipelines pass around typed objects. You can kinda almost get the same behavior from some UNIX commands by first having them dump everything into JSON and then having the other end parse the JSON for you, but you're still relying on a lot of text parsing. Personally I think it is high time the UNIX world put together a new toolset.

<comment><USER12> Why replace your hammers, screwdrivers, and chisels just because someone invented a 3D printer? They have tradeoffs. Powershell has some good ideas, and benefits from having been invented altogether, rather than evolving over four decades. But in practice it's not as efficient for doing simple things. It's oriented towards much more complex data structures, which is great... but there's no need to throw out your simpler tools just because you think they look ugly.

<comment><USER14> They're full of footguns, esoteric behavior, and have arcane names. They're actually pretty awful tools now that it isn't the 70s anymore.

<comment><USER15> Yet when I visit the unix sysadmins' office I see people chaining commands to administer hundreds of boxes. On the windows side I rarely see powershell prompts. Powershell looks so much better in theory. However it's just an okayish scripting language with a good REPL. Unix tools are a far better daily driver.

<comment><USER14> That's probably because for daily tasks we have much better tooling in Windows already that doesn't require us to use the command line and interactive construct it. I can easily administer the configurations of thousands of computers through AD, for instance, and while I could use PowerShell to do so, using ADUC is just easier most of the time.

If you do a lot of work with Exchange though, you'll probably end up using PowerShell much more, because the web UI for it is not so great.

No matter what you think of the specific implementation, a lot of PowerShell's ideas are good ideas. Unfortunately UNIX culture is such that they'll probably never implement any of them.

<comment><USER15> > That's probably because for daily tasks we have much better tooling in > Windows

ssh, docker, ansible, kubernetes, grafana, prometheus, etc... All coming from Linux/unix. This statement is clueless. Most of the cloud is not running microsoft, and for a good reason.

To automate, we have python, which has a much better syntax. It's pointless to use powershell.

And it takes a microsoft head, without knowledge of programming language's history to say that powershell's ideas actually come from powershell. Method chaining/fluent interface with a pipe instead of a dot does not look that new.

Also, some attempts have been to implement posh clones on unix. Being redundant with either perl/python or bash/zsh, none succeeded.

</comment></comment></comment></comment></comment></comment></comment><comment><USER16> Removing leading zeroes doesn't require Python. One easy solution would be sed:

        $ echo -e '0001\n0010\n0002' | sed 's/^0*//'
        1
        10
        2

<comment><USER17> Yeah plus seq can generate sequences with leading zeroes (something like seq -f %04.f 1 20).

So instead of scripting, he could have generated a sorted list of numbers from the files he had. Created a file with the sequence of numbers for the range and diffed/commed the whole thing. Voilà...

<comment><USER18> The seq provided in the GNU toolset has a -w flag to turn on "equal width" mode, so one can also get zero padded numbers out (from GNU seq) by turning on that mode and zero padding the input:

        $ seq -w 0001 0003
        0001
        0002
        0003
        $

<comment><USER19> Nice, this works automatically like this:

        $ seq -w 98 102
        098
        099
        100
        101
        102

</comment></comment></comment></comment><comment><USER20> A change in structure might be helpful:

        $ ls data
        0001.csv 0002.csv 0003.csv 0004.csv ...
        $ ls algorithm_a
        0001.csv 0002.csv 0004.csv ...
        $ diff -q algorithm_a data |grep ^Only |sed 's/.*: //g'
        0003.csv ...

<comment><AUTHOR> Excellent point, haha!

</comment></comment><comment><USER21> For learning to get things done with Unix, I recommend the two old books "Unix Programming Environment" and "The AWK Programming Language". There are many resources to learn the various commands etc., but there is still no better place than those books to learn the "unix philosophy". This series is also good:

[https://sanctum.geek.nz/arabesque/series/unix-as- ide/](https://sanctum.geek.nz/arabesque/series/unix-as-ide/)

</comment><comment><USER22> I think the best part about using Unix tools is it forces you to break down the problem into tiny steps.

You can see feedback every step of the way by removing and adding back new piped commands so you're never really dealing with more than 1 operation at a time which makes debugging and making progress a lot easier than trying to fit everything together at once.

<comment><USER23> It's basically functional programming. I find that my approach to writing code is very similar to how I work with the shell. The main difference, I guess, is that the command 'units' are slightly bigger, in the form of functions, but the way I iterate my solution to a problem is basically the same.

</comment></comment><comment><USER24> I've often done this, usually not for a large dataset, but it's sometimes helpful to pipe text through Unix commands in Emacs. C-u M-| sort, for instance, will run the selection through sort and replace it in place.

If you're going the all python route, and even want to be able to run bash commands, and want something where you can feed the output into the input, I'd strongly recommend jupyter. (If you want to stay in a terminal, ipython is part of jupyter and heavily upgrades the built-in REPL and does 90% of what I'm mentioning here.)

You can break out each step into its own cell, save variables (though cell 5 will be auto-saved as a variable named _5) but the nicest thing is you can move cells around (check the keyboard shortcuts) and restart the entire kernel and rerun all your operations, essentially what you're getting with a long pipeline, only spread out over parts. And there are shortcuts like func? to pop up help on a function or func?? to see the source.

It's got some dependencies, so I'd recommend running it in a virtualenv via pipenv:

        pipenv install jupyter  # setup new virtualenv and add package
        pipenv run jupyter notebook
        pipenv --rm  # Blow away the virtualenv

Also, look into pandas if you want to slurp a CSV and query it.

<comment><USER25> I doubt you'll find many Emacs users that would prefer "C-u M-| sort" over "M-x sort-lines".

</comment></comment><comment><USER26> The problem with this is that there isn't a standard format forced on the args that following the command name "cut".

What makes it worse is that there are seemingly patterns of standard format that get violated by other patterns. It's often based on when the utility was first authored and whatever ideas were floating around during the time. So sometimes characters can "clump" together behind a flag, under the assumption that multi-character flags will get two hyphens. Then some utilities or programs use a single flag for multi-character flags. Plus many other inconsistencies-- if I learn the basic range syntax for cut do I know the basic range syntax for imagemagick?

Those inconsistencies don't _technically_ conflict since each only exists in the context of a particular utility. But it's a real pain to sanity to see those inconsistencies sitting on either side of a pipe, especially when one of them is wrong. (Or even when it's a single command you need but you use the wrong flag syntax.) That all adds to the cognitive load and can easily make a dev tired before its time to go to sleep.

Oh, and that language switch from bash to python is a huge risk. If you're scripting with Python on a daily basis it probably doesn't seem like it. But for someone reading along, that language boundary is huge. Because the user is no longer limited to runtime errors and finicky arg formatting errors, but also language errors. If the command line barfs up an exception or syntax error at that boundary I'd bet most users would just give up and quit reading the rest of the blog.

Edit: clarification

<comment><USER12> Learning the idiosyncrasies of the tools involved is one of the tradeoffs. But there's no getting around it. These tools have been around for far too long to change them all in some misguided attempt at consistency--the semantics of most tools are so different, it wouldn't even make sense to try to enforce some consistency anyway.

You don't have to know every flag for every tool. You don't need to know if you can glob args together in a certain tool. These are different tools developed across decades by different people for different purposes. The fact that you can glue them all together on an ad-hoc basis is magical!

You learn by learning how to do one thing at a time--cutting characters 10-20, or grepping for a regex, or summing with awk, or replacing strings with sed, or translating characters with tr--and adding it to your mental toolbox. It's okay to have a syntax error because man is there and you can easily iterate the command to make it do what you want.

You aren't writing a program to stand the test of time. You're solving a problem in the moment!

</comment></comment><comment><USER27> This was a nice read and a good introduction to text processing with unix commands.

I agree with the other user re python usage - that you may as well use it for the whole task if you're going to use it at all - but I don't think it's a major flaw. It worked for you right? I would suggest naming the python file a bit more descriptively though.

Interesting to read the other suggestions about dealing with this without python.

<comment><AUTHOR> Thanks! Glad to hear!

</comment></comment><comment><USER28> $ join -v 2 <(ls | grep _A | sort | cut -c-4) <(ls | grep -v _A | sort | cut -c-4)

The shortest one I could come up with, no need to use python.

`join -v 2` shows the entries in the second sorted stream that don't have match in the first sorted stream, the rest is self-explanatory I hope.

Edit: $ join -v2 -t_ -j1 <(ls | grep _A | sort ) <(ls | grep -v _A | sort)

Is even shorter, it takes first field (-j1) where fields are separated by '_' (-t_)

<comment><USER29> Slightly shorter:

        ls -v|cut -d_ -f1|uniq -c|awk '$1<2{print $2}'

Tested by creating 500 sets of dual files and removing 10 `_A` randomly.

        for i in $(seq 1 500); do j=$(printf %04d $i); touch ${j}_data.csv; touch ${j}_A.csv; done
        for i in $(seq 1 10); do q=$((RANDOM % 500)); r=$(printf %04d $q); rm -v ${r}_A.csv; done
        removed '0438_A.csv'
        removed '0327_A.csv'
        removed '0150_A.csv'
        removed '0173_A.csv'
        removed '0460_A.csv'
        removed '0194_A.csv'
        removed '0073_A.csv'
        removed '0293_A.csv'
        removed '0404_A.csv'
        removed '0153_A.csv'

And then using the code above to verify the missing files

        0073
        0150
        0153
        0173
        0194
        0293
        0327
        0404
        0438
        0460

<comment><USER30> You could use uniq -u to avoid the awk.

<comment><USER28>
        ls|cut -d_ -f1|uniq -u

You win.

<comment><USER31> I like this solution - I'm not very used to using "cut" \- or more generally to map from "files" to "fields/lines in a text stream".

I'm more inclined to ask:

given a list of files with this name, does a file of a different name exist on the file system?

But the more Unix approach is really:

how can I model my data as a text stream, and how can I then pose/answer my question?

(here: list all filenames in folder in sorted order - cut away the text indicating type - then count/display the non-repeat/single entries)

My solution would probably be more like (with bash in mind, most of this could be "expanded" to fork out to more utils, like "basename -s" etc) :

      for data_file in *_data.csv
      do;
        alg_file="${data_file%_data.csv}_A.csv";
        if [[ ! -f "${alg_file}" ]];
        then
          echo "Missing alg file:\
          ${alg_file} for data \
          file: ${data_file}";
        fi;
      done

Ed: this is essentially the same solution as:

[https://news.ycombinator.com/item?id=19161358](https://news.ycombinator.com/item?id=19161358)

Although more verbose. I think I prefer omitting the explicit if, though - <USER2> just using "test" and "or" ("[[", "||" ).

</comment></comment></comment></comment></comment><comment><USER32> My favourite one is 'pkill -9 java'. Fixes my laptop if it starts lagging.

<comment><USER33> Does that kill electron instances too? ;)

</comment></comment><comment><USER34> I thought this was a neat demo of building up a command with UNIX tools. The python inclusion was a bit odd, yes.

I learned about sys.stdin in Python and cutting characters using the -c flag

<comment><AUTHOR> Thanks!

</comment></comment><comment><USER35> After moving back to working on a Windows machine the last several years and being "forced" into using PowerShell, I now find myself using it for these sorts of tasks on Linux.

I now use PowerShell for any tasks of equal or greater complexity than the article. It's such a massive upgrade over struggling to recall the peculiar bash syntax every time and the benefits of piping typed objects around are vast.

As a nice bonus, all of my PowerShell scripts run cross-platform without issue.

<comment><USER36> I've dabbled in PowerShell before, but I've always found the objects you get from cmdlets to be so much more opaque than the plain text you get from Unix output, which makes it harder to use the iterative approach to development the article and other commenters describe. Do you have any tips for poking around in PowerShell objects / a workflow that works for you?

<comment><USER35> I've tried to love it while using it as an interactive shell, but it's hard for me to lose the Unix muscle memory and remember their verbose commands.

For anything more than a single pipe, or anything that requires loops or control flow, I switch to Powershell in Visual Studio Code with the PowerShell extension which has intellisense and helps to poke around the methods on each object. From there you can select subsets of your script and run with F8 which helps me prototype with quick feedback.

</comment></comment></comment><comment><USER37> All the pipes and non-builtin commands (especially python!) look like overkill to me, I must say.

        for set in *_data.csv ; do
            num=${set/_*/}
            success=${set/data/A}
            if [ ! -e $success ] ; then echo $num ; fi
        done

ETA: likely specific to bash, since I have no experience with other shells except for dalliances with ksh and csh in the mid-90s.

<comment><USER38> Yup, I'd probably have gone with a `for` loop also. A bit shorter:

      for set in *_data.csv; do
        [[ -f "${set/data/A}" ]] || echo "${set%_data.csv}"
      done

Edit: though I just write it out like this for formatting on HN. In real life, that would just be a one-liner:

for set in *_data.csv; do [[ -f "${set/data/A}" ]] || echo "${set%_data.csv}"; done

<comment><USER39> Just because I like GNU parallel:

        parallel -kj1 'f="{}"; [[ -f "${f/data/A}" ]] || echo $f' ::: *_data.csv

</comment></comment></comment><comment><USER40> I usually do text processing in Bash, Notepad++ and Excel. Each has its own pros and cons, that's why I usually combine them.

Here you have the tools I use in Bash:

grep, tail, head, cat, cut, less, awk, sed, sort, uniq, wc, xargs, watch ...

<comment><USER41> As an aside I once found out you can replace 'sort | uniq' entirely with an obscure awk command so long as you don't require the output to be sorted. Iirc it performs twice as fast.

      cat file.txt | awk '!x[$0]++'

<comment><USER25> The awk commands prints the first occurrence of each line in the order they are found in the file. I can imagine that sometimes that might be even better than sorted order.

</comment></comment></comment><comment><USER42> If you are using python in your pipeline, might as well go all in!

      from pathlib import Path

      all_possible_filenames = {f'{i:04}A.csv' for i in range(1,10)}

      cur_dir_filenames = {Path('.').iterdir()}

      missing_filenames = all_possible_filenames - <USER3>

      print(*missing_filenames, sep='\n')

</comment><comment><USER25> The article solves the problem: for which numbers x between 1 and 500 is there no file x_A.csv? It looks like in this case it is equivalent to the easier problem: for which x_data.csv is there no corresponding x_A.csv?

        cd dataset-directory
        comm -23 <(ls *_data.csv | sed s/data/A/) <(ls *_A.csv)

<comment><USER43> This will fail for any filenames that contain newlines

<comment><USER25> Correct. It is intended for the filenames in the article. More generally, I try to write all my shell code to silently produce hard to track down errors when a filename contains newlines, in order to punish me for my carelessness if I ever accidentally create such a filename.

</comment></comment></comment><comment><USER44> I got paid $175/hr as a data analyst contractor to basically run bash, grep, sed, awk, perl. The people that hired me weren't dumb, just non-programmers and became giddy as I explained regular expressions. The gig only lasted 3 months, but I taught myself out of a job: once they got the gist of it they didn't need me. Yay?

</comment><comment><USER45> Nicely done using Unix utils. You can have a pure sed solution (save the `ls` invocation) that is much simpler, albeit obscure, that hinges on the fact that every number has a `data.csv` file.

Given a sorted list of these files (through `ls` or otherwise) the following sed code will print out the data files for which A did not succeed on them.

      /data/!N
      /A/d
      P;D

This works on the fact that there exists a data file for all successful and unsuccessful runs on data, so sed simply prints the files for which there does not exist an `A` counterpart.

If you want to only print out the numbers, you can add a substitution or two towards the end.

      /data/!N
      /A/d
      s/^0*\|_.*//g;P;D

Edit: fixed the sed program

<comment><USER45> Actually the following is even shorter

      /A/{N;d;}

So all together this gives the following

      ls|sed '/A/{N;d;}'

</comment></comment><comment><USER46> given the limited scope of files in the direcctory... not sure why it was necessary to use grep, instead of the built in glob?

      ls dataset-directory | egrep '\d\d\d\d_A.csv'

which FWIW wouldn't even work, on multiple levels: you need -1 on ls and no files end with A.csv

      vs

      ls -1 dataset-directory/*_A?.csv

ref: [http://man7.org/linux/man- pages/man7/glob.7.html](http://man7.org/linux/man-pages/man7/glob.7.html)

Update: apologies, apparently my client cached an older version of this page. at that time the files were named A1.csv and A2.csv

<comment> olog-hai Some ls man pages state the following about the -1 option: "This is the default when the output is not directed to a terminal."

I've never needed to use -1 when piping ls's output to another command.

</comment></comment><comment><USER47> Instead to create a script in Python to convert numbers in integers, you can use awk: "python3 parse.py" becomes "awk '{printf "%d\n", $0}'"

<comment><USER37> Why even use awk rather than the shell's (well, bash's) builtin printf?

        $ printf '%d\n' "0005"
        5

<comment><USER47> You can apply the awk command on a pipe, and so it is applies on each line of the file/stream.

<comment><USER37> Right - though that's solvable with xargs:

        $ echo "0005" | xargs printf '%d\n'
        5

That said, my suggestion doesn't work anyway since the leading 0 marks it as octal, d'oh (as mentioned elsewhere in the thread).

</comment></comment></comment></comment><comment><USER48> If you don't mind "cd dataset-directory" beforehand, a shorter and possibly more correct version would be:

      comm -1 -3 <(ls *_A.csv | sed 's/_.*$//') <(seq -w 0500) | sed 's/^0*//'

The OP's solution doesn't seem correct because of the different ordering of the two inputs of `comm': lexicographical (ls) and numeric (seq).

<comment><USER48> Although -w is supported by both GNU and BSD versions of `seq', BSD's ignores leading zeros in input. Thus a more portable approach is:

      comm -1 -3 <(ls *_A.csv | sed 's/_.*$//') <(seq -f %04.f 500) | sed 's/^0*//'

</comment></comment><comment><USER49> Easier would be just use 'cat list_of_numbers | sort | uniq -u' to get the unique entries.

<comment><USER50> Shorter still:

        sort -u < list_of_numbers

<comment><USER49> This is not the same. For sequence [5,5,4,3,3,2,1,1] "sort -u" returns [1,2,3,4,5], while "sort | uniq -u" returns [2,4].

<comment><USER50> Huh, I didn't know that! Thanks.

</comment></comment></comment></comment><comment><USER51> Useless use of seq spotted. Seq does not exist on many systems. Bash has {0001..0500} instead.

Nice writeup though.

<comment><USER52> Well, to be fair, bash does not exist on many systems either.

For example I have used dragonflybsd and freebsd today and they both had "seq" but no "bash".

<comment><USER53> They have jot(1)

</comment></comment></comment><comment><USER54> I learnt a lot from the book Data Science at the Command Line, now free and online at [https://www.datascienceatthecommandline.com/](https://www.datascienceatthecommandline.com/)

</comment><comment><USER55> Set operations are very useful. Here's a summary:

[http://https://www.pixelbeat.org/cmdline.html#sets](http://https://www.pixelbeat.org/cmdline.html#sets)

<comment><USER55> [https://www.pixelbeat.org/cmdline.html#sets](https://www.pixelbeat.org/cmdline.html#sets)

</comment></comment><comment><USER56> Not the most efficient solution but this is what springs to mind for me:

        seq 1000 | xargs printf '%04d_A.csv\n' | while read -r f; do test -f $f || echo $f; done

</comment><comment><USER57> Use F# with a TypeProvider. Of course, I imagine it would take some work learning F# but once you learn it the sky is the limit in what you can do with this data.

</comment><comment><USER58> More power to those who enjoy writing control flow in shell, but if I need anything beyond a single line I'm going with an interactive ipython session.

</comment><comment><USER59> You could use one sed command to replace your grep, cut, and python. It feels cheap to use python do massage data in a post about Unix command line.

</comment><comment><USER60> Is there a nice alternative for seq or jot ? Something neater than for-loop in awk ?

<comment><USER61> In bash, you can create sequences with {A..B}. E.g.

echo {1..10}

or to count backwards

echo {10..0} boom!

</comment></comment><comment><USER62> ls | rb 'group_by { |x| x[/\d+/] }.select { |_, y| y.one? }.keys'

[https://github.com/thisredone/rb](https://github.com/thisredone/rb)

</comment><comment><USER63> For heavier duty text processing, try

emacs -e myfuns.el

When it comes to mashing text, nothing beats emacs.

</comment><comment><USER64> awk one liner: ls | awk '{split($1,x,"_"); split(x[2],y,"."); a[x[1]]+=1} END {for (i in a) {if (a[i] < 2) {print i}}}'

<comment><USER65> Zsh one liner (probably works in Bash too):

        for a in {0001..0500}; do [[ ! -f ${a}_A.csv ]] && echo $((10#${a})); done

The only trick I'm using is _base transformation_ to remove padding in the echo...

<comment><USER39> I didn't realize Zsh (and Bash) was capable of removing zero padding in that way.

Everybody has there own style, but I would prefer to print the missing file pattern and avoid loops.

If you have GNU parallel installed (works in bash)

         parallel -kj1 '! [[ -f "{}" ]] && echo {} || :' ::: $(jot -w %04d_A.csv - 1 501)

or if preferred

         parallel -kj1 '! [[ -f "{}" ]] && echo {} || :' ::: {0001..0500}_A.csv

<comment><USER65> > _I didn 't realize Zsh (and Bash) was capable of removing zero padding in > that way._

Well, it's for transforming an integer in a different bases like octal, binary until base 24 (or more don't recall), but it can be abused to strip padding zeroes from variables. Using _printf_ should probably be cleaner but usually I only recall the the C syntax...

I think I have _parallel_ installed but I tend to use _xargs_ out of habit, mostly because I was forced to use _xargs_ in locked out production systems.

If the number of files wouldn't be so big, I'd simply expand them on ls and capture stderr:

        ls {0001..0500}_A.csv 1> /dev/null

It's a little nosier with the error messages but it's fast. With 500 files I'm sure I'll exhaust the shell parse(?) buffer:

        (ls {0001..0500}_A.csv 2>&1 1> /dev/null) | awk -F\' '{print $2}'

and too much complications to suppress stdout and pipe only stderr. ^__^;

</comment></comment></comment></comment><comment><USER44> The people that created the command line weren't L33T H4XOR NOOBS. They were brilliant PhD scientists. Let's not confuse the two.

</comment><comment><USER66> > I am starting to realize that the Unix command-line toolbox can fix > absolutely any problem related to text wrangling.

Am I the only one who thought, "No shit, Sherlock"?. This is a fundamental of UNIX that many people don't seem to grasp.

<comment><USER67> Everybody realises this at some point. Nobody ever thought "I can use this for anything" when they first saw a shell. It takes time.

</comment></comment><comment><USER5> > I am starting to realize that the Unix command-line toolbox can fix > absolutely any problem related to text wrangling.

How many problems related to text wrangling arise simply by working with Unix tools?

"This philosophical framework will help you solve problems internal to philosophy."

<comment><USER12> What a useless comment. The OP is an interesting walkthrough of solving a highly specific problem in a clever way using a common but often poorly understood toolset. Then you come in and leave a snarkbomb trashing the idea that learning how to use this toolset is worthwhile without providing any reasoning or alternatives.

Do you also trash posts about learning how to build your own furniture or troubleshooting car engines?

What elevated domain do you operate in that only has perfectly elegant solutions to beautifully architected problems that use only tools perfectly crafted to solve those exact problems? Doesn't sound like very interesting work to me.

<comment><USER5> [http://www.art.net/~hopkins/Don/unix- haters/handbook.html](http://www.art.net/~hopkins/Don/unix- haters/handbook.html)</comment></comment></comment>


Americans Support an Internet 'Kill Switch' According to Survey   - <AUTHOR> http://www.securityweek.com/americans-support-internet-kill-switch-according-recent-survey

<comment><USER2> Americans support a lot of crap they probably shouldn't and don't know a whole lot about. This is why we have a Republic, not a Democracy.

Edit: To put this a bit less glibly, there's a whole range of things that the "average American" doesn't know enough about to make a fully qualified decision. That's not to trash "average Americans" - I wouldn't do a whole lot better at making coherent, well-thought-out policies which take all available information into account if put on the spot - but it is to say that just because something polls well doesn't mean it should be policy. Our government is supposed to represent the best interests of the people, not blindly enact their will.

<comment><USER3> "average American" doesn't know enough about to make a fully qualified decision.

If voters aren't qualified for a direct democracy, what makes them qualified for an indirect democracy AKA republic? Wouldn't they just pick the wrong representation, given the inherent ineptness of the public? If the public is truly a fringe, and the government is supposed to represent the fringe, wouldn't the government just be a fringe?

"Our government is supposed to represent the best interests of the people, not blindly enact their will."

If they're not enacting the will of the people, then who's will are they enacting? What qualifies these folks to know the best interest of everyone else?

<comment><USER2> Well, first, it's an imperfect system, which is all we'll ever get. We're still human, after all.

I say "don't know enough" as a specific and non-judgmental quantifier, not an overall statement - in other words, while otherwise competent and capable, and possessing areas of expertise, the average person may not know enough about specific issues and the diversity of interests surrounding those issues to draft adequate policy, nor judge the quality of drafted policy. That's not to say they couldn't learn, but people simply don't have enough time to fully engage the issues of the day.

Because of this, we've elected to have a republic. Ideally, this means we choose gifted and dedicated people to represent us and to, as their full-time job, build up a body of knowledge and wisdom such to devise policies which best address pressing issues (Ideally.). This is a concession to the fact that we as citizens simply do not have the time to become experts both in our chosen craft and in the business of state. As to their qualifications, they're nothing but dedication, integrity, and intelligence. Ideally.

Obviously this isn't what we have, but it's the thinking behind what we wrote down a few generations back, and I think it's still valid.

</comment></comment></comment><comment><USER4> The sad thing is that anyone using an _Internet 'Kill Switch'_ would do far more damage than most of the "cyber attacks" that we know about. And if the ability was built, it would definitely become a target for cyber attacks.

<comment><USER5> > _become a target for cyber attacks._

This, this, a million times this.

</comment></comment><comment><USER6> These surveys are usually spurious, because they don't state what questions were asked and who constitutes "the public". It's possible to get whatever results you want either by cherry picking the kind of people you ask, or by framing the questions in a way which is biased towards certain responses.

I bet if they asked a question such as "do you believe that the president should have the right to prevent you and everyone you know from using Facebook, Twitter and Gmail for an unspecified period of time?" the support level would be rather minimal.

If it was framed as "do you believe that the president should be able to temporarily suspend internet services to protect vital national security interests in an emergency?" probably the majority would be in favour. Protecting things in an emergency sounds like a good thing, it's vague enough so that it doesn't sound like something which would apply to you personally, and if it's only temporary then who cares?

These days many people who would not be classed as archetypal "nerds" are highly addicted to internet use, and in the past I've seen people having something close to a panic attack if they can't access their emails for half an hour.

</comment><comment><USER7> I wonder if those surveyed realized that this would cut off their access to Facebook, Twitter, etc, or if many were too clueless to realize that.

<comment><USER8> In the article it mentioned "portions" of the internet. So I have to assume every one polled figured it would not be the part of the internet they are on but the part they(the bad guys)are on. If there was such a thing.

Like someone else mentioned. The consequences for such a thing are so much larger than what they are trying to stop.

</comment></comment><comment><USER9> For anybody who'd like a summary of the issue itself, I think Bruce Schneier covers the whole idea of an Internet Kill Switch pretty well here:

[http://www.schneier.com/blog/archives/2010/07/internet_kill_...](http://www.schneier.com/blog/archives/2010/07/internet_kill_s.html)

</comment><comment><USER10> I wonder what counts as an "emergency" and "cyber attack" and what is the extent of a "portion"?

These terms seem too vague for me, take this example: Someone tweets that Obama has killed a baby kitten, Obama could shut down Twitter because it's an emergency, he was falsely accused. He could shut down Facebook too, and all services that could be connected to Twitter to stop this cyber attack from spreading.

No disrespect, but I seriously doubt that the majority of the people(not just Americans) actually know how the Internet works, it's like asking a first graders to solve a differential equations.

The Internet just works fine by itself it does not need supervision from the US government, in fact it is one of the reasons it works so well.

</comment><comment><USER11> Americans also support a government-sponsored lobotomy for themselves, to insert Patriotic God-Juice into their heads.

Why is this a surprise?

The U.S. is a success _despite_ the majority of the American people, not because of them.

</comment><comment><USER12> I thought they very _idea_ of the internet was to create an unkillable network. This idea seems counter-productive and regressive.</comment>


Why It's Important For Us To Keep Explaining The Net And Its Civil Liberties - <AUTHOR> http://torrentfreak.com/why-its-important-for-each-of-us-to-explain-and-keep-explaining-the-net-and-its-civil-liberties-130331/ <comment><USER2> If Rick Falkvinge stopped saying "the copyright monopoly" and instead just said "copyright", then everything he said would make more sense, and more people might understand it more quickly, and he would be less frustrated. There is no such thing as "the copyright monopoly". Each individual copyright is a monopoly. But that's "monopolies", not one "monopoly".

There is perhaps a "copyright industry", consisting of companies and institutions that have a financial interest in the maximal copyright and maximal copyright enforcement. But those companies and institutions don't particularly have a monopoly on anything, other than their individivual copyrighted works.

For a clear statement of how copyright relates to the interests of the general public, read "Misinterpreting Copyright—A Series of Errors" (<http://www.gnu.org/philosophy/misinterpreting-copyright.html>) by Richard Stallman, which, unlike much of what Falkvinge writes, actually makes sense all the way through.</comment>


Show HN: KillSwitch - Uptime Monitoring With Google AdWords Integration - <AUTHOR>

Coincidentally, I wanted to launch this yesterday but HackerNews was down--
I built KillSwitch (http://Killswit.ch) with the busy agency in mind. Many digital agencies don't host the sites they advertise for and this can cause a problem when there is downtime. It's not uncommon for a client website to go down for hours in the middle of the night while traffic is being sent to the site. KillSwitch monitors the site every minute for connectivity and in the event of downtime notifies stakeholders. If the site is not returned to service after a user-adjustable threshold, the AdWords campaign is paused automatically. Upon restoration of connectivity, it's started again.
This is the most M of MVP's but I've really enjoyed building it. If you have any feedback, I'd be happy to hear it. email@example.com
Stack: PHP with Laravel 4 (http://laravel.com/) Bootstrap 3 Dashboard (Gumby on the homepage) Amazon Web Services Ubuntu 12.04 Local dev with Vagrant & Virtual box
Deployment with Capistrano <comment><USER3> One question for other Adwords users - have you noticed a penalty for putting the campaign in pause mode?

Years ago (3+) we used to pause our campaigns during known holiday times - <USER2> sell to mostly business-minded users and found that, on Christmas Eve and Christmas Day, we were getting clicks that would never, ever convert (3+ years of data). So we started pausing campaigns during those times. We found that when we re-enabled those campaigns we would get much worse results for as much as two weeks after we re-enabled them.

Being that this is HN, I realize that 100 people can't want to argue/disprove/etc (or be "captiuous") but I'm telling you now that we knew that it was due to pausing rather than "external factors". I'd just like to hear from anyone else to see if they had the same experiences.

<comment><AUTHOR> Hey Scott,

The adrank could be affected by things like this but here's what I would ask you to consider: Google is very sly about measuring on page performance metrics. If you have downtime, how many visitors do you think are going to return to the Search Engine Results Page? Google knows this and could use it as a an indicator of a poor performing ad. My service is different from manual pausing for this reason. Here's a good article on the type of work they're doing to create adrank: [http://vuurr.com/impact-of-google-adwords-quality- score-on-c...](http://vuurr.com/impact-of-google-adwords-quality-score-on- cost-per-click/)

Specifically, I've not had a "penalty" for pausing ads myself but the ads were typically performing well. I pause the bad ones pretty quickly.

Cheers,

Josh

</comment></comment><comment><AUTHOR> I'd also be happy to entertain any questions you might have about the solo- founder undertaking and how I arrived at this particular niche of monitoring.

</comment><comment><USER4> Clickable - [http://Killswit.ch](http://Killswit.ch)

<comment><USER4> Some feedback

1\. Don't force the user to enter the phone number. I don't mind giving out my email address, but my phone number is only if you win my trust.

2\. You need to have some form of trial before requiring a credit card. If that's not possible then at least a video demo and screenshots.

edit: formatting

</comment></comment><comment><USER5> I like the name. :)

<comment><AUTHOR> That doesn't surprise me. You've got good taste.

</comment></comment><comment><USER6> Looks like a much needed service.

<comment><AUTHOR> When I was doing agency work the thing that disappointed me the most was that we had LARGE clients with BIG budgets who would have SERIOUS downtime for no particular reason. It was so frustrating. This service comes from that frustration.

</comment></comment><comment><USER7> I'll give it a spin.

<comment><AUTHOR> Let me know if you have any questions. firstname.lastname@example.com</comment></comment>


College is a waste of time and money for kids - <AUTHOR> http://www.ft.com/cms/s/0/cfb4d37a-d907-11dc-8b22-0000779fd2ac.html <comment><USER2> I was with him right until he say not to travel. Travel is the single biggest step in developing a proper perspective of life in this world, and what part you'll choose to play in the future. At 25, you can tell if someone has traveled or not by having a 15 minute conversation with them.

The traveled ones are always much more rational in thought, and open in mind, and aware of the possibilities. Non traveled folk tend to be more rigid, narrow in though and unable to grasp concepts that may be foriegn to their history.

As to the rest, my biggest beef is that he assumes all people are born entrepreneurs - even if he says otherwise. One must remember that although college can be a bore for a bright individual, it can be a saving grace for a person that is, well average.

<comment><USER3> I'm sorry but I disagree. Traveling is certainly something that is worthwhile, but it's hardly the determining factor in one's personal development that you're making it out to be.

I've been to a lot of places, and I still value things like the books that I've read, personal relationships, real experience etc etc a lot more.

Sometimes all travel contributes to is pretentiousness.

<comment><USER4> sorry, but taking a cruise to bahamas,or a two weeks trip to europe doesn't make one a "well traveled" being. Living in a different country for a while, it will open you a whole world of differnt perspective, and experience things that you just don't when visiting a place for a few days.

yes, just the fact of knowing a foreign language, (well), tells a lot about a person.

<comment><USER5> Agreed. It really depends on the kind of travel. Watching other people commute is not mind expanding. That said moving somewhere totally different and working there for a year is.

</comment></comment></comment></comment><comment><USER6> I've found that college is great for the 'pluggers' in life. Those kids you remember from high school who asked question after boring question and spent hours on their home work while you spent class time hacking your graphing calculator and spent study hall exploring your high school's network.

Sure they got better grades than you, but instead of an A on your test you got a B- and worked 10x less to get it.

College is also great for the socially motivated who love to party, 'hook up' and generally spend the entire day trying to sleep the previous night's craziness off and plan for the upcoming night debauchery.

<comment><USER4> Believe it or not, you can learn, get decent grades, party and get laid, and have scholarhips all at the same time.

Those people that skip this, are soreley missing a great experience of their life.

Remember, work will be there for the rest of your life. Don't just rush into it. From 24 to 64, that's 40 years or working for most people. you can start your startup just as well as 25, as you could at 20, but probably you would be much smarter at that age.

<comment><USER7> it amuses me that people advocate the importance of college -- supposedly an educational institution -- based in part on parties and sex. and in part based on avoiding work.

if you can pay for college, you could pay less to not work or go to college. and still party and have sex, if that's what you care about.

<comment><USER4> It amuses me how some people say that college is not worth their time and money. Maybe you stop taking easy clases? Get the challenging ones? Don't overpay for school?

Nobody is measuring the importance of school, on how much fun you have with it, but what I am saying is that you can learn a lot, and yet have a great time, and yes even fun. Unless you are sociophobe, scrooge, or something, most humans want some fun on the side.

I thought CS was not that difficult, so I decided to minor in math, which I found it not that difficult so I challenged myself even more, and I decided to minor in Physics, and here I met my match. A hard ass professor, that would teach a lot, but it was almost impossible to get an A. My first ever class, where I felt no matter how much effort I put into it, I couldn't get an A. It taught me about humility, and learning my limitations.

And, yes, I did all that, plus lot's of volunteering, helping with alumni affairs, while having a part time job (washing dishes, and then just a lab proctor) at the same time.

There is nothing more awesome when you learn a lot, and yet can have fun and get laid at the same time.

To all you say that college is waste of time:

1\. Maybe you should study something that you are going to get a great return out of it, or it will be challenging enough.

2\. You might want to check if you have problems getting socially adjusted.

So, unless you quit school and become a millionare by now, you have no cred on saying college is a waste of time. College is a waste of time for many, b/c they don't use it. To those that are smart enough, it is a great learning experience. To those that think are too smart for college, maybe they are not taking the right/classes or programs. If you are a genius, what prevents you from finishing it in two years, get a masters, get a phd. and become the next albert einstein, invent cold fusion, a cure for H1V or something great? You can do that in college. It allows very smart people, to have access to great equipment, grant money and other resarch, things that you probably wont have in your own basement.

<comment><USER7> i didn't say college is a waste of time, i pointed out one way the pro-college arguments are flawed. repeating myself in different words: some arguments advocate a lifestyle (college) in part for certain benefits, that are easily available from rival lifestyles. so they are not an advantage over college's rivals. so they should be left out of the discussion.

</comment></comment></comment></comment></comment><comment><USER8> I could, and did, write a book (<http://www.aarongreenspan.com/authoritas.html>) about this subject, namely, why the American educational system is broken. I found college to be particularly difficult, but even so, I still think that James Altucher is wrong. Not only is his analysis wrong, but his facts are wrong. Unbelievably expensive though it may be, I know of no college that costs $400,000 over four years. Students are not cleanly divided into "the well-rounded generalists" and "the focused entrepreneurs." There are considerable virtues to liberal arts education, even if Mr. Altucher is too dense and narrow-minded to appreciate them.

<comment><USER9> I am repeatedly amazed at what kinds of people leave comments here.

</comment></comment><comment><USER10> College gives kids thousands of other kids to hang out with, at a critical point in their lives when they are just entering into adulthood.

College gives kids a chance to create their own social identity, as seen through the eyes of their peers, rather than their parents.

If you are a social recluse, and spend most of your college years hiding behind your computer screen, then yeah, you might as well not go.

If you recognize that human beings are herd animals, and that in order to learn how to be a leader in the herd, you need to have a herd to practice on ...

I learnt some really important life lessons in college, most of them outside of class. I heartily recommend it. Challenge your expectations of what you can and can't do. You may be surprised by how much more you can grow.

<comment><USER11> >I learnt some really important life lessons in college, most of them outside of class.

What makes you think you would not have learned the same things (or equally valuable things) if you did not go to college, but did stop living with your parents?

<comment><USER10> As yters points out correctly, the real world is not like a college environment, where you are surrounded by other people your own age.

There is something about being surrounded by other people just as clueless, inexperienced, and eager to experiment as you, that stimulates the creation of convention-defying ideas.

That said I would like to think I did not need college to become an intelligent, aware, adult. But it sure did help and it sure was fun.

</comment></comment></comment><comment><USER12> Look. There is no "right" or "wrong" way to live.

People need to stop focusing on the black and white and become aware of the greyscale.

<comment><USER13> there. is. no. box.

<comment><USER14> I thought it was, "There is no spoon"

<comment><USER15> I thought it was - 'there is no cake'.

<comment> tac-tics Delicious Cake. I must eat it.

<comment><USER16> The cake is a lie.

</comment></comment></comment></comment></comment></comment><comment><USER17> I've taught myself math and computer science topics by myself from books, and I've learned math and computer science in classes, and I can say that you make much faster progress and have far fewer foggy, unclear areas if you have access to a professor.

That's assuming you really take advantage of the teaching assistants, professor, and books out there. If you just party and take sociology classes, sure, college is going to mostly be a waste.

<comment><USER18> If by access to a professor, you mean someone who knows the material extremely well and can help you with questions or steer you in the right direction, you can get that for free from IRC.

I've been leeching off my friends' college educations for years, and although I'd agree that it would be impossible to learn some things (like advanced programming topics) strictly from books, someone who knows the material and exactly how to explain things to me is just as good as a traditional instructor.

It's also better training for the real world, where I don't have time to take a course on everything I need to know.

<comment><USER17> for free off IRC? not the things I wanted to learn.

</comment></comment></comment><comment><USER19> Hes off the mark by a long shot.

Is college right for everyone? No, and on both sides of the equation. A very small group of people are too smart for it, get too constrained by it, etc. There is a much larger group of people who are not smart enough for it, or don't work hard enough to get there. For everyone else, it makes sense. To say that you won't make up the money (and he says 200,000 - 400,000, that is a 50,000 to 100,000 a year college! There are only about dozen that expensive! I'm in one of the top 20 in the country and it would cost me 120,000 for 4 years coming from out of state, if not for a scholarship), is ludicrous. I'm going to work from the age of 25 to the age of 65 unless I do something spectacular and can retire early, which MOST PEOPLE DON'T. So for most people it is 120,000 divided by 40 years, or $3,000 a year more they would have to make in order for it to be worth it, obviously most college grads are better off money wise if they go. And staying in state is even cheaper. His point is kind of ridiculous, just because he didn't like college, doesn't mean that he should recommend against it. I sure feel bad for his kids who will be trying to make it big on their new version of MySpace when all their friends graduate and start making real money.

<comment><USER20> $50,000 per year is not actually that high in the grand scheme of things as a total cost to attend a private college, since most undergrads depend heavily on their parents for financial support.

If you subtract a base tuition of $35,000 from that, you're left with $15,000 per year as total living expenses. That's barely over $1200, and has to cover not only rent and food, but textbooks, computers, and other costs of campus life. Even assuming your kid doesn't do much partying, or need access to a car, $15k is hardly a lavish annual budget.

<comment><USER19> Yea, but that would assume a base tuition of 35,000, which there is no reason to spend. Go to a state school if the money is an issue, out of state to Penn State, considered one of the more expensive state schools, is 24,000 a year. That leaves 26,000 a year for living expenses. In state is even less. At my school it is 18,000 out of state and 7,000 in state and we're top 25 nationwide in both of my majors. My apartment is one of the nicer ones on campus and it is $700 a month, plus $500 for food and expenses, plus 2,000 for books and supplies, I'm barely up to 35,000 and that is with generous allouances and an expensive apartment. In state you could be doing it for 20,000 a year. The people who claim college is outrageous are far from the ground level. I don't know anyone who couldn't work it out by taking loans and going to a state school, everyone quotes these private schools and their prices but that is a whole different ball game. The reality is that college is not all that expensive in the short term, and it is nothing in the long term.

<comment><USER20> Going to a small, private college is also a completely different experience than attending a large state school. The entire pedagogical style is difference: small (<15 students) conference courses and one-on-one meetings with profs are the norm, rather than the exception.

I'm not saying that private schools make sense a strict cost/benefit level; if you read my other posts in this thread, you'll see that I'm nearly as dubious as the author of the original article about the real economic benefits of a four-year degree. I just don't think that $200k is an unusually high total cost for four years of college.

</comment></comment></comment></comment><comment><USER20> I think most of the comments here are missing the key argument in this article: namely that in addition to college being a poor fit for many potential students, there are some much better, more productive ways that young people could spend the time in the years immediately following high school.

The suggestions to encourage your kid to practice one skill, or start a business, are spot-on, at least from an economic POV. Learning a trade and having some business experience are much more effective at increasing your future potential earnings than a liberal arts education.

Personally, I dropped out after my freshman year to go work for a startup I've had a great time working in IT since then, and consistently out-earned my friends who stayed in school, even now that they've had 4-6 years in the workforce to catch up on experience.

That being said, I think that there is still a strong argument to be made for people _eventually_ attending college. I still look forward to college as something I'll do in the future as a sort of "working vacation." I.e., just like faculty take a sabbatical to focus on their research, or spend time with their family, I hope to take a couple of years off to devote myself to intense, full-time study of some subject other than computers.

<comment><USER21> Oh, blah. I'll grant you that people (in this country) would be better served by slowing down a bit, but I wouldn't encourage a high school kid to start a business instead of going to college.

Maybe it's a good idea for a kid to work on his "startup" when he's barely old enough to shave, but probably not. The chances of success aren't higher when you're young and under-educated (likely the opposite), and you'll be squandering the one time in your life when you're allowed to get away with things and make mistakes.

Don't want to go to college? Fine. Travel the world. Have adventures. Play in a band. Make friends. Meet women. You're not likely to get the chance later, and you'll be a more compelling, charismatic person for the experience. Then go to college.

Once you've graduated, you'll have the rest of your life to start companies (and the charisma and friends you've picked up will help you on your way).

</comment></comment><comment><USER22> I don't think he goes far enough. Looking back, my university experience was indeed sort of a waste of time. However, the time I spent in college was orders of magnitude better than the previous four years of high school.

-edit-

Ok, I just closely read the piece instead of just skimming it. I do think college can be a waste of time for many people. However, the suggestion to instead outsource Facebook/MySpace application development to India is ridiculous. Also, what college costs $400,000? Ivy League tops out at about $40K/yr. UC Berkeley is about $9K per year.

</comment><comment><USER23> If all you want to get out of college is to learn to program or other "trade" skills, then yeah it's probably a waste of time, but that's not what college is all about (unless you go to a vocational school)

<comment><USER24> Seriously. Imagine if children treated kindergarten with the same attitude that some people treat college with, the trade school approach to education. They'd never get off the dime.

</comment></comment><comment><USER25> I agree with him, but I don't think the alternatives he proposed are reasonable for most people.

</comment><comment><USER26> This is a great article, even though I disagree with most of it. I think it's a great article because the conventional wisdom about college does need to be challenged. Just like the notion that you have to attend a prestigious college to be successful needs to be debunked, the notion that you have to attend college at all should be challenged.

I'd agree wholeheartedly with anyone who said that college should _not_ be a default choice. There are all kinds of amazing opportunities for learning now that don't require a degree. He had me while he was arguing that college can be a big waste of time for a lot of people - but he starts to lose me when he starts to argue _against_ college.

Personally, I have my doubts about anyone learning advanced math or physics without an intellectual environment similar to the one you'll get at college. I learned from my peers in a study group at least as much as from my professors. I suppose you don't strictly have to be enrolled - and maybe you could view webcasts and form a general study group outside the university - but that would be an argument against a formal degree, not an argument against a college education.

<comment><USER26> One thing to add - he's not totally off base about the 400K price tag. If you could earn 50K/year as a high school grad, that's 200K in lost earnings (though it would be taxed). Stanford's tuition is about 40K/year now, right? So there's another 160K. Not too far off (I wouldn't count living expenses, because those would be incurred regardless of whether you were in college).

A high school grad with no skills isn't going to earn 50K a year as a dishwasher. But someone smart enough to get through Computer Science at MIT might easily blow through the 50K/year estimate. So the opportunity cost could actually be much higher than 400K.

<comment><USER27> I think including potential wages is at least a little weaselly. Besides the fact that you can work while going to school, get scholarships, etc, you could just as easily say that any lost earning potential while taking classes should quickly be made up by the difference in your wages from having a degree.

Even if you do, 50K/year as a high school graduate (at least in the US) is being pretty optimistic. I don't think most of my friends in non-technical programs didn't even make that out of college. The average starting salary for a BS CS grad even is probably still only around 50K.

<comment><USER26> This is why the right answer is "it depends". And it's why I liked the original article, even though I disagree with the conclusion that college is a waste of time. It might be a waste of time, it might be useful but not worth the cost, and it might be enormously useful and well worth the cost.

I have a feeling a really good hacker who can do consulting work, or join a startup, could easily earn more than 50K, and might be ahead of college grads by the time they do enter the workforce. Think of it this way - you might make more with PhD than a BS, but you won't necessarily earn more than you would with a BS+6years experience, and even if you do, you have to account for lost earnings.

Of course, High School only vs. BS is a very very different equation from BS only vs Ph.D. Personally, I do think that smart people who want a career in a technology related field would be well advised to attend college.

</comment></comment></comment></comment><comment><USER28> As you peeps have pointed out, 400k is ridiculous. I go to an expensive private college, and it's on the order of 120 for 4 years (though it's actually a 5-year school).

And if his kids are really entrepreneurial, as he claims, then they'll have the motivation to apply for scholarships like I did. I have received absolutely no financial help from my parents, and yet have accumulated very little debt.

I have learned most of my knowledge by teaching myself, and generally goof off during class if I even go (and no, that doesn't put me into the B group). But I've also been exposed to an incredible variety of people and built a huge network of friends and acquaintances in almost every industry.

Another point he seems to be missing is that to make a Facebook or Myspace app go viral, you need an initial network of users, typically your own friends. I hope his kids have fun trying to promote their app to all their high school friends who have gone off to college and lost contact.

I think what makes me most upset about this article, though, is that he's not using his views to make his own decisions, he's using them to make his kids' decisions. I feel really bad for his kids.

<comment><USER29> A) When you look at the high numbers you need to ignore scholarships as many don't qualify. B) Its a 5 year school so 30 * 5 = 150k not 120k C) You could be making a lot of money if you worked instead of spending all that time in school.

Finally, you could get significant returns if you invested your first years tuition by the end of 5 years.

<comment><USER27> The fifth year is quite possibly a year of work-study, like in my degree. Either that or adding a second degree/masters, which is going to increase earnings potential hugely.

I have seen statistics before about how having a degree positively influences your earnings potential, but I have yet to see anything but "feelings" from those claiming going to college has a negative effect.

If we're going on anecdotes, I seriously doubt I could have made that much going into work straight out of high school, and I seriously doubt there are many people that could.

I would bet there's a more people that could qualify for significant scholarships than could make significant money going straight into work.

<comment><USER29> I think there is value to looking at anecdotes because the average collage student is better connected and more intelligent than those who skip it. Most of my friends have collage degrees. Of those who don't one was making 120k/year before he died. One charges 100$+ per hour and is constantly booked. And the last guy is 29 and makes 140k / year at yahoo. IMO a large reason for the disparity in lifetime earnings between those with a collage degree and those without are the type of people who go to collage.

For most people staying in school is probably a good idea. But, when your options are to stay in school or drop out and make 70+k / year staying in school seems less useful. Would these people make more money in 5 years with a collage degree well possibly, but they are making good money now and investments compound with time so it would have to be a huge increase to be worth it.

PS: Doctors get paid a lot latter in life but on average they don't retire younger than most people.

</comment></comment></comment></comment><comment><USER30> I read an article once that suggested that colleges and universities are part of a mafia-type organization that extorts parents/students to pay high tuition fees...otherwise the student won't be able to get a decent white-collar job as an adult (the whole college degree required deal)

<comment><USER31> Nah, there's just not a good way to rate and rank potential job candidates without actually taking the time to get to know them. Finishing college is just an imperfect proxy for being smart and working hard, and companies use it as a baseline because it puts the onus on students/employees to attain. If there was a way for companies to more accurately rank candidates (without actually requiring more work on the company's part), they'd use that instead. Universities are more than happy to facilitate this.

See also [http://sethgodin.typepad.com/seths_blog/2007/10/the- greatest...](http://sethgodin.typepad.com/seths_blog/2007/10/the-greatest- ma.html)

</comment></comment><comment><USER32> He says, "I can't remember anything good coming from my freshman year – other than starting a business with a few of my classmates, which inspired me for subsequent businesses."

So, he found his calling in College? A waste of time? Could he have started this first business without attending college?

</comment><comment><USER18> I like how the article suggests that the cost of developing a viral Facebook application is $200 for outsourcing to India.

I agree with the premise of the article, though. Except for the $400,000 price tag. Where does that come from?

</comment><comment><USER33> I agree, I'm glad I haven't wasted time on college, learning everything myself has been more beneficial. In certain cases, like that of doctors or lawyers college is essential!

</comment><comment><USER15> So people who live on farms in France don't have the internet? I think he there still might be a flame.

</comment><comment><USER34> Great piece. Helps re-encourage me that I'm not throwing my life away.</comment>


Is the iPhone causing Apple to lose the plot? - <AUTHOR> http://www.mobilecrunch.com/2009/07/29/is-the-iphone-causing-apple-to-lose-the-plot/ <comment><USER2> "It's your phone, so why can't you install whatever the hell you want on it"

Primarily because you don't own the software that runs on it. You license it. I suspect Apple would have no legal recourse against a project that ported Android to the iPhone along with clean room implementations of all the required bits like radio firmware, bootloader, etc. It's a much different situation when we're talking about modifying Apple's software.</comment>


Ask HN: How to be a good bash script writer? - <AUTHOR>

Assume elementary knowledge - pipes, redirection, awk/sed/cut/standard Unix tools.
How does one get to writing scripts that are -
(a) portable
(b) easy to debug
(c) clean and easy to modify
Where can I find good sample scripts? <comment><USER2> You might like the Advanced Bash Scripting Guide. To quote:

"This tutorial assumes no previous knowledge of scripting or programming, yet progresses rapidly toward an intermediate/advanced level of instruction... It serves as a textbook, a manual for self-study, and as a reference and source of knowledge on shell scripting techniques"

[http://tldp.org/LDP/abs/html/](http://tldp.org/LDP/abs/html/)</comment>