Ask HN: Should I leave my startup? - <AUTHOR> I've been with the company 7 years and am employee #3. I started as a senior engineer and worked up to VP Engineering but for all practical purposes I'm the CTO.
I manage 5 engineers down from 15. The 10 that left did so because we couldn't afford to pay them market rates. In my opinion we were understaffed even at 15.
It would take > 3 senior people to replace me. I know the company can't afford it. I know for certain 3/5 of the remaining devs would definitely leave if I left (an the other 2 probably would after that). Plus I have a lot of proprietary knowledge because I'm a specialist in multiple areas.[1] I don't think the company can survive me leaving. That means 20+ people losing their job.
I make $109k but haven't had a raise in 4 years. The company can't afford it. I was promised $170k almost a year ago (never happened).
I live in a medium COL area. Cheaper than SF but still expensive. I have a mortgage to pay.
I started at <1% options and I now have >5%. On paper it is work over $1MM but I can't see liquidity happening for a few years.
The company owes me more than $60k back pay. I went 6 months at minimum wage and have a note promising to pay me back but there is no interest built-in so they have no incentive.
My non-compete an invention clause are very strict and I live in a state where they are legal. We do consulting to pay the bills (the lean startup way) so my non-compete pretty much covers every industry you can think of.
I have 4 weeks vacation but my average I can use is 1 because there is always a deadline looming and the company loses a lot of revenue for every week I'm not here.
On the plus side: I love the team, I think they are great. And I have a high degree of freedom I probably wouldn't have elsewhere.
[1] Note: I try very hard to spread the knowledge but every time I get close they leave for better offers. <comment><USER3> It sounds like you and the CEO both want something, which is often the basis for a reasonable compromise. You want to be let out of the strict non-compete clause, and you want your back-pay and ideally a raise. Meanwhile, the CEO doesn't want his company to fall apart in the wake of your departure. You could approach your CEO, and ask him to deliver a plan to pay down your backpay. During that discussion, you can tell him that since you're underpaid and haven't gotten a raise for years, you'd like to amend your employment contract so that the non-compete and invention clauses only apply to your corporate work (and not your consulting work). You can explain that this would let you do projects on the weekend that would help you make some more money to compensate for the undermarket salary.

Not tying the reason to your desire to be able to get another job easily also helps the CEO from freaking out as much. Also, limiting the non-compete edits to the consulting work helps him feel that his corporate work isn't threatened, while getting you what you want.

Amending your non-compete would also give you flexibility if you choose to leave, so that would be my primary goal here. To increase your leverage, you can also ask for a raise. Assuming he says that giving you more money is impossible, that increases pressure on him to concede on the only term that is non-monetary.

<comment><AUTHOR> Thank you for the advice. I have tried similar in the past, but I have stopped shy of putting him in a corner.

I have gotten him to agree to let me work on side projects in exchange for the company getting 20%. But I found the hours I need to put into my main job are so long that there is no time left for the side projects (unless I neglect my wife, daughter, and/or my health) so it's an empty promise.

And obviously I can't contract on the side. That would be direct competition.

<comment><USER3> Taking a step back, it seems like you want to present a perfect solution to your boss that he can just adopt. I mean this in a friendly way, but that's not your job. Your job is to tell your boss what you want. Your boss's job is to figure out how to meet your needs.

Along those lines, I would just tell him politely that you'd like a plan to get your backpay over time, you'd like a raise although you understand that funding may preclude that, and you'd like your non-compete to be altered so at least it doesn't apply to your consulting work, since the scope of the consulting work is so broad that you wouldn't be able to ever get another job. You can mention that given the natural instability of a startup, this fact is stressing you out and making you lose sleep.

I would also ask to "paper" the backpay, so you have more than a verbal commitment on that.

If you talk nicely and respectfully about these issues, it is very likely that he will agree to the one term that doesn't cost him any money. (He will likely stall on all the other terms.)

</comment></comment></comment><comment><USER4> From your additional comments, it sounds like the CEO has access to money from his personal wealth, but is not willing to put it to work at his own company. This sounds like he has not enough confidence in his own abilities and the team to make a decent ROI. I consider this a big red flag.

I'd recommend reconnecting with the 10 engineers who already left to see if they can help you line up a new gig. Once you have an option, you can have a conversation with your CEO and try to negotiate a better offer that would convince you to stay. If he fails to do so, he at least had a chance and you will find out quickly how much you are worth to him.

Good luck!

<comment><AUTHOR> It's a good idea.

But I just want to correct one thing. The CEO has generously put a lot of his own money into the company. Nearest I can figure close to (if not over) half his net worth. I don't blame him for not wanting to use his whole net worth. If anything that is part of the issue, he's used his money to lengthen the runway but it hasn't been able to take off. He's also used that as a guilt- trip on me a few times.

<comment><USER5> Who has the bigger call on you: your wife and daughter, or your ceo? Who is more important? Answer that question, then man up and do what you obviously need to do.

ps -- I'd talk to a lawyer (it's free for a consult! Write down the details so you don't miss anything and can be dispassionate, then ask one or two to review your situation and options!) about filing a wage claim for that $60k, or some other type of compensation.

One last thing. You know what you call people whose leaving would crater a company? Owners. If you're that crucial and only have < 5%, you're a sucker. Sorry man. This is a lesson; don't let it happen again.

<comment><AUTHOR> Thank you for the advise.

My daughter and my wife are certainly doing well but not as well as they can be. It's not like I can't provide for them at $110 per year.

I think I will be talking to a lawyer if I do decide to leave.

And if you think 5% is bad... I negotiated up from 0.5%. I'm not sure I agree that 5% is bad. It is above average for a CTO from what I understand and I'm sure many of those CTOs are just as critical to their companies as I am to mine.

As far as the wage claim... I don't think I can. Technically it is on the books as a deferred bonus. The company paid me minimum wage for that time and I stupidly agreed to it. I'll talk to the lawyer, though.

<comment><USER5> I disagree. It's simple: if you're crucial to a small company, you're an owner and you get comp like an owner. If you have less than 5%, two options: (1) you're not actually as crucial as you think you are, or (2) you're being ripped off. (Could be some of both!) Buttering up employees and telling them they're crucial is, btw, free. Just like titles. Don't let your boss pay you with free things.

<comment><AUTHOR> Do you have any numbers or articles I can refer to to back up #2?

There are actually a lot of investors in the company and it raised a fair amount of money before I came on board.

I won't rule out #1. However, I'm basing that assumption on a lot of inside knowledge of how the company works. It's an evidence based assumption not an emotional one. And it's based a lot on how long I have been here and what has come to depend on me. I absolutely 100% do not think I can move to another company and be as irreplaceable, it is a niche I moved into by being the only one who could pick up the pieces when each of the other people left.

<comment><USER5> You're not gonna change your comp with a benchmark. You're gonna change your comp with a simple 3 step plan:

1 - get a new job

2 - <USER2>

3 - see how indispensable your ex-ceo sees you as now

</comment></comment></comment></comment></comment></comment></comment><comment><AUTHOR> Some notes because I ran out of room:

I am a full-stack engineer with deep knowledge of devops, architecture, cloud computing, middle-tier, back-end, and front-end development. I wouldn't call myself expert in any of them but I'm at least a level III in all of them and a level V in some. I also have a fair amount of engineering management experience.

I have a computer science degree from an accredited and regionally well ranked school but it doesn't rank high nationally.

I've also written several books and regularly speak at conferences.

I think it is fair to say that finding another job would be low risk if it weren't for my non-compete.

<comment><USER6> On the bright side, if they're a job shop and live on consulting work, they probably can't afford to go after you for a non-compete.

It sounds like you're not in a good situation. Good luck!

<comment><AUTHOR> The CEO has a fair amount of money personally, I wouldn't put it past him to go after me with his personal funds. He's a generally nice guy but a ruthless business person, especially if someone torpedoes his company.

On that note, he has gone without pay much longer than me. It's not like he's been taking paychecks this whole time. But he's also a millionaire and I'm barely getting by after my entire savings went away from all the missing pay checks.

<comment><USER6> So, first, _you aren 't the one torpedoing the company_. That was done by whoever failed to hire a CTO or people to maintain the business on the tech side.

Second, _the CEO 's situation is not your situation_. They have money and legal responsibility for the business, you don't. They have a more favorable equity situation and voting power, you don't. You have to look out for yourself.

</comment></comment></comment></comment><comment><USER7> Yes.

The non-compete is less of an issue than you may think. Your new company may even be willing to defend it for you if you are valuable enough to them and you are upfront about it. It happened to me (not in software/engineering). Plus, your current company needs to have the money and be willing to spend it to enforce the non-compete.

Your current salary is unimpressive. The $60K and the options sound worthless. The biggest mistake seems like letting it get to this point.

My recommendation would be to find that next job to support your mortgage and whatever else, and get out as soon as possible.

<comment><AUTHOR> That is a worry of mine, that 7 years at this company looks bad on a resume.

Also, I was not trying to impress anyone with my salary. I am aware it is probably at least 50% below market. I was hoping the company would make due on it's offer to give me a raise and I could negotiate from a stronger position.

<comment><USER8> > That is a worry of mine, that 7 years at this company looks bad on a resume.

7 years looks solid & respectable in a space where there's 100% churn every 24 months. You've got a great story to tell.

Relative to negotiating your departure and future compensation, Stuart Diamond offers excellent advice> [http://www.goodreads.com/book/show/8095067-getting- more](http://www.goodreads.com/book/show/8095067-getting-more)

</comment></comment></comment><comment><USER9> What is your exercise window on the options? Are they ISO or RSU?

If you have ISO options with a 90 day exercise window, you'll need to pay the AMT tax on the difference between current valuation and strike price. If you joined as employee #3 and valuation of options is now $1m, that could be a substantial tax bill. If the company never exits you will have paid a bunch of tax money for shares you'll never be able to liquidate.

Also consider that if you feel you're so important to the company that leaving might cause it to fail, then your leaving will decrease the possibility of any exit and effectively devalue your options.

If you have a 90 day exercise window, perhaps you can negotiate it to 10 years, in exchange for reducing the back pay you are owed. That $60k is leverage you can use in any negotiation, whether for options terms, severance package, or non-compete agreement.

<comment><AUTHOR> The tax burden on the options would be well more than I can afford. And like you mentioned, my leaving would have a very negative impact on valuation.

My window is 2 years. So at least there is that. Although (not sure if I mentioned that), a bulk of my options came within the last year and I have a 1 year cliff with four years total vesting. So at best 30% of my total options are vested.

<comment><USER9> How many more options would you get if you stayed the remainder of the year? Perhaps that could be worth it.

And perhaps another thing you could negotiate for is vesting acceleration.

<comment><AUTHOR> Back of napkin math... staying an extra year would be slightly less than 50% more (the second 25% of my largest grant would hit and I'd have half of it).

This is part of what makes the decision so hard. And it's worth noting the valuation of the company is not terrible. If I had this valuation after 4 years I'd be ecstatic. Problem is it's been 7 and I have a lot of sunk cost.

</comment></comment></comment></comment><comment> alex- It sounds like you are not happy. If that is the case, why not explore what opportunities you have for a better position (whatever better means to you)? If you find something that you are confident is better for you - then you should pursue it, after all you have decided it is better for you.

You might find in the exploration that you actually don't want to move on yet. Maybe you will miss the people, flexibility or sense of ownership. Maybe the kind of role you are seeking would require a move you don't want to make etc.

Alternatively maybe you find your perfect gig (great people/problems/pay/etc).

<comment><AUTHOR> That is probably what I will end up doing.

</comment></comment><comment><USER10> Looks like it was fun while it lasted. You got valuable skills, a mix of challenges, from technical to management. It would be completely fine for you to move on and join another company.

The key thing is: do you believe this company has a chance to succeed? It sure doesn't sounds like it. If so, then why do you stay? Loyalty is nice, but after trying for 7 years, it's ok to move on.

Don't worry about the non-compete, unless you really go work for a _direct_ competitor. Don't take clients away, don't take business away.

<comment><AUTHOR> To clarify: I give it a decent (but still less than 50%) chance of having a reasonable liquidity event. But I give it no chance of having that event before I sink another year or two of opportunity costs into it. It's a decision I weight constantly. And yes, I know I'm probably buying into the Sunk Cost Fallacy.

</comment></comment><comment><USER11> I would leave in your situation in a heart-beat. If leaving is not a credible option the company won't pay you a fair market rate and won't treat you with respect. I am not a lawyer, but if the company comes after you with a non- compete it seems reasonable that you could sue them for your back pay. Assuming your non-compete is for a year or so this is almost a wash in terms of salary especially when factoring how much more money you'll make over the long run with a higher salary.

The fact that you are even asking this question means you probably know the answer already - you have the most information about your situation so ultimately you should treat all advice here with a grain of salt.

</comment><comment><USER12> > It would take > 3 senior people to replace me.

Everybody thinks this and it's rarely (if ever) true.

> my non-compete pretty much covers every industry you can think of.

They can't afford to pay you but they can afford to sue you?

> The company owes me more than $60k back pay.

You are never going to see this money.

> I know for certain 3/5 of the remaining devs would definitely leave if I > left

You don't know this for certain and even if it is true, not your problem.

<comment><AUTHOR> Your first point is a fair one. Although in this situation I think it falls under the "rarely" category, for a few reasons:

\- I do architecture, middle-tier, front-end, backend, DBA, devops, and engineering management. I have only met a few people my entire life at my level with all of those skill sets.

\- I work an ungodly number of hours and a single new hire is unlikely to be willing to do the same.

\- I have a significant amount of proprietary knowledge that while not unusually is very unusual to find in a single person.

I can't say for certain it would take three people but I can say for certain it is either multiple people or one extremely rare person who is going to have a lot of demands and take a long time to find.

And I say this as someone who does hiring. I have interviewed literally hundreds of people.

Per your second point, as I answered previously. I'm not entirely sure the CEO and/or investors wouldn't fund the suite personally. I can't go into detail here on why I feel that way. And even a small chance at a suite is not something I'm willing to risk.

Per your third point, it was more. They have actually paid me back some so I'm optimistic about the rest.

Per your forth point, either its true or they lied and I can't see why they would lie. The team team trusts me and they confide in me. More so they they do with the CEO or even HR. It may not be my problem if they leave but it doesn't bode well for the company.

Edit: as an aside we have had a job listing out for someone to take over just part of my job. We haven't even had anyone apply... and no, I didn't sabotage the listing. HR wrote it.

<comment><USER12> > I can't see why they would lie.

To encourage a workaholic coworker with an overinflated ego to leave so they could replace him with someone they like better?

Not saying this is the case, but from what you've said, the possibility is certainly there. Also, how do you know what they are/aren't saying to HR?

<comment><AUTHOR> I had a longer reply but I deleted it because I was here for advice not to defend myself and I would get nothing but terrible useless advice if I lied.

The possibility of your explanation being legitimate is certainly not there. It only makes sense in a world where someone following me out the door is a good thing. And seeing as I have a lot of stock options it is most definitely not a good thing. It is motivation to stay not leave.

I'm sorry you are so jaded you think someone can't be good at their job.

I'm also sorry you apparently live in a world where complaints to HR go unacknowledged. Because at our company they get addressed head on, not buried.

I'll rest easy at night knowing that 7 out of 10 of the developers who left have kept in touch after and thanked me for my mentorship. Something they had no obligation to do. Because if you treat your team right, as I do, and you consistently have their back and mentor them as I do, you don't need to guess if they are telling the truth.

</comment></comment></comment></comment><comment><USER13> Sounds like you loved working at the company and got a lot of responsibility. However in our field there needs to be progress, and it sounds your company is more in zombie mode. Have a heart to heart with your CEO and ask where he sees the company 5 years from now. The market is really good now and if you are just doing soso you will not survive a down turn.

</comment><comment><USER14> "My non-compete an invention clause are very strict and I live in a state where they are legal. We do consulting to pay the bills (the lean startup way) so my non-compete pretty much covers every industry you can think of." Show it to a lawyer if it is very broad it most likely would be tossed out in court.

</comment><comment><USER15> Well if you haven't been paid wouldn't that be breech in contract in which case I don't know how the contract would be enforceable.

<comment><AUTHOR> The cases where I wasn't paid, my employer has a written agreement that I am temporarily taking a salary decrease in exchange for a deferred bonus with no date or time limit on the bonus so it's not quite that clear cut.

I'm not a lawyer though, I'm sure if I want to get dirty I could find a way to bend it to my will but I'm not eager to get in the mud, I just want to build great software and get paid a fair salary for it.

</comment></comment><comment><USER16> Is the company working on an investment round or the future outcome depends only on revenue?

<comment><AUTHOR> The company is break even, no sign of an investment round at the moment. I don't think the CEO has any desire to fund raise.

We've had multiple seed rounds (about $5MM total) but haven't had one in over four years.

Edit: break even with all the software engineers being underpaid by at least 15% and myself being underpaid by 50%.

<comment><USER16> Well, it is a bet. You can define a time window where you accept the current working terms expecting enough profits/an exit/whatever that give the team better salaries and incentives. You can estimate if this outcome is possible, it is just a dream, or a lottery prize. It would be good to communicate partially or completely your position since it will be irrational to continue in this situation for the long term.

I think beyond this time window you should leave.

</comment></comment></comment><comment><USER14> Your CEO is not reading HN?

<comment><AUTHOR> No, he doesn't.

But even if he did. I've been careful not to include any answers that don't have plausible deniability and I have changed a few key numbers in such a way that it doesn't change the question significantly.

Edit: even if he did, I think with a few minor exceptions if he read this it gives me more leverage not less.

<comment><USER17> You obviously don't think very highly of your boss. Anyone who knows you should be able to immediately tell who you are just by writing style and pretension alone.

<comment><AUTHOR> Not true, I don't think that is obvious at all. In fact, I think very highly of my boss. If I didn't, I would have left years ago. It's one of the things that makes the idea of leaving so difficult.

Second, that is a very specific soft skill that not everyone has. You might have that skill, I don't think everyone does. If I saw my boss' writing I'm very sure I would not recognize it.

Edit: adjectives I've used to describe him in this tread... nice guy, generous, ruthless business person. With the exception of that last one they are positive, and depending on who you talk to, ruthless is a positive word in business.

</comment></comment></comment></comment><comment><USER14> What % of revenue is generated via consulting?

<comment><AUTHOR> Near 100%. The non-consulting amount is not statistically significant. Why?

<comment><USER14> From the outside it looks like the owner is using "startup" as a cover to run a consulting shop with significantly underpaid stuff. In any potential liquidity event given the revenue structure and company age it's likely that it pretty much will be valued as a consulting shop with 6 consultants.

<comment><AUTHOR> We do have products that are made in-house but none that generate even enough profits to cover a single salary.

It didn't start that way. The consultancy came much later as a temporary alternative to shutting down. Unfortunately (fortunately?) it's been successful enough that it shows no sign of being temporary.

<comment><USER14> Sounds like you would be much more successful as a consulting shop then :)</comment></comment></comment></comment></comment>


Ask HN: What's a young hacker to do? - <AUTHOR>

Aside from being careful with our grades, what should young (high school) programmers be doing and aspiring to? I program as often as I can and (of course) check Hacker News for the latest. Recently, however, I've hit a wall. When I code, I worry that I will have nothing to show for it, though I might complete a project. What did you do as a high schooler and what do you think we should do? <comment><USER2> I think its important focus on the development of your soft/social skills in addition to your technical skills. The combination of both will help you get far in interviews. Otherwise, if you go the entrepreneurial route, they will help you understand your customer. At the end of the day, we all need to communicate, and ideas tend to be only as valuable as we can communicate them. (this isn't a criticism, you seem to write well!)

Also, have fun with your programming and experiment. Getting experience in fields and languages that you enjoy will enhance your likelihood of finding a job in those areas. The sooner you find your passion, the more experience you can build up before you look for full-time income.

<comment><AUTHOR> I remember speaking to a university student at a Q&A session. Someone asked him, "what class did you take in high school that you feel helped you most here?" and he replied immediately, "AP English." He said it gave him the interpersonal and critical thinking skills that he needed to be successful as an adult.

So, yes. I guess communication is a big deal. Thank you!

</comment></comment><comment><USER3> I'm a sophomore in high school and I completely understand your dilema. I would say just _do_ it. Build the app (or other type of project) you've always wanted to, and complete every part of it. You'll need to hire a designer if you want people to use it, but many professional designers are willing to trade services, so you don't need to spend anything there. What you do after you release your product depends on what it is. If it's a web app, then launch it on a good server, but something that doesn't cost too much. I'd recommend an EC2 server which is relatively low cost and gives you a free trial long enough to see what users will think of it before you make too big of an investment. If it looks like it's going to be successful, then you just succeeded in your goal. If not, you only made a small investment and try not to get discouraged; build something else.

Apart from that, I would recommend you join forrst if you haven't already. If you haven't and need an invite, I can giv you one. Forrst really helped my programming skills go from just a hobby to something that people ask to hire me for without me ever having to search for clients.

Forrst also helped me get my name out there. It's made my posts visible to enough developers that I've had my stuff tweeted about by people with over 20,000 followers.

But above all I'd recommend to just learn as much as you can and get as much experience as possible. If you want to make a career out of it (which seems like a good fit for you), there's no such thing as too much knowledge or experience, it's what can separate you from the crowd. If you can finish college with a degree in another science field _and_ 6 or so years of real life programming experience, you'll be leagues ahead of the competition.

Best of luck!

<comment><AUTHOR> Thanks for the advice. I just now made a forrst, since they let you in if you have an active github.

Best of luck to you as well!

</comment></comment><comment><USER4> I'm in High School right now (the Irish equivielent anyway).

Last year, I entered a Twitter-related project into a Science competition and I won my category. I've created software ranging from Copyright-protection to <http://spreadapp.co>.

My point is, there is no limitation to what you can do, the sky is the limit. Think of a cool project you'd like to work on, and make it to the best of your ability, and don't let anybody hold you back. You'll find at the end that you have a great piece of software (or a product) that you can have to show for your work, or sell!

Good luck in the future!

<comment><AUTHOR> Thanks. Best of luck to you as well!

<comment><USER4> No Problem, thanks :)

</comment></comment></comment><comment><USER5> As long as you work on things that interests you, then just keep working at it.

When I was in middle/high school, building forum communities from PHP and MySQL were all the rage. So I attempted to create my own community by buying a Linux server, installing PHP and MySQL, and installing the appropriate forum software. Then I proceeded to add mods to it and write my own mods. It was a fun project to learn Linux and PHP on. The best part was that most mods were 'modular' which made them easy to install and dissect the code of.

</comment><comment><USER6> I didn't particulary get into any of the two, but if you like algorithmics you should check out Topcoder and <http://ioinformatics.org>.

Topcoder is full of highschoolers or young undergraduates. IOI is specially targeted towards them, though I think you need sponsorship from your educational institution to participate.

Faring well in any of the two is something cool to add to your CV, and is highly valued by many firms in Silicon Valley.

<comment><AUTHOR> I did Google Code Jam this year, which was really rewarding. Topcoder looks like it could be even better, especially with the frequency of competitions. As for IOI, my school only has a robotics team (which I am a member of), so I might start a group to compete in that. Thanks!

</comment></comment><comment><USER7> Consider applying for an internship with the government? [http://www.nsa.gov/careers/opportunities_4_u/students/index....](http://www.nsa.gov/careers/opportunities_4_u/students/index.shtml)

<comment><AUTHOR> Thanks for the link! I definitely have thought of interning, it's just that most internships are for college students only.

I'm thinking of bothering some Seattle tech companies as well to see if I can convince them to take me for a summer.

<comment><USER7> The program is specifically designed for High School students. I would check it out, if you are interested.

</comment></comment></comment><comment><USER8> Build something that makes money from other businesses to get filthy rich.

</comment><comment><USER9> Hey Guys, this thread is amazing. I'm also a high-school student, I currently reside in Illinois.

Please feel free to email me, and we can arrange a meeting.

Email: firstname.lastname@example.com</comment>


WiGLE: Huge WiFi Networks Geolocation Map - <AUTHOR> https://wigle.net/ <comment><USER2> For some reason I feel violated that it shows my wifi network's name and mac.

<comment><USER3> For me, that's something like "linksys" and AA:AA:AA:AA:AA...

Feel free to steal my config.

The more the merrier.

<comment> T-A You may want to change "linksys" to something slightly less common:

[https://www.lifewire.com/is-your-wireless-networks-name-a- se...](https://www.lifewire.com/is-your-wireless-networks-name-a-security- risk-2487658)

[https://wigle.net/stats#ssidstats](https://wigle.net/stats#ssidstats)

<comment><USER4> My final project for undergrad used this service along with WiFi packets sniffed from coffee shops to show the privacy implications of leaving WiFi on in public.

Since phones/laptops send packets containing the SSID of saved networks it is possible to locate home/workplace of a device, assuming the networks at those locations have a somewhat unique SSID.

From that perspective, I'd say a common SSID is better to have than something unique.

<comment><USER5> They only send SSIDs of 'hidden' networks. There is no reason to send SSIDs of normal visible networks because they are visible if you need to connect.</comment></comment></comment></comment></comment>


We don't write tests. There just isn't time for luxuries. - <AUTHOR> http://jamesgolick.com/2007/8/28/we-dont-write-tests-there-just-isnt-time-for-luxuries

<comment><USER2> I'm a chronic manual tester. But I want to start getting into automated tests.

What are the most important things to test? What are some good intros to pick up the basics on writing automated tests?

<comment><USER3> I personally think that test/behavior driven development is the way to go, so my advice will be geared in that direction. With that in mind, the "reference" text on it is probably "Test Driven Development: By Example" by Kent Beck. I've only skimmed it, myself, but it seems solid and Kent Beck is a smart guy.

The cool thing about testing (and TDD in particular) is that even introducing a bare minimum of that sort of discipline into your work consistent will quickly pay pretty big dividends. A quick way to get started is to read this Wikipedia article: <http://en.wikipedia.org/wiki/Test-driven_development>. Focus on understanding the TDD cycle. Mocks and stubs are useful to know, but you can safely consider them "advanced" and skip them if they confuse you initially.

Then, pick a test harness for your platform, say RSpec (<http://rspec.rubyforge.org>), if you're a Ruby person. Read the documentation and search for blogs and tutorial articles on the framework you've chosen. Then just start designing a simple module using it!

As you practice, you'll find it useful to read up on Mocks and Stubs and dig deeper into the philosophy and best practices. Dave Astels has some great writings out there.

<comment><USER4> I personally found Kent Beck's book very tedious and boring. In it , simple classes are bludgeoned into submission through the over use of tests.

From Ganssle's article at <http://www.ganssle.com/articles/deconstructing%20xp.htm>

"XP's motivating guru, Kent Beck, also has a new book out. Test Driven Development (Addison Wesley, 2003, ISBN 0-321-14653-0) focuses on XP's testing practice.

I bought the book because I'm fascinated with testing. It's usually done as an afterthought, and rarely hits the hard conditions. Unit tests are notoriously poor. The programmer runs his new function through the simplest of situations, totally ignoring timing issues or boundary conditions. So of course the function passes, only to inflict the team with agony when integrated into the entire system. XP's idea of writing the tests in parallel with the code is quite brilliant; only then do we see all possible conditions that can occur.

Beck argues for building tests first and deriving the code from these tests. It's sort of like building a comprehensive final exam and then designing a class from that. An intriguing idea.

But X what if the test is wrong? Test driven development (TDD) then guarantees the code will be wrong as well.

Worse, TDD calls for building even the smallest project by implementing the minimal functionality needed to do anything. Need a function to compute the Fibonacci series? Create the simplest possible test first X in this case, check to see that fibonnaci(0) is 0. Then write a function that passes that test. Try fibonnaci(1); that breaks the test, so recode both the test and the function. Iterate till correct.

The book shows how to build a factorial program, which results in 91 lines of code and 89 of test... after 125 compilations! Klaxons sound, common sense alarms trip.

Agile proponents love the interactive and fast action of this sort of method. But programming isn't about playing Doom and Quake. If you're looking for an adrenaline rush try bungee jumping.

125 compilations for a trivial bit of code is not fast. Dynamic? You bet. But not fast.

"

Test _Driven_ Development is not quite the same as having unit tests or automated tests, which I believe the poster asked for. The jury is still very much out on how useful TDD (and XP for that matter) really is.

Many talented programmers believe XP/TDD is essentially hype. . This is not to say Pius is wrong, just to add a note of caution, be skeptical of the XP hype machine.

"Try it and decide for yourself" is probably the best advice possible

<comment><USER3> I totally agree with your last statement.

</comment></comment></comment></comment><comment><USER5> "Everybody Tests" is a good point. Some devs throw away their tests, and other devs keep the tests around and run them periodically, but devs who write code iteratively (which seems like pretty much all of them) write something that could be easily turned into a unit test.

If you don't like TDD (and I have to admit I'm not a fan), you can still produce tests while you write code (you probably already are). Whenever you write a bit of code, you probably do something to check whether it is producing the expected outcome - even if it is as simple as writing a little script to generate some output that you visually inspect as it flashes by on the screen. If it looks good, you keep developing. If it fails, you figure out why.

Just store your these mini-testing scripts in a unit test, and verify the expectation/output in a short statement. If you use a framework, you'll have some methods for this available to you - if you don't, no biggie, just write something that honks loud and gives you a message diagnosing the failure.

It's not going to satisfy the TDD crowd, and honestly your coverage won't be as good as if you had done TDD, but there you go - you just got yourself some unit tests, and you didn't have to disrupt the mental flow that you prefer to TDD.

</comment><comment><USER6> I wrote some opinions on this in other thread:

<http://news.ycombinator.com/item?id=46526>

Unless your software will kill people if it fails or something, my advice is: observe what proportion of your development time you spend debugging. If it's one half or more, unit testing would probably benefit you. If it's significantly less than 50%, then you're probably better off riding bareback, because writing and updating those tests will take at least half of your time.

The problem I see with the "you'll test anyway, you may just as well automate it" argument is that unit tests are no substitute for testing with real users.

</comment><comment><USER7> "the break even point for an automated tester is when they've run their tests twenty times"...

I find that (at least in early development), when I'm in the zone I refactor several times a day (differnent parts of the system of course). If my break even point is twenty times, I'd never get there. Every refactoring would virtually obliterate all of the previously automated tests.

</comment><comment><USER8> I've recently begun using automated tests of basically all my code. Both frontend and backend, when I'm doing web stuff. I usually write the code to a minimum first, and then implement tests. It's a huge relief to use these tests when I refactor and refine code. I definitely think automated testing is the way to go in most cases.

</comment><comment><USER9> Sometimes it just does slow you down overall.

Remind me again, how many web startups have failed because of a bug? How many? Hm.

<comment><USER10> Are you implying that the number is very small or very large?

<comment><USER9> SMALL. Are you implying that it's not?

Or maybe you're with the aaron crowd in believing that it has to be absolutely perfect or it's useless. Heck, even that only could really apply to the design and architecture of an app.

Perfection is rarely necessary.

edit: And what's with so many downvotes? I thought yc was more mature.

<comment><USER11> How many teams fail to execute and lose precious time to market debugging things that they could've caught earlier?

I am not a 100% TDD fanboy either, but it's not something to dismiss wholeheartedly.

<comment><USER9> That's why I said sometimes. If you read the article, it claims that this is never the case. I presented my disagreement with this.

<comment><USER11> Sorry, I think you're right. There are cases when it can slow you down overall.

Adding tests up front, from my experience, adds about 40% overhead to a project. (time spent actually writing the tests) You may get some of this time back later, as the author claims, as the automated test suite is run.

I've also been on projects where the code & entire test suite was re-written & scrapped after 6 months of initial development, pretty much making the whole test up front thing of little value.

<comment><USER9> To be fair, it seems that everyone other than I took this in the context of more traditional software project, not an ill-specified one man startup project.

I was referring to the later.

<comment><USER12> ok, so now I get it. TTD is better suited for those large companies where people don't know what the f are doing, so they need to test everything (just making sure), and when speed doesn't matter, as big companies everything goes slow.

The same enviroment and thinking that brought us java, and made it a clusterfuck of "abstractions", and "design principles".

I'd rather just code ad design at will, and be able to refactor, change, or throw away code without being held hostage of these tests which will break at some point, not because your code is not right, but because they were design to test something according to a thinking, and when that something changes, all those tests are obsolete.

<comment><USER13> I think the problem that you might be facing is more related to the quality of your tests than testing itself. If your tests are written correctly, you should never have to throw them away when you refactor.

In fact, that's one of the major benefits of testing: a refactoring safety net!

<comment><USER3> Right. How can you refactor code that has no tests? Basically, you've got no clue if your new refactored code works because all the precious time you've spent using the old code went towards ephemeral tests that weren't written down. That makes refactoring a _lot_ more risky than it needs to be.

<comment><USER3> What I meant to say is "How can you _effectively_ refactor code that has no tests?"

Refactoring is inevitable. I find that on projects where I've written tests, refactoring goes a _lot_ better.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> I have always believed that there is an inverse correlation between the amount of precise attention devoted in design/development and the amount of time spent in any debugger. Nothing here changes that thinking.

</comment><comment><USER15> I dislike writing more code for my tests than the "actual" code. I do not eat the "test driven development" drivel hook-line-and-sinker. Sorry, call me an unenlightened developer.

<comment><USER12> I am in the same boat. Maybe TTD was invented for people certain kind of people that like to "plan" stuff well in advance? Maybe my brain is not wired for TTD, but I tried to like it, and I couldn't.

I can see how TTD is important in critical software (avionics, banking, trading) etc... but for the average web 2.0 speed of execution should be paramount, and I think TTD just slows you down. You end up writting x2 the code, plus everytime you have to change something, you have to do it twice.

In mobile client development, if found TTD pretty useless, unless you are doing mission critical things, where speed of writing a software is less important than the accuracy of it. (imagine phones crashing, not a good think).

<comment><USER3> Just a quick note on "mission critical." I've always found that to be an interesting term. I'd argue that software components critical to your business are mission critical, practically by definition. It doesn't matter if I'm an old-school avionics company or a Web 2.0 social bookmarking site . . . my software is mission critical to my business.

<comment><USER16> Crashes are a lot worse for business if you're an avionics company than if you're a social bookmarking company.

<comment><USER17> I second that. If you have something like a display bug or some strange inconsistency in a website, having that bug there is actually a chance for you to show how responsive you can be to user bug requests. That is, having and quickly fixing a bug could turn out to be a good thing. That is not the case if you are NASA or your local nuclear power plant.

<comment><USER14> "having that bug there is actually a chance for you to show how responsive you can be to user bug"

and how sloppy you were up until now</comment></comment></comment></comment></comment></comment>


Musopen Wants to Give Classical Music to the Public Domain - <AUTHOR> http://www.kickstarter.com/projects/Musopen/record-and-release-free-music-without-copyrights

<comment><USER2> _Right now, if you were to buy a CD of Beethoven's 9th symphony, you would not be legally allowed to do anything but listen to it. You wouldn't be able to share it, upload it, or use it as a soundtrack to your indie film- yet Beethoven has been dead for 183 years and his music is no longer copyrighted._

Wow, talk about missing the point: Beethoven has been dead for 183 years, but the musicians in the orchestra that recorded it certainly haven't been.

Having one random orchestra (even one with international renown) record the works of Beethoven, and releasing those to the public domain will only help people who want to re-use that single interpretation of the works in their indie film.

The whole project seems to me to be of minimal value; what they're really doing is hiring one orchestra to produce a very low budget version of some classic works, so that some future group of people can avoid paying the license costs for those (likely mediocre) versions.

<comment><AUTHOR> I agree with your point, but I think it's an interesting project, nevertheless. Mostly, I'd like to see if there really would be enough people to make this happen. Also, chalk it up to morbid curiousity as to how good their results would actually be; there doesn't seem to be any hints as to which "internationally renown[ed]" orchestra they have their eye on.

In any case, the value of this project would be exactly how much people are going to assign to it. If nothing else, maybe I'd end up with more legit music in my collection. :)

<comment><USER3> the orchestra depends entirely on how much money we raise, and what pieces of music the donors want. I can hire the london symphony, but I personally would rather get something less well known and more music.

</comment></comment></comment><comment><USER3> If you don't want to donate but want to help, we are also looking for votes from our Pepsi Refresh grant

<http://www.refresheverything.com/musopen></comment>


Complexity in art, music, and cellular automata - <AUTHOR> https://medium.com/@_NicT_/its-actually-pretty-simple-cda17a589f39#.dijtvye37 <comment><AUTHOR> Hi there, have been reading a little about complexity theory here and there for a while now and thought it would be fun to put together some thoughts in a blog post. Sharing here in case anyone finds of interest.</comment>


JavaScript development environment? - <AUTHOR>

What do folks use for JavaScript dev environment?
I am trying to setup new extjs based system and so far I like everything I see, except I can't figure out how to properly develop the thing. I am very much Eclipse person (last 8+ years) and while I am willing to move away from Eclipse I would like to have things like refactoring, type browsing, class hierarchies and integrated debugging.
Do this exists either as a standalone product or an Eclipse plugin?
Many thanks for the insight and suggestions <comment><USER2> For the features you're looking for, the other commenters gave you the two best-bets that I'm aware of. Personally, I use Visual Studio 2010 and really like it, primarily because of the multi-pane, multi-monitor support. In other words, I use it as a really fancy text editor. I do code heavily in ExtJs. For integrated debugging with ExtJs, Firebug is totally indispensible.

Getting started with Ext can be hard. I'd suggest starting with the demos and tweaking the code from there for your needs. The API reference online is very useful. StackOverflow is about on par with the forums for questions.

Good luck.

</comment><comment><USER3> I use Netbeans, it has really good JavaScript support, all the standard stuff you would expect, plus it reconizes most of the JS libraries and will do auto complete for them. They used to have in IDE Javascript debugging in 6.7 but 6.9 and 7 do not have the plugin, they say it is coming back at some point. It was really nice to set break points in the IDE run the browser and be able to step through the code in the IDE. You could step through the front end, to the back end, to the database, and back again.

</comment><comment><USER4> Aptana Studio is based on Eclipse: <http://www.aptana.com/products/studio3>

Another JavaScript IDE is WebStorm: <http://www.jetbrains.com/webstorm/>

</comment><comment><USER5> When I use in a proyect js, I use eclipse with a simple java web proyect and tomcat with the javascript files inside the folder etc, but I only use it for some things, so I think you should use what fit to your needs man :D</comment>


Wolfire Games: Are games art? - <AUTHOR> http://blog.wolfire.com/2009/01/are-games-art/

<comment><USER2> Step one: Define art.

Step two: Define game.

Step three: Read off from your definition whether games are art.

It is only people's insistence on running this algorithm backwards, or that there are universal definitions for either term, that keep this pointless debate alive. The answer is yes and no and maybe, for some good definition of either. It can be worth thinking about, but don't think you can actually answer the question once and for all.

You'd think after the thirtieth or fortieth medium that we'd all understand this. "Well, I guess musique concrete is art and I guess interpretative dance is art and I guess these 'moving picture' things are art and I guess the crucifix in a jar of piss is art, but I DRAW THE LINE at video games! No!" Sheesh. What a crock.

<comment><USER3> Step one: Cut a hole in the box.

Step two: Put your junk in that box.

Step three: Make her open the box.

And that's the way you do it!

</comment></comment><comment><USER4> There are two largely accepted definitions of art. The first is the view of art simply as the mastery of a certain action or craft. The second is the view of art as something that changes the person that sees it.

By the first definition, video games are inherently piece of art so long as there is an objective standard behind games. There's an art form behind, say, the comprehensive control scheme from Super Mario 64, which is in my opinion the best platforming control scheme I've ever used. There's an art to creating responsive and beautiful worlds.

By the second definition, games still qualify, but we're in a poor position in that thus far games are still parroting movies by-and-large. The large advancements that we've seen in gaming - I'll cite Shadow of the Colossus and Grand Theft Auto IV as the two pinnacles - are ones that, in terms of message, are still parroting movies. Shadow of the Colossus gives us the sense of loss, Grand Theft Auto IV gives us the sympathetic - and fully-fleshed - character. These things can be compared dismissively towards movies, since they're far more primitive in nature.

That's not to say that video games will remain as such. People are entering the field with ambitious minds right now. Many games are starting to hint at an expansion towards the sorts of things that gaming and gaming alone has to offer. I think that the "are games high art" argument will be moot 20 years from now, if not sooner. The gap is closing.

(I'll also say that if not the games themselves, music in video games is the least compromised that any forum of music has been for a century. I'm thinking of Nobuo Uematsu, whose soundtracks are regularly 4 discs+ long, who releases piano arrangements and orchestral and rock arrangements of all his pieces, and who - unlike, say John Williams - is given an incredibly wide scape of emotions to portray with every game he works on. His output is impressive, as is the output of Koji Kondo, though he isn't nearly as evocative in my opinion.)

</comment><comment><USER5> Interestingly I think this article misses a fundamental point.

I have a dear friend, a long time artist (who went to Goldsmiths and the Royal Academy) and who paints 8 paintings a week, rain, snow or shine.

He has always talked about the need for art to emote, or at least provoke thought in (in the case of much modern art) the viewer. His art, he says, is trying to capture a moment in his life and display that in a way that others might glimpse his life even though they could never be him.

By these criteria perhaps video games present the most formidable medium in order to achieve the lofty goal. While one cannot directly control the exact interaction a good video games can steer a player in the same way a good director or leads the audience or an author leads her readers. By setting up scenarios and situations which create the opportunity to feel or think an idea that is entirely novel to us because we have been guided to it.

The power of interaction can bring this to a new level over traditional mediums because of the exploration of the player. In basic computer science we are taught about the difference between data, information and knowledge. The knowledge being that information which a person is not only able recollect but also directly apply. As an IT trainer I know from experience the way people gain knowledge is by taking an abstract piece of information, and discovering how it works. The phrase "let them work it out for themselves" is common, because it reflects the simple truth.

If art is about experience, emotional or cognitive, interaction offers the most concrete vehicle of human learning to reach states of exerience as never before. I would contend that video games are indeed art of the highest order.

</comment><comment><USER6> I really didn't get Shakespeare until I saw it performed. His plays are plays for a reason, after all. That's enough (for me) to blow away the "requires authorial control" theory: no two performances are the same.

Or any book: everybody reads it and has a different mental image of what's going on. (How many people read Dune and saw it as Lynch did?) In a book, you're given fixed words and given free reign on the visual and auditory axes. In a game, you're given fixed audiovisuals and given free reign on plot. I don't see how one could be inherently artistic, and the other not. Only a movie reviewer (where virtually everything is fixed) could claim that lack of viewer participation is the defining attribute of art.

</comment><comment> matthew-wegner I guess Ebert's definition meant to have another word:

"Video games by their nature require player choices, which is the opposite of the strategy of serious film and literature, which requires [EXCLUSIVE] authorial control."

Games simply share this authorial control (some games share more than others, like Spore or Little Big Planet). If you take Ebert's definition literally then, indeed, games are art, and it kind of backfires on him.

<comment><USER7> There's a strong misunderstanding going on here of what "authorial control" means, though. Anything you can do in a game, even a free-form game, is taking place within a system of rules set down by the game's designers and programmers. Games are interactive, and obviously accept input from the player, but they are as strongly, and really, as exclusively authored as any other medium.

Ebert comes at it entirely from a film point of view -- video game players can (depending on the game) have control of the camera, of the movements and decisions and dialog choices of the protagonist, or even in some cases, control of the narrative! Every movement in a game, though, is restricted (or it might be better to say, "allowed for" or "enabled"), by the designers. Unless it is a poorly constructed game, you can't go walking off out of the game's predetermined accessible areas. Every line of dialog was written by a writer, or recorded by a voice actor (or, if it uses some sort of procedurally generated speech system, is operating under rules built by a programmer and game designer). Some games are more forgiving of "emergent" moments than others -- having gameplay systems which are built to intersect and collide in unexpected ways -- but all of those underlying systems were designed and built with the purpose and expectation of creating interesting collisions.

The problem is that Ebert is framing games like he'd frame a piece of serious film or literature -- defining authorial control in terms of narrative, or at least in terms of character, which is ridiculous. If Ebert ventured into a gallery space and encountered an interactive art installation which created some sort of projection which responded to his body movement, voice, and an artist-chosen soundtrack, would he declare that it inherently wasn't "art" because it wasn't a painting?

I think he's confusing the presence of a protagonist and the use of filmic storytelling tools and cinematography for a classic narrative work, and then sees players diverting that narrative and shaping that narrative and tearing it apart, and decides that there can't be anything valid there.

Narrative can be a great tool, or even a framework, to build into a game, but the rules of conventional narrative not what games are inherently about, and that makes them a misguided criteria by which to judge them.

<comment><USER4> The big point to be made here is that every form of art is inherently about the medium. Games have different rules from movies. Camera work doesn't matter, but creating a world such that every camera view looks equally beautiful does. The authorial control isn't the control of the actual actions but of the _results_ of said actions.

<comment><USER8> Yes. And camera control does matter. But for most games programming the camera is more like good craftsmanship than art. (This is the way it should be for those games.)

<comment><USER4> The one really good camera movement I've seen in gaming is in Shadow of the Colossus, where the camera blurs when it moves too quickly. It means that suddenly there's a visual effect for switching camera angle. And for whatever reason, most shots in that game are gorgeous without the player trying.

</comment></comment></comment></comment></comment><comment><AUTHOR> Good response to roger ebert and his ranting about the topic.

<comment><USER4> Yeah. Shame on Ebert. I love the guy and he's an incredible writer, but I think he's wrong on this one.

</comment></comment><comment><USER5> I was disappointed the author flaked out of drawing his own conclusions. I felt that he raised some reasonable, if obvious, points about the dismissal of new mediums throughout the history of art.

</comment><comment><USER9> Is sound art? Is a colored plane art? Is a rock art?

It depends.

</comment><comment><USER10> Yes. Art is in the eye of the beholder. The end. Can we let this silly discussion die please? (I didn't read the article)</comment>


Don't Play with the Odds: How the Birthday Paradox Hit Us While Coding Solid - <AUTHOR> http://blog.getsolid.io/birthday-paradox-coding-solid/?utm_source=hn&utm_medium=referral&utm_campaign=bdaypb

<comment><USER2> I wrote about this some time ago[0] and submitted it here on HN[1], where there was an interesting discussion. That article includes a complete derivation[2] that's simply quoted in this submission.

[0] [http://www.solipsys.co.uk/new/TheBirthdayParadox.html?HN-201...](http://www.solipsys.co.uk/new/TheBirthdayParadox.html?HN-20150513)

[1] [https://news.ycombinator.com/item?id=4753014](https://news.ycombinator.com/item?id=4753014)

[2] [http://www.solipsys.co.uk/new/TheBirthdayParadox.html#toc_na...](http://www.solipsys.co.uk/new/TheBirthdayParadox.html#toc_name002)</comment>


The Need for Both Makerspaces and Incubators in Africa - <AUTHOR> http://whiteafrican.com/2013/02/22/the-need-for-makerspaces-and-incubators-in-africa/

<comment><USER2> Makes me think of "The Boy Who Harnessed the Wind." A very cool book, I would highly recommend it.</comment>


Show HN: Have I Been Pwned donations page is awesome - <AUTHOR> https://haveibeenpwned.com/Donate <comment><USER2> He has great taste in beer.

In all seriousness I hope people of means on here donate the $100 to take his wife out to dinner. Partners of folk doing this type of work never get recognition for the work they do to enable it

<comment><USER3> I always look up the blockchain addresses for the donation links out of curiosity and it's awesome to see that the creator this project has received 12 + btc! Well earned for providing such a great service and doing it without expectation of this reward.

</comment></comment><comment><USER4> My main complaint with this site is that looking up by email address isn't particularly helpful since mine shows up in several breaches. What I'd like is an itemized list of services involved in a list that my email appears in so I have something actionable to work with.

I just checked my password manager, and I have over 100 accounts, most of which use the same email. Changing my password on all of those services isn't practical each breach, nor is opening a bunch of separate email accounts to make tracking down affected accounts easier.

I like my current password manager's (Bitwarden) feature of telling me which accounts have weak or compromised passwords, but it doesn't include breaches where passwords weren't leaked, like the most recent major leak.

I love this site, but I don't find it all that useful honestly. Maybe this just means I need to get a new email address and slowly migrate accounts to it.

</comment><comment><USER5> This is a great service and Troy gives a lot to the community generally.</comment>


ISPs "exaggerate the cost of data" - <AUTHOR> http://www.pcpro.co.uk/news/broadband/370393/isps-exaggerate-the-cost-of-data

<comment><USER2> I have worked for ISPs where people who are making decisions about "bandwidth" _do not understand_ the difference between "bits" and "bits per second."

These decision-makers would say that "Our costs for providing backhaul / transport / whatever is going up - no matter how much we upgrade, its capacity is exhausted either immediately or very quickly. We have to recover that cost somehow, or we won't be able to continue providing any service."

However, they don't understand that their equipment and transport are constrained at the level of "bits per second at peak" and not "sum of bytes transferred during a billing period." So, they end up making bad decisions about "charging $x per GB after y GB transferred per month."

<comment><USER3> How long until we get to peak and off-peak rates for data usage by ISPs, much like some electrical utilities?

<comment><USER4> That's certainly how it works for most Australian ISPs. And the off-peak times only run from about 02:00 to 07:00. I'm lucky enough to have an unlimited ADSL account with TPG, or my quotes would be a couple hundred GBs for the same cost.

Luckily, I think competition is going to be making things better for the Australian ISP consumer. That, and recent improvements/additions in undersea cables.

<comment><USER5> I can't speak for your experience with TPG, but a lot of users I know on TPG hate the fact that they have "unlimited downloads" but in reality get limited bandwidth due to an oversold service.

I decided I prefer a limited plan where I know what service level (particularly latency and throughput) I can expect. For instance, plans with Adam and Internode are not _that_ much more than a TPG plan for more usage than I'll forseeably use (I get 500gb a month or so at 20mbps + VOIP for $100), but the difference in gaming latency and actual speeds are both considerably more stable and faster than friends on TPG.

<comment><USER4> My experience with TPG has generally been very good. I connect at 17-18Mb/1Mb and consistently download at 1.0+ MB/sec. Usually 1.4MB/sec, though I've seen it drop to ~800KB/sec.

Given that your plan is 60% more than mine, I'll gladly pocket the difference and live with the minor problems I've had. As another poster mentioned, I chose to vote with my wallet. Generally small quotas + absurd off-peak times (where the majority of the quota is allocated) make my blood boil and I'll support anyone who helps break up the status quo.

</comment></comment></comment></comment></comment><comment><USER6> The report was commissioned by content providers so I suspect the truth lies somewhere between.

Still, at least with my phone I have my choice of carriers and will be switching from AT&T to Verizon this month. I don't have that same choice in choosing a "high speed" ISP... I wish some comcast competition would move into town, the consumer might end up with a better quality product.

<comment><USER7> The truth doesn't have to lie somewhere in between. Look at the last paragraph:

 _Even for ISPs running their own network, such as BT, [ISP CTO] Davies claims the figures of €0.01-0.03 per GB are "rubbish". "It's an order of magnitude greater than that," he claimed._

This is coming from a guy claiming that the "existing smartphone data tariffs of around €10 per GB" are too low. They're profit-seeking, and relying on their regulators to be as bad at math as they are.

<comment><USER8> The problem, at least in Canada, isn't that the CRTC is bad at math, it's that almost all of the people making decisions used to be executives for the ISP and or Telecoms industry.

</comment></comment></comment><comment><USER9> > Even for ISPs running their own network, such as BT, Davies claims the > figures of €0.01-0.03 per GB are "rubbish". "It's an order of magnitude > greater than that," he claimed.

Right. So let's start seeing those prices get in the 0.10~0.30€/GB now shall we?

<comment><USER10> I'm sure you're right that prices are pretty low overall, vs the high rates of overage fees, but I wanted to point out that the same people who can't tell bits from bits per second and dollars from cents probably aren't using "an order of magnitude" in the full scientific sense, but rather in the "a whole lot more" sense.

I'd be interested in what an independent investigation would reveal.

<comment><USER11> Considering that almost everyone has a stake (i.e. as either a user or a provider), I'd imagine that an independent investigation could only be done by a government entity forcing disclosure of this information by a private company.

Actually, does anyone know of a government funded ISP that makes this information public?

</comment></comment></comment><comment><USER12> «Even for ISPs running their own network, such as BT, Davies claims the figures of €0.01-0.03 per GB are "rubbish". "It's an order of magnitude greater than that," he claimed.»

awesome, so it only costs about a quarter for a gigabyte. so i use about 20 GB a month. can i just pay 5 bucks a month? yeah, i didn't think so. i'm pure gravy for them, and what does comcast charge me for 4 to 10 Mbps? $70

and don't get me started about the fixed cost versus the variable cost—the variable cost is certainly not more than a few cents per GB but the fixed costs can be enormous.

i wish the internet was just a utility.

</comment><comment><USER13> This is heartening news because connectivity is becoming a commodity. Will anything come of it?

Recently net neutrality ended and now we have 250 GB/Month bandwidth caps as Comcast customers. I'm not worried about ever hitting that cap, but I am worried about the cap tightening over the next couple of years.

Everyone talks about Google and Facebook's control over their information, but I don't hear enough concern about ISPs' collective control over access to information.

<comment><USER14> Here in Aus we have always had caps. They've done nothing but increase over the years. My SOHO plan with a quality ISP is $80 for 150GB, for comparison, but you can get cheap no-frills residential for $40 for the same quota. I doubt the caps where you are will come down much, if at all.

Last year my quota was 60GB, 1-2 years before that it was 25, so on and so forth.

<comment><USER15> That and these days you can get 200, 500, 1000GB caps, which really are about as good as unlimited as you can get. You'd really have to push to download that much data in a calendar month.

</comment></comment></comment><comment><USER16> I'm not quite sure what point the consultancy is making on the future mobile- data rates. The article says:

 _However, it claims forthcoming 4G technologies will significantly reduce those costs. "Forward-looking estimates which take account of the transition to LTE [Long Term Evolution], additional spectrum and traffic subscriber growth... puts the cost to the mobile network operators at under €1 per GB,"_

Does that include the cost of the 4G licenses? In the UK these haven't been issued yet. If an operator pays £4bn [1] for the license and has 10m customers it needs to charge an extra £1 to every customer every month for 33 years to cover the cost of that license. That's not an inconsiderable amount before we begin talking about data.

[1] Costs based on UK 2000 licenses: <http://news.bbc.co.uk/2/hi/business/727831.stm>

</comment><comment><USER17> Within the UK I think that you tend to get what you pay for from the ISP. They just create irrelevant metrics and restrictions to prove to customers why they should upgrade to a better plan. Perhaps they should just make the ISP's publish typical speeds for a particular location, so that you can make an informed comparison. This is exactly what better informed customers already do, instead of just going with BT.

</comment><comment><USER18> This is the report being talked about for those that are interested [PDF]: [http://www.plumconsulting.co.uk/pdfs/Plum_October2011_The_op...](http://www.plumconsulting.co.uk/pdfs/Plum_October2011_The_open_internet_- _a_platform_for_growth.pdf)

</comment><comment><USER19> In other news: "water wet", "pope catholic", "bears shit in woods".

</comment><comment><USER20> For mobile data you can get variable cost plans in Austria that you pay about 4$ per Gb for. So the cost cant be higher than that.

</comment><comment><USER21> Why are we even having this discussion? It is just a bunch of companies who want a subsidy because.. well because they want a subsidy.

The best way to end it is to slap them with a higher tax until they shut up.

But that would require both balls and common sense.

<comment><USER22> I don't understand this logic, what would the higher tax accomplish?

<comment><USER23> Punishing the greedy capitalists?

<comment><USER24> Or adding yet another hike to your internet bill. Do you really think taxes prevent companies from going up on prices? Of course not. It won't affect their profits at all. They'll just charge customers more to offset the tax.</comment></comment></comment></comment>


I quit my job. Today is Day #2. - <AUTHOR> http://www.nevertryneverfail.com/i-quit-my-job-today-is-day-2/ It appears that my hosting is failing... Posted same article here.<p>https://medium.com/@keslertanner/i-quit-my-job-today-is-day-2-c5f00278e426 <comment><USER2> I'm excited for this guy's optimism, but everyone should be aware this is the typical elation you have when you start to do something new that excites you. This fades, and the fade is much harder when it's dealing with your own enterprises, because it's difficult to produce the motivation to carry an entire company when you're discouraged. At least when you encounter similar problems while employed, you can be reasonably assured that if you can muster the effort to hit the minimums, you'll continue to get your paycheck. Disillusionment becomes much more nefarious when there's a direct inverse relationship to your livelihood.

Being an entrepreneur is not easy, and while it has moments of elation and excitement, like anything else, it also has moments of despondency and doubt. This doesn't mean it's not worth it. It just means, be prepared. Bask in the enjoyment when it's there, because you'll need the memory to keep you going when real life hits.

<comment><USER3> Agreed. The post reads like a honeymoon. I hope the author writes about his experiences a month/year after

</comment></comment><comment><USER4> Day #3: The database is down. I don't know how much longer I can hold out. If anyone reads this, please tell my family that I love them.

<comment><USER5> ...If anyone reads this, please tell me how you got my database back up.

</comment></comment><comment><USER6> Best of luck! Always enjoy a personal piece on branching out, even though I've been freelancing for a decade. Something to do with re-affirming past decisions, perhaps?

Always good to be reminded to break tasks down to manageable sizes also, as I've been caught by that old obstacle of 'task too big, do something easier' more than I should have been recently, given the number of times I've come across it before.

<comment><USER7> I face that same obstacle over and over again, too. Curses!

</comment></comment><comment><AUTHOR> It appears that my hosting is failing... Posted the same article here.

[https://medium.com/@keslertanner/i-quit-my-job-today-is- day-...](https://medium.com/@keslertanner/i-quit-my-job-today-is- day-2-c5f00278e426)

<comment><USER8> Please drop SumoMe too. It's absolutely horrible.

</comment></comment><comment><USER9> Google cache caught it:

[http://webcache.googleusercontent.com/search?q=cache:ykshUmL...](http://webcache.googleusercontent.com/search?q=cache:ykshUmLmRA4J:www.nevertryneverfail.com/i-quit- my-job-today-is-day-2/)

</comment><comment><USER10> I quit my job almost a year ago and never worked a day since, and I never will anymore, at least in the traditional sense. I went from middle class to moneyless and I could't be happier with my life, which involves travelling all over the world, meeting people, visiting friends and partying a lot.

I have never felt more free. One day I will settle down somewhere, have kids and teach them how to live free like I did, so they have to never go through the tyranny of an office job :)

<comment><USER11> You must have some sort of income? How do you afford basics like food and lodging?

<comment><USER10> If I need money, I could basically earn money by playing music or whatever on the street, or DJ-ing in bars/clubs, as this is my passion.

I travel by hitch hiking, there are communities that live completely without money, either by being self-sustainable growing their own food and trading/bartering with locals, or by dumpster diving at super markets or asking i.e. bakeries if they have leftovers they would otherwise throw away.

There are more and more communities popping up all over the world, consisting of people who don't agree with the fact that a lot of still useful stuff is thrown away nowadays (Ending is better than mending?). They come with 'freeboxes' where you can take or exchange clothes and other items for free. If I need or want a change of clothes, I put my old ones in there (washed of course) and take others.

Being moneyless doesn't specifically mean that I don't touch any money or think it's evil. It's nothing but a tool for me, and being independent of it for the rest gives me a sense of freedom I've never felt before. No worries anymore that I would starve, freeze to death or that things would be stolen from me. I actually expected to be quite dependent on CouchSurfing or BeWelcome, but until now every time I am in one city, I seem to meet people who know people in the next that are more than willing to host me for a few days. Example: I am currently in Warsaw and tomorrow going to Vilnius, to meet a friend I met on Lanzarote in February.

Partying is easy, I don't go to places that ask an entrance fee, or I befriend people that can put me on the guest list. I volunteer at festivals, which gives me free entry, including food and drinks.

I also realize that not everyone could do this, because a lot of it is dependent on the goodness of others, although I'm never too lazy to give back by sharing the skills and knowledge I have, or doing all kinds of tasks, like cooking/dishes or helping a farmer out for a while.

That said, almost every day I'm amazed about the awesome people I meet everywhere, people who pick me up while hitch hiking offering me a room at their home if I ever plan to visit their country, or offering to pay for food or buy me a train ticket. Even more astounding to me is that when I offer to pay for coffee or gas in exchange for a ride, this is mostly refused.

For information about the free (as in beer) world, check out websites like nomadwiki, hitchwiki, trashwiki and moneyless.info. One can even hitch boats and planes :)

<comment><USER12> So what brings you to hackernews?

<comment><USER10> I'm a hacker. Currently a life hacker as well ;)

I've worked for 14 years in IT, but always had this dream to go travel the world and see as much as I can. 25 days off a year or a sabbatical year weren't enough time to realize this.

I also love to explore different ways of life, finding ways within the current system to survive and live a fulfilling life. A lot can be built from stuff that others throw away, an example on HN today is the Poor Man's 3D-printer.

Based on all the experiences I go through now, I eventually want to set up my own community, in a way that it is scalable and not dependent on other systems, so it can provide a working alternative for people who suffer from stress due to unfulfilling jobs and feeling stuck in the life they're living. I see a lot of people fighting capitalism for example, or disagreeing with their government, but without an alternative, they still feel they are dependent on these systems and therefore are not willing to bite the hand that feeds them.

It's in hackerspaces and the hacker scene where I see a new world being built :)

</comment></comment></comment></comment></comment><comment><USER13> I like the optimistic start - keep it up. please promise to follow up when you are on day 100. and here's a good quote to keep in mind as you go through the process: "It is an insufficiently noted aspect of adult life that everyone's pretty much trying to keep their morale up every day." Everyone struggles with morale. Good thing to keep in mind.

<comment><AUTHOR> Thanks for the quote! I plan on writing a follow up with how things progress.

</comment></comment><comment><USER14> Working for yourself is awesome. I have done it twice during my life, once for 5 years and again now for nearly 4.

I was never happier at work. But, there are significant downsides (of which you are probably aware).

1) At a job personal weaknesses get covered up or smoothed over. Aren't so good at <insert failing>? The company probably has someone else to take care of that. So it might merit a "to improve" on your review. When you work for yourself these failings have real unavoidable economic consequences.... often fatal in the business sense.

2) A law of nature. A life for one is simply a meal for another. Nobody cares if you stay in business unless they think they need you in the future. So you are food for everyone. Your vendors, your customers, your employees. All are (generally) out to take as much as they can from you. This is the game you are largely insulated from when you work for someone else and the reason long term business owners probably seem a bit jaded and suspicious.

3) You are responsible for everything. Everything. From filing taxes to making sure there aren't objects that can be tripped on on the work floor. Everything. There is zero buck passing. I personally believe buck passing is a large part of the workday for most employed people at reasonable sized companies and this skill doesn't transfer well.

4)You are mortal. So you get sick. No more sick days. Now you lose income unless you are properly set up with employees and a business structure which takes time, money and often some trial and error. Just because you made 20 grand on a jackpot 2 weeks of work doesn't mean that event will be duplicated or scaled. Something that is hard to remember sometimes.

Overall though... it is worth it in my opinion if you are tough and can take some blows. The upside is unlimited. The freedom is exhilarating. You truly can do your best work at a level you probably never did employed. But, please be aware it isn't for most people. The sacrifices can be immense.

</comment><comment><USER15> _clicks link_

ERROR ESTABLISHING DATABASE CONNECTION.

There's a joke in here somewhere...

<comment><USER16> Welcome to your new startup. Your first challenge is 'database connection'. You may begin.

<comment><AUTHOR> Haha thanks :) Looks like I will be switching solutions sooner than I anticipated.

<comment><USER17> In addition to changing hosts, there are a few ways to dramatically increase the traffic a blog can handle:

\- Caching aggressively to static files. \- Using a static site compiler. \- An HTTP caching layer, like Varnish or CloudFlare.

All of the above eliminate the need for a database connection per pageview.

</comment></comment></comment></comment><comment><USER18> A little iron that 'never try never fail' is currently giving a database connection error. Always try, but always catch exceptions. ;-)

</comment><comment><USER19> I did this about 3 years ago for about 4 months. Some notes:

Working for myself was great. I'm still amazed at the sheer number of things I made in that time. It was some of my best work, and I really want to get back to that time.

Freelancing isn't the same. It's better than being employed, but it's not complete freedom. It is a constant struggle to maintain a relationship with the client that is both congenial but not familial. When things get familial is when they start with the attempts to guilt-trip you into extra work or doing things against your principles. It would take an extremely generous deal, of the likes I doubt anyone would ever be willing to pay for someone like me, to get me to go back to working as an employee.

I had a schedule problem. I woke up when I felt like it, I worked until I was exhausted, and repeated the next day. This meant I was working on about 7 hours of sleep (the least I've ever been able to run on for an extended period of time and still feel good, so that was a plus), and 18 hours of work, with brief breaks for food, showering, etc., fit into the cracks around the edges. Yes, that adds up to 25 hours. My schedule was constantly shifting. There are several good and bad things that came out of this. It isolated me from a lot of people, which was good for work but bad for my psyche. It kept me out of the rush hours, which was great for everything but made me hate weekends when all day is rush hour in Philadelphia (I live in the DC area now).

I didn't have a marketing plan at all. If I did anything, I was posting to my poorly-followed and poorly-maintained blog. I think I posted ONE project during that time to HN. Now that I'm significantly more engaged in marketing, I'm seeing much more traction with my projects, even though I don't have anywhere near the same amount of time to focus on them.

"If you build it, they will come" IS NOT TRUE. You have to advertise. How you advertise is up to you, but I will tell you that paying for Google AdWords is far more consistent and reliable than trying to play the Show HN lottery. You can make plans off of ad traffic. You can make corrections. Show HN is a one- shot deal, and if you don't get it right the ONE time everyone is paying attention, then you'll likely never get it again.

Finally, 4 months just isn't enough time. It's enough time to build something, but building the thing is only 20% of the work. Ideally, I'd want a year to get everything done right.

</comment><comment><USER20> Site looks to be hosted at BlueHost. My advice, jump ship to another host _, or more powerful setup as soon as possible.

_[http://en.wikipedia.org/wiki/Endurance_International_Group](http://en.wikipedia.org/wiki/Endurance_International_Group)

<comment><USER21> Github recently started providing distributed hosting for their webpages. You can use it with your own domain. I already use it for my open-source project and are seriously considering to switch my home-page.

</comment></comment><comment><USER22> And the first thing I did was write a blog post

</comment><comment><USER23> Ha!

Today is the first day since quitting my cushy Bank coding job on Friday. I'm uncertain about the future now, but I feel so happy about coding for me, for fun.

<comment><USER24> Congrats! What are you working on?

</comment></comment><comment><USER25> Well, it's pretty clear what HN is interested in. 10 minutes later the database is crashing!

</comment><comment><USER26> I've been freelancing for 3.5 years, part of which included a 6 month office contract that ended a little over a year ago. Just wanted to give a few hard numbers since nobody ever gave me any:

* When I left the working world, I had no savings. Long story, but I was so far in debt due to paying business expenses with personal credit cards, saving wasn't on my radar. Won't get into details, but ya, don't do that (don't mortgage your future). Having no savings is common, but it's only okay if you have some sort of income lined up.

* The first year I earned income however I could with repair experience I had gained at my last job. Most of that entailed flipping broken Macs through eBay, which is getting marginal but if you need quick cash you can buy one online, replace the HDD or whatever, and sell it on Craigslist. Don't expect to average more than about $50-100 per day though, and probably less than that unless you can pipeline multiple repairs at once. I also did a little troubleshooting and picked up $100 here and there, which I no longer do, due to the liability of being on call.

* The second year I went to the school of hard knocks, picking up a degree in online contracting. I started with freelancer.com, taking $100 gigs a couple times a week. Then I moved to elance.com and got a few fixed-rate contracts in the $500-1200 range. Unfortunately I underestimated and several 2 week contracts turned into 6 weeks. Two overseas clients never paid (one at no fault to elance - I did too much without asking for payment, and one I scouted on my own) which set me back at least $2000 and a couple months of my life.

* Just when I was about to hit my breaking point, an old friend called me up and asked if I would come interview with his employer. That turned into the 6 month contract at a company that provides mainframe services, but was dabbling with more of a startup business model. They offered me my hourly consulting rate on a full time schedule and I was able to save again.

* In the year since, I've been focussing on contracts in the $1500+ range, which is probably the minimum for rural America. It makes ends meet but I will be owing taxes soon on money that I made at the 6 month contract, because the contract was split across a year boundary. I managed the first year just fine but ate into my savings the second year. So be sure to set aside a third of your income for taxes.

On the face of it, this is all irrational behavior and I'm told daily that I should just go get a job. That's all fine and good, and to be honest, showing up somewhere and doing time each day in exchange for currency can be less stressful than worrying about where next month's mortgage payment will come from. But, I find that I can't generate more than about 6 hours of concentration each day sustainably. So the additional 2 hours that jobs extract are to ensure that nothing is left for pet projects. They have more to do with turnover than productivity.

I kind of look at it now in a meta way, like, if you take a step back from your life and picture where you are and where you'd like to be, does that depend on money? I mean for me, that's a very serious question. I don't care about money. I know I should, but it always seemed like a pretty inaccurate way of measuring one's life, since the people who have a lot of it never seem to do anything good with it. I've decided to instead focus on having enough. If the forces align each day to give me enough food/water/shelter to continue, then I consider that a good day.

So I don't know what else to call it than survival. And I'm doing it for just exactly the same reasons that survivalists do it, as refuge from "this", whatever intangible kind of ominous matrix-like manifestation you want to call our culture. I don't think I have to explain it to anyone here because you already get it. I've really been thinking a lot lately that if enough of us can get unplugged and survive long enough, that we might be able to make the real contributions to society that we know we are capable of.

My biggest problem now is that since I work for the client instead of a boss, I'm finding it hard to carve out enough time for my own projects. The world still wants dollars, and part of the reason that office life is so brutal is because that's what it takes to stay afloat today. Contracting pays 2-4 times as much as a day job, but I find focussed concentration takes a toll on downtime as well. I have the inspiration to tinker but not the motivation. This is really starting to haunt me, more perhaps than scratching out a living. I keep wishing that there was some kind of reprieve, like if I could just get ahead of the curve long enough, that maybe naturally I would find time to invent stuff. The answer seems to be residual income, but the few times I've had it, it's been snatched up by others and I developed a pavlovian aversion to investing in it. That's probably my biggest regret, and so I would offer a word of warning that if you go the survival route, be sure to have something brewing in the background or else you will find yourself back where you started.

Sorry this got so long, I've never been a very good editor.

</comment><comment><USER27> Let this be a lesson for everyone...

If submitting to HN, pick a static cached page of the content to serve or use varnish or something. ;)

<comment><USER28> Don't just install Varnish in front of WordPress and assume the best. WordPress by default places a cookie in the browser session and Varnish (also by default) will vary the cache results by cookies -- in the case of PHP session IDs, it means Varnish will let each request through.

Plan your caches carefully and start from the CMS side.

</comment></comment><comment><USER29> Eerily similar to my current life. I quit 11 days ago, spent a week with my family, and now working on some personal projects.

Good luck!

<comment><AUTHOR> Wow, that is quite the coincidence. Good luck with your endeavors as well!

</comment></comment><comment><USER30> Great article! Looks like you have a long road ahead of you, but best of luck! :)

</comment><comment><USER31> Attention lost. Woops!

"Error establishing a database connection"

<comment><USER31> Not my fault this was broken. What is this? HS?

<comment><USER32> I think you're getting downvoted because you're being snide about it, not because you stated the fact.

<comment><USER31> How is this being snide? Anyways, thanks for your input. I actually think we come to read articles that sound interesting, and if these are not accessible, our attention is lost, so we move on; not all of us have all day. Anyways, someone was just on a power trip; they think that because they can down vote, they can do so for anything, even when it points out actual problems with a blog/site.</comment></comment></comment></comment>


Is the cute Meitu app really a threat to our online privacy? - <AUTHOR> https://www.techinasia.com/talk/cute-meitu-app-threat-online-privacy <comment><USER2> This is a really bad/misinformed article that says to use a VPN against shitty applications requesting every possible permission on earth and exfiltrating all of the collected data.</comment>


Memory layout of multi-dimensional arrays - <AUTHOR> http://eli.thegreenplace.net/2015/memory-layout-of-multi-dimensional-arrays/ <comment><USER2> Related to this is the concept of arrays of structures (AoS) vs. structures of arrays (SoA). Do you store all of the data that makes up one "object" in a single data structure, or do you store all of each "member" of each object in separate flat arrays?

If you need to use all of the "members" of an "object" at once, AoS is more efficient. But if you only need to use 1 or 2 members for a certain task, you will waste lots of memory bandwidth on data that isn't needed for the current operation. In this case, an SoA layout can improve your performance by an order of magnitude, by insuring that only the data that is actually necessary for your task is loaded into cache.

The same principle applies to row vs. column-oriented databases, which obviously are orders of magnitude more sensitive to IO bandwidth usage.

One of the most damning criticisms of OOP, from this perspective, is that it encourages you to wrap up lots of potentially unnecessary data into monolithic "objects" without concern to memory layout or usage patterns, giving you AoS _and_ bloated data structures.

<comment><USER3> Good points. However, object oriented programming doesn't necessarily mean arrays of objects. I'd disagree that it encourages it in any way.

I realize that's a common paradigm in Java and C++, but it's an architectural decision that you make, not a part of a programming paradigm. The array-of- structs prevalence really stems from C, not object-oriented approaches.

You can do both in a object-oriented manner. For example:

        points = Points(xarray, yarray, zarray)

vs

        points = [Point(x0, y0, z0), Point(x1, y1, z1), ...]

If you're mostly operating on a series, it makes more sense to structure objects and methods around the former "plural" approach, anyway.

</comment></comment><comment><USER3> Excellent summary!

Layout of N-dimensional arrays becomes even more important when you start considering on-disk formats, where seek times are relatively slow.

People often talk about out-of-core calculations without fully considering how their data is laid out on disk. If your access pattern doesn't match the on- disk ordering of your large, N-dimensional array, you're in for a large surprise. Accessing a portion of an array that takes milliseconds in RAM can easily take multiple hours if you naively write a large multidimensional array to disk with an ordering that doesn't match your access pattern.

Of course, this is why chunking exists, but surprisingly few people are as familiar with it as they should be.

For example, at work, I often need to use a specific legacy format that's basically a large 3D fortran-ordered array stored on disk. I often get asked why I'm loading the entire thing into memory instead of taking an out-of-core approach. The problem is that our access pattern deals entirely with accessing a single "trace" (z-direction) at a time. For a fortran-ordered array stored on disk, that's the worst-possible access pattern. Each element in a single trace is a long ways apart throughout the entire file. Loading a single trace takes far long than loading the entire file into memory on most (non-SSD) systems.

I've recently been on a bit of an evangelical kick for trying to persuade more people to think about the implications of on-disk layout. (For example: [http://geology.beer/2015/02/10/hdf-for-large- arrays/](http://geology.beer/2015/02/10/hdf-for-large-arrays/) While that's specifically about HDF5, most of it is explaining memory/disk layout and its implications.)

</comment><comment><USER4> This is a fascinating topic I have had to spend a lot of time studying as I was writing [http://nd4j.org](http://nd4j.org) (numpy port for the jvm)

When you want to represent say 6d data you have to account for all sorts of interesting details.

One interesting detail is strides along particular dimensions.

For example given an ndarray of

2 x 3 x 1 x 5

You will have a stride to get to the next element along a dimension in the linear buffer.

The strides along k dimensions are determined by ordering.

The crazy thing here though is that 1 in the middle for the shape.

If you find a 1 in the shape you actually skip that stride when indexing into the array.

Numpy's internals go in to this quite a bit: [https://scipy- lectures.github.io/advanced/advanced_numpy/](https://scipy- lectures.github.io/advanced/advanced_numpy/)

The great thing about the linear storage of these arrays also helps with distributed systems and reading blocks of bytes.

Lots of interesting little things in here to consider.

Without going ok another tangent this is also highly relates to blas. There is a lot more to this in practice. This is a wonderful introduction to the topic though.

<comment><USER5> You can also use space-filling curves to produce nice cache behavior under certain assumptions. If you're only needing to view a small, contiguous portion of a very large array, for instance.

<comment><USER2> GPUs use tiling and morder-order storage for textures for this reason.

[https://fgiesen.wordpress.com/2011/01/17/texture-tiling- and-...](https://fgiesen.wordpress.com/2011/01/17/texture-tiling-and- swizzling/)

<comment><USER6> ... Morton Order [https://en.wikipedia.org/wiki/Z-order_curve](https://en.wikipedia.org/wiki/Z-order_curve)

<comment><USER2> ...oops...

</comment></comment></comment></comment></comment><comment><USER7> Was hoping this touched on Z-ordering. If you have a processor that can efficiently interleave bits (i.e. combine 0110 and 1011 to get 01101101), you can lay out 2D and 4D arrays in Z-order, which will improve spatial locality (nearby elements in both the X and Y dimensions are also nearby in memory/cache/TLBs).

You can actually efficiently achieve this even on processors without an interleave instruction if you're only walking stepwise -- basically, mask out the odd (resp. even) bits separately, setting the even (resp. odd) bits to 1 before adding 1, or to 0 before subtracting 1, then recombine.

</comment><comment><USER8> There's more about multi dimensional numpy iterators in "beautiful code" [https://www.safaribooksonline.com/library/view/beautiful- cod...](https://www.safaribooksonline.com/library/view/beautiful- code/9780596510046/ch19.html)

</comment><comment><USER9> relevant:

[https://software.intel.com/en-us/articles/optimize-data- stru...](https://software.intel.com/en-us/articles/optimize-data-structures- and-memory-access-patterns-to-improve-data-locality)</comment>