Is San Francisco becoming toxic for new startups? - <AUTHOR> http://blog.thefamily.co/post/90661830903/is-san-francisco-becoming-toxic-for-new-startups

<comment><USER2> I've never met anyone who lives in SF who thinks buying/auctioning public parking spaces is a good idea. Every single resident I know thinks it's just a shitty thing to do and makes the city worse for everyone (other than the people leaching money off the system I guess). Note that the author of this post lives in Paris. Parking in SF is certainly a pain, but auctioning off spots is just going to lead to fistfights in the street.

<comment><USER3> Also I don't understand why they didn't first launch in Paris, which has exactly the same problem, possibly worse.

</comment></comment><comment><USER4> >San Francisco City Attorney Dennis Herrera has just sent a cease-and-desist letter to MonkeyParking a competing company that offers another mobile application related to parking.

>In fact there seem to be nothing illegal about sharing or even selling parking information.

Way to side step the fact that MonkeyParking was basically a service that encourage people to hoard parking spaces for money. That's less about selling parking information than selling/leasing public property.

<comment><USER5>
        >Way to side step the fact that MonkeyParking was basically a service that encourage people to hoard parking spaces for money.

Has MonkeyParking been around long enough to determine if people are "hoarding" spaces to broker the info? After a certain length of time the four or five bucks you could earn is offset by wasting time sitting in your car waiting for info purchaser. I don't live in a city as congested as SF but I could never imagine waiting more than about 5-10 minutes (even that's a stretch) before deciding "fuck it, I don't need 4 bucks that bad".

<comment><USER4> See my answer above on why I think it would be worth it, also consider the unemployed/underemployed.

I don't think the app had been out long enough to get a large user base. I only saw it a 2-3 weeks ago and then the gov't acted pretty quickly. So no data on my end. I think most people that read the article about the app imagined the worse case scenario, which in my opinion was likely to happen.

Another scenario would be for tenants who have reserved parking to park out on the streets, wait for someone that's will to buy the place in their apartment. They would then move their car back to their reserved parking space until they see an easy open space to take that's near them. This scenario doesn't take much effort from the person abusing the system.

</comment></comment></comment><comment><USER6> Startups aren't good in themselves, they're good if they bring about helpful innovations or good products etc.

Skirting regulations to privatize a public resource and make money off of it harms society as a whole, so it's better to have as few startups like that as possible.

Also, this article claims that this one, widely supported government action is indicative of a "broader trend", but it offers no further evidence and I'm having trouble thinking of many other cases. Arguably, SF has been rather forgiving of companies which skirt regulations without really harming the public - like AirBnB, Uber, or even Google buses.

</comment><comment><USER7> Kinda funny that the post bemoans regulation then says: The truth is, incumbent rent-seekers do have powerful lobbies even in the US. The oil industry tried hard to stop Governor Schwarzenegger from passing more stringent green air legislation in California.

One person's responsible legislation is another person's gov't innovation killing regulation.

<comment><USER8> Many people flock to libertarian political views (and consequently, business models) because they can't see where their selfishness ends and the rest of the world begins.

Consequently, it's not surprising that the defense of their positions ultimately devolve in to convoluted rationalizations for 'do as I say, not as I do'.

</comment></comment><comment><USER8> I was approached several months ago to build an app with the same business model as MonkeyParking's queue-barging-for-cash. I walked away: the world needs less Ayn Randian it's-not-nailed-down douchpreneurship, not more.

</comment><comment><USER9> I've heard people say before that usually when an article's headline is a "provocative" question, the answer is no. That seems to hold true here.

<comment><USER10> Betteridge's law of headlines [http://en.wikipedia.org/wiki/Betteridge's_law_of_headlines](http://en.wikipedia.org/wiki/Betteridge's_law_of_headlines)

</comment></comment><comment><USER11> This was a pretty exaggerated post from what I read. It went from a city not allowing public parking spaces to be sold, to the United States turning into Japan.

It is hard to take it seriously with such exaggerations.

</comment><comment><USER12> Was it here or somewhere else, I found a very interesting piece about the price of free parking. And how too much parking makes a downtown even worse than it could be. Because it makes it too cheap to use the car.

[http://www.cato-unbound.org/2011/04/13/donald-shoup/who- shou...](http://www.cato-unbound.org/2011/04/13/donald-shoup/who-should-pay- parking)

It also said that SF wanted to introduce a demand based system for parking, which meant it would be a few cents if nobody needed the space and $6 if everyone wanted a space. Of course such a system wasn't realized.

BTW, they could just build a parking space and auction those parking spots, at least it wouldn't be rent seeking. But I guess the prices for a square feet in SF are a little too bit for such a project.

<comment><USER13> > It also said that SF wanted to introduce a demand based system for parking, > which meant it would be a few cents if nobody needed the space and $6 if > everyone wanted a space. Of course such a system wasn't realized.

Demand based parking meters exist in SF. They've been tested in various areas since 2011. They also have an app that helps you find parking using the same sensors that adjust the price.

See: [http://sfpark.org/](http://sfpark.org/) or [http://en.wikipedia.org/wiki/SFpark](http://en.wikipedia.org/wiki/SFpark)

<comment><USER12> Ah, thanks.

</comment></comment></comment><comment><USER14> Interesting read. I think it poses a good question about what can be rented/sold as it pertains to information. I think MoneyParking is a bit different than Sweetch from what I can tell so I don't think they have much to worry about. I agree with the other comments that the article is pretty slanted and not very well sighted. But it's a blog. Can't really expect much journalistic integrity from a blog these days.

</comment><comment><USER15> If your business model is anything like MonkeyParking's, you're a fucking parasite. Good riddance.</comment>


Canadian robot melds brain surgery, rocket science (2007) - <AUTHOR> https://www.reuters.com/article/us-surgery-robotics-idUSN1742478220070417 <comment><AUTHOR> The recent news of Simone Giertz[0] (Queen of _something_ robots) diagnosed with a brain tumor[1] made me wonder about neurosurgical robots.

neuroArm[2] is litterally a scaled down Canadarm[3] used on the space shuttle and now ISS.

[0] [https://en.wikipedia.org/wiki/Simone_Giertz](https://en.wikipedia.org/wiki/Simone_Giertz)

[1] [https://news.ycombinator.com/item?id=16959754](https://news.ycombinator.com/item?id=16959754)

[2] [https://www.nasa.gov/mission_pages/station/research/benefits...](https://www.nasa.gov/mission_pages/station/research/benefits/neuro_Arm.html)

[3] [https://en.wikipedia.org/wiki/NeuroArm](https://en.wikipedia.org/wiki/NeuroArm)</comment>


Ask HN: Advices, Literrature, hacks for prepping to be a parent - <AUTHOR> My wife and I are expecting a child in a few months from now. I'd love to hear thoughts, book recommendations, links, hacks, advices or what have you on the topic from the HN community. <comment><USER2> My random advice from the internet:

0\. The best thing you can say to your child is "I love you and I'm very proud of you." The second best thing is "Did you have fun?" The third best thing is "Where do you want to eat?"

1\. "No battle plan survives contact with the enemy" applies to parenting [and I'm not implying children are enemies, even surly teenagers].

2\. Your child is a person with their own independent life. Living vicariously through your child will happen. Learn how to recognize it and stop. Don't use "we" when referring to the events relating to your child's football team.

3\. There is no hacking children. Each is an individual person not a mechanical system. Each grows at their own pace. Or rather paces, because their are many dimensions to growth and maturity both as an organism and as a person. Children tend to turn out ok despite our plans or our lack thereof.

4\. Don't get caught up in comparing your child to other children. Don't get caught up in their getting ahead. There isn't a race. There isn't a prize for the first kid to reach first grade. Even if their were, nobody would inscribe it on a tombstone.

5\. Get to know your child as a person because your child is an amazing person.

6\. Everybody is still a child. Enjoy the excuse to act like one and just spend time playing. No matter how old your child gets or how mature, remember they're still a child. It's wonderful.

7\. Practical advice given to me by Julie when my beloved was pregnant: For the next year, baby and mama are first and you are third. Unless you have a dog, then you are fourth.

Congratulations and good luck.

<comment><AUTHOR> Thank you for the comprehensive list! Especially like the first and third one. Just those 2 probably can weight a whole lot in a child well being.</comment></comment>


Mill CPU: Memory (2013) [video] - <AUTHOR> http://millcomputing.com/technology/docs/memory/ <comment><USER2> I'm as much of fan of Ivan and the Mill as anyone else but this talk is from 2013.

I think the title should be altered to reflect that. Unfortunately the previous submissions of this talk didn't generate any discussion so there's probably no real point in linking them here.

Whatever the case, I'm not sure what there is really to discuss. The Mill project hasn't published any new material in some time.

<comment><USER3> Added.</comment></comment>


Backblaze Vaults: Zettabyte-Scale Cloud Storage Architecture - <AUTHOR> https://www.backblaze.com/blog/vault-cloud-storage-architecture/ <comment><USER2> Reading the comments, is anyone else bothered by this reply from a Blackblaze representative:

"Right now, Backblaze has only one datacenter, so the short answer is "no". :-)

The longer answer is that for online backup, there is one copy of your data on your laptop, and another copy in the Backblaze datacenter in Sacramento. If a meteor hits our datacenter in Sacramento pulverizing it into atoms, you STILL would not lose one single file, not one - because your laptop is still running just fine where ever you are with your copy of the data. In the case that occurs, we will alert our users they should make another backup of their data."

There are a million one things other than a comet strike that can go wrong in a data centre. I would not trust a backup provider that does not replicate my data across at least two data centers.

<comment><USER3> Yev from Backblaze here -> It's true, we just have the one. Backblaze was bootstrapped, so we cannot over-expand and maintain profitability, which allows us to stay in business. We're pretty up-front about having the one datacenter. We'd LOVE to add more in the future, but truthfully, it would at least double our costs, and we'd need to raise prices. We're thinking of ways of avoiding that while maintaining our current business-model. If you are looking for something more geo-redundant, take a look at services like Amazon S3, they are great, but the downside there is they charge per GB to make up for the extra costs, so depending on the amount of data, it can get pricey. Either way, we recommend a 3-2-1 backup policy (3 copies of your data, 2 onsite but on different mediums and 1 offsite) as a good start to a backup strat. We're just one solution of many, though, we like to think we're the easiest one!

<comment><USER4> How about an add-on cost/service that tags your data as needing datacenter redundancy, and only replicating that to a new datacenter. It has the benefit of not requiring as much up-front investment, as it's used it pays for itself, and you have a bunch of current customers you can upsell to. The architecture to segregate redundant from non-redundant backup customers could be a pain, but as long as you have tools to migrate data between systems (I imagine you do), then it could just be running two separate backblaze clusters in the first datacenter, one which supports redundancy and one which doesn't, and then just migrate customer data between the clusters as the add/drop the redundancy service. That saves you from having to cherry-pick specific files/customers from the the cluster to duplicate in the other datacenter, you just make sure one cluster is always redundant.

<comment><USER3> We're definitely looking at options like this, but the engineering work that it would take to implement solutions like that are not insignificant, and a lot of our engineering muscle has been working to roll out the Vaults over the past year and change! It could certainly be another revenue stream for us, but building out a new datacenter is expensive, especially if you don't buy/guarantee build-out ahead of time, so we'd have to forecast how many people would want that service to prepare accordingly, again not insignificant stuff, but is definitely possible in the future!

<comment><USER4> The nice thing about using separate clusters is that you can build them out in chunks. Build X new capacity in your main datacenter as a new cluster, and X new capacity in a different datacenter, and replicate. Need more redundant capacity? Build Y new capacity in your main datacenter, and Y new capacity in a different datacenter, not even necessarily the same backup datacenter as before. You end up with one main non-redundant cluster, and a bunch of smaller redundant clusters spread over one or more additional datacenters.

If you're _really_ lucky, you siphon off customers from the non-redundant service for this at the same or faster rate as they are signing up for the non-redundant service, allowing you to not have to build that out much for a short while.

</comment></comment></comment></comment></comment><comment><USER5> "For Backblaze Vaults, we threw out the Linux RAID software we had been using and wrote a Reed-Solomon implementation from scratch. It was exciting to be able to use our group theory and matrix algebra from college. We'll be talking more about this in an upcoming blog post."

I hope I'm not the only one uncomfortable about this. I mean, I understand the need for greater flexibility and features that MDRaid doesn't provide, but this wording stinks of NIH and rebuilding the wheel from scratch "because it was fun", discarding the maturity and reliabilty of established software. And data storage is _all_ about reliability.

I hope I'm just reading too much from this, and it isn't actually representative of Backblaze's engineering practices.

<comment><USER6> Brian Wilson from Backblaze here (not the "Brian Beach" who is author of the blog post) - this was not a case of NIH. We use lots and lots and lots of existing software like Debian, Java, Ext4. We use tools like ansible and Zabbix. But this one thing just didn't exist for us in the form we needed it. We looked, we really did.

We did write the Reed-Solomon ourselves in a "clean room" so we did not have to pay any licensing fees and we clearly didn't steal anybody else's source code, but that is a very small amount of code. Like 80 lines of Java. Seriously. We referenced the technical papers we read to implement it in that blog post, but here it is again: [http://www.cs.cmu.edu/~guyb/realworld/reedsolomon/reed_solom...](http://www.cs.cmu.edu/~guyb/realworld/reedsolomon/reed_solomon_codes.html) And we unit tested the living heck out of that code, plus we mathematically verified various parts.

But I'm open to an alternative solution if you can suggest one? Remember our three highest priorities are: 1) reliable, 2) low cost, 3) simple. The "low cost" includes things like we do not want to pay ongoing licensing fees to other companies.

<comment><USER5> Thanks, that's basically what I wanted to hear. :)

My first thought was that you could've reused the R-S code from mdraid or dm- raid or ZFS, but on second thought 1) it may be too specialized to be reusable, and 2) it's GPL (or CDDL), so you can't just plonk it into your own code.

And yeah, if it's just 80 lines of Java, I'm worrying about the wrong things.

<comment><USER7> > 2) it's GPL

Backblaze is a web service, so for better or worse, the GPL doesn't apply here, since we never have access to their binaries. The AGPL would apply, but that's not the license used.

</comment></comment></comment></comment><comment><USER8> That was a great read. I was surprised that the data on a drive tries to go back to the same drive when it fails. One of the things that GFS did (I presume they still do) and Blekko does is that when a drive fails its data is reconstructed on other working drives so replacement leaves no long term degradation risk. If you don't do that then while your drive is dead you have lost some data resiliency until it gets replaced, as opposed to just waiting until it has been successfully recreated elsewhere[1].

Its no wonder storage companies like EMC are hurting when you have innovators like these guys out there.

[1] Which give a 3x replication system and a sharded or (chunked) file can happen pretty quickly.

<comment><USER6> Brian from Backblaze here. We do angst over the idea of a "hot spare" where the very second we fail a drive it can begin rebuilding elsewhere. But that takes up redundancy even when it is not used (an extra drive waiting) which raises cost.

At our current scale it is becoming less and less of an open debate, because we now have 7 day a week staffing at our datacenter and the datacenter techs jump right in and replaced failed drives often within an hour or so. A "hot spare" would only save a couple hours of rebuild time. But remember, your mileage will vary - until you reach half our scale you cannot afford even a Monday-Friday datacenter tech, so you might only be able to replace failed drives on Mondays and Wednesday, which widens your exposure.

<comment><USER9> Have you considered rebuilding into already available space in the cluster?

Something similar to how ceph or swift handles rebuilds? you get rid of the individual disk sitting around as a spare. though it would break the idea of a tome being a specific collection of disks. you would need to be able to identify and move a shard around your cluster into other vaults and a shard would need to be smaller than the raw disk size.

this would increase network overhead as well. (more movement.)

I'm probably just rambling here so you can probably ignore me. (you have awesome tech there though)

<comment><USER6> > I'm probably just rambling here

:-) Not at all! Don't assume we're some perfect team of scientists that know all the correct solutions before we start coding. We often angst over these decisions and designs, knowing that once we write the code a lot will be set in stone (hard to change) for a numbers of years. The reason it becomes hard to change is we don't have a huge development team that can afford to rewrite the software every year, so we try to get it correct and then go on to work on new things or polishing up corners that need polishing.

</comment></comment></comment></comment><comment><USER10> I'm really looking forward to the Reed-Solomon article. It seems that very few RAID-like applications are built to handle arbitrary data and parity stripes.

<comment><USER11> Here's a nice explanation of how ZFS does triple parity raid with reed- solomon:

[http://people.freebsd.org/~gibbs/zfs_doxygenation/html/da/dc...](http://people.freebsd.org/~gibbs/zfs_doxygenation/html/da/dc9/RaidZ.html)

[http://people.freebsd.org/~gibbs/zfs_doxygenation/html/d1/d7...](http://people.freebsd.org/~gibbs/zfs_doxygenation/html/d1/d7d/vdev__raidz_8c.html#_details)

<comment><USER10> _Note that the Plank paper claimed to support arbitrary N+M, but was then amended six years later identifying a critical flaw that invalidates its claims. Nevertheless, the technique can be adapted to work for up to triple parity._

So better than most, but still not arbitrary. And the limiting factor seems to be write performance.

</comment></comment></comment><comment><USER12> Getting rid of RAID makes things a lot easier since you don't have to suffer through rebuilds, which causes a lot of I/O for the entire RAID. You still have to repopulate the drive, but you have fine-grained control of when to do it and even which files have the highest priority.

For those looking to build something similar, check out ceph or gluster.

Is a single file spread across multiple data centers? At the claimed 99.99999% annual durability, doesn't the chance of a natural disaster that could take out the entire data center start being a major factor?

I realize that the customer also has a copy of the data so you don't have to take the same precautions as something like S3, but it'd be sad if a datacenter got taken out by a meteor or airplane crash the same day that the customer's laptop was stolen.

Finally, a question for backblaze devs. In your opinion, how often do you need to scrub a drive to check for problems?

<comment><USER3> Yev from Backblaze -> That meteor question comes up a lot ([https://www.backblaze.com/blog/vault-cloud-storage- architect...](https://www.backblaze.com/blog/vault-cloud-storage- architecture/#comment-1901229080)). We currently do have one data center, but this design allows us to bring others online. If the datacenter was hit by a meteor all our customers would get an email blast urging them to create a local backup immediately. The chances that both the DC and the user would get hit by the same natural disaster are relatively small. Still, it's not a storage service like S3 so geo-redundancy plays a smaller role. We do plan on building out other datacenters in the future, but since we're bootstrapped, we have to do that when the time is right, otherwise it would be very easy to over-extend ourselves and start losing money.

 _edit_ -> I ignored your Backblaze dev question, sorry. We have multiple processes running at all times on the pods, and they go shard-by-shard. We're always optimizing, but the short answer is, we're always looking for errors.

<comment><USER13> It doesn't take a meteor. I got hit when EV1 had a fire in one of their DCs, fortunately we had a local backup from which we restored and kept on running but a lot of companies were not in that position and had a hard time surviving. EV1 was hurt badly by this, it's not just meteors. What happened was that a transformer on the floor exploded, took a dividing wall with it and cause a (surprisingly!) relatively minor fire.

What took down the DC for several weeks was the firedepartements investigation. They took their time to figure out the root cause of the fire which is their right but the collateral damage of that was substantial.

So don't just plan for meteors.

<comment><USER12> The nice thing from backblaze's perspective and their customers is that downtime is far more tolerable than it would be for most businesses. Most disasters that are going to impact a data center aren't going to destroy the physical hard drives, assuming the data center has the usual safeguards in place.

<comment><USER6> Brian from Backblaze here-> this is definitely true. If you ask to have a 5 TByte restore prepared, it will take us a full 22 hours to get that all assembled for you. If you want us to FedEx the prepared restore on a USB hard drive, it will take ANOTHER 24 hours, and if you are in Europe it's more like 48 hours.

And what's luxurious about "backup" as a business is this doesn't bother many customers. As long as we keep communicating to them on the progress, and we assure them they are going to get every solitary bit/byte/jpeg/mp3/movie back - they often tell us to take our time and do it right. For "backup" accurate and durable is about a thousand times more important than "instant gratification".

</comment></comment></comment></comment></comment><comment><USER14> _At our current growth rate, Backblaze deploys a little over one Vault each month._

That's a full rack of storage pods a workday. Some back-of-the-envelope math says that's almost a tractor-trailer worth of hardware a month. Wow.

<comment><USER3> Yev from Backblaze here -> We're VERY proud of our datacenter techs, and you'd hear more about them if they weren't so shy. It IS a monumental achievement though, considering that a few years ago we only had two guys running our entire farm.

<comment><USER15> The client I'm working for failed to get a web server procured with 6 months lead time. And that's a company 100X bigger than Backblaze. I wish they understood how much money they are wasting by being stingy.

<comment><USER3> All you can do is keep sending them these posts :)

</comment></comment></comment></comment><comment><USER16> I find the work Backblaze is doing is wonderful, mostly because of how open they are with their data. Watching their numbers on certain HDD failures has really helped steer some of my recent purchasing decisions. I'm also really interested to see more details on their custom RAID replacement.

Essentially raid is dead to me, and ZFS/BTRFS, etc seem to be the only way to go forward, so I hope they gpl the code.

For anyone from BackBlaze reading this, I'm curious though, have you found that backplanes are becoming a primary bottleneck? Because that seems to be the case (Sata 6gb/s hurts after using things like fusion-io or even thunderbolt.) Any insights into the future of backplanes?

</comment><comment><USER17> Backblaze seems so forward thinking with the hardware, but if you've ever tried to restore a file using their web interface, it's an exercise in frustration.

If you want to pull a single file, you'll be navigating through a windows 95'esq tree. They store snapshots, but if you want to change the snapshot, you wait a minute for each while it loads. Even going back to a snapshot you were just looking at, you wait the whole load time.

Now if you actually need to restore something. You can download a zip file. They will only let you make the zip file so big, so you have to break up your restore into multiple zip files. You will have to do that manually, there is no way to have BB auto-generate the parts for you. These are zip files and not one of the many archive formats that allow for parts.

Besides that, you will need double the amount of storage to recover this data since you'll need to store the zip and the extracted backup.

The way around this is to pay BB to put the data on a USB drive (flash up to 128GB or external up to 4TB) at $99 and $189 respectively. A 128GB external usb on Amazon, first result is $120. They actually won't give you a 4TB unless your restore needs it, but the price is still $189. Labor I guess? According to their own FAQ, you will wait 2-3 days for them to ship these drives, so hopefully you don't need that restore anytime soon.

I really liked Backblaze up to the point I needed to use it for it's real purpose. It seems like nobody at BB cares about the restore process or that it doesn't sell new subscriptions.

\--

Also, maybe someone from BB can explain why the secure.backblaze restore website loads tracking pixels from googleads.g.doubleclick, a.triggit, s.adroll,facebook, ads.yahoo,x.bidswitch, ib.adnxs and idsync.rlcdn. Are you selling my need for a new harddrive or something?

<comment><USER6> > It seems like nobody at BB cares about the restore process

Brian from Backblaze here -> I care! It just keeps getting bumped by something higher priority. I have a spec for how to speed up the restore tree browsing, it's just waiting for us to have a spare moment. For a while the Vaults took precedence.

Part of running Backblaze without VC funding is we can only hire programmers when we can afford it out of profits, and we're up to about 6 programmers (the result of a recent burst of hiring) which handle all of: Windows, Macintosh, iOS, Android, and in the datacenter they built the pods, the Vaults, and the web front end. But we'll get there, I swear.

> or that it doesn't sell new subscriptions.

This is unfortunately the heart of the problem. The most important thing to get smooth as glass is the BACKUP part, that sells new subscriptions. If we have your data safe, we can always hobble through a restore even if it is a little slow and clunky we can get all your files back after your laptop is stolen. If it held up sales, we'd jump over and do the one week of work to speed it up.

</comment></comment><comment><USER18> Always impressed by the write-ups the Backblaze team does. Very informative and very clear on what you guy are achieving and how you do it.

Crazy amount of hardware involved, and Backblaze is the "small kid on the block" in relation to FB, Google, and Amazon.

<comment><USER3> Yev from Backblaze here -> Yea, we hope they start doing more stuff like this too! More information = everyone wins!

</comment></comment><comment><USER19> Backblaze is fantastic for sharing all their hardware development efforts. We've built our own pod for onsite storage and it's amazingly awesome.

<comment><USER3> Yev from Backblaze here -> Nice! Glad it's working for you :)

</comment></comment><comment><USER20> What does 99.99999% annual durability mean in practical terms? That everyone should expect to lose a few bytes per year? Or that only one in a million customers will be affected by data loss?

(I've never been good at statistics.)

<comment><USER21> That number is probably the estimated probability that over the course of a year, a single vault loses every file contained on it (which is what would happen if a single vault had 4 drives simultaneously and irrecoverably fail).

As a consumer, the type of failure you would experience, and the probability of experiencing that failure, given that Backblaze has suffered a vault failure, depends on how they distribute your data amongst their vaults. They don't explicitly say how they do this, so it's impossible to know for sure, but we can consider the two extreme scenarios.

Scenario 1: Each customer is assigned a single vault, and all your files are on it. In this case, if Backblaze lost a vault, you would either luck out and have your files on another vault and be completely unaffected, or get really screwed and have all your files on the bad vault, and lose them all. They've got 150 PB of storage, and each vault stores 3.6 PB of data, so we can estimate that currently you may have something like a 1 in 40 chance of having your data on any given vault. So under this scenario, you would have a 1 in 400 million chance of losing all your files.

Scenario 2: Each customer's files are uniformly distributed across all vaults. In this case, if Backblaze lost a vault, all customers would lose a fraction of their files. Again, using our estimate that they might have 40 vaults, you would have a 1 in 10 million chance of losing 2.5% of your files.

So up to now, we're basically just doing the math without questioning the assumptions of the model. In reality, I think your practical risk is mostly concentrated in things outside of the model: ie, an event that affects all of their vaults simultaneously, like a fire, earthquake, meteor strike, etc. If I had to make a bet about what that number is, I'd put it in the 1/10,000 to 1/100,000 range. In other words, orders of magnitude higher than losing data because some hard drives failed, or a backblaze employee spilled his coffee, or something like that.

<comment><USER20> Thanks. IMHO the greatest risk of data loss is bugs in the homegrown software and/or operator error during maintenance operations, not a natural disaster. We infallible software engineers always underestimate that stuff, but it's usually the cause.

Also, I'm not worried. If that probability only concerns data loss on Backblaze's side, even if it's 1/10,000, then that's still not the probability of actual customer data loss. Because for that to happen there'd have to be a simultaneous loss of data on the customer side as well. That probably extends the durability considerably.

<comment><USER22> _We infallible software engineers always underestimate that stuff, but it 's usually the cause._

My former boss used to say that 90% of all problems are cabling. His percentage may be off, but the sentiment certainly isn't.

</comment></comment></comment></comment><comment><USER23> I know this is a hijack that doesn't have anything to do with the article, but every time I see a Backblaze article I can't help but say it: Give us a Linux Client already!!!!

<comment><USER6> Stay tuned.

<comment><USER24> Very much interested in this. I asked a while ago if I could build one myself, was initially told it was OK but was later told it's against your ToS.

Days, Weeks, Months, or Years?

<comment><USER3> Pretty sure we can't say much other than stay tuned at this point :)

</comment></comment></comment></comment><comment><USER25> Still no linux support?

Any plans for that or even just an API so I can write one?

</comment><comment><USER26> Their website requires TLS but doesn't support TLS 1.2. What technology are they using to serve their website that cannot support a 7 year old standard?

<comment><USER27> I'm happy their webserver has better cipher suites than it did last week.

<comment><USER3> Yev from Backblaze here -> we're working on it!

<comment><USER6> Brian from Backblaze here -> seriously, I just beat that team up today again, we'll get there.

</comment></comment></comment></comment><comment><USER28> FTA: "If one of the original data shards is unavailable, it can be re-computed from the other 16 original shards, plus one of the parity shards"

So you would have to read 17x the data to recreate it. Given disk latency, network bandwidth, etc. I'm guessing it'll take quite a while to recreate a 6TB HDD if it fails.

<comment><USER14> The drive operations (including waiting for the head to seek, etc) could take place in parallel. There's also the chance the data needed was in cache. So it might not be horribly bad.

<comment><USER28> We're talking about 102TB of data (6TB x 17), so a cache won't dent this figure too much (especially, given that each of those disks is one of 45 drives in the pod, which means the cache is shared across all of them). Then, each of the drives will also be serving files (or storing them), which means disk heads will be seeking around all over the place while rebuilding the failed drive...

<comment><USER29> Actually, it shouldn't take particularly long. The key is to distribute.

Sending 102TB of data to the pod that's rebuilding would take forever, this is true.

Instead you have each peer pod be responsible for 1/17th of the parity calculations.

1\. 17 pods each read 17 megabytes and sends them across the network.

2\. Pod A gets all 17 of megabyte 1. Pod B gets all 17 of megabyte 2. etc.

3\. Each pod calculates their megabyte of the replacement drive and send it off.

4\. Repeat until 6TB have been processed.

So this way each pod reads 6TB from disk, sends 6TB across the network, receives 6TB across the network, and calculates one nth of the data for the replacement drive.

It scales perfectly. It's no slower than doing a direct copy over the network.

Just make sure your switch can handle the traffic (which it already has to handle for filling the vault in the first place).</comment></comment></comment></comment>


Tradeoffs for Women in Technology - <AUTHOR> http://hmrx.posterous.com/being-a-woman-in-tech-sucks-yes-but-seriously <comment><USER2> It is curious that the problem is diagnosed as a social pathology to overwork and then the solution proposed is mentoring more women. Wait, back up here: you're taking a functioning sausage grinder and proposing throwing more folks you care about into it so that the sausage that successfully makes it to the other end is a little pinker than the mostly blue sausage we get right now. How about working to fix the sausage grinder model, instead? (Well, strictly speaking they are not mutually exclusive, but if you had to concentrate your energy, I know which one I'm picking.)

There is nothing intrinsic about startups that requires getting up at 6 AM and working 18 hour days that leave no room for meeting one's more important obligations to family, friends, the community, God, etc. The fact that we generally believe that this is intrinsic, and model it as the "correct" behavior, is a social pathology in our industry. Fix that and plenty of people will have improved lives, including (quite possibly) the marginal new female founders the detoxified culture will attract.

<comment><USER3> The interesting thing about "startup" social pathology is not just that I fall in willingly (intrinsic motivation to ascribe to this set of normative behaviors), but that many others in the Valley often do the same (extrinsic motivation to ascribe to this set of normative behaviors).

The cultural miasma is this is the way things are done. You can always buck the prevailing culture and be a black swan, but you have to look at the risks and benefits of swimming that way.

And many more folks than me buy in. Many are heavily invested in perpetuating this model because it gives them a level of perceived (rather than actual) control - if they aren't actually forming the startups, and they need a set of ways to evaluate potential success, than overwork may be one of those obtainable metrics you can check off relatively easily.

This means the sausage grinder may always be a sausage grinder no matter the tinge of the sausage coming out on the other end. I have no idea how to go about fixing this other than to begin not by continuing to bitch about work/life balance but to begin to fix my own equation (intrinsic - what are my goals? How will I reach them?) and others (extrinsic - are there talented groups out there who aren't getting a shot because we aren't telling women they can strut in and say "fuck it, fund me?").

I figured I'd start with a more open solution where women are informed about the risks and returns by those who've been there. I've been fortunate enough (for various reasons) that I've been able to get those meetings and hack the existing network here.

If anyone has more concrete suggestions I'd love to hear them - and better yet see them implemented, pronto. The generic "let's fix the sausage grinder model" comments tend to put my teeth on edge because they lack 1. specificity and 2. actionability and thus 3. utility.

Why we persist with collaborative maintenance of the startup 'mythology' or 'reality' (one person's myth is another person's faith system) is, you're right, the interesting question.

I can only answer that for myself.

Early stage, high-growth, hyperactive tech startups where you're doing a new feature release every 2 weeks and an entirely new product on average once every 12 weeks means, sorry, working near 18 hour days.

This model may not be 'correct,' but it sometimes works. And as a startup entrepreneur, my primary interest is in finding things that work.

It's not like I haven't examined a multiplicity of options. I've read 37signals, and used their method for 2 years prior.

Bootstrapping got me to a partnership with a top US health insurer to build a product within 3 months after launching my company, keynotes at Mayo and Eli Lilly, and consulting for the National Board (which administers the USMLE or 'boards' to docs) so I'm not knocking this as a potential solution - and one to which I might return (have a day job, do the startup lunches, nights, and weekends).

The pathology to overwork is a personal tendency which you accurately diagnose. In my case it's been augmented by choosing a role in an early stage tech startup where that group pathology persists.

I'm not sure fixing the social pathology is a viable solution, but like other pathologies, teaching coping skills and how to manage emergent behaviors may be.

</comment></comment><comment><USER4> Law and medicine are perhaps worse because these professions impose relatively rigid career tracks upon their members. It's well and good to work 80+ hours/week at a "white shoe" law firm when you're in your late 20's without children. But, even the time off required to push out a baby and recover will have a negative impact on your billable quota. A couple years ago, a friend's firm adopted some new, supposedly mommy-friendly policies in hope of keeping more women on partner track. A few months later, they did a round of lay-offs which included her. She had been working part-time post-child after having received glowing reviews pre-pregnancy. Other firms have done better by offering non-partner track roles which paid less, but offered 40-hour work weeks -- a realistic option taken by a couple of women I know. Perhaps the whole up-or-out partner track model of traditional law firms must change. It's clearly from an era where the rising star was expected to have a stay-at-home wife to take care of things. It can't be the only model by which law can be practiced successfully. This model also adversely affects those opting into law as a second career.

Physicians finish residency around age 30 -- about the average age for college-educated women to have a first child. A friend of ours had to delay residency by two years because a pregnancy required bed rest at the time she was supposed to be finishing up her internship year. The next year's match class was already full. So, after a MD/PhD and fellowship, she will have her first real job at age 37. Maybe the traditional model of "lots and lots of school" followed by being a full-fledged doctor could be replaced by some sort of stepping-stone model which where one could opt out of the rat race for a couple of years in the middle. I'm imagining a path that allows on to practice as doctor-lite (like a nurse practitioner) and later become a full-fledged doctor.

I don't think one can "have it all" -- meaning that a parent can't compete successfully with those willing to sacrifice their lives to work and still show up at the after-school soccer games. Society will have to figure out alternatives for those models of success built-up in the era where it would have been redundant to prepend "stay-at-home" to "wife."

I'm not suggesting that there's anything morally/ethically wrong with the way law firms work today. If a private business wants to limit the available work force through an incentive system that's not fully utilizable by 50% of the population, that's its right. I'm not so comfortable with making a similar statement about medicine because the rites of passage are implicitly set by government regulation rather than market forces. It's not as if I can start-up a certification program for high-end health professionals which will compete with the AMA's MD designations.

<comment><USER5> I don't disagree with the substance of your post, but I wanted to note that the AMA, and the national and state licensing boards are non-government entities. While the states do make it illegal to practice medicine without a license (or practice law without a license), for the most part they let these professions regulate themselves.

<comment><USER4> Can I start-up a competing licensing body? That's my issue -- the AMA is given an implicit monopoly by the Federal/State governments.

</comment></comment></comment><comment><USER6> Except for the childrens thing, what is the bit that makes it harder for women? The article was very long, and children were only mentioned once, quite in the beginning.

Also, why does the startup require the long hours etc? Is there a law in the universe for it? Then it seems curious that the law requires one to work exactly 18 hours per day on a startup. What if the universe decided that to run a startup successfully one would have to work 30 hours per day on it? Then nobody would be able to do startups...

<comment><USER3> Hey Tichy - good questions. Never thought this post would end up on HN, so I didn't explicate the problem(s) and challenges systemically.

There's no universal law, but as mentioned earlier, there is a cultural norm, or, as patio11 pointed out, a tendency towards social pathology that rewards 'overwork.'

All startups may not require the long hours - I don't know and can't speak for ALL founders. However, I know many for whom this is true. It's certainly true that in this stage of growth (pre-toddler) for MY startup it requires that amount of work to complete the specific things I have on my calendar.

<comment><USER6> "a tendency towards social pathology that rewards 'overwork.'"

Reward in what way?

<comment><USER3> <http://www.paulgraham.com/die.html>

</comment></comment></comment></comment><comment><USER7> I think she's in a good position of bootstraping with 30-hours work week from home while raising kids and living on husband's salary.

<comment><USER3> Ouch. Thanks Dennis. Now if you want to toss a husband my way for an arranged marriage that would be great! <end sarcasm/>

</comment></comment><comment><USER8> Do most start-up founders really end up making a choice between the start-up and their relationship/family? Are there examples of people who haven successfully been able to maintain the energy in both? If not, that's pretty depressing.

<comment><USER4> It's relatively easy for a parent to work 12+ hours/day as long as there's a spouse at home who can dedicate his/her life to the children and household.

<comment><USER3> Exactly. And in our culture that's usually still women (although more stay at home dads are stepping up - see: <http://www.rebeldad.com/stats.htm>).

So, as a woman in startup tech, if you choose to have work and family life simultaneously, it means finding a partner who will do that (is willing and financially stable), and/or outsourcing a significant portion of your childcare to family, nannies, etc.

<comment><USER6> I am still not quite sure what you are complaining about? At least as a woman you have a choice, you can opt out and spend more time with the kids. That's not supposed to be a burden imposed by society, it is supposed to be quality time and a gift. Why else have children, if you don't even like it? If you don't like it, the choice is easy, just spend all the time at your startup.

Unfortunately everything has a price - spending more time at home has a price (earning less money with the startup?), spending more time at the office has a price (no kids). But you have a choice.

In another comment you said that your particular startup requires the long hours. Maybe you could restructure the startup? Maybe you could spend some money on employees? I don't know - but overall I don't see how you being a woman makes it in any way special. If anything, you have more choices than men?

<comment><USER3> Good point Tichy - not complaining about being a woman, which I very much enjoy. Being a woman doesn't make, by virtue of your sex-linked chromosomes, doing a startup in anyway special. And as to choices, exactly. That's the point of the post (sorry if that wasn't clear). We all accept (or should accept) responsibility for our own choices. I like your closing comment ;). More choices, more challenges. Synonymous?

<comment><USER6> In hindsight I thought OK, just having kids probably is easier for men. As others commented, it's easy if the spouse stays at home to take care of the kids. It seems kind of equivalent to employing nannies - except it is probably not easy to find adequate nannies?

And the man working long hours also might not benefit from his children much. Except maybe later in life, when they are older and he works less?

I didn't want to come across as harsh. I just suspect that in hindsight it is common to wonder what for did one sacrifice family and children? I am 37 (male) now, and I don't really have a strong inbuilt desire to have children. But one could just go for it and see what happens? If not now, then when? Other things are probably more easy to delay.

I admit that I am also becoming cynical about the traditional roles. I am not that thrilled about having to provide as much money as possible to compete for child rearing women. It may still look as if men have the better position in society, but it doesn't really make much sense. Men don't have any bargaining position at all. They are 100% replaceable (sperm is cheap) - so struggling to be as successful as possible financially is pretty much the only option. Men have to take the risks - women pretty much can opt to take them for fun, or not. More or less, at least...

</comment></comment></comment></comment></comment></comment><comment><USER9> "Not only am I an unpedigreed, first time (at 30, Jesus) female founder of a tech company, I am a non-tech, non-CS founder who doesn't code."

I don't care if you have four dicks. if you are not technical, you have no business running a tech startup.

<comment><USER3> Thanks Isc 12. I'll quit what I'm doing immediately and work on growing four interesting male parts instead.

<comment><USER9> huh? I said male parts won't help. a non-technical person, male or female, running a tech company would be like me running a company that, say, relies on sales to enterprises.

you can't just "know business" - to run a company, you need to understand what that company does.

<comment><AUTHOR> Successful tech companies often need to do much more than just technology to be successful. There's no reason such a company shouldn't have at least one non-technical founder. I agree that it would be a mistake if all the founders were non-technical, or if the non-technical person were running the technical side of the show. But nothing suggests that's true here.

<comment><USER3> Exactly! We've built our company based not just on tech, but market needs. I've seen startup friends build kick-ass stuff that goes nowhere because no one knows about it (the marketing and biz/dev functions) 2 years after launch. Then they're demoralized, die off, and go back to a day job and hacking nights and weekends.</comment></comment></comment></comment></comment>


Secret Files Expose Offshore's Global Impact - <AUTHOR> http://www.icij.org/offshore/secret-files-expose-offshores-global-impact

<comment><USER2> Coral cache may be helpful: [http://www.icij.org.nyud.net/offshore/secret- files-expose-of...](http://www.icij.org.nyud.net/offshore/secret-files-expose- offshores-global-impact)

This is really huge. Offshore accounts are thought to total up to $32 trillion. The Guardian / ICIJ archive accounts for only a portion of this, though I suspect the BVI are attractive to many for their stability (association with Britain provides its advantages).

On the technical side, I'd be interested in what tools are being used to analyze the archive. A related Guardian article notes:

"Unlike the smaller cache of US cables and war logs passed in 2010 to WikiLeaks, the offshore data was not structured or clean, but an unsorted collation of internal memos and instructions, official documents, emails, large and small databases and spreadsheets, scanned passports and accounting ledgers.

"Analysing the immense quantity of information required "free text retrieval" software, which can work with huge volumes of unsorted data. Such high-end systems have been sold for more than a decade to intelligence agencies, law firms and commercial corporations. Journalism is just catching up."

[http://www.guardian.co.uk/uk/2013/apr/04/offshore-secrets- da...](http://www.guardian.co.uk/uk/2013/apr/04/offshore-secrets-data-emails- icij)

Anyone have any specifics on this "free text retrieval" software, its capabilities, and how it compares with, say, standard Linux/Unix text search and processing tools?

<comment><USER3> They provide good detail about the software they used.

"The major software tools used for the Offshore Project were NUIX of Sydney, Australia, and dtSearch of Bethesda, Md. NUIX Pty Ltd provided ICIJ with a limited number of licenses to use its fully featured high-end e-discovery software, free of charge. The listed cost for the NUIX software was higher than a non-profit organization like the ICIJ could afford, if the software had not been donated."

Securiing their communications proved more difficult than free-text search, however:

"The project team's attempts to use encrypted e-mail systems such as PGP ("Pretty Good Privacy") were abandoned because of complexity and unreliability that slowed down information sharing."

<comment><USER2> Ah. My read of the PGP comment was that the offshore-banking individuals / organizations had tried to use PGP but bailed. Seems that your interpretation it was the _investigative team_ that tried but failed is the actual case.

Interesting. Sadly, far too common. I've been using PGP for well over a decade, know a small handful of people (outside of technical mailing lists) who have and can access their keys, and have actually been chewed out by some of these for sending encrypted mail.

</comment></comment></comment><comment><USER4> I love how the article spins it such that offshoring is only for sinister, evil people. The reality is that most people use offshoring as a hedge against insolvent governments, and depending on their industry, the very real threat of criminal prosecution for benign activity.[0]

The moral question isn't 'how can this be prevented?', but 'how can it be made more accessible for the average person?'. I see huge startup opportunities right now around helping people cheaply and easily diversify internationally.

[0] [http://maxkeiser.com/2013/01/17/a-broken-justice-system- most...](http://maxkeiser.com/2013/01/17/a-broken-justice-system-most- americans-commit-about-three-felonies-a-day/)

<comment><USER5> Speaking strictly for Americans, this simply isn't true. There are very, very, very few benefits to keeping your money offshore. Tax fraud is benefit #1. "Asset protection" (which generally means you're hiding money from your spouse or business partners) is benefit #2. Unless you actually own property in the country where you keep your money, or do real business there, those are basically the only benefits. There are also massive costs: (1) the literal costs that the foreign nation (and bank) will charge you to hold your money; (2) the risk that the money will actually be available when you want to access it; (3) the risk that the foreign government and bank will cooperate with the U.S. government when it comes knocking (this risk has greatly increased since the passage of MLAT and other recent treaties).

I could definitely see how citizens of other countries could legitimately use such accounts to get around immoral laws in their home countries. Whether this makes up "most people" who use offshore accounts, as you claim, is beyond my knowledge.

I would bet that a great percentage of the volume of money moving through these offshore havens are indeed a hedge against insolvent governments: the governors' own. There's a reason these island nations are where despots like Gaddafi, Mugabe, Marcos, Bashir, Afwerki, Mbasogo, Mubarak, Kagame, and others, and their extended families, park their money.

<comment><USER4> > Speaking strictly for Americans, this simply isn't true.

You are mistaken. Allow me to enlighten you:

\- <http://en.wikipedia.org/wiki/Executive_Order_6102>

\- <http://www.progress.org/fold13.htm>

\- <http://www.youtube.com/watch?v=ZsIGtSRDy54>

Not to mention the rhetoric inherent in statements like "it's patriotic to pay taxes", or the implicit assumption in policies like tax withholding, that government owns the product of your labor, and 'grants' you a portion of it.

No, history shows us that broke governments are dangerous governments, and if you don't believe that the US is going broke, you are blind and helpless.

<comment><USER6> Yowsa. You need go hole up in the mountains with your survival gear, guns, gold bullion and collection of Glenn Beck books.

<comment><USER4> Way to add something constructive to the discussion.

</comment></comment></comment></comment></comment><comment><USER7> I wonder where they got this information trove from?

Also, I highly recommend the book _Treasure Islands_ by Nicholas Shaxson ([http://www.amazon.com/Treasure-Islands-Uncovering- Offshore-B...](http://www.amazon.com/Treasure-Islands-Uncovering-Offshore- Banking/dp/0230105017/)) to understand the enormity of the problem and its historical context (yes, the growth in activity is easily traced).

<comment><USER2> [http://www.guardian.co.uk/uk/2013/apr/04/offshore-secrets- da...](http://www.guardian.co.uk/uk/2013/apr/04/offshore-secrets-data-emails- icij)

A related article at The Guardian provides more details. Specifically: "The ICIJ's exploration of offshore secrets began when a computer hard drive packed with corporate data arrived in the post. Gerard Ryle, ICIJ's director, obtained the small black box as a result of his three-year investigation of Australia's Firepower scandal, a case involving offshore havens and corporate fraud."

As for the Firepower Scandal: <http://en.wikipedia.org/wiki/Firepower_International>

"Firepower International was advertised as a Hong Kong-based company owned and operated by Global Fuel Technologies Ltd, specializing in technology purporting to reduce the fuel consumption and environmental impact of petrol- operated vehicles.[1] There were other offices in Sydney, China, Rhodes, Athens and Papua New Guinea, according to the now-defunct official company website. But "in reality it was a handful of people in an industrial estate in Perth", who were conducting a complex of fraudulent operations.[2] The original entity—Firepower Operations Pty Ltd—was a $1 company, first registered in December 2004, owned by Firepower Holdings Group Ltd, a company with an address in the British Virgin Islands.[3]"

Oh what a wicked web we weave ...

</comment></comment><comment><USER8> Nice try Guardian: [http://order-order.com/2013/04/04/a-tip-off-for-the- guardian...](http://order-order.com/2013/04/04/a-tip-off-for-the-guardian- investigations-team/)

<comment><USER9> The hypocrisy is so common. I just read on that site that the French politician leading the tax avoidance crack down resigned as he had 600,000 euros in an off short account!

</comment></comment><comment><USER10> Currently the ICIJ web site is buckling from the traffic, but the article and the various summaries elsewhere make for interesting reads.

I can't say I'm not surprised about these revelations, there have been other leaks, articles and essays discussing the problem, but not with the specific details. I'd love to get the full set of documents and run it against specific individuals/corporations in my area, but it would probably confirm what I already suspect.

</comment><comment><USER11> Is this scoop an unintentional advertisement for Swiss banking ? I did a text search for India , Sharad Pawar , Gandhi and Yadav --- nothing showed up. But the rich in India always favor Swiss banking for their illicit wealth.

<comment><USER12> They also like Mauritius.

</comment></comment><comment><USER7> ICIJ have a nice and useful ( _cough_ ) "Interactive: Stash Your Cash" too (<http://www.icij.org/offshore/interactive-stash-your-cash>)

<comment><USER13> Wow. Phoney lawsuit: brilliant.

(not condoning this behavior in any way)

</comment></comment><comment><USER14> Why they haven't released that data for public download on the Internet?

<comment><USER15> Because they are private banking records - some of the account info released is from private citizens doing nothing illegal (offshore banking is explicitly legal in many countries)

One way to look at it is: how would you like it if it was discovered that a drug lord was using the same bank as you to launder money, and therefore your bank account balances and recent transfers were released to the public, even though you had nothing to do with the drug lord other than storing your money at the same bank?

<comment> trust-me I see no problem at all with my bank account balances being published on the Internet. In Norway everybody's total balance is published by the government. I really find irrational some ordinary people desire to keep stuff like net worth, salary, etc. secret.

<comment><USER16> This is not an irrational fear, nor a cultural one. It's mostly tied with local laws and rules. For example, the US being such a hotbed for class actions and frivolous law suits, I can imagine that some wealthy individuals would rather keep their balance from lawsuit 'trolls' ... But that is just my opinion

<comment> trust-me Sure, for the wealthy there are reasons, but not for the 99%.

<comment><USER17> Let me get this straight, because I don't want to misread you. What you are saying here is the wealthy deserve separate protection under the law and anyone in the 99% should have their records published because they don't "need" privacy?

<comment> trust-me Nope, where did I say that anything about making new laws? It want matter much anyway as the really rich people and privately hold companies don't keep the majority of their money in own bank accounts.</comment></comment></comment></comment></comment></comment></comment>


Show HN: Infreqdb – S3 backed key/value database for infrequent read access - <AUTHOR> https://github.com/turbobytes/infreqdb <comment><USER2> Be careful using S3 for lots of small writes. The price is utterly dominated by the PUT calls which cost $0.01/1000.

<comment><AUTHOR> Yes. I should include this info in the README calcs. For my usecase I intend to do 4 - 10 PUTs per hour, since they are batched. 10 PUTS/h = 7300 PUTs/month = $0.073/month. The key here is _infrequent_

<comment><USER2> Be most careful with retries. Saw a case where the bucket wasn't writable and the code just retried in a loop. You get charged for the API call, successful or not.

</comment></comment></comment><comment><USER3> Stuff like this is great! And you said you mentioned you are primarily doing it for timeseries data - then you can easily batch writes to handle high volume throughput too!

We did this with S3 as the storage engine, 100M+ records for $10/day: [https://www.youtube.com/watch?v=x_WqBuEA7s8](https://www.youtube.com/watch?v=x_WqBuEA7s8)

And discord had a very nice article on this as well: [https://blog.discordapp.com/how-discord-stores-billions- of-m...](https://blog.discordapp.com/how-discord-stores-billions-of- messages-7fa6ec7ee4c7)

Great work, I think there is a lot of exciting stuff you can add to it!

<comment><AUTHOR> That's cool. Last week I tried googling for similar stuff but all I could find was people asking "How to run postgres on S3"...

</comment></comment><comment><USER4> We use a similar approach with TrailDB. You can make S3 access totally seamless with user-space page fault handling, which is pretty cool: [http://tech.adroll.com/blog/data/2016/11/29/traildb- mmap-s3....](http://tech.adroll.com/blog/data/2016/11/29/traildb-mmap-s3.html)

<comment><AUTHOR> This is pretty cool. Do you invalidate all pages if the file changes upstream on S3?

<comment><USER4> In our case all blobs in S3 are immutable, so no need to invalidate anything.

</comment></comment></comment><comment><USER5> How is this different than AWS Athena ? - [https://aws.amazon.com/blogs/aws/amazon-athena- interactive-s...](https://aws.amazon.com/blogs/aws/amazon-athena-interactive- sql-queries-for-data-in-amazon-s3/)

Also why don't you dump the log data into a NoSQL like dynamoDB instead of S3 ?

<comment><AUTHOR> Athena looks cool. Didn't know about it. It probably describes what I'm trying to do.

> Also why don't you dump the log data into a NoSQL like dynamoDB instead of > S3 ?

Price.

<comment><USER6> Maybe you should take another look at pricing? DynamoDB is like $0.002/GB/month more expensive than S3 for storage. Requests are more expensive, but you're also not reinventing every wheel and if they're as infrequent a set of requests as this seems designed for it's still only a few bucks a month.

<comment><AUTHOR> My _infrequent_ reads and writes are huge, and possibly spikey.

> Write Throughput: $0.0065 per hour for every 10 units of Write Capacity > (enough capacity to do up to 36,000 writes per hour)*

> A unit of Write Capacity enables you to perform one write per second for > items of up to 1KB in size

As I understand it, for $4.68/month I can only add 36 MB/hour, and thats assuming my objects are in exact multiple of 1KB.

<comment><USER6> Ahh, gotcha. That makes sense, this setup makes a lot more sense if you're dealing with very spikey reads.

</comment></comment></comment></comment></comment><comment><USER7> So, a database that's has a file format of write-once content-addressed "shards" and is persisted/distributed by "offlining" those shards into object storage, and then "onlining" them back into a given DB node's MRU cache at query time; and is mostly read-only, but can update by downloading the relevant shards, modifying them locally, re-hashing the modified shards to get new names for them, and then uploading them again under those new names.

Is this basically equivalent to Datomic, then? (Not that that's a bad thing. The world needs an open-source, non-JVM-targeted Datomic.)

<comment><USER8> > The world needs an open-source, non-JVM-targeted Datomic.)

Couchdb?

</comment></comment><comment><USER9> What do you think about this being able to work with Gzip compressed JSONs stored on S3? Cool project, thank you.

<comment><AUTHOR> Thanks.

I think flat JSON files wont be efficient. My goal is to have the cache on disk, and each cached file would be big with lots of keys on it. In order to use JSON files, I would either have to keep the whole parsed data in memory, or parse the whole JSON each time I want to lookup a key.

If the data fits in memory then sure JSON is more convenient.</comment></comment>


Ask HN: Is there any place on Internet where one can get feedback on essays? - <AUTHOR> Hello, is there any place where one can submit essay drafts and get feedback? I'm a non-native speaker and like to write occasionally. I was thinking if there is a place on internet where I can submit my writings and get feedback. Any help is appreciated. <comment><USER2> Here's an app that helps with simplifying your writing: [http://hemingwayapp.com/](http://hemingwayapp.com/)

While it won't help you to extract meaning and make everything concise, it could help with sharpening your words.

</comment><comment><USER3> I don't know about free feedback, but there are plenty of freelance non- fiction writers and editors out there. Undoubtedly plenty of them would be quite happy to be paid to give feedback on someone else's writing.

</comment><comment><USER4> I have been looking for the same and did not really find any, mostly services for students to get their reports judged.

Subreddits are often good and if you write that you wrote and you want feedback a lot og people are then really nice :)

<comment><AUTHOR> Can you kindly let me know some reddites that you found useful?</comment></comment>


Twisted, Tornado, the Zero-Step, and Activation Energy - <AUTHOR> http://writeonly.wordpress.com/2009/09/12/when-great-features-arent-enough-twisted-tornado-the-zero-step-and-activation-energy/

<comment><USER2> When I saw that horrible hand-drying contraption, my first thought was "am I required to put my hands above shoulder-level to get them dry?" My second was "will it amputate my hands at the wrist?"

Tornado sorta makes me feel better about development, mainly because Twisted is phenomenally intimidating, like a black monolith.</comment>


Python 3.5 type hinting in PyCharm 5 - <AUTHOR> http://blog.jetbrains.com/pycharm/2015/11/python-3-5-type-hinting-in-pycharm-5/ <comment><USER2> Pretty impressive that the syntax:

       def greeting(name: str) -> str:

Turns out to be _backwards_ compatible in Python3 before 3.5 (though not Python2, yet another reason, if you actually needed one, to start new stuff in Python3)

<comment><USER3> How is it backwards compatible? Was there syntax added to the language in preparation?

<comment><USER2> I have no idea how they managed to find something that didn't throw a syntax error, yet looked reasonable. Perhaps someone who followed the process will chime in.

</comment></comment></comment><comment><USER4> Recently stumbled upon IntelliJ using the Go plugin package after a Rubyist friend of mine spoke very highly of RubyMine. I have to say I was incredibly impressed. I looked at it a couple years ago when I was mostly Python and it was ok, but it's really stepped up its game - both feature-wise and visually.

The IntelliJ IDE set is really far better than I understood. IMHO.

<comment><USER5> The things you could do in Go vs in (Python | Ruby) are actually are results of difference between static vs dynamic typing. Yeah, static typing requires more key presses and some thinking, but in the case of tools (navigation, refactoring) it makes a huge difference. Ruby support would be years ahead of what it is now if the tools could understand the types in all cases.

</comment></comment><comment><USER6> I can see this being very useful in teams. Hell, its even useful for a single developer as a form of future documentation. I generally write code today and think, when I read this in 6 months and have no memory, how can I help my future self? For new code bases I'll be adding this.

<comment><USER7> I'm the author of the article. When I gave a preview of this at the PyRVA meetup, I gave this exact illustration, even using "my future self". So...jinx. :)

<comment><USER6> haha maybe we're twins!

</comment></comment></comment><comment><USER8> I wonder why, given they call it 'type hints', they didn't go for Haskell-like type comments:

        #greeting :: (str, int) -> str
        def greeting(name, age):

In fact, I wonder why more languages don't adopt this syntax, since it greatly simplifies functions with very large types.

<comment><USER9> I love Haskell, but I'm not sure this would actually be better for Python. To be Haskell-like, the comment would actually be different than what you wrote:

#greeting :: str -> int -> str

and this is specifically related to the functional nature of Haskell. `greeting "hello"` now instantly becomes a _function_ with type :: int -> str. So the Haskell syntax is about currying, it's about how the function you wrote is really just a cascade of functions.

Doing something silly like making the "arguments" reside in an artificial tuple would definitely be bad practice in most Haskell cases, and so syntactically looks a bit weird and out of place, like it's shoehorned on just because Python function arguments are not like Haskell arguments.

Since Python is not based on this model of a function, I think it's a lot better to keep the syntax in the actual def statement. Adding the return type annotation at the end is nice.

Most probably, this stuff was added to the signature because there were already natural tools like the inspect library that would provide the immediate hooks for making use of it. But even if the implementation didn't need it to be part of the signature like that, I still think it's better for Python's paradigm.

<comment><USER8> I don't think this is a case of shoehorning anything. It is true that in Haskell you use currying more often, while in Python you usually supply a tuple with all the arguments at once.

However there are also cases in Python where you get currying, like decorators. And for complicated types like:

        # greeting :: (str -> (str, str), int -> List[str]) -> Dict[str, int -> int]

being able to separate the signature from the declaration certainly makes reading it easier.

<comment><USER9> But in Python, to do currying, you either use decorators, functools.partial, or forced early binding (like nested function factories), etc. Just because currying is a thing you can do in the language (in a much different way than in Haskell where it is fundamental to every function) doesn't seem like a good reason to me to adopt Haskell-like type signatures.

Just for example, in Python it's extremely common to use default arguments, and in fact a lot of stuff in the typing module exists to make it easier to do something like make an Optional type that can either be `None` or a `str` or something. So whatever type signature you're going to mimic from Haskell, you won't be able to give optional types or default arguments without making it totally un-Haskell. And of course no one is going to go off and change Python to work based on Haskell-like type classes (nor should they).

</comment></comment></comment></comment><comment><USER10> There's quite some dissonance between the artificial "generics" vs normal types. Why the concrete str type for strings, but not for lists? I realize there are implementation level arguments for it, but the resulting compromise does not seem good.

And specifying the concrete str type goes diametrically against the "don't check against a concrete type, embrace duck typing" mantra that Python programmers are taught early.

<comment><USER11> Probably because Python doesn't distinguish characters and strings. When you index into a string, you get a string back (that happens to be length one, of course), which means that a string is a list of strings. Thus, exploding string into a list type isn't really possible as the type of `str` is something like `List[str]`, which doesn't make a whole lot of sense.

</comment></comment><comment><USER12> something that's always had me a bit confused:

the type hinting in Python is just that -- _hinting,_ right?

It's for IDEs like PyCharm to use to generate better docs and to be more intelligent about suggestions when writing code. But what it does NOT do is inform the interpreter/compiler about type information, correct?

I.e., Python3 code with perfectly implemented type hints would only make it easier on the developer, and not the interpreter/compiler, yeah?

<comment><USER13> Oh, I was somehow hoping this would allow you to have multiple function signatures. i.e. two functions foo, both accepting one variable named bar, but one would implement str and one int, for example.

<comment><USER14> You can have that since 3.4, thanks to PEP 443 ([https://www.python.org/dev/peps/pep-0443/](https://www.python.org/dev/peps/pep-0443/)). Here's a (slightly contrived) example: [https://gist.github.com/zsiciarz/e15199d9be1b0e07faf6](https://gist.github.com/zsiciarz/e15199d9be1b0e07faf6)

</comment></comment></comment><comment><USER15> "Type hinting" without checking is kind of lame. If you're going to have declarations such as

        def zero_all_vars(vars: Iterable[LoggedVar[int]]) -> None:

people are going to get them wrong some of the time. Dumping the checking problem on some IDE, with a very vague specification of what an error is, is not progress. If functions have type specifications, you should not be able to call them with the wrong type, ever. This is the job of the compiler and the run time system, not the IDE.

Somebody should stop Python's little tin god before he creates another giant step backwards like the Python 2 to 3 mess.

<comment><USER16> So just run the compiler and runtime type checker, what's the problem?

<comment><USER17> Usually optional tooling tends to be ignored.

For example, C would have been a much safer language if lint was part of the compiler, instead of being optional.

<comment><USER16> Sure, but how is that "another 2 to 3 mess"? If anything, the backwards- incompatible changes made the language more robust.

<comment><USER17> I was replying to the issue that this being two separate tools:

"This is the job of the compiler and the run time system, not the IDE."

Making this optional will mean the majority won't bother to run the checker.

How many do bother to run lint, jslint, core.typed or dialyzer?

<comment><USER18> that doesn't make sense? Who would go to the effort of adding type hints and then not run the checker?

<comment><USER17> Just spend some months working in enterprise environments....

</comment></comment></comment></comment></comment></comment></comment><comment><USER19> I have fond memories of using Python 8-10 years ago, but looking at it today, I'm not sure I see compelling reasons not to just use C# instead.

I can have real static typing if I want it. I can also have dynamic typing if I want that. I can use lambda functions. I have LINQ. I can use ScriptCS or some of the new Roslyn stuff if I want to use it as a scripting language. Unix support is not as great, but they are coming along with that. There's no GIL.

Is there some killer feature for Python that I'm overlooking?

<comment><USER20> Compared to C#, it doesn't require you to run Windows in production.

<comment><USER21> C# doesn't require you to run Windows in production, even before .NET was open-sourced. Mono works well for a lot of use cases.

<comment><USER22> Mono performance was at least an order of magnitude worse than MS .Net. At least before the open sourcing of .Net.

</comment></comment></comment></comment><comment><USER23> I like this. I use Ruby as a scripting language more often than Python and I would be happy to see similar additions to Ruby to improve code readability and IDE support (although RubyMine) does an awesome job of warning of possible code problems just using code inspection).

I sometimes wish that one programming language would work for all of my requirements but I keep needing: scripting languages for quickly getting stuff done; practical languages like Clojure and Java, and type strict languages like Haskell (that I would like to use more, but development takes longer).

Looking ahead a few years, the ideal for me would be a loose scripting language that had awesome static analysis tools. So, for example, I would get the safety of Haskell when hacking in Ruby.

</comment><comment><USER24> It's cool that PyCharm added support for this -- the point of the typing module and the type hinting PEP is for IDEs to adopt it for linting purposes.

For those of you using vim, emacs, and the standard UNIX editing toolchain, the mypy linter provides command-line support. This blog post from April 2015 describes the first version of mypy that includes support for the official typing library.

[http://mypy-lang.blogspot.co.uk/2015/04/mypy-02-released.htm...](http://mypy- lang.blogspot.co.uk/2015/04/mypy-02-released.html)

</comment><comment><USER25> So how do we get some of these benefits if we don't use PyCharm, but use Vim? Are there standalone tools, or VIM plugins?

<comment><USER26> If you can deal with a different file navigation setup (and outside a terminal obviously), VIM emulation in PyCharm is very good.

Having recently switched, PyCharm really does make larger projects a lot easier to refactor / manage.

Edit: Also, to actually answer your question, this is a relatively new feature so I don't think any of the VIM plugins support it yet. Jedi has had type hinting from doc strings for some time now. I'd imagine they'll update that to use this soon.

</comment></comment><comment><USER27> Can this be fixed?

[http://stackoverflow.com/questions/25588642/pycharm-false- sy...](http://stackoverflow.com/questions/25588642/pycharm-false-syntax-error- using-turtle)

</comment><comment><USER28> Has someone tried GNOME Builder? I found it quite good for C, but the Python mode of PyCharm is still superior regarding search and introspection, yet it is not Free Software…

</comment><comment><USER29> Finally! without the additional type information, it's almost impossible to do any meaningful refactoring...</comment>


CamelotPro – A PRO layer on an OSS library to extract tables - <AUTHOR> CamelotPro is a layer on camelot-py library to extract tables from Scan PDFs and Images.<p>CamelotPro is no different from the original Camelot to code, it comes with extra flavor="CamelotPro" in read_pdf(), along with regular "lattice" and "stream". <comment><AUTHOR> Hello All,

I'm new to the community, heard this is the best place to receive valuable feedback as well as attract early adopters.

To make it one stop place for developers to extract tables, we built camelotpro, a wrapper on OSS, camelot-py.

Thanks, Aks [https://github.com/ExtractTable/camelotpro](https://github.com/ExtractTable/camelotpro)

</comment><comment><USER2> Welcome Akshwohini,

That was a good work, also easy to try instructions found on the repo page. While I enjoy the free credits for now, I've concerns about the future, as there is no information provided on how to purchase more credits, pricing et al

<comment><AUTHOR> Hey there,

Thanks for trying it out. The wrapper is as a part of making easy adoption of [https://extracttable.com](https://extracttable.com) . The pricing and documentation of API behind scenes can be found at the above website . Also, I want to mention that the credits received for camelotpro can also be used for extracttable.com api. For any issues, please contact at email@example.com

P.S: Noted to mention these details on camelotpro repo as well.

Thanks, Aks</comment></comment>


Google and Gmail blocked in Iran, im a iranian programmer wth am i doing - <AUTHOR> http://www.smh.com.au/world/google-and-gmail-blocked-in-iran-20120924-26hex.html

<comment><USER2> Are you really posting from Iran? Are you familiar with the Tor Project:

<http://www.tor-project.org/>

In any case, please be careful there, friend. Be very careful.</comment>