Blockspring: Do anything in a spreadsheet - <AUTHOR> https://a16z.com/2015/07/29/blockspring/ <comment><USER2> The "Suddenly any financial analyst in your company, _given the right permissions_ " comment is telling. The companies that would be most likely to use a service like Blockspring would be the types of companies that would _never_ be willing to give and trust data access and processing to a random third party. The Privacy policy does not help alleviate these concerns (tl;dr: we keep everything). It is also unclear if the business offering ([https://www.blockspring.com/business](https://www.blockspring.com/business) ) offers self-hosting.

Case in point, one of the example scripts in the blog post ([https://open.blockspring.com/pkpp1233/get-amazon-new- price-b...](https://open.blockspring.com/pkpp1233/get-amazon-new-price-by- product-id) ) requires you to input a _Amazon Product Access Key_ and a _Amazon Product Secret Key_ as parameters.

<comment><USER3> I recently discovered the joys of ExcelDNA, which lets you do spreadsheet extensions using C#. One of the first things I did was spin up a plugin that allows users to request data elements from CSV files hosted on a server, essentially a giant remote VLOOKUP. The product uses TCP/IP so it can run anywhere - client self-hosted, or remote over the internet. Throughput was pretty high - 70k elements per second over a network. This, and I'm a novice programmer.

So I look at Blockspring, and my first thought is that I don't quite get it, although if they ran on proprietary, internal data sources, it would make sense. (Do people really download Amazon listings into Excel? I suppose). Or, if they could get a company like Factset or Bloomberg to let them do direct data pulls from their databases, then you'd get masses of high quality data to analyze.

In terms of the core "into excel" functionality - maybe I am understating my novice capabilities, but ever since I discovered ExcelDNA it seems like you can do almost anything with Excel and C#.

<comment><USER4> > if they could get a company like Factset or Bloomberg to let them do direct > data pulls from their databases

this exists. bloomberg has a plug-in that lets you pull much of its data directly into excel using normal-looking formulas. the catch is you have to pay $20,000/year.

</comment></comment></comment><comment><USER5> [https://www.blockspring.com/about/privacy](https://www.blockspring.com/about/privacy)

User Content. We collect your personal information contained in any User Content you create, share, store, or submit to or through the Service, which may include photo or other image files, written comments, information, data, text, scripts, graphics, code, and other interactive features generated, provided, or otherwise made accessible to you and other Users by Blockspring via the Service in accordance with your personal settings.

<comment><USER6> Sounds right. They need permission to display your files back to you, or to others if you want them to.

</comment></comment><comment><USER7> This is pretty awesome, and it's exactly the sort of product that makes me wonder if the "in the future everyone will need to code" predictions are going to pan out. With better tools that can 'intelligently' help a non-coder[1] use APIs and algorithms, the coding bit of coding becomes somewhat redundant. We'll still need the problem solving and analytics skills, and some of the organisation but actually sitting and typing in syntax could well vanish. It's feasible that a suitably easy to use programming language would make coding so trivial it actually stops being a 'skill', in the same way that wordprocessors and printers have made copy typists redundant.

[1] As a caveat to that I would also argue that a competent Excel user actually is a coder, just with a higher level macro language than most other developers, but still.

<comment><USER8> This gets me thinking.

I wonder if why spreadsheets become something like programming-lite is that it allows for rapid prototyping and iterating.

This because the user can manipulate the content of variables directly, and watch the result change as they do. This then allows them to test smaller parts of the logic before aiming the result at another piece of logic and thus building a larger structure piece by piece.

The problem with using spreadsheets for this is that as you move stuff around the logic parts break.

So what if we had something like a variable grid and a logic sheet? So that any variable read or written was up in a grid like that of a spreadsheet, for easy reference and manipulation, while the logic was of somewhere else. Thus moving things around to make room for more logic didn't influence the variables etc.

And to take a step back, i wonder if this ease of prototyping was one of the things that made micro-computers so potent. This because they booted directly into a Basic interpreter (prototyping), while at the same time there was no persistent storage you could screw up with crap code (just flick the power and get back to it).

<comment><USER9> I think flow-based programming was supposed to be something like that. As spreadsheets show us, we haven't really figured out how to scale this approach to more complex applications.

<comment><USER10> Spreadsheets show us that 2d grids and a lack of other programming concepts make it very hard - it's a distinct subset of flow programming:

A unit of code an take an arbitrary number of inputs from a 2d grid and give you 1 output. Real Flow programming lets you have multiple inputs and outputs and aren't constrained to 2d. I think the biggest constraint is representing the dimensionality and interaction ... but we have classes and deep package structure in Java, so it's probably more of a effect than cause, but still a blocker to wide adoption.

</comment></comment></comment></comment><comment><USER11> The code in spreadsheets is always a 2nd-class or 3rd-class citizen. It makes it ridiculously hard to both test and review and there are essentially no good checks on what output you're getting from the input.

I've worked in orgs where C-level execs without a technical background were writing crazy complicated spreadsheets and nobody had any idea how they worked or if they were correct. When you bury the code, you bury the necessary processes around the code.

This is dangerous.

<comment><USER12> Why is everyone hating on Excel, and spreadsheets in general? They are such easy tools for creating, displaying and auditing calculations (like financial models), manipulating rows of similar data, and generating regular charts and reports.

Sure, their ease of use makes them popular with, and accessible to, people who don't think like we do. This doesn't mean they are bad tools for everyone. Python's duck typing may horrify Java programmers, but it doesn't mean that all Python code is bad or has type errors.

<comment><USER13> > Why is everyone hating on Excel, and spreadsheets in general?

Speaking for myself, because spreadsheets are intrinsically "information silos." Traditionally, this has been due to the data being locked up in the spreadsheet's proprietary files. This offering only exacerbates the issue by incorporating data from external sources which lack data management guarantees available in other solutions.

> They are such easy tools for creating, displaying and auditing calculations > (like financial models), manipulating rows of similar data, and generating > regular charts and reports.

When their use is restricted to this, spreadsheets do provide value. The problem is that many times the spreadsheets start off as a data integration scratch pad (as it were) and then become the "system of reference."

Used as a tool which enables one-off types of value it's hard to argue against them (so I won't :-)).

> Python's duck typing may horrify Java programmers, but it doesn't mean that > all Python code is bad or has type errors.

This is a false dichotomy[1], or a straw man[2] if you prefer, as both Python and Java are general purpose programming languages.

Excel is not a general purpose programming language. It is not a programming language. It is a program which allows people to use a _very specific DSL_ to benefit from it in novel ways. And when they do, any chance of autonomous integration is lost.

1 - [http://www.philosophy-index.com/logic/fallacies/false- dilemm...](http://www.philosophy-index.com/logic/fallacies/false-dilemma.php)

2 - [https://en.wikipedia.org/wiki/Informal_fallacy](https://en.wikipedia.org/wiki/Informal_fallacy)

</comment></comment></comment><comment><USER14> I think this concept has far reaching effects on optimizing white collar operations if you're able create social features based on the data.

I'd want to know who else in the company is using this data? Who has used it on the past? Have they done work that is similar or even a duplication of the work I'm doing?

These information management issues are currently hidden, but result in lost productivity. Just this past month my friend at Google found out another person had already done his analysis and he could learn from the previous work. Just knowing someone had previously used the same dataset could have saved him 7 weeks of work.

<comment><USER2> > _Just knowing someone had previously used the same dataset could have saved > him 7 weeks of work._

Information about data is not black-and-white in the corporate world. Even in the same company, internal data is usually on a need-to-know basis, with necessary NDAs. A service that tells you who else is using the data would defeat the purpose.

As an extreme example, what would happen if a service allowed data-sociality with HIPPA-protected data? Not good things.

<comment><USER15> I don't think open access lists for HIPAA data would be a bad thing. Why do you? In fact, it sounds like a great idea. Shine some light on who is accessing what data.

<comment><USER16> The fact that a certain specialist has accessed a patient's records would reveal something about their health. Even the frequency of general access conveys information about your health.

<comment><USER15> I'm thinking more internally, non-doctor users.

</comment></comment></comment></comment></comment><comment><USER17> A nice hack for someone who wants to use this tool (granted that it is free for non-corporate customers):

If you are a data-scientist and would like to extract public-data from 1 of the APIs they provide, you can use this tool, pull that data (take it in raw form) and then export as both a spreadsheet and as CSV.

You can then use the CSV with your data-analysis toolset (Python probably has support), and voila! You've saved yourself the headache of wrangling APIs.

Apart from that, I'm going to go with the opinion of the other comments made that you probably don't want to be putting your private Excel info on another acquisition-driven startup.

</comment><comment><USER18> I doubt anyone with hardcore spreadsheet data would trust a third party plugins. My employer rolls most software (incl Excel plugins) in-house for that exact reason - can't let outsiders near the data.

</comment><comment><USER19> This is such an interesting angle on making technical tools more accessible. I can't say I have a use for including functions in spreadsheets — but when I imagine where this could go it's exciting.

</comment><comment><USER20> [https://open.blockspring.com/browse](https://open.blockspring.com/browse)

For anyone who wants to view the actual blockspring UI/UX along with what APIs are available.

I personally prefer seeing the product rather than reading marketing materials. In many cases (not specific to blockspring) they're completely different in terms of design, messaging and stated features.

<comment><USER20> As an early user of blockspring I was excited to login and add another one of my APIs (bitcoin related) that would be perfect for having in spreadsheets. Unfortunately I couldn't find any obvious way to add an API. Naturally I hacked it by going to one of my existing APIs (thanks google, I couldn't find a URL from the UI otherwise) and cloning it. Here's the UI of the create API page[1] for anyone who's interested.

[1] [http://i.imgur.com/wQoyhT1.png](http://i.imgur.com/wQoyhT1.png)

Edit: You can create new APIs on blockspring by going directly to a route[2] which still works. I saw Alexa had an API but it needed authentication credentials, and since I already had found Alexa's undocumented API[3] I added it to blockspring[4] just for fun.

[2] [https://open.blockspring.com/blocks/new](https://open.blockspring.com/blocks/new)

[3] [http://data.alexa.com/data?cli=10&url=news.ycombinator.com](http://data.alexa.com/data?cli=10&url=news.ycombinator.com)

[4] [https://open.blockspring.com/montanaflynn/alexa- rank](https://open.blockspring.com/montanaflynn/alexa-rank)

<comment><USER21> Wow great lenghts but in the footer you can just click "Create block" :)

</comment></comment></comment><comment><USER22> How does this differ from building plugins? Excel already has a programming interface and I'm pretty sure you can script Sheets with python. Is it just easier?

</comment><comment><USER23> So they made PowerQuery that you have to pay for?

<comment><USER24> You don't need PowerQuery when you can have full SQL inside Excel, and in a very elegant way: [http://thingiequery.com/](http://thingiequery.com/)

</comment></comment><comment><USER25> There was Resolver One, which apparently got acquired by PythonAnywhere, which allowed Python scripted spreadsheets. It was AWESOME when I tried it, but tad too slow and no takers in the office because spreadsheets get passed around and need to be verified/rerun by people who do not know any programming. Even VBA at times.

Their OSS version is apparently still available at [https://github.com/pythonanywhere/dirigible- spreadsheet](https://github.com/pythonanywhere/dirigible-spreadsheet) (haven't tried it.)

</comment><comment><USER26> How much does this cost to use? I was not able to find a pricing page on their website.

<comment><USER27> $7/user/month for team of $50/user/month for enterprise (extra admin support and training). [https://open.blockspring.com/organization/setup](https://open.blockspring.com/organization/setup)

</comment></comment><comment><USER28> Feels narrow and I don't see the payout.

"They just need an easier on-ramp to their internal database to do things like comparing projected revenue to actual revenue in order to close out this month's financials."

This is a small gap. On one side, you have people copy-pasting CSV files into excel and doing it one-off.

On the other side, you have specialized SaaS products for business workflows. (Wouldn't you rather just use some accounting software to run your companies financials, rather than muck around in Excel?)

In the middle, you have people writing either code or Excel Macros to automate some of this. Looks like BlockSpring is meant to automate that group away. Just having spent 3 minutes with blockspring, there's friction. (Log into this service, fiddle with the parameters, clean up data, remove mess, etc)

I can see the IFTTT / Zapier comparisons but this is a lot more work and far less reward. Those tools give immediate results.

At the end of my integration with Blockspring, I'm left with a table of data that I STILL have to analyze in Excel.

<comment><USER29> Huge market.. e.g. look at Domo. Same idea.

</comment></comment><comment><USER30> Goldman Sachs already has a lot of financial and internal data available in spreadsheet plugins via Slang, and it's generally very useful for financial models (at least when I was there). This just seems like a way to generalize that for the masses and for other types of data.

</comment><comment><USER31> Does everybody need to learn to code?

One's health is pretty important to one's staying alive - not everyone learns to be a doctor.

Finance and accounting is the lifeblood of a business, not everyone learns to be an accountant.

Everyone can do it, you should - no you must - do it too. I think it devalues the profession.

<comment><USER32> Basic skills in medicine and accounting are useful to have and sort of prevalent. I'm sure you put on a band-aid before. Basic skills in automating certain computer tasks seem to be valuable to many people -- almost all office workers, I'd wager.

I don't see how that devalues the profession. There's a huge difference between what people can achieve using stuff like this and what I do every day.

Maybe basic skills in stuff like this will even translate to a better appreciation of what professional developers do. A guy can dream...

<comment><USER33> When it comes to programming, 'putting on a band-aid' is analagous to 'entering user/pass credentials'. Yes, everyone can put on a band-aid, just like everyone can log on to something. Neither take skill or understanding, and both are 'just things you do'.

On a tangent, a great many people have positively woeful understanding of biology. I think biology should be a mandatory subject in high school. After all, the one thing that absolutely every human has and uses throughout their entire life is their body; that can't be said of pretty much any other high school subject.

</comment></comment></comment><comment><USER34> You learn to code that way or you learn to code this way, either way you're gonna pay the piper.

<comment><USER35> I started learning to code via spreadsheets (anyone remember Borland Quattro Pro?!)

</comment></comment><comment><USER24> I don't think "anything" is a good term. Its good use cases are only of read- only data. Reading (and preprocessing) data from your services/databases inside a spreadsheet is quite awesome.

</comment><comment><USER36> You've been able to do this in Google Sheets for years with a tiny bit of JS programming.

<comment><USER24> I tried, and it is impossible. Google Apps Script is awfully slow and limited.

</comment></comment><comment><USER37> No no no. People at my company already use spreadsheets where they don't belong (in the place of databases, note taking apps, regular expressions, and one time someone tried to do page layout with it). The last thing we need is for these bozos to program with it. I see the need for an intermediate entryway into programming (Hypercard), just don't tack it on to an already bloated program like excel.

<comment><USER38> I think you need to sit down next to those "bozos" for a day or two and see their actual workflow. I've seen and worked closely with people using Excel instead of databases; hell, myself I used it as a map editor for a 2D game (for page layout I prefer PowerPoint).

The reason Excel is so popular is not because non-coders are dumb. It's because "tools" provided by IT department are usually piece of crap that makes work more difficult for people using them. Take using Excel instead of databases - why spreadsheet is better? Because it has easier data entry, and it's free-form. When you need to modify your workflow to include some additional notes, you don't have to write to the manager of the manager of the internal procedures to maybe consider updating the schema somewhen before next decade. You just add a new column and you're done.

I'm sure when someone creates a program that actually solves the problems those people encounter every day, they'll gladly switch to using it. But it's easier to say that users are stupid instead of actually caring about what their needs are.

<comment><USER39> I don't think the users of spreadsheets are dumb. I think they don't have an appreciation for organization of data to allow for efficient machine- readability. Your use of a spreadsheet as a game editor shows that you're one of the exceptions that proves the rule.

The free-form nature of spreadsheets, in my experience, almost always makes for human readable repositories of data that are nearlyuseless from an automated processing perspective. I cringe when I'm asked to export data from a database into a spreadsheet for users to edit, because I know I'm going to get back an unparsable mess that I'm going to have to invest time in to make usable again.

The spreadsheet is, effectively, shifting the user's inability to understand why data needs to be organized well off to a programmer (or some other grunt) who ends up fixing all the data up. (Or, alternatively, nobody fixes it up and the business ends up making bad decisions based on garbage data that came from a spreadsheet that's a disorganized mess.)

Edit: Grammar.

<comment><USER40> > _I don 't think the users of spreadsheets are dumb. I think they don't have > an appreciation for organization of data to allow for efficient machine- > readability._

Well said...it's that concept of machine-readability that is so core to efficient data investigation and interoperability with other enterprises (such as web apps)...theoretically, such a concept is learnable without a decent jump into the computer science/programming waters...but honestly, I haven't run into many lay persons who have figured it out well enough for themselves.

</comment></comment></comment></comment><comment> kuni-toko-tachi I've yet to see a product that turns non software engineers into software engineers. These types of products only make salespeople commissions and then sit unused. Crystal Reports is a prime example. I've seen it sold many times to people at companies who buy into the notion that software engineers can be replaced by drag and drop tools. The people who buy into the sales pitch lack the desire or discipline to learn it. Software engineering is difficult work and requires patience, which is often lacking in people who think a half-assed design document written in MS Word is anywhere equivalent to writing bug free scalable code that needs to run 24/7.

</comment><comment><USER41> Can you explain the tech behind this? How do you go about building an excel and google docs plug-in with custom formulas?

<comment><USER21> A good starting point: [https://developers.google.com/apps- script/quickstart/docs](https://developers.google.com/apps- script/quickstart/docs)

<comment><USER41> what about for excel?

</comment></comment></comment><comment><USER42> >What is your exit strategy?

>Acquisition by Google.

<comment><USER43> Please don't quote unless it's an actual quote.

<comment><USER42> >Please don't quote unless it's an actual quote. - bpodgursky</comment></comment></comment>


How I discovered my left AirPod was bad - <AUTHOR> https://www.jeffgeerling.com/blog/2017/how-i-discovered-my-left-airpod-was-bad <comment><USER2> > I can walk about 50-60' while listening to music with no dropouts!

What kind of magic are these things? My bluetooth connection from my phone to headphones or the car will start to drop if too much of my body gets in between the radios.

<comment><USER3> Is that inches or feet? With my bose qc35 I can walk 5 to 10 meters away from my phone/computer. (10 if clear view, 5 if wall)

1.5 meters seems very little and 18 meters seems too much.

<comment><USER4> At work I can leave my phone in my office and go get a drink from the kitchen without the audio dropping out. It's probably a good 15 meters away with 2 walls in between.

<comment><USER5> Ditto. I can walk to the restroom and my QC35 don't disconnect. 15+m, 2 walls (all tile bathroom walls) in between. Not so much as a crackle.

</comment></comment></comment></comment><comment><USER6> Well, since we are going to have a headphone jack debate at some point in this thread anyway, I've got a question.

How are wireless headphones dealing with something as simple as switching to a different device? I can just replug my wired headphones, but what about wireless? I know AirPods do some of that W1 magic, but that's only if you are dealing with Apple devices. What if I just randomly show up to a friend's house with my wireless headphones and want to listen to music on his ThinkPad?

<comment><USER7> Varies by device but it's basically just repairing Bluetooth. I have a pair of Marshall wireless headphones and double-clicking the button on the right earphone puts them into pairing mode. From there it's two clicks on a MacBook to pair from the Bluetooth menu or a few more than that to pair to a mobile device.

<comment><USER8> Can I just take a moment to say how hilarious it is that all the guitar companies have licensed their name out for consumer audio products?

The audio system in my car is "Fender".

<comment><USER9> The big-name manufacturers are now lifestyle brands.

Serious players know that if you want a Strat or Tele, you get a better instrument from G&L or Suhr than from Fender. Gibson's reputation is in tatters after years of poor quality control and dubious design decisions. A group of Marshall's technical staff left the company to form Blackstar Amps after Marshall went public.

Most of the big musical instrument brands are going the way of Chrysler and Chevrolet - they trade on an old brand image, but there's little of the original DNA left. It's clearly working in the short-term, but I suspect that these brands will fade away with the Baby Boomers that have a nostalgic attachment to them.

</comment></comment></comment></comment><comment><USER10> Are these things as leaky as the headphones that come with a new iphone? The reason those are unsuitable in a work environment is that your colleagues will hear almost as much your music as you do.

</comment><comment><USER11> I love the saccharine tone of the article. If this happened to another brand of earphones and the guy was dealing with a different electronics store, it would be filled with rage and tears.

<comment><USER12> Yep. I had similar problems with my AirPods and wasn't about to be Apple's debugger for free for hours on end - I just returned them.

That doesn't make for as interesting a story, but it's clear they were rushed out the door (poor soldering according to the teardowns, tons of problems like this report, extremely limited supply months later) and it really makes me wonder about the future of Apple when they can't even nail hardware anymore.

I know the canard of "don't buy V1 of an Apple product" but it seemed to have been better in recent years. Until this, at least.

</comment></comment><comment><USER13> I'm guessing lessons won't be learned here until this happens out of warranty. Headphones shouldn't be this complicated.

<comment><USER14> In my dream world, Apple would make some sort of standardized bluetooth receiver similar to AirPods minus the actual audio hardware. It would just be the transceiver, electronics, sensors, and DAC. Then you could attach it (via e.g. a UE or Shure audio cable connector) to whatever IEMs you wanted.

This would not only address the fact that the actual audio on apple buds is very mediocre, but would also allow for some separation of concerns and easier replaceability/customizability.

<comment><USER15> They already make things like that. [https://www.amazon.com/Anker-Bluetooth- Transmitter-Receiver-...](https://www.amazon.com/Anker-Bluetooth-Transmitter- Receiver-Headphones/dp/B00E174RTS?tag=headphcompar-20)

Some companies like Westone, also make a replacement bluetooth cable since their IEMs all have replaceable cables. [https://www.amazon.com/Westone-MMCX- Bluetooth-Cable-78548/dp...](https://www.amazon.com/Westone-MMCX-Bluetooth- Cable-78548/dp/B01H3XS2ZQ)

I am pretty sure the correct answer is: Stop making phones so thing and bring back the 3.5 headphone jack.

If the 3.5 really is a hindrance, come up with a connector that allows a simple adaptor to 3.5 jacks and _still allows cabled charging_. I already have a couple of L-adaptors for my headphones with straight connectors, so carrying a 3.5-to-Lightning doesn't bother me. But not being able to charge at the same time. WTF?

iPhone SE forever!

<comment><USER16> The thinness wasn't the problem - there are hundreds of phones thinner than an iPhone 7 that still have the 3.5mm jack (e.g. Galaxy A8 is 5.9mm compared to iPhone 7's 7.1mm).

The problem was the increased space required for the taptic engine.

</comment></comment></comment></comment><comment><USER17> That was a really interesting read, and almost amazing the lengths that Jeff was willing to go to debug the problem. But it's also worth considering, how much work was possibly wasted during this arduous debug process? I say that not to knock the work Jeff did, but really to appreciate the creative process which is "debugging" and this is a great example of where a different strategy could save a lot of time.

Anytime you debug a problem, you create in your mind a list of likely candidates for the problem, and sort them based on a combination of how likely the problem is being caused by 'X' multiplied by how hard it is to prove the problem is _not_ caused by 'X'.

I may have missed something, but from the story it seems like testing with one AirBud versus the other AirBud active would have isolated the problem immediately to the left pod. If you see different performance on left vs. right pod, you know immediately the problem almost certainly has nothing to do with your RF environment. So before going to extreme lengths to modify the RF environment, you absolutely must ask yourself; how can I prove the problem is not RF?

Well, you have one shared RF environment. So naturally you need two or more DUTs (device under test) to run the test against and see if the problem is consistent across different DUTs.

What's so neat about this particular case is that Jeff already _had_ two DUTs, since the pods can apparently operate completely independently. Having two DUTs, it was a massive missed opportunity to test them independently before going down the RF rat hole.

Please don't read this as a critique of Jeff, his write-up is a great place for learning to happen!

</comment><comment><USER18> I have a ton of lag when I use my Bose Bluetooth speaker with my MacBook Pro, or when I airplay to my Apple TV. I tried using Bluetooth Explorer, but I can't figure out what's causing the interference. Anyone have tips on how to debug this?

<comment><USER19> From my experience, macOS just seems to be really conservative with avoiding drop-outs in the audio. Are you using the speakers for music, or more general use? I've found it tends to value quality over responsiveness with music in particular.

<comment><USER18> It's for music.

</comment></comment></comment><comment><USER20> Thanks for the hint with the Bluetooth Explorer!

<comment><AUTHOR> It's one of those random apps that you'd never even think about unless you hear it mentioned deep in a thread on some random forum... I think it was mentioned in a rather old MacRumors thread discussing Logitech mouse interference. Definitely a good tool to have around, and saved me buying or borrowing an expensive spectrum analyzer :)

</comment></comment><comment><USER21> Great article, a few people have complained about drop-outs that others are not seeing - I wonder if this is a semi-common problem?

</comment><comment><USER22> I love these kind of articles. For some people it might be a waste of time, but to me is really good to see such amazing willingness to discover the problem. I am tempted to get a pair of AirPods, not totally sure yet, but if in the future I will get one, I will definitely remember this post in case I have troubles with them.

Well done!

</comment><comment><USER23> all this for headphones!

<comment><USER24> Are you in with the in-crowd?

</comment></comment><comment><USER25> Maybe someone can answer this:

Why isn't there some sort of cache on the earbuds that allows the device to send the packets to it and buffer as needed (for, say, music)?

<comment><USER26> It would only work for music; for video or games or anything else a large buffer would mean terribly unsynchronized and delayed audio. Humans will feel something is off with just a few tens of milliseconds of buffer.</comment></comment>


Dynamic Linking - <AUTHOR> http://harmful.cat-v.org/software/dynamic-linking/ <comment><USER3> Dynamic linking is a key innovation that we abandon at our own peril. Only those who are comfortable living in the world dynamic linking creates can agitate for its removal. In a way, advocates of pure static linking are like anti-vaccine advocates.

Here's a killer application of dynamic linking: NSS. It's _wonderful_ to be able to extend applications with new notions of names and addresses.

Remember the bad old days when /etc/passwd parsing was hardcoded and you had a choice of either local users or NIS, and that choice was baked into every application? PAM is another good example.

By the way: there's no such thing as static linking in Windows. All system calls go through ntdll.dll. (In fact, the actual system call numbers are scrambled with every service pack.) This property allows _wonderful_ things, including a pure _userspace_ implementation of 32-bit compatibility for an otherwise pure 64-bit kernel.

<comment> vezzy-fnord _In a way, advocates of pure static linking are like anti-vaccine advocates._

Holy fuck, this is one of the most absurd exaggerations I've read in some time. I'm assuming you have a very strong and convincing reason to justify this statement.

 _Here 's a killer application of dynamic linking: NSS._

...

Really? NSS?

OK, so static NIS/YP may not be good at all, but that doesn't change that NSS and the nscd (name server cache daemon) have been amongst the buggiest and most brittle components in glibc, on top of glibc's sequential DNS resolution. It's so bad the Busybox developers had to write a single-threaded fork-based alternative just to deal with its inherent resource exhaustion issues (see the comments in this software: [http://busybox.net/~vda/unscd/](http://busybox.net/~vda/unscd/)).

And this: [http://www.nico.schottelius.org/blog/nscd- bugs/](http://www.nico.schottelius.org/blog/nscd-bugs/)

musl libc (ever growing in popularity in embedded and other fields for its standards compliance, robustness and implementation simplicity) doesn't support NSS and doesn't intend to, instead researching several alternative methods, the third one probably optimal: [http://wiki.musl- libc.org/wiki/Open_Issues#NIS.2FLDAP.2Fothe...](http://wiki.musl- libc.org/wiki/Open_Issues#NIS.2FLDAP.2Fother_user_databases)

And if you think this problem hasn't been solved so much more elegantly before, look into the details of Plan 9's ndb and factotum.

 _PAM is another good example._

bsd_auth is much better: [http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD- current/man3/...](http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD- current/man3/auth_approval.3?query=authenticate&sec=3)

<comment><USER3> > musl libc

I'm conflicted on Musl. I like the idea. I like the focus on robustness. I like Felker's style [1]. But for God's sake, Musl refuses to implement basic operating system primitives we've had for decades, like dlclose(3), all because the author, from his necessarily limited perspective, just can't understand why someone would want those facilities.

I'm sticking with glibc until the Musl people decide they want robustness _and_ feature parity.

By the way: PAM isn't "add in the bloat of additional backends". Dynamic loading of backends is the _opposite_ of bloat.

There's absolutely no reason Felker can't greatly improve libc robustness _and_ maintain at least source-level compatibility with a libc we've used for decades and that has many valuable features. Musl's sin is being gratuitously opinionated.

[1] [http://ewontfix.com/](http://ewontfix.com/), good for the most part. Wrong about 32-bit PIC being terrible. (What we really need is -fvisibility=hidden and -Bsymbolic by default.)

<comment><USER4> Source level compatibility with glibc is a terrible idea, it is a huge and bloated system, which quite clearly has made a lot of mistakes - which it is gradually fixing some of. C is no longer a general purpose programming language, and needs to be used more carefully when it is needed, and Musl helps you do that. Use a safe language if you want more stuff.

<comment><USER3> Puritanism in system libraries helps nobody. What you consider "bloated" others consider essential. If you're responsible for the lowest-level userspace library on a system, one that every component uses to talk to every other component, you have no business foisting your opinion on the world.

glibc, buggy and hacky as it is, doesn't try to impose its opinions on you.

<comment><USER4> Source level compatibility means replicating the bugs, which is pointless, just use glibc if you want it to be identical. Musl is basically like the libcs of the BSDs, ie less stuff it just made up, just the standards compliant stuff. Add the other stuff to your own library.

<comment><USER3> > Source level compatibility means replicating the bug

No it doesn't. For example: we can fix the the glibc pthread_mutex_lock bug without affecting ABI or source compatibility.

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> When Heartbleed hit, to fix your dynamically linked system, you replaced openssl and called it a day.

With your statically linked system, you replace every single binary on your system that linked to openssl during build (if you can figure that out).

<comment><USER6> The security argument goes both ways though. Lets say I update my music player and through some complicated chain it ends up also updating my system libc and openssl, and lets say one of those libraries accidentally introduced a vulnerability. If those libraries were statically linked, it's just my music player that's in trouble. With dynamic linking, instead of my music player having a security issue (not a big deal, probably), my entire system is now vulnerable including my web browser, even though I didn't update my web browser. (much bigger deal)

If you consider biological ecosystems, diverse genetics are almost always more robust against infection. You could potentially make that analogy about the ecosystem of computers as well.

<comment><USER7> In your situation, static building would also result in rebuilding libc and openssl, and from there your web browser, as most package management systems, and package managers, would make sure that the updated libc were relinked in to as many programs as possible as soon as they were available.

</comment></comment></comment><comment><USER8> Most systems are actually a mix of statically linked and dynamically linked executables.

For example, the systems I compile can all boot and function without the need for a dynamic linker. These static binaries are on a root filesystem that is in memory, not disk. Hence I can boot from removable media and then remove the media so the slot is open.

If I need larger, complex, dynamically linked programs I mount a filesystem that has them along with the dynamic linker and then chroot. I build these filesystems separately, usually storing them on removable media.

</comment><comment><USER9> I think it all depends on the granularity of dependencies and how you cope with multiple versions. If every app have their own version of a lib, you get problems with security updates and bloat. If you force a single version of each lib, you get a really slow testing and itegration cycle. If you choose something in between those extremes I think dynamic linking does have merit.

</comment><comment><USER10> I'll take an "insecure" ldd over having to recompile every single binary when there is a glibc/libjpeg/libpng/libwhatever security fix, any day.

I'm sure the ldd problem could be solved by writing a slightly smarter "ldd" that doesn't actually execute the ELF binary, as well.

<comment><USER11> Dynamic linking ends up being a wash or a net negative for # of recompiles when you consider that developers of important dynamic libraries break binary compatibility all the time on non-security-related changes.

<comment><USER3> > developers of important dynamic libraries like glibc break binary > compatibility all the time on non-security-related changes.

You couldn't have picked a worse example. glibc goes to great lengths to maintain backward compatibility. glibc is practically the reason ELF symbol versions exist.

[https://www.redhat.com/f/summitfiles/presentation/June2/Deve...](https://www.redhat.com/f/summitfiles/presentation/June2/Developer%20Tools/Drepper_Writing%20Shared%20Libraries.pdf)

<comment><USER12> And yet in practice, the number of steam games on linux that I can't run because the developers compiled them against a newer version of libc than I have is in the dozens.

<comment><USER3> Compatibility works the other way around. You can run binaries compiled against an _older_ libc on a _newer_ glibc, but there's no reason to expect that binaries compiled against a newer libc should be able to run against older versions of libc. If you demand that, you demand software stasis.

<comment><USER11> It's a _libc_ implementation for christ's sake. If ever a library _should_ remain in stasis, it would be a libc implementation.

<comment><USER13> There's a clue in your formulation that guides you to the right answer - <USER2> useful library should ever remain in stasis.

Or as it's generally said, the only software that never changes is software no one uses.

<comment><USER11> That's just, like, your opinion, man; tell it to the retrocomputing folks.

IMHO, the reluctance to "finish" a project and the stigma against something being "in maintenance mode" is responsible for most of the bloat and messes in modern software. Perhaps it's related to "CADT": [http://www.jwz.org/doc/cadt.html](http://www.jwz.org/doc/cadt.html)

It's also, ironically, the cause of numerous security flaws. Maybe we wouldn't have to worry so much about security flaws in our libraries if we weren't constantly tearing them down and remaking them.

Regardless, in the case of libc, I genuinely cannot think of a reason to break binary compatibility, backwards _or_ forward. The POSIX standard is set in stone, the interfaces are stable, the data types are opaque, etc. Of course you can create new implementations of memcpy etc that take advantage of new instruction sets, but I don't see why you need to break binary compatibility to do so.

<comment><USER14> > in the case of libc, I genuinely cannot think of a reason to break binary > compatibility, backwards or forward

Well, since you can't think of one, there must not be one.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER15> What exactly is being proposed here? Would every app statically link against some version of UIKit or the Android Java libs? How would the app run on different versions of the OS? How do you maintain forward compatibility?

It's mystifying that the debate here centers around performance or RAM usage. The benefit of dynamic linking is architectural: it enables different components to evolve independently.

I understand Plan 9 has an unusual approach, where things that are traditionally library calls (getMousePosition) are instead filesystem operations (read /dev/mouse). But this seems terribly rigid to me: we've just replaced structs with a static byte stream format. How do you evolve this?

<comment><USER16> The interesting bit is that Limbo on Inferno, Plan9's successor, only uses dynamic linking.

</comment></comment><comment><USER17> No, dynamic linking isn't "harmful."

</comment><comment><USER18> Take it to its logical conclusion - statically compile the kernel in each executable. Build the task scheduler in the CPU. Make every program run on pure bare metal.

<comment><USER19> [http://www.techrepublic.com/article/unikernels-offer-a- strip...](http://www.techrepublic.com/article/unikernels-offer-a-stripped- down-alternative-to-linux/)

[https://news.ycombinator.com/item?id=8606968](https://news.ycombinator.com/item?id=8606968)</comment></comment>


Ask HN: How does a JIT-compiled function communicate with a bytecode function? - <AUTHOR>

consider we have two functions compiled into bytecode first. if the first function is JIT-compiled, how does the second function call the x86 code of the first function ? how does it extract the results ?<p>basically how does this leap into the "x86 land" and from there back into the "bytecode land" occur ? also how is the "x86 land" stored and accessed ? <comment><USER2> By using function pointers!

The concept of "function" or "procedure" doesn't really exist concretely. Sure, some processors provide opcodes to view the accessible memory as a "stack", to accommodate some specific, if popular, block structure implementations, namely C and Fortran's.

The most critical part is mapping the language's concept of "function", an executable body of code where all the referenced variables have specific instances/allocation, reachable at the time of its invocation, _with_ the processor's, or most likely, C's view of how the processor underneath haphazardly implements block structure.

If your JIT compiler can seal a body of code and guarantee to implement the environment (the mapping from address/name to value) in away accessible to the dumb processor, then you will win.

This requires that the compiler hacker implement code to convert values from HLL types, to lower level ones. And convert the implementation of closures and environment, to something more palatable to the lower implementations.

Below is a simple "JIT" interpreter. It uses pre-compiled C code in a shared library, along with its own "natively" compiled operations. A real JIT compiler following this protocol would just make sure it's able to find the functions by any other means, other than compile-time name lookup, as this one does. It can maintain its own explicit symbol table, hashing a function name along with type signature, and dispatch on it at run time.

In the code below, ADD and SUB are the usual addition and subtraction operations, precompiled "for speed", while DIV and MUL are "interpreted" byte- codes. In the current design, both types of functions (native and bytecode) are mapped into a single namespace; typically, there are predicates to distinguish between the two. In most Common Lisp implementations, the two co- exist side by side, and you can call and interpreted function just as you can call a native one.

This is just a quick 1-hour hack, let me know if it's not clear.

Here is the runtime library.

      /* --- compiled-functions.c   */

      /* Compile with:  gcc -fPIC -g -c compiled-functions.c  */

      extern int add (int x, int y);
      extern int sub (int x, int y);

      int add (int x, int y) {
        return x + y;
      }

      int sub (int x, int y) {
        return x - y;
      }

Here is the JIT compiler:

      /*  jit-test.c   main driver */
      /* compile as gcc jit-test.c compiled-functions.o */
      #include <stdio.h>
      #include <stdlib.h>
      #include <string.h>

      /* library functions */
      extern int add (int x, int y);
      extern int sub (int x, int y);

      /* compiled opcode values */
      #define ADD 0
      #define SUB 1
      #define MUL 2
      #define DIV 3

      /* string opcode names */
      char *opcode_names[] = {"ADD", "SUB", "MUL", "DIV"};

      #define OPCODE_COUNT 4

      /* the values returned by the parser */
      struct input {
        int op, arg1, arg2;
      };

      /* rudimentary parser, converts strings to internal structure. */
      struct input *parse_input (char *op_name, char *arg1, char *arg2) {
        int i, opcode = -1;
        struct input *parsed_input = (struct input *) malloc (sizeof (struct input));

        /* convert opcode name to opcode value */
        for (i = 0; i < OPCODE_COUNT; ++i) {
          if (strcmp (op_name, opcode_names[i]) == 0)
            opcode = i;
        }

        if (opcode >= 0) {
          parsed_input->op   = opcode;
          parsed_input->arg1 = atoi(arg1);
          parsed_input->arg2 = atoi(arg2);
        }
        return parsed_input;

      }

      int main (int argc, char **argv) {
        int result;

        /* this points to an unknow jit function linked against us */
        int (* jit_fn) (int, int);    /* <</comment><comment>--- Function pointer! */

        struct input  *input = parse_input ("DIV", "4", "3");

        if(input) {
          printf ("%s %d %d ==> ", opcode_names[input->op], input->arg1, input->arg2);
        }

        switch (input->op) {
        case ADD:
          jit_fn = &add;
          result = jit_fn (input->arg1, input->arg2);
          break;
        case SUB:
          jit_fn = ⊂
          result = jit_fn (input->arg1, input->arg2);
          break;
        case MUL:
          result = (input->arg1 * input->arg2);
          break;
        case DIV:
          result = (input->arg1 / input->arg2);
          break;
        }

        printf ("%d\n", result);

        return 0;

      }

<comment><USER2> Also, you might benefit from reading this paper:

[http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.25.2...](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.25.238)

The original block-structure implementation in Algol is still accessible in Pascal and Oberon literature.

</comment></comment><comment><USER3> Do you have a specific virtual machine in mind, or are you asking for general approaches to the problem? A simple approach is to only JIT function bodies, and deal with all function calls at the VM level. Then all function calls within JITted functions call back into the VM, e.g. by being compiled to some variety of _vm_do_function_call(function, params). Actual VMs include a bunch of other optimizations, of course; which ones are possible depends partly on the language's semantics (e.g. can we assume functions are statically defined, or can they be redefined at runtime?).

</comment><comment><USER4> There is no leap back into an AST, JIT works on hotpaths and applies optimizations to an AST (bytecode) directly to machine code, there is no reinterpretation of machine code. 'Communication' if you will is simple - everything has an address.

</comment><comment><USER5> Dependent method calls are compiled into the single image version of x86 code, these dependencies are resolved and compiled the first time the call is found in the bytecode.</comment>


Show HN: N-Body simulation using WebAssembly - <AUTHOR> https://novoselrok.github.io/nbody-wasm/ <comment><USER2> Uses the Barnes-Hut algorithm to avoid O( n^2 ) force calculations in favor of O(n log n). Simple nearest neighbor tracking cannot be use in gravity simulations as gravity has an effective infinite ranged potential. [https://en.wikipedia.org/wiki/Barnes%E2%80%93Hut_simulation](https://en.wikipedia.org/wiki/Barnes%E2%80%93Hut_simulation)

</comment><comment><USER3> I just get a blank, dark screen. Recent version of Firefox.

<comment><AUTHOR> Hey, author here. Any errors in the console? (Sometimes the Emscripten Module fails to load)

<comment><USER4> I got that too when i opened the tab in background. Chrome 70, console says:

 _Active resource loading counts reached to a per-frame limit while the tab is in background. Network requests will be delayed until a previous loading finishes, or the tab is foregrounded. See[https://www.chromestatus.com/feature/5527160148197376](https://www.chromestatus.com/feature/5527160148197376) for more details_

Works fine after page reload.

<comment><AUTHOR> Ok, I'll test it on Chrome 70. Emscripten Module loading is not still as reliable as I would like :/</comment></comment></comment></comment>


Seagate Announces 16TB HDD - <AUTHOR> https://nexthive.com/seagate-biggest-hdd-16-tb/ <comment><USER3> From the article: "In a world where SSDs have become more and more common, one might ask what the point of even developing newer and bigger HDDs is."

Solid state drives still cost about 10 times more per gigabyte. I'm just a web developer, but a spinning-disk drive looks 100 times more complicated to me to manufacture. I know at the beginning that hard drives would be cheaper, simply because SSDs were newer, and there was R & D to amortize. But by now I would have thought that SSDs might have matched hard drives or even undercut them. Not only has it been over a decade, but hard drives themselves are using new R & D too. For example this article mentions that this new hard drive uses a new technique, called HAMR. Can the old factory machinery from 2005 be used to manufacture these new HAMR disks?

<comment><USER4> >Solid state drives still cost about 10 times more per gigabyte

I think you should go shopping, because the time where SSDs cost 10x as much as an HDD is over.

When a WD Blue 1TB SATA 6 Gb/s 7200 RPM 64MB is $46 on Amazon and a Crucial MX500 1TB 3D NAND SATA 2.5 Inch is $134.

Unless you're comparing the cheapest 1TB HDD you can find to a Samsung 970 pro 1TB or something... which really isn't a reasonable comparison imo.

<comment><USER5> >When a WD Blue 1TB SATA 6 Gb/s 7200 RPM 64MB is $46 on Amazon and a Crucial MX500 1TB 3D NAND SATA 2.5 Inch is $134.

those are cherrypicked numbers because 1TB is at the _very_ low end for HDD sizes. compare with a more typical HDD size (3TB or 4TB) and you'll see the differences become more obvious.

<comment><USER6> I don't know about you, but 500-1000 GB is more than enough for my SSD needs, so the comparison at 1TB is much more relevant to me than at 4TB.

<comment><USER7> Wasn't the parent discussion about a new high-capacity HDD and the per- capacity cost of HDD vs SSD? I don't think that people satisfied with under 1TB are relevant to that topic.

In that market, people are comparing a pile of disks in an array to a pile of solid state storage needed to replace its capacity. The bulk price of storage inverts when it is cheaper to store hundreds of TB on huge SSDs rather than on huge HDDs. That would be the death of HDD, since nobody really wants spinning disks in their datacenter.

<comment><USER8> > a pile of disks in an array to a pile of solid state storage

It doesn't even need to be a pile of disks. Even for a PC the differences are big.

My computer has a 1 TB SSD which is a decent size for an SSD. It's still a bit tight for me, so I complement with a spinny rust NAS. If I had an HDD instead it would have been like 8 TB and I wouldn't necessarily need the NAS. I think that's what GP is hinting at when he says "me SSD needs". I think he's also complementing with HDD somewhere for bulk storage (be it external drive, NAS or even cloud storage)

<comment><USER6> If I'm GP, yeah; I use 250-500GB SSDs as primary disk and have a small NAS with a RAID1 of two 5 TB disks for slow archival storage. For that slow archive, though, I don't care what factor SSD price is to HDD price (as long as it's still >1.0).

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> I don't quite recall when it happened, but at some point my reaction to announcements like this changed from, "Yay, more space!" to "Good, the size I need will be even cheaper when I upgrade 2-3 years from now."

<comment><USER10> I record and produce my own music as a hobby. Each project can easily reach gigabytes in size, and my biggest projects are eight gigabytes or so (eight drum mics, guitar, bass, vocals, and synthesizers, all uncompressed 24/96 audio, often with a number of different takes). I go through HDDs very quickly. I know that for development and most uses bigger HDDs are unnecessary, but for media production these improvements are really valuable.

<comment><USER11> Why not use FLAC? It gets 50-70% compression and is lossless.

<comment><USER12> Offhand I actually don't know of too many DAW programs that can record the individual tracks in FLAC format. Reaper does. Checking the Wiki ([https://en.wikipedia.org/wiki/List_of_hardware_and_software_...](https://en.wikipedia.org/wiki/List_of_hardware_and_software_that_supports_FLAC)), it looks like Cakewalk Sonar does as well. But that leaves a lot of major players (Protools, Logic, Cubase, Ableton Live, FL Studio, etc.) that cannot.

It's not something I personally would do because (as others imply) IMHO the CPU trade-off is not worth it. Usually I use a blend of plug-ins (which take up no space) and recorded hardware, consequently the amount per track ends up being more like 100-600MB and not gigabytes. This is rather manageable.

However, I will note that sample based software synthesizers and sample packs can be _huge_ these days, even despite in many cases lossless compressing being applied to the sample library. Omnisphere 2 for instance comes with a 60GB+ sample library (and that's before you add the add-ons like Moog Tribute). At the current extreme end, orchestral sample company Spitfire offers a string library ([https://www.spitfireaudio.com/shop/a-z/hans-zimmer- strings/](https://www.spitfireaudio.com/shop/a-z/hans-zimmer-strings/)) that is 183GB in size and a sampled piano that's a whopping 211GB _compressed_ ([https://www.spitfireaudio.com/shop/a-z/hans-zimmer- piano/](https://www.spitfireaudio.com/shop/a-z/hans-zimmer-piano/)).

<comment><USER10> And that's why when I use Omnisphere (which I love, BTW), I send my projects to my best friend's computer; there's no way in hell I could fit that on my computer's internal SSD.

Even the relatively small software instruments (I've been really into Soniccoture's Glass/Works, for instance) use up like 8 GB apiece.

</comment></comment></comment></comment></comment><comment><USER13> As a professional storage-system developer, my reaction to every announcement of larger disks is something close to horror. Capacity keeps increasing while performance remains darn near constant, so the gap between the two keeps getting wider. Yes, we can use flash to absorb the I/O demand, treat disk more like tape, yadda yadda, but all of that takes significant effort. You do want your storage system to be _correct_ despite the greater complexity of heterogeneous hardware, don't you?

Looking on the bright side: infinite job security. Yay?

<comment><USER14> The part that makes me recoil in horror is the potential damage wrought by one failure - at what point does the rebuild time for your RAID Array start to collide with the MTBF for the drives? ;-)

<comment><USER13> This is why the really big storage systems use erasure coding. At scale, RAID-5 and even RAID-6 are vulnerable to these kinds of overlapping failures. Not theory; seen it happen.

</comment></comment></comment><comment><USER15> I know that big disks are useful for a lot of people/applications. However, I topped out at 500GB about 10 years ago. That gives me plenty of headroom. Since then, I have gone from a 500GB HDD to a SSD, but my space usage hasn't really gone up. The largest disk I own is a 1TB disk in my Time Capsule (which I could probably stand to upgrade, but I don't feel like taking it apart).

I think this is probably because I stream everything, and don't download much content. Even most of my applications are web-based these days. Granted, I'm sure that a lot of storage in the cloud is required to service me...

<comment><USER16> you also do not game a lot. if you look at [https://www.game- debate.com/games/index.php?g_id=9339&game=R...](https://www.game- debate.com/games/index.php?g_id=9339&game=Red%20Dead%20Redemption%202) you will see that you need 100gb for just that single game.

I mean my own music library which is rather small has 250gb...

<comment><USER17> I game a lot, but none of the games I play even approach half that size. not all games are bloated, disk space eating messes.

<comment><USER16> it actually depends. you can either have a game with extremly fast loading, which means that assets won't be compressed a lot or you can compress them and go for file size, you can even reduce the quality. it always depends on your target audience.

but still modern games for pc/mac are around ~10gb per game which would limit your installed game base by a lot.

(also working with docker (pc, mac not linux) or vms in general will quickly eat your 500gb hard disk)

<comment><USER6> I have a 200 GB or so SSD in my gaming computer. Yeah, I basically just install one AAA game at a time.

</comment></comment></comment></comment></comment><comment><USER18> For general file storage spinning disk HDD are still better cost, but there is a decreasing need for general file storage for the average person. For most people one terabyte is more than enough and it may as well be an SSD that is also their boot drive.

Personally I'd love to replace my multiple 2TB drives with 8TB drives if the price comes down a bit.

<comment><USER19> Games are the only reason most people will need more.

Most new AAA games are approaching 100GB each. And on the extreme end Gears of War 4 for example takes up over 250 GB.

<comment><USER20> Datacenters as well. Peoples data storage needs have gone up only now we put all the data on someone elses computer.

</comment></comment></comment><comment><USER21> As a person who doesn't know much about HDD technology, I always wonder: what were the advances that allowed this particular bump to happen?

<comment><USER22> From TFA:

>The Exos 16 TB hard drive using HAMR technology is now the world's biggest HDD in terms of capacity overtaking the 14 TB Barracuda Pro.

>HAMR, which is the acronym for heat-assisted magnetic recording, to be precise. This replaces the regular PMR, perpendicular magnetic recording, found in most HDDs. To the average consumer, this doesn't mean much at all. However, Seagate believes that HAMR is the key to making significantly larger capacities readily available shortly.

<comment><USER23> What is the theoretical realistic limits for HDD storage?

From the sounds of it this 16TB HAMR-enabled drive is just the first of many we will see year-to-year to continue HDD sales.

<comment><USER24> Seagate says they'll get up to 100 TB with HAMR. Curious at what point it's no longer financially reasonable to keep researching ways to increase density.

<comment><USER25> We produce and process more data in a few days than we did in a year even as recently as a decade ago (and, by "we", I mostly mean companies that want to store every detail of every human life for analysis in order to maximize profits). So far, the increasing need for storage has kept pace with the ability to store it.

</comment></comment></comment></comment></comment><comment><USER20> I just checked out the price of that 14tb seagate drive and you can get 2 of their 10tb drives for the same cost of 1 14tb drive. Who is buying this 14tb drive right now if it costs that much?

<comment><USER26> People who need mass storage and either have limited space or want to maximize the space they do have. A thirty-two drive rack full of 14TB drives is 448TB of storage space instead of 320TB (thirty-two 10TB drives). And if you only need 320TB of storage you only need a rack that holds twenty-three drives - <USER2> you might downsize to a twenty-four drive rack.

I updated a lot of my 4/6TB drives to 8TB drives for this reason: too much physical space was being taken up on my desk by having so many 4TB drives.

</comment></comment><comment><USER27> The HDD industry, Basically Seagate, Toshiba, and WD together are milking it for as long as possible. The Data created per year, or even growth per year far out-weight the growth of HDD capacity.

At some point later, surely NAND will offer more storage in 4U than a 40U Rack. Lower Energy, higher speed.

I have been asking this a lot, at what price point, will the TCO of NAND, where it offer superior speed, higher density per Rack and its capacity cross and it makes sense to store them all in NAND.

</comment><comment><USER28> The link doesn't work for me

[https://web.archive.org/web/20181205181735/https://nexthive....](https://web.archive.org/web/20181205181735/https://nexthive.com/seagate- biggest-hdd-16-tb/)

</comment><comment><USER29> The Library of Congress is only 10 terabytes.

<comment><USER30> Disagree, I sold them tape cartridges by the PB

</comment></comment><comment><USER31> If it's anything like their 8tb offerings it's unreliable rubbish.

<comment><USER32> Statistics with representative sample size disagree. [https://www.backblaze.com/blog/2018-hard-drive-failure- rates...](https://www.backblaze.com/blog/2018-hard-drive-failure-rates/)

<comment><USER31> I don't know if you're looking at the same list I am, because the large Seagate disks on here have some of the highest failure rates outside WDC.

<comment><USER33> The Seagate 8TB has an AFR of around 1%, this is not only really good, but also makes any anecdotal evidence from a consumer quite useless.

</comment></comment></comment></comment><comment><USER34> when are we getting 8 tb ssds. its been awhile. ssds should have caught up to spinning rust by now in capacity

<comment><USER35> Enterprise 15 TB SSDs are already available. It would be easy to create 8 TB consumer SSDs but they don't see demand. There's tons of empty space inside 2.5" SSDs: [https://www.anandtech.com/Gallery/Album/6783#3](https://www.anandtech.com/Gallery/Album/6783#3)

<comment><USER36> Right, but a huge part of the consumer SSD space has moved to m.2, where there isn't a lot of space to just pack in another couple dies. While conceivably you might be able to make a 8TB drive in m.2 its going to be easier in 2.5", and people with the money to buy that large of a SSD probably don't want it on SATA. Which is why, as you point out they exist, here is a 8TB u.2

[https://www.newegg.com/Product/Product.aspx?Item=1E4-006U-00...](https://www.newegg.com/Product/Product.aspx?Item=1E4-006U-00009)

for only $2600.

</comment></comment></comment><comment><USER37> Anyone have an opinion about Seagate? I had one of their HDD and it died on me within a year.

Not sure if this was a 'one-off' or other people had issues with their quality.

Similar issue with samsung on SSD.

<comment><USER38> I've had a drive from every major manufacturer fail at some point. Brands don't matter, backups do. Buy hard drives in pairs and put an item on your calendar to pop the other one in once a month, sync it, and put it back in a fire-resistant safe.

<comment><USER39> > Buy hard drives in pairs and put an item on your calendar to pop the other > one in once a month, sync it, and put it back in a fire-resistant safe.

Alternatively, buy one hard drive and keep a copy in Amazon Glacier Deep Archive for $1/TB-month.

<comment><USER30> And then wait weeks and pay real money to pull it all back if you have 8TB :)

<comment><USER8> Yeah it's something like $700 to retrieve 8 TB

<comment><USER30> Meanwhile for highly redundant tape cartridges, you can store that amount for maybe $100-200, but the device to read it back is about 10x more expensive used unless I am just struggling the find the real deals :/</comment></comment></comment></comment></comment></comment>


What happened to Google's effort to scan university library books? (2017) - <AUTHOR> https://www.edsurge.com/news/2017-08-10-what-happened-to-google-s-effort-to-scan-millions-of-university-library-books <comment><USER2> FWIW: you have to be affiliated with an academic institution to get access to most full texts on Hathi. However, one such academic institution is the Library of Congress. I would think anyone reading Hacker News in the US would qualify as a researcher at the Library of Congress; the catch (admittedly a big one) is that you have to present yourself in person to get a library card ("reader registration card" in LoC parlance) -- meaning that unless you live in the immediate DC area you have to travel there. And they're only valid for two years.

Link to how to get a library card at the LoC:

[https://www.loc.gov/rr/readerregistration.html](https://www.loc.gov/rr/readerregistration.html)

<comment><USER3> I've always wanted to visit the Library of Congress and now I have even more of a reason to do so. Any other advice or suggestions for my visit?

<comment><USER4> It may be the only way to find some of thing things preserved via citizen archives. I enjoyed the talk Ian MacKaye gave there about archiving the live performances of Minor Threat & Fugazi. Here [1] is a post on the subject.

[1] [https://blogs.loc.gov/thesignal/2013/05/ian-mackaye-and- citi...](https://blogs.loc.gov/thesignal/2013/05/ian-mackaye-and-citizen- archiving/)

<comment><USER5> Open only to citizens?

<comment><USER4> Hmm, that seems unlikely but I honestly don't know. Possibly free for US citizens but a small fee for visitors? I can't imagine it would be closed to visitors, though you should be able to contact them to find out.

The talk I so enjoyed is on YouTube with many others: [https://youtu.be/AvqtY_7Q7hI?list=PLEA69BE43AA9F7E68](https://youtu.be/AvqtY_7Q7hI?list=PLEA69BE43AA9F7E68) (video 30 is MacKaye)

</comment></comment></comment></comment></comment><comment><USER6> " _A Data Capsule is a secure, virtual computer that allows what's known as "non-consumptive" research, meaning that a scholar can do computational analysis of texts without downloading or reading them. The process respects copyright while enabling work based on copyrighted materials._ "

And that is completely ridiculous; a technical solution to an invented problem.

<comment><USER7> > And that is completely ridiculous; a technical solution to an invented > problem.

Congratulations, you've just described most software projects by libraries and archives.

It's funny, because back in the 60s-80s, libraries were leaders in building shared data systems and networked infrastructure. The history of OCLC describes this well.

But once the web came around, they had an identity crisis, were unable to react to technology trends, and largely got conned into predatory and restrictive arrangements by service providers (Elsevier, ProQuest, etc). The same thing actually happened in the past with microfiche and led to libraries destroying huge, valuable portions of their collections, that could have been better preserved with the advent of scanning technologies.

<comment><USER8> Most librarians would agree with you that the current situation is terrible, and if you look at library literature they've generally felt that way from the beginning. They are highly constrained by copyright law.

re: microfiche, it lasts much longer than digital scans -- if kept in good conditions it has a usable half-life on the scale of >100 years, while digital files need much more active maintenance to both prevent bit-rot and to ensure the file-type is still readable (eg countless file formats have been abandoned and are only currently accessible via emulators of older machines).

And if you want to bring the cloud into this, most libraries don't have the funding to bring in the technical know-how to manage a private S3 instance only accessible from that building.

<comment><USER9> > prevent bit-rot

What I do is regularly copy my files forward onto newer media. I started this back in the 1970s, and it is the only reason I still have a copy of the FORTRAN-10 source code of Empire:

[https://github.com/DigitalMars/Empire-for- PDP-10](https://github.com/DigitalMars/Empire-for-PDP-10)

All the other stuff I wrote at the time is lost because I stored it on a magtape, and the Caltech magtape drive had drifted so far out of spec the tapes could only be read on that machine which was lost.

I managed to preserve that by copying it over a serial line to a PDP-11 and storing it on a PDP-11 floppy. I later was able to save my 11 code by copying it over a much later serial line to an IBM PC, to put on 5.25 disks. As time went by, the files migrated to zip drives, then CD-ROMs, then a long sequence of hard drives (my older hard drives can't be read with modern IDE interfaces, even if the connector fits, I have no idea why).

I remember reading boxes of 5.25 floppies and burning them onto CD-ROMs, a long and tedious process. But now, nothing will read 5.25 floppies any more, but copying a year old hard drive to a new one is a simple process, especially since the new drives are usually much larger than the older ones.

Hence I have most of the stuff I worked on since the early 1980s. The old Zortech bulletin board stuff is gone, though, even though I still have the hard drive for it. Nothing can read that old drive. Not that there's anything particularly interesting on it, but I enjoyed running the BBS for many years.

</comment></comment></comment></comment><comment><USER10> Speaking from experience, the Hathi metadata is a hot mess. If I had to guess, I would say that it was largely sourced from whatever institutions catalog without a great deal of regularization.

There are levels of dirtiness in data. Hathi's metadata reached a level of dirtiness where the filthier fields could actually be mined for _other_ data, given enough labor. While I was able to extract some benefits from this at some cost, it was no substitute for having things done well.

Many of the scans are described as "unusable" by the faculty and having looked at many of their complaints I would agree.

<comment><USER7> I worked on this data in a former life and it is indeed a mess and derived from institutional records. But it's not any more a mess than most other forms of library-derived metadata because MARC and related formats are essentially pseudo-schemas that are _at best_ haphazardly applied.

From what I recall, there were attempts at regularization by the big universities involved with project, but they're both not very good at tech projects and especially not good at finding and keeping the high-level talent that would be needed for something of this scale.

But if you think bikeshedding is bad in tech, take a look into some disputes between library cataloguers over their arcana. Much of what they produce is valuable at level of a trained cataloguer who can navigate its inconsistencies and vagaries, but rarely so at the machine level.

</comment></comment><comment><USER11> I used to work with HathiTrust and attended a few workshops; my experience was that the notion of a "data capsule" that ensures "non-consumptive use" is quite onerous, in practice, and that getting that access can acquire prior approval and supervision of research activities:

[https://wiki.htrc.illinois.edu/display/COM/HTRC+Data+Capsule...](https://wiki.htrc.illinois.edu/display/COM/HTRC+Data+Capsule+Environment)

I think it's actually a clever technical solution to a hideous legal problem, and it makes a lot of research possible that would be totally illegal otherwise, but it 100% gets in the way of totally legitimate research as well.

<comment><USER6> Does it make you feel any better that most of the texts over a decade old are permanently out of print and thus valueless, and that the hideous legal problem only applies to a tiny fraction of the works in question?

<comment><USER12> Not to mention how much of Amazon's rainforest have been/are being razed to ground to make way for pulp farming for the dead trees. It's scary and stupid both.

<comment><USER13> Are you being sarcastic? Amazon rainforest wood is not used for paper, the trees are cut down for farmland, not for the wood.

There are far better sources of wood, but land is more scarce.

<comment><USER12> Who said anything about using old trees for book pulp? Pulp farming requires arable land which is taken away from rainforests.

</comment></comment></comment></comment></comment><comment><USER14> I was an engineer in Google Books in 2011 and got an impression that this effort turned out to be not as useful as expected. This took away energy from the legal fight. Having access to every printed book seemed amazing in principle, but it didn't translate to amazing demand for the service. My theory is that it didn't lower the barriers to knowledge much -- there's not that much useful information left in offline-only storage.

<comment><USER15> I think there's a lot of information sleeping in depositories only on paper. The thing is people are busy with the urgent stuff and stay away from the important if they can't get something immediate from it.

I've used extensively Google Books for research of 17th century Mathematics (not just me, also the whole department were I was at that point). I couldn't have afforded to visit all the libraries to look up the originals but thanks to this repository I could download several copies of obscure works, which gives your very interesting insights: sometimes you hit a personal copy of a previous researcher, or there's an ex-libris from an old institution that you know your author was connected to, or you just discover something no one had noticed before simply with a clever string search. You can even backtrack to the forgotten true primary source of a mistakenly repeated fact for instance. It's been a blessing, honestly.

If you are into whatever not too transited alley of history, you realise soon that your (former?) company has produced a treasure trove for future generations that just shouldn't be shut down ever: there are many books there not digitized anywhere else. So thank you very, very much for your work there. It definitely belongs to the important.

</comment></comment><comment><USER16> As an academic library user, it would be especially helpful if one could read obscure/old titles stored offsite in digital form; currently one has to have it shipped to campus, pick it up, and maybe realize it doesn't actually contain the needed info — a waste of energy on multiple parties.

<comment><USER17> I find archive.org fairly useful for a lot of older books. If they are past the copyright date, then many books can be found and downloaded as PDFs. I've found lots of old Language & Literature books from 80-140 years ago. :-)

</comment></comment><comment><USER18> See also: [https://www.theatlantic.com/technology/archive/2017/04/the-t...](https://www.theatlantic.com/technology/archive/2017/04/the- tragedy-of-google-books/523320/)

</comment><comment><USER19> Another story, with a lot more detail: [https://www.wired.com/2017/04/how- google-book-search-got-los...](https://www.wired.com/2017/04/how-google-book- search-got-lost/)

</comment><comment><USER20> > "Somewhere at Google there is a database containing 25 million books and > nobody is allowed to read them."

That is the key result: the greatest library in human history is locked shut, primarily because it was perceived as a threat to the legacy business model of paper book publishing (and royalties based on sales of paper books.)

Restricting libraries to "non-consumptive use" betrays the fundamental purpose of a library: to enable people to read the books in the collection; it makes little sense with physical books and even less with digital texts which don't wear out.

</comment><comment><USER21> I heard of HathiTrust Research Center but didn't know almost all the books were scanned by Google.

Google was nice to want to share. But I think the original and ongoing concern was improved image recognition and NLP. Ie. Machine Learning the whole world knowledge.

<comment><USER22> > Google was nice to want to share.

Niceness had nothing to do with it. From the article, "As part of the deal, Google's partner libraries made sure they got to keep digital copies of their scanned works for research and preservation use."

</comment></comment><comment><USER23> It sucks that we could have basically had a virtual Library of Alexandria thanks to Google, but because of book publishers lobbying in order to try and conserve their monopolies, it's now locked away to the general public.

</comment><comment><USER24> Libgen happened.

</comment><comment><USER25> What happened?

tldr; lawyers...

<comment><USER26> like it was a surprise</comment></comment>


The Curse of the Ship of Gold - <AUTHOR> https://narratively.com/the-curse-of-the-ship-of-gold/ <comment><USER2> So is the moral of the story to melt down any gold you find and disguise its origin? I found it sad that the insurance companies were trying to make a claim from the 19th century as well. That seems pretty out there.

<comment><USER3> Makes sense to me.

Another moral: Don't hide from the Feds in the US.

Edit: And that reminds me. A decade or so ago, I chatted with people who claimed to be doing the _Cryptomonicon_ thing. That is, pouring molten gold into a cave on their property, and issuing gold-backed electronic currency. They were likely crazy, or putting me on.

</comment></comment><comment><USER4> This is only somewhat related, but does someone know a resource that describes what happened to the continental financial/currency systems when a ship with gold on board sunk and couldn't be recovered? I've read that this is how ship insurance started, but couldn't find anything about the effects of disasters, discoveries and conquests in the old and new world.

<comment><USER5> Relevant to the article: [https://en.m.wikipedia.org/wiki/Panic_of_1857](https://en.m.wikipedia.org/wiki/Panic_of_1857)

</comment></comment><comment><USER6> Hopefully someone will make a movie about this guy similar to: www.imdb.com/title/tt8106596</comment>


Ask HN: What are you working on? - _-___________-_ As we start a new decade, what are you spending your time on? Are you starting something new? Finishing something old? What are you working on? <comment><AUTHOR> Low code for developers and data scientists</comment>


Digital Sight Management, and the Mystery of the Missing Amazon Receipts - <AUTHOR> https://mssv.net/2020/08/16/digital-sight-management-and-the-mystery-of-the-missing-amazon-receipts/ <comment><USER2> Insightful article. And worldscraping is a delightful term, with lots of positive potential if there were an open platform.

Unfortunately I agree that big tech will definitely try to own AR-scraped data, and make it even more of a walled garden than smartphones, but I am pessimistic about government regulation preventing that, and keeping data open.

Governments around the world have so far only shown interest in"opening" the data into their own surveillance systems. And centralization of the platform by big tech makes it easier for them.

The only hope I have is from either of:

a) a new upstart that builds on an open platform from the very beginning, to compete with big tech (unlikely due to the significant hardware investment requirements)

b) the loser among FAMANG's AR attempts, who decides to open everything up as a last ditch attempt (plausible)

c) hackers jailbreaking AR glasses and creating an open platform

</comment><comment><USER3> I am surprised that such a provocative article is getting no discussion. "Worldscraping" is the cyberpunkiest new word I've learned in quite a while. Coincidentally, my new copy of Simulacra & Simulation should be arriving from Amazon today...

<comment><USER4> Author here: This is the second time it's been on HN's front page (I posted it the first time), and it didn't get any comments then either! I was pretty surprised "worldscraping" hadn't been used before tbh.

<comment><USER5> FYI I'm currently getting ERR_CONNECTION_CLOSED / "mssv.net unexpectedly closed the connection" when trying to read the article.

Edit: Working as expected now but I grabbed a screenshot if you want it for any reason. Not that it would be much help now.

<comment><USER4> It's hosted on Wordpress.com so hopefully they'll sort it!</comment></comment></comment></comment>


Kinect Air Guitar - <AUTHOR> http://www.chrisoshea.org/lab/air-guitar-prototype

<comment><USER2> Not bad for a simple experiment. Maybe even more interesting is how much one developer can do when supported by open source frameworks and libraries.

</comment><comment><USER3> I sure hope this kind of "Kinect hacking" was Microsoft's original idea: watch the community come up with all kinds of _crazy_ applications, and then use them in the next Xbox and blow everyone away.

</comment><comment><USER4> I think Microsoft know's the resuts if it tries to stop the hacking of Kinect. Their own games are really not that engaging so far - this could help. I just hope they don't try some Apple move with development.

</comment><comment><USER5> That's really clean. Good job and keep going. Also, you need air drums.

</comment><comment><USER6> I think now with Kinect, the only limitation is your imagination...

<comment><USER4> Zombocom

<comment><AUTHOR> <http://html5zombo.com/>

</comment></comment></comment><comment><USER7> he should mention that playing air instruments while listening to music, helps keep body, spirit and mind in shape. air instrument playing has health benefits. :-)</comment>


Ask HN: Tool to auto-generate basic docstrings in Python? - <AUTHOR> I know that most documentation generators require docstrings to already been in the source code. Are there any tools that will insert docstrings based on variable name and type? <comment><AUTHOR> The only half-decent tool I've found is PyMent: [https://github.com/dadadel/pyment](https://github.com/dadadel/pyment)</comment>


Ask HN: How to create mandatory quiz online lectures? - <AUTHOR> I am a professor looking to up my online teaching game. In Coursera courses, I remember lectures where the video would pause periodically and ask you a question or two to check your understanding. And continue once you answered those questions correctly.
I am looking to create a similar interface for my courses. Is there already a company that is offering these solutions? IT at my uni is just support, so they won't be able to handle anything complicated.
I want to record my lectures, that pause after at some pre-determined time. The students are given questions out of a pool of possible questions. Their answers are recorded and they are encouraged to go back and listen to the module again if they got some answers wrong. We are currently using Microsoft teams, but looking for a better solution. <comment><USER2> A quick google search of "live poll for class" shows several results. The top results are:

[https://www.polleverywhere.com](https://www.polleverywhere.com): Poll Everywhere is the easiest way to gather live responses in any venue… conferences, concerts, classrooms, and company off-sites — anywhere with internet.

[https://directpoll.com](https://directpoll.com): DirectPoll lets you create live polls in an instant.

And then some results for "top free polling tools" and the like.

</comment><comment><USER3> I haven't used the teacher side, but [https://edpuzzle.com](https://edpuzzle.com) does video lectures with graded questions alongside. I'm not sure if it supports pooling or revising answers though.

</comment><comment><USER4> Please check [https://prixest.com/app](https://prixest.com/app)</comment>


Car's Yearly Tune-Up Costs as Much as a Civic - ajay-d https://www.nytimes.com/2020/05/25/business/bugatti-chiron-price.html <comment><AUTHOR> A veyron's brake job costs more than a BMW S1000RR motorcycle, which can keep up with that car on most tracks. [https://www.youtube.com/watch?v=ujSvsyY15GE](https://www.youtube.com/watch?v=ujSvsyY15GE) And the last time i checked, you had to ship the car to france to have the work done.

I own a slightly less race-specced bike than the S1000RR, but i take a lot of pleasure in being able to get off the line in under 3 seconds at about an order of magnitude less cost than most cars that can do that.

Stating the obvious: hypercars aren't interested in performance-dollars very much.

<comment><USER2> The Veyron was built for speed not the track. It goes 65MPH faster than a S1000RR, but made huge sacrifices to hit those speeds in 2005.

PS: The pre-production Chiron Super Sport 300+ hit 304.77 mph showing how far things have come vs the initial 253.81 mph speed of the Veyron.

</comment></comment><comment><USER3> It is literally a symbol of having more money than you care about. The cost IS the feature. This is just ad copy.</comment>