Ask HN: Did anyone leave Software Engineering as your profession? If yes, why? - <AUTHOR> I am a Software engineer right now with over 4 years of experience. I have been having this feeling lately, that I feel the excitement I experience is not enough and I need more. So, I have been thinking about doing other things.
Has anyone else here felt the same? Or felt like being done with Software engineering? If so, I would like to know when did you figure this out and what are you doing right now, and how did you end up doing what you are doing?
Asking for Software Engineering in particular, because I have seen people from other professions find coding as a passion and wanted to know if there are people who went the other way. <comment><USER2> I'm currently on my first break after 6 years with no job lined up and I'm not sure I want to go back to software.

Unfortunately I don't have any answers but as the other responses point out software is a gilded cage because it would be very hard to get anywhere near an equivalent salary doing anything else.

I had become increasingly disillusioned with software, mainly the maddening bureaucracy, problematic management and lack of teamwork over my previous two jobs. But I also had something of an existential crisis when I realised I think that so much of what we do is just rewriting or updating apps to use the new hot tech in order to have something to do. I feel like so much of what's exciting people at the moment are solutions in search of problems, if StackOverflow can run on a few servers to serve an incredibly high traffic site why do we need k8s and serverless and firebase and microservices and kafka and whatever else (granted I have no idea what most of these things are or do)? Why in the name of God do I need an entire build pipeline and SPA framework to deliver some static HTML to users? I feel like Rust is interesting and potentially worthwhile but I don't have much interest in the domain of problems Rust excels at. Granted all this is coloured by the burnout I'm recovering from.

I think I will probably end up going back to software, maybe as a contractor, but I'm tempted to try something new. My belief is that it would be easier to do something socially meaningful in another field but even in scientific research most work is pointless and the reward/funding system is entirely dysfunctional.

<comment><USER3> I never really understood how insane it can get until my current job. We literally have a system diagram with like 30-40 different components (proxies, lambdas, pretty much all aws services), for a system that would work just as well with 3 (front/back/database).

At this point it's become a kind of joke for me because anything else would be too depressing.

<comment><USER4> Search for baklava code. Or architecture astronauts.

It's comes from leadership and trickles down to individual engineers. If your leader is deep into this mindset of completely over engineering things, you are screwed. I quit a job like this because the system was just too much. It was making me hate my profession and that's when I knew it was time to go.

You can get so far with a simple monolith, there is no need to suffer the whims of someone justifying their position.

<comment><USER5> Hi,

I've had the luck to work on a 10 years old Django System. Everyone from the 6 people working on it said it was the purest sh.t they saw.

On the other hand, I need to learn new stuff. I tried Express or other JS fmw but these are not new things.

You can get pretty sh.tty unusable code with a monolith, while state-of-the- art is expected to be understood and seek by people that crave for more.

Tbh, I think it all depends on how well a team is prepared, and ofc it is more complex with the later tools, but they definitely avoid a lot of useless and shame code.

It is also very common that business hasn't validated their initial client base.

</comment></comment></comment></comment><comment><USER6> I took a break for a couple years at one point, also about 4-5 years after I started. I went and threw boxes around warehouses for UPS, then I did a year of law school, then I realized that software wasn't so bad, so I came back to it.

But I came back with different goals. I wasn't pushing myself to be the best engineer in the world... I just did the work. I wasn't trying to make software used around the world... I just took jobs and did my best.

And I found that with a more relaxed attitude, I was better. My work was better, my performance and satisfaction overall was vastly improved. And my career went to a better place. I still don't have the passion for it that some people do, and won't miss it when I retire. But I have solid skills, a solid work ethic, and a solid career.

<comment><USER7> I want to do something like this. Take a hiatus but work a minimum wage job so that I appreciate what I have.

<comment><USER8> No need for extreme measures. Why not do some gardening in the weekend? Some manual labour?

</comment></comment></comment><comment><USER9> This isn't for everyone, but after working as a software dev for 3 years I went to law school. Because I had a technical background, I was immediately pegged to be a patent attorney. I know, most software developers hate patents, but most of the work is not as evil as it seems in the press.

There are often real disputes, between real companies, on patented ideas that are incredibly similar. Having a solid technical foundation is necessary to fully understand what's going on. As a patent litigator, you generally get to work on 2 or 3 cases at a time, often involving different technology, and you get to become the expert in them. I personally worked on Bluetooth, H.264, and crazy image processing algorithms, among many others. The downside of course, is that while you deal with tech and learn about it, you're now on the outside. You don't make anything, and the job is stressful, but for the right person, it can be a good fit.

<comment><USER10> It seems that you still have found some middle ground between software engineering and law by creating Docket Alarm. :-) Congratulations, it seems like a nice and useful platform. Apparently, it is Django-based, though I'm curious about what are other components of your stack (e.g., for mobile app and analytics).

<comment><USER9> > It seems that you still have found some middle ground between software > engineering and law by creating Docket Alarm.

Yup, for me, it was the right balance. If you want to start a company, learn engineering, then learn something else, then build something for that something else.

> Apparently, it is Django-based, though I'm curious about what are other > components of your stack

I use Django, jquery, elasticsearch, and google app engine 4 years ago. For analytics, I have used so many platforms and have built a bunch of custom stuff. Sadly, no native mobile app, but the site is fully responsive.

I'm pretty skeptical of taking my past decisions and re-applying them today. Today, parts of it (especially jquery) seem outdated, I'd probably choose vue. Even Python, which is generally great, is causing pain today (moving 2.x to 3.x; performance for a number of corner cases). App Engine used to be an incredible platform, but google has stripped so many features from it that it's hard to describe it today as a real PaaS.

</comment></comment></comment><comment><USER11> I've thought about it many times, but it's hard to walk away from the money. I'd have to start at zero in any other field, and to reach even half the salary in another standard professional career, I'd have to go to school for a few years. If I had any business ideas or an entrepreneurial spirit, I could try to set out on my own, but that's unlikely. In lieu of going insane from the tedium and politics of a software career (not that other careers are any better in this regard), I take a lot of time off; usually about a year every three years. I always go back to software, though, because it's easy. I'm tempted to do something completely different for a quarter of the salary, like become a dive instructor. Beats sitting behind a desk writing the same web app for the 50th time and dealing with sprint planning meetings.

</comment><comment><USER12> I was burnt out as a programmer a long time ago. Using my IT knowledge I moved into the field of business analysis and then on to project management. After a while I got tired of the politics so I switched to working as a network / operations engineer. More recently I went back to university to do post-grad research in model based software engineering and now freelancing as a consultant / project manager.

I have never felt that I had to stay stuck doing something that I no longer enjoyed. For me learning new material (and enterprise scale networks was a big challenge) and facing new challenges has always been more inspiring than merely picking up a fat wage. The biggest problem being that, for most people, their spending increases faster than their income. Exercising a small bit of restraint and having some savings opens up so many great opportunities and adventures.

I can understand the fear of starting from zero. But that doesn't need to be the case. What I have always done is to focus more on domain knowledge and then segueing from one area to a adjoining one. Technical knowledge, alone, quickly becomes outdated. But domain knowledge continues across multiple generations of technology.

</comment><comment><USER13> Not sure if everyone feels this way, but I kind of hate "software engineering". I feel like once the logic or core principle is figured out, the rest is just the chore of implementation. It's like the fun part is visualizing how everything should work in your head, then you have to do the boring part of typing it all out, debugging, dealing with stakeholders etc which forms 90% of "software engineering".

I'm a lot happier now that I can outsource some of that work to other people. Maybe in the future some of the boring 90% can be automated away, but the human factors remain.

<comment><USER14> Try being an analyst or architect. This way, the "boring" work gets done by coders, while you define the work, figure out big problems etc. It can be quite fun.

</comment></comment><comment><USER15> I moved to sales & investing after 10 years of coding. I still write code every day, but only for myself. For me, programming is beautiful if I am in control of the product getting built. It's a complete package and the creative aspects is what gives me the thrills. Being a code monkey does not.

<comment><USER16> How'd you do that move?

<comment><USER15> Jumped right in. Struggled a lot in the first six months. Then it got better.

</comment></comment></comment><comment><USER17> My old VP Engr quit and opened a falafel shop. Still going strong after 20 years.

</comment><comment><USER18> A friend I know went from software engineering to civil engineering for difficulties in finding work. But I find many more that do the opposite, go from civil to software.

I was first inclined into software before deciding on civil in college after the dot com bubble burst.

I recommend all software inclined engineers do anything but software. Software Programming is a very powerful tool in most industries but is least required in the software industry.

So I recommend you pick up another skill. And start working on it from a software perspective.

<comment><USER19> > A friend I know went from software engineering to civil engineering

Do you mind if I ask how he achieved that? Did he go back to college? I ask because I've been thinking about transitioning to mechanical engineering, but going back to college for 4 more years and racking up 10s of thousands of dollars in debt is off putting.

<comment><USER18> The first degree was from India. So he had little college debt. Second degree was from public university. With two years at county college. And he worked part time.

</comment></comment></comment><comment><USER20> I've gotten bored multiple times and changed companies, organizations, teams, or worked more on my promotion to change the nature of my work.

Try to see if you can find a more interesting domain to apply your skills in. For me, moving to a senior role offered a more interesting scope and facing more design work and advising mixed with some coding, mentoring, etc.

If you're tired of the field entirely, maybe there are non technical roles where you can apply a set of your skills in something like management.

</comment><comment><USER21> I quit software for mainly two reasons: the constant squabbling about what framework of the day to use and the lack of training people have. I know it may sound petty, but when people doesn't even know how to use the current framework, why on earth should the add more they don't know anything about?

</comment><comment><USER22> Surprisingly common to feel like this. You've done 4 years. Hope you've been saving hard. You should be able to take a year out and rest up and see what takes your fancy - you might find after a good break your interest in software is rekindled.

p.s. I believe all software engineers should be factoring in a year out for every five years worked.

</comment><comment><USER23> Did 10 years at 4-6 companies, from startups to big co. Felt like most coworkers were big pussies who couldn't even change their own car oil. Their time is just so valuable. I was always handy, did all my own auto/home repairs, etc. so took the leap and started buying distressed properties as-is in "bad" neighborhoods. Learned a lot about evicting squatters, keeping homeless and druggies from trespassing and loitering, fixing roofs, replacing leaky cast iron plumbing, repairing refrigerators and stoves, fishing romex up 3 floors, renting to millennials, gentrifying a street, etc.

<comment><USER24> Call me a pussy but your new job sounds a hell of a lot worse than most software Dev jobs to me. Each his own I suppose.

<comment><USER25> I think Op is some parody / troll account. It's too unhinged and it's trying too hard to offend. I don't see any other explanation.

<comment><USER24> Is using the word "pussies" trying hard to offend? It seems like a pretty legitimate post, I know many people that think like this guy, only most of them don't read hacker news..</comment></comment></comment></comment>


Yahoo Announces Non-Exclusive Search Agreement With Google - <AUTHOR> http://www.techcrunch.com/2008/06/12/yahoo-announces-non-exclusive-search-agreement-with-google/

<comment><USER2> As much as monopolies hurt business and innovation I set two benefits to this:

1) As a publisher using Adsense I have much more relevant ads with more advertisers flocking to one supplier.

2) As an advertiser I have access to a nearly unlimited amount of channels in almost every market.

I'm sure one day we might look back at this as a negative thing but for now I still hold respect for Google.

</comment><comment><USER3> yahoo made a smart move - problem creators now have to calm down for a while - it will give some time to breath and move their strategy as a success ... kind of wait and watch.

<comment><AUTHOR> as pg once said that if enough time and freedom is give to jerry yang to run the company the way he wants then he can prove his worth - i think the time has come that he proves himself and get back the reputation and business as earlier ... good luck to him.</comment></comment>


Beautiful Progressive Web App for Finding Coastal Tides - <AUTHOR> https://hightide.earth <comment><USER2> If it meets the guidelines, this might make a good 'Show HN'. Show HN guidelines: [https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html)</comment>


90% of apps on Google Play contain third-party trackers - <AUTHOR> https://arxiv.org/abs/1804.03603 <comment><USER2> The latest release of GPGTools for the Mac also contains a third party tracker. From their FAQ[1]:

===== Why am I seeing an outgoing connection to analytics.paddle.com/paddle.com. Why are you tracking your users' activity?

We are absolutely not tracking our users' activity. We do perform an initial remote verification of the activation code, which is the reason why you are seeing this connection. Paddle does provide the feature to track user activity but we explicitly don't enable that. Unfortunately the endpoint for the API (vendors.paddle.com) and the endpoint for their analytics feature (analytics.paddle.com) share the same IP, so dependent on what the result of your macOS's firewall reverse DNS lookup is (built-in, Little Snitch, etc.) you might either see a connection to vendors.paddle.com or analytics.paddle.com =====

And that's on a desktop platform.

</comment><comment><USER3> Does that include all apps that 'need' google play services?

<comment><USER4> Of course it does.

Dear Freshdesk, and other services who care about their customer, and privacy: if you're reading this... Please distribute an app through F-Droid! Or, at least an APK.</comment></comment>


JSON Schema - <AUTHOR> http://json-schema.org/ <comment><USER2> I was unaware of the usefulness of JSON Schema until recently I was writing some Vega Lite json specs in VS Code.

I don't have any relevant json extensions, but VS Code started auto-suggesting values for options (e.g. a drop down for whether the chart type is a 'bar', 'line', etc.) and pointing our mistakes like 'the size should be an integer not a percent'.

This was the lightbulb moment for me - as if by magic, my code editor was double checking my json document was valid and helping me write it. Amazing, and super useful.

It took me a while to work out how it was doing it, but all vega lite examples start with the magic:

"$schema": "[https://vega.github.io/schema/vega- lite/v2.json"](https://vega.github.io/schema/vega-lite/v2.json").

<comment><USER3> This light bulb is exactly why many people prefer statically-typed programming languages, especially above a certain codebase size/complexity and team size.

Make my tools do the grunt-work for me, thanks, so I can focus on the actual problem domain.

<comment><USER4> > This light bulb is exactly why many people prefer statically-typed > programming languages, especially above a certain codebase size/complexity > and team size.

I agree, with the caveat that _sometimes_ you need an escape valve.

Basically I want static typing 99% of the time and I don't want guff from the compiler (or fanboys) in that 1%.

<comment><USER5> That's what (unchecked) exceptions are for.

They're supported by all general purpose programming languages because we — as developers — are smarter than the compiler.

<comment><USER6> 1\. GP is talking about type safety in general, not just exceptions.

2\. Not "all general purpose programming languages" support exceptions. To name a few that don't: C, Rust, Go.

<comment><USER7> Go has the empty interface, if you really need it.

<comment><USER8> The empty interface is not an unchecked exception, but panic() is. [https://golang.org/pkg/builtin/#panic](https://golang.org/pkg/builtin/#panic)

<comment><USER6> I don't actually know Go, but can panic() be caught? If not, it's not an exception.

<comment><USER9> With recover(), yes.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> Is there something wrong with just using TypeScript's type system for expressing the required format of JSON data that would make me want to use the significantly more verbose JSON schema format instead? I've written a couple of code generators (e.g. [1]) in the past which use the former approach and found it a very convenient way to express my models.

For example this:

        {
            "title": "Person",
            "type": "object",
            "properties": {
                "firstName": {
                    "type": "string"
                },
                "lastName": {
                    "type": "string"
                },
                "age": {
                    "description": "Age in years",
                    "type": "integer",
                    "minimum": 0
                }
            },
            "required": ["firstName", "lastName"]
        }

can be expressed like this:

        interface Person {
            firstName: string;
            lastName: string;
            age?: number; // Age in years
        }

I can see that there are additional constraints expressible, such as the fact that age has to be an integer and has a minimum value. However those could all conceivably comprise a superset of TypeScript's interface definitions. JSON Schema just feels to me like XML Schema all over again.

[1] [https://www.npmjs.com/package/jsonidator](https://www.npmjs.com/package/jsonidator)

<comment><USER11> TypeScript provides, primarily, a type _verification_ system.

JSON-schema provides a type _assertion_ system.

Both serve as type documentation systems.

TS says "if you create objects in my world, I will use my type system to verify that you're creating the right kind of objects, and you can look up their definitions to see what's in them". JSON-schema says "if you give me external, un-annotated objects, I can tell you whether they comply with a type definition, and you can look at that definition to see what's in them".

TS has a limited assertion facility, but it's not nearly as capable as JSON- schema. Projects like io-ts ([https://github.com/gcanti/io- ts](https://github.com/gcanti/io-ts)) add more functionality to those assertions, but they're still much less easy to use, in my opinion, than JSON- schema for this purpose. If you're using TS types to assert, though, you can definitely describe more complex constraints, so it may make sense as an alternative to JSON-schema for some people.

As others have pointed out, JSON-schema is also much easier to use on non- TypeScript platforms than TS assertions, which, even if you use an assertion library like io-ts, will still be limited in use to JavaScript (and JS- to-$other_language runtimes).

<comment><USER12> A dependent type system can do both.

<comment><USER11> ...yes it can?

Would you expand on how dependent type systems are relevant to the differences between TypeScript/JSON-Schema, or how the use of a DTS could help ameliorate the issues people have in those areas?

<comment><USER12> Not in an HN comment, but here's a paper called Power of Pi [https://cs.ru.nl/~wouters/Publications/ThePowerOfPi.pdf](https://cs.ru.nl/~wouters/Publications/ThePowerOfPi.pdf) that probably explains it the best. It shows the usage of it for deriving a parser and generator from a single specification, and also crypto protocols and database usage. It uses Agda for its examples, but it can be translated to Idris as well.

</comment></comment></comment></comment></comment><comment><USER13> For a long time, json-schema was stuck at draft-04, and it looks like in the last few months under new authorship it's up to draft-07. The mapping of these names to IETF RFC revisions is more complicated, and explained on the site [1].

Json-schema is useful for describing JSON data structures in JSON, so you can bring a codegen and generate corresponding structures in your language, or perform some payload validation at an API gateway; and many higher-level API specs -- like OpenAPI/Swagger -- make use of Json-schema underneath.

However, I'm not sure I fully understand why they branched out into also becoming a hypermedia description language [2] when there's already half a dozen others in this space [3], and reading some of the discussion about this [4] makes my head hurt.

[1] [http://json-schema.org/specification.html](http://json- schema.org/specification.html) [2] [http://json-schema.org/latest/json-schema- hypermedia.html](http://json-schema.org/latest/json-schema-hypermedia.html) [3] [https://sookocheff.com/post/api/on-choosing-a-hypermedia- for...](https://sookocheff.com/post/api/on-choosing-a-hypermedia-format/) [4] [https://github.com/json-schema-org/json-schema- spec/issues/4...](https://github.com/json-schema-org/json-schema- spec/issues/48)

</comment><comment><USER14> This is what I'm using for schema validation in FormAPI [1]. The best part about using a standard is that there's a lot of open source libraries you can use. It's really easy to validate a schema in Ruby [2]. I use json-schema- faker [3] to fill out PDFs with test data. I also use AlpacaJS [4] to generate HTML forms based on the JSON schema.

Anyway, I'm really glad I didn't try to roll my own thing for schema validation. Alpaca is insanely powerful, and it would have taken me weeks or months to build the online forms from scratch.

[1] [http://formapi.io](http://formapi.io)

[2] [https://github.com/ruby-json-schema/json-schema](https://github.com/ruby- json-schema/json-schema)

[3] [https://github.com/json-schema-faker/json-schema- faker](https://github.com/json-schema-faker/json-schema-faker)

[4] [http://www.alpacajs.org/](http://www.alpacajs.org/)

<comment><USER15> We have a similar setup for extending our app with custom fields / forms using ruby-json-schema and react-jsonschema-form [1].

ruby-json-schema has worked out quite well, we added a custom schema reader so that we could generate enums dynamically based on records in the database via custom refs eg. app://schemas/organizations?type_eq=education&tags=one, any of these internal refs are inlined at the API layer.

One issue we did run into with ruby-json-schema, is the schema caching is not thread safe. We opted to clear the cache after each validation, this ended up causing race-condition issues. In the end we had to use a mutex on write and cache the validation result (not ideal, but our app isn't that write heavy).

[1] [https://github.com/mozilla-services/react-jsonschema- form](https://github.com/mozilla-services/react-jsonschema-form)

</comment></comment><comment><USER16> Creating a JSON schema for your data and committing it is a great way to codify assumptions about the data consumed or produced by your program. Once you have a JSON schema for your data, you can generate models and serializers in many languages with quicktype: [https://app.quicktype.io#s=coordinate](https://app.quicktype.io#s=coordinate)

</comment><comment><USER17> Reminds me of XSD - the XML schema definition language that itself is XML. Any takeaways (good/bad/ugly) from that experience that are applicable here?

<comment><USER18> My first thought on seeing the headline was "continuing to reinvent the XML ecosystem...in JSON"

<comment><USER4> Do you consider that a negative?

I personally think it's great. The faster the ecosystem is reinvented the sooner we can stop wasting our lives reading it.

<comment><USER18> It just strikes me as a circular development pattern.

<comment><USER19> XSD is only part of the value-offering... XSD + XSLT is the value provider for format heavy, robust, declarative, document transformations.

JSON-schema solves many of the validation issues of JSON in the absence of an XSD-for-JSON solution, to allow message processing and other validation assertions. From that perspective it's 'duplicated effort', but only to the extent that JSON messaging applications are maturing to the point they need the kinds of guarantees that were so clear during the development of XML/SGML. It's a whole different language supporting a range of JSON-powered clients that don't necessarily require transformation support or other guarantees.

This is about JSONs ecosystem maturing to the baseline, not about reinventing the wheel :)

</comment></comment></comment></comment></comment><comment><USER20> Hi all! I'm one of the new team that works on JSON Schema since draft-4.

I could spend time addressing each issue raised here, but that could take some time...

@niftich We've (the new authorship) actually been working on JSON Schema since late 2014!

If you're using JSON Schema, or are interetested in finding out more, we DO now have a slack server; invite link at the bottom of json-schema.org. Come talk with us.

We're looking for companies who are using it in production to weigh in on new developments and releases. Our current release plan means we will have an RC with time for feedback. We want to hear your issues, comments, and suggestions!

One common problem at current is the level of support. Because draf-4 has been around for a LONG time now, it became the defacto standard, and has the most support. Developers come along and use an older library which only supports draft-4, but want to use new functionaltiy found in draft-7. Many implementations support draft-7, and some even implement new functionaltiy before it's finalised. SO remember to check which version of JSON Schema the library you're using supports.

HyperSchema already exsisted at that point, so we inheirted it. It focuses on providing specification for correct HTTP usage and HATEOAS driven APIs, where as other higher level API specification languages allow you to describe any type of API.

It looks like a good number of you have good questions, ideas, or issues. Thanks for all those replying supporting JSON Schema and those showing a balanced viewpoint.

Come chat with us!

</comment><comment><USER21> This is what mongodb picked up for schema validation in its last release: [https://docs.mongodb.com/manual/core/schema- validation/](https://docs.mongodb.com/manual/core/schema-validation/)

</comment><comment><USER22> This was incredibly handy when I was building a Slack bot that helped my colleagues figure out what microservice did what, since they often had puntastic names and there were enough that it was hard to keep track of ([https://github.com/leemachin/wtf-is](https://github.com/leemachin/wtf-is)). It made it really easy to validate the structure of the metadata file and return errors that made at least some sense to the person trying to integrate it.

It's not particularly friendly to the maintainer compared to switching to a statically typed language, but it would have taken a fair bit more effort to wire that up without this.

</comment><comment><USER23> Does anyone actually use JSON Schema for real?

<comment><USER24> Not since I discovered GraphQL. It's hard to imagine any other abstraction on top of REST APIs that packs as much robust functionality in something so easy to use.

<comment><USER25> I'm just curious - GraphQL is more of a query standard, while JSON Schema is structure validation. How could one use GraphQL for, say, validating input?

</comment></comment></comment><comment><USER26> My startup built an extended version of JSON Schema that also supports conditional logic and data references to power a customer-configurable Workflow system. Basically a lightweight way to script a series of tasks with more advanced validations and data pre-population. Works great, customers love it.

<comment><USER27> I built this same thing in the past month for automated questionnaire rendering in react, saving and merging the responses into word documents.

<comment><USER26> whoa what's up man?! Wickedfire paths cross in the most random ways haha

<comment><USER27> Hah, seriously man! I've seen you here a couple times, would love to catch up. Ping me? Email in my profile

</comment></comment></comment></comment><comment><USER28> This stuff is great. I used it to design an API in Python.

A request comes in and the wrapper checks for a schema. if it is not a GET request and one does not exist it errors out even before the method is called. If one exists, the schema is validated automatically. If the target method is called, you can be safe assuming that the input is valid (in terms of type and does it exist or not).

Same goes for output. Server will error out before it lets incorrect output be returned to the user. Great way to present what the application accepts to a developer. This is what it will accept and return, because this is what we use to validate input and output!

</comment><comment><USER4> Anybody using this in production?

We (reluctantly) went with XML for an external facing B2B API, because JSON schema was stuck in draft and did not seem to be in widespread use. JSON without schemas would have been a nightmare.

<comment><USER29> We "use" this in the form of Swagger, which is a web service that can download a specification of how an HTTP API works and generate a minimal UI to use that HTTP API from that specification. While the specification of the API is in "Swagger", the actual documentation of types used by the API is JSON-Schema.

The UI is okay; however, we've had two big downsides:

* The UI needs to be able to fetch the JSON schema document from the API; that means that it needs to correctly respond with CORS headers if you host the web interface on a separate domain. (We do, s.t. each API service doesn't need to repeat the mundane task of hosting it and keeping it up to date.)

* The schema needs to be useful. Far too often, people will document an API endpoint as taking "JSON" and returning "JSON"; this is true and correct, but not _helpful_. What keys exist? What are the valid values for those keys? etc. The author of the schema needs to take the time and thoroughness to document the API, and all too often either they do not (the workmanship is sloppy) or they can not (management provided deadlines that were too tight to get the job done right).

Neither of those are really problems with JSON-schema per se. You also don't _need_ JSON schema to document a JSON API: you can always do so with prose, by using the type system of a language your devs are familiar with and providing a set of rules to map those structures/types to JSON object, etc.

My biggest nit with JSON schema: it's painful for humans to interpret. (And we actually do it in Swagger, which is in YAML, so it's slightly easier to parse and we get comments. But it's considerably verbose compared to most type systems; the upside is that it _can_ encode more information such as descriptions, or valid ranges/values if the raw type like "number" or "string" doesn't suffice.)

</comment></comment><comment><USER30> It's such a shame that many of these standards ignore or are ignorant of the elementary and fundamental sum type. Instead I am left needing to employ some ad-hoc encoding using string tags or similar.

<comment><USER31> Does oneOf not cover it?

<comment><USER30> oneOf looks like it could be used to encode sum types together with a custom string tag, but that's not the first class treatment that records get.

<comment><USER32> Mark ([https://mark.js.org](https://mark.js.org)) is a new notation that extends JSON with a type-name, which can be a more structured way to encode type info, instead of using custom string tag.

</comment></comment></comment></comment><comment><USER33> I'm just now becoming familiar with JSON Schema. It's very interesting. However, before I or my team new it existed, we created this for filtering and validation, which has its own sort of schema: [https://spireteam.github.io/whitelister/](https://spireteam.github.io/whitelister/)

<comment><USER34> Your site's CSS makes it unreadable on wide screens: [https://imgur.com/a/WlYOA](https://imgur.com/a/WlYOA)

</comment></comment><comment><USER35> I can imagine uses, but I was looking for some kind of faq that answered what problem this was solving. Aren't there other serialization specification formats that come with type validation built in (like protocol buffers and thrift), that can be serialized to json as well? Maybe I am confused about what this does.

</comment><comment><USER25> These guys - [https://cswr.github.io/JsonSchema/](https://cswr.github.io/JsonSchema/) \- analyse JSON Schema and propose some mostly non-breaking adjustments.

</comment><comment><USER36> JSON Schema is useful. Here is a post with a Python example of using it to validate some data:

Using JSON Schema with Python to validate JSON data:

[https://jugad2.blogspot.in/2015/12/using-json-schema-with- py...](https://jugad2.blogspot.in/2015/12/using-json-schema-with-python- to.html)

</comment><comment><USER37> Techies are so f'ing useless. We keep reinventing every frikking wheel. I should get out of tech, it's depressing

<comment><USER38> What, JSON shouldn't have schemas because they exist for other formats? Or JSON shouldn't be used because XML is good enough?

While I agree that tech should reevaluate its sisyphean upgrade treadmill, I'm not sure how that relates here.

<comment><USER39> > Or JSON shouldn't be used because XML is good enough?

Mu. Neither JSON nor XML should be used, because S-expressions are good enough, and more attractive to boot.

<comment><USER38> [https://eli.thegreenplace.net/2012/03/04/some-thoughts-on- js...](https://eli.thegreenplace.net/2012/03/04/some-thoughts-on-json-vs-s- expressions)</comment></comment></comment></comment>


Show HN: Robots Reading the News – The Podcast - <AUTHOR> tldr; Podcast of well-spoken robots reading recent news, here's how to listen in iTunes: http://i.imgur.com/67VFszI.png
Here's the podcast link you see in that pic: https://dl.dropboxusercontent.com/u/8948326/podcast/index.rss
FADE FROM BLACK - COMMERCIAL GUY SPEAKS
Is this you?
(Imagine a frumpy looking fellow sitting at a computer screen, trying desperately to type in CNN.com or FoxNews.com or NPR.org or RealUltimatePower.net into his browser, but accidentally spilling a can of paint all over himself instead. [cue sad sounding trumpet] )
Reading the news is hard! Wouldn't it be better if robots read the news to you?
Now they can!
Introducing "Robots Reading The News"! The podcast that has my mom saying:
"Boy, my son is lazy!"
And my dad saying:
"Who would ever want to listen to a robot read the news?"
With reviews like that, you've gotta try it. Just follow the instructions at the top of the post.
ORDER NOW - ROBOTS ARE STANDING BY!!!!
I built it using Node with: request, cheerio, unfluff (parsing out articles) and say libraries.  Python's sumy for NLP summarization plus some text thesaurus spinning, and Ruby's dropcaster for RSS creation in Dropbox.
request https://www.npmjs.com/package/request
cheerio https://www.npmjs.com/package/cheerio
unfluff https://www.npmjs.com/package/unfluff
say https://www.npmjs.com/package/say
sumy https://pypi.python.org/pypi/sumy
dropcaster https://github.com/nerab/dropcaster <comment><USER2> direct link for vlc streaming [https://dl.dropboxusercontent.com/u/8948326/podcast/2016-08-...](https://dl.dropboxusercontent.com/u/8948326/podcast/2016-08-04.mp3)

<comment><AUTHOR> I'm still messing with it, so that link probably won't work anymore. :) But this one will (until I get the bug and mess with it even more, haha): [https://dl.dropboxusercontent.com/u/8948326/podcast/2016-08-...](https://dl.dropboxusercontent.com/u/8948326/podcast/2016-08-05_12-51-59-AM.mp3)</comment></comment>


Ask HN: How do you organize data on your disk? - <AUTHOR> Your projects, repositories, documents, media and other data files — what does the directory hierarchy that holds them look like? How did you come up with it? <comment><USER2> Corporate (banking) - across multiple departments and users - but I'd like to share:

Bucket

|- filename1

|- filename2

|- filename3

|- filename4

|- filename5

|- ...

I'm not joking. This is because I worked in an organisation where directory structures and silo'd access across each department were removed, and everything got thrown into a pool of documents with files that needed to be correctly names, and attached meta data filled out.

Most people hated it.

But I loved it, because it brought a discipline of accurate filenames and attaching correct meta-data. It brought strong disciplines on security/permissions. It brought an audit trail of who had opened or edited what and when.

Whenever I see a department of a company with endless directory upon directory of files with data replicated across them, I know the department has big people trouble - stuff is being done manually and operational controls are weak. Like seeing Excel sheets with merged cells = non-normalised tables that cannot be pivot-tabled = staff don't know basic Excel functionality = stuff is being done manually, time wasted, effort duplicated.

Everything was accessed by search. This also brought strong levels of discovery: Formerly, a document in another department could have been useful to me, but I'd not know of its existence unless I'd caught wind of someone working on this problem before. Now, with correct permissioning, I would see it through search. This saved incredible amounts of time. I also know that what I wrote 10 years ago isn't stuck in a directory that no one knows exists, it is at the fingertips of anyone who needs it.

</comment><comment><USER3> I'm really horrible at this. As neat as I keep my git repos, I'm a chaotic keeping my disk organized. My desktop started to regularly get crowded with files since I had a computer with an OS that made that possible. Still move everything to "Stuff" whenever it gets too crowded and I don't want to make the decision yet whether I should delete this particular file or not. At least I have a "Code", "Git" and "Stuff" folder now.

</comment><comment><USER4> My home directory looks something like this:

      bin/  - scripts I use
      build/  - software I build
      doc/
         biz/
           taxes/
             YYYY/
             ...
           budgets/
           receipts/
           house/
         travel/
         personal/
         notes/
         ...
      projects/
         projectA/
           docs/
             invoices/
             contracts/
             proposals/
             presentations/
             ...
           bin/
           src/
             repoA/
             ...
           archive/  - old data
           people/
             johndoe/  - sent to me by someone else
               YYYYMMDD/
               ...
             ...
         ...
         projectN/
      crap/   - Junk organized by date
         YYYYMMDD/
         ...

</comment><comment><USER5> I put most of the things i download on the desktop, and when it's to cluttered i create a folder like "downloads2" and repeat until the "downloads3" is needed ;D

<comment><USER6> I started cleaning my act up after finding "shit to sort" inside a "shit to sort" folder, which was nested under another "shit to sort" folder. I remember also seeing another "shit to sort ACTUALLY THIS TIME!!!" folder within there somewhere.</comment></comment>


Why are Gilt and Flash Sales Sites so Profitable? - <AUTHOR> http://primitus.com/blog/why-are-gilt-and-flash-sales-sites-so-profitable/

<comment><USER2> "This is an easy business to start; it's a really hard business to scale."

It will be interesting to see how Gilt, Ruelala, ideeli, etc. continue to address this. They all seem like they've been frantically trying to diversify (travel being the most notable one IMO).</comment>


Obama administration defends massive phone record collection - <AUTHOR> http://www.reuters.com/article/2013/06/07/us-usa-wiretaps-verizon-idUSBRE95502920130607

<comment><USER2> _"It's called protecting America," added Senator Dianne Feinstein_

The Congressperson who leads the group providing oversight on things like this thinks you're stupid.

<comment><USER3> > "It's called protecting America," added Senator Dianne Feinstein

I wish she would hug us a little less tight. We need a less protecting, so we can breathe.

</comment></comment><comment><USER4> I would have been nice had they defended this before it happened.

You know, in public. Motherfuckers.

</comment><comment><USER5> Unfortunately, Obama is so amoral and charismatic, he'll wiggle out of this one as he always does.</comment>


Unexpected physics engine stability joy - <AUTHOR> http://altdevblogaday.org/2011/04/11/unexpected-speculative-stability/

<comment><USER2> The writer makes this sound exciting but I'm having a hard time trying to understand this article.

Could someone explain this in simple terms to someone with only a tiny bit of physics engine experience?

<comment><USER3> Contact generation is the process whereby, after a collision is detected, the points of contact are defined, which are necessary to simulate collision response(responses themselves are anything from a simple separation of the bodies to an in-depth physics sim).

In the most naive form of contact generation, you move the bodies, let them intersect each other, and only afterwards do something to remove them by finding the points of intersection after they've already interpenetrated. Even with simple situations, this is the source of numerous glitches, and makes physics sims unstable, since the two interpenetrating bodies will both see that they're "inside" each other, create forces to get out the situation, and then collide with other bodies, repeating the process, gaining energy, and making the sim explode, or at least jitter endlessly. (Collisions between many close-by bodies also present other contact quality problems - bodies might "miss" each other and only contact some of the other shapes.)

The speculative contact system described is one approach towards resolving these kinds of instability problems: Basically, it uses velocity information to "look ahead" at where the body will move, and then it adjusts the velocity vector to fit so that the two bodies touch exactly. This can be repeated for all potential collisions, until the minimum movement for each body is found, at which point contacts may be generated.

I happened to discover exactly this technique independently, trying to make a simple game collision system as robust as possible even at high velocities. Velocity appears to be a critical piece of information for robustness, as all the heuristics I tried to separate bodies already interpenetrating led to major glitches when the heuristic guessed wrong. I also tried a very similar method of "rolling back time" by moving the body backwards along its existing velocity with a step-by-step approximation, until the collision no longer happened; this turned out to not be as good as modifying the _future_ velocity vector to touch exactly, as it was cruder, left points of contact inexact, and thrashed the original velocity values, leading to things stopping dead and "sticking" as soon as they collided anything at all.

<comment><USER2> That helped a lot. Thank you!

</comment></comment></comment><comment><USER4> Why are you green?

<comment><USER5> I think new users are green. New as in: user.account_age <= GREEN_ACCOUNT_AGE

<comment><USER6> >>> print(GREEN_ACCOUNT_AGE)</comment></comment></comment>


Dear White People, You Suck at Diversity - <AUTHOR> http://techcrunch.com/2015/12/13/dear-white-people-you-suck-at-diversity/ <comment><USER2> The problem with promoting diversity as a goal is that you can define diversity however you want. The author mocks others for including women but not "women of color". Similarly, I've heard other argue that diversity of socio-economic class is more important than other definitions. Because are you really promoting diversity by including only those with that went to the same school and had the same upbringing as everyone else? Who gets to define what should be promoted?

Depending on the organization, diversity can be a key driver of success but when it's forced upon society as a moral imperative, it brings out those that choose to use social constructs and recent history as a means of consolidating their own power. Be wary of those who claim to speak for any one group of people, especially if you never got a ballot.

</comment><comment><USER3> '"In fact, we just hired a young woman from Stanford who's every bit as good as her peers," Sequoia Capital investor Mike Moritz said in an interview with Bloomberg, titled Sequoia's Moritz: Looking for Women to Be Partners. "And if there are more like her, we'll hire them. What we're not prepared to do, is to lower our standards."'

Pro tip: don't brag about how great you are at diversity by starting from the assumption that women are inferior.

<comment><USER4> I honestly didn't read that into what he said, am I missing some context that alludes to this meaning? It seems like he's trying to call out those people who would demand he hire a less qualified woman simply _because_ of her sex.

Isn't holding the belief that there exist qualified women putting them on equal standing? And then because there are qualified women wouldn't hiring without concern for sex the fairest hiring method? If he believed that women were inferior and wanted to promote diversity then wouldn't he would be making the exact opposite claim?

If he truly holds the beliefs he claims to in the quote then the problem of diversity could be solved at the educational level by encouraging more women to go into tech. Assuming merit is uniformly distributed throughout society, if 10% of your applicants are qualified but 90% of your applicants are men then you would expect your staff to be 90% men assuming no bias whatsoever.

<comment><USER3> Mostly I think it's simply a boneheaded comment. Because it implies that hiring women means lowering your standards, except for a few exceptional individuals.

</comment></comment></comment><comment><USER5> I'd be relatively happy if diversity in tech had progressed sufficiently that second order effects like intersectionality were the most important issues remaining.

Pretty sure we're not there yet. I'd argue that focusing on the big first order problems still has the most upside in general if not for everyone.

</comment><comment><USER6> Which races should we emulate to get better at diversity?

<comment><USER7> As those who control we (white men) have the responsibility to fix the issue.

<comment><USER8> > As those who control we (white men) have the responsibility to fix the > issue.

White men don't control anything , rich men who happened to be white do (in US). Making everything about race IS racist. The article is certainly racist.

</comment></comment></comment><comment><USER9> Not the only one sucking at it, considered the title...

</comment><comment><USER10> Oh, the irony of an article asking people to become more sensitive to the social issues facing diverse peoples by using a racist generalization in the title.

<comment><USER11> Racism toward the people in power can't exist, is the usual argument. But ultimately I can't see how this helps the cause. They are brow beating the only people that are trying.

<comment><USER7> Only people in a position to try, and doing a lousy job so far.

The title is pure clickbait. The article is about the challenges faced by women and trans people of color when they attempt to enter the tech industry. It raises some good points.

<comment><USER10> > Only people in a position to try, and doing a lousy job so far.

This doesn't justify or excuse racist generalizations, and it doesn't qualify the idea that you "can't be racist" towards some people.

> The title is pure clickbait.

They generalize about "white people" multiple times in the article.

> It raises some good points.

It does it using racism and classism and with a lecturing tone. Any good it was supposed to do is completely nullified, not the least of which because it insults the people it purports to educate.</comment></comment></comment></comment>


AOL Instant Messenger developer team laid off, reports NYT | The Verge - <AUTHOR> http://www.theverge.com/2012/3/13/2868404/aol-cutting-over-100-employees-aim-patch <comment><USER2> Good run, guys.</comment>


Python for C++ and Java programmers - <AUTHOR> http://www.aleax.it/Python/accu03_Python_Intro_alex.pdf

<comment><USER2> Should this be labeled (2003)? What's changed since then, any of this that should be revised for whatever's considered the current "stable" Python?

<comment><AUTHOR> It should have been. Still it's the best learning source I've seen, especially when combined with docs.python.org, particularly the tutorial, what's new, and brief tour of the standard library

</comment></comment><comment><USER3> Looks like a good high level intro to the language, but it's a bit dated (2003).

<comment><USER4> I think I saw an old style Python class in there.</comment></comment>


A ZFS developer's analysis of Apple's new APFS file system - <AUTHOR> http://arstechnica.com/apple/2016/06/a-zfs-developers-analysis-of-the-good-and-bad-in-apples-new-apfs-file-system/ <comment><USER3> I would recommend anybody interested in filesystems to watch Jeff Bonwick (ZFS inventor) explain the design of ZFS: [https://www.youtube.com/watch?v=NRoUC9P1PmA](https://www.youtube.com/watch?v=NRoUC9P1PmA). They have a few very nice war stories explaining why they found it useful to have the user data checksummed as well.

<comment><USER4> I might be wrong, but I read APFL's no-checksum decision as founded upon Apple's hardware-and-software strategy. Given their situation, they can decide that checksumming is a hardware problem. They can build their latest devices with ECC, and require customers to only use Apple-developed USB keys. (I would not be shocked to learn that, in late 2017, Apple starts selling USB-C storage keys preformatted as APFS.)

<comment><USER5> Checksumming at that level is a bit pointless, because then you can't repair the data. Instead of being able to recover from a mirror or parity data, all you get is "it's corrupt, oh well".

<comment><USER6> Checksumming at the file system level solves the problem of corruption that occurs off the media (on the bus). The media has checksumming on it that allows detection and recovery from errors that occur on the media itself.

The question I'm sure Apple engineers have were - How often do we see BitRot occurring off media, and is the media that we're deploying sufficiently resistant to bit-rot?

And, with APFS's flexible structure, this is a feature that can be added at a later time. Probably made sense to deliver in 2017 something that was rock solid that they could build on, than to either (A) push out the delivery date, or (B) not fully bake all features of the file system.

<comment><USER7> And here's a nice summary of the bitrot that you see in systems with disks. Just like network devices, anyone with enough gear sees these failures.

[https://www.usenix.org/legacy/event/fast08/tech/full_papers/...](https://www.usenix.org/legacy/event/fast08/tech/full_papers/krioukov/krioukov_html/main.html)

</comment></comment></comment></comment></comment><comment><AUTHOR> Fun fact: Dominic Giampalo (who wrote the BeOS file system) is on the APFS team. His book "Practical File System Design" is an excellent description of a traditional UNIX file system design. May be out of print now but I think used copies turn up on Amazon.

<comment><USER8> > Fun fact: Dominic Giampalo (who wrote the BeOS file system) is on the APFS > team. His book "Practical File System Design" is an excellent description of > a traditional UNIX file system design. May be out of print now but I think > used copies turn up on Amazon.

It looks like he has a PDF up on his website:

[http://www.nobius.org/~dbg/practical-file-system- design.pdf](http://www.nobius.org/~dbg/practical-file-system-design.pdf)

<comment><USER7> He's had it up at that url since 2004, with a major update in 2008... says the Wayback Machine.

</comment></comment></comment><comment><USER9> APFS is slated for a 2017 release, yet development started as recently as 2014. By comparison, development on Btrfs started all the way back in 2007, yet many still consider it to be unsuitable for widespread deployment, particularly in mission-critical settings.

If Apple can actually pull off this turnover so quickly, does that suggest the complaints about Apple's declining software engineering quality were overblown?

Edit: Ted T'so in this talk(1) (at the 8 min mark) discusses the taskforce that birthed Ext4 and Btrfs and its estimate (based on Sun's experience with ZFS, Digital's with AdvFS) of 5-7 years at a minimum for a new file system to be enterprise ready--an estimate which definitely proved optimistic with regard to Btrfs. Will APFS be different?

1: [https://www.youtube.com/watch?v=2mYDFr5T4tY](https://www.youtube.com/watch?v=2mYDFr5T4tY)

<comment><USER10> [Disclosure: I work for SUSE]

> an estimate which definitely proved optimistic with regard to Btrfs.

SUSE has had Btrfs support for enterprises since SLE11SP2[1,2] (2012). And it's been the default for the root partition since SLE12[3] (2014). So it wasn't overly optimistic at all, it was actually a very accurate estimate. The same support apples for openSUSE, but I think they had it for longer.

[1] [https://www.linux.com/news/snapper-suses-ultimate-btrfs- snap...](https://www.linux.com/news/snapper-suses-ultimate-btrfs-snapshot- manager) [2] [https://www.suse.com/communities/blog/introduction-system- ro...](https://www.suse.com/communities/blog/introduction-system-rollbacks- btrfs-and-snapper-sles-11-sp2/) [3] [https://www.suse.com/documentation/sles-12/stor_admin/index....](https://www.suse.com/documentation/sles-12/stor_admin/index.html?page=/documentation/sles-12/stor_admin/data/sec_filesystems_major.html)

<comment><USER11> Do you have any good source to point people at when they say btrfs is unstable and corrupts data? (and I mean, either positive or negative data, I don't want to be biased either way) I'm kind of tired of people posting comments like that, which are based on "google is full of people saying this", or "I know one person who lost data" (but no idea which kernel was it - could well be from 2014).

If I look at btrfs patches on lkml, on one hand side I can see some fixes for data loss, but on the other they're usually close to "if you change the superblock while log is zeroed and new superblock is already committed and there's a power loss exactly at the point, you'll get corruption" \- which are just really obscure edge cases people are unlikely to ever hit.

So what can I look at to get a realistic picture of what's going on? (what would SUSE point me at)

(for the negative results, I know of the recent filesystem fuzzing presentation where btrfs comes out worst, but honestly I don't consider it interesting for real world usage - car analogy, I'm interested how the car behaves on a typical road, not which fizzy drinks added to the gas tank will break it)

<comment><USER12> RE: the fs fuzzing, David Sterba a Btrfs developer @ Suse replied to this: [https://www.spinics.net/lists/linux- btrfs/msg54454.html](https://www.spinics.net/lists/linux-btrfs/msg54454.html)

As for unstable and corrupts data, this is just not true. Many more users using it on stable hardware don't have problems. I've used Btrfs single, raid0, raid10 and raid1 for years, and haven't had corruptions at all ever that I didn't myself induce.

I have stumbled upon, just days ago, parity corruptions in raid5 however. The raid56 stuff is much much newer and hasn't been as well tested, and has been regarded as definitely not ready for prime time. So that's a bug, and it'll get fixed.

Bunches of problems happen on mdadm and LVM raid also due to drive SCT ERC being unset or unsupported, resulting in bad sector recovery times that are longer than the kernel will tolerate. That results in SATA link resets, rather than the fs being informed what sector is bad so it can recover from a copy and fix up the bad sector.

So there are bugs all over the place, it's just the way it goes and things are getting quite a bit better.

It is totally true that Apple can produce their own hardware that doesn't do things like lie to the fs about FUA or its equivalent of req_flush being complete when it's not, or otherwise violating the order of writes the fs is expecting in order to be crash tolerant. But we're kinda seeing Apple go back to the old Mac days where you bought only Apple memory and drives, 3rd party stuff just didn't happen then. The memory is now soldered on the board and it looks like the next generation of NVMe and storage technologies may be the same.

Windows and Linux will by necessity have file systems that are more general purpose than Apple's.

</comment></comment></comment></comment><comment><USER13> You might also find value in the comments from when this was published via Adam Leventhal's blog -

[https://news.ycombinator.com/item?id=11934457](https://news.ycombinator.com/item?id=11934457)

</comment><comment><USER14> What I found most interesting about the review is that Apple chose not to implement file data checksumming for the reason that the underlying hardware is very 'safe' and already employs ECC anyway.

<comment><USER15> Serious question: what value does filesystem checksumming offer for the average user who has no redundancy?

I mean, it'll tell you that your only copy of a file got corrupted, but it'll still be corrupted...

<comment><USER16> The average user might have no redundancy, but they still ought to have a backup. Checksum failure tells them they need to restore.

At the very least, a checksum failure might tell them (or the tech they're consulting) that they have a data problem, rather than, say, an application compatibility problem.

<comment><USER17> "Why is my machine crashing?" "Well, somelib.so is reporting checksum failures" is a much better experience then "weird, this machine used to be great but now it crashes all the time"

<comment><USER18> somelib.so what? And what's a "checksum"? Error messages need to be comprehensible to the average user.

<comment><USER19> "Error: Buy a new Mac."

<comment><USER18> Assuming your intent is not to troll: "The file xyz.txt is corrupt. Click here to restore from a Time Machine backup."

</comment></comment></comment></comment></comment></comment></comment><comment><USER20> The author is apparently proud of the fact that they have "literally never seen or heard of [the OS X document revision system] until researching this post". The dismissive tone of the entire article is hard to stomach. Ars has really gone downhill since the departure of John Siracusa.

<comment><USER21> John Siracusa's departure was after Ars already had gone downhill significantly. Also he was not an editor or regular contributor outside of his OS X reviews, so his departure doesn't have much effect on the day to day. It's more of a symptom rather than a cause.

The cause is the change in editorial standards for the site which has been decreasing for years, and so every year is at a new all time low. Not to mention the really invasive ads and even sponsored content.

<comment><USER22> People have been complaining about the editorial standards continuously since at least 2000, but that doesn't magically lend more weight than any other subjective assessment unsupported by evidence.

As for advertising, how do you expect them to hire writers and editors or run a high-traffic website without advertising? They've offered paid subscriptions for years and subscribers don't see ads at all but not enough people have taken them up on it.

<comment><USER21> Give me a break, I am not complaining about advertising period. Ars has always had advertising and it used to be fine. But over time the ads have gotten more and more intrusive. There's more and more auto play ads with sound, and ads that take over part of the screen, etc.

The lack of subscriptions are probably a symptom of the editorial quality going down. You can say there's no evidence of it, but when lots of people complain about it that _is_ evidence of it. I was a long time Ars reader and the b.s. finally got too thick that I have stopped going to the site regularly in the last year.

It used to be a site for intelligent, balanced articles about tech. Now it's got shills like DrPizza who basically just reprint whatever Microsoft's PR department emails him.

</comment></comment></comment></comment><comment><USER23> One of the (many) things that struck me in the article was his dismissive remarks regarding copy on write within a filesystem. That seems like a fantastically useful feature for development, etc, where you are almost always building, then copying the artifacts into the deploy/test directory. Avoiding the disk IO in those situations seems like a pretty sound win to me in terms of giving a performance boost for free.

<comment><USER24> > _was his dismissive remarks regarding copy on write within a filesystem_

? ZFS is copy-on-write.

<comment><USER23> Yeah, I know ZFS is CoW, I use snapshots all the time when building jails/containers precisely for that reason. I was just struck by a few paragraphs from the article that really stuck out:

> With APFS, if you copy a file within the same file system (or possibly the > same container; more on this later), no data is actually duplicated. > Instead, a constant amount of metadata is updated and the on-disk data is > shared. Changes to either copy cause new space to be allocated (this is > called "copy on write," or COW). btrfs also supports this and calls the > feature "reflinks"—link by reference.

> I haven't seen this offered in other file systems (btrfs excepted), and it > clearly makes for a good demo, but it got me wondering about the use case. > Copying files between devices (e.g. to a USB stick for sharing) still takes > time proportional to the amount of data copied of course. Why would I want > to copy a file locally? The common case I could think of is the layman's > version control: "thesis," "thesis-backup," "thesis-old," "thesis-saving > because I'm making edits while drunk."

CoW is one of those features that is superficially questionable until you start noticing the bits and pieces of workflow it really makes faster and easier. Given the keynote was towards an audience of developers, I'm really surprised that there wasn't a demo showing how much faster deploys, etc are with such tech.

<comment><USER25> He is dismissive of file-level CoW. ZFS will also create new blocks if you cp a file, only dedup will remedy that. Also APFS only seems to do file-level CoW with a special syscall.

<comment><USER26> Note he was also not even aware of support for the same feature in btrfs until I pointed it out to him in an earlier HN discussion of his original blog post -- the version on Ars already pretends otherwise, without being marked as a fixup (though I am ready to blame this on Ars fully; his blog version includes an UPDATE tag and has better flow).

</comment></comment></comment></comment></comment><comment><USER27> > APFS addresses this with I/O QoS (quality of service) to prioritize accesses > that are immediately visible to the user over background activity that > doesn't have the same time-constraints. This is inarguably a benefit to > users and a sophisticated file system capability.

Could someone clear up how this is can be determined on the filesystem rather than scheduler level (I suspect it cannot be, or the article is making bogus claims)?

<comment><USER28> Why couldn't they be? Those are in the same process and can talk to each other.

HFS+ implements scheduling of background QoS threads, especially on HDDs - <USER2> can see it working with 'spindump'.

</comment></comment><comment><USER29> Do they explain the reason why they needed a new filesystem instead of using some existing one (same OpenZFS)?

<comment><USER30> We can speculate on the management decision, but from an engineering point of view Dominic Giampaolo said that he didn't dive into btrfs, ZFS, or HAMMER in order to not get potentially bad influence. At least that's how I read his answer as cited by Adam. It's interesting that Larry (Oracle CEO) being a good friend of the late Steve Jobs didn't result in finding an acceptable ZFS licensing agreement for Apple. I mean, they've incorporated DTrace (into the kernel of all things), so why not ZFS as well?

<comment><USER29> _> from an engineering point of view Dominic Giampaolo said that he didn't dive into btrfs, ZFS, or HAMMER in order to not get potentially bad influence._

I'm not sure I quite understand this approach. It's sounds like a pure NIH kind of method (which is admittedly common for Apple). I.e. of course one can always try to reinvent the wheel, but why is it bad first to analyze what already exists and to evaluate good / bad sides of that? Or his approach is simply always to make everything from scratch and not to look at anything else?

<comment><USER31> >why is it bad first to analyze what already exists and to evaluate good / bad sides of that?

It's a legal defense strategy.

ZFS is covered by multiple patents.

If someone who have never read about any of ZFS's designs and patents independently reproduces one of ZFS's patented features, then the courts could rule that that was non-infringement.

That's why the author said of Giampaolo: "...but didn't delve too deeply for fear, he said, of tainting himself". Reading too much about a patented product effectively taints yourself from being able to freely create your own designs.

<comment><USER32> Independent reinvention is not in general a defense to a claim of patent infringement. Is there some fairly specific set of circumstances you're referring to here?

<comment><USER33> No, but it helps you to avoid the potential treble damages bit for wilful infringement.

</comment></comment></comment></comment></comment></comment><comment><USER34> Because Apple systems are non-expandable, I usually store my large data on external drives (USB, NAS, etc). Any idea if APFS will work on them, either direct connect or iSCSI?

<comment><USER35> File system is usually independent from storage media, with some exceptions

(and I would kind of avoid formatting external drives in "funny" FSs as I might want to read them in some other OSs. Unfortunately this usually means FAT32)

But I do have external media as HFS to work with Time Machine

<comment><USER36> For what it's worth, there's NTFS drivers for Linux and OS X (ntfs-3g). NTFS often gets a mixed reception, but regardless of your opinion of it, it's still massively better than FAT32.

Another option is ext3. The only caveat there is the Windows ext drivers don't fully support ext3 (unless I've missed an announcement). However they do fully support ext2 and ext3 is backwards compatible so you can effective get ext3 support in Windows.

Sadly though, all the good stuff requires 3rd party libraries. It's a real pity everyone can't agree on a standard to replace FAT32. :(

<comment><USER37> Is NTFS support good enough for writing? I haven't kept up. At one time you could read NTFS on Linux well enough, but writing was not supported (or was "experimental") I think mostly because the permissions attributes were quite different from the Unix approach? And also because it was reverse-engineered and not officially endorsed by MS?

Did Microsoft ever open-source the NTFS specs and drivers?

<comment><USER38> This was true 15 years ago, I hope it is not true anymore.

I know there was a way to write to ntfs from Linux, but it required to install ntfs driver file from Windows.

I hope there is a native ntfs support that supports writing.

<comment><USER10> > I know there was a way to write to ntfs from Linux, but it required to > install ntfs driver file from Windows.

That isn't true. You need to use ntfs-3g, which is a free software implementation of NTFS (that allows both reading and writing). It's been stable for 10 years. Using NTFS doesn't require anything from windows and doesn't require proprietary software.

<comment><USER38> It is true.

I stated how it was in the past[1]. Anyway according to [1], it looks like ntfs-3g still uses a proprietary version of ntfs.sys

[1] [http://superuser.com/questions/139452/kernel-ntfs-driver- vs-...](http://superuser.com/questions/139452/kernel-ntfs-driver-vs- ntfs-3g/357949#357949)

<comment><USER10> I just downloaded the source for ntfs-3g[1], and it doesn't appear to have any binary blobs in it. In addition, it's under the GPLv2 so integrating proprietary components is unlikely to be legal. The answer you linked quite clearly says that the company _offers_ a proprietary _version_ of ntfs-3g. The answer does _not_ say that ntfs-3g is proprietary.

Also, Trisquel (an FSF-approved GNU/Linux distribtion, meaning that it doesn't have any proprietary software within 100km of the distro) has packages for ntfs-3g[2]. So it's _definitely_ entirely free software.

So again, you're wrong on this point. In addition, I strongly believe that you were never correct on this point. Maybe you confused ntfs-3g with the proprietary version that company sells?

[1]: [http://www.tuxera.com/community/open-source- ntfs-3g/](http://www.tuxera.com/community/open-source-ntfs-3g/) [2]: [http://packages.trisquel.info/search?keywords=ntfs&searchon=...](http://packages.trisquel.info/search?keywords=ntfs&searchon=names&suite=belenos&section=all)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER39> Great article!

 _Apple contrasts [space sharing] with the static allocation of disk space to support multiple HFS+ instances, which seems both specious and an uncommon use case._

Really? I depend on this use case every year to safely test drive pre-release OS versions.

<comment><USER40> That use of multiple partitions I think qualifies as uncommon which is not to suggest that it's not useful.

<comment><USER41> Extremely commonly used by engineers within Apple, however =)

</comment></comment></comment><comment><USER42> The original article was on HN's front page when it was originally written.

</comment><comment><USER43> > A ZFS developer's analysis of the good and bad in Apple's new APFS file > system

What about linking to the actual source rather than a 3rd party: [http://dtrace.org/blogs/ahl/2016/06/19/apfs- part1/](http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1/)

<comment><USER30> Well, the article on Ars seems to be authored by Adam, so it's probably a cleaned up single piece of his blog posts.

<comment><USER44> Correct—I reached out to Adam to ask if Ars could syndicate the piece, and then I did some minor cleanup and clarification editing on it (mostly style conformance, but also some minor grammar tweaks and a few sentence re-writes). After checking with him to make sure my changes didn't change anything substantive, we ran the piece this AM.

Link wherever you'd like, of course, but the more traffic this pulls in, the more ammo I have to be able to get Adam contributing to Ars as a regular freelancer!

(edit - hi, adam!)

(edit^2 - corrections corrected. Apologies for the errors. I am just a simple caveman. Your mathematics confuse and annoy me!)

<comment><USER30> If you want to publish more ZFS articles, I'd love to read about following topics from Adam and/or members of OpenZFS:

\- using DTrace for ZFS operations insight

\- state of OpenZFS, comparing illumos, FreeBSD, NetBSD and Linux

\- myths and/or often cited problems (ECC, COW unfit for DB load, VM images etc.)

\- ZFS version and feature flags wrt portability between illumos and FreeBSD versions

\- pool flexibility work that will make it easier to remove devices

\- comparison to btrfs, hammer{1,2} and flash filesystems

\- the topic of zfs pointer rewrite

\- garbage collection and safely removing traces of a file/directory in a COW fs

\- rebalancing story compared to HAMMER and future work in this space

\- built-in ZFS encryption (independent of system crypto volume support)

I should say that I'd only support an article like that if Ars allows parts of the written text to be incorporated into OpenZFS wiki/documenation.

<comment><USER44> Some of these are probably a little too deep to get much traction on the ars front page, but there are some solid ideas here (especially the oft-sited problems one). Thanks for the feedback!

We _did_ run a big piece by Jim Salter a couple of years ago on next-gen file systems that focused on ZFS and btrfs ([http://arstechnica.com/information- technology/2014/01/bitrot...](http://arstechnica.com/information- technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/)), but yeah, I'd love to have more filesystem-level stuff showing up. The response is generally very, very strong—turns out people really like reading about file systems when the authors know what they're talking about!

edit -

> I should say that I'd only support an article like that if Ars allows parts > of the written text to be incorporated into OpenZFS wiki/documenation.

That's more complicated, unfortunately. I am not a lawyer etc etc and I am only speaking generally here, but Ars and CN own the copyright on the pieces we run (though syndications like Adam's piece today are different), and wholesale reuse of the text without remuneration isn't something that the CN rights management people like. Fair use is obviously fine, so quoting portions of pieces as sources in documentation is not a problem, but re-using most or all of something isn't (necessarily or usually) fair use.

(again, not a lawyer, my words aren't gospel, don't take my word for it, etc etc)

<comment><USER30> I'm also not a lawyer, but my thought process is like this: in the open source spirit, given this is not a book to be profited from, and profiting from technical books is very hard anyway, developers of some software could contribute technical content which then instead of compensation would only get editor time and in return be allowed to be included in the project's documentation. Real World Haskell, Real World OCaml somehow managed to convince the publisher this is fine. Again, IANAL, just thinking out loud.</comment></comment></comment></comment></comment></comment>


Leveraging machine learning to fuel new discoveries with the ArXiv dataset - <AUTHOR> https://blogs.cornell.edu/arxiv/2020/08/05/leveraging-machine-learning-to-fuel-new-discoveries-with-the-arxiv-dataset/ <comment><USER2> This is a long time coming. Glad that they've done the lifting to expose these resources to higher-order analysis.

I took a look at this potential a couple of years ago but on PubMed:

[https://techbio.org/b-tracing-psych-signals- lit.php](https://techbio.org/b-tracing-psych-signals-lit.php)

</comment><comment><USER3> Any potential for something like this to be added to gpt-4's training data set. Being far from an expert, I assume the figures and any non-text data could pose a problem, but it also seems like providing it with a huge high- quality source of scientific reasoning could lead to some pretty amazing and revolutionary results. gpt inspiring research directions or even coauthoring. Anyone who knows more about this have any thoughts? Have I just drunk the gpt cool aid or is their some potential for it to revolutionize science quite soon?

<comment><USER4> My money is on the next iteration of gpt being multi-modal (See image-gpt). So this kind of thing would fit right in. I don't think this would lead to the kind of scientific revolution you are thinking of, given the tendency of gpt to confabulate things instead of basing itself on facts. That may be entering cool-aid territory :)

</comment></comment><comment><USER5> For anybody on mobile thinking the link has redirected to the index, you need to scroll way down to read the actual article.

<comment><USER6> Thank you

</comment></comment><comment><USER7> The value of content tagging (e.g., PhysML) and keyword tagging (e.g., ScienceWise) is apparent in aggregate, like for searching. That benefit is to consumers while the burden is currently on the content creators.

I don't know of any incentives within academia or grant processes that would motivate content authors to tag their content. With the exception of creators of the tagging systems. That means bulk analysis (whether using a grammar or natural language machine learning method) is key.

Citation graphs (which is mostly what's been done previously[0]) pale in comparison to text analysis. The possibility of enabling complex searches would be a big leap forward in science.

[0] [https://physicsderivationgraph.blogspot.com/2020/05/literatu...](https://physicsderivationgraph.blogspot.com/2020/05/literature- review-for-using-arxiv-as.html)

</comment><comment><USER7> There have been efforts to tag keywords in arXiv [0] and to identify sections of articles [1]. A conference on Mathematical Knowledge Management [2] was held last week; some participants have already been analyzing ArXiv.

Hopefully integration with Kaggle expands the number of teams taking advantage of the knowledge in the corpus.

[0] [http://sciencewise.info/](http://sciencewise.info/) [1] [https://github.com/OMdoc/OMDoc/wiki/PhysML](https://github.com/OMdoc/OMDoc/wiki/PhysML) [2] [https://cicm-conference.org/2020/cicm.php](https://cicm- conference.org/2020/cicm.php)

</comment><comment><USER8> This is really cool. I worked on the AI Index Report and had to bug karpathy to get his copy of arxiv papers to do analysis. (He's been slowly collecting papers for arxiv sanity for years). Getting them all from the arxiv API would have taken months.

This will enable tons of useful stats gathering about the fields represented on arxiv. Hopefully it will also lead to new scientific insights as well!

<comment><USER9> arXiv also offers bulk access to the papers, you can download them via S3 'requester pays'

[https://arxiv.org/help/bulk_data](https://arxiv.org/help/bulk_data)

</comment></comment><comment><USER10> Very cool, I really look forward to seeing science move forward along these paths. All the same - have to post the obligatory (and recent) xkcd rebuttal- [https://xkcd.com/2341/](https://xkcd.com/2341/)

<comment><USER11> I'm a scientist and my problems are a lot more like the first panel than the second.

<comment><USER12> I think this is the second time Randall has made that joke: [https://xkcd.com/1831/](https://xkcd.com/1831/) . I wonder if he has a grudge against data scientists or something

<comment><USER13> I think it's aimed at people who make assumptions that all other fields are just a subfield of their field</comment></comment></comment></comment>


Mathematical Foundations of Computer Networking - video lectures - <AUTHOR> http://blizzard.cs.uwaterloo.ca/keshav/mediawiki-1.4.7/index.php/Lecture_videos

<comment><AUTHOR> Although the title says "Computer Networking", I think this video lectures also covers some of the essential maths for A.I. and machine learning.</comment>


Sublime Text packages for Front-end web development - <AUTHOR> http://martineau.tv/2014/07/sublime-text-for-front-end-development/

<comment><USER2> Thanks, there are a lot of overlapping packages out there and it helps to see a working collection.

And does anyone have suggestions on D3.js packages that aren't just code snippets? Sublime Text 2 has been great so far for JS but it could be more specific.

</comment><comment><USER3> I would be curious to see a similar list, but for Atom</comment>


A case study of the tactics of climate change denial, in which I am the target - <AUTHOR> http://blogs.discovermagazine.com/badastronomy/2012/02/02/a-case-study-of-the-tactics-of-climate-change-denial-in-which-i-am-the-target/ <comment><USER2> Briggs responds: <http://wmbriggs.com/blog/?p=5154>

My experience is that usually in a contentious debate like this, the person who can keep a cooler head is right. It's not a perfect heuristic but it's a good starting point. Galileo can afford to be calm whereas Cardinal Bellarmine has to rant and rave.</comment>


Twitter to Sell 50% of All Tweets for $360,000 a Year Through Gnip - <AUTHOR> http://www.readwriteweb.com/archives/twitter_to_sell_50_of_all_tweets_for_360kyear_thro.php

<comment><USER2> Let's see, at the 1000 tweets per second figure from the article that's 365 * 24 * 3600 * 500 = 15,768,000,000 tweets for $360,000. Or 0.002 cents per tweet. So 500ish tweets are worth about a penny, and your $360K would be for 15 gigatweets which would weigh in at about 2.2 TB[1]. So my tweets aren't worthless, they're just very very cheap.

[1] That's a hard drive manufacturer terabyte, not a real one

<comment><USER3> Sweet, so my account is worth about a whole $1 in tweets

<comment><USER4> That is $1 per sale :). Maybe your account is worth $10-$20 total.

</comment></comment></comment><comment><USER5> We really need (and will inevitably get) an open, distributed protocol for status updates. It's insane for everything to be routed through one (or 2, or 3) companies.

<comment><USER6> Who would host the servers though? A standards organization? I doubt they could afford it.

That, or it will be like email, where everyone has @example.com appended to their username.

Neither situation strikes me as desirable.

<comment><USER7> In the future, it's quite likely that a standards organization _could_ afford it.

Processor speed, disk capacity, network bandwidth, and available software are all growing much more rapidly than online populations.

In some years' time I might be able to run an operation like Google, Facebook, or Twitter from my bedroom.

<comment><USER8> The current Google perhaps, but not the Google of the future.

<comment><USER7> I'm not sure I _want_ the Google of the future.

</comment></comment></comment></comment></comment><comment><USER9> The key value is not as much in the data itself, as much as the _timeliness_ of the data. Access to the halfhose allows you to answer a _very_ valuable question:

"What's happening _right now_?"

This question is worth a lot of money, and something that doesn't have a good algorithmic solution(e.g. Google News.) Twitter is probably the only company that has a privacy-compliant solution to this, hence making it a very monetizable product.

<comment><USER10> But Twitter already provides an API for that. You can ask what are the current top 10 trending topics at any time.

</comment></comment><comment><USER11> _All these platforms,..., they all realize that the data they have is extremely valuable to everyone from API partners to marketers .... I think all these companies could see that there's more money in data services than there could be for them in advertising._

The value would be back into the API and not into weird sponsored trendic topics. Twitter seems to go back towards Alex Payne's vision (data hose platform) and away from Biz Stone's (twitter as a media with _celebrities_ etc...). They could also set up separated Twitters Hoses: like there could be automatic sensors data input for the Internet of things for instance, separated from human input. Any link where Twitter guys are speaking of this?

</comment><comment><USER12> What are the technical measures to prevent people to just scrape 100% of the tweets, if any?

<comment><USER13> Simple rate-limiting / DOS prevention would be plenty given the volume of tweets.

<comment><USER14> Not that hard to get around. IPs are easy to come by.

<comment><USER15> I disagree. Twitter is pushing upwards of 3000 tweets per second [1], it'd be nearly impossible to scrape at that rate and go without notice, regardless of how many IPs you've got at your disposal.

[1] <http://mashable.com/2010/06/25/tps-record/>

<comment><USER14> You don't need to scrape for each tweet. Scrape every active user every couple hours/days depending upon their tweet frequency. Not much point in having a database up to date every second since processing it in real time is almost impossible anyway.

Twitter would notice a major scraping operation, but if it's done correctly they wouldn't be able to distinguish between user IPs and bot IPs.

edit: Barracuda already did more than 10% of users just for a white paper: [http://www.barracudanetworks.com/ns/news_and_events/index.ph...](http://www.barracudanetworks.com/ns/news_and_events/index.php?nid=387)

150,000,000 registered users only takes 170 days at 10 users a second for a first pass. Focus on frequent tweeters for subsequent scrapes. Even among the ~20% of twitter accounts that are active, most don't need to be scraped daily, and the most active accounts are likely spam.

<comment><USER16> > Scrape every active user

translates to "Scrape every user" unless you know of some magical way to get list of "active" users.

Guess howmany active users there are? Guess how many servers you need running to get through those in 1 hour. My guess is something much more than $360,000 worth / yr.

<comment><USER17> _> unless you know of some magical way to get list of "active" users_

look for users who tweeted in last X days? also look for their repliers+buddies, since they too are likely to be active. doesn't seem hugely complicated to me?

<comment><USER16> > look for users who tweeted in last X days?

Requires you to check aka scrape __every __user to see which ones tweeted in last X days.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> Does this include private tweets and DMs?

<comment><USER19> In the past, firehose access has meant public tweets only. I presume Gnip is getting the same data as the Library of Congress and other firehose consumers, but details haven't been spelled out yet.

</comment></comment><comment><USER10> I work for a media aggregator, and we do a lot of different media (microblogs like twitter included). We index close to 100M documents a month. We also charge a fraction of 360k/yr. I'd like to know who the target market is?

Anyone here from people genuinely ready to spend that kind of cash for 50% of Twitter?

</comment><comment><USER20> I'm trying to find my "privacy outrage" / "who actually owns your content" soapbox but having trouble.

In fact, I find it kind of refreshing that Twitter is just flat out saying all your base belong to us, and anybody that wants it gets access for $360,000 a year.

This in contrast to Google, Facebook, Yahoo etc that muddy the waters whenever it comes to how they're actually using / sharing your data.

<comment><USER21> I'm not sure how you'd manage "privacy outrage" over Twitter messages, since they're almost always public.

<comment><USER20> The concept of "Privacy" means to me that the average user is aware of how their profile, their content, etc is going to be used by the host.

There's a difference between the perception of your feed being public, and Twitter selling your feed data to a corporation to utilize for targeting, advertising, reselling to employment agencies in the future long after your stupid teenage profile is deleted, whatever.

When a solid 10% of the users are kids (and a much much larger percentage is entirely clueless) it's worth questioning and the people that do know what's actually going on have a responsibility to ask questions.

That's my best attempt to get fired up about privacy on twitter, you forced my hand - oh won't you please think of the children?

Ref twitter age - [http://royal.pingdom.com/2010/02/16/study-ages-of-social- net...](http://royal.pingdom.com/2010/02/16/study-ages-of-social-network- users/)

<comment><USER22> I'd rather not get into a discussion of what "privacy" means, but Twitter (and Facebook) are almost certainly violating most of their users' perceptions of what Twitter does with their tweets. That said, if they anonymize the tweets and hold the buyers contractually obligated not to de-anonymize them, I think most users would be OK with that. Each user owns their tweets, as does Twitter, but only Twitter owns all of the tweets.

<comment><USER21> It's a broadcast medium. People literally keep score with each other about how many people they can get to follow them, and how many people they can get to RT their messages. I think you're simply dead wrong about this.

<comment><USER23> 1\. Reality and perceptions don't always collide. Like the people that post messages on Facebook about calling into work sick when they aren't... only to have their boss read the message and fire them over it.

2\. What about the people whose Twitter accounts are private?

3\. The people 'racing' with each other for followers or retweets are by definition more public than most other people. Unless you are going to claim that all or most Twitter users fall into that category. Trying to use them to categorize the user base of Twitter as a whole seems a bit off.

4\. Twitter _is_ a broadcast medium, but what we are talking about are the perceptions of the people using it, not the reality of the situation. There are plenty of people that broadcast stuff publicly that they wouldn't want their parents to read. Why would they do so? "My parents aren't on Twitter." I'm sure the same thing applies to bosses and the workplace.

</comment></comment></comment></comment></comment></comment><comment><USER24> I think a distinction needs to be made, I agree that a company using this many of the Tweets as part of a commercial effort should pay.

I also think that a research effort with the ability to process 100% of Tweets can most likely afford to pay. Something like the 5% though, I think a great deal of research can be done on the Twitter platform may be prevented because of the cost. For research maybe you could charge the bandwidth required to deliver the stream, no idea what kind of ballpark this would be in.

<comment><USER25> _I also think that a research effort with the ability to process 100% of Tweets can most likely afford to pay_

Processing 100% of all tweets is not actually very hard or expensive: 2000 messages/sec is small potatoes if you want to do it in realtime, and even easier if you are just doing batch analysis queries. You could do it (with reasonable performance) for much, much less than $360,000/year (let alone 2x or 3x that for the 100% feed).

<comment><USER24> Not that amount, but afford to pay some amount for access, obviously for the massive full price little if any research that can't be monetized will be conducted. It's a net loss for us all and no gain for Twitter.

</comment></comment></comment><comment><USER26> This is pretty interesting. There are a lot of interesting projects and data analysis that come from analyzing tweets but the price tag seems far beyond what any of these would be willing to pay.

Previously some serious time would be spent scraping content from the feeds but given that it's only 2% of the content and would take months (at least) to gather a significant amount makes it less than ideal. Although I would assume for a vast majority of cases, months are worth less than $360k.

</comment><comment><USER27> isn't there already a free stream for new tweets that could be stored and analyzed? is this just a sale of the 50% of tweets that have already passed? why so pricey for historical market research data when current data is free?

<comment><USER28> From my understanding, the current data is not free. Yes, it's fully accessible, but to get access to the whole thing you would need to either scrape the site which Twitter will most likely disallow, or use the API, whose access Twitter controls.

</comment></comment><comment><USER29> jrockway to post 50% less content to Twitter for free.

</comment><comment><USER30> That's a lot of tweets about Justin Bieber!

</comment><comment><USER31> i'm selling my gtalk, Y!m and Skype statuses. $10000/year. Anyone interested?

</comment><comment><USER32> Anyone keen to discuss this issue further? Join the live chat at <http://twich.me/twitter360k>

</comment><comment><USER33> I can't believe anyone would pay coin for that crap. Might as well get one year of the NYT crossword puzzles for the same price, it's a better value. Or randomly generate "I am having a coffee with my cat" in all possible combinations.

P.S. if this just a "display" does it mean there is no any value to the stale links?

<comment><USER34> You missed the point of the whole thing. This is not about displaying tweets, it's about the ability of a analyzing trends, news, popularity, etc. via tweets. It's a huge deal to be able to get 50% (let alone 100%) of all the tweets on a given moment.

<comment><USER35> Not only is it not about displaying tweets, it's specifically disallowed:

> Customers will only be allowed to analyze the messages, not display them</comment></comment></comment>


NYC Subway REST API - <AUTHOR> http://www.nycitypulse.com/

<comment><USER2> It is possible to get this data directly from the MTA: [http://datamine.mta.info/](http://datamine.mta.info/)

It is a good bit messier than this, but it is free, unlike this site. And after all, you never know how long sites like these will be around.

<comment><USER3> It appears the point of this service is to make it less messy. For that convenience, there's a charge. And now we've both stated the obvious.

<comment><USER2> I didn't think that the URL for the MTA's own data source was particularly obvious, so I stated it. Apologies if doing so somehow offended you.

<comment><USER3> HN contributors have a great habit of creating something (some sort of MVP) and charging for it. HN commenters have a habit of immediately pointing out the free way of doing something that takes more time, skills, or effort. It's not offensive, but it gets old and it's not very supportive of the contributors.

<comment><AUTHOR> Thanks @robbiemitchell for appreciating my work. I put many hours and a lot of effort to make this project happen, and as a student, need a way to support myself so I can continue to grow it. So @untog, it is a little offensive when you just ask contributors to give out their work for free.

</comment></comment></comment></comment></comment><comment><USER4> Another project that does this - [https://github.com/jonthornton/MtaSanitizer](https://github.com/jonthornton/MtaSanitizer). Free and self-hosted.

</comment><comment><USER5> Putting your API on the same hostname and presumably the same codebase as your website is not a great long-term strategy. And mentioned before, consider HTTPS to protect the API key.

<comment><AUTHOR> Thank you johndunhart for the advice, I will definitely take into account this in the next iteration of the product.

</comment></comment><comment><USER6> Does google maps/citymapper consume this sort of real-time train location data?

If not, perhaps you could use the NYC subway REST API to cross reference route calculations

<comment><USER7> The real-time feed and static schedule are both available from the MTA directly: [http://datamine.mta.info/](http://datamine.mta.info/)

Both are in GTFS format, and AFAIK Google does consume it.

</comment></comment><comment><AUTHOR> This API uses the same source of real-time data as Google Maps and CityMapper

</comment><comment><USER8> Looks like the Cities->Want to see your city here? menu link is broken

</comment><comment><AUTHOR> Sorry about that. What city would you like to see ?

</comment><comment><USER9> it uses http so anybody can snoop api key ?</comment>