Small guys find most breakthroughs (Letter to FT) - <AUTHOR> http://www.ft.com/cms/s/0/c49b785e-8a48-11e0-beff-00144feab49a.html

<comment><USER2> Here's the letter, liberated from the reg-wall:

From Mr Mano Manoharan.

Sir, In "Getting IP right" (Editorial, May 19) you say: "Giants must not stop others from standing on their shoulders" if we are all to benefit from the advancement of ideas.

My reading of the history of UK innovation would suggest individuals and small- and medium-sized enterprises are the source of more breakthrough invention, rather than giants.

So practical action, as put forward by Ian Hargreaves, to encourage more SMEs to begin the process of innovating should be welcomed. As should action to discourage giants poaching intellectual property from vulnerable SMEs.

I speak from experience, having funded a successful breakthrough and now patented innovation, that rationalises the ink used to print packaging, only to see giants shamelessly trying to copy us, and even being told chillingly by one; "We will drown you in court fees."

My shoulders certainly need a cheap way of protection from the weight of these particular giants!

Mano Manoharan, Joint Managing Director, LFH illuminating brands, London NW1, UK

<comment><AUTHOR> Thank you for posting this. I had thought that letters were not put behind the paywall.

<comment><USER2> They aren't, but you still have to register.</comment></comment></comment>


Security Implications of JIT Compilation - <AUTHOR> http://zee-nix.blogspot.com/2011/06/welcome-to-virtual-world.html

<comment><AUTHOR> This is strange, I thought I submitted [http://wingolog.org/archives/2011/06/21/security- implication...](http://wingolog.org/archives/2011/06/21/security-implications- of-jit-compilation)</comment>


The science behind piracy: Guilt portion of the brain fails to fire - <AUTHOR> https://www.monash.edu/news/show/the-science-behind-piracy <comment><AUTHOR> What makes this interesting, for me, is the phrase _" these findings suggest that people are more willing to break laws and moral rules surrounding intangible property because they have more problems with representing intangible objects"_.

I would leave out "and moral rules". A possible, IMO better, interpretation is that we do not feel copying intangible stuff is morally wrong.

It would be interesting to check what our brains do when we think of stealing electricity, say by charging our car from a neighbor's power outlet. I have the feeling that we find that more tangible and thus more morally wrong.</comment>


Show HN: Clubhouse – project management built for software teams - <AUTHOR> https://clubhouse.io <comment><USER2> We're currently using Clubhouse (and very happy with it), and one thing it does extremely well is the Github integration; you can assign steps in your workflow to events in Github, including merges and pull requests. For example, in our setup, if you tag a commit or branch with a Clubhouse issue number, then the issue automatically moves into "in development", then if we open a PR from a branch to our development main branch, it moves into "ready for review"; once the PR is closed it moves into "ready for deployment". As a developer, it means you don't have to micro-manage your issues most of the time, especially if you have more than 3 steps in your workflow. This is a very nice experience, and I don't think I've seen this in other products (apart from tinkering with APIs and Github webhooks of course).

<comment><USER3> Sounds like a good reason for Clubhouse.

Do you think the same would be possible with Trello connected to Github either directly [http://blog.trello.com/github-and-trello-integrate-your- comm...](http://blog.trello.com/github-and-trello-integrate-your-commits/) or via Zapier?

</comment></comment><comment><AUTHOR> Hey HN, if you want to give Clubhouse a try, you can use this link for a $100 credit: [https://clubhouse.io/signup?referral=hn](https://clubhouse.io/signup?referral=hn)

(I'm a co-founder, also happy to answer any questions.)

<comment><USER4> I'm curious to hear how you would describe Clubhouse relative to the competition. Are there aspects that are fundamentally differentiated? Or is it more a matter of making lots of little improvements, which add up to a much better experience overall?

I ask because project management software is a super crowded market, with a lot of products that seem only subtly different from each other. On the surface Clubhouse also seems like it's only subtly different from other products. I'm curious about strategies for navigating these kinds of markets. Is high-level differentiation useful? Or are the details more important? Or is it really all about marketing and branding?

Or, if you guys just wanted to make something you like and are hoping enough other people like it too so you can pay the bills, and you're really thinking strategically about growth fight now that's cool too.

I haven't really tried Clubhouse yet but it looks really nice. And I certainly have yet to fall in love with anything out there. I will definitely give Clubhouse a try.

<comment><AUTHOR> Good questions! It is a crowded market, but nobody has really nailed it yet. Clubhouse is being built specifically for software teams, and we're trying to provide just enough structure to let you see what's going on at a high level, but keep it flexible enough so that you can work the way your team wants to.

Part of what makes us different is the little details that add up to a better UX, but our GitHub integration is pretty sophisticated: we don't just link stories to pull requests, we can actually automate your Clubhouse workflow using your existing GitHub workflow. We've also put a lot of work into cross- project visibility, so most other tools only let you see one project at a time, so it's difficult to know what a single person is doing across multiple projects, but in Clubhouse it's very easy. We're also starting to get into more interesting features, like providing data-driven guidance around milestones and deadlines, the ability to create roadmaps and more organic story dependencies, more analytics/dataviz... we still have a lot to do.

<comment><USER3> Everyone who visits your landing page thinks 'Trello' and here you try to explain why Clubhouse is advanced (without mentioning Trello) which is not so easy to digest.

Why don't you just create a nice comparison matrix on your site where you compare Clubhouse to Trello feature by feature and highlight the advantages (which are mentioned here and there in your posts in this thread). Would help a lot.

<comment><USER5> It's on the list of things to do at some point soon.

Here's a blog post that one of our users wrote a few months ago that covers some of the same ground: [https://medium.com/humans-in-space/clubhouse-is-the- best-pro...](https://medium.com/humans-in-space/clubhouse-is-the-best-product- management-tool-this-year-3b6f75c093c3#.54h382exp)

</comment></comment></comment></comment></comment><comment><USER3> Trello is free for unlimited users, this thing here is free for 3 users? So what's the point?

We use Trello for years and never had the need to get the paid versions and those are the same price as Clubhouse.

Could somebody enlighten me— _why?_

EDIT: It was predictable that this post is getting downvoted instead of just replying to some valid points

<comment><USER6> Don't downvote this. I want to hear the differences between the products too.

</comment></comment><comment><USER7> We've been using Clubhouse for about two weeks, having transitioned from Asana, which we found lacking in many respects. (Before then, we used Trello, which we also found to be not fit our workflow very well.)

So far, Clubhouse has been working really well. They are emulating the Slack aesthetic pretty hard, and that's not a bad thing. It's pretty responsive and well designed. I find the story card view friendlier than Trello's.

Clubhouse is really a clone of PivotTracker, and I don't mean that as a criticism. They even use the same terminology (points, stories, epics, workflow) and categories (bug, chore, feature). However, there's less of a focus on iterations, and the UI is decidely smoother from my perspective.

There are some features I'd like. For example, I would like their Slack integration to PM me personally about changes; Clubhouse can currently post updates to a Slack channel, but monitoring _all_ the story activity is just not an option when you're on a team working on many different things.

There are also some rough edges in the UI. You can save your story view as a workspace, but if you alter anything (e.g. hide an epic), you have to save it under a new name, then remove the old workspace. I expect stuff like this to improve over time.

<comment><USER8> AFAIK 'points, stories, epics, workflow' are generic 'agile' terms rather than specific to one particular tool.

<comment><USER9> They are, but if you look at Clubhouse's UI there are exact analogues to almost all of Pivotal's UI. Didn't Pivotal invent some of these terms, too? I could be wrong.

<comment><USER5> We tried to use rather generic agile terms.

Here's a reference to most of the same concepts on the Jira site, for instance: [https://www.atlassian.com/agile/delivery- vehicles/](https://www.atlassian.com/agile/delivery-vehicles/)

(I've worked with Pivotal on projects in the past and I worked at Thoughtworks for a while, so we're definitely using language that should be familiar to users that are familiar with the flavor of "agile" promoted by those companies.)

</comment></comment></comment></comment><comment><USER10> We user Pivotal Tracker but at 15 people switching to this would be over twice as expensive.

It looks really nice, I'd love to try it. But we're a small startup (many of the 15 people are part time or consultants) and doubling our expenses on this is more than we can afford.

<comment><USER5> We tried to price things fairly but still make enough to grow and invest in the what we're building.

PT uses tiered pricing, so if you have 16 people we're only $10 more a month than PT and at 26 people we're $40 less expensive each month.

<comment><USER10> Yeah, I'm just an unfortunate edge case where I'm right under the Pivotal price break.

15 * $8.50 = $127.5 vs Pivotal $75 flat. But if I had one more user Pivotal would jump immediately to $150 and Clubhouse would only go up $8.50.

So pretty much you are cheaper at the low end of the ranges but more expensive at the high ends.

Don't get me wrong, this looks awesome! I really want to try it. Maybe when we hit the next user barrier on Pivotal I'll use it as an excuse to switch :)

</comment></comment></comment><comment><USER11> As a former preview user of Clubhouse, I can say that it's easily my favorite work/story manager to date. It gives you what you need for keeping track of your work in an agile environment, and more or less just stays out of your way beyond that. It's totally worth a try if you're fed up with JIRA, Pivotal tracker, et al.

</comment><comment><USER12> Seems like a nice combo of things I've stitched together (at 3 different companies now) with Trello + Zapier + GitHub.

<comment><USER13> I'm interested to know how you identify the trello cards in the commit message? Are you using your own codes to identify them? I'm thinking of building something similar myself.

<comment><USER12> We squash commits before merging so we use the PR as the tie between cards and commits.

</comment></comment></comment><comment><USER14> This feels like a more opinionated Trello, which is not necessarily a bad thing. One oddity on the pricing page. You say the standard plan is $8.50 for unlimited users. Then if you read the small grey text underneath, you see that this price is per user per month. This doesn't really make a lot of sense.

<comment><AUTHOR> Good point, we've removed that line. Thanks for your feedback!

</comment></comment><comment><USER15> After searching through these kinds of products for the past couple weeks, this product looks great and is priced reasonably (unlimited projects!).

Theres a few features it's missing though. It needs a calendar view, daily agenda for scheduling tasks in your work day, and time logging/tracking. These three features would make it perfect for me.

Additionally there's some UX issues in my opinion in terms of how you get from one view to another. For example, if I click on a story, the story modal comes up, how do i get to a view where I can see all the stories for that project? I haven't tested this yet, but it seems that the dashboard doesn't show upcoming due dates, which is one of the most important reasons I would check the dashboard.

All considered, this app still seems great and I'll continue using it for the next few weeks.

</comment><comment><USER16> Based on the description it really sounds great. I didn't really try it, but I intend to, because even though there are a lot of options, unfortunately, they are all lacking. It seems the general topic of work and organizing work is much more difficult then we are willing to admit.

</comment><comment><USER17> Does it cooperate with an issue tracker? I.e., is it easy to convert tickets into tasks?

Also, I just watched the demo gif. To move a task from "unstarted" to "started" you have to drag and drop it? Doesn't that get clumsy when the "unstarted" column is really big (which it will probably be in realistic sitations).

<comment><AUTHOR> You can use an integration like Zapier or our API [1] to automatically generate stories based on some external event. As already mentioned you can update the state within the story dialog, or using our GitHub integration you can partially automate the process by setting up rules like so:

\- When pull request is opened on master, move story to Ready for Review

\- When branch is merged to master, move story to Ready for Deploy

We use these rules ourselves, and they not only reduce process overhead, they make your cycle time data more realistic vs. doing these steps manually.

[1] [https://clubhouse.io/api](https://clubhouse.io/api)

</comment></comment><comment><USER18> This looks like a good alternative for Trello. One thing I am missing from Trello is time estimation / planning / tracking. With a bit of help with extensions you can make it work, but it's not a nice experience overall.

Does this tool has this covered, and/or how do other people deal with this?

<comment><USER5> We have estimation in place, and planning is done at a high level via Epics, but not at the granular level of "you can get 25 points done this week, based upon past performance" (yet).

Related: We have cumulative flow diagrams in place at the Epic level so that you can see where your bottlenecks are and visualize progress.

Suggestions are welcome on how to best add tracking to the system without it being too overbearing.

(I'm also a founder here.)

</comment></comment><comment><USER19> I just want to say that your product looks great and the presentation is excellent. I knew what it was in under 10 seconds. I feel like I already know how to use it, in a weird way.

Definitely trying it out tonight! I'm a single dev, but still could be useful to manage projects!

</comment><comment><USER20> Is it just me, or can you only set up one workflow per organization? Does that mean that you can't have multiple teams/projects using their own styles within one account?

<comment><USER5> It's a constraint imposed, by design, so that you view progress easily across projects. (i.e. view the progress of the frontend, backend, and UI team against a common epic)

Some of our users have extra steps in their workflow (for example, "waiting for app store review") that some of the project teams hide in their workspace (each column in the workflow can be hidden by an individual user).

In the future we might support different "departments" within an organization to allow for multiple ways of working, but it's not at the top of the list of things to do currently.

<comment><USER21> I feel like you're missing out on all the money that big companies with more than a handful of dev teams could be spending on you.

<comment><USER5> We're focusing on teams of 3-150 developers right now.

We have some of the building blocks in place for bigger teams, but it's not a priority right now.

</comment></comment></comment></comment><comment><USER22> Good Product. Been looking for something similar (multi-projects project management) for years, but we would need something to self host due to company policy. Any plans?

<comment><USER5> The software is built so that we can ship an on-prem version at some point in the future, but we don't have the processes in place yet to make that a reality.

Nothing planned yet, but if there's enough interest we'll consider it.

</comment></comment><comment><USER23> Do I need to sign up to try? I see no live demo option? If you want people to try best to provide a no strings attached option (I.e. No account needed).

</comment><comment><USER24> Too many things inside the cloud. Does nobody create software to install on bare metal=

<comment><USER25> It's much easier to dev, test, deploy, and migrate when you are only supporting your own technology stack. It's so much easier than supporting all the random platforms everyone has.

Additionally, the account management side is so much easier. It's a pain in the ass to maintain license servers, and it costs more. If you're shooting for fast early user adoption and/or small and medium size businesses it's a lot less daunting to charge $x/user/month than $1000x + yearly maintenance, or all of the other on premise licensing schemes.

</comment></comment><comment><USER26> $10 (or $8.5) per user per month is relatively expensive.</comment>


Ask YC: How can we catch up to our peers at the big name universities? - <AUTHOR>

As an undergrad at a "second tier" university, I want to know what I need to do to compete with CS students at the top schools. Specifically, I'm concerned about admission to grad school. Bonus points for criteria other than GPA. <comment><USER2> Research, research, research.

Seriously, there's nothing university departments like to see in a graduate entrance application than published research. It doesn't have to be related to anything anyone at the department in question is working on (although that wouldn't hurt); simply demonstrating that you are capable of working independently, discovering new things, writing a coherent paper about them, and getting it published will put you ahead of the vast majority of applicants.

Edit: Sole authorships, while not necessary, are ideal -- with co-authorships (particularly involving undergraduate students) there's always a question as whether each author made any significant contribution.

<comment><USER3> Research could also involve independent study courses and/or an honor's thesis. The latter could take two years to complete, but when you go to apply for grad schools it's something you've completed based on your interests and drive. It's the perfect example of completing something that grad school will require. You will have formed a committee, presented your ideas, executed a plan of work, and defended the results. It's practice for grad school because it's a mini-version of it.

Independent study courses are a great way to explore your interests more intensively especially as you get ready to do your own work. If you're working on research with a professor, some schools will allow you to get credit in this form. Or you could simply do more focused reading with a professor and get credit for it. One of my best experiences as an undergrad was an independent study with two profs (philosophy and psychology) to talk weekly about mind and brain issues. I needed to know them to see if it would fly, and I organized the reading list with their input, but afterwards it functioned as a normal course with me as the only student.

Through a thesis committee and independent study courses, you'll also get to know professors for those all-important letters of recommendation (as Paul recommends). For those, it's better to know 6 profs well so you can then pick the ones that know you and can best speak highly of you. In a class, especially one with more than ten students it's very hard for a prof to see how you stand out. Better to start working more with them directly. Then they'll have much more to write about. In a similar vein, take small seminar classes wherever and as early as possible. That's a straight-forward way to get to know profs (to see who you want to work with or have on a committee) and for them to get to know you.

Also, find ways to go to other schools during the summers to conduct research more in line with your interests and to be surrounded by like minded peers. Schools have programs that will pay you a little bit and put you up in a dorm. If you research those programs quickly, you could find something for as soon as this summer.

<comment><USER3> P.s. More directly in line with your original question: It's very hard for undergrads to get face time with profs at the better name schools. If you get that opportunity, it will shine through in your letters, but more importantly, in your experiences.

</comment></comment></comment><comment><USER4> <http://philip.greenspun.com/careers/women-in-science>

It's not what you asked for, but if you haven't read the likes of it before, you should. I wish I could have read it when I was your age. I wonder if I would have believed it.

If you still need advice on graduate admissions, I can tell you that pg and cperciva are on the right track. Undergrad research is particularly great, as it gives you lots of contact with an adviser who will write great recommendation letters for you.

<comment><AUTHOR> This document is pretty shocking, I won't lie. Do you think it's totally true?

<comment><USER5> I think it's a bit exaggerated. Certainly the _average_ assistant professor isn't denied tenure: that only happens in about 10-20% of cases as I understand it.

[http://incoherently-scattered.blogspot.com/2008/02/how- commo...](http://incoherently-scattered.blogspot.com/2008/02/how-common-are- tenure-rejections.html)

<comment><USER4> I'd agree that it's a little bit exaggerated. And yet... I'd claim that the real reason why "only" 20% of assistant profs are denied tenure is that the system now weeds them out at the postdoc level. It's now considered normal to require candidates to perform _multiple_ postdocs before awarding them a tenure-track position. The only sure way to escape that treadmill is to write your own grant and get it funded... which, by no coincidence at all, is also the secret to being a successful assistant prof.

You can count the number of first-year grad students per year. You can count the number of graduating Ph.D.s per year. And you can count the number of tenure-track openings per year. And then you know the percentage of people who will eventually leave, or be kicked out, of academia. The rest is detail.

Most of us leave. For a motivated student to be _forced_ out is kind of rare. They're quite happy to have you. You're really, really cheap.

<comment><USER5> Mostly true, but I think it makes a big difference whether you leave as a 23-year-old PhD student or as a 42-year-old assistant professor.

<comment><USER6> Problem is, leaving after a couple of years of a PhD program is the academic equivalent of chewing your own leg off, and most people won't do it -- even if it's the right decision.

You have to be exceptionally mentally strong to leave that culture without grabbing the brass ring, and even then, it can haunt you with feelings of failure and inadequacy for years (people like Jerry, Larry and Sergey are the obvious exceptions.)

</comment></comment></comment></comment></comment></comment><comment><USER7> The order of importance of the many components that would make up your grad school app:

1\. Research experience --- The most important factor, though not essential. Not having it is not a handicap. In fact most undergrads admitted for grad school do not have research experience, and their app focuses on no. 2 and 3 below. However if you have done some significant reseach, that fact will dominate the rest of your app, and make 2 and 3 irrelevant to a large degree.

2\. Recommendation letters --- there are a couple of factors here. A letter from a better known researcher is more valuable. A letter from a professor who is known to your potential advisor is more valuable. Finally, a letter from a professor who has a history of recommending strong candidates carries more weight (admission committees keep track of such things).

3\. Undergrad school and GPA --- A 3.6 from MIT is more valuable than a 4.0 from a second tier school. Your GPA over your junior and senior years carries more weight than the GPA over your first two years. This means you can't afford to let up, but you can make up for the follies of youth (or freshman year).

4\. Your statement of purpose (the essay you write explaining why you want to pursue a phd in a particular dept) --- This should be focused

5\. GRE scores. These are much less important, and are used mostly as a preliminary filtering mechanism.

Another important issue is that given two applications of equal quality (that is, the two candidates are roughly the same by the criteria above), the more focused one will have an edge. By focus, I mean that between an app saying "I'm interested in theoretical computer science" and one saying "I'm interested in probabilistically checkable proofs and error correcting codes" (these are specific topics of research in theoretical CS), the second one will make the cut every time.

As an applicant from a "second tier" school, you are significantly handicapped. The good news is that this system can be hacked quite well. The best course of action depends on how much time you have left before applying for grad school and the subject you plan to apply in, but I would strongly suggest the following:

a) Increase your gpa as much as possible, between now and when you apply. If your gpa is low right now, showing a substantial improvement just before you apply sends a strong message of maturity.

b) Do research. Find a project in your school to get involved in. Find an informal advisor. If you feel such opportunities don't exist in your school, then find the closest top-tier school in your area. Go visit and find a project there. The goal of this involvement would be to produce a publication and a couple of recommendations. Ideally you should do it in conjunction with point c below.

c) Focus your application. This is the single biggest factor that can boost your chances. Decide the subfield within your major that you want to apply to. Find potential advisors within that area. Read up on their research, contact them about it. Do not spam. If possible go meet these people and get involved. If a professor wants to take you on as a grad student and has funding, then nothing on your application matters and you are in.

Its very possible to get admitted to a top tier program. It takes hard work and knowing how the admission process works. You should also talk to people in your field, as some things vary across disciplines.

<comment><USER6> You're right about 1 and 2, but you've got the order wrong for 3-5.

Most grad schools view GPA and GRE scores as an initial, binary filter (with the emphasis on GRE scores), and then use the other criteria to rank the remaining applicants in a somewhat flexible way. A 4.0 from MIT is obviously not going to hurt you, but I wouldn't go so far as saying that a 3.6 from MIT is "better" than a 4.0 from a second-tier school. That's just snobbery.

<comment><USER7> Those numbers were just an example to say that both the gpa and school are factors. That is, an admissions committee would not do a straight comparison between gpas, but would factor in a weight for the school.

This is not snobbery, just the facts of life. I am not saying that a 4.0 student from an unknown school is worse than a 3.6 MIT student. There is no way to objectively know that.

Also gpa matters more than you seem to think. A 4.0 from MIT would do more than not hurt. It would get you admission into a top tier program. The reason graduate programs give importance to gpa is because it takes a sustained effort over 4 years to have a great gpa when you apply. This shows an ability to work hard over a sustained period of time which is a prerequisite for success in grad school and research.

Finally, as I originally said, most undergrads do not have any significant research experience when they apply, and so the recommendations and gpa become more important in those cases. There is a strong correlation between recommendations and gpa, with the difference that a gpa is an objective measure. Most competent students manage to find people who will write positive recommendations for them. Its much harder to finesse your gpa this way. Admission committees are well aware of these issues.

<comment><USER6> I understand what you believe to be true, but I'm not speaking from a hypothetical perspective. I've seen the 3.6 MIT student get rejected, while a 4.0 student from a "lesser" school gained admission. It happens.

As I said, a high GPA from MIT isn't going to hurt you, but it's not a golden ticket, either. If nothing else, grad schools like the GRE because it's a standardized metric. Still, neither GRE score nor GPA are particularly good predictors of success in graduate school, and the committees know this.

<comment><USER7> I think we possibly have a different understanding of what is a lesser or second tier school. I based my opinion on the original poster's response to pg's comment, where he states that his cs program is like "trade school". I doubt that a 4.0 from such a program would count for much in comparison to an applicant from a top CS department.

As for your anecdotal example, could it not be that the other factors were the deciding ones in this case (like the student from the lesser school having much stronger recommendations or published a paper)? In any case, all I meant by making up this 3.6/4.0 example was that the school matters too.

If I read you correctly, you also seem to be saying that GRE scores are more important than GPA. This is not true at all. In fact GRE scores count for very little compared to your GPA (which counts less than recommendations and research). Remember that we are discussing admission to top-tier schools. MIT does not even consider the GRE in their admission procedure.

Also, what makes you assume that I'm talking from a hypothetical perspective? At my undergrad school, I have personally known about 50 people who've applied and gone to grad school, and GPA correlates very strongly with the ranking of the department they joined. Since then these views have been confirmed by the faculty (including my advisor) that I have talked to at my grad school.

In any case, we have our grad recruiting weekend coming up next week, and I'll do some field research and report back :)

<comment><USER6> My post was based on my own knowledge of the recruiting process (and the background of many fellow grad students) a top-tier school:

1) GPA and GRE scores are typically the basis for an include/exclude decision to filter a large pool of applicants down to a smaller pool. They don't typically play a primary role in determining the _ranking_ of the candidates after this step. That's a far more holistic (read: subjective) decision.

2) In terms of #1, committees tend to weight both GPA and GRE equally, with a slight preference given to the GRE, because it is the only direct quantitative comparison available between all applicants.

But this debate is mostly navel-gazing, because admissions committees are famously capricious; the _exact_ method used depends on the members of the committee, and their particular biases. The only point I'm trying to make here is that your chances to get admitted to a top-tier school aren't necessarily higher if you went to MIT as an undergrad. That's a myth that is dearly loved -- by MIT undergrads.

(And by the way? Two seconds of googling tells me that you're wrong about MIT and the GRE:

[http://web.mit.edu/admissions/graduate/requirements/testing_...](http://web.mit.edu/admissions/graduate/requirements/testing_req.html))

<comment><USER7> First, I really don't understand what we are arguing about here, since my initial comment already says that research and recommendations are much more important than GPA and GRE. Lets just agree to disagree on the rest.

Second, I never questioned the basis of your opinions, you questioned mine (indirectly).

Finally, maybe you should have spent more than two seconds googling, since MIT CS does not ask for your GRE scores. See the relevant part in this document (which is available on the page you linked): [http://web.mit.edu/admissions/graduate/pdfs/MIT_department_i...](http://web.mit.edu/admissions/graduate/pdfs/MIT_department_info.pdf)

(Edit: Did you go to grad school in CS? Because otherwise the whole discussion loses relevance.)

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> The way you get admitted to good grad schools is by impressing specific profs. It's much more personal than undergrad admissions. And profs I know say the thing that impresses them most is good recommendations from other professors they know and trust. So talk to current profs who like you and ask them who they know at grad schools you want to go to who would make good advisors for the kind of work you want to do.

<comment><AUTHOR> Our CS program here is more like "programming trade school." I doubt recommendations from those profs would be worth much -- what about from the math department?

<comment><USER6> Get the best recommendation you can, from the best researcher who knows you personally, and who can vouch for your _research skills_. Classroom performance doesn't count.

If you have to choose, it's better to have a great recommendation from someone unknown, than it is to have a mediocre recommendation from a superstar.

</comment></comment></comment><comment><USER9> <http://www.cs.cmu.edu/~harchol/gradschooltalk.pdf>

<comment><USER10> This is a really nice link, I found it really nice to read although I have got into grad school already :-), nice to be reminded why we are here in the first place.

</comment></comment><comment><USER11> What? You don't need to (other than perhaps geographically).

<http://paulgraham.com/colleges.html>

If the gospel according to PG doesn't persuade you, I'm happy to offer some additional anecdotal evidence. Some of the smartest hackers I've ever met are college dropouts or went to a 2nd rate college.

As Twain said, "Never let your schooling interfere with your education."</comment>


Generate the ideal setting for your vim #vim #python #ruby #PHP - <AUTHOR> http://vim-bootstrap.appspot.com/

<comment><USER2> The ideal setting is an Internal Server Error?</comment>


Michael Hastings' Dangerous Mind - <AUTHOR> http://www.laweekly.com/2013-08-22/news/michael-hastings-crash/full/

<comment><AUTHOR> So, the government _did_ contribute to his death, in at least an indirect, inadvertent way

> _Hastings was intensely interested in government surveillance of > journalists. In May, the story broke about the Department of Justice > obtaining the phone records of Associated Press reporters. A couple weeks > later, Edward Snowden 's revelations about the National Security Agency's > massive surveillance program became public. Hastings was convinced he was a > target._

<comment><USER2> > Hastings was convinced he was a target.

This kinda hints that he had some dirt on government operations.</comment></comment>


Why Tracking Cookies Are Dangerous If You Have Nothing to Hide - <AUTHOR> https://www.hackerfoss.com/2019/10/why-tracking-cookies-are-dangerous-if.html <comment><USER2> 90% of readers are mentally inserting the word "Even" as they read that headline. Why didn't you put it there for them?

<comment><AUTHOR> Thanks, pal. I have updated the title.</comment></comment>


Android Market Now Over 10,000 Applications Strong - <AUTHOR> http://www.mobilecrunch.com/2009/09/07/android-market-now-over-10000-applications-strong/

<comment><USER2> Certainly the android phone is a competitor to the iPhone now. The whole community including the app store is what makes the iPhone successful. As many people have said, even if a phone comes out that is as good or better as the iPhone, it will only be equal to or better than the iPhone when it has the same community of support and apps.

I personally think that the Android's biggest issue is that it does not have a very solid main place where everyone comes for apps and even if that happened, it would need a lot of publicity, just like the iPhone got.

<comment><USER3> _I personally think that the Android's biggest issue is that it does not have a very solid main place where everyone comes for apps_

that may have been true in the early days of the tmobile g1, but i can't even recall any apps i've installed that weren't from the android market on my tmobile mytouch. i would wager many mytouch users don't even know it's possible to install apps that don't come from the market.</comment></comment>


How we created a tolerometer for Indians - <AUTHOR> https://medium.com/@tmatthewj/how-we-created-a-tolerometer-for-indians-b6d136f35029 <comment><USER2> Pretty cool idea although I'm not fond of the execution - I think people can easily be biased or lead into favorable answers on the questions you have asked - may be the next step would be to design a Milgram experiment for tolerance?

Another aspect of this kind of social experiment or social science project is, how do you establish that you are non-partisan and non-biased? As well- intentioned as it is, there is a chance that this can be misconstrued as a propaganda that says "India is more tolerant than you think".

<comment><AUTHOR> Agree with you. The execution could've been much better. To start with, it always make sense to do a validated questionnaire experiment than come up with a new survey like we did.

Talking about Milgram and similar experiments, though they are known to help achieve unbiased observations about human behavior, it actually doesn't rule out the validity of a well executed questionnaire experiment. Good questionnaires take care of authenticity using a lie score that tries to detect if the subject is trying to make himself / herself look good with the answers.

<comment><USER2> So, do you have a lie score for this experiment?</comment></comment></comment>


Bitcoin 'More Than Just a Fad,' Morgan Stanley CEO Says - <AUTHOR> https://www.bloomberg.com/news/articles/2017-09-27/bitcoin-more-than-just-a-fad-morgan-stanley-ceo-gorman-says <comment><USER2> Key quote from Gorman in the article: "I haven't invested in it."

<comment><USER3> Well, he would sound like he's pushing an agenda if he says "I have invested in it".

<comment><USER2> Are you saying that you think Gorman has actually invested in Bitcoin, but he lied because he didn't want to make it seem like he was touting his book?

<comment><USER4> I think people at his level trade on traditional assets (stocks, metals, whatever) but always with an insider's edge. He probably does not have that edge in Bitcoin.

</comment></comment></comment></comment><comment><USER5> I hope Bitcoin is more than just a fad because it would provide a really convenient way of harnessing solar energy in space without having to beam it all back to Earth. But this can only happen if Moore's Law slows way down (which is happening) and if mining ASICs hit an efficiency plateau lasting for years (is this happening?).

Currently, it costs about 9MWh per BTC, or about $1000 in electricity per bitcoin to mine on a pretty efficient S9 antminer. S9s currently go for about $2/Watt and require about 100 J/TH and weigh, I don't know, maybe 1kg per kilowatt (or 1000W/kg).

Solar modules are about 30 cents per watt on the spot market, produce about 100W/kg (if weight optimized, i.e. not such a thick aluminum frame). Since all the heat must be dumped into space at the ASIC's operating temperature, radiators are about 100W/kg and probably cost as much as solar modules ($0.30/W) if mass produced.

So 1000W of mining capacity would weigh 21 kg and cost $2000+$300+$300= $2600. To launch 21kg at $100/kg to GSO (this is an incredibly low cost, but could be achieved with SpaceX's or Blue Origin's long-term ITS/BFR or New Armstrong rockets) would cost about $2100. So a total of about $4.7/W.

Assuming the lifetime of that whole thing is about 12 years (a reasonable time for a risk-free doubling of an investment), that's about 12 BTC, or about $47,000 at today's prices, or a factor of 10 return.

(Today's BTC returns ~40 cents per kWh invested, this scheme would break-even at about 4 cents per kWh generated.)

...but: 1) There's a whole bunch of glue, like a network link and spacecraft subsystems that must be included as well as spacecraft processing.

2) Moore's Law in the strict sense is over, but progress still continues.

3) ASIC improvement is possible.

4) We don't have Musk's or Bezos' huge reusable rockets quite yet.

5) If return was this well-proven, you'd likely have a bunch of competition from even cheaper sources of energy.

6) I still think Bitcoin is far too volatile and speculative to justify this kind of investment. Given the reduction in mining rate, if you knew Bitcoin would be able to justify this kind of investment over decades, then you'd probably be better off just buying a bunch of BTC right now and holding it.

But maybe the future of cryptocurrency will eventually enable a less volatile and more proven return, making stuff like this worth investing in. I hope so.

</comment><comment><USER6> Beyond all this noise that is patticularly annoying in this space, I really like articles trying, at least, to provide more fundamental analysis. Last week Chris Burniske provided an article about cryptoasset valuation: [https://medium.com/@cburniske/cryptoasset-valuations- ac83479...](https://medium.com/@cburniske/cryptoasset-valuations-ac83479ffca7)

</comment><comment><USER7> * Call it a fad

* Price goes down

* Buy

* Say it's not a fad

* Price goes up

* Sell

* ...

* Profit?

<comment><USER8> I don't think you need the "...", usually that indicates a mysterious step will never actually materialize.

But the Profit cycle is fully complete here.

<comment><USER9> [http://knowyourmeme.com/memes/profit?full=1](http://knowyourmeme.com/memes/profit?full=1)

<comment><USER10> Comment is still valid. There is no mysterious step here.

</comment></comment></comment></comment><comment><USER11> Good thing that all these Wall Street CEOs share these nuggets of wisdom with us uncouth plebeians. Their charity is out of this world! Now I just need to know what Jim Kramer thinks of this and I'm good.

<comment><USER12> Like it or not, major figures in the financial industry saying "this is legit" helps bitcoin's image which has knock-on effects. Bitcoin still struggles from the "its only used to buy drugs" stigma even with some on here.

<comment><USER11> The point of crypto is making these figures irrelevant.

<comment><USER13> ...and replacing them with cryptocurrency CEOs and HN and Reddit commenters whose economic experience is limited to googling the word "fiat" and talking points on free markets? Is that what you mean?

Occasionally people are listened to because they have something to offer, like, say, running one of the world's largest investment banks. His remarks immediately made Bloomberg, CNBC, and so on, which is just free PR. From the head of Morgan Stanley. Who in the current world, and not the glorious future one you want, commands respect and attention. Talk about biting the hand throwing your passion a bone.

And what's your plan to make Gorman and other investment banks irrelevant with cryptocurrency? You're talking about something like, what? A third of the world economy? I made that number up, but it's probably even more. Nobody will have financial planning services and investments in your cryptographic future? End capitalism? How should I explain what you want to the soybean farmer next door who needs an ag loan from _one of those very banks_ to buffer crop loss from an imported Asian pest? The free market will figure it out, and the entire system of capital allocation we all know is fundamentally broken and useless? If only he mined Litecoin?

The single biggest thing holding crypto back is tone deaf arrogance like this from its biggest proponents. Slow down, buddy, you've successfully monetized files of bits. Completely disrupting the entire world economy and making banks irrelevant is a wee bit far fetched and even if it isn't, should be approached with a little more caution than a lot of crypto folks give it. You're talking about the (imperfect) global system that feeds billions of people and keeps them alive. Show it some respect; only hundreds of thousands of people, nigh, millions, worked on it before the engineers with vim showed up and declared they know better.

Cryptocurrency is good. Let's be reasonable when discussing it.

<comment><USER14> Shh, half the fun of cryptocurrency threads is watching its proponents re- learn every single harsh lesson banking and financial industries have learned over the past couple hundred years! It turns out a lot of regulations and whatnot involved with Fiat™ don't just exist for arbitrary reasons.

<comment><USER11> Sure but it's also naive to think that banks aren't trying to fuck you over.

<comment><USER13> And surely, nobody in the basically unregulated cryptocurrency world has ever done that, right? Bad actors are bad actors, and it's disingenuous to point at some and ignore others.

<comment><USER15> One of the largest cryptobank (exchange) is ran by someone who openly praised one of the major ponzi scams, even after it shut down and was known to be a scam. [1]

Said cryptobank is now allegedly using fake orders to buy up significant amounts of BTC (recently a 50,000 BTC order [2]) during market crashes.

[1] [https://www.reddit.com/r/btc/comments/6xpddt/as_20m_more_tet...](https://www.reddit.com/r/btc/comments/6xpddt/as_20m_more_tether_are_issued_today_i_think_its/)

[2] [https://twitter.com/Bitfinexed/status/908718464671641602](https://twitter.com/Bitfinexed/status/908718464671641602)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER16> This seems like a decent place to ask:

What is the most legit and safe method for creating a wallet and just buying a bit of Bitcoin? I wouldn't mind being able to drop a couple hundred on it and seeing where it goes, but it's hard to trust websites/subreddits that look like posts were written by nerdy 15 year olds.

<comment><USER17> Coinbase is the only answer here, and it makes me frustrated to still see the "buy a hardware wallet" recommendations.

I recently had my hardware wallet (a trezor, which is the nicest one) fail during a firmware update. Now, obviously I have a backup of my seed, but for those few minutes that backup became a VERY expensive little widget.

When I travel, I have to worry about where my backups are stored. Storing them in a bank vault means I have to worry about the bank vault.

If I have a hardware wallet, I'm essentially betting that I am better at opsec than coinbase, a >$1billion company who devotes a non-trivial amount of resources to opsec. Keeping my own wallet means I am arrogant enough to believe that I can out-opsec the team of security people they have there.

That's absurd.

There ARE reasons for a hardware wallet (which is why I have one). The bCash split recently is why I have mine.

Keeping your own wallet and "being your own bank" is a really fun cyberpunk fantasy, but there are plenty of practical reasons to use banks.

If you are asking the question about how to buy bitcoins in 2017, then the answer is unquestionably that you should both buy and store you coins on coinbase.

<comment><USER18> >If I have a hardware wallet, I'm essentially betting that I am better at opsec than coinbase, a >$1billion company who devotes a non-trivial amount of resources to opsec.

I'm sure people who were using Mt Gox felt the same way. Part of the equation is you're betting you place your own interests at a higher priority than the people who run Coinbase. Which is almost certainly true.

<comment><USER19> Coinbase is U.S. based, insured and licensed. Very different from Mt. Gox, which started as a trading depot for Magic: The Gathering cards.

[https://support.coinbase.com/customer/en/portal/topics/79436...](https://support.coinbase.com/customer/en/portal/topics/794360-security/articles)

</comment></comment></comment></comment><comment><USER20> if I invest in Bitcoin today, am I still an early adopter?

<comment><USER21> I think invest is the wrong term. Investments have some kind of builtin return like interest or dividends. You're buying bitcoin in the same way you buy gold or a lottery ticket. Maybe you have some solid reasoning on why price will go up but you aren't getting any returns like in an investment.

<comment><USER22> Boiled down to the base principles it's the same... money goes in, more money might come out, or you might get less money out. The only difference is predictability and odds.

<comment><USER23> The entire profit motive from Bitcoin "investing" comes from selling it to another human for more than you bought it for. They're digital Beanie Babies.

Real investments have a return derived from capital. A house, for example, can be rented out for money. Or a business can earn profit and distribute it to its shareholders. You can try to buy and sell houses like people did during the RE bubble for profit, or try to buy and sell stocks like a day trader, but these are still fundamentally different from an "asset" which has no means by which to generate a return on its own.

<comment><USER24> > Real investments have a return derived from capital. A house, for example, > can be rented out for money.

You can buy goods and services with Bitcoin. It's just another currency. Do people claim that forex is not investing?

<comment><USER23> > Do people claim that forex is not investing?

Yes, all the time. It's clearly speculation (in the technical sense)

</comment></comment></comment></comment></comment></comment><comment><USER25> I'm here for the shit posting.

<comment><USER26> I thought I was the only one!

<comment><USER25> Popcorn?

</comment></comment></comment><comment><USER27> > "The concept of anonymous currency is a very interesting concept -- > interesting for the privacy protections it gives people, interesting because > what it says to the central banking system about controlling that."

I don't think he means bitcoin in particular, it could be any crypto currency. Bitcoin is still a ponzi scheme in bubble phase.

<comment><USER28> hey that's not fair... it's a pyramid scheme, not a ponzi

<comment><USER29> It's neither of these things.

<comment><USER30> Most of its value is speculative and if you buy you have to hope there's a bigger sucker somewhere in the future in order to get your payout, that's not exactly Ponzi-shaped, but I can definitely see why people might consider it a decentralized form of the pattern.

<comment><USER29> According to your interpretation, wouldn't gold also qualify as a decentralized ponzi scheme? You're just hoping some bigger sucker buys your over inflated metal right?

<comment><USER31> People actually use gold, as it's optimal (or solely suited) for a number of electric and chemical tasks. Also, it's pretty and used as a status symbol. It has utility, which has been overblown in the market for a long time (like diamonds).

<comment><USER29> The utility of bitcoin is as a secure currency, it's not possible to forfeit or dilute, and limited in nature.</comment></comment></comment></comment></comment></comment></comment>


Sales drop for sites after Amazon implements Affiliate API throttling - <AUTHOR> http://www.pcworld.com/businesscenter/article/208463/sales_drop_for_sites_after_amazon_query_throttle.html

<comment><USER2> _One website developer who asked not to be named said that during a busy hour, his site might max out with the new limits....Each time a visitor searches for a product, the site queries Amazon using the Product Advertising API for that information._

Isn't it bad practice to make an API call for every page view? It would be better to make one API call and cache the results locally for a set interval (e.g. 5 minutes). The pricing data is not likely to change very much in 5 minutes or even an hour, and for a busy site (apparently his site occasionally makes 2000+ requests to Amazon per hour) it would reduce the number of API calls dramatically. For example, if users do 10,000 searches in an hour for 100 distinct products, instead of doing 10,000 API calls, he could just do 1200 (12 calls x 100 searches).</comment>


A list of papers on deep learning with anime style images - <AUTHOR> https://github.com/deeppomf/DeepLearningAnimePapers <comment><USER2> Looks like this hasn't been updated in a while. It's not a popular subfield but there have been interesting things published since. I keep a list of Danbooru2018-related applications & papers in [https://www.gwern.net/Danbooru2018#applications](https://www.gwern.net/Danbooru2018#applications) , and there's a slightly active subreddit at [https://www.reddit.com/r/AnimeResearch/](https://www.reddit.com/r/AnimeResearch/) for applications/papers in general.

<comment><USER3> Came across your work by accident not too long ago and am a huge fan of it (no, not just StyleGAN). I'm too shy to send an e-mail or a Tweet, so given that here is the opportunity: thank you.

<comment><USER4> i second that, gwern is amazing :)

</comment></comment></comment><comment><USER5> This list is similar: [https://github.com/dimpurr/awesome-acg-machine- learning](https://github.com/dimpurr/awesome-acg-machine-learning)

It has a bit more focus on manga datasets like [http://www.manga109.org/en/index.html](http://www.manga109.org/en/index.html)</comment>


Over 1000 frames in, XKCD 1190 is still going - <AUTHOR> http://xkcd.com/1190/

<comment><AUTHOR> Animated version here: <http://xkcd.aubronwood.com/></comment>


What's so hard about PDF text extraction? - <AUTHOR> https://filingdb.com/b/pdf-text-extraction <comment><USER2> PDFs are the bane of my existence as someone who relies on machine translation everyday. The worst is that so many event flyers and things, even important local government information will just be dumped online as a pdf without there being any other effort to make the contents available. I don't know how blind people are supposed participate in civil life here..

<comment><USER3> The problem is accessibility features are totally invisible to normal users. Someone with good intentions creates a pdf, and it works for them. They don't use screen reader tools or know how they work so they don't even realize there is a problem.

<comment><USER4> And the problem with _that_ is that the screen reader tools are $1200, because of the huge associated R&D costs and incredibly small target market.

The sad thing is that the very complexity required to implement a screen reader is soley because of the technical nightmare information accessibility currently is.

It's reasonable to think that "if only" everything could be made ubiquitous and everyone (= developers) could become collectively aware of these accessibility considerations, maybe there would be a shift towards greater semanticity.

Thing is, though, that NVDA is open source, and iOS has put a built-in free screen-reader in the hands of every iPhone user... and not much has changed.

So it's basically one of those technologies forever stuck in the "initial surge of moonshot R&D to break new ground and realign status quo" phase. :(

<comment><USER5> How much good would moonshot-level R&D even be capable of doing? Without realigning the world around a new portable format for printable documents, isn't this 99% Adobe's problem to solve? Or are the hooks there to make much more accessible PDFs, and the issue is that various popular generators of those documents (especially WYSIWYG ones like MS Word) either don't populate them, or perhaps don't even have the needed semantic data in the first place?

For my part, I would love to see PDFs which can seamlessly be viewed in continuous, unpaged mode (for example, for better consumption on a small- screen device like a phone or e-reader). Even just the minimal effort required to tag page-specific header/footer type data could make a big difference here, and I expect that type of semantic information would be useful for a screen reader also.

<comment><USER6> Could governments insist that PDF software they buy be screen-reader friendly? If this were rigorously done, you'd have all government documents be readable by default, and then anyone else who ran the same software commercially would, too.

You could also impose requirements on public companies to provide corporate documents in accessible formats- these sorts of documents are already regulated.

There's various levers that could be pulled, maybe those aren't the right ones. But you could do it.

<comment><USER7> The problem is the PDF spec itself is not screen reader friendly.

<comment><USER8> PDF is mostly just a wrapper around Postscript, isn't it?

You could just put the original text in comments or something, wrapped in more tags to say what it is.

<comment><USER9> PDF is a document format, Postscript is a Turing complete programming language (and rather a fun one IMHO).

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> Here where I work we are parsing PDFs with [https://github.com/flexpaper/pdf2json](https://github.com/flexpaper/pdf2json). It works very well, and returns an array of {x, y, font, text}.

If you are familiar with Docker, here is how you can add it to your Dockerfile.

ARG PDF2JSON_VERSION=0.71 RUN mkdir -p $HOME/pdf2json-$PDF2JSON_VERSION \ && cd $HOME/pdf2json-$PDF2JSON_VERSION \ && wget -q [https://github.com/flexpaper/pdf2json/releases/download/$PDF...](https://github.com/flexpaper/pdf2json/releases/download/$PDF2JSON_VERSION/pdf2json-$PDF2JSON_VERSION.tar.gz) \ && tar xzf pdf2json-$PDF2JSON_VERSION.tar.gz \ && ./configure > /dev/null 2>&1 \ && make > /dev/null 2>&1 \ && make install > /dev/null \ && rm -Rf $HOME/pdf2json-$PDF2JSON_VERSION \ && cd

<comment><USER11> Your command is cut off

<comment><USER12> Maybe it was scraped off of a PDF? ;P

In all seriousness, though: it doesn't look like it was cut off; I think the final cd is just to return the docker builder thingee to the home directory? The file was already built and installed by that built.

</comment></comment></comment><comment><USER13> A lot of people here seem to knock PDF, but I love it. Anyone who has tried to use OpenOffice full time probably does too. We have 'descriptive text' in MS various formats, or even html/css. The problem is every implementer does things in slightly different ways. So my beautiful OpenOffice resume renders with odd spacing and pages with 1 line of text in MS Office. With PDF, everyone sees the same thing.

<comment><USER14> > A lot of people here seem to knock PDF, but I love it.

People will disabilities that rely on screen-readers don't love it. There is no such a problem with HTML/CSS which should be the norm for internet documents.

> With PDF, everyone sees the same thing.

Yes, provided you can see at first place...

<comment><USER15> > There is no such a problem with HTML/CSS which should be the norm for > internet documents.

It should be. But meanwhile everybody seems to think it is perfectly ok that there is a bunch of JavaScript that needs to run before the document will display any text at all and how that text makes it into the document is anybody's guess.

<comment><USER16> It was an amazing shift in priorities that I feel like I somehow missed the discussion for. We went from being worried about hiding content with CSS to sending nothing but script tags in the document body within 5 years or so. The only concern we had when making the change seemed to be "but can Google read it?". When the answer to that became "Uh maybe" we jumped the shark.

My bashful take is that nobody told the rest of the web development world that they aren't Facebook, and they don't need Facebook like technology. So everyone is serving React apps hosted on AWS microservices filled in by GraphQL requests in order to render you a blog article.

I am being hyperbolic of course, but I was taken completely off guard by how quickly we ditched years of best practices in favour of a few JS UI libraries.

</comment></comment></comment></comment><comment><USER17> Always use PDF/A-1a (Tagged PDF) which contains the text in accessible format. For many governments this a legal requirement.

With tagged PDF it's easy to get the text out.

<comment><USER18> LaTeX does not support PDF/A btw.

<comment><USER19> Not by default, but the pdfx package enables this, Peter Selinger has a nice guide: [https://www.mathstat.dal.ca/~selinger/pdfa/](https://www.mathstat.dal.ca/~selinger/pdfa/)

<comment><USER20> The linked instructions cover PDF/A-1b (have plain text version of contents), but Tagged PDF is more than that -- it's about encoding the structure of the document.

There is a POC package for producing Tagged PDF here [https://github.com/AndyClifton/accessibility](https://github.com/AndyClifton/accessibility) but it's not a complete solution yet.

Here is an excellent review article that talks about all the other options for producing Tagged PDFs from LaTeX (spoiler — there is no solution currently): [https://umij.wordpress.com/2016/08/11/the-sad-state-of- pdf-a...](https://umij.wordpress.com/2016/08/11/the-sad-state-of-pdf- accessibility-of-latex-documents/) via [https://news.ycombinator.com/item?id=24444427](https://news.ycombinator.com/item?id=24444427)

</comment></comment></comment></comment><comment><USER21> Don't get me started on PDF's obsession with nice ligatures.. For the love god, when you have a special ligature like "ti", please convert the text in the copy buffer to a "ti" instead of an unpasteable nothing.

Nothing is more annoying than having to manual search a document that has been exported from PDF and have to make sure you catch all the now incorrect spellings when all the ligatures have just disappeared "action" -> "ac on", "finish" -> " nish".

<comment><USER22> I'm a bit of a typography buff, so just chiming in to say that there is nothing inherently wrong with ligatures in PDF!

As far as I understand, PDFs can be generated such that ligatures can be correctly cut'n'pasted from most PDF readers. I have seen PDFs where ligatures in links (ending in ".fi") cause problems, and I believe that's just an incorrectly generated PDF; ligatures done wrong.

Considering that PDF a programming language designed to draw stuff on paper, going backwards from the program back to clean data is not something that one should expect to always work.

</comment></comment><comment><USER23> In my case, I think the correct expression would be "what's so hard about _meaningful_ PDF ext extraction"

My company uses the services of, and has some sort of partnership with, a company that makes it's business out of parsing CVs.

Recently we've seen a surge in CVs that after parsing return no name and / or no email, or the wrong data is being fetched (usually, from referees).

So, out of curiosity I took one (So far) pdf and extracted the text with python.

Besides the usual stuff that is already known (As in, the text is extracted as found, e.g., if you have 2 columns the lines of text will appear after the line at the same level in the document that is in the other column) what I found - obviously take this with a grain of salt as this is all anecdotally so far - is that some parts of the document have spaces between the characters, e.g.:

D O N A L D T R U M P

P r e s i d e n t o f t h e U n i t e d S t a t e s o f A m e r i c a

These CVs have the characteristics to be highly graphical. Also anecdotally, the metadata in the CV I parsed stated it was from Canvas [1]

[https://www.canva.com/templates/EAD7WY_6Ncs-navy-blue-and- bl...](https://www.canva.com/templates/EAD7WY_6Ncs-navy-blue-and-black- professional-resume/)

<comment><USER24> How meaningful the text is is going to depend on how the PDF was generated.

Consider that creating a PDF is generally just the layout software rendering into a PDF context — no different as far as it is concerned than rendering to the screen.

Space are not necessary for display (although they might help for text selection so often are present). It is not important that headers are drawn first, or footers last — so often these scraps of text will appear in unexpected places....

PDF has support for screen readers, but of course very few PDFs in the wild were created with this extra feature.

<comment><USER25> You're completely correct but unfortunately, this doesn't matter in practice. It's true that thanks to formats like PDF/UA, PDFs can have decent support for accessibility features. Problem is, no one uses them. Even the barest minimum for accessibility provided by older formats like PDF/A, PDF/A-1a are rarely used. Heck, just something basic like correct meta-data is already asking for too much.

This means getting text out of PDFs requires rather sophisticated computational geometry and machine learning algorithms and is an active area of research. And yet, even after all that, it will always be the case that a fair few words end up mangled because trying to infer words, word order, sentences and paragraphs from glyph locations and size is currently not feasible in general.

Even if better authoring tools were to be released, it would still take a long time for these tools to percolate and then for the bulk of encountered material to have good accessibility.

This recent hn post is relevant: [https://umij.wordpress.com/2016/08/11/the- sad-state-of-pdf-a...](https://umij.wordpress.com/2016/08/11/the-sad-state-of- pdf-accessibility-of-latex-documents/)

</comment></comment></comment><comment><USER26> _It is not uncommon for some (or all) of the PDF content to actually be a scan. In these cases, there is no text data to extract directly, so we have to resort to OCR techniques._

I've also seen a similar situation, but in some ways quite the opposite --- where all the text was simply vector graphics. In the limited time I had, OCR worked quite well, but I wonder if it would've been faster to recognise the vector shapes directly rather than going through a rasterisation and then traditional bitmap OCR.

<comment><USER27> Here's another 'opposite' \- I had to process PDFs to find images in them.. and the PDFs were alternating scans of text + actual images.

</comment></comment><comment><USER28> I'm parsing PDFs and extracting tabular data - I am using this library [https://github.com/coolwanglu/pdf2htmlEX](https://github.com/coolwanglu/pdf2htmlEX) to convert the PDF into HTML and then parsing thereafter. It works reasonably well for my use-case but there are all kinds of hacks that I've had to put in place. The system is about 5-6 years old and has been running since.

The use-case is basically one where there is a tabular PDF uploaded every week and the script parses it to extract the data. Thereafter a human interacts with the data. In such a scenario, every ~100 parses it fails and I'll have to patch the parser.

Sometimes text gets split up, but as long as the parent DOM node is consistent I can pull the text of the entire node and it seems to work fine.

<comment><USER29> Did you publish your fixes so this could help others too? it seems like this repo is unmaintained atm.

<comment><USER28> They are very specific changes for my use-cases unfortunately

</comment></comment></comment><comment> heresie-dabord Read all the comments here about people struggling with PDF. All the energy and code wasted! I have watched this madness for my entire career.

PDF is being used wrongly. For information exchange, data should be AUTHORED in a parseable format with a schema.

Then PDF should be generated from this as a target format.

There is a solution: xml, xsl-fo.

<comment><USER30>
       > There is a solution: xml, xsl-fo.

You're right about that, but sadly years of "xml-abuse" in the early naughts has given xml a bad reputation. So much so that other, inferior, markups were created like json and yaml. We ain't ever going back.

Meanwhile, pdf just worked-- until you the first time you crack it open and see what's inside the pdf file. I'll never forget the horror after I committed to a time-critical project where I claimed... "Oh, I'll just extract data from the PDF, how bad could it possibly be!"

<comment> heresie-dabord > xml-abuse

Bad programmers, as usual, frustrated by their own badness.

Today's coders want us to use Jackson Pollock Object Notation everywhere for everything.

> We ain't ever going back.

Not so, friend. ODF and DOCX are XML. And these formats won't become JPON anytime soon.

</comment></comment></comment><comment><USER31> > Turns out, much how working with human names is difficult due to numerous > edge cases and incorrect assumptions

I think a good interview question would be, given a list of full names, return each person's first name. It's a great little problem where you can show meaningful progress in less than a minute, but you could also work on it full time for decades.

<comment><USER32> If I got that question in an interview, I would write a program that asks the user what their first name is. That's the only correct solution to that problem.

<comment><USER31> The fact that it's unsolvable is what makes it a good interview problem. Seeing someone solve a problem with a correct solution doesn't really tell you anything about the person or their thought process.

<comment><USER33> [https://news.ycombinator.com/item?id=24447182](https://news.ycombinator.com/item?id=24447182)

</comment></comment></comment></comment><comment><USER34> It seems like it should be doable to train a two-tower model, or something similar, that simultaneously runs OCR on the image and tries to read through the raw PDF, that should be able to use the PDF to improve the results of the OCR.

Does anyone know of any attempt at this?

Blah blah blah transformer something BERT handwave handwave. I should ask the research folks. :-)

<comment><USER35> I've thought about it, but haven't tried.

As an experiment, we once tried converting an OCRed dictionary (this one: [https://www.sil.org/resources/archives/10969](https://www.sil.org/resources/archives/10969)) into an XML dictionary database. (There are probably better ways to get an XML version of that particular dictionary, but as I say, this was an experiment.)

Despite the fact that it's a clean PDF, and uses a Latin script whose characters are quite similar to Spanish (and the glosses are in Spanish), the OCR was a major cause of problems: Treating the upside down exclamation as an 'i', failing to separate kerned characters, confusion between '1' and 'l', misinterpreting accented characters, and so on and so on. And for some reason the OCR was completely unable to distinguish bold from normal text, even though a human could do so standing several feet away.

So I did think of extracting the characters from the PDF. If it had been a real use case, instead of an experiment, I might have done so.

Write-up here: [https://www.aclweb.org/anthology/W17-0112/](https://www.aclweb.org/anthology/W17-0112/)

<comment><USER34> Interesting! We're working on OCR on menu photos, which has some parallels in structure, but has a much smaller common vocabulary than a dictionary, almost by necessity. :-)

Many menus are also available in PDF form, so we're trying to figure out if it's worth bothering with the PDF itself, or if we should just render to image and thus reduce the problem to the menu-photo one.

</comment></comment></comment><comment><USER36> So interesting to see this just 2 days after this on HN:

"The sad state of PDF-Accessibility of LaTex Documents"

[https://news.ycombinator.com/item?id=24444427](https://news.ycombinator.com/item?id=24444427)

PDF's _have_ accessibility features to make semantic text extraction easy... but it depends on the PDF creator to make it happen. It's crazy how best-case PDF's have identifiable text sections like headings... but worst-case is gibberish remapped characters or just bitmap scans...

</comment><comment><USER37> I have a very similar project where I've extracted text and tables from over 1mm PDF filings in large bankruptcy cases - bankrupt11.com

I still haven't found a good way of paragraph detection. Court filings are double spaced, and the white space between paragraphs is the same as the white space between lines of text. I also can't use tab characters because of chapter headings and lists which don't start with a <tab>. I was hoping to get some help from anyone who has done it before.

<comment><USER38> I imagine looking for lines that end prematurely would get you pretty far. Not all the way, since some last lines of paragraphs go all the way to the right margin, but combined with other heuristics it would probably work pretty well, especially if the page is justified.

<comment><USER37> Not a bad idea!

</comment></comment></comment><comment><USER39> If you are want to extract tabular data from a text-based PDF. Check out the Tabula project: [https://tabula.technology/](https://tabula.technology/)

The core is tabula-java, and there are bindings for R in tabulizer, Node.js in tabula-js, and Python in tabula-py.

<comment><USER37> I love used tabula and recommend Camelot over it. The Camelot folks even put together a head to head comparison page (on their website) that shows their results consistently coming out ahead of Tabula.

My other complaint with Tabula is total lack of metadata. It's impossible to know even what page of the PDF the tables are located on! You either have to extract one page at a time or you just get a data frame with no idea which table is located on which page.

<comment><USER40> The best I've used is PDFPlumber. Camelot lists it on its comparison page[1] but I've had better results.

Both are better than Tabula though.

[1] [https://github.com/camelot-dev/camelot/wiki/Comparison- with-...](https://github.com/camelot-dev/camelot/wiki/Comparison-with-other- PDF-Table-Extraction-libraries-and-tools#pdfplumber)

<comment><USER37> Thanks - I cannot get Camelot to run in parallel (I use celery workers to process PDFs), there is some bug in Ghostscript that SEGFAULTS. I'll try using PDFPlumber instead! By the way Apache Tika has been the best for basic text extraction - even outputs to HTML which is neat.

</comment></comment></comment></comment><comment><USER41> I've come up with an idea of PDDF (Portable Data Document Format). The PDF format allows embedding files into documents. Why not embed an SQLite database file right in PDF document with all the information nicely structured? The both formats are very well documented and there are lots of tools on any platform to deal with them. Humans see the visual part of PDF, while machine processing works with the SQLite part.

Imagine that instead of parsing a PDF invoice, you just extract the SQLite file embedded in it, and it has a nice schema with invoice headers, details, customer details, etc.

Anything else in PDF would work nicely either - vector graphics, long texts, forms - all can be embedded as an SQLite file in a PDF.

<comment><USER42> Haven't you heard Mozilla foundation? You cant just embed SQLite database!!!1 Its all about those developer aesthetics! [https://hacks.mozilla.org/2010/06/beyond-html5-database- apis...](https://hacks.mozilla.org/2010/06/beyond-html5-database-apis-and-the- road-to-indexeddb/)

<comment><USER43> I think that PDFs and developers aesthetics live on different planets...

(As a side note Mozilla reasoning is that no one knows how to turn SQLite into a backward compatible standard, not even SQLite developers. So while they recognise how WebSQL is a fantastic feature it is inadequate for the web platform the same way it would be to add a python interpreter to the browser)

</comment></comment></comment><comment><USER44> I once took a job for $25 to replace some text on a set of PDFs programmatically assuming it would probably take an hour max, only to end up spending 8 hours before I found a decent enough solution. I have never touched PDF manipulation tasks after that.

</comment><comment><USER45> PDFs are terrible for screen readers for very much all the reasons listed here.

This article does make me wonder though if we'll ever get to a point where OCR tech is sufficiently accurate and efficient that screen readers will start incorporating OCR in some form.

<comment><USER2> It will have to out-pace the DRM tech that will stop you capturing pixels from pdfs deemed "protected"

<comment><USER46> Can you link to something that describes this tech?

<comment><USER47> There are way too many approaches to enumerate in a reply, but the StackOverflow answer covers just a few of the approaches I've seen used on Windows: [https://stackoverflow.com/a/22218857/17027](https://stackoverflow.com/a/22218857/17027)

The GPU approach is considerably harder to work around, fwiw. Still possible, of course.

<comment><USER48> You will always be able to decode it, today running VM or just Chrome with head on a box is simple way to bypass these techniques most of the time.

Even if DRM on video output becomes common(HDMI has them unlike VGA), Ultimately video protocols have to emit analog signal they will have to render to photons your eye can see, CAM rippers use this. It will be possible always to decode.[1] [2]

[1] Until neuralink kind of tech becomes mainstream and every content owner requires the interface to consume the content, and the interface can biologically authenticate the user .

[2] They can trace a CAM ripper, basis unique IDs embedded watermark etc, but they can never "stop" them from actually ripping, they can only block/penalize the legal source the rip originated from.

</comment></comment></comment></comment></comment><comment><USER49> If you are interested in to extract pdf Tables, I recommend you Tabula, here an example [https://www.ikkaro.net/convert-pdf-to-excel- csv/](https://www.ikkaro.net/convert-pdf-to-excel-csv/)

<comment><USER50> also tetpdf (paid) works really well (used it for extracting transactions from account statement pdfs) (demo works for 2-3 pages). I actually used a combination of tabula and tetpdf.

</comment></comment><comment><USER51> Recently: [https://news.ycombinator.com/item?id=22473263](https://news.ycombinator.com/item?id=22473263)

</comment><comment><USER52> Using ML it is possible to parse PDF pages and interpret their layout and contents. Coupled with summarisation and text to speech it could make PDFs accessible to blind people. The parsing software would run on the user system and be able to cover all sorts of inputs, even images and web pages, as it is just OCR, CV and NLP.

The advantage is that PDFs become accessible immediately as opposed to the day all PDF creators would agree on a better presentation.

Example: A layout dataset - [https://github.com/ibm-aur- nlp/PubLayNet](https://github.com/ibm-aur-nlp/PubLayNet)

and: SoundGlance: Briefing the Glanceable Cues of Web Pages for Screen Reader Users [http://library.usc.edu.ph/ACM/CHI2019/2exabs/LBW1821.pdf](http://library.usc.edu.ph/ACM/CHI2019/2exabs/LBW1821.pdf)

A related problem is invoice/receipt/menu/form information extraction. Also, image based ad filtering.

</comment><comment><USER53> In Linux, pdftotext works pretty well for the basic extraction needs I've had over the years. I have many times used this extremely simple Ruby wrapper [https://gist.github.com/rietta/90ae2187606953bee9735c00f3a6e...](https://gist.github.com/rietta/90ae2187606953bee9735c00f3a6e766).

</comment><comment><USER54> Related: Why GOV.UK content should be published in HTML and not PDF ([https://gds.blog.gov.uk/2018/07/16/why-gov-uk-content- should...](https://gds.blog.gov.uk/2018/07/16/why-gov-uk-content-should-be- published-in-html-and-not-pdf/))

</comment><comment><USER55> Just last week I was trying to extract text from a PDF and it had extra spaces between all the characters. Eventually in frustration I exported to PNG, made a new PDF, and OCRed it.

It felt dirty.

<comment><USER2> pdfsandwich helps make this a one command process

[http://www.tobias-elze.de/pdfsandwich/](http://www.tobias- elze.de/pdfsandwich/)

<comment><USER56> This looks quite useful but it appears it hasn't been updated since 2010, and I'd imagine there's been some advancements in this domain in the last decade. If anyone knows of similar tools for aligning and improving book scans, I'd love to hear of them. Thanks for this in any case!

</comment></comment></comment><comment><USER57> I remember a prior boss of mine was once asked if our reporting product could use PDF as an input. He chuckled and said "No, there is no returning from chaos"

</comment><comment><USER58> Isn't the reason extraction is almost impossible that everything in a PDF is placed inside a box? So every line has it's own box and paragraphs don't exist. Paragraphs are just multiple boxes placed together. The context of sentences is completely gone.

<comment><USER59> Basically. The text placement syntax is essentially "put this glyph or string of glyphs at these coordinates". So it could even be as granular as a textbox for each individual glyph.

</comment></comment><comment><USER40> I've (partially) done this (for ASX filings, not EU).

God what horrible mess it is. They are right about OCR being the best approach sometimes, but then there is tables. Tables in PDFs are.. well there is active academic research into the best way to extracting data.

</comment><comment><USER7> Going the other way is terrible too. I had to work on making text in pdfs that our software generated selectable and searchable. PDF simply doesn't have any way to communicate the semantics to the viewer, so things like columns, sidebars, drop letters, etc. Would all confuse viewers. It didn't help that every pdf viewer uses its own heuristics to figure out how text is supposed to be ordered. Ironically, Acrobat was one of the worst at figuring out the intended layout.

<comment><USER60> I've worked at a html > pdf engine, I don't remember that much issues with text itself beyond making the pdf metrics the same as html so that linke breaks occurred on the same places.

did you write the column one after another (first all left, then all right) or going left right line of text then top down?

<comment><USER7> It was a few years ago, but I think it was all one column then another.

And it worked ok in some viewers, but not others.

</comment></comment></comment><comment><USER61> PDF are a pain, and Konfuzio is an AI software to make PDF content machine readable. On our journey to structure PDF content or even scans we have been supported by large enterprises in the banking, insurance and audit industry. We are in closed beta for data scientists, so feel free to request a demo and free user account.

[http://www.konfuzio.com](http://www.konfuzio.com)

Disclaimer: I am the co-founder of Konfuzio, a start-up founded in 2016 based in Germany.

</comment><comment><USER62> While it still has issues, I have pretty good luck using pdftotext, then in the cases where the output isn't quite right, adding the --layout or --table options.

<comment><USER63> Yeah, I've had good luck with pdftotext -layout in the past. And in one case I've used pdftohtml -xml (plus some post-processing), which is useful if the layout is complex or you want to capture stuff like bold/italic.

</comment></comment><comment><USER64> Despite all recent OCR, AI and NLP technology, PDFs won't make "computational" sense unless you know in advance what kind of data you are looking for. The truth of the matter is that PDF was designed to convey pretty "printed" information in an era where "eyes only" was all that mattered. Today the PDF format just can't provide the throughput and reliability that interop between systems requires.

</comment><comment><USER65> > Copying the text gives: "ch a i r m a n ' s s tat em en t" Reconstructing > the original text is a difficult problem to solve generally.

Why not looking for stretches characters with spaces between them, then concatenate, check against a dictionary and if a match is found, remove the spaces.

> "On_April_7,_2013,_the_competent_authorities"

Same here.

<comment><USER66> It's doable, but it's only the easiest cases. If it's a one off script, or some automation you give as a base to a human to review afterwards it can be a good first step.

If it's supposed to be a somewhat final result, ran against a dataset you have little control over (people sending you PDFs they made, vs. PDFs coming from a know automated generator) you'll hit all the other not so edge cases very fast.

Like, otherwise invisible characters inserted in the middle of your text, layout that makes no logical sense and puts the text in a weird order but was fine when it was displayed on the page, characters missing because of weird typographic optimization (ligatures, characters only in specific embeded fonts etc.). Basically everything in the article is pretty easy to find in the wild.

</comment></comment><comment><USER67> I built and successfully sold an entire start up around solving all of these problems and more. I'm interested to learn about projects where I can contribute my knowledge. Hit me up if you need some help with something significant. contact details in my about.

</comment><comment><USER68> Probably preaching to the choir, but for me step one is to use popper tools. pdftotext -layout

</comment><comment><USER69> I've written a little overview of the open source options for text extraction available in C# [https://dev.to/eliotjones/reading-a-pdf-in-c-on-net- core-43e...](https://dev.to/eliotjones/reading-a-pdf-in-c-on-net-core-43ef)

At some point I need to port PdfTextStripper from PDFBox, it seems to be among the most reliable libraries for extracting text in a generic way.

</comment><comment><USER70> PDF should really be called UPDF (Un-Portable Document Format). Everybody who is using PDF for data exchange makes it clear that he/she is either incompetent or actively trying to make data exchange impossible. It is no accident, that many governments publish statistics as PDF files (instead of CSV for example), which makes it very hard to parse and search through the data.

</comment><comment><USER71> The same interpretation hassles crop up when trying to extract text from PCL documents. This happened to me when implementing a SaaS feature allowing customers to use a printer driver to insert documents into a document repository.

The weirdest one? an application where the PCL code points used the EBCDIC character set. For once, being an old-timer helped.

<comment><USER35> EBCDIC; EBCDIC... Now that's a name I've not heard in a long time.

</comment></comment><comment><USER42> >PDF read protection

Click print, copy all you want in the print preview window. Brilliant protection scheme.

</comment><comment><USER72> I really wish this company had some sort of product. However I spoke with them and they are essentially just a dev shop that specializes in PDF manipulation. Just to save other people some time if you think they simply have a product you can use.

</comment><comment><USER73> Since there are so many books available both as PDF and in other formats, where it's easier to extract the text, shouldn't it be possible to train a neural network on them?

</comment><comment><USER74> What I hate the most are those job sites they extract the text from your PDFs. They are close to 99% failure rate, requiring me to manually tweak the content they extracted

<comment><USER75> * Please upload an up-to-date resume detailing your education and job history.

* Please fill out this series of forms on our website detailing your education and job history.

* Congrats, you have been selected for phone screening! Please be prepared to discuss your education and job history with the recruiter.

:'(

</comment></comment><comment><USER76> I wonder if there is a flag or feature one can set in {major-vendor text editor} when exporting to PDF to make it as accessible and machine-readable as possible?

</comment><comment><USER77> What software is the current leader in OCR solutions?

<comment><USER40> Abbyy FineReader is generally better than any of the major cloud vendors in my testing.

See also this previous discussion: [https://news.ycombinator.com/item?id=20470439](https://news.ycombinator.com/item?id=20470439)

<comment><USER78> Also quite expensive since it is at the head of the pack IIRC. There is probably some value in making a competitor with new deep learning techniques provided you have a sufficiently diverse training set. It would take years to build tho.

<comment><USER40> I've done some work on deep learning approaches - specifically table extraction.

It's not at all obvious how to make this work - there is a lot of human judgement involved in judging what a header is vs what are values, especially with merged header column/row columns.

<comment><USER78> Yeah I remember Abbyy also has an interface to define layouts for this kind of problem. I.E., this thing is a table and here are the headers etc.

Sorry, I was not trying to say deep learning would be a substitute for all such issues, just that new approaches may help a smaller team build those tools more efficiently.

I don't know if Abbyy combines its layout tool with training a model for customers, but it seems like a reasonable thing to build and expose.

</comment></comment></comment></comment></comment><comment><USER79> Isn't the solution to render a PDF one wants to extract text from, with a high dpi, and then OCR the result?

<comment><USER35> Have you ever looked at the output of an OCR engine? Character (T) at position 1.03409 38.953934 Character (h) at position 9.89402 38.927831 etc. And hypothesized line, para etc. boundaries that (at least in my experience) might have come from a Ouija board.

</comment></comment><comment><USER80> "What's so hard about PDF text extraction?" \- if Weird Al ever wrote an Elvis Costello parody

</comment><comment><USER81> At our firm, evolution.ai, we prefer always to read directly from pixels. Gives you much more dependable results, for the reasons layed out here.

</comment><comment><USER82> excellent text extraction using pdfbox.apache.org

</comment><comment><USER83> PDF is so bad for representing text that when you tell people about it, they think you're trolling or wrong.

[https://news.ycombinator.com/item?id=24112836](https://news.ycombinator.com/item?id=24112836)

<comment><USER84> Representing text isn't PDF's job. PDF is meant to be "virtual paper".

<comment><USER85> Yup, the use case of PDFs is "export to PDF and check it immediately before sending to the printer".

All other use cases of PDF are better served by other formats.

<comment><USER86> > All other use cases of PDF are better served by other formats.

If that were the case, we would all be using PDF/A, or better still a subset of that. The fact that PDF has much more stuff in the spec than that suggests that a large number of people find PDF to be the best format for what they're trying to do.

<comment><USER87> > a large number of people find PDF to be the best format for what they're > trying to do.

Or they just don't know any better. I could certainly see this being the case for many office workers.

<comment><USER86> I mean, I'm certainly an advocate for using PDF/A. I agree with you about what PDF _should_ be used for, but it's hard to imagine the spec got that inflated even though nobody thought it needed all those features.

<comment><USER87> As we all know, it is an Adobe technology first and foremost. That could explain why somebody thought it needed those features, despite the fact that it really shouldn't.

More clarification of what I mean, in the form of a dangerous conversation:

OfficeCorp Exec: Hey adobe, you know what would be great! Fillable form fields! And dynamic stuff!.

Adobe Sales: Great! Wonderful feedback, we can do it!

Edit: Fix tone. Or try to. Maybe I'm just overthinking what I say?

</comment></comment></comment></comment></comment></comment></comment><comment><USER88> Tldr, text can be encoded as plaintext, image or binary.

<comment><USER89> There's a little more nuance than that. Even if text is drawn using plaintext data there's no guarantee that the characters/words appear in the correct order or have the proper white space between them.

<comment><USER88> The best method is probably to render the PDF and use OCR.

<comment><USER90> Unfortunately that's obnoxiously inefficient if you're trying to run it through text-to-speech in real time.

</comment></comment></comment></comment><comment><USER91> Personally I think PDF text should NOT be extractable. It is used to show printed document.</comment>


Templates in plain English - <AUTHOR> http://cppbetterexplained.com/templates-in-plain-english/ <comment><USER2> Ok jpg instead of copy-able source. Bad move.</comment>


If Facebook/Google wanted to programm you - <AUTHOR> ... how would they do that, if they wanted you to engage in behaviour that is beneficial to you, while preserving that addiction that keeps you sticking with them?<p>Example: They want to help you be less fanatic about race, gender, etc....
right now the algorythm suggests more and more extreme views to keep you hooked, but it's not doing something beneficial to you apart from providing the links <comment><USER2> Relevant: [https://journals.sagepub.com/doi/full/10.1177/17470161155995...](https://journals.sagepub.com/doi/full/10.1177/1747016115599568)

>The Facebook study, entitled Experimental evidence of massive-scale emotional contagion through social networks (Kramer et al., 2014), was a collaborative endeavour between Facebook and Cornell University's Departments of Communication and Information Science. In it, Facebook researchers directly manipulated Facebook users' news feeds to display differing amounts of positive and negative posts from the people they followed in order to determine whether their subsequent posts were affected by the positivity or negativity of the set of posts they were viewing. This effect, that more positive or negative posts read by a user could change their own emotional state positively or negatively, is the 'emotional contagion' referenced in the article.

Other links:

[https://www.theatlantic.com/technology/archive/2014/06/every...](https://www.theatlantic.com/technology/archive/2014/06/everything- we-know-about-facebooks-secret-mood-manipulation-experiment/373648/)

[https://www.bbc.com/news/technology-29475019](https://www.bbc.com/news/technology-29475019)

</comment><comment><USER3> While the goals are slightly different, I highly recommend reading[1] about the methods used by GCHQ's JTRIG unit (including the full Snowden document[2]). They are not really about _specific_ metho9ds to manipulate groups of people, but the general ideas behind solving that type of problem.

A particularly important topic discussed in [2] is "Attention Management". The first step humans use to process sensory input is deciding which (small) part of that data is important. We end up _blatantly ignoring_ [3] most sensory data, because our capacity for attention is surprisingly small _and very easily manipulated[4]_ if you know how to flood someone's input buffers[5].

The more troubling problems start when it isn't a human deciding how to manipulate attention, but instead it's a bunch of machine learning trying to "optimize engagement" (attention).

[1] [https://theintercept.com/2014/02/24/jtrig- manipulation/](https://theintercept.com/2014/02/24/jtrig-manipulation/)

[2] [https://theintercept.com/document/2014/02/24/art- deception-t...](https://theintercept.com/document/2014/02/24/art-deception- training-new-generation-online-covert-operations/)

[3] how long does it take you to notice the difference between these two flickering frames: [https://www.cse.iitk.ac.in/users/se367/10/presentation_local...](https://www.cse.iitk.ac.in/users/se367/10/presentation_local/Change%20Blindness_files/plane.gif)

[4] [https://www.youtube.com/watch?v=GZGY0wPAnus](https://www.youtube.com/watch?v=GZGY0wPAnus)

[5] Apollo Robins is shockingly good at using banter to DoS someone's attention: [https://www.youtube.com/watch?v=dTa7rC1oUnk](https://www.youtube.com/watch?v=dTa7rC1oUnk) , [https://youtu.be/1kkOKvPrdZ4?t=1882](https://youtu.be/1kkOKvPrdZ4?t=1882)</comment>


Humans are scary - <AUTHOR> http://miniver.blogspot.com/2016/04/humans-are-scary.html?m=1 <comment><USER2> I like this as a way to consider that maybe we'll be the super powerful alien race that others races should be hiding from.</comment>


White dwarf flies around a black hole every 28 minutes - <AUTHOR> https://arstechnica.com/science/2017/03/white-dwarf-flies-around-a-black-hole-every-28-minutes/ <comment><USER2> Every 28 minutes! I can't begin even imagine the kinetic energy of that orbit. It's just stupefying. Not to mention the centripetal force maintaining the white dwarf in that orbit. Every time I think outer space is done blowing my mind, I read another story like this.</comment>


Why Python is a perfect language for Kibitzr - <AUTHOR> https://kibitzr.github.io/why-python.html <comment><USER2> It seems like every language you learn teaches you a new facet of programming.

I've never picked up Python but kind of always meant to -- what's it teach you?

<comment><AUTHOR> For me it's mostly pep 20 [https://www.python.org/dev/peps/pep-0020/](https://www.python.org/dev/peps/pep-0020/) conveyed through builtin and third party libraries

</comment></comment><comment><AUTHOR> Kibitzr is kinda opening a new niche in tooling.</comment>