The Troubling Dean-to-Professor Ratio - <AUTHOR> http://www.businessweek.com/articles/2012-11-21/the-troubling-dean-to-professor-ratio? <comment><USER2> I hope some enterprising journalist works out some way to wiggle in there and try to figure out exactly what all these administrators are doing, because to me, it's not a merely a rhetorical question. What are all these administrators doing? I'm 100% serious. Really, what on Earth are they doing?

A great deal of the truly hard work of running a University, like scheduling several thousand students into several thousand classes, is done either entirely by or with great assistance from computers. In the absence of prior knowledge of the situation, had you asked me whether a university would need more or fewer administrators today per student or professor than in 1960, I would have immediately answered "fewer".

Moreover, to be clear, I'm not trying to make an implicit argument here that they must be doing worthless things. That argument, if I were to make it, could only come after this question is answered. Further, I'm not asking for people to conjecture what they may or may not be doing from the outside; I can do that as well as the next guy. But if you have direct experience, I'm all ears. I'm especially interested in hearing about the experiences of people who are deep down in the system, removed from direct student or professor interaction, not a lab assistant or counsellor whose contribution is obvious.

<comment><USER3> They're honestly not doing much at all. These are quite cushy positions. Work usually gets sloshed around and offloaded onto staff who make much less but work much more.

For example, at my public law school (1,000 students), there was the dean (who many suspected had an alcohol problem and appeared drunk at commencement) who makes $330,000 per year. His job is to just go around the country/world selling the school to "recruit" (we don't actually need more faculty) in order to boost our "prestige" so we can climb up in the almighty US News rankings. He doesn't engage with faculty or students at all.

Then below him is another Dean, who is a prick. He makes something like $280,000 and also doesn't do much at all. He is also a "recruiter." At least he answers emails when you have a question.

Then there are other Deans, like Dean of "Student Affairs." She makes $130,000. Mind you this is just for a law school of 1,000 students situated on a larger university campus that will have its own Dean of Student Affairs as well. I have no idea what she does. She does get dressed up every day in fashionable clothes and walk around the building chatting up people. That's all I ever saw her do.

Below that is the "Director of Student Services" or something like that. This guy works his ass off, has a mediocre basement office, but literally runs the school. What does he earn? $80,000. Anytime anyone had a question, they had to go to him. After exams, he gets a pile of USB keys, one for each exam. Takes each grade for each class and logs it, curves the grades, reports them, etc.

It's all a big scam. A big one. Tuition at this school has risen every year -- they want to build a new building (Why does a law school need a fancy building? All you need is a room and a professor). They think it will boost their "ranking." Nevermind that "ranking" doesn't do anything for the local job market, which is still stagnant, and every year new graduates enter it looking for work to pay their $150,000 in loans (at 8% interest, non- dischargeable).

This school isn't even that bad compared to the other law schools out there, namely the private ones. Even a top school like Columbia states on its own site that students will spend $250,000 for their law degree (this includes a conservative housing budget).

The only thing more mind blowing than the scam itself is that we all sit around and don't seem to care as these people eat our young alive. In fact, they worship these institutions. It's their "alma mater," after all, it's like family, isn't it? Some even DONATE to them! The federal government enables the scam by flooding the industry with easily obtained credit, with no restrictions on tuition charged, and even letting for-profit "schools" get the money if they want.

This is a racket that is in big need of disruption. Where does it start? Perception. Once we stop perceiving these degrees as meaning anything, once we stop valuing prestigious names as absolutely required, and look more at individuals themselves when hiring, the racket will die.

<comment><USER4> It is in need of disruption, and it's a wicked problem. There's no simple solution and no easy starting point. Perception is definitely something that needs to change, but unfortunately behavior is much harder to change.

Another aspect of the problem is tenure. It is (extremely) difficult to fire a tenured professor (or admin for that matter). When things get tough universities have a hard time downsizing if necessary.

A resulting, and somewhat funny, problem is what universities tend to do with tenured professors who just aren't good at what they do anymore. What do you do when a professor with tenure isn't good at teaching or research? You promote them.

<comment><USER5> Private universities exist.

If public universities were mismanaged and inefficient, we would see private ones providing a better product at a lower cost.

<comment><USER6> I would agree with this statement, but the prestige situation is a self ensured stagnation. You do see big private competitors such as Phoenix, but in order to compete they lose out in the prestige competition.

I think the only real disruptive players in the business timelines that Universities operate in are the technical school, but they have been around long enough to play the prestige game also.

Phoenix for example will need to operate for 100 years to even begin to sit at the table with the nostalgia crew.

<comment><USER7> Phoenix is probably does more harm than good to the for-profit or at least disruptive private college community. Their targeting of federal subsidies that often account for ~90% of their income and the stories of them doing things that basically trick low income people into spending more money than they will get from the degree give everyone a bad impression. Even if these stories are all exaggerated (which I have no reason to believe they are), they aren't entirely false and it makes people (reasonably) dubious of other for- profit schools. Someone should have made an elite for-profit school first that then expanded to what Phoenix could be.

It's like google and other companies with driver-less cars. Sure, their cars might be safer than drivers statistically, but if you get a few high profile accidents that a driver could have stopped, they will probably get swamped in bad PR and get delayed many years. Luckily, this hasn't happened.

random example from google: [http://www.post- gazette.com/stories/opinion/perspectives/vet...](http://www.post- gazette.com/stories/opinion/perspectives/veterans-fight-back-unscrupulous-for- profit-colleges-are-bilking-them-of-their-gi-benefits-661697/)

</comment></comment></comment></comment></comment></comment><comment><USER8> I'm currently a student at a public university, where we just finished a multi-million dollar renovation of our five-story library. Previously all floors were study space for students, and you could almost always find study space, even during finals. The fourth floor is now exclusively administrative office space, and things are getting cramped.

This story has played out over and over in various corners of the school. An office would spring up in a dorm here or there. The IT department wanted to expand one of their offices into our heavily-used English tutoring center. It's an epidemic, as far as I'm concerned...

<comment><USER9> Sounds like my university (University of Calgary). The library is nice but packed. For a period they even had to reopen the old library for extra study space. Next victim appears to be Machall which is our student centre/cafeteria. The university is even planning to sell off unbuilt land for development. All while we have some classrooms with painful seating!

On the plus side our tuition is only ~7k. In that regard I have no right to complain compared to many of the USA universities.

</comment></comment><comment><USER10> Holy crap. I went to Purdue from 1999-2004 I was out of stat and I paid around $17k in tuition and $4k in room+board, so it did nearly exactly double since I went there. It's one thing to hear that college costs are going up really fast, but it puts it into perspective when you see the actual school you went to.

I picked Purdue partly because I could go there without incurring any debt, which wasn't the case with a lot of private schools. Had I been born a few years later I would have had to go to an in-state school to do this.

<comment><USER11> It will be interesting to see what a conservative guy like Mitch Daniels does as president. As an ag and engineering school I'd guess Purdue is not as uniformly liberal as some universities, but he certainly won't get a completely welcoming reception.

<comment><USER10> When I was there at least, it was slightly right of mainstream US, and _way_ right of the typical University.

</comment></comment></comment><comment><USER12> At the college where my wife is a professor (small liberal arts school of perfectly average quality... very typical for its type) the pay of the administrators is considered competitive while that of the professors is considered less than competitive.

The average administrator makes a low 6 figures, the average professor makes about $50,000 to $60,000, teaching positions are constantly frozen, contributions to retirement frequently withheld (this way they can claim they maintain salaries while still cutting costs... looks better on the surface) while the number of administrators just keeps creeping up.

I suspect that as smaller colleges turn to a more business oriented way of operating, out of desperation to increase enrollment and raise funds, this trend will continue. The administrators at my wife's school operate more and more like C-level executives at a corporation while the faculty are becoming the worker bees of the hive. The two groups used to occupy a similar level with full professors being on par with the higher level administrators.

<comment><USER13> Could this be perceived value of skills? Admins/managers can work outside the education sector and so a wider market governs rates?

<comment><USER14> I've never understood what it is that university administrators can do outside the academic sector. As far as I can tell, they're basically the most useless bureaucrats on Earth.

</comment></comment></comment><comment><USER15> Disclaimer: I'm on the "dean" side of the ledger as a contractor for an Australian university.

One of the bloggers on my little network calculated that approximately 28c of each dollar of public funding for Australian universities is spent on academic staff:

[http://clubtroppo.com.au/2012/11/19/the-university- coalface-...](http://clubtroppo.com.au/2012/11/19/the-university-coalface- gets-28-cents-in-the-dollar/)

He went on to discuss reform options and possibilities:

[http://clubtroppo.com.au/2012/11/21/university-reform- part-i...](http://clubtroppo.com.au/2012/11/21/university-reform-part-i-what- are-the-options/)

[http://clubtroppo.com.au/2012/11/23/university-reforms- part-...](http://clubtroppo.com.au/2012/11/23/university-reforms-part-ii-the- barriers/)

[http://clubtroppo.com.au/2012/11/26/university-reform- part-i...](http://clubtroppo.com.au/2012/11/26/university-reform-part-iii-so- what-can-be-done/)

</comment><comment><USER16> Well, for non-STEM majors at least the value of going to a top college is 70% being able to say you were good enough to get in. So you've got a positional good and thanks to the government lots of people who've been loaned the money to bid it up. And since most of these places are non-profit you won't see high returns to investors, meaning the one place for the money to go is to superfluous staff.

There's an element of Baumol's cost disease[1] too, but according to a paper I read recently (I'm too lazy to dig it up, sorry), that only accounts for >20% of the rising costs of tuition.

[1]<http://en.wikipedia.org/wiki/Baumol%27s_cost_disease>

</comment><comment><USER17> I have no idea why they're using "average full professor salary" as a proxy for comparison, since full professors are a tiny minority of faculty.

<comment><USER18> Presumably because Deans are about the same level as full professors? I doubt that a fresh PhD would be appointed a dean anywhere.

</comment></comment><comment><USER19> Something I have always wondered is why deans and other administrators make such great salaries relative to professors. My hunch is that it is simply because they can, but I feel like I must be missing something. Does anyone know if these tough jobs in ways I don't appreciate?

<comment><USER10> A lot of the really high up positions are heavily involved in fundraising, and the argument there is that "If person X can increase our endowment by just 1% more than person Y, it justifies the huge salary"

<comment><USER19> Ah, the football coach argument.

<comment><USER10> Nah, the football coach argument is much easier: There is lots of money in college football, and nearly all of it is banned from going to the players. Therefore there are nearly unlimited funds for coaches.

</comment></comment></comment></comment><comment><USER20> And they say education needs more money. Not when so much of it goes to waste like this, it doesn't. If they audited every school from top to bottom, they'd probably save at least 50% of what those schools are getting right now.

<comment><USER5> > And they say education needs more money. Not when so much of it goes to > waste like this, it doesn't

What numbers are you using when making that statement? Do you think that the additional cost of administrators is anywhere close to the amount in reduced funding from the state?

</comment></comment><comment><USER21> The scale of this problem varies quite a bit based on institution type and based on the primary sources of revenue for the institution. A medium size two year institution has a very different (lighter) organizational structure from a large four year residential institution. In general, emphasis on the full featured residential experience leads to more "programs" (product bundling) and more administrators to run those programs. This might not be a bad thing especially for endowment driven institutions that have chosen a high cost strategy. These schools can afford to pay the large number of administrators and pay their faculty well pretty much without trouble.

The problem truly arises with the second tier of institutions that are trying their best to "keep up" with first their peer institutions and then the top tier in the US News rankings. Their budgets are generally more tuition driven which starts to put some constraints on what is possible. These schools build the same expensive high touch programs and resources to attract the best students and try to keep up in the rankings. Higher ed is like a school of fish - everyone wants to swim together but only a few really have the money to do it. This is where the cracks in the model really start to show.

There are definitely opportunities for disruption and change in higher ed. Intelligent application of technology both to teaching and learning and to the other elements of the current bundle (research, certification, etc.) will lead to disaggregation and (probably) reduced employment for higher education administrators. The wildcard is how accrediting organizations will respond to the changes on the horizon. They act as a brake on innovation, and since most federal and state funds are tied to accreditation, institutions are loath to change too quickly.

</comment><comment><USER22> I see this as a direct boost of confidence to Khanacademy, MITX, and the likes.

</comment><comment><USER23> _The bottom line: From 1993 to 2009, U.S. universities added bureaucrats 10 times faster than they added tenured faculty._

\-- Makes sense.

Non-profit=The profits go out the door in "costs"

</comment><comment><USER24> Wow that is almost, but not quite as bad as the Memphis City School system, which has more administrators per student than any school system on the planet.

</comment><comment><USER24> I think it is rather laughable all the tea party kooks here stepping up to justify this kind of bloated top heavy organization. Little wonder US universities are headed south in terms of the quality of their grads. All part of the same sports virus infects all of US education. It it makes money, it gets priority. No wonder US education costs are so high, yet teachers and students get shafted.

</comment><comment><USER25> Unfortunate that this thread will probably dissolve beforehand, but coincidentally there's an IAmA on Reddit by an assoc. dean of admissions at Colgate about to start: [http://www.reddit.com/r/IAmA/comments/13vel5/im_karen_gianni...](http://www.reddit.com/r/IAmA/comments/13vel5/im_karen_giannino_senior_associate_dean_of/)

</comment><comment><USER26> Have a read of Daniel Bernstein's travails at UIC.

<http://cr.yp.to/uic.html>

It's excruciatingly entertaining.

</comment><comment><USER27> The problem extends way beyond the college level. I once saw a very small school (graduating fewer than 20 students from sixth grade) add a non-teaching position every year for five years.

</comment><comment><USER27> And let me add that there were probably too many deans 40 years ago.

</comment><comment><USER28> Sounds like a bubble.

</comment><comment> nsp</comment>


Tech lessons learned from the wisdom of crowds - <AUTHOR> http://news.com.com/2100-1014_3-6143896.html?part=rss&tag=2547-1_3-0-5&subj=news <comment><AUTHOR> A very interesting bit from the article:

"Microsoft: When Todd Proebsting, director of Microsoft's Center for Software Excellence, tested a prediction market internally, managers quickly gave it their blessing.

The goal: to have 25 members of a development team predict when a Microsoft product would ship (this was an internal product, not one sold externally). The prediction market was set up in August 2004, and the product that "had been in the works for a long time" was scheduled to ship in November 2004.

Each "trader" received $50 in their account to start with, and was told that the more accurate their prediction, the more money they would make. The market opened with an initial price of on-time delivery set to 16 2/3 cents.

"The price of 'before November' dropped to zero right away," Proebsting said. "The price of 'on time' in about two to three minutes dropped to 2.3 cents on the dollar." Translated, that's more than 30-to-1 odds against on-time delivery.

Then the woman who was responsible for scheduling started trying to convince her colleagues who were buying and selling future delivery dates. "She was able to talk (on-time delivery) up to around 3 cents," Proebsting said. "People really enjoyed moving the price...They loved this."

"The next day the director comes into my office and said, 'What have you done?'" Proebsting said. But further investigation showed that the product actually was behind schedule, even though nobody was telling management, and it eventually shipped in February."

It is amazing what the crowds are capable of! Often times, groups can be smarter than the smartest individuals in them.</comment>


Facebook refuses again to pay exploit bounty - <AUTHOR> http://ehrazahmed.blogspot.in/2013/09/exploit-for-removing-any-account-from.html

<comment><USER2> I'm a software engineer at Facebook working on security and privacy. This is simply a hoax. The html source shown in the video clearly says "No test user was deleted". We've verified in our logs that the victim account was manually deactivated by visiting [https://www.facebook.com/deactivate.php](https://www.facebook.com/deactivate.php). Anyone can visit [https://www.facebook.com/whitehat/accounts/](https://www.facebook.com/whitehat/accounts/) and verify that the query parameter used by this endpoint is selected_test_users not selected_users. We've also audited our code to verify that there's no variant of this exploit that works against that endpoint or any other that we've found. In fact, the most recent code change to this endpoint was in April and was routine maintenance that had no security implications.

</comment><comment><USER3> Hope he doesn't try to prove his point like the other guy did with Zucker's account.</comment>


Habitat — A new approach to automation - <AUTHOR> https://www.habitat.sh/ <comment><USER2> I've been waiting for this announcement for a few weeks. Since the home page is quite lean on an actual explanation, let me give a rough summary of interesting points as i understand it. I apologize in advance if I have anything wrong.

1\. It is a from-scratch source build system written in rust that borrows a huge amount from nix. This means repeatable software builds with isolated dependencies.

2\. It attempts to move some configuration management primitives into the build pipeline so that the deployment config has enough flexibility for real- world use.

3\. It intends to replace single host process supervision with a distributed model based on gossip.

This project is most definitely ambitious, and I'm not sure the whole goal is realizable but some aspects are particularly interesting.

1\. The build process seems quite useful for building containers, especially if it gets some traction. The main drawback of nix is that no one knows how to use it. If this gets the backing of the hordes of chef-aware system administrators, we may have a good solution for isolated repeatable builds, especially if they focus on deterministic reproducible builds[1] as well.

2\. Distributed systems are notoriously hard to configure in a general way. It seems like every company has their own magic combination of config management, monitoring, and scripting for keeping systems like zookeeper running. Its pretty clear that we don't have the right primitives for sharing this work. I don't think a container management system has the right primiteves for this either, because I don't think you can solve for all distributed systems in a generic way. You really need custom logic for the particular software you are trying to distribute. I'm not convinced that a gossip-based process-supervisor is the best approach for this, but it could give us a place to start collaborating on these primitives.

[1] [https://wiki.debian.org/ReproducibleBuilds](https://wiki.debian.org/ReproducibleBuilds)

<comment><USER3> >1\. It is a from-scratch source build system written in rust that borrows a huge amount from nix. This means repeatable software builds with isolated dependencies.

Except the builds aren't at all repeatable in a meaningful way, judging by the example code on the home page. The 'do_build' function runs 'npm install'. This means that 1) the build container has network access (i.e. it's nondeterministic by definition) and 2) the declared dependencies do not fully describe the dependency graph. So, I really don't see how much it could possibly borrow from Nix if it threw out the most important part of it.

<comment><USER2> They used npm install in their example code? Well that was a poor choice. Having dealt with npm and node apps in my own build system, I have not found a good solution for deterministically building node apps. Does nix have a solution for this? Thankfully the prepackaged software with reasonable source build systems don't do craziness like this. See the redis package for example:

[https://app.habitat.sh/#/pkgs/adam/redis/3.0.7/2016061320525...](https://app.habitat.sh/#/pkgs/adam/redis/3.0.7/20160613205253)

<comment><USER4> The nix solution is npm2nix, which runs npm against a package.json file and generates a nix expression for all the npm modules you need.

This mostly works, but occasionally you need to override the generated derivations to add implicit dependencies that npm doesn't capture, or otherwise tweak packages that rely on quirks of npm.

<comment><USER5> So basically it "does not work"...

<comment><USER6> Well, let's be clear, if it doesn't work, you know at compile time, not at runtime. I don't care how broken things are at compile time if I'm guaranteed that everything will work at runtime. That's kind of the whole point of nix. It's not impossible to write code with errors, but you should not be able to build as long as the errors exist.

<comment><USER4> Also, it _does_ work. It's just not completely automatic.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> The article link points directly to the Habitat demo. For those of us that don't already know what Habitat is, a better place to start might be the about page: [https://www.habitat.sh/about/why- habitat/](https://www.habitat.sh/about/why-habitat/)

Habitat seems like an entirely new Chef product to enable what they're calling Application Automation.

Relevant quote: _" [...] automation must travel with the application, rather than be provided by the platform. Everything the application needs, from build dependencies, run-time dependencies, configuration, dynamic topologies, deployment strategies, secrets management, security auditing, and so on belongs with the application, because we're doing it for the application. Runtime and infrastructure layers exist to support the application, but the efficient building, deployment, and management of the application are decoupled from those layers."_

Incredibly ambitious.

<comment><USER8> Containerization is disrupting a lot of their current market, so they need to adapt in some fashion. Given the fact that the only hammer they have at hand is ruby systems automation tooling it is not a big surprise they are aiming at these targets.

Unfortunately, IMHO, the scope is a mile wide and an inch deep in terms of what chef brings to the table here. I can walk down that list and either think immediately of another tool that is dedicated to the specific task or the needs are so slight that a few ansible or bash scripts take care of the need. Sounds like they are aiming to occupy some space in the general "tooling & orchestration" arena, but it will be an uphill climb.

<comment><USER9> This is an excellent point - kind of goes against my earlier comment. Maybe they have to move this way because that's their only choice - if they don't move into automating this, containers will disrupt them (or already has)

</comment></comment></comment><comment><USER10> Found this elsewhere, interesting to note, Habitat looks to be built mostly with Rust

[https://github.com/habitat-sh/habitat](https://github.com/habitat-sh/habitat)

<comment><USER11> This is Adam - Rust has been amazing to work with, and was the prefect choice for this platform. It's great.

<comment><USER12> Could you elaborate a bit more on why you chose Rust, and why you think it was the right choice for this project?

<comment><USER6> On the heels of this, I've been exploring Rust a bit lately but haven't been able to find any solid patterns / style guides for idiomatic Rust. Do any such resources exist? For example, the official rust book describes Arc, Mutex, etc but.. Are there more of these in the standard library? What purpose do they serve? What are some good idioms to apply when using them? I can surely make them work, but maybe I should be using some other thing?

<comment><USER13> You might want to join users.rust-lang.org.

Like most languages idiom is learned while working with the language. Something I've found for instance is to try and use Iterators to get around sharing rules as they can encapsulate sharing patterns in a lot of cases.

The specific modules you mentioned are for multi-thread support, those are only necessary for sharing memory between threads.

Anyway, I think this is off-topic for this thread.

</comment></comment></comment></comment></comment><comment><USER14> Home Automation is what first came to mind when I read the title. With this in mind, reading the blurb on the website sounded like, oh they're talking about building home automation software...

"build automation, IT automation, deployment automation, etc" would have saved me some confusion. Or simply compare yourself to other solutions out there will help me understand what category this new piece of utility and name fall into.

<comment><USER15> Yep. Would made perfect sense for the name as well.

</comment></comment><comment><USER16> I wonder if this project is doing too many things. It includes a package builder _and_ a supervisor. Is there any benefit to integrating the two?

For a project that's smaller in scope but addresses the part about packaging the automation with the application, Joyent's ContainerPilot [1] looks interesting.

[1]: [https://www.joyent.com/containerpilot](https://www.joyent.com/containerpilot)

</comment><comment><USER17> The GitHub project: [https://github.com/habitat- sh/habitat](https://github.com/habitat-sh/habitat)

</comment><comment><USER18> I'm not sure what I'm looking at. What is being "automated" here? The headline and home page don't give much context.

<comment><USER7> The main points around automation seem to be[1]:

* Flexible versioning and configuration rollout

* Ability to deploy anywhere

* Ease of management

Concretely, they're touting their gossip-based configuration and topology management layer as a way to send configuration updates to service groups, and then letting the service groups apply the changes to all the group members. Contrast this with a more traditional way of propagating changes across a network by enumerating every single host/VM/container. I'd imagine another large component of the application automation would come in the form of tooling allowing the ability to deploy anywhere (bare metal vs. VM vs. container vs. IaaS/PaaS providers), but haven't seen that just yet. Maybe it's being covered in the live announcement.

[1] [https://www.habitat.sh/about/why-package-automation-with- app...](https://www.habitat.sh/about/why-package-automation-with-app/)

<comment><USER18> So but am I automating devices in my home/building IoT-style? Or am I automating deployments of software? Or hardware? "gossip-based configuration and topology management" means nothing to me unless I know actually what the overall context is here.

<comment><USER7> From what I'm reading, I believe this is primarily geared towards trying to automate software deployments.

</comment></comment></comment></comment><comment><USER19> I watched the videos, did the tutorial, read the docs, read the tech site stories but I still don't see why I would use Habitat over the current Docker workflow and ecosystem (including third party) for building, shipping and deploying applications. Both are a type of packaging that puts the application, its dependencies and configuration in a bundle that can run exactly the same everywhere. Docker has Dockerfiles and docker-compose.yml files, Habitat has plan files (seems like a shell script, Dockerfiles and compose files are much simpler). Docker has a central repository/registry "docker hub", habitat has depots. "docker run redis" == "hab start core/redis", etc... etc...

The docs/articles say habitat apps can run in containers, but why bother if your app can already be run in a container that achieves the same thing with even greater isolation (cgroups). Why put a self contained app in a self contained container application?

I'm not trying to put down Habitat, I'm trying to understand its role. As I currently see it, why would I choose this over the more mature Docker environment with a massive established ecosystem and user base?

Again I'm just trying to understand what I'm missing, if I'm missing something please point me in the right direction.

</comment><comment><USER20> This looks really interesting. Im waiting for 8:30PDT for the full announcement though!

Im curious about how it works under the hood, it seems to solve 'all the problems' ;)

My skepticism comes from some of chefs past product releases, which have seemed rather, half baked.

Products like chef metal (provisioning), pushy (push jobs) or delivery...

This is in no way negative as I am a long time chef user and fan of chef.

</comment><comment><USER21> Hmm, I very much like the concept but when I see the need for supervisors and a habitat service, it makes me wonder if this is really what is says it is.

If I need to pre-install all these services on my hosts before any of this will work, then how is that really different from uniformly deploying all my applications on an application server such as WebSphere? An EAR too contains all my configuration data, my bindings, my security settings etc. So...if you uniformly run WebSphere everywhere in your enterprise, it sort of accomplishers the same thing.

HOWEVER, if the habitat configuration is capable of talking to my F5 load balancer to create a resource pool, self register my app with my corporate DNS, create a DB schema and populate it with my tables, talk to my networking equipment and open all relevant firewall ports and do so when the application is deployed to a new environment, as well as undo all these actions if it's ever moved elsewhere, then you have piqued my interest...

<comment><USER22> There's no pre-installation - the supervisor is bundled with the application deployment itself.

</comment></comment><comment><USER20> Theres a live announcement now [https://bignews.chef.io/](https://bignews.chef.io/)

</comment><comment><USER9> I love the Chef ppl, but...

Is it just me or are there a ton of other folks doing this? The idea of introducing yet another layer of stuff is scary, and there's lots of work left to do on Chef. In a lot of ways, they feel just like Nomad (coincidentally also on the front page)... I wish they would spend some time on their core products versus expanding into an overfilled market.

<comment><USER23> That's what I thought, too. Chef (software/ecosystem) needs a bit more love IMHO.

However it shows that Chef (Company) is still able to design, build and ship a new product, as an organization. That's a very positive sign. Until recently I thought when comparing Chef with Hashicorp: Hashicorp is so much faster in building and releasing usable stuff but they seem to feel the pain in growing too fast and too broad in terms of products.

So, in the end: It's a great thing, just also don't forget to put more love into Chef (software) and don't dissipate developers/time/budget :)

</comment></comment><comment><USER24> As nice as the 'console' on the page is, the fact the window gets cleared when you click to go to the second step makes it look like it lost what you did on the first page. To me that makes it look like it's broken - the steps should maintain the state and build on each other, not start over each time.

</comment><comment><USER25> Is this like Docker?

<comment><USER11> Nope! You can use it with docker, and we think it makes the experience better. We love docker.

<comment><USER26> Is there anywhere that actually lays out the tech that powers habitat? ie:

This thing uses kubernetes or this thing uses mesos This other thing uses docker/rkt/containerd/lxd This is the set of things we wrote in rust, etc, etc.

</comment></comment></comment><comment><USER27> How does this compare with Heroku's buildpack system?

</comment><comment><USER28> That jumping header is quote annoying, I guess the developer did not calculate a proper padding to accomodate for a sticky menu once it becomes fixed positioned

</comment><comment><USER29> We updated the link from the live demo page ([https://www.habitat.sh/try/](https://www.habitat.sh/try/)) to the homepage.

</comment><comment><USER30> Ah, another limiting abstraction layer to learn and to fight.

Bash is plenty good.

<comment><USER3> But all of the package recipes are just Bash scripts, which I think is just terrible, btw. Using a real programming language would have been a much better choice.

<comment><USER31> > Using a real programming language would have been a much better choice.

The problem is that everyone disagrees on what a tolerable real programming language is. Some refuse to use Python; others Ruby; others Scheme; others Lisp.

At least every can tolerate shell more or less.

</comment></comment></comment><comment><USER32> Can I also buy a sofa? [http://www.habitat.co.uk/](http://www.habitat.co.uk/)</comment>


Google Squared is Live: Who Knew Structured Data Could Be So Unhelpful? - <AUTHOR> http://www.readwriteweb.com/archives/google_squared_is_live_who_knew_structured_data_co.php <comment><USER2> >> Hopefully it's still early days for all of these kinds of tools.

Definitely - I think each product is a great concept, but they need to learn a lot from the way people are actually using the service, and use that to refine their programs.</comment>


Salt: Combining ACID and BASE in a Distributed Database - <AUTHOR> http://muratbuffalo.blogspot.com/2015/02/salt-combining-acid-and-base-in.html

<comment><USER2> This is neat and all, but when can I get a database that does this?</comment>


Memo to Stripe: Winning the hearts of Valley startups is not winning payments - <AUTHOR> https://pando.com/2014/01/24/memo-to-stripe-winning-the-hearts-of-valley-startups-is-not-winning-payments/ <comment><USER2> The article is from 2014 when Stripe's valuation was $1.75b, a 44x multiple to revenue. From the article "sky-high valuations for unproven companies aren't unheard of in the Valley."

Now in 2019 valuation is $22.5b. I think the article is no longer current.</comment>


Ask HN: Would you sign up if a Phone Number is required? - <AUTHOR> Would you create an account on a App that requires a Phone Number?
The reasons for asking for a Phone Number are to prevent abuse, better control spam and make the accounts more secure.
More apps are now requiring a Phone Numbers to create an account. I also know some people are against sharing to much information.
Would you provide a number if it was justified? Are the reasons above justifiable? <comment><USER2> Heck no. They will sell the number to marketers etc. They will get hacked and let my number out. They will send texts or calls or alerts. Their policies will change when new company buys them.....

</comment><comment><USER3> Why require the number? So you can send an activation/recovery code over SMS? Virtual numbers and pre-activated SIMs are easily accessible nowadays. You'd mitigate but not prevent abuse nor spam. In terms of security, SMS is NOT recommended to communicate sensitive data.

That being said, only a tiny few of us are concerned about privacy, most app users aren't, otherwise FB wouldn't have a billion users.

<comment><AUTHOR> These are really great points. The thought process is that it will be used for authentication and recovery. I would say Virtual Numbers are a concern but most cost money and an email is free to create. Which was why I said better control spam. If they want to pay money to be a clown and troll I've already won anyway.

I think you are right about the concern of privacy. The scary part is most don't even read the Terms and Policy Agreement.

</comment></comment><comment><USER4> I've given my phone number to microsoft and google but that's about it.

<comment><USER3> Which is to say: everyone has your number now ;)

</comment></comment><comment><USER5> For a professional service where I can give the number of my employer, I might, but I'd be reluctant, as a few companies will start calling every day in an attempt to sell you the service.

</comment><comment><USER6> That's strictly need-to-know. So for most it's not. but then, I also use a fake empty Facebook account for sites that use social login.

</comment><comment><USER7> no. I don't trust many apps other than maybe apple. I assume anything I enter will be sold to the highest bidder.

</comment><comment><USER8> i think it's a good idea. It makes bans more permanent.</comment>


DataSelfie – hack your Facebook data - <AUTHOR> https://vimeo.com/201178499 <comment><USER2> How is the analytics done locally? If it's not, saying the data is stored locally seems potiental meaningless.</comment>


Interactive Demo of Purely Functional Turing-complete language "Fexl" - <AUTHOR> http://fexl.com/demo/?sample=print

<comment><AUTHOR> Feel free to push the Evaluate button to see the actual output.</comment>


NEETS - Navy Electricity and Electronics Training Series - <AUTHOR> http://jacquesricher.com/NEETS/ <comment><USER2> Since there's no immediate explanation I imagine someone will click through to the comments to find one. This really should be at the top of the linked-to page (if the author is present to see this comment).

\</comment><comment></comment><comment>----

The Navy Electricity and Electronics Training Series (NEETS) was developed for use by personnel in many electrical- and electronic-related Navy ratings. Written by, and with the advice of, senior technicians in these ratings, this series provides beginners with fundamental electrical and electronic concepts through self-study. The presentation of this series is not oriented to any specific rating structure, but is divided into modules containing related information organized into traditional paths of instruction.

The series is designed to give small amounts of information that can be easily digested before advancing further into the more complex material. For a student just becoming acquainted with electricity or electronics, it is highly recommended that the modules be studied in their suggested sequence. While there is a listing of NEETS by module title, the following brief descriptions give a quick overview of how the individual modules flow together.

\</comment><comment></comment><comment>----

Slightly larger Module descriptions can be found here: <https://gist.github.com/4053057> (didn't want to pollute commentspace by pasting it all here)

</comment><comment><USER3> A friend of mine gave me a dog eared copy of the Navy electronics training manual when I was in high school and it was fabulous. Here was a book written to approach the largest audience with a variable set of backgrounds. It helped me pass my Novice exam and later much of the stuff I just "knew" confounded my fellow students while pursuing my EE degree. Nice to find these updated versions although the motors & generators section is much as I remember it.

</comment><comment><USER4> This is exactly the type of education experience we should be optimizing as a society: self-study, in-depth, yet practical material geared toward leaving the student able to perform some service of value in the world.

I'm a huge fan of a broad liberal education, but many educational institutions have lost the plot. When a free PDF on the web can double your salary ability and a 50K education from the local university will leave you only in debt? The feedback loop for higher education is broken.

I'd love to see this integrated into something with more video material. In fact, if I had money for philanthropy, putting together a job-oriented free electronics course would be near the top of my list of things to do.

Side note: there are instructions for sending completed training material to the Navy to have them grade it. I wonder how many packages the Navy gets from students all over the world consuming this material and looking for a grade?

</comment><comment><USER5> Thanks! I've been slowly teaching myself electronics for the past few years (as a hobby). I recently got an oscilloscope and it opened a whole new world for me!

I'd also suggest the following playlist on YouTube. The army/navy/air force videos are amazing! <http://www.youtube.com/playlist?list=PLAA9B0175C3E15B47>

<comment><USER6> I am doing the same thing. Any tips for cool things to do with the oscilloscope?

<comment><USER5> I cut one of the ear buds from an old pair of headphones. Connected it to the scope and got a cool visualization of the music being played.

Beyond that, the biggest benefit of the Oscilloscope was it let me learn about and build oscillators and op-amps. For instance, I made a relaxation oscillator a few days ago (produces low frequency square waves).

I recently also got two electronics learning lab. I got the 300-in-1 from Elenco. Radioshack also has a similar one. The material that comes with the lab is strange - on the one hand, the projects are pretty sophisticated. On the other, there is little to no explanation on how they work. But I've found that this is great for self-study. The oscilloscope has been very handy for this part.

P.S.1 Someone showed me a cool trick with an oscilloscope at maker faire. You need two signals out of phase. Go into X-Y mode and voila ... you get some cool annimated drawings :)

P.S.2 Be careful and safe playing with electricity!

<comment><USER7> Having a term to google may help others: it's called a Lissajous Figure

</comment></comment></comment></comment><comment><USER8> I'm on my phone so typing is a little bothersome, but if this is the same as the printed manuals they've had for years I would highly recommend it. Most of my study on electronics outside of hands on work was with this series. If you're thinking of taking 6.002x on edX these might help out a lot as a primer for those who might otherwise be jumping in blind.

</comment><comment><USER9> My brain immediately parse the acronym as Not in Education, Employment or Training(NEET) before I read the rest of the title.

</comment><comment><USER10> Thanks for posting this. Not only does it appear to be an excellent resource, but downloading all of the PDFs finally gave me a reason to try out curl.

<comment><USER11> I used this small Python program to download the entire set of PDFs:

      import urllib
      for i in range(14173, 14197):
          url = "http://jacquesricher.com/NEETS/%d.pdf" % i
          print url
          urllib.urlretrieve(url, "%d.pdf" % i)

<comment><USER12> With curl:

      curl -O 'http://jacquesricher.com/NEETS/[14173-14196].pdf'

Here's a way to do 4 downloads in parallel:

      for ((i=14173; i<14197; i++)); do
        echo http://jacquesricher.com/NEETS/$i.pdf
      done | xargs -n 1 -P 4 curl -O

Another trick for parallel execution is to generate a makefile then use make -j.

</comment></comment></comment><comment><USER13> I was the electrical officer on a Navy ship. Since my undergraduate degree was in physics, I thought it was incredible that my guys were learning this stuff. Glad to see it's made it to the Internet.

</comment><comment><USER14> I went to school in San Diego where there are a lot of former sailors. The former Navy tech guys always dominated in my EE classes after learning from these materials.

</comment><comment><USER15> Interesting to see these posted on HN. I studied NEETS modules while working on radar and weapon systems electronics as Data System Technician in the Navy (DS). They cover a lot of ground and most teach things in a pretty simple way (as the often need to train people who weren't the most academically successful). They are a bit dry, though.

</comment><comment><USER16> I almost joined the Navy's <http://en.wikipedia.org/wiki/Nuclear_Power_School> . Back then it seemed like it would be really technically challenging but I chickened out at the last moment. :( oh well. Given how my life has gone so far maybe it would have been a good option.

It seems like this NEETS training would have also been neat. Thanks for this link. The nuclear school material is SECRET, so I won't expect to find it, lol. :)

</comment><comment><USER17> Here's a quote from the introduction:

"If there are roots to western science, they no doubt lie under the rubble that was once ancient Greece. With the exception of the Greeks, ancient people had little interest in the structure of materials. [...] Recent experiments have revealed that there are, indeed, several basic particles, or building blocks within all substances."

The document doesn't read like a usual textbook, it feels lighter though maybe a little less rigorous. Interesting read nonetheless!

</comment><comment><USER18> This is great. Thanks for posting. Much less rambling than the art of electronics!</comment>


Stop asking me to sign up - <AUTHOR> http://www.gkogan.co/blog/stop-asking-me-to-sign-up/

<comment><USER2> I believe the buttons are correct - a signup with a modern startup web software (ie. an app) feels much less like "starting to use our software" but more like "sign up for this email campaing we're running". Signing up usually requires an email address and then results in "Greg from Blahblah-App" follow- up emails. To me it feels like many startups are truly so desparate for some users, that they will follow "best practices" and spam the hell out of anyone who will leave their email address. I am not a fan of this culture.

</comment><comment><USER3> Some good points...</comment>


First vaper reported dead after mysterious lung disease - <AUTHOR> https://www.theverge.com/2019/8/23/20830412/vape-death-lung-disease-cdc-e-cigarettes <comment><USER2> I find the difference in vape use in the US and UK really interesting. We (the UK) have much lower rates of teen vaping, and the NHS is actively pushing vaping as a pillar of its stop-smoking programmes. Vaping just doesn't seem to have much of a cool image, it's just that thing that your ex-smoker friends do instead of smoking.

Meanwhile in the us there seems to be a full blown moral panic going on, higher use by teens, and it seems to have some kind of "cool" factor.

I'm curious whats going on with these lung problems, and why it seema to have suddenly become an issue.

<comment><USER3> The reporting on this has been super shoddy. I've read countless articles that fail to mention the "I bought some sketchy stuff on the street and put it in my vape pen" aspect of it or don't mention it until the last sentence. There is a clear anti-vape movement in the US. I suspect it stems from busy bodies that are frustrated that their anti-smoking campaign, while mostly successful, has been countermanded by the vaping movement. But it's just a hunch.</comment></comment>


Better filler text - <AUTHOR> I'm tired of "Lorem ipsum". Anyone have any better (more amusing etc.) filler text to use during development? <comment><USER2> [http://loremgibson.com/](http://loremgibson.com/)

[http://www.buzzwordipsum.com/?paragraphs=4&type=sentences](http://www.buzzwordipsum.com/?paragraphs=4&type=sentences)

[http://www.cipsum.com/](http://www.cipsum.com/)</comment>


The "What Can I Do For You?" Culture - <AUTHOR> http://jessekatz.tumblr.com/post/10519620171/the-what-can-i-do-for-you-culture

<comment><USER2> I moved here from LA. I found a small group of nice people, but it overall, it felt like I was in the movie Cruel Intentions</comment>


The Curse of Dimensionality in Classification - <AUTHOR> http://www.visiondummy.com/2014/04/curse-dimensionality-affect-classification/

<comment><USER2> Another good dimensionality reduction technique to consider is Latent Dirichlet Allocation. I use this approach for natural language or other "bursty" data sets. "Bursty" data sets are characterized by having Zipfian distribution over features, but certain long-tail features achieving a higher probability of multiple observations given initial observation in an instance. For example, "armadillo" is relatively rare, but an article mentioning an armadillo once has a high chance of mentioning it again.

A cool thing about LDA is that it allows you to express the latent characteristics of a given document as a point in Euclidean space. This gives you the ability to use spatial distance metrics such as cosine distance to express document similarity. I specifically use this for recommending large- scale UGC communities based on their latent characteristics. Furthermore, since you've turned your language data into spatial data, you're able to use spatial classifiers such as SVMs more effectively over natural language data, which is normally a bit better suited for Bayesian classifiers.

I'm a huge fan of Gensim for its LDA library. It's even capable of distributed computing using Pyro4. It's relatively trivial to deploy an LDA pipeline for extremely large datasets using EC2 and the Boto AWS library.

Edit: If you haven't heard of it, scikit-learn is an awesome Python library for highly performant machine learning using Python's C extensions for numerical computing (scipy, numpy). It's easy to take the data you get above and perform learning on it using the classifiers provided.

<comment><USER3> Can you provide a link to an article discussing how you can treat the latent characteristics as a point in Euclidean space?

<comment><USER2> Here's some basic background: [http://faculty.washington.edu/jwilker/559/SteyversGriffiths....](http://faculty.washington.edu/jwilker/559/SteyversGriffiths.pdf)

</comment></comment></comment><comment><USER4> Neural nets are a great way to reduce dimensionality. In particular, deep autoencoders. here's an old Hinton paper on it: [http://www.cs.toronto.edu/~hinton/science.pdf](http://www.cs.toronto.edu/~hinton/science.pdf)

</comment><comment><USER5> I really like the idea of the Web site as a whole: explaining concepts from computer vision in a simple way.

When I was starting my masters course I was interested in learning what the concept of bag of words in computer vision was all about. Although it is straightforward technique, there are few examples on the Web explaining how to implement it (clustering the feature vectors and etc.)

<comment><USER6> Although not widely known, the best way to learn about specific computer vision topics in detail is usually through the websites for tutorials held at the premiere computer vision conferences. These are CVPR, ICCV, and ECCV. For example, here are the tutorials held at ICCV 2013: [http://www.iccv2013.org/tutorials.php](http://www.iccv2013.org/tutorials.php)

If you click through, you'll see that most of them have links to slides, and some even have video coverage and/or links to software as well. They're also usually presented by experts in that area.

Finally, note that even if the most recent conferences don't have relevant tutorials for what you're looking for, you can still often find good material by going back further in time (e.g., tutorials presented 2 or more years ago). Although the state of the art may have advanced since then, the fundamentals usually don't change very frequently.

</comment></comment><comment><USER7> I'm familiar with this idea - but it's nice to see explained with cute little puppies and kittens.

</comment><comment><USER8> one counter-example: face recognition using 100k features ([http://research.microsoft.com/pubs/192106/HighDimFeature.pdf](http://research.microsoft.com/pubs/192106/HighDimFeature.pdf))

<comment><USER9> Not really. The article mentions that using linear methods (i.e., LIBLINEAR) is one way to avoid the curse. LIBLINEAR is specifically designed for situations in which you have many features and relatively few training instances. When using a linear classifier it may make sense to simply generate as many features as you can, and then use, i.e., lasso regression in order to do feature selection. [http://www.csie.ntu.edu.tw/~cjlin/liblinear/](http://www.csie.ntu.edu.tw/~cjlin/liblinear/)

<comment><USER10> But the article used a linear model to demonstrate the curse, and the model was overfit just with 3 dimensions. There is clearly something missing: for example, for text data it is not uncommon to have thousands or hundred thousands of dimensions, and algorithms work fine.

I think the missing piece is regularisation. It doesn't have to do feature selection and actually reduce the number of dimensions, but you're right that using L1 for such data is usually a good idea.

<comment><USER11> The article had very few data points, that's why it worked with 3 dimensions. The deciding factor is how N (effective number of data points) compares with p (effective number of features).

</comment></comment></comment></comment><comment><USER12> is there some kind of test to know if we are past the optimal number of dimensions? I guess overfitting could be detected by the ratio between volume and area of the classification boundary.

<comment><USER13> You could make a plot like Figure 1. Look for the turning point (do some calculus if you can, i.e. d(perf)/d(dim) = 0).

<comment><USER14> Derivatives require continuity. It would be sufficient to simply look at which number of dimensions gave you the best cross-validated classification rate.

</comment></comment></comment><comment><USER15> Can someone explain to me Figure 6 please? How does projecting the 3d space (and 2d green plane) lead to the regions around the cat's heads?

<comment><USER16> I will try:

Imagine flattening the 3D space (Figure 5) down onto the 2D space of the floor.

Then, the image is trying to illustrate which sections of the 2D space have been selected as 'cat' by the classifier (green plane).

Visually, you can imagine that the cat heads have each pushed down a chunk of the green plane (the dog heads have held the green plane up off the ground).

This illustrates that the third dimension was important for separating out the cats: they certainly aren't separated by a clean line in those two dimensions.

However, it also illustrates that the separation might be slightly contrived: looked at in 2D, the green plane seems to have plucked out the odd cats correctly but without logic.

One goal is to show that using a high number of dimensions will guarantee that you can separate dogs and cats, but that this is just an over-fitted solution to the data set that you have: it will not continue to work when you apply it to further data.

<comment><USER15> > Visually, you can imagine that the cat heads have each pushed down a chunk > of the green plane (the dog heads have held the green plane up off the > ground).

Thank you, that helps. I was mentally projecting the whole plane down and could see why it wasn't all-green (or all-not).</comment></comment></comment>


SQLite timeline items related to "json" - <AUTHOR> http://www.sqlite.org/src/timeline?r=json

<comment><USER2> The JSON support in SQLite is in a very early stage of design and development still. There is a lot of cool stuff coming. But there is also still a lot of work to do.

Please check back in a few weeks.

If you would like to leave comments describing the kinds of problems you would like JSON support in SQLite to solve, I promise to read them all and give them careful consideration.

<comment><USER3> Hi, I spend _at least_ $60 x 12 months /year on a NoSQL database that HN absolutely _loves_ to hate. I don't mind the service, but my first love was SQLite (thanks to Android, and Python), and my second love was JSON (partly why I ended up choosing the database, and switched to the NoSQL database). Some of this is probably slightly off-topic, but since I _love_ the product, I would very happily use SQLite once again if for:

1) JSON support, as is being offered as described in the post. I want to do the following in SQLite (look to MongoDB(!) documentation for syntax): a) Inserts. You know, regular inserts. b) Upserts. The 'hated' database (Mongo, obvs) has this easily-configurable upsert ['replacements'?], that's incredibly easy to use. c) Bulk upserts, inserts and deletes.

2) Cheap-ish online database that you can probably use Amazon S3 for, which I am going to make my institution (University/employer) pay for anyway. Things I care for are: a) you take care of all the deployment details and provide support when I need one and b) don't throw me off your service when I cross your limits because I've overused them because I'm a data-hoarder but I'll lose my life's worth of work if you throw me off your service.

I would also love it if your offered a choice of setting up couple of indexes very easily (using JSON, obvs), and offered a visual explorer of data in the database (relevant because a _certain_ mongo service offers an awesome data explorer).

Please understand that a very large majority of your users is _NOT_ doing complex joins or using other advances features of the database. As an engineer, I would _love_ it if my users used the most advanced features and let me code the funnest, cutting-edge algorithms, but that's not what I want as a user. All I ask is simple inserts, deletes, upserts, an online service to handle them, and an ability to do this all using JSON. This is not going to make the most exciting days of your work-life, but I would be an extremely happy person if I were able to use these services.

Sidenote: Since I may never have chance to an SQLite person again, I'll bring this up even though this is likely WAY off-topic. I am a researcher who likes to HOARD data. I like storing data in my Uni's server, that has a limit on file size/total size. _IF_ you created the ability to store the same database across several files, I would be most grateful to you.

PS: Love your documentation, guys. You really set the bar for the awesomest, most easily-understandable documentation of extremely technical and advanced subject matters in the field. All my attempts to make my code readable are attempts to merely emulate your awesomeness. : )

<comment><USER4> I doubt SQLite can ever be used on S3. There are two problems with that, first is that to preserve consistency SQLite requires working locking, which S3 by default can't provide (for sane reason using SQLite on NFS is not advised).

Second problem is that S3 does not allow to modify files, each time you make a change you need to reupload the whole thing.

I think you might have better luck trying DynamoDB.

Regarding the post, I don't know what to think. The SQLite was banned that way because it supposed to be light, I feel like adding things like JSON is going away from that principle.

</comment></comment></comment><comment><USER5> Oh hey, neat.

[http://www.sqlite.org/src/artifact/950ed4e8deb8ad4c](http://www.sqlite.org/src/artifact/950ed4e8deb8ad4c)

So from looking at the test, you can get JSON out of select queries now, not a full blown JSON type like PostgreSQL.

<comment><USER6> The test is written in TCL, which has slightly non-obvious syntax. In essence it tests SQL functions json_array and json_object which seem to return JSON arrays and objects as string.

</comment></comment><comment><USER7> If you are interested in mobile JSON databases, we've been working on our network-enabled native databases for iOS, Android, C# and PhoneGap, since 2010. The library size is less than 500kb and we are compatible with the Apache CouchDB sync protocol. We also have a server side Sync Gateway that has fine grained access control, webhooks, etc.

Did I mention everything is open source? There is a mini-hack here that will let you choose your platform and be up and running in ~20 minutes with what's basically the stem cell you're gonna be winning hackathons with now that you know.

[https://github.com/couchbaselabs/mini- hacks](https://github.com/couchbaselabs/mini-hacks)

Ps we couldn't have done it without SQLite as we started out using that for our storage engine. Now we are moving to ForestDB because it's more like the data structures we present in our API.

<comment><USER8> Seems like a bad place to run a sales pitch.

</comment></comment><comment><USER9> It's a curious thing that Dr Richard Hipp co-created the "unql" specification then effectively did nothing more with it after that.

[http://www.infoq.com/news/2011/08/UnQL](http://www.infoq.com/news/2011/08/UnQL)

<comment><AUTHOR> There's code, you can compile it and try it out. For fun I'm thinking to write a little python wrapper:

[http://unql.sqlite.org/index.html/wiki?name=UnQL](http://unql.sqlite.org/index.html/wiki?name=UnQL)

</comment></comment><comment><USER10> Newbie here. Can somebody explain me what does the JSON SQL function extension? Any example?

<comment><USER11> At first glance it seems like some functions that like json_object which can return (query) results as a JSON array or json_insert to I presume puts JSON directly into SQL tables.

Nothing too spectacular and nothing you couldn't do with any decent library already, but some convenience.

<comment><AUTHOR> Every big feature starts out small.

</comment></comment></comment><comment><USER12> This is so exciting! To anyone who doubts the value of this effort - do you have any good option for an embedded, document-based DB system already? Thanks Dr Richard Hipp!

<comment><AUTHOR> UnQLite. Not affiliated with SQLite, but its an embedded json document store. Supports a cool procedural query language.

[http://charlesleifer.com/blog/introduction-to-the-fast- new-u...](http://charlesleifer.com/blog/introduction-to-the-fast-new-unqlite- python-bindings/)

</comment></comment><comment><USER13> I like it but I am just a bit skeptical about why there is an investment being done in building this.

Aren't they already too late to the party? I mean, there would have been tonnes of libraries which would do this given how popular SQLite is. Shouldn't they have left it to those libraries and stuck to investing on the core database thingy ( too vague but I hope you get what I mean ).

:)

<comment><USER14> I'd rather have this built in SQLite, and with its standard code quality and extensive testing, than have to use some additional library.

And what does "too late to the party" even mean? Who said the party ended or ever ends?</comment></comment>


Ask HN: What are the most linked-to sites from HN? - <AUTHOR>

Has anyone looked into the sites that are most linked-to from HN?
I'm also curious about the ones that are most popular (some combination of number of times submitted and avg # of upvotes).
Another thing I'm wondering about is if there is a significant impact on popularity of a submission if it isn't mobile-optimized.
Couldn't find anything about this through searching... <comment><USER2> For your first question: [https://news.ycombinator.com/item?id=7079653#up_7083023](https://news.ycombinator.com/item?id=7079653#up_7083023)

(Remembered that I had commented on this when it came up earlier; found it by doing an hnsearch that included the names of some popular sites.)</comment>


Ask HN: Will 3D printing make warehouses a thing of the past? - <AUTHOR> In retail: From mid/late 1900's-2000 it seems the trend was retailers who could spread out their physical locations the fastest to get the most discount stores in front of customers was the winner; Over the last ~20 years the equation seems to have changed to be who spreads out their warehouses full of discounted goods to ship to customers is the winner. Is 3D printing the next trend? <comment><USER2> Lean supply chain management and contract manufacturing is likely the big upcoming trend in this space, as it is easier and cheaper than ever to have experts create the thing that you want or source the components or whatever.

I think the QA as well as fit & finish of 3D printing is not high enough yet (and unclear whether the investment to get to that point is worth it) for mass produced consumer goods. More likely it will spawn a big uptick in hardware ventures (already has) since it also massively reduces the cost of prototyping and can reduce the cycle time of early modeling and prototyping.</comment>


Ask HN: How does the service "Sell Hack" work? - <AUTHOR>

I'm very curious how this service (http://www.sellhack.com) might compile and verify email data. <comment><USER2> Test it on yourself</comment>


Ask HN: Is New York ever going to beat Silicon Valley? - <AUTHOR>

Is New York's 'tech renaissance' going to ever put it on par with Silicon Valley? Or is it all just a bunch of hot air?
I mean specifically regarding web startups, as opposed to hardware stuff where there doesn't seem to have been as much attention. By 'on par' I mean in a position where it can produce at least a handful of multi-billion dollar web exits, even if SV ultimately has a greater volume.
Are there any key blogs that cover the NY Tech Scene in particular? <comment><USER3> It's possible, but unlikely for the same sorts of reasons that the Bay Area is ever likely to be 'on par' with New York in terms of theatre. Bay Area culture accepts failed startups just as New York culture accepts failed plays. Likewise, the couch surfing ramen fueled lifestyle is fine for programmers in the Valley just like actors in New York.

Every city has a cluster of socially acceptable reasons for being broke. In New York, being an entrepreneur tends to be not among them.

</comment><comment><USER4> Hard to say, but I think one of the main hinderances is availability of capital, especially to early stage startups. It's not that there isn't money in NY, it's just generally far more available to later-stage companies (and arguing which is better is a whole other issue). Clearly, if a startup is having trouble raising on the East coast, they have two options: give up, or move West. I think that as long as this is happening, the scales are likely to continue to tip in SV's favor.

</comment><comment><USER5> Take into account the large start-up culture employers - Google, Apple, FB, Yahoo to name a few. And Standford as a thinktank. Funding bank: VCs and angels. These are hubs where talent revolve around and from which money flows. In the absence of established hubs, thinktanks, and VC pool, there will be little neck-neck competition with SV/SF. Which is to say it is better to have diverse startup cultures in the US, some small, some monoliths, each different and unique.

</comment><comment><USER6> As far as I can tell, the current tech/startup rankings are SF > The Bay Area > Seattle > NY.

SF/The Bay Area win because everybody you meet will likely be another startup/tech person. The constant influence drives creativity, spontaneous idea generation, acquisitions, pr, hr, etc. Also, acceptable weather.

Seattle's tech scene is 90% Amazon employees. You have other smaller companies and startups out there, but Amazon dominates. To start a company, you'll be fighting Amazon for employees. If you hire an ex-Amazon employee, you have to spend six months deprogramming them from broken Amazon processes and lifestyles. Bonus for Seattle: no personal or corporate income tax (just a 9.5% sales tax).

NY's tech scene is almost hidden. You won't run into it day to day. You'll run into actors, models, fashion designers, lawyers, quants, tourists, and people trying to find their place in the world. You won't hear someone talking about json or rails or functional programing in coffee shops or on the subway.

Summary conclusion: If you're a high strung aspie nerd who wants to win win win -- SF or The Bay Area. If you don't need tons of employees or can import employees from around the country, Seattle beats SF to avoid taxhell and dirtyhell and smellyhell and homelesshell. If you want to date, have a life, have culture, and cross pollinate your company with The Real World, NY wins.

Transit conclusion: In The Bay Area or Seattle, you'll want to have a car. If you never leave SF, you can get by without a car. In NYC, only crazy people have cars (just wait until you can afford a door-to-office helicopter).

Place-to-live conclusion: SF's housing/rental market is broken. Seattle has more sane rents and places to live, but nothing is connected by transit. NY has adjustable rent depending how far out you want to live, and everything connected by transit.

Starting a company conclusion: Seattle wins because of no income tax, still having a relatively high techie population, and young we-can-do-this atmosphere. California rips almost 10% out of your paycheck for personal taxes and more with corporate taxes, but you're surrounded by the best startup people in the world. NY has a more sane personal income tax level around 6%, but you'll be importing startup people or herding them yourself (just convince some actors to act as programmers until they can actually program).

Actual conclusion: If you want the easy build-to-flip or fail-by-acquihire approach, be in SF. If you want to be around good people and not be taken advantage of by your local governments, use Seattle to start/grow your company. If your company deals with real people having real problems and you want to live an adventure, use New York City.

Am I qualified to have these opinions? Check my profile.

<comment><USER7> As far as I can tell, the current tech/startup rankings are SF > The Bay Area > Seattle > NY. - <USER2>

CORRECT VERSION: Peninsula > South Bay > East Bay > SF > NY > Boston > Austin.</comment></comment>


Facebook Announces FBJS2 - <AUTHOR> http://developers.facebook.com/news.php?blog=1&story=189 <comment><USER2> I've never written a FB app but aside from getting some of facebook's canned ui components, why would one use FBML and FBJS?

It seems better to just run your whole app in an iframe and just use html/css/jquery and have complete control while hitting the REST api yourself.

<comment><USER3> Here's a good explanation:

<http://www.ccheever.com/blog/?p=10>

FBML allows you to do specific Facebook-only features easily, like friend requests and searching for your friends. It also has documentation and tutorials, which is not trivial when making the FBML/iframe decision.

</comment></comment><comment><USER4> Why did they create fbjs in the first place? It gave js development in fbml an unnecessary learning curve. Other companies had already mastered the js sandbox by then so there was no reason for these js wrappers.

<comment><USER5> The only other example of a JS sandbox that works that I know of is Caja and Caja hadn't been publicly announced or released when FBJS originally launched.

In general, its pretty useful to be able to sandbox JS on a page so you can allow 3rd parties to do scripting on your page without putting their stuff inside iframes.</comment></comment>


Ask HN: How do you plan and track the books you want to read? - <AUTHOR> Hey,
I think I have a problem. My want-to-read list on Goodreads currently has more than 2100 books. I probably have more than 50 online classes that I want to take.
I constantly find new things that I want to learn about. I have multiple lists on Goodreads with titles of "Urgent reads" and "The most urgent reads". Things get reprioritized so often that I don't actually end up reading or learning much.
I recently started building syllabuses around different topics for myself to put together relevant things together and then I try to force myself to focus on a single syllabus at a time.
I am experimenting with another system for books. I have a table of books, where when I get excited about a book, I upvote a book. Only after a book is upvoted more than 3 times, I download a sample of the book (if available on Kindle). Only after I read a book, I buy the book. This helped a little, but not enough. Now, I have 10 books that I started reading.
I am curious about how others approach this. What is your framework? How do you self-discipline? Are there any good tools? <comment><USER2> Buy the books, have a big threatening pile of them glaring at you all the day; from time to time pick one out to read. This is the principle of the _antilibrary_ which Nassim Taleb attributes to Umberto Eco

[https://fs.blog/2013/06/the-antilibrary/](https://fs.blog/2013/06/the- antilibrary/)

</comment><comment><USER3> Some interesting thoughts here:

Rational Reading: Thoughts On Prioritizing Books

[https://www.lesswrong.com/posts/Kmch6T2YscMyLFJD9/rational-r...](https://www.lesswrong.com/posts/Kmch6T2YscMyLFJD9/rational- reading-thoughts-on-prioritizing-books)

</comment><comment><USER4> It's FOMO. Delete/un-enroll/unsubscribe from all the current "want'to'be" list. Then just go by your day and pick up things you realise you should learn. Buy a physical book on that topic and read.

Uninstall all reading apps on your phone. Less is sanity.

Make notes. Upload into Anki. Write a blog on what you learned today.

</comment><comment><USER5> I use the app "Things 3" and set deadlines, works pretty Goode... only downside: Its only available in the Apple ecosystem and its not free.

Btw.: its also my preferred Task manager

</comment><comment><USER6> by shelf and nightstand

it's nice to read from paper after looking at screens most of the day</comment>


How to sell your game online without using an app-store - <AUTHOR> http://positech.co.uk/cliffsblog/?p=884

<comment><USER2> A couple of notable quotes:

1\. _I've sold tens of thousands of games direct online, since I started in 1998_

2\. _It took me maybe 5 years before I could live from my direct sales, and was able to quit my job._

10's of thousands doesn't sound like much over 12 years, and it took him 5 years to go full time.

This reminds me of the story of Pangea software. An indie Mac shareware game company. They launched with the iPhone App Store: [http://www.bizjournals.com/austin/stories/2008/11/17/story1....](http://www.bizjournals.com/austin/stories/2008/11/17/story1.html)

 _Although his game development business [Pangea] launched in 1987, the revenue generated in six months by just two of its iPhone games has matched the retail revenue of all of Pangea's preceding personal computer games combined, he says._

In 6 months they made more than __21 years __of being an indie shareware developer. That is the promise/possibility of a highly traffic'd app store. And yes, results not typical, but I think the point still stands.

</comment><comment><USER3> There are guys like Terry Cavanagh who've had some success by making limited versions in Flash that market the real version.

The Flash industry can deliver an enormous quantity of eyeballs onto a game. My own best has had almost 6 million people play it although I track games much larger than that.

<http://www.kongregate.com/games/TerryCavanagh/vvvvvv-demo>

</comment><comment><USER4> So is anyone taking this approach with iPhone or Android apps? And to what degree of success?

It's starting to get under my skin that I lose 30% of my app's (<http://goo.gl/USEr>) revenue for nothing more than a listing in the app store and payment processing.

<comment><USER2> You can't with iPhone since the App Store is the only official way to get onto an iPhone.

I can't speak to android, but suggesting that the iOS App Store is "nothing more than a listing and payment processing" is sort of like saying you'd rather sell your widget directly from your basement rather through Wal-Mart so you don't have to give up the retail markup to Wal-Mart.</comment></comment>


Schrodinger's Cat Video and the Death of Clear Text - <AUTHOR> https://citizenlab.org/2014/08/cat-video-and-the-death-of-clear-text/

<comment><USER2> Salient point:

      The target is presented with a dialogue to upgrade their
      flash installation [via ISP-based MITM - DenisM].
      If this upgrade is accepted the malicious SWF enables the
      installation of a 'scout agent' which provides target
      validation.

</comment><comment><USER3> A related article is [https://firstlook.org/theintercept/2014/08/15/cat-video- hack...](https://firstlook.org/theintercept/2014/08/15/cat-video-hack/), via [https://news.ycombinator.com/item?id=8184061](https://news.ycombinator.com/item?id=8184061).

</comment><comment><USER4> Adopting HTTPS is very important for the Web, but the lack of HTTPS in this case is just a red herring. The real problem is that it's way too easy to exploit systems through the web browser. If the entire Web switched to HTTPS, government attackers would just find another way to deliver their exploits besides injection into plain text websites. Working on solving this problem is just as essential as transitioning to HTTPS.

<comment><USER5> _If the entire Web switched to HTTPS, government attackers would just find another way to deliver their exploits besides injection into plain text websites._

Exactly. HTTPS relies on a centralised CA infrastructure, which is probably (indirectly) under the control of government anyway, so I don't see how taking away the ability to MITM plain text connections will help; and governments would prefer that their exploits be done over HTTPS since a properly-forged certificate will look "secure", and that makes the end-users more likely to blindly trust the authenticity of the content they download... it also means that firewalls and other systems designed to detect suspicious content in traffic (as much as they're derided for being used for censorship-related purposes, ones under the control of the user can be beneficial) won't catch the exploit either. I think that would be even worse.

HTTP causes some suspecion of the content, but HTTPS gives the impression of being secure, decreasing the level of suspicion while implicitly forcing the user to trust the CAs.

HTTPS with self-signed certificates that are exchanged in-person would probably provide the most secure authentication, but that doesn't really scale.

<comment><USER6> I'm assuming, as there is a live PE executable modification payload listed, and Windows is supposed to throw warnings for unverified / unsigned updates and installers, that the CA infrastructure is already compromised.

Either that or most Windows users still just habitually click through security warnings, which I sincerely hope is no longer the case.

</comment></comment></comment><comment><USER7> With tools like these out there, what can we do to protect ourselves? Besides not downloading binaries over HTTP (which still wouldn't protect you if a CA has been compromised) what other steps can someone take? I hate how vulnerable and yet utterly essential our browsers have become.

<comment><USER8> Ship the public key by post like internet banking in the early 2000s.

This is actually how OpenBSD operates. If you buy an official CD set, the thing ships with keys which are then used to sign downloaded packages.

When the keys go via a side channel, the probability of compromise decreases considerably.

</comment></comment><comment><USER9> Maybe a stretch, but is this somehow connected:- [https://news.ycombinator.com/item?id=8160844](https://news.ycombinator.com/item?id=8160844) ?

I oddly remembered this comment from my memory trove.

</comment><comment><USER10> The internet 1.0 was designed so grad students could argue about Star Trek while pretending to publish research</comment>


Twilio (phone call web API) is crazy fun: MicroISV on a Shoestring - <AUTHOR> http://www.kalzumeus.com/2009/12/29/twilio-phone-call-web-api-is-crazy-fun/ <comment><USER2> Patrick, I have two questions:

1) If you use any IM clients that have an API for checking your activity, could you hook that up to the site for a more thorough data source than just an if statement based on time of day?

2) Why put Nginx or Passenger in front of the Sinatra app running Thin/Mongrel, instead of just having the app itself running on some port bare?

<comment><USER3> 1) I don't routinely use IM clients, because either I'm working or I'm with the people I hope would be IMing me.

2) I suppose I could do that, but I like minimizing the number of externally visible services on general principles, and Nginx is like a Russian-built tank.

</comment></comment><comment><USER4> If you like playing with stuff like this, Cloudvox (cloudvox.com) is also worth a look. You can write callflows in various languages (including straight up JSON), which makes it very nicely hackable.</comment>


Ask HN: What do you want to see in a deployment management system? - <AUTHOR>

Deployment management can be a painful, slow and repetitive task if overlooked. There are tools available to ease our suffering: chef, cfengine, puppet, capistrano. While these can help, at times they feel like monstrosities to overcome in themselves. I use chef to deploy/maintain and apps and configs on <100 servers. The initial installation and learning curve was not as smooth as I'd have liked but now its doing a good job. Even still, I feel its lacking in some ways for my needs and the amount of moving parts required to keep it running frustrates me at times.
In an ideal deployment management tool, what would you like to see? Is it ease of installation? A hosted solution?
I'm looking to write something to simplify the process greatly not only for my own benefit but for others out there as well. I would like some insight into what your needs are or what you think a good deployment tool should be capable of.
Thanks HN <comment><USER2> \- easy web-based 1-click deploy. I choose the version. Click deploy. Done.

\- Don't worry about updating database changes, I can do that manually when needed, it's too complex anyway.</comment>


Ask HN: What password manager tools do you use or recommend? - <AUTHOR> What password manager tools do you use or recommend? <comment><USER3> BitWarden has come a long way and now it does everything I need: iOs integration (in ios12), Android, extensions for all browsers, and even the (fairly pointless but customary in 2018) electron-based desktop app. It's even opensource, so one could host a private instance - which I'd be keen to do if it weren't based on the annoying Microsoft stack.

The only feature I miss is the little gamification that LastPass has, where you can check how many of your credentials had the same details - <USER2> that score day by day was a nice challenge. I was tempted to build it myself (BW has an API too) but not got around it yet.

<comment><USER4> Seconding BitWarden. It's the only open source/self host able password manager I could find that provides all the features that I wanted from closed source/hosted password managers (1pass/lastpass). The chrome extension and iOS app are great, have auto fill support unlike some other open source password managers (KeeWeb).

Self hosting it is super easy to setup, they just give you a docker compose file, and a few commands later it's ready to go. It's also fairly easy to modify the compose file to deploy it to a Docker Swarm cluster, which is how I have mine setup.

</comment></comment><comment><USER5> This won't be popular here, but I use LastPass extensively (300+ saved credentials) and works like a charm: 2FA via TOTP, incredibly reliable Chrome extension (I almost never have to copy/paste passwords from it, the auto completion aggressively works even with the websites that explicitly try to prevent it or change frequently), beautiful ios integration and seamless synchronization across 6+ devices. I take a csv backup on an encrypted usb stick monthly just in case. I don't remember any of the passwords I use (besides the LastPass master password), including the most important ones such as gmail and bank accounts (for which I also rely on TOTP whenever possible). Always automatically generated and stored in LastPass.

I would love if someone convinced me to move to something that would work just as well, for me user experience is the most important thing, I don't want to deal with complicated workflows, considering password completion is something I have to do ~50 times a day or more, so even just a few additional seconds become exponentially annoying for me.

</comment><comment><USER6> I use 1password right now. It's really nice.

If I were starting over I'd give pass and its various open source UIs/apps a chance: [https://www.passwordstore.org](https://www.passwordstore.org)

</comment><comment><USER7> I wrote my own, MemPa [1, 2].

I'm not sure I'd recommend it because it's basically iOS-only now and with no iOS12 integration, but I'd definitely recommend a deterministic approach for at least your most important sites (banks, google?, facebook?). The main reason is availability in case of real need, when for example your phone could be broken or inaccessible. I wrote about my use case here [3].

Happy to discuss deterministic approaches, please make sure to read also the counter arguments [4].

In addition, I strongly encourage setting up 2FA, especially on the most important sites. My choice for passwords is also related: I really want my password to be something I know (not something I need a device/password manager for) because with 2FA you prove already something you have. Another totally valid option is to setup 2FA on your password manager and call it the day. For completeness, I'm also building a security key called Solo, open source and FIDO2 [5].

[1] [https://mempa.io](https://mempa.io)

[2] [https://hackernoon.com/mempa-a-modern-deterministic- password...](https://hackernoon.com/mempa-a-modern-deterministic-password- manager-2c0f28fa108b)

[3] [https://hackernoon.com/how-i-manage-my-passwords- technical-v...](https://hackernoon.com/how-i-manage-my-passwords-technical- version-8549dc1bde1e?gi=c53454131b71)

[4] [https://tonyarcieri.com/4-fatal-flaws-in-deterministic- passw...](https://tonyarcieri.com/4-fatal-flaws-in-deterministic-password- managers)

[5] [https://solokeys.com](https://solokeys.com)

</comment><comment><USER8> I switched to keepassxc[1] a while back.

[1] [https://keepassxc.org/](https://keepassxc.org/)

<comment><USER9> I use KeePass with DropBox on Windows, Linux and Android.

</comment></comment><comment><USER10> A notebook or a folded up sheet of paper with hints you need to remind you of your passwords but not explicitly stated. Completely invulnerable to security flaws, it's only breachable by physical access.

</comment><comment><USER11> Im using [https://psono.pw](https://psono.pw)

its free, open source and you can host it easily yourself with docker. Full disclosure, Im the key developer behind it.

</comment><comment><USER12> I use an app called oPass, it's iPhone only though ([https://opassapp.info](https://opassapp.info)).

</comment><comment><USER13> I personally use iCloud Keychain, and it's more than enough for me.

</comment><comment><USER14> I use PasswordGorilla ([https://github.com/zdia/gorilla/wiki](https://github.com/zdia/gorilla/wiki))

</comment><comment> java-man Passwørd Safe

[https://github.com/andy-goryachev/PasswordSafe](https://github.com/andy- goryachev/PasswordSafe)

</comment><comment><USER15> Firefox native password manager. Works fine and it syncs with my phone automatically.

</comment><comment><USER16> I use mnemonic devices (Rhymes, non-wester geography, latin or greek history), with really long strings: ([https://xkcd.com/936/](https://xkcd.com/936/)) The system works pretty well. Zero password reuse. Always Remember, Can't be compromised with a password manager... the problem is the companies and their poor security! (LinkedIn, Experian)

I'm really looking forward to the days when we don't have passwords anymore.

NOTE: And they said I would never use Latin!

<comment><USER17> If Gmail gets hacked and they leak your password, thereby requiring you to change your password, how will you remember the changed password?</comment></comment>


DIY - Soldering BGA IC on PCB  - <AUTHOR> http://dev.emcelettronica.com/diy-soldering-bga-ic-pcb <comment><AUTHOR> DIY: soldering BGA ICs on PCB in your garage/lab without professional facilities. It is a headache to have a BGA IC mounted on the PCB if it is a small project. Today I want to share something about soldering.</comment>


Recent Searches To Influence Google's Results - <AUTHOR> http://googlesystem.blogspot.com/2008/04/recent-searches-to-influence-googles.html <comment><USER2> As per the comments in the article, you'd think this would be helpful, but it's actually quite annoying.</comment>