Ubuntu Unity vs. Windows 8 new start menu - <AUTHOR>

Ubuntu Unity caused a stir as the top Desktop Linux disto. Not because it does not work but because of lack of customization. During this time Linux Mint took over the number one spot. Microsoft should be watching this and how users were in shock over all the changes to gnome 3.
Windows 8 has a new start menu that they took from Joli OS,MeeGo, or Smeegol Linux. What they did was take the start menu and flattened it out to dumb it down even more to turn it into a Tablet/cloud OS. Makes me wonder what they are going to do when the tablet market crashes before Windows 8 comes out. I purchased a 27 inch iMac and two laptops, one for windows and one for Linux. Microsoft should have spent all the money in making windows 7 more secure since that has always been one of the major flaws of windows.
What are your thoughts on Windows 8 compared to Ubuntu for a productive Desktop workstation? <comment><USER2> The Unity UI, deployed this year for Ubuntu Linux, is the canary in the coal mine. Canonical wants to put Ubuntu on smartphones and has developed Unity as a touch-friendly UI - but forced on desktop users too. There has been dramatic resistance from previously loyal Ubuntu users that are either refusing to upgrade or migrating to other Linux distributions that preserve the conventional desktop experience. I expect that Microsoft will likewise face massive laptop/desktop user resistance to Metro. Note that Apple has not made this mistake - rather keeping two distinct UIs, the first for keyboard-mouse laptops/desktops and the second for touch-mobile iPhone/iPad/iPods.

One can only wonder at the gamble Microsoft is making, especially in view of the reception of Ubuntu Unity throughout 2011. Does the technical elegance of a single UI across all platforms have more merit than specialized UIs?

<comment><AUTHOR> Thank you for your comment

</comment></comment><comment><USER3> Gnome 3 makes me feel like a smartphone interface is being shoved down my throat even though I'm using a laptop. It's tolerable when I have a browser and little else opened, but it becomes hell to deal with when I have a browser, Netbeans, IM, a few folders, Word, terminal, etc. opened.

I haven't tried Windows 8 personally, so I can't really comment on that.

<comment><AUTHOR> I agree, the lack of customization is very irritating. If everything would get out of my way when I'm trying to do something productive I would be happy.

</comment></comment><comment><AUTHOR> Windows 8 wants to tie you to an email address for your login. They have multiple screens that try to force you to setup a hotmail or live account. With all the social networks I wonder how many people use multiple email addresses.</comment>


Musk dares Alameda county to arrest him - <AUTHOR> https://www.marketwatch.com/story/musk-vows-to-reopen-california-factory-2020-05-11?mod=mw_latestnews <comment><USER2> What a ridiculous title. Musk didn't dare anyone. He only asks that if anyone is arrested it will be him.

<comment><AUTHOR> I disagree. I am biased since I wrote it, but allow me to defend it:

1\. County says "don't open" 2\. Musk says "will open anyway; arrest me"

dare definition - defy or challenge (someone) to do something

I believe the qualification stands. He dares local government to actually arrest him. Either way it will be big news and neither side can afford to back down at this point.

<comment><USER2> County indeed said "don't open". Musk said "will open anyway". But he didn't say "arrest me". He said if you want to arrest someone because of this. Arrest only me and not the workers. That is not a challenge to the government. That is an appeal to their humanity. That the government should only hold Musk accountable and not the workers.

<comment><AUTHOR> Yes. But what are they supposed to do? They can't simply ignore it and expect rule of law to be treated seriously. Hence, it is a dare.

<comment><USER2> They can do many things. Laws are ignored everyday by officials.

They can also close down the factory without issuing an arrest. A dare is very specifically doing something to get a reaction out of somebody. Musk most certainly isn't hoping that the state will arrest him or anybody working at his factory. Quite the opposite.

<comment><AUTHOR> You do have a point in general, but in this specific case, Musk publicly called them out. Not doing anything in such situation is about as risky as doing something dramatic. I do not envy any decision maker here.

</comment></comment></comment></comment></comment></comment><comment><USER3> I do hope he does consider Arizona as well as Nevada and Texas for new HQ options.

I think the tantrum is a bit unbalanced... same for Alameda county for that matter. For the most part there's been way to much opportunism from politicians over this issue. It wasn't meant to, nor could it stop the spread of COVID, it was always meant to slow progression over time so that it doesn't overwhelm emergency resources, of which, the US always had more than comparable western nations.

Beyond this, the risk for food shortages internationally may kill more than COVID ever would have if we don't start opening things back up again... printing checks against the future won't do much if there's no food being distributed in channels that are now closed. Unless you like a lot of fiber.</comment>


No evidence of publication bias in climate change science - <AUTHOR> https://link.springer.com/article/10.1007/s10584-016-1880-1 <comment><AUTHOR> Write-up by the authors at [1]

[1] [http://sciencenordic.com/no-publication-bias-global- climate-...](http://sciencenordic.com/no-publication-bias-global-climate- change-research)</comment>


Why Code Breaks - <AUTHOR> https://www.git-tower.com/blog/reasons-why-code-breaks <comment><USER2> Completely agree with this. I'm working on a startup with a somewhat legacy codebase where we're not following any of these practices, and it's completely a mess.

For example, the only way to know something happened is when the public website goes down, and when that happens I have to stop what I'm doing to SSH into the server, read journald logs to try to find out what happened, and tell that to the devs. Only recently I decided to just pipe them to a public log file and let other devs do their thing with that just so I can focus on my tasks.

Higher ups just won't let us invest time on setting up proper logging because "there are other priorities".

There's also a lot of dead code, files where some lines have 20+ trailing spaces, JWT sessions with our custom invalidation tracking, a hand-made XML parser (wtf)... It's honestly a complete mess, and I don't expect this startup to survive even a month when it's launched.

So much could have been better had they followed these practices from the beginning, instead of ignoring __all 8 of them __.

</comment><comment><USER3> Water is wet.

In my experience tests are the primary mitigation for this issue.

<comment><USER4> I'm having an uphill battle trying to institute some of these practices on my current team. My takeaway is that it may not be as obvious as we think.</comment></comment>


Show HN: Universal mobile client for devs. Join Private Beta - <AUTHOR> https://github.com/archyhq/archy <comment><AUTHOR> Short demo what kind of client you can create with Archy.ai: [https://video.twimg.com/dm_gif/817220933689585664/JSz2qxRPlt...](https://video.twimg.com/dm_gif/817220933689585664/JSz2qxRPltY4drcpMsggTPZVSlcrbYkAyyDH_h7O-pRb3E33l1.mp4)

</comment><comment><AUTHOR> I feel that closed beta is not in favor at HN. But I'm trying to get some early users to test product and API before can open it to public. I believe in HN community )</comment>


10th largest site MySpace announces microformats support - <AUTHOR> http://marshallk.com/10th-largest-site-online-to-launch-microformat-integration-network

<comment><USER2> Is MySpace considered so unimportant by the web community that its name has to be prefixed with "10th largest site" in order to try to get attention?

<comment><USER3> It does when posted on the 5,220th largest site.

<comment><AUTHOR> Only 5,220? that seems rather high. I thought this site was much more obscure than that.</comment></comment></comment>


WordPress.com launches redesign - <AUTHOR> http://www.noeltock.com/web-design/wordpress/making-sense-of-the-wordpress-com-redesign/

<comment><USER2> While I agree with a lot of your synopsis — a general de-personalizing — it's hard to deny that the new homepage is stylish and sexy. Which, in my opinion, is what WP needs to portray most these days.

</comment><comment><USER3> I was hoping for a responsive design from WP. Sadly, no such luck.

<comment><USER4> Looks like they have a completely different mobile version with a huge call to action which makes sense for the service.</comment></comment>


Evolution of Sexual Intimidation: Male Baboons Beat Up Females, Increases Mating - <AUTHOR> http://www.newsweek.com/male-baboons-sexual-violence-control-evolution-human-behavior-632890 <comment><USER2> I look forward to the day a majority of people realize that our biology is, to a large extent, something to be overcome. Our "natural" biological impulses are often incredibly destructive, for both ourselves (e.g. sugar cravings) and for those around us (e.g. the impulses in this article).

<comment><USER3> _Our "natural" biological impulses are often incredibly destructive_

Tribalism is easily the most destructive of our impulses, especially in the way in which it can interact with ideology. Through ideology, such tribal impulses can camouflage their own effects and short circuit the "better angels of our nature" to excuse the dehumanization end even the slaughter of the out group.

I think that's the last great barrier to humans finally "waking up from history."

In particular, the way tribalism and in-group/out-group psychology interposes itself into political discussions is especially pernicious -- especially in the ways it manifests over the internet in social media. Fear isn't the mind killer. It's ideology combined with tribalism! If humankind ever destroys itself, in all likelihood, some group will have ideologically convinced itself that it should do so for some incredibly noble reasons.

<comment><USER4> Indeed. Nationalism, playing the man instead of the ball, blindness to hypocrisy because "the other side does it", whataboutery, and all round aversion to principles and reasoning. It long predates social media, but social media makes it worse, and the dilution of a basic consensus source of truth into fragmented tribe-truths - I don't know where it leads but it's not a good place. Soon, whatever deluded tribe gets their hands on the rudder will drive the ship of state until it hits the rocks of reality.

<comment><USER3> _Soon, whatever deluded tribe gets their hands on the rudder will drive the ship of state until it hits the rocks of reality._

There's a word for this happening throughout human history. It's called "government."

</comment></comment></comment></comment><comment><USER5> The article ends with: "It doesn't mean it has an evolutionary basis."

If not evolution, then what? Doesn't all behavior have an evolutionary basis?

<comment><USER6> No. There is a technical term for this - can someone help me out? I can't remember it....

<comment><USER7> Yes. They're called non-adaptive traits or spandrels: [https://en.wikipedia.org/wiki/Spandrel_(biology)](https://en.wikipedia.org/wiki/Spandrel_\(biology\))

</comment></comment></comment><comment><USER8> Don't really see anything new here. Grew up on a farm. Cats beat the crap out of each other in mating season. Horses as well. The stud will typically bite the crap out of her or run her through a fence repeatedly

</comment><comment><USER9> Makes sense. Evolution favours the strong and works to ensure survival, not happiness.</comment>


Dirty money piling up in L.A. as coronavirus cripples global money laundering - <AUTHOR> https://www.latimes.com/california/story/2020-04-29/coronavirus-slows-money-laundering-to-a-crawl <comment><USER2> This is the best article I've seen in the LA Times in 20 years.

</comment><comment><USER3> Plot for next Ozark season?

<comment><USER4> My first thought exactly

</comment></comment><comment><USER5> Anyone have a paywall-free link?</comment>


Netflix loses $4.58 million, as DVDs decline and streaming grows - <AUTHOR> http://thenextweb.com/insider/2012/04/23/netflix-reports-operating-loss-of-0-08-per-share-beating-expectations/

<comment><USER2> I am glad to see the loss of Starz had not real impact. Maybe this will help to send the message back to those licensing content to Netflix that they don't have as much power as they thought they did - which will mean reasonable license fees to Netflix and more (good) content for the subscribers.</comment>


In Praise of the Office Kegerator - <AUTHOR> http://pando.com/2013/12/21/in-praise-of-the-office-kegerator-the-future-of-better-jobs/

<comment><USER2> Marx would say -- this is taking us back to the old pre-capitalist days. Capitalism reduced the employer/worker relation to money alone. Now, in the leading edge, it's not just a lotta money, it's ski days, free lunch, and beeeeer!

Is that a step forward or a step backwards?

</comment><comment><USER3> I saw this article on Pando when it came out yesterday, and you've nicely explained something that I've been wondering about. There has to be some explanation to the beer trend other than frat-boy insanity (and yes, I like a good brew too). I think you've hit it.

</comment><comment><USER4> The SV kegerator the equivalent of the old mahogany desk -- a new status symbol. But at least it's for everyone in the company, not just the boss with the corner office.

</comment><comment><USER5> I wish they'd just hand over more money, or at least let us choose -- get the coffee, the massages, etc., which the company buys at a discount, or else get the money.</comment>


Legislator Drafting Bill to Outlaw Non-Compete Agreements in Massachusetts - <AUTHOR> http://www.xconomy.com/boston/2008/12/16/legislator-drafting-bill-to-outlaw-non-compete-agreements-in-massachusetts/

<comment><USER2> Massachusetts is already hostile to noncompete agreements; they have a "blue pencil" standard that allows judges to rewrite noncompetes instead of requiring the original agreement to be enforced verbatim, and noncompetes in MA have to be specific to the work the employee was doing.

Like I said last time when this subject came up, consider whether this development is a great thing; there's unlikely to be a federal ban on noncompetes, which means other jurisdictions may be nominally more attractive for sales and knowledge businesses because they're legally favorable to business owners.

<comment><USER3> Noncompete agreements shift some power from employees to owners of capital. A jurisdiction that forbids noncompete agreements will tend to be less attractive to owners of capital and more attractive to employees. But a _business_ generally requires both employees and owners of capital. Whether jurisdictions that forbid noncompete agreements are more attractive to a certain kind of business therefore depends in the short term on whether employees or owners of capital have more power in that kind of business, and in the long term on whether employees or owners of capital are more important to the success of that kind of business.

As pg has repeatedly asserted at great length, the capital requirements for software businesses are at an all-time low, so we would expect noncompetes to repel such businesses.

<comment><USER2> To be clear, I'm anti-noncompete.

However, I'd ask you to consider that in Graham's world, this distinction between "employees" to "owners" gets fuzzy; _you're_ supposed to be the owner, not an "employee". You can't do everything yourself. You might want to hire someone to help QA your application. You might not want that person to take their test plan, bug list, and design documents to your next nearest competitor, or to use your internal wiki as a blueprint to start their own competing company.

<comment><USER4> > You might want to hire someone to help QA your application. You might not > want that person to take their test plan, bug list, and design documents to > your next nearest competitor, or to use your internal wiki as a blueprint to > start their own competing company.

Confidential information agreements, which are legal in CA, seem to handle that case.

<comment><USER3> Inevitably, though, your competitor will learn a lot when they hire someone experienced in the field.

<comment><USER4> Yes, but that's because they hired someone experienced, not because they hired someone who knew your plans and designs.

In my experience, CA employers are very careful about IP from former employers.

The "IP theft" cases that I'm (somewhat) aware of weren't "took stuff to new employer" but "started own biz using prev employer's stuff".

<comment><USER2> The real textbook case isn't the knowledge workers themselves, but salespeople joining competitors and taking their rolodex.

The same thing applies to consultants, who can parlay relationships their previous employers built up for them.

<comment><USER5> If rolodex is built on company's time and dime then it's company's IP. No need for non-compete in that case.

Reltionships are fair game. I don't think any corproation should be able to dictate whom I can and can not talk to. This is where their business interests end and my personal liberties begin.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> Legislation that forbids two consenting adults from engaging in mutually beneficial interactions is always bad.

One does not have to be in favor of non-competes to dislike legislators removing or curtailing a civil right like the freedom of contract.

<comment><USER7> An agreement between a big company and an individual really isn't a case of "consenting adults". The company has too much power. In the case of top officers, then I agree, it should be negotable. Put another way the choices are

1) negotiated noncompete 2) noncompete is a given 3) noncompete forbidden

We now have (2). In theory (1) would be better. (3) is bad for the same reasons that (2) is. But more people benefit from (3), so this legislation could be an improvement.

Re. (1), if a company had to pay me to not compete in the future, I wonder what it would really be worth to them? I bet it's worth more to the individual.

Many years ago I saw the managers at a former employer prevent an ex-employee from taking a new job based on one of these agreements. They were not really concerned about the competition. They wanted to teach him, and the rest of us, a lesson.

<comment><USER2> So then, noncompetes should be allowed for 4-person startups, but disallowed for Walgreens.

<comment><USER8> In general I think all regulations should only apply to large companies. It sounds absurd at first glance, but starts to make sense once you think it over.

<comment><USER9> Then large companies will just set up small straw-companies to do non- regulated deals with you.</comment></comment></comment></comment></comment>


The New and Improved C# 6.0 - <AUTHOR> http://msdn.microsoft.com/en-us/magazine/dn802602.aspx <comment><USER2> Oh dear, the timing of this article is very unfortunate since two of the features it describes (Primary Constructors and Declaration Expressions ) were announced to be cut from the C# 6.0 complier just yesterday.</comment>


Chronic fatigue syndrome is in your gut, not your head - <AUTHOR> https://www.sciencedaily.com/releases/2016/06/160627160939.htm <comment><USER2> A misleading headline for a diagnostic test with moderate sensitivity and unstated specificity.

I desperately want to believe that chronic fatigue syndrome is as simple as a microbiological imbalance and I hope that we can see some stronger proof but this is an illness that has foxed us for a long time.

From UpToDate, my current preferred summary resource ([https://www.uptodate.com/contents/clinical-features-and- diag...](https://www.uptodate.com/contents/clinical-features-and-diagnosis-of- systemic-exertion-intolerance-disease-chronic-fatigue-syndrome) but extortionate subscription required, sorry):

"The proposed diagnosis was chronic brucellosis from the 1930s to the 1950s and hypoglycemia from the 1950s to the 1970s. Other hypotheses have included chronic Epstein-Barr virus (EBV) infection [11-13], purported chronic Lyme disease, total allergy syndrome, multiple chemical sensitivity syndrome [14], chronic candidiasis [15], and xenotropic murine leukemia virus-related virus (XMRV) and related retroviruses, such as murine leukemia virus (MLV) [16,17]. None of these etiologic agents have been scientifically linked to the SEID/CFS."

Extraordinary claims require extraordinary evidence.

<comment><USER3> Also, the article says: "The researchers have no evidence to distinguish whether the altered gut microbiome is a cause or a whether it is a consequence of disease".

If the altered gut microbiome is a consequence rather than a cause, the cause isn't necessarily even in the gut.

</comment></comment><comment><USER4> My mom suffered from CFS for years, so this got my attention. My current best explanation for my mom's illness is, unfortunately, that it was in her head, given her many other mental and emotional problems. But could it be the other way around---that the CFS had a biophysiological origin and was a cause (or at least exacerbater) of the mental problems? Somehow I'd like to believe that as it would make her illness comprehensible and more easily treatable. But maybe this current find won't pan out as an explanation.</comment>


When Google talks about "do no evil", who do you think they were talking about? - <AUTHOR> http://avc.blogs.com/a_vc/2008/05/why-cant-micros.html

<comment><USER2> It's pretty impressive of Fred to talk openly about Microsoft being evil. Very few VCs would do that, because they wouldn't want to alienate such a powerful partner and/or potential acquirer for the startups they fund.

Founders: this is the kind of VC you want on your side.

</comment><comment><USER3> "Microsoft messed with the technology industry for a decade..."

He's talking like it's past tense. OOXML was just a few weeks ago. They worked hard for that bad reputation. And honestly given the billions they've made, I'm not sure I'd choose different.

</comment><comment><USER4> I want to reiterate my advice to Xobni, Sell and run! These guys are beasts they will make a copy of Xobni and make you history, remember Netscape.

<comment><USER5> It's funny to see people making this argument. When was the last time that MS released anything truly new or made any "competitor" history? Netscape was a long time ago.

<comment><USER4> It's easy to say that when never played on MSFT turf; asked anyone of those companies building plug-ins or Add-ins for their platform. Ask your self why the Europeans are so hard on MSFT these days >1 billion in fines.

</comment></comment></comment><comment><USER6> Of course, it was "don't be evil", but that's just semantics. Mr. Buchheit (sp?) came up with that famous motto. He meant it literally, from what he says: don't be like the other guys. Don't be evil. I don't think that's the same as "be perfect", but at least try to have pretty good intentions -- that's far and away better than most big companies.</comment>


Ask HN: What are some non-obvious traits of a good coworker/team member? - <AUTHOR>

I'm starting my first technical internship today. In the past, the jobs I've worked have been menial and temporary, and I haven't put that much effort into creating a long lasting presence or becoming a true team member.
I really like this company and want to be well received despite my lesser technical ability.
What are some traits or individual actions that have made you respect and value somebody as a part of your team? What are some big things to avoid as a new member of a tight-knit company? <comment><USER2> Do what's asked of you. Do a bit more if you can. Help others. Ask questions if you're stuck, but only if you can't find the answer using Google. Join in when people are talking things through. Speak out when you think something is wrong (or right). Question assumptions. Respect the opinions of the rest of the team. Show gratitude when someone helps you. Be honest. Accept some of the shit jobs - everyone does them. Don't let people treat you like crap - you're there to learn stuff. Try not to take sides. Join in the non-work things. Have fun.

All this boils down to a single very important rule: _Don 't be a dick._

<comment><USER3> Very good advice. Be a good guy / good employee, not just a strong technical contributor.

Couple of qualifiers to: "Do a bit more if you can"

\- don't over-reach, doing stuff no-one's asked for could be a negative, and

\- don't be too pushy about "new ideas"; you might be right but as a new start you do not want to rub your colleagues up the wrong way until you've built a good track record and some respect within the org

Edit - plus buy the team cakes and chocolate sometimes :-)

</comment></comment><comment><USER4> Taking ownership of a problem they caused. Giving more credit away than they take.

</comment><comment><USER5> Put your hand up - always be the first to volunteer for a project or help out

</comment><comment><USER6> You can't go wrong with bringing fika.

</comment><comment><USER7> Be nice, polite, keen and humble.</comment>


How can one find the best quality item with the least amount of markup? - <AUTHOR> <comment><AUTHOR> OP here:

Certain goods have high status and markup but aren't high quality.

Can you think of a strategy of finding great quality items that don't have a luxury "tax" bumping up margins far than a typical good, due to the status they communicate to others.

In other words, how does one find high quality items without status included in price?

Let's say I'm buying a wallet and I would like to find a very high quality wallet (workmanship, materials, design). Let's also say I'm willing to spend a limitless amount for the wallet.

How would I find the sweetspot of a top quality wallet, but with the least amount of markup due to perceived luxury/social status?</comment>


Fancy Dorms Aren't the Main Reason Tuition Is Skyrocketing - <AUTHOR> http://fivethirtyeight.com/features/fancy-dorms-arent-the-main-reason-tuition-is-skyrocketing/? <comment><USER2> This has been studied in other places. It appears to have a lot more to do with the subsidy of student loans: [http://www.nber.org/papers/w21967](http://www.nber.org/papers/w21967)

I.e., more dollars chasing the same amount of education. A bit more explanation here: [https://fee.org/articles/student-loan-subsidies-cause- almost...](https://fee.org/articles/student-loan-subsidies-cause-almost-all- of-the-increase-in-tuition/)

<comment><USER3> I have a hard time believing this. I teach at a community college in Minnesota. Our state funding per student has been halved in the last 20 years. Salaries are not exorbitant. We went almost 5 years without a pay raise. We aren't frivolously spending money. We raise tuition because the bills have to be paid. The money isn't coming from the state. There's really only one way to raise the needed money and that is tuition increases.

I wonder how it can be determined that tuition wouldn't have increased as much as it has in the absence of eussidized loans. Would salaries be even lower than what they are? Would administration not have increased as much?

<comment><USER4> Have you considered that state funding has been lowered due to the fact that federal loans to students have increased?

 _We aren 't frivolously spending money. We raise tuition because the bills have to be paid._

The ballooning of the 'administration' staff at my wife's university over the last 20 or so years is the same type of staff explosion seen at larger public companies. Except that those companies are responsible to share holders, and thus constrict when profits cannot be maintained. Whereas the University can forgo layoffs by increasing tuition, and know that the federal (and private) loan system will fill the gap. I cannot remember there ever being even a 2% layoff of staff at the university, let alone something you'd see from MS/IBM/etc.

<comment><USER3> I think you have cause/effect backwards. Funding per student decreases because it's easy for politicians to say we increased funding by x% without mentioning that the number of credit hours has increased by (x+y)%. The politicians can then claim that universities are being wasteful and bloated because despite increases in funding tuition is going up.

<comment><USER5> it's not just one thing. here's like 5 different factors that all result in tuition increases

1\. state budgets for universities are cut for political reasons not having much of anything to do with the university itself. budget shortfalls must be made up from other sources, primarily from tuition increases.

2\. increased student enrollment due to increasing economic and social pressure for people to get a college degree. more students requires more facilities and more staff, which requires more funding.

3\. natural aging of physical plant of university facilities, combined with other long-term maintenance costs (pensions, for example) that have historically been underfunded and now need more funding.

4\. a price spiral relationship between availability of student lending and tuition costs. tuition costs increase so student lending amounts increase so tuition costs increase etc.

5\. real actual greed. this isn't universally true but it can't be dismissed. some universities will just raise tuition simply because they can.

</comment></comment></comment></comment></comment><comment><USER6> A pet peeve of mine is when an article displays data to bolster an argument but then cites the source to something extraordinarily large and generic (e.g. "SOURCE: DEPARTMENT OF EDUCATION") which makes actually finding the primary data nearly impossible.

</comment><comment><USER7> Missing from the title: ...at public schools. Which, if you've been following the amount of funding cut from state schools at all, hopefully isn't too surprising.

<comment><USER8> Agreed, after the Great Recession state funding to universities (and K-12, colleges) dropped dramatically. Arizona leads the way with almost a -50% drop in funding, of course tuition goes up by almost 90% [1]. The article has this data from 2000-2014 but most of the damage was 2008-present.

[1] [http://money.cnn.com/2015/05/13/pf/college/public- university...](http://money.cnn.com/2015/05/13/pf/college/public-university- tuition-increase/)

</comment></comment><comment><USER9> Tuition is rising at San Jose State to: get us "great partnership deals" on new equipment like expensive teleconference TVs or smartboards that are rarely used.

I have had CS professors who struggle to get their computers on the projector because of old worn-out bulbs while math professors get rooms with smartboards they don't even use, and similar.

Tuition rises because of bogus costs and needs that aren't needs at all or because of overspending on poorly used/misused equipment that is sorely mismanaged.

I could ramble on and on about problems I've seen first-hand, but rising tuition is a serious issue not only because of costs being higher but also because schools are just plain casually misusing/abusing the money in the first place, some even carelessly.

</comment><comment><USER10> For any endeavor, including education, if we shift the source of funding from democratic government to private individuals, then those needing funding will shift their focus from pleasing the public to pleasing wealthy parties.

For example, there was a new park being designed in NY. Lacking traditional public funding, they sought private funding. As a result, instead of a design that serves the public, the park is designed according to the desires of a wealthy donor.[0] In higher education, instead of making progress and giving more and more people a better education, as prior generations did, we are going the wrong way (AFAIK). I believe it's fundamentally undemocratic, which is an American ideal much more than small government, which is a partisan ideology.

To me, the adherence to ideology (in this case, smaller government) over practical good results (educating people and opportunity), always is a terrible idea and can lead to dangerous results.

The free market allocates goods to those willing and able to pay most for them. That's efficient for many things such as soda and cars, but undesirable for basic human needs and societal goods such as education and healthcare, which we don't want to deny to people because they are poor. Funding by the public (via taxes) is a good solution for some things.

[0] That was the story as I read it maybe 1-2 years ago; it's possible the situation changed since then.

EDIT: added sentence, reworded another

<comment><USER11> I don't see how your comment is relevant to the topic. This isn't a public vs private debate. If anything it's a debate between federal loans and state appropriations, or an issue of administrative mismanagement.

There is another kind of ideology, however, which may be at play here: an automatic reaction to pour public money on problems. Public institutions can be irrational actors that cator to narrow interests just as private ones can. The public sector needs to be regulated as well, and it largely is not.

<comment><USER10> > This isn't a public vs private debate

We might be referring to different forms of "private". The article says that as public funding was pulled back, tuition was increased to fill the gap. Tuition is private funding, though by many (mostly) wealthy people rather than a few uber-wealthy. With funding coming from that private source, wealthy customers, educational institutions need to focus on pleasing them and invest in fancy amenities rather than educational resources.

> The public sector needs to be regulated as well, and it largely is not.

It is regulated by voters and their representatives - very imperfectly, and we should improve it and not be satisfied, but I've yet to find a large human endeavor for which that wasn't the case.

> There is another kind of ideology, however, which may be at play here: an > automatic reaction to pour public money on problems

Given the radical cuts in government budgets on the federal and local level, I don't see that approach having much influence currently so I'm not worried about it. Also, spending public money is a strategy, not an ideology. Democratic socialism is an ideology, and I do dread dogmatic ideologues of all stripes, but that one also isn't a threat right now.

</comment></comment></comment><comment><USER12> You can also see this in aspects of the university that are supported by research funds instead of tuition. States slashed their support under the assumption that the Federal government would pick up the tab with faculty writing more grants.

</comment><comment><USER13> That site managed a breakthough in bad behavior with tracking blocked. The page turns black every five seconds, then reloads.

</comment><comment><USER14> In 2004 or so, right after I started college, the state of Texas repealed their laws that capped tuition at public universities at a very low amount. After that, tuition at my university, UT Dallas, utterly _skyrocketed_. It rose far more than the other public universities: we very quickly became one of the most expensive public universities in the state, possibly _the_ most expensive, and on par with private universities.

I was insulated from that because I had a scholarship, but if I didn't, I might have had to drop out.

</comment><comment><USER15> So, if the article is to be believed, the tuition costs were already high, but hidden by financing them through taxes, and now they are coming out for all to see by explicitly stating them as tuition costs and putting them on those who consumes the service instead of distributing them between all taxpayers.

If so, then despite the obvious negative impact on students, I think by itself such restructuring is a good thing - if the costs are hidden, they'd never be discussed and evaluated and subjected to both public and market scrutiny. If nobody knows how much tuition really costs, it's virtually impossible to have meaningful discussion about it. If the public thinks tuition should be subsidized, these subsidies should be out there in the open, so the public knows how much it costs per student, and experiences it openly, not has to dig it up from the piles of budget allocations.

One question, though, is if it was state financing drop that causes tuition to raise in public schools, wouldn't that effect be much less in private schools? So far I don't think the data confirms this.

</comment><comment><USER16> It's worth asking what drove the states to cut aid. Some states have certainly had lower tax revenues than expected. But some states (probably) felt safe moving the funding problem onto loans and/or sources such as lottery proceeds.

<comment><USER17> Illinois it's been a pension time bomb building for 30 years. "Luxuries" like higher education are way down the line from legally mandated payments to retires teachers and policemen

<comment><USER18> Is Illinois one of the states that's made a habit of raiding their pension programs over the decades, promising to pay back in later[1], and now later has come?

[1] (although they never mention the interest that would have accrued if the principal hadn't been stolen...)

<comment> rm_-rf_slash It's more like Illinois pensions are second to the Ten Commandments in terms of universal truths that must not be questioned under any circumstances.

It's one thing to make a promise and intend to keep it. It's another thing to stick to the promise until it's beyond the realm of possibility to fix it, and a lot of people end up getting hurt.

</comment></comment></comment></comment><comment> rm_-rf_slash As someone whose family is more steeped in higher education than most people, I have come to believe that the biggest problem with higher education is the degree with which we have allowed it to become the universal gatekeeper of professional careers.

I think college is a great experience, and I do hope for a day where everyone can experience higher education in their life, but the basic mechanisms of college are NOT necessary to work in a professional field. It is only our culture that prevents it from happening.

The nucleus of this problem is our obsession with pedigree. We want everything to be awesome and perfect and follow a neat little narrative where we all end up rich and famous and successful because we attended the right schools and worked with other people who attended the right schools. But life is never that clear cut, and this narrative ignores the reality that most of your ability to do a job is learned on that job. I didn't know a lick of ColdFusion or even more than a few lines of JavaScript when I graduated college, and within just a few months I could handle anything my employer threw at me.

A good amount of that success was because of my education, but most of it was because they were willing to take on the risk of someone that had to learn and grow, instead of insisting on a ColdFusion expert that could have done everything from day one.

College is great, but it's not for everyone. My GI Bill-aided grandfather didn't need the college he dropped out of to teach him how to run a successful restaurant and support a family. And most of us understand that a lot of what we get tested on from the age of 5 is bullshit.

The way forward is not fixing education, it's fixing employment. We need employers willing to take risks with people who aren't perfect by default. To do this we need to encourage and maybe even subsidize paid apprenticeships. Real, paid work experience is better than any welfare or job retraining or $80k degree. It shows what you are worth and what you are capable of doing.

As Ronald Reagan said, on one of the few occasions that he actually managed to make a damn good point:

"The best social program is a job."

<comment><USER19> > As someone whose family is more steeped in higher education than most > people, I have come to believe that the biggest problem with higher > education is the degree with which we have allowed it to become the > universal gatekeeper of professional careers.

Blame Griggs vs. Duke Power Company[1], which is the reason employers are afraid to use proficiency tests in hiring (and hence are forced to rely on proxies like an applicant possessing a four-year degree).

[1] [https://en.wikipedia.org/wiki/Griggs_v._Duke_Power_Co](https://en.wikipedia.org/wiki/Griggs_v._Duke_Power_Co).

<comment> rm_-rf_slash Interesting, didn't know about this case. Not sure how a degree would be any less discriminatory given the makeup of college graduates, and if I can cite this case the next time I'm asked to whiteboard FizzBuzz, but either way, I still believe it would be better than a purely educational focus to instead subsidize apprenticeships so that employers see if someone is simply incompetent for lacking the right skills or if they have a lot of potential and are worth investing in.

</comment></comment></comment><comment><USER20> The analysis here is pretty insipid. If primary sources of funding get cut, and your reaction is to go out on a capital projects spree, what do you expect to happen?

I went to a big state school where most classrooms weren't painted since 1969. Now there's a 50" tv ok every flat surface and, the dorms are very fancy, and the mystery meat and hamburger vending machine has been replaced with better food and 24x7 cafeterias.

Guess what? This shit costs more.

</comment><comment><USER21> This article fails to mention the exploding student population size.

Over time the per-student funding from states has gone down significantly. But that doesn't mean the state slashed funding for higher education! For University of Washington the budget from the state is almost entirely flat. Up about 3%.

Student Size: +100% State Tax Revenue: +40% State Higher Education Funding: +15%

What would you consider the "dominant factor" here? It's several factors. But I would not consider, and I quote, "a steady decrease in support for higher education on the part of state legislatures" to be the dominant factor. In fact, that statement is flat out wrong. The state legislature hasn't decreased support at all. They're providing more support than they were! The state, and the state's coffers, have no way to keep up with the growing number of students.

[http://f2.washington.edu/fm/financial-report- archive](http://f2.washington.edu/fm/financial-report-archive)

(I did a little digging last night specific to the University of Washington. Numbers from memory and a little fuzzy. But within the ballpark.)

<comment><USER22> It depends on how you look at it. There _has_ been a decrease in funding _per student_.

<comment><USER15> But it is presented as a state government action to cut funds, when the actual events were that action was to increase funds but not enough to catch up with increasing student numbers. Given that student numbers are not under control of the state government, describing it as a state action to cut funds in misleading.

It is misleading in the same way as to say that the company substantially raised the average salary of employees when in fact what happened is that CEO salary was doubled. Technically yes, that means average salary has increased too. But such statement omits important details that make it misleading.

<comment><USER22> Oh, see what you're saying now. You're right. It _has_ been "slanted" as a funding "cut".

That said, I think you'd agree that the government has failed to fund the education of its citizens to the level it used to. I got my college education when it was "cheap" so if I was selfish [OK, _more_ selfish than I am] I would be inclined to go with the "I got mine, tough for you. I had to walk uphill both ways to school, blah, blah. Thanks for paying for my retirement."

</comment></comment></comment></comment><comment><USER23> Nate Silver totally fails to note the enormous financial transfer from professors (all new professors are cheap adjuncts) to administrators. I know adjunct professors that are on food stamps. Academia is a hellhole and college administrators should be ashamed.

<comment><USER24> Often the administrators are former or current faculty themselves. I've seen a professor hold a position as head of the university technology department(a non-academic support role) and teach classes. VPs, Deans, department heads, leaders of various "Centers of Excellence", leaders of various initiatives, Provosts, and even Presidents can be former faculty.

The former professors just view their role as administrator as another step up the academic ladder. There's even a website poking fun at university titles:

[http://universitytitlegenerator.com/](http://universitytitlegenerator.com/)

</comment></comment><comment><USER25> This article is largely on point, but I get the feeling they are playing games with Tuition v. Total cost. At my state university Tuition is separate from dorm costs, so fancy dorms effect loan amounts but not "tuition".

</comment><comment><USER26> What's the enrollment change? Is this the same amount of money spread over more students?

</comment><comment><USER27> One university decided to change signs on toilets. Single label costs $400, it has to be in brail and match various regulations.

I think education is ripe for disruption. We will see rise of bootcamps and online education. Also university in foreign country costs peanuts.

<comment><USER28> Your comment makes an important point though: who will take care of people who are visually impaired or who need other accommodations?

The online MOOCs aren't going to introduce braille.

I am kind of glad they spent that money to accommodate those students. Aren't you? That's not what is raising the cost of education.

EDIT: I did work on a mouse that provides haptic feedback including texture years ago, so I am aware that it is possible to provide braille like feedback over the web.

EDIT2: Many people have pointed out that MOOCs do add accessibility features. This is true, and good. I think the MOOCs are largely run by well meaning people.

<comment><USER29> > The online MOOCs aren't going to introduce braille.

In general, putting content online makes learning more accessible. As a specific example, Khan Academy has done important work developing tools in this area: [http://engineering.khanacademy.org/posts/tota11y.htm](http://engineering.khanacademy.org/posts/tota11y.htm).

I see where you're coming from - many "dispuptive" services have (shamefully, in my opinion) resisted making their services accessible, but I don't think MOOCs really fall into that class.

<comment><USER30> Disclosure: I work for Uber, so I'm more than I little biased.

Tell me; How many blind or deaf people did you know who took a cab before Uber? I went to RIT, which hosts a school for the deaf, and while there wasn't really a call for cabs at the school, I knew a couple people who did co-ops in big cities and were frustrated by how hard it was to get around. When asked why they didn't take a cab, they explained how much of a pain it was to communicate with their drivers, even with the fairly advanced assistive devices (every student had a sidekick, at the time a very advanced phone, and could usually pass around notes using such). It was still too hard; the barrier of entry on the communication was too high, and taxi drivers were easily frustrated trying to communicate fares and routes to deaf students. I can't even imagine a blind person hailing a cab. How would they know where to hail? How would they know when it stopped in front of them?

Now, I see posts on the RIT subreddit from time to time from deaf students who've just taken an Uber. It's not a problem anymore. There's a distinct group of blind users of the uber app; It works, the interface is consistent, and I just tried it myself (it wasn't easy, but I called a cab, and I think it'd work decently with practice).

<comment><USER29> That's fair. I mentioned Uber because I remembered the criticism from the National Federation of the Blind [1]. However, it's tangential to my point, so I've edited my post to remove it.

[1] [https://nfb.org/groundbreaking-settlement-end- discrimination...](https://nfb.org/groundbreaking-settlement-end- discrimination-against-blind-uber-riders-who-use-guide-dogs)

</comment></comment></comment></comment></comment><comment><USER31> What fancy dorms? You'd pay like $200/month or less for such a place in most cities...

<comment><USER32> When I was applying to college in 2006, one of the big selling points at the University of Alabama was that the honors dorms had a lazy river.

I wonder if the "fancy" dorms are only a thing in non-city universities. It would make sense that building fancy, new dorms in a downtown area in a place like Boston or New York just wouldn't be worth it.

<comment><USER33> Universities get a property tax exemption that allows them to do things with land in downtown areas that wouldn't be economically rational for typical real estate investors.</comment></comment></comment>


South Park has reportedly been banned in China after criticizing censorship - <AUTHOR> https://www.businessinsider.com/south-park-removed-from-china-internet-report-says-2019-10 <comment><AUTHOR> Response from Trey and Matt: [https://twitter.com/SouthPark/status/1181273539799736320](https://twitter.com/SouthPark/status/1181273539799736320)

<comment><USER2> National treasures those two.

</comment></comment><comment><USER2> I'm surprised it wasn't already.

No fun in China allowed unless approved by local party officials first!</comment>


Silicon Valley's Latest Unicorn Is Run by a 22-Year-Old - <AUTHOR> https://www.bloomberg.com/news/articles/2019-08-05/scale-ai-is-silicon-valley-s-latest-unicorn <comment><USER2> Comments moved to [https://news.ycombinator.com/item?id=20614672](https://news.ycombinator.com/item?id=20614672).</comment>


How Discord Indexes Billions of Messages Using Elasticsearch - <AUTHOR> https://blog.discordapp.com/how-discord-indexes-billions-of-messages-e3d5e9be866f <comment><USER2> Pretty interesting stuff. I'm amazed that there are only 14 nodes in the cluster with 1TB SSD each; message size must be fairly small on disk (less than 295 bytes, unreplicated). I'd also recommend looking at maybe tuning the shard_request_cache; there are some possible improvements to be made there, if you're running that many indices. Finally, are all of the indices of approximately the same size? Are they time-boxed?

edit: Put together a logging cluster consisting of 14 nodes, 12 data + 2 indexer/search API, with ~40TB of consumer-grade SSDs. Ran ~14k indices based on log type & timestamp, with a whole raft of custom field configuration to handle aggregations and different tokenizations. In sum, Elasticsearch looks easy to configure and tune but is amazingly hard to do well - but incredibly rewarding.

<comment><USER3> > but is amazingly hard to do well - but incredibly rewarding.

Isn't this true of everything. The greater the reward the more effort required in most scenarios? I do agree, Elasticsearch can take some trial and error along with experimentation to get right!

<comment><USER2> Yeah, that's nearly a tautology. I guess in this sense what I mean is that, there's a much deeper Trough of Disillusionment when it comes to learning how to properly configure & tune ES to your needs, as opposed to a lot of other tools I've used over the years. It's very easy to set up, but also very hard to get right, which I'd imagine causes some people to settle or give up.

</comment></comment></comment><comment><USER4> Well-written, props. Some architecture questions you might want to think about:

\- Why only one elasticsearch shard (+replica) per index? You run some major risks of write-time hotspotting here, especially if you ever have to completely rebuild your indices from scratch (pro tip: plan for it now). If you have any basic metric of a discord server's size prior to index creation, you can pre-optimize the number of shards in its index to ensure that writes are distributed across the cluster.

\- What happens when one server's index grows too large to fit on one (elasticsearch) shard? Practical shard size limit in ES is ~50GB due to concerns related to reallocation and recovery. You could also theoretically hit the 32GB memory limit.

Also, as another poster mentioned, consider dedicated master nodes. You're going to lose data nodes from time to time, don't let it cost you consistency by bringing down a master with it.

</comment><comment><USER5> We've been running large ES clusters here at Revinate for about four years now. I've found the sweet spot to be about 14-16 data nodes, plus three master-only nodes.

Right now, we're running them under OpenStack on top of our own bare metal with SAS disks. It works well but I have been working on a plan to migrate them to live under Kubernetes like the rest of our infrastructure. I think the answer is to put them in StatefulSets with local hostPath volumes on SSD.

We use CoreOS for our Kube cluster and are huge believers in it so I will definitely be leveraging that for ES. One thing I have done for my Cassandra clusters is to write a custom health check running in a container that monitors Cassandra's health and locks the cluster with locksmithctl to prevent reboots when the database is unhealthy. This will be easy to translate to ES. Correspondingly, I will add a curl call to the ES unit file to disable shard reallocation when the service is shut down during a reboot. This prevents costly reallocation of shards when you know that a node will be back online shortly.

</comment><comment><USER6> OT but: so many engineering blogs lack a link to the company's main site. It's a small thing but it bugs the hell out of me. I was just reading the (also currently front page) article on Segment's blog and actively noticed and appreciated the presence of such a link.

Is it a Medium thing? Does it disallow linking out from a Medium hosted, privately branded blog?

<comment><USER7> This has been true for several years, even before Medium was popular. Most blogging platforms behave this way, even self-hosted ones.

</comment></comment><comment><USER8> At some point (roughly around your size, maybe a little larger) you'll want to add dedicated master nodes with no data.

3 small boxes, master eligible, holding no data.

I love the writeup, having a way to reproduce data if your clusters fail is a definite must with ES.

<comment><AUTHOR> Yup. We're thinking about that too. We've been monitoring node performance closely to see when we'll need to shift the topology around a bit. I'd want to keep our clusters small and the topology uniform before I investigate dedicated master nodes though.

<comment><USER9> FYI: From my experience (on older ES versions) when you hit problems with it, they will be pretty bad, catastrophic problems. We run dedicated master nodes, but have the master servers running multiple ES instances, so they can serve as masters for multiple clusters.

</comment></comment></comment><comment><USER10> I really enjoy Discord's blog. Their Cassandra write up was excellent as well. A couple of thoughts and questions:

\- Having many clusters and assigning messages to a specific cluster seems like an interesting solution.

\- I'm curious how they managed to lazily index messages.

\- Since only message, channel and server ids are stored in ES, have there been any problems reindexing data after an index fails?

<comment><AUTHOR> The first time you run a search in a server (or the first time you run a search in a server after the index fails) - will trigger a full re-index of that server. Ctrl-F "Historical Index" in the blog post for more details! If you've never used search in a server - the messages are not indexed in real time until you do for the first time. Both these things make the system "lazy".

The worst case to an index failure is that the search query is delayed as the index rebuilds itself. We throttle the rate of historical indexing into ES to a safe level so that we're not degrading performance of other components of the system.

<comment><USER10> Oh, I think I get it now-- is it that the _initial_ indexing is lazy, but all indexing after that is done automatically by the historical index workers? Basically when a user searches for something do you check that ES has something for that user, if it doesn't start off the initial indexing process, and from there the workers do their thing?

<comment><AUTHOR> The historical index workers index the history of the server, whereas we have a real-time index worker that is consuming and bulk inserting messages in real time. Searching for the first time in a server turns on real-time indexing and triggers a historical index of all previous messages in that server.

<comment><USER11> Can you talk about the bulk insert for real time messages ? How does this get triggered - does it run every X minutes or X messages (your code looks like it is every X messages).

Are you using DB triggers to fire the job ?

</comment></comment></comment></comment></comment><comment><USER12> > Elasticsearch supports automatic shard rebalancing, which would let us add > new nodes to the cluster, fulfilling the linearly scalable requirement out > of the box.

I'm assuming they mean "I can add a new node to the cluster, and NEW SHARDS FROM NEW INDEXES can be distributed amongst the new nodes".... So far as I know elasticsearch can't rebalance shard location or composition automatically based on cluster membership change events...right? I mean, the cluster reroute api will let you manually move a shard, but thats all I know of.

<comment><AUTHOR> ES can indeed rebalance shards to new nodes as they join the cluster. More here: [https://www.elastic.co/guide/en/elasticsearch/reference/curr...](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk- allocator.html)

<comment><USER12> huh, the more you know. not sure how I've missed this. I'll definitely look for some discussions/use cases of how this actually plays out.

<comment><USER13> I suspect you were thinking of changing the number of shards, which can be an important part of rebalancing an index. Unfortunately, Elasticsearch can't do that.

<comment><USER14> There is a convenient reindex API available in more recent versions, although in most cases where you can rebuild your index anyway, it's faster to do that.

<comment><USER13> Ahh. I was wondering when they were going to get around to implementing that. I might have to upgrade to 5.x soon.

<comment><USER15> FYI, the reindex API was added as experimental in 2.3, so it's been there for about a year [1]. Even as experimental it worked great.

[1] [https://www.elastic.co/guide/en/elasticsearch/reference/2.3/...](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/docs- reindex.html)

<comment><USER13> Ha. I just came here to edit my comment. I remember seeing it before and just checked the docs again. In 5.2 it's STILL considered experiential. Craziness. It took me about an hour to build The basic functionality myself.

<comment><USER16> Thank you for noticing this, it is not considered experimental anymore. I integrated a change[1] to update the docs; it should percolate through to the site within thirty minutes.

[1]: [https://github.com/elastic/elasticsearch/pull/23621](https://github.com/elastic/elasticsearch/pull/23621)

<comment><USER12> FYI, the re-index API is awesome.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> On the other side of the spectrum, I recently did a mini review of available F/OSS options for full-text indexers /search tools. Requirements: natively compiled, one-shot/single-process per search, no long-running containers so EL, SOLR, and anything else based on mighty Lucene and Java was out of scope.

The reviewed tools were: clucene, Apache Lucy, Indri/Lemur, Xapian, and Zettair. Of those, only Xapian and Zettair could be made to return useful search results within time constraints. Sadly, I didn't get Apache Lucy to build, which along with Xapian is the only option still being maintained. clucene could be built, but delivered erratic results (it found "impeachment" but not "indictment" in the US constitution text), and was too difficult to extend into a full stand-alone app (though it apparently is being used in Informix still).

<comment><USER18> Have you seen Bleve? [https://github.com/blevesearch/bleve](https://github.com/blevesearch/bleve)

I have only tried it for an hour or two, and can't speak for its search result quality, but it seems promising.

On my laptop, with an index of 10,000 documents each containing 200 random words, it takes about 110ms to perform a fuzzy search. Half of that is the actual search, the other half is the process/initialization overhead.

<comment><USER17> Thanks. Will try.

</comment></comment></comment><comment><USER19> Also notable (IMHO) about Discord is that they run Elixir/Erlang on the backend

</comment><comment><USER20> >4 primary metrics that we use to decide when the cluster needs to be grown

I would imagine you could measure some sort of throughput/response time values to tune this instead of relying on cpu usage because having 100% cpu usage doesn't mean app is poorly performing it could mean app is performing optimally with all the resources available to it. Similarly heap_free shouldn't be measured on its own, neither should be tuned on its own, ideally jvm should be left to its own devices to tune these values based on your inputs in form of params like MaxGCPauseMillis, GCTimeRatio ect.

</comment><comment><USER21> I'm interested in this bit:

> Search API: An API endpoint that clients can issue search queries to. It > needed to do all the permission checks to make sure that clients are only > searching messages they actually have access to.

We've had to implement this exact thing for our index, and were never quite sure if we were doing it right. To "prepend" a security clause/condition into an incoming search query felt awkward and hacky.

<comment><AUTHOR> How would it be any different from prepending a security clause/condition to a SQL statement for a traditional rdbms?

</comment></comment><comment><USER22> _> Self-healing: We don't have a dedicated devops team (yet), so search needed to be able to tolerate failures with minimal to no operator intervention._

Do people not hire small business support technicians anymore? I worked for a guy when I was a teen who took support contacts to manage small business networks. He would call me and send me to fix a router when something broke. He couldn't have charged more than $2K per month, basically just to be on- call. That's 24k a year for an on-call ops tech, that's fairly cheap I think.

(Also, "self-healing"? Is this different from hooking up a service request with timeout to an APC with remote power and rebooting the sucker?)

 _> Of course, this entire search infrastructure would be incomplete without a way to discover clusters and the hosts within them from the application layer._

((If you want something that isn't complicated to use or maintain, why would you rely on technologies which use complicated decentralized algorithms just to _find_ them??))

</comment><comment> charlie-r Their Elasticsearch type mapping shows they're using the English fulltext analyzer for message content, I'm curious if they're planning on supporting fulltext search for other languages in the future.

<comment><AUTHOR> Maybe one day. It'd be trivial to add per-locale templates. The hard part really is exposing that in our new guild flow (hey - what language is this server gonna be in) - and also handling if someone chooses to change their server language (this would have to trigger a re-index). We'd also have to replicate those language analyzers in our client to do proper highlighting.

</comment></comment><comment><USER23> Great writeup, thanks! For the search query, are you guys using Elasticsearch's query strings or have you wrote your own? If using elasticsearch's, have you had any issues so far?

<comment><AUTHOR> We're just passing elastic search JSON data using their query DSL: [https://www.elastic.co/guide/en/elasticsearch/reference/curr...](https://www.elastic.co/guide/en/elasticsearch/reference/current/query- dsl.html)

<comment><USER23> I'm sorry, I haven't used Discord. From reading the post, I got the impression that you allow queries like:

        from:jhgg "Fallout 4" -Witcher

Is that so? If so, you created a parser for this format yourselves and are converting that to ElasticSearch's Query DSL?

</comment></comment></comment><comment><USER24> Are the RBAC rules stored in Cassandra or in the application..?</comment>