Microsoft Clarity – a product to visualize user interactions - <AUTHOR> https://blogs.bing.com/webmaster/november-2018/Introducing-Clarity-a-web-analytics-product <comment><USER2> OSS analytics packages are unfortunately such a rarity - Piwik seems to be the only serious game in town. All I really want is an anonymized picture of feature uptake and some inkling as to how well we're doing in the UX and perf departments. I want to have these things without that annoying EU cookie widget, without shipping any data to third party services, and without loading any third party scripts.

Unfortunately this seems to just be the frontend tracker, and the backend will remain a closed MS service.

<comment><USER3> Snowplow[1] is an alternative to this. While they have a managed solution, their backend is open source and can be self-hosted. And they have a forked version of the Piwik client side library designed ot work with Snowplow[2].

Snowplow is a bit more generalized than just Piwik, and it shows in Piwik having a more robust featureset for website analytics specific. But Snowplow has a lot more usefulness if you're having to merge a bunch of data sources together to get a picture of what's happening.

[1] [https://snowplowanalytics.com/](https://snowplowanalytics.com/)

[2] [https://github.com/snowplow/snowplow-javascript- tracker](https://github.com/snowplow/snowplow-javascript-tracker)

</comment></comment><comment><USER4> _When Manali started using Clarity, she realized that a lot of users were abandoning the page before reaching the bottom, which has important information about the recipe. After replaying sessions of users who abandoned her blog, she noticed that users who were only interested in the recipe, which is at the bottom, scrolled through the long post and gave up midway and abandoned the page._

They should find a more compelling example. "Recipe at the bottom of long page with life story and huge images" is one of the canonical examples of user- hostile web design. EDIT: As in, no one should need a product to tell them this is clearly not what people want.

</comment><comment><USER5> Awesome that they are open sourcing this. I've always felt funny about paying a ton of money to FullStory for this. I wonder what this means for FullStory and HotJar?

<comment><USER6> I expect they already offer more and better differentiation to not be too terrified of this. I work for an analytics company and when our core features are copied by larger companies it's pretty exciting and is a driver of innovation for ourselves.

</comment></comment><comment><USER7> It's nice to know how your users use your apps and web-sites, but I think it's morally wrong to track user interactions in such detail. User testing is more effective _and_ they know they are being watched. It's probably a good idea to show a disclaimer that says everything the user does on the site will be recorded, or better, have the user opt-in to being recorded.

<comment><USER6> I work for an analytics company - that said, their malware example is one unlikely to come up in most user testing. I say most because I've seen some do user testing over Skype with the user's real environment.

Another piece of anecdata, I had a customer experiencing a bug I am confident I wouldn't have uncovered even with intense technical communication without having session replay... our UI was breaking a POST request for a user and there was seemingly no way for it to do so, it turns out in a number field, Firefox had these up and down arrows to move the number up, but the attribute that received the value when doing this was different from expected when entering the number by keyboard.

Anyways, the scary side of session replay does arguably outweigh the upsides. If you don't know or care, and even if you do care it's hard to make sure you never capture things like credit card numbers, etc...

At my current company I've helped push a lot of things that make it darn near impossible for us to capture PII, but nothing is perfect.

As someone else in this thread said, it would be nice if someone built an end to end OSS analytics tool that was anonymizes by default, but I think even the concept of anonymization is hard to tackle depending on how one defines anonymous. The set of features one uses, and the frequency and/or order of use can likely be identifying by some definition.

Opt in, for the example I described (bug hunting) would be a totally reasonable use, but I bet you'd have a hard time justifying the cost of such a tool if you use it for such sparse cases, and you lose out on many other serendipitous discoveries. Again, if you think it's morally wrong, there's not much to argue with in terms of using such a feature :)

</comment></comment><comment><USER8> Who in the world signed off on that promo video? The music was _way_ too loud in comparison to the voice-over. Irritating.

<comment><USER9> It's not louder, at least doesn't sound it to me, but yeah the background music should be lowered just a bit more. I'm surprised the guy isn't yelling as he talks, felt like I was watching something out of Rick & Morty, one of those awkward ads.

</comment></comment><comment><USER10> Creepy. Notice how it doesn't just record where a visitor had scrolled to or has clicked, but it apparently sends either DOM content or actual screenshots to the server. Otherwise they would not have found out that the malware put extra stuff in the user's Bing results.

</comment><comment><USER11> Hopefully they use it on their blog, so they can discover 22% of the screen real estate is smothered with a sticky header.

</comment><comment><USER12> Going to [https://clarity.ms](https://clarity.ms) gives 502 Bad Gateway

<comment><USER13> You need to put "www" at the beginning. On the video they showed the full URL: www.clarity.ms

</comment></comment><comment><USER14> Is this free to use?

</comment><comment><USER15> How does clarity differ from other tracking or session replay tools like hotjar or fullstory? It looks like it's another Microsoft-ish-ly bad version of other existing things.

<comment><USER16> It's free to use, whereas tools like FullStory cost a fortune, even when compared to heatmap tools like HotJar.</comment></comment>


Reply of the Zaporozhian Cossacks - <AUTHOR> https://en.wikipedia.org/wiki/Reply_of_the_Zaporozhian_Cossacks <comment><USER2> "Turkish devil", indeed; note the irony, the title of the person who wrote the purported letter is "Kosh otaman", a title with a Turkic word:-)

There's confusion between "Kazakh" and "Cossack", much was at stake on definitions like this, especially in earlier times when Turanic ideas were en vogue. Tolstoy's novel is about the people mentioned here, who are Christian.

<comment><USER3> Turkic origin is not the only possible root - there is also a Scandinavian theory that ties hetman and otaman together (their meaning is similar and at some point was used interchangeably) and also a Germanic root.

Source is in Russian (couldn't find an English one): [https://ru.m.wikipedia.org/wiki/Атаман](https://ru.m.wikipedia.org/wiki/Атаман)

<comment><USER4> > (couldn't find an English one)

Here are Wikipedia pages in Ukrainian[1] and English[2]

[1] [https://uk.wikipedia.org/wiki/%D0%9E%D1%82%D0%B0%D0%BC%D0%B0...](https://uk.wikipedia.org/wiki/%D0%9E%D1%82%D0%B0%D0%BC%D0%B0%D0%BD)

[2] [https://en.wikipedia.org/wiki/Ataman](https://en.wikipedia.org/wiki/Ataman)

<comment><USER3> Thanks! Didn't search hard enough.

</comment></comment></comment></comment><comment><USER5> I love this story and painting, but I wonder.. did the Cossacks hold their ground, or did the Sultan eventually get them?

<comment><USER6> One can say he did not, and they did. The most likely context of this is one of the countless wars between Ottomans and Russia for the control over Ukraine, south of Russia and Crimea. Most likely this one: [https://en.wikipedia.org/wiki/Russo- Turkish_War_(1676%E2%80%...](https://en.wikipedia.org/wiki/Russo- Turkish_War_\(1676%E2%80%931681\)) Note that there were different Cossack communities there, and some were recognizing Ottomans as rulers, and some did not. The initial letter was probably written to the latter, and is by the standards of that time pretty mild - basically just asking the Cossacks to bend the knee, like their fellow people on the other end of the river did, and stop harassing the Ottoman territories. But these Cossacks were having none of it, and by the end of the war Russia ended up owning their side of Ukraine (separated by the Dnepr river) while Ottomans ended up owning the other part. Of course, it soon led to the next war, of which there was about a dozen overall. Cossacks as a community (or rather very diverse set of communities) survived most of them, until the Communist revolution, which largely destroyed their way of life. Though it is said to be reviving now, not sure whether it is authentic anymore.

</comment></comment><comment><USER7> That's one of the most lively paintings I've ever seen. I suspect there are depths of symbolism of which I'm totally unaware and probably a bunch of insults to ethnic groups, but to my untrained eye it reminds me of Bruegel the Elder meets Norman Rockwell.

<comment> andrewl-hn One interesting detail is the rolled banners. Some of them are yellow and blue - same flags that modern Ukraine uses. Others are black and red - the colors used by Ukrainian rebel armies against Polish Lithuanian Commonwealth, Ottoman Empire, and Russia. Centuries later black and red banners were used by Euromaidan protesters.

</comment></comment><comment><USER8> > The second version of "The Cossacks" the artist tried to make more > "historically authentic". In 1932 it was presented by the Tretyakov Gallery > to the M. F. Sumtsov Kharkiv Historical Museum.

The differences in the two versions is interesting. It's mostly some subtle differences in hair styles and clothing design. I'm curious what the exact motivation/reasons were.

</comment><comment><USER9> > the crick in our dick

gold</comment>


WordPress disable the new admin bar - <AUTHOR> http://wpsnipp.com/index.php/functions-php/disable-the-new-admin-bar/

<comment><USER2> Why?</comment>


Achieving a profitable product/market fit - <AUTHOR> http://wensing.tumblr.com/post/2976690885/achieving-a-profitable-product-market-fit

<comment><USER2> Good post. Points out the real need of other methodologies like Customer Development and Business Model Generation for finding _profitable_ product/market fit.

Leancamp's a good example of this too. It has product/market fit, and passes the Net Promoter / Sean Ellis tests. But the business model itself isn't easily sustainable or scalable, which is where I've had to place my attention the last few months.

</comment><comment><USER2> David Heinemeier Hansson of 37 Signals made the same point about pricing at SXSW last year. He encouraged people to charge more, and pointed out that 37 Signals' experience is that the quality of customer feedback and customer relationship goes up with the more you charge.</comment>


Is Chinese-style surveillance coming to the West? - <AUTHOR> https://www.theguardian.com/commentisfree/2019/may/07/chinese-style-surveillance-exported-west <comment><USER2> Its already here. Look at all the proactive "banning" of the voices on the other side.</comment>


Ask HN: What's the best way to teach my kid to code? - <AUTHOR>

Hey folks. Apologies if this question has been asked before (if it has, please point me to the answers). I want to get my 5 year old son into coding. He's pretty handy with basic tasks on the computer, and more importantly is inquisitive and creative.
He also has the attention span of a 5 year old, so if he wants to learn anything, it not only needs to asume no prior knowledge of programming, but also needs to quickly and repeatably deliver some escalating level of gratification in order to keep a kid (or for that matter, most adults) interested.
What resources are around geared towards kids learning to code that don't assume too much knowledge, are hands on and interactive, and are actually fun for a 5 year old? If you're a parent who's taught their kids to code, what resources and/or techniques worked for you? What didn't?
(A future programming prodigy is grateful in advance for the benefit of your insight.) <comment><USER2> I started my 6 year old daughter on Scratch. I think it will fill the bill for your son. It's very visual and includes enough control structures to allow you to make real programs (games and media presentations are its forte.)

<http://scratch.mit.edu/>

Now that she's 8 and has a better understanding of math she's moved on to <http://inventwithpython.com/>

<comment><USER3> There may also be some good puzzle games around which teach relevant skills. Boolean algebra, stuff like that.

Simple programmable robots are also good, like the old LEGO turtle. If it's accessible, kids will often self-determine things they want to make it do and then work at it until it happens (provided they don't get stuck too badly).

</comment></comment><comment><USER4> If he likes Legos, the Lego Mindstorms kits are awesome. You get to build projects and then program them to do things. I'm not sure how the kit is currently set up, but 10 years ago it shipped with a basic, graphical programming language. You snapped together programming blocks to create the logic. Would probably be perfect for a younger kid.

When your kid outgrows the graphical language (I did after a while) people have ported Java and C to the Mindstorms too.

</comment><comment><USER5> <http://hackety-hack.com/></comment>


Easy Faxing Anywhere with Onlinefaxes.com - <AUTHOR> http://www.onlinefaxes.com/ <comment><AUTHOR> Faxes are sent to your email as a PDF, where you can view, download and share them. You never have to worry about missing a fax or not knowing when it arrives. All faxes can be accessed from anywhere there's an internet connection by computer, smartphone, or tablet.</comment>


New York Times paywall goes up January 2011 - <AUTHOR> http://mashable.com/2010/05/14/nyt-paywall-january-2011/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Mashable+%28Mashable%29&utm_content=Google+Feedfetcher

<comment><AUTHOR> I don't know. It sounds like a pretty reasonable system to me. Information wants to be free, but fair accurate reporting certainly is not.

<comment><USER2> Free as in speech, not as in beer.

</comment></comment><comment><USER3> When Murdoch announced that the Times of London would become a subscriber-only site, my reaction was "there's no way I'm paying for the same news I can get anywhere". With the NYT, I'm not so sure. There's still the basic issue of the gulf between "free" and "not free" regardless of price, but the NYT does things that I just don't see anywhere else.

If I were Bill Keller, I'd be thinking seriously about how much value traditional news actually adds to a paid product. I certainly place very little value in reading regurgitated wire feeds and press releases. Come to think of it, I'd be far more inclined to pay for the New Yorker, Salon or Slate than the NYT. Hmmm.

</comment><comment><USER2> It's only a partial paywall. A redirect, referer change or resetting cookie file would be able to fix their broken website when the time comes.

<comment><USER4> If you read their articles enough to hit the paywall, why not just pay to support the product they're creating?

<comment><USER2> Information wants to be free.

<comment><USER4> Journalists are the ones who free that information in the first place. The less remunerative that career is, the less information there will be. I think it helps to consider your subscription to be more of a donation than payment for a product. Then again, I work for a non-profit news organization. (Feel free to donate.)

</comment></comment></comment></comment><comment><USER5> I'd pay for information that helps me make money... I don't think I'd pay for information that's just a different presentation of the same news that everyone else is covering.

I WOULD however pay a subscription for Glenn Beck to be TAKEN OFF the air, and the threat of having him come back on would be enough to keep me paying regularly.

<comment><USER4> _"I don't think I'd pay for information that's just a different presentation of the same news that everyone else is covering."_

The New York Times does plenty of original journalism.</comment></comment>


The Great Legacy Code Crisis of 2016 - <AUTHOR> http://devops.com/blogs/great-legacy-code-crisis-2016

<comment> at-fates-hands From the article:

 _" What is scary is that by legacy I am not even talking about mainframes or Visual Basic desktop applications. I'm talking about last generation .NET and J2EE platforms which are too heavyweight, slow and bulky to change. Change, iteration and experimentation is what's important in the new world and these platforms are just not optimised for that."_

Not sure what the author considers "last generation" but over the last three years I worked on several large enterprise applications that were built on .Net and Java.

The Java application was being moved from an older JSE 6 codebase to Spring MVC, a robust, MVC Java framework. The project went extremely well and there were very few bumps in the road. Six months in and out and the project was done. We completely redesigned and migrated the entire application.

The other two .Net projects I worked on were a total breeze. One was a really old webforms enterprise app and the move to MVC4 was pretty easy. The other was a more recent move from MVC3 to an Azure based platform. Again, very easy (I've always liked the Microsoft platform) to migrate and update.</comment>


Why did TermKit die? - <AUTHOR>

Original story: http://news.ycombinator.com/item?id=2559734<p>If you check out the project on GitHub you'll see that there were no commits since July, and there are no proper forks. <comment><USER2> I'm also sad it's died, it was quite a neat concept and project.</comment>


Reverse Engineering Challenges - <AUTHOR> http://challenges.re/ <comment><USER3> This site and accompanying book ([http://beginners.re/Reverse_Engineering_for_Beginners- en.pdf](http://beginners.re/Reverse_Engineering_for_Beginners-en.pdf)) seem a nice effort. Briefly reading the book, it seems that you need to be an intermediate or advanced beginner, or know something about assembly beforehand.

Some terms (opcode, ISA) are not really explained (except in the glossary) before they are used and there are perhaps too much detail in the expense of the bigger picture.

Criticisms aside, the book and the challenges seem interesting and the efforts of the author must be appreciated!

<comment><USER4> Get past the idea that you need to be advanced to grok assembly language. Assembly is in a lot of ways _easier_ than a lot of higher-level languages. When I was growing up in the early 90s, a lot of my friends started on x86 assembly as a first language --- and x86 is the most annoying instruction set to learn.

The right way to learn this stuff is to dive in. You'll be over your head for a few hours, but you'll get your bearings. There are topics this technique doesn't work great with, but assembly reversing isn't one of them.

An additional benefit: assembly is one of those things that you might not use all the time in your career (although I've ended up using it quite a bit), but that will nonetheless illuminate lots of other things about computer science. There's a reason Knuth used it as a language to express algorithms in TAOCP.

I can sum up the core idea of assembly for you in just a few sentences:

* You're given 8-32 global variables of fixed size to work with, called "registers".

* Virtually all computation is expressed in terms of simple operations on registers.

* Real programs need many more than 8-32 variables to work with.

* What doesn't fit in registers lives in memory.

* Memory is accessed either with loads and stores at addresses, as if it were a big array, or through PUSH and POP operations on a stack.

* Memory is to an assembly program what the disk is to a Ruby program: you pull things out of memory into variables, do things with them, and eventually put them back into memory.

* Control flow is done via GOTOs --- jumps, branches, or calls.

* A jump is just an unconditional GOTO.

* Most operations on registers, like addition and subtraction, have the side effect of altering status flags, like "the last value computed resulted in zero". There are just a few status flags, and they usually live in a special register.

* Branches are just GOTOs that are predicated on a status flag, like, "GOTO this address only if the last arithmetic operation resulted in zero".

* A CALL is just an unconditional GOTO that pushes the next address on the stack, so a RET instruction can later pop it off and keep going where the CALL left off.

Everything else is just a detail.

When working with assembly, a lot of people will just get the programmer's reference manual for the instruction set in a PDF (they're published for free). Here's a short one for X86:

[http://ref.x86asm.net/coder32.html](http://ref.x86asm.net/coder32.html)

Here's ARM:

[http://infocenter.arm.com/help/topic/com.arm.doc.qrc0001m/QR...](http://infocenter.arm.com/help/topic/com.arm.doc.qrc0001m/QRC0001_UAL.pdf)

Here's AVR:

[http://www.atmel.com/images/atmel-0856-avr-instruction- set-m...](http://www.atmel.com/images/atmel-0856-avr-instruction-set- manual.pdf)

Even if you'd never written a line of assembly, if I asked you to express the procedures of a simple Ruby program in assembly and gave you the instruction set reference and those sentences, you would figure out how to get the job done in a couple hours. No monads or linear algebra required!

<comment><USER4> Hm. 12 sentences. Someone here can do it better in fewer sentences, I think.

<comment><USER5> Much of assembly programming is variants of these commands:

        - mov <dst>, <src>            | dst = src
        - sub <dst>, <src a>, <src b> | dst = a - <USER2>
        - jump <label>                | continue running at <label>
        - jump if equal <label>       | continue running at <label> if both sources of
                                        the last command were equal
        - call <label>                | save the current location, then continue
                                        running at <label>
        - ret                         | continue running after the previous CALL
                                        instruction

<dst> can be:

        - any memory address
        - any of 8-32 available temporary integer variables called "registers"

<src> can be any valid <dst> or a hardcoded integer known as an "immediate"

<comment><USER4> This is both more detailed than mine and uses fewer sentences. My only nit is that I feel like understanding status flags is really important.

<comment><USER5> I had several nitpicks with mine, but realized once the learner starts asking questions about gaps they're in a pretty good place to find an answer on their own.

I do agree flags is less obvious, but also takes a bit of space to list what kind of flags one might see:

"jump if equal" is facilitated by an implicit "flags" register which is set after most arithmetic or comparison operations like "sub". Possible flags include:

\- carry: the last arithmetic operation overflowed

\- zero: result of the last operation was zero

\- parity: result of the last operation was odd

\- sign: result of the last operation was negative

</comment></comment></comment></comment></comment></comment><comment><USER6> Although probably not well-known, [http://crackmes.de/](http://crackmes.de/) is a site I've been using to create / upload and also work on challenges for various RE applications for about 10 years now. I'd definitely recommend it to newcomers wanting to learn more about RE in general.

</comment><comment><USER7> My Sophos Anti-Virus blocked this site, mentioning a 'Mal/HTMLGen-A' on it.</comment>


Augur: Mining Human Behaviors from Fiction to Power Interactive Systems (2016) [pdf] - <AUTHOR> http://hci.stanford.edu/publications/2016/ethan/augur-chi2016.pdf <comment><USER2> why is stanford paying attention to garbage like ethereum contracts?</comment>


Treasury to acquire 30% of company for $700M. Company is worth $120M - <AUTHOR> https://www.washingtonpost.com/business/2020/07/01/treasury-loan-yrc-worldwide-cares-act/ <comment><USER2> The $700M is a loan. Misleading headline...

<comment><USER3> The full headline is

"In unusual deal, U.S. Treasury to acquire 30 percent of trucking company in exchange for $700 million loan"

I wonder if a better title would be

"Treasury to acquire 30% of 120M company in exchange for 700M loan"

<comment><USER3> Actually - I can't find where the 120M figure was obtained from. Does anyone else know where that number in the title came from?

<comment><USER4> Good question, maybe the article link changed? This article talks about the company's valuation: only $70 million as of Tuesday's close, but up a lot today after this news. [https://www.cnn.com/2020/07/01/business/yrc-federal- loan/ind...](https://www.cnn.com/2020/07/01/business/yrc-federal- loan/index.html)

</comment></comment></comment></comment><comment><USER5> [https://outline.com/gSACeH](https://outline.com/gSACeH)

</comment><comment><USER6> Shocking...

The Trump administration just lent $700 million to a trucking company sued for ripping off taxpayers

[https://www.cnn.com/2020/07/01/business/yrc-federal- loan/ind...](https://www.cnn.com/2020/07/01/business/yrc-federal- loan/index.html)</comment>


My First 10 Minutes on a Server - <AUTHOR> http://www.codelitt.com/blog/my-first-10-minutes-on-a-server-primer-for-securing-ubuntu/ <comment><USER2> This one is pretty decent but if you want the ultimate guide check out this one:

[https://www.inversoft.com/guides/2016-guide-to-user-data- sec...](https://www.inversoft.com/guides/2016-guide-to-user-data-security)

It covers 10x what all the other guides cover in terms of server and application security. It was posted a few weeks ago on HN but didn't make the front-page.

<comment><USER3> My worry here is that, in posting what seems to be a book, people just won't even do it because we don't have time to do it, unless this is a primary part of their jobs.

If a 10 minute guide gets users 90% of the way, then they're more likely to do it. And that's good enough to cover a majority of automated attacks.

Update: I take it back – they've provided scripts to run this stuff. I will explore these. Thanks for the link.

<comment><USER4> Back in the day, The Linux Documentation Project had a trove of hundreds of HOWTOs covering every facet of using Linux. By today's standards they seem like "books", but in reality they were step-by-step instructions for anything you could ever want to do in Linux. No digging through forums, no combing through man pages, no following broken outdated blog posts that didn't explain what you were doing. I find it sad that these are seen as detrimental today.

<comment><USER5> I feel like arch wiki is that thing today. It's a bit disorganised and things get outdated while nobody's looking, but there's almost anything you'd every want in there.

<comment><USER4> It seems like there a bunch of articles there but they're more like notes on how to use a tool specifically with Arch, rather than a guide for everyone.

Compare this page [http://www.tldp.org/HOWTO/Quota.html](http://www.tldp.org/HOWTO/Quota.html) to this page [https://wiki.archlinux.org/index.php/Disk_quota](https://wiki.archlinux.org/index.php/Disk_quota) . TLDP organized HOWTOs like mini-books; tables of content, multiple authors, versioned releases, and of course, you could download them all and search through them by category. And they didn't assume things like the distribution setting up a bunch of the tools and system for you, so you learned how the tools _actually_ worked.

<comment><USER5> Sure, it's a different format. But you can always dig into man pages or other documentation to learn the details. Arch wiki is great for practical solutions - for example [https://wiki.archlinux.org/index.php/HiDPI](https://wiki.archlinux.org/index.php/HiDPI) is the place to go to for hidpi display configuration. TLDP has not been seriously updated since hidpi even came out. Similar to [https://wiki.archlinux.org/index.php/Power_management](https://wiki.archlinux.org/index.php/Power_management) \- there's no recent TLDP equivalent.

I appreciate TLDP for what it is and the details they go to. But Arch wiki can easily be read as a FAQ for all modern systems. And it will usually send you to other places for details (sometimes TLDP as well)

</comment></comment></comment></comment></comment></comment><comment><USER6> > We don't even have a password for our root user. We'll want to select > something random and complex.

So you're taking something secure by default -- no password means no login allowed, and making it less secure. And if you have hundreds of these servers, you'll need to rotate them whenever someone on the team leaves. This is painful.

Simple solution: leave root password blank, don't forget your sudo password. If you can't get in, use grub or a liveCD. Or tie auth to ldap or kerberos so you _can't_ forget. This is one area where Windows has a distinct advantage: AD more or less requires admins to think at the level of network of servers, and provides a baseline set of services always present.

<comment><USER7> This part struck me as odd, too, especially when you consider the justification: That if you lose access to your sudo account/password, you must have some other way to get into the system.

Backups should include everything required to rebuild every server in the company. No server should be 'too critical' to wipe and start again if required. So maybe I'm being too much of an armchair warrior here, but the reason I don't like this one is less about the actual security implications, and more because of what it says about the fragility of the overall setup.

<comment><USER8> >Backups should include everything required to rebuild every server in the company. No server should be 'too critical' to wipe and start again if required.

True. But compare the effort required for, say, solving an accidentally misconfigured /etc/sudoers on a database server. You could argue that would never happen with proper testing, but shit happens and and having a particularly long root password in Keepass is a small price to pay to save that sort of headache.

</comment></comment></comment><comment><USER9> This is good advice on achieving the most secure SSH configuration: [https://stribika.github.io/2015/01/04/secure-secure- shell.ht...](https://stribika.github.io/2015/01/04/secure-secure-shell.html)

"My goal with this post here is to make NSA analysts sad."

<comment><USER10> Actually, I've improved on this somewhat by splitting configs to 6.5+ vs. older. Corrections welcome. At some point, I will get around to publishing it.

 __Configs __

OpenSSH 6.5+ Server

UsePrivilegeSeparation sandbox KexAlgorithms firstname.lastname@example.com,diffie-hellman-group14-sha1 Ciphers firstname.lastname@example.com,firstname.lastname@example.com,email@example.com,aes256-ctr,aes128-ctr MACs email@example.com,firstname.lastname@example.com,hmac- sha2-512,hmac-sha2-256

OpenSSH Server Legacy

#UsePrivilegeSeparation yes KexAlgorithms diffie-hellman-group14-sha1 Ciphers aes256-ctr,aes128-ctr MACs hmac-sha2-512,hmac-sha2-256

OpenSSH 6.5+ Client

UseRoaming no IdentitiesOnly yes KexAlgorithms firstname.lastname@example.com,diffie-hellman-group14-sha1,diffie-hellman-group- exchange-sha256,diffie-hellman-group-exchange-sha1 HostKeyAlgorithms ssh- firstname.lastname@example.com,ssh-ed25519,email@example.com,ssh-rsa Ciphers firstname.lastname@example.com,email@example.com,email@example.com,aes256-ctr,aes128-ctr MACs firstname.lastname@example.com,firstname.lastname@example.com,hmac- firstname.lastname@example.com,hmac-sha2-512,hmac-sha2-256,hmac-sha1

Host * IdentityFile ~/.ssh/id_ed25519 IdentityFile ~/.ssh/id_rsa HashKnownHosts yes VisualHostKey yes VerifyHostKeyDNS ask AddressFamily inet ForwardX11 no ForwardX11Trusted no

OpenSSH Client Legacy

UseRoaming no IdentitiesOnly yes KexAlgorithms diffie-hellman-group14-sha1 HostKeyAlgorithms email@example.com,ssh-rsa Ciphers aes256-ctr,aes128-ctr MACs firstname.lastname@example.com,hmac- firstname.lastname@example.com,email@example.com,hmac-sha2-512,hmac- sha2-256,hmac-sha1

Host * IdentityFile ~/.ssh/id_rsa HashKnownHosts yes VisualHostKey yes VerifyHostKeyDNS ask AddressFamily inet ForwardX11 no ForwardX11Trusted no

</comment></comment><comment><USER11> > I check our logwatch email every morning and thoroughly enjoy watching > several hundreds (sometimes 1000s) of attempts at gaining access with little > prevail.

This is something that actually bugs me a bit. These attacks are so common, getting emails like this every day contributes to alarm fatigue. ([https://en.wikipedia.org/wiki/Alarm_fatigue](https://en.wikipedia.org/wiki/Alarm_fatigue))

I'd love to see the Linux nightly security scripts replaced with something that only sends out emails when there's an specific _actionable_ event I need to pay attention to. Ideally in a way that can easily be aggregated over all the machines I manage.

<comment><USER12> Yep and this doesn't demonstrate anything about security. Showing brute force scanners trying out "root/letmein123" doesn't teach anyone the importance of good security, just the importance of not using super-common user/passes.

I cannot figure out why anyone would care or find anything useful in these logs. Change the port, call it a day. Getting worked up about random SSH attempts (or random HTTP "exploit" attempts) seems to be for admins with too much free time.

</comment></comment><comment><USER13> > sudo ufw allow from {your-ip} to any port 22

I'm surprised nobody mentioned this is a great way to shoot yourself in the foot if you don't have a static IP.

<comment><AUTHOR> Ooo. Fair point. I'll add that now and link to your comment.

<comment><USER13> Great! Thanks for the article, it makes a good reference when setting up VPS's.

</comment></comment></comment><comment><USER14> I don't mean to sound flippant but why can't these "lock down your new box" tutorials just be a bash script? Shouldn't they be?

<comment><USER15> Hardening (along with any other server setup/configuration) should be implemented using some idempotent configuration management software (Chef, Puppet, Ansible, etc). It's 2016, there's no need to configure servers manually.

<comment><USER16> It depends.

I'm a security guy, so it would be embarrassing and possibly bad for my career if any of my servers got hacked, but I actually don't set up servers that often--it's not part of my job. These Web 2.0 configuration management solutions change pretty fast and don't care about reverse compatibility. So between my infrequent setups my configuration scripts pretty much always break.

Contrast this with bash, which cares a whole lot about reverse compatibility, and it's a no brainer. I've got scripts where the only modifications I've made since 2005 were to add functionality or increase security, never to fix existing functionality that was broken by a change to the system. I'll take that over Chef or Ansible (both tools I've used) any day.

<comment><USER17> Would be very interested to review server hardening Bash scripts. Have you published anything on Github or elsewhere?

<comment><USER16> I have, but I don't want to associate this HN account to my GitHub account, sorry. :/

<comment><USER18> throwaway github account then? :)

</comment></comment></comment></comment></comment></comment><comment><USER19> Why do people install fail2ban then disable password based authentication entirely? I legitimately don't understand the purpose.

Also, they complain about log spam (from failed SSH attempts) this is one reason to move SSH to a different port. It does NOT increase security, but it DOES reduce log spam from bots trying for easy targets.

<comment><USER20> I always change the ssh port to something other than 22. It has always seemed to work well for me for most automated attacks. Perhaps this is not advisable for some reason?

<comment><USER21> There is a slight security concern to binding it to a port above 1024 in that a non-privileged user can bind to that port and MITM.

Here is a good summary of the different options you have with ssh and choosing a port:

[http://serverfault.com/questions/619898/should-i-change- the-...](http://serverfault.com/questions/619898/should-i-change-the-ssh-port- to-1024)

</comment></comment></comment><comment><USER22> Fail2ban? sshguard? unnecessary. Just disable ssh passwd auth (which generally is a good idea) -> done/done

If you don't like lognoise from ssh scanners (even if you disable passwd auth), move your sshd port to some random high port and make note of it in your ~/.ssh/config

Generally: if in doubt, take the more simple and elegant solution to a problem.

<comment><USER23> I agree that fail2ban for ssh seems unnecessary. But, it also provides monitoring for other services like http and common exploits.

I'd be interested in learning from the community if fail2ban adds much value. As I've looked into the service, it seems like simply running the latest security patches obviates the need for fail2ban.

<comment><USER20> I run a web server with some 50+ Wordpress installs on it. You better believe Fail2Ban is necessary. Without it all resources would be consumed by brute force attacks. If someone knows of a better way I would like to hear about it.

<comment><USER24> We use the WordFence module to block brute force attacks, seems to work fine.

I cannot believe that Wordpress _still_ ships without basic rate limiting on its login form.

</comment></comment></comment></comment><comment><USER25> I prefer the "Securing Debian Manual" – it's an _official manual_ from the Debian project.

[https://www.debian.org/doc/manuals/securing-debian- howto/](https://www.debian.org/doc/manuals/securing-debian-howto/)

</comment><comment><USER26> Be aware fail2ban does not handle IPv6 at all with its default configuration on Debian/Ubuntu.

[https://github.com/fail2ban/fail2ban/issues/1123](https://github.com/fail2ban/fail2ban/issues/1123)

<comment><AUTHOR> You're right. Took mwpmaybe's suggestion for the time being and only will accept IPv4 from ssh for now till that's fixed.

</comment></comment><comment><USER27> Great tactical advice, but what a sad situation to be in. "Run this command, then run this command, then run this command ..."

There should be a single configuration file (or set of files) that declaratively describes the whole state of the machine. That way the exact situation of the server can be reviewed by just looking at files, instead of trying to poke and prod at the machine to see what commands have been run over the last X weeks.

<comment><USER28> There is a set of files that describes the state of the machine, it's called the filesystem. Anything less doesn't describe the whole machine. The 'poking and prodding' is just a convenient way of querying the very small parts of the filesystem that are relevant to that query.

That said, a script that pokes and prods the right places and reports a machine's 'security factor' and prompts improvements would be cool (and probably already exists).

<comment><USER29> > That said, a script that pokes and prods the right places and reports a > machine's 'security factor' and prompts improvements would be cool (and > probably already exists).

[https://cisofy.com/lynis/](https://cisofy.com/lynis/)

</comment></comment></comment><comment><USER30> Since I changed the default SSH port of 22 to something else (like 4422), I no longer get any of these drive-by attacks and don't need fail2ban anymore.

I also like to set up a simple Monit configuration to alert me about high cpu usage or when the disk space is about to run out. Instead of emailing me these alerts (and also weekly reports) I've configured Monit to post them to my Slack team of 1.

[https://peteris.rocks/blog/monit-configuration-with- slack/](https://peteris.rocks/blog/monit-configuration-with-slack/)

<comment><USER31> You should still use fail2ban.

[https://news.ycombinator.com/item?id=11854576](https://news.ycombinator.com/item?id=11854576)

</comment></comment><comment><USER32> What's the reason for using a firewall?

Assuming that services which shouldn't be accessible to the outside only listen to localhost not the network (e.g. MySQL on a LAMP stack), isn't that sufficient?

(Honest question, I don't have much experience with syadmin.)

<comment><USER33> So there's a couple of reasons to add a firewall.

1\. If an attacker gets unprivileged access it can slow them down (if properly configured) in getting new tools onto the system or adding a shell.

2\. If a configuration error results in a service being started on a network accessible interface by accident the firewall gives you a bit of defence in depth protection against unauthorised connections to that server.

3\. you can also use it for logging activity to feed into other systems.

<comment><USER12> >1\. If an attacker gets unprivileged access it can slow them down (if properly configured) in getting new tools onto the system or adding a shell.

That only works if you have an outbound firewall. Which is very onerous - you'd either have to whitelist destinations (package repos, but what if you want to validate arbitrary certificate's CRLs?) or whitelist applications (but not wget etc.)

</comment></comment></comment><comment><USER34> Here's an Ansible role (I made it) that automates the steps described in the article: [https://github.com/LucianU/ansible- secure](https://github.com/LucianU/ansible-secure).

<comment><USER35> See also [https://github.com/geerlingguy/ansible-role- security](https://github.com/geerlingguy/ansible-role-security) mentioned above

</comment></comment><comment><USER36> Can somebody help me out with this question: The default config for unattended-upgrades seems to not enable reboot even if a reboot would be required to activate the upgrades. Wouldn't that had made quite a few important upgrades in the last years effectless if they server did never get rebooted?

<comment><USER37> Not sure if this is still true, but I've also seen cases where Ubuntu will happily continue to install kernel updates as they come down the pipe, right up until /boot is full of old kernels and ramdisks.

</comment></comment><comment><USER38> For those saying "why fail2ban?", fail2ban can be used for a great deal more than just watching the sshd log. You can activate fail2ban rules for apache and nginx which help significantly with small DDoS, turning spurious traffic/login attempts into iptables DROP rules. And a lot of other daemons.

<comment><USER39> At least one log parsing tool I've seen in years past was vulnerable to log injection attacks. Hilarious proof of concept to own a box by way of PTR record.

I haven't checked to see whether fail2ban suffers from this model or not.

</comment></comment><comment><USER40> I'd be more curious to see a "My first 10 minutes on an Ubuntu desktop" version of the article.

<comment><USER41> Or alternatively: "My first 10 minutes on an Ubuntu desktop (for users who don't hate Unity, which includes myself)".

In my case: Change the Switch workspace keys from Ctrl+Alt+Arrow keys to Super+ Arrow keys. Remove LibreOffice, install WPS Office. Remove Transmission, install Deluge. Install indicator-multiload, indicator-sound- switcher. Install Kodi. Install Steam.

A lot more stuff, but I have not written it down. =)

<comment><USER42> First time I hear about wps office, why do you prefer it over libreoffice? mobile support?

<comment><USER41> It's faster (C++ vs Java) and more compatible with the documents I have to open.

It also pleasantly surprised me once:

I received a Powerpoint presentation, and went to a customer meeting, where said presentation had to be presented.

I plugged the HDMI to VGA adapter for the VideoBeam to the laptop, started the presentation, and the presentation was running in the external display, while the laptop display was still showing the normal 'Powerpoint' view. I could load the web browser in the laptop display to check some things while the presentation was still running undisturbed and the speaker and the audience was happily unaware of it.

That level of professional use in software was something I did not come to expect in Ubuntu for any third party software. In fact I don't know if the other Office suites (including MS) have that feature. I guess they do, but still.

It is wonderful when everything just works as intended.

</comment></comment></comment></comment><comment><USER43> > First we'll want to make sure that we are supporting IPv6

How does that help security?

<comment><USER44> That comment was in regards to ufw as it doesn't support adding IPv6 rules by default in commands in older Ubuntu.

If you don't enable IPv6 and the server supports it (odds are it does) all of the benefits of using ufw at all are totally ignored on IPv6.

</comment></comment><comment><USER45> I'm finding that another important step is this one:

apt-get install etckeeper && cd /etc && etckeeper init

Keeps your /etc under version control so you know what kinds of configuration changes you've perpetrated.

<comment><USER46> I used to use this, but I found that it's significantly less useful than a git repo with my server-specific Chef cookbook in it.

Manually modifying servers was never a good idea; it's worse now, even with tools like this.

<comment><USER45> I suppose that's true, but in the field where I work (social science), servers are mostly spun up to scratch an immediate and idiosyncratic itch, so configuration tends to happen organically.

I agree that's probably not a good idea, and learning Pupchefsible is well worth the effort. In the meantime, though, there's at least some degree of reproducibility with etckeeper.

<comment><USER46> I do this stuff professionally, and I've learned the hard way that you either have a reproducible environment or you don't. etckeeper _isn 't_ reproducible. Actually rolling back with something like etckeeper is much, much more likely to break something (by deleting a config file used by a newer service, say) than to save you. If it did something like separate branches for each service or component within /etc I might be more sympathetic...but at that point you have half of a CM system already and might as well just go the rest of the way.

If you need reproducibility without a CM framework, keep backups of your machines.

</comment></comment></comment></comment><comment><USER47> I have been meaning to write up a similar guide.

I would like to recommend using just iptables instead of ufw, I had a case on my vps where an update to ufw failed and then the firewall was not working.

With iptables, install iptables-persistent package so they are saved when you do restarts. Do not try to block entire country ip ranges as this slows the machine down substantially.

fail2ban is great, I would recommend looking at some of your system logs to figure out new rules to add.

<comment><USER48> Ferm [1] is wonderful as an iptables frontend. Apart from making it simpler to read, it can avoid a lot of repetition with handy shortcuts such as ability to group things. E.g.:

        proto tcp dport (smtp ssmtp qmtp pop3 pop3s imap2 imap3 imaps) ACCEPT;

.. creates rules for each of the ports listed. You can use multiple groups in the same statement as well (lets say all the services above for some crazy reason also listened to udp - you'd just replace "tcp" with "(tcp udp)").

Being able to set variables is also fantastic. E.g.:

        @def $DEV_PRIVATE = (eth0 eth1);

[1] [https://github.com/MaxKellermann/ferm](https://github.com/MaxKellermann/ferm)

<comment><USER49> Also, FireHOL [1] is an interesting option. It also has a DSL for managing firewall rules. The custom service port definition is a little weird, but overall I like it better than ferm.

[1] [https://firehol.org/](https://firehol.org/)

</comment></comment></comment><comment><USER50> Why don't they disable root logins with password period and only allow SSH key authentication?

Also if you put a passphase on your SSH key does that mean you have to enter it every time you want to SSH to the server (in order to unlock the key) or does it stay cached on most SSH clients (ssh on mac terminal, putty on windows, etc).

Isn't watching failed logins kind of useless? I think it is more important to see what successful logins were made.

<comment><USER51> If you put a passphrase on your ssh key you only have to enter it when you initially add it to your ssh-agent.

<comment><USER52> yes, ssh-agent will let you enter the password once, and then it won't prompt you anymore.

(see [https://help.github.com/articles/working-with-ssh-key- passph...](https://help.github.com/articles/working-with-ssh-key-passphrases/) )

</comment></comment></comment><comment><USER53> Its a tradition to nitpick these kinds of lists. Here's my take.

>I generally agree with Bryan that you'll want to disable normal updates and only enable security updates.

Hmm, fairly certain the Ubuntu (and others) don't do major product updates or API breaking updates via apt-get. You shouldnt have to worry about breaking anything if you use normal updates. This seems a bit too conservative for me and leads to problems down the line of being on an ancient or bugged library and then having to do the update manually later, usually after wasting a couple hours googling why $sexy_new_application isn't working right on that server.

He setup an email alert, but not an smtp to actually send it. Also, OSSEC takes a few seconds to install and is much nicer than emailing full logs.

Lastly, fail2ban is becoming a sysadmin snake-oil/fix-all. Its use is questionable in many circumstances. There's a real chance of being locked out of your own server with this. If people are recommending it, they should be giving noob-friendly instruction to whitelist their IP at the very least.

</comment><comment><USER54> Not sure if others feel this way but adding this line to sudo never felt right to me...

deploy ALL=(ALL) ALL

I usually instead limit the deploy user to a smaller subset of commands e.g. the init.d script to control a service.

obviously if someone gained access to deploy user we're probably sol anyway... but it just makes it seem safer... we have a to login as an ops user to install or update things on the boxes.

<comment><AUTHOR> Someone on /r/netsec rightly pointed out that you shouldn't ever add a user directly to sudoers anyways. You should add them to the sudo or wheel group. I've since updated the article.

What I've described is a more of a base, but according the Principle of Least Privilege you could go even one step further and do what you're suggesting. You'd probably want to have a couple of users though. An admin user, a deploy user, and a maintain user all with different privileges.

<comment><USER55> > _you shouldn 't ever add a user directly to sudoers anyways_

What was the reason for that? I have the deploy user able to run a couple of individual commands without a sudo password (scripts that run canned updates, to be initiated from a buildserver), but I don't see how it would improve things to use a group instead that only holds that user.

</comment></comment></comment><comment><USER56> I created a script that does almost everything automatically using Ansible and Ansible Vault : [https://github.com/guillaumevincent/Ansible-My- First-5-Minut...](https://github.com/guillaumevincent/Ansible-My- First-5-Minutes-On-A-Server)

</comment><comment><USER57> I always worry that adding 2FA could make your machine inaccessible if anything happens to google-authenticator in this case. Maybe it's a little bit of paranoia but I don't like the idea of giving control over my ability to log into my server.

<comment><USER58> It'd be perfectly reasonable if libpam-google-authenticator relied on Google's infrastructure, but despite the infelicitous name, it does not; it just implements the server side of TOTP. The authentication flow is identical to any other correct TOTP implementation, and you can use any compatible client; no integration with Google services or infrastructure is required. (In fact, I don't think it's even possible.)

Speaking of TOTP without Google, if you use iOS and find the Google Authenticator app unsatisfactory, try Authy. It's good stuff, and well worth a few bucks.

<comment><USER59> Hurricane Electric's network tools iOS app is free and has an OTP client built in, with iCloud Keychain sync.

OTP Auth is another excellent, free OTP client.

Edit: clarified both are free.

<comment><USER58> While I'm not in the market for an OTP client, this is the network tools app I've been looking for since I bought my first iPod Touch. If you have remote beer-buying support, point me to it.

<comment><USER59> You're welcome. I only discovered it myself recently when testing their IPv6 tunnel service.

no beer required :-)

</comment></comment></comment></comment></comment><comment><USER60> For protecting against brute-force login attempts, I use sshguard [1]

I really think this should be installed by default on distros like Ubuntu.

[1] [http://www.sshguard.net/](http://www.sshguard.net/)

<comment><USER12> I can't see any benefit, what am I missing? Put SSH on a port that's not 22 and done, no more mass scanning. The only thing SSHGuard has ever done for me is to lock me out when I was accidentally using the wrong key.

<comment><USER61> If SSH is on a non-standard port, it is still possible to brute-force access to the server. You will see fewer automated attempts but you are still vulnerable to a motivated attacker who port scans you and finds the SSH port. Such an attacker is less common than automated scans but is more of a threat. With Sshguard, you are no longer vulnerable to this type of attack at all, no matter which port you run SSH on.

<comment><USER12> But brute forcing any reasonable password or key is so far-fetched as to not be something to even consider.

<comment><USER61> If you can make hundreds of login attempts per second, and you can keep at it for days/weeks/years, you can get through some pretty big password dictionaries with lots of variants (e.g. password/p4ssword/passw0rd/p4ssw0rd/etc.).

</comment></comment></comment></comment></comment><comment><USER62> One of the suggestions is to make sure your public key has the .pub extension, and they imply that if someone didn't include the extension they would be reprimanded - any reason for this in particular?

<comment><USER63> Having a reliable convention like that reduces the risk of someone accidentally copying their private key to a server.

</comment></comment><comment><USER64> > "You should never be logging on to a server as root."

Can someone explain me, let's say I disabled password logins and only allow login via a key, what are potential downsides of logging in as a root?

<comment><USER65> Well for one, every command you run has root privileges (instead of requiring sudo). Every process you spawn has root privileges. You're safer keeping your privileges limited and sudoing when required. Also you lose your real-life audit log if multiple people log in as root.

<comment><USER64> But when I tried using a regular user to run anything on a server - it still requires me to do sudo, so the process will anyway have root privileges, no? So I end up typing sudo most of the times becauase most of processes do not work without root priveleges

</comment></comment></comment><comment><USER66> Forgot to check if the server isn't backdoored. You will be surprised how many providers add many backdoors and monitoring systems you don't need (m looking at you AWS guys).

</comment><comment><USER67> I would be annoyed with a cryptic Audi password. I would prefer 'BatteryHorseStaple' passwords. Anything I can't remember gets written on a post it note and put next to my screen with what it is for. This is my behaviour and the problem with cryptic passwords is that there are others like me, willing to keep a good password secret and not willing to be so secret about a clumsy, easy to crack by machine but impossible to remember password.

<comment><AUTHOR> Get a PW manager instead. No passwords should be stored in your head (because every one should be different) and they should be stored behind encryption -- definitely not plain text nor sticky notes.

[http://keepass.info/download.html](http://keepass.info/download.html)

</comment></comment><comment><USER68> It may be useful, at step 0, to check out the server and see _basic server orientation_. Which Linux is it (cat /etc/*-release)? How much ram and disk (htop, df)? How is the filesystem setup (mount)? What packages are already installed (dpkg -l)? What processes are running (ps aux, htop)? What did the last root, including me, do (history)? I also like to know where is the box physically, roughly (tracert, run locally).

</comment><comment><USER69> My biggest concern with being on a VPS like Linode, once you're all done securing yourself and binding services to the local LAN IP, is an attack from within the network. The VPS you own is also accessible by others on the same subnet, contrary to what you might assume.

I'd love to see a ufw guide for whitelisting only your own internal IPs to be allowed access to any services for ultimate security.

<comment><USER70> Not sure what you mean but ufw by default blocks everything on your interface so other machines in the local subnet shouldn't have access.

If you want to have more security and no (or just a single) outgoing service configure OpenVPN with TLS and put all your local services in a local subnet for your machine. So not even a portscan can find something.

</comment></comment><comment><USER71> Very useful, most of this stuff is pretty common for anyone who has done any regular sysadmin work, but definitely good to have a checklist.

</comment><comment><USER72> echo "set background=dark" > /etc/vim/vimrc.local

<comment><USER26> All users in your team are forced to use dark terminal backgrounds?

<comment><USER73> You can actually autodetect whether the terminal background is light or dark. For any xterm-compatible terminal, write '\x1b]11;?\x07' to the terminal, and it'll write back a string telling you the foreground color (for instance, '\x1b]11;rgb:0000/0000/0000\x07', which if written back would set the foreground color). If the color matches 'rgb/RRRR/GGGG/BBBB', compute the luminance of that color, and assume a dark background if <0.5 and light otherwise.

<comment><USER74> I didn't know about this, but from now on, when writing CLIs that use color, I'm going to take this into account!

<comment><USER73> Awesome; more tools should do that. Some caveats, though:

* You might not get a response from every terminal, so limit how long you wait.

* If you don't already have echo turned off, turn if off before sending the sequence, because otherwise it'll be visible as though the user typed it.

* You don't know that the color will use the "rgb:RRRR/GGGG/BBBB" format (a terminal can return anything XParseColor can understand); just read the string from the escape to the terminator, look for 'rgb:', and ignore formats you don't understand.

* To calculate whether a color is "light" or "dark", see [https://en.wikipedia.org/wiki/Luma_%28video%29](https://en.wikipedia.org/wiki/Luma_%28video%29):

        dark = (0.299*red + 0.587*green + 0.114*blue) < 0.5;

<comment><USER74> Have you measured how long typical terminals take to respond?

Regarding the third point, it might be a good idea to just feed it to XParseColor and process it from there.

<comment><USER73> > Have you measured how long typical terminals take to respond?

Arbitrarily long. Consider that a user might run your application over SSH via a high-latency network connection. Better to just handle it asynchronously. Your input loop needs to watch for escape sequences anyway, so watch for that one and process it when or if you see it.

Sadly, that only works for interactive screen-oriented applications, not run- and-exit command-line applications that want to use color.

> Regarding the third point, it might be a good idea to just feed it to > XParseColor and process it from there.

That assumes you have libX11 and an X Display available. The former is a heavy dependency for a CLI application, and the latter requires you to connect to the X server.

I'd suggest just manually handling the common case of "rgb:R/G/B" (where each component may use 1-4 digits and requires scaling accordingly), and then deal with anything else if your users actually encounter it in the wild.

<comment><USER74> I assume XParseColor would therefore limit portability to OS X or Wayland Linux. I stand corrected.

I was speaking of this regarding screen-oriented termbox/curses applications, not run-and-exit applications.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER75> Technically you don't need the root password, you can always password recovery if you have access to the box. And how exactly did you lock yourself out of every account with sudo? Of course there's always "messed up my ldap or general network settings, can't log in to fix them". There's nothing wrong with setting your root password to a random string and throwing it away, after verifying your sudo works, I guess.

I will admit to being lazy, and with full automation its faster to spawn a new virtual image and let ansible run its course than to do root password recovery where you boot and tell the bootloader to make the init system /bin/sh and hand edit /etc/shadow and /etc/passwd and then reboot again, etc etc. I mean I can set up a new image almost as fast as I can reboot an old image, and I set up images a lot more often than I do password recovery, so...

Scrap the ssh commentary and set up ssh company wide as per stribika plus or minus local modifications:

[https://stribika.github.io/2015/01/04/secure-secure- shell.ht...](https://stribika.github.io/2015/01/04/secure-secure-shell.html)

"On large scale, you'll be better off with a full automated setup using something like Ansible"

At ANY scale you're better off, unless you're experimenting or time isn't money. It'll take longer to add the time to document and test what you're doing by hand than to convince ansible to do it for you. If you don't document or test you're just doomed, so its not like you can avoid that effort. With automation this is like "first two minutes on a server" not ten.

Some people like to drop a .forward in root's homedir sending mail to your sysadmin mailing list or yourself. I THINK but might be wrong that if you do that you don't have to tell logwatch whom to email to, it'll go to root then forward to the right people. More than logwatch assumes firstname.lastname@example.com exists as an email address.

You're missing setting up your centralized rsyslog or local equivalent, your munin/nagios/zabbix or local equivalent... I still configure zabbix by hand because I'm old fashioned but its possible to automate that.

NTP is also missing. You can make Kerberos a very sad faced puppy if time isn't synced. And its easy to set up to point to local trusted servers.

(Note, a post that's nothing but complaining still means the linked article is at least 99.9% correct, it is a nicely written wide ranging TODO list)

<comment><USER73> > And how exactly did you lock yourself out of every account with sudo?

A single typo in /etc/sudoers or any /etc/sudoers.d file will lock you out of all sudo usage. visudo helps with that, but a single mistake (including in a sudoers.d file installed by a configuration management system or package) will lock you out.

<comment><USER75> Yes sir, and that's why the paranoid sysadmin ssh's in, sudo su's up, THEN runs ansible and tests that it works before unleashing the ansible (or puppet) across the entire network.

Also if you do the "group auth" thing in sudoers then you edit that file approximately once per employment and never touch sudoers again. Of course that abstracts the problem into "I deleted the wheel (or sudo, or ...) group on the ldap server and now I can't sudo up to fix it". And that's why you make snapshot backups on the NAS, so you can roll back the image of the LDAP server (or whatever you use locally) (and edited to add, and don't do something dumb like use the ldap image running on the openstack to authenticate logins into the openstack... that would be painful indeed)

<comment><USER76> I prefer lazy sysadmin that avoids all that manual ssh/sudo by making puppet/etc validate sudoers before updating it. You can syntax check with visudo -c.

</comment></comment></comment></comment><comment><USER77> I have a VPS, when I first got it, it had an additional user setup for some unknown reason. I didn't know it was there until my server was hacked by a bot. I'd suggest adding one step of checking the /home directory or other places to make sure no 'unknown' accounts have been set up.

</comment><comment><USER78> Thanks for this article - very clear, well-motivated, and concise. I'm saving this for myself and others.

</comment><comment><USER79> I guess I'm a pretty big noob, but why do people recommend so strongly on password protecting your private key? Losing it pretty much dooms you whether or not it's password protected. It might get you a few hours or so to react and invalidate the public key, I guess...

<comment><USER80> If you've got a good password on your key, then nobody will be able to use it in years. It most certainly helps.

<comment><USER79> Years? How long a password would you need to make GPU cracking take years?

<comment><USER80> A very short one. Reasonable 10+ character passwords should remain out of reach for years.

Also, AFAIK there doesn't currently exist any very effective GPU cracking software for SSH passphrases.

<comment><USER81> I went to lookup the algorithm GPG uses to encrypt private keys, to help answer the original question, but couldn't seem to find that information anywhere. Do you know what it is?

<comment><USER79> According to a quick stack exchange search, OpenSSL uses 3DES for encryption of private keys.

</comment></comment></comment></comment></comment></comment><comment><USER82> This is very similar to my "How To Set Up Your Linode For Maximum Awesomeness" guide:

[http://feross.org/how-to-setup-your-linode/](http://feross.org/how-to-setup- your-linode/)

</comment><comment><USER83> If you open up access from/to port 80 or 443, you also open up access to all trojans/spyware/telemetry/auto-update created in the last ten years. You'll want to limit access per user and process.

</comment><comment><USER84> It might nice if there were some cloud vendor specific addendums. For example on rackspace you almost always want to install the monitoring daemon (it's actually fairly decent and small foot print).

</comment><comment><USER85> Anyone remember Bastille Linux? [https://help.ubuntu.com/community/BastilleLinux](https://help.ubuntu.com/community/BastilleLinux)

</comment><comment><USER86> Instead of using unattended-upgrade, I prefer to subscribe to mailinglists and see when there are new securtiy updates.

One could combine that with something like rundeck where you run apt-get upgrade.

</comment><comment><USER87> Why not make a _certified_ secured best practice 99% covered snapshot and share it as part of the one-click installs that most VPS providers offer nowadays?

</comment><comment><USER88> I think 2FA is generally bad practice and quite sad it is ubiquitous in e.g. banking and people try to shove it everywhere. It is analogous to password rules, 8-14 characters, numbers, capital letters and other signs. Yet it is very rare you can use a 40+ character passphrase. It gives a false sense of added security, while being annoying at the same time imo. It is very common, for me at least, not to have access to my phone all the time, because I left it at home, in the car etc. Not to mention if you lose it (or someone steals it) you have a huge pita to deal with.

<comment><USER11> 2FA doesn't have to be annoying. Take a look at Yubikey devices as an example of how to do this right. The reality is that it is actually really useful at preventing some common attack vectors: password reuse, keyloggers, etc.

It's even better if you're using a hardware dongle that supports U2F (or can be used as a smartcard for SSH), because that can even prevent active MITM attacks.

<comment><USER88> no, my point is exactly that the 2 in 2FA is inherently annoying, because you need to have physical access to 2 different devices at the same time.

How does it prevent password reuse? You can use the same (weak) password to lock your phone and login to your banking account (which is again, a false security). However it could be easily circumvented by random generating secure passwords for users (which needs clever advertising like 2FA, because they prefer convenience otherwise). In this case your phone is a single point of failure. You could even argue it increases the attack surface.

<comment><USER89> > How does it prevent password reuse?

It does not _prevent_ password reuse, it mitigates the risks of password reuse in that it adds the requirement of having physical access to a device, which is a show-stopper for _most_ attackers.

If you're using a password manager with sufficiently complex passphrases, the biggest remaining risk factor are targeted malware attacks (something like a keylogger), which is something that typical SMS- or TOTP-App-based 2FA implementions won't help you with, fair enough. Implementations where certain security-sensitive activities require separate confirmation and where the details are transmitted through a separate channel would mitigate this attack to a certain degree as well. As an example, some banks in Europe provide their customers with card readers with a PIN pad that shows transaction details on a separate display. Banks routinely include transaction details in SMS-based TAN mechanisms, which works as well, but is obviously not quite as good.

> You could even argue it increases the attack surface.

How?

<comment><USER88> Meant in the general sense. More complexity, more opportunity for attacks and/or implementation bugs. For starters you have a phone number associated with an account already. I would wager losing your phone is nearly impossible to prevent, while picking your passphrase is up to you. Losing your phone could alone compromise your security, but the very least SMS leaks the info where you bank.

I'm aware I am a minority with this opinion, but I would be really grateful if I could at least opt out from phone based 2FA.

</comment></comment></comment></comment></comment><comment><USER90> Is there a similar guide for Windows servers out there?

<comment><USER91> [http://decentsecurity.com/](http://decentsecurity.com/)?

</comment></comment><comment><USER92> It would be useful to discuss what prevents the server from being rooted without a trace during the 10 minutes it takes to execute these steps.

<comment><USER75> Production boxes are not allowed to be plugged into bare internet or DMZ unless they were seasoned and tested on the LAN first, and the LAN allows no external traffic in (stateful firewall). If people on your own LAN are trying to pown you, you got bigger problems to solve before installing another box.

In the old days this was manually moving ethernet cables, now a days this is changing which VLAN the virtual image talks to or if you use something like openstack that implements its own firewall at the virtualization level you allow no external traffic in until the config and testing is done.

Also you need to verify your install media is not powned, which means you need access to the md5sum of the media (and how do you know someone didn't MITM the correct md5sum?) and you need to verify your md5sum program isn't powned which means you need to verify your verification strategy isn't powned which means this gets recursive real quick.

</comment></comment><comment><USER93> Hmmm ... why does root need a password? `sudo su`

<comment><USER94> _> You'll only need this root password if you lose your sudo password._

Also, good to have in case sudoers file is corrupted, _e.g._ via bad edit, typo, _etc._.

<comment><USER95>
        cd /etc
        git init
        git add *
        git commit -am "before i screwed up"

</comment></comment></comment><comment><USER96> 1.

        useradd -m deploy

2\. "PasswordAuthentication no" probably won't work as you expect if UsePAM is on.

</comment><comment><USER97> May I suggest Ansible or whatever other provisioning tool, you can subtract 9+ minutes from the title...

<comment><USER98> please don't.. provision once and snapshot, and deploy the snapshot

</comment></comment><comment><USER99> What was wrong with 5 minutes? :-)

<comment><AUTHOR> Nothing! (Except I doubt I can manually get it all done in 5 minutes =) ) Thanks for your great article.

We just ended up adapting your approach with a couple modifications (like 2FA) and extending it to be more of a primer and explain the steps a bit more so that the younger engineers understood what each step performed was doing. I found myself pointing them to your article, but then having to explain what was being performed and it's purpose (not a bad thing - just different audience).

As has been mentioned, in the real world, an Ansible Playbook should be performing these, but teach a man to fish, etc...

<comment><USER99> I love the article, and thanks for the credit.

</comment></comment></comment><comment><USER100> Nice guide, better comments, leaving this here for later reference.

</comment><comment><USER98> sure makes me glad all that (and so much more) happens in the first negative 10 minutes on any server I deploy.

If you are doing this after your server has launched you are doing it wrong.

</comment><comment><USER101> Pretty useful, thanks

</comment><comment><USER102> No production server should ever be manually configured.

<comment><USER103> How should this be done by, say, a small team of three with no SysAdmin?

<comment><USER95> Take a day off and get familiar with ansible, it will save you tons of time in the long run and simplify your dev/staging/prod environment for years to come

</comment></comment></comment><comment><USER104> Don't just change SSH key requirements, also change SSH port.

Port 22 is possibly the most heavily scanned port around.

</comment><comment><USER105> _Sigh_ ... "principal of least privilege"

</comment><comment><USER106> I'd been using this article for a couple years when I was a Linux server neophyte and now some of these things seem obvious to me. A good article for total noobs.

<comment><USER106> Just realized I'm getting downvoted because I thought this was the first 5 minutes article. Whatever!</comment></comment>


What SQL Analysts Need to Know About Python (2016) - <AUTHOR> https://segment.com/blog/sql-vs-python/ <comment><USER2> I'm worried about the inverse - what Python data analysts should know about SQL. Because I've met tons of analysts who wouldn't be able to even run a basic select.

I've seen tons of (often non-reproducible) code written in place of a simple SQL query. I really wish bootcamps and other learning platforms focused on SQL a bit more.

(I am a Python data analyst who properly learned SQL only after several years in the industry.)

<comment><USER3> Or worse, they use cursors. Cursors should never be used in SQL. Ever. That's my philosophy.

<comment><USER4> What is a better approach? I have Python that directly connects to an Oracle database, and the Oracle blog tutorial[0] for using their Python package always uses a cursor.

[0] [https://blogs.oracle.com/oraclemagazine/perform-basic- crud-o...](https://blogs.oracle.com/oraclemagazine/perform-basic-crud- operations-with-cx-oracle-part-2)

<comment><USER5> They talk about client-side cursors, these are part of the Python DB API. What you shouldn't use is server-side cursor, which you create with DECLARE in PosgreSQL. These keep their state on the server and are intended for optimizations for special cases like streaming data processing or realtime updates. Basically, for deep internals of realtime systems, and not regular queries.

<comment><USER6> _What you shouldn 't use is server-side cursor, which you create with DECLARE in PosgreSQL._

That entirely depends on the DB. Oracle for example always executes with a server-side cursor; all declaring it does is give you a handle to what it was doing anyway.

<comment><USER7> The problem isn't the cursor itself, but that it is usually a symptom of procedural thinking vs set based thinking. In an RDBMS it's typically far faster to puzzle out the joins, CTE's, and set based expressions and functions to use to winnow down a dataset vs a cursor based procedural logic on a row by row basis.

<comment><USER8> Thank you for clarifying that; I was wondering where the concern was coming from.

Without disagreeing with any of the above, one important consideration is what you're going to do with the query. If all you want to know is a column's mean or some other simplified statistical value, there's really no sense in pulling all the data into Python just to calculate it. Do it inside the DB itself with SQL.

On the other hand, if you need that data to do other work (i.e. populate the table in a webpage, or generate a new descriptive data set or whatever), then the trade-off for pulling it into Python/pandas and running a mean in addition to the other work becomes much smaller.

My approach is usually to do as much data filtering and parsing as possible inside SQL, but things like complex parsing and string manipulation (especially!) I'll do with Python. I can do some simple string work in SQL, but I can almost always do it faster and cleaner in Python.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> A few years ago I joined a Rails shop, and one thing that always struck me was how many of the engineers didn't know SQL. Most of them had learned to code on Rails, and had always had SQL abstracted away via ActiveRecord.

I know this is not the point of this article, but as data analyst/scientist roles continue to climb in popularity, I'm curious if there won't be a similar trend with Python.

<comment><USER10> Isn't that one of the selling points of Rails? Time to market is king. Optimize your SQL query performance after you've released and proven that it's an actual bottleneck. Why spend more time and money on an optimized product that might never see the light of day?

<comment><USER11> There's a difference between basic optimization, and needing to refactor most of your business logic because your assumptions about databases are boneheaded.

In general, it doesn't matter what you're doing, your basic design patterns need to fit around how a database works. If you don't know this, you'll hit scalability issues far too soon, and it'll take too long to fix them.

> Optimize your SQL query performance after you've released and proven that > it's an actual bottleneck.

I've seen one project fail because the design patterns around using the ORM were incorrect. Then I joined another project where the bottleneck (from incorrect use of the ORM) was so bad the product couldn't scale beyond being a demo. It took 2 months to refactor, all because one of the programmers used an ORM incorrectly to save a few hours at the beginning.

</comment></comment></comment><comment><USER12> I've found that SQL and data frames are pretty exceptional together in combination. I'm a big fan of a Python module called PandaSQL [https://pypi.org/project/pandasql](https://pypi.org/project/pandasql). What makes this work so well (for me, at least) is that I can combine pandas operations that transform data frames with SQL operations that transform data frames in the same pipeline, and if I _really_ need to, I can just break it all apart with python and reassemble it back into a dataframe later. I don't need to recreate Boolean algebra with complex and potentially buggy dataframe operations[1], and I don't need to recreate loops, conditionals, stats operations and so forth with complex and potentially buggy SQL.

For clustered work, I've found that Spark sql data frames give a lot of the same functionality (not quite all, though I _think_ that's because there are some pandas operations that require a full in-memory dataframe and don't lend themselves to distributed solutions).

[1] there have been so many attempts to replace SQL with a different relational-like language. the end result is a new syntax that doesn't work if you try to pull your queries out and run them against a database independently. I'm going on a tangent in a footnote here, but I remember reading "your data will outlast your application." I personally strive for a usable database outside the context of the application it was originally created to support. Trust me, eventually someone will want a set of reports that would (in many cases) be fair easier to write as queries if you'd made sure your back end database was a properly designed relational database.

<comment><USER13> PandaSQL was last updated 3 years ago. Do you know any other alternatives?

<comment><USER12> No, though it has been a while, and your question may prompt me to take a look around the landscape. Not that I have had a problem with pandasql, it does work nicely.

This is one of those technologies that I was hoping would make its way into the framework, kind of how connection pooling for databases was once an external module but is now often built into the various web frameworks. DataFrames with common columns are such a natural match to relational tables that it seems that a way to call SQL would be (ok, in my opinion, should be) part of pandas (I think that this is the case for R data frames).

I also just really enjoy writing SQL. To some extent, this is a personal preference - some people just mentally line up with certain ways of thinking. There are things that are clearly better to do in python, things that are clearly better to do in SQL, and then a grey area. I personally lean much farther to the SQL side of that grey area, but there's certainly nothing incorrect about going the other direction.

</comment></comment></comment><comment><USER14> Python + SQL seems like a good match for many analysis problems, but Excel + SQL is not bad either. I like the ability to combine complex SQL views or SQL functions with Excel pivot functionality, querying the database directly from Excel

<comment><USER15> Maybe I've not fully understood pivot tables (quite possible), but this example[0] seems to support my understanding... aren't pivot tables basically just aggregate functions such as `sum` and `avg` applied to a window of the table data (pivoted by rows -> columns)?

Some of the executives I work with like to relate all their work in Excel and they just love pivot tables. I showed one of them the output of a table of data on web page backed by a database and they asked if I could export the data to Excel to make a pivot table that would then be displayed on the web page. Of course, I implemented their pivot table as queries against the database and created a new view to display it.

I've never understood why I would need Excel to make Pivot tables when I already have SQL.

[0] [https://blogs.msdn.microsoft.com/spike/2009/03/03/pivot- tabl...](https://blogs.msdn.microsoft.com/spike/2009/03/03/pivot-tables-in- sql-server-a-simple-sample/)

<comment><USER8> You're absolutely right. As someone coming into Python and SQL from a purely Excel background, the first realization I had was that I had been, in essence, programming Excel. Not in a rational VBA kind of way, but in how I was linking all of my cells (and sheets and books) and then using pivot tables, etc., to extend Excel beyond a single column/row perspective. Microsoft PowerPivot (or whatever it used to be called) was my gateway drug into the world of SQL.

I can generate similar utilities to Excel/PivotTables using Plotly Dash or Pandas or even DataTables, but even after a few years of learning and practising it would take me less than 1/10th the time to generate an equivalent excel pivot and chart and have it displayed in front of a group.

I'm not saying it's better, just that those execs are so comfortable with that visual-feedback excel approach that it'll be an uphill battle to convert them to a programmatic one. Pivot tables provide very rapid means of filtering, modifying output, and aggregating information than is otherwise possible inside Excel. To those coming from a two dimensional excel spreadsheet world, discovering pivot tables is like viewing the world in 3D.

I don't see why you would go to Excel pivot tables from SQL; you already have a more powerful tool at your disposal, if you're comfortable with it. Going from Excel to SQL? That hurdle is a bit higher.

<comment><USER16> Well, because Excel is quick. I know R and SQL, and I still use Excel a lot.

If I want a quick overview of a problem, query db from Excel -> PowerQuery -> PivotTables. It's easy to use, it's very easy to find solutions in google for PowerQuery, and it just feels slick.

If I need in depth analysis then is when I switch to programming.

</comment></comment></comment></comment><comment><USER17> Tangential question: I'm curious how many people (here on HN, or in general) learned SQL before they learned more traditional programming (e.g. Python, Java, C)? I learned traditional programming (through college) and only stumbled upon SQL years later (someone left a "How to use Microsft Access" book around at work).

I absolutely love SQL, and am in the middle of writing a book on how to use just SQLite for large, complicated data work. My target audience is programmers, but also non-programmers, because I think SQL is vastly easier to learn than something like Python. The tradeoff of course is that SQL is much more limited a framework, but it's more than enough to do data work (all the stuff that isn't visualization or complex stats).

When I lived around Palo Alto, I met a few recent grads who didn't study STEM in college, but ended up working for tech firms in content and analytics roles. They'd know Excel and Tableau, but through co-workers, would pick up SQL to make their data work easier. But they hadn't yet had learned general programming, or tried Python/Javascript.

But I can't imagine many other professional or academic career paths in which someone who ends up learning and using both Python and SQL started out with SQL, then learned Python (i.e. general programming). Would would they have done with SQL in school, or as a hobby?

<comment><USER18> I'm a data engineer and I learned SQL before I learned python. I started out as a marketing specialist (mostly working in Google AdWords), then I graduated up to BI Analyst, and eventually to data engineer. I've picked up 100% of my python in my current job.

<comment><USER17> What did you think of Python, and was it difficult to grasp the differences between the two languages/paradigms? I ask because, for myself, any general language I pick up (Ruby,Python,JS,R), I have a good idea of underlying concepts like memory pointers and garbage collection, even if it's all abstracted by the language. With SQL, I have an incredible ignorance of the most basic programmatic concepts, like how to define variables or custom functions. The deepest underlying concept I have of SQL is that it has a query planner that does all the thinking for me.

<comment><USER18> Yeah, the learning curve was pretty steep for me. I've been learning some C# for my job, and it's been a lot easier after learning basic Python. SQL is kind of weird in the sense that starting out is very very very easy, but mastering it is incredibly hard. I've been writing SQL everyday for about 3 years and I'm still blown away with what the SQL experts at my company can do with the language. I see presentations at conferences that make me feel like an absolute beginner.

</comment></comment></comment></comment><comment><USER3> SQL+Python is extremely powerful, and the author makes some good points (multivariate regression should be done in python and not SQL for example), but the query example in the blog is not a good one. Every modern DB has aggregate and statistical functions like ntile, percent_Rank, median, min, max, etc. I'd honestly rather run these functions against the database than do it in python. Especially if you're working with billion+ row datasets. In fact if all I'm doing is these kinds of statisical/aggregate functions then I definitely wouldn't use python at all.

<comment><USER8> I wonder if people do timings for the variances between a function in the DB vs Python? Unless your queries are so small (like individual records) that a round-trip to the DB is impacted by network lag, I find it's almost always faster to ask the DB to do as much of the lifting as possible.

The downside, of course, is that your code potentially becomes unreadable, harder to migrate between database backends, and more difficult to debug.

<comment><USER19> Pushing computation back into the database has been a major trend in the last few years. Tools like DBT have made that transition easy in terms of clean code and testable SQL.

</comment></comment></comment><comment><USER8> One thing that hasn't been mentioned yet, that I found when I started using Python and SQL together, is the importance of well-considered indices. ORM's aren't always too clever when they generate an index, and often (for repeated queries) you can dramatically cut down the processing time by thoughtfully generating an index.

If I know I'm going to be asking the database a certain question a bunch of times I'll even generate a temporary index to speed up my analysis, and then delete it when I'm done. No sense running a SQL query for an hour if I can cut it down to 5 minutes with an index on the target column.

EDIT: I know this only applies to PostgreSQL, and there are numerous alternatives, but a big shout-out to pghero which helped me identify a bunch of duplicate and missing indexes in one of my databases, saving both time and hard drive space. Incidentally, it also sent me down the road to learn (but definitely not master) index optimization. [https://github.com/ankane/pghero](https://github.com/ankane/pghero)

<comment><USER20> I recommend "Use the Index, Luke"[1] for people who are relatively new to SQL performance issues and want to learn how to make their queries fast.

[1] [https://use-the-index-luke.com/](https://use-the-index-luke.com/)

</comment></comment><comment><USER21> uMatrix prevents it from loading.

Against my better judgement, I turned off uMatrix and loaded the page. Then had a look at uMatrix.

Wow, I think we have a winner for a site with the most scripts, most 3rd party domains, and the sheer number of XHRs.

Sorry for this being off topic.

<comment><USER22> Do you have a sane setup you can share with uMatrix? Last time I looked at it, I was a bit overwhelmed and shelved it for later.

<comment><USER23> So, I don't know about setting rules beforehand that work very well universally, but what I personally think is the better method is to just slowly build up your ruleset as you use it every day. I don't think lots of people know how to do this so here it is:

1) Upon using a website you frequent, enable one root at a time until it functions properly. (or you can do the more granular per grid block enable) 2) Click umatrix, then click at the top where it says "uMatrix $ver" which will take you to the dashboard. 3) In the dashboard click on "my rules" 4) on the right click the "commit" button.

What this does is commit your changes to the permanent ruleset and websites you frequent will start to "just work". A word of warning, if you blindly commit after browsing for a while when you may have done some temporary allows on random websites, those will be commited too. So I suggest either reviewing the commits first (always a good idea) and deselecting the ones you don't want, or having a new session for each of your frequented websites that you then commit from.

I have been planning to write up a tutorial on things like this for family and friends, maybe I'll post it to show hn.

<comment><USER21> When I visit a site that doesn't work. Usually I carefully whitelist things that I am willing to whitelist until it does work. If I can't get it working, then in most cases, I don't consider the content of that site to be valuable enough to whitelist things that seem sketchy.

In the case of this site, uMatrix just flat out blocked the entire site. it's not that the side page didn't load, or that nothing happened because of, say, missing javascript. I got a huge warning page in FireFox that uMatrix was flat out blocking it. Probably because the primary site is already blacklisted, maybe?

<comment><USER23> Yes some sites will be completely blacklisted, and you have to manually whitelist them if you trust them enough to try them. I usually just skip them if they do that though.

</comment></comment></comment></comment></comment><comment><USER24> I had a problem where there was some fixed width data in a sql database - basically someone put mainframe data in a database.

There were 3 fields that had multiple entries in them, fixed width delimited. I had to split the fields by width and re-combine them, then also recombine them with another set of data with weird delimiters and rules.

It took a day and half (not full time) to figure it out in python. I can't even imagine tackling the problem in a non-repl language.

<comment><USER25> Turns out a good database is _really_ good at data munging. A solution for postgresql might look something like this:

        CREATE VIEW my_table_improved AS
        SELECT SUBSTRING(the_column, 0, 8) as col1,
               SUBSTRING(the_column, 8, 8) as col2,
               SUBSTRING(the_column, 16, 8) as col3
        FROM my_table

After which you can query my_table_improved as a normal table. col1, col2, and col3 contain the data split out from the_column. In practice you'd probably also want to do some type conversion, e.g. if col1 is supposed to be an integer you can simply update the view to select `CAST(SUBSTRING(the_column, 0, 8) as INTEGER) as col1` instead. In production use you might find this to be slow at which point you will want to create indexes for your new columns. Something like this (adjusted to which queries you're running of course) should work:

        CREATE INDEX ON my_table ((SUBSTRING(the_column, 0, 8)));

Of course this is a lot of work if you're unfamiliar with SQL, and above examples aren't quite complete yet for your use case, but it should get you an idea of how SQL is the exact right tool for the job here. which is somewhat the point of many commenters here: get yourself familiar with SQL and save yourself a metric tonne of work in the future.

An alternate approach might be to write a query that migrates the fixed-width format to a format where each entry is in their own column. The ease of this mostly depends on if applications depend on that column being in that format.

side note: above sql code is untested but should be roughly correct.

<comment><USER24> If it was just one column, that would be easy enough. Or if one object was on one row, it would be easy enough. In my case, up to 6 rows could be required to represent one object, and I had to slice 3 columns with an arbitrary number of slices.

Oh, and there were two types of sub-record per object, and they had to be processed in database order.

<comment><USER25> Well, as long as you can build a query to get the data in the right format (which you almost inevitably can) you can make a view out of it. But honestly the true solution here would be to migrate away from such a brain damaged format.

</comment></comment></comment></comment><comment><USER26> If you are working a lot with databases, and would like to automate some of the workflow in python, I'd highly recommend Pony ORM, which really feels like LINQ for python, and feels very close to the original SQL.

<comment><USER27> Pony is the LAST product I'd ever recommend to anyone, ever. It's probably the worst open source software I've ever used.

That being said, thanks to Pony I was called in to fix a shitshow and it likely earned me an additional 25% in consulting fees... so maybe I do like it in that regard.

But in all seriousness, stay far away.

<comment><USER26> I suggest pony because it would be quite good for people who have a deep understanding of SQL, because the translation is quite nice.

The alternative - SA is really overkill for most things.

<comment><USER27> SQLAlchemy is a buzzkill but it's not as heinous as Pony which is a straight up ticking time bomb in your project. Read the source code for 2 minutes and it should be enough to shy you away.

As far as lightweight Python ORMs, I really enjoy Peewee and it's usually my first choice.

<comment><USER26> I tried PeeWee, but it does not have support for SQLServer, which I need to work with on a daily basis.

</comment></comment></comment></comment></comment><comment><USER28> Is 'analyst' an actual job title or just one skill required in a job? What does an 'analyst' do?

<comment><USER19> This article is talking about a role typically called "data analyst". They help product managers, marketing, operations, etc by running reports, building dashboards, (business intelligence), building data models, running an exploratory analysis, analyzing A/B tests (product analytics) or things like building marketing attribution models or timeseries forecasting.

It's the job that's 80% of what a data scientist does but without Python or R. But mostly it depends because no one can decide what these roles should be called or what their responsibilities should be.

<comment><USER29> A Data Analyst is a Data Scientist who can't prove Bayes Theorem.

</comment></comment></comment><comment><USER30> What, that psycopg2 is busted?</comment>


Microsoft Just Won the Cloud Wars - <AUTHOR> https://www.inc.com/heather-wilde/microsoft-just-won-cloud-wars.html?lipi=urn%3Ali%3Apage%3Ad_flagship3_feed%3Bww%2BttRTOSLaYPVvYaWMZvA%3D%3D <comment><USER2> Rather click-baity title -- Microsoft bought a company which specializes in optimizing your cloud spend. So, if you're a company who leaps before you look when provisioning resources in the cloud, that might be helpful. Otherwise, you're thoughtful and manage your spend like a rational business should do -- situation normal.

</comment><comment><AUTHOR> yea right.

Fed up with sky-high cloud service provider bills? Microsoft is too...</comment>