Ask HN: How do you install developer tools on a fresh system? - <AUTHOR> I used to have a bash script I would run to install everything (libs/tools) you're average web developer would need (node, sublime, google chrome, , python libs etc etc). Unfortunately I lost it after some time. Does anyone use anything similar? <comment><USER2> I don't do fresh installs very often, but when I do, it's generally because I want it to be a genuine fresh install. As such, I don't install a tool until I need it. I find that tools I once thought I absolutely needed are not tools or libraries that I use anymore.

I have three exceptions to this: Vim (my editor of choice), dotfiles (which I store in a git repository and put in place using stow, installed via a simple bash script), and Vagrant, so I can do development testing against a VM.

As Docker matures, I may use it in place of Vagrant, but it's not ready to fill the same role quite yet.

<comment><USER3> Have you tried the beta for OSX or Windows yet? I haven't needed vagrant in the slightest since it was released.

<comment><USER2> If they can get xhyve based VMs stable when co-existing with VirtualBox, then yeah, that will go a long way towards replacing Vagrant for me.

That said, I'm not a huge fan of Docker's feature churn; having to re-install, reconfigure, and re-learn tooling that is central to my workflow every other month gets old pretty quickly.

<comment><USER4> That's currently blocker #1 for us when it comes to the new Docker for Mac/Win. VirtualBox works as a nice abstraction layer on top of 3 OSes (Mac/Win/Linux) which simplifies things with Vagrant. Unfortunately it seems VirtualBox has been in maintenance mode as of lately so that might have contributed to Docker's decision to go with HyperV/xhyve.

</comment></comment></comment></comment><comment><AUTHOR> Oh god I used "you're" instead of "your" and can't edit it.

<comment><USER5> The horror!

<comment><AUTHOR> I have become what I hate :(

<comment> aug-riedinger You only hate not being able to edit your comment on HN. So do I.

</comment></comment></comment></comment><comment><USER6> I recently reinstalled my NixOS laptop. I just installed the distribution, added my SSH keys, cloned a repository, made a handful of symlinks, and then told NixOS to set everything up.

It's actually a collaborative repository, so that both of us in our company can improve the computer configuration, install new tools or language runtimes, etc etc.

The shared configuration has stuff like: our user accounts and public SSH keys; local mailer setup; firewall; conf for X/bash/emacs/ratpoison/tmux; list of installed packages (including Chromium, mplayer, nethack, etc); fonts and keymaps; various services (nssmdns, atd, redis, ipfs, tor, docker, ssh agent, etc); some cron jobs; a few custom package definitions; and some other stuff.

In Emacs, I use "use-package" for all external package requirements so that when I start the editor it installs all missing packages from ELPA/MELPA.

Aside from dealing with the binary WiFi blob this Dell computer demands, reinstalling was a pleasure.

<comment><USER7> You can use nixOS as a development machine? How did you install a window manager or desktop environment?

<comment><USER6> Sure! It's a full distribution. You install window managers the same way as any other package. The NixOS manual is quite comprehensive.

</comment></comment></comment><comment><USER8> I have my dotfiles for that. Split into different categories (brew, npm, pip) together with all the config files I need. brew and brew cask (with brew- bundle [0] for Brewfile support) take care of getting all libraries and applications onto the system.

For the development itself I'm either shipping my entire config (.vimrc for example) or use systems like spacemacs, sublimious or proton that only need 1 single file to re-install the entire state of the editor.

The install script itself [1] is then symlinking everyhing into place and executes stuff like pip install -r ~/.dotfiles/pip/packages.txt.

It takes a bit of effort to keep everything up to date but I'm never worried of loosing my "machine state". If I go to a new machine all I have to do is clone my dotfiles, execute install.sh and I have everything I need.

On servers I am using saltstack [2], a tool like puppet, ansible and friends, to ensure my machines are in the exact state I want them to be. I'm usually using the serverless version and push my states over SSH to them.

[0]: [https://github.com/Homebrew/homebrew- bundle](https://github.com/Homebrew/homebrew-bundle)

[1]: [https://github.com/dvcrn/dotfiles/blob/master/install.sh](https://github.com/dvcrn/dotfiles/blob/master/install.sh)

[2]: [https://saltstack.com](https://saltstack.com)

<comment><USER9> I have no strong opinion here, but I am curious to hear yours. What were the discriminators in choosing saltstack over the alternatives?

My head spins with these tools and every time I pick one I seem to eventually run into a road block that is a no-go. The most recent effort was ansible and the no-go was its strict dependency on python2.7.

<comment><USER8> I came from puppet and salt felt a good amount "lighter". I'm mostly a python developer and salt states written in pure yaml with jinja2 templates, or alternatively directly in raw python for more complex stuff feel like home.

The yaml makes it very easy to understand even if I come back to a state after months. It's just a list of things that should happen with a few template tags sprinkled in.

Not saying salt is better than puppet or friends. It's purely based on preference. I can't say anything to chef or ansible since I never tried them. Salt has a crazy active community and while there are things I don't like about it like the "name" of a state, it's still doing it's job just fine so I sticked with it.

<comment><USER10> ha, i moved from chef to ansible for much of the same reasons. at the end of the day, it doesn't matter.

(however, from my experience salt and ansible stay readable because they're yaml and not arbitrary code/DSL, whereas chef "recipes" are ruby and usually devolve into complex programs if you're not careful)

</comment></comment></comment></comment><comment><USER11> If you consistently use the same Linux distribution, consider building metapackages for that distribution.

I created a set of Debian packages that depend on suites of packages I need. I download and install "josh-apt-source", which installs the source in /etc/apt/sources.list.d and the key in /etc/apt/trusted.gpg.d/ , then "apt update" and "apt install josh-core josh-dev josh-gui ...". That same source package also builds configuration packages like "josh-config-sudoers".

<comment><USER12> Can I use that with external PPAs?

Or I have to install the PPAs before I can use the metapackage (which kind of defeats the idea of using a metapackage instead of a script)?

<comment><USER13> While you could have third party repos installed via apt packages the problem is you can't trigger an apt update after they install and before the things that depend on them install easily.

</comment></comment></comment><comment><USER14> The question asks only about *nix systems, I assume, but it worth mentioning that there is a great tool for Windows too, just in case if someone needs it - [https://chocolatey.org/](https://chocolatey.org/)

<comment><USER15> Automating from-scratch setup with chocolatey: [http://www.boxstarter.org/](http://www.boxstarter.org/)

<comment><USER16> Looks cool but I would be uncomfortable using this over HTTP, not HTTPS. Don't really want to risk a MITM when installing software on your machine (and giving it UAC _and_ your password).

</comment></comment></comment><comment><USER17> Use a configuration management tool (I picked [https://saltstack.com/](https://saltstack.com/) , mostly because of the docs & community support) but there's lots to choose from - Chef, Puppet, Ansible, and so on.

There's a learning curve, and plenty of 'where did my afternoon go?' rabbit holes you can lose yourself in. But the upside is that you can have consistent, repeatable, and rapid builds, with modularity as a bonus.

Don't be afraid with any of these kinds of tools to brute force complex components if you're in a hurry - ie. ignore the pure / idiomatic way, and use the tool's primitives to dump a shell script on a remote box and then run it.

<comment><USER18> I've found with Ansible at least, I was initially tempted to make large complicated roles for things like "application server" or "development desktop" but what ended up working much better was very granular roles such as "nginx server" and "emacs" (often just a single task such as "yum: name=nginx state=installed") that can be combined in playbooks. This makes it easier to avoid duplicating tasks in different roles, or having a lot of complex conditional cases in your roles.

<comment><USER19> This is the "roles and profiles pattern" in puppet, it generally makes sense.

Also, +1 for using CM to set your workstation back up, I really need to get around to writing some Puppet manifests to do that for mine

</comment></comment></comment><comment><USER20> I'm using a shell script together with the nix package manager for that. The shell script just ensures that all packages are there (e.g. doing `nix-env -i fpp wget iterm2 jekyll ghc ruby nodejs composer php`). I can pin the version of all packages by configuring `NIX_PATH` to point to a specific `nixpkgs` (the package repository) commit. So that all people have exact the same versions of everything.

Package customizations like a .vimrc is also handled by nix (I recently blogged about how I do this: [https://www.mpscholten.de/nixos/2016/05/26/sharing- configura...](https://www.mpscholten.de/nixos/2016/05/26/sharing- configuration-across-systems-with-nix.html)).

The shell scripts together with the package customizations (e.g. my custom vimrc) are managed by git.

<comment><USER21> I've recently learned you can put all the packages in one custom Nix expression in ~/.nixpkgs/config.nix, like below, then load (and reload/update/change) it with one `nix-env -i all`, or faster with: `nix-env -iA nixos.all`:

        # ~/.nixpkgs/config.nix
        {
          packageOverrides = defaultPkgs: with defaultPkgs; {
            # To install below "pseudo-package", run:
            #  $ nix-env -i all
            # or:
            #  $ nix-env -iA nixos.all
            all = with pkgs; buildEnv {
              name = "all";
              paths = [
                fpp wget iterm2 jekyll
                ghc ruby nodejs composer php
              ];
            };
          };
        }

and then keep only this one file in git. (Though still working on how to possibly also keep .inputrc, .bashrc, .profile, etc. in it.)

</comment></comment><comment><USER22> I've found ansible to be okay at setting up my environment. I'm able to configure everything from my zsh themes, terminal font size, window manager shortcuts, thunderbird logins, and so forth. The playbook takes about 30 minutes to run and after that I have almost everything ready.

Unfortunately I don't have a public GH repo I can point at as I don't want to expose everything I use to the internet. However the principle is the same as provisioning servers with ansible.

The only thing different I do is I use GPG keys to decrypt and untar things like thunderbird profiles rather than using Ansible vault. I restore GPG keys + SHH keys from offline, encrypted USB backups.

<comment><USER23> Have you considered using Gitlab? I really like GitHub, but can't justify the prices right now (just starting out in my career). But while Gitlab isn't as popular, or maybe quite as polished (though it's getting there fast), it does have free private repos. I've used this to store private data like this.

Check it out if you haven't already.

<comment><USER22> I use both GL and GH regularly (I have private repo at GH as well). I'm moving towards GL slowly but my experience is that it is much slower than GH atm.

</comment></comment></comment><comment><USER24> That's a very good reason to only install apps with a real package installer.

On OSX, this is a no-brainer with brew[1] and brew cask[2]

# On my old mac

      $ brew list
      $ brew cask list

=> then I save relevant parts for future references

      brew install npm
      brew install zsh
      brew cask install sublime-text
      brew cask install google-chrome
      brew cask install intellij-idea

[1] [http://brew.sh/](http://brew.sh/) [2] [https://caskroom.github.io/](https://caskroom.github.io/)

<comment><USER25> >brew list

Protip: "brew list" will list all installed packages, including dependencies, which you might not want. What you probably want is "brew leaves", where it lists all installed packages _that are not dependencies of another installed package_.

This makes a difference in cases where a dependency is no longer needed in the latest version.

On a related note, why does the majority of package managers make the common and simple task of "list all manually installed packages" so incredibly hard?

For a fun brain twister, try to list all the manually installed packages on your system by just reading the man pages and no internet. Ubuntu is nightmare mode for this challenge.

<comment><USER26> For Ubuntu, look no further: [http://askubuntu.com/a/492343/145754](http://askubuntu.com/a/492343/145754)

</comment></comment></comment><comment><USER27> I use a shell script for a new debian[0] installation and also have other scripts for kubuntu[1], opensuse[2] and other software installations. I store my dotfiles[3] and other useful scripts that I can customize for each development environment. Hope that helps!

[0] [https://github.com/svaksha/yaksha/blob/master/yksh/apt- debia...](https://github.com/svaksha/yaksha/blob/master/yksh/apt-debian.sh)

[1] [https://github.com/svaksha/yaksha#2-folders](https://github.com/svaksha/yaksha#2-folders)

[2] [https://github.com/svaksha/yaksha/tree/master/yksh](https://github.com/svaksha/yaksha/tree/master/yksh)

[3] [https://github.com/svaksha/yaksha/tree/master/home](https://github.com/svaksha/yaksha/tree/master/home)

<comment><USER28> Just wanted to compliment you on the documentation in those scripts. I'm that documentation stickler guy at work, so seeing it in the wild (especially for personal files) makes my day.

<comment><USER27> Thanks :)

</comment></comment></comment><comment><USER29> I just install stuff when I run into something I need but don't have. Keeps my system slim.

</comment><comment><USER30> Setting up a new system is where NixOS really shines. Once you have one system working it is trivial to duplicate it on new metal.

1\. Install NixOS

2\. Copy configuration.nix*

3\. Copy dotfiles

4\. # nixos-rebuild switch

5\. Enjoy your old setup on new hardware--no secret sauce needed!

*A hardware-configuration.nix should have been generated by the installer. By default this is sourced by configuration.nix, in which case configuration.nix shouldn't need editing.

<comment><USER31> I've been interested in Nix and Guix for some time; would you be able to comment on the two?

<comment><USER30> Probably the definitive document making the case for Nix (and Guix by extension) is Eelco Dolstra's PhD thesis [0] . The introduction is a good read by itself.

I have never used Guix/GuixSD but have heard good things about it. Behind the scenes it uses the Nix package manager and offers a subset of Nix packages licensed as free software. Whereas Nix/NixOS uses the Nix expression language, Guix/GuixSD uses a Guile Scheme front-end. I've heard the Guix CLI is quite nice and a bit more polished than Nix's but Nix is currently in the process of overhauling the CLI. See [1] for a more detailed comparison.

Both of these projects have very active development but don't have the volume of listings as you would find in e.g. AUR or the Debian repos. That said, I've been pleased and actually surprised at just how many packages exist. If you can't find what you want, contributing new packages isn't too hard (often just a case of finding something similar in the repos and changing the relevant values). You can check the current Nix [2] and Guix [3] packages here to see if enough of your needs are covered before giving one of them a try.

[0] [http://grosskurth.ca/bib/2006/dolstra- thesis.pdf](http://grosskurth.ca/bib/2006/dolstra-thesis.pdf)

[1] [http://sandervanderburg.blogspot.com/2012/11/on-nix-and- gnu-...](http://sandervanderburg.blogspot.com/2012/11/on-nix-and-gnu- guix.html)

[2] [https://nixos.org/nixos/packages.html](https://nixos.org/nixos/packages.html)

[3] [https://www.gnu.org/software/guix/packages/](https://www.gnu.org/software/guix/packages/)

<comment><USER31> Thanks!

</comment></comment></comment></comment><comment><USER32> Manually. I wipe rarely, and change tools often for various reasons (even ignoring version upgrades), making building and maintaining an installation script not worth it.

For awhile I did maintain a windows batch script that installed things off of a share at work. I was dealing with pre-release Windows 8, and wiped frequently for upgrades. Even that probably wasn't worth it, but I didn't have a second machine at the time, and wanted to run it overnight instead of blocking my ability to work.

</comment><comment><USER33> I have a repository[0] that holds all my configuration and installs some language-specific tools. Otherwise I just manually install any packages I need. I may consider automating this at some point but I don't use that many tools so it hasn't been particularly onerous.

[0] [https://github.com/michaelmior/dotfiles](https://github.com/michaelmior/dotfiles)

<comment><USER34> ditto. In addition to dot files, my repo has a `system_setup.sh` which installs everything that can be installed on the command line and sets up symlinks and and and. Every time i add a new tool to my arsenal (brew install x usually) i also add it to that file. This repo means i can be up and running on any new system in under 30 mins. Most of that time is for some manual downloads, and git checkouts I have to do too.

</comment></comment><comment><USER35> [https://chocolatey.org/](https://chocolatey.org/)

<comment><USER35> windows 10 comes with packet manager, so you can just execute the following statements on an fresh windows install (powershell):

[http://pastebin.com/HmiqDDbi](http://pastebin.com/HmiqDDbi)

<comment><USER36> Interesting. This looks like NuGet on steroids.

</comment></comment></comment><comment><USER37> Depends on the system.. for OSX, first VSCode, second Homebrew, after that VS Code. I use Homebrew to install the version-switchers for my language of choice (usually node/nvm).

From there, I'll setup a ~/bin directory with various scripts as utilitarian. I may source some of them in my profile script.

\----

Windows Git for Windows, Git Extensions, ConEmu, Visual Studio (Pro or Community, depending on environment), VS Code. I should look into chocolatey, but admit I haven't. NVM for windows.

\----

Linux/Ubuntu generally apt, and ppa's as needed.

\----

FYI: I keep my ~/bin symlinked under dropbox, as I tend to use the same scripts in multiple places. I will separate ~/bin/win, ~/bin/osx and ~/bin/bash, and have them in the path in appropriate order... linux/bash being default. I'll usually use bash in windows these days too, and set my OSX pref to bash. It's the most consistent option for me, even with windows /c/...

</comment><comment><USER38> I use Ansible with Brew and Brew Cask. I've found using Brew for everything makes it easier to upgrade all applications for security reasons and it also gives a high level view of my system. Here's the relevant config file of the things I install:

[https://github.com/arianitu/setup-my- environment/blob/master...](https://github.com/arianitu/setup-my- environment/blob/master/roles/common/vars/main.yml)

The ansible script also links to my dotfiles, which can be found at:

[https://github.com/arianitu/dotfiles](https://github.com/arianitu/dotfiles)

<comment><USER39> I use a similar setup and find it quite dependable. I notice your bootstrap script does not automate the XCode installation. Here's how you can automate that in case you're curious: [https://github.com/timsutton/osx-vm- templates/blob/ce8df8a74...](https://github.com/timsutton/osx-vm- templates/blob/ce8df8a7468faa7c5312444ece1b977c1b2f77a4/scripts/xcode-cli- tools.sh)

</comment></comment><comment><USER40> I find I end up with a lot of cruft and my tools of choice change over time, so I don't worry about it. A decent package manager makes this approach tolerable.

Homebrew and Homebrew Cask on OS X handle at least 90% of what I want to install.

</comment><comment><USER41> As a rails developer, I've used and recommended [https://github.com/thoughtbot/laptop](https://github.com/thoughtbot/laptop) :)

<comment><USER42> We've adapted this at 18F. It's not a fork but it is based on and inspired by Thoughtbot's original project. Strong recommend. [https://github.com/18f/laptop](https://github.com/18f/laptop)

</comment></comment><comment><USER43> Not directly related, but this seems like the right thread to ask: I've been trying to move from Linux to OS recently, and the one thing I can't stand is the .DS_Store and other files which OSX just throws all over the place, in every directory whether local, network or external drive.

Is there a way to stop it? (installable on a fresh system? I've been experimenting with reformatting, so that's not a problem)

<comment><USER44> This might have what you're looking for, anyway it's a good starting point for a lot of commands to customize an OS X instance. [https://github.com/mathiasbynens/dotfiles](https://github.com/mathiasbynens/dotfiles)

Edit: AFAIK .DS_Store is only created when you use Finder. In the past couple of years I've only used the Finders to drag'n drop stuff between ~/Desktop and ~/Downloads, terminal for the rest, so .DS_Store files might not be a problem in practice.

Edit2: Google helps answer your original question [http://stackoverflow.com/questions/18015978/how-to-stop- crea...](http://stackoverflow.com/questions/18015978/how-to-stop-creating-ds- store-on-mac)

<comment><USER43> Thanks. I was aware of asepsis, not aware of DeathToDSStore, but neither works on El Capitan unless you turn off SIP.

I find it so surprising that there's no way to disable this.

</comment></comment></comment><comment><USER45> I use the Nix package manager. I use it on both Linux and OSX.

Setting up tools is quick and easy:

1\. Install Nix

2\. Copy my config to ~/.nixpkgs/config.nix

3\. Run "nix-env -i all"

</comment><comment><USER46> I've been using [https://github.com/superlumic/superlumic](https://github.com/superlumic/superlumic) for setting up my Mac machines because it supports a few more constructs that are common for Macs such as plist file modifications. Since it's Ansible-based you use YAML files to configure everything and it works well with Homebrew at least. I spend enough time working on configuration management professionally and don't want to spend any more time than I must to keep sinking time into my workstation's configuration than I have to, so more complicated DSLs / systems like Puppet, Chef, Salt are out for me despite working with those professionally.

In the future I'd like to try NixOS for managing OS X but it seems rather immature at this point for people that want stuff to Just Work primarily.

</comment><comment><USER47> I usually develop in VM's where the machines are very well defined so it's easier to build a build machine. In the end though this usually means that the absolute package requirements are held by the project repo so what needs to be installed is based on that.

So if I need to start a project or rebuild a system to do a project I can generally use the project itself as guidance on what to install. After two or three builds I have usually cleared all the dependency hurtles. The only thing that really breaks down is not having your dotfiles available, but for me those are backed up/in SCM.

If you are familiar with building projects from scratch it becomes a lot easier to understand dependencies and grow the system to what it needs to be, and with VMs it is that much easier to start with a blank slate that you are capable of blowing away and not even trouble secondary workflows.

</comment><comment><USER48> I did do ansible [https://github.com/CraigJPerry/home- network](https://github.com/CraigJPerry/home-network) but I've decided it's overkill for my dev machines.

I still like ansible as a layer on top of AMI images when I spin projects up in the cloud though. I don't want to have to go install iotop when I want to use it, I want to just know that my tools are present and ready to go. But I consider this a different type of machine from my dev hosts.

As I see it, the best solution on a dev machine is to make it easy to install software I might need in future. That means having a package manager available and access to my preferred configuration. On linux I just need my dotfiles repo available on the box, on windows I also need chocolatey installed.

</comment><comment><USER49> I keep a series of basic setup scripts for Ubuntu on my github[0]. I do wget and pipe them to bash, which I'm not tremendously proud of...

[0]: [https://github.com/cjjeakle/devbox- setup](https://github.com/cjjeakle/devbox-setup)

</comment><comment><USER50> I use bash+rpm.

RPM's are insanely easy to create, and the work helps with deployment/version tracking on the end machine. For example I have an .spec that downloads a given version of codeIgniter, unzips it, tweaks some permissions, and then rolls it into an RPM and tosses it into a network accessible repo. The RPM has a pretty complete list of dependencies, so they automatically get pulled in.

So my devel script looks something like:

      cat <<EOF
      "EOF" > /etc/yum.repos.d/local-repo.repo
      [local-repo]
      name=local-repo
      baseurl=http://xxxx
      enabled=1
      gpgcheck=...
      EOF

      PACKAGES="codeigniter otherstuff"

      for PKG in $PACKAGES; do
       sudo yum install -y $PKG;
      done

</comment><comment><USER51> A few days old but I wanted to contribute anyways:

I do a really fresh/clean install. I install tools as I need them and find myself leaving tools behind often and trying things out. I get setup on new machines pretty often: whether I installed a new linux distro on my chromebook, finally dual-booting Linux on my PC, or just setting up a dev VM.

Anyways, I recently abandoned using Tmux because I prefer using i3 as my window manager and it works for me much better because now I can not only tile terminal windows but also other utilities.

I've also moved on from using CMDer straight to using ConEmu on my Windows machine. Every reinstall is basically doing inventory of my tools.

Obviously, I keep a repo of dotfiles and other settings files.

</comment><comment><USER52> Not as Bash script but some time ago I documented complete environment setup for Windows / Mac OS X / Linux

[https://github.com/Corsaair/redtamarin/wiki/DeveloperEnviron...](https://github.com/Corsaair/redtamarin/wiki/DeveloperEnvironment)

The Windows part have a Batch script setup to automate the install of cygwin, apt-cyg, wpkg, etc.

It is very specific to the Redtamarin project, compiling C++, compiling Java, compiling AS3 to bytecode, etc.

there are also other doc for hardware setup, SSH to/from Windows, running "remote" build from the LAN, etc.

but the setup should work for about anything, comments to improve it welcome :)

</comment><comment><USER53> I use [https://github.com/denisphillips/boxible](https://github.com/denisphillips/boxible) it's like GitHub's Boxen project but instead of Puppet it uses Ansible.

</comment><comment><USER54> At work, we support umpteen OSes in our build (a lot of C++ stuff, so it's picky about where it builds and runs). For the Linux side of things (where most of the dev work is actually done), I'm in the process of building Docker images for our range of build machines, so that a developer can put up a releng-equivalent build environment in a couple minutes, pulling images from a locally-hosted registry.

Our current system is built on a combination of VM templates and documentation for the devs to set up their own build machines manually. Anything that I can't containerize will be stuck there, too.

</comment><comment><USER55> I'm on Archlinux and I keep a list of the packages installed from pacman and from AUR and some config files in [https://github.com/Keats/dotfiles](https://github.com/Keats/dotfiles) ([https://github.com/Keats/dotfiles/blob/master/zshrc#L39-L40](https://github.com/Keats/dotfiles/blob/master/zshrc#L39-L40) for the aliases).

I haven't reinstalled in a long time so the install script might be broken though

</comment><comment><USER56> I maintain Ansible playbooks for my own systems should the worst happen.

<comment><USER57> Just to make sure I understand, your play book isn't for a VM but a host os?

<comment><USER56> Yep. You can use 'connection: local' for example.

</comment></comment></comment><comment><USER58> I think you might like this ([https://github.com/snwh/ubuntu-post- install/](https://github.com/snwh/ubuntu-post-install/)) project a lot.

Well, personally, I made my own project somewhat similar to the one I linked to but mine wasn't that savvy. Plus mine also installed stuff from npm (linters mostly), ruby-gems (jekyll blog), pip and grabbed some sources and compiled them and symlinked them to proper places using GNU Stow.

PS: He's the developer of Paper GTK Theme.

</comment><comment><USER59> Debian 8 has been very brittle, what gave me some experience on creating fresh systems at home...

I have my configurations on a version controlled puppet repository. This helped a lot, and I'd recommend anybody to forget /etc versioning, and use a proper configuration control system even when it's not exactly a requirement. I'm about to ditch my /etc backups now.

A certain source of pain is software that must be kept up to date. I have Firefox and GHC on this category. Both hurt, but it's not worth it to repackage them.

</comment><comment><USER60> I use Ansible to install all my desktop basically. This coupled with my dotfiles (that includes a bash for gnome configuration) makes my workstation ready in 15 minutes or so.

This is my Ansible repo for reference: [https://gitlab.com/edgard/ansible- ubuntu](https://gitlab.com/edgard/ansible-ubuntu)

And dotfiles: [https://gitlab.com/edgard/dotfiles](https://gitlab.com/edgard/dotfiles)

</comment><comment><USER61> I'm one of those guys with an install.sh file that I got from my mentor. I just run it on every new install. Last time being when I installed Ubuntu 16.04.

<comment><USER62> Mine is called "apt-all.sh" and installs gcc, clang, qemu and some libraries:

[https://github.com/vmorgulys/sandbox/blob/master/stackcity/t...](https://github.com/vmorgulys/sandbox/blob/master/stackcity/tool/debian/apt- all.sh)

It's a big list of "apt-get -y install".

</comment></comment><comment><USER63> I started to use Vagrant together with [https://puphpet.com/](https://puphpet.com/) to create a dev machine. I have a MacBook Air and won't pollute it anymore with a dev environment. With a virtual machine, everything stays separated in a nice and tidy way.

Installing dev tools on OS X are a matter of minutes then, because everything else comes with the Vagrant box.

Edit: I'm a web dev guy.

</comment><comment><USER64> I use Chef, Ive got a cookbook here that uses policy files to install homebrew and homebrew casks and set up some stuff. It slao tests it in test kitchen, converging an os x system in virtulbox! [https://github.com/willejs/chef- workstation](https://github.com/willejs/chef-workstation)

</comment><comment><USER65> (OSX) I keep a bash script in a gist on GitHub that I can copy down and run on a new system. It installes Homebrew, rvm, nvm, and then a bunch of homebrew and homebrew-cask packages which is pretty much all of my development environment. Every so often I'll `brew list` and paste in the latest dependency list to keep it up to date.

</comment><comment><USER66> Question: why are you doing a fresh install? If you install everything through your package manager, you don't need a fresh install to clean things up. My operating system has lived through several different machine upgrades. I just do a `cp -a` to copy the files from one hard drive to another, set up the bootloader and go.

<comment><USER67> Theoretically that may be the case, but its far from reality.

<comment><USER68> It's not the same hard drive, but the files in /var/log/installer/ show I installed Ubuntu 10.10 on 24 October 2010. It's been upgraded since, and copied to a new drive at least once.

/etc/popularity-contest.conf has the same timestamp, so I'm curious whether Ubuntu (or Debian) keep any statistics on the lifetime of a system.

</comment></comment></comment><comment><USER69> I use Arch Linux and follow the basic guidelines for installation and modules. The rest is synchronized with my Dropbox and I created a bash script to remove the configuration files and symbolic link them to their respective folders in my sync folder. Thus I can also maintain the same configuration on my other computers, too.

</comment><comment><USER70> It's about an even split between installers from an MSDN account, and a ninite bundled installer for the rest.

<comment><USER71> Ninite bundled installer has made setting up new computers for family members so mindlessly easy. So glad it exists.

</comment></comment><comment><USER72> I use a ShutIt script:

[https://github.com/ianmiell/shutit-home- server/blob/master/S...](https://github.com/ianmiell/shutit-home- server/blob/master/Shutitfile)

it's platform independent and automates the install of everything I need.

<comment><USER72> More info:

[https://github.com/ianmiell/shutitfile/blob/master/README.md](https://github.com/ianmiell/shutitfile/blob/master/README.md)

[https://github.com/ianmiell/shutitfile/blob/master/CheatShee...](https://github.com/ianmiell/shutitfile/blob/master/CheatSheet.md)

[http://ianmiell.github.io/shutit/](http://ianmiell.github.io/shutit/)

</comment></comment><comment><USER73> I created a personal list, but half of the install is manually made: [https://framagit.org/briced/conffiles/raw/master/INSTALL.md](https://framagit.org/briced/conffiles/raw/master/INSTALL.md)

</comment><comment><USER74> Here are mine: [https://github.com/ashishb/dotfiles/blob/master/setup/setup_...](https://github.com/ashishb/dotfiles/blob/master/setup/setup_new_mac_machine.sh)

</comment><comment><USER75>
        sudo apt-get install emacs

<comment><USER76> OP asked how to install developer tools, not how to install an OS ;)

</comment></comment><comment><USER77> I have a bash script and a PowerShell script stored in a web-accessible place. I pull it down and it bootstraps Chef to run chef-solo on the machine from a repository stored in a trusted location.

Dotfiles are stored in Dropbox, too, which is handy for keeping zsh and Sublime synced.

</comment><comment><USER78> Here's my brain dead script: [https://github.com/pathikrit/mac-setup- script/blob/master/se...](https://github.com/pathikrit/mac-setup- script/blob/master/setup.sh)

</comment><comment><USER79> I'm very interested in a way to containerize (runing them within a vm) these tools that way nothing needs to be installed, and could go easily from machine to machine, without polluting the filesystem.

<comment><USER18> Honestly I think this is overkill. For a dev machine, I install all my tools locally that way "yum update" or equivalent keeps everything up to date easily.

Use VMs or containers for local simulation of your deployment targets.

</comment></comment><comment><USER80> I've had a painless time setting up my mac with [https://github.com/msanders/cider](https://github.com/msanders/cider)

</comment><comment><USER81> I use Thoughtbot's Laptop. [https://github.com/thoughtbot/laptop](https://github.com/thoughtbot/laptop)

</comment><comment><USER82> I tried boxen from github. Like others have mentioned, tools like these have a big learning curve and boxen was pretty high maintenance after setup in my experience.

</comment><comment><USER83> I use ansible: [https://github.com/dschep/box](https://github.com/dschep/box)

<comment><USER84> Ansible is great, we use it at work extensively. But I am sticking with Vagrant for OSS project as it (Ansible) does not work with Windows host.

</comment></comment><comment><USER85> 1\. Install Vagrant.

2\. Install git.

3\. Check out the complete, working development environment and run its Vagrantfile.

</comment><comment><USER86> choco install as the need arises</comment>


Ask HN: What are the best Emailing Services out there? - <AUTHOR> If they have a free-lower tier that would be awesome. <comment><USER2> Are you looking for personal e-mail, transactional e-mail or a service for newsletters and such?

For personal e-mail you could try either one of Zoho Mail, Outlook, Gmail or iCloud Mail.

For transactional e-mail you could try SendGrid, Mailgun or Amazon SES. All of them have free tiers. I've been using Sendgrid for a while and could recommend them.

When sending out newsletters you could use Mailchimp or SendGrid, both also offering free tiers.

</comment><comment><USER3> Echoing what others have said. For transactional emails my go to is Sendgrid, for newsletters Mailchimp.

With Sendgrid you get one month of up to 40,000 for free. After that the (forever free plan) is up to 100 emails a month I believe.

</comment><comment><USER4> Rackspace and FastMail have a great offering for personal emails. For transactional I also recommend Sendgrid and MailGun.</comment>


Ask HN: What are the consequences if you don't put your phone into flight mode? - <AUTHOR> Do consequences/dangers exist (If any), or does the pilot just hear that annoying "da da da" noise in his/her headphones? <comment><USER2> With the current digital phones there is almost no chance of the phones interfering with the airplane's operations. When phones were analog their frequencies did live in some of same ranges as the plane's equipment.

Analog phone used lower frequencies to cover longer ranges. Back then we had of fewer towers so you had to use lower frequencies. The lower the frequency the more ground you can cover, the better reliability. Airplanes used the similar frequencies for all the same reasons.

Now both planes and phones use better equipment with better FCC oversight on was frequencies and transmission type each uses. Airplanes rely more on satellite communication then terrestrial towers which further helps to prevent radio communication collision.

</comment><comment><USER3> The moment you take your phone off flight mode, the plane will nose dive steeply until all meet their doom.

Unless of course you put flight mode back on again.

</comment><comment><USER4> very very little, while they make out that its all scary there is no way that theyd ever let something off the ground that could be brought down by a few rouge radio waves. id wager that on most any flight youve been on there were one or two people who didnt turn their airplane mode on

<comment><USER5> _rouge_

Small nitpick: you probably mean _rogue_, rouge alludes to red (powder).

</comment></comment><comment><USER6> People can call you, and that's just awful.</comment>


Unlocking Product Market Fit - <AUTHOR> http://blog.clarity.fm/unlocking-product-market-fit-from-the-author-of-lean-analytics/

<comment><USER2> This is so true:

"You need incredible vision and conviction to be successful. You also need incredible timing and an absurd amount of luck."

Whether or not we'd like to admit it, luck does have a lot to do with success.</comment>


Ask HN: If you had 200M dollars and a goal, how would you achieve it? - <AUTHOR> In a hypothetical situation, you were just granted a fund and a goal. You have complete freedom to operate in the world.
What are the 10 big steps that would get you to your goal, as a causality?
You pick the goal. <comment><USER2> With that money you can chill and travel around the world for fun. And it will be your goal.

<comment><AUTHOR> Ok, we have 1 answer about being selfish. I expected this.

</comment></comment><comment><USER3> Hire the best people in the world and let them solve it for you.</comment>


Rendering: don't try this at Chrome - <AUTHOR> https://medium.com/prezi-engineering/rendering-f857b6e09551 <comment><USER2> found these relevant discussions on planning to implement 2D context loss events on Chromium dev channels: [https://code.google.com/p/chromium/issues/detail?id=322335](https://code.google.com/p/chromium/issues/detail?id=322335) [https://groups.google.com/a/chromium.org/forum/#!topic/blink...](https://groups.google.com/a/chromium.org/forum/#!topic/blink- dev/CQIwXxKpO_Q)

</comment><comment><USER3> It's interesting to see that when IE was the most popular browser it was the most problematic. Now it's Chrome.

<comment><USER4> It's always the one with the most users. :)</comment></comment>


What happened, Gmail? - <AUTHOR> https://uxplanet.org/what-happened-gmail-e5f35e423b1b <comment><USER2> The author's experience with GMail seems to be the opposite of mine. The new features for disappearing/unprintable/etc. emails sound like they're just going to cause annoyances while Gmail sorting things into promotions and updates is _absolutely fantastic_. And I use the "forums" tab too. This _is_ the "good method for telling users what they need to read" that the author wants - that's what Primary is. This _is_ the way to help manage the number of emails you get. This _is_ the efficient way to categorize and sort the emails that you get. It's really not Google's fault that you've got 6gb of emails in the promotion and updates tabs without realizing it - they've been prominently showing you the new emails you've received in those tabs every time for years now.

Here's the feature I keep wishing for in Google: when I attach a file, it should let me rename the file. I want to store it on my compute with a name that's meaningful for me and to send it to someone with a name that's meaningful for _them_. Eg: I want to have cv_google.pdf for my copy of my CV that I'm sending to Google for a job application but I want to have them receive it as cv_tgb.pdf to know that it's from "tgb". Similarly, when I download a file I want to be able to rename it.

<comment><USER3> I have been using Google Inbox since the launch and I absolutely LOVE it. It finally solved my email problems.

\- I have notifications disabled for all but the important emails. So I respond to emails in time and I don't keep checking my phone anymore.

\- Easy to read and discard emails. Especially promotions. I actually do check them - once a day and once a day only and then delete them all.

\- Social/Updates ...etc are neatly organised. I check them during a downtime and not often. They don't alert me.

\- Organising emails by trips are great. I can find all the relevant emails in one bundle.

\- Absolutely love the GTD features - Marking as Done and Snoozing emails. My Inbox is always zero. I use that as pretty much a task manager as well.

Overall, classification of emails make emails actually usable for me. I used to be meticulous with creating email rules and filters in Outlook so I get to the ones I need to and ignore the rest for my sanity. Now it's done automagically and pretty spot on.

<comment><USER4> My problem with it is that I don't trust it. Outlook does the same thing, I'm not singling Google out for this. Anything that automatically sorts my email for me, I don't trust it to get it right 100% of the time. And the last thing I want to do is a miss a critical email because Google or Microsoft told me it wasn't important to me.

Case in point: I run a website for my town and some user actions are sent to me via SendGrid emails. Not a lot, but maybe one or two per day. I've made the email subject descriptive enough that I rarely need to open them to know what happened. I'm guessing that because it's a mailer from SendGrid and that I rarely open them, my email provider decided that they weren't important emails and stopped sending them to my inbox. When I figured it out a few days later and went to check the other tab, I found a couple messages had been sent of people wanting to purchase ad space that the email provider had also decided wasn't important enough to notify me of.

So now I have to check both tabs religiously (in reality I just turned it off) because I can never trust that I'm not missing something.

<comment><USER5> The idea with whitelisting is viable. Use a firstname.lastname@example.com email for these and then in the classic interface set a filter that will land these mails always in your inbox with an additional label.

Been doing this kind of categorizing with filters since I discovered the feature and never looked back.

<comment><USER4> But then that's no better than just managing my email by hand. The whole point of these smart inboxes is to not have to do that.

<comment><USER5> In my setup there are no blanket rules. Only super important and especially annoying stuff gets the filter treatment. All the rest is autosorted.

</comment></comment></comment></comment></comment></comment><comment><USER6> Email was meant to be kept simple, and Google is making this exponentially hard with every single "design" iteration. Get off Gmail - it'll break your email.

The article has very good points, especially when it touches the issue of "hidden" mail, like the promotions tab: there are labels and tabs. There should be only one system, and no, don't hide things from the user at this level. When there's 17k mails in promotions, eating up 6GB alert them, let them know. (This ties back to how WordPress keeps hiding anything technical from users, which is also bad[^1])

Filtering on Gmail is outrageous, especially when you compare it to Sieve. No option to match on custom header, seriously?

Things like unprintable email is a bad joke. Unprintable? What if I connect to Gmail with mutt? It gives the sense of a false security.

I'm aware of all the arguments against email, but so far nobody could come up with a robust, reliable (see SMTP retries), async, world wide, federated solution, that even touches the level of email.

Here's a revolutionary idea: instead of trying to come up with a new email, stop breaking the current one, and keep using it.

[^1]: [https://www.rarst.net/wordpress/technical- responsibility/](https://www.rarst.net/wordpress/technical-responsibility/)

<comment><USER7> If I want to take my 20GB of email out of Gmail and put it elsewhere for the next 15 years without having to think about it, where do I go?

<comment><USER8> You might also consider registering a vanity domain to go with your Fastmail (or other) subscription. It'll allow you to keep your email address even if Fastmail gets evil / bad in the future.

When I bought my domain, it was before Gmail was as ubiquitous as it is today. My email was hosted by a mom and pop ISP. Then I moved to Gmail and later to Fastmail.

Having my own domain is inexpensive, fun, and lets me maintain email portability over time. Namecheap is a good, easy registrar. I moved to them a while back and was impressed at their documentation and help during that domain transfer.

<comment><USER7> Good advice. I assume I can forward my Gmail because that address has been my address for as long as gmail has existed. This is just like cell numbers. You attach one to everything in your life and transitioning is near impossible. Glad the government forced free number transfer. If only emails worked the same way.

<comment><USER8> Yes, I'd forward emails until you can get everyone trained over to using your new email address. I've done that in the past with mostly inactive emails that I got through professional or academic associations. Presumably Gmail still makes auto-forwarding reasonably easy.

<comment><USER9> Why not just leave the forwarding on? I still have my first e-mail address from the 90s forwarding all incoming mail to my current one, though it's very rare by now that anything comes.

<comment><USER8> You could! The only reason not to is if you wanted to close the account for some reason.

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> Among "needed innovations", the post lists: "A good method for telling users what's important to read and what can wait for later."

To me, it's downright scary that someone would want Google telling them what is important in their own mailbox. Personal responsibility bad, Hand-holding good, apparently.

<comment><USER11> Just as scary as having Google filter out mail that is totally not important, i.e. spam?

In Gmail you can (un)label email as being important, which will then train an algorithm just like a spam filter. Besides that, it will learn to recognise email you reply to often as being important.

I very much like this feature; I set up the Gmail app on iOS to only send me push notifications for important mail. For me that strikes a good balance between no distractions and not missing out on important email.

<comment><USER12> Spam can be removed by objective filters, that simply classify the email against known patterns/rules. Who you are or what you like is irrelevant to them.

This is not that: it's you, giving a giant personal information sponge, a bigger tap.

<comment> loup-vaillant > _Spam can be removed by objective filters,_

Those "objective filters" prevent me from sending email from home (I have to relay through a non-residential IP).

I receive about a dozen spam email per day (with occasional surges and lapses). My server accepts everything, and a simple local filter from my mail user agent (Evolution or Thunderbird, mainly) let few through, and false positives are very rare.

I'm not sure why the giant providers need to work any differently.

<comment><USER13> A dozen spam messages per day? Lucky you. I did a tally on yesterday's harvest on my server and found the following:

\- 57 rejected messages designated as spam by SpamAssassin

\- 137 greylisted messages, most of which will end up being spam as those addresses which I communicate with regularly will be in the whitelist.

\- 181 connection attempts blocked at the gate due to protocol violations (most of them due to fake HELO, usually trying to connect using my own server's FQDN)

\- 144 delivery attempts blocked at RCPT due to the use of blacklisted recipient addresses. This is why using recipient-specific sender addresses makes sense when communicating with commercial, organisational or governmental institutions: it makes it possible both to track down who leaked or sold addresses to spammers as well as to block those addresses entirely.

This domain has been handling mail for close to 23 years now, the server is used daily by about 8 people, it also forwards mail for a few others.

I generally don't see more than one or two spam messages per week in my actual INBOX.

<comment> loup-vaillant Ah, 8 people. If we remove the protocol violations, we get (57 + 137 + 144) / 8 = 42 spam message per person per day. Between 3 and 4 times my amount. I may be lucky, but frankly that doesn't sound extraordinary.

I also get no more than 1-2 spam message per week in my inbox.

</comment></comment></comment></comment></comment></comment><comment><USER14> Extra First Class World problems: Inbox is not optimized for iPhone X. Everytime I read this I cringe.

How about the thought that your phone broke common UI behaviour and not your software which needs to adapt to a new screen culture.

I'm sure very soon Inbox will support iPhone X. But there are many other problems in Inbox which I see more important like Unspam emails without Gmail.

<comment><USER15> > Inbox is not optimized for iPhone X.

This has led me to think Inbox is on its way out. Lack of updates is the classic Google move prior to killing a product.

<comment><USER16> > This has led me to think Inbox is on its way out.

Any time an app takes forever to update to a new apple screen resolution is a good indicator the app is on its way out.

I mean, think about it.... if you were all-in on your app, wouldn't the highest priority fixes on your backlog be "get the damn thing to run in the native resolution of your target device"?

</comment></comment></comment><comment><USER17> Article is a little bit biased as a one who worked on messaging apps claiming that messaging (eg Slack) solve communication is just wrong. Outlook screenshot is very simple and clean but... it mentioned as an example of nightmare.. Looks like author got in the his own bubble just like google.

<comment><USER17> > An efficient way to categorize, filter and search content.

No, no one likes to categorize emails, this is work for someone else or for power users that's not a case for GMail.

> A tool to help users fix mistakes they've made, such as sending someone the > wrong email, or spelling something incorrectly.

For a long time Telegram didn't want to introduce this feature since if you sent something this should be in other's inbox. If someone can modify your inbox than you won't be able to trust that some random email won't disappear.

> Allowing users to handle their business better through Gmail (e.g. sign > documents, approve things, review things).

Uh, guy just want to put everything to GMail while claiming that this is not an "innovation".

> Allowing users to design their emails in a better way.

Do you want to receive emails from your lawyer in comic sans? There is a reason why FB, Slack, Telegram, etc don't allow you to style your text easily.

> Letting users know if someone read their presentation and what parts > interested them (DocSend).

Breaking fundamental flow of an email. No one will see if you read and no one will be upset. There are a reason why slack dosen't have read status.

GMail probably have a lot of problems now, but (sorry) this all just unprofessional judgment.

<comment><AUTHOR> I think these suggestions make some sense since they are still problems that exist. The way you'd imagine solving them could be different if you were on that team, with the data and would iterate to find the solution. The bullet points under the innovation section are not suggestions for solutions, on the contrary, they are problems that users still have. For each of these problems, there are services that are trying to solve that problem. Take scheduling a meeting as an example. Google's solution was to bring the calendar into Gmail and increase complexity. X.AI's solution is to make a bot that coordinates. Some other companies will have different solutions. The point is, what is the best solution for a user who is in the frame of mind of emailing. It is very easy to stuff things from different places one on top of another and it's called sustained innovation (if I'm not mistaken). But what ideally should happen is finding new ways to adapt these extra services, including these problems in a way that is matched with the main use case. Otherwise, it gets complex.

</comment></comment></comment><comment><USER18> I created an account to comment on the good experience I was having with the update.

I did a side-by-side comparison of the old UI and the new UI and I've changed my stance. The old one is a better "email" UI, but google is trying to mature the email experience.

A big problem here is that email showing it's age. Google is eager to hide less common tools behind menus. They want to delete all of the old "manual" buttons/tools so their smart filters can take over. It wont work. The truth is everyone likes email because it's fairly consistent, but we're forever trapped in 90's tech.

Instant messengers have transformed the way that we communicate, but email hasn't caught up. Maybe... it shouldn't.

<comment><USER6> > email experience

Email, at this point, is infrastructure. There is no need for an "experience" with it.

</comment></comment><comment><USER19> What got me is they moved the archive button. I have been trained to click on the email, move my mouse up a few pixels, and click archive. Now when I do that, my cursor resides on the spam or delete button. Oops! I don't think the Google of yesteryear would have made that mistake -- the google that designed Chrome closing tabs to stay the same size so you could close multiple tabs quickly.

</comment><comment><USER20> Something I really hate about Gmail is the way it places attachments at the bottom of the page. So if I open a long thread in which the latest email had an attachment, I've got to scroll down to the bottom to locate it - it's so unintuitive that I often find myself scanning the page looking for the attachment.

</comment><comment><USER21> I feel with each Gmail "improvement" Gmail gets slower. I miss the days when Google's priority was speed and not pretty. I feel less productive when I'm always eating for pretty Gmail to load and respond to actions in the ui.

</comment><comment><USER22> It flickers. It animates. It's annoying. Thank goodness they did not break keyboard shortcuts.

It's more intrusive as a gui than the old one.

<comment><USER6> As long as google keeps the imap and pop3 gateways alive, we have a choice.

Use them, utilize other webmails, even run it localhost, like rainloop; desktop clients, like geary, evolution, claws, thunderbird; command line, like mutt.

Don't let the gmail interface drive the way you want to deal with email.

<comment><USER23> Their imap gateway is nonstandard to say the least. The labels vs folders thing is already a mess, but it also speaks a protocol that is just different enough from regular imap that most clients (e.g. thunderbird) have a different implementation with quirks for google-imap.

Using those clients is sub-standard with google's imap implementation, and switching between them is more painful due to how each one handles googles "labels not folders" quirk

<comment><USER6> This is true, I'm well aware of this, but at least there's still a choice.

Unfortunately nothing is truly stopping Google to pull a Slack move and close the gateways... unless the amount of IMAP/SMTP users are significantly higher, than we know, or that those users are tech influencers Google don't dare to loose.

<comment><USER24> I would wager that G Suite users will keep IMAP/SMTP alive via Outlook and other enterprise deployments.

</comment></comment></comment></comment></comment><comment><USER25> Is it just me or does the new Gmail UI feel sluggish/unresponsive? Scrolling the mail list is lagging a bit, the left side menu takes time to initiate the appear animation and the animation itself is too fast. When hovering the mouse over mail list, the interaction icons do the lagging effect, which is really not nice for eyes.

However the right side menu for Tasks, Calendar and etc. is nice, I like having them in one place instead of keeping multiple diff tabs open.

I'm not sure if this lag effect is intentional or a performance problem, maybe React instead of Angular could help :D

</comment><comment><USER26> Inbox looks really nice and is (to me) very simple and I kinda wished Google would have uses Inbox as a template because Inbox has also two things which makes my life really hard: 1 I can't sent emails to groups in my contacts and 2 in the Sent folder it doesn't say to whom I've sent the email but that I sent them, which I already know.

Both issues are also true for the Gmail application on Android.

</comment><comment><USER27> It's also slower. The scrolling is choppy and I shouldn't need a top of the line computer to scroll smoothly through some emails.

</comment><comment><USER28> I just went through my Inbox to figure out why I still use email.

For personal stuff, I think WhatsApp, Messenger, iMessages are clear winners. 90% of my emails seem to be marketing I'll never read or notifications I ignore. I try my best to unsubscribe, but email really needs maintenance to work nicely.

For work almost everything is Slack. Sales team and a few others still seem to use email for some reason. Email is vital for communication with clients and it's pretty good for a big announcement where you can then choose to "reply all" or "reply", not sure Slack solves that nicely with threads. Also email seems to be used for anything considered too important for Slack, but I can't see any good reason why it should be that way.

When you boil it all down, it looks like a lot of use of email is for legacy reasons, i.e. some people still insist on using it. There is legitimate usefulness over other options when it comes to communicating between companies though. Even if Slack figured out a way to fix that, they'd probably require the reciever to be paying for Slack.

I guess the ability to send a stranger a message is email's weakness and it's strength. Nothing out there has solved that yet.

<comment><USER29> an email address and a phone number are still the only things you can safely assume everyone has. for many of my older relatives and non-tech acquaintances, the only options are email, sms, or phone calls.

i dislike phone calls in general for all but the most time-sensitive and/or intimate conversations. i get a bit anxious on the phone and i don't like being essentially blocked for the duration of the call.

i dislike sms because i don't have an iphone, so i cannot respond using a full sized keyboard without using workarounds like pushbullet, which i don't feel like setting up for the small number of people who insist on using sms.

i don't love communicating via email but i would contest the claim that it is a legacy technology on the basis that it does not have a clear successor.

there is no modern service that matches the ubiquity of email; the penetration is unequaled even by the likes of facebook. i would also point out that (personal) email addresses provide highly stable means of communicating with infrequent contacts. while people's phone numbers change over time, people seem to accumulate personal email addresses and at least forward the old ones to their main account. if i got someone's personal email address five or even ten years ago, i have a pretty good chance of reaching them today. on the other hand, i assume that phone numbers that i haven't used for a couple years are dead links. with messaging apps, you don't always have a clear indication of whether the person currently has the app installed with notifications enabled.

</comment></comment><comment><USER30> Been thinking of switching back from Inbox to Gmail for one single annoying and IMO stupid design decision: if there are multiple recepients for the email the main reply button will by default perform a reply all. Had me look like a dork a few times, replying to everyone that I can't make it to a party or something.

Anyone else been bitten by this?

<comment><USER31> I think the party host who is sending his invitations without using bcc is to blame here.

I use gmail mainly for business related mails and the default reply-all makes a lot sense - almost all the time other recipients ( managers, clients etc) are there to be kept in loop. If you forgot and just reply to it, then you have send the same mail again - or worse you never notice it.

<comment><USER30> The party host is not using bcc because he/she wants everyone to be aware of who has been invited to the party.

I guess it boils down to your default usage of email; I've never had an email client before Inbox that does 'reply all' by default.

</comment></comment></comment><comment><USER32> Where is this "unprintable e-mail" and "self-disappearing e-mail" feature? I don't see the options on my Gmail. I would love to write a Chrome plugin to destroy these features.

Once information enters my premises, I do whatever the hell I want with it as long as I'm not sharing it with others or violating any NDAs. In particular, you do not get to define the media and methods I use to access, consume, and save information. You just get to send me a binary number containing information, and it's your choice whether you want to do that or not. That's it.

If you don't agree to me having personal freedom, don't send me information and don't contact me.

</comment><comment><USER33> Between the Gmail and Reddit redesigns Reddit's redesign was so much worse.

</comment><comment><USER34> Those that don't use inbox, why? I'm sure there are features of gmail that are missing from inbox but what are they? Why do Google keep these two "forks" of the same product?

<comment><USER35> I find the lack of visual density and regularity in Inbox very annoying. Gmail (compact) fits 3/4x times more emails on the same screen, with a single text line of identical size for each. It's very quick and easy to scan through.

Inbox looks like a jumbled mess to me, with randomly sized images and pre- rendered images, and it makes me scroll a lot more.

Annoyingly, the new UI is slightly less dense than before, but it's still much better than Inbox.

<comment><USER34> I think the key idea behind inbox is that you don't have more than say 5 or 10 emails in your "to do" view. I never had to scroll. I only look at the inbox view (i.e. things I need to deal with) - and that is always fewer than one screen. If I need something from the "already dealt with" screen I do a search, never a visual scan. This might be due to the volume of emails received obviously, but that's why I was wondering. It might also depend on the usage pattern, e.g. I reply to probably fewer than 1 in 100 emails for example.

If I had more emails, or more emails I actually had to address by replies, it might be different. I never noticed the "images" in inbox though. Are those for things like purchases or travels that it renders differently? My emails just look like subject lines, with about the same density in inbox as in classic gmail (Quick check: An inbox email on desktop is a 32px div with 6px padding, while a gmail email is a 39px table row - so quick estimate is 5px more per email for inbox)

<comment><USER35> Yes, for some reason Inbox expands some attachments inline for me, and pre- renders some links (with a picture from the page a short summary). It doesn't seem to do it systematically, and it could be something that can be turned off in options (it didn't seem to be when I first checked). It seems to pick the most useless items for expansion (recurring bills! 150px vertically for this month's G drive bill!).

A quick an dirty screenshot-on-my-current-machine shows inbox compact emails at 60px vertical, vs 50 for the new gmail. All of the difference seems to be whitespace, the fonts are more or less identical. There's extra waste for "Today" and "This month", and with a couple of expanded pictures, I can't even fit 10 emails vertically on my screen (MB 12"). Gmail shows 20 emails in the same space just fine.

It's quite possible that if you have just the right volume of email, the Inbox format is perfect. It's also possible that most people don't mind scrolling. Me though, I like to see as much as I possibly can at a glance.

</comment></comment></comment></comment><comment><USER36> My gripe is when I star something in gmail it shows up 3 times in my outlook todo list. I suppose it's partly microsoft's fault.

Hmm maybe there is a fix: [https://www.msoutlook.info/question/copies-in- important-fold...](https://www.msoutlook.info/question/copies-in-important- folder-gmail)

</comment><comment><USER37> I don't understand replacing the onmouseover shortcut to all mail from/to a given address with shortcuts to other google apps actions (add to contacts, compose, schedule event, hangouts, etc.) I'm mostly a keyboard shortcut person but that's a mouse action I use all the time.

</comment><comment><USER38> While we are griping, the mobile website is complete crap on iOS, constantly mistaking the initial touch to start a swipe motion. So you accidentally open emails while trying to scroll through them. I've never seen any other website do this, so I'm assuming someone got too clever.

<comment><USER39> Youtube (another Google product!) does this and it is infuriating! The youtube iOS app is utter garbage, but the site itself is nearly unusable because of the initial-touch mistake.

</comment></comment><comment><USER40> I moved from gmail to fastmail and am much happier.

</comment><comment><USER41> I like the new design.

My only issue is that the sidebar expands when I mouse over it. Does anyone know if this can be fixed so just the icons show?

</comment><comment><USER42> What about sorting e-mails? By size, for example?

<comment><USER43> I think that's a very uncommon use case and I can't think of a time I've ever wanted to do that. I've sorted by "has-attachment" but why would you need to sort by size?

<comment><USER44> Well it becomes important when your .PST file is about to hit the 2GB limit and you need to find large attachments to delete.

[https://www.lifewire.com/outlook-pst-files-size- limit-117334...](https://www.lifewire.com/outlook-pst-files-size- limit-1173344)

(NB: I'm being facetious, and agree with you - this, 15ish years ago, is really the last time I remember needing to sort by size)

</comment></comment></comment><comment><USER45> I'm sure the Gmail team has done great research on this, and this might have worked for the majority of the users, but just not for some power users such as Hacker News readers. And I think, that's because we might have one false belief, that is User Experience/User Interface is one big, beautiful, monolithic top of the mountain everyone wants to reach. It is not: Power users and the normies might have different needs. A normie like your mom and mine don't know how to turn off annoying notification emails from Facebook, so they might need a tool to help them hide spammy emails. They need something easy to hit, hard to miss, because they use their fingers to point. But you and I use keyboard shortcut and know exactly not to give our serious email addresses away, we don't need that.

An UX that is great for your mom and mine might be bad for you and I.

I noticed that on one of my project that got on Hacker News and Hackaday, then got viral on reddit and media and such a week or so later. It has both a github and a couple of videos linked on the site. What I noticed from my rudimentary Cloudflare log and Youtube analytics was that the HN crowd read and looked at the github, but most didn't watch the video. The normies crowd watched the video, but most didn't read. To me, it was kinda funny.

I think that's part of the crisis that GNOME 3 suffered lately. I seriously tried GNOME 3 for a while and their macOS approach - simplify, trying to make good choices by default - genuinely sucks for me. I want to see my options, don't hide them away from me. I want to be able to make choices. The problem that the GNOME 3 team doesn't realize is that their users aren't made up of majority normies. They are the ones who are savvy enough to install a Linux distro on their desktop. They are the ones that will go great lengths to customize something to make it work exactly the way they wanted. And GNOME 3 fails to deliver that [1].

At first when I worked on my project, I wanted to create something very simple that "just works." As time went by, it turned out to me many of my users, the ones that stick and support my project aren't the ones that just install and forget. They read the docs, wiki and the source code to tease out what I was trying to do. So I realized having a great, updated wiki is a very valuable asset. It takes great pain to do it, but it is worth it.

The problem with project that are extremely popular like Gmail is that you tend to carter for the mass. But are they loyal? Are they the ones that actually use and love your product? Are they the ones that influence other people's choices? I think those are the questions worth asking and considering when designing products.

1: [https://www.reddit.com/r/linux/comments/8etezq/_/](https://www.reddit.com/r/linux/comments/8etezq/_/)

<comment><USER6> > so they might need a tool to help them hide spammy emails.

They need a tool to make the source of the spam stop spamming them, not a tool to hide this. Sweeping the dirt under that carpet is not an actual solution or fix.

</comment></comment><comment><USER46> Also what genius decided that left menu will slide on mouse hover when there is hamburger icon for that.

</comment><comment><USER47> when gmail started filtering my emails I just deleted all the filters/tabs/whatever. Though it took me bit to figure it out.

</comment><comment><USER48> I don't understand people complaining on free services. I believe it's something you can do, but not as seriously as this article makes it sound. Gmail is a toy.

</comment><comment><USER49> the new self erasing and unprintable emails make me really hope we can opt out of the new gmail.

</comment><comment><USER50> the missing bloody contacts is driving me nuts, otherwise its ok i guess.

</comment><comment><USER51> does the new design allow me to sort my emails in date order received?

</comment><comment><USER52> Material Design is one of the worst things that ever happened to the design of websites, not only Google's but everyone's, since now so many are trying to copy it. I have no idea why anyone would want to see it anywhere.

<comment><USER53> Couldn't agree with you more. We should have a drink :)

</comment></comment><comment><USER54> Better than Inbox, anyway.

<comment><USER33> Better than Reddit's redesign

</comment></comment><comment><USER53> @google, check this: [https://dribbble.com/shots/4020485-Inbox- Client/attachments/...](https://dribbble.com/shots/4020485-Inbox- Client/attachments/921027). It's better than yours.

Never a fan of material design.. seriously, Google is NOT good at design. Period. I hoped, in this Gmail iteration, Google would finally take the plunge and kill the flat icons and minimalist design and once for all. It's quite the opposite unfortunately. It got flatter and uglier. All so plain and washed out, one doesn't know what he should look at first.

<comment><USER55> That design is...awful. Tons of wasted space, prominent screen real estate given to interactions the user will never engage and less than 50% of the screen is given to the content I am there to see?

> one doesn't know what he should look at first.

Pot, meet kettle.

<comment><USER53> So what I heard is that people would just prefer something like the Thunderbird, basic but functional - [https://addons.cdn.mozilla.net/user- media/previews/full/152/...](https://addons.cdn.mozilla.net/user- media/previews/full/152/152814.png?modified=1458845197) and you don't care if it's ugly. However, You can't just think like gmail are only used by engineers who designed it. Most of the users are not.</comment></comment></comment>


Adobe Drops Licensing Fees, Gives Away Flash For Devices - <AUTHOR> http://blog.wired.com/monkeybites/2008/04/adobe-drops-lic.html <comment><USER2> This was published back in April??

Weird, I hadn't heard of it.

</comment><comment><USER3> The Flash Player is still proprietary, and Gnash is far from being a full replacement. Hopefully, HTML 5 video and faster DOM graphics will replace Flash soon.

<comment><USER4> Ew, HTML5. What we really need is for IE to support SVG. Instead, they made silverlight.

<comment><USER3> With "faster DOM graphics", I meant to include SVG. I can't believe IE still has no native support for it :-( I see that depending on your requirements, there are partial SVG and Canvas implementations for IE in JavaScript, as well as higher-level JS APIs that work across the browsers.

<http://starkravingfinkle.org/blog/2006/03/svg-in-ie/>

<http://excanvas.sourceforge.net/>

<http://www.liquidx.net/plotkit/>

<comment><USER3> Took me a while to find the link again, but this work to implement Silverlight in JS and SVG is closest to flash I've seen for building RIAs: <http://github.com/toshok/firelight/wikis/home>

</comment></comment></comment></comment><comment><USER5> You have to give these guys credit for AIR. I'm very impressed by the applications I have seen built on it-- especially balsamiq.

I'm definitely looking forward to building on it.</comment>


I Didn't Get Hired. Here's Why - <AUTHOR> https://googleyasheck.com/i-didnt-get-hired-heres-why/ <comment><USER2> "I Didn't Get Hired. Here's Why" Doesn't say why.

<comment><USER3> Even worse, he says: "I don't know why."

</comment></comment><comment><USER4> I have seen this guys ridiculously popular repo [https://github.com/jwasham/google-interview- university](https://github.com/jwasham/google-interview-university) I just don't get why is he fixing on a single company like google. I believe there are many other companies where he can be much happier. From what I read, engineers at google struggle to live up to expectations there. And those are the guys, that would be rockstar engineers in other companies.

</comment><comment><USER5> Then apply somewhere else that will value your talents. Google isn't the only company out there.

I couldn't figure out why he was so focused on just getting hired at Google. Either try, try again, or move on to something else. That's kind of how it goes. Who knows, he might thrive at a smaller company that he never thought about before.

</comment><comment><USER6> Saw that coming 2 posts ago.

But that completely wrong title really brings it something.

<comment><AUTHOR> I would have renamed it, but that was the exact title on the article.</comment></comment>


Efficient pagination of a SQL table with 100M records - <AUTHOR> http://allyouneedisbackend.com/blog/2017/09/24/the-sql-i-love-part-1-scanning-large-table/ <comment><USER2> In my current job I maintain a database of 2B records that is (unfortunately) on spinning rust and on a last-gen Intel desktop CPU system. This database is a read-only data warehousing sort of setup.

If you need to do operations like this on a database with this setup I'd suggest checking out the following config options:

        1. max_heap_table_size
        2. read_buffer_size
        3. sort_buffer_size
        4. join_buffer_size
        5. thread_concurrency
        6. tmp_table_size

I'd also suggest looking into TEMPORARY TABLEs with ENGINE=MEMORY.

I have not included values in this because the values that make sense for you are likely not the values that make sense for me. Check out MySQL's documentation for what these values effect [0]. The defaults in even MySQL's huge config are very outdated for the horsepower that modern computers bring to the table. It's funny that MySQL's once massive 4GB of RAM config file is now the appropriate setup for my laptop.

[0] - [https://dev.mysql.com/doc/refman/5.7/en/server-system- variab...](https://dev.mysql.com/doc/refman/5.7/en/server-system- variables.html)

<comment><USER3> To set these variables appropriate for your system, I recommend the MySQL tuner script:

[https://github.com/major/MySQLTuner- perl](https://github.com/major/MySQLTuner-perl)

<comment><USER4> Is there similar tuning script for Postgres?

<comment><USER5> Yes, pg_tune

<comment><USER6> Is there a similar script for Redshift?

</comment></comment></comment></comment></comment><comment><USER7> Two big pitfalls seem to hit almost all of these "low-level optimization of how the database handles..." type posts: the importance in optimizing the system, and the use of other technologies.

Write activity impacts database query caches. Many applications do not require realtime-accurate results from the database and generate a substantial number of the same paginated queries. For these cases, it is very important to consider higher-level caches within the system -- CDN, page-level caches, page block-level caches, etc. as caching your 99% traffic pattern will provide DB platform headroom to support your 1% traffic pattern.

Where the results you are paginating are based on any sort of matching (SQL WHERE), most read applications see a sizable benefit in integrating a search platform. Data selection for display is handled in the search layer, and underlying data retrieval for display happens either from the search layer or via the backing database using inexpensive lookups via primary key.

One of the key considerations not covered in the article is the need for result consistency when paginating, e.g. if the underlying data changes. It is the need for this consistency, not the desire for performance, that I see as the primary reason to include primary key identifiers or timestamp values in your pagination strategy.

<comment><USER8> > include primary key identifiers or timestamp values in your pagination > strategy

And then we got that _one customer_ that somehow backfilled old timestamped data and complained our pagination broke :(

<comment><USER7> Many database platforms support column-level privileges or permissions which could be used to prevent that sort of backfill, by preventing UPDATE on the column that holds the creation timestamp.

I've also dealt with databases without column-level privileges. Where not handled by a column-level privilege system, it may still be possible to block this sort of UPDATE using a TRIGGER designed to fail.

Privilege grant (or drop of the trigger) could be used when the system is in an offline maintenance mode should you require the ability to correct that protected data, reinstituting the control when the maintenance is complete.

The criticality of the data and the level of automation in use would probably be the factors I would use to decide whether this overhead was warranted. Hopefully you had backups available.

<comment><USER8> Well in this case it was timeseries data, so technically supported (we "upsert", and timestamp is simply a part of the composite key).

It occurred when the clock on the hardware got messed up :(

</comment></comment></comment></comment><comment><USER9> You know, if I had access to the documentation of a major DB system, I would put something about this right on the documentation page for LIMIT and OFFSET. It's just _such_ an attractive nuisance; who can blame a novice developer/DB user for making the mistake of thinking that LIMIT and OFFSET are a good solution, really?

<comment><USER10> LIMIT is great, and a necessity on MySQL. It's OFFSET that causes problems. If I were king of databases I'd make OFFSET > 1000 cause an error directing the user to the doc page explaining why what they're doing is a bad idea and how to increase the limit anyway.

<comment><USER11> Offset 1000 won't hurt you if it's a simple indexed scan; MySQL will mess you over far worse if you use a subquery in your where clause, you can be looking at 1M+ rows scanned via the power of nested loops.

I've invested several months of my life optimizing arbitrary sort and filter for MySQL results over tables varying from 100k rows to 100M (using time windows to scope things). As long as you can efficiently cut down the underlying data set into the region of 100k using some kind of window - usually recency based - and convince MySQL to filter by this before it sorts or does any other kind of filter - then limit / offset pagination doesn't hurt.

You can then use higher-level pagination on the recency window if it becomes necessary.

</comment></comment></comment><comment><USER12> It gets worse when you want to allow sorting the paginated results by something other than the integer primary key, such as a name. Duplicate names with different ids means having to use GROUP BY.

I typed out the example queries below from memory, hopefully I haven't made any logic errors. Note:

a) Requires additional indexes to make it efficient.

b) Accounts for inserted and deleted rows.

c) You cannot use page numbers. You can only do first page, next and previous pages, and last page.

d) Example is for 20 items per page. The query pulls 21 (+1) items, so app can determine whether there is a next or previous page.

e) In a web app or api for example, your parameters get crazy, such as: /users?page=next&sort=name&sortId=20&sortVal=George

      -- first page
      SELECT name, id
      FROM users
      GROUP BY name, id
      ORDER BY name, id
      LIMIT 21;

      -- next page (ex: last item on current
      -- page has id 20 and name 'George')
      SELECT name, id
      FROM users
      WHERE name >= 'George' AND id > 20
      GROUP BY name, id
      ORDER BY name, id
      LIMIT 21;

      -- prev page (ex: first item on current
      -- page has id 21 and name 'Harry')
      SELECT name, id
      FROM users
      WHERE name <= 'Harry' AND id < 21
      GROUP BY name, id
      -- need DESC, reverse order in app for display
      ORDER BY name DESC, id DESC
      LIMIT 21;

      -- last page
      SELECT name, id
      FROM users
      GROUP BY name, id
      -- need DESC, reverse order in app for display
      ORDER BY name DESC, id DESC
      LIMIT 21;

<comment><USER13> How about page 9?

<comment><USER12> As mentioned:

> c) You cannot use page numbers. You can only do first page, next and > previous pages, and last page.

To get to page 9, you start on first page. Then "next page" 8 times. This is why you see this kind of UI fairly often. First (<<), previous (<), next (>), last (>>). No numbers.

Technically you can do _relative_ page numbers (ie: current page + 8) by increasing the LIMIT of the query and skipping over results. It's really messy to handle and you can never use "real", absolute page numbers, because inserted and deleted records can change how many next or previous pages there will be.

<comment><USER13> > To get to page 9, you start on first page. Then "next page" 8 times.

Are you sure it will be faster than just using offset/limit on the server side?

</comment></comment></comment></comment><comment><USER14> This page provides a more complete overview of different pagination methods.

"Five ways to paginate in Postgres, from the basic to the exotic"

[https://www.citusdata.com/blog/2016/03/30/five-ways-to- pagin...](https://www.citusdata.com/blog/2016/03/30/five-ways-to-paginate/)

<comment><USER15> And the associated recent discussion (also linked elsewhere here):

[https://news.ycombinator.com/item?id=15446855](https://news.ycombinator.com/item?id=15446855) (Oct 2017, 42 comments)

</comment></comment><comment><USER16> it's kind of crazy that here in 2017 there are still discussions and debates on what exactly is the best way to paginate a large table in a database.

I'm leaning with @jerf that the LIMIT/OFFSET approach can trap you on the 100M record scale, however in all honesty, most of us don't deal with databases at this size. As @alvil stated, his approach works fine for 1M records, which is way more then most personal and application databases will contain.

Maybe the solution is for the database vendor to actually implement and document a definitive solution for this. Maybe re-engineering the LIMIT/OFFSET approach under the hood to take advantage of this new way.

As for the article... I really have a personal problem when authors get lazy and take the easy way out by assuming everyone knows what they are talking about. Take this portion of the article

</comment><comment></comment><comment></comment><comment></comment><comment>

Simplified algorithm:

We get PAGE_SIZE number of records from the table. Starting offset value is 0.

Use the max returned value for user_id in the batch as the offset for the next page.

Get the next batch from the records which have user_id value higher than current offset.

</comment><comment></comment><comment></comment><comment></comment><comment>

The author took the time to write an entire article debating and demonstrating their solution to the whole pagination problem and they couldn't take 5 minutes to show the code behind these steps in their solution?

As I stated, it's just a personal thing.

UPDATE: Down the internet rabbit hole I go. Here is a very nice article similarly demonstrating and debating the LIMIT/OFFSET approach in MSSQL between their OFFSET/FETCH and CTE.

[https://sqlperformance.com/2015/01/t-sql- queries/pagination-...](https://sqlperformance.com/2015/01/t-sql- queries/pagination-with-offset-fetch)

<comment><USER17> Exactly, the "correct solution" would be my first attempt as it uses the (clustered) index scan to count pages/records, it's likely to be the fastest option whereas the first method should not even be an option. This should be immediate for someone who understands how to develop in virtually any RDBMS.

This is also, more broadly, something I've been thinking about recently. It feels (to me) like industry knowledge is getting lost somewhere, because you see people trying to reinvent the wheel in a lot of technical areas. One of them is data: there are proven approaches to most problems faced by the average organization, especially when it comes to designing and managing a data platform. Yet, you see people rolling their own approaches/designs/methodologies to basic things like ingestion, ETL and data modelling when the traditional approaches would suffice and would take half the time to implement and one third of the effort to maintain. It's great that people try to innovate, but what I see regularly is more like people trying to solve a solved problem without bothering to educate themselves on how it was solved so far.

</comment></comment><comment><USER18> Citus also has a good article on different paging methods: [https://www.citusdata.com/blog/2016/03/30/five-ways-to- pagin...](https://www.citusdata.com/blog/2016/03/30/five-ways-to-paginate/)

</comment><comment><USER19> See also: [http://use-the-index-luke.com/no-offset](http://use-the-index- luke.com/no-offset)

<comment><USER13> But how would you build a paginator?

How would you give a link to page 9 when you're on page 1?

<comment><USER20> You can't: _On slide 43 you can also see that keyset pagination has some limitations: most notably that you cannot directly navigate to arbitrary pages. However, this is not a problem when using infinite scrolling. Showing page number to click on is a poor navigation interface anyway—IMHO._

As someone who often skips a few pages when navigating such interfaces, I say "thanks, but no thanks".

</comment></comment></comment><comment><USER21> > In my case with 100 000 000 records, the query is never finished. The DBMS > just kills it. Why? Probably, because it led to the attempt to load the > whole table into RAM.

What? Why? It should just instantiate a cursor and return the first batch size worth of records.

I'm not sure why the dB pulls load the entire result set into ram to return results unless it requires a sort on a non indexed field.

<comment><USER22> Mysql would fall back to a disk based sort once the data set is too large.

Perhaps it's his client that is OOM-ing? Back in the day, I remember the PHP mysql client loading the entire result set into a local buffer unless you explicitly asked for the output to be streamed.

Edit: Found reference to PHP behavior I mentioned: [http://php.net/manual/en/mysqlinfo.concepts.buffering.php](http://php.net/manual/en/mysqlinfo.concepts.buffering.php)

</comment></comment><comment><USER23> Similar article from 5 days ago: [https://news.ycombinator.com/item?id=15446855](https://news.ycombinator.com/item?id=15446855)

<comment><USER24> Thanks, I missed this!

</comment></comment><comment><USER25>
      > You need to walk thru the table,
      > extract each record,
      > transform it inside your application's code
      > and insert to another place.

The writer said his database died in the middle of the query when he selected the 100,000,000 rows without first breaking them into pages.

It looks like he's using MySQL. I've used PostgreSQL for over a decade, and I just don't see this happening, though I've tested only on a table with a few million rows. But anyway I doubt PostgreSQL loads the whole table into memory when you select it. It seems to load just parts at a time, as needed. And for what it's worth, an offset of a few million rows still took just a few seconds.

Actually what I would first try to is dump the table to a file, transform it if possible with Linux command-line tools like sed and awk, and then load the file into the new table.

Can anyone confirm if Postgres would die like MySQL did on too big a dataset --- selecting everything without paginating but fetching just one row at a time in the application?

<comment><USER26> > I've tested only on a table with a few million rows

I've tested postgres with > 250m rows. It has problems. I've also tested to 1b rows, same issue.

Now throw in some sum, count and group by functions, more issues.

Add a join? Now you've major problems. Add several joins? Woah there!

I've been able to bring a 5 node citusDB cluster to it's knees. Which the only solution was to scale out massively to double digits servers. But you can't do that on a limited budget.

The caveat here, this was about 2 years ago now. I don't know if there have been any improvements since that time.

I don't have a postgres database now, I've since migrated over to memsql as the majority of my work is OLAP.

<comment><USER27> I run a 270m rows database for my IRC bouncer, and run fulltext searches over it all the time, with many joins, expensive ranking, etc. On a single machine.

I've never been able to bring PGSQL to its knees.

</comment></comment></comment><comment><USER28> Related (2009) : [https://www.slideshare.net/suratbhati/efficient- pagination-u...](https://www.slideshare.net/suratbhati/efficient-pagination- using-mysql-6187107)

</comment><comment><USER29> That's great if you need to do some kind of crawl through the DB for background work, but fails to support key elements (filtering, non-primary-key sorting) that are probably hard requirements at the API level.

<comment><USER30> You can still support arbitrary filters and ordering, as long as you always sort by the ID last. The nice thing is the queries actually get more and more efficient as you approach the last page. (The normal approaches have the same cost across all pages, or even get more expensive toward the latter pages.)

The only thing this approach does not support is jumping to an arbitrary page number.

</comment></comment><comment><USER31> I wonder if an indexed "PageNumber" column could work? Obviously wouldn't work for arbitrary queries, but if you had a single, commonly run query it could be used to speed up that one...

</comment><comment><USER32> I'd really go with the first solution. However I'm usually using Postgres. And of course it would need to be done in batches at the application side.

</comment><comment><USER33> Can u share a real-life use case for this problem, even in google, folks don't get past the 5th page. Last five can be accessed by reverse sorting.

</comment><comment><USER34> <sarcasm>And here I thought clustered index and a OFFSET was the solution</sarcasm>

</comment><comment><USER35> On the frontend would one store page=1,page=2 or page=user_id(10000th),page=user_id(20000th) ?

<comment><USER36> With this approach you can't have page numbers on the front-end, just first, prev, next and maybe last. You can't jump to particular page. It is old technique and I like it, but if it would be usable it would be used by everyone but it is not.

<comment><USER37> What is the purpose of having pages numbers and jumping to a particular page?

<comment><USER13> For example you have a forum post with 1000 messages starting in 2007, and you are dying to know what happened in the discourse in 2011. So basically you need to jump in the middle by bisection.

1000 is tiny by DB standards but solutions like this one are often bolted without considerations.

<comment><USER37> That's a great example. I understand the use case now.

I note this use case can be solved using keyset pagination instead of offset/limit, because the ordering of messages is stable: messages are never inserted in the middle of the list; they are appended at the end. We can attribute a number to each message, and use this message to filter and sort.

<comment><USER13> But what if a message gets deleted from middle of the list? This happens with forum messages.

<comment><USER37> You keep it and mark it "deleted", and you show "deleted" in place of the message. This is what most forums do already, independently of the underlying pagination mechanism.

<comment><USER13> So basically I'm pretty sure now that we need a new kind of SQL INDEX, as in, pagination index.

To do that aforedescribed accounting for us, and more.

<comment><USER37> It wont't be a new kind of index. It would be a new interpretation of OFFSET and LIMIT, which would ignore deleted rows.

<comment><USER13> Why would it? It will be rebuilt when a row is deleted or even marked deleted.

If we have an pagination index over (topicId, deleted) on postTime asc.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER38> storing the max id returned from the batch for use as an offset for the next batch is something I've done ever since I even started programming - is it supposed to be revolutionary advice?

</comment><comment><USER36> I usually use query similar to this one (in PHP/MySQL), it's much faster than traditional offset query. You can compare it for yourself using EXPLAIN. Sure not for 100M records, but for 1M or so :)

      $items = db("SELECT a,b,c FROM item
               JOIN (SELECT id
                  FROM item
                  WHERE
                  <your-where-conditions>
                  ORDER BY id DESC
                  LIMIT <offset>, <limit>)
                  AS x ON
                  x.id = item.id");</comment>