Intel has released new CPU microcode for download - <AUTHOR> https://downloadcenter.intel.com/download/27431/Linux-Processor-Microcode-Data-File <comment><USER3> Despite the name, this is not Linux-specific microcode. If you're running SmartOS or another illumos derivative, run ucodeadm(1) on the microcode.dat file (which will need to be renamed to have an "intel" prefix per the man page -- e.g., "intel-code.txt"). You can then run "ucodeadm -v" to validate that the new microcode has been loaded. (Note that this does not persist across a reboot, but we at Joyent are currently in the process of upstreaming the microcode into illumos, at which point it will.)

<comment><USER4> And slow down my computer? No thanks.

<comment><USER5> A microcode update is applied directly to the processor, typically during the boot process, and is a common practice to fix CPU bugs. It changes the microcode running on the hardware itself.

This is totally different from the kernel PTI fixes, which attempt to deal with Meltdown by changing the way the operating systems for Linux/Mac/Windows/DragonFly (and hopefully soon *BSD) interacts with the hardware and page tables.

Also you're making a statement without the above comment even stating if this update is designed to address Spectre or Meltdown.

<comment><USER6> What he said isn't wrong, this new ucode changes fencing and branching prediction semantics regardless of using the instructions added for spectre.

The problem is varied. While most people should run the new ucode and pending kernel and toolchain fixes, not all must. Most businesses buy computers on rated performance, and they are about to take an unexpected performance haircut. Intel ucode isn't really optional, as there is no public change log and it is unknowable what critical stability fixes are within. so even in a closed system you can't selectively ignore it without some amount of negligence. This presents a big problem for HPC and other closed systems.

IBM, a more professional and customer focused HPC vendor, is offering a firmware flag to preserve pre-spectre behaviors. Most closed monolithic HPC clusters can ignore these vulns and the associated nvidia one.

<comment><USER5> I wonder if this will end with Intel selling a lot of new chips, or Intel having to issue massive recalls. I doubt Intel will be able to be able to get away with discount/trade-ins without some serious legal complications.

I doubt we'll see entire data centers move to AMD as that would get stupid expensive. For the short term, we might see data centers order extra AMD blades and changing their provisioning systems to pin high IO tasks to those machines. And that's for people who co-locate or host their own stuff.

I'm interested to see how this affects Infrastructure providers who abstract a lot of that away, like AWS, DigitalOcean, Rackspace and others.

They might even be working on analysis and auto-migration to move high IO nodes to higher performance machines transparently. It will be interesting to see what hosting providers start blogging about in the next few weeks.

<comment><USER6> The first would set a strange precedent without some kind of good faith amends by intel, and so far they have signaled apathetic callousness. I think the legal implications haven't even begun to arise yet because large users are still scrambling to understand the impact and deal with what they have on hand. I speculate some kind of recall or prorate program will come out of it in time.

Dieselgate is the closest thing I can think of recently where a product was marketed primarily on performance that was later rescinded through mechanical and/or software clamping. I wonder if there will be any FTC action for consumers.

Separate from the above thoughts, intel was already testing customer patience with Skylake. It wasn't a substantial performance increase, but it was a substantial pricing increase. A lot of that, because, in merging the 4S (E5-4xxx) and 8S (E7) into one marketing line, a majority of users are paying big premiums for unwanted UPI scalability to get clock speed, cores, and caches they do want. They also gambled poorly on clamping I/O lanes to fight GPU, and with omnipath which is a niche at best. History will most likely judge Krzanich poorly. I expect AMD and IBM will have a good couple years coming up.

</comment></comment></comment></comment></comment></comment><comment><USER7> Intel, AMD and Via microcode is being archived by CPUID, by the user community: [https://www.win-raid.com/t3355f47-Intel-AMD-amp-VIA-CPU- Micr...](https://www.win-raid.com/t3355f47-Intel-AMD-amp-VIA-CPU-Microcode- Repositories.html)

 _" Collecting all available Production CPU microcodes is important for upgrading/downgrading purposes, for creating universal tools that can help people understand what microcode they use, for research on how the general technology works, for developers with no vendor representative who want to experiment on a given platform etc.

Disclaimer: All the microcodes below come only from official BIOS/UEFI updates, Intel Linux Microcode Updates, Linux Distributions, Windows Updates etc which were provided and made public by various manufacturers! It is always advised to request and/or wait for your OEM/OS to release newer fixes. The microcodes are gathered and provided with the sole purpose of helping people who are out of other viable solutions. Thus, they can be extremely helpful to those who have major problems with their systems for which their manufacturer refuses to assist due to indifference and/or system age."_

<comment><USER8> I always assumed they were not just signed but encrypted, and padded to avoid meta analysis. Is that not the case?

<comment><USER9> This recent talk shows that AMD ucode wasn't signed _or_ encrypted up until 2013, and they reverse engineered large portions of the structure.

[https://www.youtube.com/watch?v=lY5kucyhKFc](https://www.youtube.com/watch?v=lY5kucyhKFc)

</comment></comment></comment><comment><USER10> The linked Intel page is a mess.

It is titled "Linux* Processor Microcode Data File Version: 20180108 (Previously Released) Date: 1/8/2018"

Below, however, you can see _highlighted_ "A newer version of this software is available. Click here to get the latest version of this software." This, though, points to an earlier ("Version: 20171117 (Latest) Date: 11/17/2017") version: [https://downloadcenter.intel.com/download/27337/Linux- Proces...](https://downloadcenter.intel.com/download/27337/Linux-Processor- Microcode-Data-File)

(one hour later) UPDATE: appears to be fixed. Well, not really, only if the link specifies the product, i.e. [https://downloadcenter.intel.com/download/27431/Linux- Proces...](https://downloadcenter.intel.com/download/27431/Linux-Processor- Microcode-Data-File?product=873)

</comment><comment><AUTHOR> To update the intel-ucode package to the system:

\- 1. Ensure the existence of /sys/devices/system/cpu/microcode/reload

\- 2. Copy intel-ucode directory to /lib/firmware, overwrite the files in /lib/firmware/intel-ucode/

\- 3. Write the reload interface to 1 to reload the microcode files, e.g. echo 1 > /sys/devices/system/cpu/microcode/reload

<comment><USER11> don't forget to update your initramfs, else on reboot it won't be applied

on debian: update-initramfs -u

check with dmesg | grep microcode

<comment><AUTHOR> Thanks, and on RHEL/CentOS/Fedora:

      dracut -f

</comment></comment></comment><comment><USER12> FWIW, microcode is now included the patch that VMware released today.

Gonna go test it out now...

PSA: VMs have to be cold booted after patching and set to HW v11+ for PCID support

EDIT: Just fired up my first Windows VM after patching ESXI and I'm now showing all green using the PowerShell script.

Here's the link that I'm referring to: [https://www.vmware.com/us/security/advisories/VMSA-2018-0004...](https://www.vmware.com/us/security/advisories/VMSA-2018-0004.html)

<comment><USER13> Yikes. So they refuse to vMotion to a host that's on the new microcode? I'm not even clear on how that would be supported -- it should Just Work if the host is in the cluster. Are hosts unable to rejoin a cluster after rebootign with this new microcode because they're effectively part of a different processor compatibility now? Does EVC affect how this impacts the ability to vMotion?

<comment><USER14> The problem (requiring the cold reboot of the VM) is that they masked the PCID bit off in the CPUID, so the VM thinks it is running on a CPU that doesn't support PCID. Obviously they've stopped doing this with the new update, but you have to cold reboot (not warm) for the newly enabled feature to be visible.

Presumably they did this to make it possible to hot migrate VMs between old hosts that don't support PCID, and new ones that do. Which, with hindsight, was not a good tradeoff.

<comment><USER15> Given that Linux (and most other VM guest OSes of note) support CPU hotplug, shouldn't vMotion have just given the VM a new, second CPU with PCID, and then removed the first one?

<comment><USER14> Given the history of CPU hot plug, it has always been generally assumed that all the CPUs in an SMP are identical (hence the S in SMP). I'm not aware of any OS that supports differing feature sets, and if one exists, it almost certainly goes with the least common denominator feature set.

</comment></comment></comment></comment></comment><comment><USER16> Literally tells us nothing about what's in it. Not even a changelog. Not even a sentence hinting as to what might be in it. Incredible.

<comment><USER17> > Literally tells us nothing about what's in it. Not even a changelog. Not > even a sentence hinting as to what might be in it. Incredible.

They do have ./releasenote:

      Intel Processor Microcode Package for Linux 20180108 Release
      -- Updates upon 20171117 release --
      IVT C0            (06-3e-04:ed) 428->42a
      SKL-U/Y D0        (06-4e-03:c0) ba->c2
      BDW-U/Y E/F       (06-3d-04:c0) 25->28
      HSW-ULT Cx/Dx     (06-45-01:72) 20->21
      Crystalwell Cx    (06-46-01:32) 17->18
      BDW-H E/G         (06-47-01:22) 17->1b
      HSX-EX E0         (06-3f-04:80) 0f->10
      SKL-H/S R0        (06-5e-03:36) ba->c2
      HSW Cx/Dx         (06-3c-03:32) 22->23
      HSX C0            (06-3f-02:6f) 3a->3b
      BDX-DE V0/V1      (06-56-02:10) 0f->14
      BDX-DE V2         (06-56-03:10) 700000d->7000011
      KBL-U/Y H0        (06-8e-09:c0) 62->80
      KBL Y0 / CFL D0   (06-8e-0a:c0) 70->80
      KBL-H/S B0        (06-9e-09:2a) 5e->80
      CFL U0            (06-9e-0a:22) 70->80
      CFL B0            (06-9e-0b:02) 72->80
      SKX H0            (06-55-04:b7) 2000035->200003c
      GLK B0            (06-7a-01:01) 1e->22
      <snip>

But I'm not sure how to interpret it... BDW is probably Broadwell and HSW is probably Haswell etc., IVT is... interrupt vector table? and I'm not sure what the other characters are.

<comment><USER18> The characters after the shorthand are probably chip revision.

{HSW,BDW,SKL,KBL,CFL} are probably Haswell, Broadwell, Skylake, Kaby Lake, Coffee Lake.

{HSX,BDX,SKX} appear to be the codenames for some variants of Haswell/Broadwell/Skylake - I find mixes of references to those being the Xeons, associated chipsets, and otherwise.

GLK appears to be Gemini Lake, a codename for some of Intel's really low power SoCs.

<comment><USER19> Skylake-X?

<comment><USER20> SKX = Skylake-X = Skylake Xeon, as distinct from SKL, which is the "client" die. The most important distinction is that SKX has AVX 512 and SKL does not.

</comment></comment></comment></comment></comment><comment><USER21> Looks like they updated the microcode for every single processor they ever released, going all the way back to the Pentium.

The list includes: The Pentium 4, Pentium III, Pentium II, Pentium Pro and the original Pentium.

I didn't even know the Pentium had upgradable microcode.

<comment><USER22> I thought you were joking and had a close look at the list. Turns out you're absolutely right!

Intel® Pentium® Processor 100 MHz, 50 MHz FSB Intel® Pentium® Processor 120 MHz, 60 MHz FSB Intel® Pentium® Processor 150 MHz, 60 MHz FSB Intel® Pentium® Processor 75 MHz, 50 MHz FSB Intel® Pentium® Processor 90 MHz, 60 MHz FSB

<comment><USER23> I wouldn't be so quick to say that.

it's the cumulatively package including all microcodes. Thus it applies to the listed CPUs.

Without having a look, you cannot say which one got an update and which CPU didn't.

<comment><USER24> Practically though, I downloaded the microcode from the link for my 2009 era i7 920 Family 6 Model 1A stepping 5 CPU (Bloomfield) and applied it to my AMI BIOS and the tool gave the date as 2013 AND after rebooting the Powershell Get-SpeculationControlSettings command shows I do not have the hardware microcode support.

For those interested I used the instructions here -> [https://www.delidded.com/how-to-update-cpu-microcode-in- ami-...](https://www.delidded.com/how-to-update-cpu-microcode-in-ami-bios/) and to get my family/model/stepping I used [https://www.intel.com/content/www/us/en/support/articles/000...](https://www.intel.com/content/www/us/en/support/articles/000005651/processors.html)

</comment></comment></comment></comment><comment><USER25> Anybody know how to make this work in OSX, for those of us who don't want to "upgrade" to the train wreck that is High Sierra?

<comment><USER26> Does High Sierra update the microcode of the CPU?

For Sierra, a patch is likely coming; there is "Security Update Developer Beta 2018-001" which you can get using their beta program.

<comment><USER27> What about those of us still on 10.11? That one is horrible already but the horror stories I see on a daily base from colleagues with 10.12/10.13, thanks but no thanks... I'm not a free QA engineer for Apple.

<comment><USER28> I was under the impression that the recent security update for El Capitan 10.11.6 contains required fixes. I am doing a clean install and upgrading all the way and check the PoC for any red flags. I will update here my findings. +1 for not upgrading to High/Sierra.

Update: According to [https://support.apple.com/en- us/HT208331](https://support.apple.com/en-us/HT208331) the Security Update 2017-005 contains fixes for Meltdown.

Update 2: According to [https://support.apple.com/en- us/HT208403](https://support.apple.com/en-us/HT208403) the Safari Update 11.0.2 contains fixes for Spectre.

So folks on 10.11.6 El Capitan should be good.

<comment><USER29> From the first link, the Meltdown fix is only for High Sierra:

Available for: macOS High Sierra 10.13.1 Impact: An application may be able to read kernel memory (Meltdown)

</comment></comment></comment></comment></comment><comment><USER30> As someone living/fighting through the performance degradation that the kernel patches have done to IO loads at AWS, I'm wondering if their Xen patches have had this or may include it in the future.

Would really really rather that there were no more negative changes.

<comment><USER31> Some of the hypervisor fixes were dependent on the new microcode

</comment></comment><comment><USER32> Downloads for Intel® Xeon® Processor E5-2630 (15M Cache, 2.30 GHz, 7.20 GT/s Intel® QPI)

</comment><comment><USER33> Tested rev 0x042a on my Xeon E5-2658 v2 on my Asus X79-Deluxe mainboard. Flashed by using UBU v1.69.10 run Ashampoo SpectreMeltdownChecker both vulnerabilities are green with a tick now. It say's "your processor is safe" but release date of the patch is 2017-12-01 and goes up to 2017-11-16 which means, they know about this bug at least the beginning of November 2017 and they hide it about 2 months. What to think about that?

</comment><comment><USER24> I downloaded this microcode from the link and got the latest BIOS for my Asus motherboard. I found my CPU family/model/stepping using [https://www.intel.com/content/www/us/en/support/articles/000...](https://www.intel.com/content/www/us/en/support/articles/000005651/processors.html)

I used a tool and instructions from [https://www.delidded.com/how-to-update- cpu-microcode-in-ami-...](https://www.delidded.com/how-to-update-cpu- microcode-in-ami-bios/) to update my AMI BIOS. It recognized the microcode and added it to the list. Unfortunately, the tool showed my microcode as having a date of 2013 even if it was listed and included in the download. So I have to wait for updated microcode I assume. I flashed it anyway. Post boot tests showed I was not protected by microcode... I have a Bloomfield CPU.

This technique will probably work for someone else with a newer CPU.

</comment><comment><USER34> Wish Intel would say if this does anything useful without also updating the Kernel, or do you have to have both.

<comment><USER35> I presume you would need to do both, since there are multiple attack vectors. Some problems like meltdown can be mitigated via how the kernel handles page table's when switching to kernel mode, while some problems like spectre's branch manipulation need microcode updates.

</comment></comment><comment><USER36> I opened a request on Intel's Community Site requesting microcode changelog documentation. Hopefully I'll get an answer.

[https://communities.intel.com/message/518872#518872](https://communities.intel.com/message/518872#518872)

Please do click the "I have the same question" button on that thread. Hopefully if it gets enough heat Intel will answer our questions.

As best I can tell at the moment these changes are necessary in order to enable the IBRS and IBPB changes that are currently being discussed on LKML. AFAIK, IBRS and IBPB mitigate against Spectre V2 performance regressions that are introduced with the retpoline changes.

</comment><comment><USER37> Does AMD also need some microcode update for this?

<comment><USER14> [https://www.amd.com/en/corporate/speculative- execution](https://www.amd.com/en/corporate/speculative-execution)

They claim no, due to microarchitectural differences in their branch predictor behavior. This is variant 2 we're talking about here.

<comment><USER37> So I guess just kernel updates for Spectre are needed.

<comment><USER14> Supposedly just for 1 of the 2 main variants of Spectre.

Though there are many likely more timing channels that could be reasonably called Spectre variants that have not yet been fully explored...

</comment></comment></comment></comment><comment><USER38> Odd, this doesn't appear to apply on my 2500K even though it's listed as "valid for this product":

      [    0.000000] microcode: microcode updated early to revision 0x29, date = 2013-06-12

Works fine on my 7200U though.

</comment><comment><USER39> Is there any meaningful way to know what the differences are between previous microcodes?

<comment><USER40> Only what Intel provides, the updates are AFAIK encrypted.

<comment><USER41> And no reverse engineering possible?

I'm a bit surprised there is no enthusiast blog that tries to document intel microcode changes, but maybe I just haven't found it yet.

<comment><USER42> This might be interesting reading for you: [http://inertiawar.com/microcode/](http://inertiawar.com/microcode/)

Signed with 2048-bit RSA, and probably encrypted too.

<comment><USER41> Yes! I love it, even if there were meager returns; basically identifying the algorithms for encrypting/signing the blob and that's all.

</comment></comment></comment></comment></comment><comment><USER43> If they can fix it in microcode, why did they have to patch the kernel?

<comment><USER44> You need both.

According to [https://access.redhat.com/articles/3311301](https://access.redhat.com/articles/3311301):

> CVE-2017-5715 (variant #2/Spectre) is an indirect branching poisoning attack > that can lead to data leakage. This attack allows for a virtualized guest to > read memory from the host system. This issue is corrected with microcode, > along with kernel and virtualization updates to both guest and host > virtualization software. This vulnerability requires both updated microcode > and kernel patches. Variant #2 behavior is controlled by the ibrs and ibpb > tunables (noibrs/ibrs_enabled and noibpb/ibpb_enabled), which work in > conjunction with the microcode.

</comment></comment><comment><USER45> CPU: Intel(R) Xeon(R) CPU L5639 @ 2.13GHz

Not listed :/

<comment><USER46> I'm running freenas on a Xeon. I'm assuming i am going to have to install something from supermicro and wait for a freenas (and freebsd) update. Is there anything else I should update?

<comment><USER47> Similarly, my primary (personal) server is an X10SL7-F with an E3-1231v3 (running FreeBSD). I'm waiting around for Supermicro but I would suggest not holding your breath.

On second thought, I suppose the same goes for Asus with regard to my main workstation (Z10PE-D16/WS with a pair of E5-2620v4's).

My ThinkPads are much older (T420/W530) and I'm not really expecting anything for them from Lenovo -- especially any time soon, although they do put out the occasional BIOS update for 'em.

<comment><USER45> You don't need to wait for BIOS updates to install microcode - <USER2> sysutils/devcpu-data and sysrc microcode_update_enable=YES

</comment></comment></comment></comment><comment><USER48> no debian 9?

<comment><USER49>
        apt-get install intel-microcode

[https://wiki.debian.org/Microcode](https://wiki.debian.org/Microcode)

then follow the "releasenote".

<comment><USER50> I was just explained the other day on Hacker News how CPU microcode gets delivered with the Kernel. That it gets installed automatically on every boot. Why is it a separate package (which it turns out I don't have).

<comment><USER35> Well it is a separate package because it is fundamentally independent of the kernel. For example, your Debian system might want to use a different kernel like GNU's Hurd, kFreeBSD, or NetBSD, so by keeping those packages separate, they can easily be used interchangeably. Also if you are on an AMD system, you wouldn't want Intel microcode, but you might still want the same Linux kernel.

Also another big issue issue is the Intel microcode is proprietary, so separating it from the kernel means that user could selectively pick and choose if wants to have a totally free system, which would mean not loading microcode updates with a libre kernel. This is done for instance with Parabola and Trisquel distros, which is needed to obtain FSF's totally free certification.

<comment><USER50> Does this mean that my Ubuntu installation runs stock Intel microcode? It means I actually have to know about this, and then go out of my way and know how to install it? Or should it be installed by default on Ubuntu? Because the package called `intel-microcode` isn't installed for me now.

</comment></comment></comment></comment></comment><comment><USER51> HN really needs a _sticky_ feature for comments concerning security patches and other updates that can bork your machine.

In this case, someone who dreams in hardware, breathes ASM and talks in bytes, needs to clearly inform the community here concerning these questions:

\- SHOULD THIS MICROCODE UPDATE BE PERFORMED SEPARATELY FROM RUNNING: apt-get update && apt-get upgrade ?

\- WHAT IS THE IDEAL/BEST WAY TO PERFORM THIS MICROCODE UPDATE?

<comment><USER52> I like how HN's adherence to a certain simplicity has, I think, helped keep the community integrated, and relatively egalitarian and respectful (with a lot of other work going into that, as well, I'm sure).

But there have been a few times, this year and recently, where very pertinent security issues have had threads of both immediate and enduring value, with information -- mostly in the comments -- both useful and not available elsewhere that I've seen, online.

Maybe another one of HN's select, few categories. E.g. "essential". Probably populated solely at the moderators' discretion. In that, I'm in favor of the benevolent dictators model: Maybe some polite and well-argued comments about what might belong, but no voting or manipulable -- technically nor socially -- as to what gets in there.

When a processor, platform, OS is significantly borked, and the knowledge is essential to a broad portion of this community. That would be what goes in there. Starts as regular threads. If the need to know and value of them are high enough, they get tagged with that category. So that, e.g. I can more readily find that Intel Management Engine thread a couple of months later, when I'm deciding whether I want to patch (or patch further) and what mitigations to keep in place.

I don't know, and maybe I'm wrong. Just an idea.

P.S. I don't know whether the front page would link "essential" somewhere, or whether it would be like some other qualifiers, that don't have a front page presence. Again, the simplicity of the front page, versus the value of the information and the need to know.

P.P.S. I really am afraid, though, of the arguments its presence might engender, as to what belongs in it, and the disharmony this might introduce and foster. There's a LOT of value to the existing simplicity (of the interface, if not always the elephant behind it).

<comment><USER53> You can make this page! Start curating HN posts and comments and the other resources relative to your essential needs and see what magic falls out

</comment></comment></comment><comment><USER54> Does this release has anything to do with Linux P page table isolation patches ([http://pythonsweetness.tumblr.com/post/169166980422/the- myst...](http://pythonsweetness.tumblr.com/post/169166980422/the-mysterious- case-of-the-linux-page-table))?</comment>


A method for improving Milky Way exposures in light pollution - <AUTHOR> https://www.lonelyspeck.com/the-milky-way-in-los-angeles-light-pollution/ <comment><USER4> > _Shouldn't the image taken at ISO 6400 show more noise than the image taken > at ISO 1600? If the exposures were the same brightness, yes. But these > images were not taken with the same exposure. The first image started with > twice as much shutter open time (30 seconds versus 15 seconds) and two stops > more gain on the sensor (ISO 6400 vs ISO 1600). The result is that the first > image has significantly more light data. This makes the final signal-to- > noise ratio of the processed images higher on the first image. The higher > the signal to noise ratio, the less noisy the resulting photo. So in this > case it was actually better to overexpose and compensate later in post > processing, despite the initial unprocessed images appearing unusable._

This is really neat. I've got a full frame camera with a sensor that has very good dynamic range and I'd learned to under-expose and then pull detail from the shadows in post processing as information can't be saved from blown highlights. This sort of flips that on its head, except the featured overexposed shot didn't have blown highlights, it just looked like it did. Instead it had a wealth of low-light data.

<comment><USER5> What I've learned over the years with digital cameras -- you can't trust the preview image. The histogram tells much more of the story (if not the whole truth). My photographs have become much better (in a technical sense) after I learned to expose almost to the point of blowing highlights, making sure there is little blank space at the right (where the highlights are displayed) of the histogram as possible (hence the term Expose To The Right). As someone downthread stated, this improves the signal-to-noise ratio of the captured data. There's an unwritten assumption that the camera's built-in auto-exposure capability isn't sophisticated enough to expose the scene properly, and the photographer has to take control. This is often the case.

And speaking of SNR, there are a few tricks astro-photographers have been using for years that occasionally find their way into mainstream photography. One is image stacking, where you combine multiple photographs of the same scene (useful for landscapes if a little blur isn't a problem for you) into a single image. This pushes the shot noise (the random distribution of noise) down even further. I've gotten clean images out of cheap cameras, and even from cheap cell phone cameras (given enough images).

The other technique is dark frame subtraction, where you take an photo with the lens covered at the same ISO/shutter/aperture as the rest of the set. Digital sensors have readout noise and heat noise which tends to be very predicable. Subtracting this noise gives you a cleaner shot as well.

On a side note, I do wish when talking about ETTR and even digital photography in general that the term "overexpose" wasn't used when explaining the technique of overriding the cameras default metering with exposure compensation. Overexposure would be blowing the highlights, or at least the highlights of details that the photographer considers important (e.g. blowing the highlights of street lights is usually okay). The "over" portion makes it sound like the photographer is doing too much of something, when in reality ETTR is a technique of giving enough exposure, but not too much.

<comment><USER6> > One is image stacking

Since you seem to have experimented with this, I'd like to ask: don't you need a certain "minimum" time to capture the faintest of stars, because their signal otherwise wouldn't rise above the noise floor? I've been curious about this with stacking in atrophotography for a while.

> The other technique is dark frame subtraction, where you take an photo with > the lens covered at the same ISO/shutter/aperture as the rest of the set. > Digital sensors have readout noise and heat noise which tends to be very > predicable. Subtracting this noise gives you a cleaner shot as well.

For long exposures this heat noise becomes more problematic: long exposures heat up the sensor quite a bit. Have you tried building your own customised electric coolbox to keep temperatures down? It can be a fun project!

I actually used this in a different context: during long Skype sessions at home with my people abroad, my phone heats up _a lot_ and the video would start stutter. Putting it on a cooling block actually fixes that. This had more to do with the system not having to scale back CPU performance to let the phone cool down, but a side-effect according to people on the other end of the line, was that the image quality improved too

<comment><USER7> > don't you need a certain "minimum" time to capture the faintest of stars, > because their signal otherwise wouldn't rise above the noise floor?

I'm not an expert or even very knowledgeable on image stacking, but I'll share my experience.

I've taken a stacked picture of the Orion constellation. 15 exposures at 5 seconds each, and combined them to get this photo:

[https://www.flickr.com/photos/abliskovsky/31756623694/](https://www.flickr.com/photos/abliskovsky/31756623694/)

In none of the individual exposures is the Orion Nebula visible, but in this photo it's pretty clear. So in the end I'm fairly sure that your statement is true, but the signal required is very, very faint.

<comment><USER8> Do you know of any good resources for "getting into" image stacking for a complete newbie? 2 years ago I tried stacking images I took of Jupiter in my 16" dob (taken with a Canon DSLR), but it was terrible and I'm not confident I used the app I found (I _think_ it was AstroStack..) for image stacking correctly.

<comment><USER9> RegiStax is popular for solar and planetary imaging. I've used it several times on H-alpha solar photography. It has a fairly well defined workflow and there are lots of video tutorials on how to use it.

<comment><USER8> Awesome, I will check it out. Thanks! Also looks like it runs under Wine on Linux, which is good news for me!

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> While idly thinking about light pollution a while ago, I thought of a ridiculously impractical solution:

Mandate the replacement of all outdoor night-time illumination with LEDs that are pulse-width modulated at a low duty cycle. Synchronize them all to an accurate global clock (e.g. from a GPS receiver), so that for instance, all of the lights are simultaneously turned on for the first tenth of each UTC millisecond. Then an image sensor with a sufficiently fast global shutter could disable itself during every brief pulse of light, so that it picks up 90% of the incoming starlight, but almost none of the light pollution.

<comment><USER11> Fix the duty cycle to a particular slice of the phase of AC voltage (e.g. the first 0-pi/2) and you accomplish the same thing much more practically, at least within a single power network.

<comment><USER12> It's going to be a bit trickier than that, because power is usually distributed as 3-phase, not single-phase.

So that's 3 different offsets, depending on which phase any given light is wired into.

Plus that's probably going to be slow enough to cause visible flicker.

<comment><USER13> Have it trigger on a voltage threshold. 3 phase means you're hitting 0V 6x per cycle so that'd bring the pwm up to 720Hz.

</comment></comment></comment></comment><comment><USER14> If you're into deep space astrophotography, this is not really news.

It's all about exposing for the stuff you care about. When imaging DSOs (deep space objects), you often don't care about the relatively bright stars nearby. But you do care about that faint little object in between. So you adjust exposure so the stars are usually blown out (overexposed), but the DSO along with the light pollution background have peeled off the left hand edge of the histogram.

So now your DSO is sitting comfortably in the middle of the histogram, where you can process and extract it easily.

In practical terms, for DSOs you tweak exposure until the big light pollution hump on the histogram becomes disconnected from the left hand edge of the graph. Turn exposure up gradually until there's a gap at the left hand wall. That usually does it. Overexposure is bad, but underexposure is super-duper- evil, so you must avoid it. OTOH, don't push things too far to the right - <USER2> still want some chroma info in the star halos.

</comment><comment><USER15> Would it be possible to filter out the pollution by subtracting an out of focus image from the in focus one? The pollution is already spread out while the stars are point light sources. So wouldn't that leave the stars untouched while greatly reducing the light pollution?

<comment><USER16> All you need to do is make a copy of your image, blur that copy enough to get rid of the stars, and then subtract it from the other image. It's going to affect larger structures like the milky way, though.

</comment></comment><comment><USER17> Overexposing and underexposing (formally known as Bracketing) is typically a great technique in landscape photography too because it captures as much of the dynamic range as possible. Most DSLRs can bracket right out of the box. Set it up to take 3 exposures (1 underexposed, 1 normal and 1 overexposed) with about 2 or 3 stops in between. Then merge all 3 exposures in Photoshop and you'll get an image with a TON of information embedded in it for you to tweak in post-processing.

Also, while I have't tried this, I'd think that using an ND filter in light- polluted areas like this could help a little bit with astrophotography.

<comment><USER18> There are light pollution reduction filters made specifically for astrophotography. I was shocked at how good these photos were w/o using such a filter. I don't think you ever could have recovered so much of milky way using film w/o a filter in such a polluted sky. (I was certainly never able to.)

Googling for an example, amusingly enough, the first link is to the same site:

[https://www.lonelyspeck.com/hoya-intensifier-review-an- affor...](https://www.lonelyspeck.com/hoya-intensifier-review-an-affordable- light-pollution-filter-for-astrophotography/)

That filter is not specifically for astrophotography, just one that happens to work well. Here's a discussion of other options:

[https://www.cloudynights.com/topic/494162-light-pollution- fi...](https://www.cloudynights.com/topic/494162-light-pollution-filter/)

</comment></comment><comment><USER19> Here's something I've wondered about a lot, but haven't been able to find a definitive answer: Isn't ISO pretty much software gain control? Why ever increase the ISO on the camera instead of just taking photos at the native ISO and then post-processing the photo to a higher exposure? That way, you don't store the blown-out pixels (if you do overexpose).

<comment><USER20> ISO is not just software gain. On a lot of higher-end cameras (read: just about any ILC) there's a definite analog gain stage prior to ADC.

Since there's a fixed level of sensor noise, when you increase the analog gain you decrease the level of noise in the shadows relative to the sensor noise, essentially increasing SNR. This isn't true for all cameras, however.

This chart [[http://www.photonstophotos.net/Charts/PDR_Shadow.htm#Canon%2...](http://www.photonstophotos.net/Charts/PDR_Shadow.htm#Canon%20EOS%206D,Nikon%20D750,Sony%20ILCE-7RM2) ] demonstrates the difference between three cameras, one of which is known to be largely "ISO invariant" (the Nikon D750), one which is not (the Canon 6D), and one which looks like it has a 'native' ISO of 800 (the Sony A7R II)

As you can see, the D750 doesn't gain much (heh) from increasing ISO -- the total image noise in the shadows isn't improved by increasing the ISO through the range. If you're shooting a 6D in a low-light situation, however, you see improvements in shadow quality up to about ISO 3200, after which it turns out that you will be fine if you just raise the shadows in post. If you're shooting an A7R II you might as well just leave the ISO dial on 800 all the time.

That's not all, though, since it turns out when you increase ISO you basically lose 1EV of highlight off the top of your image, as you allude to, and you also lose around 1EV of DR. (Chart here: [http://www.photonstophotos.net/Charts/PDR.htm#Canon%20EOS%20...](http://www.photonstophotos.net/Charts/PDR.htm#Canon%20EOS%206D,Nikon%20D750,Sony%20ILCE-7RM2))

It is, like all things in life, a balance.

<comment><USER19> Fantastic, that's exactly what I wanted to know, thank you. This is very useful for setting my max auto ISO.

</comment></comment></comment><comment><USER21> As a photographer who's shot the Milky Way a lot, I can say this is excellent advice.

</comment><comment><USER22> Or get an IDAS-LPS, CLS or UHC filter (either on the lens or as a sensor clip on). Or stack a bunch of images with proper dark substraction. If you stack enough light sub frames you can efficiently reconstruct dynamic range. Of course the ETTR advice is good too, but you can go much further even with consumer equipment.

</comment><comment><USER23> I have trouble visualizing how the view of the Milky Way as seen from Earth reconciles with pictures showing whole galaxies. Are we seeing it edge on? Is there any cool animation showing it first from the outside, then rotated to match how we see it in the sky from Earth?

<comment><USER24> Imagine a vinyl record, and that you are sitting on the record, near, but not at, the outer edge of the record. South of you, but close to the horizon, is the centre of the record - containing a massive black hole. North of you are all the stars in the galaxy between you and the edge of the galaxy. Most of the stars in the galaxy are scattered along the disc, but with a fairly large variation. The closer you get to the centre, the greater the number of stars - that is the brightest part of our night sky.

Some ancient clusters of stars have drifted a fair bit away from the disc, but are still gravitationally bound to it, and its dark matter. If you were on another galaxy looking from above, our galaxy would look like a spiral with arms. If you looked edge on, it would look more like a cigar.

</comment></comment><comment><USER25> Interesting. I would've liked to see the result of 1600 iso at 30s too - <USER3> for comparison.

On a note about light pollution, the mention of sodium street lamps immediately made me think of filters - stars are suns, so should have wide spectra - why not just filter out the orange bit? Apparently I'm not the first with the idea (obviously):

[http://www.nezumi.demon.co.uk/nonad/spectra.htm](http://www.nezumi.demon.co.uk/nonad/spectra.htm)

[https://petapixel.com/2016/12/14/purenight-filter-cuts- light...](https://petapixel.com/2016/12/14/purenight-filter-cuts-light- pollution-better-night-sky-photos/) (has some nice with/without filter images)

<comment><USER26> The PureNight filter is coincidentally a crowdfunded product by Lonely Speck, the people (Ian Norman and Diana Southern) who contributed this story article.

</comment></comment><comment><USER27> When i was a child all the street and business lights were turned off at about 11pm.

Now they are on 24/7

Incredibly wasteful, although i suppose it is safer to go about ones business at night.

It certainly gave the opportunity to view the stars even if one lived in city areas.

<comment><USER28> I'm curious if in many decades self driving vehicles will eliminate the need for most night time street lighting.

<comment><USER29> Street lighting predates cars.

</comment></comment></comment><comment><USER30> I wonder where are the star trails. Wouldn't 30s exposure make earth movement noticeable?

<comment><USER20> The answer is the 500 rule. To avoid star trails, exposures should be less than 500/focal-length-in-135-equivalent.

In other words, if he shot this on his 'favourite' Rokinon 24mm, an exposure time of 20 seconds is reasonable to avoid all star trails.

<comment><USER30> Didn't know that one. Thanks a lot!

</comment></comment></comment><comment><USER5> I'm quite surprised any images of the Milky Way are possible in such a light polluted environment and I'll be using these techniques when I can to see if I can replicate the results.

</comment><comment><USER31> This articles reminded me the wonders that surround us while we go on living our boring human lives, so easily forgetting about the marvel of life and nature. Thanks for sharing this.

</comment><comment><USER32> Resource limit reached, anyone have a cached version?

<comment><USER33> [https://web.archive.org/web/20170912050030/www.lonelyspeck.c...](https://web.archive.org/web/20170912050030/www.lonelyspeck.com/the- milky-way-in-los-angeles-light-pollution/)</comment></comment>


The Unplumbed Depths of Government Data - <AUTHOR> http://motherboard.vice.com/read/the-unplumbed-depths-of-government-data

<comment><USER2> People often hear about govt managed civil engineering projects (like expanding freeway) ending up way over projected cost/schedule.

Some of it is due to not so good management/estimate but often it's due also to bad record keeping by govt also.

Under/near most of major US roads, there's many different pipes/cables/etc for water/electricity/data. Different contractors are adding/modifying something but there's no absolute/good authority on what goes where.

So next time a major project is started, people have to guesstimate where cables/pipes are. And if you happen to run into incorrectly charted (or uncharged) pipes/cables, schedule is thrown off and of course people/equipment that were scheduled (and paid for) end up waiting or doing more work.

<comment><USER3> > So next time a major project is started, people have to guesstimate where > cables/pipes are.

When surveys are done now, they spray paint the results onto the ground/pavement. Not sure how new it is but the color scheme was put into a national standard in 2013 "AS 5488-2013 Classification of Subsurface Utility Information". A different kind of open data! [https://en.wikipedia.org/wiki/Utility_location](https://en.wikipedia.org/wiki/Utility_location)</comment></comment>


Huawei Backdoors Explanation, Explained - <AUTHOR> https://blog.erratasec.com/2020/03/huawei-backdoors-explanation-explained.html <comment><USER2> Not huawei but I have worked with networking gear from china and other countries (cloud scale) that has similar backdoors. Basically an account where a whitelisted ip can use to ssh to the device. It was basically a "support" account. If you have a critical problem you get on the phone with their engineers and have them help you troubleshoot with the backdoor access. Of course no one calls it a backdoor,more like a side door for support.

I suspect that's what they're talking about here. I also suspect a much more clever actual backdoor in their products. People think a backdoor is something that gives direct shell access,but in reality compromising data integrity or compromising something else so that the attacker can access it easily or a one way "back window" where the device or other devices are taken offline is also a "backdoor".

Huawei is distracting from concerns of actual backdoor. They can easily say "would you consider windows auto-updates a backdoor? How about windows remote assistance?" ,they're basically misdirecting to a defensible side door by calling it a backdoor.

<comment><USER3> Not to dimimish your main point, but I do actually consider Windows Update to have _some_ similarities to a back door.

It was abused by Microsoft to push unwanted telemetry and advertisements to mostly unsuspecting users of Windows 7.

<comment><USER2> Sure, that does not diminish my main point at all. For public opinion "MS does it too" is enough of a defense.

<comment><USER4> Except all of Microsoft updates are meticulously reviewed by security teams around the world and would find any backdoors being installed, activated or manipulated.

<comment><USER2> Reply to the parent commenter. I never disputed that. But since you replied to me: MS never allowed external security teams to audit its auto-update infrastructure. Can they not deploy any software with auto update? Until recently,their crypto api had a bug (backdoor? Lol) that allowed forging signatures for auto updates...so.. I don't see your point. Much like huawei, the speculation is that these types of backdoors will be used in times of war or severe political tension.

<comment><USER4> US Government has audited Microsoft's infrastructure.

</comment></comment></comment></comment></comment></comment><comment><USER5> Maybe I was expecting a technical overview or reverse engineering of how their backdoors work, but really the article was non-technical. If I understand correctly, it can be summarized as: The WSJ made a claim that Huawei was using a backdoor into a backdoor, so that Chinese intelligence would have access to their devices globally. However, the claims were unsubstantiated and attributed to anonymous U.S. officials. Then, Huawei made a video clarifying what backdoors are, stating that a "backdoor" which is used by law enforcement is really a "frontdoor".

</comment><comment><USER6> I thought that blog post on explaining the explaining was very misleading, what am I missing? He or she says that if the us related leaker about Huawei was legit, they allow use of their name. But maybe it's a secret that say the us govt figured out that there really is a back door there? There is too much meta here. Yes, the us has been claiming that we can't trust Huawei. But keeping what the us knows unclear can be to our advantage.

<comment><USER7> I agree, I have no clue what point the author's trying to make here. After criticizing the WSJ's reporting and heavily implying it's reporting is inaccurate, even going so far as to suggest the WSJ article is unethically false and labeling it "fake news", the author then goes on to confirm the key accusations in the WSJ article but reframes them in a way that minimizes their significance. The author goes on to building a straw-man argument that all telco manufacturers/operators do similar things for law enforcement so we shouldn't be concerned about Huawei's practices but then goes through some mental gymnastics explaining China's authoritarian government surveillance and intel operations are sanctioned by Chinese law so Huawei aiding the Chinese government is no different than a US telco honoring a request from law enforcement. This article is disingenuous at best and comes off as blatant astroturfing. It's also interesting to note, the author mentions working directly with Huawei engineers in the article.

<comment><USER8> > I have no clue what point the author's trying to make here

You mean, you don't know the agenda. The "point" was data, namely: \- the Washington game that leads to unreliable journalism \- the fact that all Huawei's competitors have the same law enforcement backdoors, the same support contracts, and there's no evidence or even accusation that Huawei is doing anything different \- my own personal experience watching Huawei support engineers using their "backdoor" "sidedoor" "frontdoor" access to gain intelligence information.

> implying it's reporting is inaccurate

Unreliable, not really inaccurate. It's clearly bad journalism violating clearly expressed ethical standards. That doesn't mean it's wrong, it doesn't mean the government official's accusations are false. It instead means that we can't rely upon them.

> straw-man argument that all telco manufacturers/operators do similar things > for law enforcement so we shouldn't be concerned about Huawei's practices

I'm not sure you read the article. I make it clear that we should be concerned about Huawei even if their hardware, software, and support access are no different than any other vendors, because the Chinese government can lean on them in ways democratic governments cannot lean on their own vendors.

> comes off as blatant astroturfing

One of us does not understand "astroturfing". This is clearly an anti-Huawei piece that nonetheless tries to understand things from Huawei's point of view. I haven't worked with Huawei's engineers, I was working on a mobile companies systems when I saw Hauewei's support people log in via their VPN and gather national intelligence information.

The point is that everything can be true: Huawei can in fact be no different any competitor, doing at least as good a job preventing backdoors, and yet still be a national security threat due to backdoors. I believe it's good policy to forbid Huawei equipment in 5G deployments even if I doubt they have any special technical backdoors.

<comment><USER9> >"because the Chinese government can lean on them in ways democratic governments cannot lean on their own vendors." //

In comparison with USA this is where you lost me, aren't NSA's national security letters just as much a way for the government, or whoever has weight at the NSA, to access so-called front-doors? Depending how you define democratic the NSA can "lean on vendors in ways democratic governments cannot". And you intimate that access "from China" might equally be being done on behalf of USA's secret agencies.

That's fine if you trust USA, and it's leadership (covert and public).

To me, in the UK, when USA are saying "don't use Huawei" the reason that seems to be most likely is 1) financial, 2) because then we would potentially be subject to Huawei's backdoors instead of USA's backdoors. And as a citizen I'm pretty certain Five-Eyes/GCHQ have every tiny bit of meta-data about my tech use for the last year: so China can know who I call and when too, giving up that as well in exchange for reliable 5G seems like it's not really losing me much.

Seems in the UK we're more at risk from USA's financial meddling than from China's?

</comment></comment></comment></comment><comment><USER10> To be slightly snarky, Rob is arguing two points here:

* The WSJ allegations--that Huawei has allowed unauthorized use of its lawful intercept capabilities by Chinese intelligence--are anonymous and therefore unreliable.

* Rob personally knows (based on his own experience, omitting any details that would enable us to validate the story)that Huawei has allowed unauthorized use of its lawful intercept capabilities.

...er, what?

In a less contrarian framing, this could be rewritten as, "The WSJ article is accurate, and I'm even willing to put my name on the public record as having seen similar things."

It's fashionable these days to criticize anonymous sourcing--and, hey, I agree a reader should be cautious about such things--but this has to be the first time I've seen someone criticize an anonymously sourced article while explicitly validating its claims.

</comment><comment><USER11> Don't care what they call it.

</comment><comment><USER12> There is something very, very disturbing here, sort of like a "(legal) disturbance in the (legal) force", to join a Star Wars quote with Law...

Let me explain.

First, let me get across that I am neither for nor against Huawei, neither for nor against The Wall Street Journal, neither for nor against this article/video nor the claims made in it.

My first problem, if I have a problem, begins with human language...

You see, we can call something a "front door", we can call something a "back door". We can say that "front doors" are used lawfully, by law enforcement (or others), for the purpose of "lawful interception". And we can say that there are "back doors" (generally associated with illegal, unlawful activity -- but this may not be the case in all circumstances), and that there are "maintenance doors", that workers use.

 _But the problem is, this use of language dumbs down our conversation._

A simpler view of the universe would say that there are methods which can be used to access data, and that actors employing such methods are either authorized or not authorized.

In other words, you have "access method", and "authorized" or "unauthorized".

That's it.

Authorized is you have permission to use it.

Unauthorized is you don't.

That would be the simplest view of things.

But this is not what's disturbing...

You see, there's a fairly deep legal question in relation to this...

To understand it, let's suppose I was a manufacturer of network equipment. And let's suppose I was coerced by a government (U.S., Chinese, Other Government), by whatever means (legal, sanctions, threat of violence, threat of loss of commerce, ?) to add a backdoor/front-door/access method (call it what you will) to the network equipment, for this government's law enforcement community.

OK.

So now that backdoor/front-door/access method (again, call it what you will) -- is there and all.

Now, here's the legal question...

If it's there, by virtue of it being there, by virtue of the government actor knowing how to use it as a means of access, is the government (foreign or domestic) then legally authorized (do they have permission) to use it?

?

In other words, if I, as a network equipment producer, then sell my backdoored/front-doored/access method enabled product to a service provider -- do the legal rights for that backdoored/front-doored/access method -- go to the service provider, who must again explicitly grant them to the government (again, foreign or domestic) in order for them to use that backdoored/front- doored/access method, and still be authorized?

To make the story short, authorized access, (from a legal point of view), is not just one, but a series of contracts and contractual agreements (and the interpretation of those contracts!) made between many parties, including, but not limited to the manufacturer, the service provider, and the users to whom service is provided to.

In other words -- there's no easy answer!

The devil is in the details!

Even a Lawyer could not answer this question... it would have to be determined by the Courts, and then it could go many ways, depending on the number of actors involved, the contractual agreements between them, and the legal arguments raised...

We have FISA Court -- but even FISA is not a blanket authorization _in the presence of other contracts_! (FISA might be a blanket _presumed_ authorization if no other contracts were present, but generally speaking, that won't be the case...)

In other words, two cases involving a lawful intercept from two different equipment manufacturers could go two seperate ways in the courts!

A legal mess, to be sure...</comment>


Ask HN: Is Google is broken for discovery? - <AUTHOR> My own thoughts based on a recent series of queries below.
TLDR I think it is most definitely.
Recently I was fortunate enough to be able to afford buying a home.
An empty home needs a lot of furniture to fill it. I've graduated from Ikea furniture I suppose, or more correctly I don't think I'd be able to make a successful pitch to the wife that this is the correct place to shop for furniture.
So, off I go to Google to try and see what the landscape is for furniture. The usual suspects all show up. Cranes and Bottles, Possibly Barn, Mercys, Blossomdales. And that was it. I scroll down, different variants of pages for these, ads... again more duplicate pages of brands I already know, more ads, so on and so forth. The "next page" is replaced by a "see more", and when I tap it just more of the same.
I literally couldn't find anything organic other than major brands, who not coincidentally were likely paying for adwords as well.
This isn't what I want to see in a search engine, I doubt it is for anyone else. I could have just typed in those domains, I know them all already.
I feel like at this point, it's time for a back to square one search results engine. Monopolies eventually kill off all brand goodwill and it looks like we've hit the tipping point. <comment><USER2> I generally do a google search now and tag 'reddit' at the end, to find useful conversations around the topic I'm interested in. It's been working great so far.

<comment><USER3> also, you can use site:reddit.com to get results only from reddit domain

<comment><AUTHOR> Ha, exactly what I did in the end. I thought I was clever.

But if this is the case then what value is google providing other than a good search interface on subreddits?

</comment></comment></comment><comment><USER4> That's the whole point inevitably, if it was good we won't stay on a page/multiple result pages for long. Also the rationale behind removing direct image links, so as to keep you on a JS executing page.

"Institutions will try to preserve the problem to which they are the solution." — Clay Shirky [https://kk.org/thetechnium/the-shirky- prin/](https://kk.org/thetechnium/the-shirky-prin/)

</comment><comment><USER5> I wrote about this before:

[https://austingwalters.com/is-search-solved/](https://austingwalters.com/is- search-solved/)

Basically, Google doesn't have an incentivize to ever provide some unique results to you. They solve for the "general" use case, and the more you click through the better for them (higher likelihood you'll click an ad).

I'm actually working on my own kind of search, it's the basis for a few of my websites.

</comment><comment><USER6> local furniture stores near me

turns up a good number of stores that are, in fact, near me. Brick and mortar, local, boutique places. Places I've never heard of. Yeah, the ads are for the usual suspects. But not only were there a lot of places that seem at least somewhat interesting on their own, there were a lot of articles from local news sources about furniture shopping in Dallas, TX. Which is also really convenient because that's where I live. Articles about the benefits of going local over big brands (internet driven or otherwise), where, when, and how to find good bargains locally, which shops are better for what categories of furniture. Etc.

I mean, Google certainly has some problems, but I don't think this is one of them.

<comment><AUTHOR> Could be location helping. Not a lot of craft furniture stores in the Southern Part of Silicon Valley. Yet another side effect of buildings being worth 1k a sqft

</comment></comment><comment><USER7> Home furnishings are a bit special. The market is flooded with cheap Chinese goods. There's no way to tell what's good, and there are way too many choices/knockoffs. Plus it's too heavy to return if you don't like it. You really have to see it in person to have the confidence that it's right.

This is also true for cabinets, flooring, lighting, etc.

<comment><AUTHOR> I agree with you that's its probably the worst case. But let's explore, I also need a new TV, well not need but it's been around 10 years figure I want to see what's up in the TV market. This time there was one new review site I was happy to see called rtings that I hadn't heard about. But between thats was 50 versions of better buy and the like.

Perhaps it's false nostalgia but I feel like search results used to have some variety.

<comment><USER8> I was about to say that headphones were in the same place as furniture, where any ratings cannot be trusted, and the only way to guarantee a quality product was trial and error. Maybe the problem space is larger than I realized but I just do not buy enough stuff to notice.

</comment></comment></comment><comment><USER9> When I want to discover new stuff, I just keep excluding things I don't want. For furniture, just add "chair -ikea -x -y -z" until you no longer see results from them.

<comment><USER10> I wish I could permanently ban results from websites that game (or pay for) their rank and whose content I am not interested in(generally the big guys like web.md, etc..) I think at some point google had an x for each result and you could remove it from the search. They didn't keep it. Does anyone know of any plug-in or script to do that? I know i could do -web.md but there are way to many sites that occupy the first pages of results I want removed

<comment><USER11> I don't know a plugin, but this seems like a good idea. Maybe I'll try to circle back around this weekend and build out something.

<comment><USER10> Also can you think about shareable blacklists similar to how AdBlock works? And easy enable and disable. I'm thinking that if I don't get any good search results I could compare with a search without blacklisting. Thanks

</comment></comment></comment></comment><comment><USER12> You can't expect a tool to work when you don't know how to use it.

You're blaming google because you're using the tool wrong. If you're searching for furniture, type in 'furniture'. If you're searching for different furniture brands type in 'different furniture brands'. If you want to discover furniture brands type in "discover furniture brands".

</comment><comment><USER13> I suspect the difference in experience relates to the idea that a recent homebuyer is valuable marketing product. OP is probably getting flooded because of financial circumstances.

I agree that discovery is broken. I too cannot find a brand or organic non- toxic children's furniture except by very roundabout searching through similarities on Amazon, etc.

<comment><AUTHOR> Totally. This was another thing that bugged me to no end trying to separate the wheat from the chaff so to speak. I went with JontyCraft which I originally found on amazon and then cross referenced myself. Seems like they are US based and when they arrived the chair was extremely solid no toddler is going to break it I think adults if they could fit it could sit fine it's so well made.

</comment></comment><comment><USER14> Google has changed it's search to favor corporate interests. It doesn't reflect the real internet anymore.

Search for movies, you get a list of newspaper reviews instead of the forums or independent reviews. Everything, even recipes, gets links to newspapers. It's so useless now.

What I do is "-news" or add "forum, reddit, etc" in my searches. Or I just simply jump to page 5 of the search result. Or I started to use duckduckgo more.

The same thing with youtube. If an event happens, I used to just search for the event and I'd get the most revelant/most viewed links. Now, it's pages of CNN, MSNBC, WashingtonPost, NYTimes, etc links.

It seems like google just wants to be a glorified newstand. I can't believe how horrible it has gotten. It takes effort to get what you want now.

<comment><USER15> I suspect it's a side effect of their anti spam efforts. Just making sure big brands fill the top-n slots kills a lot of spam. At the expense, of course, of most of the good stuff.

</comment></comment><comment><USER16> Google's major problem is that it's in an adversarial relationship with the SEO industry. For any given keyword there is a lot of money to be made by being in the top 3 results, which means the top results tend to be dominated by businesses with the resources to mount a focused and persistent SEO campaign. This leads the results to become saturated with content marketing rather than "organic" content.

</comment><comment><USER17> Recently I went to Berlin for a startup bootcamp and met the CEO of Mablo[0]. Their aim is to solve this problem, so that people can find products that fit their exact needs. The point is the problem that you described above is real, everyone else also faced it and there is a company that is trying to solve it.

I agree with the other comment. It is not that search engines are broken. But having ads does not incentivize any search engine to make products discoverable.

[0] [https://mablo.net](https://mablo.net)

</comment><comment><USER18> 33 out of the 36 results on the first page of a google shopping search for "furniture" are all from Living Spaces for me, with a pretty homogeneous aesthetic

<comment><AUTHOR> Right? How is that useful to anyone?

</comment></comment><comment><USER19> Google has a problem that they are not content generator, and their every attempt to create community around content generation failed, knol, google+, google answer, blogspot etc. Their reliance on other people to create relevant content could not solve the problem no matter how efficient their algorithm is. I wish they give one more go at content creation with much more distributed control and privacy awareness.

</comment><comment><USER20> when I search Google for "furniture" one of the first results is the craigslist furniture section for my city, much of which is free.

</comment><comment><USER21> Google doesn't provide a great mechanism for discovery or human curated content. That's why many use Pinterest and Instagram for discovery.

<comment><AUTHOR> Pinterest at least for me is the worst browsing experience ever. Sucks you in with thumbnails that you can't enlarge and regwalls and annoyance I personally avoid it like the plague

</comment></comment><comment><USER22> When I search for furniture I get a huge variety of different stores and brands, many of which I've never heard of before. Have you tried searching from a private browser window?

</comment><comment><USER23> Search is not about discovery. It's about already knowing what you want. (Or it should be, Google's steps away from this are what irritates everyone.) Try Pinterest.

</comment><comment><USER24> I do seem to get many bad links, but I think it may be Search Engine Optimization? Or possibly it seems Google takes me to more social media links than in the past.

</comment><comment><USER25> It is in fact common that what I look for will not be found easily in any search engine.

</comment><comment><USER26> Did you try other search engines too ?

<comment><AUTHOR> What other search engines would you recommend?

</comment></comment><comment><USER27> I feel like Pinterest is a better platform for these type of queries.

</comment><comment><USER28> if you 're doing image search and not interested in stock photo sites, try bing or yandex</comment>


Ask HN: What to do about a startup holdout? - <AUTHOR> Hey guys,
An acquaintance of mine has been working on a startup in the social media space for more than 3 years. Early on he managed to build the app and raise some funding, but the product got no traction whatsoever on the user side of things, even after many pivots. The core premise just doesn't resonate with people at all. Anticipating failure, the team left about 18 months ago. Since then he's been living abroad, alone, chugging away at this startup and paying rent from the money he raised. He routinely uploads motivational posts to Facebook stating the importance of perseverance and "never giving up".
I wanted to ask HN - do you know anyone who's a startup holdout? Who maybe should've given up a long while ago, and became way too invested? What would you say to them? Or maybe they're right to keep going?
Thanks! <comment><USER2> Well, why should he give up if there is still some money to burn? It will be like, leaving the money on the table? Doesn't make sense at all. I think he's doing everything right.

Obviously, he's really busy with other things while he's posting all this. Probably doing another startup, or just freelancing.</comment>


How much should I pay for a domain? - <AUTHOR> I am negotiating a domain name for my new startup. The seller is asking for $2000/month plus 5% equity for leasing the domain with a buyout option of $500k. All the money put towards the lease will go towards the buyout.  ALl This is a one word dictionary domain. It will REALLY help out the startup as the word clearly describes the business.<p>I definitely plan on negotiating.  I wanted to get thoughts and feedback on what you have seen in the past. <comment><USER2> That you are even considering this is proof that vc funding is fucking crazy. Half a million dollars, for a domain name?

</comment><comment><USER3> you can register a domain for ~$10, depending on which registry you use.

anything more than that is a strict value call on your end. Is getting the exact domain that you want worth the extra thousands and thousands of dollars?

also, it strikes me as odd that you are calling the other person a "seller" when they are offering you a lease on the domain. are you aware that "lease" is just another word for "rent"?

honestly this sounds like a terrible idea that you haven't fully thought through.

</comment><comment><USER4> Can you measure or prove the impact the domain would have on the business? If not, you'll end up spending a great deal of money on something you don't know if it matters or not.

IMO, domains are growing less and less important as more people find sites by clicking on links, not by typing in the domain name.

So many successful companies start with getMyApp.com instead of MyApp.com.

</comment><comment><USER5> You haven't given much info. Do you know much about internet traffic? You might be overestimating the organic reach of your domain.

That being said, it's not too bad a deal, you are only out $2K if you have a taste and the domain doesn't deliver.

BUT cut out the equity piece. These type of shareholders are at best pure dead weight and at worst a pain in the ass.

One other note, even with a contract, there are ways the seller can screw you (eg. changing their bank acct details & going incommunicado) so if you are leasing make sure the domain is controlled by a neutral 3rd party for the lease term so you are protected if someone with a better offer comes along.

<comment><AUTHOR> Yeah, that's what I am thinking as well. Having the domain be monitored by a third party is an excellent idea. I know some domains go for millions. The traffic on the domain right not is negligible since its a holding page.

</comment></comment><comment><USER6> It's hard to offer guidance around pricing without knowing anything about your company, or how much it is/could be worth. Unless you're the next Uber, spending even $2000/month sounds _insane_ to me.

My opinion is that you should save the money, choose a different name (or variation of the name), and put the $2k/month towards something that will provide real value to your startup. There's a ton of one dictionary word domains available if you have a little creativity, and, since you're new, you're presumably not really tied to any one name.

</comment><comment><USER7> Think of it as a full-time marketing employee. Would you give your best marketing person 5%? Probably not, but $2000/month is probably ok for a good domain.

</comment><comment><USER8> Obligatory pg article link: [http://www.paulgraham.com/name.html](http://www.paulgraham.com/name.html)

</comment><comment><USER9> How far along is your startup?

<comment><AUTHOR> Pre-stage. Not even the design has started yet.</comment></comment>


At Marathon in North Korea, Curiosity Goes a Long Way - <AUTHOR> http://www.nytimes.com/2015/04/17/sports/at-marathon-in-north-korea-curiosity-goes-a-long-way.html?

<comment><USER2> This is an interesting quote:

> Not everyone's reaction could have been staged on the course, said Roeland > Loof, 33, a fellow Dutch runner. Especially the children. Children do what > they want.

> "In the U.S. and Europe, we're as brainwashed" about North Korea, Mr. Loof > said, "as they are here."

Interesting the article didn't elaborate on this more, but it's true. People need to spend more time meeting, talking to people and experiencing life, and less time in front of the TV digesting everything the media (under the direction of the government) tells us to believe.

<comment><USER3> I hate this kind of false equivalence. It prevents anyone from discussing the actual problems, because, well, "You do it too! You're just as bad!"

No. The West is not just as bad as North Korea. The West has more freedom of speech, freedom of information, and general freedom to be human. Saying otherwise denies what North Koreans suffer, and is precisely what the North Korean government would say in a PR stunt.

<comment><USER2> > I hate this kind of false equivalence. It prevents anyone from discussing > the actual problems, because, well, "You do it too! You're just as bad!"

>The West is not just as bad as North Korea. The West has more freedom of speech, freedom of information, and general freedom to be human.

On the flip-side, what you propose prevents us from seeing the problems with our own society. "At least we're not as bad as North Korea" So many times when things go bad here, when the government takes away more and more freedoms, all we hear is that at least we still have it better than Russia/North Korea/Iran/insert bogeyman.

I'm not proposing brushing their problems under the rug (nor ours). I'm simply proposing that we see people in these countries as people. There are real people in North Korea, in Iran, in Russia, who grow up, get married, have kids, have celebrations, and generally live life.

So much of our propaganda is about dehumanizing our enemies, turning them into subhuman bogeymen in order to justify whatever our governments decide to do to them/us.

And no one said anything about the west being as bad as North Korea. The only comparison made was that we're brainwashed in our perceptions of others. Which we are. Your response shows that very well, which basically parrots everything our media ever says about North Korea.

<comment><USER3> > I'm simply proposing that we see people in these countries as people.

Of course. Nobody is saying we shouldn't. If you're debating _this_ premise, you're debating a premise nobody else raised, one which only exists in your head.

> Your response shows that very well, which basically parrots everything our > media ever says about North Korea.

I'm not parroting, and saying I am just convinces me you're not interested in a discussion, but in being right, and insulting people who disagree.

My conclusions are based on evidence, mostly from people who went there as tourists, shepherded by the government itself, and _still_ saw horrible things. That, plus the refugees and the stories they tell, paint a consistently terrible picture of what life is like in North Korea.

It's possible they're all lying, all the photos are doctored, and North Korea has just as many lights as South Korea when night falls. However, that's so far from being the least hypothesis that claiming it without a lot of new evidence is absurd.

<comment><USER2> > I'm not parroting, and saying I am just convinces me you're not interested > in a discussion, but in being right, and insulting people who disagree.

Then why do you say nothing of individual North Koreans, and not even touch on any of the points I raised in the first post you replied to?

I said "People need to spend more time meeting, talking to people and experiencing life, and less time in front of the TV digesting everything the media tells us to believe." and your reply was "No. The West is not just as bad as North Korea. The West has more freedom of speech, freedom of information, and general freedom to be human."

I suggested a conversation, a meeting of peoples, which is exactly what the article is about. It's not about a North Korean PR stunt, it's not about an officially sanctioned marathon. Yes, all that happened, but it's about what happens when North Koreans are allowed to meet westerners.

> My conclusions are based on evidence, mostly from people who went there as > tourists, shepherded by the government itself, and still saw horrible > things. That, plus the refugees and the stories they tell, paint a > consistently terrible picture of what life is like in North Korea.

Again, you're debating how terrible North Korea is. That's not the point.

> It's possible they're all lying, all the photos are doctored, and North > Korea has just as many lights as South Korea when night falls.

Yes, North Korea is less developed, there's less lights (granted, nearly every place on the planet has less lights, especially per square mile, than South Korea). Again, that's not the point.

All I'm interested in is seeing the PEOPLE that are on the other side. This article is interesting because of the interactions with people in an isolated, some would say repressive, state that is opening up ever so slightly to foreign interaction. That's the point.

<comment><USER3> > Then why do you say nothing of individual North Koreans, and not even touch > on any of the points I raised in the first post you replied to?

You said nothing of individual Westerners, and didn't touch on any of my points.

</comment></comment></comment></comment></comment></comment><comment><USER4> Fascinating. Curious what it would take to register, I found the Pyongyang Marathon signup form here: [http://www.pyongyangmarathon.com](http://www.pyongyangmarathon.com)

Apparently they accept amateur runners and you can do a 10k, half marathon, or full marathon.

<comment><USER5> > Please Type 12345 in the field below

that's the best captcha I've ever seen.

<comment><USER6> the google captcha just says click to confirm you are not a robot. [https://www.google.com/recaptcha/intro/index.html](https://www.google.com/recaptcha/intro/index.html) Maybe the second best.

<comment><USER7> Actually the movement of your cursor alone can go a long way to telling whether someone is a bot or not[0]. The ACM linked a better article a while back that went more in depth about it, but there is more going on with that click that meets the eye. [0][http://www.wired.com/2014/12/google-one-click- recaptcha/](http://www.wired.com/2014/12/google-one-click-recaptcha/)</comment></comment></comment></comment>


Show HN: EasyWrite- Write clearly using only 1,000 commonly used words - <AUTHOR> https://github.com/adeekshith/easy-write <comment><USER2> "This encourages you to write only using frequently used words"

"This helps you to write only using top words"

These two sentences could have different meanings though. What is a top word? The best ones? The frequently used ones? Without context it might not be clear.

Encourages and helps can also be very different.

\---

Anyway it is quite cool I guess. Maybe you could add automatic synonyms for words that are within the 1000.

</comment><comment><AUTHOR> Inspired by Cleartext for the web. Read my blog post on this if interested: [http://www.deekshith.in/2016/04/easy-write- intro.html](http://www.deekshith.in/2016/04/easy-write-intro.html) Let me know your feedback. Thanks :)

</comment><comment><USER3> its an interesting idea - this bit made me laugh -

"beautiful" instead of "gorgeous"

since i never spell "beautiful" right!

<comment><AUTHOR> Haha, yeah may should have said "lood looking" instead of "gorgeous".

</comment></comment><comment><USER4> Why is this good? Being specific is better than being general.

<comment><AUTHOR> [https://xkcd.com/thing-explainer/](https://xkcd.com/thing-explainer/)</comment></comment>


MacBook Air 2013 display lag - <AUTHOR> http://www.youtube.com/watch?v=9KlOkbFlnsk&feature=youtu.be

<comment><AUTHOR> I have some display issue with my new MacBook Air 13" (Mid 2013). Core i7, 8Gb, 512SSD. LCD panel model: LP133WP1-TJA7 (LG). Picture on the screen is redrawing very slow when you scroll or drag. Contrast objects leave traces. Seems like very slow display response time. This effect burns my eyes. Apple phone support says that it's normal, after watching my video. Any ideas?</comment>


Revisiting 1M Writes per second - <AUTHOR> http://techblog.netflix.com/2014/07/revisiting-1-million-writes-per-second.html <comment><USER2> My other favorite "scalability" study is from WhatsApp:

[http://www.youtube.com/watch?v=c12cYAUTXXs](http://www.youtube.com/watch?v=c12cYAUTXXs)

That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp

(that walk title was create before the acquisition and was mean to imply message count, after the acquisition it got a secondary meaning).

The one thing that is fascinating about it, is how small their team was compared to the volume and complexity of the operation.

<comment> dav- The thought of even touching systems at such scales terrifies me.

<comment><USER2> Agreed. Someone asked once how to you simulate load like that to test and he answered "we can't" we try to do gradual deployment and rolling upgrades.

</comment></comment></comment><comment><USER3> I love using Cassandra; it's been a dream for analytics. Thank you Netflix et al. for not only moving the project forward, but providing research and commentary like this and others available at [http://wiki.apache.org/cassandra/ArticlesAndPresentations](http://wiki.apache.org/cassandra/ArticlesAndPresentations)

<comment><USER4> Can you talk a little about how you're using it for analytics and why you like it over other similar choices?

</comment></comment><comment><USER5> Would be great to know the exact data set size (not size in the database). I can't get an order of magnitude sense of what I am looking at without that.

I know I can divine it from the parameters to stress, but I have no idea if the row keys generated by different clients overlap and I don't know the default number of columns nor their size.

I think it's also important in this kind of benchmark to describe the distribution of access especially for a read intensive benchmark. Without that you really don't know what your are looking at. I am a fan of scrambled Zipfian.

<comment><USER6> It seems like this info should be front in center in the test. 1M/s 100kb writes is much more impressive than 1M/s 16 byte writes.

That said, there's a previous benchmark linked to at the top of the post:

[http://techblog.netflix.com/2011/11/benchmarking- cassandra-s...](http://techblog.netflix.com/2011/11/benchmarking-cassandra- scalability-on.html)

 _The client is writing 10 columns per row key, row key randomly chosen from 27 million ids, each column has a key and 10 bytes of data. The total on disk size for each write including all overhead is about 400 bytes._

There are 3 replicas, so figure that in as well.

</comment></comment><comment><USER7> So the test was run on a 285 node cluster with 60 clients. It would be nice to know how they arrived at those numbers. Was there some sort of formula used to calculate how large each group should be? How much trial and error was involved.

</comment><comment><USER8> Couldn't help but notice: $398.70 per hour = $9568.80 per day = ~$3.5m per annum. They obviously get a discount...but still. What kind of discount do guys like this get?

<comment><USER9> I'd be surprised if anyone got discounts as deep as Netflix considering their usage.

For what it's worth, 3.5m/yr is about 0.08% their revenue.

<comment><USER8> Well, that puts things in perspective.

</comment></comment></comment><comment><USER10> This is running Cassandra 1.2.x which is over 18 months old.

Here's some performance notes on the latest (2.1rc4): [http://www.datastax.com/dev/blog/cassandra-2-1-now- over-50-f...](http://www.datastax.com/dev/blog/cassandra-2-1-now- over-50-faster)

<comment><USER11> Will 2.1 actually be usable? It seems like 2.0 has missed that mark in every single release.

<comment><USER12> Are you saying 2.0 isn't usable ?

Because there are many companies (including ours) who would strongly disagree with you about that.

</comment></comment></comment><comment><USER13> I wonder where the term sidecar originated from and what is the precise definition? Is this something invented by the netflix OSS people or does it predate that?

<comment><USER14> It's from here: [https://www.google.com/search?q=motorcycle+sidecar&source=ln...](https://www.google.com/search?q=motorcycle+sidecar&source=lnms&tbm=isch)

<comment><USER13> I was hoping for a more precise software concept. Maybe it is loosey goosey.

<comment><USER14> The definition I use is a piece of software that runs independently to encapsulate infrastructure libraries. In other words it gives you a way to access the libraries without having to import them into your code.

</comment></comment></comment></comment><comment><USER15> How many companies can possibly afford the cost and management pain of running a 285 node database? Few would have 285 servers of any type? So if this is what it takes to get 1M writes on Cassandra that is some poor ROI. 10K writes/sec is not impressive.

<comment><USER12> I don't understand your point.

The type of companies who could afford this are the types of companies who need 1M writes/second. Which are few and far between. And yes 10K is not that impressive but 1M is. And with Cassandra you could continue to improve that number just be rolling out more nodes.

<comment><USER16> It is quite cheap to average a million writes per second. I've done it with five servers on AWS, and that was spatially indexing billions of GeoJSON polygons through storage while running queries against the index.

Many companies need far in excess of a million writes per second. Basically, most machine-generated data sources, whether it is personal location data or any other kind of telemetry. Many companies that do not generate that data themselves buy and consume it. I know of companies doing over a billion writes per second.

Cassandra is pretty good for this type of thing among open source software but it is not nearly as efficient as it could be in terms of write throughput. If the storage engine is correctly designed, you should be able to drive 10GbE all the way through storage -- call it 1 GB/sec per node. However, that does mean you can't do things like mmap()-ing files; those interfaces are slow due to poor scheduling by the OS when the throughput is very high.

<comment><USER17> > It is quite cheap to average a million writes per second. I've done it with > five servers on AWS, and that was spatially indexing billions of GeoJSON > polygons through storage while running queries against the index.

How big was your total dataset. It's cheap to average a million writes per second if the dataset fits in RAM, or at least if the index set does, with the right database. It can be less cheap for a data set far larger than RAM, as for most databases write amplification becomes a significant problem.

<comment><USER16> For that case, several terabytes IIRC. It was not in-memory, that database engine was pushing records through disk storage. Obviously there were no R-trees or any other kind of slow secondary indexing; the database itself is deeply and fundamentally spatially organized, even for text and numbers data.

While there is some write amplification it is less than most databases. It only takes few disks before the scheduler can get significantly more bandwidth out of the disks than a 10GbE network has to drive that activity, so there is extra capacity. The bottleneck on most server hardware is the silicon between the storage and memory if you are doing it right.</comment></comment></comment></comment></comment>