Replacing exploit-ridden firmware with a Linux kernel [pdf] - <AUTHOR> https://schd.ws/hosted_files/osseu17/84/Replace%20UEFI%20with%20Linux.pdf <comment><USER2> These slides are from the talk given at the Open Source Summit (formerly known as LinuxCon) last week in Prague. The abstract given at the talk page [https://osseu17.sched.com/event/ByYt/replace-your-exploit- ri...](https://osseu17.sched.com/event/ByYt/replace-your-exploit-ridden- firmware-with-linux-ronald-minnich-google) is

 _With the WikiLeaks release of the vault7 material, the security of the UEFI (Unified Extensible Firmware Interface) firmware used in most PCs and laptops is once again a concern. UEFI is a proprietary and closed-source operating system, with a codebase almost as large as the Linux kernel, that runs when the system is powered on and continues to run after it boots the OS (hence its designation as a "Ring -2 hypervisor "). It is a great place to hide exploits since it never stops running, and these exploits are undetectable by kernels and programs._

 _Our answer to this is NERF (Non-Extensible Reduced Firmware), an open source software system developed at Google to replace almost all of UEFI firmware with a tiny Linux kernel and initramfs. The initramfs file system contains an init and command line utilities from the u-root project ([http://u-root.tk/](http://u-root.tk/)), which are written in the Go language._

<comment><USER3> >"UEFI is a proprietary and closed-source operating system, with a codebase almost as large as the Linux kernel"

Does anyone know how they were able to compare the size of the codebase given that UEFI is proprietary/closed-source?

<comment><USER4> UEFI is just a spec. Intel did release an open source implementation: [http://www.tianocore.org](http://www.tianocore.org)

<comment><USER5> My understanding is that most of the UEFI implementations out in the wild is based on this one, since the spec is pretty big and definitely not something a random mobo vendor would like to implement.

</comment></comment></comment></comment><comment><USER6> Ever since I read that Intels UEFI reference implementation has a comparable complexity to the Linux kernel[1] I thought it's absolut nuts to have something with such a complexity just to set up the hardware for another operation system. And that's just UEFI and not even touching ME and other stuff.

[1]: [https://mjg59.dreamwidth.org/10014.html](https://mjg59.dreamwidth.org/10014.html)

<comment><USER7> On the other hand, isn't quite a lot of the code in Linux kernel for the same thing, handling hardware complexity?

<comment><USER8> Even if it is, that still means you're doing it _again_.

</comment></comment></comment><comment><USER9> Nitpick: NERF isn't coreboot, not even "coreboot 2". It's a separate project.

Nice project though, and sad that it's needed in the first place. Oh well.

</comment><comment><USER10> Power9 servers appear already do this. They have minimal firmware to load the on-rom linux, which then uses userland application (petitboot) to figure out which installer media or already installed linux system you want to kexec() into:

[https://sthbrx.github.io/blog/2016/05/13/tell-me-about- petit...](https://sthbrx.github.io/blog/2016/05/13/tell-me-about-petitboot/)

This seems logical, if you are going to run linux, why reimplment all the drivers in UEFI.

</comment><comment><USER11> So far AMD has ignored fixing this pretty much because "only privacy nerds" care about it. Well, some of those privacy and security nerds happen to work for companies like Google and even have influence over those companies' hardware purchases.

So now AMD has just lost a big chuck of potential Google business, and who knows what kind of other potential contracts, because it continues to ignore this issue instead of actually showing some leadership on it.

Maybe it's time for AMD to actually differentiate from Intel on this issue?

<comment><USER12> Devil's advocate: if large customers care, AMD can easily give them a "special" microcode image with it disabled. Intel does the same thing with the "High-Assurance Platform" for the intelligence community, which is effectively a ME killswitch.

However, these fixes are generally not offered to the public even when they already exist (again, see: Intel HAP).

The rules of the game are different when you're one of the Super 7 instead of just some random pleb. And increasingly those guys are no longer interested in leaving this threat vector open, which says something.

<comment><USER3> >" Intel does the same thing with the "High-Assurance Platform" for the intelligence community, which is effectively a ME killswitch."

Do you have any links or documentation you could share regarding these exceptions? Thanks.

<comment><USER13> Here you go: [http://blog.ptsecurity.com/2017/08/disabling-intel- me.html](http://blog.ptsecurity.com/2017/08/disabling-intel-me.html)

<comment><USER14> To see the Int community have a special mode to disable ME simply confirms that the ME is exploited, if any further proof was needed.

</comment></comment></comment></comment></comment><comment><USER15> I find the use of Go for the userland pretty cool. It makes sense in this case considering if you disable cgo the only external dependency you have is the Linux syscall table.

I've implemented a number of system tools in Go and it is pretty well suited to this sort of job IMHO. One particular case that I've struggled with however, is when I've needed to modify some characteristic of my running thread using an OS mechanism. For instance, say I have a number of go-routines running and then I want to make sure a filesystem is unmounted from all mount namespaces the kernel has. My only pure-Go option right now, is to execute another go program (or re-execute the same program with some arguments) that will enter the namespace via syscall, do the work in that namespace and then quit. If you have a bunch of namespaces this seems wasteful.

This is because you don't have complete control over what Go does with the OS- thread you are working in. Yes, you can lock your go-routine to a particular OS thread but you can't stop Go from using that somewhat special and potentially more privileged thread, to create other OS threads to service go- routines. Thereby potentially sprinkling your go-routines with different capabilities and/or namespaces.

I ended up using cgo, which was a shame. Perhaps someone knows some neat trick to work around this?

<comment><USER16> Hey there.

In snapd (which is implemented in mostly go) we have this problem a lot. The real issue is that certain system calls fail if more than one thread exists in the calling process. One of those is setns(2), as is documented in the manual page.

What I ended up doing is to use a small C preamble that parses command line arguments, figures out where to go and uses setns before the go code even begins initializing.

This solved the particular case we were working on but in my opinion golang's opinionated approach to threading is not suitable for writing many system tools in it.

My wishlist item for golang 2.x is a build mode where threading is 100% under developer control but this seems to be at odds with the design for non- blocking IO.

<comment><USER15> Interesting. I don't have access to the code I was talking about anymore, but I'm pretty sure that I did have different threads in different mount namespaces.

Looking at the man-page briefly it looks like the restriction you are talking about applies only to user namespaces.

Interesting to know that there are even more situations to be considered.

</comment></comment></comment><comment><USER17> The question is,

Why Linux? You already have Linux for OS, why the same on firmware?

Why not Minix? ( Used in Intel Me Already )

Why not OpenBSD? ( Very Secure )

Why not NetBSD? ( Extremely Portable )

Why not FreeBSD?

Why not Coreboot?

Why not the OpenSource UEFI implmentation?

Why not switch back to simple and easy BIOS?

<comment><USER9> > Why Linux? You already have Linux for OS, why the same on firmware?

The Linux they're flashing to the firmware ROM is a custom minimal build. Using that, they can then boot a standard distro kernel. And depending on your threat model, you might not need to update the firmware kernel as often as the distro kernel, as it's only used for booting. Note that this is for servers; for an embedded system you might as well boot directly to the final kernel.

> Why not Minix? ( Used in Intel Me Already ) Why not OpenBSD? ( Very Secure ) > Why not NetBSD? ( Extremely Portable ) Why not FreeBSD?

A few guesses:

\- Linux is the one most familiar to the developers of this. And to the Google sysadmins, presumably.

\- Compatibility: If the distro kernel resides on, say, an XFS filesystem on an MD-RAID device, the firmware boot kernel needs drivers for that.

\- It uses kexec which is a quick and easy way to boot a Linux kernel from another Linux kernel. If you want to boot from another OS you'd have to figure out something else.

> Why not Coreboot?

Coreboot is preferable, but since the low-level firmware on modern x86 server platforms is tightly controlled with hardware signing keys, no data sheets released etc., coreboot hasn't been able to run on an Intel x86 server platform for the past decade. This project (NERF) is a pragmatic compromise, by replacing only the upper parts of the firmware stack and removing/disabling the ME as much as possible.

> Why not the OpenSource UEFI implmentation?

It's designed-by-committee crap which is far less battle tested than Linux.

> Why not switch back to simple and easy BIOS?

It's less evil than UEFI sure, but I guess it's unfortunately only a question of time until it's removed from PC firmwares (Apple already did it many years ago, AFAIU). Besides, AFAIK switching to BIOS mode does nothing to disable the ME.

And if you're going to do something from scratch, you might as well use something sane like coreboot.

<comment><USER18> > I guess it's unfortunately only a question of time until it's [BIOS is] > removed from PC firmwares

As far as the bit I know, Legacy Boot, an option on (most/all) current PCs that simulates BIOS and which many people confuse with disabling UEFI, is actually a mode of UEFI. For more detail, look up Compatibility Support Module, and the difference between Class 2 and Class 3 UEFI.

<comment><USER9> Thanks, I stand corrected.

</comment></comment></comment></comment><comment><USER19> Mods, please change title to replace Coreboot with "Linux kernel". This is not Coreboot and doesn't involve Coreboot at all :)

<comment><USER20> I watched the talk video and I don't remember it was stated anywhere that Google will replace UEFI/Intel ME. The correct title is "Replace your exploit- ridden firmware with a Linux kernel".

</comment></comment><comment><USER21> Doesn't Coreboot implement UEFI? I suppose a more accurate title is "replace proprietary UEFI implementation"... nonetheless, the idea of putting a full Linux kernel in the firmware (if I'm reading the article correctly), presumably just to boot another one in the actual OS, despite the openness, sounds like it would _increase_ complexity even more.

I think Linus' opinion remains relevant over a decade later:

[http://yarchive.net/comp/linux/efi.html](http://yarchive.net/comp/linux/efi.html)

[https://plus.google.com/+LinusTorvalds/posts/QLe3tSmtSM4](https://plus.google.com/+LinusTorvalds/posts/QLe3tSmtSM4)

I'd actually love to see a "replace UEFI with regular BIOS" project ;-)

<comment><USER9> > Doesn't Coreboot implement UEFI?

No. Though I think there is some work to allow people to build a coreboot + upper layers of UEFI (presumably using the open source tianocore UEFI implementation) combination in order to boot operating systems that require UEFI. Just like it's possible to build a coreboot + seabios combination in order to boot OS's that require BIOS services (such as DOS).

> nonetheless, the idea of putting a full Linux kernel in the firmware (if I'm > reading the article correctly), presumably just to boot another one in the > actual OS, despite the openness, sounds like it would increase complexity > even more.

UEFI is very complex. I.e. it contains a TCP/IP stack (v4 & v6), and whatnot. The Linux IP stack is certainly a lot more battle tested than the UEFI one. And further, the idea is to allow the owner of the hardware to update the boot Linux kernel and not be beholden to the whims of the Intel & the HW manufacturer, who may not have the owners interest as a primary concern.

<comment><USER5> > UEFI is very complex. I.e. it contains a TCP/IP stack (v4 & v6), and > whatnot.

For better or worse. Most regular BIOSes contains IPv4 for PXE anyway.

UEFI just allows you to do PXE a little more cleanly directly from firmware without needing a disk to bootstrap the process with other stuff like iPXE.

<comment><USER9> > Most regular BIOSes contains IPv4 for PXE anyway.

Indeed. But that's UDP only, not TCP which is a lot more complex. UEFI contains TCP as well (IIRC recent versions of UEFI support boot over HTTP, similar to iPXE).

> UEFI just allows you to do PXE a little more cleanly directly from firmware > without needing a disk to bootstrap the process with other stuff like iPXE.

That's correct. OTOH when your firmware + bootloader stuff requires a TCP/IP stack, HTTP, support for booting from all kinds of software RAID setups, filesystem support in order to be able to find and load the kernel, and whatnot, replacing all of that with a minimal Linux kernel + userspace doesn't sound so crazy anymore.

</comment></comment></comment></comment><comment><USER22> From the presentation, looks like that - as of today - a ChromeBook are way more secure than a MacBook. Is that right?

<comment><USER23> I don't think much is publicly known about Apple's UEFI implementation; what if it's already stripped down?

</comment></comment><comment><USER24> They confuse exploit with vulnerability. Exploit = ready made program to take advantage of a security hole.

<comment><USER25> Yeah, but it's also extremely common to conflate those two. It's not as bad as how everything was somehow a "zero day" for a few years there.

</comment></comment><comment><USER26> What are the reasons the hardware the microcode is running on isn't directly exposed to the user? Why shouldn't we teach our compilers to generate code for the lowest level possible?

<comment><USER27> There is actually an significant technical issue with user-supplied microcode on modern fast CPUs: the control store is not fully writable and only small parts of it can be patched in the field. Reason for that is that the control store is probably the fastest block of memory in modern PC as it has to have latency on the same order as CPU clock period and usually has very wide data bus. Implementing this as entirely writable RAM would be uneconomical.

</comment></comment><comment><USER18> Slide 3 refers to "ISH" and "IE" among "X86 CPU(s) you don't know about". What are they referring to?

<comment><USER23> ISH: Integrated Sensor Hub

IE: Innovation Engine

There's also the PCU Package/Power Control Unit that no one talks about.

</comment></comment><comment><USER28> Does this indicate us end-users can apply it too to our own gaming PCs for example?

<comment><USER9> AFAIU at this point it's quite early, and the way they're working on it at the moment is to remove the ROM flash chip (which, depending on the motherboard, may require desoldering the chip) and flash it with a SPI programmer.

If you're up to that stuff, I think they'd be happy to have your help. Otherwise maybe you should wait and hope they'll figure out how to flash it from Linux, and make it robust enough that there's a relatively low risk of bricking your system.

</comment></comment><comment><USER29> So many layers with a single goal: preventing the user from owning the hardware.

</comment><comment><USER30> Is'n't GRUB supposed to play that role? Why a full kernel?

<comment><USER31> I think grub would usually be called after the boot firmware (like UEFI, or coreboot, or a traditional BIOS) has already initialized the hardware. Some of these systems can also directly load your operating system, and I would guess the one of the few thing missing from the boot firmware would be the filesystem support, and not much more

</comment></comment><comment><USER32> Compiling on demand is cool, but why is it needed?

<comment><USER33> If I understand correctly to make it more auditable. Still: tool-chain remains prebuilt and trusting trust issues. More auditable is better than less auditable.

<comment><USER32> Why is compiling on demand more auditable than just having open source BIOS?

<comment><USER33> UEFI is open source to some extent. There is an Intel project [1] that every company's UEFI is based upon. Even having fully open source BIOS is not enough by itself. The binary that is on your system can be anything. You can somewhat trust it if you can verify it, only if the project would use reproducible builds. But the hardware can still lie to you. What I also have in mind is the trusting trust issues [2].

[1] [https://github.com/tianocore/edk2](https://github.com/tianocore/edk2)

[2] [http://wiki.c2.com/?TheKenThompsonHack](http://wiki.c2.com/?TheKenThompsonHack)

</comment></comment></comment></comment><comment><USER34>
      > The problem
      > ● Linux no longer controls the x86 platform
      > ● Between Linux and the hardware are  at least 2 ½ kernel

Please release it for general public, so that everyone has the choice (for Intel+AMD CPUs).

      > code you don't know about: Kernel -3, Minix 3

Wtf "Minix 3" running invisible behind the scene

<comment><USER35> Parts of Intel ME are probably derived from Minix 3.

[http://blog.ptsecurity.com/2017/04/intel-me-way-of-static- an...](http://blog.ptsecurity.com/2017/04/intel-me-way-of-static- analysis.html)

</comment></comment><comment><USER36> How I read the title: replacing a exploit-ridden software, with another exploit-ridden software (but this one is open source)

<comment><USER37> Linux has undergone a lot more (hostile and friendly) penetration testing and bug-fixing than UEFI.

<comment><USER38> Linux is still certainly more "bug-ridden" than UEFI, because of size alone.

<comment><USER37> From the abstract:

> UEFI is a proprietary and closed-source operating system, with a codebase > almost as large as the Linux kernel...

And this kernel build is a trimmed-down one, removing functionality that isn't necessary to initialize the hardware and boot the full kernel.

</comment></comment></comment></comment><comment><USER39> 64x</comment>


Why F# - <AUTHOR> http://davefancher.com/2013/01/27/why-f/

<comment><USER2> _At one point the discussion turned to language productivity and Phillip remarked that writing C# feels like completing government forms in triplicate. As he elaborated I experienced a sudden burst of clarity into one of the major things that had been bothering me about C# – it's verbosity!_

It's like a parallel universe to the Scala/Java relationship.

<comment><USER3> Even more:

1\. Common ascendants

Scala is heavily influenced by OCaml and Haskel.

F# is descendant of OCaml, heavily influenced by Haskel.

2\. Past

Don Syme designed generics for .NET.

Martin Odersky designed generics for JVM.

3\. Erlang influence

Acca - heavily influenced by Erlang and OTP, MailboxProcessor looking quite similar to Erlang processes.

4\. Multiple paradigms

Both languages combine FP and imperative. Scala is more OOP, F# is more FP.

And (in my opinion) F# is looking more "clean" than Scala.

Simple thing in F#:

       let (|>) g f = f g

In Scala becames much more complicated. Also I think that discriminated unions are "prettier" than case-classes.

<comment><USER4> > _Scala is more OOP, F# is more FP_

That depends on your background. As far as FP goes, I wouldn't see myself working in a FP language that doesn't have type-classes or something similar. Clojure has protocols and multimethods, Scala has type-classes based on implicits, while F# doesn't have anything like that.

> _F# is looking more "clean" than Scala._

I don't know about that, I got used to Scala code. As far as the languages are concerned, I hate for instance how F# imported some language features from C# that really needed to die a long time ago, like static class members and operator overloading based on said static class members. Scala on the other hand cleaned all that crap. My problem with F# is basically that it has 2 type systems in the same language. Same problem that Ocaml has really, except Ocaml can also do some pretty cool tricks with structural typing for OOP that F# cannot.

But anyway, these issues are actually irrelevant because the choice for developers will end up being the answer to " _.NET or the JVM?_ "

<comment><USER3> I like what Odersky did with statics in Scala.

I agree that F# is not pure as Haskel and limited in comparison with OCaml and Haskel.

But the things that works in F# are working fine. In comparison with Scala: curring (by design) and tail-calls optimization are looking "more FP" in my opinion.

<comment><USER4> Yeah, but LISP doesn't do currying and I find Scala's support for currying just fine, although the syntax in F# is definitely much nicer.

Also Scala does optimize self-recursive tail-calls and it's pretty easy to work with trampolines. This is a platform limitation btw, as you can't optimize tail-calls without support from the VM. Mono, the open-source .NET implementation, also has problems with optimizing tail-calls (last time I checked) and I don't know about .NET 4.5, but .NET 4 was not optimizing tail- calls on 64-bits Windows. I remember reading a blog post about them fixing it on 64-bits OS in the latest version, but it wasn't the kind of feature you could rely on in the past.

I do have hopes that TCO will make it into the JVM at some point, as there are experiments for it available as part of OpenJDK. Unfortunately the JVM moves at a glacial pace in regards to such features. This is both a blessing and a curse.

I do agree with you. Not having full TCO in a functional language kind of sucks.

<comment><USER5>
       and I don't know about .NET 4.5, but .NET 4 was not
       optimizing tail-calls on 64-bits Windows.

Actually there's a .tail CLI Opcode in .NET for tail call optimization which is not usually emitted, but the JIT does tail call optimization on 64 bits (not on 32 bits though). You can witness this when you have a C# getter calling itself. On 32 bits, you get a StackOverflowException, on 64 bits it keeps running happily forever. For the specifics: [http://blogs.msdn.com/b/clrcodegeneration/archive/2009/05/11...](http://blogs.msdn.com/b/clrcodegeneration/archive/2009/05/11/tail- call-improvements-in-net-framework-4.aspx)

It's lower level and more limited than full blown FP aware TCO and it may not catch every call pattern, but it's been there for a while.

</comment></comment></comment></comment></comment></comment><comment><USER6> Isn't the first C# example (the class) a pessimization? Why not e.g.

        public class CircleMeasurement
        {
            public double Diameter { get; private set }
            public double Area { get; private set }
            public double Circumference { get; private set }
            public CircleMeasurement(double diameter, double area, double circumference)
            {
                Diameter = diameter;
                Area = area;
                Circumference = circumference;
            }
        }

Arguably there's still a fair bit of repetition due to the constructor, compared to the F# code (and I'm not sure object initializers can be used with private-set properties), but a third of the repetitions have been skipped through implicit backing fields.

<comment><USER7> Also most of that code can be written automatically with Resharper...

<comment><USER8> Resharper isn't going to read it for you.

<comment><USER9> This. I love IDEs, but arguments like 'the tool can generate $GARBLE for me!' are what keeps the haters a-hatin' and the mediocre programmers mediocre.

</comment></comment></comment></comment><comment><USER10> I am a .NET developer by trade, and any other .NET developers I know that have heard of F# love the idea of it and would happily use it. The sad part is that this is often as far as it goes.

I think C# is a fantastic language, but if the development of F# got to the point where I could swap out either language when building an ASP.NET website with no worries about compatibility or missing out on anything (ReSharper being a big one) I would happily use F# on a project, and I'm willing to bet that many other developers would.

What seems to be holding off many people, including myself, is a lack of knowledge and information about F# from the higher-ups.

</comment><comment><USER11> Being able to simultaneously develop in an imperative language (C#) and also a functional language (F#) at the same time on a project can be very appealing depending on what one is doing. With .net or mono, it makes it pretty easy to switch between them.

Sometimes I like having the power to use a functional language for a certain task, but would rather not develop an entire project with one. C# does have functional abilities (Linq), but it's not quite the same.

<comment><USER12> Agreed. I recently wrote my first F# code because the boilerplate associated with emulating pattern matching/discriminated unions in C# was so unappealing. I very much like the ability to wrap a piece of pure F# code in a C# API. I actually think the resulting F# is easier to explain to another developer than a huge collection of visitor classes.

I also feel that I wouldn't want to write the whole app in F#, because I don't feel like the OO/FP approach feels as natural for some parts of the app as plain old C#.

<comment><USER13> Could you elaborate on your last sentence? Especially with F# 3, the OO features should be syntactically the same as C#. You can use F# as a light- syntax C#, really, with a few minor caveats.

For curiosity, what in particular do you find to be unnatural in F#?

<comment><USER12> I'm just not really a fan of mixing OO or imperative and FP code in the same code unit, with the exception of a few things like LINQ which might be called FP. For anything with a lot of mutation or tight loops, I'd rather stick with C#, and save myself the hassle of explaining the code to other devs later on. The idea of one language for everything is nice, but C# has carved out a place for itself, and maybe F# should try to focus on those areas where it can differentiate itself by effecting shorter, more self-describing code.

</comment></comment></comment></comment><comment><USER14> Given that the two comments by pohl and yareally are appreciative of a functional language that interoperates well with a host language, I would like mention Felix, a language I have been lloking at the past few weeks.

It is whole program optimized, strongly typed, polymorphic, ML like language without a default evaluation order that can interact effortlessly with C and C++ code and compiles down to C++, another differentiator is the presence of coroutines

I can point to two tutorials. One explains Felix by beginning from C++ [1], the other describes Felix the language [2].

[1] <http://felix-lang.org/web/nutut/intro/intro_index.fdoc>

[2] <http://felix-lang.org/web/tutorial.fdoc>

No analogies are correct, but roughly

Felix is to C++ what Scala is to JVM (or F# is to C#). Note however Felix predates all these languages.

@Ingaz the analogy is a little more literal because Felix compiles down to C++ (unlike OCaML)

<comment><USER3> Interesting.

But I always thought that OCaml to C++ is what F# to C#.

<comment><USER15> Specially given OCaml's industry support, mainly in finance.

</comment></comment></comment><comment><USER13> Sounds much like my own experiences. I recently wrote some APIs in C#, and was reminded just how verbose and limited my writing becomes.

Even if I'm writing code that's not very functional, using F# as a better C# seems to work very well.

The main reasons to use C# are for legacy interop and perhaps if the code isn't a major focus and hiring someone capable of writing F# would actually be a limitation. Some folks also feel F# is clunky for imperative code; I haven't really felt that way.

</comment><comment><USER16> That's great. Will F# get support from Microsoft for our enterprise? Nearly every server is either HPUX, AIX or Solaris. There are a few blade servers running Linux. Java runs on every one of these platforms with great vendor support. What can we expect from Microsoft if we switch to F#? This is the reality of large monolithic parastatal enterprise.

<comment><USER17> The .Net Platform is a massive enterprise mainstay.</comment></comment>


Protect the Internet from Stop Online Piracy Act et al. - <AUTHOR> http://thedailycougar.com/2012/01/05/stop-sopa/

<comment><AUTHOR> so much for that. their webhost, MT, opposes sopa; i put up a banner as their developer, with an article filled with links. they took them down; still no news for 20 days now. called the "daily" cougar.</comment>


Hollywood promotes war on behalf of the Pentagon, CIA and NSA - <AUTHOR> https://medium.com/insurge-intelligence/exclusive-documents-expose-direct-us-military-intelligence-influence-on-1-800-movies-and-tv-shows-36433107c307 <comment><USER2> Is this a surprise to Americans (genuine question)?

It's plain as day that Hollywood films are strongly designed to influence the American's perception of war and their position in it/contribution to it when you watch it as a non-american.

That's before we even get into the "flag tax".

The more films you watch over the years the more cringeworthy it gets. Whether that's the whole one-man-army, or some other recipe, it feels like the viewer is being treated like an idiot.

<comment><USER3> Add cigaret smoking and alcohol consumption at every emotional turn in the narrative because that's "how people behave in the western world".

</comment></comment><comment><USER4> These types of relationships in Hollywood is well documented in the recursive movie Argo:

[https://en.wikipedia.org/wiki/Argo_(2012_film)](https://en.wikipedia.org/wiki/Argo_\(2012_film\))

[https://www.wired.com/2007/04/feat-cia/](https://www.wired.com/2007/04/feat- cia/)

[https://www.amazon.com/dp/0688163025/ref=as_li_ss_til?imprTo...](https://www.amazon.com/dp/0688163025/ref=as_li_ss_til?imprToken=-yJXNV- zyuc.ya-u1QuDPA&slotNum=0&tag=slatmaga-20&camp=0&creative=0&linkCode=w61&creativeASIN=0688163025&adid=1KDQS75WP88HYMY6Z8PV&)

</comment><comment><USER5> The essence of this article might not be hugely surprising to anyone, but it's still good that they did all the FoI requests and everything.

</comment><comment><USER6> That sucks, right? Now, who do you think is paying them to do that?

<comment><USER7> We are. The gun lobby hasn't paid a dime to Hollywood to make gun violence prevalent throughout the medium. John Wick double taps a hundred guys and we gladly hand over our money.

The Pentagon is just facilitating consumer demand with the handshake that they get to be the good guys, like John Wayne was when killing those other pesky non-white people.

</comment></comment><comment><USER8> Probably, because the US wars on behalf of Israel.

Is this news to anyone at this point? I'm sure we all know it's not Koreans or Eskimos running Hollywood.

<comment><USER9> That is a deeply stupid trope and conspiracy. War for profit, the glorification of war, and even propaganda have long predated the very existence of Judaism.

The whole concept is a feudal peasant propaganda going onwards like a zombie or a stray bullet, propelled onward through the centuries by absolute unthinking fools. Everyone except a royal essentially was stuck blaming some scapegoat for problems like the king's advisors but not the king himself without getting their whole family brutally murdered. In the European case that was generally the Jews but the Chinese and others had eunuchs because they didn't have any descendants to get offended on their behalf.</comment></comment>


The Nintendo Switch Switch - <AUTHOR> https://blog.cynthia.re/post/nintendo-switch-ethernet-switch <comment><USER2> At my college, we have an airgapped network for security classes, and when assignments are due, we often run out of ethernet cables. I had just mentioned I would bring a switch with me the next day and people were confused why I'd be goofing off with a Switch so close to the deadline.

Now we can avoid that kind of confusion! :)

<comment><USER3> > when assignments are due, we often run out of ethernet cables. I had just > mentioned I would bring a switch with me the next day

As I understand it, a switch will solve a shortage of ethernet ports, but will make the cable problem worse by requiring one extra cable. What's the purpose of the switch here?

<comment><USER4> I think they mean out of ethernet-cables-that-are-usable-to-get-network- access. So there's loose cables aplenty, but a shortage of cables that work in the sense of the far end being plugged in, due to a shortage of ports.

From the perspective of a student walking into the room with their laptop, that's a shortage of cable-ends that they can plug into their laptop to gain network access.

<comment><USER5> Either way, I'm surprised port security isn't enabled in an educational environment like this.

<comment><USER6> Why should it be, or even be detectable if OP attaches a switch? Afaik, those don't even have different MACs and even if they do, it is unlikely that the university has all the students laptops MACs.

<comment><USER7> Switches have MAC addresses.

</comment></comment></comment></comment></comment></comment><comment><USER8> I'm surprised to not see anyone talking about how Switch homebrew works and which devices are compatible. There's both hardware and software patches to worry about.

Here's the most up-to-date guide with version numbers and bootloader choices and such: [https://nh-server.github.io/switch-guide/](https://nh- server.github.io/switch-guide/)

</comment><comment><USER9> I'm [vaguely] interested in unconventional Android devices.

People often talk about "plenty of android devices with X" where x is e.g. a keyboard. But a search of Google, Ali Express, etc don't yield much except for conventional phones.

Is there a list or other source where I can discover these mythical interesting devices?

<comment><USER10> This is probably not quite what you're talking about, but:

I'm convinced there's a bunch of interesting single-purpose Android devices that simply aren't available to the general public. The manufacturers are large companies that you've probably heard of that don't do business on places like AliExpress. Unfortunately, the devices in question are "enterprise" products, and sold accordingly.

For example, the self-service kiosks at Taco Bell seem to be Android-based, though I can't confirm this. Good luck finding a supplier who will sell you those in lots of less than a thousand.

Swinging around to your original question, I occasionally hear about MIPS- based Android tablets made for Indian or Chinese markets. I'm sure they're on AliExpress, but I doubt they're clearly labeled as MIPS.

<comment><USER11> Norwegian Air's in-flight entertainment systems (screen on the back of the seat in front of you) are modified/custom Android tablets. When you open the flight tracker you can see the regular Android buttons at the bottom and pull down to see the notification area (which has been modified to contain no content). Didn't find a way out of their sandbox, maybe next time I'll bring a male-male usb cable and see if I can get in with adb..

</comment></comment></comment><comment> core-questions Cute project! Sometimes it's nice to see a simple hack that elucidates perfectly just how amazing it is to actually be able to run our own software on a variety of devices. With Linux support, these devices will be useful for decades to do random things.

<comment><AUTHOR> thank you :) I have other evil network related ideas for this in the near future :p

</comment></comment><comment><USER12> i am a bit disappointed that what has been done is basically installing linux and setting up a bridge using usb ethernet dongles. not what i was expecting. its still somewhat amusing though.

<comment><USER13> Yeah, and only two ports so it's not useful as a switch. It could be useful as a router though.

<comment><USER14> Very expensive extension cable? :-)

</comment></comment></comment><comment><USER15> Woah, didn't realize you could install Linux on the switch now. Got me thinking about the uses for a portable tx2 tablet...

<comment><USER16> There is a hardware vulnerability in the first ~18 months of Switch systems produced. It's a USB boot mode (like DFU mode in iPhones) from the underlying Tegra chipset and is triggered by pressing the Tegra "Home" button (not the Switch Home button!), one of the Volume buttons, and Power while injecting the software you want to boot over USB. That button is exposed as one of the pins (#9, iirc?) of the right-side Joycon rail, so all you have to do is short that to ground with anything you like. 3D printed jigs are cheap and common.

Your system will get banned from Nintendo online services if you run any Switch-mode homebrew—for good reason since piracy and online hacking have unfortunately become rampant—so the most I've done with mine is run the standalone Hekate to dump my system's unique keys and internal storage a few times. I'm still enjoying the system for its intended use too much to want to go offline forever, so I'm just holding on to an exploitable system for a few years until the Next Big Thing comes along.

As for obtaining an exploitable system, all you need is one manufactured before July 2018. I keep the following in the Notes app on my phone for when I see a used system for sale in the wild:

Exploitable Switch Serial Ranges

Serials beginning with XAW1:

\- XAW1007XXX and below are safe to buy

\- XAW1008XXX not safe to buy, probably patched

\- XAW1009XXX and above definitely patched

Serials beginning with XAW7:

\- XAW70017X and below are safe to buy

\- XAW70018X not safe to buy, probably patched

\- XAW70019X and above definitely patched

Serials beginning with XAJ4:

\- XAJ40052X and below are safe to buy

\- XAJ40053X not safe to buy, probably patched

\- XAJ4006XX and above definitely patched

Serials beginning with XAJ7:

\- XAJ70042X and below are safe to buy

\- XAJ70043X not safe to buy, probably patched

\- XAJ7005XX and above definitely patched

Happy hunting!

<comment><USER17> > Your system will get banned from Nintendo online services if you run any > Switch-mode homebrew

What you can do is set up a separate Homebrew-enabled partition on your SD card without any wifi networks to connect to, and keep your normal, untouched partition for normal use.

For similar reasons, installing Android/Linux is safe.

</comment></comment></comment><comment><USER18> Living in the US, I'm a bit jealous of the 100mbit upload...

<comment> kube-system If you're in a Fios area, Verizon's running a deal on 940/880 Mbps right now for $74.99.

<comment><USER18> Cable or DSL only here and I pay about that for 120/16.

</comment></comment></comment><comment><USER19> The author's main motivation is probably the cool name "Nintendo Switch Switch" but a bridge is not a switch, so it should be called a "Nintendo Switch Bridge".

<comment><USER20> Being a little pedandic here aren't we? For all intents and purposes a bridge is basically a switch with two ports. At the end of the day this is running on a Linux kernel so it can basically be any networking device you want it to be, even a router.

<comment><USER14> Or an extension repeater adapter?

<comment><USER19> No, even a two-port bridge is smarter than physical extension/repeater. I.e. the bridge will not forward ethernet frames to the other port if the destination if assumed to be in the originating network segment.

</comment></comment></comment></comment><comment><USER21> Putting PiHole on this would have been wonderful.

<comment><AUTHOR> _notes down_

</comment></comment><comment><USER22> Bravo. So is the Fast (megabit) Ethernet speed bottleneck with the USB ports?

Do you think you could get gigabit Ethernet speeds?

</comment><comment><USER23> Interesting CSS you've got there on your blog page ;)

<comment><AUTHOR> blame my friend Ben, I just cloned his blog and Cynthiaified it :P [https://blog.benjojo.co.uk/](https://blog.benjojo.co.uk/)

<comment><USER24> I think he was alluding to the HN look and feel

<comment><USER23> I'd be inclined to believe she understood that.

</comment></comment></comment></comment><comment><USER25> Was this inspired by the meme?

</comment><comment><USER26> No one:

Absolutely nobody:

A Nintendo Switch that works as a switch

</comment><comment><USER27> Thats nothing I turned my playstation into a space station.</comment>


Just started in a project, how to be productive on the day one? - <AUTHOR> https://medium.com/@alabeduarte/just-started-in-a-project-how-to-be-productive-on-the-day-one-cb149aaf8172 <comment><AUTHOR> We always forget about this after we know how to run everything and it becomes part of our job, nobody cares about documenting stuff, until the next dev come along... and also forgets it... then the cycle goes on...</comment>


Apple iPhone Landing on T-Mobile April 12 for $99 - <AUTHOR> http://www.wired.com/gadgetlab/2013/03/apple-iphone-tmobile/ <comment><USER2> "no contract, followed by 20 months of $20 payments on top of monthly pre-paid data plans."

<= how is that not a contract?</comment>


Zed Shaw's Son of Sam Email Server: Write Email Apps in a Modern Style - <AUTHOR> http://www.zedshaw.com/projects/sos/ <comment><USER2> re: performance, any smtp mail server worth its salt will be limited by I/O write performance (email should be sync'd to stable storage before signalling acceptance). CPU spent pushing data over pipes/unix sockets won't register. I don't see any measurements in there, so it sounds like "X Windows is slow because it has to talk over a socket" analysis.

"The short version: Classes and methods beats pipes and processes."

Well, it depends. For latency on when processing a single large email, yes, perhaps. One-pass processing will give you that (although I think he skipped over how SoS manages to hand off incoming email to SpamAssassin without "passing it over a pipe to a perl process")? Surely we are trying to compare like with like, right?

But mail processing is a parallel activity. Reducing latency on a single message is not the same thing as improving throughput. What do we find out about that?

"But, sendmail and postfix are still way better at huge scale bulk, so it relays to them."

Ah, OK. So this is a boutique solution, for personalised handling of small- site stuff. Guess what? We already have that, since (as noted by the spamassassin example) you can set up the big solutions to hand off a message to you to process in the language of your choice.

Sorry for -ve rant. I didn't see any -ve comments in this section and wanted a little critical balance. Nice software is always nice, and you don't have to have a reason to want to write it.

But I don't like unjustified justifications. If you're going to claim better performance, give numbers.

<comment><USER3> What is "-ve" ?

<comment><USER2> Negative, sorry for abbreviation.

</comment></comment></comment><comment><USER4> Now all he needs are IMAP and POP3 implementations and we might finally have a mail stack that mere mortals are capable of setting up without screaming profanities.

<comment><USER5> I found Dovecot pretty easy to install, but setting up spam filtering (especially SMTP-time) is a lot of work.

<comment><USER4> Any of the puzzle pieces individually aren't bad, it's putting them all together then adding additional pipes to stuff like (as you mention) SPAM filtering, etc. that make it such a huge pain in the ass. In an ideal world, there would be some package available where you simply specify the credentials for the machines you want to use for each piece (smtp, pop3, imap, secure pop3/imap interface, etc.) and it will setup everything for you.

In an ideal world.

</comment></comment></comment><comment><USER6> Does anyone else have trouble reading Zed's blog because of the high contrast white-on-black text theme? I have to look away after around a minute, and even then I have ghosts burned into my retinas.

<comment><USER7> r.e. title image, does anyone else think the chick on the left looks like she has a dick?

<comment><USER8> Sounds Freudian, dude...

<comment><USER8> <http://en.wikipedia.org/wiki/Freudian_slip>

That, but used differently.

</comment></comment></comment></comment><comment><USER9> I wonder why he chose Mako as a template system as opposed to Jinja (django style). Looking at the Mako syntax.. it just doesn't make any sense.

<comment><USER10> Django templates are designed to be designer (non-programmer) friendly, becase of its CMS/newspaper origins.

Mako's goal is to create a powerful, programmer friendly and FAST templating language .

A lot many times having code in templates is a faster way to develop, especially if the presentation layer is mostly intended to be used by a programmer.

Also, you _can_ write mako code which has good separation of concerns.

How is : <% for r in rows %> ${r.name} <% endfor %>

less readable than :

{% for r in rows %} {{r.name}} {% endfor %}

<comment><USER11> jinja2 is as fast as mako. mako is popular, though, so I don't begrudge anyone choosing it as a default for a project.

<comment><USER12> jinja2 also lets you easily change your delimiters to make it suit your style, which is nice.

</comment></comment></comment></comment><comment><USER13> Does Twitter have message-ids like email?

I ask because email has the property that all replies are linked to the original message, creating a dependency tree of ID's. Any system that doesn't create a unique message-id is a step back from email.

<comment><USER14> Twitter messages have unique URLs (permalinks). It doesn't exactly have replies, though; you can send a message @someone (public, but intended especially for the given person) or send a private direct message, but there's no standard way to indicate which message you're replying to.

This is an intentional step back from email. Twitter is a bad place to have long back-and-forth conversations by design.

<comment><USER15> The Twitter timeline knows the specific message a reply is connected to.

<comment><USER16> ... but is actually just inferring, and frequently gets it wrong.

<comment><USER17> With in_reply_to_status_id and proper client support, that should go away, hopefully.

</comment></comment></comment></comment></comment><comment><USER18> Interesting. I wrote something very much like this for internal use at my company. Basically we have a lot of legacy alerts and messages and reports that are "dumb" (i.e. if a cron job runs once a minute to detect problems within one minute it will keep on mailing until it's fixed because it has no state) so the new mail system sits in front of it all and says, hmm, saw that a minute ago, won't bother the humans with it again, they're probably busy actually fixing it.

It's not, and never will be, a general purpose mail server - old-fashioned SMTP servers have their flaws but it's as good as it gets if you want mail that can reach anyone, anywhere.

</comment><comment><USER19> Tis the season to write SMTP implementations.

[http://undeadly.org/cgi?action=article&sid=2008111208464...](http://undeadly.org/cgi?action=article&sid=20081112084647)

</comment><comment><USER20> <http://smtpd.develooper.com/>

<comment><USER21> And FWIW, qpsmtpd handles _a ton_ of mail with no apparent problems.

With that in mind, scaling a mail server is about speeding up disk access, not implementing it in a "fast" language. I'm sure Zed's thing can do just as well.

</comment></comment><comment><USER22> Is it okay to give a technology the name of a serial killer?

<comment><USER23> Bell Labs named three of their operating systems after castrated men, a notoriously bad movie, and Hell itself. Names that make marketers cringe have a long tradition.

<comment><USER5> And let us not forget SATAN.

<comment><USER24> Which was written by Wietse Venema, who went on to write the finest general purpose SMTP server in existence. But, he chose a more suit-friendly name (Postfix) for the mail server.

<comment><USER23> Don't forget Dan Farmer also worked on SATAN. And FWIW the name history on Postfix was: VMailer, then (in parallel) IBM Secure Mailer (forgotten) and Postfix (alive). Wietse's previous work (tcp_wrappers?) didn't have particularly suit-unfriendly names either. Maybe it was Dan who suggested SATAN?</comment></comment></comment></comment></comment>


How to promote your business for free (checklist) - <AUTHOR> https://ghostwriter.ai/2020/05/25/how-to-promote-your-business-for-free-checklist/%22 <comment><AUTHOR> Hi founders! Struggling to promote my business on a budget, I've listed my workflow to keep it measurable. I share my thoughts and my experiments, as well as my checklist. Let me know your ideas about how I can make it better.</comment>


YC's 2016 Holiday Gift Guide - <AUTHOR> http://www.ycgiftideas.com/?2016 <comment><USER2> The thing that really strikes me about this list is how nearly everything on it is offered at a premium price.

One thing I loved about the tech world while I was growing up was that it was consistently doing more with less. Wikipedia displaced Encyclopaedia Brittanica by being both better _and_ free. Hotmail was a clear win over AOL and Yahoo's offerings and then Gmail was faster and handled spam better while giving more at every price point (including free). Computers became both faster and cheaper. Amazon and Netflix broke down even the movie industry and ended the $70 per tape movie sale price.

The offerings on the linked page are _mostly_ things that would be out of the budget of an American family living on a median income. I won't call out any specific examples or hurt feelings, but a random sampling of several companies of any part of the list (other than non-profits) will show a clear trend.

It's hard to say if the same was true 20 years ago and those few companies that broke out into famous successes happened to be particularly aimed towards a broad appeal, but it's clear the current trend is doing more with more.

<comment><USER3> "Amazon and Netflix broke down even the movie industry and ended the $70 per tape movie sale price."

Is that a typo? When were movies ever $70?

<comment><USER4> In the 80s and early-to-mid 90s, most movies weren't offered for sale to consumers. Big blockbuster movies sometimes were, but those were the exception. Places like Blockbuster or your corner video rental store bought VHS movies for around $70/ea and then rented them to consumers for $3 or so. That changed somewhere around the late 90s, when people began buying movies on VHS and DVD, and the price dropped significantly.

Source: I worked for a small video store in the early 90s.

<comment><USER5> >Places like Blockbuster or your corner video rental store bought VHS movies for around $70/ea //

Isn't that just rental prices vs consumer prices. It's a tort to by the consumer copy and rent it out so large companies could never get away with that instead they bought a more expensive copy that is basically the tape with license to rent it??

Turns out it's more complicated than that: [https://entertainment- industry.knoji.com/the-history-of-the-...](https://entertainment- industry.knoji.com/the-history-of-the-vhs-movie-industry/) & [http://boards.straightdope.com/sdmb/showthread.php?t=692441](http://boards.straightdope.com/sdmb/showthread.php?t=692441).

Would be interested in similar info with global comparison, eg how was the situation different in UK, Germany, etc..

</comment></comment></comment></comment><comment><USER6> Unless there's a significant financial incentive to do so (ex: 2-to-1 match), don't give people gift cards. Gift cards are the biggest scam ever pulled by the retail industry, or more specifically, perpetuating the idea that giving cash is taboo.

The one exception would be giving a preloaded credit card to someone who otherwise doesn't have access to one (ex: a niece/nephew that's far away) and only if there are no fees associated with initially loading it. I think AmEx has some kind of cyber monday promotion for that today. If you have to pay anything more than $0 then forget about it though.

All that aside, Markhor looks interesting. I don't recall seeing them on HN (guess it's less "tech" than most YC companies) but the loafers look nice. Anybody here have any experience with them? Always on the look out for a better pair of shoes...

<comment><USER7> > _Unless there 's a significant financial incentive to do so (ex: 2-to-1 > match), don't give people gift cards. Gift cards are the biggest scam ever > pulled by the retail industry, or more specifically, perpetuating the idea > that giving cash is taboo._

The problem is that cash often feels like not giving a gift at all. If you exchange fifty-dollar bills with a sibling, it feels pointless. Gift cards can alleviate that feeling. If you give your sibling an iTunes gift card it doesn't feel like nothing was exchanged _even if_ they give you one back.

Gifts are also an opportunity to give someone something they wouldn't typically buy for themselves. Ideally that would be a thoughtful gift but if you can't think of one, cash is likely to end up in the "general fund", while a gift card to a restaurant or a massage or a bookstore will be used for its intended purpose (with some risk of it being lost or sold on some card resale site).

<comment><USER8> My (to be fair, minor) issue with gift cards is that they're essentially saying "hey, have some money...but you can only spend it at this one place of my choosing, not yours".

I'm aware that this is pretty low on the list of problems you'd hate to have but it's there nonetheless. Let's say a relative has heard me talk about some Arduino project I'm messing with or something to do with building a computer. They don't know much about either of those things but they know I like "techie" stuff and gadgets so they get me a gift card from Best Buy, as that is the only electronics store they're really aware of.

Now I've got a $100 gift card to a store that either doesn't sell any of the things I'd want to buy or charges a hefty markup over what I'd normally be willing to pay. Sure, I could sell the gift card to some bargain hunter for $95. They'd get $5 off, I'd get most of the value to spend elsewhere...

...but that's a hassle and not at all what the gift-giver would want to necessitate anyway. The other major option is to just suck it up and either buy something I don't really want/need out of the available Best Buy catalog or maybe buy that $100 item that I would've just found for $70 at Microcenter or Newegg.

I think you nailed it when you pointed out issues of perception and tradition. Giving cash to someone other than an aging/retired parent or a young child comes off as odd. It's almost like you didn't care enough to think of a gift that they would actually enjoy or shows that you know them. I just think gift cards are about the same in that regard, only less desirable or useful.

<comment><USER9> >"hey, have some money...but you can only spend it at this one place of my choosing, not yours".

In some circumstances, that's actually an advantage, and works as a positive for the recipient.

When I was in college, money was tight. Had to be very careful with every dollar. When relatives would give me money for holidays, it would just go into the general fund and had to be treated with the same care as the rest of it. That money was certainly appreciated and made my life easier, but not a lot of fun was had with it.

But when I'd get gift cards to Amazon or Best Buy or Borders, suddenly it was a license to go crazy. To recklessly buy electronics, books, and luxuries I couldn't otherwise justify. It wasn't just a gift of money. It was a gift of money without worries attached.

<comment><USER10> That's why I'm buying my grandparents a Subway card. You can reload it online and they love Subway they just don't want to splurge to go there whenever they go in to town. I know if I load this up for them they'll have a treat whenever they're out to a doctor's appointment. It has the added benefit of me being able to reload it when they run low.

</comment></comment></comment></comment></comment><comment><USER11> Somewhat related: "Gift guides" is one service that I considered unsolved. Every holiday (Christmas, birthday, father's day, mother's day, etc) I struggle to come up with gifts for the people I care about. In many ways it's demoralizing. If I care so much about someone, why can't I think of a good gift for them?

There are lots of factors here, but I think it basically boils down to, as adults we generally get what we need/want during the course of the year. So gift giving is really the art of giving things people hadn't thought of, but still would like. That's a tall order!

In recent years I've tried searching for "experience" gifts, which I believe to have more value. Nights out to exciting dinners, fun shows, events, etc. But the sites that serve those needs (Groupon et. al) are really rather scummy. All the businesses on there tend to be poorly reviewed; just using those sites to push polished turds of a deal.

I look all the time for gift guides and helpful sites and always come up empty handed. The lists on Amazon mainly push Amazon's products, which are garbage, or really common/useless stuff (there's no-one in my life who needs a kitchen stand mixer; they all already have one). I've read idea threads on Reddit, and those mostly boil down to "Man, all I want for Christmas is a pair of shoes. I'm really poor." followed up by a bunch of helpful Redditors. Not that that is a bad thing, but it's useless as a gift recommendation.

Before the deluge of rhetoric replies: yes, rampant consumerism, corporate marketing holidays, donate in someone's name, etc, etc. I happen to enjoy giving gifts, and would love to give more meaningful, personalized gifts (like those experience gifts I mention). I'm just really bad at it. And regardless of that rhetoric, can you honestly tell me that you would pass up an opportunity to build a successful "gift guide" service? I can't even imagine how lucrative something like that would end up being.

<comment><USER12> I think the main reason it's unsolved is that there's a lot of personalization that needs to happen. If you (as a son, let's say), can't figure out what to give your dad, how can a site? The reason why a lot of holiday guides give common or useless stuff is because they have to generalize across age ranges, ethnicities, locations, and preferences.

The best gift guide would also probably be one that gets to the core of what you are lacking -- you might want a specific book, for example, but what you're looking for is how to learn a language, and that specific book didn't really matter (unless it was the best one to help you learn). This is really hard to do though, unless you are Santa.

<comment><USER11> Yes, that's definitely true, it's not a simple task by any stretch of the imagination. I think the experience gifts is perhaps a more viable option. For example, a mystery dinner type deal (mystery theater + dinner) I think is a good gift for a lot of friends and family. Cooking classes are pretty general too. I've been gifted cooking classes before and they were really fun (they focus on making it date night sort of deal so it was geared towards couples). But both of those I've had challenges gifting. I couldn't find any well reviewed mystery dinner type deals, and the few good cooking class type things I found did not offer a gifting option. Buying tickets to events, like concerts, is tough too, again because they don't generally offer gift options. If I could buy a "blank check" sort of ticket to the local playhouse, that'd be nice.

My big gripe is the sites that get mostly there. They ask questions about the intended recipient, like age, gender, and tastes. At the end of it, though, the recommendations are really bland. I don't mind sorting through a list of possible gifts, with only a few good results. But the results I get are the kind of generic "gifts" you see on the end-caps of Kohl's during holiday season. As I mentioned, people generally already have what they need. Gifting often comes down to finding something the person wouldn't have otherwise thought about. So it's necessary to show unique options. Again, the accuracy doesn't need to be terribly high. If I only find one good option amongst 40, that's fine. Instead I get 200+ kitchen stand mixers...

</comment></comment></comment><comment><USER13> What, no STEM toys!? Have you visited the toy gadget section of your typical store, e.g. Target, BestBuy lately? It's full of crappy stuff marketed under the buzzword "STEM" for unknowing parents. Examples: Construction kits from Vex Robotics, WowWee's CHIP Robotic Dog, etc. etc. Problem with these "STEM revolutionary" toys is that they are not open ended, even with the so-called programmable ones kids get tired after a few hours of play. Another good example of such a toy is the BB-8 robot. You love this when you get it but quickly lose interest after you realize that it's a glorified remote- controlled car.

For truly inspiring toys like this one ([https://www.kickstarter.com/projects/withpiper/piper-a- minec...](https://www.kickstarter.com/projects/withpiper/piper-a-minecraft- toolbox-for-budding-engineers)) you still have to hit Kickstarter or search the Web. Why is that?

<comment><USER14> Erector / Meccano is a nice open ended STEM toy that could use some disruption and metrication and modernization and no themed disney licenses please.

Unfortunately there is this strange attractor that sucks everyone who trys it into "I wanna make kids version of 80/20 because 80/20 is both cool and insanely expensive" leading to "heres my custom aluminum extrusion of immense expense that works with nothing out there, but now I know why 80/20 is incredibly expensive" leading to "remember those guys?"

<comment><USER15> I personally think MakeBlock ([http://www.makeblock.com/](http://www.makeblock.com/)) does a great job of being a modernized metricized replacement for Erector sets.

It's not particularly cheap - probably costs ~$300 to get a decent sized kit or a smaller kit with enough spare parts to experiment.

All of the mechanical parts are machined reasonably well and fit together in a large number of useful configurations - I really like them for rapid mechanical prototyping.

The electronics are basically arduinos with beginner-friendly connectors - haven't used them myself but have generally heard good things about them.

</comment></comment></comment><comment><USER16> A note to any future startup founders reading some of the broadly skeptical and pessimistic comments on this HN discussion thread:

The people commenting here are _not_ your customers.

For every 1 person posting a snarky comment, there are 100+ simply browsing the gift guide. A few are saying "Hey, that's neat!", and a few are making purchases today or in the near future, either for themselves or as gifts. It's their behavior that you should worry about.

If you think you can make something people want, then please focus on that. Don't let your non-customers discourage you.

(Disclosure: co-founder of [https://www.pantelligent.com/](https://www.pantelligent.com/) \-- one of the products on the gift list.)

<comment><USER17> Couldn't agree more. I'd like to add: Don't let not-customers butter you up! Find your customers and get their feedback. Lots of people will think yours is a great idea: just not great enough to pay you for it!

<comment><USER16> Very true!

</comment></comment></comment><comment><USER18> I see that Acre (a home construction company) is on there. Kind of odd for a gift giving site.

"Merry Christmas! I bought you a house!"

<comment><USER19> And like all "modern prefab" houses, it is extremely overpriced.

$400k for 1200 sq. ft is nuts.

<comment><USER20> It's all relative. In Des Moines it's outrageous (although still worth it!), but in the Bay Area or similar market it's a steal. Comparable homes out here can be 2-3x.

We did our fist home at $145K for 1200 sqft and can offer than again late next year. Margins are too slim to grow at that rate, re:Tesla.

<comment><USER19> You're confusing the value of land vs the value of the structure itself.

The $1 million properties in the bay have structures on them that are worth maybe $150-200k tops, whereas the rest of the value is in land.

A high land:structure value ratio is not uncommon for the US coasts, in high demand areas (especially that have older houses, built in the 20s and 30s that will need significant renovations to be brought up to a safe living standard)

That's why it's outrageous. If you're considering paying 400k just for the structure, that means you already own a piece of land worth at least $800k where to put it. So we're talking about being a millionaire to be able to afford something like that. So let's be honest for a second and not pretend like you're not targeting the top 1-1.5% demographic here.

I noticed you're a founder for the company.

Construction is a perfectly competitive market. It shouldn't have economic profits, which means it should be paying your costs and salaries and that's it. I'm not sure what was the rationale for your getting into the business, but the modern prefabs are a dime a dozen and the thing they all get wrong is affordability. If you don't think I can go to another state and bring 4 construction workers to build the same type of structure for $200k, sourcing my own materials and paying them salary, you're just deluding yourself.

I wouldn't be surprised to learn that companies like that go bankrupt by the dozens, because they all miscalculate the economics of the market.

But I do wish you good luck in your business.

<comment><USER20> Most of the value is in the land, and the homes are no different than $200K homes in the midwest. However, they are worth more and cost more because of very high demand and limited supply of labor and land. Putting a standard new home on a property that just sold for $1.6M (a home that should, anywhere else cost $300K) allows you to flip that property for $3.2. That's in Redwood City in a decent, but not over the top (for the Peninsula) neighborhood. Same home in a different market is worth quite a bit more.

Bare minimum for non tract construction is about $250/sqft or 3x the rates in the middle of the country. For high performance homes, $600+ /sqft is not uncommon.

Construction is not perfectly competitive, especially here in the bay. Guys (builders/trades people) pick their rates because demand on them is so high. Regulation, permitting and crazy processes mean the barrier to entry are higher than ever before. It's not fixable with day labor and a spunky attitude.

That said, our current line is the Tesla Model S of homes, late next year we'll be able to offer our Model C range for a much broader market.

Thanks for the well wishes!

</comment></comment></comment></comment></comment><comment><USER21> "A monthly subscription to The Flex Company will make a relieving gift for any ladies on your gift list who suffer from menstrual bloating, cramping, or leaking." I couldn't make this up.

<comment><USER19> Make sure to never tell them about it for maximum creepiness. Imagine how delighted your extant female friends will be to start receiving tampons in the mail.

</comment></comment><comment><USER22> I feel like I'm looking at a Sharper Image catalog. Lots of neat, expensive, completely useless things (for the everyday person) that I'm surprised anybody buys.

<comment><USER23> And the few things here that seem legitimately useful (the sous-vide circulator, for instance) are competing with proven, solid, reliable products (e.g., the Anova precision cooker: 4.5 stars on Amazon with 1100+ reviews).

<comment><USER24> It made me remember that I as considering buying a sous-vide circulator as a gift. So I clicked through, found that this one in particular was out of stock, and ordered the Anova instead at half the price. So, +1 for advertising, -1 for fulfillment execution :(

</comment></comment></comment><comment><USER25> Some of these look pretty cool, I'd never have put most of them as YC companies. I wish there was a way to filter by location I'm in, or even just say "Not in SF" and see what's still visible. I assume anything subscription is out, and I'm not going to click individually through everything that requires shipping and see who has sane/any shipping prices to the UK.

</comment><comment><USER26> Super excited to see Watsi on the list of suggested donations. I became a monthly contributor a year and change ago, and every time I get a raise, I raise my contribution. I love the work they do, and the way they'll select the patient for me -- I find the process of combing through each one emotionally a bit too onerous!

</comment><comment><USER27> +1 for MTailor. Not affiliated at all, but I ordered several shirts and a suit in the last year and it fits amazingly (and it's quite convenient to order from the app). Do be aware though, that it will take ~ 4 weeks (slower than buying from a store, but as fast as getting it tailor-made)

</comment><comment><USER28> Never heard of Click and Grow before this. Seems really cool and will definitely consider for gifts

</comment><comment><USER29> Awesome idea this list. What an impressive (quantity and quality) - list of companies. Some of them are known to me as YC portfolio companies - because I buy from them or have heard of - but loads of them, I would not have guessed that they are YC funded.

</comment><comment><USER30> Random: I'm surprised that Boosted Board is still using "perfect for the last mile of your commute" in their marketing slogan.

All the influencers I've seen with it (like Casey Neistat) use it to commute pretty long distances! Their biggest benefit is, you can quickly travel from point a and b, especially in a conjested city, on something that you can carry under your arm or in a backpack. No need for bikes, public transit or cars.

</comment><comment><USER31> I'm a big fan of Boosted, but I wouldn't put them on a holiday gift buying guide. Shipments have been halted for a month due to a quality issue, and even if they fixed it today, the waitlist is so long there is no way you would receive a board this year.

</comment><comment><USER32> Interesting mix of gift stuff. Some of it definitely piqued my interest and I will probably return and consider it. It's all sort of semi-luxury goods, but that's, well a holiday gift kind of thing. And it's hard to make money selling to people who don't have a lot of money.

Wheely's Cafe is probably the most _interesting_ gift there to me. It appears to be a franchise-in-a-box. I wonder if there's one in Seattle! Probably an _ideal_ gift for an 18 year old with a work ethic and lack of academic passion.

</comment><comment><USER33> Soundboks link does not work. It points to [https://bookface.ycombinator.com/company/974](https://bookface.ycombinator.com/company/974) which is locked.

<comment><USER34> sorry about that - fixed!

</comment></comment><comment><USER35> Aside from the Donations section (YC's nonprofits), having much of the YC portfolio collected like this really deflates a lot of the cachet about YC, its companies, and its place at the Silicon Valley cool-kids table.

I look at this list and I see problems that never needed solving, middlemen -- sorry, _curators_ \-- reselling cheap Chinese crap, and a cult of jamming Bluetooth and USB into just about anything that holds still long enough because IoT.

There is no magic, "man I wish I'd thought of that" moment here, just a few "that makes sense" products in a sea of Valley bullshit.

Perhaps YC was always like this, but they hadn't presented that information in such an easy-to-consume list before.

<comment><USER36> YC has funded 1500 startups. You're generalizing from a narrow selection, which is clearly bogus. (Did you expect quantum computing on a Christmas list? Well I guess this _is_ HN.)

Also bogus is drawing conclusions about companies' long-term strategy ("problems that never needed solving") from a holiday gift guide.

Just because some of us aren't big on consumer goods is no reason to throw out basic logic!

<comment><USER35> I am commenting on the narrow selection that YC itself assembled.

I realize there are many YC companies missing from this list, and from the consumer goods and services industry in general. But if this [ahem] curated list leads to the kind of generalizations I made, the list itself may share some blame for that.

If appeals to "basic logic" and wait-for-the-pivot are YC's best shot at getting me excited about the 2016 YC Holiday Season as presented by YC, then by all means carry on.

<comment><USER36> > I am commenting on the narrow selection that YC itself assembled.

That's patently untrue. You commented on YC.

<comment><USER2> At least the way the comment read to me, gamache's meaning was closer to "this is bad marketing for YC / this dilutes the brand" than "YC doesn't fund anyone solving real problems".

<comment><USER36> Your interpretation is more charitable than mine. Well done!

<comment><USER2> Well, I'm sure distance makes it 1000x easier! Your job is one not many could do.

</comment></comment></comment></comment></comment></comment></comment><comment><USER37> I'm glad you guys are making this stuff, but I was really hoping for things that I might personally want or enjoy.

</comment><comment><USER38> I always wonder how people can say things like this: "They've created a wifi- enabled immersion circulator called the Nomiku so every home chef can prepare simple, Michelin-starred meals."

I mean, I bet that literally zero Michelin-starred meals have been prepared with this device. And the phrase "Michelin" or "Michelin-starred" is trademarked and has a clear meaning. If I was a marketer for this company I would feel uncomfortable using a phrase like that.

With that aside, it's odd to see so many YC companies on this list with the same faddish business models of "send some stuff to you every month you don't get to pick the stuff but trust us" and "do something that's already pretty simple in your kitchen or home but now using an overcomplicated device with machine learning and wifi"

The former model isn't really a technology empowered business model at all, it's really a buying or deals club driven by marketing. And the latter is ostensibly technology but it doesn't seem to have really any of the accelerative effect on efficiency that real genuine networked software based businesses have.

Seeing all of them in one place like this, backed by YC, is pretty disheartening.

<comment><USER39> Not this specific device perhaps, but Sous Vides (as I understand it) can form part of a commercial kitchen, at high-end restaurants.

A quick google shows: [http://www.telegraph.co.uk/news/2016/09/10/michelin- restaura...](http://www.telegraph.co.uk/news/2016/09/10/michelin-restaurant- ends-up-in-hot-water-over-sous-vide-hygiene/) \- A Michelin starred restaurant, which uses a Sous Vide. They are generally quite expensive to by for home use.

I won't comment on the rest, but I'd agree that the list doesn't exactly fill me with wow.

<comment><USER38> But that doesn't make sense. If I'm selling a stainless steel bowl like they use in commercial kitchens can I say "now every person ever can whip up Michelin-starred omelettes" or something?

I mean I suppose I could, but it makes me sound like an idiot. It's also plainly factually untrue. No Michelin reviewer will ever see my meal, no meal I make will receive a Michelin star. It's literal nonsense.

And Sous Vide has been around awhile and totally solid units can be had for about $100. I remain confused about what value-add YC and/or this company is bringing to the space.

<comment><USER40> > But that doesn't make sense. If I'm selling a stainless steel bowl like they > use in commercial kitchens can I say "now every person ever can whip up > Michelin-starred omelettes" or something?

People who are selling, say, chef's knives absolutely say that kind of thing. It's kind of silly but not obviously more so than any other marketing puffery.

</comment></comment></comment></comment><comment><USER41> I just want to mention that Le Tote and Vet Pronto are awesome. We use them both and they make life easier and more pleasurable. Always great things to wear and our cat doesn't have to leave home.

</comment><comment><USER42> YC backed Scaphold.io is a backend as a service platform powered by GraphQL They have a freemium mode to check out as well as $25 code "HappyHolidays2016" (exp.1/15/17) Visit their slack channel that the founders use to teach you how to start scapholding!

[https://scapholdslackin.herokuapp.com/?email=](https://scapholdslackin.herokuapp.com/?email=)

</comment><comment><USER43> I wouldn't even accept these businesses as design clients. Gross.

</comment><comment><USER44> YC admins: The "S" in Sixa is linking to Doblet.

<comment><USER34> thank you kind stranger! updated.

</comment></comment><comment><USER45> on edit: There are so many of these companies that are basically stylized versions of Alibaba listing pages or are basically re-skinning of some business process that, turns out, doesn't actually need to be disrupted. Transporting gasoline to fuel up someone's car? Really? We're going to be energy wasteful just to shave 10 minutes off someone's week?

The other disturbing thing is how many of these gadgets are IP enabled. I really hope YC has some kind of IoT advisor who insists upon security at a low level.

<comment><USER36> Snarky dismissals of other people's work are not welcome on HN, so please don't post like this.

I tend not to post as many moderation scoldings in YC-related threads, but that was such a bad HN comment that I'll break the pattern for it.

Edit: you rewrote your comment to make it (somewhat) more substantive and less snarky. In principle that's fine, but it would have been more respectful to preserve the original context for the reader.

<comment><USER46> Have you considered that these companies might not actually be that innovative and that the parent comment was a legitimate criticism?

<comment> grey-area It's terribly easy and not very useful to dismiss things in a vague way like this, and it's often wrong, particularly if talking about an entire category or market.

e.g. google could have been dismissed as _just a search engine_ but it turned out eventually to be exactly what a lot of people wanted.

<comment><USER47> (As written now) I think it can be legitimate critic. Google really grew out of making a better search engine algorithm and all. Some of these businesses are essentially reselling things and not really focused on innovating the substantial part of the business. That can also work of course, but I can see why it's less interesting.

</comment></comment></comment></comment></comment><comment><USER48> Man, I was hoping for books. The YC staff book recommendations are usually good.

As it is, there is nothing of value on this list. Most of these are absolute rubbish. But most outstandingly rubbish is Vidcode. It claims to teach code to teens/tweens, but speaking as one myself, this looks like the same kind of condescending, manipulative, "we be down with the kids, yo" crap that attracts teens the same way that a man covered in shit attracts high society.

Look, if you're going to try to attract teens, or any group, to coding, don't try to look hip, or trendy. Treat them like they're regular people (not just a bunch of cretins that you have to stick sparkles in front of every five seconds to keep their attention), give them real tools, and allow them to attain the knowledge to build what they want to build. That's what works.

And how do I know that it works? Because that's what I got, when I decided I wanted to learn programming. That's what countless kids got in the 80s, where children would, through self-motivation, learn assembler, and type in opcodes by hand.

And maybe those people are the minority. In fact, they almost definitely are. But you'll never get the majority, because the majority _doesn 't care_. Making things that look like they're designed to appeal to an insipid stereotype won't change that.

<comment><USER36> > _It claims to teach code to teens /tweens, but speaking as one myself_

I'll cut you some slack for that reason, but you need to (re-)read [https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html) and nip that internet rant thing in the bud ("the same way that a man covered in shit attracts high society"?) if you want to continue commenting here.

<comment><USER48> Yeah. Sorry dang. I'll go re-read that now, seeing as I quite clearly need of a refresher.

And there's no need to cut me slack for my age (although it's nice of you to do so): I broke the rules, and I'm responsible for my actions. Just like any other member of HN.

Twice in three years is too many times to have to say this to you.

 _bows head in shame_

<comment><USER49> Bummer that your apology got downvoted, since this is a gracious response to being called out. Kudos. We've all been jerks on the internet — the key is learning how to curb that behavior, which it seems like you're doing pretty well.

<comment><USER48> >Bummer that your apology got downvoted

That trend has reversed. Dramatically. I suspect your comment has something to do with it.

As for why I wrote GP in the first place, it's an important part of _qwerty 's code of internet interactions_ (yes, I have a private code of internet behavior: so do you, most likely, whether or not you've formalized it. And yes, I made up that name on the spot. It sounds pretentious, but it needed a name so I could reference it, and all the others souded equally pretentious. So there you go):

"If you were wrong (whether in your argument, or in your manner), then take the appropriate action, whether that be an apology or a concession. Do so graciously: There is no shame in being wrong. There is shame in refusing to admit it, or accepting it poorly.

They're words to live by. And you can trust me on that, because I've lived by them (or tried to), and it pays off.

Was I irritated about a product aimed at my demographic, and how dramatically it missed the mark? Yes. But that doesn't excuse my expression of that anger.

</comment></comment></comment></comment></comment><comment><USER50> Pick up one of these for the hacker/developer/AI enthusiast in your family:

[https://getasteria.com/](https://getasteria.com/)

<comment><USER14> WRT the whole christmas gift theme, "Estimated delivery date: Summer 2017."

The explanation of why it can't run on a phone and why it needs a new currency is a little unconvincing.

"asteria can connect with any bot, command any service" But it can't run on a phone, why?

Certainly "its an open alexa/siri/now" sounds very interesting.</comment></comment>


Beijing will have its revenge on Hong Kong - <AUTHOR> https://www.ft.com/content/12119746-e67d-11e9-b112-9624ec9edc59 <comment><AUTHOR> Readable version, sans paywall: [https://archive.is/eosgd](https://archive.is/eosgd)

I was particularly struck by a claim tucked in the final paragragh that "One senior police official says privately that as many as a quarter of his officers are joining peaceful protests in their spare time."</comment>


A surprising campus free speech problem: left-wingers fired for their opinions - <AUTHOR> https://www.vox.com/policy-and-politics/2018/8/3/17644180/political-correctness-free-speech-liberal-data-georgetown <comment><USER2> You are in college for one reason and one reason alone: education. College shouldn't be used as a platform for politicking. If you wish to do politics or activism do it outside the campus. Just like how you would if you were in school or a work environment. Why should colleges be an exception to the rule?

It's ridiculous that college students are used for furthering political agendas. Campuses have become a hotbed for politicians and activists alike. They no longer need to do the hard work of visiting neighborhoods and interacting directly with people about their problems. They have a readily available resource in the form of college students whom they can influence. Lest we forget about peer pressure students face: you are in our group as long as you align with our political ideology/activism.

And what is the point of inviting speakers when everyone is pretty much divided and will never accept the other's views? This entire process just creates more fissures than bridge gaps.

It's sickening that we as a society allow this in the first place. And this isn't just a societal issue but also an ethical issue: should taxpayer money be used to fund such events when it was provided to further education standards?

This isn't just restricted to the US; it's pretty much a worldwide phenomenon.

</comment><comment><USER3> Who are the people that we are sending off to college?

I don't believe that the vast majority of kids are intellectually nor morally challenged enough before they enter the college atmosphere and those that do give profound thought to life (i.e. why is it OK for the president to curse and say borderline racist things in his speeches but thought provoking books with similar sounding rhetoric are banned from schools) are typically ostracized from the rest of society for not conforming.

Along with social media and the competitiveness of this country for getting ahead in life, college kids, possibly unwittingly, fight tooth and nail to align with others in solidarity of causes and opinions that were never properly reflected upon before by the individual but because they want to maintain friendships or make new ones or even put themselves out there for the sake of personal vanity or gain, you now suddenly see the blind leading the charge against opposing opinions and rhetoric, but to what end?

Instead of people telling others how they should feel about every topic and what side they need to take, people really need to focus on why they care so much in the first place about anything in general and equally important, recognize when they are being trolled just for the sake of getting a reaction out of them.

</comment><comment><USER4> I notice the usual undertone here that universal free speech is an inherently good thing. While I personally believe that wholeheartedly, I've spoken with a non-negligable number of students who've come to the conclusion that free speech is not an inherent right shared by everyone, and that they are justified in trying to quash the free speech of those they disagree with.

The dispute over campus free speech isn't going to go anywhere until it's acknowledged that not everyone wants free speech.

</comment><comment><USER5> >Some campus free speech critics, I suspect, aren't operating in good faith. For them, the entire debate is a way to attack universities as hopelessly and dangerously liberal — to undermine higher education for nakedly partisan reasons.

This is almost assuredly so. However, I find it curious that partisans would seek to attack an institution, unless the institution were itself seen as nonpartisan. And wonder whether there is any truth to that perception. What causes said perception?

</comment><comment><USER6> The article is claiming that a new study was just released, but the closest thing I can find is a Knight Foundation study that was done in March. Can anyone find the study the article is referring to? It doesn't seem to be in any of their links.</comment>


So You'd Like to Make a Map using Python - <AUTHOR> http://sensitivecities.com/so-youd-like-to-make-a-map-using-python-EN.html Use Pandas, Matplotlib, and Basemap to extract, clean and re-arrange data, before creating a series of thematic maps. <comment><USER2> Cool article, explains how you can do anything using Python, although doesn't mention Mapnik. However, for most people, these days I would recommend to try TileMill ([https://www.mapbox.com/tilemill/](https://www.mapbox.com/tilemill/)) to make a map. The CartoCSS can let you style anything based on attributes and it also lets you add and style raster data.

<comment><USER3> Can you draw brightly colored lines on roads for a bus map? I like to mess around analyzing bus routes, but so far I haven't been able to find a computer-based tool for drawing them effectively.

<comment><USER4> Tilemill ingests geographic/spatial data, it doesn't have much provision for interacting with it.

You might find that someone has already drawn the bus routes you are interested in:

[http://wiki.openstreetmap.org/wiki/Buses](http://wiki.openstreetmap.org/wiki/Buses)

(Then you just have the task of pulling the data out of OSM and translating it into a format that you can work with)

<comment><USER2> Also, I just realized, if the bus routes are not in the OSM, he could use the new iD editor and add the bus routes to OSM under his account, and then export :)

</comment></comment></comment></comment><comment><USER5> My biggest problem with maps these days is the data license for commercial use. I dont need very detailed map, usually administrative level 2, but it's hard to find accurate sources that dont make you pay thousands of dollars per small userbase. We create our own app and distribute it, therefore cannot exactly estimate our userbase. Does anybody know of a decent source with good, fairly detailed world maps and liberal license ? Doesn't have to be free.

<comment><USER4> Openstreetmap?

Just requires a simple attribution.

(there are certainly lots of things to work on in the data)

<comment><USER5> The freely available dumps to SHP are not up -to-date or have insufficient details and my several efforts to produce an SHP myself have failed. I'd like to be able to produce a shp dump that contains only selected "layer" for given country.

<comment><USER6> I think OGR (gdal.org) has an OSM driver that you can install optionally to go from OSM -> SHP. Problem is that SHP and OSM are not 100% compatible formats, so data massaging may be necessary in many cases.

Also, Geofabrik.de will export it for you for a fee.

<comment><USER5> I tried QGis 2 with built-in OSM support but I was unable to properly download data through it. Perhaps I was doing something wrong, I'd love to read a good tutorial on this. If you happen to find one, let me know!

<comment><USER6> It's been a while since I played with OSM data. OGR 1.10.0 supports OSM, and you can grab it from various sources (gdal.org or gisinternals.com/sdk/ if you want latest windows binaries).

Converting OSM -> SHP should be relatively similar to other formats, and there are a good number of tutorials out there for using OGR. Here's their page on OSM: [http://www.gdal.org/ogr/drv_osm.html](http://www.gdal.org/ogr/drv_osm.html)

</comment></comment></comment></comment></comment></comment><comment><USER7> In our company we use python to make maps, but we go with the traditional GIS approach, dependencies?: postgis and mapnik. The first two examples would be solved by a single postgis query, the last one maybe would require some extra work. But nice work anyway, bookmarked.

</comment><comment><USER8> Surprised to see that they're using basemap instead of cartopy. There's nothing wrong with using basemap, but it can be a bit clunky, i.m.o.

Then again, cartopy is only a year or two old, so it doesn't have the traction that basemap does. It's gained a fairly large following very quickly, though.

<comment><AUTHOR> Doing the same thing using Cartopy is next on my list!

</comment></comment><comment><USER9> There's also Vincent[1], which has some mapping capabilities and is built on top of Vega (a "visualization grammar" for d3js).

[1]: [https://github.com/wrobstory/vincent](https://github.com/wrobstory/vincent)

<comment><AUTHOR> Yep, I'm very interested in exploring the (Geo)pandas / Vincent / Vega intersection.

</comment></comment><comment><USER10> Cartography? Hasn't everything already sort of been discovered, though by, like, Magellan and Cortés?

<comment><USER11> Cartography is about making _maps_ , not discoveries.

<comment><USER12> Except for figuring out about Pangaea and continental drift.

<comment><USER8> That has nothing to do with cartography. That's geology.

Geology often uses cartography to display data and various spatial analysis methods to analyze data, but what you're describing is purely geology.

Sorry if I sound snippy... But as a geologist, we get a bit annoyed when one of the major discoveries of our field is attributed to "those darn geographers"! :)

</comment></comment></comment></comment><comment><USER13> Very cool, article. I've always loved maps and mapping and python is my preferred language. The only thing I would mention is that it would be nice to have a pic of the results earlier in the article, that's just from the "let's look at this article, seems cool but what exactly is he teaching me" angle. I'm more likely to try the code if I can see the results up front. Otherwise it was a really cool example.

</comment><comment><USER14> [http://www.robots- everywhere.com/re_wiki/index.php?title=Geh...](http://www.robots- everywhere.com/re_wiki/index.php?title=Gehandler) I wrote a sort of google earth API wrapper thing in python if anyone wants it. Windows only though.

</comment><comment><USER15> What are blue plaques?

<comment><USER16> Historical markers: "on this spot in 18xx something happened" kind of thing.

[https://en.wikipedia.org/wiki/Blue_plaque](https://en.wikipedia.org/wiki/Blue_plaque)

</comment></comment><comment><USER17> Very interesting, it had never occurred to me that there were probably python libraries for mapping. My ArcMap license expires in less than two weeks, perhaps I will give this a shot before I re-up.

</comment><comment><USER18> This is exactly why I prefer R for static maps. Would have taken like a quarter of the time, if that.

<comment><AUTHOR> A) The article's (quite clearly) not geared towards concision.

B) Personally, I'm delighted that there are various ways to do this. It's good for innovation.

</comment></comment><comment><USER19> Cool! I have been looking for an article like this. Thank you for sharing.</comment>