Python 3 Q&A by a prolific core developer - <AUTHOR> http://ncoghlan_devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html

<comment><USER3> Unconvinced.

</comment><comment><USER4> I think all the outrage is that Python 3 behaves like a new language in how everything needs to be ported (albeit at a magnitude less complexity than going up the language chain or across the playing field, like C to Ruby or C# to Java) but it still incurs the new language cost, while being the new version of the old language.

If people just thought of Python 3 as the replacement to broken text model Python, like how XHTML tried to supplant HTML and failed, and then HTML5 supplanted XHTML with some quirks, the same thing happens here.

Also, I adore the comments about the GIL. I understand the benefits of using a single language in every circumstance, but (even) a JIT scripting language is not sufficient for all problems. I guess it depends on if a polyglot code base is more complex than handling multithreading and the corresponding memory complexity and gimmicks, and that is developer centric.

Still, I would much rather just stick processor bound code in openMP C functions and call those from Python when need be. It seems like the "right" answer to the performance problem, without losing much productivity.

</comment><comment><USER5> It's just one of the little things, but I'm still very conflicted about _print_ as a function. The reasoning is sound, but I just can't seem to get used to it. It feels especially cluttered and clumsy with new-style string templates:

        print('ver: {}'.format(', '.join(str(i) for i in sys.version_info)).upper())

versus:

        print 'ver: {}'.format(', '.join(str(i) for i in sys.version_info)).upper()

Despite the more consistent naming (ConfigParser -> configparser), simplified api (iteritems() -> items()) and all other syntactic improvements, I somehow still find 2.x code more enjoyable. Writing small scripts in Python has kind of lost its charm for me.

This, of course, is all very subjective and I'll probably grow over it in a few thousand lines of code. I hope you're all less sensitive to the little things that annoy you.

</comment><comment><USER6> Unreadable on iPhone, what's up with its stylesheet? Safari won't even zoom...?

<comment><AUTHOR> Yeah, it seems to be the default RTFD style…try: [http://www.instapaper.com/text?u=http%3A%2F%2Fncoghlan_devs-...](http://www.instapaper.com/text?u=http%3A%2F%2Fncoghlan_devs- python- notes.readthedocs.org%2Fen%2Flatest%2Fpython3%2Fquestions_and_answers.html)

</comment></comment><comment><USER7> I wish python 3 would make default parameters immutable. It seems to go against the "simple", "explicit" python philosophy, not to mention that (it seems) most people tend to learn about the feature through debugging.

description: <http://effbot.org/zone/default-values.htm>

<comment><AUTHOR> Yeah sometimes I wish there were a warning whenever someone defines a function with [] as def arg.

<comment><USER7> I know pylint and pychecker do this, but not pep8. Opinions on those?

</comment></comment></comment><comment><USER8> Python3 and Perl6 can both fuck off.

I look forward to the day PyPy is considered the real Python. Look at PyPy's homepage (<http://pypy.org/>), doesn't even mention Unicode as a significant feature. Instead it talks about speed, security, concurrency, and compatibility with the current _real_ Python (2.7.2) - all the things real Python programmers care about and expect the Python developers to focus on.

PyPy may be some way off but I want to find the developers and hug them for setting the right vision and _trying_.

I just donated $50 and if you hate Python3 you should too.

<comment><USER9> Or if you really like Python 3 (as I do!) you can donate those $50 to Python 3 support in PyPy: <http://pypy.org/py3donate.html>

</comment></comment><comment><USER10> From the article:

 _Back in reality, though, complaining about the GIL as though its a serious barrier to adoption amongst developers that know what they're doing often says more about the person doing the complaining than it does about CPython._

It was a solid, well-argued piece up to this point. You do yourself and the Python community a disservice by writing off your critics as ignorant. It sounds petulant and childish, and is wrong.

There are valid arguments on both sides of the GIL argument, but neither side's advocates are ignorant or bad programmers.

<comment><USER11> In my experience with Python, the issue of the GIL comes up far too frequently. I use Python first as a prototyping environment, and once I have a proof of concept I like to optimize for performance (without re-writing the entire program in a more appropriate language). Most of the stuff I do is CPU- bound, but also involves large amounts of data. So, without writing C code as an extension, I can either suck it up and deal with single-core performance, or I can use the multiprocessing module and hope that inter-process communication isn't too expensive or difficult to code (and no, it's not always as simple as just using multiprocessing.Queue). I am under no delusion that the GIL will ever be "fixed", but I am disappointed that I spent years working with Python before I got into more CPU-intensive tasks and eventually hit the brick wall that is the GIL. And of course, comments like you mentioned, which attack my competence as a coder, are not constructive.

<comment><USER12> I'm interested to hear about your CPU-bound Python programs that also involve large amounts of data. Normally such problems are I/O bound and are better solved in a data parallel way. (Graph problems can sometimes be an exception, but those are still generally I/O bound.)

For writing extensions, have you considered Cython?

<comment><USER11> Sure thing, here are two examples:

1) My current project at work is a GPU-accelerated keyword-matching engine. The project was started before I joined the company, so I had no say in the choice of Python. Keywords change infrequently, while we analyze a continuous stream of incoming text. There are several million keywords, ranging from small to enormous in size. Aho-Corasick ([http://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_string_mat...](http://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_string_matching_algorithm)) is a pretty ideal algorithm for this scenario, which we use for the GPU matching kernel.

AC requires some preprocessing of keywords into a deterministic finite automaton (basically a suffix trie). This is very expensive for a large number of keywords with a large number of characters. The DFA grows to something like 10GB while being built.

Meanwhile, the main engine loop has to be running continuously, while updating keywords in the background. The engine is a service available to other systems on our network, so it uses multiple threads for concurrent I/O. The problem is that the GPU performance is so ridiculously high that the CPU can't keep it fed with data. I've profiled it and this is not a memory-bound problem...the CPU simply cannot keep up with the document streams that we send to it.

The concurrent I/O threads cannot reasonably be split across processes because they need a shared memory space for the data structures driving the engine. So clearly, the background keyword updating is a problem if it runs in the same process as the rest of the engine. I spent a lot of time trying to figure out how to get the keyword updating working in its own multiprocessing process. It's a complete hack to work around the failings of Python (I can go into more depth about the implementation issues if you'd like). And this is why I loathe the GIL.

We use Cython for some aspects of the code, but the keyword updating has yielded very little gain. It's difficult to rewrite parts of the keyword updater as more optimized Cython because it uses some language features that do not seem to be supported in Cython.

2) For a personal project, I need to do a lot of timeseries processing. I'm using Python to prototype, with the intention of either optimizing it eventually or possibly rewriting it in a more suitable language. I've found parsing timestamps to be particularly CPU-intensive, while working on gigabytes of data. Most data I send to a multiprocessing process will have to be returned in some form eventually, so communication costs are huge. So huge, in fact, that I only see a 10% speedup from splitting the workloads evenly across six cores. Profiling reveals that the majority of the "processing" time is actually just waiting on data getting sent back to the main process. This would not be a problem with a shared memory space.

<comment><USER13> re 2) - <http://docs.python.org/library/mmap.html> ?

<comment><USER11> Interesting, thanks for the tip! I'll look into it and think about whether it makes sense for my timeseries analysis.

</comment></comment></comment></comment></comment></comment><comment><USER14> Why can't I use Python 2 modules from Python 3 program? Is there any effort to make it possible?

<comment><AUTHOR> There is no such thing as "Python 2" or "Python 3" modules. There are just modules that will go belly up because they are incompatible with one or another.

I presume you mean that you can't access your old Python 2 modules from a new Python 3 installation. That's just because Python installations usually don't share their modules (i.e. site-packages). You can try to install them using Python 3 (usually just python3 setup.py install) and see if they work or not.

<comment><USER15> There's a deeper question here: why doesn't the interpreter contain a compatibility layer for the old code? There's no really good reason for the old modules to go belly up in the new interpreter. At least no other than the man months needed to implement that layer. But then, that probably would still be orders of magnitude less than migrating all the libraries that keep people from migrating...

<comment><AUTHOR> That's essentially what __future__ imports are for – things that go further would be simply too much work and Python 3 was about shedding weight, not shipping essentially two interpreters. (while they are very similar from outside, the innards couldn't be more different)

<comment><USER15> Cool. So this way porting things to Py3k got easier. That perfectly explains why all the critical libs have been ported fast and no one remembers about Python 2 anymore, doesn't it?

Shedding weight is sometimes a step in a good direction. But we need to draw a line at some point. As the history of Python 3 adoption shows, that point maybe was not optimal.

Edit: forgot an angle:

Also, __future__ is fixing the incompatibility of what people are using with something that they can't use. I'm not sure if it's the right problem to solve.

<comment><AUTHOR> As it has been pointed out for several times: the adoption of Python 3 is doing just fine. It has never been expected that everybody will be using Python 3 today.

But look at all those current efforts at Canonical, Django, Twisted…it's not like nothing is happening and we expect a knot to burst. Far from it! Porting started slowly and has gained a momentum by now that has surprised myself. It's not like we changed the language completely like perl6 did.

In the result, we'll have a better Python for it.

</comment></comment></comment></comment></comment></comment><comment><USER16> The following situation happens with other data than just unicode. This seems an argument for static typing in general.

      The reason this approach is problematic is that it means the traceback for an unexpected UnicodeDecodeError or UnicodeEncodeError in a large Python 2.x code base almost never points you to the code that is broken. Instead, you have to trace the origins of the data in the failing operation, and try to figure out where the unexpected 8-bit or Unicode code string was introduced. By contrast, Python 3 is designed to fail fast in most situations[...]

<comment><USER17> Well the lighter solution and more Pythonic solution is strong typing. If a Unicode string and a byte string could not be combined then the problem would not exist AFAIK. For example "a" + 1 is an error, why is u"a" + "s" not an error?

<comment><USER18> > For example "a" + 1 is an error, why is u"a" + "s" not an error?

"a" + b"s" is an error in Python 3.

<comment><USER17> Sorry, that was my point. Strengthening the types in python3 solves the problem without static typing being needed at all.

<comment><USER18> ok.

Static types solve it earlier though, especially for little-exercised code paths which could be forgotten or missed in testing.

</comment></comment></comment></comment></comment><comment><USER10> In the last few hours, this has been silently added to the post, and I cannot post a reply on that site:

 _Jumping on the internet to say that "they" (specifically, the people you're not paying a cent to and who aren't bothered by the GIL because it only penalises a programming style many of us consider ill advised in the first place) should "just fix it" (despite the serious risk of breaking user code that currently only works due to the coarse grained locking around each bytecode) is also always an option. Generally speaking, such pieces come across as "I have no idea how much engineering effort, now and in the future, is required to make this happen"._

Fine, then don't bother. But don't insult us or call foul for our pointing out the downside impact _on us_ of this decision. Your assumption that we are naive rubes who don't know how to code is really, really wrong.

I have a very good idea how much engineering effort would be involved in fixing the GIL, and I am well aware that Python has involved many person- millennia of gratis work, and am appreciative of both. However, I still disagree with the Python devs' obviously entrenched position that fixing the GIL isn't worth the effort, and I will continue -- even when shouted down by the likes of you -- to advocate for the GIL's removal or some equivalently good solution. (As I said earlier, I am not opposed to STM solutions, but the current one performs unacceptably without special-purpose hardware.)

Why? Because I have a single, selfish interest in this. I depend heavily on Python now, and want the language to be better. I have written many lines of Python 2 code that rely on the threading primitives in the standard library. Perhaps it was foolish of me to expect that the threading model offered by the standard library, modeled on Java's threading primitives, would some day work in the same way as Java threads do in practice. Nonetheless, I am left with a real world problem: my CPU-bound threaded Python code does not scale well to multiple cores. I need the GIL fixed, or to rewrite my Python code, or to migrate to another language that supports the standard model of threading programming that real-world programmers have been using for several decades, and which has built-in support from all major operating systems. Or, sure, wait for STM to be ready for prime time and migrate my thread-based semantics to the new STM-based semantics.

The best path right for me right now is migration to Jython or IronPython. But then we are still unsupported orphans, living in the third world of Python 2.X.

I guess it comes down to: do you want people to actually use this language to write programs they want to write, or do you want Python to be an advocacy platform for "correct" programming? Python's pragmatism has always appealed to me, so the ivory tower reaction to the practical concerns around the GIL really seem dissonant. (And this is coming from an MIT AI Lab Lisp guy who would rather write everything in Lisp. But Lisp lacks Python's wonderful, high-quality third party libraries and developer-centric pragmatism regarding unit testing, documentation, etc.)

I know you are tired of hearing people bitch about the GIL, but, really: people write multithreaded programs. They should work as advertised, using native OS threading primitives and taking advantage of the native OS thread scheduler. Why does Python offer threading primitives if the language is not meant to support, from a practical standpoint, multithreaded programs?

<comment> old-gregg _...the standard model of threading programming that real-world programmers have been using for several decades..._

What standard? And whose "real world"? The need for threads has always been controversial even among OS kernel devs. UNIX/Linux/BSDs have twisted and non- trivial threading histories peppered with religious wars similar to this one. And which "several decades" are you talking about?

There is no such thing as "standard threading model". To some, a thread is just a flavor of fork() with a wrong parameter and plenty of "real world" programmers continue to believe that kernel-level threads is a hack. And please, do not make it sound like Python threads are useless. Far from it.

Python threads are not what you are used to. That's pretty much TL;DR of your comment.

 _...Python's pragmatism has always appealed to me, so the ivory tower reaction to the practical concerns around the GIL really seem dissonant..._

I feel like they are being dragged into it though. The original motivation behind GIL support has always been a pragmatic one: removing GIL will make the entire codebase more complex, harder to hack on and will complicate and slow down the development/maintenance of the libraries. That's pretty pragmatic.

But a fairly vocal groups of users started to claim, similarly to you, that programming with threads is _supposed_ to work like they expect it to work according to make-believe "threading standard", to which GIL supporters (correctly, IMO) replied that shared memory + locks is not the only/best approach to concurrency. It is easy to be offended by this answer but it doesn't invalidate their point.

<comment><USER10> _Python threads are not what you are used to. That's pretty much TL;DR of your comment._

Correct. I am used to programming language threads that work the way computer scientists and programmers have typically described them -- for example, as in this (I hope uncontroversial) Wikipedia article:

<http://en.wikipedia.org/wiki/Thread_(computer_science)>

When I say "standard model of threading", I am not talking about nuances of call conventions to the underlying OS thread primitives. I am talking simply about running multiple streams of instructions, bytecodes, or other units of computation in parallel, within a single OS process.

<comment><USER19> That Wikipedia article defines threads in terms of operating systems. Only one small part of that article concerns how threads are exposed to programming languages.

You can't talk about running multiple streams of instructions or bytecodes in parallel without talking about the nuances of how they share memory. Semantics of a multithreaded memory model are a highly "opinionated" thing -- there are lots of possible ways to define it, and the definition can have widespread effects on efficiency, ease of programming, and the guarantees that the runtime can provide. For example, an important aspect of a Python memory model would be that no Python program can SEGV the interpreter due to a race condition.

I recommend the following reading to get an appreciation for how much really goes into a memory model and how far from "simple" or "standard" it is:

      http://en.wikipedia.org/wiki/Memory_model_(computing)
      http://en.wikipedia.org/wiki/Java_Memory_Model
      http://www.kernel.org/doc/Documentation/memory-barriers.txt

Python is a lot harder to define a good memory model for than say Java, because in Python lists and dictionaries are primitive objects. If you say:

      x['A'] = 1

...that is a single operation that must not corrupt the dictionary, even if multiple concurrent threads are mutating it. In practice, this means that you need to either make every such mutation wrapped in a lock (which adds a lot of locking overhead) or you need to use lock-free data structures (which are still relatively experimental and architecture-specific).

<comment><USER10> I agree that a so-called dynamic language like Python is at something of a disadvantage because it must make atomicity guarantees that lower-level languages like C need not.

I still don't think it's reasonable to conclude that typical programmers are fine with their threads not really running in parallel, or that the GIL isn't worth bothering to fix, even though fixing it would be hard. In my original post yesterday, I pointed out that as the language footprint has grown, Python's disadvantage in this respect has increased: it is much harder to remove the GIL now than it was in, say, the 1.5 era when there actually was a (problematic) GIL removal patch.

We've gotten way off track, but the original point I was trying to make was that 1) the GIL really is a problem for not-purely-theoretical programs written by competent developers, and 2) that the 2->3 transition, by complicating the language and increasing the workload for the alternative implementations, has made it less likely than ever that the GIL problem would be resolved.

And, indeed, Nick explicitly confirmed this by saying the GIL is basically a dead issue for the CPython devs. His post made many good points about the merits of the 2->3 transition, and in particular pointed out some ways that 3 has reduced work for the alternative implementations, but I remain unconvinced overall. And not out of ignorance or incompetence, as he implied.

<comment><USER19> I still think your position is unreasonable, because your inherent assumption is that the GIL is a "problem" that needs a "fix." This terminology is appropriate for a situation where the status quo could be improved without giving up any of the benefits of the current implementation. But this is not the case; removing the GIL in the way you advocate would add CPU and memory overhead that everyone would pay, even in the single-threaded case. And this is to say nothing of the practical problems of maintaining compatibility with existing C extensions.

The GIL is not a bug, it's a threading model. You wish the threading model was something else. You insist on your particular vision of an alternative threading model without acknowledging its downsides. You make no indication that you have actually considered or tried the alternative concurrency models that CPython _does_ support, like multiprocessing, greenlets, or independent processes. You make no objective arguments for why your desired threading model is better than the ones that are currently available, except that you could avoid changing your code. You accuse Python of failing to live up to some accepted standard for what a "thread" should be, when in fact no such standard exists, especially for high-level, dynamically-typed languages like Python. If anything, newer languages are moving _away_ from shared-state concurrency; see Erlang, Go, and Rust.

I don't think you have malicious intentions, but I urge you to reflect on what you are demanding and whether it is reasonable. What may look to you like "obvious" brokenness that demands an "obvious" fix is really a lot less clear- cut than you seem to think it is. I feel for the Python developers who have to deal with this complaining all the time.

<comment><USER20> To Python-level code, Python's threading model is pretty much exactly the same as that supported in all "fast" languages such as C and Java (even Go, ever pragmatic, has locks). Given that Jython already allows true multithreading and PyPy is trying to emulate it with STM, it's reasonable to see the GIL more as an implementation bug that won't be fixed for practical reasons than as a threading model... even if Python _also_ supports alternate threading models that are perhaps better for most applications anyway (if strictly less powerful).

<comment><USER19> > To Python-level code, Python's threading model is pretty much exactly the > same as that supported in all "fast" languages such as C and Java

Yes, but Python also exposes higher-level operations like table manipulation as language primitives.

> Given that Jython already allows true multithreading

That may be, but as I mentioned this has an inherent cost, both in CPU and in memory. Therefore it is not a strict improvement over CPython, just a different direction.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER21> From the article: "The only downsides of this approach are that it means that CPU bound Python code can't scale to multiple cores within a single machine using threads, and that IO operations can incur unexpected additional latency in the presence of a CPU bound thread."

Response: The only downside? As a Python user suffering from JVM envy, I have to say that that's a SERIOUS downside!

Here's why: (1) Python is slow. Almost any real life program in pure Python will have CPU-bound components. (example: BBCode parsing on my forum).

(2) Most programs that need to scale won't need to scale beyond a single machine (the most active web forum on my continent runs on a single quad core server)

Therefore the need to be able to scale CPU-bound python programs to multiple cores on is a very real need. Even though we accept that removing the GIL is hard, let's not insult real-life Python users by suggesting that their needs are not real.

<comment><AUTHOR> The article doesn't say multi-CPU scaling isn't necessary. It says that threads are usually the wrong answer anyway.

There are great process based ways to scale out – look no further than Erlang to see it's true.

<comment><USER21> I use Python for my day job. I've experimented extensively with Java and Scala. I must say this: Threads are AWESOME. Threading is the most flexible model of concurrency because you can can build programs that EFFICIENTLY implement "alternative concurrency models" like message parsing and STM with threads. Threading is supported natively by every OS. And fast.

Erlang is hyped as the ideal model for concurrency, but in practice is a niche product that's primarily useful for programs that are almost pure IO - <USER2> servers, routing components like proxy servers and packet switches.

The Erlang model does NOT apply to python, anyway, since Python processes are nothing like Erlang processes. Unlike Erlang processes, Python processes are very heavyweight and message passing between them is costly.

<comment><AUTHOR> If you're using python, the performance gap between processes with message passing and threads with locking is the last of your problems, believe me.

The big difference is that processes are much more robust and testable. The cases where threads are really needed are fringe cases and – while it's a pity – Python doesn't seem the right language if you don't want to go the Jython/IronPython way.

The bigger problem is that people are used to go for threads by default although only few are able to write bug-free threaded code. It's obsolete but prevalent performance wisdom and the fact that threads were really popular in the Java world.

<comment><USER22> This is a really short-sighted perspective.

Proper support of threading inherently allows more performance and flexibility than multiprocessing. On top of them, you can build powerful, Pythonic abstractions like concurrent.futures.ThreadPoolExecutor.map and STM, and on top of those, even more powerful abstractions that help the developer avoid concurrency bugs.

I'm really excited for PyPy. That is a project full of people who are not afraid to quickly iterate on powerful ideas that can make Python a high performance language that it deserves to be, instead of resorting to calling MT programming a "fringe case" and ad hominem attacks.

<comment><AUTHOR> I wish you'd read the article before making your accusations.

<comment><USER22> I have. There's an ad hominem attack in the middle of it. Otherwise it's a great article.

I understand that everyone here is acting in good faith and wants Python to be better, and the article otherwise contains lots of great information presented in a reasonable manner. You bring up lots of good points too. But other statements like the ones I mentioned are overly broad or brash.

<comment><AUTHOR> I think it's just attrition by explaining the GIL problem over and over again. Nick is one of the major core developers and probably just fed up by the topic. So this bit of snark is all the pay he'll ever get for his work on CPython.

<comment><USER23> I really liked the article. But look at how much of this thread is spent talking about that bit of snark. I think that mixing the snark in with all of the rational reasons for Python 3's existence and not working on the GIL made some people less receptive to rational arguments. In other words, I don't think it was worth it.</comment></comment></comment></comment></comment></comment></comment></comment></comment>


How the Linux philosophy affects you - <AUTHOR> http://opensource.com/business/15/2/how-linux-philosophy-affects-you

<comment><USER2> _Store Data in Flat Text Files_

With this tenet in mind, what is the preferred "Linux" method of engineering parts of an application that house sensitive information?

<comment><USER3> /etc/shadow would be a good example of storing sensitive information (kind of).

</comment></comment><comment><USER4> Owners of Mike Gancarz's original book might find the title kind of weird.</comment>


The disintegration of the parent-child bond - the-enemy http://www.theglobeandmail.com/life/parenting/the-disintegration-of-the-parent-child-bond/article28191786/ <comment><AUTHOR> > One cause of [children and teenagers'] fragility is a weak parent-child > relationship.

I don't buy that the relationship between parents and children has changed all that much in the last 25 years. I certainly don't believe parents love their children less, as the author is implying ("they cannot get unconditional love and acceptance from their peers or from a report card"). Plenty of Gen-Xers were latchkey kids and we don't seem to have suffered for it.

The explanation I personally favor is that children and teenagers aren't allowed to do anything the slightest bit dangerous anymore -- even the playgrounds are padded -- so they can't calibrate their sense of what really is dangerous, and therefore everything feels dangerous.

<comment><USER2> > I don't buy that the relationship between parents and children has changed > all that much in the last 25 years.

Certainly social workers in some countries (including my own) think otherwise. When we got our child there were heaps of "how to bond with children" guides given to us.

> The explanation I personally favor is that children and teenagers aren't > allowed to do anything the slightest bit dangerous anymore -- even the > playgrounds are padded -- so they can't calibrate their sense of what really > is dangerous, and therefore everything feels dangerous.

Even if playgrounds are padded, children have plenty of opportunity to learn the cause of their actions with their parents in a safer environment. I don't see how children would develop any less just because they don't end up being disabled due to playing in dangerous environments.

<comment><USER3> > I don't see how children would develop any less just because they don't end > up being disabled due to playing in dangerous environments.

Speaking from my own experience, I really wonder how I would have developed the way I did without being allowed to handle (very dangerous) tools from an early age, roamed all over Amsterdam from the ripe old age of 6, toyed with chemistry, explosives, electronics, household power and petrol engines all of which could have easily killed me.

I'm probably _more_ careful rather than less careful because of this and none of my friends or me ended up disabled. Long before we got to the stage where we could have become disabled we learned how to respect the stuff we played with.

Trying to imagine the alternative universe where all my tools had been plastic, safe imitations, where I was only allowed to work with batteries, being limited to the local playground (supervised, of course) and chemistry would be limited to playing around with salt & vinegar.

I'm treating my own children in much the same way, I _trust_ them to use their heads when it matters and I show them the consequences of failing to do so in as controlled (but very real) a way as possible.

Of course other parents think I'm nuts but that doesn't overly bother me.

<comment><USER2> > Speaking from my own experience, I really wonder how I would have developed > the way I did without being allowed to handle (very dangerous) tools from an > early age, roamed all over Amsterdam from the ripe old age of 6, toyed with > chemistry, explosives, electronics, household power and petrol engines all > of which could have easily killed me.

Neither, but that does not mean that your parents did not bond with you. Here is what happened in my generation: the time I spend with my parents and family was significantly bigger than most children today. For me personally the reason for that was that my parent took me to their office and I could spend time there. For many others in my generation it was because one of the parents stayed at home. Here is what's happening right now: children enter Kindergarten at 6 months of age or when they turn one. The only person that can tell them what to handle or not is most of the time a randomly assigned caretaker.

Note that when people talk about bonding they do not talk about a 6 year old child, they talk about a newborn to toddler of less than two years of age.

You're from the Netherlands where there is a proper maternity leave. Not everybody is in that fortunate situation and I assume the article is written with this in mind.

//EDIT: also to leave another note on older children: I'm not sure how it was for you in the Netherlands, but parents and teachers used to be proper authorities to me. This is something that is becoming less and less the case. There are brats around our flat that would not even listen to police if it came to it.

<comment><USER3> > Neither, but that does not mean that your parents did not bond with you.

You have no clue.

</comment></comment></comment></comment></comment><comment><USER4> My impression of the general trend is the opposite. When I think of parenting 50 or 100 years ago, I think of it as more formal and detached. Children addressing their parents as "sir" and "ma'am". Fathers who were emotionally distant and leaving much of the parenting to their mothers. Discipline that involved respecting the authority of all adult figures.

When I think of parenting now I think of extensive concern with kids' emotional well-being, parents who take kids sides in quarrels with teachers, etc.

I don't know if these impressions are broadly true, but it's interesting that my impression is of an opposite trend.

I agree with the overall premise of the article, which is that kids are more fragile than they have been in the past. I'm just not sure that a decreased parent/child bond is the right explanation.

<comment><USER5> > When I think of parenting 50 or 100 years ago, I think of it as more formal > and detached. Children addressing their parents as "sir" and "ma'am".

I don't think that's detached, but precisely the opposite. Treating one's parents like anyone else says, 'you don't matter any more to me than anyone else does,' whereas treating one's parents as one's superiors says, 'you are my parents, older and wiser than I am, and I treat you unlike some random people on the street.'

<comment><USER6> Right, but the familial bond isn't expressed by formal, ritualistic practice. That only creates a relationship of command.

<comment><USER5> > Right, but the familial bond isn't expressed by formal, ritualistic > practice.

Seems like human society has for millennia worked by having formal rituals, and it's bound people together pretty well. I'm more than a little cautious about throwing all that out.

> That only creates a relationship of command.

What's wrong with that? A 'relationship of command' is _exactly_ what the relationship between a parent and a three-year-old must be: when the parent shouts, 'don't touch that stove!' the kid must immediately shrink back; there's no time for inner reflection or external debate on the merits and demerits of stove-touching. As the child grows, the relationship becomes less and less commanding, but a wise adult will always listen to his parents (because they really do have more experience) and a wise adult will not command his grown child except as a last resort (because it's the only thing with a chance of getting through).

</comment></comment></comment></comment><comment><USER7> Really? I thought the trouble with kids these days was that they are all mamas-boys and daddys-princesses who can't accomplish anything on their own and who's parents micromanage their whole day and won't leave em the hell alone?

<comment><USER8> I don't necessarily see helicopter-parenting and absentee-parenting as mutually exclusive. The typical helicopter parent is always nagging, harassing, and invading the privacy of his/her children -- which is a sort of omnipresence, sure, but it's not a constructive emotional presence. You might as well not even be there in that moment; you'd accomplish the same thing sending a preprogrammed nag-bot into your kids' rooms to chide them and/or dote on them constantly. In fact, it's possible your kids are actually tuning your presence _out_ when you're in their faces all the time.

The magic combination seems to be: 1) let your kids discover the world (and make mistakes) on their own, just enough to let them learn and grow up; 2) be there for them when they need you. At one point in our history, this combination seemed intuitive. These days, after several generations of successive, fear-based sales of self-help books, advice, and programming, our culture has short-circuited parental instincts.

<comment><USER9> I see helicopter-parenting _and_ absentee-parenting as min-maxing; these are the munchkins of parenting.

Just as in role-playing, the parent gives little to no concern to other issues that are important for the child's development. The helicopter-parents focus on something to build into the child or to preserve the kid from; the absentee-parents merely minimize effort/time spent on the child.

</comment></comment></comment><comment><USER10> TL;DR:

Do things with your kids. Connect to them. Love them, not " _because_ ", but unconditionally.

Everything I have read, and everything I have seen, suggests that parents have to actually parent their kids. This isn't to say they need to discipline, or treat; it is to say that they cannot be idle. It's too easy to sit your kid in front of the TV and let them be "themselves", and that doesn't do anything.

</comment><comment><USER11> 30 years ago when I was 13, I experienced all the things described by the author. I over valued my peers' opinion, I disregarded my parents' words of advice.

Isn't it normal to feel these things? It's how an individual grows.

My secret to good parenting is compassion, consistency and honesty - the rest is just detail.

<comment><USER12> Same experience here. I am thankful that I was able to make mistakes and learn from them. With support and the values instilled by my parents, I was able to minimize the negative impacts of my youthful mistakes.

</comment></comment><comment><USER13> I'm not sure there's a parent-child bond being disintegrated in modern American culture -- if anything, a lot of parents seem more intentionally involved in their kids lives than past generations. However, I do think we need to look at a slightly higher-level problem: an adults-children bond.

As a father of two small children, I have noticed (along with my wife) that a lot of people in various social settings seem rather uncomfortable having our kids around. It's almost like this unspoken question is hovering around us, like "why aren't they in preschool" or "why aren't they with a babysitter" or "why do you have to bring them in here? This store is for adults only!"

A lot of businesses don't have good facilities for taking care of kids (changing tables, etc.). Only recently have we seen local malls remodel and include private nursing areas, etc. So sure maybe there's some progress here, particularly as a result of greater awareness of the needs of breastfeeding mothers and so forth, but I think there's still plenty of prejudice against kids in certain sectors of society.

Somehow, and I could be mistaken, it seems to me that in past cultures (particularly in rural or tribal settings), it was a given that children were heavily involved in the "business" of home and community. Children would work the same farm as the parents, for example. Obviously a lot of that kind of family cohesion was lost during the Industrial Revolution, so maybe it could be argued that things were worse off in 1916 than 2016. I don't know. It just "feels" to me, subjectively, like children and teens need to feel more included in "adult" society, and adults need to be more actively interested in the world of children across the board.

</comment><comment><USER14> "some [teenagers] are more concerned about what their peers think than what their parents think. Others are more concerned about their inflated self- concept than about what their parents think. Kids need to value their parents' opinion as their first scale of value, at least throughout childhood and adolescence."

It turns out that this article is written by a parent, rather than by a teenager.

</comment><comment><USER15> This is an extremely complex issue that can't be broken down into a few bullet points and fixed. Beyond generalizing all parents, the author of this article overlooks many factors that form children relationships with them and others. For example, how is the parents relationship with there parents, economic class of the family, culture, etc..

</comment><comment><USER16> I'd like to open with this tweet which I think captures some of the issues perfectly [1]:

\---

my aunt: why u kids always on them phones cant u have a real conversation

me: _puts down phone_ _crosses legs_ why did u melt the ice caps

\---

I'm a young adult, and I can say that anecdotally, if there's a strain in the relationship between my parents and myself it happens because I grew up in a world which is fundamentally different from the world they knew. This is partially due to technology, but it also comes from the fact that I (we) need to live in the world that "they" created.

My mother didn't understand why I wanted to sell the stock in Exxon Mobil that I was given by my grandmother. But I am the one who will have to come of age in a world of rising sea levels and global temperatures, at least partially created by that firm's wilful deception[2]. I avoid animal products in meals for the same reason. But in trying to tell my parents this, they seem to believe I am disingenuous, or they don't take the issues as seriously as I do.

Perhaps I'm reading too much into these issues. Perhaps that this is just the most recent iteration of the generational conflicts that have occurred throughout time. But it feels like the world I will inhabit is radically different from my parents', and I just don't know if there will ever be a good way to bridge that gap.

[1] [https://twitter.com/what_eve_r/status/681246826746523649](https://twitter.com/what_eve_r/status/681246826746523649)

[2] [http://www.scientificamerican.com/article/exxon-knew- about-c...](http://www.scientificamerican.com/article/exxon-knew-about- climate-change-almost-40-years-ago/)

<comment><USER17> > a world which is fundamentally different from the world they knew

Hyperbole. Let's hear about these [fundamental] changes. Life is 'a bit' unforgiving and discounting experience of survivors of this natural regime [is non-optimal].

> .. why did u melt the ice caps

So you could waste your life obsessively staring into a 3x4 glowing surface.

[edits]

</comment></comment><comment><USER18> I have to wonder what impact having both parents in the workforce has? It would seem that working is no longer a choice but an economic necessity for both parents. And that's only the families where the parents remain together. I'd imagine single parents find it even harder to spend time with their kids.

</comment><comment><USER19> > _Many teens would be the first to tell you that they love their parents. But > they are not seriously concerned with what their parents think. Or more > precisely, some are more concerned about what their peers think than what > their parents think._

Hasn't this always been the case?

> _As one Scotsman told me, "We don't even think much about 'generations.' We > just all enjoy doing things together."_

Unless that Scotsman is 13, that's pointless. He may love doing things together; his children might be counting down the minutes 'til they can hop back online and talk to their friends.

> _If you have the opportunity to move closer to your child's aunts, uncles > and grandparents, do it. (We did.)_

That's good, but I'm not sure what this has to do with the parent-child thesis.

</comment><comment><USER6> >The first and most obvious evidence is the extraordinary rise in the proportion of young people diagnosed and treated today for anxiety and depression.

I'm not sure why the author hasn't considered the possibility that, just like in other areas of mental health, recognition that these issues exist and better means of diagnosing and treating them have come to light. It may well be the case that there were a great deal of anxious and depressed children around, but went undiagnosed and untreated for a long time.

</comment><comment><USER20> A classic psychology tract, constructed on the default premise:

1\. Here's how things _are_.

2\. Here's how things _should be_ , if only people would listen to psychologists.

In this way, psychology has occupied the vacuum created by the departure of religion, in the minds of people who believe life should have a particular outcome, and who think someone is (or should be) in charge.

Meanwhile, parents, children, indeed everyone, are doing their best to fulfill personally chosen goals. Those goals are freely selected by independent agents in a morally neutral universe ruled only by evolution. Some plans will succeed, some will fail, but consulting authority is always a mistake, because in evolution, _there is no authority_.

Psychology has a long history of condemning behaviors it thinks the public dislikes, then changing their tune when public tastes change. It was once a mental illness to run away from your master in the antebellum South (Drapetomania). It was once a mental illness to be a homosexual (not very long ago). More recently, it was a mental illness to be creative and intelligent (Asperger syndrome). All these and many other imaginary ailments have been abandoned as public tastes changed.

H. G. Wells said, "Civilization is in a race between education and catastrophe." It was a positive step to abandon religion. It will be another positive step when we abandon psychology and begin to think for ourselves.

</comment><comment><USER21> "some (teens) are more concerned about what their peers think than what their parents think." I thought peer influence being as powerful or more than parent influence in teenagers was already accepted, not something new.

<comment><USER12> Exactly. If by "some" the author means "all," then I'd say the quote is spot on.

</comment></comment><comment><USER22> >His main idea is that many of the problems we see with North American kids today – _the defiance, the disrespect, the disconnection from the real world_ – can be traced to the lack of a strong attachment between parents and their kids.

 _We_? _Problems_? Under what calculus are defiance, disrespect, and disconnection, "problems"? Maybe the current state of the world (or maybe just Canada) deserves defiance, disrespect, and disconnection?

The observation being made here is purely subjective; one side believes they deserve something (obedience, respect, attention) that they may, in fact, not.

More likely we are seeing a rejection of the state of the world and many of the people who caused the state to be the way it currently is.

<comment><USER5> > Under what calculus are defiance, disrespect, and disconnection, "problems"?

They are not the proper attitudes of those with lesser knowledge, skill and experience to those with greater such. That they are prevalent is indicative of, among other things, bad parenting.

Guess what? Your folks were your age once. They experienced the same emotion- addled thought processes. They believed ludicrous things to be true for the same reasons that you do (and I did, and sometimes still do). And they can look at your situation from a distance and say, 'you know what? relax!' and _be right_.

<comment><USER23> Adults aren't wiser and more experienced children. They're children with the creative and vital burnt out of their souls and replaced with a respectable act that gets put on for other adults.

Defiance, disrespect, and disconnection are absolutely the correct way to interact with those who insist on destroying a fundamental part of who you are.

<comment><USER5> > Defiance, disrespect, and disconnection are absolutely the correct way to > interact with those who insist on destroying a fundamental part of who you > are.

A fundamental part of who you were when you were born was crying every time you were the least bit upset; it was evacuating your bowels without even trying to contain them. A fundamental part of who you were as a small child was being a psychopath who didn't fully realise that other people are, well, _people_. A fundamental part of who you were as a teenager was an underdeveloped brain — and another was the overdeveloped hormone production which didn't help your decision-making skills any.

None of those are unique to you: they are the common lot of all of us.

The whole point of good parenting is to destroy all the fundamental parts of children which are animalistic and contrary to survival in the world as it is, and to inculcate the development of secondary parts which are humanist and support survival. Now, many parents fail at those tasks, but those who succeed produce healthy, functional adult human beings. Bad parents produce selfish, brutal, cruel, unkind, animalistic demihumans.

<comment><USER23> There's a huge difference between showing children a more humane way to interact with the world, and crushing your child's spirit until they do what you want them to. Doing the latter is how you end up with breaking people to the point where they don't notice that they have emotions for a decade.

Evil people don't think they're evil - instead, they think things like "spare the rod, spoil the child". Just... goddamn, be careful with spreading that line of thinking, okay?

</comment></comment></comment></comment></comment><comment><USER24> Honestly, I think a serious contributor here is the number of families with two full time working parents more than anything else. My wife and I were actually talking about this last night.

We both work til about 5:30 or 6 every week day. Our kids go to after school programs where they play with friends, do homework, etc before we pick them up. We get home around 6 and they need to get to bed by 8. That gives us 2 hours every weeknight to make/pickup dinner, do something together as a family, talk about our day, and get ready for bed.

If either of us got off at around 3 every day both of the kids could come home. We could go to the store together and get things to make for dinner together. I could start cooking and they could help or do homework and then when my wife got home dinner would be ready for us to sit down and have a family meal. We'd end up eating healthier and being less rushed. On a lot of days the kids could have their friends come over for a couple of hours until dinner time and potentially that child's parent(s) could come over to visit while the kids played.

Instead, we're left with this weekly rush. I honestly don't have much of an idea who the parents of my kids friends are at school...because I don't ever see them. I barely see my kids friends. This is before we factor in things like soccer practice.

On weekends, over 2 days we try to balance house chores with family time with going to the store to get everything we'll need for meals for the week before anything else that might come up.

I cannot imagine that we're the only family with 2 working parents that experiences this. The seeming solution to the problem is for one of us to scale back, but that means figuring out something at work that will allow us to do that and ensuring that we're able to adjust costs to take a hit somewhere (assuming that asking to scale back doesn't put the entire job in jeopardy). The other solution is to try to invest what's left of that spare time into creating a side income so you can afford to scale back...which just feeds the problem.

As I've gotten older I've started to place a significantly larger amount of respect on stay at home dads or moms, because the longer we've been pushing this dual career thing the more I realize what we're giving up. It seems like so much chatter, movie references, TV references, etc over the last 2-4 decades has mocked stay at home parents as either being unimportant, uneducated, people who sit around watching TV and what not that as a society we've pressured an entire generation into feeling like raising a family instead of having a job makes you somehow less.

And I think it's breaking more of society than we want to admit. This isn't a gender roles thing either. I don't think it matters WHICH parent spends more time at home as long as one of them does.

We fuel this issue by generally meeting our significant others in college...where we are both training to have full time careers and taking on student loans that we'll have to work to repay. All just seems to be a huge, interconnected mess of a pattern.

I'm not actually sure what you could do to fix it, aside from providing companies tax credits for allowing "parent friendly" jobs to incentive them to have positions where the day ends when kids get out of school.

</comment><comment><USER25> Funny how every new generation is so much worse than the one that came before.

<comment><USER26> "The children now love luxury. They have bad manners, contempt for authority; they show disrespect for elders and love chatter in place of exercise." \- Socrates

<comment><USER5> Some people read that and think, 'Ha! Old fogeys always complain, but nothing changes.' Others read it and think, 'Huh, folks have gotten a lot worse over time.'

</comment></comment></comment><comment><USER27> Lame and unconvincing.

</comment><comment><USER28> If I had lived in a household like this, I would have become estranged from my parents the moment I could. It's authoritarian drivel. This is fodder for narcissistic parents.

<comment><USER29> The article uses very broad strokes, making points which could certainly lend themselves to narcissistic parenting. One stand-out line is "[b]ut your child's first allegiance must be to you, not to her best friend". It stresses very hard that the responsibility is on the child to adhere to the relationship as defined by the parent, whom is painted as infallible in this regard. Parents certainly take on a lot of responsibility as caregivers, but they're never perfect, and aren't always deserving of allegiance.

Children are first and foremost _people,_ and they deserve to be treated as such. They should be able to define their own relationships, even if that means forming close bonds of friendship at an early age. Maybe the friendships won't last, maybe they will, but that's not for anyone to decide but those involved. Taking that away from a kid, telling them their friendships are somehow superficial and don't matter because so much of it happens through "magic computer-boxen and apps" that seem to be the root of all of today's ills, well that sounds pretty domineering and narcissistic to me.

If a parent wants a strong bond with their child, they can take all the steps they want to encourage the growth of that bond, but trying to _force_ it, as with any other relationship, will only lead to the opposite result.</comment></comment>


Apple declined to implement 16 Web APIs in Safari due to privacy concerns - <AUTHOR> https://www.zdnet.com/article/apple-declined-to-implement-16-web-apis-in-safari-due-to-privacy-concerns/ <comment><AUTHOR> can APIs be selectively disabled in chromium based browsers? if not disabled, replaced with stubs via the extension mechanism? or other method.</comment>


Creating a new Blogging CMS Integrated With The Best of HTML5 - <AUTHOR> http://protoco.de/protoblog-a-simpler-blogging-cms/

<comment><USER2> Do we really need _another_ CMS?

<comment><USER3> Yes?

<comment><AUTHOR> show me some CMS's that do just this and we'll stop making a new one haha.

</comment></comment></comment><comment><USER4> I'm pretty sure we already have this. This looks like its already been done by Squarespace.

<comment><AUTHOR> yes but you have to pay for it and for what it offers, it's a rip off, at least from a web developer's perspective.

</comment></comment><comment><USER5> Awesome stuff! Been wanting this for a while.</comment>


Before They Were Titans, Moguls and Newsmakers, These People Were...Rejected - <AUTHOR> http://ca.finance.yahoo.com/personal-finance/article/yfinance/1500/before-they-were-titans-moguls-and-newsmakers-these-people-were--rejected

<comment><AUTHOR> I definitely relate to the teenage angst of college rejections, and still feel it sometimes (now in my third year). My school is considered one of the "New Ivies", and many of my classmates were similarly rejected or waitlisted at all the other (top tier engineering) schools they applied to except this one. We've ended up with a student body that's apathetic, and still somewhat resentful. I wonder how common this is at other "New Ivies"...

But you know what? To the entrepreneurial types among us, it only makes us work harder here. :)

</comment><comment><USER2> Somewhat related video (found on minimotivation.com):

<http://www.youtube.com/watch?v=_tjYoKCBYag></comment>


Why Robinhood Is Fundamentally Flawed and Bad for Users - <AUTHOR> https://blog.willrobbins.org/the-flawed-economics-of-robinhood-why-users-are-better-off-without-it-a004913b3b4e#.f4ta9kae0 <comment><USER2> most of your point is valid to other broker too.

compare RH free trade to other broker.

I find it useful.</comment>


The wonderful world of insects - <AUTHOR> https://delanceyplace.com/view-archives.php?3973 <comment><USER2> "The big question here is how the wasp mother can control the ladybug, transforming her into a zombie babysitter... The answer is that the wasp mother injects the ladybug not just with the egg but also with a virus. "

This strengthens my belief that insects are like tiny little robots.

<comment><USER3> The really interesting question is why don't we see these amazingly detailed and precise parasitic mechanisms elsewhere, like in mammals? Are we just doing a terrible job of finding them, or is there something more interesting going on? Del Giudice recently published a paper arguing that the baroque complexity of non-insect nervous systems may in part be an anti-parasite mechanism, by making it too hard for a distributed group of small simple parasites to find hacks: "Invisible Designers: Brain Evolution Through the Lens of Parasite Manipulation" [https://www.gwern.net/docs/genetics/selection/2019-delguidic...](https://www.gwern.net/docs/genetics/selection/2019-delguidice.pdf) , Del Giudice 2019 (discussion: [https://slatestarcodex.com/2019/08/19/maybe- your-zoloft-stop...](https://slatestarcodex.com/2019/08/19/maybe-your-zoloft- stopped-working-because-a-liver-fluke-tried-to-turn-your-nth-great- grandmother-into-a-zombie/) ).

<comment><USER4> Toxoplasmosis is (not) your friend here [0]

[0] [https://www.mayoclinic.org/diseases- conditions/toxoplasmosis...](https://www.mayoclinic.org/diseases- conditions/toxoplasmosis/symptoms-causes/syc-20356249)

<comment><USER3> Toxo is the exception that proves the rule here. Toxo is perhaps the best known example of any parasite having human behavioral effects, yet the causal role of toxo is still not accepted, and what it's postulated to affect is pretty random: schizophrenia, car accidents, that sort of thing. That doesn't look like _any_ kind of planned behavioral control which might affect toxo reproduction. It just looks like a parasite causing some damage post- infection. It's unclear if it even manages to replicate the mouse effect of making humans more attracted to cats (since we contract it mostly from cats, obvious confound). You certainly cannot compare its capabilities, in rodents or primates, to the astonishingly precise and complicated behaviors that insect parasites can orchestrate. And it's not like humans in the wild don't host plenty of parasites!

</comment></comment></comment></comment><comment><USER5> Has anyone read this book? I've been having a craving these days for books on animals, having just finished "The Rise and Fall of Dinosaurs", and this looks pretty good.

</comment><comment><USER6> Lots of underrated beauty in the insect world as well. Check out the last series of images in this video about manual focus stacking for short dof shots - [https://youtu.be/2GmQ2Hj9WOs?t=198](https://youtu.be/2GmQ2Hj9WOs?t=198)

Another one from the same guy, have to poke around - [https://www.youtube.com/watch?v=qmMcCjEU68Y](https://www.youtube.com/watch?v=qmMcCjEU68Y)

Lots more of these kinds of vids on youtube of course...

</comment><comment><USER7> Some tiny wasps are used for pest control in farms, natural and clean way

</comment><comment><USER8> To borrow a line from Spock: 'Fascinating...'</comment>


European Lisp Symposium 2016 - <AUTHOR> https://www.youtube.com/playlist?list=PLA66mD-6yK8xqd5XKxzwVBIqf1d3QQXpv <comment><USER2> Best talks in my opinion:

* [https://www.youtube.com/watch?v=5bQhGS8V6dQ&index=15&list=PL...](https://www.youtube.com/watch?v=5bQhGS8V6dQ&index=15&list=PLA66mD-6yK8xqd5XKxzwVBIqf1d3QQXpv)

* [https://www.youtube.com/watch?v=dK3zRXhrFZY&index=11&list=PL...](https://www.youtube.com/watch?v=dK3zRXhrFZY&index=11&list=PLA66mD-6yK8xqd5XKxzwVBIqf1d3QQXpv)

Notable mentions:

* [https://www.youtube.com/watch?v=6GO1rust6SU&index=8&list=PLA...](https://www.youtube.com/watch?v=6GO1rust6SU&index=8&list=PLA66mD-6yK8xqd5XKxzwVBIqf1d3QQXpv)

* [https://www.youtube.com/watch?v=--gqVzhLYoI&list=PLA66mD-6yK...](https://www.youtube.com/watch?v=--gqVzhLYoI&list=PLA66mD-6yK8xqd5XKxzwVBIqf1d3QQXpv&index=7)

* [https://www.youtube.com/watch?v=JU-BL4XAOHo&list=PLA66mD-6yK...](https://www.youtube.com/watch?v=JU-BL4XAOHo&list=PLA66mD-6yK8xqd5XKxzwVBIqf1d3QQXpv&index=4)

I liked OpenGL lightning talk as well:

* [https://youtu.be/2ZEYxxFR_Gs?t=27m34s](https://youtu.be/2ZEYxxFR_Gs?t=27m34s)

<comment><USER3> > > [https://www.youtube.com/watch?v=5bQhGS8V6dQ](https://www.youtube.com/watch?v=5bQhGS8V6dQ)

no wonder CANDO is impressive, the scope and fields involved are pretty rare...

I'm very eager to learn if C. Schafmeister work will bear 'disruptive fruits'.

ps: a talk about clasp [https://drmeister.wordpress.com/2015/06/15/i-gave-a- talk-on-...](https://drmeister.wordpress.com/2015/06/15/i-gave-a-talk-on- clasp-and-my-chemistry-at-google-in-cambridge-mass-last-week/)

<comment><USER4> [https://www.youtube.com/watch?v=8X69_42Mj-g](https://www.youtube.com/watch?v=8X69_42Mj-g)

This talk is even better. "C++ templates are to Commmon Lisp macros what IRS tax return forms are to poetry". I don't know both well enough to tell if it's true, but damn it's a good metaphor.

<comment><USER5> I know both well enough and I think it's very true :).

Christian Schafmeister's talks are great and I'm very impressed by his work. It was a great experience to meet him personally on this year's ELS. One thing I learned from chatting with him during the conference break is that his work on Clasp is actually pushing LLVM forward - when implementing Common Lisp / C++ cross-linking he found some bugs that were there because apparently no one before has been trying to do the stuff he's doing - so he had to nag some Apple engineers to get to work and fix them ;).</comment></comment></comment></comment>


Ash HN: question regarding code generation in compilers - old-gregg

I am no compiler writer but I like to read about them a lot. :-)
I have a question to ask which I don't know a better venue than HN. Basically it is
about code generation for AMD64 platform.
For a long time I have been seeing common complains about "inferior" and "poorly designed" architecture of x86 command system. Most often people would complain about lack
general purpose registers.
When AMD launched their 64-bit extensions (and even Intel with SSE before that) they have added a lot of that, yet telling GCC, for example, to utilize those capabilities doesn't
make general purpose C programs any faster. Targeting AMD64 vs plain vanilla i386 doesn't
give you hardly and performance boost.
Why? <comment><AUTHOR> The lack of performance boost I think is caused for a few reasons: The extreme amount of optimization on both the architecture makers and compiler writers. They hyper-optimized to the architectures they had, so giving them general registers support didn't buy you much.

As for why it was lambasted for its lack of general purpose registers? General-purpose registers make writing compiler back-ends cleaner and reduces the need for some stubs.

</comment><comment><USER2> Longer pointers (64 vs 32) means that programs are bigger and thus are more likely to bust the cache, causing sometimes significant performance hits.

</comment><comment><USER3> Also, the bottleneck is memory access, not cpu power, so optimising for this or that cpu isn't that important after all.</comment>


Yahoo.com redirecting to m.www.yahoo.com? - <AUTHOR>

Am I the only one experiencing this? <comment><USER2> welcome onboard. i believe its nw a family affair</comment>


Jeremy Zawodny: NoSQL is Software Darwinism - <AUTHOR> http://blog.zawodny.com/2010/03/28/nosql-is-software-darwinism/

<comment><USER2> Am I the only one that finds certain aspects of NoSQL databases easier to use than SQL? Despite having used different SQL ORMs and writing two on my own, I'm a big fan of MongoDB. It's fun using it for small projects, too, even if I know scalability and speed won't be an issue.

<comment><USER3> OK - I have to bite. Who writes 2 ORMs? What was wrong with the first one?

<comment><USER4> You were surprised? In my experience most hackers love redoing the last project, only better. There's all the joy of victory over your past mistakes, and less of the pain of struggling with an unfamiliar problem.

They love it so much they usually have to be counseled out of doing it when it isn't a good idea for strategic or other reasons.

<comment><USER5> This is usually called the "second system effect", especially when the second is overambitious and never gets finished.

<comment><USER6> That made me think of TextMate 2.

</comment></comment></comment></comment></comment><comment><USER3> NoSQL is the duck-billed platypus in the evolution of databases. Absolutely fascinating, provides great material for PhD students, but a total dead end. Might survive in some isolated corners of the world; not one to bet on to be the next dominant species.

<comment><USER7> A while ago I have held the same opinion; that is up until I have faced a database problem that genuinely wasn't very RDMBS-friendly so I was "forced" to have a look at the alternatives. Ever since then I have realized that there there are large domains of problems that can be solved a lot easier with "nosql" configurations than with RDBMS systems. I mean stuff that I've used to solve with SQL databases before.

My point is: it is always a good idea to have a larger perspective.

<comment><USER8> I had a similar change of perspective in 1997 when I worked at a well known Internet retailer. The entire catalog of items we sold (as well as customer reviews and other data) were all stored in a number of key value stores (Berkeley DBs) that were routinely built and pushed out to each web front end. This was very fast and for our purposes was much better than storing this information in a centralized SQL database.

<comment><USER3> If you'd had viable open source SQL databases in 1997, would you have spent the engineering time on BerkeleyDB, or would you simply have replicated the master database onto each server? In 1997, you weren't choosing NoSQL vs SQL, you were choosing open-source vs commercial.

Anyway, you're essentially storing pre-generated pages, which isn't a use case that I think anyone considers particularly database-appropriate (SQL or NoSQL). Using memcached to cache the data that is in active use seems more efficient, faster, and gives you the option to force through out-of-sequence updates (though again, that wasn't available to you off-the-shelf in 1997.) Complete re-generation of data might have worked for Amazon in 1997, but is this what they're using today?

<comment><USER8> With open source SQL databases there is no "simply" when it comes to replication. Even today, MySQL replication is brittle, and master/slave inconsistencies are the rule rather than the exception. Slave crashes often cause replayed transactions due to lack of atomicity in writing master.info and relay-log.info. The replication landscape with PostgeSQL is varied and essentially a bag on the side. Last I counted there were more than 10 different ways of doing it, a number involving trigger based log shipping. It wasn't about open-source vs commercial, it was about scaling the reads.

The detail pages weren't pre-generated, they were based on read-only catalog data, which I think is entirely database appropriate. I imagine that complete re-generation of data is no longer done, but I'd be willing to bet that Berkeley DBs are still used in production somewhere.

<comment><USER3> A read-only database isn't a database in my book.

I agree that built-in replication can be difficult to administer even today, but you're being completely revisionist here. Replication wasn't introduced into MySQL until 2000. In 1997, you would by necessity have rolled your own replication system tailored to your needs (much simpler than solving the general-case problem). That's basically what you did anyway, but you solved it in the most trivial way possible: you 'replicated' by doing a complete database dump and re-distributing the entire DB. If you'd had a viable open- source relational database, you could have scaled the reads and got more developer productivity by distributing a SQL database (e.g. SQLLite) rather than a key-value database (BDB).

I appreciate your standing up and giving a concrete example of NoSQL usage - nobody else has been brave enough to do so. But it seems that the reasons for it were highly specific to the time: there were no viable open-source databases, Amazon was just introducing the idea of customer reviews (i.e. pre Web 2.0) so data was primarily read-only, memory was comparatively expensive and memcached didn't exist, and you had a comparatively small product catalog where complete re-generation was an option. I don't think you can carry forward the optimizations you made in that framework into today's world.

<comment><USER9> See my reply to the grandparent.

I actually was responsible for that system, and moving away from BDB's being pushed to servers sometime in '00 or so.

As you said, these weren't really databases by any stretch of the imagination, simply snapshots, and built for a very specific type of query. (by asin, by time, reverse ordered)

The building of the DB's was a pain in the ass, because the sheer scale of them was so big that you had to do clean builds (instead of incrementals) fairly often without them wasting space. There was also all sorts of voodoo magic going on to work around various BDB issues.

The system did eventually move to a service architecture (as all of AMZN did), for two main reasons:

1) pushing that much data to more and more servers was getting insane, even on their inner networks.

2) we wanted faster turnaround for new reviews

3) rebuilding the BDBs was becoming more and more cumbersome with scale

All that said, the original system did take us pretty darn far, both in scalability of traffic and scalability of data, farther than most websites will ever reach.

Fun times working there, you really get to work on some unique problems.</comment></comment></comment></comment></comment></comment></comment>


Heliogen, backed by Gates, says it has achieved a solar breakthrough - <AUTHOR> https://edition.cnn.com/2019/11/19/business/heliogen-solar-energy-bill-gates/index.html <comment><USER2> Am I missing the breakthrough? This just reads like boilerplate tech marketing, replete with questionable ai usage. I suspect without its well known backer anyone would notice.

<comment><USER3> They seem to be using computer vision techniques to dynamically aim the mirrors. Presumably static or open-loop techniques don't focus as precisely, and using a closed loop also allows them to reduce the stiffness (and this material use) in their mounts and/or fewer mirrors.

<comment><USER4> Closed loop enables higher temperature. I doubt material saving is worth it.

<comment><USER5> Pointing hundreds of mirrors is expensive. Requiring a lot less rigidity for the motion system is notable.

I don't doubt you could have done this open loop with a very rigid (expensive) platform before (including solving problems of ground subsidence). But now you can do it with a reasonably priced platform closed loop, which is what makes it actually possible.

<comment> core-questions Could you expand on what closed/open loop mean in this context?

<comment><USER6> Open loop means that they're just using a mathematical prediction of where to point the mirror at a specific time, given the relative position between the heliostat, the target and the sun. adjustments to the mirrors are typically done every 10 to 15 seconds, which is sub optimal. Closed loop means that they're actually using Computer vision to dynamically sense the specific angles between the sun and the target and then adjust the mirrors accordingly, no mathematical model required, and the adjustments can be done tens of times per second.

<comment> core-questions Thanks!

</comment></comment></comment></comment></comment></comment></comment><comment><AUTHOR> Excerpt: A secretive startup backed by Bill Gates has achieved a solar breakthrough aimed at saving the planet.

Heliogen, a clean energy company that emerged from stealth mode on Tuesday, said it has discovered a way to use artificial intelligence and a field of mirrors to reflect so much sunlight that it generates extreme heat above 1,000 degrees Celsius.

=====

The article describes a bit about the role the AI is playing, and Heliogen didn't seem to me to have used "AI" as a buzzword

<comment><USER7> I fundamentally cannot perceive where AI plays a role. Isn't it just a solvable geometry problem?

<comment><USER8> "Heliogen uses computer vision software, automatic edge detection and other sophisticated technology to train a field of mirrors to reflect solar beams to one single spot."

In principle I think you are right, but that would mean for each plant you build you need the exact position of each mirror, the oven chamber and the motion of the sun through the day and year. I think the "AI" approach scales better, as it is rather simple "AI" application.

<comment><USER9> Can't you then solve this by having a bunch of photodiodes under the edge of each panel, indicating if the panel is placed correctly?

<comment><USER10> From my reading of the article, one of the big issues to overcome was cost of CSP vs Photo Voltaic . They're even using smaller flat mirrors that are less sophisticated since per panel costs add up fast. One of the advantages is that this takes 4 cameras across the full plant, not a per panel cost.

That's just my reading of it, not asserting anything myself.

<comment><USER11> How many photodiodes it takes before you reach the cost of 4 cameras?

</comment></comment></comment></comment></comment></comment><comment><USER12> One of my discarded project ideas was exactly this, but on a smaller scale. A steerable mirror on a mount with a camera for targeting. The idea was to place one on the ground outside a north facing window. The device would then busy itself reflecting the sun up into the room making a nice sunny patch on the ceiling to light the room. They could also be used to deliver extra light to windows with plants in them to grow plants which require more light than is available or to modulate the daily light to fake a different season.

Sitting in 2020 the technology is near trivial. The problem is that some knucklehead would point a bunch of them at the same window and set curtains and pigeons on fire. (You'd do this because you had a large room with few windows, so dispersed mirrors would give you multiple dispersed lighting spots in the room… and incinerated pigeons.)

I was unable to make the leap to what happens if you are a deliberate and well reasoned knucklehead! In a controlled situation a megawatt of light might be just the thing.

Addendum: The mass field of mirrors introduces a _lot_ more work. Some of the immediately interesting problems:

• you need a "everyone stop shining at the target" mode for emergency stop, but the mirrors need to move away from the target in such a way that they don't create convergences elsewhere. Obviously sweeping your portal to hell down the tower would be a bad idea. So you have a collection of autonomous, free thinking mirrors, but when commanded to stop hitting the target they need to not make the same decisions.

• your camera probably needs to be able to stare into the heart of a thousand suns (literally). But it also needs to see in ordinary daylight. Maybe a pinhole filter that moves over the lens would be appropriate.

• with 999 mirrors shining on the target, you will not be able to see your own contribution to aim. There probably needs to be a "hey! Everyone else look away for a second" command for unit to aim and calibrate itself. See complexities from the first bullet and not incinerating things you like.

So yeah, they could have started with my garden heliostats, but then it would be a crack the knuckles, bring in the eggheads, and get to work operation to solve all the hard problems.

<comment><USER13> Add similar software to a swarm of drones each carrying mylar film stretched on a lightweight rigid frame with a vertical tilt actuator, and you have a seriously dangerous weapon system on sunny days.

<comment><USER5> It's hard to point what's basically a giant sail on an airborne drone.

<comment><USER14> Add a second sail and use the AI magic to move it to roughly counter the force on the first one, helped with the rotors for fine control.

I don't really know if this would be possible, but sailing ships can rig their sails to propel them against the wind.

</comment></comment></comment></comment><comment><USER15> Another interesting problem this might solve is birds.

Yes, domestic cats kill orders of magnitudes more birds, but that argument a. depends on relative scale that people don't grok, and b. coming from an energy company reinforces the preconception that they're callous.

Add radar and a sufficiently smart AI, and they could turn a PR problem into a marketing pitch by showing birds flying through unscathed while the mirrors shift to avoid them.

<comment><USER16> If you can track individual birds like that I'd have thought you could do targeted scaring to deter the birds before they get fried. Moving mirrors quickly enough seems like it would require quite significant adaptation of the apparatus?

<comment><USER17> I worked with a security company that did essentially that with air canons and radar. Unfortunately, scared birds aren't the most predictable creatures, but it seems to work pretty well.

</comment></comment></comment><comment><USER18> I think the article is overplaying the promise of CO reductions.

Around 60% of CO2 (see 1) emissions during cement manufacturing are just a byproduct of the chemical process converting calcium carbonate to lime (see 2) - nothing that solar power can help with.

1) [https://precast.org/2013/05/concrete-and- co2/](https://precast.org/2013/05/concrete-and-co2/) 2) [https://www.ipcc- nggip.iges.or.jp/public/gp/bgp/3_1_Cement_P...](https://www.ipcc- nggip.iges.or.jp/public/gp/bgp/3_1_Cement_Production.pdf)

<comment><USER4> Cement is 8% of global CO2 emission. 40% of that is still greater than air travel (about 2.5%).

<comment><USER19> I thought that the problem with air travel is not (just) the amount of CO2 emitted, but also the altitude, which makes the green house effect bigger per ton of CO2.

<comment><USER20> It's not about CO2 emissions, but rather the high-altitude NOx that contributes more to climate change than low-altitude emissions. However, it's debatable whether the amount is very significant, at least compared to other sources of NOx.

[1] [https://en.wikipedia.org/wiki/Environmental_impact_of_aviati...](https://en.wikipedia.org/wiki/Environmental_impact_of_aviation#Carbon_dioxide_\(CO2\))

</comment></comment></comment></comment><comment><USER21> Cool, an Archimedes Death Ray with AI.

</comment><comment><USER22> I'm wondering, where does AI come into play? Isn't this achievable with just basic geometry.

<comment><USER4> I wouldn't call closed control loop "AI", but you need some control to deal with unpredictable ground heave/subsidence.

</comment></comment><comment><USER23> Is the company backed by Bill Gross or Bill Gates or both?

</comment><comment><USER24> Just how much AI is required to keep a mirror aligned with a target. If I see another article on AI, cloud or blockchain then I'll eat my keyboard :]

<comment><USER25> Just barely enough so it can make it into the marketing material without seeming like a total scam.

</comment></comment><comment><USER26> @elonmusk please work with integrating battery tech with these guys!!

<comment><USER27> Backing up solar production with batteries is pointless. Except for a really specific cases. The costs are too high, the stress on the grid is too high, the grid need to be overscaled, the batteries need to be changed each 15 years the solar panels need to be changed each 20 years.

And even if we overcome this, we will need so much solar panel and batteries that we simply don't have enough raw materials to do so (to replace a 1GW nuclear core you need to cover half of Paris with solar panels, source [http://www.assemblee-nationale.fr/15/rap- enq/r2195-t1.asp](http://www.assemblee-nationale.fr/15/rap-enq/r2195-t1.asp)).

In France, 95% of the stored electricity is done using reversible-damns (you pump water up to store, you empty the water and generate electricity when you need it), only 2% using batteries. We basically put damns everywhere we could on the territory. We have 2 days max of storage. How can we now store weeks/months of solar-panel/wind electricity we don't need to use it when we need it?

Batteries are inefficient for grid storage, oversized, heavy and too short term. That's it. We now know for decades that damns are the easiest way to do it and we can't do more than a few days of storage.

At one moment we simply have to face the maths and physics. Trusting the "next big thing" in battery storage/solar production, even if we do 5x more energy, even if we develop long lasting ones, will not work in the end. Our needs are too big and the amount of minerals and other raw materials we need is way too big. That's it.

<comment><USER26> Yeah true. I heard of another Bill Gates backed startup that actually stored the energy in a form of potential energy which was way more cost effective. Keep in mind though dams are super catastrophic to the environment as far as water flow which can be detrimental to moisture and animals that depend on water in certain areas

<comment><USER17> I'm sure you could do it with a closed system, e.g. pump water up a cliff, and when you need power, bring it back down. I'm guessing using dams on existing rivers is just more convenient than building a closed system from scratch since it can double dip on hydro infrastructure.

</comment></comment></comment></comment><comment><USER28> Again, we've been able to do this with four photodiodes for something like 30 years.

Of course it worked the first time they turned it on - they were using 4 photodiodes. The "AI" and "computer vision" is just to get funding and hype.

I'm going to start taking solved problems and applying BS "AI" to them to get funding, sheesh.

<comment><USER29> I somehow doubt Gates gets fooled by hype. Do you think he just stumbled his way to success? Maybe there is a little more complexity to the problem than you anticipate.

<comment><USER28> There isn't.

"Bill Gates is smart, therefore he can't be fooled or wrong" is not a strong argument.

<comment><USER30> Actually being smart would be leveraging all the hype for maximum coverage, granted the actual product has actual value/benefits.</comment></comment></comment></comment>


Whats a good group on Linkedin to meet engineers?   - <AUTHOR>

<comment><USER2> That's a little too vague, man

Linkedin is not a great place to meet engineers, maybe the worst. Most engineers have taken down their profiles from too much spam from recruiters.

</comment><comment><AUTHOR> @gamechangr, thanks for the headsup. I am in search of general programmers with some background in php,flash, C++. I am sure I can always look here at HN. Thanks.

<comment> hippee-lee You don't have any contact info in your profile. I consider myself a web generalist with experience in php. This (web) programming being my second career I'm not sure how to rate my ability but if you have a well defined spec that won't change I can provide a solution ...

Anyway, I'm not so confident that I have deleted my LinkedIn profile. In fact I am working on building my soft skills and that site has been very helpful to me lately. You can easily find me there or with google. Maybe I can help you?</comment></comment>


WeWork All Hands Transcript - <AUTHOR> https://www.vox.com/2019/10/24/20929236/wework-layoffs-adam-neumann-marcelo-claure <comment><USER2>
        Employee 3

        Hi, my name is [redacted]. (audience applause)

        Marcelo Claure

        Well first you've got to tell me why the people like you so much? (audience laughs)

        Employee 3

        Because I asked a question at the last All-Hands meeting.

        Marcelo Claure

        Okay.

        Employee 3

        I'm with the Development Finance Group, I do data analytics and reporting and today's my one-month anniversary at WeWork. (applause)

        Marcelo Claure

        You're very popular for one month.

Not asking anyone to dox employee 3, but I'd really love to know what the backstory there is.

<comment><USER3> I'm willing to bet that person made an impression.

Places I've worked, when you start they ask you to stand in front of the company and say or do something whimsical or answer a question (name a piece of sports equipment you would be and why you chose that). It's possible they said or did something funny people still remember.

I'm only going on that it's their one month work anniversary. However, WeWork is 3-4000 employees so not sure.

<comment><USER2> Dev finance group and comment at a precious all hands, I was guessing person called out previous CEO in public.

</comment></comment></comment><comment><USER4> > remind all of us that we came here to do something meaningful in the world

Couldn't go on after this BS. This is the first line in the Official cliche/bullshit all-hands playbook.</comment>