Multicore Programming in PyPy and CPython - <AUTHOR> http://morepypy.blogspot.dk/2012/08/multicore-programming-in-pypy-and.html

<comment><USER2> Although it adds another piece of software on top, I've been using Python with Redis a lot lately, and it's wonderful. Here's a 'base' for building the larger program. It doesn't make a single python instance multithreaded, but it easily allows me to connect other computers on my network to join in the processing fun.

        filePath = "/home/user/file"
        p = Process(target=readFile, args=(filePath,))
        p.start()

        numProcesses = 6
        for x in range(numProcesses):
            p = Process(target=mapper, args=())
            p.start()

So I have a file reader to prevent a seek storm on a large file, I read something (like separations between tags that I want to process more), I put the binary data in a "ready:NAME" key, with "NAME" going into a "readyList", and things go wonderfully. It makes my data persistent without writing to files or pickling, and I can easily share data between process instances.

My question is - what advantage does a multicore programming offer that my take wouldn't? I can import process and chain things out over my 8 cores, but why would I do that if I can have persistent data with the ability to more easily add processing power when I want to?

<comment><USER3> well it allows you to share in-memory, mutable state between processes. if you have problems that don't need that then yes, parallelization is easy (the python-only version of the above would be to use multiprocessing). but maybe i've missed the point (seem to be making error after error at the moment...)

[edit: to clarify a bit more, the example given in the article linked is making each execution of a loop run in parallel, even if the loop affects some variable. so you would hope to get speedups in code that isn't written explicitly to be executed by multiple processes - without messaging etc. in particular, it guarantees safety while giving the chance (sometimes it doesn't work out) for speedups.]

<comment><USER4> Message passing is usually a "better" method of IPC anyway. The situations that genuinely require shared memory are rare.

<comment><USER5> Perhaps in your line of work, but in scientific computing, it's quite common. Scientific codes have parallelism at multiple levels, but loop-level parallelism is extremely common. Simply, you have large vectors or matrices that are operated on by a loop. The iterations of the loop are (mostly) independent, so you can use multiple threads to do them in parallel. Assuming shared memory, of course; the communication cost of transferring the vectors or matrices to another process over typical IPC mechanisms would kill the benefit of the parallelism.

OpenMP (<http://openmp.org/wp/>) for C, C++ and Fortran is the most common means of exploiting such parallelism in the high performance computing world.

<comment><USER6> I'm curious about what field of scientific computing you work in. I am only familiar with MPI and hybrid MPI/OpenMP codes. I haven't come across any OpenMP only programs ever in chemistry or physics.

If you have a loop with independent iterations then you can use message passing or shared memory to approach your problem (as well as loop vectorization). The main problem with message passing is that for very large datasets you can run into memory limitations.

<comment><USER5> I don't, really. I do system software for high performance and distributed computing. The scientific applications are given to me, and are largely benchmarks. (Although, now, I don't do much actual HPC, but high-throughput, low-latency stream processing.)

I have a friend who has a galaxy simulation that uses OpenMP but not MPI. The reason is simple: she doesn't have the expertise to make it distributed. Slapping a few OpenMP directives on the most expensive loops is easier than figuring out how to make it distributed using message passing. How much parallelism you extract out of a program is often a function of how much time and effort you can invest into it. Some people get "good enough" performance improvements by scaling on a single node.

</comment></comment></comment></comment></comment></comment><comment><USER7> It would be good to first make it possible to write Python code that can execute concurrently and use plain old locks and conditions -style synchronization. When that works efficiently, things like transactional memory can add value, but not before.

What comes to automatic mutual exclusion "AME", that sounds pretty far fetched to me. I don't think there's a production-worthy implementation of anything like this out there, all parallel programming environments need some kind of annotations from the programmer to keep stuff in sync. Is anyone aware of any research into this kind of systems?

If anyone is interested in parallel programming features in languages, Haskell's parallel programming facilities should be worth looking at. Not because the language or functional programming would be some kind of magic medicine, but simply because there's been years of research and hard work put into making Haskell work in a concurrent environment.

<comment><USER8> So are you saying the current STM prototype should be put on hold until a "plain old locks" Python comes along?

</comment></comment><comment><USER9> Here's his EuroPython talk <http://m.youtube.com/watch?v=pDkrkP0yf70>

And here's another talk about an alternative approach By Robert Hancock <http://lanyrd.com/2011/pygotham/shppw>

</comment><comment><USER3> this gives a route for AME on STM in CPython via the LLVM. it assumes (afaict - it took me a while to grok the article and i'm still not sure i have it right) that the CPython GIL will be removed. but we already have a failed attempt to remove the GIL via LLVM (unladen swallow). so the obvious question is - how meaningful is that failure? does it mean that removing the GIL via LLVM is hard? how hard? because without removing the GIL, AME and STM are pointless, right?

[edit: ok, thanks for the reply. it turns out it was dropped from the roadmap for unladen swallow because the garbage collection library they thought would help turned out not to - see [https://code.google.com/p/unladen- swallow/wiki/ProjectPlan#G...](https://code.google.com/p/unladen- swallow/wiki/ProjectPlan#Global_Interpreter_Lock) \- so i guess this question is much less interesting. sorry.]

unrelated note / question. this is the first time i've heard that HTM is restricted to the cache(s) (although it makes a lot of sense in retrospect). how serious a limitation is that for other languages? is haskell planning to use it? what about clojure (JVM - guess not)? and what happens to HTM if the cache overflows?

AME: automatic mutual exclusion, STM: software transactional memory, HTM: hardware transactional memory, LLVM: itself [fixed, thanks], GIL: global interpreter lock, JVM: java virtual machine, uff.

<comment><USER10> Acronym nitpick: LLVM originally stood for _low-level virtual machine_ , not _little-language_. It doesn't really have a connection to the little-language or domain-specific-language communities, but instead originates in a desire to provide a compiler target that was lower-level than generating C code, but less platform-specific than directly generating machine code (C-- is another project in that space). But it's since been renamed to just LLVM, a brand that doesn't officially stand for anything.

</comment></comment><comment><USER11> I find this incredibly interesting. I have a production server running a Flask app, performance is not stellar, and I've been meaning to try running it with Pypy w/ a web server, but I have yet to find a good resource for this. Anyone has any tips? Seems kinda counterintuitive with all these performance numbers with Pypy > Cpython when I can't even run it in production.

<comment><USER12> It's strange that you have a problem with Flask in production. What I've heard, there shouldn't be much of a problem with it. It should easily be able to handle 1,000 basic requests per second.

Have you done any performance analysis to figure out where the bottlenecks might be? Have you tried the flask-debugtoolbar (<http://pypi.python.org/pypi/Flask-DebugToolbar>)? How many requests do you get? What's the load average of the process? Is there an un-indexed SQL query you don't know about? In the most extreme case, have you put a debugging sleep(1) call in your code which you've forgotten to remove?

Without knowing more about where your Flask app is being bogged down, it's not that worthwhile to switch to pypy.

</comment></comment><comment><USER13> Haha wtf. After learning Haskell and seeing stuff like lparallel for CL, I just fundamentally can't pretend this is even worth learning. The GIL and the lack of foresight in designing Python are problems that are too fundamental to the language.

I love python, I've used it for a long time, but using it as anything other than for readable shell scripting is a waste of time. It's using the wrong tool for the job. If you are worrying about speed aside from algorithms, learn a different language.

<comment><USER14> Do you have examples of lack of foresight?

<comment><USER15> There are many examples of things that people now call "lack of foresight", but I think people quickly gloss over just how long Python has been around. It started in December 1989, when the first Pentium was still three years into the future. If Guido had tried to "foresight" his way to a language with an acceptable multicore solution at that point (on any hardware he could afford), the only possible result is total failure. I don't think there are many attributes the language currently possesses that were truly that avoidable. Most people's snap suggestions result in incurring costs to the language that it may not be able to recover from when it is young.

Personally, I think the truth is that Python is what it is; a very, very, very good OO language that has reached the mature stage of its life. It probably won't be able to make the leap to "true" multicore, and besides, even if it does it won't be likely to be very successful anyhow because it'll still be a very slow-but-powerful language.

(There's very little point in taking a language very near the bottom of the Shootout and adding 4 or even 8 way parallelism to it, when you could just rewrite the target hotspot in C and get the same performance on one core. The math just doesn't favor trying to add lots of "multicore" to such a _slow_ language. Python's one of my favorite languages, but that doesn't mean I can't see where it is weak.)

<comment><USER16> PyPy (together with LuaJIT and tracemonkey) has been banned from shootout, partly because we complained about unfair rules. Looking at shootout to asses the relative performance of languages where you cannot compare high- performance implementations of a language is a very very bad idea. Looking at shootout to compare would very likely be a bad idea anyhow, but right now it's just completely useless.

<comment><USER15> I was confining myself to CPython on the grounds that it is still what people complain about in terms of adding multicore to it. People don't complain about PyPy not having it since it's still developing, and it is not yet known whether PyPy will be fast enough to be worth worrying about multicore. To be honest I'm skeptical, but open to the possibility. But I rather suspect in the end that Python will forever be a slower language than the competition; all the dynamicness doesn't come for free.

Also the nicer primitives that IMHO the really-useful multicore languages are building on are either impossible (Haskell) or too late (Go/Erlang) for Python to deeply embrace due to massive legacy libraries and code. If you want to make threading "work" in Python, go nuts, it will bring much benefit, but Python will simply never be the first choice for tasks in which multicore performance and safety is the first or second priority, rather than the fifth or sixth. And that's _fine_. I'd really rather see Python become a better Python than see it become a crappy Go or something.

<comment><USER16> PyPy (on it's good day) is within 2x mark from the equivalent C. We're planning on closing this gap, but also we're planning on making every day the good day (right now you kind of have to know what you're doing in case you want to hit the sweet spot). This might be an interesting read for you: [https://www2.cisl.ucar.edu/sites/default/files/2cameron_spar...](https://www2.cisl.ucar.edu/sites/default/files/2cameron_sparr_FINAL.pdf)</comment></comment></comment></comment></comment></comment>


The smug style in American liberalism [2016 repost time] - <AUTHOR> https://www.vox.com/2016/4/21/11451378/smug-american-liberalism?repost-time <comment><AUTHOR> This article is basically a "by liberals, for liberals" attempt to inspire some philosophical self-reflection. It kept coming to mind over these last few weeks, and especially last couple days, of stories stirring up so much rage, counter-rage, questionable arguments, emotional triggering, etc. I've always identified left but I'm definitely starting to feel confused by the actions of other leftys. I definitely don't identify with the right so overall I'm feeling tribeless as of late. Thought it worth reposting this.

For completeness, here is last round of discussion: [https://news.ycombinator.com/item?id=12906783](https://news.ycombinator.com/item?id=12906783)

<comment><USER2> > always identified left but I'm definitely starting to feel confused by the > actions of other leftys. I definitely don't identify with the right so > overall I'm feeling tribeless as of late.

You took the words right out of my mouth. Lifelong liberal here, and lately I feel the left has abandoned me. Whenever I call something into question I'm told that I'm a nazi, or that I'm "just not angry enough" and some dishes have to be broken for the cause, etc. I no longer feel part of the crowd, but I am certainly far from a right winger, so.. tribeless myself.

The left is turning into a fascist group. Ironically while waving the flag pretending to fight it. In fact, the liberal movement is turning into the type of populace the government has always wanted. No right to an opinion, speech and expression are heavily regulated, and we rely on the government for everything. We tell others exactly how to live their lives and punish them when they don't.

This wasn't what I signed up for.

<comment><USER3> >> The left is turning into a fascist group.

The fringe left, while the moderates remain silent. I've found it interesting that of late there has been a minor attempt to redefine fascism as a right- wing ideology so that fascists of the left can't be accused of fascism.

<comment><USER2> It's more than minor, Google changed their definition a while ago, and dictionaries have been following suit.

I always thought it would be Republicans who would make 1984 a reality!

</comment></comment></comment></comment><comment><USER4> When I taught chess, we encouraged young players not to play "hope chess." Which means, making a move based on what you hope the opponent will do, not based on what you actually think is the best (or even a good) move. Playing a game based on logic, not based on hope, is both part of having a good strategy for any game and part of growing up. You can't just hope life's obstacles away.

And yet, by their actions and their rhetoric, "American liberals" seem to be implying that hoping that Trump gets impeached is, in fact, a viable political program, despite "rationally" knowing that the odds are against it, and that their denial of the reality is more likely to lead to 8 years of President Trump than 0 years.

The smugness is just a symptom of a much deeper childishness in the Democratic party.

</comment><comment><USER5> In assessing the current poisonous political environment with respect to all parties...

"It was so much easier to blame it on Them. It was bleakly depressing to think that They were Us. If it was Them, then nothing was anyone's fault. If it was us, what did that make Me? After all, I'm one of Us. I must be. I've certainly never thought of myself as one of Them. No one ever thinks of themselves as one of Them. We're always one of Us. It's Them that do the bad things." -Terry Pratchett, "Jingo"

</comment><comment><USER6> The smugness comes from the American left party moving right to try and capture some "center" that does not exist except in the professional and donor class.

The "left wing" party busting unions 60 years ago would be unthinkable. The left wing under FDR, not coincidentally the most popular US president in history, did not hatefully dismiss the poor working class as stupid, it gave them jobs and lifted them up. Rural areas voted Democrat. But we don't have the FDR Democratic party anymore, we have the (Bill) Clinton Democrats. Which is pro-deregulation, pro-incarceration, anti-union, pro-business, anti-worker, pro drug-war, pro-foreign war, pro-austerity (all policies which are bad for the poor, downtrodden, working class people) but they say the right things educated people believe about minorities and talk about how their policy is "evidence based" and boy if you cross them you'll sure face the full force of their weaponized smug.

Trump flanked Clinton from the left on trade. I'll let that sink in. He said he was going to fight for American workers and reject trade deals that helped big business. Clinton waffled about it (because she and the Democrats support it).

I love this article. The Democratic party has a whole lot of smug to offer; if you join them you can feel like you're in the cool kids club, the journalist's club, the wealthy professor's club, the doctor and lawyer and banker's club. You're on the "right" side of things. But beyond a sense of identitarian membership in the "expert, thinking, correct" class, it doesn't have much else to offer.

If you suggest a program to make public college and universities tuition free as it is done in many other countries, the "compromise" response from the democratic party is a means-tested boondoggle: first if you prove to a faceless government bureaucrat that you are poor and get all your paperwork together (which is humiliating and difficult) then tuition is waived. Which is completely out of touch with the attitudes of the poor; if you ever talk to a poor person, the difference between a handout they have to beg for by admitting their poverty and a universal program that everyone gets just for being an American (like roads, police, etc) is the difference between visceral humiliation they will refuse to face or hate using and something they can get behind.

Look at the new slogan: better skills, better jobs, better wages. Skills first of course, implying the under and unemployed are lacking skills. Implying skills retraining programs (and not job creation programs, like the new deal) actually work. Implying there are actually jobs for those workers once they get "skilled-up". Implying people who currently have good jobs and are wealthy are there primarily because they are skilled (there are plenty of pundits and opinion authors at famous newspapers and middle managers at large companies whose continued employment directly contradicts this theory). Implying a corporate tax cut is designed to help workers. On and on and on. The democratic party is smug towards workers because it considers itself above them. It isn't the party of the workers anymore, it's the party of the people who think workers are stupid.

<comment><USER7> > weaponized smug

That's a beautiful phrase. I'm stealing it.

> It isn't the party of the workers anymore, it's the party of the people who > think workers are stupid.

Very true.

</comment></comment><comment><USER8> Telling climate scientists that you know better is smug.

</comment><comment><USER9> Labor has no pricing power. Everything else follows.

</comment><comment><AUTHOR> Flagged?</comment>


Tesla reveals revolutionary new wiring architecture to help robots build cars - <AUTHOR> https://electrek.co/2019/07/22/tesla-revolutionary-wiring-architecture-robots-model-y/ <comment><USER2> So the idea is to make ecu in the vicinity to be responsible of the components, and drive them through communications.

Nothing new, trucks are applying this technique already.

The big drawback is that it increases SW and systems complexity. You cannot believe how few of the SW guys in the automotive industry have a clue about CAN (or flexray) communications and its failures mode. Its unbelievable how much code does not react properly on Communications failures / init / lag.

<comment><USER3> I know someone who used to work on BMWs and he said when they first came out with the lan-based car wiring problems were almost impossible to diagnose. Basically the shops would just swap parts until something worked. I think it's gotten better now as things have matured.

I expect because tesla is doing it in SV, they have enough software guys available to figure things out (and maybe frontload the diagnostics to make servicing things easier).

That's how technology works - make cheaper and less sophisticated hardware do more and better.

<comment><USER2> I don't see how this would help, because the default behaviour in a computer software is to inform the user that an error occured, please press enter to continue.

When you write software for a car, you always have to think about the impact of your code in the car, and pressing enter or crashing the sw is not the safe way to respond. So if the comms fails, you have to ask yourself what should the thing you're writing do and what the driver would see or think.

The problem with this architecture is that when you have a physical sensor, it is obvious when it fails and you can easily add contingencies. When you have a sensor other CAN or whatever, you are adding new failure modes: sensor OK but communication not reliable, or sensor info not yet received. You thus need more code to handle those new failure modes which are tricky for the average sw guy.

<comment><USER3> This is generally how everything computerized is developed.

The hardware is simplified and the software becomes more sophisticated. In this case, the recurring costs are minimized (simplified wiring harnesses that are installed via automation) in favor of an increase in non-recurring costs (more up-front engineering time).

The failure modes of these kinds of designs are by now a solved problem. The fact that automated assembly of simpler wiring harnesses occurs might mean the hardware reliability goes up.

</comment></comment></comment></comment><comment><USER4> Tesla has finally discovered subassemblies and the concept of minimizing wiring.

Other car companies have been using both for years...even decades, in Toyota's case (the original pioneer behind using subassemblies in automotive design).

</comment><comment><USER5> How is this different architecturally from typical structured wiring in networking, or say the arrangement of USB controllers within a typical laptop / docking station?

Although I'm not thrilled by the idea of my drivers side puddle lamp having it's own [IP] address!

<comment><USER6> AFAIK systems "bus based" are very similar, even for home/office electric cabling.

In the home/office "domotic" ones there are usually two couples of wires, two are the mains (poass-through) and two are "signal", every device is connected to both.

The closest thing in typical networks is - I believe - PoE: [https://en.wikipedia.org/wiki/Power_over_Ethernet](https://en.wikipedia.org/wiki/Power_over_Ethernet)

In automotive, normally the frame is 0V or "ground" and you need only a single (big) wire with +12 V and the signal cable/wires.

</comment></comment><comment><USER7> Tesla's big advantage is that they're not just building autos, they're redefining how they get built. It's not about the car, it's about the machines that make them and the interfaces between them. Effectively, the car is built using an electromechanical design language (DSL) specialized for automated manufacturing.

</comment><comment><USER8> > ts unbelievable how much code does not react properly on Communications > failures / init / lag.

I think this is the main reason those we do some kind of engineering work have jobs.

Those issues are prevalent in anything built with some constraint in time and/or money, that would be just almost everything built and in production on Earth.

</comment><comment><USER9> This is just Tesla claming they invented something like when Apple does the same thing for something that others have been doing for a while. It's PR.</comment>


Ask HN: reading source code on an e-reader? - <AUTHOR>

I'd like to review some source code without staring at a monitor.  Does anyone know of a tool that will convert C/C++ source code to EPUB for use in an e-reader? <comment><USER2> I don't really think this sort of question is HN worthy. But if you convert to PDF then run through Calibre you can convert to EPUB.</comment>


Show HN: What WordPress Theme Is That? - WP Theme Discovery - <AUTHOR> http://whatwpthemeisthat.com/ <comment><USER2> Cool!</comment>


HP Will Cut as Many as 30,000 More Jobs After Split - <AUTHOR> https://recode.net/2015/09/15/hp-will-cut-as-many-as-30000-more-jobs-after-split/ <comment><USER2> I believe it was a point of pride at the old HP (pre-Reagan years) that they never did employee layoffs. The company my father worked for, DuPont, had the same policy. Seems like ancient history now. I must be getting old.</comment>


Show HN: WinLess, a small graphical less/more-like for Windows - <AUTHOR> http://runtimeterror.com/rep/winless <comment><AUTHOR> I made this little utility this afternoon in a couple of hours or so. I was trying to check the diff of some project that i had made several modifications to which had 1000+ lines of changes and after trying to check it with "more" in the Windows command line, i thought "why am i doing this?". I remember using xmore (see [0] but i'm certain i used a different version that had some extra functionality) in X11 and googled a bit to see if there is any equivalent graphical tool for Windows, but couldn't find any so i made my own and decided to share :-).

[0] [https://cyber.dabamos.de/unix/x11/#xmore](https://cyber.dabamos.de/unix/x11/#xmore)</comment>


Rockstar, Patent-Holding Firm Partially Owned by Apple & Microsoft, Sues Google - <AUTHOR> http://daringfireball.net/2013/11/rockstar_suit <comment><USER2> I just don't buy that everyone is equally a bad actor here. Even if we take for granted that Google are two faced liars who secretly love patents, their public messaging is completely different. Google repeatedly has come out against them. The members of Rockstar brag about their patents, even flaunted them in the iPhone announcement. That difference is enough to make me draw a line in between the companies.

Of course, as Gruber's mocking of Android's openness has shown, he believes attempting to live up to an ideal and failing somewhat is worse than not even trying. At least when it comes to Google.

<comment><USER3> So both sides have different approaches, and are known malevolent actors, but one is better in your book because of their PR?... :(

<comment><USER4> Not both sides.

Apple and Microsoft has been waging anti-android patent war for ages, suing HTC, Samsung, Barns&Noble and many other Android manufacturers.

They are publicly pro-patents and when they are failing in the marketplace are suing anyone who poses a threat to their business.

Google is publicly against patents, releases a lot of source code under Apache license that grants patent rights and, as far as I know, haven't launched an offensive patent suit (i.e. against someone who hasn't sued them first).

The Motorola lawsuits that Gruber (and quite a few anti-Google people) is so keen to quote happened years before Google acquired Motorola.

At this point I don't even understand why people expect Google to just take it and not launch offensive against Apple and Microsoft. You can be pro-peace but if someone points a gun at you and shoot, you shoot back and that's what Google should do: shoot back.

<comment><USER5> I'd like to know where Apple has been public about patents because I find that surprising. If anything, there seems to be hardly any corporate communication addressing these types of things letting the media speculate to their heart's content.

Your premise about Motorola is incorrect. Google was required to approve the patent assertions against Apple. From FOSS Patents:

>A few months after that Zeitgeist talk, in January 2012, Google authorized a Motorola Mobility lawsuit against Apple over six patents in the Southern District of Florida. The merger agreement was publicly available and absolutely unequivocal about the fact that Motorola needed Google's consent prior to bringing new IP assertions while the merger was under antitrust review.

But even if that weren't the case, there would be nothing stopping them from settling or withdrawing the case.

</comment></comment></comment></comment><comment><USER6> So in Gruber's mind, if someone drops a bomb on your country, and you launch a retaliatory strike, you're just as bad.

The timeline here would be instructive. Microsoft and Apple sued and threatened Android OEMs first. Apple's CEO is on record as practicing 'thermonuclear war'. Steve Ballmer gleefully cheered about their patent licensing. Both companies love patents. Jobs "and booooooy have we patented it!"

Weaksauce attempt by Gruber at drawing an equivalence.

<comment><USER5> Gruber is pretty clear about what's in his mind, so there's no need for terrible analogies.

>If you want to argue that the whole patent system stinks, and that all of these tech giants are abusing it, I agree. But if you want to argue that Apple and Microsoft are in the wrong, and poor Google and their Android partners are victims of one-sided abuse, I'm going to have to disagree. If there's a difference between Apple/Microsoft and Google in this war, it's not over nobility, but rather over how well each side has played the game.

These companies pursue what's in their self-interest, and attempts to portray Google as being somehow above it are disingenuous, regardless of how high- minded the PR makes them seem.

<comment><USER6> Disingenuous? Let's talk reality. Apple sells hardware, Microsoft has a licensed software business. Android is a fundamental threat to both businesses. Google doesn't make any money from Android.

Now, who has an incentive to launch patent lawsuits as an attempt to restrain competitors? The one whose business model is fundamentally threatened by open, free operating systems, or those who don't make any money from them? Which actors have an incentive to try and impose costs on open source? It is fundamentally not in Google's interest to ban iPhones, as they make money on iPhones just like they do the desktop Web. A commoditized mobile industry, or a competitive mobile industry is a net win for Google, only a monopoly monoculture is a loss for Google. Left to their own devices, Google could undermine iPhone by just shipping Android for free, no need for legal fights to undermine it.

Microsoft in particular had a plan to try and stop Linux when it started eating up the server-side market by launching or threatening patent IP issues, for much the same reason as the current attacks on Android. The reality is, the only recourse for proprietary vendors to go after something that is free is to use the force of government and the courts.

Which company has a founder/CEO who openly proclaimed in inflammatory rhetoric that he was going to "destroy" and go "thermonuclear"? Which company has threatened and asserted more patent attacks? You cite a single instance in which Google inherited a long running fight between Motorola Mobility and Apple, but completely ignore the fact that Apple has sued pretty much every Android vendor, and Microsoft has shaken down many. What do you call Microsoft's shakedown of B&N's Nook? You think that's morally equivalent to Motorola?

This is about as intellectually honest as the people who try to compare Fox News to CNN, or claim that the "extreme left" in Congress is just as bad as the "extreme right", despite the fact that none of the "socialists" in Congress have brought the government to a two week shutdown, or a debt crisis.

I don't call someone who abuses the system better at "playing the game", that suggests that this is a fun activity with no moral or ethical repercussions. I call someone who is better at abusing the legal system MORE immoral than someone who is doesn't do it as much.

In order words, it is not a game. Calling it one is a disingenuous attempt to paper over what is disgusting behavior and whitewash it enough that you can try to draw a moral equivalence by leveling it all out.

Yes, Google is pursuing what is in their self interest, but there is nothing immoral about releasing stuff as source for free or competing in the marketplace. It is not evil, despite the fact that Jobs tried to say it was.

What is immoral is gumming up the courts and using the force of the government to beat your competitors because you can't stop consumers from choosing other products.

<comment><USER7> > Google doesn't make any money from Android.

You might want to rethink that statement. If Google had not bought Android and built it into the ecosystem it is now, do you think Google had made more money by now?

> It is fundamentally not in Google's interest to ban iPhones, as they make > money on iPhones

Google doesn't sell apps for iOS, but it does pay Apple to be the preferred search engine in Safari. So that line tells me that you know Google makes money on Android (the way it always does, by collecting user data to sell ads – and on 'Android with Google' it gets more of that data than anywhere, except perhaps Glass).

<comment><USER6> Google makes money from the Web, primarily from Search. Android is an insurance policy against a world in which Bing is the default search engine. An insurance policy is a cost in case something bad happens.

It is well known fact that Google makes far more money on mobile ads on iOS than Android.

Google's interests are for people to use the Web as much as possible, and therefore, to make it as cheap as possible to get on the web and spend all your time there. That is, Google's interests are aligned with the interests of the consumers to have downward price pressure on mobile devices.

Apple's interest is to maintain margins, it is fundamentally against the interest of consumers in that regard, and therefore, it has a large motive to restrict trade of competitors.

Competition is in the interest of consumers, and therefore these abuses of the patent system are also not in the interests of the public. Apple should stop it, and the government should reform the system.

<comment><USER7> > Google makes far more money on mobile ads on iOS than Android.

That's small potatoes. What I'm talking about is the data Google collects while you're online and logged in with a Google account or using an 'Android with Google' device. From your geolocation to your contacts, to the websites you visit, to the searches you perform, to the documents you create in Apps, etc etc.

That is data Google can use to display targeted ads on any device that is tied to you, not only on its Google search engine, but on the majority of websites on the Internet (Google owns DoubleClick.)

[http://en.wikipedia.org/wiki/Doubleclick](http://en.wikipedia.org/wiki/Doubleclick)

<comment><USER6> "Can use". Emphasis on CAN. Display Ads constitute about 12% of Google revenues, the degree to which people imagine Google use's personal data for targeting, and the actual reality are far apart.

The reality is, Google makes more money on AdWords through Google searches, and so the default search engine is far more important.

<comment><USER7> Given that Google is a publicly traded company, I think it's a question of 'when' instead of 'if' it will trade on user data more than they do now. It only takes a flip of a switch.

That's why companies like Amazon, Facebook, and Twitter have the market value they do. Any of those can at any time flip the switch to cash in on the user data and mindshare they've collected.

Apple has a very different business model. It hasn't shown any interest in user data (past credit card information) and doesn't even have the infrastructure to exploit it. Because Apple is so inept in web services, I don't mind them having my information on file, while I worry about what Google does with my information even when I'm not logged into its services.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> Life very rarely presents us with simple black & white choices, but that doesn't mean that it's not important to make distinctions among distinctly different shades of gray. Don't let party apparatchiks like Gruber fool you with their smoke & mirrors. This is nothing more than an attempt by Apple & Microsoft to accomplish in the courtroom what they have failed to achieve in the marketplace. We have it straight from the Cupertino horse's mouth that this is a "nuclear war" designed to destroy Android.

If this succeeds it will set a very dangerous and chilling precedent for the development of any new technology going forward. Anyone that claims to value innovation and opportunity for new ideas should condemn it.

<comment><USER9> The idea that Apple is using to go on the offensive because they are "failing in the marketplace" is ridiculous, since extensive patents were touted during the iPhone's 2007 introductory keynote.

Also, what evidence is there that Google is still committed to making Android open source even since Andy Rubin left the project?

Finally, I don't know enough about Andy Rubin's relationship with Apple, but can somebody who does comment on this? [http://www.fosspatents.com/2011/09/apple-to-itc-andy- rubin-g...](http://www.fosspatents.com/2011/09/apple-to-itc-andy-rubin-got- inspiration.html)

<comment><USER8> There's a world of difference between filing protective patents and using patents offensively against your competitors. Until fairly recently, this was considered to be something that was largely beneath the dignity of the big players in the industry.

And, even if Android were completely closed, it would still occupy the higher moral ground since Google did not instigate this war and would be happy to see it end.

<comment><USER9> What if to Apple, the war started when Eric Schmidt stayed on Apple's board of directors while Google was developing a direct competitor to the iPhone?

<comment><USER8> Or perhaps, earlier still, when Apple revealed their intention to roll out a platform over which they had complete gatekeeper control over any third party software installation?

</comment></comment></comment></comment></comment><comment><USER10> _The difference between Lodsys and Rockstar is that Lodsys is a bully, suing small (and in some cases, downright tiny) companies that lack the financial wherewithal to fight back. And in fact, when Lodsys's targets do fight back, Lodsys runs away — settling for nothing in order to avoid a trial. Rockstar may be a patent troll, but they're a patent troll that at least is picking on someone its own size._

Lodsys is _almost_ the antihero here. They are just being exploitative which is dumb and obnoxious. If only they were explicitly attacking Apple, Microsoft, and Oracle's ecosystems. Dishing out the abuse towards their small innocent customers. Using the system these companies have expanded and propped up to really hurt them. And say, "Hey, this is what we are doing. Maybe you should fix this Congress."

</comment><comment><USER11> The patent system is to blame here.

Android is shipping on more units than any other OS and Google is going to win if things continue this way based on current trends. They are probably not far away from getting Android to a similar level of polish as iOS within the next few years. There's only so much you can improve on in terms of the OS software before you hit the point of diminishing returns.

At this point, the trends are clear, Google will win the mobile OS war. There is really no other likely outcome at this point unless Apple does something drastic.

In business, it is pretty much expected that public companies will act in the best interest of its own stakeholders, so this lawsuit was just something that had to happen. You can't blame Apple or Microsoft for this. This is what you do as a business to protect your products and removing your barriers to entry, or you fall off the mobile rocket ship.

Also, honest question, but why do people get so upset over this blog? I just looked up John Gruber on Wikipedia and his full time job is maintaining a blog to write about Apple products, with a history of being an "Apple Apologist". We're we expecting him to write something different?

</comment><comment><USER12> From what I've understood he is actually defending and white knighting Apple's patent trolling.

For anyone still in doubt whether Gruber is actively paid by Apple (and possibly others) to deliver viral marketing - here you go.

</comment><comment><USER13> So Rockstar - an NPE patent Troll by definition - filing lawsuits against Google & Samsung is the same as Motorola filing lawsuits according to Gruber. Except Motorola has - you know - actual products that it sells. Additionally this company was created by the Apple + M.S Cartel explicitly to go after Google and Android - not to create any value, only to harass and stifle a competitor.

Sometimes Gruber gets so wrapped up in his own B.S. that he starts defending the scummy actions of Apple - sad.</comment>


Step by step from jQuery to Backbone - <AUTHOR> http://open.bekk.no/from-jquery-to-backbone/ <comment><USER2> I like the tutorial style and I wanted to show something like this to someone, so that's really cool.

There's one thing I think you've got wrong in the example: by having functions bound to the same event (addStatus and clearInput), you paint yourself into a corner with Backbone, where you can only bind one function declaratively. However, your solution of clearing the input whenever a status is added is wrong, since (conceptually) the status collection may add statuses whenever it likes for whatever reason, but now doing so will erase the user's input box. Suppose this evolved into a realtime twitter-like thing – you would get hard to understand bug reports of people's input "disappearing".

(Moralising: It's important not to let architecture astronautics distract form the actual logic of the program.) When the NewStatusView successfully adds a status, it should be responsible for clearing its input ready for reuse. The call to clearInput belongs at the end of addStatus. A further argument for this is that it will eventually have to move into a success callback of the collection add, for example when a banned list of words in statuses lives on the server, or you actually broadcast the statuses.

Don't know why I felt compelled to make such a long comment; hope it makes a great reason even better.

<comment><AUTHOR> Great point. I'll take a look at it and maybe make some changes. :)

</comment></comment><comment><USER3> This is an excellent tutorial. It's also an excellent lesson in why you should stay away from frameworks like Backbone.

(To declare, I have written large apps with Backbone, so I'm not entirely talking from my ass.)

You've gone from a simple, succinct, and easily understood set of code to something that now relies on 2 extra frameworks (backbone and underscore), in addition to still needing jQuery. You've had to incorporate several layers of abstraction (views, cloning events) to simply do what was originally done in a few lines.

I know what the appeal of something like Backbone is. You think that adding in all this abstraction will make it easier to maintain your product. I will even agree that to an extent, it does.

However, a big problem is that the people that already care about organizing their code well, already do so. Backbone is so needlessly open-ended and directionless that a dev who doesn't already know how to write good code will just write worse code.

When I first learned Python, it was enlightening. Here was a language that says, "This is the right way to do it". It got rid of the baggage of other languages and got to business. Backbone is definitely not that for JavaScript development.

Backbone gives you so much rope to hang yourself with, and with so little sense of how things should properly fit together, that it's like wandering around a desert for days on end with a noose on your neck, until you fall into a deep enough hole and die.

I prefer to write my code so that there is an overall architecture, usually with a modular pattern when possible. Anything that needs to happen code-wise for a particular module is all contained within that code block and nothing else. That code is more or less like what was originally posted, where all the things that matter for that module are done in that module.

The other technique I use for modules to talk to each other is sheerly through event signatures. If you have a login module, it can emit events like "loggedin" or "loggedout". Any other module can subscribe to those events and do their own thing, without interfering with others, etc.

This pattern works very well and is a hell of a lot more usable than anything I've ever been able to do with Backbone.

Finally, Coffeescript is not JavaScript. It's another annoying abstraction.

JQuery is not JavaScript, but an incredibly useful library.

Stop piling one layer on top of another thinking you're making your life easier. Get back to basics, learn how to structure your code, and just get some fucking work done.

<comment><USER4> Finally someone who agrees with me. I like Backbone quite a bit and a large part of our application is now using it... However, the added code, layers of abstraction, and lack of any "right" way to do things make maintaining it more of a hassle than anything else. I like that many people are trying to build application frameworks on top of backbone (marionette, etc) but this is just making things worse as now you have 20+ choices on how to write your backbone application, thereby making it harder to maintain across developers (what are the chances the new guy you hired is not only proficient in Backbone, but also backbone using handlebars instead of underscore, and also backbone + chaplin -- or vertebrae, or thorax?)

When our js guru started porting our app the backbone, it was intensely difficult at first for the rest of our devs to make trivial changes to our front end code. What used to be a well defined information flow was now spread across dozens of files.

<comment><USER5> Don't worry, there will soon be a movement to get back to POJOs (Plain Old JavaScript Objects) and the cycle of engineers over-thinking a problem will be reset.

</comment></comment></comment><comment><AUTHOR> Author here. The blog appears to be down, check it out on Github instead: [https://github.com/kjbekkelund/writings/blob/master/publishe...](https://github.com/kjbekkelund/writings/blob/master/published/understanding- backbone.md)

<comment><USER6> I can't say thank you enough! This is the first Backbone.js tutorial that really made any sense to me.

</comment></comment><comment><USER7> Very well written tutorial, right up to this part:

 _However, the code has increased from 16 lines to more than 40, so why do I think this is better? Because we are now working on a higher level of abstraction. This code is more maintainable, easier to reuse and extend, and easier to test. What I've seen is that Backbone.js helps improve the structure of my JavaScript applications considerably._

I don't see any justification for those claims. In my experience, more code means less maintainable, because it's more to understand and provides more opportunities for bugs to creep in. I can look at the original 16 lines and understand immediately what's going on, while the 40 line result doesn't even fit onto my screen all at once, and the application logic is broken up and spread out across a bunch of boilerplate-type code.

Regarding reuse and extensibility, I don't see that here either. Sure, you can add more behavior when either element changes, but you could have done that before just as well. For reuse, what's the likely-hood that you're going to need a control like this, which seems to be tightly bound to the application requirements? If you're writing a library or generic control reuse is important, but most of the time most developers are writing custom one-off requirement-specific controls, and reuse just isn't as important as clarity and simplicity. Besides, if you _really_ wanted to reuse the original, you could move it all into a function and pass 'new-status' and 'statuses' as arguments; then any form/textarea/ul triplet can be used as a control.

Finally, testability: yes, I'll grant that your final version is more scriptable which makes it easier to write automated tests. However, you've also got a lot more to test, because now your code is subject to bugs in the 1000 lines of Backbone code you've made yourself dependent upon. What if there is some problem in the communication with your service? That's buried in Backbone and you're insulated from it, unlike in the original where $.ajax gives you nearly direct control over all aspects of the communication. The original, being much simpler, is also easier to visually inspect for bugs. Not much can go wrong in 16 lines of tightly integrated code, but once you switch over to inserting bits of behavior into a larger framework that you don't control and probably don't understand, who knows when your code will get executed (or not) and what edge cases might arise?

BTW, yes I know jQuery is a much larger and more complex dependency, but both the original and final examples depend on it equally. More importantly, your code calls jQuery rather than jQuery calling your code: your code stays in control. With frameworks like Backbone, your code gives up control to the framework.

<comment><AUTHOR> Great feedback. What I mentioned are things I've seen from working on fairly large JavaScript apps in enterprise environments. Maybe I should have made it clearer that those where my experiences.

What I failed to note is that in my experience the end result has less code and less complexity when writing my JavaScript like this.

<comment><USER8> IMO what the Backbone.js version has over the jQuery [only] version is readability in a large code base. I assume by "maintainability" this is one thing you mean.

You're only going to cover a small example in a tutorial, so the above benefit is not readily apparent.

<comment><USER7>
       statement: {

           context: { opinion: "mine" },

           subject: "Backbone.js",

           type: new Comparison("jQuery [only] version"),

           filter: new FilterContext("large code base"),

           result: "readability",

           action: raise("assume", { subject: "maintainability" }),
       };

       on("assume") {
           belief: "one thing you mean"
       };

I disagree that this spread-out, boilerplate-rich format becomes more readable the more of it that you have. I think the larger your codebase is, the more important it becomes for distinct chunks of application logic and behavior to be written clearly and concisely.

</comment></comment></comment></comment><comment><USER9> Very nice writeup, probably one of the best practical articles on transitioning code to MVC-style structure.

My only question about the result is the binding of NewStatusView.clearInput to the collection add event. If I understood correctly, it is logically incorrect in that clearing the input field should be a result of having submitted it, not of the fact that something has been added to the collection. A typical example to illustrate this issue would be a collaborative environment where multiple views/users can be adding to the same collection - say, a chat. Every submit would clear everyone else's text fields mid-typing?

<comment><AUTHOR> Thanks! And great question. Of course, in real-time environments you need to be really sure which events trigger when. However, on a normal single-page app the add event will only be triggered when you have make a request to the server, and you won't do that mid-typing, only when you're finished typing.

Edit: But, of course, as Robin_Message mentions, it's not always as clear cut as I stated, and there might be some issues with my solution.

</comment></comment><comment><USER10> If the premise of this article is to teach Backbone to people who are accustomed to jQuery, I think it fails miserably. It's long-winded, obtuse, and the example, which is supposed to firmly anchor you, is hazy and enigmatic.

The real trick of Backbone is treating it like a delegator for the UI. It handles creation of new elements, interactions, and creates a 1:1 relationship between the data and what is seen on screen; that's why it's so important in single-page JS websites: It handles the gap between raw JSON data and page visualization.

</comment><comment><USER11> As others have said, thanks, really great tutorial. I don't know whether i'm going to go all the way to backbone (maybe?), but wherever i stop i'll have better, more maintainable JS.

I also have a quick question; why from the 'Creating a view' section, why do we have:

        var NewStatusView = function(options) {
            var statuses = options.statuses;
            ...

        new NewStatusView({ statuses: statuses });

instead of just calling `NewStatusView(statuses)` and having `var NewStatusView = function(statuses){` ??

<comment><AUTHOR> I've gradually started passing a hash instead when I expect the number of arguments to increase, which I do for constructors (I hate reading code and not understand the arguments in a function call). But also, it made the change over to Backbone views easier and less "magic".

</comment></comment><comment><USER12> Now this is serendipitous! Earlier today while on my run, I was thinking about finding a resource to take me to the next step after the codeschool backbone.js series.

This is great, though I am having some doubts about whether backbone is the way to go or if angular or ember would be a better choice to invest my time in learning.

<comment><USER13> After playing with them all, we went with Angular for what it's worth. Less lines of code, IOC is pretty neat, and it just "feels good". We're happy w/ our choice thus far.

</comment></comment><comment><USER14> Really nice format. I've been using backbone for over a year and knew exactly how the code was going to end up but my understanding of 'why' is better after reading it, which is a really the most import question to answer. I hope this style catches on, maybe the format could be called a Why-To :)

</comment><comment><USER15> Thank you so much for writing this. I've always felt like I was missing something with Backbone -- I knew that it was mostly infrastructure and assumptions that helped organize and clean up client-side code, but coming from mostly a backend/rails background, I never quite understood the assumptions that were being made and the "magic" that has happening behind the scenes. This step-by-step process is really helpful.

</comment><comment><USER16> nice - like the presentation with the file diffs. awesome stuff.

<comment><USER17> I noticed that as well. Stealing the idea for my own blog. Makes it very clear what changed, which is instrumental during learning.

</comment></comment><comment><USER18> I really like this tutorial's format. I made another one in its spirit about Agility.js, another framework that is significantly more succinct: [http://www.pindi.us/blog/step-by-step-from-jquery-to- agility...](http://www.pindi.us/blog/step-by-step-from-jquery-to-agility-js)

</comment><comment><USER19> Is it just me or do these two apps not behave similarly at all? The jQuery app posts a status to a URL, the Backbone one does not. If the intention was only to show your typed status below once you click a button, why the Ajax call to begin with? Again, maybe I missed the point entirely.

<comment><AUTHOR> Ah, minor mistake. Used `add` instead of `create`. It's been fixed in the blog post, and I'll push a fix to the app soon.

</comment></comment><comment><USER20> I've recently become a bit of a fan of Backbone, and have been tying it with Vertx and SockJS. It makes my fingers tingly.

<comment><USER21> Might I ask: how are you using those together?

<comment><USER21> I'm asking because Backbone is written for CRUD apps out of the box, and real- time webapps aren't CRUD normally.

</comment></comment></comment><comment><USER22> Very nice and innovative style and osom tutorial as well. If only one could write this for spine as well :D

<comment><USER23> The author of Spine wrote this book: <http://shop.oreilly.com/product/0636920018421.do>

</comment></comment><comment><USER24> Really nice way of explaining things.

</comment><comment><USER25> Very excellent read, thanks!

</comment><comment><USER26> Very nice article! Thanks!

</comment><comment><USER27> Github link 404s. :-(

<comment><AUTHOR> Which Github link? This one: [https://github.com/kjbekkelund/writings/blob/master/publishe...](https://github.com/kjbekkelund/writings/blob/master/published/understanding- backbone.md) ?

</comment></comment><comment><USER28> Nice tutorial.

I have a question: how come you aren't making use of _.bindAll?

<comment><AUTHOR> Because I think it's cleaner having it as the third argument on events instead of having a large list of stuff in _.bindAll in initialize (and additionally, I've found it simpler to explain it to others, but I don't know why). I think it's just one of those subjective choices.

<comment><USER29> Agreed. Plus, bindAll [1] does quite a lot of magic under the hood that IMO is an overkill compared to passing the context object.

[1]: <http://underscorejs.org/docs/underscore.html#section-52>

</comment></comment></comment><comment><USER30> HAHAHA nice IE 503 error...

</comment><comment><USER31> Well, the website is down... Thanks kjebkkelund

<comment><AUTHOR> You can read it here: [https://github.com/kjbekkelund/writings/blob/master/publishe...](https://github.com/kjbekkelund/writings/blob/master/published/understanding- backbone.md)

Our Wordpress bullshit is failing hard.

<comment><USER31> I found the link, awesome writeup :) Thanks!</comment></comment></comment>


Fat pointers in C using libcello - <AUTHOR> http://libcello.org/learn/a-fat-pointer-library <comment><USER2> In my personal experience, "a bit more than a pointer" works best as a pair of (start, end) pointers (where "end" points to just beyond the last element.) The most obvious reasons for this are:

\- slices become a total non-issue since a pair of (start, end) already is a slice and you can just move start and end.

\- comparing against an end pointer is generally easier than adding up a length value first, particularly if you're slicing at the same time.

\- the end pointer value is independent of the array element type, so if you e.g. cast to uint8_t * (which arguably you shouldn't in most cases) it stays exactly the same. If you store a count you need to adjust a multiplier. If you store a byte length, you need to do a lot of divides or casts to deal with pointer arithmetics.

Also, this is a huge red flag to me:

[https://github.com/orangeduck/Cello/blob/master/include/Cell...](https://github.com/orangeduck/Cello/blob/master/include/Cello.h#L135-L140)

      #define is ==
      #define isnt !=
      #define not !
      #define and &&
      #define or ||
      #define in ,

P.S.: This also is a "try to invent a new programming language without inventing a new programming language" thing. Have your cake and eat it... either it's C or it isn't, and this library is leaving the space of "normal" C.

<comment><USER3> This is how Go does slices, though they are 3 fields:

        type slice struct {
            zerothElement *type
            len int
            cap int
        }

Though I suspect under the hood the len and cap are ordered first.

If it's good enough for C greybeard Ken Thompson and unix hacker Rob Pike, its good enough for me.

In fact I've looked for a port of go-style slices to C and haven't found one. Maybe people think sds is good enough?

<comment><USER4> Go has a rather idiosyncratic take on arrays and how they're used, which is reflected in its slices. I can't think of any other language or framework that did it this way.

<comment><USER5> Rust Vec has the same representation.

</comment></comment></comment></comment><comment><USER6> Interesting idea, but this implementation has UB:

        typedef void* var;

        struct Header {
          var type;
        };

        // ...

        #define alloc_stack(T) header_init( \
          (char[sizeof(struct Header) + sizeof(struct T)]){0}, T)

        var header_init(var head, var type) {
          struct Header* self = head;
          self->type = type;
          return ((char*)self) + sizeof(struct Header);
        }

The section "struct Header* self = head" is UB. The alignement requirement of the local char array is 1 but the alignment requirement of struct Header is that of void* which is probably 8.

<comment><USER7> Not only that but you have a pointer to a parameter returned back and used outside its scope ...

<comment><USER8> It's not a pointer to a parameter, it is just the parameter itself.

var is a typedef for void* and no & appears in the function.

<comment><USER7> > no & appears in the function.

It's an array, so you don't need & to take its address, it decays into a pointer without &.

Imagine:

        char buf[sizeof(struct Header) + sizeof(struct T)];
        char *p = buf;

Then take away the names so that you are effectively passing p as a parameter... Then returning p.

As in, an anonymous temporary being given to the function, and the function returns its address back.

It's assuming that this temporary parameter buffer will exist after it is used and the function has returned. I'm not sure what the standard says for that but it is crazy sketchy. [Edit: Googling around, it seems like maybe this is illegal in C99 but possibly legal in C11? Or that C11 changed the rules for this. Does not seem like a great thing to rely upon.]

<comment><USER9> There is no issue here (except the one highlighted by joejev).

Many standard library functions return a pointer which they got as a parameter (or another pointer offset from it, as is the case here). The compound literal is no more "temporary" than a variable that was introduced right before the function call. Lifetimes of compound literals are specified in section 18.104.22.168 of both C99 and C11.

      func((int[256]){ 0 });
      // is mostly equivalent to
      int __a__[256] = { 0 };
      func(__a__);

<comment><USER7> > Many standard library functions return a pointer which they got as a > parameter

Obviously. But this does not extend the lifetime of the buffer they are passed. Namely you can't use this as a technique to extend the life of automatic storage falling out of scope.

> Lifetimes of compound literals are specified in section 126.96.36.199 of both C99 > and C11.

This is what I was missing. So it is valid by the standard. Which is good if you have looked it up. It remains _not obvious_ when reading source without a copy of the standard on hand, or prior knowledge of that section. Passing an expression of that sort and keeping a pointer to it visually looks like the intent is to retain a pointer of more limited scope. Intuitively it would make just as much sense if the lifetime were shorter. If you are seeking clarity of intent this is not a great thing to rely upon.

</comment></comment></comment></comment></comment></comment><comment><USER10> Perhaps a stupid questoon: Why isn't a vector type similar to { ptr, count } a normal thing to pass around in C? It's what you reach for in any other language, why did it become idiomatic to pass pointers and lengths separately in C?

A C standard library has a header file for _complex math_ but it doesn't define a simple fixed size array struct? Why is that? Is it because they become pointless when there is no generics to deal with the stride?

<comment><USER11> > why did it become idiomatic to pass pointers and lengths separately in C?

I've read that it's because there used to be binary interface issues with structures. They can be returned from functions and passed as parameters but it isn't immediately clear how that happens: is it on the stack, in one register or in several registers? Even today there are compiler options that affect the generated code in those cases:

      -fpcc-struct-return

      Return "short" struct and union values in memory like
      longer ones, rather than in registers.

      -freg-struct-return

      Return struct and union values in registers when possible.

[https://gcc.gnu.org/onlinedocs/gcc/Code-Gen- Options.html#ind...](https://gcc.gnu.org/onlinedocs/gcc/Code-Gen- Options.html#index-freg-struct-return)

[https://gcc.gnu.org/onlinedocs/gcc/Code-Gen- Options.html#ind...](https://gcc.gnu.org/onlinedocs/gcc/Code-Gen- Options.html#index-fpcc-struct-return)

[https://gcc.gnu.org/onlinedocs/gcc/Incompatibilities.html#in...](https://gcc.gnu.org/onlinedocs/gcc/Incompatibilities.html#index- fpcc-struct-return-1)

<comment><USER12> >They can be returned from functions and passed as parameters but it isn't immediately clear how that happens: is it on the stack, in one register or in several registers?

Why does it have to be clear? It can be unspecified and the compiler will do what it thinks is best given the struct, e.g. return `struct {int x,y};` in registers, return `struct int[80] x}` as pointer to memory or write in-place to the caller's stack, via RVO.

<comment><USER11> > Why does it have to be clear?

Because it's part of the binary interface. Changing the binary interface prevents existing programs from interoperating. Everything breaks until the software is recompiled and that can be a major nuisance at best and impossible in the worst case.

Simple, well-defined and stable binary interfaces are a major reason why C is still widely used. Uncertainty in this area is never a good thing so people will actively avoid language features that introduce it. Looks like enums aren't favored for similar reasons: what is the underlying type?

</comment></comment></comment></comment><comment><USER13> On the "weird pointer solutions" tangent, there's the ARM authenticated pointers: [https://lwn.net/Articles/718888/](https://lwn.net/Articles/718888/)

Given that years ago we added a mandatory piece of hardware to most systems to implement virtual memory, I'm now starting to wonder what security and/or performance benefits could be achieved by delegating memory allocation to (or through) hardware.

<comment><USER14> That's a fascinating approach - how widely used is it? Would love to know whether it causes any problems for 'existing' code.

<comment><USER15> Every new (post-2018) iPhone ships with this. iOS developers can build code for the architecture but I believe Apple currently strips it out before distribution, so its use is limited to the OS for now. I would assume at some point they'll flip the switch to allow it; until then developers can use the toolchain to test if their code still works (generally it does, but messing with function pointers in ways unspecified by the standard can occasionally cause problems). 'pjmlp is fairly interested in this topic so they might be able to share some more examples of it being used if they drop by the thread.

</comment></comment></comment><comment><AUTHOR> The concept of "fat pointer" the article is about has been described by Walter Bright (D creator) as "C's Biggest Mistake": [https://www.drdobbs.com/architecture-and-design/cs- biggest-m...](https://www.drdobbs.com/architecture-and-design/cs-biggest- mistake/228701625). It's also an interesting read.

The summary version (from Walter Bright's article) is:

> C can still be fixed. All it needs is a little new syntax:

> void foo(char a[..])

> meaning an array is passed as a so-called "fat pointer", i.e. a pair > consisting of a pointer to the start of the array, and a size_t of the array > dimension.

<comment><USER16> It's worth noting that fat pointers didn't originate with Walter Bright or that 2009 article. The oldest C-with-fat-pointers I can think of off the top of my head is CCured from 2002: [https://people.eecs.berkeley.edu/~necula/Papers/ccured_popl0...](https://people.eecs.berkeley.edu/~necula/Papers/ccured_popl02.pdf)

The paper mentions fat pointers in passing, not putting the term in quotes, not defining it, and not giving a citation -- which makes it clear that the term was already well established at the time.

<comment><USER17> Fat pointers were part of Pascal (and derivatives), although I'm sure the concept has existed in one form or another going back to the beginning.

edit: Pascal pointers were just a location and size, however, not a slice-type fat pointer; however, I have always heard of _any_ pointer containing more information than a memory address referred to as a fat pointer (except tagged pointers). YMMV.

<comment><USER4> I don't think Pascal pointers had to be implemented as location+size, since most operations that required checking the size were undefined or implementation-defined anyway. Some implementations might have used them to provide runtime checks, but it was certainly not the case in Turbo Pascal, for example.

</comment></comment></comment></comment><comment><USER18> Is this blog post confused or am I confused? It keeps talking about fat pointers but the description looks much more like "arrays with their length stored before their first element," which is a massive difference.

<comment><USER19> I think the latter. Fat pointers were supposed to be 2 pointers wide, not 1...

<comment><USER2> It's just using "fat pointer" to refer to the concept of passing around a pointer with extra information concerning the data it points to. I agree that generally people would expect "fat pointer" to imply a larger pointer itself, but I don't think the label is misused egregiously enough to warrant picking at this.

<comment><USER18> > It's just using "fat pointer" to refer to the concept of passing around a > pointer with extra information concerning the data it points to.

Is it actually? Here's a quote: "The trick is to place the value representing the number of items in the array in memory just before the pointer we actually pass to functions. This pointer is fully compatible with normal pointers,"

So suppose I pass this "fully compatible with normal pointers" pointer to a function.. and it ends up being stored in a register. Now where is this location "just before the pointer"? In the previous register?

I don't think this post is describing fat pointers at all. I think it is describing arrays with a prepended length header. There is no data in or before the pointer, there is metadata before the pointee assuming you're pointing at something with metadata prepended to it.. No extra information is passed with a pointer, it's just assumed to be there at the pointee. Call it a fat pointee if you want a fancy name; at least that acknowledges the onus is on the pointee to have the right metadata in the right place. There's nothing in the pointer here.

<comment><USER15> > Is it actually?

Yes, the extra data being the length that is located in memory before the pointer.

> So suppose I pass this "fully compatible with normal pointers" pointer to a > function.. and it ends up being stored in a register. Now where is this > location "just before the pointer"? In the previous register?

You're at the wrong level of indirection; the header in in the memory before what the pointer points to, as in p - 1 rather &p - 1.

> I think it is describing arrays with a prepended length header.

Correct.

> Call it a fat pointee if you want a fancy name; at least that acknowledges > the onus is on the pointee to have the right metadata in the right place.

I would prefer that this was not called a "fat pointer", but the claim made above is that a pointer with any implicit data associated with it is "fat".

> There's nothing in the pointer here.

No data, but there is an implicit guarantee that it points to the data portion of a length-prepended array.

<comment><USER18> > Yes, the extra data being the length that is located in memory before the > pointer.

> You're at the wrong level of indirection; the header in in the memory before > what the pointer points to, as in p - 1 rather &p - 1.

You are contradicting yourself. &p - 1 is before the pointer. p - 1 is before the pointee.

> I would prefer that this was not called a "fat pointer", but the claim made > above is that a pointer with any implicit data associated with it is "fat".

> No data, but there is an implicit guarantee that it points to the data > portion of a length-prepended array.

I think that's a borderline useless definition. In my C, there's an implicit guarantee that any pointer, in a context where it may be dereferenced, points to a valid object (as long as it's not NULL and there's no programming error causing it to point at nothing well defined). Usually it points at the start of a struct, sometimes it points at list node (or whatever) embedded in a struct and I might have to work my way back with offsetof. Either virtually all of my pointers are fat or none of them are. I go by that none of them are, because whatever data I have (implicit or explicit) is not a property of the pointers I use but of the data I point them to.

</comment></comment></comment></comment></comment></comment><comment><USER20> I understand their desire to use a library, but there's a faster and safer way to do this that's more C-like if you have access to the compiler:

Just locate anything declared as an array in a particular linker section so the pointer manipulation can be done with two (or one if it's at the top of memory) comparison, possibly even to a constant.

If you do this you can even forbid pointer arithmetic except in actual []-declared memory, and can do transparent bounds checking (&array-1 can hold the array length or, possibly faster, the address of the location after the end of the array).

An advantage of this over the library route is you can prevent pointer/array punning but otherwise allow any C program to work fine. And apart from a few corner cases (there are legit non-array uses of pointer arithmetic, though very few) and noncompliant program can be changed to use [] and still work perfectly fine without this option being used.

</comment><comment><USER21> "This proposal wasn't accepted into the C standard..."

Walter often shows up on HN, so I'll ask: was this proposal merely on the Dr Dobbs article or did it actually go to a committee for review? If the latter, why wasn't it accepted?

Should C reconsider this? Especially now that C++ has std::span<> and std::string_view<>?

<comment><USER22> Currently Checked C seems to be only attempt left, and a mentality shift to at very least use the static analysis tools that come with the compilers.

Contrary to common HN wisdom, most C and C++ related surveys show that only up to 50% actually use some kind of analysis tooling.

</comment></comment><comment><USER19> Also see: BSTR

[https://docs.microsoft.com/en-us/previous- versions/windows/d...](https://docs.microsoft.com/en-us/previous- versions/windows/desktop/automat/bstr)

<comment><USER23> Because you can never have enough string types in your Windows projects.

<comment><USER24> Or enough wrappers to safely acquire and free those things in C++ without having to write many extra lines or spend a long time looking for leaks

</comment></comment></comment><comment><USER15> Previous discussion: [https://news.ycombinator.com/item?id=10526159](https://news.ycombinator.com/item?id=10526159)

(Fixed to actually point at discussion of fat pointers; thanks, 'dgellow.)

<comment><AUTHOR> The link I posted is the documentation about fat pointers, not the homepage of libcello.

<comment><USER15> Fair enough; I'll replace the links above with [https://news.ycombinator.com/item?id=10526159](https://news.ycombinator.com/item?id=10526159), which as far as I can tell points to the same page. Thanks!

</comment></comment></comment><comment> seventh-chord This seems to assume you want to actually allocate individual objects using malloc/calloc, when in reality you usually want to pool your allocations somehow, both for performance and for your own sanity.

</comment><comment><USER25> Might be a nice safety feature to tag the first few bits of the size with a magic sequence so the "free" method can sometimes catch an attempt to free a non-fat pointer passed into it.

</comment><comment><USER26> Another question for language lawyers: if you are given a pointer to char, is it defined behavior to somehow cast the data before that into a correct integer? Assuming you have

       typedef struct MyStr {
          size_t mystr_length;
          char [1];
       } MyStr;

       typedef char *PMyStr;

       size_t MyStrLen(const PMyStr p) {
          const MyStr *pmystr = ?;
          const size_t *psize_t = ?;

          return *psize_t;
       }

How do you make a legal cast to get either pmystr or psize_t?

<comment><USER15> That depends on what p points to. If it's the address of the char[1] array inside a valid PMyStr, then as far as I'm aware you can subtract the correct number of bytes (offsetof is your friend) and cast the resulting pointer to your MyStr, then get the size from there.

<comment><USER26> Ah, with your information I'm looking into the container_of macro which I'm not entirely certain is legitimate but hasn't run into problems.

</comment></comment></comment><comment><USER26> Many string functions are intended to operate on a substring. These functions would appear to need the original string passed in every time to find the length.

<comment><USER27> Every pointer into an array could pass the triple _{current, start, len}_ , or even maybe the quadruple _{current, start, len, allocated}_ , which is the kind of minimal efficiency C programmers are looking for.

</comment></comment><comment><USER28> They use a similar trick with std::string in libstdc++. The string object has a pointer to the null terminated character string, right before that string is a structure that contains the reference count. (I think libcello could add reference counted objects that way)

</comment><comment><USER29> From a cursory look, this is one of the most unsafe pieces of code I have seen, with complete disregard to memory alignment requirements and the lifetime of temporary objects passed as arguments to functions.

Definitely don't use this in production code.

<comment><USER30> From the FAQ[0]

> Can it be used in Production?

> It might be better to try Cello out on a hobby project first. Cello does aim > to be production ready, but because it is a hack it has its fair share of > oddities and pitfalls, and if you are working in a team, or to a deadline, > there is much better tooling, support and community for languages such as > C++.

[0]: [http://libcello.org/home](http://libcello.org/home)

<comment><USER7> The author has already made it clear they consider "memory allocation by unaligned offset into temporary char[] cast into a struct pointer" to be a valid strategy, so frankly I'm not very interested in their opinions on whether it's production ready.

I've seen it on HN before, whenever this project gets mentioned. People who don't know much about C confuse it for a really cool thing you can do with C, as if it's just another legit library that you can pick up and use. It's a lot of undefined behavior. People have enough problems writing safe C as it is, and on top of this complaint about alignment and misuse of temporaries, this thing makes the problem worse in other ways too, removing the few safeguards that exist by treating everything as void* for instance.</comment></comment></comment>


Show HN: vpnns – Per-app VPN using Linux namespaces - <AUTHOR> https://github.com/cernekee/ocproxy#vpnns-experimental <comment><USER2> This is something which I can't belive wasn't already supported. It seems so obvious. Most of the time, when I use a VPN, I only want certain applications to use it, and others not.

Or, the same for Tor. I ended up using Transparent Proxy and creating a separate linux user for Tor browsing, mostly because iptables stopped being able to filter by PID. Also, it provides a bit more encapsulation. (Note, in my use case I was not too much concerned about being RE-identifyable, rather that my "ISP" and anyone nefarious on my WiFi should not see my traffic, and that I appear anonymous to websites. Most users should use something like TorBrowser to minimize their uniqueness!)

Long disgression, but I really wish you could right click on the title bar of any app, and say "route this through VPN / TOR / over cable". I never got around to implement a prototype unfortunately.

<comment><USER3> Yes, but the implementations I saw either used LD_PRELOAD of iptables -m owner. Using cgroups looks like a much nicer way.

</comment></comment><comment><USER4> Seems obvious after you see it, mark of a truly genius idea. Great work. I cant wait until this is mainlined into Ubuntu or something.

<comment><AUTHOR> Once I push a new ocproxy release, vpnns will percolate into the Debian package: [https://packages.debian.org/sid/ocproxy](https://packages.debian.org/sid/ocproxy)

</comment></comment><comment><USER5> This is great, I'm really looking forward to playing along with this when time permits. I've been using Alpine docker containers + openvpn + squid, but it's not the most stable solution. Plus openvpn is a complex PITA to use (at least for me).</comment>


Sorting Your Twitter Followers by Popularity (using JavaScript) - <AUTHOR> http://robgibbons.com/media/followerPopularity/

<comment><AUTHOR> I really made this to scratch an itch, as I have a Twitter account with 9k+ followers and I wanted to see who the most popular followers were. You can't really do that with Twitter, so I learned a bit about their API and built this service myself.</comment>


Algorithm Solves Graph Isomorphism in Record Time (2015) - <AUTHOR> https://www.quantamagazine.org/algorithm-solves-graph-isomorphism-in-record-time-20151214/ <comment><USER2> Here's the updated paper with some significant corrections [http://people.cs.uchicago.edu/~laci/upcc- fix.pdf](http://people.cs.uchicago.edu/~laci/upcc-fix.pdf)

<comment><USER3> Those are just the corrections, not the full paper.

<comment><USER4> This is the full paper the OP was referring to: [http://people.cs.uchicago.edu/~laci/17groups/version2.1.pdf](http://people.cs.uchicago.edu/~laci/17groups/version2.1.pdf)

The fixes linked above are mentioned in the WARNING below the abstract and explained at 18.1.

</comment></comment></comment><comment><USER5> Slightly off-topic, this article has one of the best expositions I've seen (the Arthur-Merlin stuff) of P, NP, NP-complete, witnesses, and the idea of "evidence" for what complexity class the problem is in.

<comment><USER6> Unfortunately I found all of the story-like exposition infuriating: they never even say what the new conjectured lower bound is.

<comment><USER7> Likely it's the lower bound (of something) that an important conjecture in the paper gives? That is, it's not some new kind of a lower bound, but a way to refer to it?

</comment></comment></comment><comment><USER8> "That would be the most interesting possibility, Trevisan said, since it would make graph isomorphism the first natural problem to have a quasi-polynomial algorithm but no polynomial algorithm. "

"Proving it would amount to solving the P versus NP problem, since it would mean that graph isomorphism separates the problems in P from the NP-complete problems."

(exp((log n)^O(1)))

Interesting that there could be a much more varied landscape than was previously thought.

<comment><USER9> One thing that's bothered me ever since I heard about complexity theory is why there is no explanation given as to why polynomial and exponential complexities are given such special status. It's just taken for granted that polynomials and exponential are the 2 classes you care about... with no explanation as to why the rest aren't interesting.

EDIT: Edited to clarify, sorry the question was unclear. Thanks for all the replies, but yes, I know the basics :-) I'm not asking about why polynomial complexities themselves are special. Rather, I was trying to ask why we only ever talk about _polynomials and exponentials_. The only thing that's ever justified is polynomial-time _reducibility_ within a class (which also inherently defines the P class itself). That part is fine. But then once that's established, people suddenly start talking about exponentials. Sorry, but what just happened? There are an infinite number of complexities in between to consider! Why should I suddenly not care about everything in between? Why am I not taught a _single_ algorithm whose complexity is between these? is my question.

<comment><USER10> Well in mathematics there is a somewhat similar phenomenon concerning the growth of finitely generated groups. If you have a finitely generated group, pick a generating set as an alphabet and a natural n, then ask how many elements of the group you can build by using only words of length at most n. This is the growth function, whose growth does not change if you pick another generating set.

See [1] the Tits alternative which implies that if your group is linear (huh) then the growth function is either polynomial or exponential. Then there is a result by Gromov [2] which says that if the growth is polynomial then the group is virtually nilpotent. There was a long standing question if there exist finitely generated groups with an intermediate growth and the first example was found [3].

So it's not interesting until it becomes interesting due to a newly discovered fact.

[1] [https://en.wikipedia.org/wiki/Tits_alternative](https://en.wikipedia.org/wiki/Tits_alternative)

[2] [https://en.wikipedia.org/wiki/Gromov%27s_theorem_on_groups_o...](https://en.wikipedia.org/wiki/Gromov%27s_theorem_on_groups_of_polynomial_growth)

[3] [https://en.wikipedia.org/wiki/Grigorchuk_group](https://en.wikipedia.org/wiki/Grigorchuk_group)

<comment><USER9> Wow that's definitely interesting :-) thanks for sharing this! Makes you wonder if there's a deeper connection between these two areas!

</comment></comment></comment></comment><comment><USER11> _even small graphs can be made to look very different just by moving their nodes around_

I'm curious: for the graph isomorphism problem, how are the graphs defined then? It seems there must be a geometric component in there because if they were defined just in terms of abstract sets of vertices and edges between them, it would be rather trivial to show if two graphs are the same, wouldn't it?

<comment><USER12> Graphs are given as abstract sets of vertices and sets of edges (as adjacency lists or matrices). The problem is that corresponding vertices in the two graphs may not have the same label, so to solve graph isomorphism naively, one would need to try all n! possible bijections between the vertex sets (where n is the number of vertices).

<comment><USER13> Wikipedia has a nice picture of two graphs that are isomorph: [https://en.wikipedia.org/wiki/Graph_isomorphism](https://en.wikipedia.org/wiki/Graph_isomorphism).

It should be noted than one of the common used algorithms, called VF2, and descendants of it basically just check all possible bijections, although they use some heuristics to prune the search tree.

<comment><USER14> I think pictorial representations are what is confusing the grandparent. The problem isn't about comparing two graphs that look different visually, it's about comparing two graphs which have differently named labels (but might still have the same structure). The graph isomorphism algorithm doesn't know anything about where on the page you drew the nodes, just what is connected to what

</comment></comment></comment></comment><comment><USER15> Explanation of the current state of review: [https://cstheory.stackexchange.com/a/40373](https://cstheory.stackexchange.com/a/40373)

</comment><comment><USER16> "Monads are monoids in the category of endofunctors. What's the problem?" \-- that's how every paper looks at me!

</comment><comment><USER17> Is the algo practical, implemented by say major generic graph libraries?

<comment><USER13> No, graph libraries usually relay on some variation of VF2 and maybe (but I haven't checked) in some cases on the more complicated nauty/traces algorithms.

<comment><USER18> (basically speaking) no-one does Graph Isomorphism with VF2, they use (as you say) Nauty, or some system which derives from that.

Things like VF2 can be used for sub-graph isomorphism (and various variants of that problem). Nauty (and this paper) are useless for solving that problem.

I work in this area, and it's interesting (to me at least) that you would imagine the algorithms for graph isomorphism and sub-graph isomorphism would be quite similar and they really aren't.

<comment><USER13> Still, the more well known graph libraries seem to avoid Nauty, probably because it is not that easy to implement. From a quick search: \- NetworkX: VF2 \- JGraphT: VF2 and some newer algorithm called "color refinement algorithm" \- seems to resemble Nauty \- BGL: VF2 \- SnapGraph: Does not seem to contain graph isomporphism \- Igaph: Bliss and VF2

So are you convinced that Nauty is still the best thing that we have nowadays? I was looking at the paper last summer with the intention to implement it for Julia, but then got sidetracked with other stuff. Maybe I should resume that.

<comment><USER18> Nauty is I think the best trade-off. The improvements in Saucy and Bliss (which have mostly been added to Nauty now) are useful.

The basic algorithm isn't that hard -- I have a few implementations lying around (nothing in Julia). Traces is better, but is much harder to implement and it isn't clear the improvement is worth it for someone implementing from scratch.

There are some improvements, particularly for graph isomorphism (as opposed to automorphism), but the basic algorithm framework is the same.

</comment></comment></comment></comment></comment><comment><USER19> (2017). This is old news, I thought for a moment he had a new paper. Please add the year.

<comment><AUTHOR> OK

</comment></comment><comment><USER20> (2015)

<comment><USER2> Technically 2017 was the latest update to the paper, but still not as groundbreaking as the submission title.</comment></comment>