Show HN: Simperium, a realtime data layer - <AUTHOR> http://simperium.com

<comment><USER3> This is interesting. We built something very much like this internally at SeatMe. On the iOS side, it's almost identical. It's how we keep our iPads up to date. Might not of built something internally ourselves a year ago if there was a platform like this.

The biggest difference is that since we are not a general platform, we can make assumptions about the model and how version each release and we can built in some constrains and unique security models.

We took a lot of cues from the OData spec and Microsoft's reference design.

The rest of this comment is mostly targeted at the creators of Simperium.

You made very similar design decisions to us in a lot of ways. A lot problems though that you will face I see with your path here so I have a few tips for you.

* It sucks the iOS client isn't open source. I get scared of linking in third party libs into iOS projects because I have to account for anything you do when I go to Apple to submit my app.

* You really got to brush up on the objective-c naming conventions. Not to be harsh but `-(id)getCustomObjectForKey:(NSString * )key;` makes me cringe.

* Don't require me to have to know about your categories.

* Namespace your categories so you don't smash mine ("something like "SP_encodeBase64WithString" instead of "encodeBase64WithString")

* If you include third party libs, you MUST rename them and prefix with your prefix. I see you use ASIHTTPRequest, DDLogger, SocketIoClient, AsyncSocket, Reachability, and a few others. You will smash everyone else's implementations if they already had them included.

* Don't use ASIHTTPRequest internally (it's old and unmaintained and doesn't play nicely with ARC)

* PREFIX ALL YOUR CODE. We don't have any real name-spacing objective-c. As a framework developer, you have to be aware of that more than anyone else. I shouldn't be seeing DiffMatchPatch and SocketIoClient show up in my symbol list after linking your lib

* Your addDelegate/removeDelegate is funny. After you exhausted the need for one delegate, switch to NSNotificationCenter.

* DON'T USE XIB/NIBs. Interface builder for iOS was an after thought and it's only a 90% solution (unlike with Cocoa where interface builder was a first class product built side by side with AppKit). Especially don't make me have to include your XIBs in my bundle. At the very least give me a bundle with it in it.

* Separate your GIT repo. If I want to include your library as a submodule I have to take all the client libraries as well.

Now when it comes to the actual sync layer and how you generate JSON dictionaries and apply "patches" this is fine code.

Here are some feature requests:

* Instead of having to give you a single NSManagedObjectContext let us register them. We have a few (some use different concurrency types).

* Let us override what gets generated or if we want to ignore a field with userInfo keys in the core data model.

* Let us get an idea of your backend sync processes to be able to suspend and start them when we need and know if anything is pending. Give us a callback that you still have things to queued and when we are done so we can at our leisure set up UIBackgroundTasks on iOS 4+.

On an unrelated note, why not create this as a NSIncrementalStore and just put your code behind the persistent store coordinator instead of monitoring it? We are doing the same as you at the high level because we wrote our code pre iOS 5.0 but iOS 5 gives you an awesome new toy there.

<comment><USER4> Any argument against using CocoaPods for managing ASIHTTPRequest, Socket.IO, etc.?

Renaming seems like an ugly solution.

<comment><USER3> As a library library writer, if you are going to include dependencies, you have to rename pretty much everything.

Objective-C is dynamic and the last implementation wins if they are named the same so it's possible for third party libs to suddenly smash your implementation.

</comment></comment></comment><comment><USER5> _"With Simperium, you own your data - that's important!"_

Not to take away from some of the great projects featured on HN recently, but statements by Simperium like this make me much happier as a developer and business owner: _"We believe the best apps have both a great user experience and unique backend services."_.

From <https://simperium.com/overview/>: _It is not a backend-as-a-service. We believe the best apps have both a great user experience and unique backend services. Our focus is to provide a great data layer between your frontend and backend while integrating with other providers of tools, hosting, and services._

<comment><USER6> This is definitely a cool and interesting service, sure, but how exactly do you "own your data"?

It's sitting in Simperium's database and you're locked into their platform once you decide to use it. Maybe there's an export, but that's irrelevant because all your apps in the wild and all your users are relying on this service to move data around.

If Simperium gets hacked, you get hacked. If Simperium goes down, you go down. If AWS goes down, Simperium goes down, and then you go down.

Say your app gets huge, and Simperium can't scale (and really, "half a million users" is nothing for a free service; that's not any meaningful scale and not a number that can really be cited as a signifier of ability to scale) -- what happens then?

Say you outgrow Simperium, and it's now costing you too much. What happens then?

The above isn't really about Simperium, but really about all backend-as-a- service services. They're great for starting out, but if you actually have a successful product you're going to need to be running your own backend at some point.

If you get too big/successful/etc and things start dying and you end up like Friendster before Facebook decides to buy your company, you're going to look back and wish you built your own backend and knew what was going on behind the scenes.

<comment><USER7> The code gets open sourced, and you allow "syncing" to a local database, where everything is accessible.

I went through all of these business decisions for HubSync in late 2009, and came to the conclusion that you had to do the following to make "owning your own data" a reality:

1\. Cannot charge developers or earn money on the actual data itself in any way. (Google will not be competing in this space.)

2\. MUST, MUST, MUST allow removing any data from the cloud AND reloading it later, without losing any features/abilities.

3\. USERS need to pay for cloud sync storage/service, NOT app developers.

4\. Code must be open source, visible, and peer reviewed, just like a crypto algorithm.

If Simperium will do all of these, IMO they can raise money, and achieve a Dropbox-style exit. There's no doubt the potential is there.

2 is the most important. If you cannot remove your data from Simperium's backend _and then reload it again later_ , either to their own backend, or a competitor, you have vendor lock in of the data, and you don't own your data.

3 means you have to have to do something like Dropbox, where you give away a certain amount of storage/sync for free. Ultimately, this is why I didn't pursue HubSync further, even though the products are nearly identical (even down to syncing CoreData on iOS with a web app running in the browser).

(In a nutshell, I didn't want to get funding and devote years of my life to data sync, when I'm in the middle of working on building a live action Pixar. But it's a great idea, and I really hope Simperium or someone else makes it happen.)

</comment></comment></comment><comment><USER4> This one is the best so far:

1\. Works with any JavaScript library

2\. You can _write server-side logic_

3\. Operational Transformation baked in

Seriously I cannot find anything bad about this project. (I really wanted to.)

My questions:

1\. Is the server side written in Python?

2\. Is there offline support for JavaScript apps too?

(I tried this with my own library and I think it was a wrong direction -- it only made it more complicated than it should be)

3\. How about relationships in JavaScript objects? How does that work? (I suppose it's all automatic in Core Data objects, but how about JavaScript?)

Thanks!

<comment><USER8> 1\. Server side is all python (combination of tornado/gevent/zeromq)

2\. We've experimented with it, we'll likely add it as an option in the future since there may be security concerns depending on the app.

<comment><USER4> Thanks! Can you please tell me something about the number 3. Sorry just edited it!

</comment></comment></comment><comment><USER9> We desperately need a high quality open-source Firebase/Simperium/Spire. A simple, robust, realtime data synchronization layer with libraries for major platforms. These projects all look great, but outsourcing this infrastructure is simply not feasible for lots of projects.

<comment><USER10> Are there any operational transform libs that are transport independant? Socket.io and socketstrwam can handle communication.

<comment><USER9> ShareJS is a good example. However I don't think this is the right approach necessarily. The FireBase API is a lot simpler and in casual tests it's much faster too.

</comment></comment></comment><comment><USER11> _Pricing is free during the beta period._

Simperium team: can you give any guidance on when you'll announce pricing, or roughly what pricing is going to be?

<comment><AUTHOR> Hey Brian, we know this is really important. Our beta label is mostly for the lack of pricing since the technology itself is production-ready. We're looking for more feedback and we aim to announce pricing soon.

What do you think about Urban Airship's model based on active monthly users? What we like is that the costs are obvious and map clearly to your business.

<comment><USER11> MAU doesn't seem like a good model to me; it's not very well connected to your costs or customers' usage so it creates weird incentives on both sides. And it makes it expensive for an existing product with a large userbase to try out a small simperium-based feature.

You guys are a hosting/infrastructure service, and it's probably for good reason that such services have historically charged based on usage. For you that could be something fairly raw like "GB transferred, GB stored" or something a little more abstract like "pushes" or "versions".

<comment><USER12> I'll second this. It makes you more comfortable as a user to know that if the incentives aren't misaligned your pricing will be more stable and easier for you to build a business on. I think its no surprise that AWS can keep making its services cheaper and App Engine had to change its pricing model entirely.

<comment><USER13> What were the misaligned incentives in the GAE model?

</comment></comment></comment></comment></comment><comment><USER14> Thought it would be relevant here, I quickly hacked up a Ruby client: <http://pheuter.github.com/Simperuby/>

<comment><AUTHOR> Whoa. Nice work, that was fast.

</comment></comment><comment><USER15> This looks incredible. My one burning question: For integrations on mobile devices, how does this affect the battery life? I'm concerned that there is some open-ended connection from the client to server listening for data changes which will drain the battery while the app is open.

<comment><AUTHOR> Good question. We haven't heard any complaints about battery life so far. If you'd like you can check out Simplenote to judge for yourself: <http://simplenoteapp.com>

You do have control over the granularity of your updates. For example, we spoke to a game developer who would want to disable Simperium while a game is being played, and then enable it again at the end of levels. These coarse changes are supported since they'll resolve automatically when the client comes back online.

</comment></comment><comment><USER16> Absolutely priceless! This is an amazing product, and I can not start to imagine what developers are going to be able to do with this. Very exciting!

<comment><USER17> I'm missing something. This is just MVC style observer pattern. I get that it is well packaged/productized, but this is the kind of thing that developers have been doing for decades, with standard libraries for most of that time. Doing it with a browser is a bit newer, but ever since WebSockets it has been common place.

A nice product yes, but I wouldn't expect this changes what developers are going to be able to do.

<comment><USER18> No, it is much more than that. You should watch the video, it demonstrates the platform solving some very hard problems with very little code.

The key here is (I believe) operational transformation (<http://en.wikipedia.org/wiki/Operational_transformation>), the algorithm behind products like Etherpad and Google Wave.

The observer pattern only handles the case where you have one client accessing the datastore. OT expands those ideas to handling multiple clients. Sync is a really hard problem, which is why almost nobody can get it right. Simperium seems to come closer than most and is packaged in a nice service.

It's really exciting to imagine the possibilities.

<comment><USER17> I did watch the video, but used poor semantics. :-(

Fair enough that it is multiple observer context and I didn't spell it out. It's essentially locking in an OT consistency model/control algo. I'll give you that sync is a really hard problem, but Etherpad, Wave, Google Docs, etc. are just a few examples of some off the shelf solutions (the Wikipedia page details more and doesn't even cover the whole set of what is out there, particularly if you consider version control systems). The big commercial success for generic document sync was probably Lotus Notes, and they made the sync solution a separate product product.

Again, not knocking the product, it looks like a quality solution and one I might even recommend, but I'm not sure I grok how one sees this as opening up a new set of possibilities.

</comment></comment></comment></comment><comment><USER19> This almost seems unreal. What kind of latency would you be dealing with when your application(s) deal with many concurrent connections? For example, say a cross-platform multiplayer game with a persistent, shared world?

<comment><AUTHOR> Latency is good right now (low hundreds or less), but for launch we focused on data integrity and reliability.

That leaves room for improvement, including some low hanging fruit like eventually giving you the ability to disable versioning for certain kinds of data (like multiplayer updates).

</comment></comment><comment><USER20> The website mentions that you are using google-diff-match-patch in the JavaScript client to merge changes, it seems as the API does not require the developer to actually specify things like "the user added an A at this position" as opposed to just "commit the changes to this entire object". Is there a reason other than simplicity for this API (I guess maybe because CoreData doesn't have that abstraction, and I presume your timeline was to start with figuring out how to sync CoreData), and on iOS are you also using google-diff-match-patch (there was not the same explicit mention of it in the documentation there)? (edit: Actually, I guess the comment from zbowling about DiffMatchPatch answers the second half of that. ;P)

BTW, this is generally really awesome: I am (right now, as in I'm sitting there right now ;P) helping teach a class on cloud computing at UCSB that happens to currently be discussing database synchronization and replication; after spending a bunch of time today discussing "how PostgreSQL is implemented and the basis of different isolation levels in the SQL standard and in MVCC" I took the time to tell everyone about Simperium (which probably makes more sense if I mention that I've looked into building something similar before for my projects; I'm glad someone else finally seems to be coming at it from the correct mindset). Everyone here seems to agree: this is going in a great direction.

<comment><USER8> Thanks saurik! Yeah that's right, we've tried to make it as easy as we can for developers to use. One of our goals has been to let developers be able to work with data as they do normally if it were just local, while we handle figuring out whats changed.

</comment></comment><comment><USER21> Guys congrats on the launch! I was wondering what was taking you so long ;) Very exciting stuff!

I'm amazed that we're on the same wavelength - we've had to build very similar infrastructure for ourselves for Unipost (www.post.fm). Can't believe we didn't collaborate on this, we'd happily be your first customer :(

A few interesting differences:

\- Our approach is more like Meteor - web only, no iOS support

\- The backend is a python tornado app that handles validation and conflict resolution before saving stuff to dynamodb

\- We have a JS datastore backed by websql/indexeddb/memory that syncs with our backend datastore

\- We have "live" Backbone collections that update themselves when datastore queries return different results

\- We have a Backbone sync adapter that uses the datastore to persist data locally and kick off synchronization

\- We sync a subset of the data (eg 3 months of mail) - thats a core requirement for us

\- We sync all of the tables at once, not per bucket, cause queries are joining tables so the datastore has to be consistent at all times

\- No operational transforms cause it doesn't seem to apply to us - <USER2> "notepad" specific I think

\- No versioning as we didn't see benefits for us

\- We'll probably open source this stuff when we're done

What do you use for storage?

<comment><USER21> Another interesting approach is:

\- CouchDB

\- PouchDB: <https://github.com/daleharvey/pouchdb>

\- Backbone PouchDB Adapter: <https://github.com/jo/backbone-pouchdb>

But the Map-Reduce paradigm didn't seem to fit well onto our data model.

<comment><AUTHOR> Our goal on the client-side (compared to CouchDB) was simpler, more lightweight libraries that work well with existing tools that developers already use, like Core Data on iOS and sqlite/ContentProviders on Android.

Whereas CouchDB does master-master replication among instances of itself, Simperium can accomplish something similar with any database: e.g. sqlite on iOS, to MongoDB in our backend, and to whatever database you use in your own backend.

<comment><USER21> Just strange that you don't support offline for web apps, only iOS & Android, even though web apps can use websql.

That's probably cause you're used to writing native iOS apps, but we're attempting a web app that works via phonegap across all devices from one source code base.

So for us storing all the data in Backbone collections isn't an option - models add too much memory overhead, versus native JSON objects.

As I understand multiple collections update independently of each other? How do you deal with relational data in that case?

<comment><AUTHOR> Consistent browser support for storage (besides localStorage) is still quite frustrating. It's improving though and we're on it.

You can model relational data in Backbone quite effectively, or use another library (or your own). For example: <https://github.com/PaulUithol/Backbone- relational/>

</comment></comment></comment></comment></comment><comment><AUTHOR> Simperium dev here. We're launching our beta to gather feedback. Let us know what you think.

<comment><USER22> Jaw-dropping.

Perhaps it's too late to predict that 2012 will be the year of realtime interaction, but between this, Meteor, Firebase, etc., not only are all the tools converging in that direction, but they all appear to be drop-dead easy to use.

Thank you for this. Is there an IRC channel or Google Group for questions?

<comment><USER23> Don't forget about <https://github.com/socketstream/socketstream> which expects 0.3 to be dropped later this month. Very extensible.

</comment></comment></comment><comment><USER24> I've been a SimpleNote user for a while, great to see the platform layer finally coming out.

I have a few questions:

\- As a potential customer, I'm curious as to how sustainable is the company. I understand there was a YC seed investment in '10, but is the company well funded for the next couple of years at least?

\- The platform seems to be ahead of Meteor, Firebase, etc, in that it already seems to have implemented a basic login and security model based on expiring tokens, but from the docs it seems like "finer grained control of these permissions are under development". Does this mean that presently any user can erase/modify data from any bucket, such as global data, data from other users, etc? If so, that's a big deterred for me.

\- Are there any plans to allow querying the data? Key-value stores are fine for simple games, to-do lists, etc, but for any non-trivial app querying arbitrary fields is a basic requirement.

Thanks again, those are some great strides in the right direction.

<comment><AUTHOR> By "finer grained control" we mean control over expirations and read/write permissions for a particular token. It's definitely not possible to erase/modify data from other users.

An example of where read-only permissions are useful is the live dashboard you see at simperium.com after you sign in. The "number of syncs" and alerts at the top are all pulled live from Simperium, but the token used on that page is a read-only token. We just need to expose the ability to create these read- only tokens to developers.

Actually, as a Simplenote user, you might be interested to know that our alerts and blog posts are pulled from Simplenote via Simperium. When we tag a note as "Alert" or "Published" it instantly appears on the dashboard.

Regarding querying, we're working on something for apps that can't or don't want to keep all data locally. In the meantime you can locally query however you'd like in your database of choice.

<comment><USER25> I'm dying to use this, but the ability to create read-only tokens is important for all the applications I have in mind. Do you have a sense of where this is in your priority queue, or even when you might be able to expose an HTTP endpoint?

<comment><AUTHOR> Yes, support for this is imminent. But I believe you know that now since we're already in touch with you.

</comment></comment></comment></comment><comment><USER20> One thing that came up in the discussion about some other related offerings (such as Firebase) is the question: "what is your durability story"? When I'm spending the time to store things for my company, I make certain to ask such questions as "is this data important enough to keep on multiple servers, or is having a backup sufficient".

When storing data in services provided and managed by other companies, it becomes more difficult to be certain that the risk tradeoffs are reasonable, as I'm in essence trusting that you aren't just storing the data on a single server in RAM on memcache or something ;P. Some more details would thereby be much appreciated.

(I understand, btw, that I can also have a server keeping a mirror of all of my data, which is definitely awesome, but I also would then want to get a better understanding of whether I would be a fool not to be doing that, or whether I can feel comfortable with having data stored on your servers ;P.)

<comment><USER26> Important question saurik! Our primary store is mongo. All data is replicated in different availability zones and at least one additional replica which is kept in a different ec2 region. A running snapshot backup is also taken at least every 30 minutes. Simperium isn't a messaging service. Every change is persisted and versioned. So for the Simplesmash demo, you could have a slider that would allow you to go back in time and replay movements. A note on versions: they are stored in an RRD like fashion. At first every version is kept, and as they accumulate; older versions are compacted and become less granular. We plan on making what level of granularity is kept an adjustable option in the future.

</comment></comment><comment><USER20> Is every object on a separate timeline for purposes of operational transformation and synchronization? As an example where this would be noticeable, if I make a change to one object, and then a different one, am I guaranteed that everyone will see the first change before the second? Put from the different side, is it possible for me to not have seen changes on one object yet, but end up downloading a fresh copy of a different object that is from a later point in time? Finally, related to that latter stating, is it possible to use the data before it has fully finished downloading, in that I might only initially need a subset of the objects (and is it even Simperium's goal to keep the entire data store synchronized, or just the active subset)?

<comment><USER8> Correct, each object is on a separate timeline for purposes of operational transform. If both clients are connected at the time the changes are made, then it shouldnt be possible to see the second change before the first. We keep an ordered history of all changes to all objects, though for efficiency, if a client that wasn't present when changes were initially made syncs, changes to multiple objects may appear in a different order. For example, a modification to object A, then object B, then object A again, would appear in the correct order for all clients when those changes are made. If a different client then syncs and asks for what's changed, we may group the two changes to object A such that there would appear to be two changes, a change to B then a change to A.

There is currently limited support for keeping a subset of data synchronized (we have plans to improve this) - if you're using the client libraries in an app, we've focused on supporting the common case of keeping all the data per bucket for a user synced. In the case of keeping all data mirrored to a backend, we provide an endpoint per bucket that you can listen to all changes for all users so you can keep the entire data store synchronized.

<comment><USER20> Ok, with those semantics (and I think I understand the basis implementations that would cause them), do you perform the "resync" step (which has the property that you can at least get the catch-up changes in a single bulk and hopefully fast download for the entire bucket, even with the internal reordering that happened over time) as a single transaction (by locking the persistant store coordinator); and if not, why?

With keeping the objects on separate timelines, do you or can you run into any problems with CoreData inverse relationships? I'd be concerned that my objects would have issues where a change got synced for object A that had it linking to object B, but somehow my application crashed before it got the synchronization for object B sent. I can imagine various reasons why this concern is stupid, including "saurik doesn't understand how CoreData works" (as I only even learned of its existence a couple days ago).

<comment><AUTHOR> There's currently no explicit lock or transaction, but due to the way we're multithreading, multiple changes are propagated back to your context in one shot.

Core Data relationships (including their inverses) that come over the wire can be lazily resolved by Simperium if necessary.

</comment></comment></comment></comment><comment><USER27> Excellent product and superb demo.

Question that may be a bit off-topic but one I'd really like to know: what editor are you using there when you're editing the python service? Thanks and congratulations on your launch.

<comment><AUTHOR> Thanks, appreciated.

The editor is Sublime Text 2: <http://www.sublimetext.com/2>

</comment></comment><comment><USER28> Something I don't understand: when your fundamental value proposition is moving data between platforms ("everywhere it's needed"), why didn't you consider your MVP to include an Android library?

<comment><AUTHOR> We agree Android is important, but we found enough developers who thought iOS/OSX, JavaScript and Python were sufficiently compelling, so we focused on those for launch.

Releasing more libraries is a priority.

<comment><USER28> What I find really odd is that you have no statement of such on your web page. I search the entire thing and I can't tell if you ever plan to release an Android library. As a developer it shapes my perception of you as not a cross platform play but an iOS play and immediately rules you out for just about anything I would do. If I was you I would at very least have a note in your libraries section that Android is on your agenda.

<comment><USER20> I generally presume that any project has all kinds of things on their agenda, and that they don't need to explicitly state all of them. I often find users e-mailing me asking me "do you intend to do X", and I find myself wanting to reply "do you really consider me so closed minded as to not want to do X? I just haven't done it yet". What really matters is what is already done and what can be relied on: even if they said "we intend to build an Android library" on their site, it doesn't necessarily mean that they will ever get around to it (or that they won't decide it is even a bad idea, not that I think that would happen in this case).

</comment></comment></comment></comment><comment><USER29> That is awesome and looks 300 times easier to implement than iCloud! I wish inClass had Simperium to sync across any device and the web.

</comment><comment><USER14> Looks awesome, quick question, is it possible to mutate data from a javascript client? In the API reference, all I see for javascript are event listeners. Also, how would a javascript client receive an access token? <https://auth.simperium.com/> does not allow cross origin requests.

<comment><USER8> Yes definitely, you can update directly using the bucket.update() method, and pass in an id and the data object. The better way would be to implement the 'local' callback to return the data, then just call bucket.update(id), and the library will call your 'local' method to retrieve the current state for that id. You can read about the 'local' callback here: <https://simperium.com/docs/reference/js/#local>

For auth, we'll be adding cross-origin support for <https://auth.simperium.com>, up till now we've been focused on supporting apps with existing backends which generally use the HTTP API to generate auth tokens from their server.

<comment><USER14> perfect, thanks!

<comment><USER8> We just added support for CORS on both /authorize/ and /create/, let us know if you have any issues

<comment><USER14> sweet, will do

</comment></comment></comment></comment></comment><comment><USER30> Are there any recommendations for the maximum size of a data set used with Simperium? It looks like the perfect solution for a back office app for a small business, but I'd like to know if I will be running into a brick wall at some point.

Clients will be iOS and Mac OS X only, using Core Data.

Thanks.

Brad

</comment><comment><USER31> Mildly depressing that this is all proprietary.

I'd much much rather use something that's a hybrid of CouchDB and Unhosted...

It's definitely the future to build syncing as the core operation though.

<comment><USER32> I'm working on something similar that I want to open source. Is anyone else interested in building an open source platform to do this sort of thing?

</comment></comment><comment><USER33> How many concurrent users will this support? I have a "school" database which could potentially benefit, but might have upwards of 500+ users adding and manipulating data at the same time.

</comment><comment><USER34> What is the brown text editor used in the video? Is it only on Mac?

<comment><USER35> at 2:52? Sublime Text.

<comment><USER34> Thanks. :)

</comment></comment></comment><comment><USER36> I have no idea what this does from reading the homepage. You need use cases for where your product shines.

<comment><USER22> You should watch the video. They give concrete examples and code demos that are pretty compelling.

The Zelda-like game running across three screens separately and independently in the same game space was particularly inspiring.

</comment></comment><comment><USER37> The github links on the samples page are broken (they have 'examples' instead of 'samples' in the path).

<comment><AUTHOR> Thanks for that, fixed.

</comment></comment><comment><USER38> So, message queue as a service. With conflict handling. Interesting.

</comment><comment><USER39> Please demo this to Evernote. They need to leverage this asap!</comment>


The conceptual origins of Maxwell's equations and gauge theory (2014) [pdf] - <AUTHOR> http://www.physics.umd.edu/grt/taj/675e/OriginsofMaxwellandGauge.pdf <comment><USER2> I once took an advanced seminar course on the mathematical foundations of electrodynamics in parallel to my theoretical electrodynamics course during my third bachelor semester. I did not have any clue about differential geometry and did not understand the advanced formalism the lecturer introduced in the seminar. But I was quite shocked how easily Maxwells equations can be derived and how compact the formula was. The article suggests that Gauge theory and fiber bundels are subjects, where math and theoretical physics seem to help each other, which is absolutely facinating!

<comment><AUTHOR> Note the article is by Nobel Laureate C.N. Yang [1] who also worked with James Simons and co-authored what has become known as the "Wu-Yang Dictionary" [2].

[1] Yang-Mills Theory: [https://en.wikipedia.org/wiki/Yang%E2%80%93Mills_theory](https://en.wikipedia.org/wiki/Yang%E2%80%93Mills_theory)

[2] "Wu-Yang Dictionary" [http://www.indiana.edu/~jpac/QCDRef/1970s/Concept%20of%20non...](http://www.indiana.edu/~jpac/QCDRef/1970s/Concept%20of%20nonintegrable%20phase%20factor%20and%20global%20formulation%20of%20gauge%20fields%20-%20Wu,%20Yang%20-%201975.pdf)

 _...The mathematics of these results is in fact well known to the mathematicians in fiber bundle theory. An identification table of terminologies is given in Sec. V. We should emphasize that our interest in this paper does not lie in the beautiful, deep, and general mathematical development in fiber bundle theory. Rather we are concerned with the necessary concepts to describe the physics of gauge theories. It is remarkable that these concepts have already been intensively studied as mathematical constructs._

"Gauge Theory and Inflation: Enlarging the Wu-Yang Dictionary to a unifying Rosetta Stone for Geometry in Application" [https://www.youtube.com/watch?v=h5gnATQMtPg](https://www.youtube.com/watch?v=h5gnATQMtPg)

<comment><USER2> Thank you for the interessting material! I have heard a lecture on the geometric and topological applications to solid-state physics. This is one of the things that excites me a lot that whole areas of physics can be "geometrized".

<comment><USER3> Part of me believes that geometry is really just a subset of human thinking that comes very naturally/quickly to us, and thus we are more successful studying physics through geometry than through less 'algebraic' frameworks. So we made the most advances there just because we are biased toward doing so.

But then I think that we may be so strongly biased toward doing so because there's something fundamentally easy about evolving a brain that comprehends geometry. That information with geometric representations are fundamentally easier to evolve good mental models for than other kinds of information.

</comment></comment></comment></comment><comment><USER4> So I've read that it's enough to know that "there is a U(1) gauge symmetry in the Lagrangian" to derive the Maxwell equations from. Basically U(1) => Maxwell equations.

Is that true? Are there any other assumptions required?

<comment><USER5> It's a great question, but slightly subtle. If you already know the Lagrangian (-1/4 F^{\mu\nu} F_{\mu\nu}) all you need is Noether's theorem. If you want to come up with the Lagrangian you can do so by trying to find an expression with local gauge symmetry (the A field is invariant under addition of the four-grad of a scalar field). I'm afraid I don't have a good reference off the top of my head, but it's not difficult.

<comment><USER6> Is that why those symmetry groups are so often mentioned, because they single out unique theories? As far as I know, but I am not a physicist, gauge symmetries are more of a defect than a feature of field theories. Because we lack theories without redundancies we just declare all states in the mathematical model that are related by a gauge transformation to be equivalent to each other and to correspond to just one physical state. In consequence I always wondered why one would then stress those symmetries but it would of course make sense if they were some kind of unique labels.

<comment><USER5> Yes! The symmetries dictate the physics, gauge and otherwise. Unfortunately there isn't a nice way to see that without a ton of field theory background.

<comment><USER6> Which seems quite strange to me. I assume we would be really happy to find theories equivalent to existing field theories but without the gauge redundancies so that we have a one-to-one mapping between physical states and mathematical states. Why would a part we want to get rid of determine the part we are really interested in?

<comment><USER3> The 'mathematical states' here are descriptions of nature. A symmetry in the description doesn't correspond to identified physical states, it corresponds to a constraint on the description. For instance, if you want to prove that something is unique, you have to show how all the different ways of describing it lead to the same conclusions.

<comment><USER6> _A symmetry in the description doesn 't correspond to identified physical states [...]_

As I said before, I am not a physicist, but I am pretty sure that you are wrong here. Unless you are a physicist and you are sure about that. My understanding is that gauge symmetries are a misnomer and should better be called gauge redundancies because they essentially establish equivalence classes of mathematical states corresponding to identical physical states. This is not the same kind of symmetry as for example the Poincaré symmetry of special relativity.

Let me illustrate my understanding with a silly example that would almost certainly not actually work out mathematically but it should get the point accross. Say all you have in your mathematical toolbox are complex numbers and you come up with a theory of thermodynamics where temperatures are described by complex numbers. Everything works out nicely besides that there is no discernable physical difference between all the temperatures differing only by their imaginary parts. So you declare that there is a gauge symmetry in your theory of thermodynamics and all temperatures only differing by their imaginary parts are actually the same physical temperature. This theory has translations along the imaginary axis as a gauge symmetry.

<comment><USER7> Ah, you're looking for charge I think.

[https://en.wikipedia.org/wiki/Charge_conservation](https://en.wikipedia.org/wiki/Charge_conservation)

The whole thing spins my head around differently every time I try to learn more but I hope I am helping by pointing this out.

"The symmetry that is associated with charge conservation is the global gauge invariance of the electromagnetic field"

<comment><USER6> No, I really meant what I said, gauge symmetries are non-physical redundancies in gauge theories. Gauge fixing is the act of choosing one of the equivalent alternatives for carrying out calculations, gauge transformations allow switching between those alternatives. The first paragraph of the Wikipedia article on gauge fixing [1] actually says pretty much exactly this, unfortunately I just looked at it seconds ago.

 _In the physics of gauge theories, gauge fixing (also called choosing a gauge) denotes a mathematical procedure for coping with redundant degrees of freedom in field variables. By definition, a gauge theory represents each physically distinct configuration of the system as an equivalence class of detailed local field configurations. Any two detailed configurations in the same equivalence class are related by a gauge transformation, equivalent to a shear along unphysical axes in configuration space. Most of the quantitative physical predictions of a gauge theory can only be obtained under a coherent prescription for suppressing or ignoring these unphysical degrees of freedom._

And because you mentioned the Aharonov–Bohm effect in your other comment, further down at the end of the gauge freedom section you can find a statement that the Aharonov-Bohm effect does not enable experimentally distinguishing different gauges either.

 _Not until the advent of quantum field theory could it be said that the potentials themselves are part of the physical configuration of a system. The earliest consequence to be accurately predicted and experimentally verified was the Aharonov–Bohm effect, which has no classical counterpart. Nevertheless, gauge freedom is still true in these theories. For example, the Aharonov–Bohm effect depends on a line integral of A around a closed loop, and this integral is not changed by A → A + ∇ψ._

I also think you are wrong when you state in your other comment that charge conservation is not exact. I am pretty sure it is exact and you are mixing up CPT-symmetry and the violations of its components with the gauge symmetry of QED.

[1] [https://en.wikipedia.org/wiki/Gauge_fixing](https://en.wikipedia.org/wiki/Gauge_fixing)

<comment><USER7> Good point about charge conjugation symmetry vs "charge conservation"\- I was totally mixing up one for the other! Thanks (a lot) for driving that home. For anyone who might follow this - C in CPT - is charge conjugation - the swap of a particle for its anti-particle. Certainly not the same thing as charge conservation itself. Thanks for that correction. I tossed around a lot of stuff I shouldn't have.

Okay, so I think I get you now. I was basically working from the point of view that "gauge matters" because the current 4 vector is conserved only if the E&M Lagrangian is gauge invariant. Hence charge conservation is inextricably linked with gauge symmetry - it "matters" to the derivation of the physical dynamics.

But, yes, to your mathematical description of gauge fixing above... And to not being able to experimentally distinguish gauges. Phase differences, after all, are not gauge differences. Whatever the worth of the Aharonov–Bohm effect, it is not going to matter here. Good point. I get excited about this stuff.

To your question: "Why would a part we want to get rid of determine the part we are really interested in"

I'd answer: Why do we "want to get rid of" it?

We cannot detect which gauge we are in, and we can twice confirm that it doesn't matter which one we are in precisely because current four vector is conserved. The redundant descriptions of the Lagrangian are the way to ensure physical descriptions of reality.

I think this is a matter of what we like or don't like, not physics or math. But hey, sometimes those are clues!

Gauge symmetries are non-physical redundancies... which are essential to our mathematical description of real physics. Maybe they are a clue that there is a better, undiscovered, way to do business?

Maybe.

<comment><USER6> _I 'd answer: Why do we "want to get rid of" it?_

I'd counter with why would you want additional degrees of freedom in your theory that don't correspond to something real? To pick up my silly example, why would you use complex temperatures to do the math if the physical temperature is real and only has one and not two degrees of freedom? I can only think of two sensible scenarios.

The first one is that you have a theory with real temperatures but you can make working with that theory easier if you switch to complex temperatures for the calculations in witch case the redundancies are purely a mathematical tool. The second one is that you suspect that temperatures are actually complex and you just haven't figure out how to detect the imaginary component in experiments.

Neither option seems to be true for gauge theories - as far as I know there are no alternative equivalent theories that don't have gauge symmetries and are just harder to work with mathematically and I also think nobody or at least not many believe that gauge redundancies correspond to yet undiscovered physical things.

<comment><USER7> Okay cool.

I take your point. Maybe you are right, but I'd argue that the first option actually applies to gauge theory in the following way:

First, I have to assert that Lagrangian formulations of standard physics are generally simpler than say, Newtonian form. This is because the equations of motion all come from one principle - least action.

Second, I assert that when working with a Lagrangian, it is impossible to derive all the needed physics without introducing gauge freedom somewhere. For a simple example (that is within range of my brain+sources (see footnote1)) Try it with the magnetic field, B.

There is a constraint on B that it must be everywhere divergence free. The easiest way to ensure this is to write the magnetic field as the curl of something. This introduced redundancy in the description but ensures we do not have to worry about the fact that B is constrained to be divergence free (- now there is no alternative!) Furthermore, there is no way to derive Lorentz's force law from a Lagrangian without adding a gauge freedom term. -end example

This example is from Susskind's "The Thoeretical Minimum," page 211 (for the case of the Lorenz force law) This kind of thing generalizes well to differential forms and comes in handy deriving, e.g. Yang Mills, or making sure some particular theory is "manifestly co-variant" \- that sort of thing. It makes the thing easier, not harder to properly formulate.

One could build the theory without using a Lagrangian (or without anything "math-mechanically equivalent" like a Hamiltonian) but then you loose the simplicity of the dynamics falling out of principle of least action. This use for least action is universal across everything we currently know about fundamental physical theories.

In sum, I assert it makes sense to go with the first "sensible scenario." ...And that it leads directly to gauge redundant theories just like we have. But of course this assertion of what is easier is subjective (see footnote2).

To get even more subjective for a minute, there are these amazing coincidences, such as the fact that GR looks like E&M, and even more like Yang Mills when they are viewed in this "action oriented" way. -A way that basically requires the use of gauge freedom to get to the correct dynamics. Why turn away from that? Well you would if you found something else simpler and more beautiful. To me, it looks like with gauge theory we have something too amazing, and too pretty, to be a coincidence or to ignore. But hey, if you have something up your sleeve to beat it, don't leave us all out!

Anyway, if you see it differently, all well and good. If you think the above stuff is ...just that... then by all means, teach me better!

This has been fun.

Cheers.

(footnote1) Further investigation reveals that there appear to be theorems and classification results proved by Cartan, Weyl, and others implying that second order quasi-linear field equations for the metric tensor possessing symmetries and conservation laws of the Einstein equations necessarily arise from a variational principle. This claim taken from: Gauge invariance, charge conservation,and variational principles, by Manno, Pohjanpelto, and Vitolo -I wonder what else is out there in this regard? -I am betting there is more but I really do not know.

(footnote2) Okay, to try and get past the subjectivity, how about comparing two non-equivalent theories to see how "different they are"? Could there be some use for gauge redundancy here? Skimming around when I could during a meeting at work I find hints of this idea in "General covariance from the perspective of Noether's theorems" from Brown and Brading. [http://philsci- archive.pitt.edu/821/1/TorrettiB&B.pdf](http://philsci- archive.pitt.edu/821/1/TorrettiB&B.pdf) page 8, footnote. But again, I surely don't know!

<comment><USER6> I'll have to think about that for a day or two.</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment>


Using jQuery and node.js to scrape html pages in 5 lines - <AUTHOR> http://blog.nodejitsu.com/jsdom-jquery-in-5-lines-on-nodejs

<comment><USER2> Strange popup: "Hello, i see you are coming from hacker news.

the article you clicked on was most certainly not submitted by nodejitsu.

news.ycombinator has a long history of squashing articles and submitters that aren't funded by y-comb.

most of this is done through their "silent" banning and censoring mechanisms, that leave people not even realizing they have been silenced.

i hope you enjoy this article, and remember that HN is extremely biased and that you should keep your horizons broad."

While I would agree that HN is bias towards YC-funded projects I would not agree that it is biased against non-YC projects or news. In fact, the majority of the items on HN are non-YC. This also follows for submitters and commenters for the year or more I've been here.

On a different note. Hpricot is not representative of Ruby scraping anymore - nokogiri (<http://nokogiri.org/>) is where it's at. Which has a Hpricot translation layer if you need to change. Even when I decided to solidify on Python for everything else I will still go back to Ruby just for nokogiri when it comes to scraping.

<comment><USER3> Marak, the guy behind the Nodejitsu (and, presumably, the popup message) is known to exhibit asshole behaviour. (See: <http://news.ycombinator.com/item?id=1448309>) The popup message is consistent with what HN knows of him.

Whether being a jerk justifies banning I can't say - but his assertion that HN is biased has little justification (particularly when you consider that the writer _himself_ is biased.) Kindly ignore.

<comment><USER4> He seems to be about the same sort of asshole as a lot of other internet personalities (Theo comes to mind). i.e., he's good at what he does, knowledgeable, and he doesn't take the time to play nice with others.

It kinda comes down to what sort of "thing" HN is supposed to be -- whether it has a place for people like him -- but regardless, he's certainly not exactly hurting over his inability to play in our little sandbox here.

<comment><USER5> But he's actually not knowledgeable or good at what he does -- if you view his Github page, you'll see that his projects are 100-line thefts or wrappers around other people's work.

He's been a major drain/drag on the Node.js community, and makes the IRC channel a toxic wasteland for a good part of the day.

He trolls other sites as well... <http://www.youtube.com/watch?v=IrkDqh9ZVog>

<comment><USER6> Say what you will about his interactions with people, but "not knowledgeable or good at what he does" is an unfair characterization.

I find his hook.io project (<http://hook.io/about.html>) particularly interesting, and the BDD testing app he was building for Node Knockout has a lot of potential as well.

</comment></comment></comment></comment></comment><comment><USER7> Hey guys. The Nodejitsu team and Marak (<http://www.github.com/Marak>), the guy behind Nodejitsu are perma-banned from HN and can't respond to your queries.

He sends his regards, and if you'd like to contact him visit the #Node.js IRC channel @ Freenode

</comment><comment><USER8> I have a question: Does scraping like this execute Javascript on the scraped page? Am I able to access the output of Javascript/AJAX on that page?

As far as I know this is impossible with any other server-side scraping technology.

If so, that would be amazingly useful for a couple of my side projects, much easier than parsing their Javascript code and extracting the info I need.

<comment><USER7> You'd have to parse the page seperately and run each piece of in line scripts / linked scripts in a sandbox which can talk to jsdom, but it could be done.

<comment><USER9> Sounds like a useful general-purpose library someone could put together: server-side execution of HTML-embedded js, with some sort of configurable sandbox (e.g. decide whether you want to let it call out to the internet or not). All the components are available, but seems like the field is open for an all-in-one solution that gets the defaults and edge cases right.

I imagine Google has something at least close to that internally, given that they've dropped hints here and there about their ability to crawl post-js- processed pages, but unless I've missed it, I don't think they've released anything (my guess is because of the arms-race issue, with shady sites trying to use js to mask certain content from googlebot).

<comment><USER8> Second this idea. Nothing like this exists currently except for a couple buggy poorly documented dead projects.

Selenium might be OK for usability testing a single site, but it's useless for large scale multithreaded crawling and scraping and doesn't scale.

I, for one, would definitely pay good money for an easy library/API to scrape JS-heavy sites without the overhead of a full browser running macros.

</comment></comment></comment></comment><comment><USER10> The article lists BeautifulSoup as the Python choice for scraping, but that isn't necessarily true. I'm using <http://scrapy.org/>, for example, which is a scraping framework that uses lxml and xpath by default.

<comment><USER11> I don't know if it was edited after your post, but it lists Scrapy right next to Beautiful Soup.

</comment></comment><comment><USER12> This reminds me, I ported the core ideas of the parsley scraping language to jQuery.

<http://github.com/fizx/pquery#readme>

</comment><comment><USER13> The article reads "The challenge with using these libraries is that they all have their own quirks that can make working with HTML, CSS and Javascript challenging."

And that's true only if you only want to do page manipulation in Javascript. I'm perfectly happy with my page manipulation in Ruby w/ Nokogiri. Here's an example:

(code formatting on HN sucks, so it's on my blog, apologies)

[http://blog.knowtheory.net/post/1074676060/xml- manipulation-...](http://blog.knowtheory.net/post/1074676060/xml-manipulation- in-6-lines-of-ruby)

</comment><comment><USER14> Interesting, when I built <http://cQuery.com> (Content Query Engine), I investigated a number of options html parsing and content extraction options. I had played with Rhino and John Resigs env.js (<http://ejohn.org/blog/bringing-the-browser-to-the-server/>) to run jQuery server-side.

For portability, performance and flexability I finally settled for writing my own HTML parser and CSS selection engine from scratch.

</comment><comment><USER15> Site appears down. Is node popular enough yet for the "Node doesn't scale" talk? :)

<comment><USER2> Yes, as cliched as it is, I think it's time. I couldn't use at least 6/10 of the node challenge top 10 when it hit the HN front page (and the rest were beset by bugs and didn't work - the pixel one where you form characters stopped showing the shape I was supposed to be trying to get into after a few rounds, and the robot war one never let me buy or release my wave of robots on Chrome or Firefox). Overall it was totally disappointing experience.

<comment><USER6> I think there are a couple of factors here:

1\. It was a 48-hour coding competition, so some bugs are to be expected.

2\. The hosting for the apps in this competition was turned on literally the day before: <http://twitter.com/joyent/status/22204412477>

I don't think that "node can't scale" will get much press, just because node is pretty performant and many people conflate performance and scalability. Node's asynchronous api would seem pretty good in a services-oriented architecture, and scalability is mostly about how the app is architected. (and in 48 hours, that probably means "quickly")

On the other hand, node.js is still a very young platform and there are a lot of unsolved issues. As of right now there are only 3 or 4 dedicated node.js hosts that I know of, and all are in private beta. People are still trying to figure out best practices for design and testing.

The thing that I took away from Node Knockout was that here are a bunch of really cool web app ideas that, given time, could be built with node. I think the apps were 'frozen' pending judging, but I know many of the top app teams were planning on improving them as soon as they had the chance.

</comment></comment></comment><comment><USER16> Ignoring the drama: my current favorite scraping combo is NekoHtml underneath Scala's completely kickass combo of pattern matching and XML literals.</comment>


Ask HN: Tech stack for Fintech web app - <AUTHOR> Hello HN,<p>I want to build a Fintech Web app i am confused with the tech stack to be used, the will have a dashboard with admin ui( each customer having their own admin dashboard) i want to the ui to be beautiful and responsive, I want the app to scalable, which stack should I use? MERN/MEAN? Django? Or something else( I've resources to hire developers ) <comment><USER2> Laravel Nova ([https://nova.laravel.com](https://nova.laravel.com)) could get you pretty far. PHP is easy to hire for and scales easily.

<comment><AUTHOR> It says $199 per site does that mean I've to pay for each user? I've like 1000+ users with revenues very much higher. I'll be using custom data wrangling, analysis and prediction. Will this be good. I'm currently looking at Django as it pretty solid and have Python to which I can use Python's data science libs to do all my stuff, but I've not found any example of beautiful dashboard ( should I create the dashboard as frontend after login rather than going to Django's admin??)

Thanks for the reply!!

<comment><USER2> Per site so long as it is a single domain name with 1000 users it is fine. I like it because it is attractive and easy to customize.

If you can get most of the work done in Python then the question becomes: does Python dynamically generate the data/results for your users or do you store the data somewhere and some other tool (PHP for example) can query it to present to the user?</comment></comment></comment>


Ask HN: Gmail Down? - <AUTHOR> Can't sign in to my personal or gsuite company accounts. <comment><USER2> Not for me. U.S. East coast

</comment><comment><USER3> Not for me. Southern U.S.</comment>


Show HN: Libconcurrent – Coroutines in C - <AUTHOR> https://github.com/sharow/libconcurrent <comment> david-given I wrote a networking daemon (SMTP greylisting proxy; now unmaintained, but <plug> [http://spey.sf.net](http://spey.sf.net) </plug>) using coroutines once.

Non-preemptive multitasking made reasoning about the problem way easier, because I had no concurrency to worry about, while still allowing inversion of control flow which meant that the SMTP state machines were small and simple. It worked really well.

...until I started getting bug reports from users about weird irreproducable crashes. After a very, _very_ long time I eventually figured out that if spey was built on a system where sqlite had been compiled with pthread support, then it automatically linked in a thread-safe malloc, which tried to fetch the current thread ID, which on some versions of pthreads on some Linux architectures on some glibc versions with some kernel versions, was stored at the top of the stack. It used alignment tricks to find this.

So, every time something called malloc(), the libc was doing the equivalent of:

        threadid = *(int*) (alignup(sp, 16MB) - sizeof(int))

Except my stacks weren't allocated through pthreads, so threadid ended up being garbage. Hilarity ensued.

I eventually rebuild the coroutine library to be a wrapper around pthreads with a big lock to ensure that only one would run at a time. Which was kind of evil, but much more reliable. (Previously I was using my own coroutine implementation based on getcontext/setcontext.)

So, uh. I can't remember if I had a point any more; but the pain remains.

<comment><USER2> I use pthreads and coroutines (actually fibers) inside pthreads and lock-free bounded queues to communicate between pthreads. All pthreads execute coroutines until all of them get blocked, then pthreads go to epoll. If a pthread gets a socket readiness notification for a socket it owns or an orphan socket then it handles I/O itself, without any synchronization. If a socket is owned by a coroutine running on a different pthread then the notification is forwarded to that pthread via the bounded queue. The project is called MainMemory, and it is just here on Show HN.

</comment></comment><comment><USER3> The links to other projects with similar goals at the bottom of the Readme is brilliant. It helps developers solve their problems irrespective of whether this particular solution works...and solving problems not stars and forks is the measure of utility for a library.

<comment><USER4> It's missing this link, which I previously thought was "the" co-routine de- facto standard for C before seeing all those other links: [http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html](http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html)

There's a link to the .h file near the bottom of the page.

That technique is what's used in protothreads and Contiki OS.

</comment></comment><comment><USER5> You might also want to look at lthread. It has some interesting functions like lthread_compute_begin()/end() to run expensive computations inside a coroutine. the compute methods move the coroutine into an actual pthread and resume until compute_end() is called. It also has c++11 bindings called lthread-cpp which allows you to launch coroutines with variable arguments & types and comes with nice socket wrappers.

      void MyMethod(std::vector<int> my_vec) {}
      std::vector<int> v{1,2,3,4};
      Lthread t1{&MyMethod, v};
      t1.detach()

      void my_coroutine()
      {
         lthread_compute_begin();
           ret = fibonacci(55);
         lthread_compute_end();
      }

</comment><comment><USER6> How does this compare to [http://libmill.org](http://libmill.org)?

<comment><USER7> libmill offers a channels(FIFO queues) abstraction. This is a fibers/coroutines framework. They solve different problems. See go-lang channels for what libmill is providing.

<comment><USER6> Oops, sry, I meant [https://github.com/Zewo/libvenice](https://github.com/Zewo/libvenice)

</comment></comment></comment><comment><USER8> Cool!

A co-TA and I ported the concurrency library in JOS, MIT's teaching operating system, to regular UNIX:

[https://github.com/geofft/vireo](https://github.com/geofft/vireo)

It uses the standard-ish library functions setcontext and getcontext to avoid an assembly dependency. ("POSIX.1-2008 removes the specification of getcontext(), citing portability issues, and recommending that applications be rewritten to use POSIX threads instead.")

<comment><USER9> *context are unfortunately quite slow as they require a system call to update the signal mask. The sigaltstack trick popularized by Pth is probably still more portable and faster.

BTW makecontext and friend were made obsolete by posix for a technicality: makecontext signature isn't expressible any longer in a strictly conforming C99 applications, although in practice it will work with any C compiler.

Ironically this happened around the same time that coroutines started becoming popular again.

</comment></comment><comment><USER10> It would be helpful if the README explained how this is different than the many other coroutine libraries.

</comment><comment><USER11> How does this compare to Apple's GCD [1]?

1 - [https://en.wikipedia.org/wiki/Grand_Central_Dispatch](https://en.wikipedia.org/wiki/Grand_Central_Dispatch)

<comment><USER7> I ported GCD to Linux when it was made available/open-sourced some many months ago. Effectively, GCD uses a bunch of (named) FIFO queues, protected by semaphores, and manages some OS threads, which dequeue tasks(or blocks) from the queues and execute them. It provides some nice higher level abstractions and functionality (e.g groups ) on top of that. It also supports both synchronous and asynchronous execution semantics. The OS threads are of course managed by the OS scheduler. (unfortunately, pthread workqueues are not supported on Linux yet -- the original implemntation creates workqueues for each defined priority and that's how priorities are supported in GCD. On Linux you can use setpriority(), ioprio_set() and other such APIs that don't really offer much in practice. But I digress).

Contrast that to coroutines/fibers where tasks(i.e functions), which run in the same I/O thread(though not an absolute requirement). Cooperative multitasking is used, where a running function can yield control to other runnable functions by explicitly calling a yield like function -- that is, there is no scheduler that preemptively stops a function and runs another. You need to do that yourself.

There is some overlap between the benefits and properties of tasks that run on OS threads and fibers, but for the most part, they have different pros and cons. You can't really use in practice fibers to get the benefits you get from a GCD like framework (which utilizes multiple H/W cores to run functions/tasks concurrently), but also, you can't use GCD to improve throughput on a per- thread basis and deal with long-running and/or blocking lightweight tasks, or implement a fair-scheduling execution scheme).

</comment></comment><comment><USER12> I got a silly question:

Is this statement basically a do while/while(true) infinite loop?

        for (;;) {
             ...
        }

<comment><USER13> Yes. You're likely to run into this construct often in C.

Lots of people do it because stupid C compilers produce better code for `for(;;)` than they do for `while(1)` and stupid C compilers used to be very common.

It's also one less character spent: A rare win/win.

<comment><USER12> Cool, thanks! I have yet to learn C other than from what I've learned from programming in C++.

I am really interested in learning some C however, especially after this post hit front page a few days ago:

[https://matt.sh/howto-c](https://matt.sh/howto-c)

<comment><USER14>
      for (;;) { ... }

is a valid JavaScript and PHP syntax too. And used also as the first few chars of a JSON stream to prevent other sites consuming your internal API (e.g. Facebook uses it, Google+ uses while(1) { ... } which is longer).

</comment></comment></comment></comment><comment><USER15> Are the Coroutines scheduled across multiple CPUs?

<comment><AUTHOR> No, i suggest to check lthread.</comment></comment>


The Story of Mel Explained - <AUTHOR> http://jamesseibel.com/blog/?p=109 <comment><USER2> Something I've long been curious about:

        Mel finally gave in and wrote the code,
        but he got the test backwards,
        and, when the sense switch was turned on,
        the program would cheat, winning every time.

This doesn't sound like the sort of bug that occurs because you get a test backwards. Making a blackjack program that wins every time, and one that loses every time, seem like very different beasts. So what mistake did Mel make?

<comment><USER3> The manual for the game says:

      If SENSE SWITCH 32 is depressed, there is a better than normal chance
      of an ace being dealt as the player's first card.

Presumably the story didn't get that quite right. Sadly the code hasn't turned up.

</comment></comment><comment><USER4> Woz also coded the first Apple I in binary as he had no assembler. Of course the 6502 was a beautifully simple design with a minimal set of opcodes.

Mel's story is nuts but from the perspective of today (or even when I started in 1981) but you have to start somewhere and you can't judge what people did in the early days except in the context of the times.

<comment><USER5> Didn't the story go that Paul Allen wrote a loader for Altair Basic this way on pen and paper on an airplane?

</comment></comment><comment><USER6> > To summarize, Mel enabled the index register bit (despite not using the > index register) to cause an integer overflow, and because the instruction > was at the very top of memory, it had the side-effect of also incrementing > the instruction's opcode, modifying the instruction to change into a JUMP > command, in order to exit an infinite loop.

Reading this made me thing of some of the crazy tricks I've pulled in TIS-100. And while it worked well in a game, it would be a nightmare for that to be my day job.

</comment><comment><USER7> I read this interesting story once before and still don't understand why the author insists on typecasting Mel as a "Real Programmer".

It sounds to me like he should have been writing optimizing assemblers instead of wasting his talents doing one-off blackjack programs.

Additionally, Real Programmers write comprehensible code.

<comment><AUTHOR> That is the point - the term 'Real Programmer' is meaningless and is merely used to disparage languages / tools / platforms and boost egos. You can look at Mel, a 'Real Programmer', and realize that you can't make value judgements of what true programming is. Programming is using logic to solve problems, regardless of the tools and language.

</comment></comment><comment><USER8> Was the difference in speed between Mel's super-optimized blackjack program and a less optimized equivalent actually noticeable to the user? Or was this level of optimization crazy even back then?

<comment><USER9> The machine's main memory was a physically rotating drum. If you think of how slow that must have been, odds are very good that his optimizations made a noticeable difference.

<comment><USER10> how many RPMs for one of those drums?

<comment><USER3> 3600 rpm, with 64 words per track, so reading a word took 260 µs. Minimum cycle time (the address is up next) was 9 × 260 µs = 2.34 ms. Worst case then (just missed it) was (9 + 63) × 260 = 18.75 ms.

</comment></comment></comment></comment><comment><USER5> I am finding the explanation distracts from the prose of the original and is a jarring contrast. Perhaps this would be better as footnotes.

<comment><USER11> I think the assumption is that The Story of Mel has been circulated enough that finding a copy of it isn't a problem.

For everyone else, though, it's part of the Jargon File: [http://www.catb.org/jargon/html/story-of- mel.html](http://www.catb.org/jargon/html/story-of-mel.html)

</comment></comment><comment><USER12> This is a great link. I read this story a while ago, but missed a lot of meaning behind it. This explanation hits the spot explaining all of that.

</comment><comment><USER13> Those were the days, when self-modifying code was inevitable.

You should think that self-modifying code has died out by today, but far from it. Each time you open a web site, you could watch this old trick at work. How often do you `document.write` on a web site? To put it without diplomacy, our webware today is at the level of programming an IBM 650 with magnetic drum memory, where self-modifying code is practically mandatory. And Mel is nowhere to be found.

<comment><USER14> By and large we know better than to output javascript with document.write. We have structured templating languages even for use in a web page. Not everyone uses them yet, but the better technologies do exist.

<comment><USER15> I wish! Using document.write to inject script tags is the most reliable way to insert code to run before another script tag. Here's an example where we might load some scripts then run some code before finally calling some main function.

      <script>
        // Load optimized or debug code.
        if (debugMode) {
          document.write('<script src="build-debug.js"></script>');
        } else {
          document.write('<script src="build-opt.js"></script>');
        }

        // Enable some runtime debug flags.
        if (enableTracing) {
          // This runs *after* loading the script above and
          // before main() is called below.
          document.write('<script>ENABLE_TRACING = true;</script>');
        }
      </script>
      <script>
        // This runs *after* the script tags injected in the section above.
        main();
      </script>

</comment></comment></comment><comment><USER16> > The Story of Mel is a story about a 'Real Programmer' that came out in the > early 1980′s on Usenet

I read this as a story of "coming out" on Usenet... thought it was going to be about sexual/gender identity or something.</comment>


Interview with Larry Wall - <AUTHOR> https://developers.slashdot.org/story/16/07/14/1349207/the-slashdot-interview-with-larry-wall <comment><USER2> My experience with Perl:

We (a corporate shop) have a lot 'legacy' scripts in Perl.

I am assigned to maintain such scripts as no one else wants that role.

I discover, Perl is powerful.

Every 'legacy' script I find can be trivially regression tested (These scripts always have the same pattern, take some input, produce some output).

Managers want to move away from Perl.

I want to move closer to Perl.

<comment><USER3> The sad thing is about 90% of normal back-end computing at enterprises is take data source A,B,C,etc and generate data file Z. Cobol lives because it could do that well. Perl did that amazingly well[1]. Its really trivial to write easy to maintain code that does that in a very standard (across programs) way.

1) Data Munging with Perl was a pretty good book for it

<comment><USER4> Work is receive input, mangle input, send output once you generalize it enough. but the devil will definitely be in the details.

<comment><USER3> Oh yeah, the details are the problem. Worse, its a technical problem and a people issue for some kinds of output. Report Writing has all of the problems of graphic design and date munging with none of the rewards. Lord help you if the friggin DBA & Application Developers built the database or other data sources for transactional processing and didn't provide any help (e.g. summary tables, other data source) for batch (set) processing.

I now follow a simple law when setting up a data source: If I need attribute X from an item then I better have a way to find all items with attribute X in a hurry.

</comment></comment></comment></comment><comment><USER5> It is hard not to like him. But as a recovering linguist (not the programming kind), I find him strangely approachable.

I wish I had his code and programming abilities. He seems like such an interesting guy. Every once in a while Perl6 comes up. I read about it, and like Perl 5 but more so (at least Perl 5 I have used from time to time), but it seems like such an impressively rigorous build-up of work by hardcore veterans. Most of his jokes come off as sarcastically arrogant, and I enjoy how he pokes fun at his work and himself. I only kind of worry about the Perl Jam security talks his "doctor it hurts when I laugh" retort, because malicious doctors force you to laugh all damn day and steal PII and run up your bills online with malicious transactions. I do not want to dismiss TIMTOWTDI, but if you have a limited set of bad ways to do it, you cannot expect people to ignore it. This shows a weird kind of idealism to me in the one area we know idealists do not lurk.

Jokes aside, I will always see him in a special light. If only I could honor him better by churning out some Perl!

</comment><comment><USER6> "Perl has always considered itself primarily a programmer-centric language, while Python has always considered itself to be more institution-centric. So in a sense it's a bit dumbed down, much like Java. You'll note both of those languages make their greatest appeal to managers. :-) "

Huh? I was using Python when Perl was more popular, and this seems completely fabricated. He's trying to lump Python in with Java, which is bizarre because those 2 camps have pretty big philosophical differences.

I think that the main reason that Python became more popular because it has both clearer syntax and clearer semantics than Perl. It is compact AND readable -- not just compact. The data structures in Python also behave more like those you would see in a CS textbook.

I'm working on my own language, and I just read over the Camel Book ("Learning Perl"). Honestly I don't see much in there that I want to borrow, except maybe for ~= automatically introducing variables. What else is good about Perl, which is

        1) not in Python/Ruby/JavaScript (like hash tables)
        2) overly terse/clever?

(Honest question)

Perl's regex syntax seems to be it's biggest influence and legacy, with Python/Ruby/Java/PCRE all adopting its syntax.

And -e in Perl is also useful -- Python/Ruby/etc. aren't really good for one liners like Perl. This provides a nice gateway into the language.

But a lot of other features were rightly ignored by other languages IMO.

Also, with regard to the previous sentence, I actually consider Python OO kind of weak, but it's definitely better than Perl's AFAICT.

<comment><USER7> > What else is good about Perl, which is 1) not in Python/Ruby/JavaScript or > 2) overly terse/clever? (Honest question)

Scalar/list/hash context. It makes zero sense and the syntax seems to change randomly… until you understand it. Then contexts make a great deal of sense.

Also, having dealt with a good deal of Python, I'm inclined to lump it in with Java too—less verbose and dynamically typed, but still focused on code that's superficially easy to read or understand. There's a big difference between being able to read a function or class definition (whatever the hell it means to 'read' code; I don't even know anymore) and understand where that falls in the big scheme of things. Python and Java read from the bottom-up; individual methods make sense but then you have to piece everything together. Perl, APL, and Lisp-family read from the top-down. You start with some abstractions and idioms and work your way down to individual functions, which may be hard to grok but what matters is _what_ they do, not _how_. Perl makes writing those really, really fast, and allows building abstractions easily.

<comment><USER6> OK thanks for the answer. I agree that this context sensitivity is probably the most Perl-ish part of Perl. I just read over the Camel book section on this, and it says:

"This is the most important section in this chapter. In fact it's the most important section in the entire book."

"Not only that, but you can't make any general rules to apply what you know about some expressions to others. Each expression can make up its own rules. ... Perl is very much a language that tries to do the most common, mostly right thing for you."

"In fact, a big part of learning Perl is actually learning how Larry thinks. Therefore, once you can think like Larry does, you know what Perl should do".

Those quotes don't inspire confidence in me -- it sounds like a battalion of mostly correct special cases... mostly correct meaning "sometimes wrong". However, I do appreciate that the linguistic part of your brain is very large and powerful, and once you internalize all these rules they can lead to a beautiful expressivity. Larry is making use of a different part of your brain than other languages do.

But I have to work with 5 languages regularly, and I see this polyglot trend only increasing in the future. So I appreciate when languages are a little more predictable, and a few months away from them won't diminish my fluency.

<comment><USER8> > _In fact, a big part of learning Perl is actually learning how Larry thinks. > Therefore, once you can think like Larry does, you know what Perl should > do._

Haha, this reminds me of Ruby's Matz:

> _The principle of least surprise means principle of least my surprise._

<comment><USER9> The key insight here is that without unusual levels of care, the principle of least surprise _always_ is about what would surprise the author least, and this can be subjective.

</comment></comment></comment></comment></comment><comment><USER10> Some gems:

> It turns out if you accept the 90% of each paradigm that is practical, and > reject the 10% that puritanically rejects other paradigms, you can get a > pretty decent general-purpose language out of it

> But there's a long history of people confusing "general-purpose" with > "Turing-complete".

Yeah. Things that are formally equivalent are _not_ (usually) equally easy.

</comment><comment><USER11> I wish the language the best, but I find it a bit confusing to call it Perl 6. It is basically a new language, something different than Perl (5).

<comment><USER7> That's basically everyone's sentiment in the Perl community right now. Perl6 is a nice language (I mean, I'm not a huge fan, but it's not bad either) but it's a language that has some Perl influences but is definitely not Perl. It's like if Ruby was called Smalltalk 2.

<comment><USER12> > everyone's sentiment in the Perl community right now

Definitely not 'everyone'. And I strongly suspect it's not even 'most'.

There has been a relatively small but very vocal group with this point of view.

<comment><USER13> It is not just a sentiment, but a fact that Perl 6 being called Perl 6 has hurt Perl 5. The only reason you don't have a lot of people telling Larry to stop being stubborn about this issue is that the milk is already spilled.

<comment><USER12> I wonder if you can share the data behind this fact? I'm always looking to add missing datapoints to my brain. Much appreciated!

<comment><USER13> The data is obvious online, in that many many people have to be explained that they're different languages, with a good part of those also stating that they gave up on Perl 5 since "Perl 6 is coming"; as well as in direct talks with companies as early as 5+ years ago already when many german companies told me they're delaying further investments in Perl 5, since "Perl 6 is coming".

<comment><USER12> Thank you. I don't doubt that Perl6 has had some negative impact on Perl5.

However, I and many others believe that Perl6's positive influences on the Perl5 community far outweigh the downsides. For starters, the whole Moose:: ecosystem ([https://metacpan.org/pod/distribution/Moose/lib/Moose/Manual...](https://metacpan.org/pod/distribution/Moose/lib/Moose/Manual.pod)) would arguably not have existed were it not for Perl6. And there are quite a few other examples.

Perl5 is a vibrant, active language: [https://metacpan.org/recent](https://metacpan.org/recent)

I understand your perspective though.

<comment><USER13> Perl 6 has done many good things to Perl 5, yes.

However the name being what it is has hurt Perl 5, and that was the topic being discussed here.

<comment><USER12> Understood, thank you.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> >Why isn't PERL more windows friendly?

Really? I've had tons of fun with Perl+OLE automating the processing of Excel and Word files.

<comment><USER15> Same; I've always had more problems with Python on Windows, rather than Perl.

With Perl 5, both ActiveState's and Strawberry Perl just worked, whereas Python can't even print to the console [1] without workarounds or an extra library that's not installed by default.

[1] [http://stackoverflow.com/questions/5419/python-unicode- and-t...](http://stackoverflow.com/questions/5419/python-unicode-and-the- windows-console)

<comment><USER16> I've grown somewhat disillusioned by ActiveState, but StrawBerry Perl has been quite good to me.

</comment></comment></comment><comment><USER17> "Which large companies are still using Perl in production? I can name Booking.com, but do you know any others?" \- it's worth mentioning DuckDuckGo of course, which is quite young and chosen Perl as a way to go.

<comment><USER18> Oh By[1], the universal shortener, runs solely on perl CGI.

We wouldn't have done it any other way.

[1] 0x.co

<comment><USER19> That URL isn't even being converted into a link by my browser, it's so short ;)

</comment></comment></comment><comment><USER20> > Software interprets lawyers as damage and routes around them.

I feel like it's 1999 again, finding a quotable Larry Wall statement on Slashdot.

</comment><comment><USER21> Slightly off topic, but I've been thrust into working with some perl code and am looking for a good learning option. I like Code Academy's interface and there are also a few good Markdown tutorials out there[1][2].

Does anyone know of a similar experience for Perl?

[1]: [http://www.markdown-tutorial.com/#/](http://www.markdown- tutorial.com/#/)

[2]: [http://agea.github.io/tutorial.md/](http://agea.github.io/tutorial.md/)

<comment><USER13> The best learning option is to head to [http://perl-tutorial.org](http://perl- tutorial.org) and check out what the most recent things are. Reason for that is that due to Perl's sheer age and rate of evolution, many of the older learning resources are outright damaging to newbies.

</comment></comment><comment><USER22> With all due respect to Larry Wall he shouldn't be a source of info or inspiration when it comes to Perl6.

Watch this instead, it's pretty damn great

[https://www.youtube.com/watch?v=Nq2HkAYbG5o&feature=youtu.be...](https://www.youtube.com/watch?v=Nq2HkAYbG5o&feature=youtu.be&t=274)

</comment><comment><USER23> > LW: PHP is both the motivational poster child for worse-is-better, as well > as the demotivational poster child for worse-is-worse. Somehow PHP has > managed to convince a horde of programmers that if their programs are flakey > or hard to maintain, it must somehow be the programmer's fault. It couldn't > possibly be because they've been frog-boiled.

I've never heard the term 'frog-boiled' before, but it's the most succinct description of PHP I've ever heard.

</comment><comment><USER24> At my current job we've started using Node.js for a lot of windows based migration jobs and shell scripts, historically where we would have used Perl. We're mostly C#, but data munging and migration is just a lot easier and faster in Node.js or Perl.

I'm curious on what we're missing out on by using node instead of perl?

<comment><USER25> Node is getting pretty good. I'm a Perl developer of many, many years, and it is still my go to for command line scripts (and also web, but only because I really like Mojolicious and it happens to fit into existing systems). But, building command line tools in Node is no longer a crazy idea. It's still gonna be more verbose, by a lot, than Perl, for many classes of problem (data munging, file operations, almost anything that needs to interact with the OS), but the language has gotten nice with ES6. The libraries are pretty good.

It's really hard to beat the built-in "while <>" semantics of Perl for writing concise, clear, programs that take something in from a file or STDIN and spit it back out. Perl is effectively a DSL for writing command line scripts that live in a UNIX environment (they work OK on Windows these days, too, now that Windows has a reasonable shell with reasonable semantics for scripts like that).

If you want to write many small commands that are composable, Perl is a perfect fit. If you want to write one big application that does a bunch of things, the benefits of Perl look smaller and may even be outweighed by the bigger ecosystem of JavaScript/Node/npm; you can amortize out stuff like options handling and file/stdin handling across all of the functionality and it won't look like such a big part of your program anymore, compared to Perl (which gives you most of those things practically for free with one or two lines of code).

</comment></comment><comment><USER26>
      > my @values = [1 .. 20];
      1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
      > say @values[0,2,4 ... *];
      1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Wall says this should print all the evens, but it prints all (this is in the perl6 repl).

<comment><USER27> Which version are you using? I happen to have 2016.04 installed:

        $ perl6 -v
        This is Rakudo version 2016.04 built on MoarVM version 2016.04
        implementing Perl 6.c.

and I see:

        > my @values = [ 1 .. 20 ];
        [1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20]
        > say @values[0, 2, 4 ... *];
        (1 3 5 7 9 11 13 15 17 19)

(Of course it's the odd numbers, not the even numbers, but it just happens that the odd-valued entries are in the even-indexed positions in this list.)

<comment><USER26>
      perl6 --version
      This is perl6 version 2013.12 built on parrot 5.9.0 revision 0

Installed from the Mint/Ubuntu/... repository. Wow, that's really old.

Is there anything like Python's Anaconda, but for Perl6?

<comment><USER28> try rakudobrew or just building it yourself from rakudostar

<comment><USER26> rakudobrew worked fine, thanks.

</comment></comment></comment></comment></comment><comment><USER29> Oh, cool, finally :) Now it's the time to read it. :D Edit: have been waiting for it after seeing it on /.

</comment><comment><USER30> Isn't quite late for a JS backend given that we have WASM now?

<comment><USER15> In an IRC chat [1] on 2016-04-15, pmurias, the lead developer of rakudo-js [2] said:

> Perl 6 -> wasm won't be happening any time soon

> for now wasm is just bytecode asm.js

> if we really wanted to compile to it we would have to use enscripten and run > MoarVM on top of wasm

[1] [https://irclog.perlgeek.de/perl6/2016-04-15/text](https://irclog.perlgeek.de/perl6/2016-04-15/text)

[2] [https://github.com/pmurias/rakudo-js](https://github.com/pmurias/rakudo- js)

<comment><USER30> JS as target will always be slower and bigger(as payload) than WASM as target so my point is that JS as compilation target is no longer relevant once WASM is supported(i.e. most likely by the end of the year). It makes little sense to start a javascript backend now unless you are looking for very short term benefits.

<comment><USER31> It's JavaScript that's a much smaller and faster payload if you need garbage collection. There's talk of putting a generic garbage collector into WASM but that's still well into the future. WASM also doesn't have direct access to the DOM - right now it's more like Flash or Java applets than it is JavaScript.

<comment><USER30> Eventually wasm will provide GC.

Edit: It's on the roadmap just after MVP so I wouldn't say it's that far. Also based on the latest talks from Luke at City Hall(I can't find the link) the access to DOM could arrive even sooner than the GC. My point is that wasm is worth investing it because it's the future. Once language X compiles to wasm and perl compiles to JS you can see a double digits performance difference. I don't need to tell you which language looks more appealing. WASM has nothing to do with applets/flash etc. It was already clarified in various posts.</comment></comment></comment></comment></comment>


Fusion experiment by B.C. firm raises concerns - <AUTHOR> http://www.cbc.ca/news/canada/british-columbia/story/2011/10/03/bc-fusion-energy-project.html <comment><USER2> The author of the article clearly isn't familiar with the ideas or even the terms he's writing about. Controlled, man-made fusion has been done before, in many experimental fusion reactors around the world. This might be the first one we can extract usable energy from. [http://en.wikipedia.org/wiki/Fusion_reactor#History_of_resea...](http://en.wikipedia.org/wiki/Fusion_reactor#History_of_research)

Plasma isn't really the same as gas, it's actually a different state of matter. <http://en.wikipedia.org/wiki/Plasma_(physics)></comment>


Alt Text: How Neanderthal Are You? Take This Quiz to Find Out - <AUTHOR> http://www.wired.com/underwire/2010/05/alt-text-neanderthal/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+wired%2Findex+%28Wired%3A+Index+3+%28Top+Stories+2%29%29&utm_content=My+Yahoo

<comment><USER2> I suppose if you score a maximum on the test, you get put in charge of putting autoplay video ads on the site.</comment>


America is about to undertake "its biggest transformation in decades" - <AUTHOR> http://www.economist.com/specialreports/displayStory.cfm?story_id=15793036&source=hptextfeature

<comment><USER2> Having a trade deficit to a developing country is like giving them a loan. (My economics teacher)

The Relation the US had with China in the last years, reminds me a lot of the Japanese-Korean situation in the 1980s. Japan outsourced labor intensive work to South-Korea because it was cheaper there. Over time South-Korea became very wealthy and the labor costs were to high to continue to outsource. But this was great news, because the South-Korean now had demand for Japanese products.

The same story happens to US/China, once Chinese reach a certain wealth, they will be able to afford US products.

The other reason why I do believe this will happen rather quickly is this chart:[http://www.businessinsider.com/scariest-jobs-chart-ever- febr...](http://www.businessinsider.com/scariest-jobs-chart-ever- february-2010) If this chart is true, all the laid off consumers will drive the domestic demand down. So clearly the only possible growth is in other countries.

<comment><USER3> _The same story happens to US/China, once Chinese reach a certain wealth, they will be able to afford US products._

And which products would those be?

<comment><USER4> Ask and wikipedia answers: [http://en.wikipedia.org/wiki/Economy_of_the_United_States#Ma...](http://en.wikipedia.org/wiki/Economy_of_the_United_States#Manufacturing)

And that's not counting agriculture or services. Here's a full breakdown of the economy (not all of which is applicable to exporting):

[http://en.wikipedia.org/wiki/Economy_of_the_United_States_by...](http://en.wikipedia.org/wiki/Economy_of_the_United_States_by_sector)

<comment><USER5> As that first link notes under Manufacturing:

 _A total of 3.2 million -- one in six U.S. factory jobs -- have disappeared since the start of 2000._

So, I think this trend would need to reverse. This recession alone has displaced 8 million.

<comment><USER6> If Chinese income rises to the point where its middle class can afford to buy American products, I fully expect a lot of manufacturing to be coming back onto this side of the ocean.

<comment><USER5> I could see that happening, namely with cars, which is why I think it's important to have a strong car manufacturing industry (and another reason to save GM). However, one caveat I think is we need to place higher importance on quality, not just with cars but American products in general. If China has gotten good at the role of producer after these years, and the U.S. as consumer, and that suddenly flips... I'm not sure how smoothly that would play out. A lot of people could potentially eat our lunch.

</comment></comment></comment></comment></comment></comment><comment> daniel-cussen There is still the catharsis to go through.

What I'm referring to is a J-curve <http://en.wikipedia.org/wiki/J_curve> where the currency gets smacked hard -- say 2-1 -- and the trade deficit gets worse because terms of trade changed and you can't get rid of oil dependence overnight. Exports recover because the American worker is closer to par, in terms of costs, with the Latin American and Chinese workers. But it won't be pretty, and it'll really, really hurt. Just as things are darkest just before the dawn, things'll get bleaker for a while before Americans see this through.

<comment><USER7> I agree that this is a possibility. In the end how quickly America pulls through depends on how well it makes the 'transformation' to more sound consumer spending.

</comment></comment><comment><USER8> While this article seems pretty strong on economics(can't really judge , not an economist) , it fails to mention technology's impact , and the fact that today we have many industries being disrupted which was never the situation before the internet.

They give as an example the fact that intel is growing. but what happen if ARM wins in the processor battle in the next few years ? won't it change who is importing and who is exporting ? and at what price? The same can be said about Hollywood. they are under immense pressure from free/ad-based online video. this kinds of pressure gives an advantage to content that's cheaper to produce. this isn't very good for the u.s. industry.

So while this article is very optimistic about the exporting more, it would be a major challenge.

</comment><comment><USER9> I don't think I'm smart enough to understand. The companies in this article are proud to have shed jobs to become leaner and focused ... on what customers that have jobs to buy their products and services?

The basic conundrum, to me, is that there is no such thing as an economic recovery with no job growth. If people can't earn money, how do they spend it? If they can't spend money, how does the profits get reentered into the company's balanced sheet so that the company can invest in new products and services to grow? If a company can't grow, how does it employ new workers to buy things?

It's not that simple, but really, is it really that difficult to comprehend?

<comment><USER10> If your mental model of the country is that it currently contains zero employed people, it should not be surprising that your mental model produces nonsense results. GIGO holds on mental models too.

</comment></comment><comment><USER11> If the US is going to become export-oriented, who will we be exporting to?

The world economy has been polarized between export-oriented developing countries and the import-oriented United Stated. If No country is import oriented, what then? Japan and non-China Asia are suffering as much as US through the recession and they are ultra-export oriented.

The US became THE importer over the last twenty year. Even the other export oriented nations will have a hard-time without the US.

What's the answer here?

<comment><USER12> The answer is to create some kind of a balance. Currently the US has a large trade deficit. If the US imports goods and resources, but doesn't have goods/services/whatever to export in the same range, then the US will have the trade deficit for many years to come and the numbers add up. If it isn't possible to export more, than the US might want to reduce imports and generally build a more sustainable economy.

<comment><USER13> The balance is going to be at a different point than it has been for most of the last forty or fifty years, though. Americans will likely not be as rich relative to the rest of the world as they have been.

</comment></comment></comment><comment><USER14> Does this mean that investments going into the US will decline?</comment>