Introducing Chronos: A Replacement for Cron - <AUTHOR> http://nerds.airbnb.com/introducing-chronos <comment><USER2> I'm surprised there aren't more things like this, because Cron replacements are really valuable.

We wrote a small C program that serves as a scheduling daemon with Redis; we have a keyspace in Redis that can be used to schedule millisecond-granular periodic or one-shot tasks with a flexible "Chronic"-like specification syntax.

It is hugely more convenient and usable than cron. Once you have it, you immediately spot lots of opportunities to factor systems into scheduled jobs that you might have avoided doing if it meant you had to deal with cron.

<comment><USER3> I don't think it's uncommon at all to roll your own job scheduler. I know I've done it at the last two places I have worked, though neither had nearly as fancy a front-end as Chronos appears to have. This particular scheduler (Chronos) looks like a cross between traditional cron and HPC cluster job scheduling systems (a la Sun Grid Engine). It looks like a really cool project.

<comment><USER2> That depends on what you mean by job scheduling, right? Lots of people would call Resque a job scheduling system, but they still reach for cron when they want to run something every 5 minutes.

<comment><USER3> I'm about as unaware of the Ruby ecosystem as anyone who regularly reads HN can be, so while I've heard of Resque I'm not really sure where it fits in.

I suppose the definition of job scheduling is dependent on what your workload entails. Is it something you expect to complete in less than a second, a minute, or a big resource intensive HPC job you expect to run for hours? What sorts of dependencies do you have between the things you want to run? Will it run in the main event loop, separate thread, separate process, or on a separate host? I think this will reflect in the tool you choose for scheduling, or cause you to write your own.

It sounds to me like the target workload for Chronos is big slow batch jobs (ETL, data analysis). HPC job schedulers take care of things like resource management (run this job on this machine because it has the lowest load) and dependency management (don't run job b until job a completes), whereas cron handles running periodic tasks and that's about it. Chronos is nice because it looks like it is combining the two and includes a nice looking web based control panel to boot.

</comment></comment></comment></comment><comment><USER4> so... cron's a 200kb self contained C program, super simple, does one thing, and does it well.

Chronos is built on top of frameworks, needs a few services to run.

Certainly it has value, but marketing it as a cron replacement is wrong imo. Chronos is one of these "web-server-service-tool" thingie, but no, it's not cron.

<comment><USER5> Every time I come across one of these heavyweight, dependency-encrusted "replacements" for basic Unix tools, I'm reminded of L33tStart[0], a fictional, satirical init(1) replacement I wrote about on Reddit -- a sort of parody of systemd.

Every day, in a bizarre manifestation of Poe's Law, L33tStart seems less and less a parody.

[0] [http://www.reddit.com/r/programming/comments/14ay0r/hacker_k...](http://www.reddit.com/r/programming/comments/14ay0r/hacker_koan_on_the_proliferation_of_unix_text/c7bntjn)

<comment><USER6> Any chance you could open source L33tStart. Also what type of ponies were you using for your startups? Black or white ponies?

<comment><USER7> Mostly pink and purple, is my impression.

</comment></comment></comment></comment><comment><USER8> This is almost like saying "A Replacement for Gravity".

The benefit having something so simple and singular in function with no dependencies is that there's so little to break. While I appreciate the need for having something with more capability, calling it a "replacement" is a bit facetious.

I wish them success and call it an airplane. I'll stick to my hang glider.

<comment><USER9> What the ops team isn't telling them (DevOps only goes so far) is that there's a cron job on each node which periodically bounces the Chronos processes (and its 50 dependencies) in order to mitigate memory leaks.

</comment></comment><comment><USER10> Interesting to see that this was built on top of Apache Mesos, a dynamic cluster partitioning framework started by some researchers at Berkeley [1]. So far I've heard about Twitter (and now Airbnb) using Mesos in production. Is anyone else evaluating or using it currently?

[1] - <http://incubator.apache.org/mesos/papers/nsdi_mesos.pdf>

<comment><USER11> Wired had a great article on Mesos that mentions Conviva also uses Mesos: [http://www.wired.com/wiredenterprise/2013/03/google-borg- twi...](http://www.wired.com/wiredenterprise/2013/03/google-borg-twitter- mesos/all/)

</comment></comment><comment><USER12> Looks great, but calling it a replacement for cron is like calling word a replacement for cat.

</comment><comment><USER13> This is brilliant! I've had to do a lot of work creating queue based systems in recent years, and i really like off the shelf solutions to problems like this. Cron is a great tool, but its a mess once you have a bunch of machines and it really just does not scale.

<comment><USER14> By itself it does not. If you have a bunch of machines, you also have a config system (right?) and they (puppet, chef and so on) have the means to manage cron tasks as a class.

You still get cron, and the means to manage them across one's servers.

</comment></comment><comment><USER15> I really hope someone will succeed in rolling out a good and eventually popular replacement for Cron, because Cron completely sucks.

The crontab file has a weird syntax, that in some respects seems to look like a shell script, but isn't really one and some shell constructs work, other not. There are lots of ways you can make a mistake in writing the commands to be executed in a way that the command you intended won't run but will fail silently and you won't get any trace of an error happening. It is hard to even extract some common parts of commands and put them into a variable. I wasted hours and hours debugging weird cron errors. One case was where the crontab of one of the users was moved over to be the system crontab and strangely didn't work. Well, turns out the system wide crontab has one more field, but cron will not signal an error even in an obvious case like this, it will just fail silently (and this is the thing cron is really good at in my experience).

</comment><comment><USER16> Awesome! If anyone's ever had to use Ctrl-M or AutoSys, you know well the supreme horridness of other solutions in this space.

<comment><USER17> Brr, AutoSys gives me nightmares.

</comment></comment><comment><USER18> I know a couple guys that will be rather unhappy about the choice of moniker. <http://www.getchronos.com/>

<comment><USER19> <http://chronoss.sourceforge.net/> has been around longer than they have.

<comment><USER20> <http://en.wikipedia.org/wiki/Chronos>

<comment><USER21> Love it. The god Chronos has been around longer than time itself...

</comment></comment></comment></comment><comment><USER22> Tools like this can make a lot of ops housekeeping stuff easier, like log roll ups and deleting old backups and so forth. One doesn't want to depend on particular machines being up; but on the other hand, there's not a 99.999999% uptime requirement.

</comment><comment><USER23> Always happy to see more tools like this - the bigger the toolbox, the easier our lives are.

Of the various approaches I've had to depend on in recent memory, from perl scripts querying a central Db right through to eye wateringly expensive Control-M or Autosys in larger envs, It's plain old cron, fronted by config management (cfengine, puppet, chef, salt - it doesn't matter which) that has proven most dependable, easiest to train others on and simplest to debug.

</comment><comment><USER24> We've built Luigi at Spotify to solve a lot of similar problems: <https://github.com/spotify/luigi>

Might be worth checking out if you are building large data flows. We probably run 10k Luigi "tasks" every day, of which the majority is Hadoop jobs. They are all organized in a large dependency graph expressed within Python, and you also get visualization, exception handling, atomic file operation, etc.

</comment><comment><USER25> Many people do much of the fancy stuff with system managment products (Tivoli or BMC patrol spring to mind from past experiences). With that they can do centralised monitoring of the server and services and software and automate responses to defined conditions allowing automatic responses with the right thought and alerting callout when exceptions to the rules occur and when things need looking at.

Now all that said, I'm not uptodate on that side of things and even less uptodate on open source alternative, though my quick look at this does indicate that it is a start in the right directon and can only get better. So quick look and glance over gave me a good gut feeling, which is always nice to have.

</comment><comment><USER26> Anyone know why launchd hasn't taken off in this regard? I've used it on my personal systems with success, but rarely (if ever) see it mentioned as a cron replacement. Instead, you frequently see these solutions done from scratch.

<comment><USER7> Link for the ignorant and lazy, as I once was: <http://en.wikipedia.org/wiki/Launchd>

Seems like a decent idea. According to Wikipedia, Ubuntu considered it in 2006 when they were looking for a new unit system, but didn't like that it was under Apple's own license, which was shortly thereafter changed to the Apache license.

I'll remember it if I want to do something unconventional on a system.

</comment></comment><comment><USER27> I've seen it in action, and it can schedule down to the milisecond across machines.

</comment><comment><USER28> And....that blog is going offline in a few weeks. Thanks Posterous/Twitter!

<comment><USER29> pls...they used a custom domain and they have plenty of time to migrate to a new blogging service and point their DNS to it

</comment></comment><comment><USER30> Im not sure what mesos brings to the table (possibly distributed dependency management), but for a powerful replacement for cron with dependency management, logging, etc. one could use systemd timers [1]

1\. [http://jason.the-graham.com/2013/03/06/how-to-use-systemd- ti...](http://jason.the-graham.com/2013/03/06/how-to-use-systemd-timers/)

</comment><comment><USER31> I was planning to do something similar after I learned about Mesos although I haven't found the time. At the moment I'm using a few processes with MySQL publishing with RabbitMQ to handle distribution and node failures. It doesn't have nearly as nice an interface but RabbitMQ isn't terribly difficult to manage. Thanks for open sourcing!

</comment><comment><USER32> Seems like Chronos covers a lot of the same functionality as Rundeck.

</comment><comment><USER33> I am using a similar tool for job automation, history & alert, and it is called Jenkins :)

<comment><USER34> Yeah, I'm amazed that people consistently run around reinventing the wheel, when there are 18-wheeled chariots out there free for the grabbing.

</comment></comment><comment><USER35> As far as Cron replacements go, I prefer whenjobs [1].

[1] <http://people.redhat.com/~rjones/whenjobs/>

</comment><comment><USER36> in Java?

Something is deeply wrong with this world.)</comment>


The Living Truth: On Alexander Herzen - <AUTHOR> http://berlinbooks.org/brb/2016/12/the-living-truth/ <comment><USER2> There are three plays by Tom Stoppard about Herzen.

[https://en.wikipedia.org/wiki/The_Coast_of_Utopia](https://en.wikipedia.org/wiki/The_Coast_of_Utopia)

I saw them last year and I remember finding the subject matter daunting because it was so much complex material and still represented such an abbreviated, high-level view of his life and times!

Also, there's a huge amount of tragedy there.

<comment><USER3> The plays are a very good read with the added bonus that you can read at your own pace.</comment></comment>


Throw Away Your Bug Tracking System - <AUTHOR> http://wiki.apidesign.org/wiki/Bugzilla

<comment><USER3> Kind of inflammatory post that boils down to one reason--there's no point in having a bug tracking system if you're not going to address the bugs.

I disagree with this article. Especially if you're open source, you gain a lot by having a bug tracking system:

1\. It's another way to get user feedback. Are people excitedly opening lots of new enhancements, or frustratedly reporting lots of crashes? Building community is extremely important.

2\. It's a way to MEASURE the priority of bugs. Are 30% of your users running into this bug and piling on with "me too" comments? Then you'd better fix it. If you keep closing bugs because you dont want to fix them, you wont see that feedback.

3\. It's a way to organize your own thoughts and priorities. Is the new feature X that you want REALLY more important than fixing the existing functionality? How broken are things currently?

There's nothing wrong with leaving a bug open and making the comment that "it will be a while before we fix this" I'm more likely to submit a patch in that case, as a user.

<comment><USER4> Basically, this article is about optimizing the developer experience rather than the user experience. This seems to be a problem endemic in open source software anyway, which is why so few people use it.

<comment><USER5> "Basically, this article is about optimizing the developer experience rather than the user experience."

Did you _ever_ use Bugzilla? Nothing is more detrimental to the user experience...

Users would rather send support e-mails or use phone support. For the lack of that in open source project, keep bug trackers simple. E.g. Github's bugtracker is quite ok, it is fine for writing a bugreport and reacting to it. But doesn't have all the bells and whistles that makes it complicated for users.

<comment><USER4> The article is not specifically about bugzilla, but issue trackers in general.

</comment></comment></comment></comment><comment><USER6> As a user I find the way some of those bug tracking systems are used infuriating. This seems to happen to me often:

      - You submit a bug or find one that's active and comment on it
      - there is no activity
      - then vendor posts needsinfo
      - info posted soon after
      - then ignored for while longer
      - <USER2>
      - ignored for while longer
      - then closed because release is obsolete, need retesting in new release

This is mostly in RedHat's bug tracker, but elsewhere also.

<comment><USER7> Why not help out triaging and fixing bugs then?

<comment><USER6> At points I've submitted patches or very clear descriptions but they get ignored still. In other projects there is often a steep learning curve to get even a small patch in, like in Mozilla, particularly if it's a language you don't normally use. I think in most projects it's simply not feasible to drop in to do a couple of patches you really have to involved long term.

The only exception to this are security bugs where I've avoided the bug tracker and typically got fast responses and fixes.

<comment><USER7> OK, join the project and put the patches in yourself.

<http://fedoraproject.org/wiki/PackageMaintainers/Join>

<comment><USER6> OK, how do I join RedHat Enterprise? which is what we actually use and pay money for therefore where I find the bugs.

I probably should join Mozilla, Ubuntu, KDE and Gnome too - that's going to be a lot of catching up to do just to fix one bug in each.

Can you see what I'm saying? feedback is valuable to those projects but they are ignoring it, if the only way people can get a bug fixed is by spending 10s or 100s of hours getting involved, there is something wrong.

<comment><USER7> If you contact Red Hat support you'll get immediate help. If you have a support contract you shouldn't be filing bugs in Bugzilla, since the support team will do that for you, and chase the developers to get things fixed. Your support contract will make all this clear.

<comment><USER6> That's not been my experience, it's like in bugzilla but your oscillating between "waiting on customer" and "waiting on redhat" and the people your talking to don't really understand the problems and so have to ask someone else everytime.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> As a user I'd rather see a bug tracked even if it's not addressed. Sure, it's frustrating to find a bug that's received no attention, but having the bug tracked in a central location is one level of feedback better than nothing at all.

<comment><USER9> As an end user, it's very helpful for me when a bug has been tracked even if it's not addressed. When I run into some kind of weird behavior, I can just check the bug tracker. If someone else has already reported the bug, it saves me a significant amount of troubleshooting.

It's also pretty typical that users will post workarounds and patches in a bug report. As you might imagine, this is hugely useful on a number of levels. It makes it easier for project maintainers to fix the bug in question, and end users with the problem are able to quickly resolve the issue.

</comment></comment><comment><USER10> This came to mind: <http://www.jwz.org/doc/cadt.html>

But that said, I feel for this guy. Big backlogs with low priority enhancements masquerading as bugs are no fun.

There are some pretty big open source projects running without bug trackers, they just keep an eye on the mailing list, and if a problem keeps cropping up, it might get handled.

</comment><comment><USER11> Don't call it a bug tracking system. Call it a todo list.</comment>


Big Fish launches real-money gambling app in UK via Betable - <AUTHOR> http://techcrunch.com/2012/08/16/big-fish-raises-the-stakes-with-real-money-gambling-on-the-iphone-with-big-fish-casino/

<comment><USER2> One correction - TechCrunch claim Big Fish would be "the first time a real- money gambling game will be available on the app store". That's simply not true: there are several real-money games already available in jurisdictions where online gambling is legal. The large betting exchange BetFair, for example, have a casino games app with real money: [http://itunes.apple.com/gb/app/betfair- casino/id505191581?mt...](http://itunes.apple.com/gb/app/betfair- casino/id505191581?mt=8)

<comment><USER3> Correct, this is not the first real-money gambling app on iOS. However, it is the first existing iOS game wholly owned by a third party that has implemented Betable to facilitate real-money bets. So that's pretty big news for us :)

</comment></comment><comment><USER4> One issue that a service like this is going to run into relates to security. The fact that this application needs to be secure from local tampering and unauthorized betting is obvious -- what may not be as obvious to some readers is the location-based access control that needs to be built into legal gambling applications.

Several casinos in the United States (Nevada, from what I've seen) have already built and released apps to allow, for example, poker or sports betting from within the associated casino. Some of these also allow betting from wireless connections within the state of Nevada.

This is where the idea starts to get really sticky.

If you're allowed to gamble from certain geographic locations, but not _entire regions_ , how do you enforce access control? GeoIP is somewhat reliable, but border towns can fall on either side of the coin.

Wireless networking is _generally_ not super long-range, but the DEFCON wireless shootout proved that it's possible to sustain a wireless signal from _hundreds_ of miles away, given the correct (in this case, desert) conditions.

And that's not even mentioning transport layer issues. Could I set up a VPN in England to use this Big Fish app and gamble pseudo-legally from my phone in the States? If I do so, who's liable? What about simple SOCKS proxies? Dedicated/colocated boxes in the "allowed" region?

I think that this is really cool technology, and I'm excited to see it succeed. That said, as a security guy, I can't help but wonder who would be liable in these edge cases.

And, as I mentioned at the beginning of the post, I'd love to know exactly what security precautions are in place to prevent unauthorized bets and tampering. For example, let's say you're using this phone in a Starbucks. Will an SSL error (I can only hope they're using SSL) prevent the connection occurring at all? Will it give the user a chance to accept the change? Can I just submit a spoofed request to transfer me money, or is there some sort of "two factor" key on the phone itself?

It's an interesting problem, and I'm sure Big Fish has come up with interesting solutions.

<comment><USER3> Hey David,

While I can't give you a technical answer as I'm just the marketing guy, I can tell you that simply setting up a VPN in England is not going to work. We have a number of state-of-the-art gating techniques for identifying and prohibiting players from using our service illegally. We check for much more than Geo- location and IP. Obviously I can't get into as much detail as I'd like, but that's the gist of it.

Also, as Betable is doing all of the gambling, in the event that an extreme edge case occurs we would be liable, not the game developer working with us.

Unfortunately I can't answer the following questions about security precautions, but I can see if one of our engineers can hop on the thread to respond. I'll get back to you.

<comment><USER5> _Obviously I can't get into as much detail as I'd like, but that's the gist of it._

Come on, we all know that security through obscurity is weak. Tell us how the security system works and the community will test it for you!

<comment><USER6> "...I can't give you a technical answer as I'm just the marketing guy..."

"Unfortunately I can't answer the following questions about security precautions, but I can see if one of our engineers can hop on the thread to respond."

I don't think he's not explaining it because it would be insecure. He's not explaining it because he doesn't know it well enough.

</comment></comment></comment></comment><comment><USER7> And onward we march towards the Bingo Event Horizon... [http://www.whatgamesare.com/2012/06/the-bingo-event- horizon-...](http://www.whatgamesare.com/2012/06/the-bingo-event-horizon- behaviourism.html)

</comment><comment><USER8> I love what Betable's doing and I'm excited for their team. I took part in their first hackathon in SF a few weekends ago and built an app on their platform.

The brilliant part of their business is that they're the casino. You give them the inputs, they give you the outputs but all of the actual gambling, for every app, all of the odds - everything - is all handled by them.

<comment><USER3> Thanks kine! Good to see you at the hackathon btw :)

</comment></comment><comment><USER9> This will promote a flood of apps exploring real-money gambling on all platforms. Google and Apple are probably worried on the due diligence on this kind of games, if the odds are respected, if there is no possibility of rigged games, etc etc

<comment><USER3> We actually take this very seriously at Betable. We do due diligence on all games that go live with our platform. There are two reasons: the legal reasons, and because the long-term health of our ecosystem depends on it. If players get ripped off, they won't come back. We're going to make sure every game is legal, fair and fun for those reasons.

<comment><USER4> Does this include checksums of server-side code, or random interval testing?

I am not sure how deeply you can go into your vetting process publicly, but I'd love to know how you can prevent a shadier company from pulling a bait- and-switch to change the odds of their games after you've approved them as a trusted vendor.

<comment><USER3> I can't go into more detail about our vetting process, but I can tell you that game companies cannot change the odds of their game without our approval.

<comment><USER4> Fair enough - I sort of figured you wouldn't be able to comment on it.

You guys have a really smart team, though, so I'm sure you have something awesome figured out. Congrats on the exposure this should net you! :)

</comment></comment></comment></comment></comment><comment><USER10> Lol Big Fish seeking real-money whales

<comment><USER11> yeah the "whale" terminology cracks me up too</comment></comment>


WebRTC is going to break the whole game open anyway - <AUTHOR> http://blog.andyet.com/2013/09/27/webrtc-is-going-to-break-the-whole-game-open

<comment><USER2> WebRTC is to messaging/socket communication APIs as WebGL is to 3D graphics APIs? So big game changer is that web browsers will implement functionality that native applications have had for decades? I don't understand this obsession of running everything inside a web browser with HTML and JavaScript and then declaring it as some kind of revolutionary step forward. What we want to do as developers is make it easy and secure to run applications on end-user desktops. Web browsers are the dominant delivery platform right now (for consumer applications anyway) but that does not mean that it will always be this way or that it is the optimal solution. There are a lot of W3C standards ... are they all really so great? Does anyone remember VRML?

</comment><comment><USER3> A central idea of this article is that

>Google, Microsoft, Apple, and Facebook are truly the new telecoms—each within the last few years has built or bought a communication platform of their own.

Some aspects of this analogy are true, but it's weak overall. Telecom is an extremely capital intensive business. Online messaging is not.* As a result, if telecom companies screw you over, it's hard for a startup to replace them. With online messaging, if a service screws you over, it's relatively easy to switch to a new standard/provider. That feedback mechanism should hopefully keep the messaging platforms better than the telecom platforms.

*and yes, online messaging runs on lots capital, but the capital - fiber lines and whatnot - is already pretty open</comment>


Disable SMT/Hyperthreading in all Intel BIOSes - <AUTHOR> https://marc.info/?l=openbsd-tech&m=153504937925732&w=2 <comment><USER2> Does this mean AMD hyperthreading has a performance + security advantage over currently shipping Intel processors?

Edit: [https://www.amd.com/en/corporate/security- updates](https://www.amd.com/en/corporate/security-updates)

 _> 8/14/18 – Updated: As in the case with Meltdown, we believe our processors are not susceptible to these new speculative execution attack variants: L1 Terminal Fault – SGX (also known as Foreshadow) CVE 2018-3615, L1 Terminal Fault – OS/SMM (also known as Foreshadow-NG) CVE 2018-3620, and L1 Terminal Fault – VMM (also known as Foreshadow-NG) CVE 2018-3646, due to our hardware paging architecture protections. We are advising customers running AMD EPYC™ processors in their data centers, including in virtualized environments, to not implement Foreshadow-related software mitigations for their AMD platforms._

For those on AMD platforms, how do you disable software mitigations for Foreshadow? Is this automatically done by browsers, operating systems and hypervisors?

<comment><USER3> With the newer AMD processors having as many real cores as they do, does the cost-benefit analysis of HT/SMT change? I read in a comment here a few weeks ago that turning it off on the newer AMD CPUs can yield better performance because of improved cache-coherency on some workloads (My memory of what I read might be totally wrong).

<comment><USER4> I'll defer to whatever the benchmarks say of course, but I don't see why HT would affect cache coherency for normal workloads. If you disable HT you'd still have the same number of threads/processes running on the system, so you still have to schedule the same amount of work and do the same number of context switches.

<comment><USER5> Suppose you have 8 runnable processes and a 4 core / 8 thread system.

With 4 execution units, 4 processes run at a time, and they all get swapped out every scheduling tick (losing all their cached lines). OTOH each process gets a full measure of cache to use during its slice.

With 8 execution units, 8 processes "run" at a time, they interleave based on stalls and CPU resources, and the OS doesn't need to reschedule anything every tick (so they hopefully keep their cache lines hot). But each process gets a half measure of cache to use.

In reality, code tuned to use a full measure of cache will be better off matching the number of processes to the number of execution units available, so you'd run half the number of processes with HT disabled. And cache-tuned code tends to fall off a performance cliff when it exceeds cache available, so it may easily run more than twice as fast, depending on the work.

The win from HT depends on most code not being tuned to full measures of cache, and having a lot of memory stalls or other heterogeneous work that other work can fit into. And most code is like that. Cache tends to have a declining marginal return - you have to add exponentially more cache to avoid cache misses - [https://en.wikipedia.org/wiki/Power_law_of_cache_misses](https://en.wikipedia.org/wiki/Power_law_of_cache_misses) .

</comment></comment></comment></comment><comment><USER6> For anyone (understandably) confused about the attacks and mitigations related to L1TF, I've found the linux kernel documentation on the mitigations[0] to be a great resource.

One interesting thing is that to mitigate L1TF hyperthreads only need to be disabled if you are running VMs, the userspace mitigations are effective regardless of HT status. However, there's a catch, you can leave hyperthreading enabled if you disable the Extended/Nested Page Table virtualization feature. However it is noted that this will result in a significant performance impact.

[0] [https://www.kernel.org/doc/html/latest/admin- guide/l1tf.html](https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html)

However this does not mean that HT with VMs is totally secure, as there may be more vulnerabilities relating to HT yet to be disclosed/released as alluded to by Theo. (For context, see the previous discussion [1] around the Lazy FPU switching vulnerability where Theo made the decision to enable mitigations in OpenBSD[2] prior to the public disclosure of the bug based (Theo/OpenBSD was _not_ party to the embargo))

[1] [https://news.ycombinator.com/item?id=17304233](https://news.ycombinator.com/item?id=17304233)

[2] [https://marc.info/?l=openbsd- cvs&m=152818076013158&w=2](https://marc.info/?l=openbsd- cvs&m=152818076013158&w=2)

</comment><comment><USER7> We have disabled Hyper Threading(HT) on all public facing servers(running OpenBSD). However, our compute nodes running Linux kernel are benefiting about 80 to near 100% boost for specific scientific workloads. So, we run our INTERNAL NETWORK ONLY compute nodes with HT on. In places where security is not primary concern, why not make use of HT for extra efficiency?

Think and plan before you blanket disable HT on all servers running intel CPUs...

If an attacker is able to run any code on these private servers, I have bigger problems to deal with than HT as attack vector..

<comment><USER8> If you fully trust the software you're running, I see no reason to disable HT. At this point, I don't think I'd have it running on anything publicly facing, though. That said, I still have it enabled on my work PC & home PC.

<comment><USER9> Do you fully trust all the JavaScript you run?

<comment><USER8> Nope, which is why I generally have JS disabled.

</comment></comment></comment></comment><comment><USER10> Does that mean hyperthreading is effectively unpatchably insecure?

Cloud Providers are gonna have a bad time if this is true.

<comment><USER11> You can give both hyperthreads in a physical core to the same tenant, no?

Scheduling different VMs to run on the same hyperthreaded core at once seems like it can't be good for either VM's performance, even if there were no security concerns. Hyperthreading is much more useful for running multiple threads of the same app, accessing similar instruction caches etc.

(There's also a question of safety _within_ the VM, but a huge number of cloud users are running effectively one user within their VM.)

<comment><USER10> The latter question is very important indeed. If you for instance render websites in your vm they, if i understand correctly, can potentially read secrets from other processes, like db credentials and other stuff...

If the only real solution is to turn off HT/SMT that, seen positively, should net us a lot faster VMs then...

<comment><USER12> >If the only real solution is to turn off HT/SMT that, seen positively, should net us a lot faster VMs then...

you also doubled the cost of each VM (in terms of cpu), but you didn't double the performance of each VM, so it's a net negative.

<comment><USER10> It might be Intel in the end having to pay that cost...

</comment></comment></comment></comment></comment><comment><USER11> > _SMT is fundamentally broken because it shares resources between the two cpu > instances and those shared resources lack security differentiators._

I thought the root of one of the Foreshadow problems was that caches are shared _across cores_ , and therefore even with hyperthreading disabled, you still gain information about a process on another core. Am I misinterpreting it?

It does seem like the paranoid thing to do is that each _socket_ gets to be used by only a single user. (I half-jokingly suggested at work that we replace our internal cloud with a Beowulf cluster of Raspberry Pis...)

It also seems like you could design OSes in a way which is more robust to this, e.g., certain cores are only for the kernel and processes running as root, and system calls are inter-processor interrupts, so privileged kernel (or userspace root) data doesn't go into untrusted caches at all.

<comment><USER13> Foreshadow is caused by the L1 cache which is not shared across cores. It may be only a matter of time before L3 attacks are discovered but I don't know of any today.

<comment><USER11> Oh - I forgot the L1 cache isn't shared across cores. That makes sense, thanks.

</comment></comment></comment><comment><USER14> Scary looking headline on a discussion forum with instructions to perform a task that the average user would not really understand, with no explanation of attack vector or even consequences for any user who doesn't want to take the time (and energy, frankly at this point) to follow security news.

I'm pretty close to not caring anymore. I hope somebody figures out how to at least fix the security news infrastructure, if fixing security is still a ways off.

EDIT: Scratch that, I assume attack vector is a browser since they mentioned JavaScript.

<comment><USER15> This is not security news at all. This is Theo de Raadt's personal E-mail sent to the OpenBSD development mailing list for system developers. It is never intended for the consumption by the general public.

<comment><USER14> Hah, okay. I'm not familiar with OpenBSD so I didn't know who Theo was. Well, that would be good to know, let's say, on the HN headline.

Thanks!

</comment></comment></comment><comment><USER16> Does anyone know the best way to disable hyperthreading in OSX? The only thing I found was this: [https://www.whatroute.net/cpusetter.html](https://www.whatroute.net/cpusetter.html)

<comment><USER17> I'm not sure if you are running OSX on a multi user server, but is this something we need to worry about on our laptop? It's not clear to me what needs to be done in a single user situation.

<comment><USER2> The OpenBSD post references possible JavaScript (browser) attacks.

</comment></comment></comment><comment><USER18> Maybe this is what finally gets me to upgrade from my ~2012 i7-3770. Not because of performance improvements, but to avoid performance degradation from all these security patches...

<comment><USER19> I'm not so sure I'd bother, more of these attacks keep coming out, odds are you'll just buy a CPU that'll be vulnerable to the next one. Maybe AMD would save you from some.

</comment></comment><comment><USER20> FTA

>>> We are having to do research by reading other operating systems.

Si Intel cooperates with business partners like apple/windows and not with open source. Does it mean that Apple and Windows can claim to be more secure because they have access to the information needed to fix Intel's issues ?

<comment><USER13> Intel cooperates with organizations that obey embargoes and don't badmouth their partners in public, like Red Hat, Canonical, and probably the Linux Foundation. Intel does not cooperate with OpenBSD.

<comment><USER21> OpenBSD has never disobeyed an embargo. Argued for them to be reduced, criticized them, but not disobeyed them.

<comment><USER22> i'm pretty sure this is not true: The most recent example i remember is: [https://lobste.rs/s/dwzplh/krack_attacks_breaking_wpa2#c_pbh...](https://lobste.rs/s/dwzplh/krack_attacks_breaking_wpa2#c_pbhnfz)

<comment><USER23> This is common misinformation. Even in this case, OpenBSD did not break the embargo. After protesting, they received the permission of the researcher to publish:

      Note that I wrote and included a suggested diff for OpenBSD already, and that
      at the time the tentative disclosure deadline was around the end of August. As
      a compromise, I allowed them to silently patch the vulnerability.

([https://www.krackattacks.com/#openbsd](https://www.krackattacks.com/#openbsd))

</comment></comment></comment></comment></comment><comment><USER24> There are some comments here talking about the possibility of getting better performance without HT. Here's an article from a test (on Intel only) of that theory: [https://www.phoronix.com/scan.php?page=article&item=intel- ht...](https://www.phoronix.com/scan.php?page=article&item=intel-ht-2018)

In the end, the conclusion is: "Long story short, Hyper Threading is still very much relevant in 2018 with current-generation Intel CPUs. In the threaded workloads that could scale past a few threads, HT/SMT on this Core i7 8700K processor yielded about a 30% performance improvement in many of these real- world test cases."

</comment><comment><USER25> Will be interesting to see what Apple does about this in their next software update. I can't imagine many people will be happy if the next software update forcibly disables hyper threading.

(For those who aren't familiar with Apple devices, Apple don't expose settings like this to a user, which are usually available in the BIOS on a PC)

<comment><USER26> Will the loss of HT have apparent consequences on most Intel-based Apple hardware? Very few of them are servers under constant multithreaded load, throughput-oriented. I suppose almost all Macbooks and most Mac Pros will not visibly slow down.

<comment><USER27> I would expect disabling HT on dual-core systems to have a noticeable performance impact on the desktop.

</comment></comment></comment><comment><USER28> It's not clear that he's using "SMT" to refer to AMD specifically as he goes on to talk about "Intel CPUs" and disabling it in "Intel BIOS". Does the Zen architecture have the same issue?

<comment><USER29> I don't think AMD is impacted by Foreshadow

</comment></comment><comment><USER30> [ marc.info not responding for me, found post linked elsewhere. Edited due to markdown.

URL: [http://openbsd-archive.7691.n7.nabble.com/Disable-SMT- Hypert...](http://openbsd-archive.7691.n7.nabble.com/Disable-SMT- Hyperthreading-in-all-Intel-BIOSes-tp349646.html)

Here it is: ]

\---

Title: Disable SMT/Hyperthreading in all Intel BIOSes

Posted by Theo de Raadt-2 on Aug 23, 2018; 11:35am

Two recently disclosed hardware bugs affected Intel cpus:

\- TLBleed

\- T1TF (the name "Foreshadow" refers to 1 of 3 aspects of this bug, more aspects are surely on the way)

Solving these bugs requires new cpu microcode, a coding workaround, _AND_ the disabling of SMT / Hyperthreading.

SMT is fundamentally broken because it shares resources between the two cpu instances and those shared resources lack security differentiators. Some of these side channel attacks aren't trivial, but we can expect most of them to eventually work and leak kernel or cross-VM memory in common usage circumstances, even such as javascript directly in a browser.

There will be more hardware bugs and artifacts disclosed. Due to the way SMT interacts with speculative execution on Intel cpus, I expect SMT to exacerbate most of the future problems.

A few months back, I urged people to disable hyperthreading on all Intel cpus. I need to repeat that:

DISABLE HYPERTHREADING ON ALL YOUR INTEL MACHINES IN THE BIOS.

Also, update your BIOS firmware, if you can.

OpenBSD -current (and therefore 6.4) will not use hyperthreading if it is enabled, and will update the cpu microcode if possible.

But what about 6.2 and 6.3?

The situation is very complex, continually evolving, and is taking too much manpower away from other tasks. Furthermore, Intel isn't telling us what is coming next, and are doing a terrible job by not publically documenting what operating systems must do to resolve the problems. We are having to do research by reading other operating systems. There is no time left to backport the changes -- we will not be issuing a complete set of errata and syspatches against 6.2 and 6.3 because it is turning into a distraction.

Rather than working on every required patch for 6.2/6.3, we will re-focus manpower and make sure 6.4 contains the best solutions possible.

So please try take responsibility for your own machines: Disable SMT in the BIOS menu, and upgrade your BIOS if you can.

I'm going to spend my money at a more trustworthy vendor in the future.

</comment><comment><USER31> Makes sense from the adage that more complexity means more exploits.

</comment><comment><USER15> Does anyone know the best way to disable hyperthreading on Linux?

<comment><USER15> Okay, I found it. A SMT knob was added alongside in the L1TF fixes.

        /sys/devices/system/cpu/smt

        /sys/devices/system/cpu/smt/active

        /sys/devices/system/cpu/smt/control

        active:  Tells whether SMT is active (enabled and siblings online)
        control: Read/write interface to control SMT. Possible

        values:

        "on"		SMT is enabled
        "off"		SMT is disabled
        "forceoff"   	SMT is force disabled. Cannot be changed.
        "notsupported"	SMT is not supported by the CPU

        If control status is "forceoff" or "notsupported" writes are rejected.

</comment></comment><comment><USER32> I would love to but how in the world do I do this when I'm using Windows and Lenovo doesn't give me an option in the BIOS?

<comment><USER33> One way to achieve something similar would be via a software tool which would set the process affinity to only run on real cores.

Or you could only run Chrome (untrusted JavaScript) on core 2 and 3, and run the app that has your secrets on core 0 and 1. (It is my understanding that 2k cores are real, and 2k+1 is their matching, "virtual" core) This way you get both hyperthreading and security. I'm not a security expert though.

[https://bitsum.com/docs/pl/Using%20the%20GUI/using_the_gui.h...](https://bitsum.com/docs/pl/Using%20the%20GUI/using_the_gui.htm#default_affinities)

<comment><USER32> I'm not sure it would be that easy since I believe e.g. I/O can go through the System process (or other processes even), which has full affinity. We'd likely have to set thread affinities for all processes/threads. But then it would clash with manually-set affinities, and I'm also not sure if it would have worse performance than actually disabling hyper-threading or not.

Right now I'm looking at what making a UEFI application to disable HT before boot might involve... not sure if that's too late in the boot process or not.

</comment></comment></comment><comment><USER34> So, realistically, how much performance did your average OpenBSD server just lose from following this mitigation?

<comment><USER35> Performance is not a top priority for OpenBSD.

If you read [https://www.openbsd.org/goals.html](https://www.openbsd.org/goals.html), the word "performance" does not appear.

<comment><USER36> I think this may be missing the point of the grandparent comment; rather than interpreting it as an accusation of OpenBSD sabotaging its users' performance, I think we're all just curious at the relative importance of hyperthreading for real-world workloads, on any OS, in grim anticipation of the potential worst-case scenario where hyperthreading's security woes continue to worsen and worsen.

</comment></comment></comment><comment><USER37> Does anyone know if disabling SMT has had an effect on vmd(8) performance in -current?

</comment><comment><USER38> Is not it better just to disable Javascript instead of cutting your CPU performance?

<comment><USER39> Some of us like using websites.

<comment><USER40> Many actually improve when you disble JS

<comment><USER41> But a non-zero amount will completely break or not show anything at all (even though the actual content to be displayed is just static text and doesn't need _any_ JS)

</comment></comment></comment></comment><comment><USER42> Can I ask, Is Intel going to sue us if compare benchmarks with SMT to no SMT? Like they mentioned about the microcode benchmarks. Intel is in trouble.

</comment><comment><USER43> Eh hyperthreading seems to be questionable anyway. On all the application benchmarks I've run except for a few, it slows things down.

<comment><USER44> Care todo share ir provide more specific info?

</comment></comment><comment><USER45> I've long felt that there's something less than half-baked about the multi CPU architecture we're currently using. The hacky contortions HFT coders have come up with to avoid things like False Sharing strike me as a big red flag.

[https://mechanical-sympathy.blogspot.com/2011/07/false- shari...](https://mechanical-sympathy.blogspot.com/2011/07/false-sharing.html)

How about an architecture more like Erlang's, where you have independent processes with their own CPU core, where each has their own memory, but where you have much faster communications supported at lower hardware levels? Why not have a multi-processor architecture designed for direct support of Hoare CSP-inspired languages?

Hypercube topology: [http://web.eecs.umich.edu/~qstout/pap/IEEEM86.pdf](http://web.eecs.umich.edu/~qstout/pap/IEEEM86.pdf)

<comment><USER13> That's like the Cell processor. For every year that you program for Cell you need at least two years of therapy.

<comment><USER45> If you're going to change the substrate or paradigm, then you need to do a dynamite job of supporting your users. Sony did not do that.</comment></comment></comment>


Europe's old money mingles with tech startups - <AUTHOR> https://www.wsj.com/articles/europes-old-money-mingles-with-tech-startups-11583511875 <comment><USER2> Unfortunately they tend to bring old-money sensibilities along with them, meaningful employee equity is rare even at executive levels in European startups.

<comment><USER3> employee equity in particular in startups isn't really as unambiguous as the term suggests. Given how many startups die out you may as well interpret it as a creative way for company owners to shift risk on employees. After all, the failing startup is the norm, by a huge margin. And one step further these days it looks like going public is then the next step to load risk off from employees and founders onto retail investors.

to be honest I'm not really too sad that this culture hasn't really taken hold over here. Most 'old money' in Europe isn't old billionaire heirs anyway but the small and middle-sized business who treat their employees quite well, employ them for decades, don't lay them off at the first sign of trouble and so on.

If tech in Europe is looking for a business culture I don't mind them going to the old money and family business in the German South rather than Silicon Valley.

<comment><USER4> Right now, base salary is typically higher in the US for software engineers than in Europe, _plus_ equity is better.

<comment><USER5> You also need to pay Americans more to compensate for health insurance, insane holiday policy, poor worker protections, huge student debts that can't be bankrupted out of, and so on.

<comment><USER6> I have worked for almost 15 years in SV at multiple tech companies (FANG and non-FANG) and at all the places, I have seen health insurance, at least 15 vacation days per year, 9-10 holidays and generous salary to pay off the student debt.

So your comment just reads like a somewhat inaccurate meme.

<comment><USER7> You are comparing the best the US has to offer with the standard for European employees.

Here in Germany, you have health insurance, no student debt, multi-month paid maternity/paternity leave, 30 days of vacation per annum, paid sick leave, multiple bank holidays.

Notice how I didn't specify a company or location?

Because this is the standard for every full-time employee in the whole country, irrespective of location or profession.

The German model has two draw-backs: Pay never reaches the levels in the US and it is relatively hard to land your fist job. Also, some gig economy jobs fall through the cracks.

For an honest comparison and a complete picture, you can't take a software engineer from SV and compare it to a software engineer from Berlin.

At least not just.

For a complete picture, you should also compare a software engineer from Boise to a software engineer from Düsseldorf. And a call center employee from Oklahoma with a call center employee from Mecklemburg-Vorpommern.

The ones who "made it" will always be better off in the US. But everybody else, not so much.

<comment><USER6> If you want to compare worst or average of the US, why not compare it to the worst or average of Europe - say, Romania or Spain or Moldova or Greece? Why compare it to one of the best places in Europe?

Or, to use your words - the Germans will always be better off in Europe. But everybody else, not so much.

<comment><USER8> The EU is probably not as homogenous as the US, still, in Bucharest, yearly salary is 20kEUR (92kRON) for your average Sw Eng (not to mention cost of living would be low).

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> > Europe has a higher proportion of fortunes controlled by heirs, not > innovators, than do the U.S. or Asia

I wonder what they consider to be heirs? The difference can be just one generation.

<comment><USER10> It even gets murkier when you consider people like Susanne Klatten, one of the heirs of BMW. Definitely a heiress, but then she did pretty well with what she inherited. So, entrepreneur or heir? Or both?

<comment><USER11> Maybe there's more than the English Wikipedia article about her, but it doesn't show much more than continuing existing stakes.

</comment></comment></comment><comment><USER12> [https://archive.md/JITqe](https://archive.md/JITqe)

</comment><comment><USER13> I feel like there's been more dumb money going to tech, lately. Some PE, pensions, Saudi via Softbank's Vision Fund. Now add Europe's old money.

</comment><comment><USER14> I think it's really positive that these companies / networks open themselves for startups, still it would be better if they would try to change their companies so that all startups have a fair chance to work with them. This would be beneficial to them as they would have a larger sample to choose partners from. Right now these VCs present maybe 50-100 companies to them (LaFamigila for example has 34 investments as of now, I think), which is still a very biased and restricted sample, so they are almost surely missing out on some great innovation opportunities.

Doing B2B sales to large enterprises in Germany I found that it takes a lot of side channels & social engineering to sell into large organizations as a startup. Maybe this is just part of the "game" but I can't stop thinking that those companies miss out on so much innovation because they simply don't have good processes in place to systematically and fairly build relationships with small, innovative tech companies.

Founders from different regions, how is your experience with this?

<comment><USER15> The thing with big companies is that you have to fit in whatever reality they are telling themselves in that moment. Then you need big credibility points to give them a way to say you were the best at xyz, pity that you failed. They care 0 about innovation, unless they are telling themselves they need it but even then it's likely to require the small startup to have accumulated a lot of credibility first.

</comment></comment><comment><USER16> It seems that the old money has finally got the angel investing bug. It will become a flood if the returns are there or shrivel up if not.

<comment><USER17> Depends. I know wealthy Europeans with inherited wealth and most of them are putting it into real estate.

<comment><USER18> Yeah you can pretty much forget about wealthy Europeans doing anything at the speed of any business cycle. They are scared oldest sons managing a family office who have no idea what they are doing aside from breathing and making a baby in wedlock.

These people will put down $1 million dollar deposit to make it seem like they are serious about completing a deal, and will just let the timeline expire forfeiting the entire deposit!

Forget about the negative interest rates. Unless you can tap into the high yield corporate debt market, then dealing with these rich people is hopeless.

Source: Been around a lot of wealthy Europeans that talked a big game. It is the most ridiculous stable business climate on Earth. I do not recommend, especially for tech.

<comment><USER19> I don't know anything about that world...

But honestly that sounds reasonable to me. Old money wants to keep their money, so they are risk averse. New money is in the process of making their fortune so they are more interested in betting big.

<comment><USER18> Yes that is precisely what is going on, but they know they want to get out of real estate and yachts and public equities, and want to feel close to the action.

Scared money doesnt make money.

</comment></comment></comment></comment></comment><comment><USER20> have a feeling grifters are going to take advantage of this</comment>


Aziz Ansari releases DRM-free standup special for $5 - <AUTHOR> http://azizansari.com/ <comment><USER2> It's interesting that all this has come from comedians. Perhaps not surprising, given that TV shows are owned by networks, and bands are largely indebted to record labels- I never thought about it before, but a comedian is far more of a "free agent" than anyone else- for instance, Aziz is one of the main stars in a hit NBC comedy (well, as much of a hit as NBC gets these days), but he's free to do stuff like this as much as he wants. I very much doubt a band would be able to get away with that.

I've been a fan of Aziz for a long time- I'll be buying.

<comment><USER3> It has begun quite a while ago with podcasts. Adam Carolla runs a whole podcast production shop in addition to his own hugely popular show and even some spin-offs, then there's the Nerdist (same thing), Joe Rogan has a popular show, Doug Benson, Bill Burr and many others.

Comedians just seem more... entrepreneurial. It would be interesting whether it's because there's less structure (existing or required) in the comedy world, or because comedy attracts more independent performers.

<comment><USER2> There's probably a lot to be said for comedians generally acting alone or in pairs. Even when you get a group of four together you're considerably less flexible.

<comment><USER4> I did stand-up comedy for a while. Allow me to offer my own experience about the entrepreneurial streak you're seeing from the comedy sector:

1\. As a comedian, you have no choice BUT to get creative. Nobody is going to give you stage time as a no-name comic so your choices are either (booked) open mics or starting your own show.

2\. There isn't an "agent" culture for comics like there is for bands or actors. You write your own material. You're alone on stage. You're alone every step of the way.

3\. The money stinks as a comic. I can remember being paid for a show in food. FOOD. And there were limits on what I could order. At a mexican restaurant. Since there isn't a lot of money in comedy shows, every dollar you spend on getting to/from a show or paid to someone to help you book a show is almost a eat-or-pay-rent type of decision. If you go to a comedy show, buy a CD from the comic after the show. It's likely that 5 CD's sold puts more money in the pocket of the comic than whatever they earned from the club.

4\. Finding another comic to share expenses is almost a losing proposition from the start. Unless you're on stage together as part of an act, it's impossible. Only headline acts can bring their own opener/feature and you aren't a headline act.

SO, to sum up, the money stinks in comedy, you can't easily achieve scale, and you need to be your own writer/agent/booker/show-runner to make a living.

<comment><USER5> "The money stinks as a comic" A classic case of oversupply perhaps. Far too people quitting their day jobs to do stand-up full-time because someone in the office laughed at their jokes.

<comment><USER6> Everyone learns to perfect their craft through constant failure. Musicians/artists have the luxury of failing over and over while in the isolation of their bedroom. Comics have no choice but to fail repeatedly in front of a live audience, and nobody wants to pay you while you're learning to fail.

<comment><USER4> The common wisdom for comics learning is that your first 100 sets don't count. You basically count as a win any time you are able to stay onstage for the required amount of time.

To paraphrase: your first 100 jumps off the board count if you hit water.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> I know Louis CK was successful in this model, but at the time I expressed hesitation about others following his lead.

He made a good amount of money, but given Louis CK is probably the most popular working comedian today, it didn't seem like ENOUGH money to justify the entire business model (assuming no one else would make as much doing the same).

But now Aziz Ansari is doing it. Jim Gaffigan has announced plans to do so as well.

I know we'll have to wait for their numbers to come in to find out if Louis CK was a fluke or a trend, but I'm really hoping I was wrong and it's the latter.

If this is the future of selling online content, sign me up.

<comment><USER8> I think you were wrong.

Considering Jeff Foxworthy has sold over 15 million albums[1], recently had a primetime show in syndication, and managed to make three of his drinking buddies into 8-figure yearly draws, I would say Louis CK is not even close to the most popular working comedian around. He has a rabid base, yes, but I would argue he's not even a mainstream comic (ie I would bet 9/10 people off the street would not know who he is).

I think this bodes very well for those who can mobilize a few hundred thousand people. What I think will be even more interesting is how this will affect those that can only mobilize a few thousand.

[1][http://www.slate.com/articles/arts/number_1/2005/04/jeff_fox...](http://www.slate.com/articles/arts/number_1/2005/04/jeff_foxworthy.html)

<comment><USER9> I've known and enjoyed Louis CK since he first broke on late night TV. Also know less well Jim Gaffican but I've seen a few of his specials. Even seen Foxworthy's 'drinking buddy' Ron White live.

Am I the only one here who has never even heard of Aziz?

<comment><USER10> I'd never heard of him and didn't particularly like the preview. I still paid the $5 because I would like to see this business model take off and the show _might_ be funny. To be honest, I didn't find Louis CK's $5 offering all that funny.

</comment></comment></comment></comment><comment><USER11> From the confirmation e-mail:

"OH SHIITTTT!!!! Thanks for buying my new comedy special!!! THAT $5 IS MINE SON. Just kidding, but seriously it is."

<comment><USER12> The profanity in the order form was refreshing as well:

"Confirm email (Don't Fuck This Up)"

I'm guessing they didn't AB Test profane and clean copy on the forms.

<comment><USER13> I'm also guessing that someone offended by the profanity would likely not be a happy customer. Which highlights an interesting potential limit of A/B testing for conversions. It may take some time after the point of sale to determine whether you've created a happy (and therefor loyal, returning) customer or not. So imagine in this case that profanity is likely to turn away 2 customers but presence of profanity will tilt 1 additional conversion -- so overall conversion rate of non-profane form could be better (2 vs 1 on some margin). But what if the non-profane converts are ultimately dissatisfied with the product whereas the the profane convert is likely to be happy? Perhaps the long term relationship with the latter customer is worth more than the two unhappy customers, even though a naive A/B test would push you to limit profanity for the short term conversion benefit.

<comment><USER14> This is something about A/B tests that always bothers me. They always seem to assume that if variation A is best for a proxy stat (signups, views, etc) it's also best for what I'm really interested in (longterm value). I still use them though - not sure there's much you can do about it anyways.

<comment><USER15> On the other hand, if you can eventually measure long-term value and you can keep track of users over a long enough cycle, then you can eventually run an A/B test on long term value. This depends on accurately tracking users, having a quantifiable way to calculate long-term value, and eliminating other sources of error, but it should be absolutely doable.

You can also measure how well your proxy stat does correspond to long-term value and adjust your expectations accordingly.

</comment></comment></comment></comment></comment><comment><USER16> "I've deliberately hired non-Indian people to run the tech support stuff as to buck stereotypes"

<http://azizansari.com/faq>

</comment><comment><USER17> Aziz Ansari is so funny. I loved him as the fruit vendor who hates Australians on Flight of the Conchords, and his Randy character was pretty much the only good thing about that train wreck Funny People. Nice to see him succeeding via Parks and Recreation and el Internet.

</comment><comment><USER18> Is anyone here already working on an easy to customize template for these guys to build websites that easily charge for DRM-less shows?

I remember last time someone linked to something similar, but they had DRM and draconian payment methods (no paypal etc).

<comment><USER6> It seems like youtube or vimeo is missing a really big opportunity here.

<comment><USER19> Why should a comedian have to submit to a behemoth like writers do? To get on iPad or Kindle or Nook, you have to go through their stores and risk rejection or wind up giving them 30% (a fee I do not think is in any way justified just for hosting; they damn well can't _market_ anything, just provide one listing out of jillions).

I say more power to the comedians. Submit to no one, share with no one. I wish writers could do the same.

<comment><USER18> If you set your video to unlisted on youtube (only those with links can see it), then setup a paypal/google wallet account that automatically mails the link when someone pays you. Then all you'll be paying for is the paypal/google wallet fee.

Wouldn't have a sexy landing page. But would also save ~ $200k (how much Loius CK said he payed for his website).

<comment><USER20> But once that link is online, anyone can reach to it. Even the people who didn't pay for it. Correct me if I'm wrong. They need a link generator for every payment. Or have the video show up in my not yet developed "Paid Videos" section on my youtube profile.

<comment><USER18> I would need to share my link with someone else. Just like I can share the non DRM video I downloaded from Loius CK. The whole concept is that people will rather pay for it even if they can get it for free, if they think it's worth it. And it has been working so far.

So you shoulnd't need to create artificial blocks to try to stop people from pirating. I mean, look at the humble bundle, you can just click that you don't want to pay for and download the games. But people pay anyway and they make millions off it. So why would you create artificial blocks that pisses people off?

<http://www.humblebundle.com/>

</comment></comment></comment></comment></comment></comment><comment><USER21> If this is your first time hearing of Aziz Ansari and you enjoy his comedy you might also want to check out his show 'Parks and Recreation' on NBC.

It's one of the best comedies on TV right now. I'd even say that it is approaching Arrested Development-levels.

</comment><comment><USER22> This is a great trend which I'm sure more artists would follow suit.

But it puzzles me why did they build their own digital delivery solution from scratch instead of using something like GumRoad?

<comment><USER23> Probably because none of those startups have serious traction outside of hype in tech circles. Once one big player uses it, it can be seen as legit and others may follow suit.

</comment></comment><comment><USER24> $5 AND DRM Free? Could you imagine...stealing a movie that costs $5 and is DRM free? That'd be like stealing a movie that's $5 and DRM free! There's no other way to complete that analogy because that's the shittiest thing you could ever do!

</comment><comment><USER25> I'd like to see someone set up a turnkey solution, soup to nuts (i.e. website, paypal or other payment service, content delivery) with some simple pricing model. It would then be easy to standardize in terms of format, TOS, etc.

</comment><comment><USER26> I just bought it and was redirected to an error page even though the paypal went through, though a few minutes later I received an email with watching/streaming details. Just an FYI for anyone else this may happen to.

<comment><USER27> I paid, received no email at all, then got a support reply the next day (today), which successfully let me log in, but left me with no downloads and no streams. Incredibly shitty UX. Anyone else encounter problems?

As someone who bought both this and the Louis CK download, the difference in UX is huge. I wonder to what extent my experience is not unique _and_ to what extent the success of Louis CK's experiment was due to the seamlessness of their website experience. These guys should probably using Stripe; I'm beyond impatience with PayPal.

</comment></comment><comment><USER28> This is the model of new media for any type of artist.

Create enough value that you establish yourself as an expert in whatever your art is and then exploit secondary pursuits. I constantly refer back to Diddy getting famous with music, but also establishing himself as a partying/lifestyle expert and then making a killing on Ciroc. 50 cent did it with vitamin water and fitness, Athletes do it with endorsements.

It is the Tony Robbins model. His books didn't make him rich but they allowed him to charge 500k a speaking gig.

Your initial art is your marketing that develops your right to charge money for future art.

<comment><USER19> >>>It is the Tony Robbins model. His books didn't make him rich but they allowed him to charge 500k a speaking gig.

And Robbins failed with Breakthrough on NBC. So not all success is guaranteed even when it seems to be.

<comment><USER28> Agreed that success is not guaranteed. The other two factors are context and success expectation. You can't win NBA MVP and expect to successfully go and sell cookware (unless you are known for being a cook outside of the league) it is not what people trust you for. You also need to look at the size of the audience of your accepted expertise. Tony Robbins is known, but apparently not enough to float a primetime television show.

<comment><USER19> >>>Tony Robbins is known, but apparently not enough to float a primetime television show.

Which actually surprised me, given his infomercials made his fame.

</comment></comment></comment></comment><comment><USER29> Everyone here keeps saying to turn this into a platform. Though I agree that developers can capitalize on this opportunity, there's a road we can go down that in my opinion may make things too crowded. Yes, it would be great to have a central hub where I can browse through popular comedian's trailer videos and then download my favorite one for $5 DRM free. But, I think that'll just create so much competition between the comedians that they won't see the level of traction that Louis C.K. saw. Then again, it's a tradeoff and you can't really have both can you?

</comment><comment><USER30> Haha! Never heard of him before, but I'm buying based solely on this: <http://www.youtube.com/watch?v=b92vAucJg2Y> (36 seconds)

<comment><USER31> Never heard of him either but looks like my flight tomorrow just got less boring

</comment></comment><comment><USER32> I just purchased it. That was simple as possible (enter e-mail in box, redirected to PayPal, redirected to page with stream and download buttons). There's a streaming limit (3 views) and download limit (5 downloads).

The download is hosted on Amazon S3 where I'm barely getting 500k/s. The stream and download limits would then limit his delivery costs to ~$1.50 per customer with S3's bandwidth pricing. PayPal is taking out $0.41. He could be getting as little as $3 of each $5 for the special.

<comment><USER19> >>>redirected to PayPal

Which allowed you to buy it -- _for now_. See, there's a weak link in that chain. Just two weeks ago there was a huge uproar over PayPal rejecting payments for books they unilaterally and unjustifiably deemed "offensive."

And they can do that to any comedian too.

<comment><USER32> So can any payment processor, direct or third party. Right up to the top, with Visa and MasterCard declining to process for Wikileaks. Your point?

<comment><USER19> Wikileaks is not content. That's more political than anything else, with the Patriot Act used to threaten the processors.

<comment><USER33> Sounds like a good idea, lets stop all transactions going to any organization that might have political goals.

</comment></comment></comment></comment></comment><comment><USER34> Some startup needs to aggregate, give cloud storage, and iPad/roku/boxee apps for these things.

<comment><USER35> You mean like iTunes?

Eliminating the middleman is the whole point of these people that advertise and sell their DRM-free shows straight from their websites.

<comment><USER36> Eliminating the middleman is _not_ the point. Avoiding the negative consequences of having a middleman is the point, and eliminating the middleman is merely a means to that end. If you can be a middleman without (or with fewer of) those negative consequences, then you're working toward the same goal.

<comment><USER37> You're splitting hairs. Louis CK's special raked in more than a million dollars. The website cost him about 30k to produce. iTunes would have sucked up TEN TIMES that.

<comment><USER36> I did not say that iTunes was the answer. I said that a startup that offers inexpensive distribution services to entertainers and artists using this low- price, no-DRM model could be the answer.

</comment></comment></comment></comment></comment><comment><USER38> It's interesting to compare the economics of comedy vs music. I'm much more likely to enjoy a song the second, third and fourth time I hear it. I might enjoy the second time hearing a joke, but it goes downhill pretty fast after that. I'm guessing this is normal.

If a comedian's routine is recorded and widely disseminated, a lot of people might become fans and want to see him/her, but they're probably not going to want to see the same jokes again. The comedian will need to come up with new material.

Writing and perfecting a good comedy bit might be comparable in time and effort to writing and perfecting a good song. Anyone know?

How do these considerations affect which business model works best for different types of artists online?

<comment><USER39> If you get a chance, check out "Talking Funny", a very nerdy but informative DVD of conversations between Jerry Seinfeld, Louis CK, Chris Rock, and Ricky Gervais. In it they discuss this very subject.

Seinfeld says his method is to repeat material for years before throwing it out. Chris Rock sees his comedy specials schedule as the equivalent of an album ("like Prince"), so every tour he does completely new material. Gervais lands in the middle - enough freshness to keep things interesting but also enough old stuff to keep the crowds happy. Louis CK is constantly changing his act - if I remember correctly he says he rarely keeps a joke for more than a few shows.

<comment><USER38> Cool, thanks for that.

I realize that this isn't going to be a new subject for anyone in an entertainment business. But as a programmer, it's not something I've thought about nearly as much.

Tolerance for old material probably varies by audience, too. Those catchphrase comedians get amazing mileage out of unbelievably scant material; enough to become fodder for other comedians. ( [http://www.nbc.com/saturday-night- live/video/comedy-tour/131...](http://www.nbc.com/saturday-night- live/video/comedy-tour/1313758) )

I wonder if these big-splash events where a comedian sells a clip DRM-free for a simple payment might be better suited to comedy than to music, given the more time-sensitive nature of the value of a comedy bit.

</comment></comment></comment><comment><USER20> Fully support the artist and the idea. Gave them my $5. I won't have starbucks tomorrow.

</comment><comment><USER40> Why isn't there a standardized self-publish DRM-free platform for this stuff yet? I have lots of time I want to spend distracting myself, but not enough content to fill it without pirating.

Got a Rails 500 error after paying, but just refreshing the page got me through.

The page has a humans.txt, but it looks like it's just the blank template(?)

</comment><comment><USER41> Can I borrow someone's copy when you're finished with it?

Update: <http://azizansari.com/tos> says, _All downloads are for use on one device only. You may keep one backup copy of the digital content downloaded from this site on any digital media of your choice._

<comment><USER37> This is hilarious. He is being downvoted for pointing out one of the biggest serious issues with digital content licensing: The problem isn't just the DRM, but also that the licensing agreements usually (always?) destroy the traditional right to resell your book/movie/album when you are sick of it, or just when you would rather that a friend have it (say, when you finish reading a book).

Paying say $15 for an ebook that is nontransferrable, when the physical book is ALSO $15, is an absurdity that I can't imagine will stand the test of time. Our entire human history is on the side of the feeling that when you buy something, you can then give that particular thing away to someone else.

Current legal and market norms don't serve this basic human instinct yet, and it remains to be seen which shifts first. My money is on humans insisting to keep giving things away to their friends.

</comment></comment><comment><USER42> I thought that opening was really similar to Louis CK's $5 standup and that grape drink joke - that's Chappelle (around 1:50 - <http://www.youtube.com/watch?v=NuTjQLfU6Gk>).

</comment><comment><USER43> Interesting note in the terms of service (<http://azizansari.com/tos>)

Linking to our site "You must not establish a link from any website that is not owned by you."

That rules out 90% of social media..

<comment><USER44> Including this thread. Oh my.

</comment></comment><comment><USER45> Much better. Thank you, Aziz.

</comment><comment><USER46> I just put together a directory of these $5 specials: <http://www.comedyspecials.org>

If people think it's a good idea I'll add a bunch of other fancy stuff :)

</comment><comment><USER47> Bah, I bought this and tried to view it on an iPad. After a few failed attempts I try on my computer and it says I've used all my downloads? Boo.

</comment><comment><USER48> Why are people not realising he's doing this as a gesture against DRM as oppose to some kind of business statement. .. he's a comedian.

</comment><comment><USER49> Ok - so we know there's demand from top-level talent. Who's going to make it into a platform?

</comment><comment><USER50> Can anyone confirm the format? I'd like to toss him $5, but only if I can play it on my iPad.

<comment><USER51> DRM Free:

Thomass-MacBook-Pro:~ thomasknowles$ file Downloads/azizansari- dangerouslydelicious-hd.mov.crdownload Downloads/azizansari- dangerouslydelicious-hd.mov.crdownload: ISO Media, Apple QuickTime movie

anyone feel free to step up and add anymore info.

</comment></comment><comment><USER52> Never heard or seen this guy's act but bought it. Love to support such creative gesture.

</comment><comment><USER53> Heh, am I the only one who actually WENT to that tour?

<comment><USER54> No! I saw it here in Toronto and it was hilarious.

Just bought it for $5 to re-watch!

</comment></comment><comment><USER55> Louis CK did it first, but I'm glad Aziz followed!

</comment><comment><USER56> That's great that you can watch a clip on the website! I like Aziz Ansari.

</comment><comment><USER57> I will be interested to see the results of this. My prediction is that it's a novelty and the appeal will eventually drop off when the sales do (IE: people torrent it instead of buying)

<comment><USER58> Why do you think that? Any reason other than the classic idealized economic model of humans as heartless, brainless utility maximizers?

<comment><USER59> Does there need to be another reason?

</comment></comment></comment><comment><USER60> Normally, I'm a huge Aziz detractor but I think this has a shot at being more successful than Louis CK's special. I think Aziz is more popular with children of the Internet.

<comment><USER61> Why are you a huge Aziz detractor? First time I heard of this Aziz guy. Watched the preview, enjoyed it.

<comment><USER62> I'm guessing he's confusing him for the character he plays on Parks and Recreation.

<comment><USER60> Uh no. I've seen Aziz's standup live. I thought it was terrible. But not surprised to see no one else agree - he is really popular with some folks.

</comment></comment></comment></comment><comment><USER63> Now all he needs to do is something funny.</comment>


Argonne National Laboratory awards Cray contract for 180 Petaflops supercomputer - <AUTHOR> http://investors.cray.com/phoenix.zhtml?c=98390&p=irol-newsArticle&ID=2033626 <comment><USER2> To put this in perspective, as of 11/2014, the aggregated computing power of the top 500 supercomputers is 308.9 Petaflops ([http://www.top500.org/](http://www.top500.org/)).

</comment><comment><AUTHOR> Having worked at ANL in the past on other supercomputers, this is an especially impressive and interesting contract. 180PF, even in 2018 is by any definition an absolute monster of a system.

It's also very interesting that the contract was awarded to Cray and not to IBM. ANL's first two supers, Intrepid and Mira, are both IBM Blue Genes.

This must be a very exciting day for ANL and Cray. Congrats!

<comment><USER3> Both Cray and IBM were in the competition for the next supercomputers for:

\- Lawrence Livermore National Laboratory (LLNL)

\- Oak Ridge National Laboratory (ORNL)

\- Argonne National Naboratory (ANL)

IBM won LLNL and ORNL to create the upcoming Sierra and Summit Supercomputers. We heard down the grapevine that ANL wanted an IBM based system as well, but due to the rules of the competition they were given the Cray system.

So the rumour goes anyways, I'm not in the inner circle.

<comment><AUTHOR> That doesn't surprise me at all. The DOE keeps more options open by selecting different vendors for the same generation installs. In the last generation, LLNL and ANL were IBM and ORNL was the Cray.

That said, I'm long since out of whatever circle it was that I was in (certainly not the inner one!) as well so I speak with no authority whatsoever.

<comment><USER4> NERSC just put Hopper, a cray xe6, online for general use last year. It clocks in at only 1.3 petaflops.

</comment></comment></comment></comment><comment><USER5> This title is incorrect. Intel is the prime contractor (for the first time since ASCI Red) and Cray is the subcontractor. [http://energy.gov/articles/us- department-energy-awards-200-m...](http://energy.gov/articles/us-department- energy-awards-200-million-next-generation-supercomputer-argonne-national)

</comment><comment><USER6> For some reason, I don't think it's coincidence that Guangzhou Supercomputer Center got put on the US Technology "Denial List" and Argonne awarded Cray a 180 Petaflop sale.

<comment><AUTHOR> The DOE announced the awards for ORNL and LLNL last November, and there wasn't a similar announcement about then, I don't see why today's is any more relevant.

That said, I doubt it comes as a surprise to the Chinese government, but it's kind of an ironic stick in the eye given what the DOE's mission is (and what the supercomputer at LLNL is used for). I suppose I don't blame the US government as I'm guessing the law prevents US companies from exporting devices used for nuclear weapons, but it's definitely a dick move.

<comment><USER7> Currently, there is only one thing the Chinese supercomputers are good at. Running Linpack. Give it time, and the Chinese will build up the library and experience to fully take advantage of the resource. In the mean time, we're not going to help them much.

The single biggest consumer of Supercomputing cycles is the DOE for Fusion^H^H^H^H^HSuperNova simulations. They'll chew this up in no time. At least this is being located at the LCF, so it will be open to all sorts of academic usage, too.

[ added ] Teach me not to read the news earlier. Not a bright move, cutting off the Chinese from Xeons and the like will only drive them to develop all the tech themselves. While I do not believe we should be handing them tech on a silver platter, it is actually worse to cut them off.</comment></comment></comment>


Introducing the Dropbox bug bounty program - <AUTHOR> https://blogs.dropbox.com/tech/2015/04/introducing-our-bug-bounty-program/

<comment> Stefan-H Bug bounty programs seem to be more and more necessary for a mature security program. I feel like this can span the gamut from resources for responsible disclosure all the way to robust bug bounty programs that offer pay outs. The barrier to entry for the former should be low and something that every organization with security personnel can arguably support, but once you start doing pay outs, the need to run a tight ship in regards to how bugs are evaluated becomes far greater.

<comment><USER2> Yeah, it seems like a welcome evolution.

Back in the old days far too many companies had very poor processes for dealing with security issues. Either it was impossible to give the report (stuck in CS hell), you'd never hear feedback (e.g. fixed, WONTFIX, etc), you may be threatened with criminal/civil prosecution, and you most certainly wouldn't be credited or acknowledged.

Now, if nothing else, at least the process is well documented end-to-end. Even if we ignore the "debate" over compensation, at least all the other problems are now solved. If you throw on top of that a small "thanks" payment for finding the bug, you now not only solve the old problems but you give people a legitimate reason for responsible disclosure beyond morals.

That's really the core of this, doing the right thing should make rational sense as well as moral sense. Then the bug finder has no real reason to do anything except the right thing. The only reason not to disclose responsibly now is spite (or in Google's case a strict adherence to policy no matter how little sense it makes).

<comment><USER3> Unless someone makes a bug bounty program for Dropbox bugs that offers more than Dropbox does. I don't know what the fair market rate for these bugs are, but it could well be higher than what Dropbox is offering.

</comment></comment></comment><comment><USER4> Is there a list somewhere of problems/bugs that have been discovered with dropbox?

<comment><USER5> Regarding the HackerOne program you can have a look at the bottom of this page: [https://hackerone.com/dropbox](https://hackerone.com/dropbox) and have a look for bug-report titles. These are the reports that were publicly disclosed and valid reports (won't-fixes are not in that list).

Or you might go for a search for "dropbox" here (that comes in handy if there are a lot of reports): [http://h1.nobbd.de](http://h1.nobbd.de)

</comment></comment><comment><USER6> $216 for pointing out a bug in software of a multi billion dollar (valuation) software company?yeah, I'll get right on that

</comment><comment><USER7> Why now?

</comment><comment><USER8> Looking forward to the day you offer a feature most call "Privacy". Client side encryption anytime soon?</comment>


Too many AI researchers think real-world problems are not relevant - <AUTHOR> https://www.technologyreview.com/2020/08/18/1007196/ai-research-machine-learning-applications-problems-opinion/ <comment><USER2> It would be nice to have a link to where the research was finally published.</comment>