Anatomy of a Ceph meltdown - <AUTHOR> http://chneukirchen.org/blog/archive/2018/01/anatomy-of-a-ceph-meltdown.html <comment><USER3> Most of this post-mortem has nothing to do with Ceph or Gentoo, and everything to do with competent system administration and change management. No one appears to have stopped to consider the ramifications of unannounced upgrades to the fileserver or OSD's. Once you're frantically rebuilding world, its over. the lessons learned completely ignore communications between sysadmins and management or, god forbid, even customers. "do everything whenever you like" is not system administration.

>If we notice something is going wrong with Ceph, we will not hesitate to shut down the cluster prematurely.

once bitten..twice shy? how does this solve future problems in the HA filesystem for your customer?

>We should not update Ceph on all machines at once.

HA/HPC 101. rolling upgrades are fine, slow and steady, with lots of testing and a documented rollback procedure if things go screwy. large enough/critical systems often have dev test and prod

>We will build glibc with debugging symbols.

you will plan your OS upgrades separate from your Ceph upgrades and with properly communicated actions before, during, and after the events.

>We will track Ceph releases more closely

as opposed to...what? being a sysadmin, Im on more than a dozen release mailing lists. I track load balancer known issues and patches to make sure i dont roll out, say, an F5 firmware with known failure to properly proxy HTTP2

<comment><USER4> To be fair, the author says right at the beginning:

"Please remember that we are all unpaid volunteers who have our own studies and/or day jobs, and no one has had more experience with Ceph than what you get from reading the manual."

So while you might have a valid point, it's hard to develop those skills if you are not a proper sysadmin. If you are just an unpaid volunteer, it means that most likely the environment you work in can't afford (or won't pay) for actual sysadmins whose paid job is to know these things.

Maybe the discussion should be, why were there unpaid volunteers doing sysadmin work instead of an actual sysadmin, but I guess that is a normal occurrence in universities all over the world?

<comment><USER5> In defense of ceph the documentation is fantastic and ceph clusters have only gotten easier toanage with every release. I am a homelabber, not a sysadmin and I run a cluster and have only run into one or two issues that the docs or patch notes didn't cover. Also the mailing list is incredibly friendly and helpful. The IRC channel is not as useful iny experience though, seems like with many there are a ton of idlers.

One does need to keep up with the release notes, mainly for the past couple releases, especially with the latest release, luminous which marked stable bluestore as a more performance alternative to filestore.

<comment><USER4> I don't even think Ceph is at fault here, since they are not the ones making a critical piece of infrastructure be the responsibility of unpaid volunteers.

Even if said volunteers where seasoned sysadmins, I don't think they should be stressing over something so critical while being unpaid. Obviously you don't always get to be paid in money. Sometimes you feel like doing it for the greater good, or because you like the institution, or whatever the reason. However I still think as a university you should not rely on unpaid staff to handle critical systems because it's not a nice thing to do. I don't think most people _enjoy_ stressing over a hobby.

</comment></comment></comment></comment><comment><USER6> Ceph meltdowns seem to be the worst. I've seen two other projects struggle with Ceph and eventually both gave up and moved on to something else. The main problem seems to be lack of documented usage guidelines and not much written on what to do when something blows up...and things seemed to blow up with both Ceph systems frequently and in many cases without any discernable error logging.

The after actions on both systems seemed to indicate that they had simply been provisioned wrong, or had violated some other best practice, but that advice never seemed to be written down anywhere and the documentation on the Ceph site was woefully inadequate.

One team simply moved on to using HDFS and rewrote their entire approach to assume HDFS and it's been pretty solid since. I think the other group moved to using Gluster.

I really hope it's getting better, because the idea is really great, but I wouldn't recommend it to anybody right now based on what I've seen.

<comment><USER7> There are lots of other options too like LizardFS:

[https://lizardfs.com](https://lizardfs.com)

Seems really nice and easy to deploy.

<comment><USER8> That one is derived from MooseFS ([https://moosefs.com/](https://moosefs.com/)).

What turned me away from LizardFS was when they forked the project the very first thing they did was rewriting the code to be in C++, this makes much harder to merge future improvements, and feels like they make some decisions for the wrong reason (the user doesn't care what language it is written in)

MooseFS is quite good and has fairly good performance, but its weakness is that is also that the central point is its master, and the open source version doesn't have HA.

I know that LizardFS supposedly has HA (it didn't have it at the time I looked into it), how is that implemented?

<comment><USER9> MooseFS can have a second master that takes over if the first one should fail, though.

<comment><USER8> I'm familiar with 1.x which was very basic, glad they added that. The previous approach with metadataloggers had the issue that before you can convert it to a master you had to merge changelogs which could take a while when you have many files.

</comment></comment></comment></comment></comment><comment><USER10> I don't understand why people use Gentoo in production servers. Is it good practice? I can see so many downsides.

<comment><USER11> Like what?

<comment><USER12> It's almost impossible to achieve any sort of reproducibility when all packages are built from source.

Portage is a great tool, but I always thought it was better suited to build packages and operating systems as opposed being an end user package management solution.

<comment><USER13> I've learned recently that even using Nix when building from source is something of a false hope of reproducibility. Many packages use the fetchgit function and fetchgit violates the contract.

<comment><USER14> fetchgit does no such thing. To use it, you need to specify the git revision _and_ the hash of the output; it's a constant-output derivation just like fetchurl.

Reproducbility is still a difficult problem, but that isn't one of them, and Nix comes far closer than Gentoo.

<comment><USER13> [https://twitter.com/shlevy/status/956313877800566784?ref_src...](https://twitter.com/shlevy/status/956313877800566784?ref_src=twcamp%5Ecopy%7Ctwsrc%5Eandroid%7Ctwgr%5Ecopy%7Ctwcon%5E7090%7Ctwterm%5E0)

<comment><USER15> I do hope somebody either makes that start warning, then removes it, or at least adds it to a linter of some sort.

</comment></comment></comment></comment></comment></comment></comment><comment><USER16> I hate to be blunt, but the moral of this story is that you should not trade good operational practice for some personal preferences. Distros generally don't matter, until they do. Choices made at the distro level, like, hey lets build world! don't mesh well into providing stable and reliable service.

You can run any distro you want, safely, with best practices. Which means you aim to not make upstream changes which have a high probability of breaking things ... like glibc ... and you focus upon maintaining a stable platform for your service. You can do that with any distro. Any unix(-like) thing really. There's no magic, just common sense.

The flip side is that some practices encouraged by various distros range between glacial change (read as stability), or near relativistic change (read as here be dragons). Gentoo encourages the latter, and Debian/CentOS encourage the former. This does not mean one is better than the other, just that you have to pay more attention with some distros, to maintaining operational discipline.

<comment><USER17> I would have run the Ceph in a VM, with the disks passed through so I could downgrade/crossgrade to different Ceph versions while having a rollback plan that is the identity function.

</comment></comment><comment><USER18> The "restart after upgrade" lesson is one we learned the hard way too, though typically at the OS level, not the application level.

We had cases where machines with 200+ days of uptime changed from one operations team to another, and the new team did a reboot -- and found that some NFS mounts or ip routes or firewall routes had been added manually at run time, without persistent configuration (like calling mount directly instead of adding stuff to /etc/fstab). Of course, those things were lost after a reboot, and had to be reconstructed somehow.

And this is in an organization with several professional sysadmins per team, not "just" a team of volunteers.

As for the choice of OS: it makes sense to use one that people are familiar with, and that fits their style of operation. I don't want to pass any value judgment on that topic.

All in all, impressive debugging work!

<comment><USER19> I used to be all about how long my uptime was.

Around a decade ago, I started hating big uptime. "That machine has been up 200 days? WAY past time for a reboot!" In my previous job I implemented a policy of rebooting at least every 6 months, or whenever kernel or glibc/etc updates were done.

This was very effective at ensuring that changes to systems were _ALWAYS_ reflected in boot scripts, etc... Way better than finding out that all sorts of problems existed on a hundred servers during an emergency when power went out, and nobody could remember the change that was made a year or more ago.

<comment><USER20> Even that is too long. Once a week at a scheduled time is best. That way if anything goes wrong, it's relatively easy to find out what changed because people's memory is still fresh.

</comment></comment></comment><comment><USER21> I have a dumb question when it comes to Ceph:

Whatever happened to the good ol' days of best practices? At least with Microsoft products, say, something like SCCM, you have metrics that can tell you what works & what doesn't work. Or, a software product on Microsoft's platform like PDQ Inventory/PDQ Deploy. I know with a decent 4GB memory/60GB HDD/1Gbps network, I can manage 40-50 desktop PCs fairly easily.

Do people not do baselines anymore? There's so much random documentation for Ceph. It's poorly documented. There are no logical help switches for it. And there's no "Oh, you should have x mons for this many OSDs"

It's just poor. Poor all around. :-(

<comment><USER22> If you're operating it in an environment where you need it to be up with multiple 9s or you're losing money, you call Red Hat and pay for a consulting engagement. As an open source piece of software, I think it's significantly harder to keep up with the level of public documentation that MSFT can with some of their products.

They'll provide RH certified packages that have been tested in-depth on certain OSes (not Gentoo), provide someone on-site to make sure you're following their best practices around configuration, upgrade process, etc.

Ceph is a very complex distributed system and it's very difficult (IMO) to nail down what the best practices are around hardware and resources - <USER2> depends almost entirely on the workload, and requirements out of it. If you're operating it solely for sequential RGW workloads, it looks _completely_ different than what you'd want for high performance RBD devices.

They do have some documentation online, to the extent that they can: separate networks, 5 MON nodes (to avoid one going down and ruining your day), etc. I just think it's significantly harder given that it's used on such a diverse set of workloads, which is why they are more than happy to sell consulting for this.

<comment><USER23> I was forced by management at a previous gig to use a RedHat consultant to set up a satellite server. In the end, I had to help him set it up after it became clear he wasn't able to do it on his own. It was a huge waste of money, and I'd caution anybody to pay close attention to contract management if you're going to get them in.

<comment><USER24> Satellite is a product sold to management, not to people who do not mind the command line.

That can be reflected a little bit in Red Hat technicians. Do no let that affect your judgement of their technical and operational skills.

</comment></comment></comment></comment><comment><USER25> Obviously as others have mentioned there were a lot of mistakes here around engineering best practises: using a bleeding edge source based distribution for critical storage infrastructure, allowing drift of package versions, and not testing releases.

I've done some work in Ceph and in my opinion it is too ambitious: block, object, and file system storage, custom raw partition format (Bluestore), along with a somewhat legacy C/C++ codebase makes the system fairly fragile. I would only recommend it if you "know what you are doing" or are willing to get some help from Red Hat.

The other comment I'd make is that 12TB unreplicated (assuming they are using 3x replication for Ceph) is actually not a huge amount of data and in my opinion a ZFS setup would be cheaper and more stable. It is not too challenging to do a HA ZFS setup, and also ZFS's many mirroring, scrubing, and checksuming abilities makes it very resilient. Copy on write and ZFS snapshots are also a great features that can save on disk usage.

I feel the sweet spot for Ceph is probably 50TB - 1PB where you need both block and object storage and are unwilling to use cloud solutions. Lower than 50TB and the overhead and risk of managing Ceph makes it less practical than traditional solutions. From the other direction CERN did testing up to 30PB with Ceph but they had to make significant code changes and had Ceph committers in their team. To compare Hadoop is running in clusters of up to 600PB (but you will be accessing the data very differently).

[1] [https://cds.cern.ch/record/2015206/files/CephScaleTestMarch2...](https://cds.cern.ch/record/2015206/files/CephScaleTestMarch2015.pdf)

</comment><comment><USER26> I distinctly remember being dialed into a meeting by phone where someone was explaining the project requirements of what would be our first Ceph deployment. Over and over it was explained there was simply NO provision for restarting nodes without the remaining nodes trying to rebuild for the configured n+x redundancy. I completely disagreed and said it made no sense at all, until I read up all I could on the matter. No provision for patching. Period. If a node goes down, the cluster tries to rebuild.

I'm not sure what reality this design comes from, but it's not one I care to inhabit.

<comment><USER27> This is absolutely not true.

The normal flags you're looking for are: noout, noscrub, nodeep-scrub. Do your maintenance. Unset those flags.

The noout is what tells Ceph to essentially not shuffle data when OSDs go down. This is also referenced in the article.

<comment><USER28> Even that might be too rigid. A better way is to have a time-based setting. For x < N seconds, ignore the missing node. N could be 600 or 900. During that time, you have degraded performance, but the system is still serving. If you exceed that time, then perhaps the node is gone for good, so you should rebuild.

Think of the case where you have more than the three servers describe in this incident. You really don't want to disable cluster-wide automated repairs because of some other kind of maintenance. Something else WILL happen during your scheduled work and... ooops.

Why 600 or 900? Unavailability events in larger clusters mostly tend to last only 10-15 minutes, for routine things such as regular node reboots, per page 2 of [https://static.googleusercontent.com/media/research.google.c...](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36737.pdf)

<comment><USER27> Ceph already has this, the setting is called mon_osd_down_out_interval, and the default is 300 seconds (you can easily change it).

The reason for "noout" is you generally want to minimize IO operations while you're performing your maintenance. You should be closely monitoring your cluster while performing maintenance, and if something else goes wrong, you abort and unset noout, wait for it to finish rebuilding, and reassess.

EDIT: This used to be 300 in Jewel, and seems to have changed to 600 in a later version.

<comment><USER28> That's cool. Thanks!

I'd still argue that you want to provision N+2 capacity, so you can withstand a planned event and an unplanned one at the same time, without having to go through manual tweaking. I'd leave that for truly exceptional cases, such as when the whole cluster is hit by some nasty bug or has entered a spiral that calls for drastic measures.

</comment></comment></comment></comment></comment><comment><USER29> Nice debugging skills, but I can't help think that's what you get for running Gentoo in production. You should be using what everyone else uses (i.e. Debian or CentOS) so they run into problems before you.

</comment><comment><USER30> I didn't read past the first paragraph before I formed the opinion that running all your VMs on a file system like this is a terrible idea.

If you can't afford an enterprise SAN (I'm not even talking a NAS, I mean a real fiberchannel-based block-store) for your virtual environment (and I get it - many cannot) then just do yourself a favor and run on local disk.

The reduction in moving parts will dividends. I promise.

<comment><USER31> Agreed, and this post does a good job describing why:

[https://thornelabs.net/2014/06/14/do-not-use-shared- storage-...](https://thornelabs.net/2014/06/14/do-not-use-shared-storage-for- openstack-instances.html)

</comment></comment><comment><USER32> According to [https://www.fs.lmu.de/angebot](https://www.fs.lmu.de/angebot), this server hosts

* a bunch of wordpress pages

* mailing lists

* mail server for council adresses, likely mostly unused

* git server

for a few student councils at LMU. Git is likely used only by the CS council, maybe maths and physics.

Is it _really_ necessary to have 3 * 12TB for this, and all this complexity? Fault tolerance? Gentoo? Would a single server running boring software and occasional backups not have been more appropriate for something like this? To me it looks like a few CS students tasked with managing digital went completely overboard.

</comment><comment><USER33> Ceph is a very finicky beast and I'd advise anyone to stay away unless you're wiling to dedicate significant resources to master it. I've seen more than one Ceph system gone belly up during my days at IBM. IBM is throwing significant resources at it have people and procedures developed to tackle the usual and less usual operations... and it still caused headaches.

It will make you hate your job whether there's some kind of networking problem (configuration, bad NIC or just plain ol' Softlayer dropping packets while support claiming no problems), some kind of HW problem (one bad OSD node in semi failed state will pretty much kill cluster throughput) or just some bad disks causing it to rebalance in the middle of high I/O time. It always made me nervous even with the most basic 'let's swap the failed OSD' operations. The blog post itself is a classic showcase of putting all eggs in one basket. They'd be better off with a simpler setup. Btw running only 1 monitor node will make you cry when the single monitor decides to take a break which will happen sooner or later.

</comment><comment><USER13> Another story where the root of the problem is code that lacks robust type checking, allowing for subtle memory corruption.

</comment><comment><USER34> For anyone else that doesn't know Ceph: [http://docs.ceph.com/docs/master/](http://docs.ceph.com/docs/master/)

</comment><comment><USER35> What would be a more fool-proof distributed storage? I tried GlusterFS briefly, but also heard some meltdown stories about it.

It almost seems like if you need some persistent storage running plain NFS would be the safest thing if you don't have a dedicated team.

<comment><USER36> That's what I've done. Hardware RAID + XFS + NFS + backups, done. It wasn't HA, but at least I understood how to get it back up.

<comment><USER37> > That's what I've done. Hardware RAID + XFS + NFS + backups, done.

Even if this was HA, it doesn't scale to HPC type needs. You end up with a single node bottleneck for all data you need. You also would have some sort of direct attached storage all connected to a single node.

Lustre/Gluster/Ceph/GFS and maybe HDFS (ideally with commercial MapR type NFS access) are probably the only viable options in the HPC use case.

If you have a lot of money to burn Isilon/NFS is an option as well.

<comment><USER36> Yep, I haven't worked in a HPC/big data environment but neither are the people who wrote the original article. They have three OSDs and only two(?) servers accessing them.

</comment></comment></comment></comment><comment><USER38> I'm not surprised here to see comments here blaming them for using Gentoo in production, even though I argue that wasn't fully the cause of their outage. Hell, I use Gentoo on my personal boxen because I love it's package management, and even I winced when I read that. "Oh man, I hope they're keeping on top of their system administration. Damn, nope got bit by the libstd++ change, _and_ the recent profile change. This is gonna be painful."

<comment><USER39> I'm also a fan of Gentoo, but realistically, if they were running Debian or Red Hat or a derivative, they would be able to use the recommended releases directly from Ceph:

[http://docs.ceph.com/docs/master/install/get- packages/](http://docs.ceph.com/docs/master/install/get-packages/)

My interpretation of the writeup suggests that most of their problems would have been avoided by running the latest supported release of Ceph, on a supported distribution.

I ran Gentoo servers myself for a few years, but I had to give it up when I realized I wasn't getting much benefit for all the extra effort I was putting in. It was a great way to learn how free software packages interact with each other, but it became a fairly significant time sink to rebuild the world every so often.

</comment></comment><comment><USER40> Unless they host their file system on public facing IPs, it would be simpler to use private ipv4 addresses versus ipv6. With IPv6, you are running your code against less hardened paths and depdenending the type of IP address assigned to your server, behavior can change like this.

</comment><comment><USER41> everyone blaming gentoo here is completely wrong. we had a very similar meltdown where the osds were hosted on rhel 7. they would not start. we had mds failures as well on rhel.

ceph is just a buggy piece of shit. it configures its options like a windows registry. at one point I literally built ceph on my gentoo system and ran the mds over the vpn. it was the only mds that wasn't crashing. nothing about gentoo is to blame here, it's more stable with ceph if anything.

<comment><USER42> This has been my experience with ceph as well. Hell, the documented setup commands for Debian didn't even work out of the box.

I also agree with everyone that gentoo was probably not a great choice, but ceph doesn't even seem production ready unless you have a dedicated team just to manage it.

<comment><USER43> Part time sysadmin here (founder, so biz duties, dev duties, PM and product duties, so not a ton of time for sysadmin stuff) and I've had no major problems with ceph. Had some performance issues but figured out how to tune scrubs appropriately for my workload, and found out the hard way that prosumer SSD do not make good journal devices. I run a three node 18 osd ceph cluster and it's been no problem. I may eat my words when I do a rolling upgrade from Firefly to jewel (multiple versions) but not dreading that process much.

I don't understand why people are complaining about docs here. The docs are good. Read them, read config options, read until you understand what config options actually do. Passively watch the mailing list. I think expecting to run your own petabyte capable storage cluster without a little effort is a bit misguided...</comment></comment></comment>


How Gaming Got Its Dice (2013) - <AUTHOR> http://playingattheworld.blogspot.com/2013/02/how-gaming-got-its-dice.html <comment><USER3> > There is no intuitive way to extract percentile results from rolling a small > number of d6s.

Here is an algorithm for rolling a single decimal digit using on average about 2.19 d6 rolls (which can be done in parallel if you have more than one d6):

      function one_digit
        d = roll()
        while d == 6
          d = roll()
        if roll() <= 3
            return d - <USER2>
        else
            return d + 4

Do this twice to get a percentile. This seems reasonably intuitive to me.

<comment><USER3> The problem of doing the job of a d10 with a d6 can provide a nice illustration of an application of bases other than 10. Might be useful for an elementary school math teacher trying to explain why the kids should learn this stuff.

Write the numbers from 0.0 through 0.9 in steps of 0.1, but write them in base 6, to produce this table:

      dec base 6
      0.0 0.0000
      0.1 0.0333...
      0.2 0.1111...
      0.3 0.1444...
      0.4 0.2222...
      0.5 0.3000
      0.6 0.3333...
      0.7 0.4111...
      0.8 0.4444...
      0.9 0.5222...

Roll a d6 to generate base 6 digits, which you take as the digits of a base6-imal fraction in [0,1). Roll until you have enough digits to determine which [k/10, (k+1)/10) interval it falls in, using the above table as an aid.

In general this can be used to make a table for simulating a dN with a dM, by writing i/N in base M for i = 0 to N-1.

When M is 2 this gives a nice way to simulate a dN with coin flips.

The above can be made easier to use by representing it in a graph instead of a table, where you start at a root node and follow edges labeled by dM values, and the leaf nodes give the dN values.

That would be easy and fast to use.

<comment><USER4> I've never heard of this method before! That's a _really_ nifty trick, thanks for putting it here!

<comment><USER3> Here's another approach you might enjoy. We wish to simulate a dN with a dM. I'll use simulating a d10 (N=10) with a d6 (M=6).

1\. let L = a common multiple of N and M. The least common multiple of N and M is a good choice here, so for our example we'll use L = 30.

2\. Make a list of the values you are trying to choose from. Repeat each value L/N times, so that the length of the list is L. For our example, L is 30, N is 10, and L/N is three, so our list consists of the values 1 through 10 each repeated 3 times: [1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 8 8 8 9 9 9 10 10 10].

3\. Viewing this list as M chunks of length L/M, select one of the chunks at random via a roll of our dM. For our example that is 6 chunks of length 5. Here is the list again with parenthesis drawn around the 6 chunks: [(1 1 1 2 2) (2 3 3 3 4) (4 4 5 5 5) (6 6 6 7 7) (7 8 8 8 9) (9 9 10 10 10)]

4\. Take the chunk selected by the roll of the dM. If all L/M numbers in the chunk are the same, that is the value of our simulated dN roll and we are done. Let's say we've picked the second chunk, (2 3 3 3 4). That contains more than one distinct value, so we go on to step #5.

5\. Replace the list with the items from that chunk, each repeated M times. In our example, our new list is [2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4].

6\. Go to step #3.

Note that the second time through this is how we chunk the list: [(2 2 2 2 2) (2 3 3 3 3) (3 3 3 3 3) (3 3 3 3 3) (3 3 3 3 4) (4 4 4 4 4)]. 4 of the 6 chunks contain only one distinct value now, so there is a 2/3 chance that we find our d10 value on the second iteration.

If you actually implement this, you don't need to actually make the lists like I did above. You just need a count of how many times each thing appears. So instead of [1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 8 8 8 9 9 9 10 10 10] you would have [3 3 3 3 3 3 3 3 3 3], meaning you have three '1', three '2', and so on.

To pick the second chunk, you skip 5 items and then take 5. That means we skip all 3 '1's, and 2 of the '2's, and take 1 '2', then all 3 '3's, and 1 '4'. That gives us [0 1 3 1 0 0 0 0 0 0].

BTW, note that this also works for simulating a smaller die with a larger die. For example, simulating a d6 with a d10. There is a really simple way to do that, of course: roll the d10 and discard anything above 6. With that method, you get a d6 output on 6 out of 10 rolls. Let's call this method the trivial method.

If we use the list method as described above, we get a higher output rate. Here is a little table:

      1 1 1 (1)
      1 1 2   111 111 111 111 111 111 112 222 222 222
      2 2 2 (2)
      2 3 3   222 222 222 233 333 333 333 333 333 333
      3 3 3 (3)
      4 4 4 (4)
      4 4 5   444 444 444 444 444 444 445 555 555 555
      5 5 5 (5)
      5 6 6   555 555 555 566 666 666 666 666 666 666
      6 6 6 (6)

On the left, the first 3 columns just show the initial [1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 5 6 6 6 6 6] list (N distinct values each repeated L/N times) broken down into groups of L/M, which is 3 in this case.

If our d10 roll is 1, 3, 5, 6, 8, or 10 we count that as a successful d6 roll of 1, 2, 3, 4, 5, or 6, respectively, because those chunks all have only one distinct value. I've marked those in the table with the resulting d6 value in parenthesis. So, on the first d10 roll we get a d6 value 6 out of 10 times, just like with the trivial method.

Where we beat it is on the next roll. For each chunk that does not yield a d6 value on the first row, I've added the resulting list, grouped into chunks. Note that in all 4 of the possible lists, 9 out of the 10 chunks have only one distinct value, and so 9 out of 10 times the second roll yields a d6 value.

If the second roll fails too, the trivial method will again only succeed 6/10 times on the third roll. Again we beat it, with a 9/10 success rate, and so on.

</comment></comment></comment></comment><comment><USER5> Interesting that once there was a trend in tabletop wargaming towards "realistic" probabilities expressed as a percentage (e.g. "these riflemen have a 70% of hitting the target"). I think this isn't as common anymore.

Nowadays, for whatever reason, I see a trend back to the humble d6, both in historical wargaming and fantasy skirmish games. Many wargames use d6 in disguise (see: Bolt Action, SAGA, Warhammer). I think it has to do with two things: the d6 is readily available (though, one has to wonder, why is this an issue when everything else in wargaming isn't as easy to find, like the miniatures and tokens!), and the other fancier dice didn't pan out _in practice_.

<comment><USER6> > and the other fancier dice didn't pan out in practice.

For wargaming I think what happened is that "special dice and more detailed simulation" moved into "computerized simulation" (which allows a lot more depth with less player-managed overhead) in the late 1980s through the 1990s, and most of that whole end of the hobby left the tabletop world.

<comment><USER7> Yeah, although the more complex simulations have largely have largely faded away from the computer space as well. (With a few exceptions.) And very niche board games are probably more viable than very niche computer games. Modder communities for some old computer games notwithstanding.

Personally, I confess to finding complex simulations (whether of the board game or flight/combat genres) more appealing in theory than I do in practice.

<comment><USER8> They're also a pain to make. I work on complex simulations for a living, and the number of parameters and data sets you need to get good results is staggering.

If you end up just going "We think it's about 100 meters" that's a lot of effort for not much reward.

</comment></comment></comment></comment><comment><USER9> > There is no intuitive way to extract percentile results from rolling a small > number of d6s.

Shadowrun 5e -> Roll 21d6 defense test against a 18d6 attack.

In all seriousness, good article. I always wondered where the need of the d20s came from, and always wondered why d10s and the likes didn't catch on. I always had shadowrun to blame for an ungodly amount of d6's laying around my house

<comment><USER10> Is 5e any good? I loved the flavor update in 4e, but felt like they threw out a lot of what was good with the mechanics from 3e.

<comment><USER9> Yeah, besides the huge amount of errata and bizarre inconsistencies between what you would expect a roll to be vs what it actually is 5e is ok.

The amount of flavor is sad, but at lest deckers are cool now since [everything] is wireless.

</comment></comment></comment><comment><USER11> Does anyone know where I could find the Gygax article mentioned, "Four & Twenty and What Lies Between"? I loved his explanation of probabilities with dice and cards in the 1989 "Advanced Dungeons & Dragons, Dungeon Masters Guide" (apologies for my book-scanner errors):

 _As the DM, the tools of your trade are dice — platonic solid-shaped or just about any other sort. The random numbers you generate by rolling dice determine the results based on the probabilities determined herein or those you have set forth on your own. In case you are not familiar with probability curves, there are two types which are determined by your dice: linear (straight line), 'which has equal probability of any given integer in the number group, and bell (ascending and descending line), which has greater probability towards the center of the group of numbers than at either end. The two curves are illustrated thus:

[Image of linear curve for 10-sided dice, with each number having a 10% chance of coming up.]

Linear probability develops a straight line of ascending probability when used as a cumulative probability as shown above.

Bell distribution, when used to delineate the probability of certain numbers appearing, develops o curved line like this:

[Image of a bell curve for a 3d6 roll, with 3 and 18 being the least probable and 8 through 12 being the most probable.]

A single die, or multiple.dice read in succession (such as three dice read as hundreds, tens and decimals) give linear probabilities. Two or more dice added together generate a bell-shaped probability curve.

Before any further discussion takes place, let us define the accepted abbreviations for the various dice. A die is symbolized by "d", and its number of sides is shown immediately thereafter. A six-sided die is therefore "d6", d8 is an eight-sided die, and so on.-Two four-sided dice are pressed by 2d4, five eight-sided dice are 5d8, etc. Any additions to or subtractions from the die or dice are expressed after the identification, thus: d8 8 means a linear number grouping between 9 and 16, while 3d6 -2 means a bell-shaped progression from 1 to 16, with the greatest probability group in the middle (8, 9). This latter progression has the same median numbers as 2d6, but it has higher and lower ends and a greater probability of a median number than if 2dl2 were used. When percentage dice are to be used; this is indicated by d%.

The d4 can be used to generate 25% incremental probabilities, random numbers from 1 to 4, with 1 it generates a linear 2-5, etc. It can be used to get 1 or 2 (1 or 2 = 1, 3 or 4 = 2) or in conjunction with any other dice to get linear or bell-shaped probability curves. For example, 2d4 = 2-8, 3d4 ^ 3-12, d4 d6 = 2-10, d4 d20 (as dlO) = 2-M.'when rolled in conjunction with another die, the d4 can be used to determine linear number ranges twice that shown on the other die, thus: d4 reading 1 or 2 means that whatever is read on the other die is the number shown; but if the d4 reads 3 or 4, add the highest number on the second die to the number shown — so if d8 is the second die 1 to 16 can be generated if a dl 2 is used 1 to 24 can be generated. If a d20 is used either 1 -20 (assuming the use of a standard d20 which is numbered 0-9 twice without coloring one set of faces to indicate that those faces have 10 added to the number appearing) or 1-40 (assuming that one set of faces is colored) can be gotten by adding 0 if 1 or 2 is rolled on the d4 and 10 or 20 (depending on die type) if a 3 or 4 is rolled. Linear series above this are possible simply bv varying the meaning of the d4 number; 1 always means add 0, but 2 can be interpreted as add the value (highest number) of the second die, 3 can be twice value, and 4 can be thrice value. Thus, a d4 reading 4 in conjunction with a d8 (linear curve 1-32) would mean 24 d8, or 25-32.

What applies to d4 has similar application with regard to d6, d8, dl2, and d20. The d6 has 16 2/3% intervals, d8 has 12 1/2% intervals, and d20 can have (1-2 = 1,3-4 = 2,5-6 = 3), while 1 to 5 can be easily read from a d20 (1-2 = 1, 3-4 = 2,5-6 = 3,7-8 = 4,9-0 = 5).

The d20 is used often, both as dlO and d20. The bell-shaped probability curves typically range from 2-20 to 5-50, i.e., 2, 3, 4 or 5d20 added together. Also common is the reading as above with one decimal place added to the result to get 20-200, 30-300, etc. In the latter case, a roll of 3 on one die and 0 (read as JO) totals 13, plus one place, or 130.

Non-platonic solid-shaped dice are available in some places. The most common of these is a ten-sided die numbered 0-9. As with the d20, this con be used for many purposes, even replacing the d20 if a second die is used in conjunction to get 5% interval curves (1-20). Also, the die can give 0-9 linear curve random numbers, as the d20 can.

Other dice available are various forms of "averaging" dice. The most common of these has six faces which read: 2, 3, 3, 4, 4, 5. The median of the curve it generates is still 3.5, that of a normal d6, but the low and high numbers, 2 and 5, are only half as likely to appear as 3 or 4. There is a 33 1/3% chance for either of the two latter numbers to be rolled, so the probabilities of absolutely average rolls are far greater. Other such dice have zeros on them, several low numbers, and so on. These sorts of dice, along with poker dice, "put & take" dice, or any other sort can be added in order to give you more flexibility or changing probabilities in random selection or event interpretation. For example:

The author has a d6 with the following faces: SPADE, CLUB, CLUB, DIAMOND, DIAMOND, HEART. If, during an encounter, players meet a character whose reaction is uncertain, the card suit die is rolled in conjunction with 3d6. Black suits mean dislike, with the SPADE equalling hate, while red equals like, the HEART being great favor. The 3d6 give a bell-shaped probability curve of 3-18, with 9-12 being the mean spread. SPADE 18 means absolute and unchangeable hate, while HEART 18 indicates the opposite. CLUBS or DIAMONDS can be altered by discourse, rewards, etc. | Thus, CLUBS 12 could possibly be altered to CLUBS 3 by offer of a tribute or favor, CLUBS 3 changed to DIAMONDS 3 by a gift, etc.

In closing this discussion, simply keep in mind that the dice are your tools. Learn to use them properly, and they will serve you well._

[Edited for some scan errors.]

<comment><USER12> _Gary Gygax writes an article for Lowrys Guidon entitled "Dice...Four & Twenty and What Lies Between", in which he expounds on the application of various polyhedral dice in wargaming--specifically, Tractics._

From: [http://www.commonroomgames.com/timeline.html](http://www.commonroomgames.com/timeline.html)

Don't know where to source a copy however.</comment></comment>


New Twitter Code For Retweet: ♺ - <AUTHOR> http://www.stoweboyd.com/message/2009/01/new-twitter-cod.html <comment><USER2> An inconvenient solution to a problem that doesn't exist.

Aside from the dubious claim that this is "more distintive" can somebody explain the point to me?

<comment><USER3> I guess I can now sympathize with everyone who complained about too many econ/politics/non-ruby/xyz posts to HN.

Front page story about a suggested change from 'RT' --> '♺' on twitter? If I had less faith in HN I would label this a jump the shark moment.

<comment><AUTHOR> It definitely belongs in the "just for fun" category.

<comment><USER3> I guess there's nothing super-wrong with posting links like this, my surprise was more that it got so many upvotes.

</comment></comment></comment></comment><comment><USER4> Call me old-fashioned, but I prefer using "RT" as it's pretty easy to type, it's in popular usage on Twitter (so no need to explain the crazy recycling symbol), plus it's only one extra letter

<comment><USER5> _"RT" as it's pretty easy to type_

That's pretty much it. Anything that I can't type on a Mac using alt and possibly shift is pretty out there.

Not only that but who the heck has access to that unicode symbol on a cell phone?

<comment><USER6> Not just that, a lot of computer can't even display it correctly. Maybe twitter could auto-replace it with a small image of the sign, but that might break some twitter applications.

</comment></comment></comment><comment><USER7> Fail. 'RECYCLING SYMBOL FOR GENERIC MATERIALS' (U+267A) is not generally available on platforms other than Mac OS X. On that platform it is only provided by the system font Apple Symbols (as a fallback) and versions of the included Japanese font Hiragino (what you're actually seeing). It is not included in Arial Unicode MS on Windows, which in turn is irrelevant as Arial Unicode MS is only included with Office, and (sigh) Mac OS X.

From looking at the Vista fonts I have available, it doesn't appear to be available there either. If it does work, I'd be curious to know what fonts it's provided by.

Let this be a lesson: Don't use weird Unicode characters unless you know that everyone that needs to see them can see them.

FWIW, if you can't stand the little blank boxes, here's a Free font set that covers most of Unicode: <http://dejavu-fonts.org/>

<comment><USER8> I found this app which you can use to find out which of your local fonts support a given symbol:

[http://www.fileformat.info/info/unicode/font/fontlist.htm?te...](http://www.fileformat.info/info/unicode/font/fontlist.htm?text=%E2%99%BA+-+Unicode+Character+%27RECYCLING+SYMBOL+FOR+GENERIC+MATERIALS%27+\(U%2B267A\))

On my Vista system, the 0x267a (recycling) symbol is supported by "Malgun Gothic", "Meiryo", "Microsoft JhengHei", "Microsoft YaHei", and "Segoe UI".

Not sure which of these are always available.

</comment></comment><comment><USER9> If I can't do it on my iPhone (via Tweetie) ... forget it

<comment><USER6> You could just copy and paste it from an other message - oh, wait...

</comment></comment><comment><USER10> Just because "recycle" and "retweet" both start with "re" does not mean that the recycle symbol can be co-opted for retweeting.

But, meh, it's catching on and it's part of the evolution of language. Soon we'll all be speaking Twitterese.

kthxbai.

</comment><comment><USER11> looks like a square on my WinSvr2k8Std. Any screenshots?

<comment><USER12> <http://en.wikipedia.org/wiki/Recycling_symbol>

Of course this doesn't give you the _full experience_ of unicode! But that is in fact what it is.

<comment><USER13> ♲ ≠ ♺

<comment><USER12> I pulled a wishy-washy. You caught me.

</comment></comment></comment></comment><comment><USER14> It may be one less "character" than RT, but it is one more byte in UTF-8. 0xE2 0x99 0xBA. What does twitter mean with the "140 character" limit, what's a character here? I mean, <http://www.joelonsoftware.com/articles/Unicode.html> was written in 2003.

</comment><comment><USER15> It's probably telling that I can't get the character to display properly in any interface... but whether of the practicality of the suggestion, or the level of my char-fu, I'm not sure.

<comment><AUTHOR> I don't know how to type it yet, either... ;-) But it looks nice.

<comment><USER16> ♳ ♴ ♵ ♶ ♷ ♸ ♹ ♺ ♻ ♲ ♼ ♽

Just copy+paste ;)

On OS X, you can enable the Character Palette which is extremely handly for inputting unicode.

Settings->International->Input menu->Character Palette[Check]

<comment><USER15> Nope, just little blocks. Some interfaces display it as one solid dark monolith, some as empty squares, and a special few as festive little glyphs with mysterious markings inside. (XP SP2, every browser under the sun and then some.)

<comment><USER17> You probably just need to track down the right font that has them.

<comment><USER18> Sure, because everyone wants to have to mess with their global font settings in order to use a website properly

<comment><USER17> Probably true, but it shouldn't be that difficult to set up a unicode font as a fallback for all other fonts.

</comment></comment></comment></comment></comment></comment></comment><comment><USER19> Stowe Boyd is showing his lack of a tech background here. The recycle symbol is 1 byte longer in UTF8 encoding and nobody on Windows can read it. I call Fail.

</comment><comment><USER20> Too bad most of Windows users won't be able to see it correctly. And too bad those are the vast majority of Twitter users. (And it's complicated to type in.)

</comment><comment><USER21> I rather save space with interrobang: ‽

</comment><comment><USER22> lots of clients don't render unicode well. I think TweetDeck doesn't show the rest of the message after barfing on unicode, meaning instead of a retweet you'd just get a blank tweet. RT is standard</comment>