UPDATE: I now know my issue was database corruption, but discerning it was somewhat tricky--apparmor appeared to be the cause for longer than it should've.I didn't note when first posting that even after putting mysql in complain mode and sending apparmor both stop and teardown commands my syslog still showed the apparmor message...feeding my irrational fear of the protection layer--still not sure how this happened. I finally got mysql separated from apparmor, but it still couldn't lock its own files. Ergo database corruption--dang. My backups worked fine on a new server.INITIAL POST:Mysql server is being blocked by (I think) apparmor, but I'm at wits' end to determine why/how. I'm not overly familiar with apparmor.I know I shouldn't uninstall apparmor--for at least two reasons--but I've used enough profanity (and given this issue too much time) to not consider it.  Hopefully I'm merely missing something simple and will learn here.The failure began today and follows no system changes. MySQL's error log laments permissionsCan't open and lock privilege tables: Table 'servers' is read onlyI've been unable to find anyone with this issue who isn't currently moving their default database store.  I moved mine as well--two years ago. The apparmor config is unchanged since 2014/04/21:/files/bak/tmp/ rw,/files/bak/tmp/* rwk,/files/bak/mysql/ rw,/files/bak/mysql/** rwk,I've verified filesystem permissions:# find mysql/ -type d -exec chmod 700 {} \;# find mysql/ -type f -exec chmod 660 {} \;# chown -R mysql: mysql I reloaded apparmor, installed apparmor-utils, pushed mysql to complain# aa-complain mysql# apparmor_statusapparmor module is loaded.5 profiles are loaded.4 profiles are in enforce mode.   /sbin/dhclient   /usr/lib/NetworkManager/nm-dhcp-client.action   /usr/lib/connman/scripts/dhclient-script   /usr/sbin/tcpdump1 profiles are in complain mode.   /usr/sbin/mysqld1 processes have profiles defined.0 processes are in enforce mode.0 processes are in complain mode.1 processes are unconfined but have a profile defined.   /sbin/dhclient (495)...but viewing syslog still suggests apparmor is blocking mysql after service mysql start:apparmor=STATUS operation=profile_replace name=/usr/sbin/mysqld pid=13899 comm=apparmor_parserBefore I found the apparmor issue I tried restoring DBs from backups, which also failed with write permissions:Can't create/write to file '/files/bak/mysql/dbCooper/wp_cb_contact_form.MYI'I verified the filesystem is 'rw' (even though the above find -exec would have failed anyway):mount/dev/xvdf on /files/bak type ext4 (rw,noatime)I've even tried stopping apparmor, but the syslog still shows can't open and lock privilege tables after this:# service apparmor stop  [...redacted teardown msg...]# /etc/init.d/apparmor teardown  * Unloading AppArmor profiles# service apparmor statusapparmor module is loaded.0 profiles are loaded.0 profiles are in enforce mode.0 profiles are in complain mode.0 processes have profiles defined.0 processes are in enforce mode.0 processes are in complain mode.0 processes are unconfined but have a profile defined.Is it possible for mysql to lock database files and fail to unlock them when the daemon crashes?  If so how would I clear the lock?I'm currently running my DB withmysqld --skip-grant-tables...so I know the executable can run, and the databases are at least somewhat valid (the sites all appear normal). Am I missing something dumb?I tried trimming this post to relevant info--glad to post the entire (cfg|log) where useful.Thanks for readingIf this post should be moved to another Stack Exchange (ServerFault?) I apologize--and mods feel free to correct it.
apparmor: mysql permissions--with no recent changes
permissions;mysql;corruption;apparmor
Database corruption.Occam's Razor prevails. I moved backups to a new server and updated the db location/apparmor config. I cringed as I restarted everything. I'd spent hours convincing myself AppArmor was a cavernously complex and difficult beast, but my reticence was completely without cause--it worked perfectly on the first try.Amazing how it just works when no files are corrupted--should be an Apple ad.Now if I could just recover the WordPress site my web developer updated extensively while I was running under --skip-grant-tables but before I realized what the cause was.:-/