text:
You are here
FromDual TechFeed (en)
Max_used_connections per user/account
How many connections can be opened concurrently against my MySQL or MariaDB database can be configured and checked with the following command:SHOW GLOBAL VARIABLES LIKE 'max_connections'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | max_connections | 505 | +-----------------+-------+
If this limit was ever reached in the past can be checked with:SHOW GLOBAL STATUS LIKE 'max_use%'; +----------------------+-------+ | Variable_name | Value | +----------------------+-------+ | Max_used_connections | 23 | +----------------------+-------+
But on MySQL instances with many different applications (= databases/schemas) and thus many different users it is a bit more complicated to find out which of these users have connected how many times concurrently. We can configure how many connections one specific user can have at maximum at the same time with:SHOW GLOBAL VARIABLES LIKE 'max_user_connections'; +----------------------+-------+ | Variable_name | Value | +----------------------+-------+ | max_user_connections | 500 | +----------------------+-------+
Further we can limit one specific user with:GRANT USAGE ON *.* TO 'repl'@'%' WITH MAX_CONNECTIONS_PER_HOUR 100 MAX_USER_CONNECTIONS 10;
and check with:SELECT User, Host, max_connections, max_user_connections FROM mysql.user; +------+---------------+-----------------+----------------------+ | User | Host | max_connections | max_user_connections | +------+---------------+-----------------+----------------------+ | root | localhost | 0 | 0 | | repl | % | 100 | 10 | | repl | 192.168.1.139 | 0 | 0 | +------+---------------+-----------------+----------------------+
But we have currently no chance to check if this limit was reached or nearly reached in the past...
A feature request for this was opened at MySQL wit bug #77888Solution
If you cannot wait for the implementation here we have a little workaround:DROP TABLE IF EXISTS mysql.`max_used_connections`; CREATE TABLE mysql.`max_used_connections` ( `USER` char(16) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL, `HOST` char(60) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL, `MAX_USED_CONNECTIONS` bigint(20) NOT NULL, PRIMARY KEY (`USER`, `HOST`) USING HASH ) ENGINE=MEMORY DEFAULT CHARSET=utf8 ; DROP EVENT IF EXISTS mysql.gather_max_used_connections; -- event_scheduler = on CREATE DEFINER=root@localhost EVENT mysql.gather_max_used_connections ON SCHEDULE EVERY 10 SECOND DO INSERT INTO mysql.max_used_connections SELECT user, host, current_connections FROM performance_schema.accounts WHERE user IS NOT NULL AND host IS NOT NULL ON DUPLICATE KEY UPDATE max_used_connections = IF(current_connections > max_used_connections, current_connections, max_used_connections) ; SELECT * FROM mysql.max_used_connections; +--------+-----------+----------------------+ | USER | HOST | MAX_USED_CONNECTIONS | +--------+-----------+----------------------+ | root | localhost | 4 | | zabbix | localhost | 21 | +--------+-----------+----------------------+
Caution: Because we used a MEMORY table those values are reset at every MySQL restart (as it happens with the PERFORMANCE_SCHEMA or the INFORMATION_SCHEMA).
The Upcoming Leap Second
The press, be it the general daily newspaper or the computer magazines, is currently informing the public about an upcoming leap second, which will be taken in the night from June 30 to July 1 at 00:00:00 UTC. While we Europeans will enjoy our well-deserved sleep then, this will be at 5 PM (17:00) local time on June 30 for Califormia people, and during the morning of July 1 for people in China, Japan, Korea, or Australia. (Other countries not mentioned for the sake of brevity.) This is different from last time, when the leap second was taken in the night from Saturday to Sunday (2012-July-1 00:00:00 UTC), so it was a weekend everywhere on the globe.
We have got several requests from our customers about this upcoming leap second, whether they need to take any special precautions or whether they "are safe". Well, obviously nobody is "safe" from the leap second in the sense that it would circumvent them, everybody will encounter it on their systems. The concern is whether they have to expect any trouble.
For the people operating MySQL (or any other DBMS), this issue is threefold:
Let us look at the operating system first, dealing with Linux only.
(All other operating systems don't show up with significant numbers in our customer base.)
Linux measures the time in seconds (since Jan 1, 1970, 00:00:00 UTC), and it does not include the leap seconds in this counting. When the leap second is taken, the timestamp value (those seconds) will simply not be advanced at the end of the regular second, but it will re-use its current value in the leap second. As a result, the conversion of timestamps into common time reckoning will still produce values from 0 to 59 for the minute as well as for the second, there will not be a 60. Another consequence is that there is no way to tell the leap second from the preceding regular one.
MySQL always takes the time information from the Linux kernel, so it will also use the same timestamp value (or "now()" result) for both the regular and the leap second. The MySQL manual describes this on its own leap second page, which has become inaccurate by some recent code changes: We could not reproduce the results given there (and will probably file a documentation bug about this). However, that difference does not affect the principle of the page's first paragraph.
About the application, it is hard to claim anything - there are too many of them. However, it is obvious that an application might run into trouble if it managed to store a new timestamp value every second and assumed they are distinct: they will not be (unless the application manages to skip the leap second). Let's hope any application programmer creating such a high-resolution application was aware of the problem.
So does it all look fine?
Not completely:
Following the last leap second (just three years ago), several administrators noticed that their machines became extraordinarily busy, which even let the power consumption rise significantly.
This was caused by a bug in the Linux kernel, it let user processes resume immediately if they were trying to wait on a high-resolution timer.
The exact details are beyond the scope of this article, your favourite search engine will provide you with more than enough references if you ask it.
For our purposes, the important fact is that this also happened to MySQL server processes ("mysqld") because InnoDB was doing such operations.
Back then, Sheeri K. Cabral (well-known in the MySQL blogosphere) published a cure which her team had discovered:
This loop can be broken by simply setting the Linux kernel's clock to the current time.
date -s "`date`"
While this looks like a no-op at first sight, I assume that it will reset the sub-second time information and so will have a small effect, which obviously is sufficient to terminate that loop. (In practice, you should stop your NTP daemon before changing the date, and restart it afterwards. The exact command will depend on your Linux distribution.)
Well, this was three years ago. The issue was reported to kernel developers, a fix was developed (which gives credit to Sheeri's report) and applied to newer kernels, and it w as also backported to older kernels. From my search, it seems that all reasonably maintained installations should have received that fix. For example, I have seen notices that Ubuntu 12.04 (since kernel 3.2.0-29.46) and 14.04 do have it, as does RedHat 6.4 (since kernel 2.6.32-298); for other distributions, I did not search.
In addition to a current kernel, you obviously need current packages including the leap second information. Typically, this would be "ntp", "ntpdate", "tzdata", and related ones. Don't forget to restart your NTP daemon after installing these updates!
So I wish all our readers that the leap second may not get them into trouble, or that (if worst comes to worst) the cure published by Sheeri may get them out.
However, I have to remind you of the old truth which is attributed to Mark Twain (sometimes also to Niels Bohr):
"It is difficult to make predictions, especially about the future."
FromDual Backup Manager for MySQL 1.2.2 has been released
FromDual has the pleasure to announce the release of the new version 1.2.2 of the popular Backup Manager for MySQL and MariaDB (fromdual_bman).
You can download the FromDual Backup Manager from here.
In the inconceivable case that you find a bug in the Backup Manager please report it to our Bugtracker.
Any feedback, statements and testimonials are welcome as well! Please send them to firstname.lastname@example.com from 1.2.x to 1.2.2 # cd ${HOME}/product # tar xf /download/fromdual_brman-1.2.2.tar.gz # rm -f fromdual_brman # ln -s fromdual_brman-1.2.2 fromdual_brman
Changes in FromDual Backup Manager 1.2.2 FromDual Backup Manager
It contains mainly fixes with brman catalog and physical backups.
You can verify your current FromDual Backup Manager version with the following command:fromdual_bman --version
- Archiving with physical backup bug fixed.
- Connect replaced by OO style and error exit fixed.
- Create catalog fixed.
- Archivedir without archive option does not make sense.
Wir suchen Dich: MySQL/MariaDB DBA für FromDual Support
FromDual ist das führende unabhängige Beratungs- und Dienstleistungsunternehmen für MySQL, Galera Cluster, MariaDB und Percona Server in Europa mit Hauptsitz in der Schweiz.
Unsere Kunden stammen hauptsächlich aus Europa und reichen vom kleinen Start-Up bis zur europäischen Top-500 Firma. Sie erhalten von uns Support bei Datenbank-Problemen, direkte Eingriffe als remote-DBA, Schulung für ihre DBAs und Entwickler sowie Beratung bei Architektur- und Design-Entscheidungen. Ausserdem entwickeln wir Tools rund um MySQL, schreiben Blog-Artikel und halten Vorträge bei Konferenzen.
Da unsere qualitativ guten Dienstleistungen immer mehr Kunden anziehen, brauchen wir Kollegen (m/w), welche selbst und mit uns wachsen wollen.Stellenbeschreibung
Wir suchen deutschsprachige Mitarbeiter (Sie oder Ihn) auf Junior- oder Senior-Level für Dienstleistungen rund um MySQL (hauptsächlich Support und remote-DBA Arbeiten) in Vollzeit. Primär solltest Du sicherstellen, dass die geschäftskritischen MySQL-Datenbanken unserer Kunden wie am Schnürchen laufen - und falls nicht, diese schnell wieder ans Laufen kriegen...
Unser/e "Wunschkandidat/in"
- hat Erfahrung im Betrieb kritischer und hoch verfügbarer produktiver Datenbanken hauptsächlich auf Linux,
- kennt Replikation in allen Variationen aus der täglichen Arbeit,
- weiß, wie die meist verbreiteten MySQL-HA-Setups funktionieren und wie man sie wieder effizient repariert, wenn ein Problem auftritt,
- ist sattelfest in SQL,
- bringt Erfahrung mit Galera Cluster mit,
- kann Bash skripten und einfache Programme in mindestens einer verbreiteten Programmier-/Skripting-Sprache (PHP, Bash, ...) erstellen.
Wir suchen Verstärkung, die von soliden Grundlagen aus auf dem Weg zu diesem Ideal ist.
Was wir von Dir erwarten:
- Kenntnisse in MySQL, Percona Server oder MariaDB oder Bereitschaft, sich diese anzueignen
- wissen, wie man kritische Datenbank-Systeme betreibt
- Verständnis, was beim Betrieb von Datenbanken falsch laufen kann
- selbständige Arbeitsweise (remote) mit Kommunikation über IRC, Skype, Mail und Telefon
- Kenntnisse des Linux Systems
DBA- oder DevOps-Erfahrungen wären z.B. eine gute fachliche Basis.
Du schätzt den direkten Kontakt mit Kunden, hast ein gutes Gespür für deren Probleme, kannst zuhören und findest schnell die eigentlichen Probleme. Du bist gewohnt, proaktiv zu handeln bevor etwas passiert, und führst den Kunden wieder auf den richtigen Pfad zurück.
Um Deine Arbeit erledigen zu können, arbeitest Du in einer europäischen Zeitzone. Deine Arbeitszeit kannst Du, der betrieblichen Situation entsprechend, flexibel gestalten. Wir erwarten, dass Du Deinen Beitrag zum Bereitschaftsdienst leistest. FromDual hat voraussichtlich keine Büroräumlichkeiten in Deinem Wohnort. Ein Umzug ist jedoch nicht notwendig: Wir ermöglichen Dir das Arbeiten von zu Hause aus oder unterstützen Dich bei der Suche einer geeigneten Arbeitsräumlichkeit in Deiner Nähe. Gute schriftliche und mündliche Englischkenntnisse sind erforderlich.Was wir Dir bieten:
- Deinen Leistungen angemessenes Gehalt.
- Möglichkeit Dich zum Top MySQL-Datenbankspezialisten zu entwickeln.
- Selbständiges Arbeiten.
- Verantwortung für Deine Projekte und Kunden zu übernehmen.
- Gute Kameradschaft im Team, sowie lockerer und angenehmer Umgang.
- Stellenbezogene Weiterbildungsmöglichkeiten.
- Teilnahme an Open Source Anlässen.
- Arbeit von Deinem bevorzugten Wohnort aus.
Du solltest in der Lage sein, die meiste Zeit selbständig zu arbeiten, zu denken und zu handeln und Dir neues Wissen selbständig anzueignen (durch Web-Suche, die MySQL-Dokumentation, Ausprobieren, etc.). Solltest Du dennoch einmal nicht weiterkommen, werden Dir Deine Kollegen von FromDual gerne helfen.
Wenn Du jemanden brauchst, der Dir die ganze Zeit Dein Händchen hält, ist FromDual nicht die richtige Wahl.
Wie geht es weiter
Wenn Du an dieser Chance interessiert bist und Du denkst, dass Du die passende Kandidatin oder der passende Kandidat bist, würden wir uns freuen, von Dir zu hören. Wir wissen, dass niemand 100% auf diese Stellenbeschreibung passt!
Bitte schicke Deinen ungeschönten Lebenslauf mit Deinen Gehaltsvorstellungen an firstname.lastname@example.com. Wenn Du mehr über diese Stelle erfahren oder wenn Du mit mir persönlich sprechen möchtest, ruf mich bitte an unter +41 79 830 09 33 (Oli Sennhauser, CTO). Bitte nur Bewerber, KEINE Headhunter!
Nachdem Du uns Deinen Lebenslauf zugeschickt hast, darfst Du Deine Fähigkeiten in einem kleinen MySQL-Test unter Beweis zu stellen. Nach bestandenem Test laden wir Dich für die finalen Interviews ein.
Controlling worldwide manufacturing plants with MySQL
A MySQL customer of FromDual has different manufacturing plants spread across the globe. They are operated by local companies. FromDuals customer wants to maintain the manufacturing receipts centralized in a MySQL database in the Head Quarter in Europe. Each manufacturing plant should only see their specific data.
Manufacturing log information should be reported backup to European Head Quarter MySQL database.
The process was designed as follows:Preparation of Proof of Concept (PoC)
To simulate all cases we need different schemas. Some which should be replicated, some which should NOT be replicated:CREATE DATABASE finance; CREATE TABLE finance.accounting ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `data` varchar(255) DEFAULT NULL, `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `data_rename` (`data`) ); CREATE DATABASE crm; CREATE TABLE crm.customer ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `data` varchar(255) DEFAULT NULL, `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `data_rename` (`data`) ); CREATE DATABASE erp; -- Avoid specifying Storage Engine here!!! CREATE TABLE erp.manufacturing_data ( id INT UNSIGNED NOT NULL AUTO_INCREMENT , manufacture_plant VARCHAR(32) , manufacture_info VARCHAR(255) , PRIMARY KEY (id) , KEY (manufacture_plant) ); CREATE TABLE erp.manufacturing_log ( id INT UNSIGNED NOT NULL AUTO_INCREMENT , manufacture_plant VARCHAR(32) , log_data VARCHAR(255) , PRIMARY KEY (id) , KEY (manufacture_plant) );
MySQL replication architecture
Before you start with such complicated MySQL set-ups it is recommended to make a little sketch of what you want to build:Preparing the Production Master database (Prod M1)
To make use of all the new and cool features of MySQL we used the new GTID replication. First we set up a Master (Prod M1) and its fail-over System (Prod M2) in the customers Head Quarter:# /etc/my.cnf [mysqld] binlog_format = row # optional log_bin = binary-log # mandatory, also on Slave! log_slave_updates = on # mandatory gtid_mode = on # mandatory enforce_gtid_consistency = on # mandatory server-id = 39 # mandatory
This step requires a system restart (one minute downtime).Preparing the Production Master standby database (Prod M2)
On Master (Prod M1):GRANT REPLICATION SLAVE ON *.* TO 'replication'@'192.168.1.%' IDENTIFIED BY 'secret'; mysqldump -u root --set-gtid-purged=on --master-data=2 --all-databases --triggers --routines --events > /tmp/full_dump.sql
On Slave (Prod M2):CHANGE MASTER TO MASTER_HOST='192.168.1.39', MASTER_PORT=3306 , MASTER_USER='replication', MASTER_PASSWORD='secret' , MASTER_AUTO_POSITION=1; RESET MASTER; -- On SLAVE! system mysql -u root < /tmp/full_dump.sql START SLAVE;
To make it easier for a Slave to connect to its master we set a VIP in front of those 2 database servers (VIP Prod). This VIP should be used by all applications in the head quarter and also the filter engines.Set-up filter engines (Filter BR and Filter CN)
To make sure every manufacturing plant sees only the data it is allowed to see we need a filtering engine between the production site and the manufacturing plant (Filter BR and Filter CN).
To keep this filter engine lean we use a MySQL instance with all tables converted to the Blackhole Storage Engine:# /etc/my.cnf [mysqld] binlog_format = row # optional log_bin = binary-log # mandatory, also on Slave! log_slave_updates = on # mandatory gtid_mode = on # mandatory enforce_gtid_consistency = on # mandatory server-id = 36 # mandatory default_storage_engine = blackhole
On the production master (Prod M1) we get the data as follows:mysqldump -u root --set-gtid-purged=on --master-data=2 --triggers --routines --events --no-data --databases erp > /tmp/erp_dump_nd.sql
The Filter Engines (Filter BR and CN) are set-up as follows::-- Here we can use the VIP! CHANGE MASTER TO master_host='192.168.1.33', master_port=3306 , master_user='replication', master_password='secret' , master_auto_position=1; RESET MASTER; -- On SLAVE! system cat /tmp/erp_dump_nd.sql | sed 's/ ENGINE=[a-zA-Z]*/ ENGINE=blackhole/' | mysql -u root START SLAVE;
Do not forget to also create the replication user on the filter engines.GRANT REPLICATION SLAVE ON *.* TO 'replication'@'192.168.1.%' IDENTIFIED BY 'secret';
Filtering out all non ERP schemata
We only want the erp schema to be replicated to the manufacturing plants, not the crm or the finance application. This we achieve with the following option on the filter engines:# /etc/my.cnf [mysqld] replicate_do_db = erp replicate_ignore_table = erp.manufacturing_log
MySQL row filtering
To achieve row filtering we use TRIGGERS. Make sure they are not replicated further down the hierarchy:SET SESSION SQL_LOG_BIN = 0; use erp DROP TRIGGER IF EXISTS filter_row; delimiter // CREATE TRIGGER filter_row BEFORE INSERT ON manufacturing_data FOR EACH ROW BEGIN IF ( NEW.manufacture_plant != 'China' ) THEN SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Row was filtered out.' , CLASS_ORIGIN = 'FromDual filter trigger' , SUBCLASS_ORIGIN = 'filter_row' , CONSTRAINT_SCHEMA = 'erp' , CONSTRAINT_NAME = 'filer_row' , SCHEMA_NAME = 'erp' , TABLE_NAME = 'manufacturing_data' , COLUMN_NAME = '' , MYSQL_ERRNO = 1644 ; END IF; END; // delimiter ; SET SESSION SQL_LOG_BIN = 0;
Up to now this would cause to stop replication for every filtered row. To avoid this we tell the Filtering Slaves to skip this error number:# /etc/my.cnf [mysqld] slave_skip_errors = 1644
Attaching production manufacturing Slaves (Man BR M1 and Man CN M1)
When we have finished everything on our head quarter site. We can start with the manufacturing sites (BR and CN):
On Master (Prod M1):mysqldump -u root --set-gtid-purged=on --master-data=2 --triggers --routines --events --where='manufacture_plant="Brazil"' --databases erp > /tmp/erp_dump_br.sql mysqldump -u root --set-gtid-purged=on --master-data=2 --triggers --routines --events --where='manufacture_plant="China"' --databases erp > /tmp/erp_dump_cn.sql
On the Manufacturing Masters (Man BR M1 and Man BR M2). Here we do NOT use a VIP because we think a blackhole storage engine is robust enough as master:CHANGE MASTER TO master_host='192.168.1.43', master_port=3306 , master_user='replication', master_password='secret' , master_auto_position=1; RESET MASTER; -- On SLAVE! system cat /tmp/erp_dump_br.sql | mysql -u root START SLAVE;
The standby manufacturing (Man BR M2 and Man CN M2) database is created in the same way as the production manufacturing database on the master.Testing replication from HQ to manufacturing plants
First we make sure, crm and finance is not replicated out and replication also does not stop (on Prod M1):INSERT INTO finance.accounting VALUES (NULL, 'test data over VIP', NULL); INSERT INTO finance.accounting VALUES (NULL, 'test data over VIP', NULL); INSERT INTO crm.customer VALUES (NULL, 'test data over VIP', NULL); INSERT INTO crm.customer VALUES (NULL, 'test data over VIP', NULL); UPDATE finance.accounting SET data = 'Changed data'; UPDATE crm.customer SET data = 'Changed data'; DELETE FROM finance.accounting WHERE id = 1; DELETE FROM crm.customer WHERE id = 1; SELECT * FROM finance.accounting; SELECT * FROM crm.customer; SHOW SLAVE STATUS\G
The schema filter seems to work correctly. Then we check if also the row filter works correctly. For this we have to run the queries in statement based replication (SBR)! Otherwise the trigger would not fire:use mysql INSERT INTO erp.manufacturing_data VALUES (NULL, 'China', 'Highly secret manufacturing info as RBR.'); INSERT INTO erp.manufacturing_data VALUES (NULL, 'Brazil', 'Highly secret manufacturing info as RBR.'); -- This needs SUPER privilege... :-( SET SESSION binlog_format = STATEMENT; -- Caution those rows will NOT be replicated!!! -- See filter rules for SBR INSERT INTO erp.manufacturing_data VALUES (NULL, 'China', 'Highly secret manufacturing info as SBR lost.'); INSERT INTO erp.manufacturing_data VALUES (NULL, 'Brazil', 'Highly secret manufacturing info as SBR lost.'); use erp INSERT INTO manufacturing_data VALUES (NULL, 'China', 'Highly secret manufacturing info as SBR.'); INSERT INTO manufacturing_data VALUES (NULL, 'Brazil', 'Highly secret manufacturing info as SBR.'); INSERT INTO manufacturing_data VALUES (NULL, 'Germany', 'Highly secret manufacturing info as SBR.'); INSERT INTO manufacturing_data VALUES (NULL, 'Switzerland', 'Highly secret manufacturing info as SBR.'); SET SESSION binlog_format = ROW; SELECT * FROM erp.manufacturing_data;
Production data back to head quarter
Now we have to take care about the production data on their way back to the HQ. To achieve this we use the new MySQL 5.7 feature called multi source replication. For multi source replication the replication repositories must be kept in tables instead of files:# /etc/my.cnf [mysqld] master_info_repository = TABLE # mandatory relay_log_info_repository = TABLE # mandatory
Then we have to configure 2 replication channels from Prod M1 to their specific manufacturing masters over the VIP (VIP BR and VIP CN):CHANGE MASTER TO MASTER_HOST='192.168.1.98', MASTER_PORT=3306 , MASTER_USER='replication', MASTER_PASSWORD='secret' , MASTER_AUTO_POSITION=1 FOR CHANNEL "manu_br"; CHANGE MASTER TO MASTER_HOST='192.168.1.99', MASTER_PORT=3306 , MASTER_USER='replication', MASTER_PASSWORD='secret' , MASTER_AUTO_POSITION=1 FOR CHANNEL "manu_cn"; START SLAVE FOR CHANNEL 'manu_br'; START SLAVE FOR CHANNEL 'manu_cn'; SHOW SLAVE STATUS FOR CHANNEL 'manu_br'\G SHOW SLAVE STATUS FOR CHANNEL 'manu_cn'\G
Avoid to configure and activate the channels on Prod M2 as well.Testing back replication from manufacturing plants
Brazil on Man BR M1:INSERT INTO manufacturing_log VALUES (1, 'Production data from Brazil', 'data');
China on Man CN M1:INSERT INTO manufacturing_log VALUES (2, 'Production data from China', 'data');
For testing:SELECT * FROM manufacturing_log;
Make sure you do not run into conflicts (Primary Key, AUTO_INCREMENTS). Make sure filtering is defined correctly!
To check the different channel states you can use the following command:SHOW SLAVE STATUS\G or SELECT ras.channel_name, ras.service_state AS 'SQL_thread', ras.remaining_delay , CONCAT(user, '@', host, ':', port) AS user , rcs.service_state AS IO_thread, REPLACE(received_transaction_set, '\n', '') AS received_transaction_set FROM performance_schema.replication_applier_status AS ras JOIN performance_schema.replication_connection_configuration AS rcc ON rcc.channel_name = ras.channel_name JOIN performance_schema.replication_connection_status AS rcs ON ras.channel_name = rcs.channel_name ;
Troubleshooting Inject empty transaction
If you try to skip a transaction as you did earlier (SQL_SLAVE_SKIP_COUNTER) you will face some problems:STOP SLAVE; ERROR 1858 (HY000): sql_slave_skip_counter can not be set when the server is running with @@GLOBAL.GTID_MODE = ON. Instead, for each transaction that you want to skip, generate an empty transaction with the same GTID as the transaction
To skip the next transaction you have find the ones applied so far:SHOW SLAVE STATUS\G ... Executed_Gtid_Set: c3611091-f80e-11e4-99bc-28d2445cb2e9:1-20
then tell MySQL to skip this by injecting a new empty transaction:SET SESSION GTID_NEXT='c3611091-f80e-11e4-99bc-28d2445cb2e9:21'; BEGIN; COMMIT; SET SESSION GTID_NEXT='AUTOMATIC'; SHOW SLAVE STATUS\G ... Executed_Gtid_Set: c3611091-f80e-11e4-99bc-28d2445cb2e9:1-21 START SLAVE;
Revert from GTID-based replication to file/position-based replication
If you want to fall-back from MySQL GTID-based replication to file/position-based replication this is quite simple:CHANGE MASTER TO MASTER_AUTO_POSITION = 0;
MySQL Support and Engineering
If you need some help or support our MySQL support and engineering team is happy to help you.
Logging Galera Cluster conflicts
We typically suggest our customers to use our MySQL/Galera Cluster my.cnf configuration template to avoid MySQL configuration and performance problems.
And we are paranoid as well. Thus we enable all useful logging:wsrep_log_conflicts = 1
But this has also some consequences of more visibility...
If you monitor carefully your Galera Cluster for example with the FromDual Performance Monitor for MySQL and MariaDB, you might probably see some strange values increasing from time to time:mysql< SHOW GLOBAL STATUS LIKE 'wsrep_local_%r_s'; +---------------------------+-------+ | Variable_name | Value | +---------------------------+-------+ | wsrep_local_cert_failures | 42 | | wsrep_local_bf_aborts | 13 | +---------------------------+-------+
Those values are indicators that some transactions (Galera write sets) did to not succeed and were aborted by Galera. In this case the paranoid logging helps to find, what exactly was aborted and possibly helps to find out, if this can or should be fixed:150410 1:44:18 [Note] WSREP: cluster conflict due to certification failure for threads: 150410 1:44:18 [Note] WSREP: Victim thread: THD: 151856, mode: local, state: executing, conflict: cert failure, seqno: 30399304 SQL: UPDATE login SET lTsexpire = UNIX_TIMESTAMP(NOW()) + lTimeout WHERE lSessionId = 'va3ta7besku82k56ncv3bnhlj5' *** Priority TRANSACTION: TRANSACTION 464359568, ACTIVE 0 sec starting index read mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s) MySQL thread id 4, OS thread handle 0x7f1c0916c700, query id 8190690 Update_rows_log_event::find_row(30399302) *** Victim TRANSACTION: TRANSACTION 464359562, ACTIVE 0 sec mysql tables in use 1, locked 1 2 lock struct(s), heap size 360, 1 row lock(s), undo log entries 1 MySQL thread id 151856, OS thread handle 0x7f1c09091700, query id 8190614 172.20.100.11 sam_angiz query end UPDATE login SET lTsexpire = UNIX_TIMESTAMP(now()) + lTimeout WHERE lSessionId = 'va3ta7besku82k56ncv3bnhlj5' *** WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 835205 page no 3 n bits 72 index `PRIMARY` of table `fromdual`.`login` trx table locks 1 total table locks 2 trx id 464359562 lock_mode X locks rec but not gap lock hold time 0 wait time before grant 0 150410 1:44:18 [Note] WSREP: cluster conflict due to high priority abort for threads: 150410 1:44:18 [Note] WSREP: Winning thread: THD: 4, mode: applier, state: executing, conflict: no conflict, seqno: 30399302 SQL: (null) 150410 1:44:18 [Note] WSREP: Victim thread: THD: 151856, mode: local, state: committing, conflict: no conflict, seqno: -1 SQL: UPDATE login SET lTsexpire = UNIX_TIMESTAMP(now()) + lTimeout WHERE lSessionId = 'va3ta7besku82k56ncv3bnhlj5'
In the above Galera conflict 2 login transactions where running at the same time. They both come with the same Session ID and want to update the expiry timestamp. Now how to solve or fix this:
- First check, if this table has a Primary Key (tables without a PK causes full table scans which can last for long time, increasing the chance for conflicts).
- Second check, if there is a (UNIQUE?) index on lSessionId. A missing index leads to full table scans which increases the chance for conflicts.
- Third check WHY 2 logins from the same Session ID can arrive at the same time (within 1 second) on 2 different Galera nodes (Ajax requests, etc...). Try to avoid such situations.
Galera Cluster last inactive check and VMware snapshots
From time to time we see at Galera Cluster customer engagements the following, for me scary, warning in the MySQL error log:[Warning] WSREP: last inactive check more than PT1.5S ago (PT7.06159S), skipping check
We mostly see this in VMware set-ups. Some further enquiry with the Galera developers did not give a satisfying answer:
This can be seen on bare metal as well - with poorly configured mysqld, O/S, or simply being overloaded. All it means is that this thread could not get CPU time for 7.1 seconds. You can imagine that access to resources in virtual machines is even harder (especially I/O) than on bare metal, so you will see this in virtual machines more often.
This is not a Galera specific issue (it just reports being stuck, other mysqld threads are equally stuck) so there is no configuration options for that. You simply must make sure that your system and mysqld are properly configured, that there is enough RAM (buffer pool not over provisioned), that there is swap, that there are proper I/O drivers installed on guest and so on.
Basically, Galera runs in virtual machines as well as well virtual machines approximates bare metal.
We were still suspecting that this is somehow VMware related. This week we had the chance to investigate... At 01:36 am node Galera2 lost connection to the Cluster and became NON-PRIMARY. This is basically a bad sign:150401 1:36:15 [Warning] WSREP: last inactive check more than PT1.5S ago (PT5.08325S), skipping check 150401 1:36:15 [Note] WSREP: (09c6b2f2, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.42.2:4567 150401 1:36:16 [Note] WSREP: view(view_id(NON_PRIM,09c6b2f2,30) memb { 09c6b2f2,0 } joined { } left { } partitioned { ce6bf2e1,0 d1f9bee0,0 }) 150401 1:36:16 [Note] WSREP: view(view_id(NON_PRIM,09c6b2f2,31) memb { 09c6b2f2,0 } joined { } left { } partitioned { ce6bf2e1,0 d1f9bee0,0 }) 150401 1:36:16 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 150401 1:36:16 [Note] WSREP: Flow-control interval: [16, 16] 150401 1:36:16 [Note] WSREP: Received NON-PRIMARY. 150401 1:36:16 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 26304132) 150401 1:36:16 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 150401 1:36:16 [Note] WSREP: Flow-control interval: [16, 16] 150401 1:36:16 [Note] WSREP: Received NON-PRIMARY. 150401 1:36:16 [Warning] WSREP: Send action {(nil), 328, TORDERED} returned -107 (Transport endpoint is not connected) 150401 1:36:16 [Note] WSREP: New cluster view: global state: dcca768c-b5ad-11e3-bbc0-fb576fb3c451:26304132, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3 150401 1:36:17 [Note] WSREP: (09c6b2f2, 'tcp://0.0.0.0:4567') reconnecting to d1f9bee0 (tcp://192.168.42.1:4567), attempt 0
I suspected, after some investigation with the FromDual Performance Monitor for MySQL and MariaDB, that the database backup (mysqldump) could be the reason. It was not. But the customer explained, that after the database backup they do a VMware snapshot.
And when we compared our problem with the backup log file:2015/04/01 01:35:08 [3] backup.fromdual.com: Creating a snapshot of galera3 2015/04/01 01:35:16 [3] backup.fromdual.com: Created a snapshot of galera3 2015/04/01 01:35:23 [3] backup.fromdual.com: galera3: backup the changed blocks of disk 'Festplatte 1' using NBD transport 2015/04/01 01:36:10 [3] backup.fromdual.com: galera3: saving the Change Block Tracking's reference for disk 'Festplatte 1' 2015/04/01 01:36:10 [3] backup.fromdual.com: Removing Arkeia's snapshot of galera3
we can see that our problem pretty much started with the end of the WMware snapshot (01:36:10 + 5.08 = 1:36:15). By the way: For such kind of investigations it is always good to have a ntp daemon for time synchronization running. Otherwise problem investigation becomes much harder...
Some more and deeper investigation shows that we loose from time to time nodes during VMware snapshots (galera3). But they recover quickly because they can do an IST. In worst case we can loose 2 nodes and then the whole Galera Cluster has gone.192.168.42.3 / node Galera3 2015-04-10 01:44:00 [3] backup.fromdual.com: Creating a snapshot of galera3 2015-04-10 01:44:08 [3] backup.fromdual.com: Created a snapshot of galera3 2015-04-10 01:44:15 [3] backup.fromdual.com: galera3: backup the changed blocks of disk 'Festplatte 1' using NBD transport 2015-04-10 01:45:39 [3] backup.fromdual.com: galera3: saving the Change Block Tracking's reference for disk 'Festplatte 1' 2015-04-10 01:45:39 [3] backup.fromdual.com: Removing Arkeia's snapshot of galera3
150410 1:44:07 [Note] WSREP: (158f71de, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://galera1:4567 tcp://galera2:4567 150410 1:44:07 [Warning] WSREP: last inactive check more than PT1.5S ago (PT7.06159S), skipping check 150410 1:44:08 [Note] WSREP: Received NON-PRIMARY. 150410 1:44:10 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 30399299) 150410 1:44:11 [Warning] WSREP: Gap in state sequence. Need state transfer. 150410 1:44:11 [Note] WSREP: Prepared IST receiver, listening at: tcp://galera3:4568 150410 1:44:11 [Note] WSREP: Member 0.0 (galera3) requested state transfer from '*any*'. Selected 2.0 (galera2)(SYNCED) as donor. 150410 1:44:11 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 30399309) 150410 1:44:11 [Note] WSREP: Requesting state transfer: success, donor: 2 150410 1:44:11 [Note] WSREP: 2.0 (galera2): State transfer to 0.0 (galera3) complete. 150410 1:44:11 [Note] WSREP: Member 2.0 (galera2) synced with group. 150410 1:44:11 [Note] WSREP: Receiving IST: 8 writesets, seqnos 30399291-30399299 150410 1:44:11 [Note] WSREP: IST received: dcca768c-b5ad-11e3-bbc0-fb576fb3c451:30399299 150410 1:44:11 [Note] WSREP: 0.0 (galera3): State transfer from 2.0 (galera2) complete. 150410 1:44:11 [Note] WSREP: Shifting JOINER -> JOINED (TO: 30399309) 150410 1:44:11 [Note] WSREP: Member 0.0 (galera3) synced with group. 150410 1:44:11 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 30399309) 150410 1:44:11 [Note] WSREP: Synchronized with group, ready for connections 150410 1:44:13 [Note] WSREP: (158f71de, 'tcp://0.0.0.0:4567') turning message relay requesting off 150410 1:45:42 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.47388S), skipping check 150410 1:45:43 [Note] WSREP: (158f71de, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://galera1:4567 tcp://galera2:4567 150410 1:45:44 [Note] WSREP: (158f71de, 'tcp://0.0.0.0:4567') reconnecting to 54de92f8 (tcp://galera1:4567), attempt 0 150410 1:45:44 [Note] WSREP: (158f71de, 'tcp://0.0.0.0:4567') reconnecting to c9d964d3 (tcp://galera2:4567), attempt 0 150410 1:45:48 [Note] WSREP: (158f71de, 'tcp://0.0.0.0:4567') turning message relay requesting off 150410 1:47:26 [Note] WSREP: (158f71de, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://galera1:4567 150410 1:47:27 [Note] WSREP: (158f71de, 'tcp://0.0.0.0:4567') reconnecting to 54de92f8 (tcp://galera1:4567), attempt 0 150410 1:47:31 [Note] WSREP: (158f71de, 'tcp://0.0.0.0:4567') turning message relay requesting off
192.168.42.1 / node Galera1 2015-04-10 01:47:24 [3] backup.fromdual.com: Creating a snapshot of galera1 2015-04-10 01:47:29 [3] backup.fromdual.com: Created a snapshot of galera1 2015-04-10 01:47:40 [3] backup.fromdual.com: galera1: backup the changed blocks of disk 'Festplatte 1' using NBD transport 2015-04-10 01:48:43 [3] backup.fromdual.com: galera1: saving the Change Block Tracking's reference for disk 'Festplatte 1' 2015-04-10 01:48:44 [3] backup.fromdual.com: Removing Arkeia's snapshot of galera1 150410 1:44:02 [Note] WSREP: (54de92f8, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://galera3:4567 150410 1:44:04 [Note] WSREP: (54de92f8, 'tcp://0.0.0.0:4567') reconnecting to 158f71de (tcp://galera3:4567), attempt 0 150410 1:44:12 [Note] WSREP: Member 0.0 (galera3) requested state transfer from '*any*'. Selected 2.0 (galera2)(SYNCED) as donor. 150410 1:45:43 [Note] WSREP: (54de92f8, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://galera3:4567 150410 1:45:44 [Note] WSREP: (54de92f8, 'tcp://0.0.0.0:4567') reconnecting to 158f71de (tcp://galera3:4567), attempt 0 150410 1:45:48 [Note] WSREP: (54de92f8, 'tcp://0.0.0.0:4567') turning message relay requesting off 150410 1:47:27 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.66452S), skipping check 150410 1:47:27 [Note] WSREP: (54de92f8, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://galera3:4567 150410 1:47:30 [Note] WSREP: (54de92f8, 'tcp://0.0.0.0:4567') turning message relay requesting off
192.168.42.2 / node Galera2 2015-04-10 02:09:55 [3] backup.fromdual.com: Creating a snapshot of galera2 2015-04-10 02:09:58 [3] backup.fromdual.com: Created a snapshot of galera2 2015-04-10 02:10:05 [3] backup.fromdual.com: galera2: backup the changed blocks of disk 'Festplatte 1' using NBD transport 2015-04-10 02:10:53 [3] backup.fromdual.com: galera2: saving the Change Block Tracking's reference for disk 'Festplatte 1' 2015-04-10 02:10:54 [3] backup.fromdual.com: Removing Arkeia's snapshot of galera2
150410 1:44:02 [Note] WSREP: (c9d964d3, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://galera3:4567 150410 1:44:03 [Note] WSREP: (c9d964d3, 'tcp://0.0.0.0:4567') reconnecting to 158f71de (tcp://galera3:4567), attempt 0 150410 1:44:08 [Warning] WSREP: discarding established (time wait) 158f71de (tcp://192.168.42.3:4567) 150410 1:44:11 [Note] WSREP: Member 0.0 (galera3) requested state transfer from '*any*'. Selected 2.0 (galera2)(SYNCED) as donor. 150410 1:44:13 [Note] WSREP: (c9d964d3, 'tcp://0.0.0.0:4567') turning message relay requesting off 150410 1:45:43 [Note] WSREP: (c9d964d3, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://galera3:4567 150410 1:45:44 [Note] WSREP: (c9d964d3, 'tcp://0.0.0.0:4567') reconnecting to 158f71de (tcp://galera3:4567), attempt 0 150410 1:45:48 [Note] WSREP: (c9d964d3, 'tcp://0.0.0.0:4567') turning message relay requesting off 150410 1:47:26 [Note] WSREP: (c9d964d3, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://galera1:4567 150410 1:47:27 [Note] WSREP: (c9d964d3, 'tcp://0.0.0.0:4567') reconnecting to 54de92f8 (tcp://galera1:4567), attempt 0 150410 1:47:30 [Note] WSREP: (c9d964d3, 'tcp://0.0.0.0:4567') turning message relay requesting off 150410 2:09:57 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.83618S), skipping check
The backups are done with the 2 options:
Possibly this is the reason and one should disable those features in combination with Galera. Further investigation is going on. In worst case VMware snapshotting with Galera should be avoided.
Rename MySQL Partition
Before I forget it and have to search again here a short note about how to rename a MySQL Partition:
My dream:ALTER TABLE history RENAME PARTITION p2015_kw10 INTO p2015_kw09;
In reality: ALTER TABLE history REORGANIZE PARTITION p2015_kw10 INTO ( PARTITION p2015_kw09 VALUES LESS THAN (UNIX_TIMESTAMP('2015-03-02 00:00:00')) );
Caution: REORGANIZE PARTITION causes a full copy of the whole partition!
Hint: I assume it would be very easy for MySQL or MariaDB to make this DDL command an in-place operation...
MySQL Partitioning was introduced in MySQL 5.1.
MySQL Enterprise Incremental Backup simplified
MySQL Enterprise Backup (MEB) has the capability to make real incremental (differential and cumulative?) backups. The actual releases are quite cool and you should really look at it...
Unfortunately the original MySQL documentation is much too complicated for my simple mind. So I did some testing and simplified it a bit for our customers...
If you want to dive into the original documentation please look here: Making an Incremental Backup .
If you want to use MySQL Enterprise Backup please let us know and we send you a quote...Prepare MySQL Backup infrastructure mkdir /backup/full /backup/incremental1 /backup/incremental2
Full MySQL Backup mysqlbackup --defaults-file=/etc/my.cnf --user=root --backup-dir=/backup/full backup mysqlbackup --defaults-file=/etc/my.cnf --user=root --backup-dir=/backup/full apply-log
First MySQL Incremental Backup mysqlbackup --defaults-file=/etc/my.cnf --user=root --incremental --incremental-base=dir:/backup/full --incremental-backup-dir=/backup/incremental1 backup mysqlbackup --defaults-file=/etc/my.cnf --user=root --backup-dir=/backup/full --incremental-backup-dir=/backup/incremental1 apply-incremental-backup
Second MySQL Incremental Backup mysqlbackup --defaults-file=/etc/my.cnf --user=root --incremental --incremental-base=dir:/backup/full --incremental-backup-dir=/backup/incremental2 backup mysqlbackup --defaults-file=/etc/my.cnf --user=root --backup-dir=/backup/full --incremental-backup-dir=/backup/incremental2 apply-incremental-backup
and so on...MySQL Restore mysqlbackup --defaults-file=/etc/my.cnf --user=root --backup-dir=/backup/full copy-back
Have fun with MySQL Enterprise Backup. If you need any help with your MySQL Backup concept, please let us know.
Creating Event Handlers with MySQL Enterprise Monitor
MySQL Enterprise Monitor (MEM) has by default no Event Handlers created and activated. These Event Handlers you have to define yourself according to your needs.
In this article we discuss how to create MySQL Enterprise Monitor Event Handlers with MEM v.3.0.18. For other (older) versions the steps may vary...Task: Event Handler for maximum Connections reached
We would like to be notified by MySQL Enterprise Monitor when the number of connections is near to max_connections.
For this we search first which Advisors are available at all: Configuration -> Advisors -> Availability.
Here we can see that we have an Advisor called Maximum Connection Limit Nearing Or Reached which is scheduled for every 5 minutes and has thresholds at 75, 85, 95 and 100%:
Now we know which Advisor should create and Event. As a next step we have to create and Event which should be triggered: Configuration -> Event Handling -> Create Event Handler.
Here we can create and Event with all its needed configuration: Events -> All -> server.
If we look at the Events we can even see the detailed description and how the values for the Event are collected:
Task: Event Handler for used disk space
For this Event Handler we need the Advisor Filesystem Free Space under Operating System:
In this advisor we can configure the Threshold as well:
In the Event Handler we can define which Assets shall be monitored. For example the mountpoint: /.
Local disks can only be monitored, if a local MySQL Enterprise Monitor Agent is installed. An agent-less MySQL Enterprise Monitor cannot monitor local disk resources...
Have fun using the MySQL Enterprise Monitor. If you need any help in installing or configuring MEM do not hesitate to contact us.
All these functions are also implemented in the FromDual Performance Monitor for MySQL. If you want to relay on Open Source technology only you should consider our Performance Monitor.
Nagios and Icinga plug-ins for MySQL 1.0.0 have been released
FromDual has the pleasure to announce the release of the new version 1.0.0 of its widely used Nagios and Icinga plug-ins for MySQL, Galera Cluster, MariaDB and Percona Server.
All plug-ins are basically renewed and should now work all correctly.
The new Nagios/Icinga plug-ins can be downloaded here.
In the inconceivable case that you find a bug in the Nagios/Icinga plug-ins please report it to our bug tracker.
Any feedback, statements and testimonials are welcome as well! Please send them to email@example.com of the current functionality
Details about the functionality and the usage of each plug-in you get with the option: --help.
The following Nagios/Icinga plug-in for MySQL and MariaDB are currently available:check_db_mysql.pl
This Nagios/Icinga plug-in alerts you if your MySQL database is not up and running.check_errorlog_mysql.pl and errorLogFilterRules.pm
This Nagios/Icinga plug-in alerts you if it finds some suspicious messages in the MySQL error log.
The rules which messages should be ignored can be found in the file errorLogFilterRules.pm. If you want to add your own filter rules please add them in this file as well.
This Nagios/Icinga plug-in alerts you if the actual number of nodes in your a Galera Cluster is not the expected one.check_repl_mysql_cnt_slave_hosts.pl
This Nagios/Icinga plug-in alerts you if your MySQL Slaves have not reported to their Master properly their existence with the report_host variable.check_repl_mysql_heartbeat.pl
This Nagios/Icinga plug-in alerts you if your MySQL Slave is too many heartbeats behind its Master.check_repl_mysql_io_thread.pl
This Nagios/Icinga plug-in alerts you if your MySQL Slaves IO thread is not up an running.check_repl_mysql_read_exec_pos.pl
This Nagios/Icinga plug-in alerts you if your MySQL Slaves read and execution positions differ too much.check_repl_mysql_readonly.pl
This Nagios/Icinga plug-in alerts you if your MySQL Slave is NOT set to readonly.check_repl_mysql_seconds_behind_master.pl
This Nagios/Icinga plug-in alerts you if your MySQL Slave falls too many seconds behind its Master.check_repl_mysql_sql_thread.pl
This Nagios/Icinga plug-in alerts you if your Slaves SQL thread is not up an running.perf_mysql.pl
This Nagios/Icinga plug-in gathers MySQL and MariaDB performance data.Changes in FromDual Nagios/Icinga plug-ins 1.0.0 All plug-ins
- Usage was improved. The usage can be shown with the --help option.
- Usage states now which GRANT privileges are needed for a specific plug-in.
- Examples added how to use each plug-in.
- Default socket location moved from /tmp/mysql.sock to /var/run/mysqld/mysqld.sock.
- New host/socket convention implemented in all scripts similar to MySQL client tools.
- -epn tag added for Icinga.
- Some bugs fixed.
- More filtering rules added.
- Filtering rules separated into own file.
- Entry point finding problem fixed.
- Script name fixed.
- Unknown command problem with Galera Cluster caught.
- mysqladmin ping removed and implemented in Perl.
For subscriptions for commercial use of this software please get in contact with us.
Linuxtag: Knowledge and People - and New Colleagues?
At FromDual, we are currently preparing for our participation in the "Chemnitzer Linux-Tage" in March.
While we don't yet know whether the programme committee accepted our proposed talks, we will have a booth and hope for interesting exchanges with others from the MySQL, database, Linux, ... world. Of course, we will also mention that we are looking for additional colleagues - there are so many tasks that we need more people to handle them all. (In case you got curious, look here: http://www.fromdual.com/mysql-dba-2014-12-de )
While I attended the Chemnitzer Linux-Tage already last year (thank you, Frank Hofmann, for recommending them!), it will be the first time there for FromDual as a company. I enjoyed the informal atmosphere, which made it easy to get into contact with new people, to learn, and to discuss both facts and opinions.
To those of you who didn't yet attend such events, I can only say: You are missing some great experiences! It is good to get stimulated by others, by their problems and their experiences, so that one's thoughts are broadened and start considering new fields. Often, they lead to insights that are helpful for one's own daily work.
In the opposite direction, we at FromDual hope that during the Linux-Tage we can help others to make (better) use of MySQL in all its variants, with its surrounding tools and products, so that everybody will profit.
The Chemnitzer Linux-Tage will take place on March 21 and 22, 2015, in Chemnitz, Germany.
Parts of the programme will probably be delivered in English, so even those of you who don't understand German might enjoy coming there.
If you now consider a visit, you will find further information here:
We will be happy to meet you in Chemnitz - to talk about whatever interests both you and us, maybe even about you joining us?
Download MySQL Enterprise Features
MySQL provides some great enterprise features beside the MySQL Server. The ones we are asked the most at customers are:
- MySQL Enterprise Backup (MEB)
- MySQL Enterprise Monitor (MEM) and
- MySQL Enterprise Workbench (MWB)
MySQL Enterprise Backup (MEB) is an alternative to the mysqldump backup utility. Its big advantage is its fast backup but even faster restore performance. This is a must for all MySQL users having bigger databases than let's say 10 to 20 Gigabytes and/or having hard requirements for restore times (MTTR).
Last implementation tests we did with a customer for an about 30 Gbyte database were:MEBmysqldumpBackup10 minutes18 minutesRestore12 minutes80 minutes
If you need our help implementing MySQL Enterprise Backup into your backup infrastructure please get in contact with us. MySQL Enterprise Backup also seamlessly integrates into the FromDual Backup Manager for MySQL.MySQL Enterprise Monitor (MEM)
MySQL Enterprise Monitor (MEM) is an Enterprise Monitoring Solution for MySQL which Monitors your business critical MySQL databases. Various predefined advisors rise an alert if something with your precious MySQL database does not work as expected.
Our alternative competitive product is the FromDual Performance Monitor for MySQL.MySQL Enterprise Workbench (MWB)
MySQL Enterprise Workbench (MWB) is the tool modern MySQL Database administrators use to operate their MySQL databases. Old fashion ones still use the CLI... MySQL developers can easily write and test database queries and develop their data model with the ER diagram modeller.Download Enterprise tools
But how can we get now to these precious tools? This is quite easy following the screen shots below:
As a fist step you go to www.mysql.com/downloads:
Here you can find a link to Oracle eDelivery which is the MySQL/Oracle download facility. Then you get to the welcome screen:
Before you get access to the software you have to Sign In with your Oracle/MySQL customer account if you have one. If you do not have an account yet you can Create an Account to get to the software:
Then you have to agree (2 times) to the Terms & Restrictions (this is what Oracle is really good in). Once to the Oracle Trial License Agreement and once to the Export Restrictions:
Then you get to the Media Pack Search. Here you can define what product you are interested in and on which platform you are using it. Unfortunately a sub-product filter cannot be chosen. So you get a long list to pick your final package from:
An last you can download your product of desire:
Unfortunately the packages get some silly names like V59684-01.zip instead of meaningful names. But with the following command you get some information what is included in the package:unzip -l V59684-01.zip Archive: V59684-01.zip Length Date Time Name --------- ---------- ----- ---- 2958631 2014-11-04 13:29 meb-3.11.1-linux-glibc2.5-x86-64bit.tar.gz 185 2014-11-05 08:30 meb-3.11.1-linux-glibc2.5-x86-64bit.tar.gz.asc 77 2014-11-04 13:29 meb-3.11.1-linux-glibc2.5-x86-64bit.tar.gz.md5 2130 2014-11-05 15:06 README.txt --------- ------- 2961023 4 files
Have fun trying the MySQL Enterprise Features. If you need any help installing or integrating them into your infrastructure, do not hesitate to contact FromDual.
MySQL table Point-in-Time-Recovery from mysqldump backup
Sometimes we face the situation where we have a full MySQL database backup done with mysqldump and then we have to restore and recover just one single table out of our huge mysqldump file.
Further our mysqldump backup was taken hours ago so we want to recover all the changes on that table since our backup was taken up to the end.
In this blog article we cover all the steps needed to achieve this goal for MySQL and MariaDB.
Recommendation: It is recommended to do theses steps on a testing system and then dump and restore your table back to the production system. If you do it directly on your production system you have to know exactly what you are doing...
Further this process should be tested carefully and regularly to get familiar with it and to assure your backup/restore/recovery procedure works properly.
The table we want to recover is called test.test from our backup full_dump.sql.gz. As a first step we have to do the recovery with the following command to our test database:shell> zcat full_dump.sql.gz | extract_table.py --database=test --table=test | mysql -u root
The script extract_table.py is part of the FromDual Recovery Manager to extract one single table from a mysqldump backup.
As a next step we have to extract the binary log file and its position where to start recovery from out of our dump:shell> zcat full_dump.sql.gz | head -n 25 | grep CHANGE CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000216', MASTER_LOG_POS=1300976;
Then we have to find out where we want to stop our Point-in-Time-Recovery. The need for recover is possibly due to a TRUCATE TABLE command or similar operation executed on the wrong system or it is just a time somebody has indicated us to recover to. The position to stop we can find as follows:shell> mysqlbinlog -v mysql-bin.000216 | grep -B5 TRUNCATE --color #150123 19:53:14 server id 35622 end_log_pos 1302950 CRC32 0x24471494 Xid = 3803 COMMIT/*!*/; # at 1302950 #150123 19:53:14 server id 35622 end_log_pos 1303036 CRC32 0xf9ac63a6 Query thread_id=54 exec_time=0 error_code=0 SET TIMESTAMP=1422039194/*!*/; TRUNCATE TABLE test
And as a last step we have to apply all the changes from the binary log to our testing database:shell> mysqlbinlog --disable-log-bin --database=test --start-position=1300976 --stop-position=1302950 mysql-bin.000216 | mysql -u root --force
Now the table test.test is recovered to the wanted point in time and we can dump and restore it to its final location back to the production database.shell> mysqldump --user=root --host=testing test test | mysql --user=root --host=production test
This process has been tested on MySQL 5.1.73, 5.5.38, 5.6.22 and 5.7.5 and also on MariaDB 10.0.10 and 10.1.0.
Introducing Myself: Jörg Brühe
Often, in such introductions people use the phrase of "the new kid on the block". I won't. If I am to use those words, I will arrange them as "the kid on the new block". The reason is that I don't feel as a new kid in the MySQL village (or is it a city?), let alone in DBMS country.
Ever since I left university (Technical University of Berlin, Germany), I have been involved in SQL DBMS development. After my previous product's team had been dissolved in Berlin and maintenance moved to Riga, Latvia, as a cost-cutting measure, I joined MySQL AB in 2004 as a member of the Build Team. Some of you will remember my name from MySQL release announcements, others may have seen it in bug reports, newsgroup entries, or other MySQL communication. Together with the other MySQL colleagues, I got acquired by Sun in 2008 and then by Oracle in 2010.
In 2012, I decided I wanted a culture change, from a highly (and very strictly) organized corporation to a smaller, more flexible entity. That brought me to an internet site, where I worked as a DBA, concentrating on their (more than 100) MySQL installations. It was very interesting to view MySQL from the DBA side now, I got a big load of new experiences and learned a lot about using, running, and automating MySQL (rather than building and testing it).
That might have continued, but as usual the world is changing, and my DBA job was no exception: Company strategy was to shift those tasks more to the various application teams and to do without a central point for MySQL. It is a sign how far MySQL knowledge has spread, and how little effort may be sufficient to run MySQL, that this worked. OTOH, this invalidated the assumptions on which my job was based, and I did not really fit the alternatives left in that company.
In that situation, my contact to FromDual got renewed, and both the company and me felt we should be a good match. Till now, there is no indication we were wrong: I like the work, I feel I'm productive and make customers happy, the tasks are manifold and interesting (so interesting that writing the first blog gets delayed quite long), and I expect this will continue.
"Ad multos annos!"
Impacts of max_allowed_packet size problems on your MySQL database
We recently run into some troubles with max_allowed_packet size problems during backups with the FromDual Backup/Recovery Manager and thus I investigated a bit more in the symptoms of such problems.
Read more about: max_allowed_packet.
A general rule for max_allowed_packet size to avoid problems is: All clients and the server should have set the same value for max_allowed_packet size!
I prepared some data for the test which looked as follows:mysql> SELECT id, LEFT(data, 30), LENGTH(data), ts FROM test; +----+--------------------------------+--------------+------+ | id | left(data, 30) | length(data) | ts | +----+--------------------------------+--------------+------+ | 1 | Anhang | 6 | NULL | | 2 | Anhang | 6 | NULL | | 3 | Anhangblablablablablablablabla | 2400006 | NULL | | 4 | Anhang | 6 | NULL | +----+--------------------------------+--------------+------+
Max_packet_size was set to a too small value then:mysql> SHOW GLOBAL VARIABLES WHERE variable_name = 'max_allowed_packet'; +--------------------+---------+ | Variable_name | Value | +--------------------+---------+ | max_allowed_packet | 1048576 | +--------------------+---------+
The first test was to retrieve the too big row:mysql> SELECT * FROM test WHERE id = 3; ERROR 2020 (HY000): Got packet bigger than 'max_allowed_packet' bytes mysql> SELECT CURRENT_USER(); ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... Connection id: 6 Current database: test
We got an error message AND we were disconnected from the server. This is indicated with the message MySQL server has gone away which is basically wrong. We were disconnected and not the server has died or similar in this case.
A further symptom is that we get an entry in the MySQL error log about this incident:[Warning] Aborted connection 3 to db: 'test' user: 'root' host: 'localhost' (Got an error writing communication packets)
So watching carefully such error messages in your MySQL error log with the script check_error_log_mysql.pl from our Nagios/Icinga plugins would be a good idea...
The mysqldump utility basically does the same as a SELECT command so I tried this out and got the same error:shell> mysqldump -u root test > /tmp/test_dump.sql mysqldump: Error 2020: Got packet bigger than 'max_allowed_packet' bytes when dumping table `test` at row: 2
And again we get an error message in the error log! This is also a good indicator to see if your backup, made with mysqldump failed in this case.
To get a proper dump we have to configure the mysqldump utility properly:shell> mysqldump --max-allowed-packet=5000000 -u root test > /tmp/test_dump.sql
After the backup we tried to restore the data:shell> mysql -u root test < /tmp/test_dump.sql ERROR 2006 (HY000) at line 40: MySQL server has gone away
Again we got an error on the command line and in the MySQL error log:[Warning] Aborted connection 11 to db: 'test' user: 'root' host: 'localhost' (Got a packet bigger than 'max_allowed_packet' bytes)
and further the data are only partially loaded:mysql> SELECT * FROM test; +----+--------+------+ | id | data | ts | +----+--------+------+ | 1 | Angang | NULL | | 2 | Angang | NULL | +----+--------+------+
Another symptom we can see here is that the MySQL status aborted_clients is increased in all 3 situation:mysql> SHOW GLOBAL STATUS WHERE variable_name = 'aborted_clients'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | Aborted_clients | 10 | +-----------------+-------+
One positive aspect is that with MySQL 5.7.5 the first 2 symptoms do not appear any more...Further information you can find here: Communication Errors and Aborted Connections.
Avoid temporary disk tables with MySQL
For processing SELECT queries MySQL needs some times the help of temporary tables. These temporary tables can be created either in memory or on disk.
The number of creations of such temporary tables can be found with the following command:mysql> SHOW GLOBAL STATUS LIKE 'created_tmp%tables'; +-------------------------+-------+ | Variable_name | Value | +-------------------------+-------+ | Created_tmp_disk_tables | 4 | | Created_tmp_tables | 36 | +-------------------------+-------+
There are 2 different reasons why MySQL is creating a temporary disk table instead of a temporary memory table:
- The result is bigger than the smaller one of the MySQL variables max_heap_table_size and tmp_table_size.
- The result contains columns of type BLOB or TEXT.
This method can be used if changing the table structure from TEXT to VARCHAR or the use of a RAM disk are not possible solutions.
Making HAProxy High Available for MySQL Galera Cluster
After properly installing and testing a Galera Cluster we see that the set-up is not finished yet. It needs something in front of the Galera Cluster that balances the load over all nodes.
So we install a load balancer in front of the Galera Cluster. Typically nowadays HAProxy is chosen for this purpose. But then we find, that the whole Galera Cluster is still not high available in case the load balancer fails or dies. So we need a second load balancer for high availability.
But how should we properly fail-over when the HAProxy load balancer dies? For this purpose we put a Virtual IP (VIP) in front of the HAProxy load balancer pair. The Virtual IP is controlled and fail-overed with Keepalived.
First some preparations: For installing socat we need the repoforge repository:shell> cd /tmp shell> wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm shell> yum localinstall rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm shell> yum update shell> yum install socat
Then we can start installing HAProxy and Keepalived:shell> yum install haproxy keepalived shell> chkconfig haproxy on shell> chkconfig keepalived on
We can check the installed HAProxy and Keepalived versions as follows:shell> haproxy -v HA-Proxy version 1.5.2 2014/07/12 shell> keepalived --version Keepalived v1.2.13 (10/15,2014)
Configuration of HAProxy
More details you can find in the HAProxy documentation.shell> cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak shell> cat << _EOF >/etc/haproxy/haproxy.cfg # # /etc/haproxy/haproxy.cfg # #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 1020 # See also: ulimit -n user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats.sock mode 600 level admin stats timeout 2m #--------------------------------------------------------------------- # common defaults that all the 'frontend' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode tcp log global option dontlognull option redispatch retries 3 timeout queue 45s timeout connect 5s timeout client 1m timeout server 1m timeout check 10s maxconn 1020 #--------------------------------------------------------------------- # HAProxy statistics backend #--------------------------------------------------------------------- listen haproxy-monitoring *:80 mode http stats enable stats show-legends stats refresh 5s stats uri / stats realm Haproxy\ Statistics stats auth monitor:AdMiN123 stats admin if TRUE frontend haproxy1 # change on 2nd HAProxy bind *:3306 default_backend galera-cluster backend galera-cluster balance roundrobin server nodeA 192.168.1.61:5201 maxconn 151 check server nodeB 192.168.1.61:5202 maxconn 151 check server nodeC 192.168.1.61:5203 maxconn 151 check _EOF
Starting and testing HAProxy
The HAProxy can be started as follows:shell> service haproxy start
and then be checked either over the socket:shell> socat /var/lib/haproxy/stats.sock readline prompt > show info > show stat > help
or over your favourite web browser entering the username and password (monitor:AdMiN123) specified in the configuration file above:
To check the application over the load balancer we can run the following command:shell> mysql --user=app --password=secret --host=192.168.1.38 --port=3306 --exec="SELECT @@wsrep_node_name;" +-------------------+ | @@wsrep_node_name | +-------------------+ | Node C | +-------------------+ shell> mysql --user=app --password=secret --host=192.168.1.38 --port=3306 --exec="SELECT @@wsrep_node_name;" +-------------------+ | @@wsrep_node_name | +-------------------+ | Node A | +-------------------+ shell> mysql --user=app --password=secret --host=192.168.1.38 --port=3306 --exec="SELECT @@wsrep_node_name;" +-------------------+ | @@wsrep_node_name | +-------------------+ | Node B | +-------------------+
Configuration a Virtual IP (VIP) with Keepalived
Now we have 2 HAProxy load balancers. But what happens if one of them fails. Then we do not want to reconfigure our application to work properly again. The fail-over should happen automatically. For this we need a Virtual IP which should automatically fail-over.
More details you can find in the Keepalived documentation and the keepalived user guide.shell> cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak cat << _EOF >/etc/keepalived/keepalived.conf # # /etc/keepalived/keepalived.conf # global_defs { notification_email { email@example.com root@localhost } # Change email from on lb2: notification_email_from lb1@haproxy1 router_id HAPROXY } vrrp_script chk_haproxy { script "killall -0 haproxy" interval 2 weight 2 } vrrp_instance GALERA_VIP { interface eth1 virtual_router_id 42 # Higher priority on other node priority 101 # 102 advert_int 1 # notify "/usr/local/bin/VRRP-notification.sh" virtual_ipaddress { 192.168.1.99/32 dev eth1 } track_script { chk_haproxy } authentication { auth_type PASS auth_pass secret } } _EOF
Starting and testing Keepalived
To test the keepalived we can run the following command:shell> keepalived -f /etc/keepalived/keepalived.conf --dont-fork --log-console --log-detail ^C
To finally start it the following command will serve:shell> service keepalived start
To check the Virtual IP the following command will help:shell> ip addr show eth1
And then we can check our application over the VIP:shell> mysql --user=app --password=secret --host=192.168.1.99 --port=3306 --exec="SELECT @@wsrep_node_name;"
- Brian Seitzer: Redundant Load Balancers – HAProxy and Keepalived
failed MySQL DDL commands and Galera replication
We have recently seen a case where the following command was executed on a Galera Cluster node:SQL> GRANT SUPER ON userdb.* TO firstname.lastname@example.com; ERROR 1221 (HY000): Incorrect usage of DB GRANT and GLOBAL PRIVILEGES
2014-12-09 14:53:55 7457 [Warning] Did not write failed 'GRANT SUPER ON userdb.* TO firstname.lastname@example.com' into binary log while granting/revoking privileges in databases. 2014-12-09 14:53:55 7457 [ERROR] Slave SQL: Error 'Incorrect usage of DB GRANT and GLOBAL PRIVILEGES' on query. Default database: ''. Query: 'GRANT SUPER ON userdb.* TO firstname.lastname@example.com', Error_code: 1221 2014-12-09 14:53:55 7457 [Warning] WSREP: RBR event 1 Query apply warning: 1, 17 2014-12-09 14:53:55 7457 [Warning] WSREP: Ignoring error for TO isolated action: source: c5e54ef5-7faa-11e4-97b0-5e5c695f08a5 version: 3 local: 0 state: APPLYING flags: 65 conn_id: 4 trx_id: -1 seqnos (l: 4, g: 17, s: 15, d: 15, ts: 113215863294782)
According to the error message it looks like this command is done in Total Order Isolation (TOI) mode during the Rolling Schema Upgrade (RSU).
Only on the nodes which did NOT receive this wrong command the error log message was written and further they have received a GRA_*.log file.
Analysis of the GRA_*.log (failed transactions) files:hexdump -C GRA_2_16.log 00000000 f3 fe 86 54 02 53 14 00 00 76 00 00 00 76 00 00 |...T.S...v...v..| 00000010 00 00 00 04 00 00 00 00 00 00 00 00 00 00 2a 00 |..............*.| 00000020 00 00 00 00 00 01 00 00 00 40 00 00 00 00 06 03 |.........@......| 00000030 73 74 64 04 21 00 21 00 08 00 0b 04 72 6f 6f 74 |std.!.!.....root| 00000040 09 6c 6f 63 61 6c 68 6f 73 74 00 67 72 61 6e 74 |.localhost.grant| 00000050 20 53 55 50 45 52 20 6f 6e 20 75 73 65 72 64 62 | SUPER on userdb| 00000060 2e 2a 20 74 6f 20 72 6f 6f 74 40 31 32 37 2e 30 |.* to email@example.com| 00000070 2e 30 2e 31 31 31 |.0.111 |
dd if=bin-log.000001 of=binlog.header bs=1 count=120 cat binlog.header GRA_2_17.log > GRA_2_17.binlog_events mysqlbinlog GRA_2_17.binlog_events ... # at 120 #141209 15:04:54 server id 5201 end_log_pos 118 CRC32 0x3432312e Query thread_id=45 exec_time=0 error_code=0 SET TIMESTAMP=1418133894/*!*/; SET @@session.pseudo_thread_id=4/*!*/; SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/; SET @@session.sql_mode=1073741824/*!*/; SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/; /*!\C utf8 *//*!*/; SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=8/*!*/; SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; grant SUPER on userdb.* to email@example.com /*!*/; DELIMITER ; # End of log file ROLLBACK /* added by mysqlbinlog */; /*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/; /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
It further looks like this command was issues by Connection ID number 4: conn_id: 4.
How to recover deleted tablespace?
Sometimes, MySQL tablespace file(s) might be deleted by mistake, e.g. delete the shared tablespace (ibdata1) or an individual tablespace (table_name.ibd).
In this post I will show you how to recover those files (on Linux OS) having only one condition, MySQL service should still be running. If MySQL service stopped after deleting that file, this method will not work, so it is extremely important to act as quick as possible to avoid data loss.
The following is a simple table creation (innodb_file_per_table is enabled) and the records count inside that table:SQL> SHOW CREATE TABLE t\G *************************** 1. row *************************** Table: t Create Table: CREATE TABLE `t` ( `id` int(11) NOT NULL AUTO_INCREMENT, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=23 DEFAULT CHARSET=latin1 1 row in set (0.00 sec) SQL> SELECT COUNT(*) FROM t; +----------+ | COUNT(*) | +----------+ | 22 | +----------+ 1 row in set (0.02 sec)
Now, lets delete the individual tablespace for that table:shell> rm -rf /var/lib/mysql/test/t.ibd
At this time, we can still select and modify that table!!SQL> INSERT INTO t VALUES (NULL); Query OK, 1 row affected (0.00 sec) SQL> SELECT COUNT(*) FROM t; +----------+ | COUNT(*) | +----------+ | 23 | +----------+ 1 row in set (0.00 sec)
To be more accurate, rm does not actually delete the file, rather it removes the directory entry pointing to the file's inode. The inode - and in consequence the file - will be removed only if this is the last reference, but as long as the MySQL server process has the file opened, there is another reference which is the open file handle (that's why the "mysqld" server process must still be running).
To list the opened files we can use the Linux command lsof (we filter the output to get only the deleted tablespace information):shell> lsof |grep t.ibd COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME mysqld 11401 mysql 25uW REG 7,0 98304 1010691 /var/lib/mysql/test/t.ibd (deleted)
The file has a tag of (deleted) which means that the directory entry pointing to the file's inode was deleted but there is another reference(s) to that inode, otherwise it won't be listed by the above command.
Now the question is, how can we get the on-disk path to that opened file if the directory entry was removed?
We can use the "/proc" interface to the running processes and their file handles by the following formula:
- File path = /proc/PID/fd/FD-number
According to the above formula and using the output of the "lsof" command, the file we just deleted is located here:shell> ll /proc/11401/fd/25 lrwx------ 1 mysql mysql 64 Oct 28 16:14 /proc/11401/fd/25 -> /var/lib/mysql/test/t.ibd (deleted)
To make sure that this is the on-disk path for the file we deleted, check the reference: it still points to the original path.How can we recover that file??
- First, we should make sure that no other queries are modifying that table: SQL> LOCK TABLE t READ; Query OK, 0 rows affected (0.00 sec)
- Then we copy the data blocks (/proc/11401/fd/25) to a new file (we use the original file path) and change the ownership to the MySQL system user (mysql): shell> cp /proc/11401/fd/25 /var/lib/mysql/test/t.ibd shell> chown mysql:mysql /var/lib/mysql/test/t.ibd
- Restart MySQL service (if we didn't restart MySQL service directly after recovering the tablespace all changes on that table will still be redirected to the open file handle not the just recovered copy and thus will be lost after the restart): shell> service mysql restart ..... SUCCESS! ..... SUCCESS!
- The tablespace is now recovered and we can modify the table normally: SQL> SELECT COUNT(*) FROM t; +----------+ | COUNT(*) | +----------+ | 23 | +----------+ 1 row in set (0.00 sec) SQL> INSERT INTO t VALUES (NULL); Query OK, 1 row affected (0.00 sec) SQL> INSERT INTO t VALUES (NULL); Query OK, 1 row affected (0.00 sec) SQL> select COUNT(*) from t; +----------+ | COUNT(*) | +----------+ | 25 | +----------+ 1 row in set (0.00 sec)
- We can use the same procedures above to recover the shared tablespace (ibdata1) but we should lock all tables before the recovery process by using the SQL command "FLUSH TABLES WITH READ LOCK;"
- If the MySQL server had to deal with more files (.frm, .ibd, .MYI, .MYD, ...) than its "open_file_limit", it might happen that it will close this handle. In that case, the file will also cease to exist, even though the process is still running and that's why it is extremely important to act as quick as possible.
- The same procedure can be used to recover MyISAM files (.MYI and .MYD) but note that the file handle will be released if "FLUSH TABLES;" SQL command was executed.
- The same procedure can be used as well to recover binary logs, general logs, ... etc but note that the file handle will be released if "FLUSH LOGS;" SQL command was executed.
- This method can be used to recover any deleted file on Linux not only MySQL files but if the inode files have other references (lsof).
Real life case:
One of our customers was enabling the general query log on his production system, he noticed that the file was continuously growing and to not to consume the available free disk space on his server he removed that file by "rm /path/to/general_query.log". However, the available free space was still being consumed while he couldn't see the general log file. The customer thought that the file was deleted but in fact, the file handle was still opened by MySQL server process.
To get the problem solved we only issued the SQL command "FLUSH LOGS;" - which the customer should have issued after removing the file - then the file handle was closed, thus the inode was deleted and the consumed disk space freed back to the system.


text:
Reach tens of thousands of people instantly by advertising with Ekklesia. Find out more
A senior Church of Scotland minister has written to Alex Salmond about asylum seekers who are facing oppression and persecution if they are sent back home.
The Church of Scotland has joined leading charities, faith groups and trade unions in urging Scottish MPs to uphold the Lords amendments to the Welfare Reform Bill.
Homelessnness kills, yet it seems like the world is still walking on by while it happens, says a senior Church of Scotland minister.
Most nativity plays usually feature the local Sunday school, but this year a Church of Scotland minister and a Roman Catholic priest have enlisted the help of police officers, fire fighters, council leaders and even the Kilmarnock football team.
The Church of Scotland has said it is not happy with the way a consultation into feed-in tariffs has been carried out.
A senior Church of Scotland minister has called for a co-ordinated strategy by the Scottish Government and others to tackle sectarianism in Scotland.
Affirming Christians and equality campaigners have expressed disappointment at the Church of Scotland's negative response to same-sex marriage proposals.
The Chancellor's Autumn Statement has not gone far enough to ease the burden on the poorest and most vulnerable, a leading Church of Scotland minister says.
The Moderator of the General Assembly of the Church of Scotland is due to share worship with leaders of the Jewish and Muslim communities at a mosque.
World church leaders have met in Scotland's capital to discuss plans to rid the world of nuclear weapons and to meet civic and political leaders.

subset: commoncrawl


text:
- Find a rental
- Manage your booking
- Oak Island rentals
- How to book
- Help Center
- Write a review
The Seashell 311 Womble St.
From $650 / week
Description from the manager
Description from the manager
Bedrooms: 3/
Bathrooms: 2/
Sleeps: 6/
Type: House
3 Bedrooms/ 2 Baths: 2 Queens, 1 Double. 2 CCTVs, DVD player with CD player, wireless internet access, free long distance in US. Nicely decorated, carpet, ceiling fans, popcorn popper, well-equipped kitchen, safety bars in showers and beside commodes,handrail on front porch, two containers of family games & puzzles, highchair & crib. Covered porch & screened porch with furniture, fenced-in backyard with picnic table & chairs, outdoor shower- cold only. Short walk to beach. Well maintained for your enjoyment- please do not park on the grass as it damages irrigation equipment. Available for off season monthly rentals. 2014 Changes: Nightly rentals ...Read more
About the manager
About the manager
Oak Island Accommodations
Oak Island Accommodations
- Response rate: 30%
- Response time: Within a day
- Calendar updated: today
Overall rating:
Additional Location Information
Show rates details
Also consider
- FlipKey has rentals in over 11,000 destinations around the world:
- Destin vacation rentals - 1,296 rentals from $99/night
- Myrtle Beach vacation rentals - 1,055 rentals from $25/night
- Panama City Beach vacation rentals - 1,743 rentals from $89/night
- Gulf Shores vacation rentals - 779 rentals from $82/night
- Ocean City vacation rentals - 771 rentals from $81/night


text:
$1 million Powerball ticket sold in Vt.The Associated Press | September 06,2013LYNDONVILLE - Vermont Lottery officials say a $1 million Powerball ticket has been sold in Lyndonviile.
The ticket was sold at the Circle K convenience on Main Street.
Officials say the winning ticket matched five out of five numbers but not the Powerball number.
The winning numbers were 02, 09, 26, 45, 47, and the Powerball, 11.MORE IN This Just In
The state's utility regulator is beginning work on the Legislature's demand that... Full Story
Two deny incidents
at Price Chopper
Two people were in Rutland criminal court... Full Story
Flu vaccines
being offered
Free flu shots are available for Rutland County... Full Story
- Most Popular
- Most Emailed


text:
Author Archives
Carol P. Christ leads Goddess Pilgrimage to Crete, a life transforming tour for women. www.goddessariadne.org She is a founding mother in the study of women and religion, feminist theology, women's spirituality, and the Goddess movement and teaches in the Women's Spirituality program at CIIS. Her books include She Who Changes and Rebirth of the Goddess and the widely used anthologies Womanspirit Rising and Weaving the Visions.
Dionysian Rites by Carol P. Christ
In today's blog, I offer an excerpt from my soon to be re-published memoir, A Serpentine Path: Mysteries of the Goddess. The setting is Zaros, Crete, the time of year is mid-October. We had a scrumptious dinner of fresh fish,… Read More ›
Mysteries by Carol P. Christ
Finding ourselves together in Crete after attending a conference, four friends and I set out to visit the caves of Eilitheia in Amnissos and Agia Paraskevi in Skoteino. As we drove along the coast toward Amnissos, I recalled that caves… Read More ›
Argument from "Absence" and Absence of Dialogue by Carol P. Christ
Recently in a conversation with a noted archaeologist and his male graduate student assistant, I proposed that the absence of war and the trappings of war, including images of larger than life-size warrior kings, suggested to me that we should… Read More ›
The Book Is Finished, Now On to Publicizing It by Carol P. Christ
Goddess and God in the World: Conversations in Embodied Theology by Carol P. Christ and Judith Plaskow, forthcoming from Fortress Press in 2016. In Goddess and God in the World, two leading theologians model a new method of embodied theology,… Read More ›
The Dog Days of Summer by Carol P. Christ
Sirius rises late in the dark, liquid sky On summer nights, star of stars, Orion's Dog they call it, brightest Of all, but an evil portent, bringing heat And fevers to suffering humanity. –Homer, The Iliad The dog… Read More ›
No Longer Moved … by Symbols that Once Moved Me Profoundly by Carol P. Christ
This week Judith Plaskow and I submitted the final version of our new book Goddess and God in the World to our publisher at Fortress Press. Just before completion, I added a shorter version of the following passage to my… Read More ›
Today I am publishing an early work on female language for God that I wrote with Emma Trout at the first Conference of Women Theologians in 1971. Highly contested at the conference, this essay is a foreshadowing of my subsequent… Read More ›
Barth and Woman at Yale by Carol P. Christ
I recently located a copy of an essay on Karl Barth and women that I wrote as a graduate student at Yale University in the Alverno College archives. Rereading it decades later, I am a-mazed at the brilliance and tenacity… Read More ›
If You Don't Believe Women Are Fully Human, Can You Be a Great Pope? by Carol P. Christ
In recent weeks I have felt compelled to respond to a series of "Great Pope" photos and stories praising Pope Francis for his stands on poverty and climate change appearing on my facebook page. In every case I added something… Read More ›
Changing the American Story? by Carol P. Christ
In a moving part of Goddess and God in the World, the book Judith Plaskow and I are writing together, Judith describes how the Sabra and Shatila massacre forced her to confront the fact that "her people" are just as… Read More ›
Referendum in Greece: One Small Victory for the 99% by Carol P. Christ
Yesterday the Greek people voted by an unexpectedly large margin of 6l.31% against the austerity programs insisted upon by the European creditors–despite threats from the creditors that Greece would be expelled from the European Union. This was a victory for… Read More ›
Texts of Terror in the Humanities Curriculum by Carol P. Christ
When I began to study Latin in my freshman year in high school, one of the first texts we were asked to translate concerned the "rape" of the Sabine women. Even though the Latin text used a word that looked… Read More ›
THE REFUGEE CRISIS IN GREECE: A TEST OF OUR COMMON HUMANITY by Michael Bakas, translated by Carol P. Christ
Note from Carol Christ: I returned home from the Goddess Pilgrimage to Crete a week ago to find my island, Lesbos, and my village, Molivos, overwhelmed with a refugee crisis of enormous proportions. We are a town of about 1000… Read More ›
The Reason to Hope by Carol P. Christ
Recently, Valentina Khan touched many of us when she wrote a blog entitled The Powers of Evil are Well at Work and I've Lost My Spirit. Valentina spoke of the weariness and despair many of us feel when we think… Read More ›
What if Divine Feminine and Divine Masculine Are Not Oppositional Categories? by Carol P. Christ
A friend who is a spiritual teacher speaks often "bringing back the values associated with the Divine Feminine." For her this has to do with helping women to understand the beauty of our bodies and the importance of ways of… Read More ›
Restored in Beauty by Carol P. Christ
The path leading to the Klapados Waterfall begins at the edge of an open meadow in the pine and oak woodlands of a mountain in the island of Lesbos. After driving several miles on a very rutted dirt track, we… Read More ›
Take Only What You Need and Give Away: Fundamental Principles of Sustainability Ethics
Why is it so important to take only what we really need? Because everything we take harms another life. I included this Native American teaching as one of the Nine Touchstones I offered as a counterpoint to the Ten Commandments… Read More ›
Experiencing Divinity in the World by Carol P. Christ
As I work on revisions of the new book, Goddess and God in the World, that Judith Plaskow and I are writing, I am thinking again about John Cobb's notion of the "two ultimates" as two different ways of thinking about divinity…. Read More ›
Kassiani: Placing a Woman at the Center of the Easter Drama by Carol P. Christ
For many years I been told of the beautiful Hymn of Kassiani, sung only on Easter Tuesday night, but I had never heard it until this week. For many this song is the high point of Easter week. Kassiani, also… Read More ›
Spring in the Era of Pesticides, Global Climate Change, and War by Carol P. Christ
This was not a normal winter. It rained and rained and rained. It was grey, grey, grey. Gale force winds blew in from the ocean, not once but many times. Several of my shutters were shattered. An olive tree fell… Read More ›
The Inalienable Right to Life, Liberty, and the Pursuit of Happiness vs. the Right to Religious Freedom by Carol P. Christ
In his speech announcing that he signed the Religious Freedom Restoration Act, Governor Mike Pence of Indiana did not mention the issue of the so-called "right" to refuse service to gays, lesbians, and transgendered individuals as one of the reasons… Read More ›
The Power of Female Friendship: Remembering Karen McCarthy Brown by Carol P. Christ
News of Karen Brown's recent death came via email from a mutual friend of ours, Christine Downing. There are many things that can be said about Karen's life and career, including that she won prizes for her life's work Mama… Read More ›
Can You Kill the Spirit? What Happened to Female Imagery for God in Christian Worship? by Carol P. Christ
When I first began to think about female language and images for God I imagined that changing God-He to God-She and speaking of God as Mother some of the time would be a widespread practice in churches and synagogues by… Read More ›
Ancestor Connection and DNA Testing by Carol P. Christ
If like me, you view ancestry research as a spiritual quest, you may be wondering if it is worthwhile to have your DNA tested. I found out about The Seven Daughters of Eve, the female ancestors of most Europeans, some… Read More ›
Who Is Gender Queer? by Carol P. Christ
"It seems to me that calling oneself queer can be a way of affirming the parts (or all) of oneself that do not fit into the heteronormative paradigm. In my case, though I am white and straight, I am too… Read More ›
Patriotism and Religion: Speaking Complexly about Complex Issues by Carol P. Christ
Former Mayor of New York Rudy Giulaini recently questioned the American President's patriotism when he asked if Obama had been "brought up the way you were brought up and I was brought up, through love of this country." When Chris… Read More ›
What If There Are Sex Differences But Biology Is Not Destiny? by Carol P. Christ
Theories about sex differences have been used to keep women in the home and to justify male domination. Because of this, many feminists run as fast as they can from any discussion of them. Last week while thinking again about… Read More ›
Has the Vatican Discovered that Women Should Be Running the World? by Carol P. Christ
So it is a [female] generativity that .. is … giving life to social, cultural and economic structures that are inspired by values, ideas, principles and practices oriented to the common good … The above statement from the Pontifical Council's… Read More ›


text:
Visit our other Wisconsin Historical Society websites!
Refine or narrow your results by clicking any term in the gray boxes below.
Expand your results by removing any breadcrumb in a gray outlined box above. Click on the blue "X" to remove the term.
Narrow your results by entering a keyword into the search box above. Then click the "Search Within" checkbox, then click on the orange "Search" button.
If you did not find what you are looking for or you did not get any results, try changing your search term(s). See our Search Help page for more tips.
Date: 1964
Description: Caption reads: "F. Gerald Ham, Wisconsin State Archivist, examining papers (posed re-enactment) in preparation of a guide to labor history research materia...
Date: 1936-04-16
Description: Madison Packing Co. (Neesvig) truck parked in front of their store at 307 W. Johnson Street, with Labor Temple next door, 309 W. Johnson Street. Sign on tr...
Date: 1939-12-02
Description: Group portrait of Phi Sigma Sigma fraternity formal party in the Loraine Hotel dining room.
Date: 1936-03-13
Description: View from street of Barney Miller, the driver, feeding hay to Ned, the oldest dairy horse in service in Madison, The horse is hitched to a Kennedy Dairy wa...
Date: 1952-06-23
Description: New AFSCME union officers of Local 720 of the Dane County Courthouse and Lake View Santorium employees. Left to right: Doris Williams, secretary; Rodney S...
Date: 1934-11-22
Description: Two men stand in the doorway of the East Side Motor Company garage, 2305 Atwood Avenue, featuring Willard storage batteries, Pennzoil, and Eveready and Pr...
Date: 1988-09-28
Description: Waist-up staff portrait of F. Gerald Ham, State Archivist of Wisconsin from 1964 to 1989 and the head of the Division of Archives and Manuscripts at the St...
Date: 1958-03-19
Description: Portrait of Coach John Jung, Manitowoc High School boy's basketball coach. The team participated in the 1958 high school basketball tournament.
Date: 1949-01-06
Description: Group portrait of the Madison Cardinals ice hockey club (in uniform) sponsored by the Milwaukee Road Madison Service Club. The team was captained by Lee Fi...
Date: 1947-03-08
Description: Robert O'Brien, police officer on motorcycle, in front of Police Station.
Date: 1936-03-15
Description: Frank family banquet at the home of Morris L. Frank, located at 611 Chapman Street. The family were Greenbush residents.
Date: 1952-07-23
Description: Officers of the Madison Bowling Association gather to formulate plans for the 1952-53 season.
Date: 1925 ca.
Description: Benjamin Butts posing outside the Wisconsin Historical Society where he worked as a messenger.
Date: 1958-03-11
Description: Two men in front of a wall of hay bales pose with a baby camel. The men are Zoo Director Harold Hayes (left) and Bellamy Seals (right, wearing a Zor Shrine...
Date: 1949-01-05
Description: "Roundy" Coughlin sitting between retired Police Chief William H. McCormick (left) and his successor, Chief Bruce Weatherly, at a testimonial dinner for th...
Date: 1950-06-01
Description: Portrait of Charles B. Morris, a circulation department employee at Madison Newspapers, Incorporated.
Date: 1978 ca.
Description: Adkins and Shindo families posing in yukatas (the garment) at the Adkins home. From left to right back row: Shizuo Shindo, MD, Mrs. Shindo, Loraine Adkins,...
Date: 1953-02-27
Description: University of Wisconsin heavyweight boxer Bob Hinds is shown in a match with University of Syracuse boxer Joe Perry in the University of Wisconsin-Madison...
Date: 1959-01-06
Description: Problems of insect and plant disease control were discussed at the 13th annual Insect Control Conference at the Hotel Loraine. About 120 representatives of...
Date: 1948 ca.
Description: Construction of the faculty housing project at Eagle Heights for the University of Wisconsin Madison.
If you didn't find the material you searched for, our Library Reference Staff can help.
Call our reference desk at 608-264-6535 or email us at:
© 1996-2015 Wisconsin Historical Society, 816 State Street, Madison, WI 53706


text:
Favstar gets even better if you sign in.
Gift @DisraelTV Favstar Pro to unlock their best tweet!Gift them Pro!
@morten_s_jensen @poundingtherock JESUS AGREED TO PLAY FOR THE MINIMUM TOO?!? BUFORD IS A WIZARD!
Never liked blanketing the term for all military or civil servants, bc I know a lot, and almost none joined for selfless reasons....
@fivefifths This makes more sense considering that Matthew Mc. is THE sophomore philosophy major in life's purest form.
He's retiring?!? Thanks for haunting my dreams for 5 years as an undergrad! I study at home now as a grad student! Best of luck! #VoiceofPCL
@thedillon1 Waiting on the 2nd floor (church-organ space) in my Sting makeup about to zipline to the tabernacle.
"The tragedy of life is what dies inside a man after he loses his phone for a day and finds it with no notifications." - D. Valenzuelastein
Y'all want to see Outkast but you can't sing "Elevators" .... Nah.
@humbleteej L.m.f.a.o. "iiiimm going to pepper spray you in the face? ok? that's what I'm going to do to you." .... to show her appreciation
Really need some caffeine but too close to pre-workout time, so I guess I'll just finish this day in a trance-like stoic state.
@papahogiebear Don't mind my freshly buttered, slightly sugared baby. Simple misunderstanding.
@pgatour @jonnysalami0711 BUT HOW MANY OF THEM ARE AGGY THOUGH?!??
@nikkabella0721 You wanted a taste of the glorrrry? See what it taaasttee liiike? *nacho libre voice*
@bostonjerry @no_cut_card All about the founding fathers... until it ain't. Same way it's all about being christ-like... until it ain't.
Key & Peele - I Said Bitch: http://youtu.be/5LGEiIL1__s via @youtube
JK, I don't take steroids, every pre-workout I take would be approved by multiple testing agencies. But that playlist though....
Essentially a critique of our 2-party system of Govt in the U.S. from a former Republican Capitol Hill staffer. 3.5/5 . Easy read.
Good old East Austin hipster bartenders. Yes, Im complaining bc it says bulleit, and you put bull*hit in my drink. I can read a chalkboard.
Don't require but a minimum life education and full-blown education on how to be a cop..... slash.... "hero." https://twitter.com/justanova/status/607593751679098880 …
Shin Soo-Choo!!!!
Houston Native. Austin Contemporary. South Texas roots. I always think before I speak... but not always before I tweet. UT Grad Student.
Like @DisraelTV's tweets? Send them a Favstar Pro Membership to show you care.Gift them Pro!
Stats can't be shown as @DisraelTV hasn't signed in to Favstar recently.


text:
Things to see and do - El Jadida
Leaving for Morocco
Nearby hotels
Hotel Florence from35 €Book
Hotel Cafe Verdi from46 €Book
ibis budget el jadida from338 MADBook
ibis el jadida :
Nearby tourist site
ibis el jadida
ibis el jadida
Address : Place Nour El Kamar MA - 24000 EL JADIDA
from : 750.00 MADBook
Overlooking the beach, the ibis El Jadida hotel is 5 min from the city center, 15 min from the Sidi Bouzid seaside resort, 10 min from the historic monuments in the Portuguese city, and 1 hour from Casablanca and Mohamed V airport. The hotel has 103 air-conditioned rooms including 2 for people with reduced mobility, a restaurant, a bar open 24 hours a day, a terrace, a pool, a meeting room and a public car park.
Useful Information
- Address : Place Nour El Kamar MA - 24000 EL JADIDA24000EL JADIDA
Main services
- Restaurant
- Car park
- Air conditioning
- Swimming pool
- Non-smoking rooms
- Internet
- Animals allowed
- Satellite TV


text:
The Election Commission of India under its new Chief Nasim Zaidi has take taken several decisions over the last few months to ensure smooth elections.
The election commissioners want to formulate a model code in consultation with media houses, political parties and other stakeholders.
"Our central leaders have said that after winning the Bihar election, bulldozer will be run in West Bengal," he said.
According to data complied by the Commission, in the last Lok Sabha election in 2014, 464 political parties had fielded candidates.
It also sought direction to the Centre for constituting "an interim neutral and independent collegium/selection committee to recommend the names for the appointment on the vacant post of members to the EC".
Having a poll symbol on ballot paper will lead to 'unfair advantage' to some, Hazare said without naming any political party.
The announcement was made by returning Officer S Mohandoss at the end of deadline 3 PM for withdrawal of nominations.
In Karnataka where compulsory voting was applicable for the first time in the panchayat polls, no enforcement mechanism was put into place.