text:
Aga Khan III
From Wikiquote
Sir Sultan Mahommed Shah, Aga Khan III GCSI GCMG GCIE GCVO PC (2 November 1877 – 11 July 1957) was the 48th Imam of the Nizari Ismaili Muslims. He was one of the founders and the first president of the All-India Muslim League, and served as President of the League of Nations from 1937 to 1938.
- Islam is fundamentally in its very nature a natural religion. Throughout the Quran God's signs (Ayats) are referred to as the natural phenomenon, the law and order of the universe, the exactitudes and consequences of the relations between natural phenomenon in cause and effect. Over and over, the stars, sun, moon, earthquakes, fruits of the earth and trees are mentioned as the signs of divine power, divine law and divine order. Even in the Ayeh of Noor, divine is referred to as the natural phenomenon of light and even references are made to the fruit of the earth. During the great period of Islam, Muslims did not forget these principles of their religion.
- In a letter dated 4th April, 1952 to Dr. Zahid Husain, President of Arabiyyah Jamiyyat, Karachi.
- Even a little knowledge of Islam will show that its religion is not only tolerant of other Faiths, but most respectful, and indeed, fully accepts the divine inspiration of all theistic Faiths that came before Islam. It does not only teach tolerance to its followers, but goes a step further and enjoins on them all to create the godly quality of "Hilm" that is, tolerance, forbearance, patience, calmness, and forgiveness. It is due to the spirit of tolerance of Islam that even the smallest Christian and Jewish minorities survived and kept all their doctrines during the thousand years of Muslim rule. Nothing like what happened to Muslims in Spain after the Christian conquest has ever happened to a non-Muslim Faith in any Islamic dominion. How, can Europeans be so ignorant as to have forgotten that in the first century of Islam the Khalifs ordered that all that was best in Greek and Roman cultures should be assimilated; that not only the philosophy, medicine and science of Greece, but its poetry and drama, were carefully translated into Arabic and were generally sought not only by the learned but also by the pious! The Muslim attitude towards the absorption of ideas was based on the principle of Islam which enjoins to acquire knowledge wherever available, and there is a well-known and authentic saying of the Prophet that "his followers should seek learning even if they have to go to China."
- In response to an article in the TIMES, London, which had labelled Islam as an "intolerant" religion, and held it responsible for some of the problems of the Middle East.
Memoirs of Aga Khan: World Enough & Time (1954)[edit]
- It is said that we live, move and have our being in God. We find this concept expressed often in the Koran, not in those words of course, but just as beautifully and more tersely... when we realize the meaning of this saying, we are already preparing ourselves for the gift of the power of direct [spiritual] experience.
- I firmly believe that the higher [spiritual] experience can to a certain extent be prepared for by absolute devotion in the material world to another human being. Thus from the most worldly point of view and with no comprehension of the higher life of the spirit, the lower, more terrestrial spirit makes us aware that all the treasures of this life, all that fame, wealth and health can bring are nothing beside the happiness which is created and sustained by the love of one human being for another... but as the joys of human love surpass all that riches and power may bring a man, so does that greater spiritual love and enlightenment, the fruit of that sublime experience of the direct vision of reality which is God's gift and grace, surpass all that the finest, truest human love can offer.
- Consider, for example, the opening declaration of every Islamic prayer: "Allah-o-Akbar". What does that mean? There can be no doubt that the second word of the declaration likens the character of Allah to a matrix which contains all and gives existence to the infinite, to space, to time, to the Universe, to all active and passive forces imaginable, to life and to the soul.
- Imam Hassan has explained the Islamic doctrine of God and the Universe by analogy with the sun and its reflection in the pool of a fountain; there is certainly a reflection or image of the sun, but with what poverty and with what little reality; how small and pale is the likeness between this impalpable image and the immense, blazing, white-hot glory of the celestial sphere itself. Allah is the sun; and the Universe, as we know it in all its magnitude, and time, with its power, is nothing more than the reflection of the Absolute in the mirror of the fountain.
- There is a fundamental difference between the Jewish idea of creation and that of Islam. The creation according to Islam is not a unique act in a given time but a perpetual and constant event; and God supports and sustains all existence at every moment by His will and His thought. Outside His will, outside His thought, all is nothing, even the things which seem to us absolutely self-evident such as space and time. Allah alone wishes: the Universe exists; and all manifestations are as a witness of the Divine will.
- All men, rich and poor, must aid one another materially and personally. The rules vary in detail, but they all maintain the principle of universal mutual aid in the Muslim fraternity. This fraternity is absolute, and it comprises men of all colours and all races... all are the sons of Adam in the flesh and all carry in them spark of the Divine Light. Everyone should strive his best to see that this spark be not extinguished but rather developed to that full "Companionship-on-High" which was the vision expressed in the last words of the Prophet [Muhammad] on his deathbed, the vision of that blessed state which he saw clearly awaiting him.
- Islamic doctrine goes further than the other great religions, for it proclaims the presence of the soul, perhaps minute but nevertheless existing in an embryonic state, in all existence - in matter, in animals, trees, and space itself. Every individual, every molecule, every atom has its own spiritual relationship with the All-Powerful Soul of God. But men and women, being more highly developed, are immensely more advanced than the infinite number of other beings known to us.
- Life in the ultimate analysis has taught me one enduring lesson. The subject should always disappear in the object. In our ordinary affections one for another, in our daily work with hand or brain, we most of us discover soon enough that any lasting satisfaction, any contentment that we can achieve, is the result of forgetting self, or merging subject with object in a harmony that is of body, mind and spirit. And in the highest realms of consciousness all who believe in a Higher Being are liberated from all the clogging and hampering bonds of the subjective self in prayer, in rapt meditation upon and in the face of the glorious radiance of Eternity, in which all temporal and earthly consciousness is swallowed up and itself becomes the eternal.
Quotes about him[edit]
- My grandfather was a most gifted person, and amongst his many qualities, one of them had always particularly impressed me. While the past was a book he had read and re-read may times, the future was just one more literary work of art into which he used to pour himself with deep thought and concentration. Innumerable people since his death have told me how he used to read in the future, and this certainly was one of his very great strengths.
- His grandson, Karim al-Hussaini, who later succeeded him as Aga Khan IV and Imam of the Nizari Muslims.
- My father insisted that I learnt the Koran and encouraged me to understand the basic traditions and beliefs of Islam but without imposing any particular views. He was an overwhelming personality but open-minded and liberal.
- His son, Sadruddin Aga Khan, in an interview to the The Telegraph (UK) that appeared on January 24, 1998.
See also[edit]
External links[edit]
- Aga Khan III: A Study in Humanism, an article by Professor K. K. Aziz, who compiled a 2-volume book on the life, works and speeches of the Aga Khan III.
- [1]


text:
What's the best way to organize a storage closet? That depends what you are storing. We offer many specialized products, including music storage and trophy and display cases. Tired of piles of papers strewn across your desktop? Be sure to check out our handy mail organizers. Hertz Furniture also provides a complimentary design service to assist you in planning your storage system.
What kinds of storage cabinets are out there?
Storage cabinets can come in handy anywhere, whether you need to store extra cleaning supplies, computer equipment or tools. Adjustable shelving on many of our units enables you to accommodate virtually anything that needs to be stored. With a variety of sizes and colors to choose from, you can find a storage closet or wardrobe that will look great in any classroom, maintenance room or office. Metal storage cabinets from Sandusky Lee are durable and versatile. When you invest in this excellent storage solution, you can be sure that it will serve you well for many years to come.
While metal cabinets are sturdy and dependable, you might want something more attractive for a professional setting. If you need some extra space for supplies and equipment in the principal's office or executive suite, veneer and wood cabinets from top office furniture manufacturers including DMI and Mayline are great. Many of them are part of full collections so they can be coordinated perfectly with office desks and other furniture.
How can I get more office storage?
In addition to having attractive cabinets, many professionals require some extra filing space and room for office supplies. An office credenza can also be coordinated with other furniture. It includes a combination of cabinets, box drawers and file drawers that will help keep a workspace organized while ensuring that materials are accessible.
Every educational setting requires lots of great storage. Schools need extra supplies on the premises, but not everything has to be out in the open at all times. Whether you just need an extra cabinet for the art room, or you need a full lineup of storage solutions for the whole building, Hertz Furniture has it all - and then some.
Where is a safe place to store lab supplies?
Appropriate lab storage is essential for any science classroom where students are doing hands-on experiments. When you need to store microscopes, petri dishes and chemicals, we have all of the right equipment. You can choose between units with wood and glass doors that are wall-mounted or freestanding. Locking cabinets protect valuable equipment and supplies as well as students. If your science lab has a lot of chemicals, a flammable liquids cabinet is recommended. This locking storage unit ensures that potentially harmful chemicals remain locked up until they are needed for supervised use.
What kinds of storage do you need in a school?
Students put a lot of time and effort into paintings and drawings, so you want to make sure that their hard work is not accidentally damaged. When paintings are left to dry on countertops and tables, they can easily get knocked over or smudged. With art drying racks, you can make sure that student work is protected from damage as it dries. Our high-capacity art racks are great for large schools where many students use the art room over the course of a day.
A school with a full creative arts program will also require some excellent instrument storage. Whether you have a marching band, orchestra, academic music program or all of the above, having an appropriate place to keep instruments in school is very helpful to students. Rather than carrying their instruments around all day, they can leave them locked up securely in the music room. This arrangement makes the whole experience of music in the educational environment much more pleasant.
Demonstrate your support and pride for student accomplishments with our sleek trophy display cases. Whether your soccer team has won the state championship or the debate club has made it to the finals, all students will appreciate a prominent display space that showcases their achievements. When visitors and parents pass by artfully-displayed acknowledgements of your school's awards they are sure to be impressed.
Where should you put bookcases?
When you need bookcases for an office, classroom or school library, Hertz Furniture has an excellent selection. Bookshelves can be coordinated with office decor or color-coded by subjects in a library. We have steel, veneer and wood bookshelves, so you can choose the material you prefer. Educational Edge bookcases offer a great way to bring some color into a classroom. When books are displayed prominently on proper shelving, people can see that your company or institution cares about scholarship and promotes the acquisition of knowledge.
Having great shelving and closets is always convenient - except when you need to move everything. A fully-stocked closet or bookshelf can be quite a challenge to relocate. With mobile storage units, you can simply roll everything into position without having to carry anything. This saves time and back strain.
What is the easiest way to move tables and chairs?
Speaking of mobility and storage, chair dollies and table carts are a great option to consider if your school, company or church has temporary seating and tables. Whenever a room needs to be set up for a big event, furniture can easily be rolled out of storage and moved to the desired location. Our table dollies and chair trucks are designed for easy maneuvering and compact storage.
You can never go wrong when you purchase storage solutions from Hertz Furniture. Our variety of multipurpose cabinets and custom equipment provides you with any kind of storage you might need in a school, office or church. When your possessions are stored and displayed properly, the whole building will take on an organized and attractive feel.
- Refined By
- $30 - $59.99
- $100 - $124.99
- $125 - $199.99
- $200 - $249.99
- $250 - $399.99
- Metal
Colors Available
- Black
- Brown
Table Capacity
- 8
- 10
- 12
Green Certification
- BIFMA Green
- MAS Certified LEED & CHPS
Chair Truck Type
- Folding-Chair Truck
- Stacking-Chair Truck
Certified Green
- See All Green Products
Chair Capacity
- 20-40 Chairs
- 40-60 Chairs
- 60-80 Chairs
- Under 20 Chairs
Heavy-Duty Table and Chair Caddy
View Details »
Dolly for LW-800 Folding Chairs
View Details »
Stack Chair Dolly
View Details »
Large Chair Truck-50 Cap
View Details »
Dolly for FBC-1100 Folding Chair
View Details »
Heavy-Duty Colson Caster Kit for HCT-505 Cart
View Details »
Table Truck for Round Folding Tables 8-10 Cap
View Details »
Chair Strap for CLW-22 Cart
View Details »
Table Truck for 71 inch dia Tables 8-10 Cap
View Details »
Heavy Duty Chair Carrier
View Details »
Table Truck for 96L Tables 12 Cap
View Details »
Compact Chair Dolly
View Details »
Under the Stage Conversion Dolly Kit
View Details »
Small Chair Truck-35 Cap
View Details »
Stack Chair Dolly
View Details »
Table Truck for 72L Tables 12 Cap
View Details »
NPS furniture is exceptionally durable. It is built to withstand many years of arrangement, folding and repeated use. These Certified Green items are great for your institution and the environment. When you are looking for high-quality school and church furniture, NPS risers and folding tables will get the job done right.
Whether you are in a classroom, auditorium or sanctuary, having the right chairs will make everyone's experience much more pleasant. Proper seating enables students, congregants and participants to remain focused on what is going on around them and maximize their productivity. NPS chairs are designed for easy arrangement and maximum support.
Transform your gym into a complete performance space with folding chairs and stages. A selection of portable stages which can be ganged together in a variety of formations enables you to turn any open space into an instant auditorium. Once the stage is set up, the audience will need a place to sit and watch. You can choose from our selection of folding chairs and stacking chairs to create the ideal arrangement for your space.
Not all stacking chairs and folding chairs are alike. They come in various styles and colors. There are padded folding chairs as well as metal folding chairs, and stacking chairs come with different back shapes and varying degrees of padding thickness. Both of these choices are easy to store and transport. You can order chair dollies that keep your seating organized in storage and make it easy to move to the desired location.
Stackable chairs can also be used as banquet chairs or church chairs. Elegant appearance and soft cushioning are appropriate for formal occasions. A wide selection of upholstery colors and fabrics enables you to coordinate with room decor and color schemes.
When you are hosting a school dance in the gym or having a special celebration in the social hall, it is helpful to have tables that are simple to arrange. NPS folding tables are both sturdy and durable. Blow-molded plastic folding tables are extra-tough. Their surfaces are a snap to clean and are resistant to just about everything. They have an incredibly high weight capacity as well.
Help your students conduct experiments with great lab furniture from National Public Seating. Lab tables with chemical-resistant tops create the perfect place for working with chemicals and other materials that might be harmful to regular furniture. When students are having a frontal lesson or getting everything set up, they will need an appropriate place to sit as well. Sturdy lab stools offer great seating that will support students through many years of scientific endeavors.
Having music in the in the educational environment gives students the opportunity to excel in different areas and develop their creative talents. With the right music furniture, you can help budding musicians learn to play new instruments. Sturdy music stands are essential for enabling children to read the notes properly. If instruments are being played in a seated position, you will also need proper music chairs.
The right furniture is necessary for breaks as well as instructional periods. Sturdy cafeteria tables make a lunchroom comfortable and attractive. They provide a place for students to sit and eat, and fold up easily so they can be moved out of the way when the space is being cleaned or used for another purpose.
At Hertz Furniture, we know that the right furnishings can transform a space. We are proud to offer your institution a great selection of school and church furniture that you can enjoy for many years to come. No matter what you style or budget is, you are sure to find a number of great National Public Seating products for your space.

subset: commoncrawl


text:
Battlefield 1943 Sells One Million on Xbox Live Arcadeby Brian Leahy, Mar 29, 2010 1:20pm PDT
Electronic Arts sends word that Battlefield 1943, DICE's downloadable multiplayer-only re-imagining of Battlefield 1942, has become the fastest selling Xbox Live Arcade title to move 1 million units since its release in July 2009.
The game, which costs 1200 Microsoft Points ($15), served as a bite-sized preview to future Battlefield games and allowed gamers to check out the new version of developer DICE's game engine, Frostbite, before Battlefield: Bad Company 2 was released.
It would appear that this strategy is effective. The above sales figures don't include sales on the PlayStation 3 through the PlayStation Store, and the oft-promised PC version has still yet to launch. Recent reports have suggested that Electronic Arts is considering similar paid downloads to build up buzz for upcoming retail releases.
Other companies are alsop exploring this strategy. Atomic Games president Peter Tamte told me his new title, Breach, will be marketed with this strategy: a $15 downloadable game that offers a bite-sized experience instead of a full-on retail game at $60.


text:
& Deals
& Celebrations
or Login
Join FREE
Log In
Blogger Brooke McLay from Cheeky Kitchen adds fresh blueberries and maple syrup for a twist to everyone's favorite monkey bread with this Blueberry-Pancake Monkey Bread recipe.
©2013 General Mills, Inc. All Rights Reserved. This information will only be used to send an email to your friend(s) and will not be saved. Please read our Privacy Policy.
Lemony Twist! Add a bit of grated lemon peel to the sugar for a kiss of citrus!
Berry Nice! Substitute fresh or frozen blackberries in this recipe for a truly delicious treat!
Crescent Strawberry Shortcake
Grands!® Monkey Bread
Cinnamon Sugar Snackers
Caramel Sticky Buns
Pepperoni Pizza Stuffed Biscuits
Mascarpone-Filled Cranberry-Walnut Rolls


text:
Bread and Butter Plate
The colors used on our Cayman dinnerware is reminiscent of the Caribbean Sea. Softened by the sun, muted shades of brown and turquoise appear like chiseled stones around the edges of this stoneware dinnerware set. Due to the nature of reactive glaze, each piece of this handcrafted collection will exhibit unique variations in color and pattern. Microwave and dishwasher safe. This little plate is perfect for breads and muffins and also great for hors d'oeuvres, snacks or late night treats. Bread & butter plate measures 6-1/2 inch diameter.


text:
Implementing 9/11 Commission Recommendations
Created with the founding principle of protecting the American people from terrorist and other threats, DHS and its many partners across the federal government, public and private sectors, and communities throughout the country have strengthened the homeland security enterprise to better mitigate and defend against dynamic threats.
Learn More
Implementing 9/11 Commission Recommendations
Private Sector Resource Catalog
Private sector partners who want to participate in the homeland security enterprise will find a valuable collection of resources from across the agency to assist that effort. This catalogue, published in PDF format for easy download, is available as a full 66-page document or by individual chapters.


text:
Soft Structured Carriers
Catbird Baby Pikkolo Carrier
"I bought the Pikkolo to have an option for a buckle carrier where my little one could have his legs out rather than in. The Pikkolo has an adjustable base which allows both small babies to be legs out and also any baby to face forward while still sitting ergonomically. I tried it out, immediately, with my 3yo son, and was amazed at how comfy it was, even with him. The baby was far more..." --HeatherB
Moby GO Baby Carrier
The Moby GO is an amazingly comfortable baby carrier designed with the ergonomic features and comfort that Moby is known for. Perfect for babies 15 - 45lbs.
Onya Baby The Outback Baby Carrier - Olive Green / Chocolate Chip
Whether you're out with your child wandering the urban jungle or hiking with them in the hills, the Outback offers a more performance-oriented approach to the soft-structured carrier. A durable, water-resistant, rip-stop nylon exterior and air-mesh lining makes it perfect for hot weather or hiking the trails. Parent and child will stay cooler with the air-mesh lining, which wicks away...
Onya Baby The Cruiser Baby Carrier - Dove Gray / Chocolate Chip
Cruising through your day with your child in a versatile, soft cotton baby carrier is what it's all about. Whether it's a visit to the park, a long walk on the beach, or a quick stop at the farmer's market, it's designed to carry your child on your front or back. The Cruiser has an integrated seat that fits on most chairs, so you can enjoy meals together without having to...


text:
Your Direct Impact
If you are driven by passion, komoot is the ideal workplace for you. At komoot, you own your field, you decide what is best to work on. You push your work to excellence. Whatever is best to build a world-leading product, you simply do it. No managers. No hierarchy. Your direct impact is what counts.


text:
|January 3
|Designer Doggie Days Adoption Party Set for Animal Care Services
|Event will feature cool canine blends at $25 throughout the week
|January 15
|Protect your pet from the chilly touch of winter
|Animal Care Services offers tips for keeping pets safe during the colder months
|January 31
|Pets hoping to score a family at Tail-Gate 2013!
|Super Bowl comes to Animal Care...complete with a golden "LomPawdi" trophy
|February 1
|Animal Care Services breaks historic live release record...again
|Local shelters and rescues help ACS secure a placement rate of more than 80%
|February 12
|Does your Valentine have four legs?
|Animal Care hosts special "Smooch a Pooch… Kiss a Kitty" adoption party
|October 8
|Animal Care Services hosts the first ever Alamo City "Paw"dopt-a-thon this weekend!
|Citywide adoption event will feature hundreds of adoption ready pets
|October 15
|Animal Care Services shatters lifesaving records in FY12
|Strategic partnerships help ACS end fiscal year with historic live release rate
|October 15
|Animal Care Services takes the national lead in the 2012 ASPCA Rachael Ray $100K Challenge
|Historic live release rate moves shelter into first place in field of 50.
|October 18
|Animal Care Services takes the national lead in the 2012 ASPCA Rachael Ray $100K Challenge
|With less than a month of competition to go, Animal Care Services is now in first place in the nationwide ASPCA Rachael Ray $100K Challenge.
|October 22
|Keep Pets Safe this Howl-O-Ween!
|ACS offers safety tips during the trick-or-treating season
|October 30
|Animal Care Services urges pet owners to vaccinate their pets for rabies and avoid wildlife
|Recent rabies case illustrates need for caution and vigilance
|November 1
|Animal Care Services wants you to "fall back" in love!
|Adoption event comes "just in time" for Daylight Savings
|November 2
|Animal Care Services honors Vets with Mutts for the Military
|Shelter hosts special weeklong event to thank our armed forces
|November 16
|Giving shelter pets a holiday to be thankful for!
|Animal Care Services offers Thanksgiving week adoption specials
|November 19
|A holiday to be thankful for
|Tips to enjoy a happy, healthy and safe holiday with your pets
|December 7
|Wishing you a Meowy Christmas and Happy Howlidays!
|Animal Care Services to offer yuletide adoption specials
|December 10
|Special adoption case wishing for a Meowy Christmas
|Kitten who hops on her back legs looking for a fur-ever home
|December 17
|Four legged Christmas presents?
|Residents urged to make an informed decision on giving pets as gifts
|December 21
|Fireworks and pets don't mix
|Tips for a worry free holiday for EVERY member of your family
|September 9
|Rabies Cases
|Animal Care Services urges pet owners to vaccinate their pets for rabies and avoid wildlife
|October 5
|ACS facility Public Meeting
|Public meeting to present design concept for pet adoption, spa/neuter facility on Monday, October 10, 2011, 6:00PM - 8:00PM


text:
If you are one of the five construction contractors bidding on California's High-Speed Rail project, you won the lottery.
And you didn't even have to buy a ticket.
Two million dollars will be granted to each losing company, simply for bidding.
That's $8 million dollars from the taxpayers, on a project which is short billions.
I'd like to hear your Point of View. Send me an email at firstname.lastname@example.com
The High-Speed Rail Authority says this type of payout, they call it a stipend, is not unusual for large-scale projects. And they say it encourages competitive bidding.
In reality, it's an uncommon practice and this is one of the highest loser payouts on record.
A new toll road for the 91 freeway in Southern California, also has a loser-stipend attached to it costs, but it's not even a third of what the rail authority is offering.
And in its own defense, the rail authority said California and several other states pay stipends too. But even the largest amount we could find other states paying out was half of what The Rail Authority is providing.
So why would the rail authority give $8 million dollars to the losing bidders?
Were they worried they wouldn't get anyone to bid? Are they that insecure about their plan?
The chance of winning hundreds of millions should be enough to encourage proper bidding.
This kind of reckless spending raises questions about the management of the entire project.
If they're subsidizing the losing bidders with millions then they may already be in violation of the spirit of the original law's intent, which clearly states the project must function without subsidies.
This bidding fiasco not only looks bad, but it raises a red flag that says: Stop!
Stop this project.
It's unpopular, it's underfunded and as this bidding process reveals, they're already wasting money before picking up the first shovel.
Thanks for listening.
I'd like to hear your Point of View.
What do you think? We want to hear from you on this P.O.V. segment. Share your thoughts in the comments section below or email me directly at firstname.lastname@example.com.
Kevin Hale,
Vice President and General ManagerKTTV-TV, FOX 11 NewsThe views expressed are not necessarily those of the station or its employees.
WJBK-TV | Fox 2 16550 West Nine Mile Rd.Southfield, MI 48075Main Station: (248) 557-2000Newsroom: (248) 552-5103


text:
Top Airlines Flying To Detroit
More Information About Detroit Travel
- Our Detroit hotels page lists the most popular hotels and gives you the ability to filter and get real-time rates from multiple hotel providers.
- Looking for a cheap rental car in Detroit? Visit our Detroit Car Rentals page to help you find a rental car for your trip to Detroit.
- Looking for airline fees? See what airlines are charging for extra baggage, meals, pets, unaccompanied minor, or seat assignments.
- KAYAK Mobile Travel Apps help you to find flights, book hotels, and rent cars from your smart phone or tablet. Available for iPhone, Android, iPad, Windows Mobile 7, and Kindle.
KAYAK searches hundreds of sites to find the cheapest flights to Detroit (United States). Get the best price for your airline ticket to Detroit! Here's how it works. Search KAYAK for a flight to Detroit. Sort, filter, and compare results on one web page. Then book your flight ticket direct on the airline or travel agent site of your choice!
Cheap Detroit Flights
These are the cheapest flights found by KAYAK users in the past 48 hours.
- Price Flights
- from $36 Flights Wilmington (ILG) - Detroit (DTT)
- from $88 Flights Denver (DEN) - Detroit (DTT)
- from $228 Flights New York (LGA) - Detroit (DTT)
- from $237 Flights Houston (HOU) - Detroit (DTT)
- from $255 Flights Chicago (CHI) - Detroit (DTT)
- from $288 Flights Dallas/Fort Worth (DFW) - Detroit (DTT)
- from $309 Flights Washington (DCA) - Detroit (DTT)
- from $312 Flights Orlando (MCO) - Detroit (DTT)
- from $563 Flights Los Angeles (LAX) - Detroit (DTT)
- from $572 Flights San Francisco (SFO) - Detroit (DTT)


text:
After the election Kerry Eleveld wrote a piece for The Atlantic titled "Why Barack Obama Will Be a More Effective Liberal in His Second Term."
"In response to their initial disappointment with the president's early performance, many progressives speculated that Obama was just waiting for a second term to be more liberal," he said. That was true. They were.
Eleveld continued: "A more likely explanation is that Obama was still finding his groove, figuring out which levers worked best for him in the context of governing the nation. And in some ways, he was still developing the courage of his convictions."
That, it turns out, was false. He wasn't.
You can't develop convictions that you don't have in the first place.
It's hard to remember now, more than six weeks later, but there was once a time (six long weeks ago) when liberal Democrats who naïvely chose to ignore Obama's consistently conservative first term, his consistently conservative career in the Senate, and his consistently conservative pre-politics career as a University of Chicago law professor, seriously believed that his reelection would lead to a progressive second term.
"It's time for President Obama to assume the Roosevelt-inspired mantle of muscular liberalism," Anthony Woods wrote in The Daily Beast. "This is his moment. He only has to take it."
It's his moment, all right. And he's taking it. But when it comes to Obama, liberals are once again guilty of some major wishful thinking. Obama's economic policies are closer to Herbert Hoover than Franklin Roosevelt.
"With re-election safely behind him, we hope Obama will be bolder in his second term," Peter Dreier and Donald Cohen wrote in The Nation.
Again with the Hope!
Change, not so much.
Race doesn't matter. Looks don't matter. Age doesn't matter. Style doesn't matter. Only one thing matters when you're electing a politician: policy. And the willingness and ability to carry it out. Everything you needed to know about Barack Obama boils down to the fact that he voted nine times out of ten to fund the Iraq war, at the same time that he was giving speech after speech pretending to oppose it. And that was before he won in 2008.
It didn't take long for Obama to sell out the liberal base of his party the first time. Everything became clear in December 2008, when his cabinet picks didn't include a single liberal. Well, here it is, December 2012, and can't get fooled again but we did, as George W. Bush would sorta say.
Wait a minute: I thought Obama was a Democrat. So why is he appointing a Republican as secretary of defense? Not just a Republican, but a homophobe? In 1998 Republican Senator Chuck Hagel criticized President Clinton's nominee for ambassador to the sensitive strategic hotbed of Luxembourg not only for being gay, but for being "openly, aggressively gay." Gay rights groups demanded that Hagel "repudiate" his bigoted comments, and he dutifully did so, but the point is that a truly progressive Democratic president would never have appointed a gay-bashing right-wing Republican in the first place. Yeah, America has changed, but it wouldn't be that hard to find a liberal Democrat who thought gays and lesbians were real human beings back in 1998.
The "fiscal cliff" negotiations have led to another replay of Obama's 2008 sellout, this one on economic fairness. Throughout the 2012 campaign the president promised to raise taxes on the top 2% of American households, those earning over $250,000 a year. As of November 9th he was still "sticking to his guns," calling his stance nonnegotiable. On December 17th, however, without the defeated Republicans even having to propose a counteroffer, Obama pulled a classic Democratic negotiating-against-himself maneuver. Not only did he offer House Speaker John Boehner to protect the spectacularly wealthy taxpayers who earn up to $400,000 from a tax hike, he quietly sold out senior citizens by gutting the current system that calculates cost-of-living increases for Social Security and other federal entitlement programs.
At first, few people would notice Obama's switch to a so-called "chained consumer price index." (Under the new system, if the price of steak goes up, the government assumes you'll switch to hamburger—so it doesn't count as inflation.) This year, for example, the inflation rate under the chained CPI is 0.3% less. But inflation is exponential and the effect is cumulative. By the time you hit age 92, you'd lose an entire month of Social Security benefits each year.
This, remember, was the president who was supposed to bust out as an FDR-style crusading liberal ready, willing and able to fight the right-wing Republicans and stand up for ordinary Americans.
The good news is, the anticipation is over. Liberals who worried that Obama would sell them out need worry no more. Not so deep down, they knew this would happen. Now they can settle down for four more years of depressing Republican-lite kowtowing to corporations and the one percent.
I know what they're thinking. Things would be even worse if Mitt Romney had won.
I wouldn't be so sure.
Policy-wise, a Romney administration would have been pretty much the same as Obama's second term. Who knows, he might have picked Chuck Hagel as Defense Secretary.
In terms of building the political Left, a President Romney would have galvanized liberals and progressives to fight for a fairer society that treats everyone equally and with dignity. Obama, his sellouts, and his faux liberal apologists represent two steps backwards for progressivism.


text:
I had a great time last week at Percona Live, meeting up with lots of old friends, and getting to know lots of new ones.
It was great to meet many of the people that hang around on DBHangOps face to face. Geoff even got a community award (well done)! Unfortunately I had to miss the lunch.
It was also good to see Oracle getting a community award. Our engineers are extremely hard working, and all want to help community and customers alike be successful with their MySQL environments. There was lots of great positive attitude towards the work we've been doing, it was pleasing to hear that we are on the right track.
I haven't been to a conference at that venue since the "old school" MySQL UC. Well done Percona for putting on such a great event.
And as a bonus, I even got to sit in front of Shlomi Noach on my flight from SFO to Newark:
And here I thought #perconalive was over. Got my airplane seat just behind @MarkLeith . More tech talk! pic.twitter.com/J3ukpSNk8V
— Shlomi Noach (@ShlomiNoach) April 5, 2014
Here's the slides for my talk, MySQL 5.7: Performance Schema Improvements:
I hope to be seeing many of you out at MySQL Connect later this year, not long now to submit your talk!
ps_helper and its migration to sys
MySQL's Performance Schema is an incredibly rich and versatile instrumentation engine, but apparently, with great power, comes increased complexity for the user to understand up front.
I think this is pretty natural:
- The more flexibility you provide to the user on what to monitor (and this is one of the major goals of Performance Schema) – the more there is to twiddle from a configuration perspective.
- The more data we provide in as generic a ways as possible – the more data there is.
Performance Schema is also only going to grow over time. There are many many missing bits of instrumentation within MySQL that users want us to add, and for the most part, most of those new things are now going in to Performance Schema. Things such as memory instrumentation, stored programs, transactions, prepared statements, better replication monitoring … the list goes on, and the things I mention here are just the things that have gone in to MySQL 5.7 so far.
When I started ps_helper it was nothing more than a little playground for me to research both the breadth of the data that Performance Schema provides, as well as to try and nail down exactly the kinds of views that users want and need, based around various different use cases.
Judging by the feedback that I've gotten on ps_helper so far, it turns out that I was actually pretty successful with the latter goal. I've heard about it being used in all sorts of situations – from large websites, to small installations, everybody seems to have found something useful with it. MySQL Support also regularly directs customers to install it, and return the output from many of the views, to help customers tune and debug what their MySQL instances are doing.
One of the biggest pieces of feedback that I (and others) have heard is that users really want something like ps_helper to ship with the server itself – there is even a bug to have it included opened by my old colleague Valerii Kravchuk.
Getting something like ps_helper in to the server is not a small task. We need to know that what we are providing makes sense to users, commit to maintaining it "forever", make sure that it is properly tested, is as bug free as possible, that it follows the right procedures for licenses and that it is properly structured for any future plans that we may have (of which there are many).
Recently, the MySQL Workbench team announced that they would be shipping a new schema that was something like ps_helper:
Additionally Workbench 6.1 is leveraging work from various teammates in MySQL Engineering in introducing a schema called "SYS" that provides simplified views on Performance Schema, Information Schema, and other areas.
This is the first step towards moving the kinds of views and procedures that ps_helper provides in to a shipping "MySQL product". It started with the Workbench team taking ps_helper, performing a bunch of post processing on it (to rename schema etc.), and then integrating the resulting set of objects. In discussion with them after their first try at this, we decided instead to start a new branch, that didn't need post-processing, and that starts following some of the other strict guidelines for getting a product shipped within MySQL/Oracle.
And so we get to the point of this blog finally, the mysql-sys repository was born on Github.
Why the name "sys"?
Firstly, and more importantly for some people – it is shorter and easier to type than "ps_helper".
When you look at other database systems though, and at other similar things that people have done within the MySQL Community, it's clear that there is room for a schema that can more generically provide a set of objects to administrators that help them with their day to day work.
SQLServer for example has a sys catalog, which comes with a number of dynamic management views, functions and procedures for administrators to use for various tasks.
DB2 has SYSIBM, which is a mixture of some DD and some runtime info. They also have syscat and sysstat that can be used too.
Oracle has a SYS catalog, owned by the SYS user. It's a little different to our goals here (it's more the DD owner). When comparing our goals with sys to Oracle, you could compare it to the V$ tables in many ways. However in Oracle SYS also includes a number of packages (routines) that can be used for administration tasks – the SYS.DBMS_* packages.
Looking a little closer to home, one only has to look at the excellent common_schema maintained by Shlomi Noach, which provides a whole host of views, procedures and functions to help manage and interact with MySQL, around objects, runtime metadata, and more (QueryScript for example is incredibly powerful).
With all of these in mind, we also wanted something a little more generic than ps_helper, which was created to make Performance Schema itself a little easier to understand. Something that could contain routines and views that are not necessarily Performance Schema specific (although that is what the bulk of the objects are for at the moment).
So what are the changes that have been made when migrating ps_helper to sys?
A quick summary would be:
- mysql-sys now uses the GPLv2 license.
- This is an important change, it aligns it with other MySQL products, and will follow the same procedures as other Oracle products. If you want to submit patches, you should follow the OCA. Unfortunately, this means that merges from mysql-sys to ps_helper can not happen. More on this shortly
- All objects now fully specify all characteristic clauses
- For instance, they all have a "root@localhost" DEFINER with SQL SECURITY INVOKER, and fully specify other things such as whether a routine is DETERMINISTIC, how it interacts (i.e. READS SQL DATA), or the ALGORITHM a view should use.
- All [name]_raw views have been renamed to x$[name]
- Within ps_helper, the "foo" and "foo_raw" tables all sort together, which makes it a little harder to scan the list of user friendly views from the output of SHOW TABLES. By renaming them all to x$foo, all of the views with some form of raw output sort at the end, and give the nice user friendly list individually first. Oracle also has something like this with their X$ vs V$ views, although X$ is "hidden" in that case.
- A lot of the objects have had their names changed, as well as some columns
- Many of the routines are now prefixed with ps_*, many of them have had their names changed for a little more clarity etc.
- All "count" or "count_star" columns now use "total", which was also used a lot. There was a lot of inconsistency there.
- All stored routines (functions and procedures) now have a little documentation in their COMMENT clauses.
- This means you can get their docs with "SELECT ROUTINE_NAME, ROUTINE_COMMENT FROM INFORMATION_SCHEMA.ROUTINES WHERE ROUTINE_SCHEMA = 'sys';"
- A number of new views were added, particularly around summarizing user activity, and making sure that each formatted view had an x$ counterpart.
- A WHOLE BUNCH of bugs were fixed along the way.
Unfortunately, I can not really merge those bug fixes back to ps_helper – they happened as I was going through each of the objects for the clean up, but their commits were kind of mixed in with all these other changes (including the license header changes). If I try to merge this back to ps_helper, I have to re-license ps_helper as GPLv2. I'd still like to keep ps_helper as my little playground though, to try out views and routines that could be useful one day to the sys schema.
I'm not sure what to do with the dbahelper (ps_helper) repository at the moment. I am tempted to remove all of the objects that are within mysql-sys itself, and do a similar rename of the remaining objects to "sys" (so they would build on each other). I do still want to use it as my "hey this could be an interesting object to add" playground. Your thoughts on this from the community would be appreciated.
I've now updated the ps_helper page to remove the outdated examples, and to simply point at the Github pages directly instead. The README for the mysql-sys repository now contains the most up to date examples and documentation, and should be used instead.
I hope to blog some more about the changes, and the new things that came along in the near future, but this blog is long enough for now.
Finally, I'd like to take the opportunity to thank all in the community for their great feedback on ps_helper. It helped to turn it in to a much more mature project.
I'm looking forward to its new future as sys!
MySQL Connect and Oracle Open World Presentations Online
After a slight delay (travel and catching up with "real work"), I've now uploaded the talks that I gave at MySQL Connect and Oracle Open World.
They are available on my Presentations Page, and inline below for convenience. The "Introduction to MySQL Enterprise Monitor" talk was actually a full demo, but there are some screenshots of MEM 3.0 in there if you're interested in seeing a high level picture of what it looks like now.
Thanks to all that attended my talks, I got a lot of good questions and feedback!
New Article on Performance Schema on Dr.Dobbs
Interested in an introduction to using Performance Schema to profile statement activity on your MySQL instances? Head on over to Detailed Profiling of SQL Activity in MySQL 5.6, which was recently published on the Dr.Dobbs site!
Oracle *Has Not* Stopped Publishing MySQL To Launchpad
Stewart over at Percona noticed that our trees on Launchpad were out of date, and this has been picked up and taken as gospel by others, notably Henrik today.
Unfortunately, Henrik doesn't seem to have checked Launchpad before publishing his blog – otherwise he would have noticed that all of the trees are currently up to date with the current releases.
He would also have noticed that there is now a new MySQL 5.6 branch, so that we can continue with trunk as the future MySQL version after MySQL 5.6.
So let me repeat this again, Oracle has not stopped updating the MySQL branches on Launchpad.
Sorry about the blips in updating, however.
ALTER USER … PASSWORD EXPIRE – bitten by a bug
MySQL 5.6.6 was released yesterday – the list of changes is impressive.
One of the new things added within 5.6.6 was the ALTER USER … PASSWORD EXPIRE statement, which allows an administrator to expire the passwords for a user, so that they must update their password on next login.
Unfortunately, this was released within 5.6.6 with a serious problem – already noted by Kolbe Kegel – as it updated the password column to an empty string, as well as setting the password_expired flag. This obviously has serious implications, that really means you should not use this feature, if you are planning on using 5.6.6 in production (for whatever reason, given that it is not a GA release yet).
The good news is that this was caught and fixed around a month ago, but didn't quite make the 5.6.6 cut off date (where we build, and put the release through QA testing) – here's the same kind of test on my self-built 5.6.7 release:
$ mysql -uroot -pmysql -P3307 Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 152259 Server version: 5.6.7 MySQL Community Server (GPL) Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> grant all on *.* to mark@localhost identified by 'mark'; Query OK, 0 rows affected (0.00 sec) mysql> select user, host, password, password_expired from mysql.user; +-----------------+-----------+-------------------------------------------+------------------+ | user | host | password | password_expired | +-----------------+-----------+-------------------------------------------+------------------+ | root | localhost | *E74858DB86EBA20BC33D0AECAE8A8108C56B17FA | N | | mark | localhost | *E6ACCEDB2495496B191ED488F598F04239C85E73 | N | +-----------------+-----------+-------------------------------------------+------------------+ 2 rows in set (0.00 sec) mysql> alter user mark@localhost password expire; Query OK, 0 rows affected (0.00 sec) mysql> select user, host, password, password_expired from mysql.user; +-----------------+-----------+-------------------------------------------+------------------+ | user | host | password | password_expired | +-----------------+-----------+-------------------------------------------+------------------+ | root | localhost | *E74858DB86EBA20BC33D0AECAE8A8108C56B17FA | N | | mark | localhost | *E6ACCEDB2495496B191ED488F598F04239C85E73 | Y | +-----------------+-----------+-------------------------------------------+------------------+ 2 rows in set (0.00 sec) mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) mysql> select user, host, password, password_expired from mysql.user; +-----------------+-----------+-------------------------------------------+------------------+ | user | host | password | password_expired | +-----------------+-----------+-------------------------------------------+------------------+ | root | localhost | *E74858DB86EBA20BC33D0AECAE8A8108C56B17FA | N | | mark | localhost | *E6ACCEDB2495496B191ED488F598F04239C85E73 | Y | +-----------------+-----------+-------------------------------------------+------------------+ 2 rows in set (0.00 sec) mysql> exit Bye $ mysql -umark -P3307 ERROR 1045 (28000): Access denied for user 'mark'@'localhost' (using password: NO) $ mysql -umark -pmark -P3307 Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 163413 Server version: 5.6.7 Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> select * from mysql.user; ERROR 1820 (HY000): You must SET PASSWORD before executing this statement mysql> set password = password('newmark'); Query OK, 0 rows affected (0.00 sec) mysql> show grants; +----------------------------------------------------------------------------------------------------------------------+ | Grants for mark@localhost | +----------------------------------------------------------------------------------------------------------------------+ | GRANT ALL PRIVILEGES ON *.* TO 'mark'@'localhost' IDENTIFIED BY PASSWORD '*00B6543E480F70E68EB0FE311882F1B32E7EEF43' | +----------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) mysql> select password('newmark'); +-------------------------------------------+ | password('newmark') | +-------------------------------------------+ | *00B6543E480F70E68EB0FE311882F1B32E7EEF43 | +-------------------------------------------+ 1 row in set (0.00 sec)
So, please don't use this feature until 5.6.7 is released! We're sorry that this problem slipped in to the milestone release, but thankfully it was caught before our GA release, both with our own internal testing, and external community interest (thanks Kolbe!).
The documentation will be updated to reflect this in the near future as well.
A MySQL Replication Load Average with Performance Schema
Monitoring MySQL's replication has always been a bit hit and miss, especially when trying to detect whether a slave is becoming overloaded or not. There's been a few ways to do this in the past:
- Monitor the Seconds_behind_master variable from SHOW SLAVE STATUS, which is the difference between the timestamp passed down from the master that the event was executed, to the time that the SQL thread started executing the event.
- Use something like mk-heartbeat, which operates in the same way as Seconds_Behind_Master (trying to show you the actual time difference between the master and slave), and is a little more robust in complex replication chains, and other situations where Seconds_Behind_Master falls down (such as when the IO thread is lagging).
These are fine for knowing how far out of date your slave is getting, but they don't really tell you that your slave is becoming overloaded. In other words, this only lets you detect that the slave is lagging, not that it is going to lag.
You can also attempt to monitor replication lag according to the differences in where the two threads are reading / writing to, based on the binary log and relay log positions compared against the master. This allows you to dive a little more in to where the delays may be happening in the chain, rather than relying on a single time behind metric.
Here's a (somewhat simplified and not entirely accurate – the IO thread reads via the master) representation of how an event moves through replication, and the points that can be monitored with current instrumentation (including time delayed replication within 5.6):
Now, I say "Delay" above, that's not entirely accurate, as most of those metrics actually record the file, and position within that file, that each stage is at – they allow you to measure the number of bytes difference at each stage, not the latency of each of the stages.
This allows you to generate these metrics:
- IO Thread Bytes Behind Master:
- Master (File / Position) – Slave (Master_Log_File / Relay_Master_Log_Pos)
- SQL Thread Bytes Behind IO Thread:
- Slave (Master_Log_File / Relay_Master_Log_Pos) – Slave (Relay_Log_file / Relay_Log_Pos)
- SQL Thread Bytes Behind Master:
- Master (File / Position) – Slave (Relay_Log_file / Relay_Log_Pos)
- Slave Bytes Behind Master:
- Master (File / Position) – Slave (Relay_Master_Log_file / Exec_Master_Log_Pos)
- SQL Thread Current Event Size:
- Slave (Relay_Log_file / Relay_Log_Pos) – Slave (Relay_Master_Log_file / Exec_Master_Log_Pos)
- Seconds Behind Overall (maybe):
- Slave (Seconds_Behind_Master)
Actually generating this data consistently is problematic however. As it requires two statements to be executed on two entirely different systems, you will need to ensure that you execute these as close as possible to each other in time, so ntp will need to be installed on each system, and they will need to be kicked off on a precise clock based schedule on each server. Even then, they really can only be used as an indicator, not a source of truth, simply because of this.
The expressions above are slightly simplified as well. Logs roll over, so you also need to know the size that logs can get to using either max_binlog_size or max_relay_log_size (if non-zero), as appropriate, and calculate the difference depending on the number of logs between each position – not entirely simple.
But even with these metrics in hand, they don't really tell us much – the size of an event within the replication stream can differ depending on whether statement based or row based binary logging is enabled. A "small" event could be a statement that takes a very long time to run, or a single row based event that executes in no time at all. YMMV.
So we can use these to profile the system, and graphing over time probably wouldn't hurt, but relying on them as an indicator that you should start looking at a slave because of performance issues gets tough, unless you rely on something that tells you that you are already behind, in a reactive rather than proactive manner.
What we really need is an indicator that tells us that we should be looking in to performance problems before the slave even starts to lag at all.
And so we come back to the title of this blog. The idea is simple, we need a "Replication Load Average", that can tell us how busy the slave really is – if it's not executing 100% of the time, then the chances are it's not really falling behind. If it's starting to be busy 90-95% of the time, then the chances are, it could start to lag pretty soon. It's not a new idea either, I've seen a couple of prominent people wishing for such a thing:
Peter Zaitsev: Idea: MySQL Should add "Replication Load Average" metric which would tell what time replication thread was busy processing events compared to being idle. This would help a lot to see when you're close to the limit.
Jeremy Cole: I agree however that getting saturation information is important and is something we will potentially implement in Twitter MySQL. It would be useful to get essentially a %busy/%idle stat for the SQL thread.
I'm happy to say gentlemen, this is already within 5.5, and has been since it was released.
When the SQL thread has executed all events from the relay log, it enters MYSQL_BIN_LOG::wait_for_update_relay_log(), which then waits for a condition called update_cond to be raised. This is exactly where the slave goes "idle". And luckily within Performance Schema in 5.5, we can track conditions. Conditions get timed within Performance Schema for the time they were waiting for the condition to be raised by another thread – so in the above case, the total time that the SQL thread was idle.
The condition waited for within wait_for_update_relay_log() can be mapped to the wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond event within Performance Schema – here's a snapshot from a 5.5 server that is in the middle of a replication tree (it's both a slave and a master, so has both relay logs and binary logs enabled – the wait/synch/cond/sql/MYSQL_BIN_LOG::update_cond is a similar condition that is used by the IO thread logged in to a master, to signal that new events have been written to the binary log.):
mysql> select * from events_waits_summary_global_by_event_name order by sum_timer_wait desc limit 5; +------------------------------------------------------------+------------+--------------------+----------------+----------------+----------------+ | EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | MIN_TIMER_WAIT | AVG_TIMER_WAIT | MAX_TIMER_WAIT | +------------------------------------------------------------+------------+--------------------+----------------+----------------+----------------+ | wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond | 595360 | 677596417470900396 | 15021936 | 1138128892553 | 20142735786288 | | wait/synch/cond/sql/MYSQL_BIN_LOG::update_cond | 60466 | 167695645597814052 | 187123608 | 2773387450762 | 84468739641228 | | wait/io/file/myisam/kfile | 3237163 | 844640545485132 | 696600 | 260919992 | 1122678434952 | | wait/io/file/myisam/dfile | 4156623 | 377275756973940 | 206712 | 90764968 | 2911791051108 | | wait/synch/cond/sql/COND_queue_state | 11 | 321025543065288 | 166075717824 | 29184140278662 | 59224373303796 | +------------------------------------------------------------+------------+--------------------+----------------+----------------+----------------+
OK, so we can get to the raw data.. Great.. But how do we turn that in to a load average? The key is to read the values on a set interval, compute the deltas of the SUM_TIMER_WAIT statistic, and compare the delta – which is the idle time within the interval – to the overall interval.
As a quick proof of concept, I'll use an event to do this, to call a procedure to gather the data, compute the deltas, and generate some extra statistics. Before boring you with the details of the implementation, let me first give a snapshot of the end result that we are striving for:
mysql> select tstamp, busy_pct, one_min_avg, five_min_avg, fifteen_min_avg -> from slave_sql_load_average -> order by tstamp desc limit 10; +---------------------+----------+-------------+--------------+-----------------+ | tstamp | busy_pct | one_min_avg | five_min_avg | fifteen_min_avg | +---------------------+----------+-------------+--------------+-----------------+ | 2012-07-24 14:00:29 | 79.94 | 67.10 | 66.92 | 66.25 | | 2012-07-24 14:00:24 | 39.97 | 67.10 | 66.93 | 66.15 | | 2012-07-24 14:00:19 | 79.92 | 67.11 | 66.92 | 66.34 | | 2012-07-24 14:00:14 | 84.31 | 67.09 | 66.93 | 66.24 | | 2012-07-24 14:00:09 | 38.62 | 66.98 | 66.93 | 66.11 | | 2012-07-24 14:00:04 | 83.26 | 67.03 | 66.95 | 66.31 | | 2012-07-24 13:59:59 | 79.30 | 66.73 | 66.90 | 66.19 | | 2012-07-24 13:59:54 | 39.97 | 66.81 | 66.91 | 66.09 | | 2012-07-24 13:59:49 | 79.94 | 66.81 | 66.91 | 66.28 | | 2012-07-24 13:59:44 | 79.95 | 66.80 | 66.91 | 66.18 | +---------------------+----------+-------------+--------------+-----------------+
This gives what both Jeremy and Peter were asking for – a single "busy" counter, with some load average like statistics over 3 different intervals, with a snapshot every 5 seconds in this example. This allows us to see that there are peaks and troughs in the load of the slave on the shorter interval, but flattens it's overall load with the larger averaged intervals to give you a better baseline of how busy the slave really is over time.
So first, let's start with the procedure that generates this data:
DROP PROCEDURE IF EXISTS compute_slave_load_average; DELIMITER $$ CREATE PROCEDURE compute_slave_load_average() BEGIN DECLARE v_ps_enabled VARCHAR(3); DECLARE v_update_cond_enabled VARCHAR(3); DECLARE v_update_cond_timed VARCHAR(3); DECLARE v_wait_count BIGINT DEFAULT 0; DECLARE v_last_wait_count BIGINT DEFAULT 0; DECLARE v_wait_count_delta BIGINT DEFAULT 0; DECLARE v_wait_sum BIGINT DEFAULT 0; DECLARE v_last_wait_sum BIGINT DEFAULT 0; DECLARE v_wait_delta BIGINT DEFAULT 0; DECLARE v_last_tstamp DATETIME; DECLARE v_wait_sum_tstamp DATETIME; DECLARE v_time_diff BIGINT; DECLARE v_current_wait VARCHAR(128); DECLARE v_current_timer_end BIGINT; DECLARE v_busy_pct DECIMAL(5,2); DECLARE v_one_min_avg DECIMAL(5,2); DECLARE v_five_min_avg DECIMAL(5,2); DECLARE v_fifteen_min_avg DECIMAL(5,2); DECLARE v_insert_id BIGINT; /* Disable binary logging */ SET sql_log_bin = 0; /* Check Performance Schema is enabled properly */ SELECT VARIABLE_VALUE INTO v_ps_enabled FROM INFORMATION_SCHEMA.GLOBAL_VARIABLES WHERE VARIABLE_NAME = 'performance_schema'; SELECT enabled, timed INTO v_update_cond_enabled, v_update_cond_timed FROM performance_schema.setup_instruments WHERE name = 'wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond'; IF v_ps_enabled = 'OFF' THEN INSERT INTO ps_helper_logs (module, message) VALUES ('compute_slave_load_average', 'performance_schema is disabled'); ELSEIF v_update_cond_enabled = 'NO' OR v_update_cond_timed = 'NO' THEN INSERT INTO ps_helper_logs (module, message) VALUES ('compute_slave_load_average', CONCAT('performance_schema is not configured properly, the wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond event is currently set to enabled: ', v_update_cond_enabled, ', timed: ', v_update_cond_timed, 'within the setup_instruments table')); ELSE /* Get the latest MYSQL_RELAY_LOG::update_cond wait info for the slave SQL thread */ SELECT his.sum_timer_wait, his.count_star, cur.event_name, cur.timer_end, SYSDATE() INTO v_wait_sum, v_wait_count, v_current_wait, v_current_timer_end, v_wait_sum_tstamp FROM performance_schema.events_waits_summary_by_thread_by_event_name his JOIN performance_schema.threads thr USING (thread_id) JOIN performance_schema.events_waits_current cur USING (thread_id) WHERE his.event_name = 'wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond' AND name = 'thread/sql/slave_sql'; /* Get the last summary for comparison */ SELECT idle_sum, events_sum, tstamp INTO v_last_wait_sum, v_last_wait_count, v_last_tstamp FROM slave_sql_load_average ORDER BY id DESC LIMIT 1; /* Compute the interval and event count deltas */ SET v_time_diff = (UNIX_TIMESTAMP(v_wait_sum_tstamp) - UNIX_TIMESTAMP(v_last_tstamp)) * 1000000000000; SET v_wait_count_delta = v_wait_count - v_last_wait_count; /* Compute the delta busy percentages */ IF (v_wait_sum != v_last_wait_sum AND v_wait_count_delta > 0) THEN /* There have been waits during the period, calculate the stats */ SET v_wait_delta = v_wait_sum - v_last_wait_sum; IF (v_wait_delta > v_time_diff) THEN /* The last wait was longer than our current period, estimate waits in period */ SET v_wait_delta = v_wait_delta % (v_time_diff * FLOOR(v_wait_delta/v_time_diff)); SET v_busy_pct = 100 - ((v_wait_delta / v_time_diff) * 100); ELSE /* In a normal period, calculate using raw wait delta */ SET v_busy_pct = 100 - ((v_wait_delta / v_time_diff) * 100); END IF; ELSEIF (v_current_wait = 'wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond' AND v_current_timer_end IS NULL) THEN /* Waiting 100% on a single event for the entire period, i.e 100% idle*/ SET v_wait_delta = v_time_diff; SET v_busy_pct = 0.00; ELSE /* Waiting 100% on a single event for the entire period that is not update_cond */ SET v_wait_delta = v_time_diff; SET v_busy_pct = 100.00; END IF; /* Log the initial stats */ INSERT INTO slave_sql_load_average (idle_sum, idle_delta, idle_avg, idle_delta_formatted, events_sum, events_delta, busy_pct, tstamp, current_wait) VALUES (v_wait_sum, v_wait_delta, format_time(v_wait_delta / v_wait_count_delta), format_time(v_wait_delta), v_wait_count, v_wait_count_delta, v_busy_pct, v_wait_sum_tstamp, v_current_wait); SELECT LAST_INSERT_ID() INTO v_insert_id; /* Compute the averages taking the last interval in to account */ SELECT SUM(busy_pct)/COUNT(*) INTO v_one_min_avg FROM slave_sql_load_average WHERE busy_pct IS NOT NULL AND tstamp > SYSDATE() - INTERVAL 1 MINUTE; SELECT SUM(busy_pct)/COUNT(*) INTO v_five_min_avg FROM slave_sql_load_average WHERE busy_pct IS NOT NULL AND tstamp > SYSDATE() - INTERVAL 5 MINUTE; SELECT SUM(busy_pct)/COUNT(*) INTO v_fifteen_min_avg FROM slave_sql_load_average WHERE busy_pct IS NOT NULL AND tstamp > SYSDATE() - INTERVAL 15 MINUTE; UPDATE slave_sql_load_average SET one_min_avg = v_one_min_avg, five_min_avg = v_five_min_avg, fifteen_min_avg = v_fifteen_min_avg WHERE id = v_insert_id; /* Purge anything older than 2 hours */ DELETE FROM slave_sql_load_average WHERE tstamp < NOW() - INTERVAL 2 HOUR; END IF; /* Re-enable binary logging */ SET sql_log_bin = 1; END$$ DELIMITER ;
At a high level, this gathers the wait data for the wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond event from the events_waits_summary_by_thread_by_event_name table for only the slave SQL thread, which is exposed as thread/sql/slave_sql, then grabs the last snapshot of data for comparison. It then looks at the current and previous wait info to determine how to to calculate the actual busy percentage based on how the event data has changed over the period, computes the deltas and busy percent for that last interval appropriately, and inserts that in to a table.
Finally it computes the moving averages for the 3 sets of defined intervals and updates the just inserted row with that data (so the moving average also takes in to account the current snapshot), and finally purges anything older than two hours, to store a rolling two hour window of 5 second snapshots.
It also disables binary logging (because these are local statistics that should not be propagated to downstream systems), and does some checking whether performance schema is enabled, and logs appropriately within a logging table if not.
The table structures needed for the procedure are as follows:
DROP TABLE IF EXISTS slave_sql_load_average; CREATE TABLE slave_sql_load_average ( id BIGINT AUTO_INCREMENT PRIMARY KEY, tstamp TIMESTAMP, idle_avg VARCHAR(12), idle_delta_formatted VARCHAR(12), busy_pct DECIMAL(5,2), one_min_avg DECIMAL(5,2), five_min_avg DECIMAL(5,2), fifteen_min_avg DECIMAL(5,2), idle_sum BIGINT, idle_delta BIGINT, events_sum INT, events_delta INT, current_wait VARCHAR(128), KEY (tstamp) ) ENGINE = InnoDB; DROP TABLE IF EXISTS ps_helper_logs; CREATE TABLE ps_helper_logs ( id BIGINT AUTO_INCREMENT PRIMARY KEY, tstamp TIMESTAMP, module VARCHAR(64), message TEXT ) ENGINE = InnoDB;
The full output from the table looks like this:
mysql> select * from slave_sql_load_average order by tstamp desc limit 5; +-----+---------------------+-----------+----------------------+----------+-------------+--------------+-----------------+--------------------+---------------+------------+--------------+--------------------------------------------------+ | id | tstamp | idle_avg | idle_delta_formatted | busy_pct | one_min_avg | five_min_avg | fifteen_min_avg | idle_sum | idle_delta | events_sum | events_delta | current_wait | +-----+---------------------+-----------+----------------------+----------+-------------+--------------+-----------------+--------------------+---------------+------------+--------------+--------------------------------------------------+ | 106 | 2012-07-24 13:20:34 | 491.11 ms | 982.22 ms | 80.36 | 67.07 | 65.29 | 65.98 | 689661649765993320 | 982222907152 | 605561 | 2 | wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond | | 105 | 2012-07-24 13:20:29 | 500.80 ms | 1.00 s | 79.97 | 67.07 | 65.28 | 65.84 | 689655667543086168 | 1001609993416 | 605559 | 2 | wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond | | 104 | 2012-07-24 13:20:24 | 1.50 s | 3.01 s | 39.90 | 67.05 | 65.28 | 65.70 | 689649665933092752 | 3005124404328 | 605557 | 2 | wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond | | 103 | 2012-07-24 13:20:19 | 331.17 ms | 993.52 ms | 80.13 | 67.07 | 65.28 | 65.96 | 689646660808688424 | 993523754020 | 605555 | 3 | wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond | | 102 | 2012-07-24 13:20:14 | 192.27 ms | 769.07 ms | 84.62 | 67.04 | 65.29 | 65.82 | 689640667284934404 | 769065072628 | 605552 | 4 | wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond | +-----+---------------------+-----------+----------------------+----------+-------------+--------------+-----------------+--------------------+---------------+------------+--------------+--------------------------------------------------+
And the event that kicks all of this off:
DROP EVENT IF EXISTS monitor_slave_load_average; DELIMITER $$ CREATE EVENT IF NOT EXISTS monitor_slave_load_average ON SCHEDULE EVERY 5 SECOND DO BEGIN CALL compute_slave_load_average(); END$$ DELIMITER ;
You'll also need the format_time() function from ps_helper.
Here's a graph of the data over time, including a period where the slave IO thread was turned off, where you can see the moving average windows decay more slowly than the last intervals statistics:
Finally, some notes on this method:
- Importantly, this doesn't require any knowledge of timestamps across systems, or require any synchronization for collection across systems
- Yes this is not exactly like the Linux load average – which uses an Exponential Moving Average, which weights snapshots differently the older they get, but that would also be possible
- The time delta recorded within 5.5 is only possible at a 1 second granularity, which makes us lose quite a bit of precision when compared to picosecond precision wait information, within 5.6 we can use microsecond precision with SYSDATE(6), and get even more accurate reporting – and note, SYSDATE() is used instead of NOW(), as within procedures NOW() returns the timestamp for when the procedure started, not when the function is called
- The first 15 minutes worth of aggregate data should be taken with a pinch of salt, wait for the data set to "warm up"
- This doesn't necessarily negate the need for the other statistics described above – which can all be used to trace where latency might be, and in the case of Seconds_Behind_Master or a heartbeating implementation can still give information on how far behind you are getting, once your slave does really start to lag, as well as using something like Jeremy's velocity metric (linked via is name above), which can give an idea of the general direction of of a slave once it's started lagging.
With all that said though, I think this is the *key* metric that anybody should be tracking first and foremost. It is far better to fix problems or spread load before you start lagging.
The joys of digging through the different events in Performance Schema to see what they actually record! There's gold in there still to be mined, but I think this one is a genuine 14 oz nugget.
Monitoring Processes with Performance Schema in MySQL 5.6
I've written before about how you can monitor the last wait per connection with Performance Schema in MySQL 5.5.
That solution joined the performance_schema.events_waits_current with the performance_schema.threads and INFORMATION_SCHEMA.PROCESSLIST tables.
Joining to INFORMATION_SCHEMA.PROCESSLIST is a necessary evil within MySQL 5.5, to be able to get the user and host information etc. – however it is certainly not optimal, because using INFORMATION_SCHEMA.PROCESSLIST (and it's SHOW [FULL] PROCESSLIST cousin) incurs a bunch of mutexes to be locked – the fill_schema_processlist() function attempts to lock LOCK_thread_count globally (which will block new connections whilst the entire processlist output is generated), as well as tmp->LOCK_thd_data per thread that it iterates over (which could in turn stall each thread if they try to access this) and mysys_var->mutex which protects the mysys_var struct (used around killing connections).
If you are polling the processlist output frequently (I've seen some solutions out there poll it once a second, or even lower, sometimes), then the chances are, you are stalling real application connections as well. This is why we don't use these commands within MySQL Enterprise Monitor for things like Query Analysis.
With that in mind, we have now extended the performance_schema.threads table in MySQL 5.6 to include all of the information that is available within the INFORMATION_SCHEMA.PROCESSLIST table:
mysql> desc performance_schema.threads; +---------------------+------------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------------------+------------------+------+-----+---------+-------+ | THREAD_ID | int(11) | NO | | NULL | | | NAME | varchar(128) | NO | | NULL | | | TYPE | varchar(10) | NO | | NULL | | | PROCESSLIST_ID | int(11) | YES | | NULL | | | PROCESSLIST_USER | varchar(16) | YES | | NULL | | | PROCESSLIST_HOST | varchar(60) | YES | | NULL | | | PROCESSLIST_DB | varchar(64) | YES | | NULL | | | PROCESSLIST_COMMAND | varchar(16) | YES | | NULL | | | PROCESSLIST_TIME | bigint(20) | YES | | NULL | | | PROCESSLIST_STATE | varchar(64) | YES | | NULL | | | PROCESSLIST_INFO | longtext | YES | | NULL | | | PARENT_THREAD_ID | int(11) | YES | | NULL | | | ROLE | varchar(64) | YES | | NULL | | | INSTRUMENTED | enum('YES','NO') | NO | | NULL | | +---------------------+------------------+------+-----+---------+-------+ 14 rows in set (0.02 sec)
As with all other instrumentation within Performance Schema, this is entirely non-blocking. When you upgrade to MySQL 5.6, it's time to start forgetting about SHOW PROCESSLIST, or INFORMATION_SCHEMA.PROCESSLIST. They've served us well for a long time, but it's time to wave goodbye to their contentious ways.
There's another bonus in the threads table within MySQL 5.6 as well, and that is the newly added INSTRUMENTED column. This allows you turn on and off instrumentation, per thread, dynamically. You do this by simply running an UPDATE:
mysql> select name, type, instrumented from threads; +----------------------------------------+------------+--------------+ | name | type | instrumented | +----------------------------------------+------------+--------------+ | thread/sql/main | BACKGROUND | YES | | thread/innodb/io_handler_thread | BACKGROUND | YES | | thread/innodb/io_handler_thread | BACKGROUND | YES | | thread/innodb/io_handler_thread | BACKGROUND | YES | | thread/innodb/io_handler_thread | BACKGROUND | YES | | thread/innodb/io_handler_thread | BACKGROUND | YES | | thread/innodb/io_handler_thread | BACKGROUND | YES | | thread/innodb/io_handler_thread | BACKGROUND | YES | | thread/innodb/io_handler_thread | BACKGROUND | YES | | thread/innodb/io_handler_thread | BACKGROUND | YES | | thread/innodb/io_handler_thread | BACKGROUND | YES | | thread/innodb/srv_lock_timeout_thread | BACKGROUND | YES | | thread/innodb/srv_error_monitor_thread | BACKGROUND | YES | | thread/innodb/srv_monitor_thread | BACKGROUND | YES | | thread/innodb/srv_master_thread | BACKGROUND | YES | | thread/innodb/srv_purge_thread | BACKGROUND | YES | | thread/innodb/page_cleaner_thread | BACKGROUND | YES | | thread/sql/shutdown | BACKGROUND | YES | | thread/sql/manager | BACKGROUND | YES | | thread/sql/con_sockets | BACKGROUND | YES | | thread/sql/one_connection | FOREGROUND | YES | | thread/sql/one_connection | FOREGROUND | YES | | thread/sql/one_connection | FOREGROUND | YES | | thread/sql/one_connection | FOREGROUND | YES | | thread/sql/one_connection | FOREGROUND | YES | | thread/sql/one_connection | FOREGROUND | YES | | thread/sql/one_connection | FOREGROUND | YES | +----------------------------------------+------------+--------------+ 27 rows in set (0.00 sec) mysql> update threads set instrumented = 'NO' where name != 'thread/sql/one_connection'; Query OK, 20 rows affected (0.00 sec) Rows matched: 20 Changed: 20 Warnings: 0 mysql> select name, type, instrumented from threads; +----------------------------------------+------------+--------------+ | name | type | instrumented | +----------------------------------------+------------+--------------+ | thread/sql/main | BACKGROUND | NO | | thread/innodb/io_handler_thread | BACKGROUND | NO | | thread/innodb/io_handler_thread | BACKGROUND | NO | | thread/innodb/io_handler_thread | BACKGROUND | NO | | thread/innodb/io_handler_thread | BACKGROUND | NO | | thread/innodb/io_handler_thread | BACKGROUND | NO | | thread/innodb/io_handler_thread | BACKGROUND | NO | | thread/innodb/io_handler_thread | BACKGROUND | NO | | thread/innodb/io_handler_thread | BACKGROUND | NO | | thread/innodb/io_handler_thread | BACKGROUND | NO | | thread/innodb/io_handler_thread | BACKGROUND | NO | | thread/innodb/srv_lock_timeout_thread | BACKGROUND | NO | | thread/innodb/srv_error_monitor_thread | BACKGROUND | NO | | thread/innodb/srv_monitor_thread | BACKGROUND | NO | | thread/innodb/srv_master_thread | BACKGROUND | NO | | thread/innodb/srv_purge_thread | BACKGROUND | NO | | thread/innodb/page_cleaner_thread | BACKGROUND | NO | | thread/sql/shutdown | BACKGROUND | NO | | thread/sql/manager | BACKGROUND | NO | | thread/sql/con_sockets | BACKGROUND | NO | | thread/sql/one_connection | FOREGROUND | YES | | thread/sql/one_connection | FOREGROUND | YES | | thread/sql/one_connection | FOREGROUND | YES | | thread/sql/one_connection | FOREGROUND | YES | | thread/sql/one_connection | FOREGROUND | YES | +----------------------------------------+------------+--------------+ 25 rows in set (0.00 sec)
This then enables/disables all instrumentation within Performance Schema for each thread appropriately.
Now we have all of this data directly within the threads table, we can also start joining it to some of the other tables within Performance Schema as well, to get even more data on what threads are currently doing within the database. Particularly, we can join to the new events_statements_current table as well, to get extra information on statement execution (you will need to ensure that the events_statements_current consumer is enabled within the setup_consumers table).
Here's an example of another view I'm about to add to ps_helper:
DROP VIEW IF EXISTS processlist_full; CREATE VIEW processlist_full AS SELECT pps.thread_id AS thd_id, pps.processlist_id AS conn_id, IF(pps.name = 'thread/sql/one_connection', CONCAT(pps.processlist_user, '@', pps.processlist_host), REPLACE(pps.name, 'thread/', '')) user, pps.processlist_db AS db, pps.processlist_command AS command, pps.processlist_state AS state, pps.processlist_time AS time, format_statement(pps.processlist_info) AS current_statement, IF(esc.timer_wait IS NOT NULL, format_statement(esc.sql_text), NULL) AS last_statement, IF(esc.timer_wait IS NOT NULL, format_time(esc.timer_wait), NULL) as last_statement_latency, format_time(esc.lock_time) AS lock_latency, esc.rows_examined, esc.rows_sent, esc.rows_affected, esc.created_tmp_tables AS tmp_tables, esc.created_tmp_disk_tables as tmp_disk_tables, IF(esc.no_good_index_used > 0 OR esc.no_index_used > 0, 'YES', 'NO') AS full_scan, ewc.event_name AS last_wait, IF(ewc.timer_wait IS NULL AND ewc.event_name IS NOT NULL, 'Still Waiting', format_time(ewc.timer_wait)) last_wait_latency, ewc.source FROM performance_schema.threads AS pps LEFT JOIN performance_schema.events_waits_current AS ewc USING (thread_id) LEFT JOIN performance_schema.events_statements_current as esc USING (thread_id) ORDER BY pps.processlist_time DESC, last_wait_latency DESC;
And some sample output:
... *************************** 3. row *************************** thd_id: 11617 conn_id: 11596 user: root@localhost db: mysql command: Sleep state: cleaning up time: 0 current_statement: NULL last_statement: SELECT CAST(SUM_NUMBER_OF_BYTE ... nnodb/innodb_log_file' limit 1 last_statement_latency: 2.02 ms lock_latency: 0 ps rows_examined: 39 rows_sent: 0 rows_affected: 0 tmp_tables: 0 tmp_disk_tables: 0 full_scan: YES last_wait: idle last_wait_latency: Still Waiting source: mysqld.cc:848 *************************** 4. row *************************** thd_id: 12474 conn_id: 12453 user: root@localhost db: mem command: Sleep state: cleaning up time: 0 current_statement: NULL last_statement: commit last_statement_latency: 952.49 µs lock_latency: 0 ps rows_examined: 0 rows_sent: 0 rows_affected: 0 tmp_tables: 0 tmp_disk_tables: 0 full_scan: NO last_wait: idle last_wait_latency: Still Waiting source: mysqld.cc:848 *************************** 5. row *************************** thd_id: 12400 conn_id: 12379 user: root@localhost db: ps_helper command: Query state: Copying to tmp table time: 0 current_statement: SELECT pps.thread_id AS thd_id ... e DESC, last_wait_latency DESC last_statement: NULL last_statement_latency: NULL lock_latency: 1.00 ms rows_examined: 0 rows_sent: 0 rows_affected: 0 tmp_tables: 1 tmp_disk_tables: 0 full_scan: YES last_wait: wait/synch/mutex/sql/THD::LOCK_thd_data last_wait_latency: 77.11 ns source: sql_class.h:3843 ...
Listed here are two idle connections, and one active connection (which was the one gathering the data).
The events_statements_current table stores the currently executing, or last executed (if no currently executing), statement per thread. This means that this new form of processlist can not only tell you what is currently executing, but also what last executed for sleeping threads.
All of the statistics listed, such as lock latency, row stats, temporary table stats are for either the current or last statement too. So with "conn_id: 12453″ we can see that it last executed a COMMIT (in the last_statement column), which took 952.49 µs (in the last_statement_latency column).
I can't count how many times I wanted to know the last statement that executed for sleeping connections (particularly those holding open a transaction), nor how many times customers and users asked that question whilst I was in the MySQL Support group, so this is a great step forward.
For idle connections, we now have a new "idle" wait type, which records how long sessions sit idle for as well. For sleeping connections this will always be the current wait (unless you disable the idle instrument). However for active connections, we show the last wait that happened within that statement's execution.
You can see that the last session listed is copying to a temporary table, had lock latency of 1 ms, and last waited on the wait/synch/mutex/sql/THD::LOCK_thd_data event, which basically corresponds to the LOCK_thd_data mutex I was talking about above.
But hang on? I said above that SHOW PROCESSLIST etc. lock this in a blocking way as well right? And this is also grabbing this mutex? What gives? Well, performance_schema is true table storage engine, and the statement executed is still grabbing a read lock against the performance schema table. The statement itself is also creating a temporary table, doing it's own thing within it's own thread as well, and whilst doing that, threads tend to grab their own LOCK_thd_data mutex (there is one per thread), to protect access from other threads – for instance, we LOCK_thd_data to set the current query string, which protects SHOW FULL PROCESSLIST from reading a half written query string for it's INFO. That doesn't change for Performance Schema access – but it does help to show just the kind of contention that SHOW FULL PROCESSLIST can cause!
The above view only gets a subset of the available data per statement available within the events_statements_current table as well – I've omitted all of the sort / scan information for instance. You can add these to the view as you see fit.
I'll be adding some views like this to ps_helper shortly. If you have any feedback in the meantime feel free to comment!
Helper Functions for ps_helper
I love our community.
Not long after posting my update on ps_helper, I had a couple of comments around the formatting of values within the output. Daniël van Eeden gave the suggestion that I could add a couple of Stored Functions, for formatting byte and time based values.
Of course, this was a great idea – not least for myself, because I no longer have to worry about how to format certain columns in the output.
I've added the following:
I've updated all of the views within ps_helper to use these new functions as well (I may have missed some).
Along the way, I caught a couple of bugs (a missing SUM on the COUNT columns of the file IO summaries), and added a couple of updates.
The first was to update the output for the schema_table_statistics view, to include more of the data available within Performance Schema, and also to start using some of the data from the INFORMATION_SCHEMA.INNODB_BUFFER_PAGE table that is now available within MySQL 5.6.
Now the output looks like this:
mysql> select * from schema_table_statistics limit 1\G *************************** 1. row *************************** table_schema: mem table_name: mysqlserver rows_fetched: 27087 fetch_latency: 442.72 ms rows_inserted: 2 insert_latency: 185.04 µs rows_updated: 5096 update_latency: 1.39 s rows_deleted: 0 delete_latency: 0 ps io_read_requests: 2565 io_read_bytes: 1121627 io_read_latency: 10.07 ms io_write_requests: 1691 io_write_bytes: 128383 io_write_latency: 14.17 ms io_misc_requests: 2698 io_misc_latency: 433.66 ms innodb_buffer_pages: 19 innodb_buffer_pages_hashed: 19 innodb_buffer_pages_old: 19 innodb_buffer_bytes_allocated: 311296 innodb_buffer_bytes_data: 1924 innodb_buffer_rows_cached: 2
I've also added another view within the 5.6 set, called statements_with_runtimes_in_95th_percentile.
This outputs all statement digests that have an average run time, in microseconds, that are above the 95th percentile of all statements. It's output is similar to the statement_analysis view:
mysql> select * from statements_with_runtimes_in_95th_percentile where query not like 'show%'; +-------------------------------------------------------------------+-----------+------------+-----------+------------+---------------+-------------+-------------+-----------+---------------+--------------+----------------------------------+ | query | full_scan | exec_count | err_count | warn_count | total_latency | max_latency | avg_latency | rows_sent | rows_sent_avg | rows_scanned | digest | +-------------------------------------------------------------------+-----------+------------+-----------+------------+---------------+-------------+-------------+-----------+---------------+--------------+----------------------------------+ | SELECT plugin_name FROM inform ... tus = ? ORDER BY plugin_name | * | 169 | 0 | 0 | 2.37 s | 64.45 ms | 14.03 ms | 4394 | 26 | 10816 | 23234b56a0b1f1e350bf51bef3050747 | | SELECT `this_` . `target` AS ` ... D `this_` . `timestamp` <= ? | | 118 | 0 | 0 | 170.08 ms | 5.68 ms | 1.44 ms | 13582 | 115 | 13582 | 34694223091aee1380c565076b7dfece | | SELECT CAST ( SUM_NUMBER_OF_BY ... WHERE EVENT_NAME = ? LIMIT ? | * | 566 | 0 | 0 | 779.56 ms | 2.93 ms | 1.38 ms | 342 | 1 | 17286 | 58d34495d29ad818e68c859e778b0dcb | | SELECT `this_` . `target` AS ` ... D `this_` . `timestamp` <= ? | | 118 | 0 | 0 | 153.35 ms | 3.06 ms | 1.30 ms | 13228 | 112 | 13228 | b816579565d5a2882cb8bd496193dc00 | | SELECT `this_` . `target` AS ` ... D `this_` . `timestamp` <= ? | | 118 | 0 | 0 | 143.31 ms | 4.57 ms | 1.21 ms | 13646 | 116 | 13646 | 27ff8681eb2c8cf999233e7507b439fe | | SELECT `this_` . `target` AS ` ... D `this_` . `timestamp` <= ? | | 118 | 0 | 0 | 143.04 ms | 7.22 ms | 1.21 ms | 13584 | 115 | 13584 | 10b863f20e83dcd9c7782dac249acbb0 | | SELECT `this_` . `target` AS ` ... D `this_` . `timestamp` <= ? | | 118 | 0 | 0 | 137.46 ms | 16.73 ms | 1.16 ms | 13922 | 118 | 13922 | 351ebc26af6babb67570843bcc97f6b0 | | UPDATE `mem30__inventory` . `R ... mestamp` = ? WHERE `hid` = ? | | 114 | 0 | 0 | 127.64 ms | 30.33 ms | 1.12 ms | 0 | 0 | 114 | f4ecf2aebe212e7ed250a0602d86c389 | | UPDATE `mem30__inventory` . `I ... ` = ? , `hasOldBlocksTime` ... | | 56 | 0 | 0 | 61.05 ms | 16.41 ms | 1.09 ms | 0 | 0 | 56 | cdc78c70d83c505c5708847ba810d035 | | SELECT `this_` . `target` AS ` ... D `this_` . `timestamp` <= ? | | 118 | 0 | 0 | 121.76 ms | 1.95 ms | 1.03 ms | 13936 | 118 | 13936 | 20f97c53c2a59f5eadc06b2fa90fbe75 | | UPDATE `mem30__inventory` . `M ... mpileOs` = ? WHERE `hid` = ? | | 114 | 0 | 0 | 114.16 ms | 22.34 ms | 1.00 ms | 0 | 0 | 114 | c5d4a65f3f308f4869807e730739af6d | | CALL `dc_string_insert` (...) | | 80 | 0 | 0 | 79.89 ms | 2.62 ms | 998.50 ┬╡s | 0 | 0 | 240 | 93eb9cab8ced45cf3b98400e8803f8af | | SELECT `this_` . `target` AS ` ... D `this_` . `timestamp` <= ? | | 118 | 0 | 0 | 116.19 ms | 1.32 ms | 984.60 ┬╡s | 13484 | 114 | 13484 | bd23afed9a41367591e2b71dac76f334 | +-------------------------------------------------------------------+-----------+------------+-----------+------------+---------------+-------------+-------------+-----------+---------------+--------------+----------------------------------+
And to give an example using some of the bytes formatting as well, here's an example for the latest_file_io view:
mysql> select * from latest_file_io limit 10; +----------------------+----------------------------------------+------------+-----------+-----------+ | thread | file | latency | operation | requested | +----------------------+----------------------------------------+------------+-----------+-----------+ | msandbox@localhost:1 | @@tmpdir/#sqlcf28_1_4e.MYI | 9.26 µs | write | 124 bytes | | msandbox@localhost:1 | @@tmpdir/#sqlcf28_1_4e.MYI | 4.00 µs | write | 2 bytes | | msandbox@localhost:1 | @@tmpdir/#sqlcf28_1_4e.MYI | 56.34 µs | close | NULL | | msandbox@localhost:1 | @@tmpdir/#sqlcf28_1_4e.MYD | 53.93 µs | close | NULL | | msandbox@localhost:1 | @@tmpdir/#sqlcf28_1_4e.MYI | 104.05 ms | delete | NULL | | msandbox@localhost:1 | @@tmpdir/#sqlcf28_1_4e.MYD | 661.18 µs | delete | NULL | | msandbox@localhost:1 | @@datadir/Cerberus.log | 35.99 ms | write | 57 bytes | | msandbox@localhost:1 | @@datadir/ps_helper/latest_file_io.frm | 7.40 µs | stat | NULL | | msandbox@localhost:1 | @@datadir/ps_helper/latest_file_io.frm | 9.81 µs | open | NULL | | msandbox@localhost:1 | @@datadir/ps_helper/latest_file_io.frm | 16.06 µs | read | 3.17 KiB | +----------------------+----------------------------------------+------------+-----------+-----------+
Keep the feedback coming!
I'll see you at MySQL Connect! I will … won't I?
I've had a talk accepted for the MySQL Connect conference, in San Francisco at the end of September. I'll be talking about Profiling with Performance Schema.
Here's the abstract:
The Performance Schema feature was introduced in MySQL 5.5 and has been greatly enhanced within the coming 5.6 release. It exposes a new wealth of instrumentation, to DBAs and developers alike, that enable you to find answers to many of the questions that have been impossible, or more difficult than necessary, to answer in the past. Come to this session to learn how to set up and use Performance Schema to perform everyday profiling and performance monitoring tasks, such as finding problem queries; researching blocked hosts; profiling I/O usage; analyzing resource usage by schema, table, or user; or tracing a session to see exactly where it spends its time.
There is a lot of great content planned for MySQL Connect, it should be a great event!
I hope to see you there!


text:
Marketing Solutions That Work
Attracting New Prospects?
Converting Them to Customers or Clients?
Measurable, Cost Effective Results?
When You Work With Us You Can Expect The Following Results:
- You will attract more new prospects to your business . Our process leads to the development of marketing strategies and techniques that target and attract likely prospects.
- These prospects will be persuaded to choose you over your competition. Marketing messages will be clearer and more persuasive; interesting prospects in what you offer and why you are the right choice to satisfy their needs.
- Your marketing decisions will be smarter and more cost effective . You will be able to measure results and make marketing decisions based on knowing what does and doesn't work.
Why Work With Us?
- Knowledge, Expertise. We have in-depth experience with diverse businesses large and small. We understand online, social and traditional media (print, broadcast and direct mail) and how to combine them to achieve optimal results.
- Analytical Process. We start by listening to your needs and objectives and then conduct a qualitative and quantitative analysis of your business and the competitive arena. We know how to interpret web analytics and use this information to improve your website's performance. We limit risk by testing prior to roll-out.
- Emphasis on Measurable Results. We develop and work with you to execute a marketing action plan based on your objectives and on producing measurable financial results
- Record of Success . Read what our clients say about our work.
Call 919.969.8000 to schedule a FREE 30 minute teleconsultation.
There are a limited number of appointments available so call now to assure your time.
Clients talk about how we´ve helped them:
"We hired
Bo Lofvander, CEO Back on Track on Track Products
Steve Anderson, President, Legacy Parenting Company The Total Transformation Program
"Working with Alan Hecht of Multi-Channel Marketing was one of the best business decisions we made. Alan has completely improved our online presence. He first helped us establish a marketing plan by guiding us to identify three key groups of clients. Next he created a new dynamic website which is much easier to navigate and more effective in attracting prospects. He then started a search engine marketing program for us which has increased our online visibility and more than doubled our mail list entries. We will soon be working with Alan to implement a blog which will further improve our online presence. We would and do recommend working with Multi-Channel Marketing."
Ann and Margo, Ann-Mar Genealogy Trips
"Prime Life Fibers and predecessor companies have done business with Alan Hecht for several years. His company has recently spearheaded the design and implementation of a new website and online business for us. This project was done professionally and on budget. I believe that Alan's wide range of experience and know how would be beneficial to almost any business"
Bob Deerin, President, Prime Life Fibers Inc.
Lori Carswell, Syncopation Software
"Multi-channel Marketing was able to take the mystery out of online marketing. What was of great surprise was that many of our assumptions of what was working were wrong. Multi-Channel Marketing was able to show us where and how to make every marketing dollar work better and improve our website´s online performance. Their financial analysis and recommendations have been extremely productive in helping us focus more on our profitability and bottom line. This process included ferreting out the segments that needed the most overhaul and setting to action a plan to help us grow and be more productive."
Elaine Lockhead, Owner ThinLine Inc.
"We hired Alan Hecht of Multi-Channel Marketing to generate new client leads for us via the internet. He first educated himself substantially about the particulars of my industry in order to achieve better results. He does everything he says he will do, and he has shown a patient diligence with us; guiding us when we need it, deferring to us when appropriate, always treating us as an important client. We now have an ongoing program that is delivering new clients at a highly efficient cost. He is also analyzing our data and helping us come up with new strategies for business growth. I can state with emphasis that Mr. Hecht has earned his position with us as an important partner in the struggle for prosperity in our market."
Stanley Monroe Esq,. Executive Director Carolina Partners in Mental HealthCare, PLLC
"Alan's exhaustive data-driven survey of my medical practice was essentially a SWOT analysis that helped me to examine the demographics of my current patient base and delineate a plan for future expansion, focusing on the patient populations that I am most interested in helping. He then worked with a team of creative designers to create a website that has broad appeal to these groups, and in doing so was able to work within some fairly tight parameters regarding the appropriateness of marketing materials directed to a patient population that is dealing with very personal issues. He shepherded the process every step of the way, and provided me with timely feedback on various aspects of the project. The result is a website that is tasteful and appealing, and targets the patient populations that I enjoy working with most.
Dr. Karen Clark, MD
More Testimonials