How can I detect buttons in a GUI of an app with pattern detection using python? I will explain taking as example the GUI of the Calculator app in windows. 
Given this Calculator application open and focused, I need to find a way of detecting all the buttons.I can only use non-intrusive methods, so things like button id are out of the question. This leads me to image recognition. Or better say image detection, because I don't want a specific image, but a set of images that follow a certain pattern. I know how I can click/right-click/dbl-click/etc using a given button image and image recognition [1]. I don't know how I can do the other way round: scan the GUI and find those areas that meet the requirements of being a button (rectangle, text/icon/graphic labeled etc.). A bigger tackle would be to detect items that don't have the rectangle shape (e.g. icons on a windows desktop)

The nearest thing to what I need is detecting faces in a picture.[2] But I don't know how to apply this in my case. For human face detection, I saw that hundreds of pictures of the face must be used in the Haar cascade generation (don't know how I would do this with only 10-15 button snapshots). If another kind of object has to be detected like an apple, you need to generate Haar cascade for that object again using a lot of images.

Did any of you ever tried to detect buttons, items or whatever in a GUI using only pattern detection? I just need something to tell me "this is an icon/button" so I can grab that region in a snapshot. 

[1] I use SikuliX with Python to perform actions on given patterns. 

[2] I saw that this is done easily using OpenCV and Haar cascades (in XML format). Creating the Haar cacades requires a little bit of patience and skills.
First, have a look at the squares.py sample provided by OpenCV. It should handle a fair number of button types with some tweaking.

Here is the output I got (with some tweaking) for your Calculator example:


I made the following tweaks to the squares application:

Change this code (starting on line 84):

if(result.total == 4 and 
   abs(cv.ContourArea(result)) &gt; 1000 and 
   cv.CheckContourConvexity(result)):


To this:

if(result.total == 4 and 
   abs(cv.ContourArea(result)) &lt; 1300 and 
   abs(cv.ContourArea(result)) &gt; 300 and
   cv.CheckContourConvexity(result)):


Also, since you don't have to worry about scale or rotation variations, check out using matchTemplate. Also, its corresponding tutorial (in C++) is here.

See my other answer for another example of how template matching works. Also, you might find this answer helpful for detecting the top X matches with matchTemplate.

Hope that helps!