Title: ARM Procedure return performance
Tags: arm;armv7;thumb
Question: I'm doing binary instrumentation on Thumb2 executables and I noticed that in some of my benchmarks the instrumented binaries performed better than the original ones (very strange since my instrumentation adds more code to execute).
Looking into this anomaly, I was able to come to the conclusion that the performance differential was caused by the different way I handle returns from procedures. In the original executables most of the returns are handled with
pop {..., pc}
while after my instrumentation these are translated as
pop {..., lr}
because I need to do some calculations with the return address.
I was able to further test this behavior with a simple program which invokes a subroutine for 1 million times and pop {lr}; bx lr performed 49% faster than pop {pc} (which is huge).
All tests were done on a Scaleway C1 Baremetal Server running a Marvell Armada XP quad-core ARMv7 SoC.
Does anybody have an explanation for this?
The server runs Ubuntu 16.04. Here is the source code of my tests: popcptest, bxlrtest. For compilation I used GCC Linaro version 5.4.0 with default settings.
Comment: @old_timer As I've now added to the question, I'm running on top of Ubuntu 16.04. I am unaware of any particular setting regarding branch prediction. As you suggested in your post, I tried playing around with code alignment but that did not make any significant difference (actually, it slightly increased the gap in favor of `pop {lr}; bx lr`).

Comment: As I said, I tried offsetting the alignment (both on the loop and the subroutine), but nothing really changed. The difference in performance just oscillates between 47% and 51%.

Comment: @old_timer I'm not "stuck" at all. The fact that I see a performance improvement after my instrumentation is a strange but nice collateral effect. I just want to understand why this happens.

Comment: @old_timer Splitting the `pop` makes things a little slower and using 2 `ldr` a little faster (about 5%). Also tried `ldr r7,[sp],#4; ldr lr,[sp],#4; bx lr` and this almost matches the speed of `pop {r7,lr}; bx lr`. I'm inclined to say that this is just caused by a bad implementation of the operation of loading the program counter in this particular processor, but I don't have access to any other ARMv7 chip for comparison.

Comment: **Scaleway C1 Baremetal Server**; these people are brilliant for trying to change the meaning of 'Baremetal'; unhosted Linux... that is not what baremetal means to the majority of the people who can answer your question.

Comment: Looks like a terrible microarchitecture to me. Why am I not surprised about terrible ARM32 microarchitectures?

Comment: arms are not microcoded if that is what you are implying.

Comment: it is somewhat trivial to demonstrate on cores like this two instructions (loop: subs r0,r0,#1; bne loop) different performance numbers depending on where these instructions lie in the fetch line, arm or thumb.  add more machine code and the performance changes.   take the same machine code of any one of your tests add a nop out in front of it, even near the bootstrap, causing the alignment to change and you should see a performance change in the test without any other changes, just alignment.  This is expected.  this is how processors work.

Comment: did you have branch prediction on, which ones, did you clear the branch prediction caches between tests. do you have caches on, which ones, did you clear those between tests.  do you have any chip specific prefetch logic, is that on.  did you try a sweep of several alignments of the code under test?  run it, move the alignment by 4, run it, move the alingment by 4, repeat at least 16 times.  what are you using to time this, how accurate is your measurement and how do you know, is the measurement affected by the alignment changes?

Comment: you are running on top of an operating system?

Comment: I try to get folks to use "cloud baremetal" as they are using the wrong term.  They are not only not running without an operating system they have added at least one more operating system so more operating systems not zero.

Comment: if your program is an application then you wouldnt be able to get at or change those features, you get what you get.  You can certainly control alignment though, can change the code under test as well as the loop that surrounds it to see how those affect the result.

Comment: the additional bx lr moves the whole function down.  you are making a system call to get the time (kinda have to in this situation).  I probably have access to one of those armadas, might have one.  Have some armv7s maybve will play with one, will see...really need one with sram, the alwinners have some but it is super slow might offset the results.

Comment: to get you unstuck what if you pushed say r6,r7, saved lr in r6, somewhere in the function did your math on lr, pop r6 r7 then bx lr (or bx r3 or whatever you put the result in)? one extra instruction, but looking at your code you already have an extra instruction you are burning so could possibly come out in the wash...

Comment: again there are so many moving parts that you can for example add instructions and the performance doesnt change good or bad, or you can add or remove and make it better or worse.  you also have an operating system involved, etc. keep looking.  I know the arm11 mpcore has a disturbing trait (bug), that the pop {pc} is not recognized as an unconditional branch and it begins to execute (partially through the pipe) the following instruction(s) as one would expect except it will do a prefetch if the data following the pop looks like a branch, if you are seeing this that would/could

Comment: cause the pop to take longer as it has to finish this bogus prefetch cycle then start the real fetch/prefetch cycle.  to check this put some nops or other instructions after the pop {pc} to see if that affects performance.  if you are popping r7 you can do a mov r1,r7 causing a hazard preventing that move from moving through the pipe to then get discarded when the unconditional (pop pc) branch happens.  the "Ti" that you have following is 0x6954 which is an ldr r4,[r2,#20] so maybe it starts that read before the prefetch if pc.  bx lr is clearly unconditional on the other side of this.

Comment: curious to know what happens if you break it into pop {r7} and pop {pc} two separate instructions. or use ldr instead ldr r7,[sp],#4 ldr pc,[sp],#4, can have lots of fun with this one.



{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 0, 'CommentCount': 17, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2017:00:10/31/17', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<arm><armv7><thumb>', 'Title': 'ARM Procedure return performance', 'ViewCount': 106, 'id': 47040697, 'user_AboutMe': '<p>I bash buttons and things happen.</p>\n', 'user_AccountId': 10127659.0, 'user_DisplayName': 'Matteo Piano', 'user_Reputation': 392.0, 'user_Views': 27.0, 'user_id': 7482333.0}}