Problem to link to libsecp256k1 on ARM architecture
It seems like pkg-config found libsecp256k1 in the build-libbitcoin directory. which is part of libbitcoin, so they probably build libsecp25k61 there.

The reason your code doesn't compile is because your libsecp was compiled with libgmp enabled, now this is a dynamic link. 
So if you're trying to create a binary here you need to add -lgmp and if it's not in your system libs you'll need to also give a location to the .so file 

Another option would be to recompile libsecp with libgmp linking disabled.