What is This White PVC Pipe in My Yard?
They can be quite hard to remove even if not glued.  I'd assume it is a cleanout.  

Remove the top (I usually use a hammer and a piece of wood to bang under the lip--you'll have to dig a bit).  If the top does not come off, just cut it off with a reciprocating saw. You can always put another cap on or get a coupling if you need more length.  Once the cap is off you can yell into it and have someone listen to your rain drain risers to see if it is connected.

If it smells or you can't hear the other person it could be a sewer cleanout.

You could keep it right at the level below the flagstone and have a removable flagstone if you want to keep the cleanout some what accessible.