Geomancer's Test

I activated each of the earth altars by touching them with scorched lodestones in my inventory, but nothing happens. I circled around and hit each one again, but still no large stone golem. Is there a trick?

You probably killed him without noticing, or else you missed an altar. Check your log. It'll tell you whether or not you have enough lodestones on the altars. You may have also run out of Lodestones before you got to the last altar.

He doesn't pop up right in your face, you have to go to him (he's in the area). I seem to recall that it does give you an arrow/start pointing to his location. Does it not?

I think the only trick is in spotting him. He doesn't make any sort of grand entrance or anything, just spawns into the area somewhere and begins to wander. At least, that was my experience.

I did the same as you -- circled the altars a couple of times to make sure I hadn't skipped any, etc. It wasn't until I was about to give up and head somewhere else that I noticed a faint glimmer of coloured aura waayy off in the distance, which turned out to be him.

I was expecting something along the lines of the bosses in Serious Sam or something, but he's no bigger than any of the other golems around there as far as I could tell.. ('course everything I could have used for comparison was already good and dead by that time, so YMMV)

Yeah it does since I've recently done this quest - you have to (well I did) allot of Stone Elementals or Hulking El's before able to get the lodestones necessary, but there are 3 alters so you need 3 stones.
Then just a little ways to the left (which ever direction that is on the map) you'll find the target. *when I say left, the means you're facing the last alter (the one to the left of where you enter the Larger Crystal area) which means your as well facing the Large Crystal