No, multiple hexes of the same kind do not stack as the lesser of the two will be overwritten by the more powerful one. Multiple different hexes do stack however.
As far as I know, Spectral Agony does 10 degen in addition to a number of damage every second (100 or so?). It definitely doesn't give 300 pips of health degen.
10 is the max but more will apply if you try to use regen. many times ive had so many hexes/conditions on me that i will cast healing breeze and still have 10 degen