Short intro: Most of these observations were tested empirically (mostly empty skillbars, skillbars with nothing but interrupts on them, etc). For the purpose of this overview, "Stacking" refers to the use of the same skill or same type of skill on more than one hero. Please don't take my opinions too seriously and use your own judgement.
I'm well aware a lot of this isn't news, but it's not covered in the hero AI thread and I get some questions about this stuff from my guildies sometimes, so it might be vaguely helpful to a newer GW player.
Let's start with the basics:
Nuke Priority:
Generally speaking, Mesmers have no set nuke priority for many direct unconditional damage spells. Giving them multiple damage nukes will not make them prioritise AoE nukes when they can hit multiple targets over single targets - especially in the case of Energy Surge. Giving a hero Spiritual Pain, Unnatural Signet, Energy Surge and Mistrust, for example, and leaving them to their own devices will make them cast the nukes in a completely random order (tested with 5 heroes with identical skillbars and statistics). Note that there are special case scenarios (e.g. Accumulated Pain) which I will cover below in an individual skill breakdown.
Interrupt Stacking
Everyone and his mother knows about heroes not prioritising interrupts. What's less often paid attention to is that heroes have absolutely no recognition for other hero behaviour with regards to interruption. What I mean by this can easily be understood if you do the following test: Bring two heroes with the same interrupt to any caster in the Nameless Isles and watch them both try to interrupt the enemy at the same time on the same cast.
The long term significance of this is that stacking interrupts on multiple heroes leads to diminishing returns. Power Drain is not usually effective energy management on 3 or more heroes and can actually be counterproductive (lose 5 energy in return for nothing!). Psychic Instability on a hero is not guaranteed to knock down enemies even if it was used every time the recharge was up. Cry of Frustration and Panic will and do cause problems when stacked unless you take care to lock targets before the battle starts. This is one reason why if you can be bothered to micro (I usually can't), locking each mesmer onto a different interrupt target is very helpful (and obviously doesn't always work so well for Cry of Frustration).
Note that it is NOT possible to circumvent this behaviour by using nonstacking hex behaviour as heroes will use interrupt hexes as hexes (Namely Power Flux and Web of Disruption) and cast them regardless of situation. Your best bet is to have some variance in terms of skill bars so that heroes don't do identical actions (although there's some occasional natural and random delay staggering in AI behaviour for normal-priority skills.)
Conditional Priority Skills:
Heroes are very good at using skills that are AI coded for higher priorities upon certain conditions being fulfilled. This is the foundation of Discordway, but also applies to things such as removal. Note that heroes DO appear to have awareness of other hero skill usage in this instance - Two heroes will not attempt to use Shatter Hex at the same time if you only have one Hex, hence Shatter Hex and Shatter Enchantment stack well.
Now that those basics are out of the way, let's discuss a few other things:
Energy Management
Thanks to the nature of Fast Casting in PvE, it is not uncommon for non-Keystone Mesmers to have Energy problems with a high FC attribute. Fast Casting speeds up the recharge of Mesmer spells, but not Signets, and as a result, spell-based energy management will scale better than Signet solutions (although Signets offer passive energy management). I'm not going to cover the Signets as it's easy to tell what is worthwhile and what isn't, but as for spells:
Generally speaking, Mesmers don't have very strong spell-related stacking-friendly Energy Management, as a lot of it seems to be balanced around the degenerate amount of energy that Power Drain/Leech Signet returns, but there are exceptions to the rule:
Waste Not, Want Not - The most popular solution after Power Drain. Stacks effectively, but provides a low energy return that doesn't scale particularly well. At 8-11 Inspiration, a pretty high value for most builds, this skill will return a net total of 6 energy every 10 or so seconds. This is generally sufficient for Domination builds, provided you can get the condition off. The easiest way to trigger the condition is through Knockdown, although Shared Burden and other snares are also effective, as is Chaos Storm.
Drain Delusions - Possibly the strongest and most reliable energy management solution available to Mesmers, this is usually shunned as it may remove "vital hexes". This argument carries rather less weight if the "vital hex" in question happens to be a Wastrel spell (no damage when removed, but Wastrel spells are very spammable), Panic, Fragility, Arcane Conundrum, one of the Burden spells, Overload or Enchanter's Conundrum. (I believe that very few people will get reasonable use out of Shrinking Armor or Phantom Pain in PvE.) Can be very effective. Do not assume Drain Delusions will remove every single hex on your enemy - Heroes will only use this when they are low on energy and will only use it on their current target.
Energy Tap/Drain: Energy Tap usually ends up returning around 6 energy every 20 seconds, but can improve with HSR procs. Less conditional than Waste Not, Want Not, but still kind of weak - not really recommended. Energy Drain is serviceable, but takes up your elite slot.
Mantra of Recall: Fortunately, Heroes will not maintain this, but it's still somewhat underwhelming without a heavy inspiration investment. Not recommended for longer battles, fine in shorter ones, not worth an elite slot.
Arcane Conundrum: This is only an energy management tool if it manages to hit more than one target. Heroes will use this on any current target of theirs, regardless of class. This is good Shatter or Drain Delusions fodder, but failing to hit more than 1 target will end up costing you energy and provides no other benefit on enemies without spells.
Inspired/Drain Enchantment, Inspired Hex: Drain Enchantment is extremely good energy management that has one big problem: It requires an enchantment to work. If you know the area you're going into has no shortage of them, use it freely. Has great synergy with Enchanter's Conundrum. Inspired Enchantment and Hex are generally not worth using as they return little energy and don't scale properly with fast casting - with the notable exception that they're stellar on Monster enchantments and hexes, but most people know this.
Guilt/Shame: Guilt is somewhat effective energy management with a high investment into Domination and Fast Casting and minimal investment into inspiration. Shame returns too little energy to be viable by itself, but the effect is useful for disrupting enemies. Both suffer from Mistrust targeting issues (see below). Reports say that Guilt is cast on any current target of the hero, not just enemies with skills that can trigger it, so be careful with this skill.
Auspicious Incantation - No, heroes have no special AI for this and are very poor at it without supervision. They will not cast it until relatively low on energy, upon which they probably don't have the energy to cast anything worthwhile. I wouldn't recommend it, YMMV.
Non-mesmer solutions include: Blood is Power/Blood Ritual, Hex Eater Signet (long recharge), Ether Signet, GoLE and Bloodsong/Siphon Spirit (optimally at 8+ Channeling but takes 2 slots and does not stack). Note that these don't scale with Fast Casting, so they're not pertinent to my discussion.
Dealing Damage:
It doesn't take an act of genius to throw Energy Surge into every single Domination Mesmer bar, but thanks to the nature of Mesmer damage your results may actually improve if you understand the nuances of particular skills and how to prepare for areas ahead. Everyone knows that like hexes don't stack, after all.
Domination Magic
Energy Surge - Very popular. Cheap, does 99 damage at 16 Domination magic to a nearby range, but scales rather less well under that as it drops to 81 damage by going to 13-14. It's practically unconditional, stacks well, and is marred only by the fact that the AI assigns it no particular high priority and often won't cast it immediately upon recharge, will not prioritise its use on clumped enemies, and is elite. From a damage viewpoint, this skill is rather mediocre - it is outdone in almost every way by Wandering Eye, which happens to do more damage, has a shorter recharge and prevents an attack. Note that heroes will use this skill even if unable to get the full effect of energy loss and damage from the target.
Spiritual Pain - A personal favourite, but slightly flawed. Does 79 unconditional damage at 16 with a very short recharge, is good for spiking down single targets and works well with Mind Wrack. Unfortunately, appears to have a conditional activation clause that makes it heavily prioritise nuking any spirits or summoned creatures in range - which may not necessarily be your intent. Makes clearing a little faster though, as killing multiple spirits is generally annoying (if you don't just ignore them).
Unnatural Signet - The AI assigns this no special higher priority even if the target does happen to be hexed or enchanted. Useful for spell-immunity.
Wastrel's Worry/Demise - If your target happens to have Natural Resistance, Wastrel's Worry is one of the highest dps nuking spells in the game. Demise does really good DPS, but both require extra effort (usually Panic or PI, but Mistrust/Guilt Chains work too) to get the most out of. More suitable for players, but the damage can be impressive.
Mistrust - Huge damage and flashy, but heroes will only hex their current target with this. If you intend to stack this effectively, micro your heroes by locking separate targets before the battle starts as heroes will cast this on any class with the caster designation regardless of whether it has spells that will trigger this or not. It will not cast this spell on non-casters (Rangers, Warriors, Paragons). Note: Most targets in the Isle of the Nameless, including dummies, are considered casters for some strange reason. You may confirm this on the two Guards of the Isles, which are not considered casters.
Note that Mistrust can stack with Guilt and when the spell fails, both effects will trigger. Also note that Mistrust and similiar spells (Guilt/Shame) will also activate instantly if an enemy begins casting the right type of spell while under the effect of the hex, hence they work properly with Panic.
Aneurysm - Heroes have knowledge of the amount of energy your enemies have and will not cast this unless your enemy is nearly out of energy. Whether you trust them enough to use this skill without Arcane Languor is up to you.
Overload - Due to its nature of being a Hex, this will not stack, therefore multiple copies are not useful without micro. Has a high priority conditional usage similar to that of an interrupt (Heroes will try to get the additional effect), but very low priority otherwise. Great synergy with
Shatter Delusions - Very good AoE damage on a very fast recharge, works incredibly well with Overload, but suffers from the same considerations as Drain Delusions. Stacks excellently if you are not paranoid about hex removal. Note that heroes will only use Shatter Delusions on their current target and will not manically remove every Mesmer hex in range. This may or may not be a good thing.
Enchanter's Conundrum - Extremely underused skill, often overshadowed by Energy Surge. Costs more energy, smaller AoE, conditional damage (target must not be enchanted), not so hot on Dervishes, so some target consideration is required. In return, you get more damage per activation than Energy Surge combined with a very short recharge time.
Since this is a hex, heroes will not overwrite it, and with fast casting, it will recharge faster than its duration. Obviously, this just means you should be combining it with Shatter Delusions and Drain Delusions to counter the higher energy cost and create an even larger rate of damage.
Power Spike - Very strong spike damage on an interrupt with a very short recharge. Problem: It's an interrupt and doesn't stack well.
Shatter Enchantment - Somewhat superior to the necromancer enchantment removals (thanks to having a shorter recharge due to FC), but energy hungry. The main issue is that it has strong competition in the form of Drain Enchantment and Mirror of Disenchantment (which is very strong, but not a damage skill, so I won't discuss that here). Stacks well.
Shatter Hex - Very powerful - almost too powerful now, but is still energy hungry, conditional, and PBAoE on a target ally. Does very high damage in a large radius, while removing a hex and has a short recharge. One of the best damage spells if you're using a tanking approach.
Empathy/Backfire/VoR - Action hate. Not particularly energy friendly, but have the potential to do far more damage than any other Domination spell by themselves. Nearly every single target Domination nuke should to be compared to these spells unless you intend to spike down the enemy (although Empathy on certain creatures constitutes a heavy spike).
Chaos Storm - Very energy friendly, causes scatter. Note that the AI will not use this on anything except casters with wands and staves and will not target packs to AoE with this. As a result, it must be microed. Again, note that this is actually a good way of fulfilling the condition for Waste Not, Want Not.
Mind Wrack - Before putting in Mind Wrack, ask yourself if your team is capable of reducing an enemy's energy to zero (with say, stacked energy surges). Due to this being single target and only adding about 29 damage to each non-hex skill, this is not usually worth the energy otherwise. (87 extra damage and 3 energy loss on using a Energy Surge > Spiritual Pain > Unnatural Signet chain isn't too impressive)
Illusion Magic
Accumulated Pain - Has Conditional behaviour. If the enemy has two hexes on it, this will be assigned a very high priority and has low priority otherwise. Inflicts damage similar to that of Spiritual Pain, but comes with Deep Wound on the conditional, which is very very useful on spikes.
Calculated Risk - The downside of this spell is usually negligible, but the damage effect isn't reliable. Not particularly stellar for damage as it is somewhat pointless on casters. Has a long duration and generally will not drain your Energy as much as
Signet of Clumsiness - Very small damage on a relatively short recharge with a few side effects. Not really essential, but it helps with the energy passively if you're using Clumsiness. Heroes will not try to get the conditional effect, but will generally use it effectively.
Wandering Eye - Stellar damage and range, cost and recharge. The only real problem with Wandering Eye is that it's reliant on a 4 second window to trigger, which is usually reliable, but not always (Enemy creatures with spammable spells won't attack often). The sine qua non of the Illusion Magic nuker.
Degen Hexes - Since Illusion Magic has no AoE degen hexes and most of these cost 10 or more energy on quite a short recharge, these are usually not a good idea if you value your energy. Shrinking Armor and Phantom Pain are somewhat reasonable in terms of energy cost, but don't do very much damage (80 damage over 10 seconds) - Accumulated Pain would do about twice that damage in the same amount of time.
Images of Remorse - The 2 second cast time makes this a bad joke. Heroes don't specifically aim for the added damage - there are better feeder hexes for Shatter Delusions.
Ineptitude - Popular with a strong effect. Good, but not quintessential, as it competes with other strong Illusion and FC elites. The 10 second Blind doesn't scale, but if you value the Blind that much, a Blinding Surge elementalist would be more efficient.
Frustration - Has a very long duration at high Illusion magic levels and can potentially deal 100+ damage on top of every interrupt you do, making the damage output obscene. Heroes have no particular priority for Frustration and will generally hex their current target with it. Note that the damage will only trigger on actual interruption - despite the description of certain skills, just randomly tossing out Web of Disruption or Leech Signet will not interrupt the target.
Fragility - The problem with Fragility as a damage source is that it needs to be built around (usually with fevered dreams and short duration conditions or enemy condition removal). Heroes don't prioritise targets hexed with Fragility nor Fevered Dreams with conditions, so this works better on players. In any case, Fragility is capable of causing surprising amounts of spike damage, so it should not be completely discounted. Works great as a support skill with Dervishes or Incendiary Arrows.
Further notes on the nature of interrupts
Mesmers are for interrupting. This is one of their fortes - however, have you ever wondered why your heroes love letting random Meteor Showers slip through the cracks?
There are two default settings for AI. "In combat" and "Out of combat". It's easy to tell when you're in combat as heroes will start autocasting skills that aren't usually maintained, such as Dervish Avatars and glyphs.
When a hero is out of combat, it will interrupt skills when set on Defensive mode, only provided the caster is in range of the Mesmer and the skill in question targets an ally. This means it will not interrupt preparations, enchantments, forms and so on.
When in combat, the hero will attempt to interrupt most skills that the enemy uses (with occasional random lapses) provided the enemy is the mesmer's current target. In combat, your mesmer always has a current target, even if it is not locked. If Enemy A and Enemy B are casting meteor shower on a mesmer who is currently nuking or wanding enemy C, the mesmer will NOT interrupt enemy A nor enemy B. Note that calling targets will lock every nonassigned mesmer onto the called target.
Interrupt choice: Similiar to nuking, when presented a choice between multiple options, Heroes will use viable interrupts completely at random.
Special cases for AI Interrupts:Power Return: Very short recharge time, the only reason I mention this is because it makes for a good choice if you're worried about the diminishing returns from interrupt stacking harming your ability to interrupt. Leech Signet: Heroes will only use this to interrupt spells and will never use it to interrupt non-spell skills. Signet of Disruption: Heroes will use this correctly if target is hexed Web of Disruption and Power Flux: Heroes use these as if they were normal hexes and will not attempt to interrupt foes with them. Mistrust and friends: In addition to what has been said before, note that Mistrust-type spells constitute skill failure and not interruption. This means they will not put the skill on its natural cooldown, but they can get around interrupt prevention such as Mantra of Concentration. Complicate: Note that due to the above behaviour, if your heroes do not blow Complicate simultaneously on the same skill, Complicate stacking tends to severely maim enemy groups when you call targets, since a single Complicate-ed monster will rotate through its available skills. Most enemy skillbars are partially or completely nonfunctional when 1 or 2 of their skills have been locked.