I believe that Light and Dark are not Elemental damage. Chaos is also neither Elemental nor Physical. Shadow is, I think, another name for Dark, and is therefore also not Elemental nor Physical.
Not all damage types fall into the the two categories of Physical and Elemental. It's just a label. In theory, there could someday be a damage type that was both Physical and Elemental. (I've often thought that Earth should count as both. =)
Holy and shadow damages are two that are considered neither elemental nor physical. So only base armor lvl is considered when you use these two damages.