Are you using your fires in sleights? Those get you more powerful spells. You also need cure cards.
Cure cards are what keep you from dying from those pracitcally unavoidable hits he manages to get on you. You have got to use them. I recommend at least 3.
Also, make sure that you're going at him from the back. Everything else is guarded. If you have a Fire Bandit enemy card, this would be the time to use it. They give you extra power for attacking if you attack the enemy from behind, which is what you have to do to do any damage to Vexen.