It kind of depends. If you selece the defence weapon in the beginning of the game, you'll have an extra defence when you finish the game. The same goes with your oter abilities if you pick the attack weapon or the magic weapon. And It depends on if you've found all the Defence Boosts, Attack Boosts, etc.
But I would raise my Magic if I were you. You need magic more than anything when fighting secret bosses because you need to use reflect, but for normal bosses, attack is good, but what's 1 extra attack going to do? And you could wipe them out in a second.