You only lose the blue abilities associated with that DE if you decide to change party members. Red and Purple abilities stay permanently. Blue= Stats, Red= Support, Purple = Spirits.
I'm also quite surprised you beat the game without caring for your spirits at all lol.