Personally, I feel that Xemnas might have memories of using a keyblade, which may of became slightly ajar thanks to the transformation, but since his somebody couldn't remember when he was Apprentice Xehanort, he can't wield. Roxas, on the other hand, had Sora and he definitely remembered wielding before becoming a nobody. Yeah, I feel that some of Xemnas of memories of pre-Terranort might of involved the keyblade and maybe even him or at least two people he felt he knew wielding it, but it may or may not been him in his eyes.. yet he had blurry memories of his friends Aqua and Ven. In other words, he might not remember wielding it directly (since he's two people at once), but he's knows people in the past that did, possibly including the very two people that are a part of him.
Yeah.. that's a bit confusing..