I honestly thought that Xion didn't get Ventus' memories, just that the heavy connection to Sora (and I do mean heavy, considering the kid's Heart came into contact with his before he was born) made those who knew Ventus able to picture him in appearance. Remember, Xion's a mirror that reflects what the recipient sees.
Basically, it goes with the quote above. Xion's a doll/clone/puppet of Sora conceived of memories (Sora's memories, to be exact), and the appearances it takes depends on the person's connection to said Sora's memories (or to extension, even Kairi). Therefore, people who never met Sora see the hooded Xion (which is the "IT" form; they see nothing but an androgynous shape, despite the form it takes to the audience), people who are bonded to Kairi and are bonded to Sora see the black-haired Kairi, people who are bonded to Ventus, see Ventus, etc. That explains why one person sees Xion hooded, and the other person at the same time sees a face. Nothing but a mirror of illusions.
Deep inside, it's all crystals (which, technically for SE, represents memories). Trippy what the body's made of, isn't it?