The Beast did mention Xaldin during the first visit, didn't he? You beat the crud out of the Beast, then he comes back to his senses and starts snarling about how Xaldin was egging him on. That particular visit ended with a battle against a big Heartless in the ballroom.
This is one of those reasons why I really like the KH depiction of Goofy. In most media I have seen him in, "goofy" usually meant "dull, incompetent, and clumsy." Here, he's probably the most levelheaded and sane out of the trio.