Well, if Kairi, the apprentices, and half the Organization all came from Agrabah, if Aqua disappeared there, Ansem the Wise ruled there, Roxas and Namine were revived there, and SDG's base level of operation with the FF characters had been there, then, yes, I'd be arguing it should've been in 3. As it is, Agrabah is just one random Disney level, so, yes, that would be stupid.
So, according to your argument, if Sora just randomly disappeared and was completely absent from 3, and Riku or Mickey became the main character, we couldn't criticize the game for lacking the main character or failing to resolve his plotline because we can't criticize it for what's not there, only what is there. Or if D&G were killed offscreen and never appeared as party members again with nothing filling their absence, we could never criticize that or say the game is hollow without them. That's a pretty insane line of defense you're making.