Ugh, I hate it when they hide behind a corner or underneath a table and then reach out and grab your ankle while you walk by. D: I'm surprised I don't have scars from the buggers. Although, yes, it is forgivable for the fact it is cute.
I think people only consider certain areas 'scary' because they have little to no experience with that kind of environment themselves. For example, I've grown up with loads of trees/plant-life around me for a majority of my childhood. When I moved to El Paso, Texas, it was hell for me for the first couple months because of how empty it was. After that, I adjusted and just got used to it [although, I wouldn't want to go there again due to bad memories but, that's a different story].
It mainly has to do with what you were used to as a kid, because those are the things you grow up associating with 'home' and security.