This does seem to be an issue. I have come across this a few times. I am distinctly average, and also get, oh you are Big! Or, too broad, etc. It seems to be a bit of a mindset in Japan, perhaps to avoid bottoming, though I have always managed to have fun there!