This is an odd train of thougth post, and I'll try to keep it brief. If you don't like odd train of thougth posts with little to no game mechanics support, feel free to stop reading now.
Scrolling through my blog list, I see this picture:
Brendan at Necropraxis. This post here has nothing to do with Pits and Perils, but the picture is important.
In old tales, in particular some of the Arthurian tales usually included with the Mabinogion, sometimes the heroes really are larger than life. Yes, the heroic characters are literally LARGER than everyone else. I don't have my Penguin edition handy because I'm at work, but I remember one story where Arthur appears some time in the future and remarks on just how small the knights have become compared to his time. View it as a "things were better in all ways during the Golden Age" sort of mindset.
What if, in games like D&D, this were also literally true? Why do adventuring types like the PCs gain experience points and get more powerful? Hardier in combat and better able to dish out damage? At higher levels, able to go toe-to-toe with dragons and giants? Could it be because they're just plain different from everyone else physically?
"Rolf the Mighty stood 10' tall if he stood an inch!"
I'm not proposing any sort of game mechanics changes related to this concept. It would all be "fluff" but it could be evocative fluff for the right sort of "mystical, mythical, fairytale" sort of game environment. And my mismatched scale collection of minis would sorta make sense finally.