It is a valid point. There are rabbis who forbid entering the temple mount altogether. Rishonim haven't forbade it, some of them probably went up there themselves. Achronim haven't forbade it as well. The muslims are the ones who forbade infidels from entering the temple mount. Only later in 1853 when it reopened for western visitors due to European pressure, the local rabbis decided to shut it down for Jews. Nevertheless halachicly it is valid to say Jews shouldn't enter the temple mount.
But, (and this is what the haredi world ignores and misleads people on)
it is also halachically valid to say Jews can enter the Temple Mount as there are prominent rabbanim who permit it. More than that, new information has come to light since the time the "ban" was first issued, where we now know much more about the exact locations on the Mount and what areas are truly forbidden to access.
This is of course not to say that those rabbanim would today change their ruling based on newly unearthed facts (because I think it's politically charged issue and they would find a way to ignore the new facts - for many reasons - to still ban it), but it gives quite a bit of credence to those rabbanim ruling otherwise. Halacha does not go by "gedolim" or fiat or majority despite what know-nothings may claim about it.
Halacha going by "majority" refers to when the Sanhedrin decided a case (we do not have one), and it was a Talmudic dictum sometimes employed for deciding a sugiya. It has nothing to do with post-talmudic and aharonic/present-day generating of psak.