Our Duty to Repair the 2nd Temple

[edu]

During the time that the 2nd Temple existed if one of the walls of that Temple had fallen down, we would be obligated to repair it immediately. Rabbi Avigdor Neventzal commented that in our times when all 4 walls of the Temple have fallen down, we still have the obligation to immediately repair the Temple. The obligation to repair the Temple is not eliminated if the Temple was destroyed.

Rabbi Yosef Elbaum also made a similar point. He said, if right after the 2nd Temple was destroyed, if the fortunes of war had changed and the Jews had ended up winning the war against Rome, there is no question that the Jews would have started rebuilding the Temple immediately without waiting for miracles to restore the Temple. The fact that so much time has gone by since the destruction of the Temple does not negate our obligations. When we have the ability to repair the 2nd Temple we must repair it.

[edu]

If Rabbi Neventzal in theory supports the repair of the 2nd Temple, why has he been opposed to prayer by religious Jews in the outer parts of the Temple Mount, even though they immersed in a mikva and  don't wear leather shoes on the Temple Mount?
A possible answer is what I saw quoted in the name of Rabbi Neventzal at
http://www.vilnagaon.org/mount/letter4.html#11

F] The ascent of citizens who immersed in a mikva and removed their shoes, causes the ascent of policemen that did not immerse, nor remove their shoes. This is not nearing the redemption, rather it is heaven forbid, a reason to exile us from our land, as is the desire of Sharon and Abu Mazen.
    The Youthful of the sons of Levi
    Avigdor Neventzal

At this point in time I just wish to present Rabbi Neventzal's views without presenting potential criticisms.