Rambam's source according to Kesef Mishna is the Talmud Sanhedrin, page 75a.
There the Talmud brings a debate is this halacha applicable only when the woman is married or even if she is single. Rambam rules even when the woman is single.
The Talmud in Sanhedrin 75a also states there explicitly, that we don't save this guy, even if he only wants to look at her without clothing, without actually touching her. And then ends off with an even more extreme statement, namely, what the Rambam quoted that we say let the guy die even if he only wants to talk with her behind a fence.