You are not even allowed to shake a woman's hand unless it is for parnasah. But shaking hands on a date would certainly be immodest. The rabbis that made Rabbinic legislation on this (I'm not sure if it's Rabbinic. It might be D'Oraita.) knew how a man feels when he touches an attractive woman.
A single man that has never been with a woman and then touches a woman such as holding hands or hugging will get aroused. That's just how men are. I don't know about women. Even if he is not tempted to do other things with her, he will have a good feeling in his body just be touching a woman, even if it's not touching her breasts or sexual organs.
I get a good feeling when I win something, when I pat a cat, and when my mom used to kiss me on the head, there was also a good feeling. Shaking a hand does not bring up any sexual feelings in me. If I sit there and hold a girl's hand, it's extremely limited, but maybe slightly. Kissing a girl's hand is a polite thing that is a sign of respect, and while I can see how someone who has never touched a woman in his life will consider that sexual, it really isn't, and most of the sexuality involved with that is being created by his brain, not by the action. If you don't have the spiritual level to get through a handshake without drooling, I wouldn't recommend it, maybe start with pats on the back or something more neutral. If you do, there is nothing wrong as per Torah.