According to the simple reading of the Rambam it is irrelevant as to the intent of touching as long as it is not a sexual kind of touching. According to R' Mose Fienstein Zs'l in his letters about touching passers by in a crowded subway train he seems to argue (or at best reads the Rambam differently) and says that touching is allowed only if one doesn't have pleasure from the touching.
Odkahanehchai: Just because people say something is true, does not make it true. Most Jews have no clue what the Rambam writes about this subject or even what the strict Moshe Feinstein writes about it, even he allows it if it isn't with wrong intentions. There are many misconceptions of the general public.