I think the issue here is the loose definition of the word "ban". Ban what? Banning in general? Specific types of banning? We ban nazis on here sometimes. That's a form of banning.
If you are talking about things like banning cigs and alcohol, then, by all mean, of course. No government agency should try to dictate people's rights on these types of issues. Though I've heard the argument that banning these items protects children who are innocent - even though I disagree because it goes against the rights of parenting. Still, it's an example of how loosely the word "ban" can be used and how the libertarians have a good argument against them on their hands. We can make almost any ban out to be "banning something dangerous to others".
So is banning crack or heroin right or wrong?
If we are talking about not banning islam which is nothing more than a dealth cult, then I disagree. We should ban islam. Obviously, I'm a little biased there.