The fibers are extremely sharp and worm their way through lung tissue throughout a person's life, and there is no way to get rid of them (they last forever). They cause massive scarring (asbestosis) and the lung tissue, under this extreme attack, turns to a radical solution to fight its tormentor by becoming malignant.
I am not 100% sure though whether the asbestos particles themselves do all the damage, or if it is an immune overreaction to them.