So far I have not had this issue, but I believe it is a known one. You won't be able to clean/remove it as it is the plastic starting to disintegrate. You solutions are to one- replace the part or two- wrap it. If you wrap it you could sand or try harsher chemicals to get it smoother before placing the wrap.
The good part is that a brand new piece will only set you back about 40 bucks(unless you have extended leather - that piece is a ridiculous $550 bucks)....
http://www.realoem.com/bmw/enUS/part...667&series=E85