I guess you could smooth out the castings if you wanted too but why not just paint them?
How many folks to you anticipate getting on their hands and knees to peer thought your rims to exclaim, "Look Lorene, he ground the high spot off his calipers and smoothed them off!"