Replace those loose and gritty swingarm bearings with a DirtyBike off road rubber sealed bearing kit.
 
NOTE: Swingarm bearings can not be pressed through the swingarm. The inside bearings must be removed from the inside and the outside bearings must be removed from the outside.