Magnetic soft materials such as magnetic gels or elastomers demonstrate drastic changes in the viscoelasticity synchronizing with the application of magnetic fields. The change in the viscoelasticity for magnetic soft materials is greatly enhanced by an addition of nonmagnetic particles causing by the stress transfer among imperfect chains of magnetic particles via nonmagnetic one. In this review, we survey the magnetic response of viscoelasticity for bimodal magnetic elastomers containing nonmagnetic particles, and the mechanism of the enhanced magnetic response has been explained.