Incremental optical encoders have grating line errors and subdivision errors inevitably during the manufacturing and installation. These errors reduce the accuracy of the angle measurement of encoders and cause instantaneous angular speed (IAS) signal fluctuations. Research on the compensation method for grating line error and subdivision error has great significance. However, the existing methods have limitations, such as low efficiency of error compensation and difficulty in field deployment. To solve the above problems, this article first analyzes the grating line error and the subdivision error of the incremental optical encoder and establishes the error model to reveal the relationship among the grating line error, the subdivision error and the IAS signal fluctuation. Consequently, a method to compensate for the grating line error and the subdivision error of incremental optical encoders using the IAS signal is proposed. The method has advantages of high efficiency and no need to modify the encoder. The correctness of the error model established in this article and the feasibility of the error estimation method are evaluated by simulation analysis. The grating line error and the subdivision error of the incremental optical encoder at the end of the servo motor are compensated on the RV transmission test rig. Finally, the error of the incremental optical encoder is measured by the optical rotating platform. The effectiveness of the proposed method is evaluated by comparative analysis.