To address the problem that the existing reflective grating and reading head cannot set more grid lines to further improve the stability and accuracy when the volume is small, a method for improving the displacement measurement accuracy of the reflective grating is proposed. The method corrects the grating Moire signal quality by analyzing the error of three main factors that affect the grating Moire signal, including DC, amplitude, and orthogonality deviation. The arctangent subdivision method is used to interpolate and subdivide the corrected grating Moire signal to obtain the measurement result. Meanwhile, it improves the CORDIC algorithm used in the orthogonality error correction and arctangent subdivision. Experimental results show that the subdivision error is reduced from 45″ before correction to 6″ after correction, which verifies the repeatability of the system, the consistency and hysteresis of the forward and reverse displacement. The measurement accuracy reaches ±1 μm under the same grid logarithm, which is significantly better than that of the commercial grating displacement sensor.