To recover image quality from images with rain, a rain removal algorithm for sea surface images is proposed, which is based on the residual block network. The algorithm combines two types of residual block networks for extracting deep-level information of images with rain. During the training process, the residuals between the image with rain and the original image are learned. In this way, the target value domain of the image operated by the algorithm is reduced and the sparsity is enhanced. For the training dataset, we use the outdoor rain image dataset and the sea surface rain image dataset simulated by two rain addition algorithms to expand the training samples. For the test images, three different types of rain scene images are selected, and the types of rain include rain lines and raindrops. Experimental results show that the proposed derain algorithm can be applied to different rain scenes. The signal-to-noise ratio evaluation index of the images after derain processing is above 35, and the structural similarity is above 0. 97. The clarity of the images is generally improved.