The crowd anomaly detection is an important research direction under intelligent crowd surveillance technology. In the existing methods, the first step of anomaly detection is to obtain motion information. The traditional way of uniform chunking of video frames does not guarantee the integrity of pedestrians, and the extracted features do not accurately reflect the motion status of pedestrians. In this article, an incremental crowd grouping method is proposed, which firstly combines crowd motion field with frame difference method to segment the image to obtain crowd foreground. Then, the direction group is achieved, which is based on crowd motion direction. Finally, Spatio-temporal information is combined to re-cluster the direction group to get a more detailed pedestrian group. For each pedestrian group, the crowd energy feature is used to characterize the overall pedestrian motion information, and the ring block energy histogram feature is constructed based on the energy field to weaken the effect of pedestrian limb swing, and finally combined with the image appearance features for crowd anomaly detection. Experimental results show that the proposed method achieves 83% and 92% accuracy at frame level and 64% and 83% accuracy at the pixel level in two different scenes, which is a significant improvement compared to the traditional method.