The Google Phone app could give you a quick way clean up

Googles Phone app could automatically delete your Call Notes. This joins the existing option to automatically delete call recordings. The Call Notes feature is restricted to Pixel phones in the US right now.