By clearing the contact storage , it seems to help.
But the symptoms comes back after a while.
This is a indian stock samsung galaxy note 4 Marshmallow firmware N910GDTU1DPD4 on a N910G

I have also stop google sync with google people