Don’t know what to do for the best

I have a 4 year old with my ex partner I allow access every weekend for 1 night however of a late it’s been all weekend because he’s missed a lot of weeks due to drink and drugs and me not allowing access because of this.. his ways caused a lot of heartache and real suffering for me when we were together I put up with a lot out of love.. giving chances believing his words but he was verbal and emotional abuse when Id call him out on his ways .. fast forward to now his 14 year old son has recently for the first time disrespected me .. body shamed me and hoped I’d die of cancer. . I have been there for my ex’s son and it has saddened me and shocked me. The abuse from his father is having an affect on me and I am not even with him and his older son is telling our young son he doesn’t like me .. he drinks smokes weed and swears all the time whilst my young son is in his dads care .. also my ex has outbursts of anger like if something doesn’t go right or his way , whilst our son is present .. I don’t want my son picking up on any of there bad habits and poor behaviour also just being in that negative environment makes me worry so much

Do I limit the time or stop contact all together I’m worried because someone also told me the first 7years of a child’s life are what makes them who they are as an adult .. that they are like sponges and take everything in .. also the memory starts around 5 we’re they begin to remember things I’m absolutely terrified of my young son learning the wrong way and all my hard work is going to be undone .. what should I do?