I would start weaning at 2 and maybe you and him could switch things up to make you more comfortable.
Like instead of 7 days on 7 days off and switching on Sunday's. What if you have every Monday Tuesday
He has every Wednesday Thursday and then You alternate Friday saturday Sunday
That way the longest stretch you go with out seeing them is 5 days (when it is his weekend)
7 days is a long time for even a 4 yr olds
My DSD is 13 and we have her Friday morning (she has no school on Fridays) until Sunday night every week. So we only go about 4 days with out seeing her.