Difficult to discuss without you telling us all the nitty gritty biz that went on between you, which prolly isn't a good idea here.
All I can tell you is my own situation. (my kids are 19 and 15 now).
I am/was the absent parent - very much like your situation, saw the kids almost daily, they stayed with me regularly etc.
The thing is, when we split up (amicably) we went to court (we were married still) and had what was called, if I recall correctly, a Residency Order - We both already agreed the kids would live with their dad - the judge agreed and gave us Equal Parenting Rights.
It meant that the kids dad could not take the kids abroad, move house, change schools etc. without my agreement - something we've never, ever had to actually refer to cuz we get on well and respect each other's views on stuff involving our boys.
We did it mainly cuz our eldest boy had huge psychological issues/high functioning autism and being aged 7 at the time he needed to know/understand what was going on in his life/our lives.
Maybe that's something you could look into?
Good luck.