I know exactly what you mean. When Kodi was being potty trained, I couldn't trust ANYONE in the family to really keep track of things. The only solution was to have them keep him in his ex-pen at all times, which, of course, you can't do for a whole weekend. Fortunately, I have a good friend who is also a dog trainer, so when I had to be away from home, even over night, I left him with her. SHE made sure he got out often enough that there were no accidents and no set-back in his training. (in fact, I think it helped, because he just followed along and did what all the "big dogs" did
Now that he is an adult and fully trained, my husband can handle getting him out when he needs to go with no problem. But I STILL can't trust my 19 year old son to get him out when he needs to go. And the problem is, that he will always pee in his box if he can't go outside, but really HATES to use the box to poop. If you want him to use the box (because it's awful outside) you have to stand with him and remind him to "go potty" until he finally gives in. My son will either take him outside but not give him long enough to go, or will tell him to use the box, but not not really insist.
The result has been that I've come home to "presents" a couple of times when I've left my son in charge. As a result, I just don't leave him in charge of Kodi. If Kodi can't be with me or or Dave, and needs to be left for more than 8 hours or so, I leave him with a friend that I CAN rely on! When he's home with my son, if my son isn't actively playing with him, he knows that Kodi has to be in his expen (with litter box), just as he would be if no one were home. Kodi has adjusted fine to this, and is no longer put in a position where he feels that he has no choice but to make a mistake.
My older son can be trusted, but he's away at school, and when he's home, working full time, so he's just not around much.