Agree!
We were in the very awkward situation where we turned up on the day and there was a builder doing some work in the garden. The sit was actually a repeat housesit and one we liked, but the homeowner had not told us prior to the sit and just assumed it would be fine. By that point it was too late for us to say anything as they were literally leaving as we arrived.
The only way the builder could get to the garden was through the house (it wasn’t a big house) which meant we felt quite uncomfortable being in the living room watching TV with a builder coming back and forth. Bear in mind this was also late October last year when the virus was pretty bad - we were not comfortable with other people being around us. Luckily the sit was only a few days, but he was there pretty much the whole time.
Like you say - we felt the obligation to stay at the property whilst he was there. Not only because we had to let him in etc, but also we tend to travel with some expensive stuff and even if the HO decides they can trust this person - we have no idea who they are.