The only thing I could think of for this situation is:
Anything decided as an accidental hit - "Oops! you hit this guy instead of that guy" takes a damage drop for caravan members, so first, they get hurt for less damage (seems unrealistic but just hear me out) and second, they have a set hit point threshold, if the damage you do to any of the members exceeds that limit, they take you as hostile and let fly, say if you do more than 50hp of accidental damage, they flip their collective shit.
Not sure what it'd take to implement this, and I'm not saying this NEEDS to be put in, it's just a suggestion. Personally I just try to not hit them and be really sodding careful.