Agreed mis matching armor should make it not fit correctly IMO.
Though I'd see Combat armor as combat armor, and power armor as power armor.
I'd have B-Keeper, and Police uniforms with debuffs with all helmets.
Though rather than reduce protection it could simply increase the chance of getting critical hit, and higher critical damage. Due to not fitting properly.