I know it wasn't the Avengers' fault, but the public didn't see it that way. In fact, Ultron was Tony's fault, he built the robot. I know he had good intentions, but it was still his fault. I don't even agree with the Accords, I just understand that the intention was to save human lives.
I'm not sure what Ross had to do with it. He was in the movie but he played a very small part. Your issues with him shouldn't affect this decision.