The negative hit to your affinity would be because you didn't even bother to help, which is an immoral action. Trying to help, even if you fail, is still a moral positive action. At least you tried to do the right thing and help, instead of turning a blind eye and going on with your day as if nothing had happened.
Now the question really should be will the positive effect be awarded the same whether or not you succeed when you tried to help.