this dog's mental health is obviously in shambles--i tried to cut his nails again?

[MommytoaLittleBug]

my brother's dog is very strange, unlike any dog i've ever seen. he is very timid. his nails are very long and it looks painful! so i had my boyfriend hold him down while i clipped 3 of them for a start. he was whining trying to bite me and i'm surprised he didn't pee all over. when it was over my brother tried to come and get him and he even tried to bite him. is it right to hold him down and clip his nails? or should he be left alone? i feel terrible that this poor dog is so screwed up. his mental health deteriorated after my 13 year old brother got him when he was 2 years old. he's never been abused (mom is friends with his old owner) and my brother treats him like his big baby. i figured some info about him would help with my question as to whether or not it is kind to hold him down and clip his nails. brother can't do it and neither can our mom. it looks like he's being neglected to me.