I've felt the same. On top of saying no, I need to explain, maybe even apologize. I learned it doesn't matter. Whoever I say no to is going to be disappointed regardless. Nevertheless, no is an essential part of all relationships. It has to be respected.