That's why I don't say anything. Than people get sad because they think I'm ignoring them and they're problems.

I can see people getting irritated about it, but I doubt they hate you. Just try to be understanding. It's okay to help people be happy, but sometimes it's best not to say anything.