Truth is, maybe you are annoying in that person’s eyes. Either way, the way they said it was quite rude and they could’ve told you in a nicer way. Best to just not dwell on it for too long.