Would you be angry or am i "being silly"?

If he's fighting with you, and was being mean, I think you're right to be annoyed. He's being a bit cruel by ignoring you. Wait a few weeks and if he keeps being mean then confront him. Tell him to be honest as well. If he's mean about that, you may consider if he truly loves you or not. I hope this helps.