Set over set… Fold or have heart?

So I was playing in a 1-2 no limit game and playing against a loose aggressive player for about two hours before I got pocket queens and he was opening with something like JJ QQ KK AA or AK. By the flop we were heads up and the flop was Q 4 8 rainbow. �� I checked my set, he bet 1/2 pot, and I called. The turn was an A. I bet 2/3 pot, he instantly goes all in I quickly and gleefully called with my stack of $500 and �� that he had anything other than AA. But of course he had turned a set over my set. ��*♀️ I feel like I could’ve spent more time reading him and thinking through the different hands he might have, but was I supposed to fold?