In my opinion, he was afraid that Asgard wouldn't accept him because he was a Frost Giant. He clearly didn't want to reveal that, Loki himself didn't knew until he held the Casket.