Board Thread:General Discussion/@comment-24670104-20161112191552/@comment-25521424-20161112192119

Asgardians didn't know where the Time Stone was in the universe/multiverse. About the Scepter: he was in the custody of S.H.I.E.L.D., and Thor wouldn't have took from them, but after they retrieved it from the end of HYDRA, he was supposed to bring it back to Asgard, and possibly find it a safe place. The Aether was on Svartaflehim and was reached by Jane Foster through the Convergence.

I would disagree on Earth being safe from being the location of four Infinity Stones, since it seems to be the focus all the disgraces in the universe. XD