I suspect that's related to taking things - he doesn't have a strong sense of ownership or permanence yet. What's "his" is what is in his hand, what he's using right now. There's a degree to which that kind of assumption is cultural and situation-dependent, which makes figuring out the messy territory surrounding usage and ownership complex.
In addition to what others have suggested, it might help for you to try and keep up with the things he has for awhile and mislays - that will allow you to "regift" him with the same things over and over.
It could also help for you to give him small things when you go visit other homes - either something to have for the visit or something to trade for whatever he picks up while there.
"this belongs to x, it needs to stay here"
It could be helpful to anthropomorphize a bit and say "this lives here, it wants to be here". That's going to depend on the way he thinks about objects - some kids have this sort of global sense that things are possessed of personalities (kind of a like a kid version of animism), but not all. My daughter never liked that kind of talk, but it made perfect sense to my stepson when he was little.