Where Do Asteroids Come From?