A world that's large enough and drifts close enough could even kick us out of the solar system entirely. (Or cause us to collide with a nearby planet, like Venus or Mercury.)