What you are talking about is 'spherical perspective'. It's weird, unintuitive and, quite frankly, a royal pain to work with.
The perspective grid for a full 360 cyclorama looks like this:
Most of the lines you would think would be straight are actually curved. It's possible to work it all out manually, but this is something where some rough 3D geo and a panoramic camera would save a lot of headaches.
If you don't want the 'fisheye' effect, then you could cheat the image, and keep the lines straight, with lots of different vanishing points. - but this will look exactly like a collage, since that's what it will be.