Am afraid of driving scooter or car

I am afraid of driving a scooter or car for long duration, feel like jumping off and start to panic whenever i have to go out of city. What is wrong with me? As a child i once fell off from the bicycle, so could this be reason.