It is hard to say which doctor is right without a physical exam. Most people do have some degree of septal deviation. The firmness of the response
from the PCP might be a bias against surgery, mainly because he or she might not do the surgery, so they don't really understand whether or not surgery is indicated.
Sometimes the difficulty in breathing is due to turbinate hypertrophy, and not a septal problem at all. Other causes of breathing problems can be a long nose, so the angle of entry of air is an awkward one, or a previous nasal fracture which has displaced the septum out of the vomerine groove.
PCPs generally don't do surgery, and ENTs do. We tend to recommend what we specialize in. So the deciding factor for you--surgery or not--should be the degree to which this bothers you.
Do you mouth breathe a lot? Can you exercise and still breathe comfortably through your nose? Do you have a hard time getting air through your nose, even if you don't have a cold? If so, surgery may permenently help you.
If you live in a cool, dry climate, you may get what we call "Colorado nose". The inferior turbinates are a complex of bone, blood vessels, and a mucosal lining that are loacted on the outer sides of the nasal passages. Their function is to warm and moisten air. So--in a cold, dry climate--they are always working. Turbinate hypertrophy--in which the turbinates get bigger because they are working so hard--is common in this area, and physical obstruction of the airway can occur becuase of it. In my experience. fixing the turbinate problem is much more helpful for people with nasal breathing problems than correction of a septal deviation, unless the deviation is severe.
I am a surgeon, and tend to see medication as a temporary solution rather than a permanent "fix". People can get addicted to nasal inhalers, because there is rebound swelling when you stop using it. i would recommend getting a second opinion from a different ENT or plastic surgeon who is skilled in this area. A direct exam of the septum and turbinates, plus a detailed history, which help guide their recommendations, and your decision regarding surgery.
Risks of surgery include bleeding, infection, and a less than perfect outcome. Noses are inherently asymmetrical, and perfect evennes is impossible to achieve. Improvement, not perfect correction of symptoms is a realistic goal.

Hope this helps

Diane Duncan