I've tried to cover this topic before, and I recited what one of the radiologists had told me, promising to check on the figures later. I have belatedly done so, and here are the results.
Background radiation (the low-level stuff that comes from outer space and the few naturally radiactive substaces on earth) is what all figures should be referred back to for comparison. We don't hide ourselves behind heavy lead aprons every day, and we are sensible not to. The average background radiation a person receives per year is 2.4 millisieverts per year. (That weird unit is a useful measure of the biological effect of radiation, rather than its physical effects, and so is more useful to use in answering this type of question.)
The humble X-ray is about 0.02 millisieverts, which is therefore equal to about 3 days worth of ambient radiation. Thus, apart from the slightly more contentious issue of irradiating pregnant women, you fundamentally have nothing to fear from X-rays.
CT scans are more difficult to quantify, since they vary quite a lot according to the size of the body area that is scanned, as well as how thin the 'slices' are. Nonetheless, CT scans typically are about 6-10 millisieverts. This is about 3-4 years worth of ambient radiation per scan, which is certainly more substantial than with X-rays. This suggests that it is worth doctors considering other diagnostic modalities that don't use ionizing radiation, although a single CT scan is still probably not worth worrying too much about, if the indication for it is valid.
Of course, everything has its risks. The most important thing is for the doctor to weigh up the (rather slight) risks of CT scans versus potentially not diagnosing the patient's condition timeously. Saving the patient from three years of background radiation won't earn you any plaudits if the patient is no longer around to enjoy them! Furthermore, although MRIs don't use ionizing radiation, they are more expensive, slower and not without their difficulties (especially around metalic objects!) compared with CT scans. Furthermore, CT scans are better at showing up certain types of pathologies. And although ultrasound scans are quick, cheap and radiation free they are simply not good enough imagers of many things to be a replacement more than some of the time.
The issue is impossible to resolve if we consider all cases as a whole. It is far better for each case to be individualised.