The 'rhythm method' of contraception is actually a collection of methods. Collectively, they rely on the fact that there are only a certain few days during each menstrual cycle in which conception can occur.
For conception to take place, the woman's egg (ovum) must be fertilised by one of the man's sperm cells (spermatozoa). After being injected into the vagina following ejaculation, the man's sperm wiggle their little tails like mad and propel themselves towards the egg, at the rate of 1-4 mm per minute. (Well, some of them, anyway; many are retards that swim the wrong way, or have broken 'propellers', or any number of other defects.) And here's the rub: although they can survive for many weeks in the testes, once in the vagina, sperm cells can only survive for 4-5 days.
On the female side of the equation, the woman's ovulation cycle is a complicated thing. Usually, the woman sends one egg from her ovary out into the world around 14 days after her menstruation. The expelled egg can last 12-24 hours before it is unable to be fertilised.
Thus, the earliest that fertilisation can take place is 5 days before ovulation (since the sperm can last out this period of time). And the latest that fertilisation can take place is one day after ovulation (since the egg degrades after this).
Sounds easy enough, right? Just avoid sex for the six day window period... Wrong. It's not that simple. While all the above is true, it all assumes that you actually know when the woman ovulates. But on this matter, she is notoriously unreliable.
I said earlier that the rhythm 'method' was actually several related methods. The various subtypes turn out to be different methods to try to determine ovulation.
The calendar method is the least reliable. It simply guesses the ovulation always occurs 14 days after the last menstrual cycle, and then dictates that the abstinence period should be a few days on either side. The problem with this is the levels of the hormones that control ovulation are easily changed in numerous states, such as illness, stress, and age. One study quantified this variability, and showed that even if you allow for a fertility window from day 10 to day 17, only 30% of women will fall totally within this range.
The temperature method relies on the fact that progesterone, which is secreted in large doses from ovulation to around menstruation, raises the body temperature by about 0.5ºF (just less than 0.3ºC). Assidious documentation of body temperatures (at the same time each day) can actually be surprisingly effective, but requires outstanding motivation.
Lastly, the cervical mucus is known to change around ovulation - from a thick barrier, it becomes soft and watery and penetrable to sperm. Thus, monitoring of this mucus can be a good guide to when ovulation occurs. The trouble is that it is cervical mucus (i.e. on the cervix, which is the bottom bit of the uterus; the bit that protrudes into the top of the vagina) that changes, not the vaginal mucus. And the cervical mucus is usually unable to be monitored by the woman directly. There are certain indirect ways to try to bypass these difficulties though, such as mentioned here.
In summary, how good is the rhythm method? Actually, under the best conditions, and with a highly motivated user, the effectiveness of the temperature and mucus methods are only marginally worse than that of condoms. (It can be about 96-98% effective). However, they are definitely less effective than most other conventional methods, like pills, infections, or intra-uterine devices. Also, it is difficult to keep up the motivation and attention to detail. Personally, I don't recommend it to my patients, unless they have objections to using the other methods. However, if used properly, it can be surprisingly effective.