I ran across this nice summary explaining the differences between A/B and Multivariate Testing:

A/B testing is trying 2 (or more) versions of a page component like a headline, image, button, etc. and seeing which one leads to the best increase in desired visitor behavior.

Multivariate testing is trying 2 (or more) versions of multiple page components at the same time - e.g. 2 headlines 3 images 5 buttons, etc. It's like running multiple A/B tests simultaneously.

A/B testing is a good way to have a simple "bake off" to decide which version is more effective towards a visitor goal like a purchase/conversion, lead submittal, registration, etc. It's also a good way to get started with testing before advancing to more complicated methods (and it's far better than not doing any testing at all!)

Multivariate testing tells you not only which versions of each component are "best", but reveals which components are more or less influencial towards one or more visitor goal. It tells you, for example, that a range of versions of the homepage image accounts for /- 30% in click-through to the next page, whereas the range of versions of headline accounts for only /- 4% in click-throughs (Why do you care? Next test you run you shouldn't bother with the headline... just focus on the image since it is more influential!)

You can roll your own A/B testing using homegrown software, and there are also several free/low cost PHP/ASP scripts that can do the job and get you up & running.

Tools and services that allow you to do A/B and Multivariate testing include:

Read more about this topic at this forum discussion.