A modal window (e.g. Ajax Lightbox or Greybox, see http://orangoo.com/labs/GreyBox/) will interrupt the workflow to allow a subordinate task, e.g. watch an image, confirm feedback, look up help etc.
‘A modal view strengthen’s the user’s perception of entering a separate, transient mode in which they can accomplish something‘
‘Use a modal view when you need to offer the ability to accomplish a self-contained task related to your application’s primary function.’
(Apple iOS Human Interface Guidelines)
A pop up window presents related information or tasks in an additional panel floating on top of the main content. Pop-ups in Web sites are problematic with regard to accessibility and usability (see Nielsen’s top ten of mistakes) and are often blocked by the browser. Pop ups are a common feature, however, in controlled environments such as Flash, Silverlight, and in applications.