A patch file contains both line number info and a context, i.e a few lines from the original file to help synchronize if the original file has changes slightly.
If goes to the line number stated in the patch file and looks up and down in the file for lines matching the context of the hunk. If it doesn't find a match, it removes the first and last line of the context and tries again. The number of lines removed is the "fuzz factor". If it finds a match, it remembers the line offset and applies it to the next hunk.
Clear as mud, no? :-)