[Rec] is certainly a scarier movie, and the first person camera thing was much more convincing than in Quarantine. Also, the creature at the ending is much more shocking, and the...what the hell was I going to say? I forgot. Oh, right.
Quarantine, on the other hand, does a much better job with character development, and adds a lot of nice touches with the story throughout. Storywise, Quarantine wins. Scarywise, [Rec] wins.
However, both have incredibly cheap endings, despite the absolute horror that leads up to it. Fantastic night vision sequence. The acting is about equal in both, but really only with the lead character, Angela. The other characters are much more convincing in [Rec.]
The two movies have different explanations of the "zombies," and both of these explanations are pretty damn good, and do a fantastic job of sending shivers down your spine, or whatever.