For quest, as long as you hit it once before it dies, you'll get credit.
For loot, as long as you hit it once before it dies, you'll get a chance for an item.
As for what the chance should be, could make it based on how many people hits the mob and make it statistically the same as getting loot in a round robin fashion of a group of equivalent size.