One of the issues is associating the caption with the image.
The Featured Image Plugin does this by letting you add the caption specifically on the post’s page right under the Featured Image box. This is okay. The problem with the plugin is that it doesn’t put the caption right under the image. The caption is wrapped in a <div>, and it might be possible to style it so that the caption works right, but it looks like it might need a clear:both; after to get layout to work right. I didn’t play too hard on that.
The other option would be to use the already defined caption for each image in the media library. This could be done by filtering ‘post_thumbnail_html’ (I think), but the PHP logic for fetching the caption and modifying the output would be a bit complex. But doable with some effort.