It’s been a while since I’ve had to work with the srcset stuff, but if my memory servers me, WP generates the srcset, and it is also related to the image sizes in the media libary. If you’ve changed your image sizes for thumb, medium, etc, then you can have mixed results with what shows.
There are plugins that can regenerate your media images to the current sizes, and that should optimize srcset generation.