I was actually just able to find a workaround for my issue. I know it sounds simple, almost too simple that I over looked it.
So, even though it was trying to load the image that was set in the customizer, the customizer was showing “No image selected” therefore I could not just remove the image. So what I was able to do was simply click browse, choose a new image, the image would show in the customizer and then I could choose “remove” for that image and save and publish and it seem to work.