Rather than having a site that works for all visitors, I think your idea of two sites is a better approach, and WordPress makes this easy.
In order to keep it easy to manage content, I would recommend trying a theme switcher plugin – unlike Weaver II, Weaver Xtreme does not yet support switching designs.
I think there are theme switcher plugins. There are also likely existing themes designed for visually impaired visitors, so you could use that, and then Weaver for your standard visitor.