You would have to write a child theme with a replacement single.php file and use the $excluded_terms parameter right after the $cats in the calls to next/previous_post_link.
The codex says:
excluded_terms
(string/array) (optional) Array or a comma-separated list of numeric terms IDs from which the next post should not be listed. For example array(1, 5) or ‘1,5’.
I think, for example
<div class=”nav-previous”><?php previous_post_link( ‘%link’, __( ‘<span class=”meta-nav”>←</span> Previous’,’weaver-xtreme’), $cats, ‘83,84’ ); ?></div>