Antonyms for the phrase "holding sway over" would include terms such as "submissive," "subjugated," and "powerless." These antonyms connote a lack of influence or control, whether over a specific situation or a broader societal dynamic. Other antonyms might include "dependent," "inconsequential," or "insignificant." These terms suggest that the individual or group in question has little impact on the course of events, and may be easily overlooked or dismissed. While the concept of "holding sway over" implies a certain degree of power or authority, its antonyms suggest a lack of agency or control, often at the mercy of external forces.