Abstract: The strong random fluctuations and frequent operation-mode switching in new power systems cause relay protection settings to suffer from adaptation delays and difficulty in online hidden defect identification. This paper proposes a diagnostic method that combines operation-mode adaptation and setting-coordinated defense. First, real-time switch status and power flow data are used to partition the grid topology via graph theory, and an expert model of primary and secondary operation modes is employed for online identification, enhanced by forward-looking dispatch plans. Then, based on the identified mode, the method retrieves candidate setting zones from a preset database and performs multi-dimensional verification including selectivity, sensitivity, and speed, enabling adaptive online switching of setting zones in a closed loop of “identification–verification–execution–assessment”. Field tests show that the method achieves a detection accuracy of 94.7%, a false alarm rate of 4.2%, a coordination margin index of 0.84, and a total switching delay within 300 ms, effectively identifying risks of silent failures and miscoordination, thus providing engineering support for active defense of relay protection.