Simple Java password rule enforcing
I see at least one big problem in your method: there is no separation of concerns. Everything is mixed into this method: common rule for password double-checking, business rules for password validation and UI calls for user response.

Extracting the business logic

The business logic in your case if the password validation specific rules, as you named them rule1, rule2a, rule2b, rule2c and rule2d.
Everytime you will need a password validation, you will need these rules to validate a password, therefore I would extract these rules into a BO.

import java.util.function.Predicate;
import java.util.stream.Stream;

public final class Password {
    private final String value;
    private final Predicate&lt;String&gt; rule;

    public Password(String value) {
        this.value = value;
        Predicate&lt;String&gt; rule1 = s -&gt; s.length() &gt;= 8 &amp;&amp; s.length() &lt;= 50;
        Predicate&lt;String&gt; rule2a = s -&gt; !s.equals(s.toLowerCase());
        Predicate&lt;String&gt; rule2b = s -&gt; !s.equals(s.toUpperCase());
        Predicate&lt;String&gt; rule2c = s -&gt; s.codePoints().anyMatch(Character::isDigit);
        Predicate&lt;String&gt; rule2d = s -&gt; s.codePoints().anyMatch(i -&gt; !Character.isAlphabetic(i));
        Predicate&lt;String&gt; rule2 = s -&gt; Stream.of(rule2a, rule2b, rule2c, rule2d)
                                             .filter(p -&gt; p.test(s))
                                             .count() &gt;= 3;
        this.rule = rule1.and(rule2);
    }

    public boolean isValid() {
        return rule.test(value);
    }
}


If the rules are likely to change, it would also be possible to refactor this code and take the "final predicate" as a constructor argument.

Extracting "standard" logic

In your case, it is only the rule that assert the password and the confirmation password are equals. This is not a business-specific rule but rather a commonly accepted rule everytime a user creates a password. Moreover it seems you have a "validation flow" defined (first check the equality between the two password, then check if the password in itself is valid) that can be extracted into a separate class

public final class PasswordInteractor {
    public ValidationResult validatePassword(String pwd, String confirmPwd) {
        if (!pwd.equals(confirmPwd)) {
            return new ValidationResult("Passwords must match.", false);
        }

        if (!new Password(pwd).isValid()) {
            return new ValidationResult("Password does not follow instructions above.", false);
        }

        return new ValidationResult("Password set successfully", true);
    }
}


I created a DTO ValidationResult to transmit to the upper layer a composite result since it can be more than true/false.

public final class ValidationResult {
    private final String message;
    private final boolean success;

    public ValidationResult(String message, boolean success) {
        this.message = message;
        this.success = success;
    }

    public String getMessage() {
        return message;
    }

    public boolean isSuccess() {
        return success;
    }
}


Extracting UI calls

Finally the UI calls must stay in the UI-part, namely into the JavaFX controller. Typically it could looks like this (simplified):

import javafx.fxml.FXML;
import javafx.scene.control.Label;
import javafx.scene.control.TextField;
import javafx.scene.paint.Color;

public final class PasswordController {
    @FXML
    private Label responseLabel;

    @FXML
    private TextField pwd;

    @FXML
    private TextField confirmPwd;

    public void validateForm() {
        // validate other parts of the form

        ValidationResult result = new PasswordInteractor().validatePassword(pwd.getText(), confirmPwd.getText());

        if (result.isSuccess()) {
            responseLabel.setTextFill(Color.GREEN);
        } else {
            responseLabel.setTextFill(Color.RED);
        }
        responseLabel.setText(result.getMessage());
    }
}