Plain-text file encryption and decryption
Hello and welcome to Code Review! I try to give a possible explanation of what the reviewer meant when he was talking about inverse class structure; in the main method this code is present:

MyAlgorithm myAlgorithm = MyBuilder.builder()
               .withCipherSet(CIPHER_CHAR_SET)
               .build();


But from your code MyAlgorithm is a subclass of EncryptionAndDecryption and this is the class where you defined the operations of encryption and decryption , so the method shoud have return an instance of class EncryptionAndDecryption instead of one subclass. 
Other problem: in the class EncryptionAndDecryption we have the below methods:

protected final String encryptOrDecryptText(final String cipher, final String inputText, final MyAlgorithm.Action action)

protected final void encryptOrDecryptFile(final String cipher, final Path filePath, final String baseDirectory, final MyAlgorithm.Action action)


With parameter MyAlgorithm.Action action the class EncryptionAndDecryption will be dependant from its subclass MyAlgorithm and without its subclass it cannot be even compiled.
In the class EncryptionAndDecryption I would make two distinct methods for encryption and decryption of strings, putting these related to files and directory in the subclass MyAlgorithm, deleting the enum value you are using to distinguish between the two operations.
Style problem : in the main method is present a list of if like this below:

if ("encrypt".equalsIgnoreCase(action)) {}


Normally it is the string compared to the string literal like:

if (action.equalsIgnoreCase("encrypt")) {}


You can substitute the list of if with a switch like documented in strings-switch.