Drupal 8 CKeditor behaves LTR in an RTL site
The default CKEditor text language direction is LTR.
To change it to RTL, you can try setting the JS config setting:

config.contentsLangDirection = 'rtl';


Reference: http://docs.cksource.com/ckeditor_api/symbols/CKEDITOR.config.html#.contentsLangDirection

In D7, there was a config textarea that can be set:



In D8, you can set the CKEditor config settings in:

hook_editor_js_settings_alter

Something like:

/**
 * Implements hook_editor_js_settings_alter
 */
function MYMODULE_editor_js_settings_alter(array &amp;$settings) {
  foreach ($settings['editor']['formats'] as $name =&gt; $value) {
    $settings['editor']['formats'][$name]['editorSettings']['contentsLangDirection'] = 'rtl';
  }
}


More info on creating Drupal 8 modules here: https://www.drupal.org/developing/modules/8

You can also set the config in JS, for example your theme

Something like:

(function ($, Drupal) {
  Drupal.behaviors.customCKEditorConfig = {
    attach: function (context, settings) {
      if (typeof CKEDITOR !== "undefined") {
        CKEDITOR.config.contentsLangDirection = 'rtl';
      }
    }
  }
})(jQuery, Drupal);


More info on adding theme assets here: https://www.drupal.org/theme-guide/8/assets

For your other question on the toolbar button ordering. That appears to work, just make sure you set the toolbar button order on all the "Text formats" that you want to use it with (e.g "Basic HTML", "Restricted HTML" and "Full HTML") in the CKEditor admin settings.