Change "Group windows by application" preference using command line
The defaults command for Group windows by application in System Preferences > Mission Control is e.g.:

defaults write com.apple.dock expose-group-apps -bool false


In older version, it used to be:

defaults write com.apple.dock expose-group-by-app -bool false



The value can be true or false.


However, that will not immediately change it in the UI and it may get overwritten anyway.

The safest way to change it is from the UI.

Note: As suggested in the comments one can use killall Dock directly afterwards however, killall Dock will force all minimized windows to show, as well as it's visually disruptive. I'd suggest using AppleScript to change it as it will not force minimized windows to show nor it is as visually disruptive.

The following AppleScript code will check/uncheck the Group windows by application checkbox in System Preferences > Mission Control, and will not cause the same visual disruption that killall Dock will. It can be saved as an AppleScript application in Scrip Editor.

tell application "System Preferences"
    set the current pane to pane id "com.apple.preference.expose"
    tell application "System Events"
        click checkbox "Group windows by application" of group 2 of window "Mission Control" of application process "System Preferences"
    end tell
    quit
end tell


Note; This AppleScript code was tested under macOS 10.12.5 and my need adjusting for some earlier versions of OS X.