3-methyl-4-oxobutanenitrile or 3-formylbutanenitrile?
The most important simplified criteria for the choice of a principal chain are:


greater number of substituents corresponding to the suffix (principal characteristic group)
longest chain
greater number of multiple bonds
lower locants for suffixes
lower locants for multiple bonds
greater number of prefixes
lower locants for prefixes
lower locants for substituents cited first as a prefix in the name


The corresponding wording of the rules taken from Nomenclature of Organic Chemistry – IUPAC Recommendations and Preferred Names 2013 (Blue Book) is as follows.


  P-44.1 SENIORITY ORDER FOR PARENT STRUCTURES
  
  When there is a choice, the senior parent structure is chosen by applying the following criteria, in order, until a decision is reached. These criteria must always be applied before those applicable to rings and ring systems (see P-44.2) and to chains (see P-44.3). Then criteria applicable to both chains and rings or ring systems given in P-44.4 are considered.
  
  P-44.1.1 The senior parent structure has the maximum number of substituents corresponding to the principal characteristic group (suffix) or senior parent hydride in accord with the seniority of classes (P-41) and the seniority of suffixes (P-43).
  
  (…)
  
  P-44.3.2 The principal chain has the greater number of skeletal atoms [criterion (b) in P-44.3].
  
  (…)
  
  P-44.4.1 If the criteria of P-44.1 through P-44.3, where applicable, do not effect a choice of a senior parent structure, the following criteria are applied successively until there are no alternatives remaining. These criteria are illustrated in P-44.4.1.1 through P-44.4.1.12.
  
  The senior ring, ring system, or principal chain:
  
  (a) has the greater number of multiple bonds (P-44.4.1.1);
  
  (b) has the greater number of double bonds (P-44.4.1.2);
  
  (…)
  
  (h) has the lower locant for an attached group expressed as a suffix (P-44.4.1.8);
  
  (…)
  
  (j) has the lower locant(s) for endings or prefixes that express changes in the level of hydrogenation, i.e., for ‘ene’ and ‘yne’ endings and ‘hydro/dehydro’ prefixes (P-44.4.1.10);
  
  (…)
  
  P-45.2.1 The preferred IUPAC name is based on the senior parent structure that has the maximum number of substituents cited as prefixes (other than ‘hydro/dehydro’) to the parent structure.
  
  P-45.2.2 The preferred IUPAC name is based on the senior parent structure that has the lower locant or set of locants for substituents cited as prefixes (other than ‘hydro/dehydro’) to the parent structure.
  
  P-45.2.3 The preferred IUPAC name is based on the senior parent structure that has the lower locant or set of locants for substituents cited as prefixes to the parent structure (other than ‘hydro/dehydro’ prefixes) in their order of citation in the name.
  
  (…)


According to Rule P-44.1.1, the priority list of substituent groups has to be considered first in order to decide the parent chain for this compound. You have already done that in both of your suggested possibilities: The principal characteristic group that is expressed as a suffix is the nitrile group ($\ce{-(C)N}$).

The next rule (P-44.3.2) is looking at the longest chain. Therefore, the parent structure is a butanenitrile.

The following rules (P-44.4.1.1 and P-44.4.1.2) would be looking at multiple bonds, which is not relevant in this case.

Next, Rule P-44.4.1.8 is looking for a lower locant for the attached group that is expressed as a suffix (here: nitrile); however, the nitrile group has the lowest locant ‘1’ in both suggested cases; therefore, this rule doesn't help to make a choice.

The next rule (P-44.4.1.10) again is not relevant in this case.

Finally, Rule P-45.2.1 is looking for the structure that has the maximum number of substituents cited as prefixes to the parent structure. Therefore, the preferred IUPAC name is 3-methyl-4-oxobutanenitrile (two substituents) rather than ‘3-formylbutanenitrile’ (only one substituent).



By way of comparison, if you add more substituents to the methyl group, the principal chain can switch, for example 4,4-dichloro-3-formylbutanenitrile (three substituents) rather than ‘3-(dichloromethyl)-4-oxobutanenitrile’ (two substituents).