IUPAC name of a molecule with a cyanide group and a double bond
The relevant simplified criteria for the choice of a principal chain are:

greater number of substituents corresponding to the suffix (principal characteristic group)
longest chain
greater number of multiple bonds

The corresponding wording of the rules taken from Nomenclature of Organic Chemistry – IUPAC Recommendations and Preferred Names 2013 (Blue Book) is as follows.

P-44.1 SENIORITY ORDER FOR PARENT STRUCTURES
When there is a choice, the senior parent structure is chosen by applying the following criteria, in order, until a decision is reached. These criteria must always be applied before those applicable to rings and ring systems (see P-44.2) and to chains (see P-44.3). Then criteria applicable to both chains and rings or ring systems given in P-44.4 are considered.
P-44.1.1 The senior parent structure has the maximum number of substituents corresponding to the principal characteristic group (suffix) or senior parent hydride in accord with the seniority of classes (P-41) and the seniority of suffixes (P-43).
(…)
P-44.3.2 The principal chain has the greater number of skeletal atoms [criterion (b) in P-44.3].
(…)
P-44.4.1 If the criteria of P-44.1 through P-44.3, where applicable, do not effect a choice of a senior parent structure, the following criteria are applied successively until there are no alternatives remaining. These criteria are illustrated in P-44.4.1.1 through P-44.4.1.12.
The senior ring, ring system, or principal chain:
(a) has the greater number of multiple bonds (P-44.4.1.1);
(b) has the greater number of double bonds (P-44.4.1.2);
(…)

You have correctly identified the suffix (“nitrile”) for the principal characteristic group. Thus, according to the first criterion, the parent structure could be pentanenitrile or but-3-enenitrile since both include the nitrile group.
The next criterion for the principal chain is the greater number of skeletal atoms (i.e. the longest chain). Thus, the principal chain corresponds to the pentanenitrile part and not to the but-3-enenitrile part since pentanenitrile has a longer chain than but-3-enenitrile. Therefore, the correct name is 2-ethenylpentanenitrile.
Many people would probably still call it 2-vinylpentanenitrile; according to current IUPAC recommendations, however, the prefix “vinyl” is retained only for general nomenclature, “ethenyl” is recommended as the preferred prefix.