Can we load core css files from custom theme with yml lib?
The first attempt to load the single file normalize.css is not correct. If you want to do it this way, you have to use the full path to this file.

But this is not the preferred way to do this. You don't know if in future the position of this file will change. Better use the library already defined in core, as you have done in the second attempt to load modernizr. This will work, if you put the key dependencies on the first level of the array, for example see the core library date:

drupal.date:
  version: VERSION
  js:
    misc/date.js: {}
  dependencies:
    - core/drupal
    - core/modernizr
    - core/jquery.once
    - core/jquery.ui.datepicker


If you look for normalize.css, this is defined in core:

normalize:
  remote: https://github.com/necolas/normalize.css
  version: "3.0.3"
  license:
    name: MIT
    url: https://github.com/necolas/normalize.css/blob/master/LICENSE.md
    gpl-compatible: true
  css:
    base:
      assets/vendor/normalize-css/normalize.css: { weight: -20 }


And you can use it as dependency in *.libraries.yml:

  dependencies:
    - core/normalize


Or as library in the theme.info.yml file:

libraries:
  - core/normalize


If you use a core library, you don't have to put it in your own library, you can choose the direct way to use it.