Do I need to keep Calendar app open in order to be notified about calendar events
Yes it does. If you set System Preference &gt; Notifications &gt; Calendar to Allow Notifications you should get a notification if you set the alert style to Banners or Alerts even if the app has been quit: