How to structure data: Solo Events (Lectures) and Group of Events (Conferences)
My approach:

Create one channel for events with different entry types (solo-events, conference-events)
Create one channel for conferences
Re-use the date fields in both channels
That way you can fetch and order events by the re-used date fields from both channels limiting to the entry types of solo-events and conferences with one query. That makes listings, filtering and paging easy.
Conference-events can be eager-loaded with the query as well
The entry type for conference-events has a mandatory entry field for the conference
In the control panel you can show the related conference by adding the field as a column in the overview
Speakers is another channel or structure being linked from within the conference-events

Why I would not use:

Matrix: can be easily queried but can not eager load the entry it belongs to
Category: feels wrong to always add categories for new conferences
Structures for events: Structure are not 100% user friendly when adding new entries, as they always need to be sorted manually (though there are some improvements in 3.7 coming up)