How to configure access for creating a custom LMS?
I recommend you have a look at the Group module, which may be considered as a possible alternative to the Organic Groups module, and which you seem to be familiar with already.

Have a look at my answer to "What are the features of the Group module versus Organic Group module?" for details about:


How both modules compare to each other.
Integration with various modules that you might want to take advantage of also.


Resources


Community documentation for the Group module.
Video tutorial "How to use the Group module in Drupal 8 to manage editorial access control to different areas of a web site".
Podcast "Using the Group module as an Alternative to Organic Groups in Drupal 7 and 8".


Sample configuration

Below is an example of a possible configuration, which may help to get you started with possible configurations (using the Group module):

Group type ID ! Group type ! Group role ID ! Role
--------------+------------+---------------+------------
 1            ! Class      ! 3             ! Instructor
 1            ! Class      ! 4             ! Student
 2            ! Customers  ! 5             ! Lead
 2            ! Customers  ! 6             ! Prospect
 2            ! Customers  ! 7             ! Licensed Customer
 2            ! Customers  ! 8             ! Sales
 2            ! Customers  ! 9             ! Support
 2            ! Customers  ! 10            ! Accounting
 2            ! Customers  ! 11            ! R&amp;D
 3            ! Project    ! 12            ! Project Manager
 3            ! Project    ! 13            ! Developer
 3            ! Project    ! 14            ! Customer 


Group Types

The above example consists of 3 group types:


Class.
Customers.
Project.


Group Roles

For each of the 3 group types, there are specific Roles defined, some more details about some of them:


for the Class group, these are the 2 roles that are configured:


Instructor.
Student.

for the Project group, these are the 3 roles that are configured:


Project Manager.
Developer.
Customer.



These are the Group Specific Roles: these roles can be configured per group type and will only be available on groups of that group type. As an example, for group type "Class", the defined roles are Instructor and Student (which are not available for the other group types.

On top of the group specific roles, there are also Special Global Roles: These are the special (hardcoded) roles named Outsider (= a signed in user that did not join a group) and Member  (= a signed in user that did join a group). These cannot be defined by the Drupal administrator, and they are always available for any group. These roles can have different permissions sets for each group type. Special case: the role Anonymous is related to a user that is not signed.

Groups

Each of the configured group types can have any number of Groups.

Examples of groups:


for the Class group type, there could be these Groups:


Getting started with Drupal.
Drupal for site builders.
Become an expert in using the Rules module.

for the Customers group type, there could be these Groups:


Service Requests.
Order Processing.
Sales Orders.
Relationship Management.



Permissions

For each of the configured group types, you can then also configure for each Content Type which Role has what kind of access (= none, view, create, edit, delete). Be aware: even though this looks similar to Drupal's permission configuration, these permissions are not the same permissions (they are specific to the Group module).

PS: this answer applies to either D7 or D8.