How to create an insert trigger?
Don't assume that the trigger will always handle a single row inserted in the table. If you're inserting more than one row in Rental, the code you attempted to write will fail.

The logical tables inserted and deleted contain all the rows modified by the DML operation and they must be joined or correlated to the base table (or other tables) in order to take all the rows modified into consideration.

As far as the business logic required is concerned, this is a possible implementation:

CREATE TRIGGER tr_tblRental_ForInsert 
ON dbo.Rental
FOR INSERT
AS
BEGIN
    -- Always add this to your triggers: some libraries don't like
    -- triggers altering the number of rows affected by the DML operation
    SET NOCOUNT ON;

    -- a. Does not have more than 2 books rented out: 
    -- the books that are rented out are identified by having ReturnDate as NULL
    IF EXISTS (
        SELECT COUNT(*)
        FROM dbo.Rental
        WHERE MemberId IN (
                SELECT MemberId 
                FROM inserted
            )
            AND ReturnDate IS NULL
        GROUP BY MemberId
        HAVING COUNT(*) &gt; 2
    )
    BEGIN
        ROLLBACK;
        THROW 51000, 'Members are not allowed to rent more than 2 books at a time.', 1;
    END


    -- b. That the member does not owe more than 50 dollars 
    -- to the library (the BalanceOwing attribute of the Member table).
    IF EXISTS (
        SELECT BalanceOwing 
        FROM dbo.Member 
        WHERE MemberId IN (
                SELECT MemberId 
                FROM inserted
            )
            AND BalanceOwing &gt; 50
    )
    BEGIN
        ROLLBACK;
        THROW 51000, 'Members are not allowed to rent more books if they owe more than $50 to the library.', 1;
    END

END


Please do not use this code in your project/homework if you don't understand what it does: your teacher will ask questions and you'd better have answers. If you need help understanding the code, place a comment here.