Designing a join table for max n records
In that case the application should manage the amount of products on each order. 

If you want to enforce a limit on database level you can add a column ProductOrderPosition in OrderProduct that's the ordering position of each product by each order, and you can add a CHECK constraint to limit this value, for example:

ALTER TABLE OrderProduct ADD CONSTRAINT CHK_OrderProduct_ProductOrderPosition 
    CHECK (ProductOrderPosition &lt;= 10)


If you don't want additional columns you could validate with an after insert trigger, checking that each order doesn't have more than X products.

CREATE TRIGGER dbo.utrOrderProductCheckAmount ON OrderProduct AFTER INSERT
AS
BEGIN

    DECLARE @v_ProductsByOrderLimit INT = 10

    IF EXISTS (
        SELECT
            'more than allowed!'
        FROM
            OrderProduct AS O
            INNER JOIN (SELECT DISTINCT I.OrderID FROM inserted AS I) ON O.OrderID = I.OrderID
        GROUP BY
            O.OrderID
        HAVING
            COUNT(1) &gt; @v_ProductsByOrderLimit
        )
    BEGIN

        ROLLBACK

        RAISERROR('Maximum products by order limit reached.', 11, 1)

    END

END


You definitely don't want to store pivoted Products IDs on the Order table for the reasons you mentioned before. I'll add another reason, what happens if you want to know in how many orders does a product appear? You need 10 poorly indexed joins, instead of 1 optimized SELECT against OrderProduct.