Search result based on a specific field in a related entry in a matrix
Yes, you're right to better not use the search parameter for such things.

Instead get all the product variants that match your criteria first (e.g. size, color) and then query for all related products. Be careful with the field parameter, as your relation is between a Matrix field and your product variant entries.

Code is not tested (as you might guess, hehe):

{% set category = craft.categories.slug('skisko').first() %}

{% set sizeParam = craft.request.getParam('size') %}
{% set colorParam = craft.request.getParam('color') %}

{#
 # Get product variants matching our search
 #}

{% set variantParams = {
    section: 'productVariants',
    variantStock: '&gt; 0',
    variantSize: sizeParam,
    variantColor: colorParam,
} %}

{% set variants = craft.entries(variantParams) %}

{#
 # Get the products related to these variants
 #}

{% set productParams = {
    section: 'products',    
    relatedTo: [
        'and',
        {
            targetElement: variants,
            field: 'colorVariants.myRelationField',
        },
        {
            targetElement: category,
            field: 'myCategoryRelationField',
        }
    ]
} %}

{% set products = craft.entries(productParams) %}