Same statement yields NULL when in a stored function, and non-NULL valid result otherwise
Your problem is due to the same variable name is used for the parameter and local variable. i.e, wynik used as INT in the local variable as well as DOUBLE in the parameter of the nowyRanking function.

In this case changing the parameter name will solve the issue. In the below code I renamed the parameter name wynik DOUBLE to wynikDouble DOUBLE and handled the same in the query to solve the problem. 

Answer for How should I fix my stored function?

CREATE FUNCTION nowyRanking(mój INT, przeciwnika INT, wynikDouble DOUBLE)
RETURNS INT DETERMINISTIC
BEGIN
  DECLARE wynik INT;
  SET wynik = 
    mój + ROUND(3.0E1 * (wynikDouble - spodziewanyWynik(mój, przeciwnika)));
  RETURN wynik;
END


Working fiddle: http://sqlfiddle.com/#!9/76b5b/3

Answer for Why on Earth does this happen?

Update: In your actual code you declared the wynik as INT and used in the SET wynik = mój + ROUND(3.0E1 * (wynik - spodziewanyWynik(mój, przeciwnika))); block, here the wynik value is NULL only. 

So the query is execute as SELECT 1500 + ROUND(3.0E1 * (NULL - spodziewanyWynik(1500, 1500))); that's why you are getting NULL as result.