.htaccess rewrite doesn't work I have custom module with page at /page wich expects $_GET['id'] to do it's job. 

Here's .htaccess rewrite rule that works on empty website

RewriteRule category+/(.*)$ page.php?id=$1 [L]


On Drupal 8 I suppose that line should be

RewriteRule category+/(.*)$ page?id=$1 [L]


but it gives me 404 Error when i visit /category/5
The rewrite rule will not work because page is not a valid path on the server. By default Drupal rewrites all requests into index.php. When you rewrite to page apache will expect a path page on the server. 

This is not quite the right way to accomplish what you are trying to do in Drupal. However you can achieve what you are trying to do with the following

RewriteRule category+/(.*)$ page?id=$1 [R=301,L]


That will redirect category/5 to page?id=5 and you will get 5 in $_GET['id']

Check these out as well

https://www.drupal.org/docs/user_guide/en/content-paths.html

How do I retrieve the query parameters from a path?