How to set a filesystem size limit for DB2 database? Is it possible?
Yes.  Tablespaces in DB2 have a MAXSIZE attribute that you can use.  You can specify these size attributes in the CREATE DATABASE statement (or when creating additional tablespaces), or use ALTER TABLESPACE to set them later. 

create database mydb
   automatic storage yes
   on C:\mydb
   user tablespace managed by automatic storage
      initialsize 20 M increasesize 10 percent maxsize 100 M


You could specify similar options for the system catalog tablespace and the default system temporary tablespace, but I would not recommend doing so.