usage percentage of Mongodb internal cache I've ran two performance test on a MongoDB server with identical environment/settings AFAIK.
I found that the throughput was 10% apart. 

When I inspected the mongostat logs, I found that Mongodb in the first test with faster throughput used 95% of its cache_size. In the second test, it stayed at 80%. 

I rebooted the server system between the runs, but didn't change any of the settings of the server or the client. mongodb log reports the exact same setting in both tests: 

First test: 
2019-08-14T00:42:32.588-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=96122M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),

Second test (after reboot):
2019-08-14T01:11:15.722-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=96122M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),


Here is a sample from mongostat of the first test: 

insert  query update delete getmore command dirty  used flushes vsize   res   qrw   arw net_in net_out conn                time
    *0 104209     *0     *0       0     9|0  0.0% 94.9%       0 96.7G 94.7G 107|0 128|0  27.0m   40.2m  272 Aug 14 00:51:13.759
    *0 102610     *0     *0       0     0|0  0.0% 94.9%       1 96.7G 94.6G 109|0 128|0  26.6m   39.6m  272 Aug 14 00:51:44.092
    *0 104724     *0     *0       0     9|0  0.0% 94.9%       0 96.7G 94.6G  87|0 128|0  27.1m   40.4m  272 Aug 14 00:52:13.767


And a sample from the second test:

insert  query update delete getmore command dirty  used flushes vsize   res   qrw   arw net_in net_out conn                time
    *0  94009     *0     *0       0     9|0  0.0% 80.9%       1 83.1G 80.9G 127|0  81|0  24.4m   36.3m  272 Aug 14 01:17:29.289
    *0  93680     *0     *0       0     0|0  0.0% 80.3%       0 83.1G 81.0G   0|0 107|0  24.3m   36.2m  272 Aug 14 01:17:59.119
    *0  93068     *0     *0       0     9|0  0.0% 80.9%       1 83.1G 81.0G 130|0 128|0  24.1m   35.9m  272 Aug 14 01:18:29.123
    *0  92599     *0     *0       0     0|0  0.0% 80.0%       0 83.1G 80.9G 137|0 128|0  24.0m   35.8m  272 Aug 14 01:18:59.186



What could have caused the difference in the cache percentage usage and throughput?  
is it safe to change explicitly this settings so Mongodb uses more of its cache?
Can I do this setting change from the command line ? (I'm starting mongodb from command line and not as a service)
What could have caused the difference in the cache percentage usage and throughput?
MongoDB uses both file system as well as WiredTiger's cache when required 
Is it safe to change explicitly this settings so Mongodb uses more of its cache?
Yes. It is totally safe to change the cache settings
Can I do this setting change from the command line ? 
The setting has be done in your mongod config file under storage option


Refer the below link on how you should decide on the cache size
https://docs.mongodb.com/manual/faq/storage/

Similar to the below:

storage:
  engine: wiredTiger
  wiredTiger:
    engineConfig:
      cacheSizeGB: 7


Thank you!

-Anban