Are package facts getting invalidated automatically when I install remove packages?
Package facts (as any other facts like devices, disks and their partitions, IP addresses...) don't refresh automagically.
In your case, you can use a handler and notify it from any task that can possibly invalidate the cache. Here is a quickNdirty example:
- name: Pseudo code demo for package facts refresh
  hosts: localhost
  
  tasks:

    - name: Gather the package facts unconditionally at play start
      package_facts:

    - name: Make sure package X is installed
      package:
        name: package_X
      notify: refresh_package_facts
    
    # By default, handlers run at the end of the play.
    # If you want to force them to run at a certain point,
    # you need the following meta task.
    # Note =&gt; this will run !!ALL!! notified handlers
    - name: Run handlers if notified
      meta: flush_handlers

    - name: This task will run with an up-to-date cache
      debug:
        msg: &quot;{{ ansible_facts.packages | length }} packages are installed&quot;

  handlers:

    - name: Refresh the package facts
      package_facts:
      listen: refresh_package_facts