Understanding transactions
A transaction in database management system is a collection of tasks to be done by the system with certain guarantees on results depending on the type of the transactions.
A process in OS is an implementation for a container which runs programs/code. Different kinds of processes (often just two types) have different capabilities.


There is a clear similarity between the two. However, they are different in several ways.


Transactions do not create new transactions. They have a beginning and an end.
Processes can create new processes, they have the beginning but they might not have an end


I believe that with PL/SQL, one can manage to accomplish a lot with transaction and with this a transaction might have a close resemblance to a process. For example, with PL/SQL, one can declare variables and use them in the same way that a process stores values and states in its virtual memory address. So in this way, a transaction and a process are very similar. This is about Turing completeness of PL/SQL. Transaction performs task written in PL/SQL language, and process performs task in the OS's programming language (which is the binary code of that OS).

One difference between the two is the way they communicate with each other. Transactions can only communicate indirectly with each other via the results of tasks stored in the database. Process can directly communicate with each other as well as indirectly with each other via side effect on the file system (or similar objects).

So the answer to your question:


Transaction is not a process in OS. They are similar concepts, but not the same
A program is not a process. This is because a program might be run by multiple processes
Of course, there are mathematical models behind transaction. Graphs are often used together with many other things. They use a lot of things in database management system theory. If you want to know more, perhaps a new, specific question will be more appropriate.