how to update my custom table after execution of rules
In rules you have option to add PHP code inside the Action ->PHP-> "Execute custom PHP code " in that have a code to update the custom table.

The following variables are available and may be used by your PHP code:

Variable name Type    Description

$site   Site    Site information

$node   Content updated content

$node_unchanged Content unchanged content 

you can use for updating the table if its a node

Note : here node represented as a example