How to update rooted android (nexus 5) I have rooted Nexus 5 device with Android 5.1.1.
Now system shows updates are available. But on restarting it shows error screen, virtually making it impossible to update.

Is it possible to update rooted android OS, and if yes how (without wiping out any data)?
You have to use the Android SDK, specifically the command fastboot, to flash the stock system.img and stock recovery.img.  Only flash the boot.img only if a custom kernel has been flashed, if not, this isn't required.  With USB Debugging enabled in Settings - Developer Options, you'll need to boot into the bootloader.  You'll also need the factory image for the current version of stock you are running.

You'll need to use the commmands:

fastboot flash system system.img
fastboot flash recovery recovery.img
fastboot flash boot boot.img


Rooting modifies /system, which OTA's will check and fail.  Also, any other recovery except for stock recovery will also fail to flash the OTA.