How do you root a device with Magisk when it doesn't have a custom recovery This is a self answered question to address the subject of title.
With a recovery in place, rooting with Magisk is straightforward,it gets a tad difficult when you don't have a recovery.This was vaguely addressed in Stock image dump for running Magisk necessary?.
So, I thought of making a self answered question to

Help those who don't know and
Add to  canonical questions. It's important for our site to have such questions as easy reference (not-so-subtle request to others to post more self answered questions!).

Notes:

I have tried both Linux and Android versions of the answer and they work fine (tested on Pixel 4a, Android 11). The process may look daunting but it isn't (I know nothing about Linux, yet managed to root without issues).

Moderators are free to make this a community wiki, if more answers are posted depending on the device variants :)
This answer is not applicable to Samsung / Huawei devices


First, unlock your bootloader. This again ranges from very easy to complicated procedures getting unlock codes, waiting period etc, so read that up first and you have adb installed (this would help , if you don't have it Is there a minimal installation of ADB?).

Second, and very important read these installation instructions from Magisk Developer before proceeding. There are many variants of device architectures and OEM specific issues.

Third, read above  step thoroughly and accept outcome of your actions. Especially, Samsung /Huawei users, see respective linked sections.

Fourth, this answer is applicable to devices having

boot ramdisk in which case you need a copy of the boot.img (in which case custom recovery if available can also be used, but then we are assuming it isn't available)
If your device does NOT have boot ramdisk, you need a copy of the recovery.img
( To verify, install Magisk Manager and check SAR, as explained in official Magisk instructions, linked in second bullet point or read up the forums in XDA to figure out).


Substitute recovery.img for boot.img, if your device doesn't have ramdisk. Answer talks of boot.img as an example but they are interchangeable as far as process is concerned.

This process works on all  Android versions, including 11. I did it on my Pixel 4a running Android 11 (on earlier versions, my devices had custom recovery, so I didn't need this).


Extracting boot image (or recovery)

Get the official firmware package or OTA files from official firmware packages, your custom ROM zip (if using one), or go to XDA-Developers and seek for resources, guides, discussions, or ask for help in your device’s forum.

Pick up the firmware or full OTA (not incremental OTA) file that corresponds to the Android security update of your device , not necessarily the latest (you never know if the boot.img has been changed).

Prior to A/B partition system implementation ,these update firmware/ OTA packages came with partition images, which could easily be extracted and used for manual flashing. But now, these update packages come with a single Payload.bin file that contains the individual partition images (boot.img or recovery.img).

There are many blogs that tell you how to this. I found How to Extract Android OTA Payload.bin File using Payload Dumper Tool on a PC or Android Device to be very comprehensive and clear.


This shows how you can extract the boot image using a Linux machine, Android or Windows.I don't intend copy pasting the article with photos in it's entirety, but all essential steps are listed.
Using Payload Dumper Tool to Extract Payload.bin

Payload Dumper tool is basically a Python script/program created by developer ius that allows users to easily extract  Payload.bin

Installing Payload Dumper tool
On laptop (Windows/ Linux)

The tool requires Python 3 or above, which can be downloaded from the official page.

When installing Python, make sure that it’s added to the PATH on your PC. For Windows PC, this could be easily done by checking the Add Python to PATH option within the installer. For Linux/macOS, you can use the instructions provided on this page.

Download the latest version of the Payload Dumper Tool on your PC from this link and extract its contents to a suitable location. Upon extracting you should have a new folder named payload_dumper.

Open the command-line window on your PC in the payload_dumper folder, where the tool and the payload.bin files are present. To do this:



On a Windows PC: Hold the SHIFT key and right-click on an empty space inside the folder, then select the Open PowerShell window here option.

On a macOS/Linux PC: Launch the Terminal and change its directory to the payload_dumper folder using the cd command. For example: If the payload_dumper folder is located in the Documents directory, then the command should look like: cd Documents/payload_dumper



In order to use the tool on your PC, you must first install ‘Protobuf‘ (Google’s data interchange format) which is basically a dependency. To install it, enter the following command.

On Windows (in PowerShell): python -m pip install protobuf
On Linux/macOS (in Terminal): python3 -m pip install protobuf


Now to finally extract the Android OTA payload.bin file, first,extract the downloaded ZIP file to your PC . The extracted contents will include the “Payload.bin” file, which is what you’d need to extract. Copy the Payload.bin file inside the “payload_dumper” folder. Next,enter the following command.

On Windows (in PowerShell): python payload_dumper.py payload.bin
On Linux/macOS (in Terminal):python3 payload_dumper.py payload.bin


The tool will now start extracting the individual partition images from the Payload.bin file. Once finished, you will find the extracted files inside the same payload_dumper folder and you can pick up the boot.img or recovery.img


On Android. This is a little slower but also saves one step as you would see later.

Download the Payload Dumper Tool from here. Once downloaded, extract the contents of the payload_dumper_tool_by_ius.zip file to the root of your phone’s internal storage. You could extract it anywhere, however, the root directory is more convenient. The extracted ‘payload_dumper’ folder will contain the tool’s Python scripts

Now, download the  full OTA / firmware package from which you want to extract the firmware images. Once downloaded, extract the  package on your device to get the payload.bin file. Then copy the extracted payload.bin file inside the ‘payload_dumper’ folder that you extracted during step 1 above.

Download and install the Termux app (a Terminal Emulator) on your Android device. Once installed, launch the app from your phone’s app drawer.

Install and set up Python on your Android device. To do this, enter the following commands in Termux one after the other:
pkg install python -y
pip install --upgrade pip
pip install protobuf
apt update &amp;&amp; apt upgrade -y

Now, use the following command to grant storage permissions to ‘Termux’: termux-setup-storage. As soon as you enter the command, you’ll see a prompt to “Allow Termux to access photos, media and files on your device”.  Simply press “Allow” to grant the permissions.

Change Termux’s directory to the “payload_dumper” folder using the following command, provided that you have the folder placed on the root of your phone’s internal storage.
cd storage/shared/payload_dumper

Finally, enter the following command in Termux to extract the Android OTA Payload.bin bile on your Android device: python payload_dumper.py payload.bin


Once finished, the extracted firmware images will be placed inside the same ‘payload_dumper’ tool.
You are nearly done!
Last step of patching images
(from the official Magisk instructions)


Copy the boot/recovery image to your device (this step not needed if you used Android method)
Download the latest stable Magisk apk from the official source and install on your device (grant permission for installing from other sources)
Press the Install button in the Magisk card
If you are patching a recovery image, make sure “Recovery Mode” is checked in options. In most cases it should already be automatically checked.
Choose “Select and Patch a File” in method, and select the stock boot/recovery image
Magisk Manager will patch the image to [Internal Storage]/Download/magisk_patched.img.
Copy the patched image to your PC with ADB:
adb pull /sdcard/Download/magisk_patched.img
Flash the patched boot/recovery image to your device.
For most devices, reboot into fastboot mode and flash with command:
fastboot flash boot /path/to/magisk_patched.img or
fastboot flash recovery /path/to/magisk_patched.img if flashing a recovery image
Reboot and voila!


(Text in Italics added)
For any issue with Magisk, you will find this guide by Didgeridoohan very handy.