Understanding the Bilateral Filter - Neighbors and Sigma
I would start with the many resources on this site:


Is the Bilateral Filter a Solution of Some Variational Method?
How to Validate Bilateral Filter Implementation?
Comparison Between Guided Filter (Edge Preserving Filter) and Gaussian Filter.
What Is the Bilateral Filter Category: LPF, HPF, BPF or BSF?
Understanding the Parameters of the Bilateral Filter.


Regarding your questions, Let's address them one by one.

Q: How to Determine the Number of Neighboring Pixels?

The classic neighborhood used by Bilateral Filter is along the axis. So it is basically determined by Radius parameter. In order to make sense, the radius parameter is aligned with the $ {\sigma}_{s} $ parameter (Spatial Standard Deviation). Some examples of the connection between them can be found in:


How to Get Rid of Ripples from a Gradient Image of a Smoothed Image?
Parameters of Gaussian Kernel in the Context of Image Convolution.
How to Calculate Gaussian Kernel for a Small Support Size?
Gaussian Blur In MATLAB and Connection to Image Resolution.


Q: Are All Adjacent Pixels the Neighboring Pixels?

Well, Theoretically you can chose any way to set the neighborhood set (Circle, The whole image, etc...). But as I wrote above, the most common option it a rectangle (Square in most cases).

Q: What Are $ {\sigma}_{s} $ and $ {\sigma}_{r} $ For?

The parameters determines the range of values in the range and spatial domain.
Let's assume our image values are in the range [0, 1] and the spatial indices are integers. Assume $ {\sigma}_{r} = 0.1 $, then pixels with values which are farther than $ \sim 3 {\sigma}_{r} $ will have effective zero weight since $ {e}^{ -\frac{ {0.3}^{2} }{ 2 \cdot {0.1}^{2} } } = 0.0111 $ before normalization.

Same logic for the Spatial parameter $ {\sigma}_{s} $. If we set it to $ {\sigma}_{s} = 4 $ pixels which are more than 12 indices apart will have negligible weight. This is why we limit the radius of the neighborhood and match it to the spatial parameter as adding pixels which have zero weight is waste of computing resources.