How to calculate elliptic curve parameters? I'm having a rough time understanding the math behind elliptic curves.
I want to implement ECDH where user can define a, b, and p parameters of elliptic curve.

How can I calculate generator base point G? Is it any point that satisfies elliptic curve equation $ y^2 \equiv x^3 + ax + b \pmod p$?

Also, how can I calculate order n and cofactor h given the base point?
It is possible to find the desired values in an acceptable amount of time.

TL;DR: Find the curve order, factor it, select a (random) point until you have one with the desired order and calculate the cofactor as quotient of curve and point order.

First, you can use yyyyyyy's answer to find the order $n$ of the described curve using Schoof's algorithm. Please note that this order should be prime or a small multiple (4x-8x) of the a prime for enhanced security but it doesn't have to have those properties for the schemes to work.

In the next step, you need to actually factor that order. This is feasible as this order usually will be around 500 bits of length (or a lot less if you're not going for the 256-bit security level), which can be done and was done for FREAK. Of course, if the curve order is prime or a small multiple of a prime (4x-8x) the factorization may not need to be obtained with the GNFS but may be either given (prime case) or findable by trial division (small multiple case).

Finally you just select a curve point and determine its order until you hit one with the desired order $t$ which will also yield the cofactor as $h=n/t$. Usually you'll want to have $h\leq 8$ and usually you'll also start with the points having small x-Coordinates and try all incrementally until you reach one fullfilling your conditions.



The algorithm for this can be found in a variety of places in the literature and I'll quickly restate the one from the Handbook of Applied Cryptography, assuming we're testing the point A and that $n=p_1^{e_1}\cdot p_2^{e_2} \cdot\cdot\cdot p_k^{e_k}$:


Set $t\gets n$
For $i$ from $1$ to $k$ do the following:


Set $t\gets t/p_i^{e_i}$
Compute $A_1\gets t\cdot A$
While $A_1\neq \mathcal O$ do the following: compute $A_1\gets p_i*A_1$ and set $t\gets t\cdot p_i$

Output $t$ as the order of $A$




Please note: I strongly recommend against using custom curves where the parameters are chosen by the user. These may be vulnerable to attacks documented in the safe curves project. Standardized curves (like Curve448 or Curve25519) should be preferred for easier implementation and better analyzed security properties.