Is ECC over real numbers possible? Many elliptic-curve cryptosystems today use GF(p) or GF(2^m).
What if, say, we use big floating numbers with the classical point addition formulas - is a cryptosystem possible to build on that?
Summary: ECC over real numbers can be made to work, including for toy-sized security parameters and real numbers arithmetic as directly supported by typical CPUs and spreadsheets. But much more precision would be needed for private keys large enough to hope for security. That would be inefficient if secure; and perhaps just insecure.



We can mathematically define the infinite group of points on an elliptic curve on the continuous plane, e.g. with Cartesian representation in $\mathbb R^2$ under point addition (with an additional neutral element). We can define scalar multiplication on that. For any integers $k_A$ and $k_B$ and any point $g$ on the curve, it mathematically holds that $k_A\times(k_B\times g)=k_B\times(k_A\times g)$, as used by ECDH and ECIES.

However there is at least one huge problem facing a cryptosystem based on that line of thought: when using Floating Point to represent reals in $\mathbb R$ that are coordinates on the curve, $k_A\times(k_B\times g)=k_B\times(k_A\times g)$ no longer exactly holds due to round-off error, and that worsens with larger $k_A$ and $k_B$ for reasons of numerical stability.

The underlying problems are that


When using FP arithmetic, each point addition on the curve involves (among other operations) addition, and the FP version of that is not associative; otherwise said, there are exceptions, occasionally quite notable, to $(x+y)+z=x+(y+z)$.
Independent of FP arithmetic, the effect of a small variation of $g$ on $k\times g$ grows with $k$ (somewhat like the effect of a small variation of $x\in\mathbb R$ on $x^k$ grows about linearly with $k$).


With the $k_j$ in the hundreds of bits, there are many hundreds chained point additions in the computations of $k_A\times(k_B\times g)$ and $k_B\times(k_A\times g)$, so that discrepancies between the two values at each step (due to 1) will have many chances to appear and grow (due to 2) to the point where the results are extremely different (much like for evaluation of the logistic map in the chaotic region for large number of iterations).

Increasing FP precision will help, but it is hard to tell exactly how precise a FP representation of coordinates on the curve we shall use for $g$, and intermediary values including $k_A\times g$ and $k_B\times g$, so that $k_A\times(k_B\times g)$ and $k_B\times(k_A\times g)$ coincide well-enough to derive a shared secret.
Note: the practical issue of defining that well-enough and turning it into a consistent shared secret is non-trivial, but it can be solved by rounding to agreed-upon number of bits, combined with appropriate error correction like: detecting disagreement and, should that occur, retrying or trying nearby values; sending a small public additive correction, or other form of Forward Error Correction.

Floating Point arithmetic with direct hardware support on modern CPUs (that is typically at most 80-bit with 64-bit mantissa, or 64-bit with 53-bit mantissa) will not be precise enough that results coincide well-enough for realistic $k_j$; but it can be made to work most often for $k_j$ and shared key of a few bits. Making large shared keys from small ones is easy (just make multiple passes and concatenate); but I doubt we could make large private/public key pairs from small ones.

Arbitrary-precision FP arithmetic is possible, but inneficient. Typically it uses arbitrary-precision integer arithmetic, so we would be back to this.

Note: there are other serious numerical issues, including the risk of overflow, and how to extract the shared key from potentially very large coordinates.



I do not know if there is an additional issue with security.

The problem of solving for integer $k$ given $k\times g$ and $g$ (which breaks ECDH) might not be as hard with coordinates of givens in $\mathbb R$ (as approximated by FP arithmetic) as it is for coordinates in $GF(p)$ or $GF(2^m)$. As an illustration that it can't be ruled out summarily, solving for $k$ given $x^k$ and $x$ is believed hard with givens in $GF(p)$ for appropriate choice of $p$, but is easy with givens in $\mathbb R$ (as approximated by FP arithmetic, and assuming no overflow) because $k=\log(x^k)/\log(x)$ (computed with rounding to the nearest for FP).

Also, there are conflicting interactions between the precision used and security:


Too low a precision on $g$ or/and $k\times g$ restricts the size of $k$ for a unique solution $k$, and short $k$ makes finding $k$ easy; there's no solution to that in sight beyond increasing precision.
Overly increasing precision for $k\times g$ (for constant range of $k$) could conceivably make the problem of finding $k$ easier, as we are getting more information about $k$ from low-order bits/digits of $k\times g$.