Under which circumstances does a set $X\subseteq{\bf R}^n$ has the property that for all $f:X\rightarrow{\bf R}^m$ and $A,A'\in{\bf R}^{n\times m}$ with $$\lim\_{x\rightarrow 0,x\in X}\frac{f(x)-f(0)-A(x)}{\|x\|}=\lim\_{x\rightarrow 0,x\in X}\frac{f(x)-f(0)-A'(x)}{\|x\|}=0$$ we have $A=A'$, i.e. the structure of the domain allows to extend the definition of derivatives to non-interiour points (here I considered only $0$) in a way that makes sense (forces uniqueness).

I guess we can w.l.o.g. assume $m=1$ here...

Edit: I just saw that we don't need to consider any functions $f$, it suffices to just state $$\lim\_{x\rightarrow 0,x\in X}\frac{A(x)}{\|x\|}=0$$ implies $A=0$.

Some examples: It's enough if $X$ lies dense around $0$, i.e. $0$ lies in the ineriour of the closure of $X$. Or if $v\_0,\ldots,v\_{n-1}$ are linerarly independent vectors and $X=\cup\_{i<n} {\bf R}v\_i$.