We'll start with our definition of a path and our definition of asymptotic normality. Using just these definitions and some esoteric theorems, we'll amazingly be able to characterize how our estimator should behave as we move along the path towards $P$. That's a little surprising because asymptotic normality is a property that only holds at $P$ and doesn't say anything explicit about behavior along paths. Nonetheless, we'll see that there is an implication for how the estimator behaves along paths. However, we've also assumed the estimator is also regular, which is already a definition of how the estimator behaves along paths. In comparing the derived behavior from asymptotic normality and the assumed behavior from regularity, we will see there is a difference. The difference in the two behaviors can only be made to go away (as it must if an estimator is both asymptotically normal and regular) if $\nabla_h \psi = E[h\phi]$, so that's what we conclude.

In proving that asymptotic normality and our definition of a path by themselves imply some behavior of the estimator along such a path we'll have to use two advanced, technical results. Unfortunately I haven't found an alternative way to prove this that is both rigorous and intuitive. It's not at all impossible to understand these theorems (just read the appropriate sections of vdW 1998) but if you're not interested in the math for its own sake it's probably not worth your time. In terms of understanding the arc of the proof you just need to grasp how we've used these tools to get what we need, not how they work.

Throughout I'll abbreviate $\hat\psi_n =\hat\psi(\mathbb P_n)$ (the estimate as we draw increasing samples from $P$) and $\psi = \psi(P)$ (the true parameter at $P$) to keep the notation light. I'll also abbreviate $\hat{\tilde\psi}_n =\hat\psi(\tilde{\mathbb P}_{1/\sqrt n})$, which is the estimate as we draw increasing samples from distributions moving along our path closer to $P$, and $\tilde\psi_n = \psi(\tilde P_{1/\sqrt n})$, which are the true parameter values at each of these distributions. Anything that has a hat is an estimate, anything that has a tilde is along a path.

Alright, go time. Let's see what we can say about how our estimator behaves along sequences of distributions like $\tilde P_n = \tilde P_{\epsilon = 1/\sqrt{n}}$, since these are the only ones regularity has anything to say about. By theorem 7.2 of vdV 1998 the fact that our paths are differentiable in quadratic mean ensures that we get something called local asymptotic normality 🤷:

It really doesn't matter what this means because we're just going to use it to satisfy a particular technical condition in a minute. The important part is to see that we've defined some random variable on the left: the derivative-looking thing is just some fixed function of the data, so let's rename that as some random variable $T_n$. What the right hand says is that in large-enough samples, this $T_n$ thing looks like a sum of IID variables plus some constant. We combine this with the assumed asymptotic linearity of our estimator:

Which is also a random variable that's approximately equal to an IID sum when $n$ gets big. Here we stack the two above equations on top of each other and by the central limit theorem we have that

Now we use another technical result 🧐 (Le Cam's 3rd lemma- see vdV 1998 ex. 6.7), which says that when we have exactly the situation above, we can infer

Again it's not important to understand the technical device. The idea is that we're trying to say something about how our estimator behaves as we change the distribution along our path of interest. At first this seems impossible because asymptotic normality only holds at each $P$ and doesn't say anything about what happens along paths. However, with the help of these technical devices, we've actually managed to say something about the difference between the estimate as we change the underlying distribution and the truth at $P$. In particular, this difference converges to a normal with the same variance as if we had not been moving along the path towards $P$ but had instead sat still at $P$. Crazy. The limiting normal distribution now has some mean $P[h\phi]$, which we'll pull out in the course of some algebraic manipulation during which we also add and subtract $\sqrt n \tilde\psi_n$ on the left:

Moving terms around, we arrive at

Notice that the term in brackets on the left is a constant for each $n$, i.e. it's not random.

Now, finally, we recall our definition of regularity, which was

An estimator that is asymptotically linear must satisfy the behavior in the display above denoted (AL). An estimator that is regular must satisfy the behavior in the display denoted (R). A RAL estimator must satisfy both. But the only way that both of those can be true is if they are actually saying the same thing, which only happens if the bracketed term in (AL) goes to zero. Thus, for RAL estimators,

where we've recalled $\epsilon_n = 1/\sqrt{n}$ from our original definition of our sequence along the path.

We can treat any score $h$ and any influence function $\phi$ as an element of a set we call $\mathcal L_2(P)$ (or just $\mathcal L_2$ when the measure $P$ is clear). This is the set of all functions $f(Z)$ which satisfy the condition $\int f(Z)^2 dP < \infty$ (i.e. $f(Z)$ is a random variable with finite variance). This space is a lot like the vector space $\mathbb R^p$ in many important ways:

Together, these conditions are the definition of something called a Hilbert space. Indeed, $\mathbb R^p$ and $\mathcal L_2$ are the usual examples of Hilbert spaces. The main difference between the two is that $\mathbb R^p$ is finite-dimensional, whereas $\mathcal L_2$ is infinite-dimensional. What do I mean by this? Well, a vector in, say $\mathbb R^3$ clearly has 3 components $[x_1, x_2, x_3]$. A "vector" in $\mathcal L_2$ has as many components as its functions have arguments because you can think of a function $f:\mathbb R \rightarrow \mathbb R$ like this: $f = [\dots f(-200) \dots f(-0.1) \dots f(0) \dots f(1) \dots f(1.5) \dots ]$. The only difference is that I put the "vector index" in parentheses instead of as a subscript and now I call it an "argument". We also now allow for indices that are anywhere in the domain of $f$ instead of just the integers $1,2,3$. Alternatively, we can think of the vector $x$ as a function that maps $i \in \{1,2,3\} \mapsto x_j$. So vectors, functions, whatever. It's kind of the same thing in most of the ways that matter!

Honestly, I don't think it's the greatest name, but let's explain it.

The key is to 1) think about each point $P \in \mathcal M$ as having density $p$ w.r.t. some dominating measure $P\degree$ and 2) realize that $p \in \mathcal L_2$. Because $p$ has to integrate to 1, then then norm of $\sqrt{p}$ (that is, $E_{P\degree}[{\sqrt p \sqrt p}])$ is of course 1. So we can identify $\mathcal M$ with some subset of the unit ball in $\mathcal L_2(P\degree)$. It's now easy to show that $\sqrt p$ is orthogonal to $h \sqrt p$:

$\int \sqrt p (\sqrt p h) dP\degree = \int h p dP\degree = \int hdP = 0$

so the set of functions $h\sqrt p$ ranging over scores $h$ is orthogonal (tangent) to the point $\sqrt p$ in the model space. Or, equivalently, $p$ is orthogonal to all scores $h$ in the space $\mathcal L(P\degree)$.

Once nice thing that this picture shows is that the tangent space clearly depends on where $P$ is within the model.

Nonetheless, it'd probably be more informative to call it the "score space", since the fact that the scores are tangent to the density $p$ in $\mathcal L_2(P^\degree)$ doesn't seem matter that much in terms of the role the space ends up playing in the theory we're developing. Alas, we're stuck with the names we have.

Recall that $h = \frac{d}{d\epsilon} \log \tilde p_\epsilon \big|_{\epsilon=0}$. For $\epsilon$ small enough $\tilde p_\epsilon$ is still in the model, so we can factor it into $\tilde p_{Z_2|Z_1} \tilde p_{Z_1}$(omitting the $\epsilon$ subscripts). The log of a product is the sum of logs and the derivative is linear over a sum, so we get $h(Z_1, Z_2) = \underbrace{ \frac{d}{d\epsilon} \log \tilde p_{Z_2|Z_1} \big|_{\epsilon=0} }_{h_2(Z_1, Z_2)} + \underbrace{ \frac{d}{d\epsilon} \log \tilde p_{Z_1} \big|_{\epsilon=0} }_{h_1(Z_1)}$

Now we'd like to show that $h_2 \perp h_1$ in $\mathcal L_2(P)$.

For that we have to notice that $h_2$ is a score at $p_{Z_2|Z_1}$ in the nonparametric model $\mathcal M_{Z_2|Z_1} = \{\text{all densities of $Z_2$ given $Z_1$\}}$ (by definition). If we start at $p_{Z_2|Z_1}$ and move along a path defined by $h_2$, the only way we stay within $\mathcal M_{Z_2|Z_1}$is if $E[h_2|Z_1] =0$. Otherwise the resulting perturbation will not be a density. Similarly, we need for $E[h_1] = 0$.

Now $E[h_1 h_2] = E[E[h_1 h_2|Z_1]] = E[h_1(Z_1) E[h_2|Z_1]] = E[h_1 \cdot 0] = 0$ and we've shown $h_1$ and $h_2$ are orthogonal.

We can also go the other way- if I propose $h_1$ and $h_2$ that satisfy the above, then $h = h_1 + h_2$ must be a valid score at $P$ in the original model.

We showed that the tangent space can be broken up into an orthogonal sum when every distribution in the model factors and those factors can vary independently in their own model spaces. There are cases where this breaks down, though. If you look at the following picture, you'll surmise that all the scores at $P$ (blue arrows) can indeed be created as orthogonal sums of scores from $\mathcal M_{Z_2|Z_1}$ and $\mathcal M_{Z_1}$. However, there are arrows that can be constructed the same way (grey, dotted) that take us outside of the model space- these are not scores because they don't correspond to a legal path inside the model. Therefore in this case $\mathcal T \subset \mathcal T_1 \oplus \mathcal T_2$, but we don't have the equality. When this happens, we say that our submodels are not variationally independent. In other words, there are some points in the model space where I can't legally change in one of the submodels without having to make a change in another (i.e. at $P$ in the picture we can't go just up, we also have to move a bit to the right along the diagonal).

There are combinations of parameters and models for which the pathwise derivative doesn't exist or isn't a bounded linear operator. For example, consider densities of $Y,X$ and let our parameter be $\psi(P) = E[Y |X=x_0]$ for a particular point of interest $x_0$. You can evaluate the derivative and show that $\nabla_h \psi(P) = \int y h(y,x_0)p_{Y|X}(y,x_0) dy$. The problem here is that I can always pick $h$ with norm 1 but that takes huge values at $x=x_0$ (by compensating with large negative values at other $x$). So I can pick $h$ (in the unit ball) such that the integral blows up as large as I want. That's what "unbounded" means for a linear operator. The consequence of this is that the Reisz representation theorem no longer applies and we cannot guarantee that there is an element $\phi^\dagger$ that is in the tangent space. This also destroys the argument based on the difference of two influence functions of RAL estimators because we can't guarantee that even a single RAL estimator exists.

Besides giving us a definition of orthogonality, Hilbert spaces are super nice to work in because of an important property called Reisz representability. The Reisz representation theorem says that if you have a bounded linear function $F$ that maps an element of the Hilbert space to a real number, then that function can always be represented as an inner product between the argument to the function and some other element of the Hilbert space. In other words, for every bounded linear function $F$, there is some element $h_F$ so that $F(h) = \langle h_F, h \rangle$.

This is a very surprising result at first but it's pretty easy to convince yourself of it in $\mathbb R^p$. Consider vectors $\vec x, \vec y \in \mathbb R^p$ (I'll use the vector arrow $\vec x$ in this section to be very explicit). The definition of a linear function $F: \mathbb R^p \rightarrow \mathbb R$ is that $F(\vec x + \alpha \vec y) = F(\vec x) + \alpha F(\vec y)$ for any scalar $\alpha$. Any linear function in this particular space is bounded so we don't need to worry about what that means. To show that $F(\vec x)$ is actually just taking an inner product between $\vec x$ and some other vector we can apply $F$ to the unit vectors $\vec e_{1 \dots p}$ and see what it returns. Say $F(\vec e_i) = a_i$. Now we can express any vector as the weighted sum of the unit vectors $\vec x = \sum x_i\vec e_i$, and by linearity of $F$ notice now that $F(\vec x) = \sum x_iF(\vec e_i) = \sum x_i a_i = \langle \vec x, \vec a\rangle$. The conclusion is that the operation of the function $F$ on $\vec x$ is really just the inner product between $\vec x$ and some other vector $\vec a$, which is also in $\mathbb R^p$. The other direction is also obvious: given $\vec a$, define $F_{\vec a}(\vec x) = \langle \vec a, \vec x\rangle$ and since the inner product is a linear operation in both arguments, we have that the function is linear as desired.

What's amazing is that this property actually extends to any Hilbert space (i.e. a complete linear space with an inner product). In particular, if we have a bounded linear function $F$ that maps functions $f \in \mathcal L_2$ to real numbers, then we know there is some fixed function $g \in \mathcal L_2$ such that $F(f) = E[fg]$. If you're unfamiliar with functions that have functions as arguments you can think of $F$ as "assigning" a number to each function $f$.

Almost all Hilbert spaces contain other Hilbert spaces within them. For example, we can think of any plane in 3D space as a 2D space in its own right. If we add the restriction that the contained space must have the origin in it, we call it a subspace and a useful result follows. Namely: for every vector $v$ in the Hilbert space $\mathcal H$, there is always exactly one vector $v_*$ in the desired subspace $\mathcal H_*$ and a vector $v^\perp$ that's orthogonal to every vector in $\mathcal H_*$ such that $v = v_* + v^\perp$. Since the projection is unique, it's easy to check if $v_*$ is indeed a projection by simply checking (1) whether $v_* \in \mathcal H_*$ and (2) whether $v - v_* \perp \mathcal H_*$.

If a subspace is the orthogonal sum of two other subspaces, we can always obtain the projection by projecting into each sub-subspace and then summing the result.

If we have a function $f:\mathbb R^p \rightarrow \mathbb R$ that maps vectors to real numbers, the directional derivative at $\vec x$ in the direction of a vector $\vec h$ is defined as $\nabla_hf(\vec x) = \lim_{\epsilon \rightarrow 0} \frac{f(\vec x + \vec h) - f(\vec x) }{\epsilon}$. However, it's well-known that we can also write this as the sum of the partial derivatives times the components of $\vec h$: $\nabla_{\vec h} f(\vec x) = \frac{\partial f(\vec x)}{\partial x_1} h_1 + \dots + \frac{\partial f(\vec x)}{\partial x_p} h_p$. In fact, the proof of this follows from noticing that the directional derivative is a linear operator and applying the Reisz representation theorem that we derived for finite dimensional $\mathbb R^p$. Of course, this is the same as the inner product between the vector of partial derivatives, which we call the gradient $\nabla f(\vec x) = [\partial f(\vec x)/\partial x_1 \dots \partial f(\vec x)/\partial x_p]$, and $h$. Thus $\nabla_{\vec h} f = \vec h \cdot \nabla f = \langle \vec h, \nabla f \rangle_{\mathbb R^p}$.

Now it should make sense why we refer to $\phi^\dagger$ as a gradient. It's exactly the same as $\nabla f$, except now we're in $\mathcal L_2^0$ instead of $\mathbb R^p$. Since we can write the directional derivative $\nabla_h \psi(P)$ as an inner product $E[h, \phi^\dagger] = \langle h, \phi^\dagger \rangle_{\mathcal L_2}$, the object $\phi^\dagger$ is serving exactly the same role as the gradient $\nabla f$ is above. To reiterate:

We call $\phi^\dagger$ the canonical gradient because there are usually many functions in $\mathcal L_2$ that satisfy the above (namely: every other influence function). However, $\phi^\dagger$ is the only one that is in the tangent space, and is the only one that's guaranteed to exist according to the Reisz representation theorem. That's what makes it "canonical".

In the context we've described here it might make more intuitive sense to use the notation $\nabla \psi$ to represent the canonical gradient. However, we use $\phi^\dagger$ to make the connection to influence functions for RAL estimators. A gradient (depends on the parameter) and an influence function (depends on the estimator) are actually totally different things. It just so happens that for RAL estimators they happen to occupy exactly the same space.