Let’s say our experiment is drawing a single card from a shuffled 52-card deck. Our sample space Ω is the set of those 52 physical pieces of cardboard. An outcome ω is a specific card, like the Jack of Hearts. This outcome is clearly not a number—it is an object with a suit, a color, a rank, and a picture on it. Depending on what game we are playing, we will want to extract different numerical information from this exact same card. We do this by defining a function that assigns a real number to every card in the deck. This function is what we call a random variable, denoted by \(X: \Omega \rightarrow \mathbb R\). It is a total function, i.e., it is defined everywhere. If we are playing Blackjack, we choose a random variable X that maps the card to its game value:
\(X(\text{Jack of Hearts}) = 10\)
\(X(\text{3 of Clubs}) = 3\)
\(X(\text{Ace of Spades}) = 11\)
If we are playing a simple Red/Black guessing game, we might define a different random variable Y that acts as an indicator for whether the card is red:
Or if we just care about counting the number of pips (the small suit symbols) printed on the card, we could define Z:
You get the idea. The random variable is not the card itself, nor is the variable “random” in how it behaves—it is a strictly deterministic function. The randomness comes entirely from which card you draw, and the random variable just rigidly translates that random physical outcome into a real number we can do math on. The value that the random variable function \(X\) outputs is called its realization, often denoted by \(x\), i.e., \(X(\omega) = x\). The random variable is often a many-to-one function.
[!IMPORTANT]
One of the main purposes of a random variable is group outcomes into events. If \(X\) is the Blackjack value, then \(X = 10\) groups all four 10s and all twelve face cards together into a single event.
A text book example is often rolling a single die, then \(\Omega = \{1, 2, 3, 4, 5, 6\}\), and the random variable \(X\) is defined as the number of the face-up side, i.e., \(X(\omega) = \omega\), i.e., it is the identity function.
Just like for any other function, while the co-domain of any random variable is \(\mathbb R\), its range is the set of numbers that a specific \(X\) actually outputs. Obviously this will be a subset of its co-domain. The range of a function is also referred to as its image.
One of the main properties of \(X\) is that it must be measurable. What this means is that any set of outcomes that \(X\) groups together must be a valid event in my event space. This does not mean that \(X\) has to cover each and every event, but whatever it does cover, has to be present in \(\mathscr F\). Moreover, \(X\) has to be constant on every atom of \(\mathscr F\). Let’s see this with a toy mathematical example. \[ \Omega = \{ \omega_1, \omega_2, \omega_3 \} \\ \mathscr F = \{ \{\omega_1\}, \{\omega_2\}, \{\omega_3\}, \{\omega_1, \omega_2\}, \{\omega_1, \omega_3\}, \{\omega_2, \omega_3\}, \Omega, \emptyset \} \\ X(\omega_1) = X(\omega_2) = x \quad X(\omega_3) = y \] This means that \(X = x\) covers the event \(\{\omega_1, \omega_2\}\), \(X = y\) covers the event \(\{\omega_3\}\) both of which are valid events in \(\mathscr F\). In this setup \(X\) does not cover events like \(\{\omega_2, \omega_3\}\), but that is fine. However, if we had defined our event space like this instead (verify that it meets all three rules of an event space): \[ \mathscr F = \{ \{\omega_2\}, \{\omega_1, \omega_3\}, \Omega, \emptyset \} \] \(X = x\) covers an event \(\{\omega_1, \omega_2\}\) which does not exist in our event space, which makes it an invalid random variable. Look at it another way, with \(X(\omega_1) = x\) and \(X(\omega_3) = y\), \(X\) is trying to tell these two outcomes apart, but in the event space they are fused together in an atom, they are the smallest resolution at which the event space can distinguish outcomes, a random variable cannot split them. A valid random variable has to be constant on every atom, so something like \(X(\omega_2) = x \quad X(\omega_1) = X(\omega_3) = y\) is valid.
In the last module we took our trace logs and created a sample space that was made of a collection of trace logs in 1-second windows. With that sample space, we can define a random variable \(X(\omega) = count(\omega)\) which measures the number of requests in the 1-second window. The range or image of \(X\) will be \(\{0, 1, 2, \dots, N\}\). With this same sample space, we can define another random variable \(Y(\omega) = distinct(sids)\) which measures the number of unique sessions in the 1-second window. Its range too will be the same as \(X\).
We can create another sample space where we collect all the logs belonging to a single session together. Now all logs in \(\omega\) will have the same Session ID. With this we can define a random variable that measures the click-through, where it takes value \(1\) if there is a Next-Path that is a click event in the session, and \(0\) if there isn’t. This is denoted with an indicator variable \(\mathbb 1_{click}(\omega)\). Its range is \(\{0, 1\}\).
We can also create a sample space made of individual trace logs. With this we can define a random variable that measures latency \(T(\omega) = e - s\). The minimum value will be some theoretical value based on the minimum processing time for the smallest request - say \(l\). The maximum value is interesting. Because of the timeout, the measured maximum will be \(c\), the timeout value, but this does not mean that the actual latency was \(c\), it is possible that the request would have eventually completed, so the real latency is \(\ge c\). When we measure \(T\), we will see the exact value \(c\) appear a bunch of times, other values will be more spread out. The value \(c\) is called an atom, and the act of clipping it at the atom value is called censoring or more precisely right-censoring in this case. The range of \(T\) will be \([l, c]\), but \(c\) is special.
[!CAUTION]
Two kinds of “Atoms”: Be careful not to confuse the two ways we use the word “atom”!
- An atom in an event space \(\mathscr F\) is an indivisible set of outcomes that the event space refuses to split apart.
- An atom of a random variable (like the timeout \(c\)) is a single numerical value in its range that ends up absorbing a concentrated chunk of the total probability, while all the other values are completely spread out.
The astute reader must have noticed that we changed the sample space three times here and it seems to contradict the takeway of the previous module which preached that we should build one rich \(\Omega\) to avoid having to rebuild the entire probability space over and over again. A random variable is the mechanism that helps us keep the sample space unchanging. In the 1-second window example, \(X\) and \(Y\) are both defined on the same 1-second window sample space, making two different measurements from the same outcome. That is the normal case. A new question does not need a new \(\Omega\).
The reason we ended up needing three different sample spaces is that we had three different ideas of what constituted a single outcome, i.e., our unit of observation changed. Latency is a property of one request, but an outcome in the 1-second window space is a whole table of requests, so there is no single number \(e - s\) to measure. We would have had to ask for max latency, or the mean, or the latency of the \(k\)-th row - each a different random variable, and none of them “the latency”. Same for the click-through indicator: a click belongs in a session, and a session’s logs are spread across many 1-second windows. So these are not three views of one experiment, they are three completely different experiments, each with its own \((\Omega, \mathscr F, P)\). Within any one of them, we can define as many random variables as we like without touching \(\Omega\) again.
We say “random variable”, but the variable itself is not random, it has a fixed definite measurement. So where does the randomness come from? It can come from one or more of the following:
A thermometer reading of a room is all three at once — which room, the actual fluctuating temperature, sensor noise — and modeling it well means deciding which dominates.
Statistics of a sample are themselves random variables. The sample mean \(\bar{X}\), an A/B test’s estimated treatment effect, a model’s held-out accuracy — each is a function of a random sample, so each has a distribution of its own, called its sampling distribution. (For instance, a \(p\)-value computed from a continuous test statistic under a simple null is \(\text{Uniform}(0,1)\) — though this exactness relies on the continuity; for a discrete statistic it only holds approximately.)
So far in our discussions we have seen two functions: \[ P: \mathscr F \rightarrow [0, 1] \\ X: \Omega \rightarrow \mathbb R \] \(P\) is a set function, its input is a set (event is a set of outcomes), and \(X\) is single variable function whose input is a single outcome. Let us now define a third function that will give us the probability of seeing a set \(B\) as the output of \(X\). \(B\) can be a single point on the real line, a collection of single points on the real line, a single interval, or a collection of points and intervals, and anything betwixt and in between! \(B\) is called a Borel set, the same Borel set that we encountered in the last module when building the event space for uncountable sample spaces. Except here it is 1-D on the real line. In fact, practically any set on the real line we can physically write down or describe is a Borel set. We can intuit that this third function depends on both \(P\) and \(X\), so let us call it \(P_X\). It can be easily verified that \(P_X(B) = P(\{\omega: X(\omega) \in B\})\). The probability of seeing \(B\) is the same as the probability of the event whose every outcome’s measurement falls in \(B\). In other words, the probability of that part of the domain of \(X\) whose image is \(B\). This part of the domain is also called the pre-image of \(X\) and is written as \(X^{-1}(B)\).
[!WARNING]
Don’t read \(X^{-1}\) as the “inverse of \(X\)”, \(X\) does not have to be invertible for this to work. It simply means the pre-image.
In a weird twist of notation, we suddenly see function composition being used here: \(P_X = P \circ X^{-1}\), in text books this is called the pushforward of \(P\) through \(X\). This new function is called the distribution or law of \(X\). Its range is the same as \(P\)’s range, i.e., \([0, 1]\). Its domain is the monstrous set of all Borel sets on the real line, often denoted as \(\mathscr B(\mathbb R)\). \[ P_X: \mathscr B(\mathbb R) \rightarrow [0, 1] \\ P_X(B) = P(\{\omega: X(\omega) \in B\}) = P(X^{-1}(B)) \\ P_X = P \circ X^{-1} \] All the things we learned about probability over events translate to probability over intervals on the real line. \(P_X\) is a true probability measure, just like \(P\) and follows the same three + one rules:
The value of a singleton value \(x\) is written as \(P_X(\{x\})\) which is the same as \(P(\{\omega: X(\omega) = x\})\).
Just like in \(P\), we have three types of distributions for \(P_X\) as well:
[!IMPORTANT]
We do not have anything analogous to the atomic distribution we saw in the last module where we could make \(\mathscr F\) coarse, and fuse multiple outcomes into a single indivisible event. Here \(P_X\)’s domain is \(\mathscr B(\mathbb R)\) which cannot be coarse, it will always separate singletons. What we can do in with the atomic dart board example from the last module is define \(X\) s.t \(X(L) = 1, X(R) = 2, X(G) = 3\) and turn \(P_X\) into a discrete distribution.
Asking whether a quantity is continuous or discrete is an abstraction over three concrete questions:
Think of the random variable that gives us the height of an individual student, or a random variable that gives us the latency from a single trace log. The student can be \(5.6789'\) or \(6.118'\) or anything in between, the request can take \(0.1\) seconds or \(0.10001\) seconds or anything in between. There is no gap between the allowed values. This is what we mean by continuous. In both these cases, the quantity that we are measuring is truly continuous.
The moment we measure something, we have to use some measuring instrument, and every instrument has a smallest step. A perfcounter only measures up to nanoseconds. A measuring tape only shows inches. Which means that even if the true height is smooth, our recorded number always lands on one of a finite list of values - \(71"\) or \(72"\), never \(71.524"\). Strictly speaking, measured data is always a bit chunky, never perfectly smooth. We usually pretend it’s continuous anyway, because the chunks are so tiny they don’t matter. This can be quantified, i.e., there is a mathematical way to answer whether or not the chunkiness is negligible.
Let’s say the tiniest value my measurement apparatus can measure is \(h\), then according to Sheppard’s correction, my variance will be affected as: \[ \sigma_{obs}^2 \approx \sigma_{act}^2 + \frac{h^2}{12} \] My observed variance will be an overestimate over my actual variance. To figure out whether the chunkiness is negligible, I can compare \(h\) with either \(\sigma\). Let’s say I am measuring heights of some population. The smallest length my scale can measure is \(1\) mm, then \(h = 1\). And the standard deviation of heights is \(\sigma = 2.8" \approx 70mm\) (per National Health Survey), i.e., \(\sigma^2 = 70\;mm * 70\;mm = 4900\;mm^2\), and \(\frac{h^2}{12} = \frac{1}{12} = 0.0833 \;mm^2\). If we compare the two \(\frac{0.0833}{4900} \approx 0.000017 = 0.0017\%\), unarguably tiny!
On the other hand, typical latency dashboards like Grafana lump all requests into a few wide buckets of around 250 ms. It is almost as wide as the values that I care about. Here pretending that things are smooth and reading off a “precise” number off that bucket is clearly misleading.
Sometimes, even if the chunkiness is not very small, we can still pretend that the measurements are smooth depending on the quantity we are interested in. If we are computing averages, chunkiness mostly washes out. But if I am calculating the 99 percentile of latency, where I only have a few data points to begin with, chunkiness can be a real problem.
The first two questions were about the world and about the instrument. This one is about the probability measure itself. The main question to answer here is whether a singleton in \(X\) can carry any probability at all, or do we need a range to get a positive probability, which of course depends on which events in \(X\)’s pre-image carry probability. Let’s take some examples to see whether they can be deemed as continuous, discrete, or mixed.
| Experiment | Quantity | Measurement | \(\mathbf P\) | \(\mathbf {P_X}\) |
|---|---|---|---|---|
| QPS | Discrete | Discrete | Discrete | Discrete |
| Latency | Continuous | Discrete (h small) | Mixed (timeouts) | Mixed |
| Rainfall | Continuous | Discrete (h small) | Mixed (dry days) | Mixed |
| Height | Continuous | Discrete (h small) | Continuous | Continuous |
There are three ways of describing \(P_X\) depending on what \(P_X\) looks like - point masses for a discrete law, a density for a continuous one, and a bit of each for mixed. In any case, CDF always exists and is usually the description we reach for first.
The CDF can be used to describe any type of distribution - be it continuous, discrete, or mixed - because it accumulates all the probability at or below \(x\), and this is always a valid question to ask regardless of the distribution. It is defined as: \[ F(x) = P(X \le x) \equiv P_X\big((-\infty, x]\big) \] It has three properties:
We can figure out the distribution of \(X\) directly from the shape of \(F\).
This is the simplest and the most intuitive way to describe \(P_X\). It is used to represent discrete distributions which are:
It can be represented as a table:
| Realization | \(\mathbf {P_X}\) |
|---|---|
| \(x_1\) | \(p_1\) |
| \(x_2\) | \(p_2\) |
| \(\dots\) | \(\dots\) |
It is of course possible for \(p_i = p_j\), there is nothing that says the probabilities have to be distinct. If it is possible to model the probabilities as a function, i.e., \(p_X(x) \equiv P(X = x) \equiv P_X(\{x\})\), then \(p_X(x)\) is called the probability mass function. It is worth repeating that the realization \(x\) can be the output of multiple outcomes, and \(p_X(x)\) is the probability of the event consisting of all these outcomes, i.e., probability that any one of the outcomes happens. Because \(P_X\) and \(p_x\) are both functions that output probability, it is easy to conflate the two. \(p_X\) acts on single real numbers, whereas \(P_X\) acts on Borel sets made of real numbers. They are not the same. \(p_X(x)\) is defined as the value of \(P_X\) acting on a singleton \(P_X(\{x\})\), which is to say that they give the same output when we pass a singleton set \(\{x\}\) to \(P_X\).
A density function is used to represent continuous distributions. Probability density function is very much like physical density (mass per volume). Given a volume V, we can integrate the density from 0 to V to get the mass of that body. Given a range of values between \(x_1\) and \(x_2\), we can integrate the probability density function to get the probability that the realization will lie between these two values. \[ M = \int_0^V \rho(v)dv \]
\[ P(x_1 < x < x_2) = \int_{x_1}^{x_2} f(x)dx \]
As discussed earlier, the probability of any single realization \(x_0\) is \(\int_{x_0}^{x_0} f(x)dx = 0\), but it does not mean that it is an impossibility.
Density functions are themselves not probability, only their integrals are, which is why it is possible for density for some values to be greater than \(1\). Only the integral of the density needs to be bounded by \(1\). We can think of the density as the units of probability per unit of \(X\).
Support of \(P_X\) is often denoted as \(supp(X)\). Loosely speaking it is the set of values of \(X\) which have a positive probability. In the die-roll example (yes, we are back to that to firmly understand this concept), \(P_X(\{1\}) = P_X(\{2\}) = \cdots = P_X(\{6\}) = \frac{1}{6}\). So the set \(\{1, 2, \dots, 6\}\) can be considered as \(supp(X)\). But \(P_X(\{5, 6, 7, 8\}) = \frac{1}{3}\) also has a positive probability, and so does \(P_X([0.5, 1.5]) = \frac{1}{6}\). Why don’t we consider these sets to be part of the support? Let’s refine our understanding a little bit - support is a set of real numbers, it cannot have other sets as elements, so in the die-roll case we only consider singletons that have positive mass as part of the support. But then why have all the \(6\) numbers as part of the set? To answer this lets make our definition of support a little bit more precise - it is the set whose individual points have a positive probability and whose probability is \(1\), i.e., it includes all such points, i.e., \(P_X(supp(X)) = 1\). This explains why we need all \(6\) numbers. But why don’t we have \(supp(X) = \{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10\}\) ? This set also has a probability of \(1\). This brings us to yet another precision in our understanding, it must be the smallest such set. So far we have - support of \(X\) is the set of all real numbers which have a positive probability. This covers all the requirements we have seen so far - it is a set of real numbers, the probability of the full set is \(1\) because we are gathering all such real numbers, and it is the smallest set because we are excluding all points that do not have positive probability.
For a discrete distribution with a finite number of points, it is clear to see that \(supp(X)\) is the set of all points with positive mass. But this is true of countably infinite points as well. Consider the first-heads experiment, where \(X\) can be defined as the number of coin-flips, its image is \(\{1, 2, \dots\}\), and the probability is \(P_X(\{x\}) = \frac{1}{2^x}\). This entire set of positive natural numbers is the support, even though it does not seem to have any end. I can pick any number from this set, however large it may be, and I’ll always be able to get a positive probability for it. In fact the support of a discrete distribution will always be countable.
So far, so good, this draft definition works in the case of discrete distributions. What about continuous distributions? Let’s consider the following uniform distribution density function: \[ f_X(x) = \begin{cases} \frac{1}{10}, & 10 \le x \le 20 \\ 0, & \text{otherwise} \end{cases} \] No single point has a positive probability! Lets amend our understanding one more time. It is not the values of \(X\) that have a positive probability that are part of the set, it is actually those values of \(X\) whose tiny neighborhood captures some positive probability. In this case a tiny neighborhood around \(10\), say \([9.99995, 10.00005]\) captures a positive probability, but I can construct a tiny neighborhood around \(9.99995\), that does not have a positive probability, so we exclude that. The support is \([10, 20]\). What if the density was defined for the open interval as follows: \[ f_X(x) = \begin{cases} \frac{1}{10}, & 10 < x < 20 \\ 0, & \text{otherwise} \end{cases} \] I can still construct an arbitrarily small neighborhood around \([9.99995, 10.00005]\) which will capture a positive probability. The support is still \([10, 20]\). Our new and complete definition is - support of \(X\) is the set of all real points where every arbitrarily tiny neighborhood around the point captures a positive probability.
For continuous distributions with density \(f_X\), we can be tempted to think of the support as all points with positive density. But this is incorrect. Consider a function \(g\) which is almost similar to \(f_X\) : \[ g(x) = \begin{cases} \frac{1}{10}, & 10 \le x < 15 \\ 0, & x = 15 \\ \frac{1}{10}, & 15 < x \le 20 \\ 0, & \text{otherwise} \end{cases} \] The dip at \(x = 15\) does not change the value of the integral for any interval. \(\int_5^{17} f_X(x)\;dx = \int_5^{17} g(x)\;dx\). In fact I could set this value to \(g(15) = 9999\), and it still wouldn’t change the integral. I could go so far as to change the value of the function at any number of isolated points, each with length \(0\) and it will not change the value of the integral. Or looking at it another way, any tiny neighborhood around \(15\) will always capture a positive probability, even though \(15\) itself has \(0\) density. The support is still \([10, 20]\).
In Measure Theory, these \(f\) and \(g\) are considered the same function “defined up to null sets”. Null sets are sets of points that have zero length, area, or volume. Insetead of saying “all points with positive density”, if we take all such points and then “close” that set, we have our support, \(supp(X)=\overline{\{x:f_X(x)>0\}}\). The overbar is the “closure” operation which produces a closed set.
This idea of having points with \(0\) probability being part of the support as long as any and every tiny neighborhood around it has positive mass applies to discrete distributions as well. Consider the server-retry scenario we saw in the last module. \(X\) can output any one of \(\{\frac{1}{2}, \frac{1}{4}, \frac{1}{8}, \dots\}\) with each value being its own probability, i.e., \(P_X(\{x\}) = x\). Here the support is \(\{0, \frac{1}{2}, \frac{1}{4}, \cdots\}\). \(0\) is part of the support, even though the probability mass at \(0\) is \(0\). This is because every small neighborhood around \(0\) will include some number \(\frac{1}{2^n}\), which has a positive mass.
In Math and Stats text books we’ll come across this definition - support of \(X\) is the smallest closed set of points - \(\mathbf S\), for which \(\mathbf {P_X(S) = 1}\). This is equivalent to the one we came up with. One last point on “smallest” and “closed”. Lets take our uniform distribution with support at \([10, 20]\). I can pluck a singleton out of it, e.g., \([10, 20] \setminus \{15\}\), this is smaller than \([10, 20]\) and its probability is still \(1\). In fact I can take any number of such singletons out, or I can take out all the rational numbers, or any other finite set, and end up with a smaller set whose integral is \(1\). Requiring closedness blocks exactly these kinds of deletions and makes “smallest” well-defined.
\[ E[X] = \sum_x x \; p(x) \\ E[X] = \int x \; f(x)dx \]
Both the summation and integral are over \(supp(X)\), i.e., the range where probability exists.
For the mixed case it would look something like - \[ E[T] = \int_l^c t\;f(t)dt + c \cdot P(T=c) \]
Notice that this is exactly \(E[\min(T, c)]\) — the expectation of the recorded (censored) variable, not of the true latency. Every timed-out request contributes \(c\) rather than its true, larger value, so this mean is biased low — the same downward drag the light-bulb study warned about. Recovering \(E[T]\) for the true latency needs a model that credits the fact that \(T > c\) on those requests; the censored sample does not hand it to you directly.
Variance is the squared distance from the mean:
\[ \begin{align} Var[X] &= E[(X - E[X])^2] \\ &= E[X^2] - (E[X])^2 \end{align} \]
And it is computed with either a summation or an integration depending on \(X\).