The History of Logit

Data Science

A brief history of the Logit Model

Author

Pingfan Hu

Published

May 20, 2024

Useful Links

Research Paper

The Origins of Logistic Regression

Lecture Slides

Intro to Choice Modeling

Online Article

What is Logit (Chinese)

Abstract

This blog describes a brief history of the Logit Model. It starts from the famous Malthus population problem, and explains the necessity of transforming from probability to logit.

The Malthus Population Problem

1. The Malthusian Growth Model

In the biology class of my high school, I learned that the population will grow exponentially given unlimited resources and spaces. This comes from Malthus (1798) and is called the “Malthus Population Problem” and was give the name “J Curve”, since the exponential Malthusian Model looks like the English letter J.

The Malthusian Growth Function:

\[ N_t = N_0 \cdot e^{rt} \] Where:

$K$ (not in this equation but will be used later) stands for the carrying capacity, which is the maximum allowed population in the system.
$N_t$ stands for the population size at time of $t$,
$N_0$ stands for the initial population size, and
$r$ stands for the growth rate.

The Malthusian Growth Model:

Code

# Parameters
K <- 10000
N_0 <- 100
r <- 1

# Calculation
x <- seq(0, 10, length.out = 400)
N_t_malthus <- N_0 * exp(r * x)
data_malthus <- data.frame(Time = x, Population = N_t_malthus)

# Plotting
ggplot(data_malthus, aes(x = Time, y = Population)) +
  geom_line(color = "#5654A2") +
  labs(title = "Malthusian Growth Model",
       subtitle = expression(
         paste("Parameters: ", K, "=10000, ", N[0], "=100, ", r, "=1")
         ),
       x = "Time (t)",
       y = expression(paste("Population (", N[t], ")"))) +
  ylim(0, 10000) +
  theme_bw(base_family = 'Ubuntu') +
  theme(plot.subtitle = element_text(hjust = 0.5, size = 10),
        plot.title = element_text(hjust = 0.5, size = 14))

In this model, we set a simple scenario of $K$ = 10,000, $N_0$ = 100, and $r$ = 1. It can be seen that the population grows so fast, that it reaches the cap while $t$ value is a little over 4.

2. The Logistic Growth Model

However, this exponential model is unrealistic due to the scarcity of resources and spaces. The Belgian statistician Adolphe Quetelet knew that extrapolating exponential growth leads to impossible values, and asked his pupil Pierre-François Verhulst to look into the problem. Verhulst added a term for the resistance to further growth, and the result is an S curve with a cap. He published it in three papers between 1838 and 1847. This S curve is perhaps the first S-shaped curve that we could learn in our textbooks.

Verhulst named the curve “logistique” in his 1845 paper (Verhulst 1845) and never explained the name (Cramer 2002). It is likely from the Greek “logistikos” (skilled in calculation), chosen by analogy with “arithmetic” and “geometric” growth, and in contrast to the “logarithmic” curve (what we now call the exponential) that he drew next to it. It is not related to the military word “logistics”, which comes from the French “logis” (lodging). His work was then forgotten. Pearl and Reed rediscovered the same curve in 1920 while studying the US population, and the name “logistic” was revived in the 1920s.

The “Logit Model” came a century later. Joseph Berkson (1944) proposed the logistic function as an alternative to the normal-based “probit” in bio-assay, and coined the term “logit” for the log of odds.

The word “logit” is a portmanteau, coming from logistic + unit. Berkson coined it in 1944 by analogy with “probit” (probability + unit).

The Logistic Growth Function:

\[ N_t = \frac{K}{1 + \left( \frac{K - N_0}{N_0} \right) e^{-rt}} \]

The Logistic Growth Model:

Code

# Calculation for Logistic Growth Model
N_t_logit <- K / (1 + ((K - N_0) / N_0) * exp(-r * x))
data_logit <- data.frame(Time = x, Population = N_t_logit)

# Plotting the Logistic Growth Model
ggplot(data_logit, aes(x = Time, y = Population)) +
  geom_line(color = "#5654A2") +
  labs(title = "Logistic Growth Model",
       subtitle = expression(
         paste("Parameters: ", K, "=10000, ", N[0], "=100, ", r, "=1")
         ),
       x = "Time (t)",
       y = expression(paste("Population (", N[t], ")"))) +
  ylim(0, 10000) +
  theme_bw(base_family = 'Ubuntu') +
  theme(plot.subtitle = element_text(hjust = 0.5, size = 10),
        plot.title = element_text(hjust = 0.5, size = 14))

This is the “S Curve” we’ve been seen in a lot of scenarios where there is a designed cap of population due to resource scarcity. There are also other S shaped curves, such as the normal CDF used by the probit model. Divide the population by $K$ and the logistic curve becomes a proportion between 0 and 1, which is the same function the logit model uses for probability.

The CDF of the logistic distribution is this S curve. With a visible diminishing of returns, it is a perfect model for human behavior studies for the purpose of generating a feasible marketing strategy.

The Logit Model

1. Probability vs Logit

In choice modeling, the utility of an alternative has an observed part $v_j$ and a random error $\varepsilon_j$:

\[ u_j = v_j + \varepsilon_j \]

The probability of choosing $j$ over $k$ is the probability that $u_j > u_k$. To compute it, we need to assume a distribution for the errors. If the errors are normally distributed, we get the Probit Model, which is supported by theory but has no closed form solution for the probability. If the errors follow the Type 1 Extreme Value distribution, we get the Logit Model, which has a closed form:

\[ P_j = \frac{e^{v_j}}{e^{v_j} + e^{v_k}} \]

Here the logarithm of the odds is simply the utility difference: $\log(P_j / P_k) = v_j - v_k$.

This is again the probability of choosing $j$ over $k$. To generalize the equation, the probability of choosing $j$ over all other alternatives is written as this:

\[ P_j = \frac{e^{v_j}}{\sum_{k=1}^{J} e^{v_k}} \]

This multinomial form was first suggested by Cox (1966) and Theil (1969). Daniel McFadden (1973) linked it to the theory of discrete choice from mathematical psychology, Luce’s choice axiom, which gave the logit model its theoretical foundation and earned him the Nobel Prize in 2000.

The logit model requires the errors to be independent and identically distributed. As a result it has the IIA property (Independence of Irrelevant Alternatives): the ratio $P_j / P_k$ does not depend on what other alternatives are available.

IIA can be problematic when products are close substitutes. The classic example is “Red Bus vs Blue Bus”. Suppose the choice is between Taxi and Red Bus, with probabilities of 0.66 and 0.33. Now add a Blue Bus that is identical to the Red Bus except for the color. We expect 0.66, 0.165, and 0.165, since the two buses are in fact one alternative. But IIA keeps the ratio between Taxi and Red Bus at 2:1, so the logit model gives 0.50, 0.25, and 0.25.

2. Why Logit

So, why using Logit Model? Why bother transforming the simple probability value into a somewhat non-intuitive value? An important reason is that the logit value does not have lower and upper limits, so it can be modeled as a linear function of utilities. This can be better explained by the Logit Transformation:

\[ \text{Probability} \rightarrow \text{Odds} \rightarrow \text{Logit} \]

In statistics, both probability and odds are used to describe the likelihood of an event occurring, but they have an important difference:

Probability refers to the ratio of the number of times an event A occurs to the total number of possible outcomes, while
Odds refers to the ratio of the probability of an event occurring to the probability of the event not occurring.

The probability equation writes:

\[ P(A) = \frac{\text{Number of Event A}}{\text{Total Number of Events}} \]

The probability $P$ is a real number between 0 and 1. $P = 0$ indicates that an event will definitely not occur, while $P = 1$ indicates that an event will definitely occur. In the case of rolling a die, the probability of rolling a 6 is:

\[ P = \frac{1}{6} \]

The odds equation writes:

\[ \text{Odds} = \frac{\text{Probability of event}}{\text{Probability of no event}} = \frac{P}{1 - P} \]

Continuing with the example of rolling a die, the probability of rolling a 6 is $P = \frac{1}{6}$, and the probability of rolling any other number is $1 - P = \frac{5}{6}$. According to the equation, the odds of rolling a 6 is:

\[ \text{Odds} = \frac{1/6}{5/6} = \frac{1}{5} \]

Now you can see: The ratio of the probability of successfully rolling a 6 to that of not is 1:5.

Like many other concepts in probability theory, the idea of odds originates from gambling. Suppose two people, A and B, bet on rolling a die; if A bets 1 dollar on rolling a 6, B would need to bet 5 dollars to ensure a fair game.

$1/6$ and $1/5$ is not a big difference, but the point is to show that probability is happening vs all, while odds is happening vs not happening.

Imaging another scenario of drawing a ball from a packet, where there are 3 red balls and 2 blue balls. The probability of drawing a red ball is $3/5$, while the odds is $3/2$. This time, we have a larger difference, and the odds is even grater than one. In fact, the value of odds can be from 0 to positive infinity. Till now, we transformed from probability to odds.

The next transformation is from odds to logit. This can be simply done by taking the log value of odds:

\[ \text{Logit} = \log(\text{Odds}) \]

Therefore, the full Logit Transformation is:

\[ \text{Probability} = P \rightarrow \text{Odds} = \frac{P}{1-P} \rightarrow \text{Logit} = \log(\text{Odds}) \]

To better explain the differences between these 3, I constructed a data frame and sketched 2 plots of P vs Odds and P vs Logit:

P	Odds	Logit
0.01	0.01	-4.60
0.10	0.11	-2.20
0.20	0.25	-1.39
0.30	0.43	-0.85
0.40	0.67	-0.41
0.50	1.00	0.00
0.60	1.50	0.41
0.70	2.33	0.85
0.80	4.00	1.39
0.90	9.00	2.20
0.95	19.00	2.94
0.99	99.00	4.60

A summary of the transformation:

Probability values range from $0$ to $1$.
Odds values range from $0$ to $+\infty$.
Logit values range from $-\infty$ to $+\infty$.

References

Berkson, Joseph. 1944. “Application of the Logistic Function to Bio-Assay.” Journal of the American Statistical Association 39 (227): 357–65. https://doi.org/10.1080/01621459.1944.10500699.

Cramer, J. S. 2002. The Origins of Logistic Regression. Tinbergen Institute Discussion Paper 2002-119/4. Tinbergen Institute. https://doi.org/10.2139/ssrn.360300.

McFadden, Daniel. 1973. “Conditional Logit Analysis of Qualitative Choice Behavior.” In Frontiers in Econometrics, edited by Paul Zarembka. Academic Press.

Verhulst, Pierre-François. 1845. “Recherches mathématiques sur la loi d’accroissement de la population.” Nouveaux Mémoires de l’Académie Royale des Sciences et Belles-Lettres de Bruxelles 18: 1–38.

--- title: "The History of Logit" image: "/blog/banners/2024-05-20-history-of-logit.png" date: "2024-05-20" categories: [Data Science] code-fold: true code-tools: true description: "A brief history of the Logit Model" bibliography: references.bib --- ```{r} #| label: setup #| include: false library(tidyverse) library(kableExtra) library(here) ``` ## Useful Links <div class="lnk-outer"> <div class="lnk-row lnk-purple"> <div class="lnk-label">Research Paper</div> <div class="lnk-chips"> <a href="https://papers.tinbergen.nl/02119.pdf" class="lnk-btn" target="_blank">The Origins of Logistic Regression</a> </div> </div> <div class="lnk-row lnk-blue"> <div class="lnk-label">Lecture Slides</div> <div class="lnk-chips"> <a href="https://hw.pingfan.org/emse_6035_hw/choice_modeling_slides/1_choice_modeling.pdf" class="lnk-btn" target="_blank">Intro to Choice Modeling</a> </div> </div> <div class="lnk-row lnk-green"> <div class="lnk-label">Online Article</div> <div class="lnk-chips"> <a href="https://zhuanlan.zhihu.com/p/27188729" class="lnk-btn" target="_blank">What is Logit (Chinese)</a> </div> </div> </div> ## Abstract This blog describes a brief history of the Logit Model. It starts from the famous Malthus population problem, and explains the necessity of transforming from probability to logit. ## The Malthus Population Problem ### 1. The Malthusian Growth Model In the biology class of my high school, I learned that the population will grow exponentially given unlimited resources and spaces. This comes from Malthus (1798) and is called the "Malthus Population Problem" and was give the name "J Curve", since the exponential Malthusian Model looks like the English letter J. The Malthusian Growth Function: $$ N_t = N_0 \cdot e^{rt} $$ Where: - $K$ (not in this equation but will be used later) stands for the carrying capacity, which is the maximum allowed population in the system. - $N_t$ stands for the population size at time of $t$, - $N_0$ stands for the initial population size, and - $r$ stands for the growth rate. The Malthusian Growth Model: ```{r} # Parameters K <- 10000 N_0 <- 100 r <- 1 # Calculation x <- seq(0, 10, length.out = 400) N_t_malthus <- N_0 * exp(r * x) data_malthus <- data.frame(Time = x, Population = N_t_malthus) # Plotting ggplot(data_malthus, aes(x = Time, y = Population)) + geom_line(color = "#5654A2") + labs(title = "Malthusian Growth Model", subtitle = expression( paste("Parameters: ", K, "=10000, ", N[0], "=100, ", r, "=1") ), x = "Time (t)", y = expression(paste("Population (", N[t], ")"))) + ylim(0, 10000) + theme_bw(base_family = 'Ubuntu') + theme(plot.subtitle = element_text(hjust = 0.5, size = 10), plot.title = element_text(hjust = 0.5, size = 14)) ``` In this model, we set a simple scenario of $K$ = 10,000, $N_0$ = 100, and $r$ = 1. It can be seen that the population grows so fast, that it reaches the cap while $t$ value is a little over 4. ### 2. The Logistic Growth Model However, this exponential model is unrealistic due to the scarcity of resources and spaces. The Belgian statistician Adolphe Quetelet knew that extrapolating exponential growth leads to impossible values, and asked his pupil Pierre-François Verhulst to look into the problem. Verhulst added a term for the resistance to further growth, and the result is an S curve with a cap. He published it in three papers between 1838 and 1847. This S curve is perhaps the first S-shaped curve that we could learn in our textbooks. Verhulst named the curve "logistique" in his 1845 paper [@verhulst1845] and never explained the name [@cramer2002]. It is likely from the Greek "logistikos" (skilled in calculation), chosen by analogy with "arithmetic" and "geometric" growth, and in contrast to the "logarithmic" curve (what we now call the exponential) that he drew next to it. It is not related to the military word "logistics", which comes from the French "logis" (lodging). His work was then forgotten. Pearl and Reed rediscovered the same curve in 1920 while studying the US population, and the name "logistic" was revived in the 1920s. The "Logit Model" came a century later. Joseph Berkson [-@berkson1944] proposed the logistic function as an alternative to the normal-based "probit" in bio-assay, and coined the term "logit" for the log of odds. > The word "**logit**" is a portmanteau, coming from **logi**stic + uni**t**. Berkson coined it in 1944 by analogy with "probit" (**prob**ability + un**it**). The Logistic Growth Function: $$ N_t = \frac{K}{1 + \left( \frac{K - N_0}{N_0} \right) e^{-rt}} $$ The Logistic Growth Model: ```{r} # Calculation for Logistic Growth Model N_t_logit <- K / (1 + ((K - N_0) / N_0) * exp(-r * x)) data_logit <- data.frame(Time = x, Population = N_t_logit) # Plotting the Logistic Growth Model ggplot(data_logit, aes(x = Time, y = Population)) + geom_line(color = "#5654A2") + labs(title = "Logistic Growth Model", subtitle = expression( paste("Parameters: ", K, "=10000, ", N[0], "=100, ", r, "=1") ), x = "Time (t)", y = expression(paste("Population (", N[t], ")"))) + ylim(0, 10000) + theme_bw(base_family = 'Ubuntu') + theme(plot.subtitle = element_text(hjust = 0.5, size = 10), plot.title = element_text(hjust = 0.5, size = 14)) ``` This is the "S Curve" we've been seen in a lot of scenarios where there is a designed cap of population due to resource scarcity. There are also other S shaped curves, such as the normal CDF used by the probit model. Divide the population by $K$ and the logistic curve becomes a proportion between 0 and 1, which is the same function the logit model uses for probability. > The CDF of the logistic distribution is this S curve. With a visible diminishing of returns, it is a perfect model for human behavior studies for the purpose of generating a feasible marketing strategy. ## The Logit Model ### 1. Probability vs Logit In choice modeling, the utility of an alternative has an observed part $v_j$ and a random error $\varepsilon_j$: $$ u_j = v_j + \varepsilon_j $$ The probability of choosing $j$ over $k$ is the probability that $u_j > u_k$. To compute it, we need to assume a distribution for the errors. If the errors are normally distributed, we get the **Probit Model**, which is supported by theory but has no closed form solution for the probability. If the errors follow the Type 1 Extreme Value distribution, we get the **Logit Model**, which has a closed form: $$ P_j = \frac{e^{v_j}}{e^{v_j} + e^{v_k}} $$ Here the logarithm of the odds is simply the utility difference: $\log(P_j / P_k) = v_j - v_k$. This is again the probability of choosing $j$ over $k$. To generalize the equation, the probability of choosing $j$ over all other alternatives is written as this: $$ P_j = \frac{e^{v_j}}{\sum_{k=1}^{J} e^{v_k}} $$ This multinomial form was first suggested by Cox (1966) and Theil (1969). Daniel McFadden [-@mcfadden1973] linked it to the theory of discrete choice from mathematical psychology, Luce's choice axiom, which gave the logit model its theoretical foundation and earned him the Nobel Prize in 2000. The logit model requires the errors to be independent and identically distributed. As a result it has the IIA property (Independence of Irrelevant Alternatives): the ratio $P_j / P_k$ does not depend on what other alternatives are available. > IIA can be problematic when products are close substitutes. The classic example is "Red Bus vs Blue Bus". Suppose the choice is between Taxi and Red Bus, with probabilities of 0.66 and 0.33. Now add a Blue Bus that is identical to the Red Bus except for the color. We expect 0.66, 0.165, and 0.165, since the two buses are in fact one alternative. But IIA keeps the ratio between Taxi and Red Bus at 2:1, so the logit model gives 0.50, 0.25, and 0.25. ### 2. Why Logit So, why using Logit Model? Why bother transforming the simple probability value into a somewhat non-intuitive value? An important reason is that the logit value does not have lower and upper limits, so it can be modeled as a linear function of utilities. This can be better explained by the **Logit Transformation**: $$ \text{Probability} \rightarrow \text{Odds} \rightarrow \text{Logit} $$ In statistics, both probability and odds are used to describe the likelihood of an event occurring, but they have an important difference: 1. **Probability** refers to the ratio of the number of times an event A occurs to the total number of possible outcomes, while 2. **Odds** refers to the ratio of the probability of an event occurring to the probability of the event not occurring. The probability equation writes: $$ P(A) = \frac{\text{Number of Event A}}{\text{Total Number of Events}} $$ The probability $P$ is a real number between 0 and 1. $P = 0$ indicates that an event will definitely not occur, while $P = 1$ indicates that an event will definitely occur. In the case of rolling a die, the probability of rolling a 6 is: $$ P = \frac{1}{6} $$ The odds equation writes: $$ \text{Odds} = \frac{\text{Probability of event}}{\text{Probability of no event}} = \frac{P}{1 - P} $$ Continuing with the example of rolling a die, the probability of rolling a 6 is $P = \frac{1}{6}$, and the probability of rolling any other number is $1 - P = \frac{5}{6}$. According to the equation, the odds of rolling a 6 is: $$ \text{Odds} = \frac{1/6}{5/6} = \frac{1}{5} $$ Now you can see: The ratio of the probability of successfully rolling a 6 to that of not is 1:5. Like many other concepts in probability theory, the idea of odds originates from gambling. Suppose two people, A and B, bet on rolling a die; if A bets 1 dollar on rolling a 6, B would need to bet 5 dollars to ensure a fair game. $1/6$ and $1/5$ is not a big difference, but the point is to show that probability is *happening* vs *all*, while odds is *happening* vs *not happening*. Imaging another scenario of drawing a ball from a packet, where there are 3 red balls and 2 blue balls. The probability of drawing a red ball is $3/5$, while the odds is $3/2$. This time, we have a larger difference, and the odds is even grater than one. In fact, the value of odds can be *from 0 to positive infinity*. Till now, we transformed from **probability** to **odds**. The next transformation is from **odds** to **logit**. This can be simply done by taking the log value of odds: $$ \text{Logit} = \log(\text{Odds}) $$ Therefore, the full **Logit Transformation** is: $$ \text{Probability} = P \rightarrow \text{Odds} = \frac{P}{1-P} \rightarrow \text{Logit} = \log(\text{Odds}) $$ To better explain the differences between these 3, I constructed a data frame and sketched 2 plots of *P vs Odds* and *P vs Logit*: ::: {style="display: flex; flex-wrap: wrap; align-items: center;"} ::: {style="width: 30%;"} ```{r} #| echo: false # Define the probability values P <- c(0.01, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99) # Calculation Odds <- P / (1 - P) Logit <- round(log(Odds), 2) Odds <- round(Odds, 2) # Data frame data_logit <- data.frame(P, Odds, Logit) data_logit %>% kable(escape = FALSE, align = c("c")) %>% column_spec(column = 1, width = '10em') %>% column_spec(column = 2, width = '10em') %>% column_spec(column = 3, width = '10em') %>% kable_styling(position = "center") ``` ::: ::: {style="width: 70%;"} ```{r} #| echo: false #| out-width: 90% # Plot P vs Odds ggplot(data_logit, aes(x = P, y = Odds)) + geom_point(color = "#5654A2") + geom_line(color = "#5654A2") + labs(title = "Probability vs Odds", x = "Probability (P)", y = "Odds") + theme_bw(base_family = 'Ubuntu') + theme(plot.title = element_text(hjust = 0.5, size = 14)) ``` ```{r} #| echo: false #| out-width: 90% # Plot P vs Logit ggplot(data_logit, aes(x = P, y = Logit)) + geom_point(color = "#5654A2") + geom_line(color = "#5654A2") + labs(title = "Probability vs Logit", x = "Probability (P)", y = "Logit") + theme_bw(base_family = 'Ubuntu') + theme(plot.title = element_text(hjust = 0.5, size = 14)) ``` ::: ::: A summary of the transformation: 1. **Probability** values range from $0$ to $1$. 2. **Odds** values range from $0$ to $+\infty$. 3. **Logit** values range from $-\infty$ to $+\infty$. ## References ::: {#refs} :::