Table 3841. Logistic regression versus Gaussian discriminant analysis.
|
Logistic Regression |
Gaussian Discriminant Analysis (LDA) |
Commons |
Both involve the use of a sigmoid-like function to calculate probabilities |
Definition |
Is a binary classification algorithm that models the probability of the outcome (Y) being 1 (or belonging to one class) given the input features (X). The logistic (sigmoid) function is used to map the linear combination of input features to a probability value between 0 and 1. The logistic function is defined as follows: P(Y=1|X) = 1 / (1 + e(-z)). Here, 'z' is the linear combination of input features and model parameters. The logistic function squeezes the output to be in the range [0, 1], representing the probability of Y being 1. |
Is also a classification technique used to model the probability of a sample belonging to a particular class. Unlike logistic regression, LDA assumes that the input features follow a Gaussian (normal) distribution and that the variances of the features are equal for all classes. In LDA, the class-conditional probability density functions are assumed to be Gaussian, and the decision boundary is estimated based on these densities. While LDA doesn't directly use the logistic function, it often leads to a similar sigmoid-shaped decision boundary, especially when the classes are well-separated. |
Nature of Algorithm |
Is a discriminative algorithm |
Is a generative algorithm. |
It directly models the conditional probability of the output class given the input features. |
It models the probability distribution of the input features for each class and then uses Bayes' theorem to make predictions. The probability of a sample belonging to a class is calculated using Bayes' theorem. |
Assumptions |
It makes no assumptions about the distribution of input features. |
Assumes that the input features are normally distributed within each class. |
It is non-parametric |
Assumes that the covariance of features is the same for all classes. |
Decision Boundary |
The decision boundary can be linear or nonlinear depending on the feature engineering and parameters. |
LDA typically leads to linear decision boundaries. It assumes that the decision boundaries are hyperplanes. |
Output |
Produces class probabilities directly using the logistic (sigmoid) function. |
Produces class probabilities indirectly by estimating the class-conditional densities and applying Bayes' theorem. |
Parameter Estimation |
Estimates model parameters (coefficients) by maximizing the likelihood of the observed data. |
Estimates parameters (means and covariance) for each class and the prior probabilities. |
Multiclass Classification |
Supports both binary and multiclass classification through techniques like one-vs-all (OvA) or softmax regression. |
Originally designed for binary classification but can be extended to multiclass classification. |
Robustness |
Can handle a wider range of data distributions and is more robust when the assumptions of GDA are not met. |
Sensitive to deviations from the assumption of Gaussian distribution and equal covariance. |
Interpretability |
The model's coefficients provide a direct measure of the influence of each feature on the output. |
The model is less interpretable as it focuses on class distributions and doesn't provide direct feature importance. |
Data Size |
Well-suited for both small and large datasets. |
Requires more data to accurately estimate the covariance matrices. |
Comparison |
Logistic regression |
Gaussian Mixture Model |
|
![Conditional likelihood](images3/3847b.png) |
![Conditional likelihood](images3/3847c.png) |