Distribution Fitting¶
Maximum-likelihood fitting of continuous distributions, PDF/CDF evaluation, Kolmogorov-Smirnov and Anderson-Darling goodness-of-fit, and helpers for fitting many candidate distributions to many columns at once.
fit_named_distribution(data, dist_name)
¶
Fit a named distribution to data using MLE.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
array - like
|
Observed data to fit. |
required |
dist_name
|
str
|
Distribution name: 'normal', 'beta', 'gamma', 'exponential', or 'lognorm'. |
required |
Returns:
| Type | Description |
|---|---|
tuple
|
Fitted distribution parameters (loc, scale, and shape params if applicable). |
Raises:
| Type | Description |
|---|---|
ValueError
|
If dist_name is not one of the supported distributions. |
Source code in utils/distribution_utils.py
fit_distribution(data, dist_obj)
¶
Fit a scipy.stats distribution object to the data using MLE.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
array - like
|
Observed data to fit. |
required |
dist_obj
|
scipy.stats continuous_rv_generic
|
A scipy distribution object (e.g., scipy.stats.norm, scipy.stats.gamma). |
required |
Returns:
| Type | Description |
|---|---|
tuple
|
Fitted parameters as returned by the distribution's |
Examples:
>>> from scipy.stats import norm
>>> import numpy as np
>>> data = np.random.normal(0, 1, 1000)
>>> params = fit_distribution(data, norm)
>>> len(params) # loc, scale
2
Source code in utils/distribution_utils.py
compute_pdf(data, dist_obj, params)
¶
Compute PDF values over the data range using fitted distribution parameters.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
array - like
|
Observed data (used only for determining the x-axis range). |
required |
dist_obj
|
scipy.stats continuous_rv_generic
|
Fitted distribution object. |
required |
params
|
tuple
|
Parameters as returned by dist_obj.fit(data). |
required |
Returns:
| Type | Description |
|---|---|
tuple of (np.ndarray, np.ndarray)
|
(x, pdf_vals) where x has 200 evenly spaced points over [min(data), max(data)]. |
Source code in utils/distribution_utils.py
compute_cdf(data, dist_obj, params)
¶
Compute CDF values over the data range using fitted distribution parameters.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
array - like
|
Observed data (used only for determining the x-axis range). |
required |
dist_obj
|
scipy.stats continuous_rv_generic
|
Fitted distribution object. |
required |
params
|
tuple
|
Parameters as returned by dist_obj.fit(data). |
required |
Returns:
| Type | Description |
|---|---|
tuple of (np.ndarray, np.ndarray)
|
(x, cdf_vals) where x has 200 evenly spaced points over [min(data), max(data)]. |
Source code in utils/distribution_utils.py
perform_ks_test(data, dist_obj, params)
¶
Perform a Kolmogorov-Smirnov goodness-of-fit test.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
array - like
|
Observed data. |
required |
dist_obj
|
scipy.stats continuous_rv_generic
|
Distribution to test against. |
required |
params
|
tuple
|
Fitted distribution parameters. |
required |
Returns:
| Type | Description |
|---|---|
dict of str -> float
|
{'KS_stat': float, 'p_value': float} Large KS stat or small p-value indicates poor fit. |
Source code in utils/distribution_utils.py
fit_multiple_distributions(data, dist_list)
¶
Fit multiple distributions to the same dataset and compare KS test results.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
array - like
|
Observed data. |
required |
dist_list
|
list of scipy.stats continuous_rv_generic
|
List of distribution objects to try (e.g., [norm, gamma, expon]). |
required |
Returns:
| Type | Description |
|---|---|
list of dict
|
Each dict contains: - 'distribution': str — distribution name - 'params': tuple — fitted parameters - 'KS_stat': float — KS test statistic - 'p_value': float — KS test p-value - 'error': str — only present if fitting failed |
Source code in utils/distribution_utils.py
compute_nll(dist, data, params)
¶
Compute Negative Log-Likelihood for a fitted distribution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dist
|
scipy.stats continuous_rv_generic
|
Distribution object with a .pdf method. |
required |
data
|
array - like
|
Observed data. |
required |
params
|
tuple
|
Fitted distribution parameters. |
required |
Returns:
| Type | Description |
|---|---|
float
|
Negative log-likelihood. Lower values indicate a better fit. |
Source code in utils/distribution_utils.py
compute_aic(nll, k)
¶
Compute Akaike Information Criterion (AIC).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
nll
|
float
|
Negative log-likelihood of the fitted model. |
required |
k
|
int
|
Number of free parameters in the model. |
required |
Returns:
| Type | Description |
|---|---|
float
|
AIC value. Lower is better. Penalizes model complexity. |
Source code in utils/distribution_utils.py
compute_bic(nll, k, n)
¶
Compute Bayesian Information Criterion (BIC).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
nll
|
float
|
Negative log-likelihood of the fitted model. |
required |
k
|
int
|
Number of free parameters in the model. |
required |
n
|
int
|
Number of observations. |
required |
Returns:
| Type | Description |
|---|---|
float
|
BIC value. Lower is better. Penalizes complexity more strongly than AIC for larger sample sizes. |
Source code in utils/distribution_utils.py
perform_anderson_darling(data, dist='norm')
¶
Perform Anderson-Darling test (default: normality test). Returns statistic and critical values.
Source code in utils/distribution_utils.py
fit_multiple_distributions_extended(data, distribution_list)
¶
Fit multiple distributions and compute AIC, BIC, KS test, and Anderson-Darling test.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
array - like
|
Observed data. |
required |
distribution_list
|
list of scipy.stats continuous_rv_generic
|
List of distribution objects to fit. |
required |
Returns:
| Type | Description |
|---|---|
list of dict
|
Each dict contains: - 'distribution': str - 'params': tuple of fitted params - 'nll': float — negative log-likelihood - 'aic': float - 'bic': float - 'ks_stat': float - 'ks_pvalue': float - 'ad_stat': float or None (only for normal distribution) |
Source code in utils/distribution_utils.py
fit_distributions_all_columns(df, distribution_list)
¶
Fit multiple distributions to every numeric column in a DataFrame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input DataFrame. Only numeric columns are processed. |
required |
distribution_list
|
list of scipy.stats continuous_rv_generic
|
List of distributions to fit against each column. |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
Combined results from fit_multiple_distributions_extended for all columns, with an additional 'column' field identifying the source column. |