als-statistics/readme.md

A powerful and lightweight JavaScript library for descriptive statistics, regression, clustering, outlier detection, and noise analysis using a flexible table/column architecture.

# als-statistics

**als-statistics** is a fast and lightweight JavaScript library for statistical analysis, outlier detection, regression, clustering, and time series operations.  
It is designed to work seamlessly in both **Node.js** and **browser** environments, providing an intuitive and consistent API for both quick data exploration and advanced analytical workflows.

## ✨ Key Features

- 📊 **Flexible Table Structure** — Easily manipulate tabular data with automatic type detection.
- 🔢 **Two Column Types** — `RatioColumn` for numeric operations and `Column` for strings.
- 🔍 **Built-in Filtering** — Filter rows at the column or table level using `filterRowsBy`, `filterRows`, and more.
- 📈 **Descriptive & Inferential Statistics** — Includes mean, variance, confidence intervals, correlation, regression, and more.
- ⚙️ **Utility Tools** — Use standalone helpers like `range`, `newTable`, `newColumn`, and built-in regression functions.
- 📦 **Universal Compatibility** — Supports both CommonJS and ESM imports, as well as browser-ready bundling.


---


## Installation

### Via npm

```bash
npm install als-statistics
```

### Browser Usage

Include the script in your HTML:

```html
<script src="node_modules/als-statistics/statistics.js"></script>
<script>
  const { newColumn } = Statistics;
  const col = newColumn([1, 2, 3]);
  console.log(col.mean); // 2
</script>
```

## 🚀 Quick Start: Plug and Play

**als-statistics** allows you to explore data either directly (with standalone columns/tables) or by organizing it inside a `Statistics` instance for multi-table workflows.

### 📦 Standalone Columns

Create numeric or categorical columns on the fly:

```js
const {newColumn} = require("als-statistics");

// Ratio (numeric) column
const numbers = newColumn([1, 2, 3, 4, 5]);
console.log(numbers.mean);    // 3
console.log(numbers.median); // 3

// Categorical (string) column
const categories = newColumn(["A", "B", "A", "C"]);
console.log(categories.frequencies); // { A: 2, B: 1, C: 1 }
```

> `Statistics.newColumn(values)` automatically detects the type:
> - Numbers → `RatioColumn`: full statistical toolkit.
> - Strings → `Column`: ideal for frequency counts.

---


### 🧮 Standalone Tables

Tables let you manage multiple columns in a structured way.

```js
const table = Statistics.newTable({
  Numbers: [10, 20, 30, 40],
  Labels: ["X", "Y", "Z", "W"]
});

console.log(table.columns["Numbers"].mean);        // 25
console.log(table.columns["Labels"].frequencies);  // { X: 1, Y: 1, Z: 1, W: 1 }
```

> Tables automatically assign `Column` or `RatioColumn` types based on values.  
> Use `.columns[colName]` to access each column.

✨ Tables also support:
- Dynamic filtering
- Cloning with or without filters
- Column-level comparisons
- Transposing rows into columns
- Aggregating metrics (`descriptive`)

---


### 🔍 Row Filtering

Filter rows at both **column** and **table** levels:

```js
// Filter where Numbers ≤ 20
table.filterRowsBy("Numbers", v => v > 20);
console.log(table.columns["Numbers"].values); // [10, 20]
console.log(table.columns["Labels"].values);  // ["X", "Y"]

// Further filter the Numbers column directly
const col = table.columns["Numbers"];
col.filterRowsBy(v => v === 10);
console.log(col.values);                      // [20]
console.log(table.columns["Labels"].values);  // ["Y"]

// Reset all filters
table.clearAllRowsFilters();
```

> ⚠️ Filters are **exclusion-based**: the predicate returns **true to exclude** a row.  
> To keep values instead, use `v => !condition`.

---


### 📚 Using a Statistics Instance

Group and analyze multiple tables together:

```js
const stats = new Statistics();

const t1 = stats.addTable("Table1", { Data: [1, 2, 3] });
const t2 = stats.addTable("Table2", { Data: [4, 5, 6] });

// Aggregate metrics across tables
const means = stats.descriptive("mean");

console.log(means.columns["Table1"].values[0]); // 2
console.log(means.columns["Table2"].values[0]); // 5
```

> You can also apply filters globally using:
> - `stats.filterRows(indexes)`
> - `stats.clearAllRowsFilters()`

---

## 📊 Working with Columns

The `Column` and `RatioColumn` classes represent the two main types of data: **categorical** and **numeric**. Each supports filtering, statistics, and derived metrics.

> ✅ Columns can now be **empty** (`[]`) — all methods handle them gracefully and return default values like `0`, `NaN`, or `[]`.

---


### `Column` (Categorical Data)

- **Use For:** Strings, labels, categories.
- **Created By:**
  - `Statistics.newColumn(["A", "B", "A"])`
  - `table.addColumn("Name", ["A", "B", "A"])`

#### 🔧 Key Properties & Methods:

| Feature                     | Description                                                                 |
|----------------------------|-----------------------------------------------------------------------------|
| `values`                   | Array of current values (after filters).                                    |
| `n`                        | Number of unfiltered values.                                                |
| `frequencies`              | Object with counts of each unique value.                                    |
| `relativeFrequencies`      | Object with proportions (relative frequencies).                             |
| `sorted`                   | Alphabetically sorted values.                                               |
| `percentile(p)`            | Percentile by index, e.g., `50` is median (treats values as ordered).       |
| `median`, `q1`, `q3`       | Predefined percentiles (50th, 25th, 75th).                                  |
| `count(fn, filtered=true)` | Number of values satisfying a condition.                                    |
| `filterRows(indexes)`      | Exclude rows by their indexes.                                              |
| `filterRowsBy(fn)`         | Exclude rows where `fn(value, index)` returns `true`.                       |
| `clearRowsFilters()`       | Remove all filters.                                                         |
| `clone(filtered = true)`   | Create a filtered or unfiltered copy of the column.                         |

#### 🧪 Example:

~~~js
const col = Statistics.newColumn(["A", "B", "A", "C"]);
col.filterRowsBy(v => v === "B"); // Exclude "B"
console.log(col.values);          // ["A", "A", "C"]
console.log(col.frequencies);     // { A: 2, C: 1 }
~~~

---


### `RatioColumn` (Numeric Data)

- **Use For:** Numbers (measurements, ratings, etc.).
- **Created By:**
  - `Statistics.newColumn([1, 2, 3])`
  - `table.addColumn("Scores", [1, 2, 3])`

#### 📐 Basic Statistics:

| Method              | Description                             |
|---------------------|-----------------------------------------|
| `sum`, `mean`       | Total and average of values.            |
| `min`, `max`, `range` | Extremes and spread.                 |

#### 📊 Variability Metrics:

- **Population**:
  - `variancePopulation`, `stdDevPopulation`
  - `skewnessPopulation`, `kurtosisPopulation`
- **Sample**:
  - `varianceSample`, `stdDevSample`
  - `skewnessSample`, `kurtosisSample`

#### 📏 Dispersion:

- `cv`: Coefficient of Variation
- `relativeDispersion`: StdDev / Median
- `iqr`: Interquartile range

#### 🔍 Advanced Metrics:

- `geometricMean`, `harmonicMean`, `flatness`
- `sumOfSquares`: Σ(x²)
- `normalizedValues`: Min-max scaling to [0, 1]
- `zScores`: Standardized values (z = (x - μ) / σ)
- `confidenceInterval95`: `{ low, high, width }` — 95% CI around the mean
- `spectralPowerDensityArray`, `spectralPowerDensityMetric`: Noise structure indicators
- `noiseStability`: Simple noise estimator

#### 🧭 New Tools:

| Method                         | Description                                                                 |
|--------------------------------|-----------------------------------------------------------------------------|
| `xValues`                      | Automatically generates `[1, 2, ..., n]` for regression or plotting.        |
| `regressionSlope(customX?)`   | Calculates slope (β) vs. `xValues` or custom X-axis.                       |

> `regressionSlope` uses covariance and variance internally. Returns `0` if X is constant.

#### ⚠️ Outlier Detection:

- `outliersIQR`: Outliers using interquartile method.
- `outliersZScore(threshold = 3)`: Outliers based on z-scores.

#### 🧪 Example:

~~~js
const col = Statistics.newColumn([1, 2, 3, 100]);
col.filterRowsBy(v => v > 10); // Exclude 100
console.log(col.mean);         // 2
console.log(col.outliersIQR);  // []
console.log(col.regressionSlope()); // 1 (if x = [1, 2, 3])
~~~

---


✅ Both `Column` and `RatioColumn` support row-level filtering and full integration with `Table` and `Statistics`.


# 📉 regressionSlope()

Calculates the slope of the best-fit line through a numeric column.

## 🔹 Signature

~~~js
column.regressionSlope(customX = null)
~~~

## 🔹 Description
Returns the slope of the regression line between:
- `this` column (Y)
- `customX` or the default `xValues = [1, 2, ..., n]`

## 🔹 Example

~~~js
const col = Statistics.newColumn([2, 4, 6]);
console.log(col.regressionSlope()); // 2
~~~

If `customX` is not provided, `[1, 2, 3]` is used.


## Noise Analysis

Analyze noise in numeric data:

```js
const col = Statistics.newColumn([1, 2, 3, 100]);
const noise = col.noice();
console.log(noise.noiseByZ(0.1)); // false (outlier present)
```

---


## 📋 Working with Tables

Tables allow you to manage multiple columns, filter data, create new computed columns, perform statistical comparisons — and now also run **linear regression** directly.

---


### 🛠️ Creating a Table and Adding Columns

You can initialize a table with or without data:

```js
const table = Statistics.newTable();
table.addColumn("Scores", [10, 20, 30, 40]);
table.addColumn("Grades", ["A", "B", "C", "D"]);
```

Or create with data immediately:

```js
const table = Statistics.newTable({
  Scores: [10, 20, 30, 40],
  Grades: ["A", "B", "C", "D"]
});
```

> Column types (`RatioColumn` or `Column`) are auto-detected.

---


### ➕ Adding Rows

```js
table.addRow({ Scores: 50, Grades: "F" }); // Adds to the end
table.addRow({ Scores: 5, Grades: "A+" }, 0); // Inserts at index 0
```

Returns `true` if successful, `false` if structure or types don’t match.

---


### 🔍 Row Filtering

```js
// Exclude specific rows by index
table.filterRows([1]);
console.log(table.columns["Scores"].values); // [10, 30, 40]

// Filter by values using column-level filter
table.clearAllRowsFilters();
table.filterRowsBy("Scores", v => v > 20);
console.log(table.columns["Scores"].values); // [10, 20]
console.log(table.columns["Grades"].values); // ["A", "B"]

// Reset filters
table.clearAllRowsFilters();
```

---


### 🧮 Computing New Columns

Use any logic to generate new columns:

```js
table.compute(({ Scores }) => Scores * 2, "Doubled");
console.log(table.columns["Doubled"].values); // [20, 40, 60, 80]
```

> Uses the row-wise structure: `({ Col1, Col2 }) => ...`

---


### 🔎 Filtering with `.where(...)`

Returns matching row indexes (ignoring filtered rows):

```js
const indexes = table.where(({ Scores }) => Scores > 15);
console.log(indexes); // [1, 2, 3] if "Scores" > 15
```

---


### 🔁 Transposing the Table

Turns rows into columns:

```js
const transposed = table.transpose();
console.log(transposed.columns["0"].values); // [10, "A"]
```

---


### 📈 Running Linear Regression

Perform regression right from the table:

```js
const model = table.linearRegression("Scores", ["Doubled"]).calculate();
console.table(model.result);
```

- `table.linearRegression(y, x)` is a shortcut for `new LinearRegression(...)`
- Supports `.mediator(...)`, `.moderator(...)`, and `.calculate()` chaining

---


### 📊 Aggregating Metrics (Descriptive)

```js
const means = table.descriptive("mean");
console.log(means.columns["Scores"].values[0]); // e.g. 25
```

Any column-level metric is supported, like `"stdDevSample"`, `"cv"`, `"flatness"`, etc.

---


### 🧪 Comparing Two Columns

```js
const comp = table.compare("Scores", "Doubled");
console.log(comp.correlationSample); // e.g. 1.0
```

Returns a `Comparative` object, which supports:
- `correlationSample`, `correlationPopulation`
- `covarianceSample`, `covariancePopulation`
- `pearsonSample`, `pearsonPopulation`
- `twoSampleTTest()`

---


### 📦 Cloning Tables

You can duplicate the table with filters and/or selected columns:

```js
// Clone with all columns and current row filters
const fullClone = table.clone(true);
console.log(fullClone.columns["Scores"].values); // [10, 20, 30, 40]

// Clone with selected columns
const selectedClone = table.clone(true, ["Scores"]); // Only "Scores"
console.log(selectedClone.columns["Scores"].values); // [10, 20, 30, 40]
console.log(selectedClone.columns["Grades"]); // undefined

// Clone excluding columns
const excludedClone = table.clone(true, ["-Grades"]); // All except "Grades"
console.log(excludedClone.columns["Scores"].values); // [10, 20, 30, 40]
console.log(excludedClone.columns["Grades"]); // undefined

// Clone with regex filter
table.addColumn("Scores2", [1, 2, 3, 4]);
const regexClone = table.clone(true, [/^Scores/]); // Columns starting with "Scores"
console.log(regexClone.columns["Scores"].values); // [10, 20, 30, 40]
console.log(regexClone.columns["Scores2"].values); // [1, 2, 3, 4]
```

- **Options for `clone(filtered, columnFilter)`:**
  - `filtered`: `true` (keep row filters) or `false` (use original data).
  - `columnFilter`: Array of filters:
    - `"Name"`: Include this column.
    - `"-Name"`: Exclude this column (if no includes specified).
    - `/pattern/`: Include columns matching the regex.
  - **Note:** If explicit includes (e.g., `"A"`) are used, exclusions (e.g., `"-B"`) are ignored.

---



### 🧹 Removing Columns

```js
table.deleteColumn("Grades");
console.log(table.columns["Grades"]); // undefined
```

---


### 🚀 Available Methods

| Method                         | Description                                                  |
|--------------------------------|--------------------------------------------------------------|
| `addColumn(name, values)`     | Adds new column with auto-detection.                        |
| `addRow(obj, index?)`         | Appends or inserts a row.                                   |
| `compute(fn, name?)`          | Computes a new column.                                      |
| `compare(col1, col2)`         | Returns statistical comparison object.                      |
| `filterRows(indexes)`         | Excludes rows by index.                                     |
| `filterRowsBy(name, fn)`      | Filters rows based on column values.                        |
| `clearAllRowsFilters()`       | Removes all filters.                                        |
| `clone(filtered, filterCols)` | Returns new table with selected rows/columns.               |
| `descriptive(metric)`         | Returns a new table with a summary metric per column.       |
| `transpose()`                 | Flips rows and columns.                                     |
| `where(fn)`                   | Returns indexes of matching rows.                           |
| `dbscan(eps, minPts)`         | Runs DBSCAN clustering on columns.                          |
| `hdbscan(minPts)`             | Runs HDBSCAN clustering (hierarchical).                     |
| `linearRegression(y, x?)`     | Returns a new `LinearRegression` instance for the table.    |

---


## 📊 DBSCAN Clustering

`dbscan(eps, minPts)` identifies clusters of correlated numeric columns using DBSCAN (Density-Based Spatial Clustering of Applications with Noise).

### 🔹 Parameters
- `eps` (default: `0.4`) — maximum distance between columns to be considered neighbors.
- `minPts` (default: `3`) — minimum number of neighbors to form a dense region (cluster).

### 🔹 Usage

```js
const table = Statistics.newTable({
  A: [1, 2, 3],
  B: [10, 20, 30],
  C: [5, 10, 15]
});

const dbscan = table.dbscan(0.5, 2);
console.log(dbscan.labels);        // Example: [1, 1, -1]
console.log(dbscan.clusters);     // Array of clustered Table instances
```

### 🔹 Output
- `labels`: Array assigning a cluster ID or `-1` for noise to each column.
- `clusters`: Array of new `Table` instances, one per cluster.

### 🔹 How It Works
- Measures correlation between numeric columns.
- Builds pairwise distances.
- Expands clusters around dense areas.


## 📏 Cronbach's Alpha

Cronbach’s Alpha measures internal consistency — how closely related a set of numeric items are.

### 🔹 Use Cases
- Questionnaires and surveys
- Psychometric analysis
- Testing item reliability

### 🔹 Usage

```js
const table = Statistics.newTable({
  Q1: [4, 5, 3, 4, 5],
  Q2: [3, 4, 2, 3, 4],
  Q3: [5, 3, 4, 5, 2]
});

const alpha = table.cronbachAlpha;
console.log(alpha.alpha); // Example: 0.74
```

### 🔹 Item Deletion Diagnostics

```js
console.log(alpha.perColumn);
// { Q1: 0.65, Q2: 0.68, Q3: 0.71 }
```

### 🔹 Notes
- Requires 2+ numeric columns.
- If all responses are identical, `alpha = 0`.


## 📊 Pearson Correlation

Measures the linear correlation (Pearson’s r) between two numeric columns.

### 🔹 Usage

```js
const table = Statistics.newTable({
  X: [10, 20, 30],
  Y: [15, 25, 35]
});

const correlation = table.compare("X", "Y").pearsonSample;

console.log(correlation.r);  // 1.0 (perfect correlation)
console.log(correlation.p);  // 0.0 (significant)
```

### 🔹 Interpretation

- `r` close to 1 or -1: strong correlation
- `p < 0.05`: statistically significant

### 🔹 Access
- `.pearsonSample` — uses sample variance
- `.pearsonPopulation` — uses population variance


## 📈 Linear Regression

`LinearRegression` performs multiple linear regression on `Table` data.

### 🔹 Key Features
- Multiple predictors
- Intercept term
- Mediator and moderator support
- Coefficients, p-values, R²
- Interaction terms (X*Z)

### 🔹 Example

```js
const table = Statistics.newTable({
  X: [1, 2, 3, 4, 5],
  Y: [2, 4, 6, 8, 10]
});

const model = table.linearRegression("Y", ["X"]).calculate();
console.table(model.result);
// Expect Intercept ≈ 0, X Coefficient ≈ 2
```

### 🔹 With Mediator

```js
table.linearRegression("Y", ["X"])
  .mediator("M")
  .calculate();
```

### 🔹 With Moderator

```js
table.linearRegression("Y", ["X"])
  .moderator("Z")
  .calculate();
```

### 🔹 Output

```js
model.result
// {
//   Variable: ['Intercept', 'X', ...],
//   Coefficient: [...],
//   StdError: [...],
//   pValue: [...]
// }
```

### html table

```js
table.linearRegression("Y", ["X"]).htmlTable
```

### 🔹 Notes

- Internally uses matrix algebra
- Throws if predictors are constant or if matrix is singular


## Managing Multiple Tables with `Statistics`

Use `Statistics` to group tables:

```js
const stats = new Statistics();
stats.addTable("Sales").addColumn("Revenue", [100, 200, 300]);
stats.addTable("Costs").addColumn("Expenses", [50, 100, 150]);

stats.filterRows([1]); // Filters all tables
const statsMeans = stats.descriptive("mean");
console.log(statsMeans.columns["Sales"].values[0]); // 200
```

- **Methods:**
  - `filterRows(indexes)`: Applies to all tables.
  - `clearRowsFilters(indexes)`, `clearAllRowsFilters()`: Clears filters across tables.

---


## Utilities

- **`Statistics.range(start, end, step)`**: Generates an array of numbers.
  ```js
  const r = Statistics.range(1, 5, 2); // [1, 3]
  ```