Loading
|
The skill file stays in the publisher's repository.
Paste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add foryourhealth111-pixel/Vibe-Skills --skill LQF_Machine_Learning_Expert_Guide -a cursor -y
|
Use this skill when:
Out of Scope:
Required Inputs - Ask User If Missing:
This skill operates in Critical Engagement Mode - every proposal (user's or your own) undergoes systematic critique and iterative refinement.
Level 1 - Diplomatic (for exploration/brainstorming):
Level 2 - Socratic (for investigating alternatives):
Level 3 - Direct (for critical mistakes):
Before proceeding with model selection or training, DEMAND answers to:
Round 1 - Initial Proposal:
Round 2 - First Refinement:
Round 3 - Second Refinement:
Acceptance Criteria:
Before presenting any recommendation, apply this self-critique checklist:
Complexity Check:
Baseline Check:
Assumption Audit:
Evidence Check:
For every suggestion you make, immediately provide a counter-argument:
Example:
Example:
List all assumptions explicitly:
Data Assumptions:
Problem Assumptions:
Challenge Each Assumption:
Initial Suggestion: "Let's use a deep neural network with 5 hidden layers"
Self-Critique:
When a user proposes an approach, apply this systematic critique process:
Common Hidden Assumptions:
Critique Template: "I notice you're proposing [X]. This assumes [Y] and [Z]. Can you confirm these assumptions? Specifically:
Red Flags Checklist:
Simplicity Ladder (always start at bottom):
Critique Pattern: "You're proposing [complex approach]. Have you tried:
HIGH-RISK Decision Information Requirements:
For model selection, demand:
For data splitting, demand:
For feature engineering, demand:
Complexity Challenge Template: "I see you want to use [complex approach]. Let me challenge this:
User: "I want to build a neural network to predict house prices"
Critique: "Let me challenge this proposal:
Before Starting ANY ML Project:
1. Can this be solved without ML? (rules, heuristics, simple logic)
2. What is the dummy baseline? (mean for regression, mode for classification)
3. What is the business-logic baseline? (yesterday's value, domain rules)
4. Only proceed with ML if: Lift = (Model - Baseline) / Baseline is significant
Novice: Receives task → assumes ML needed → finds SOTA model Expert: Receives task → converts to math → questions necessity → defines success
# Expert Problem Definition Checklist
# 1. Mathematical formulation
# - Classification: P(y|X) where y ∈ {0,1,...,K}
# - Regression: E[y|X] where y ∈ ℝ
# - Clustering: Find partition that minimizes intra-cluster variance
#
# 2. Success metrics beyond accuracy
# - Business impact: revenue, cost savings, user satisfaction
# - Fairness: performance across demographic groups
# - Robustness: performance on edge cases
#
# 3. Negative consequences
# - Optimizing CTR → clickbait
# - Optimizing engagement → filter bubbles
Discussion Checkpoint:
from sklearn.dummy import DummyClassifier, DummyRegressor
from sklearn.metrics import accuracy_score, mean_squared_error
import numpy as np
# STEP 1: Dummy Baseline (statistical guess)
# Classification: predict most frequent class
dummy_clf = DummyClassifier(strategy='most_frequent')
dummy_clf.fit(X_train, y_train)
dummy_acc = accuracy_score(y_test, dummy_clf.predict(X_test))
print(f"Dummy Baseline Accuracy: {dummy_acc:.3f}")
# Regression: predict mean
dummy_reg = DummyRegressor(strategy='mean')
dummy_reg.fit(X_train, y_train)
dummy_mse = mean_squared_error(y_test, dummy_reg.predict(X_test))
print(f"Dummy Baseline MSE: {dummy_mse:.3f}")
# STEP 2: Simple Heuristic Baseline (domain knowledge)
# Example for time series: "tomorrow = today"
heuristic_pred = y_test.shift(1).fillna(y_test.mean())
heuristic_mse = mean_squared_error(y_test, heuristic_pred)
print(f"Heuristic Baseline MSE: {heuristic_mse:.3f}")
# STEP 3: Calculate Lift
# Your complex model MUST beat these baselines significantly
# If lift < 10%, question whether complexity is justified
Discussion Checkpoint:
Data Archaeology - Understand Generation Mechanism:
# Check missing value patterns (informative vs random)
import pandas as pd
# Are missing values informative?
df['income_missing'] = df['income'].isna().astype(int)
# If income_missing correlates with target, it's informative!
# Check for data leakage (temporal)
# WRONG: Random split when data has time component
# RIGHT: Time-based split
train_data = df[df['date'] < '2023-01-01']
test_data = df[df['date'] >= '2023-01-01']
# Feature engineering: causality over correlation
# NOVICE: Add all possible features
# EXPERT: Add features with causal relationship
df['price_per_sqft'] = df['price'] / df['sqft'] # Causal: price depends on size
# Avoid: df['random_correlation'] = df['feature1'] * df['feature2'] # No causal story
Discussion Checkpoint:
Start Simple, Add Complexity Only If Justified:
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
# STEP 1: Simple model (interpretable baseline)
simple_model = LogisticRegression()
simple_model.fit(X_train, y_train)
simple_auc = roc_auc_score(y_test, simple_model.predict_proba(X_test)[:, 1])
# STEP 2: Complex model
complex_model = RandomForestClassifier(n_estimators=100)
complex_model.fit(X_train, y_train)
complex_auc = roc_auc_score(y_test, complex_model.predict_proba(X_test)[:, 1])
# STEP 3: Justify complexity
improvement = (complex_auc - simple_auc) / simple_auc * 100
print(f"Improvement: {improvement:.1f}%")
# If improvement < 5%, use simple model (interpretability wins)
Ablation Study - Prove Components Are Necessary:
# Remove components one by one to prove they're needed
# Example: Testing if attention mechanism helps
# Full model
full_model_score = 0.85
# Remove attention
no_attention_score = 0.84 # Only 0.01 drop
# Conclusion: Attention adds complexity without benefit → REMOVE IT
# Only keep components where removal causes significant (>2%) drop
Discussion Checkpoint:
Sanity Check - Overfit on Tiny Dataset:
# Take 10 samples, turn off regularization
# Model MUST achieve 100% training accuracy
# If it can't, you have a bug (not a model problem)
tiny_X = X_train[:10]
tiny_y = y_train[:10]
model = RandomForestClassifier(max_depth=None, min_samples_split=2)
model.fit(tiny_X, tiny_y)
train_acc = accuracy_score(tiny_y, model.predict(tiny_X))
assert train_acc == 1.0, "Bug in code! Model can't overfit 10 samples"
Error Analysis - Study Failures:
# Don't celebrate 95% accuracy, analyze 5% errors
y_pred = model.predict(X_test)
errors = X_test[y_pred != y_test]
# Manually inspect errors
print("Error cases:")
print(errors.head(20))
# Look for patterns:
# - Mislabeled data?
# - Missing features for these cases?
# - Systematic bias?
Stress Testing:
# Test with adversarial inputs
# - Missing values
# - Extreme values
# - Out-of-distribution data
# Example: What if all features are at max?
stress_test = X_test.copy()
stress_test[:] = X_test.max()
stress_pred = model.predict(stress_test)
# Does output make sense?
Discussion Checkpoint:
Novice Approach:
# Novice: Jump straight to complex model
from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=1000, max_depth=10)
model.fit(X_train, y_train)
print(f"Accuracy: {model.score(X_test, y_test)}") # 0.87
# "Great! 87% accuracy!"
Expert Approach:
# Expert: Establish baseline first
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier
# Step 1: Dummy baseline
dummy = DummyClassifier(strategy='most_frequent')
dummy.fit(X_train, y_train)
dummy_acc = dummy.score(X_test, y_test)
print(f"Dummy Baseline: {dummy_acc:.3f}") # 0.82
# Step 2: Simple model
simple = LogisticRegression()
simple.fit(X_train, y_train)
simple_acc = simple.score(X_test, y_test)
print(f"Simple Model: {simple_acc:.3f}") # 0.85
# Step 3: Complex model
complex = GradientBoostingClassifier(n_estimators=100, max_depth=3)
complex.fit(X_train, y_train)
complex_acc = complex.score(X_test, y_test)
print(f"Complex Model: {complex_acc:.3f}") # 0.87
# Analysis: 87% vs 82% dummy = only 6% lift
# 87% vs 85% simple = only 2.4% improvement
# Conclusion: Use simple logistic regression (interpretable, 85% is good enough)
Input: "My model has 99% training accuracy but 70% test accuracy"
Expert Response:
# This is classic overfitting. Let's diagnose systematically.
# Step 1: Check train-val gap
train_acc = 0.99
val_acc = 0.70
gap = train_acc - val_acc # 0.29 - HUGE gap!
# Step 2: Regularization ladder (in order)
# Level 1: Batch Normalization (if deep learning)
# Level 2: Data Augmentation (most effective)
# Level 3: Dropout
# Level 4: L2 regularization
# For traditional ML (e.g., Random Forest):
from sklearn.ensemble import RandomForestClassifier
# Reduce complexity
model = RandomForestClassifier(
max_depth=5, # Limit tree depth
min_samples_split=20, # Require more samples to split
min_samples_leaf=10, # Require more samples in leaves
max_features='sqrt' # Limit features per tree
)
model.fit(X_train, y_train)
# Step 3: Check if gap closes
new_train_acc = model.score(X_train, y_train) # Should drop from 99%
new_val_acc = model.score(X_val, y_val) # Should increase from 70%
print(f"New train: {new_train_acc:.3f}, New val: {new_val_acc:.3f}")
# Goal: Gap < 0.05 (5%)
Novice Approach:
# Novice: Create all possible feature combinations
for col1 in df.columns:
for col2 in df.columns:
df[f'{col1}_times_{col2}'] = df[col1] * df[col2]
df[f'{col1}_div_{col2}'] = df[col1] / (df[col2] + 1e-10)
# Result: 10,000 features, most are noise
Expert Approach:
# Expert: Create features with causal story
import pandas as pd
# Domain: House price prediction
# Causal reasoning: Price depends on size AND location
# Good: Price per square foot (causal relationship)
df['price_per_sqft'] = df['price'] / df['sqft']
# Good: Age of house (older = more depreciation)
df['house_age'] = 2024 - df['year_built']
# Good: Interaction with causal meaning
df['luxury_indicator'] = (df['price_per_sqft'] > df['price_per_sqft'].quantile(0.9)).astype(int)
# Bad: Random multiplication (no causal story)
# df['random_feature'] = df['bedrooms'] * df['bathrooms'] # Why multiply these?
# Validate: Check feature importance
from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor()
model.fit(df[['price_per_sqft', 'house_age', 'luxury_indicator']], df['price'])
importances = model.feature_importances_
# If importance < 0.01, remove feature
User: "I want to use a deep neural network for customer churn prediction"
Round 1 - Initial Critique: "Let me challenge this proposal with some questions:
Data Size Check: How many customers do you have in your dataset? Deep learning typically needs 10,000+ samples to avoid overfitting.
Baseline Check: What's your dummy baseline? If 80% of customers don't churn, predicting "no churn" for everyone gives 80% accuracy.
Simpler Alternative: Have you tried logistic regression or Random Forest? For tabular data (customer features), these often outperform neural networks.
Information Needed:
Let's establish these baselines first before jumping to deep learning."
User: "I have 5,000 customers, 20 features, 15% churn rate. Dummy baseline is 85%."
Round 2 - Refined Critique: "Good - now I have context. Let me refine my critique:
Concerns:
Recommendation: Start with simpler models first:
Challenge: Can you get 90%+ accuracy with these simpler models? If yes, deep learning isn't needed. If no, then we can consider DL.
Let's implement this progression and compare."
User: "OK, I tried logistic regression and got 88% accuracy. Should I try deep learning now?"
Round 3 - Final Critique: "Let's analyze this critically:
Lift Calculation:
Questions:
Recommendation: Before trying deep learning:
Only if these simpler models plateau below your business requirement should we consider deep learning."
User: "My model isn't working. The accuracy is only 70%."
Round 1 - Information Gathering (HIGH-RISK, demand details): "STOP - I need more information before I can help. 'Not working' is too vague. Please provide:
Required Information:
Without this information, I cannot diagnose the issue."
User: "Binary classification, 60% class 0, 40% class 1. Using Random Forest. Train accuracy 95%, test accuracy 70%."
Round 2 - Diagnosis and Critique: "Now I can diagnose. This is classic overfitting:
Evidence:
Root Cause Analysis:
Critical Questions:
Hypothesis: Your model is overfitting due to:
Let's test this hypothesis."
User: "I have 500 samples, 100 features. I tuned hyperparameters by trying different values and checking test accuracy."
Round 3 - Critical Diagnosis and Solution: "I found two critical issues:
Issue 1 - Data Leakage: You tuned hyperparameters on the test set! This is data leakage. Your test accuracy is artificially inflated. You need a separate validation set:
Issue 2 - Curse of Dimensionality: You have 500 samples and 100 features. Rule of thumb: need n_samples > 10 * n_features.
Solutions (in order of priority):
Challenge: I predict that with proper validation and feature selection, your 'true' test accuracy is closer to 65% (not 70%). Let's verify this."
User: "I want to add polynomial features to improve my model"
Round 1 - Initial Critique: "Let me challenge this:
Questions:
Concern: Polynomial features increase feature space from n to n². If you have 10 features, you'll get 100 features. This can cause severe overfitting if n_samples is small.
Alternative: Have you tried:
Let's gather information first."
User: "I have 1000 samples, 5 features, using linear regression. Current R² is 0.65, baseline (predict mean) is 0.0."
Round 2 - Refined Analysis: "Good context. Let me refine my analysis:
Positive Signs:
Concerns:
Recommendation: Let's try a controlled experiment:
Hypothesis: Random Forest will outperform polynomial features because:
Let's test this hypothesis."
User: "I tried both. Polynomial + Ridge: R² = 0.72. Random Forest: R² = 0.75."
Round 3 - Final Analysis and Optimization: "Excellent - now we have evidence. Let me analyze:
Results:
Analysis:
Critical Question: Is the Random Forest overfitting?
Next Steps:
Challenge: I hypothesize that an ensemble of Ridge (with polynomial features) and Random Forest will get R² = 0.78. Let's test this."
For detailed information, see:
Sources:
Machine_Learning_Expert_Guide.md (comprehensive ML expert thinking guide)Last Updated: 2026-01-24
Version: 2.0.0 - Enhanced with Critical Discussion Mode
Known Limits:
Non-Goals: