The Case

Well this isn’t rare. RAND Corporation research puts the AI project failure rate at over 80%, twice the rate of non-AI technology projects. MIT’s 2025 report found 95% of generative AI pilots at enterprises are failing. And S&P Global’s 2025 survey found 42% of companies abandoned most of their AI initiatives this year — up from 17% in 2024.

The numbers keep climbing. Not because AI doesn’t work. Because the wrong AI gets applied to the wrong problem, at the wrong cost, with nobody asking the right questions upfront.

What is the AI Confusion Tax?

The AI Confusion Tax is the hidden cost companies pay when they select an AI solution before defining the problem it needs to solve. It shows up as overspend on the wrong tools, mismatched model types, and months of data preparation that never connects to a business decision.

It’s structural, not malicious. Vendors pitch what they sell. Companies don’t know the alternatives exist. The gap between those two facts is where the tax lives.

Three patterns account for the bulk of it.

The pattern underneath

Vendors pitch what they sell. Companies don’t know the alternatives. Nobody defines the problem before selecting the solution.

Every AI decision — vendor selection, project scoping, budget approval — has this tax embedded in it. Sometimes it’s $57K a year on the wrong tool. Sometimes it’s $100K a quarter in data prep delay. Sometimes it’s a model that works technically and delivers nothing commercially.

The sticker price on an AI vendor quote is only 40-55% of your actual total cost of ownership. The rest is hidden in integration, governance, and waste from solving the wrong problem.

Understanding which AI type fits your problem — whether it’s rules-based automation, analytics, predictive ML, or generative AI — is the single most important question you can answer before signing anything.

Before you sign the next contract

One question before any AI project gets approved: does the person scoping this understand what they’re scoping?

Not — is the model accurate? Not — is the vendor credible?

Can they tell you which AI type this problem needs? Classification, regression, clustering, survival analysis? And can they explain why that type is the right one for this specific business decision?

If they can’t answer that, they’re guessing. And you’re paying for the guess.

The confusion is solvable. You just need to know what questions to ask before anyone writes a contract.

Frequently Asked Questions

What is the AI Confusion Tax?

The AI Confusion Tax is the hidden cost companies pay when they select an AI solution — usually GenAI — before defining the problem it needs to . It shows up as 12x overspend on the wrong tools, mismatched model types, and months of data preparation that never connects to a business decision.

Why do companies pay the AI Confusion Tax?

Vendors pitch what they sell. Companies don’t know the alternatives exist. Every business problem has a specific ML model that’s cheaper, faster, and more accurate than GenAI for that job — but most leaders don’t know these alternatives exist.

How much does the wrong AI cost?

Companies routinely pay 12x more for GenAI solutions to problems that $5,000–$20,000 of classical ML handles better. A vendor quoted $62,000/year for a supplier categorization problem. The same job done with topic modeling: $15–20K to build, $10/month to run.

How do you avoid the AI Confusion Tax?

Before approving any AI project, ask: can the person scoping this tell you which AI type this problem needs — classification, regression, clustering, survival analysis — and why that type is the right one for this specific business decision? If they can’t answer, they’re guessing. And you’re paying for the guess.

If your AI projects are running into walls you can’t explain, the AI Strategy Audit finds the confusion tax in your current initiatives — and tells you exactly what to fix. Or start with the AI Profit Quotient to find out where your company actually sits on the AI spectrum.

1. The Line in the Sand: Strategy vs. Steps

I get asked all the time: “Jitin, how deep do I really need to go into the tech?”

The answer is simpler than you think: You own the Pain; they own the Steps.

• The CEO & Senior Leaders: Your job is to define the Strategy and Value. You have to articulate the “Pain” (Point A) and the “Destination” (Point B). You’re the one who decides what success is actually worth to the P&L.
• The Technical Team: They own the Execution. They choose the models, clean the data, manage the integration, and build the feedback loops.

The Crux: The magic (or the disaster) happens in the middle. This is where your “BS Detector” comes in. You don’t need to know how the algorithm works, but you must be able to translate their technical jargon (like “F1 scores” or “Accuracy”) into business outcomes. If you can’t link “model accuracy” to “dollars saved,” you don’t have a strategy—you have a science fair project.

2. The Dashboard Analogy: Driving the Business

Stop thinking of AI as a “magic box.” Think of it as the instrumentation of your car. To get from A to B effectively, you need three distinct views through the glass:

1. The Rearview Mirror (Exploratory AI): “What just happened?” (Traditional data analysis and dashboards).
2. The Windshield (Predictive AI): “What’s coming at us?” (Forecasting, Churn Prediction).
3. The GPS Map (Prescriptive AI): “What’s the best route to take?” (Optimization, Recommendation Engines).

Your Job: You are the driver. You don’t need to be a mechanic to know how the fuel injection works. But you must know how to read the Windshield and the Map. Part of this is Problem-Solution Mapping. If you try to use a “GenAI” chatbot to solve a “Math” problem (like inventory optimization), you’re going to drive off a cliff. Knowing which tool to pull from the AI Umbrella is the definition of strategic competence.

3. The One Thing You Cannot Delegate

I’ve seen leaders try to delegate everything—including the outcome. This is fatal. You cannot delegate the “Delivery of Value.”

AI isn’t like buying a copy of Microsoft Office where you “install it” and walk away. * Small Involvement: Defining the “North Star” Metric. * Large Involvement: Revamping a business process to actually use the new intelligence.

You also can’t delegate the Data Viability Instinct. You need to ask: “What data do we need to start capturing TODAY so we can predict the future in six months?” If you delegate the business process change or the data strategy, you don’t own the value—you just own the bill.

4. The “Promised Land”: From Grunt Work to Growth

What happens when you get this right? When you move from “Tier 1” (playing with ChatGPT) to “Tier 3” (The Strategist)?

The Day-to-Day Shift: * Before (Chaos): You’re playing defense. Worrying about overstocking, customer tickets, and hiring frantically just to keep up. * After (Leverage): You’re playing offense. Your day is spent deciding where to invest next. Which market to enter? What feature to launch?

The Human Impact: I once worked with a Fortune 500 client who feared AI would demoralize their team. The reality? The team stopped doing the “robot work” (the grunt work on high-selling products). Instead, they focused on the creative, high-leverage stuff—like upselling new products. They didn’t shrink; they just became more human.

5. The Monday Morning Report

How do you sleep at night? You don’t need to check code commits. You need to check the Value Report.

Every Monday, you should be looking for one thing: “Are the steps taken by the technical team moving us closer to the ROI we defined?”

This requires Architectural Literacy. You need to look past the “beautiful plating” of a fancy chatbot wrapper and ask if there’s an actual logic engine beneath it. If the answer is “Yes,” keep driving. If “No,” pull over and check the engine.

Lastly Your License to Operate

You don’t need to be a mechanic to drive a Ferrari, but you do need to know if the brakes work and exactly where you’re headed. The “AI Profit OS” isn’t about replacing you—it’s about giving you the dashboard you’ve always wanted.

Are you ready to take the wheel?

Check out the AI Profit OS Course | Book a Strategic Call

1. The Plier Metaphor: What GenAI Actually Does

In my AI Umbrella framework, I categorize GenAI as a tool for handling Cognitive Load.

Pliers are great for a specific set of tasks: gripping, pulling, and twisting. Similarly, GenAI is world-class at: * Summarizing long documents. * Drafting emails or reports. * Answering basic customer queries. * Generating creative ideas.

But you wouldn’t use pliers to drive a screw (you’d use a screwdriver) or to dig a trench (you’d use an excavator). If you try to use a “Plier” (GenAI) to solve a “Math” problem or a “Relationship” problem, the tool will fail—or worse, it will “hallucinate” a solution that looks right but is dangerously wrong.

2. The Danger of “Plier-Only” Thinking

When a CEO tells me, “We have a GenAI strategy,” I usually hear, “We are trying to hammer every nail with pliers.” This is a primary reason why most AI projects fail.

Here is what happens when you use the wrong tool: * The Hallucination Trap: Using an LLM to “predict” inventory needs. LLMs are built for language probability, not mathematical optimization. * The Thin-Wrapper Problem: Building a “chat interface” on top of bad data. The “plating” looks nice, but the “food” is still terrible (as I discussed in AI Strategic Skills). * The ROI Gap: Spending millions on GPU tokens to do work that a simple linear regression (Traditional ML) could do faster, cheaper, and more accurately.

3. Building a Full Toolbox

To move from “Tier 1” (playing with tools) to “Tier 3” (The Strategist), you need to understand the rest of the toolbox.

As we explored in Decision-First AI, you must start with the Decision first, then pick the tool:

1. Need to predict churn? Use Machine Learning (The Torque Wrench).
2. Need to find the best delivery route? Use Optimization & Planning (The GPS).
3. Need to find fraud patterns in a network? Use Graph Algorithms (The X-Ray).
4. Need to summarize a 50-page contract? Now you pull out the GenAI Pliers.

4. The Executive’s Job: Choosing the Tool

Your job as a leader isn’t to know how to build the pliers. It’s to know when to use them.

In my “AI Profit OS” framework, we teach leaders to move past the hype and look at the “Umbrella.” If your technical team is suggesting a GenAI solution for a problem that requires structural data logic, you need to pull the emergency brake.

Are you building a skyscraper, or just playing with pliers?

Take the Next Step

Don’t let your strategy get stuck in “pilot purgatory” because you picked the wrong tool.

Join the AI Profit OS Cohort | Book a Strategic Call

If you need help identifying which tool in the AI Umbrella fits your specific business pain, let’s talk.

The #1 Silent Killer: Absence of Pain

If you cannot name the specific business pain you are solving, your project is already dead.

I recently audited a company where the C-Suite wanted “AI for efficiency.” That is not a pain; that is a wish. Because they couldn’t quantify the pain (e.g., “We are losing $50,000 per month in manual data entry errors”), they couldn’t quantify the Value of a solution.

The result was a project with no “End in Mind.” The team drifted, timelines slipped, and the budget evaporated because no one knew what “success” actually looked like.

The Fix: Before writing a single line of code, you must be able to fill in this blank:

“This AI solves [Specific Pain] which is currently costing us [$$$$] per month.”

The “Translation Error”: CEO vs. Data Scientist

This is the most common friction point I see in enterprise AI.

• The CEO says: “I want more revenue.”
• The Data Team hears: “I need a model with a high accuracy score.”

These are not the same thing. “More revenue” is a goal; high accuracy is just one possible tactic. For example, revenue can come from lowering churn, but only if you focus on the right customers. A 1% churn reduction on low-value customers is meaningless. You need to save the High-CLTV (Customer Lifetime Value) clients.

The Fluency Gap: The executives didn’t define the business lever (CLTV), and the tech team didn’t ask. The result is a technically “successful” model that drives zero actual revenue.

When to Kill an AI Project: The “Code Red” Checklist

Part of my job as a strategist is telling clients when to stop spending money. An AI project should be put on hold or killed immediately if it meets these criteria:

1. Invisible ROI: You are three months in and still cannot map the model’s output to a dollar value.
2. Missing Ingredients: The core data is missing or inaccessible. You can’t build a churn model if you aren’t tracking customer complaints or service failures.
3. The “30% Drift” Rule: If 30% of the project timeline has passed and the team still has “no clue” what the final output looks like or lacks stakeholder support.

The only exception is when the potential ROI is massive and justifies a high-risk, “Agile” approach to de-risk the project in smaller steps.

Case Study: How Fixing Definitions Saved a Fortune 500

I worked with a global Fortune 500 retailer whose AI initiatives were failing.

The Problem: Every country operated as a mini-company. They had different definitions for “Delivery Time,” “Stock in Hand,” and “Marketing Spend.” The Consequence: The global AI forecasting model was useless because the input data meant different things in different regions. This “Translation Error” was costing them millions in lost inventory.

The Turnaround: We didn’t start with a better AI model. We started with Definitions. By standardizing the meaning of “ROI” and “COGS” across their SAPMENA region, the data became clean. Once the data was clean, the AI model could actually forecast.

The lesson is simple: You cannot layer AI on top of organizational chaos. You must fix the business definitions first.

The “dataset-first” trap — what it looks like, and why it hurts

Here’s the typical playbook I see in companies:

• Someone discovers a new dataset.
  
• They build dashboards, then a model, then a fine model, then a fancier model.
  
• They show a demo. The demo gets applause. Then the work hits integration, governance, and the messy reality of people who must make decisions every day. The model’s outputs don’t map to a decision process. So adoption fails.

Why? Because the project optimized the wrong thing. It optimized prediction metrics — error, F1, AUC, MAPE. And those are useful. But they’re not the measure of business impact. A model with better accuracy can still be useless if it doesn’t change what someone does.

Harvard Business Review captured this idea well: decisions don’t start with data. They start with a problem, a role, a process, and a behavior. If your analytics don’t connect to that reality, you get slides and disappointment. Harvard Business Review

There are more subtle costs too. Dataset-first projects often create models that are brittle in production: they overfit to historical quirks, they require constant data wrangling, and they produce numbers no one trusts. That kills scale.

The Decision Map: simple framework you can use today

If you want to flip to decision-first, start with a small, disciplined tool I call a Decision Map. It’s a one-page artefact. Build it before you touch data.

Here’s the Decision Map — six steps. Do them in order.

• Name the decision. Who decides, how often, and what options do they choose? Example: “Field-ops decides whether to dispatch a technician to a suspected DSL fault.” Be specific. Frequency matters — hourly, daily, weekly change what you can do.

• Define the decision metric(s).What counts as success? Lower cost? Faster response time? Increase in net revenue? Pick one primary metric and one secondary. If you can’t name it in a single measurable sentence, you don’t have a decision.

• Map the current process. Where is the decision made today? Which people and tools are involved? Where does data enter? Where do delays happen? This step exposes the friction you must remove.

• Identify the minimal action the model must trigger. The model doesn’t need to be perfect. It needs to change behavior. If the model’s output is a probability, what threshold triggers action? Who gets the alert? What’s the handoff?

• List the minimal signals (data) needed. Only include data that directly reduces uncertainty for the decision. You’ll be surprised how small this list often is. Think: signal → action. Not “all the data.”

• Plan the feedback loop. How will you measure the decision metric after deployment? How will you collect labels and iterate? Decide that upfront.

If you complete this map, you’ll have done 80% of the work most teams skip. It forces alignment, and it reveals whether the project is worth doing.

P.S. - I am hosting a course where you’ll learn exactly how to implement this for your usecase.

Practical tips for teams (do this in week one)

1. Run a one-hour Decision Map workshop. Invite the decision owner, one operator, one engineer, and one product owner. Build the one-page map. If the owner can’t commit to a metric, pause the project.
2. Start with a simple rule baseline. Before modeling, define a rule that will be your baseline (e.g., “if X > T, create ticket”). If the model can’t beat that rule in decision impact, scrap it.
3. Measure decision impact, not model accuracy. Your dashboard should show business metric delta — not just RMSE. If you show the board a change in cost or conversion, you’ll get attention.
4. Prioritize deployment constraints. Decide telemetry, latency, and handoff requirements first. Models that can’t meet latency or trust constraints are useless no matter how accurate.
5. Iterate with real feedback. Don’t wait for “perfect.” Ship an MVP that can be measured, then refine. Real decisions provide labels and operational learning.

Learn how to exactly do this here.

One-page checklist (copy this)

• Decision name (The GOTO problem statement): ____________________
• Primary / Secondary metric (which can accounted for ROI calculation later ): _____
• Decision owner (I don’t want to debate on this): ________________________
• Frequency of predictions: real-time / hourly / daily / weekly / monthly
• Action / interventions which can be triggered by model: ____________________
• Minimal signals required( The core Data to begin with): ______________________
• Baseline rule (your fail-safe if everything goes wrong): ____________________
• Feedback source ( might be tricky, but you should have one): __________________

If you can fill this in, you’re set for a pilot.

Final note —
start small, measure fast, then scale with discipline

The decision-first approach is simple because business problems are simple when stated well. The hard part is discipline: saying no to shiny demos and yes to measurable change. Start with one decision that matters. Map it. Ship a small system that changes behavior. Measure the business metric. Iterate.

Research supports this: models trained with the decision in mind perform better on the actual outcomes you care about. And in practice, teams that flip the order — decision first, data second — get to value faster. arXiv+1

If you want help mapping a decision in your company, send me one line describing the decision and the current process. I’ll reply with the Decision Map template you can use in a one-hour workshop. Or if you want we can connect too.

If this post help you or think help someone in need, please do share it. Thanks!!!

What Is the AI Umbrella?

AI is a big term. It wraps up everything from crunching numbers in Excel to self-driving cars and chatbots/AI Agents. This umbrella covers many areas, and each one is changing fast. Some ideas fade, some thrive, and others turn into new things. AI is not a magic trick — and it’s not new. For over sixty years, it’s helped businesses do work better, from simple data entry to complex automation.

The AI Umbrella Framework

So how do you make AI work for real business needs? Use the “AI Umbrella” to map your problem and pick the right tool. Here’s a simple view:

It covers everything from classic number crunching to the newest breakthroughs. Think of AI like a big toolbox, not just one gadget. Well we are going to take a deeper dive in each pillar, but a small disclaimer this is not-an-exhaustive list.

The Market Numbers

Here’s what the data says about the real business side of AI:

• The whole AI market is about $235 billion.
• GenAI (think ChatGPT) is just 8.6% ($20.2 billion). The rest — traditional ML — is 91.4% ($194.6 billion).
• Traditional AI gets better results: up to 30% ROI, while GenAI averages 12%.
• Predictive maintenance delivers up to 400% ROI in six months. Fraud detection? 150% in under a year. LLMs do much less: about 12% over the same time.

Putting It All Together

Before starting any AI project, ask: What problem am I trying to solve? Use the AI Umbrella map to find what fits. Is it prediction, classification, pattern finding, or optimization? Do you need machine learning, deep learning, or just smarter software? This clarity saves money, time, and helps get actual results.

Next time someone pitches an AI solution, ask “Which part of the AI umbrella are we using?” And “How does it solve our actual problem?”, or You can learn to ask specific question for your use case,, but ask for clarity and not hype. Use the framework above as your map. AI isn’t a buzzword — it’s a toolbox. The right tool solves the right problem. And that’s how you make AI work for your business, today and tomorrow.

This is just the start. I’m building an Enterprise AI series that shows how to combine these AI pillars for real business impact.

Next may be: How one manufacturing company used ML, optimization, and robotics together to cut costs by 40%. Subscribe to see the full breakdown or book a strategic call

The Allure and Limitation of KNN

In the realm of machine learning, the K-Nearest Neighbors (KNN) algorithm stands out for its intuitive nature and ease of implementation. Its principle is simple: classify a data point based on the majority class among its ‘k’ nearest neighbors in the feature space. This non-parametric approach makes no assumptions about the underlying data distribution, rendering it versatile for various applications. KNN is very popular, but it comes with some limitations.

However, KNN’s Achilles’ heel lies in its reliance on distance metrics, which are inherently designed for numerical data. Real-world datasets often contain a mix of numerical and categorical features, posing a significant challenge for KNN. How do you measure the distance between ‘red’ and ‘blue,’ or ‘large’ and ‘small’?

Enter TrieKNN: A Novel Approach

What if we could cleverly sidestep the distance calculation problem for categorical features, while still leveraging KNN’s power? TrieKNN offers just that—a way to perform KNN on any mixed data!

TrieKNN combines the strengths of Trie data structures and KNN to handle mixed data types gracefully. Here’s the core idea:

1. Trie-Based Categorical Encoding: A Trie is used to store the categorical features of the data. Each node in the Trie represents a category.
2. Leaf-Node KNN Models: At the leaf nodes of the Trie, where specific combinations of categorical features are found, we fit individual KNN models using only the numerical features.
3. Weighted Prediction: To classify a new data point, we traverse the Trie based on its categorical features. At each level, we calculate a weighted distance based on available data, ending in a probability score in each leaf node.

Building a TrieKNN Model

Let’s dive into the implementation. We’ll start with the TrieNode and Trie classes, then move on to the KNN model and the training/prediction process.

import numpy as np
from collections import Counter

class TrieNode:
    def __init__(self):
        self.children = {}  # Dictionary to store child nodes
        self.is_end_of_word = False  # True if the node is the end of a word
        self.count = 0  # Count of how many times a word has been inserted
        self.class_counts = {}  # Class counts
        self.class_weights = {}
        self.model = None  # Model at leaf nodes
        self.indexes = []  # Store data indexes belonging to this leaf
        self.labels = []  # Store data indexes belonging to this leaf
        self.node_weight = None

class Trie:
    def __init__(self):
        self.root = TrieNode()  # Root node of the Trie
        self.data_index = 0  # Initialize data index

    def insert(self, word_val, model):
        current_node = self.root
        word, val = word_val
        current_node.count += 1

        # Adding class counts
        if val not in current_node.class_counts:
            current_node.class_counts[val] = 0
        current_node.class_counts[val] += 1

        for char in word:
            # If the character is not in children, add a new TrieNode
            if char not in current_node.children:
                current_node.children[char] = TrieNode()
            current_node = current_node.children[char]

            # Adding count of instances
            current_node.count += 1

            # adding class counts
            if val not in current_node.class_counts:
                current_node.class_counts[val] = 0
            current_node.class_counts[val] += 1

        # Mark the end of the word and increment count
        current_node.is_end_of_word = True
        current_node.indexes.append(self.data_index)  # Store the data index
        current_node.labels.append(val)
        current_node.model = model
        self.data_index += 1  # Increment data index

    def search(self, word):
        current_node = self.root
        for char in word:
            # If the character doesn't exist in the children, the word doesn't exist
            if char not in current_node.children:
                return False
            current_node = current_node.children[char]

        # Return True if it's the end of a word and the word exists
        return current_node.is_end_of_word

    def count_word(self, word):
        current_node = self.root
        for char in word:
            # If the character doesn't exist, the word doesn't exist
            if char not in current_node.children:
                return 0, current_node.class_counts  # Correctly return class_counts
            current_node = current_node.children[char]

        # Return the count of the word
        return current_node.count, current_node.class_counts

    def display(self):
        # Recursively display the tree
        def _display(node, word):
            if node.is_end_of_word:
                print(f"Data: {word}, Count: {node.count}, Indexes: {len(node.indexes)} Classes :{node.class_counts} weights:{len(node.class_weights)}")  # Display indexes too
            for char, child in node.children.items():
                _display(child, word + char)  # corrected the display

        _display(self.root, "")

    def apply(self, func):
        """
        Applies a function to all models in the leaf nodes.
        """
        def _apply(node):
            if node.is_end_of_word and node.model is not None:
                func(node)
            for child in node.children.values():
                _apply(child)

        _apply(self.root)

    def apply_weight_to_indexes(self, weight):
        """
        Applies a weight to the indexes based on the percentage of data available.
        """
        def _apply_weight_to_indexes(node):
            if node.is_end_of_word:
                total_count = sum(self.root.children[child].count for child in self.root.children)
                percentage = node.count / total_count if total_count > 0 else 0
                weighted_indexes = [(index, weight * percentage) for index in node.indexes]
                node.class_weights = weighted_indexes  # Corrected this line
            for child in node.children.values():
                _apply_weight_to_indexes(child)

        _apply_weight_to_indexes(self.root)

class KNNModel:
    def __init__(self, k=5):
        self.data = None
        self.labels = []
        self.k = k

    def fit(self, data, indexes, labels):
        # print("Fitting model with indexes:", len(indexes), "labels:", len(labels))
        self.data = data[indexes].astype(float)
        self.labels = np.array(labels).astype(float)

    def predict(self, data):
        # print("Predicting with data:", data)
        dist_ind = np.sqrt(np.sum((self.data - data) ** 2, axis=1) ** 2)  # euclidean distance
        main_arr = np.column_stack((self.labels, dist_ind))  # labels with distance
        main = main_arr[main_arr[:, 1].argsort()]  # sorting based on distance
        count = Counter(main[0:self.k, 0])  # counting labels
        sums = np.array(list(count.values()))  # getting counts
        return sums / np.sum(sums)  # prediction as probability

# Sample data
n = 10000
data = np.array((np.random.choice(['Anything ', 'By ','Chance '], p=[0.6,0.1,0.3],size=n),
                 np.random.choice(['can', 'go', 'here','lets', 'see', "it"], p=[0.1, 0.1, 0.1, 0.2, 0.4, 0.1], size=n),
                 np.random.normal(3, 1, size=n),
                 np.random.normal(5, 2, size=n))).T
y_label = np.random.choice([0,1], p=[0.7, 0.3], size=n)

# Trie training
trie = Trie()
for X, y in zip(data, y_label):
    trie.insert((X[:2], y),None)

# Apply weights to indexes
trie.apply_weight_to_indexes(0.5)

# Fit models of leaf nodes
def add_model(node, data):
    node.model = KNNModel()
    node.model.fit(data, node.indexes, node.labels)

def traverse_and_add_model(node, data):
    if node.is_end_of_word:
        add_model(node, data)  # Add model to leaf node
    for child in node.children.values():
        traverse_and_add_model(child, data)

traverse_and_add_model(trie.root, data[:, 2:])

Conclusion: A Promising Path Forward

TrieKNN presents a compelling solution for extending the applicability of KNN to mixed data types. By leveraging the Trie data structure, it avoids direct distance calculations on categorical features, enabling the use of localized KNN models for numerical data.

Further research could explore:

• Optimizing the weighting scheme for combining predictions from different Trie levels.
• Comparing TrieKNN’s performance against other mixed-data KNN approaches on benchmark datasets.
• Extending TrieKNN to handle missing data and noisy categorical features.

TrieKNN opens up new possibilities for applying KNN in domains where mixed data types are prevalent, such as healthcare, e-commerce, and social science.

Resources and further reads:
1. Nomclust R package
2. An Improved kNN Based on Class Contribution and Feature Weighting
3. An Improved Weighted KNN Algorithm for Imbalanced Data Classification
4. A weighting approach for KNN classifier
5. Unsupervised Outlier Detection for Mixed-Valued Dataset Based on the Adaptive k-Nearest Neighbor Global Network
6. A hybrid approach based on k-nearest neighbors and decision tree for software fault prediction
7. Analysis of Decision Tree and K-Nearest Neighbor Algorithm in the Classification of Breast Cancer

Introduction: A Journey into Data

Picture this: you’re standing on the icy shores of Antarctica, the wind whipping around you as you watch a colony of Palmer Penguins waddling about, oblivious to the data detective work you’re about to embark on. As a data science architect, you’re not just an observer; you’re a sleuth armed with algorithms and insights, ready to unravel the mysteries hidden within data. Today, we’ll transform raw numbers into powerful narratives using CrossTab Sparsity as our guiding compass. This blog post will demonstrate how this metric can revolutionize classification tasks, shedding light on many fascinating datasets—the charming Palmer Penguins and the serious Obesity, Credit cards data and many more.

The Power of CrossTab Sparsity

Data Overview: Our Data People at work here

Impact on Model Accuracy

After applying feature selection based on CrossTab Sparsity, we’ll compare the accuracy of our new models against our baseline models. This comparison will reveal how effectively CrossTab Sparsity enhances classification performance.

metrics = {
    "Metric": ["Accuracy", "Precision", "Recall", "F1-Score"],
    "Baseline Model with all Parameters": [
        accuracy_score(y_test, base_preds),
        precision_score(y_test, base_preds, average='weighted'),
        recall_score(y_test, base_preds, average='weighted'),
        f1_score(y_test, base_preds, average='weighted'),
    ],
    "Challenger Model with only 5 Variables": [
        accuracy_score(y_test, challenger_preds),
        precision_score(y_test, challenger_preds, average='weighted'),
        recall_score(y_test, challenger_preds, average='weighted'),
        f1_score(y_test, challenger_preds, average='weighted'),
    ]
}
display(pd.DataFrame(metrics).round(4).set_index('Metric').T)

Reduction of similar accuracy from 16 to 5 i.e 68.75% reduction

Conclusion: The Road Ahead

In conclusion, this blog has illustrated how CrossTab Sparsity can be a game-changer in classification tasks using the Obesity dataset. By leveraging this metric for feature selection, we achieved notable improvements in model performance—proof that sometimes less really is more!

Future Work: Expanding Horizons

As we look ahead, there are exciting avenues to explore:

• Investigating regression problems using CrossTab Sparsity.
• Comparing its effectiveness with other feature selection methods such as Recursive Feature Elimination (RFE) or comparision with other feature selection mehtods.

By continuing this journey into data science, we not only enhance our technical skills but also contribute valuable insights that can drive meaningful change in various industries.

Introduction

Cluster analysis has always fascinated me as a window into the hidden structures of data. During my collaboration with Kumarjit Pathak, we grappled with a persistent challenge in unsupervised learning: how to objectively evaluate clustering quality across different algorithms. Traditional metrics like the Silhouette Index or Bayesian Information Criterion felt restrictive—they were siloed within specific methodologies, making cross-algorithm comparisons unreliable.

This frustration led us to develop a universal cluster evaluation metric, detailed in our paper “Cross Comparison of Results from Different Clustering Approaches”. Our goal was to create a framework that transcends algorithmic biases, enabling:
- Direct comparison of K-Means vs GMM vs DBSCAN vs PAM vs SOM vs Anything results
- Identification of variables muddying cluster separation
- Automated determination of optimal cluster counts

In this blog, I’ll walk you through our journey—from conceptualization to real-world validation—and share insights that didn’t make it into the final paper.

Comparative Advantages and Creativity at Work

Comparative Advantage Over Traditional Metrics

Our creativity yielding boons. We wanted a simple metric to judge different kind of cluster, but we got much more from our experiments and work on this metric:

1. Variable-Level Diagnostics: Low scores pinpoint variables muddying cluster separation.
   
2. Cross-Method Benchmarking: Compare K-Means (distance) vs GMM (probability) vs hierarchical vs partition clustering fairly using a unified score.
   
3. Scale Invariance: Logarithmic term makes scores comparable across datasets of varying sizes.
   
4. Debug Cluster Quality: Identify and remove noisy variables preemptively
   
5. Automate Model Selection: Objectively choose between K-Means, GMM, PAM, Agglomerative.

Lessons Learned and Future Vision

Few take away from these experiments
1. Binning Sensitivity: Quantile-based binning was transformation. Equal-width bins distorted scores for skewed variables.
2. Categorical Handling: Native levels for categorical outperformed frequency-based grouping.
3. Non-Parametric Approach: This approach allowed us to make sense of data without being tied down by assumptions. We have seen how this metric can be a game-changer for statisticians, providing insights not just into cluster behavior but also into rare event modeling.

The plots from these experiments not only clarify how clusters behave but also offer valuable insights for identifying outliers. I believe there’s exciting potential to extend this metric into classification and value estimation modeling. Imagine using it as a loss function in both linear and non-linear methods to achieve better data segmentation! Thing for another blog someday!

What are the results

Pheww… Thats too much code right. But that would save a lot of your time in real life. So in real life you would write code as 3 steps below:

# import libraries
import pandas as pd

# Getting car data from UCI
df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data',
                 sep='\s+',header=None,
                 names=['mpg','cylinders','displacement','horsepower','weight',
                 'acceleration','model_year','origin','car_name'])
df.head()

# Syntax to do anove with validating the assumption, doing test and a post-hoc
results = do_anova_test(df=df, res_var='mpg',xfac_var='cylinders', 
                        anova_model='mpg ~ C(cylinders)+C(origin)+C(cylinders):C(origin)',
                        ss_typ=3, result_full=True)

Results form the do_anova_test

Levens Test Result:
                 Parameter    Value
0      Test statistics (W)  14.5856
1  Degrees of freedom (Df)   4.0000
2                  p value   0.0000

Bartletts Test Result:
                 Parameter    Value
0      Test statistics (T)  61.2143
1  Degrees of freedom (Df)   4.0000
2                  p value   0.0000

ANOVA\ANCOVA Test Result:
                           df     sum_sq    mean_sq         F  PR(>F)      n2
Intercept                 1.0  6195.1701  6195.1701  296.3452  0.0000  0.2727
C(cylinders)              4.0  7574.5864  1893.6466   90.5824  0.0000  0.3334
C(origin)                 2.0   241.0703   120.5351    5.7658  0.0034  0.0106
C(cylinders):C(origin)    8.0   577.4821    72.1853    3.4530  0.0046  0.0254
Residual                389.0  8132.1404    20.9052       NaN     NaN     NaN

Tukey HSD Result:
   group1  group2     Diff    Lower    Upper  q-value  p-value
0       8       4  14.3237  12.8090  15.8383  36.6527   0.0010
1       8       6   5.0226   3.1804   6.8648  10.5671   0.0010
2       8       3   5.5869  -0.7990  11.9728   3.3909   0.1183
3       8       5  12.4036   5.0643  19.7428   6.5503   0.0010
4       4       6   9.3011   7.6765  10.9256  22.1910   0.0010
5       4       3   8.7368   2.4102  15.0633   5.3524   0.0017
6       4       5   1.9201  -5.3676   9.2078   1.0212   0.9000
7       6       3   0.5643  -5.8486   6.9772   0.3410   0.9000
8       6       5   7.3810   0.0182  14.7437   3.8854   0.0491
9       3       5   6.8167  -2.7539  16.3873   2.7606   0.2919

Nice!!!

And plotting is even easier

# Numbers are clumsy for most. Making more interpretable plot on above results.
plot_hsd(results.tukeyhsd.sort_values('Diff'), title="Tukey HSD resutls Anova of MPG ~ Cylinder")

Results form the plot_hsd

Plots look good with ‘p-values’.

Conclusion

Now since we applied the above to a Non RCT we cannot conclude that Difference in mpg based on cylinder is huge specially as number of cylinders goes up. But this statement might not be as explicit as might be appearing from plot. Unless you have a strong believe that the data follows with rules and assumptions of RCTs, we should be only seeking interesting as in associated results and not cause-effect results.

Give me “The Code”

from bioinfokit import analys

import numpy as np
from scipy import stats

class KeyResults:
    """
    A basic class to hold all the results
    """
    
    def __init__(self,result_full):
        self.keys = []
        self.result_full = result_full
    
    def add_result(self,name,result):
        if name == 'tukeyhsd':
            self.keys.append(name)
            setattr(self, name, result)
        elif self.result_full:
            self.keys.append(name)
            setattr(self, name, result)


# Anova test code
def do_anova_test(df, res_var, xfac_var, anova_model,ss_typ=3,
                  effectsize='n2',result_full=False,add_res=False):
    """
    Do all sequential anova tests
    
    Step 1) Leven's/ bartellet test for checking weather variance is homogenous or not
    Step 2) Main ANOVA/ANCOVA test
    Step 3) Tukey's HSD for individual combinations
    
    :param df: Pandas DataFrame holding all the columns
    :param res_var: Variable for which we are checking ANOVA
    :param xfac_var: Grouping Variables for which we want to do the comparisons
    :param anova_model: SM formula for the model. This is life savour to make all things work
    :param result_full: To provide all the results of intermediate steps
    
    """

    results = KeyResults(result_full)
    
    # initialize stat method
    res = analys.stat()
    
    # doing levens test
    res.levene(df=df, res_var=res_var,xfac_var=xfac_var)
    print('\nLeven\'s Test Result:')
    print(res.levene_summary)
    results.add_result('levene',res.levene_summary)

    # doing bartlett test
    res.bartlett(df=df, res_var=res_var,xfac_var=xfac_var)
    print('\nBartlett\'s Test Result:')
    print(res.bartlett_summary)
    results.add_result('bartlett',res.bartlett_summary)
    
    # doing anova / ancova
    res.anova_stat(df=df, res_var=res_var, anova_model=anova_model,ss_typ=ss_typ)
    aov_res = res.anova_summary
    
    # Add effect sizes
    if effectsize == "n2":
        all_effsize = (aov_res['sum_sq'] / aov_res['sum_sq'].sum()).to_numpy()
        all_effsize[-1] = np.nan
    else:
        ss_resid = aov_res['sum_sq'].iloc[-1]
        all_effsize = aov_res['sum_sq'].apply(lambda x: x / (x + ss_resid)).to_numpy()
        all_effsize[-1] = np.nan
    aov_res[effectsize] = all_effsize
    #aov_res['bw_'] = res.anova_model_out.params.iloc[-1]
    aov_res = aov_res.round(4)
    
    # printing results
    print('\nANOVA\ANCOVA Test Result:')
    print(aov_res)
    results.add_result('anova',res.anova_summary.round(4))
    results.add_result('anova_model',res.anova_model_out)
    
    # doing tukey's hsd top compare the groups
    res.tukey_hsd(df=df, res_var=res_var,xfac_var=xfac_var, anova_model=anova_model,ss_typ=ss_typ)
    print('\nTukey HSD Result:')
    print(res.tukey_summary.round(4))
    results.add_result('tukeyhsd',res.tukey_summary.round(4))
    
    # add all result componets again if needed 
    if add_res:
        results.add_result('allresult',res)
    
    return results

import seaborn as sns
import matplotlib.pyplot as plt

plt.style.use('seaborn-bright')

def plot_hsd(hsdres,p_cutoff=0.05,title=None,ax=None,figsize=(10,7)):
     """
     Do plotting of tukeyhsd results
    
  
    :param hsdres: 'tukeyhsd' result form the do_anova_test function
    :param p_cutoff: Cutoff at which we get say a combination is significant
    :param title: Title of the plot
    :param ax: Define or get the matplotlib axes
    :param figsize: Mention Figure size to draw
    
    """

    if ax is None:
        fig,axp = plt.subplots(figsize=figsize)
    else:
        axp = ax
    
    # helper func
    p_ind = lambda x : '' if x > 0.1 else ('+' if x > 0.05 else ('*' if x > 0.01 else ('**' if x >0.001 else '***')))
    label_gen  = lambda x: f"${x[0]} - {x[1]}\ |\ p:{x[2]:0.2f}{p_ind(x[2]):5s}$"
    
    #setting values
    mask = hsdres['p-value'] <= p_cutoff
    yticklabs = hsdres[['group1','group2','p-value']].apply(label_gen,axis=1).values
    ys = np.arange(len(hsdres))
    
    # adding plot to axes
    axp.errorbar(hsdres[~mask]['Diff'],ys[~mask],xerr=np.abs(hsdres[~mask][['Lower',"Upper"]]).values.T,
                fmt='o', color='black', ecolor='lightgray', elinewidth=2, capsize=0)
    axp.errorbar(hsdres[mask]['Diff'],ys[mask],xerr=np.abs(hsdres[mask][['Lower',"Upper"]]).values.T,
                fmt='o', color='red', ecolor='pink', elinewidth=2, capsize=5)
    axp.axvline(x=0,linestyle='--',c='skyblue')
    axp.set_yticks([])
    (l,u) = axp.get_xlim()
    axp.set_xlim(l+1.5*l,u)
    (l,u) = axp.get_xlim()
    for idx,labs in enumerate(yticklabs):
        axp.text(l-0.1*l,ys[idx],labs)
    axp.set_yticklabels([])
    
    # finally doing what is needed
    if ax is None:
        plt.title('' if title is None else title,fontsize=14)
        plt.show()
    else:
        return axp

Hope this give you kickstart to find you intresting patterns. Happy Learning!

So what we do Now!

Astonished !!! Don’t be. This is what has been hiding behind those numbers. And this is why we really won’t be able to cross certain performance level. Unless you change some features or even do a lot of hyper parameter tuning, your results won’t vary much.

If you look at the average value of MAPE in each decile you would see an interesting pattern. Let us show you what we see that pattern. One day while working on a business problem where I was using regression on a discussion with Kumarjit, we deviced a different way of model diagnosis. We worked together to give this a shape and build on it.

As you can see it is absolutely evident that either of the side in the distribution of MAPE values is going wild!!!!!!! Still overall MAPE is good (18%).

Final Analysis

Now the DAR framework essentially fixes this weakness of regression method and understands the behavior of data which is stable and can be tweak in a fashion that can be use in general practice.

Plot of MAPE Bath Tub Curve after applying DAR Framework for Decile Split “mdev” from Data

How this framework with same method reduced MAPEs so much and made model much more stable…?? Well here it is:

The DAR framework splits the data at either ends of the order target variable and performs regression on these “split” data individually. This inherently reduces the so called “noise” part of the data and treat it as an individual data.

Scoring on New Data

Now you might be thinking while applying regression this sounds good but how will one score this on new data. Well to answer that we used our most simple yet very effective friend “KNN” (Though any multiclass Classifier can be used here). So ideally scoring involves two step method :

1. Score new value against each KNN / Multiclass Classifier model of the data
2. Based on closeness we score it with the regression method used for that part of data.

So now we know how we can improve the prediction power of data for regression.

Code and Flowchart

If things are simple lets keep it simple. Refer flowchart and code below for implementation of this framework. Paper here!

graph TB
    
    subgraph Testing
        p1(Finding bucket of model to choose)
        p1 --> p2([Making predictions <br> based on selected model for inference])
        p2 --> p3(Consolidate final score of prediction)
    end

    subgraph Training
        md([Fitting a <br>Regression model])==> di
        di{Binning Data via <br/> evaluating Distribution <br/> MAPE values }
        di --> md2([Fitting a Buckteing model <br/> to Binned MAPE Buckets])
        md2 --> md3([Fitting Regression <br> Models on Binned Data])
        md == Keeping main<br/>model ==> ro        
        md3 ==> ro(Final Models <br> Binning Data Models + <br> Set of Regressoin Models)
    end

    
    od([Data Input]) -- Training<br> Data--> md
    od -- Testing<br> Data--> p1
    ro -.-> p1
    ro -.-> p2

    classDef green fill:#9f6,stroke:#333,stroke-width:2px;
    classDef yellow fill:#ff6,stroke:#333,stroke-width:2px;
    classDef blue fill:#00f,stroke:#333,stroke-width:2px,color:#fff;
    classDef orange fill:#f96,stroke:#333,stroke-width:4px;
    class md,md2,md3 green
    class di orange
    class p1,p2 yellow
    class ro,p3 blue