Salesforce Implementation Strategies & Designs - Part 4

Data or Process Issue and what to do?

You will eventually run into whether an issue is a data or process issue with the company and inevitably it becomes a "reconciliation issue.”

Data Issue vs. Process Issue: These two different failure modes look identical from the outside:

What are we trying to solve for? The why – right. So our goal is to find why the system underperforms, rather than just measuring that it underperforms.

When a quote goes wrong, or confidence scores drift, or the human override rate climbs, the operational report tells you something's wrong but not what kind of wrong. Splitting it into these two categories matters because the fix is completely different:

• Data issue: The case record itself is incomplete, stale, mislabeled, or inconsistent going into the semantic layer. The classification model is working correctly, but it's being fed bad or missing inputs.

  • Symptom: confidence scores that look erratic or arbitrarily low across otherwise-similar cases; the same case type produces extremely different scores depending on who entered it or which system it came from.

• Process issue: The data is fine, but the business rules, thresholds, or workflow steps themselves are wrong, outdated, or being applied inconsistently across locations/teams.

  • Symptom: confidence scores are stable and sensible, but the outcomes (YES/NO rates, override rates) don't match expectations, e.g., one location consistently overrides high-confidence quotes while another doesn't, even on similar case types.

Why this becomes a "reconciliation issue":

Reconciliation Defn: “In technology, reconciliation is the process of comparing data from two or more sources, finding differences, and fixing them so that all systems match and show the same correct information.”

Here's the key insight for the use case: when you can't immediately tell whether it's data or process, you're forced into reconciliation. This means going back and comparing what the system recorded against what actually happened (the ground truth of the case, the client's actual decision, the correct outcome) to figure out where the divergence originated.

Design note: This is expensive and slow if it's not designed for upfront, because:

• If it's a data issue, reconciliation means auditing input sources, entry points, and integration points (is the CRM system feeding clean data? Is a field being dropped between systems?).

• If it's a process issue, reconciliation means auditing the rules themselves are the thresholds still calibrated correctly, are jurisdictional overlays out of date, are teams applying the human-override step inconsistently?

Practical implication for the pipeline design:

This is a strong argument for building the diagnostic capability into the operational reporting from day one rather than bolting it on later, i.e., every case record should carry enough metadata (source system, entry timestamp, which ruleset/version applied, which location) that when a reconciliation issue surfaces, you can immediately narrow it to "this is a data quality problem in Region (remember the jurisdictional layer) X's intake" vs. "this is a rules-drift problem in the confidence model" instead of a manual investigation each time.

This also ties back to something we touched on earlier: the feedback loop from human YES/NO decisions. If you tag why a human overrode a quote (bad data vs. bad rule vs. genuinely ambiguous case), that tagging becomes the raw material for reconciliation, you're not just measuring whether the system got it right, you're measuring where in the pipeline it broke.

I put together a simple decision-tree or checklist ("is this a data issue or a process issue?") that users can apply when their own confidence scores or override rates start drifting.

CHECKLIST:

Here's a companion checklist you can walk through whenever the tree above doesn't give a clean answer on the first pass, or when you want to document the investigation for an operational report.

Step 1: Confirm the symptom before diagnosing

• Note exactly what triggered the review: a confidence-score anomaly, a rising override rate, a client complaint, or a reconciliation mismatch.

• Confirm the symptom is reproducible, not a one-off (check at least 3–5 similar cases).

Step 2: Data Issue Checklist

I like to check the source system first to confirm this is not the issue so I can move my full attention to the more granular components that may be the issue in the target system.

• Pull the raw case record as it entered the semantic (deterministic) layer - is any field missing, null, or mistyped?

• Compare the same case type across different entry sources (Source vs Salesforce vs manual entry vs API). Do scores diverge by source?

• Check timestamps. Is the model scoring against stale or outdated data?

• Trace the field lineage. Field lineage or column level lineage, tracks how individual data fields or columns move, change, and derive their values from raw sources to final reports, eg. Dependency mappings, column gets renamed or moved/deleted, aggregations error). Ask did a field get dropped or transformed incorrectly between systems? Begin by tracing the record's transaction history back to the automation or user action that wrote the incorrect data.

·        SF Best Practice: Because data can be manipulated by users, background automations, or Apex code, salesforce provides a step-by-step triage workflow to pinpoint the source of the bad value. Use their best practices which are readily available online.

·        A Salesforce Triage example:

·        Step 1: Check Field History Tracking (The Data Audit)

·        Step 2: Map Static Metadata Dependencies (The Lineage Plan)

·        Step 3: Run a Flow / Automation Check

·        Step 4: Capture Traces with Debug Logs (The Ultimate Root Cause)

• What Field Lineage Shows in Salesforce:

·        Automatic Updates: Identifies the Flows, Apex classes, Process Builders, or rules that write data into the field.

·        Manual Entry Points: Shows the page layouts, Lightning pages, and quick actions where users can fill or modify the field.

·        Permissions: Lists the profiles and permission sets granting create or edit access to the field.

·        Risks & Conflicts: Highlights data integrity issues, such as multiple automations writing to the same field at the same time

Step 3: Process Issue Checklist

• Confirm the data is clean (Step 2 passes and data sets are clean) before assuming process.

• Compare outcomes for the same case type across locations, teams, or reviewers — is variance coming from who applies the rule, not the rule itself?

• Check the ruleset version. Is every location running the same version of the threshold/business rules, or has one location drifted out of sync?

• Review recent rule changes. Did a threshold or jurisdictional overlay change without being communicated or retrained?

Step 4: Tag and Log the Finding

• Record the finding as one of: data issue, process issue, or genuinely ambiguous. This tag is what feeds your reconciliation reporting later.

• Note the specific field, rule, or team involved, so the fix is traceable, not just "resolved."

Step 5: Close the Loop

• Data issue > fix at the source (intake validation, integration mapping), not by adjusting the confidence model.

• Process issue > update the rule/threshold and communicate the version change to all locations.

• Ambiguous (unresolved) > route to human review and flag it as a candidate for a new rule, since recurring ambiguous cases often signal a gap in the semantic layer's coverage rather than a one-off exception.