Customer data quality: deduplication, phone and email normalization, how to measure and maintain it

One customer can sit in a CRM as 3 rows: the number 0912345678, the number +84912345678, and an email with one mistyped letter. Reports count 3 customers, campaigns send 3 times, and no one knows which row is right.

This article covers quality dimensions, normalizing phone numbers and emails, deduplication, metrics with formulas and blocking errors at entry. Sources: GOV.UK, Google, the IETF, OWASP, PostgreSQL, Splink, Decision 798/QD-BTTTT and Law 91/2025/QH15 (via transcribed copies), checked on October 22, 2026.

Click Star already has an article on unifying customer data from forms, Zalo, Facebook and calls; this one does not repeat it. Click Star sells CRM, CDP and data dashboard services, so it has an interest here.

Key points

• GOV.UK (June 24, 2021), following DAMA UK, recommends 6 dimensions: accuracy, completeness, uniqueness, consistency, timeliness, validity. Valid does not mean correct. ISO/IEC 25012 has 15 characteristics.
• Under Decision 798/QD-BTTTT, parallel dialing ended on November 14, 2018; from November 15, 2018 calls to an old code get only a voice notice, kept until June 30, 2019. An ordinary 11-digit mobile number in a CRM is legacy data.
• Email syntax is a coarse filter. Only a confirmation link shows control of the mailbox; do not guess or auto-correct addresses.
• Deduplication balances missed merges against wrong ones: auto-merge only on a certain key, set the fuzzy threshold yourself, and log merges so they can be undone.
• Every metric needs a formula and an owner. No standard level is given because Click Star found no open source.

What dimensions make up customer data quality

According to the Government Data Quality Hub on GOV.UK (June 24, 2021), data quality is split into dimensions; one is not enough, so pick those that fit the intended use.

The 6 DAMA UK dimensions: accuracy means data reflects reality; completeness means the data needed is present, not 100% of fields; uniqueness measures duplicates; consistency means no conflicts within or across datasets; timeliness means data is available when needed; validity means the right format, type and range.

The GOV.UK article makes 3 points. Completeness depends on purpose. A record can be a duplicate even when some fields differ, such as 2 records with different addresses for one person. Valid does not guarantee correct (its example is eye color); Click Star's phone example is a well-formed number that now belongs to someone else.

According to the IEC Webstore page, ISO/IEC 25012:2008 is a quality model for structured data with 15 characteristics from 2 viewpoints: inherent and system dependent.

According to iso25000.com (a secondary source, no date shown), 5 characteristics are inherent (such as Accuracy), 7 belong to both groups (such as Traceability) and 3 are system dependent (such as Recoverability). The full text is sold for a fee and unread, so Click Star uses the 6 DAMA UK dimensions as its measuring frame.

Normalizing Vietnamese phone numbers

Principle: store one canonical form for comparison and keep the original text the customer typed.

The 2018 network code change. Click Star read the English transcription of Decision 798/QD-BTTTT (signed by the Ministry of Information and Communications on May 25, 2018) on english.luatvietnam.vn, not an official portal. The plan converts 21 codes of 11 digits into 21 of 10 digits; the last 7 digits stay.

Examples: MobiFone 120, 121, 122, 126, 128 become 70, 79, 77, 76, 78; VinaPhone 123, 124, 125, 127, 129 become 83, 84, 85, 81, 82; Viettel 162 to 169 become 32 to 39; Vietnamobile 186, 188 become 56, 58; Gmobile 199 becomes 59.

Milestones: start at 00:00 on September 15, 2018; parallel dialing until 23:59 on November 14, 2018; from November 15, 2018 calls to an old code get only a voice notice, kept until 23:59 on June 30, 2019, after which only the new code connects.

The main text of the copy (section 2.4) wrongly shows 2019 for the parallel dialing start; the appendix and press reports give 2018, which Click Star uses.

What this means for data. An ordinary 11-digit mobile number in a CRM no longer connects. The text omits the leading 0 that CRM numbers usually carry (0123 4567890 becomes 083 4567890). Convert the prefix and mark it "unverified", since the table does not say whether the line is still used.

Machine (M2M) lines follow their own schedule: the copy lets carriers move them to 1x codes from July 1, 2018 to December 31, 2020. Not covered here.

Validation library. The libphonenumber README (Google) describes isPossibleNumber (a quick guess by length only) and isValidNumber (a check by length and prefix), plus getNumberType, isNumberMatch and AsYouTypeFormatter. It gives only a range's original carrier, and some releases change only metadata, so update regularly.

Email: syntax only as far as needed, verification by sending mail

Syntax checks remove only obvious typos; whether an address exists and belongs to the right person must be confirmed by sending mail. RFC 5321 (IETF, October 2008) caps the part before @ at 64 octets and the domain at 255 octets.

OWASP (cheat sheet, no date shown) advises well-tested libraries over hand-written regular expressions and rejecting only clearly malformed addresses.

Comparison. RFC 5321 says domains are case-insensitive, while the part before @ must be treated as case-sensitive, though relying on that is discouraged. OWASP advises keeping the original and a canonical form, lowercasing the domain, and avoiding provider-specific rules (such as removing Gmail dots).

Click Star's suggestion: a fully lowercased comparison key to find duplicate candidates, the original kept, the policy written down, and never that key for login (risk of mistaken accounts).

Verification. OWASP requires random, single-use, time-limited tokens. For marketing, Click Star suggests counting an email as "confirmed" only after a click on the link. It does not recommend auto-correcting "gmial.com" to "gmail.com"; ask the person.

Deduplication: certain keys, fuzzy matching, merging

Deduplication means finding records of one person and deciding whether to merge them. According to the Splink documentation, the deterministic approach uses rules, is cheap and precise but misses matches; the probabilistic one scores evidence from several fields.

In the probabilistic approach you choose the threshold: higher means fewer wrong merges, lower means fewer misses; the documentation gives no universal threshold.

Certain keys. A field that almost always belongs to one person, normalized and verified, such as an E.164 number or a confirmed email.

Blocking. Comparing every pair makes comparisons grow very fast. Splink compares only pairs sharing some fields, with many strict rules instead of a few loose ones.

Fuzzy matching. PostgreSQL's pg_trgm module compares strings by 3-character groups, offers GIN or GiST indexes, and defaults to a threshold of 0.3 for the % operator. That 0.3 is a module default, not a quality standard; test it on your own sample.

Vietnamese diacritics. Unicode has 2 ways to represent one accented letter. UAX #15 (Unicode 18.0.0, August 12, 2026) calls this canonical equivalence; NFC uses precomposed characters. Convert names to NFC before comparing.

Click Star's inference: strip diacritics only to find candidates, never to auto-merge, since 2 names differing by diacritics may be 2 people.

Database constraint. According to the PostgreSQL 18 documentation, a unique index prevents 2 rows with the same value, and nulls are not treated as equal by default. An index on lower(email) also forbids 2 addresses differing only in case.

Merging with a log. Click Star's proposal, not from an outside source: mark the original merged instead of deleting it, record rule, score, time and reviewer so it can be undone, and keep consent status per purpose.

Measuring quality with metrics that have formulas

GOV.UK says each organization should measure and monitor its critical data against suitable dimensions, on an ongoing basis. The metric set below is a Click Star reference frame, not a measurement and not from an outside source.

• Missing required field rate = records missing a required field ÷ all records.
• Valid phone rate = records whose number passes isValidNumber ÷ records with a number.
• Confirmed email rate = records with an email that clicked the link ÷ records with an email.
• Hard bounce rate = permanently bounced messages ÷ messages sent, per campaign.
• Suspected duplicate rate = records in suspected duplicate groups ÷ all records.
• Wrong merge rate = wrongly merged pairs ÷ merged pairs in a manually checked sample.
• Fresh record rate = records confirmed within N days ÷ all records, with N chosen by the business.

Click Star gives no "pass" level because no open source offers one. Measure first, then set targets by trend and cost of errors.

Accuracy needs a sample verified by contact.

Blocking errors at entry and cleaning on a schedule

Fixing errors at entry keeps them out of reports and campaigns. Click Star proposes 5 layers (reference frame).

1. Form: format numbers as users type (AsYouTypeFormatter), show clear errors, block only when clearly wrong.
2. API and server: validate again, since OWASP says browser-side checks can be bypassed.
3. Database: a unique index on the normalized value.
4. Verification: send a confirmation link to the email.
5. Scheduled cleaning and metrics: daily checks, monthly sampling, suspected duplicates to a review queue.

Data lifecycle and data subject rights

Click Star read Personal Data Protection Law 91/2025/QH15 (June 26, 2025, effective January 1, 2026) in a copy on ketoananpha.vn. Check the original; this is not legal advice.

According to the copy, Article 3 clause 3 requires accuracy, correction and updating when needed, and storage for a period suited to the purpose. Article 10 clause 1 allows a request to restrict processing when accuracy is in doubt.

Article 13 clause 2 says the controller corrects after a request and requires processors and third parties to correct too. Click Star's reading, not a legal opinion: a correction must reach the merged record and every copy, so each field needs a source and entry time.

Click Star's view. Small businesses usually gain most from 3 things: 1 shared normalization function, 1 unique index on the canonical value, and 7 metrics reviewed weekly.

30-day plan

A Click Star reference frame, not from an outside source.

• Week 1: pick 5 key fields, write the normalization function, store original and canonical forms, take a first reading of the 7 metrics.
• Week 2: put the function on every entry path (forms, manual entry, files, integrations), add the unique index, and handle old 11-digit numbers.
• Week 3: deduplicate on a copy of the data, auto-merge on certain keys, send the rest to review.
• Week 4: turn on daily checks, sample wrong merges, send confirmation links, name an owner per metric.

What is not known

Click Star found no open standard for duplicate, missing-field or valid-number rates, and has no Vietnamese measurements. ISO/IEC 25012 was not read in full.

Decision 798/QD-BTTTT could not be opened on an official portal; Law 91/2025/QH15 was also read via a copy. Click Star has not read the fixed-line prefix rules, not checked libphonenumber's Vietnam metadata, and has no source on wrong merges when matching Vietnamese names.

Common mistakes

1. Treating valid as correct, though a well-formed number or email can still belong to the wrong person.
2. Normalizing differently in each place, so one customer produces several keys.
3. Overwriting the original with the normalized value, losing traceability and correction on request.
4. Auto-merging on a name match alone, or using the default fuzzy threshold untested.
5. Auto-correcting emails by guesswork instead of asking the person.
6. Ignoring old 11-digit numbers, or using converted numbers unverified.