Archived
Contacts: duplicate checking on re-import (vCard and LDIF) #223
Assignees
jcoffey-dev (John Coffey)
Clear assignees
jcoffey-dev (John Coffey)
No Assignees
No due date set.
Dependencies
No dependencies set.
Reference: coffey-labs/ihasmail-github-archive#223
Reference in New Issue
Block a user
Split out of #174 so it stops living in a comment thread.
Re-importing an address book creates duplicate contacts. Reported by the submitter's colleague after testing the LDIF import (#174 comment):
Why this is not simply #222 again
The calendar case is decided and clean: an event carries a UID and that is what makes it the same event. Contacts split in two.
dnis explicitly not an identity — it says where the entry sat in somebody else's directory, which is why the import doesn't reuse it. Matching would have to fall back to something softer, name plus email being the obvious candidate, and that guesses in both directions: two real people who share a name and an alias get merged, one person whose address changed gets duplicated anyway.Open question
Put to the reporter on #174, not yet answered: skip on UID where there is one and import where there isn't (leaving LDIF re-imports duplicating as they do now), or accept a softer name+email match for LDIF.
My inclination is the former — a duplicate you can see and delete is better than a merge you cannot undo — but this is the reporter's call to weigh in on, since LDIF re-import is mostly a one-way migration in their use and pre-processing scripts are already part of their workflow.
Blocked on that answer for the LDIF half. The vCard half could ship on its own.
Rebuilt from: GH Archive, session transcript.
The vCard half is done, in #228 — live now as
2026.9.2+pr228. A card whose UID an address book already holds is skipped; one that arrives without a UID is imported as before. Matching is per book, so the same person filed in two address books stays two cards, which is the same rule the calendar got.A vCard attached to a message got better on the way past. That's usually a card you've been sent before, and it used to report "Added 0 contacts" when you already had it — which reads as a failure rather than as nothing to do. It now says it's already in your contacts.
The LDIF half is all that's left, and it's still the question I don't want to answer alone.
Where it stands: LDIF re-imports still duplicate, and the code now says so out loud rather than leaving it to be discovered —
importLdifreportsskipped: 0and there's a test asserting that importing the same file twice imports it twice, with a comment explaining that this is a decision and not an oversight. I'd rather the gap be on the record than look like a bug to whoever reads it next.The two options, as I see them:
My inclination is still (1), because a duplicate is visible and deletable while a merge is neither. But you said LDIF import is mostly a one-way migration for you, and if in practice it's re-run during a migration — a first pass, then a corrected export — then duplicating on every attempt is genuinely annoying and (2) starts looking better.
You know which of those your migrations actually look like. No hurry, and nothing else is waiting on it.
Half of the remaining half, in #234 — live now as
2026.9.2+pr234. Your question is still yours; this is what could be done without pre-empting it.An LDIF import now tells you how much of it you already had. After the usual "Imported 200 contacts" you get a second line: "40 of them look like contacts you already had." Every card is still imported. Nothing is skipped, nothing is merged, nothing is decided.
The reason for stopping there rather than going further: what your colleague actually reported was that re-importing duplicates and that it was confusing. Those are two problems and only one of them needs your answer. Being told what happened costs nothing and removes the confusion; deciding what to do about it is the part I don't want to guess at.
The likeness test is name plus one address, normalised for case and spacing, and it is deliberately fallible in both directions — two colleagues who share a name and a shared alias will collapse into one number, and somebody whose address changed since your last export won't be counted at all. That is fine for a count and would not be fine for a merge, which is precisely why it only counts. If your answer comes back "yes, match on name and email", the matching is now written and turns into a skip rather than a tally.
It costs no extra round trip, incidentally — the scan the vCard import already does for UIDs now reads names and addresses on the same request.
One thing I considered and decided against, because you should get the chance to disagree. The
dnis available at import time; ihasmail parses it and deliberately throws it away. I could derive a deterministic UID from it, and then re-importing an unchanged export would be a clean no-op with no guessing at all — the vCard dedupe path would catch it exactly.I didn't, for two reasons. Storing a dn-derived value as the card's UID makes the
dnthe contact's identity, which is the thing you and I both argued against on #174 and you had the better reasons — position changes, structural attribute changes. And two different people can share adnacross two customer directories; imported into one book, they'd merge silently.But it is narrower and more honest than a name+email guess, and it would cleanly solve the common case of the same file, twice. If that trade appeals to you more than either of the options I put earlier, say so — it is a third answer I hadn't offered you.
Correction, and an apology for the noise: you answered this on #174 at 15:25 and I posted twice afterwards without re-reading the thread. My comment above offers you the
dnidea as something new — you had already argued for that whole class of answer four hours earlier, and better than I did. Sorry for making you read it twice.Reading what you actually wrote, three things change.
Using an attribute that can change is fine, and your reasoning is the part I was missing. I had been treating "the
dncan change" as disqualifying. You are right that it disqualifies nothing for the case that actually happens: a migration where you import, spot something wrong, pre-process, and import again. Thedndoes not change in the ten minutes between those two attempts. Neither does a name. I was optimising for a durability nobody needs — an import is not a sync."Update, not skip" is the thing I got wrong, and it is already shipped wrong. #228 skips a vCard whose UID the book already holds. You want the existing entry updated instead, for contacts, whatever the source format. That is clearly right and I would not have got there on my own: skipping means a corrected export cannot correct anything, which defeats the workflow you just described — the whole point of the second import is that the first one was not right.
So the design I will build, unless you say otherwise:
dn, since it is stable across the interval that matters and it is what the file actually carries.One thing already shipped in the meantime, in #234 (live as
2026.9.2+pr234): an LDIF import now tells you how many of the entries look like contacts you already had, without doing anything about them. That was me trying to move while I thought this question was open. It is not what you asked for and it is not the end state — but it is close to your "present a dialog after import stating how many were detected as duplicates & how many were newly created", so the reporting half is done and the counts just need to become created-and-updated once the matching lands.You also mentioned needing to delete all imported entries and start over during a migration. That is a separate gap and a real one — there is no "select everything in this address book and delete it" today beyond selecting by hand. Worth its own issue if it would help; say so and I will open one.
Update rather than skip is done, in #242 — live now as
2026.9.2+pr242. You were right and I would not have got there alone: skipping means a corrected export corrects nothing, which defeats the only workflow anyone re-imports for.A vCard whose UID the book already holds now updates that card. LDIF still imports everything and reports look-alikes, because what it should match on is the question below.
One decision I made that you did not specify, and I would rather you overrule it now than discover it later. The update is a merge, not a replacement. Properties the file carries overwrite what is here; properties it does not mention are left alone.
So if somebody added a phone number in ihasmail after the first import, a re-import of the original file keeps it. The mirror of that is the cost: if you deleted a field at the source and re-exported, it stays here. I picked this way round because losing work nobody asked to lose is the worse of the two failures — but for a migration, where the file is meant to be the truth and ihasmail is meant to be a mirror of it, you may well want the opposite. Say so and I will make the file authoritative.
Also worth knowing: the card is not moved between address books on update. It stays where it is even if you import into a different book, which matters if a contact is filed in two.
What is still yours: the LDIF matching key. From your #174 answer I take it that using an attribute which can change is fine, since it will not have changed between two imports minutes apart. That leaves the
dnas the candidate — it is in the file, it is stable across the interval that matters, and it needs no guessing. My earlier objection was that it is not an identity anywhere else, which is true and which you had already argued better than I did; the counter is that it does not need to be an identity to answer "have I imported this exact entry before".The thing I would still want your call on is that a dn-derived UID becomes the contact's identity in JMAP terms, and two customers' directories can both contain
cn=John Smith. Imported into one book, those merge. Narrower than name-plus-email, but the same kind of mistake.No rush on any of it.
Reopened. Your colleague is right, and the closure was mine to answer for: the vCard half shipped, the LDIF half did not, and I closed this anyway.
Worse than that — the question I said I was waiting on had already been answered. Moritz gave it on #174: "using one of the attributes that can change … should be fine as they won't actually change between imports". That settles the
dncompletely. I acknowledged it at the time, wrote down the design it implied, built the vCard half, and then closed the issue with the LDIF half still carrying a question that was no longer open. Sorry for the round trip.The fix is in #274.
What it does. An LDIF entry is now recognised by its distinguished name. It is the only identity the file carries, Mozilla's schema defines no UID, and — this is Moritz's point, and the part I had wrong — it does not need to be durable to answer have I imported this exact entry before. The
dndoes not change in the ten minutes between importing a migration, spotting a mistake, correcting the export and importing again. An import is not a sync.A card already here under that
dnis updated, not skipped and not duplicated, merged the way the vCard import merges: what the file carries wins, what it does not mention is left alone. The toast reports created and updated separately, which is the dialog Moritz asked for.Three things worth knowing, because one of them is a real limit.
Matching is per address book. Two customers' directories that each contain a
cn=John Smithstay two people as long as they go into two books; imported into the same book they become one. That is the one way this can be wrong, and filing them separately is the way round it.The look-alike count from #234 stays and now means something narrower: entries the
dncould not catch — one whosednmoved between exports, and anything imported before this lands. That second case matters to you in practice: contacts already imported from LDIF carry an invented identity, so the first re-import after this ships still duplicates them and reports them as look-alikes. From that import on, matching works.A file holding two entries under one
dn— which a directory cannot, so the file is malformed — now becomes one card rather than two sharing an identity.Still yours, and still unanswered from earlier: should a calendar re-import update a matching event rather than skip it (#222 currently skips)? You scoped your answer to contacts and I would rather ask than assume.
And the separate gap you named — deleting everything in an address book to start a migration over — is still not built. Say the word and I will open it.
Done, and live now as
2026.9.4+pr276.An LDIF re-import matches on the entry's
dn. A contact the address book already holds under thatdnis updated rather than duplicated, and merged rather than replaced: properties the file carries win, properties it does not mention are left alone. The result is reported as created and updated, which closes the last part of what your colleague asked for.That is your answer from #174 applied rather than re-asked. The
dnis not a durable identity and does not need to be one — it only has to survive the interval between importing a migration, spotting a mistake, correcting the export and importing again. It does. An import is not a sync.Three things worth knowing, one of which will affect you on the next import:
Contacts already imported from LDIF carry an identity invented at import time, because there was nothing to match on when they were created. Your next re-import still duplicates those. From the one after that onwards, matching works. If a migration is in progress, that next pass is the one to plan around.
Matching is per address book. Two directories that each contain a
cn=John Smithstay two people as long as they are filed in two books; imported into the same book they become one. That is the one way this can be wrong, and separate books are the way round it.The look-alike count from #234 stays, and now means something narrower: entries the
dncould not catch, such as one that moved to a different branch of the directory. Still counted, never merged — name-plus-email is a guess wrong in both directions, and a merge made on a guess cannot be undone.Two cases I could not decide for you, and would rather leave in your hands than close over:
Should a calendar re-import update a matching event rather than skip it? #222 currently skips. Your answer on #174 was scoped to contacts and the same argument appears to apply, but I would rather ask.
Deleting everything in an address book to start a migration over still does not exist beyond selecting by hand. You named it as part of the workflow; say so and I will open it as its own issue.
Closing this one, since both halves are now built and shipped. If the LDIF side does not behave as described once you try it, reopen rather than filing new — it belongs here.
Thanks again to your colleague for coming back about it. The report was right and the closure was mine to answer for.