Fork tooling: the third-party code upstream carries, listed and checked

A few of upstream's dual-licensed files carry code from other projects
under MIT or BSD terms. The fork redistributes it, so their licenses
require the notices to travel with it. THIRD-PARTY.md reproduces them.

strip.py now reads the stripped tree's comments for another copyright
holder, another license, or a note that code came from somewhere else, and
names any file THIRD-PARTY.md doesn't cover. It reports, never fails: the
notice goes in with the merge that brings the release in.

On v0.16.22 it finds 14 files, all of them covered. The rest of the report
is byte-for-byte what the committed one says, so the scan disturbs nothing
it already did.

README.md

@@ -63,7 +63,10 @@ version 3](./LICENSES/AGPL-3.0-only.txt).
 It is a fork of Stalwart, copyright © Stalwart Labs LLC. Upstream's
 copyright notices are kept on every file they cover. Stalwart's files are
 dual-licensed AGPL-3.0-only or Stalwart's Enterprise License, and INBUXA takes
-them under the AGPL-3.0 only. "Stalwart" is Stalwart Labs' name. INBUXA isn't
+them under the AGPL-3.0 only. A few of those files also carry code from
+other projects under MIT or BSD licenses, which stays under those licenses;
+[THIRD-PARTY.md](./THIRD-PARTY.md) lists it with its notices. "Stalwart" is
+Stalwart Labs' name. INBUXA isn't
 affiliated with or endorsed by Stalwart Labs.
 
 The INBUXA mark reuses ihasmail's cat-and-envelope artwork.

THIRD-PARTY.md

@@ -0,0 +1,92 @@
+# Third-party code
+
+INBUXA is a fork of Stalwart. Stalwart is original work by Stalwart Labs LLC,
+not a fork of anything, but a few of its files carry code, adapted or ported,
+from other projects under permissive licenses. Those parts stay under their own
+licenses, not the AGPL, and their notices are reproduced here as the licenses
+require. Where a project offers MIT or Apache-2.0, INBUXA takes it under MIT.
+
+`tools/fork/strip.py` lists every such file on each upstream import and names
+any this page doesn't cover yet (docs/spec/SPEC.md §2.2). The fork's own code,
+and Rust crates pulled in as dependencies, aren't listed here: dependencies
+carry their own license files.
+
+## Under the MIT license
+
+| Where | From | Notice |
+|---|---|---|
+| `crates/common/src/scripts/functions/text.rs` | [levenshtein-rs](https://github.com/wooorm/levenshtein-rs) | Copyright (c) 2016 Titus Wormer <tituswormer@gmail.com> |
+| `crates/common/src/telemetry/tracers/journald.rs` | the journald snippet | Copyright (c) 2018 Benjamin Saunders <ben.e.saunders@gmail.com> |
+| `crates/jmap/src/registry/mapping/log.rs` | [rev_lines](https://github.com/mikeycgto/rev_lines) | Copyright (c) 2017 Michael Coyne <mjc@hey.com> |
+| `crates/imap-proto/src/utf7.rs` | [MailKit](https://github.com/jstedfast/MailKit), by Jeffrey Stedfast | Copyright (C) 2013-2026 .NET Foundation and Contributors |
+| `crates/nlp/src/tokenizers/japanese.rs` | [rust-tinysegmenter](https://github.com/woxtu/rust-tinysegmenter) | Copyright (c) 2015 woxtu |
+| `crates/store/src/backend/postgres/tls.rs` | [tokio-postgres-rustls](https://github.com/jbg/tokio-postgres-rustls) | Copyright (c) 2019 Jasper Hugo |
+| `crates/common/src/network/acme/directory.rs`, `crates/common/src/network/acme/jose.rs`, `crates/common/src/network/acme/order.rs` | [rustls-acme](https://github.com/FlorianUekermann/rustls-acme) (MIT or Apache-2.0) | Copyright (c) Florian Uekermann |
+| `crates/types/src/id.rs` | [crockford](https://github.com/archer884/crockford) (MIT or Apache-2.0) | Copyright (c) 2017 J/A <archer884@gmail.com> |
+| `crates/nlp/src/tokenizers/types.rs` | test cases from [linkify](https://github.com/robinst/linkify) (MIT or Apache-2.0) | Copyright (c) 2017 Robin Stocker |
+
+Each notice above applies with this permission notice:
+
+> Permission is hereby granted, free of charge, to any person obtaining a copy
+> of this software and associated documentation files (the "Software"), to deal
+> in the Software without restriction, including without limitation the rights
+> to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+> copies of the Software, and to permit persons to whom the Software is
+> furnished to do so, subject to the following conditions:
+>
+> The above copyright notice and this permission notice shall be included in
+> all copies or substantial portions of the Software.
+>
+> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+> FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+> AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+> LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+> SOFTWARE.
+
+## Under the BSD 3-Clause license
+
+| Where | From | Notice |
+|---|---|---|
+| `crates/jmap-proto/src/types/date.rs`, `crates/registry/src/types/datetime.rs` | [upb](https://github.com/protocolbuffers/upb/blob/22182e6e/upb/json_decode.c), the date parsing marked in each file | Copyright (c) 2009-2011, Google Inc. All rights reserved. |
+
+```text
+Copyright (c) 2009-2011, Google Inc.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+    * Redistributions of source code must retain the above copyright
+      notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above copyright
+      notice, this list of conditions and the following disclaimer in the
+      documentation and/or other materials provided with the distribution.
+    * Neither the name of Google Inc. nor the names of any other
+      contributors may be used to endorse or promote products
+      derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY GOOGLE INC. ``AS IS'' AND ANY EXPRESS OR IMPLIED
+WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
+EVENT SHALL GOOGLE INC. BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+```
+
+## Credited algorithms
+
+These files implement published algorithms and credit their source. No code
+is copied, so there's no notice to carry. They're listed so the strip report
+doesn't flag them as new.
+
+- `crates/jmap-proto/src/types/date.rs`, `crates/registry/src/types/datetime.rs`:
+  `civil_from_days`, from Howard Hinnant's
+  [date algorithms](http://howardhinnant.github.io/date_algorithms.html)
+- `crates/utils/src/glob.rs`: Russ Cox's
+  [glob matching](https://research.swtch.com/glob)

docs/spec/SPEC.md

@@ -85,7 +85,13 @@ The wrapper is `tools/fork/strip.py`. Beyond `ossify.py` it:
   don't compile until they're gone;
 - verifies the result across every text file, not just Rust, and reports what
   was removed, the Cargo edits, upstream's Enterprise flags, and the feature
-  gates left for §2.3 to replace.
+  gates left for §2.3 to replace;
+- lists the third-party code left in the stripped tree, as upstream's
+  comments mark it: another copyright holder or license, or a note that code
+  was ported or adapted from elsewhere. Any file `THIRD-PARTY.md` doesn't
+  cover yet is flagged as new. It's reported, not a failure: the notice goes
+  into `THIRD-PARTY.md` in the merge that brings the release in, since the
+  fork redistributes that code and its license requires the notice.
 
 ### 2.2a Snapshots, not a git fork
 

tools/fork/README.md

@@ -15,4 +15,8 @@ It writes `OUT/tree` (the stripped source) and `OUT/STRIP-REPORT.md` and
 `.json`. Exit 0 means verified clean. Exit 1 means malformed markers, or
 something Enterprise-only survived. Read the report's Problems section.
 
+The report's Third-party code section lists upstream code under other
+licenses. Files marked **new** need their notice added to `THIRD-PARTY.md`
+at the repository root before the import is merged.
+
 It needs Python 3.12+ (for `tarfile`'s `data` filter) and git.

tools/fork/strip.py

@@ -29,6 +29,10 @@ What it does, in order (docs/spec/SPEC.md §2.2):
    removed, what was edited, what upstream's schema flags as Enterprise, and
    how many `enterprise` feature gates and edition checks remain in shared
    code for the rebuilt features to replace.
+7. Lists the third-party code left in the stripped tree, as upstream's
+   comments mark it (another copyright holder or license, or "ported from"
+   and the like), and names any file THIRD-PARTY.md doesn't cover yet. That's
+   a report, not a failure: the notice goes in THIRD-PARTY.md with the merge.
 
 Only license markers and Cargo manifests are read for meaning. The code inside
 an Enterprise file or snippet is never printed, reported or kept, which is what
@@ -291,6 +295,57 @@ def verify(tree):
     return problems
 
 
+# Third-party code as upstream marks it, in comments only: a license
+# identifier other than upstream's own, a copyright line naming anyone but
+# Stalwart Labs, or a note that code came from somewhere else.
+COMMENT = re.compile(r'^\s*(?://+!?|/?\*+|#+|--|<!--)\s*(.*?)\s*(?:\*/|-->)?\s*$')
+COPYRIGHT = re.compile(r'(?i)(?:SPDX-FileCopyrightText:|\bcopyright\b|©)')
+ORIGIN = re.compile(r'(?i)\b(?:(?:ported|derived|adapted|taken|copied|borrowed)\s+from|credits?\b\s*:|inspired\s+by'
+                    r'|licen[cs]e(?:d)?\s*(?:under|:)|\w+\s+licen[cs]ed\b|^from\s+https?://)')
+OWN = ('Stalwart Labs',)
+OWN_LICENSES = {AGPL, f'{AGPL} OR {SEL}'}
+
+
+def third_party(tree):
+    """
+    Every comment line in the stripped tree that points at someone else's code.
+
+    Runs after stripping, so only shared (AGPL) code is read. Each hit is a
+    notice the fork passes on when it distributes, so it's matched against
+    THIRD-PARTY.md, and a file that isn't listed there yet is reported as new.
+    """
+    hits = {}
+    for path in sorted(tree.rglob('*')):
+        if not is_text(path):
+            continue
+        rel = path.relative_to(tree)
+        if rel.parts[0] == 'LICENSES' or path.suffix in ('.md', '.txt', '.json'):
+            continue  # license texts, prose and data, not code
+        try:
+            lines = path.read_text(encoding='utf-8').split('\n')
+        except UnicodeDecodeError:
+            continue
+        for n, line in enumerate(lines, 1):
+            ident = IDENT.match(line)
+            if ident:
+                if ident.group(1) not in OWN_LICENSES and SEL not in ident.group(1):
+                    hits.setdefault(str(rel), []).append({'line': n, 'text': f'SPDX-License-Identifier: {ident.group(1)}'})
+                continue
+            m = COMMENT.match(line)
+            if not m or any(o in m.group(1) for o in OWN):
+                continue
+            if COPYRIGHT.search(m.group(1)) or ORIGIN.search(m.group(1)):
+                hits.setdefault(str(rel), []).append({'line': n, 'text': m.group(1)[:160]})
+    return hits
+
+
+def unlisted(hits):
+    """Files with third-party notices that THIRD-PARTY.md doesn't name yet."""
+    notices = Path(__file__).resolve().parents[2] / 'THIRD-PARTY.md'
+    listed = set(re.findall(r'`([^`\s]+?)(?::\d+)?`', notices.read_text(encoding='utf-8'))) if notices.is_file() else set()
+    return sorted(f for f in hits if f not in listed)
+
+
 def schema_flags(tree):
     path = tree / 'resources' / 'schema' / 'schema.json.gz'
     if not path.is_file():
@@ -334,6 +389,7 @@ def write_report(out_dir, report):
         f'- Left for the rebuilt features to replace: {sum(r["feature_gates"].values())} `enterprise` feature gates '
         f'in {len(r["feature_gates"])} files; {sum(r["edition_checks"].values())} `is_enterprise_edition()` checks '
         f'in {len(r["edition_checks"])} files',
+        f'- Third-party code: {len(r["third_party"])} files, **{len(r["third_party_unlisted"])}** not in THIRD-PARTY.md',
     ]
     if r['schema']:
         md.append(f'- Upstream schema flags {len(r["schema"]["objects"])} objects and {len(r["schema"]["fields"])} fields as Enterprise')
@@ -344,6 +400,12 @@ def write_report(out_dir, report):
     if r['schema']:
         md += ['', '## Flagged Enterprise in upstream\'s schema', '', '**Objects:** ' + ', '.join(f'`{o}`' for o in r['schema']['objects']),
                '', '**Fields:** ' + ', '.join(f'`{f}`' for f in r['schema']['fields'])]
+    md += ['', '## Third-party code', '',
+           'Comments in the stripped tree that name another copyright holder, another license, or a source the '
+           'code came from. Files marked **new** aren\'t in THIRD-PARTY.md yet.', '']
+    for f, found in r['third_party'].items():
+        md.append(f'- `{f}`{" **new**" if f in r["third_party_unlisted"] else ""}')
+        md += [f'  - {h["line"]}: {h["text"]}' for h in found]
     if r['problems']:
         md += ['', '## Problems', ''] + [f'- {p}' for p in r['problems']]
     (out_dir / 'STRIP-REPORT.md').write_text('\n'.join(md) + '\n', encoding='utf-8')
@@ -365,7 +427,8 @@ def main():
     if malformed:
         write_report(args.out, {'ref': args.ref, 'commit': commit, 'removed_files': [], 'removed_snippets': {},
                                 'cargo_edits': [], 'dangling_mods': [], 'problems': malformed, 'feature_gates': {}, 'edition_checks': {},
-                                'schema': None, 'ossify_log': ''})
+                                'schema': None, 'third_party': {}, 'third_party_unlisted': [],
+                                'ossify_log': ''})
         print('\n'.join(malformed), file=sys.stderr)
         fail('malformed snippet markers; nothing stripped', code=1)
 
@@ -375,18 +438,22 @@ def main():
     dangling = remove_dangling_mods(tree)
     problems = verify(tree)
     gates, checks = remaining_hooks(tree)
+    others = third_party(tree)
+    new_others = unlisted(others)
 
     report = {
         'ref': args.ref, 'commit': commit,
         'removed_files': removed_files, 'removed_snippets': removed_snippets,
         'cargo_edits': edits, 'dangling_mods': dangling, 'problems': problems,
         'feature_gates': gates, 'edition_checks': checks,
-        'schema': schema_flags(tree), 'ossify_log': log,
+        'schema': schema_flags(tree), 'third_party': others, 'third_party_unlisted': new_others,
+        'ossify_log': log,
     }
     write_report(args.out, report)
     print(f'{args.ref} ({commit[:12]}): removed {len(removed_files)} files and '
           f'{sum(removed_snippets.values())} snippets, {len(dangling)} dangling mods, {len(edits)} Cargo edits, '
-          f'{"verified clean" if not problems else f"{len(problems)} PROBLEMS"}. Report: {args.out}/STRIP-REPORT.md')
+          f'{"verified clean" if not problems else f"{len(problems)} PROBLEMS"}'
+          f'{f", {len(new_others)} files of third-party code not in THIRD-PARTY.md" if new_others else ""}. Report: {args.out}/STRIP-REPORT.md')
     sys.exit(1 if problems else 0)