Split by the person when the person is the prediction boundary

Split by person when the claim concerns predictions for people absent from training. A recording split can answer a different, useful question: how well the system handles another recording from somebody it already knows. Decide which future encounter matters before inspecting scores.

The intended reader can distinguish training from test data and follow a short Python function. A deliberately simple classifier makes the information boundary visible. A second example shows why separating speakers and devices together can require larger blocks than either identifier suggests. Neither example measures a speech model or a production service.

The same eight recordings answer two different questions

Suppose four fictional speakers contribute two recordings each. Their binary labels are fixed within speaker: A has label 0, B has 1, C has 1 and D has 0. An identifier is available to a lookup rule, which memorizes the labels of training speakers and predicts 0 for an unfamiliar speaker. The fallback is declared before opening either test set.

This deliberately unrealistic rule isolates a mechanism. A real model might recover some identity information from voice or recording conditions even when its input does not contain an explicit speaker ID. That possibility needs investigation on the actual task; this example does not prove that any particular representation identifies people.

For a recording split, train on A1, B1, C1 and D1, then test on their second recordings. For a speaker split, train on both recordings from A and B, then test on both from C and D. Both partitions contain four training and four test rows. They expose the rule to different information.

from fractions import Fraction

labels = {"A": 0, "B": 1, "C": 1, "D": 0}
recordings = [(f"{s}{i}", s, y)
              for s, y in labels.items() for i in (1, 2)]

def fit_lookup(train):
    known = {}
    for _, speaker, label in train:
        if speaker in known and known[speaker] != label:
            raise ValueError("this example requires a stable speaker label")
        known[speaker] = label
    return known

def error_rate(train, test):
    if not test:
        raise ValueError("an empty test set has no error rate")
    known = fit_lookup(train)
    wrong = sum(known.get(speaker, 0) != label
                for _, speaker, label in test)
    return Fraction(wrong, len(test))

row_train = [r for r in recordings if r[0].endswith("1")]
row_test = [r for r in recordings if r[0].endswith("2")]
group_train = [r for r in recordings if r[1] in {"A", "B"}]
group_test = [r for r in recordings if r[1] in {"C", "D"}]
print(error_rate(row_train, row_test))
print(error_rate(group_train, group_test))
assert {r[1] for r in group_train}.isdisjoint(
    {r[1] for r in group_test})

The outputs are 0 and 1/2. In the recording split, every test speaker has a stored label. In the speaker split, the rule predicts 0 for C and D, making two errors on C’s recordings. The larger error is not caused by fewer training rows. It reflects a change from recognizing represented people to handling unrepresented people.

View full-size image ↗

Both columns use the same synthetic labels and four training rows. The colored cells identify the information available before prediction; they are not measured model accuracies.

The scikit-learn grouped-validation guide motivates this distinction using dependent observations from subjects. Its splitters enforce separation of the supplied group labels. They cannot decide whether those labels describe the deployment boundary correctly.

Separate an exact example from a general performance claim

There is a small probability model behind the example. Assign each speaker an independent fair binary label, then copy it to both recordings. Keep the two partitions and the lookup rule fixed. Across the 16 equally likely label assignments, recording-test error is always 0. Speaker-test error is 0, 1/2 or 1 with probabilities 1/4, 1/2 and 1/4, giving expected error 1/2.

Each unseen label is independent of everything this rule learned. Repeated recordings do not supply independent new labels. This explains the expected difference under the stated construction; it is not a theorem that a row split always improves a realized score. If every speaker has label 0, both scores are 0. If labels vary by recording, the stable-label lookup assumption fails and the code rejects conflicting training labels.

A new-recording evaluation may be valid for a personalized system with an established enrollment process. It still needs the timing and access rules of that process. A recording collected before enrollment cannot benefit from a label acquired later. A chronological restriction may therefore be needed within each known speaker, alongside a separate evaluation for new speakers.

Audit the identity that the claim actually names

Save the split membership with the input version. Check disjoint group sets after joins, filtering and feature construction. An assertion on account strings only establishes separation of account strings. If A1 uses account a_old and A2 uses a_new, a split can pass that assertion while the same person appears on both sides.

Use a documented, access-controlled mapping when multiple identifiers represent one person. Missing identity is not evidence of a new person: assigning each missing row a different placeholder would manufacture apparent independence. Excluding unresolved rows changes the covered population; retaining them in an unresolved block has a different cost. State the chosen rule and report how much data it affects.

Class balance also has limits. If every positive example belongs to one speaker, three disjoint test folds cannot each contain a positive speaker. Stratification can seek balance without violating groups, but it cannot create missing independent examples. The guide also explains that its greedy stratified group procedure can be imbalanced even when a better allocation exists. Report group and label counts rather than treating a splitter’s name as a feasibility certificate.

Two separation requirements create connected blocks

Now suppose the claim requires both unseen speakers and unseen devices. Consider six recording identities with speaker/device pairs: r1 A/X, r2 A/Y, r3 B/Y, r4 C/Z, r5 D/Z and r6 D/W. These are a separate illustrative membership table, not extra observations for the lookup score above.

Grouping by the tuple (speaker, device) is insufficient. Every tuple here is distinct, yet r1 and r2 share A, and r2 and r3 share Y. If either shared identity must stay on one side, that requirement propagates along the chain. The first three recordings form one connected block; the last three form another.

The following small union-find routine constructs those blocks. Inputs are nonempty triples of normalized, nonempty string IDs, with unique recording IDs. Speaker and device keys use separate namespaces so an identical spelling in the two domains does not create a false link. The routine uses only membership, not labels or fitted scores.

def connected_blocks(rows):
    if not rows or any(len(r) != 3 or
            any(not isinstance(x, str) or not x for x in r) for r in rows):
        raise ValueError("expected nonempty recording/speaker/device IDs")
    if len({r[0] for r in rows}) != len(rows):
        raise ValueError("recording IDs must be unique")
    parent = list(range(len(rows)))
    def root(i):
        while parent[i] != i:
            parent[i] = parent[parent[i]]
            i = parent[i]
        return i
    owner = {}
    for i, (_, speaker, device) in enumerate(rows):
        for key in (("speaker", speaker), ("device", device)):
            if key in owner:
                parent[root(i)] = root(owner[key])
            else:
                owner[key] = i
    blocks = {}
    for i, (recording, _, _) in enumerate(rows):
        blocks.setdefault(root(i), []).append(recording)
    return sorted(tuple(sorted(ids)) for ids in blocks.values())

memberships = [("r1", "A", "X"), ("r2", "A", "Y"),
               ("r3", "B", "Y"), ("r4", "C", "Z"),
               ("r5", "D", "Z"), ("r6", "D", "W")]
print(connected_blocks(memberships))
print(connected_blocks(memberships + [("r7", "B", "Z")]))

The first output is [('r1', 'r2', 'r3'), ('r4', 'r5', 'r6')]. Adding r7 B/Z joins those blocks, so the second output contains one tuple with r1 through r7. No nonempty train/test partition of all seven rows can satisfy both separation requirements.

View full-size image ↗

Edges name the identity that forces two recordings to stay together. The bridge removes a feasible split even though it adds a recording. No link represents similarity inferred from audio.

The reason is exact: every shared-identity edge requires equal partition membership, so every path does too. Conversely, assigning whole connected components to sides preserves both separations. With k components and every row assigned to exactly one of two named sides, there are 2^k−2 assignments with both sides nonempty. Two components give two assignments; one gives none. Additional class, size or time constraints can reduce that number further.

This is a structural feasibility result. Disconnected components can still share unrecorded conditions, and holding out a device does not make it representative of all future devices. If the full graph is connected, collect data that support the intended test, narrow the claim, or define and disclose an exclusion policy. Removing an inconvenient bridge after seeing its score would add another selection decision.

Carry the boundary through selection and reporting

GroupKFold places each supplied group in one test fold and requires at least as many distinct groups as folds. If the relevant units are connected components, supply those component identities rather than speaker/device tuples. Unequal component sizes can make a useful fold allocation impossible even when the group-count requirement is met.

GroupShuffleSplit expresses its test size in groups, not rows, and successive test sets may overlap. Holding out half of two groups containing 2 and 100 rows can therefore hold out either 2 or 100 rows. Repeated scores are not automatically independent measurements of new populations.

Keep the same deployment boundary in inner model selection and outer assessment, and fit preprocessing only on the relevant training portion. Raschka’s 2018 nested-validation explanation usefully separates those two jobs; grouping specifies which observations each job may access. Record whether loss averages speakers equally or weights them by recording count. Resampling individual recordings does not recreate variation across independent new speakers.

For a reproducible exercise, enumerate the 16 label assignments and recover the expected lookup errors. Then enumerate all 64 train/test assignments of the six membership rows and keep only nonempty assignments with disjoint speaker and device sets: exactly two survive. Add the bridge and check all 128 assignments: none survive. Finally, make a speaker ID equal to a device ID without sharing either within its own namespace. The two identities should remain separate. These checks establish the declared boundary; external validity still depends on how the people and recordings were collected.