A timeout leaves the remote outcome uncertain
Run local TCP controls where the same client timeout hides different server outcomes, then test retry identities and lost deduplication receipts.
A client stops waiting for a reply. The server may already have changed its state, or it may apply the change later. A timeout tells the client that its wait did not complete in time; it does not provide a receipt describing the remote outcome.
If you can follow a request and response between two programs, this lesson shows why retrying a timed-out operation needs care. We will run an actual TCP experiment on the local machine. The side effect is one increment of an in-memory counter. Deliberate pauses create different server histories with the same client timeout, and operation identifiers let us examine duplicate effects.
Put the timeout beside the effect
Imagine a request to add one to a counter. In the first history, the server reads the request and pauses before applying it. The client times out while the counter is still zero. When the pause is released, the server increments it to one even though the client has already closed its connection.
In the second history, the server increments first and then pauses before replying. The client again times out, but the counter is already one. The client-facing result is identical. Only the laboratory observer, which can inspect the server, knows which history occurred.
Marc Brooker’s Amazon Builders Library article makes this distinction explicitly: a timeout or failure does not necessarily mean that side effects have not happened. Its discussion also explains why retries can increase load. Those are separate concerns: delaying a retry may reduce pressure, but waiting longer does not turn an uncertain first outcome into proof of failure.
Reproduce both histories on loopback
The example runs separate client and server tasks in one Python process, connected by IPv4 loopback TCP. It is an executed socket experiment, not a test across machines. An event deliberately pauses the first server handler. A laboratory coordination gate starts the client’s twenty-millisecond read timeout only after that handler reaches the intended pause.
This coordination makes the histories reproducible. A production client generally cannot observe that internal point. The timer also covers only waiting for a reply line; it excludes connection establishment and sending. Twenty milliseconds is a test parameter, not a recommended timeout for a real service.
Python’s wait_for documentation says that timeout cancels the awaited task and raises TimeoutError, waiting for cancellation to finish. Here the awaited operation is a local read. The independently scheduled server handler is not that task. Closing the client stream releases local connection resources, but the program contains no protocol that reverses a server increment.
import asyncio
import time
async def run_case(stage, retry):
if stage not in ('before', 'after'):
raise ValueError('stage must be before or after')
if retry not in ('none', 'new-token', 'same-token', 'forget-receipt'):
raise ValueError('unknown retry mode')
if retry == 'forget-receipt' and stage != 'after':
raise ValueError('forgetting is tested after the first effect')
release = asyncio.Event()
received = asyncio.Event()
handlers, events, errors = [], [], []
receipts = {}
counter = 0
start = time.monotonic_ns()
def record(side, event, token=''):
events.append(dict(side=side, event=event, token=token, counter=counter,
elapsed_ms=(time.monotonic_ns() - start) / 1_000_000))
async def handle(reader, writer):
nonlocal counter
handlers.append(asyncio.current_task())
first = len(handlers) == 1
try:
token = (await reader.readline()).decode('ascii').rstrip('\n')
if token not in ('op-1', 'op-2'):
raise ValueError('this fixture accepts only its two fixed tokens')
record('server', 'received', token)
if first:
received.set()
if stage == 'before':
record('server', 'held before effect', token)
await release.wait()
# No await between checking, changing state and recording a receipt.
if token not in receipts:
counter += 1
receipts[token] = counter
record('server', 'effect applied', token)
else:
record('server', 'duplicate suppressed', token)
result = receipts[token]
if first and stage == 'after':
record('server', 'held after effect', token)
await release.wait()
record('server', 'reply attempted', token)
writer.write(f'{result}\n'.encode('ascii'))
await writer.drain()
except ConnectionError:
record('server', 'peer connection closed')
except Exception as error:
errors.append(repr(error))
finally:
writer.close()
try:
await writer.wait_closed()
except ConnectionError:
pass
server = await asyncio.start_server(handle, '127.0.0.1', 0, limit=256)
port = server.sockets[0].getsockname()[1]
async def call(token, timeout, synchronize=False):
reader, writer = await asyncio.open_connection('127.0.0.1', port)
try:
record('client', 'sent', token)
writer.write((token + '\n').encode('ascii'))
await writer.drain()
if synchronize:
# Laboratory gate: start the read timer after the server pauses.
await received.wait()
record('client', 'reply wait started', token)
try:
reply = await asyncio.wait_for(reader.readline(), timeout)
except TimeoutError:
record('client', 'timeout', token)
return 'timeout'
if not reply:
raise RuntimeError('unexpected EOF in this control')
record('client', 'reply received', token)
return 'reply:' + reply.decode('ascii').strip()
finally:
writer.close()
await writer.wait_closed()
try:
first_call = asyncio.create_task(call('op-1', 0.02, True))
await asyncio.wait_for(received.wait(), 2)
first_outcome = await asyncio.wait_for(first_call, 2)
assert first_outcome == 'timeout'
observed_at_timeout = counter # Laboratory observer, not client knowledge.
retry_outcome = None
if retry != 'none':
if retry == 'forget-receipt':
receipts.clear() # Keep the effect, deliberately lose its receipt.
record('observer', 'receipts forgotten')
token = 'op-2' if retry == 'new-token' else 'op-1'
retry_outcome = await asyncio.wait_for(call(token, 1), 2)
release.set()
await asyncio.wait_for(asyncio.gather(*handlers), 2)
assert not errors, errors
return dict(stage=stage, retry=retry, first_outcome=first_outcome,
observed_at_timeout=observed_at_timeout,
retry_outcome=retry_outcome, final_counter=counter,
receipts=dict(receipts), events=events)
finally:
release.set()
server.close()
await server.wait_closed()
await asyncio.gather(*handlers, return_exceptions=True)Each line sent to the server carries an operation identifier. The handler accepts only the two fixed identifiers used in this fixture. It increments the counter once per identifier and saves the resulting value in a receipt dictionary. Checking the dictionary, changing the counter and storing the receipt contain no await, so another task on this single event loop cannot interleave those steps.
That is a deliberately narrow concurrency model. It provides no durable storage and no coordination between worker processes. It also does not accept arbitrary operation payloads, authenticate callers or implement a general RPC service. Its purpose is to expose the ordering around one side effect.
Inspect what the client actually learned
The next block runs seven cases. The first six combine the two pause positions with no retry, a retry using a new identifier and a retry using the same identifier. The final case deliberately forgets receipts while preserving the counter. The printed counter-at-timeout column comes from the observer; it was never returned to the timed-out client.
async def experiment():
rows = []
for stage in ('before', 'after'):
for retry in ('none', 'new-token', 'same-token'):
rows.append(await run_case(stage, retry))
rows.append(await run_case('after', 'forget-receipt'))
for row in rows:
print(row['stage'], row['retry'], row['first_outcome'],
row['observed_at_timeout'], row['final_counter'])
return rows
results = asyncio.run(experiment())
assert [row['observed_at_timeout'] for row in results] == [0, 0, 0, 1, 1, 1, 1]
assert [row['final_counter'] for row in results] == [1, 2, 1, 1, 2, 1, 2]
This image is a screenshot of an HTML report populated from the program’s actual stdout under CPython 3.12.3. It is not a service interface. The two-second waits elsewhere in the harness bound test coordination and cleanup; they do not change the twenty-millisecond reply-read deadline.
A successful local write is also weaker than an application acknowledgment. The streams documentation describes drain as write-buffer flow control. Returning from drain does not prove that the peer applied the requested operation. Likewise, a server event labeled “reply attempted” does not prove the client received or interpreted that reply.
Choose whether a retry is the same operation
After either timeout, releasing the first handler without retrying leaves one increment. Retrying with op-2 asks the server to process a distinct identifier, so the final counter is two. The caller may have intended one business action, but the server has been given two operation identities.
Reusing op-1 gives one final increment while the receipt remains available. The repeated request gets the value associated with the earlier application. This illustrates duplicate suppression for one logical action; it does not make two intentionally distinct increments interchangeable.
A real protocol must also bind an identifier to its caller and intended parameters. Accidentally reusing an identifier for a different amount or a different resource should not silently turn that new action into a duplicate. Our counter fixture has only the fixed “add one” operation, so it does not exercise payload-conflict handling.
The retry can overtake the first handler
The before/same-token case is particularly useful. The original handler is waiting before the effect. Its client times out and sends another connection carrying op-1. The second handler reaches the receipt check first, applies the increment and stores its result. When the original handler resumes, it finds that receipt and suppresses a second increment.
The “first attempt” in a client log is therefore not necessarily the first attempt to apply the effect. A correct duplicate rule needs a shared decision point for all concurrent attempts of the same operation. In this example that point is the uninterrupted check-and-update sequence on one event loop. Adding an await between the check and update would require additional synchronization and a new correctness argument.
A forgotten receipt changes the guarantee
The seventh case first applies op-1, then holds the reply until the client times out. The observer clears the receipt dictionary but leaves the counter at one. A retry with the same op-1 now looks new and increments the counter to two.
This is a controlled metadata-loss injection. No process was restarted and no disk durability was tested. It demonstrates the logical problem that arises when an effect survives but its deduplication record does not. In a durable design, retaining the effect and its receipt consistently is part of the contract. Retention limits also matter: a client cannot assume an old identifier remains protected forever after the service has forgotten it.
Timeout handling should preserve this uncertainty in the caller’s state. “Outcome unknown” is often more accurate than “operation failed.” Depending on the protocol, the next step may be a status lookup by operation identifier, a permitted retry with the same identifier, or reconciliation. The example does not choose a universal recovery rule; it shows what evidence each rule would need.
For an exercise, inspect the event list for before/same-token and identify which handler applied the increment. Then move the receipt deletion to after all attempts finish and verify that it does not change those completed outcomes, while a later retry can still repeat the effect. Record the client observation separately from observer-only state. That separation prevents a timeout from quietly becoming a claim about work the client never saw.