Resiliparse Process Guards
Resiliparse Process Guard API documentation.
- class resiliparse.process_guard.InterruptType(value)
An enumeration.
Resiliparse context guard interrupt type.
- exception = 0
Send only exceptions
- signal = 1
Send only signals
- exception_then_signal = 2
Send an exception first and follow up with signals
- exception resiliparse.process_guard.ExecutionTimeout
Bases:
ResiliparseGuardException
Execution timeout exception.
- exception resiliparse.process_guard.MemoryLimitExceeded
Bases:
ResiliparseGuardException
Memory limit exceeded exception.
- exception resiliparse.process_guard.ResiliparseGuardException
Bases:
BaseException
Resiliparse guard base exception.
- class resiliparse.process_guard.MemGuard
Bases:
_ResiliparseGuard
Decorator and context manager for enforcing memory limits on a running task.
Use the
mem_guard()
factory function for instantiation.
- class resiliparse.process_guard.TimeGuard
Bases:
_ResiliparseGuard
Decorator and context manager for guarding the execution time of a running task.
Use the
time_guard()
factory function for instantiation.- progress(self)
Increment epoch counter to indicate progress and reset the guard timeout. This method is thread-safe.
- resiliparse.process_guard.mem_guard(max_memory, absolute=True, grace_period=0, grace_period_ms=0, secondary_grace_period=5, secondary_grace_period_ms=None, interrupt_type=exception_then_signal, send_kill=False, check_interval=500)
Create a
MemGuard
instance that can be used as a decorator or context manager for enforcing memory limits on a running task.MemGuard
guards a function or other context to stay within pre-defined memory bounds. If the running Python process exceeds these bounds while the guard context is active, an exception or signal will be sent to the executing thread.If the thread does not react to this exception, the same escalation procedure will kick in as known from
TimeGuard
. In order forMemGuard
to tolerate short spikes above the memory limit, set thegrace_period
parameter to a positive non-zero value. If memory usage exceeds the limit, a timer will start that expires aftergrace_period
seconds and triggers the interrupt procedure. If memory usage falls below the threshold during the grace period, the timer is reset.MemGuard
provides the same parameters asTimeGuard
for controlling the interrupt escalation behaviour, but the time interval before triggering the next escalation level is independent of the grace period and defaults to five seconds to give the application sufficient time to react and deallocate excess memory. This secondary grace period can be configured with thesecondary_grace_period
parameter and must be at least one second.Warning
MemGuard
is supported only on Linux. You can decorate functions with it on other platforms, but calling them will raise an error.- Parameters:
max_memory (int) – max allowed memory in KiB since context creation before interrupt will be sent
absolute (bool) – whether
max_memory
is an absolute limit for the process or a relative growth limitgrace_period (int) – grace period in seconds before sending an interrupt after exceeding
max_memory
grace_period_ms (int) – grace period in milliseconds (use instead of
grace_period
if higher resolution needed)secondary_grace_period (int) – time to wait after
grace_period
before triggering next escalation levelsecondary_grace_period_ms (int) – secondary grace period in milliseconds (use instead of
secondary_grace_period
if higher resolution needed)interrupt_type (InterruptType) – type of interrupt
send_kill (bool) – send
SIGKILL
as third attempt instead ofSIGTERM
(ignored ifinterrupt_type
isexception
)check_interval (int) – interval in milliseconds between memory consumption checks
- Return type:
- resiliparse.process_guard.progress(ctx=None)
Increment
TimeGuard
epoch counter to indicate progress and reset the guard timeout for the active guard context surrounding the caller.If
ctx
istNone
, the last valid guard context from the global namespace on the call stack will be used. If the guard context does not live in the module’s global namespace, this auto-detection will fail and the caller has to be supplied explicitly.If no valid guard context can be determined, the progress report will fail and a
RuntimeError
will be raised.- Parameters:
ctx – active guard context (will use last global context from stack if unset)
- Raises:
RuntimeError – if no valid
TimeGuard
found
- resiliparse.process_guard.progress_loop(it, ctx=None)
Wraps an iterator into a pass-through iterator that reports progress to an active
resiliparse.process_guard.TimeGuard
context guard after each iteration.- Parameters:
it (t.Iterable[t.Any]) – original iterator
ctx – active guard context (will use last global context from stack if unset)
- Returns:
wrapped iterator
- Return type:
t.Iterable[t.Any]
- resiliparse.process_guard.time_guard(timeout=60, timeout_ms=None, grace_period=15, grace_period_ms=None, interrupt_type=exception_then_signal, send_kill=False, check_interval=500)
Create a
TimeGuard
instance that can be used as a decorator or context manager for guarding the execution time of a running task.If a the guarded context runs longer than the pre-defined timeout, the guard will send an interrupt to the running thread. The timeout can be reset at any time by proactively reporting progress to the guard instance. This can be done by calling
TimeGuard.progress()
on the guard instance or the convenience functionprogress()
(if the context is in the global scope).There are two interrupt mechanisms: throwing an asynchronous exception and sending a UNIX signal. The exception mechanism is the most gentle method of the two, but may be unreliable if execution is blocking outside the Python program flow (e.g., in a native C extension or in a syscall). The signal method is more reliable in this regard, but does not work if the guarded thread is not the interpreter main thread, since only the main thread can receive and handle signals.
The Interrupt behaviour can be configured with the
interrupt_type
parameter, which accepts an enum value of typeInterruptType
:If
interrupt_type
isexception
, anExecutionTimeout
exception will be sent to the running thread after timeout seconds. If the thread does not react, the exception will be thrown once more after grace_period seconds.If
interrupt_type
issignal
, first aSIGINT
will be sent to the current thread (which will trigger aKeyboardInterrupt
exception, but can also be handled with a customsignal
handler). If the thread does not react, a less friendlySIGTERM
will be sent aftergrace_period
seconds. A third and final attempt of aSIGTERM
will be sent after anothergrace_period
seconds.If
interrupt_type
isexception_then_signal
(the default), the first attempt will be an exception and after the grace period, the guard will start sending signals.With
send_kill
set toTrue
, the third and final attempt will be aSIGKILL
instead of aSIGTERM
. This will kill the entire interpreter (even if the guarded thread is not the main thread), so you will need an external facility to restart it.- Parameters:
timeout (int) – max execution time in seconds before invoking interrupt
timeout_ms (int) – max execution time in milliseconds (use instead of
timeout
if higher resolution needed)grace_period (int) – grace period in seconds after which to send another interrupt
grace_period_ms (int) – grace period in milliseconds (use instead of
grace_period
if higher resolution needed)interrupt_type (InterruptType) – type of interrupt
send_kill (bool) – send
SIGKILL
as third attempt instead ofSIGTERM
(ignored ifinterrupt_type
isexception
)check_interval (int) – interval in milliseconds between execution time checks
- Return type: