a
XC?h+ � @ s` d dgZ ddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl
m Z ddl
m Z m
Z
ddlmZ dZd Zd
Zd Ze�� Zd d
� Zdd� ZG dd� de�ZG dd� d�Zdd� ZG dd� de�Z d*dd�Z d d � Z G d d � d e �Z G d d � d e!�Z"G d!d"� d"e!�Z#e#Z$G d#d$� d$e#�Z%G d%d&� d&e!�Z&G d'd(� d(e&�Z'G d)d� de"�Z(dS )+�Pool�
ThreadPool� N� )�util)� get_context� TimeoutError)�wait�INIT�RUN�CLOSE� TERMINATEc C s t t| � �S �N)�list�map��args� r �9/opt/alt/python39/lib64/python3.9/multiprocessing/pool.py�mapstar/ s r c C s t t�| d | d ��S )Nr r )r � itertools�starmapr r r r � starmapstar2 s r c @ s e Zd Zdd� Zdd� ZdS )�RemoteTracebackc C s
|| _ d S r
��tb)�selfr r r r �__init__: s zRemoteTraceback.__init__c C s | j S r
r �r r r r �__str__< s zRemoteTraceback.__str__N)�__name__�
__module__� __qualname__r r r r r r r 9 s r c @ s e Zd Zdd� Zdd� ZdS )�ExceptionWithTracebackc C s0 t �t|�||�}d�|�}|| _d| | _d S )N� z
"""
%s""")� traceback�format_exception�type�join�excr )r r( r r r r r @ s
z ExceptionWithTraceback.__init__c C s t | j| jffS r
)� rebuild_excr( r r r r r �
__reduce__E s z!ExceptionWithTraceback.__reduce__N)r r r! r r* r r r r r" ? s r" c C s t |�| _| S r
)r � __cause__)r( r r r r r) H s
r) c s, e Zd Z� fdd�Zdd� Zdd� Z� ZS )�MaybeEncodingErrorc s. t |�| _t |�| _tt| ��| j| j� d S r
)�reprr( �value�superr, r )r r( r. �� __class__r r r T s
zMaybeEncodingError.__init__c C s d| j | jf S )Nz(Error sending result: '%s'. Reason: '%s')r. r( r r r r r Y s �zMaybeEncodingError.__str__c C s d| j j| f S )Nz<%s: %s>)r1 r r r r r �__repr__] s zMaybeEncodingError.__repr__)r r r! r r r2 �
__classcell__r r r0 r r, P s r, r Fc
C s� |d ur(t |t�r|dks(td�|���|j}| j}t| d�rR| j�� |j �� |d urb||� d}|d u s~|�r�||k �r�z
|� } W n& t
t fy� t �
d� Y �q�Y n0 | d u r�t �
d� �q�| \}
} } }
}zd| |
i |��f}W nJ t�y4 } z0|�r| tu�rt||j�}d|f}W Y d }~n
d }~0 0 z||
| |f� W nT t�y� } z:t||d �}t �
d | � ||
| d|ff� W Y d }~n
d }~0 0 d } }
} } }
}|d7 }qft �
d
| � d S ) Nr zMaxtasks {!r} is not valid�_writerr z)worker got EOFError or OSError -- exitingz worker got sentinel -- exitingTFz0Possible encoding error while sending result: %sz worker exiting after %d tasks)�
isinstance�int�AssertionError�format�put�get�hasattrr4 �close�_reader�EOFError�OSErrorr �debug� Exception�_helper_reraises_exceptionr" �
__traceback__r, )�inqueue�outqueue� initializer�initargsZmaxtasks�wrap_exceptionr9 r: Z completed�task�job�i�funcr �kwds�result�e�wrappedr r r �workera sN �
�(
rQ c C s | �d S r
r )Zexr r r rB � s rB c s. e Zd Zdd�� fdd�
Z� fdd�Z� ZS )�
_PoolCacheN��notifierc s || _ t� j|i |�� d S r
)rT r/ r )r rT r rM r0 r r r � s z_PoolCache.__init__c s t � �|� | s | j�d � d S r
)r/ � __delitem__rT r9 )r �itemr0 r r rU � s z_PoolCache.__delitem__)r r r! r rU r3 r r r0 r rR � s rR c @ s� e Zd ZdZedd� �ZdKdd�Zeje fdd �Z
d
d � Z d d
� Z edd� �Z
edd� �Zdd� Zedd� �Zedd� �Zdd� Zdd� Zdi fd d �ZdLd d �ZdMd d!�ZdNd"d#�Zd$d%� ZdOd'd(�ZdPd)d*�Zdi ddfd+d,�ZdQd-d.�Z dRd/d0�Z edSd1d2��Z e d3d4� �Z ed5d6� �Z!ed7d8� �Z"ed9d:� �Z#d;d<� Z$d=d>� Z%d?d@� Z&dAdB� Z'edCdD� �Z(e dEdF� �Z)dGdH� Z*dIdJ� Z+dS )Tr Tc O s | j |i |��S r
��Process)�ctxr rM r r r rX � s z Pool.ProcessNr c C s g | _ t| _|pt� | _| �� t�� | _| j�� | _ t
| j d�| _ || _ || _
|| _|d u rjt�� phd}|dk rztd��|d ur�t|�s�td��|| _z | �� W nF ty� | j D ]}|jd u r�|�� q�| j D ] }|�� qւ Y n0 | �� }tj t j | j | j| j| j | j| j | j | j!| j
| j| j | j"|| j fd�| _#d| j#_$t%| j#_| j#�&� tj t j'| j| j(| j!| j | j fd�| _)d| j)_$t%| j)_| j)�&� tj t j*| j!| j+| j fd�| _,d| j,_$t%| j,_| j,�&� t-j.| | j/| j| j | j!| j | j | j#| j)| j,| j f dd�| _0t%| _d S ) NrS r z&Number of processes must be at least 1z initializer must be a callable��targetr T� )r Z exitpriority)1�_poolr �_stater �_ctx�
_setup_queues�queue� SimpleQueue�
_taskqueue�_change_notifierrR �_cache�_maxtasksperchild� _initializer� _initargs�os� cpu_count�
ValueError�callable� TypeError�
_processes�_repopulate_poolrA �exitcode� terminater' �_get_sentinels� threadingZThreadr �_handle_workersrX �_inqueue� _outqueue�_wrap_exception�_worker_handler�daemonr
�start�
_handle_tasks�
_quick_put�
_task_handler�_handle_results�
_quick_get�_result_handlerr ZFinalize�_terminate_pool�
_terminate)r � processesrF rG �maxtasksperchild�context�p� sentinelsr r r r � s~
��
��
�
��z
Pool.__init__c C s> | j |kr:|d| ��t| d� t| dd �d ur:| j�d � d S )Nz&unclosed running multiprocessing pool )�sourcerd )r^ �ResourceWarning�getattrrd r9 )r Z_warnr
r r r �__del__ s
�z Pool.__del__c C s0 | j }d|j� d|j� d| j� dt| j�� d� S )N�<�.z state=z pool_size=�>)r1 r r! r^ �lenr] )r �clsr r r r2 s ��z
Pool.__repr__c C s | j jg}| jjg}g |�|�S r
)rv r= rd )r Ztask_queue_sentinelsZself_notifier_sentinelsr r r rr s
zPool._get_sentinelsc C s dd� | D �S )Nc S s g | ]}t |d �r|j�qS )�sentinel)r; r� )�.0rQ r r r �
s
�z.Pool._get_worker_sentinels..r �Zworkersr r r �_get_worker_sentinels s �zPool._get_worker_sentinelsc C sP d}t tt| ���D ]6}| | }|jd urt�d| � |�� d}| |= q|S )NF�cleaning up worker %dT)�reversed�ranger� rp r r@ r' )�poolZcleanedrK rQ r r r �_join_exited_workers s
zPool._join_exited_workersc C s0 | � | j| j| j| j| j| j| j| j| j | j
�
S r
) �_repopulate_pool_staticr_ rX rn r] ru rv rg rh rf rw r r r r ro . s �zPool._repopulate_poolc
C sf t |t|� �D ]P}
|| t|||||| fd�} | j�dd�| _d| _| �� |�| � t� d� qd S )NrZ rX Z
PoolWorkerTz added worker)
r� r� rQ �name�replacery rz �appendr r@ ) rY rX r� r� rD rE rF rG r� rH rK �wr r r r� 7 s ��
z Pool._repopulate_pool_staticc
C s* t �|�r&t �| ||||||||| �
d S r
)r r� r� )
rY rX r� r� rD rE rF rG r� rH r r r �_maintain_poolJ s
�zPool._maintain_poolc C s4 | j �� | _| j �� | _| jjj| _| jjj| _ d S r
)
r_ rb ru rv r4 �sendr| r= �recvr r r r r r` V s zPool._setup_queuesc C s | j tkrtd��d S )NzPool not running)r^ r
rk r r r r �_check_running\ s
zPool._check_runningc C s | � |||��� S r
)� apply_asyncr: )r rL r rM r r r �apply` s z
Pool.applyc C s | � ||t|��� S r
)�
_map_asyncr r: �r rL �iterable� chunksizer r r r g s zPool.mapc C s | � ||t|��� S r
)r� r r: r� r r r r n s z Pool.starmapc C s | � ||t|||�S r
)r� r �r rL r� r� �callback�error_callbackr r r �
starmap_asyncv s �zPool.starmap_asyncc
c sl z,d}t |�D ]\}}||||fi fV qW n: tyf } z"||d t|fi fV W Y d }~n
d }~0 0 d S )N�����r )� enumeraterA rB )r Z
result_jobrL r� rK �xrO r r r �_guarded_task_generation~ s z Pool._guarded_task_generationr c C s� | � � |dkr:t| �}| j�| �|j||�|jf� |S |dk rPtd�|���t �
|||�}t| �}| j�| �|jt |�|jf� dd� |D �S d S )Nr z Chunksize must be 1+, not {0:n}c s s | ]}|D ]
}|V q
qd S r
r �r� �chunkrV r r r � � � z Pool.imap..) r� � IMapIteratorrc r9 r� �_job� _set_lengthrk r8 r �
_get_tasksr �r rL r� r� rN � task_batchesr r r �imap� s4 �������z Pool.imapc C s� | � � |dkr:t| �}| j�| �|j||�|jf� |S |dk rPtd�|���t �
|||�}t| �}| j�| �|jt |�|jf� dd� |D �S d S )Nr z Chunksize must be 1+, not {0!r}c s s | ]}|D ]
}|V q
qd S r
r r� r r r r� � r� z&Pool.imap_unordered..) r� �IMapUnorderedIteratorrc r9 r� r� r� rk r8 r r� r r� r r r �imap_unordered� s0 ������zPool.imap_unorderedc C s6 | � � t| ||�}| j�|jd|||fgd f� |S �Nr )r� � ApplyResultrc r9 r� )r rL r rM r� r� rN r r r r� � s zPool.apply_asyncc C s | � ||t|||�S r
)r� r r� r r r � map_async� s �zPool.map_asyncc
C s� | � � t|d�st|�}|d u rJtt|�t| j�d �\}}|rJ|d7 }t|�dkrZd}t�|||�}t| |t|�||d�} | j �
| � | j ||�d f� | S )N�__len__� r r �r� )
r� r; r �divmodr� r] r r� � MapResultrc r9 r� r� )
r rL r� Zmapperr� r� r� Zextrar� rN r r r r� � s,
����zPool._map_asyncc C s" t | |d� |�� s |�� q d S )N)�timeout)r �emptyr: )r� �change_notifierr� r r r �_wait_for_updates� s zPool._wait_for_updatesc C st t �� }|jtks |r\|jtkr\| �||||||| |
| | �
g | �|��|
�}| �||� q|�d � t �
d� d S )Nzworker handler exiting) rs �current_threadr^ r
r r� r� r� r9 r r@ )r� �cache� taskqueuerY rX r� r� rD rE rF rG r� rH r� r� �threadZcurrent_sentinelsr r r rt � s �
zPool._handle_workersc
C s� t �� }t| jd �D �]
\}}d }z�|D ]�}|jtkrDt�d� q�z ||� W q( ty� } zH|d d� \}
} z||
� | d| f� W n t
y� Y n0 W Y d } ~ q(d } ~ 0 0 q(|r�t�d� |r�|d nd} || d � W d } }}
qW d } }}
�q,W d } }}
qd } }}
0 qt�d� z6t�d� |� d � t�d � |D ]} |d � �qPW n t �y� t�d
� Y n0 t�d � d S ) Nz'task handler found thread._state != RUN� Fzdoing set_length()r r� ztask handler got sentinelz/task handler sending sentinel to result handlerz(task handler sending sentinel to workersz/task handler got OSError when sending sentinelsztask handler exiting)
rs r� �iterr: r^ r
r r@ rA �_set�KeyErrorr9 r? )
r� r9 rE r� r� r� ZtaskseqZ
set_lengthrI rO rJ �idxr� r r r r{ sJ
� �
zPool._handle_tasksc C s� t �� }z
|� }W n" ttfy4 t�d� Y d S 0 |jtkrLt�d� q�|d u r`t�d� q�|\}}}z|| �||� W n t y� Y n0 d } }}q|�r6|jt
k�r6z
|� }W n" ttfy� t�d� Y d S 0 |d u r�t�d� q�|\}}}z|| �||� W n t �y& Y n0 d } }}q�t | d��r�t�d� z,t d�D ] }| j
�� �sl �qv|� �qVW n ttf�y� Y n0 t�dt|�|j� d S ) Nz.result handler got EOFError/OSError -- exitingz,result handler found thread._state=TERMINATEzresult handler got sentinelz&result handler ignoring extra sentinelr= z"ensuring that outqueue is not full�
z7result handler exiting: len(cache)=%s, thread._state=%s)rs r� r? r> r r@ r^ r
r� r� r r; r� r= �pollr� )rE r: r� r� rI rJ rK �objr r r r~ : sZ
�zPool._handle_resultsc c s0 t |�}tt�||��}|s d S | |fV qd S r
)r� �tupler �islice)rL �it�sizer� r r r r� v s
zPool._get_tasksc C s t d��d S )Nz:pool objects cannot be passed between processes or pickled)�NotImplementedErrorr r r r r* s �zPool.__reduce__c C s2 t �d� | jtkr.t| _t| j_| j�d � d S )Nz closing pool)r r@ r^ r
r rx rd r9 r r r r r< � s
z
Pool.closec C s t �d� t| _| �� d S )Nzterminating pool)r r@ r r^ r� r r r r rq � s
zPool.terminatec C sj t �d� | jtkr td��n| jttfvr4td��| j�� | j �� | j
�� | j D ] }|�� qXd S )Nz joining poolzPool is still runningzIn unknown state) r r@ r^ r
rk r r rx r' r} r� r] )r r� r r r r' � s
z Pool.joinc C s@ t �d� | j�� |�� r<| j�� r<| j�� t� d� qd S )Nz7removing tasks from inqueue until task handler finishedr )
r r@ Z_rlock�acquire�is_aliver= r� r� �time�sleep)rD � task_handlerr� r r r �_help_stuff_finish� s
zPool._help_stuff_finishc
C sX t �d� t|_|�d � t|_t �d� | �||t|�� |�� sXt| �dkrXtd��t|_|�d � |�d � t �d� t �
� |ur�|� � |r�t |d d�r�t �d� |D ]}
|
j
d u r�|
�� q�t �d� t �
� |ur�|� � t �d � t �
� |u�r|� � |�rTt |d d��rTt �d
� |D ](}
|
�� �r*t �d |
j � |
� � �q*d S ) Nzfinalizing poolz&helping task handler/workers to finishr z.Cannot have cache with result_hander not alivezjoining worker handlerrq zterminating workerszjoining task handlerzjoining result handlerzjoining pool workersr� )r r@ r r^ r9 r� r� r� r7 rs r� r' r; rp rq �pid) r� r� rD rE r� r� Zworker_handlerr� Zresult_handlerr� r� r r r r� � sB
�
zPool._terminate_poolc C s | � � | S r
)r� r r r r � __enter__� s zPool.__enter__c C s | � � d S r
)rq )r �exc_typeZexc_valZexc_tbr r r �__exit__� s z
Pool.__exit__)NNr NN)N)N)NNN)r )r )NNN)NNN)N),r r r! rw � staticmethodrX r �warnings�warnr
r� r2 rr r� r� ro r� r� r` r� r� r r r� r� r� r� r� r� r� r� � classmethodrt r{ r~ r� r* r< rq r' r� r� r� r� r r r r r � st
�
P
�
�
�
�
-
;
5c @ sJ e Zd Zdd� Zdd� Zdd� Zddd �Zdd
d �Zd d
� Ze e
j �Z dS )r� c C s> || _ t�� | _tt�| _|j| _|| _|| _ | | j| j< d S r
)
r] rs ZEvent�_event�next� job_counterr� re � _callback�_error_callback)r r� r� r� r r r r � s
zApplyResult.__init__c C s
| j �� S r
)r� Zis_setr r r r �ready� s zApplyResult.readyc C s | � � std�| ���| jS )Nz{0!r} not ready)r� rk r8 �_successr r r r �
successful� s zApplyResult.successfulNc C s | j �|� d S r
)r� r �r r� r r r r � s zApplyResult.waitc C s, | � |� | �� st�| jr"| jS | j�d S r
)r r� r r� �_valuer� r r r r: � s
zApplyResult.getc C sZ |\| _ | _| jr$| j r$| �| j� | jr<| j s<| �| j� | j�� | j| j= d | _d S r
) r� r� r� r� r� �setre r� r] �r rK r� r r r r� s
zApplyResult._set)N)N)
r r r! r r� r� r r: r� r� �types� GenericAlias�__class_getitem__r r r r r� � s
r� c @ s e Zd Zdd� Zdd� ZdS )r� c C sh t j| |||d� d| _d g| | _|| _|dkrNd| _| j�� | j| j = n|| t
|| � | _d S )Nr� Tr ) r� r r� r� �
_chunksize� _number_leftr� r� re r� �bool)r r� r� �lengthr� r� r r r r s
�
zMapResult.__init__c C s� | j d8 _ |\}}|rv| jrv|| j|| j |d | j �< | j dkr�| jrZ| �| j� | j| j= | j�� d | _ nL|s�| jr�d| _|| _| j dkr�| j
r�| �
| j� | j| j= | j�� d | _ d S )Nr r F) r� r� r� r� r� re r� r� r� r] r� )r rK Zsuccess_result�successrN r r r r� & s&
zMapResult._setN)r r r! r r� r r r r r� s
r� c @ s: e Zd Zdd� Zdd� Zd dd�ZeZdd � Zd
d � ZdS )
r� c C sT || _ t�t�� �| _tt�| _|j| _t �
� | _ d| _ d | _
i | _| | j| j< d S r� )r] rs Z ConditionZLock�_condr� r� r� re � collections�deque�_items�_index�_length� _unsorted)r r� r r r r D s
zIMapIterator.__init__c C s | S r
r r r r r �__iter__O s zIMapIterator.__iter__Nc C s� | j �� z| j�� }W nv ty� | j| jkr
sN �
- =++E