U
��.e$+ � @ s� d dl mZmZmZ d dlmZ d dlmZ ddl m
Z
ddl m Z ddl m Z m
Z
ddl mZmZmZ dd l mZmZ dd
l mZ dd lmZ dd lmZ ee �ZG d
d� de�ZdS )� )�absolute_import�division�unicode_literals)�unichr)�deque� )�spaceCharacters)�entities)� asciiLetters�asciiUpper2Lower)�digits� hexDigits�EOF)�
tokenTypes�
tagTokenTypes)�replacementCharacters)�HTMLInputStream)�Triec sd e Zd ZdZd�� fdd� Zdd� Zdd� Zd�d
d �Zd d
� Zdd� Z dd� Z
dd� Z dd� Z dd� Z
dd� Zdd� Zd d � Zd d � Zd d!� Zd"d#� Zd$d%� Zd&d'� Zd(d)� Zd*d+� Zd,d-� Zd.d/� Zd0d1� Zd2d3� Zd4d5� Z d6d7� Z d8d9� Z d:d;� Z dd?� Z!d@dA� Z"dBdC� Z#dDdE� Z$dFdG� Z%dHdI� Z&dJdK� Z'dLdM� Z(dNdO� Z)dPdQ� Z*dRdS� Z+dTdU� Z,dVdW� Z-dXdY� Z.dZd[� Z/d\d]� Z0d^d_� Z1d`da� Z2dbdc� Z3ddde� Z4dfdg� Z5dhdi� Z6djdk� Z7dldm� Z8dndo� Z9dpdq� Z:drds� Z;dtdu� Zdzd{� Z?d|d}� Z@d~d� ZAd�d�� ZBd�d�� ZCd�d�� ZDd�d�� ZEd�d�� ZFd�d�� ZGd�d�� ZHd�d�� ZId�d�� ZJd�d�� ZKd�d�� ZL� ZMS )��
HTMLTokenizera This class takes care of tokenizing HTML.
* self.currentToken
Holds the token that is currently being processed.
* self.state
Holds a reference to the method to be invoked... XXX
* self.stream
Points to HTMLInputStream object.
Nc sF t |f|�| _|| _d| _g | _| j| _d| _d | _t t
| �� � d S )NF) r �stream�parserZ
escapeFlagZ
lastFourChars� dataState�state�escape� currentToken�superr �__init__)�selfr r �kwargs�� __class__� �C/usr/lib/python3.8/site-packages/pip/_vendor/html5lib/_tokenizer.pyr " s zHTMLTokenizer.__init__c c sP t g �| _| �� rL| jjr6td | jj�d�d�V q| jr
| j�� V q6q
dS )z� This is where the magic happens.
We do our usually processing through the states and when we have a token
to return we yield the token which pauses processing until the next token
is requested.
�
ParseErrorr ��type�dataN)r �
tokenQueuer r �errorsr �pop�popleft�r r! r! r" �__iter__1 s
zHTMLTokenizer.__iter__c % C s t }d}|rt}d}g }| j�� }||krH|tk rH|�|� | j�� }q"td�|�|�}|tkr�t| }| j �t
d dd|id�� �nbd| kr�d ks�n |d
kr�d }| j �t
d dd|id�� �n d | kr�d
k�s�n d| kr�dk�s�n d| k�rdk�s�n d| k�r,dk�s�n |t ddddddddd d d d d d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d
g#�k�r�| j �t
d dd|id�� z t |�}W n> t
k
�r� |d6 }t d|d? B �t d7|d8@ B � }Y nX |d9k�r| j �t
d d:d;�� | j�|� |S )r# z'expected-tag-name-but-got-right-bracketr$ rO z<>�?z'expected-tag-name-but-got-question-markzexpected-tag-namerH T)r r: �markupDeclarationOpenStater �closeTagOpenStater
r r � tagNameStater' r; r rA �bogusCommentStaterp r! r! r" rc i s@
�
�
�
� zHTMLTokenizer.tagOpenStatec C s� | j �� }|tkr0td |g dd�| _| j| _n�|dkrX| j�td dd�� | j | _nn|t
kr�| j�td dd�� | j�td d
d�� | j | _n0| j�td d d |id
�� | j � |� | j | _dS )NrZ F�r% rY r& r[ rt r# z*expected-closing-tag-but-got-right-bracketr$ z expected-closing-tag-but-got-eofrO � r' r; r r r� � scriptDataDoubleEscapedStater r� r
rA rp r! r! r" r� � s
z.HTMLTokenizer.scriptDataDoubleEscapeStartStatec C s� | j �� }|dkr2| j�td dd�� | j| _n�|dkrZ| j�td dd�� | j| _nt|dkr�| j�td dd�� | j�td dd�� n>|tkr�| j�td d d�� | j | _n| j�td |d�� d
S � Nr� rO r$ rH r_ r# r` r2 �eof-in-script-in-scriptT)
r r: r' r; r � scriptDataDoubleEscapedDashStater �(scriptDataDoubleEscapedLessThanSignStater r rp r! r! r" r� � s*
� �
�
z*HTMLTokenizer.scriptDataDoubleEscapedStatec C s� | j �� }|dkr2| j�td dd�� | j| _n�|dkrZ| j�td dd�� | j| _n�|dkr�| j�td dd�� | j�td dd�� | j| _nF|t kr�| j�td d d�� | j
| _n | j�td |d�� | j| _d
S r� ) r r: r' r; r �$scriptDataDoubleEscapedDashDashStater r� r� r r rp r! r! r" r� s.
� �
�
z.HTMLTokenizer.scriptDataDoubleEscapedDashStatec C s | j �� }|dkr*| j�td dd�� n�|dkrR| j�td dd�� | j| _n�|dkrz| j�td dd�� | j| _n�|dkr�| j�td dd�� | j�td d d�� | j| _nF|t kr�| j�td d
d�� | j
| _n | j�td |d�� | j| _d S ) Nr� rO r$ rH rt r_ r# r` r2 r� T) r r: r' r; r r� r ro r� r r rp r! r! r" r� s2
� �
�
z2HTMLTokenizer.scriptDataDoubleEscapedDashDashStatec C sP | j �� }|dkr8| j�td dd�� d| _| j| _n| j �|� | j | _dS )Nrs rO r$ r/ T)
r r: r' r; r r � scriptDataDoubleEscapeEndStater rA r� rp r! r! r" r� 0 s
z6HTMLTokenizer.scriptDataDoubleEscapedLessThanSignStatec C s� | j �� }|ttd�B krR| j�td |d�� | j�� dkrH| j | _
q�| j | _
nB|t kr�| j�td |d�� | j|7 _n| j �
|� | j | _
dS r� )r r: r r> r' r; r r r� r� r r� r
rA rp r! r! r" r� ; s
z,HTMLTokenizer.scriptDataDoubleEscapeEndStatec C s0 | j �� }|tkr$| j �td� �n|tkrJ| jd �|dg� | j| _n�|dkr\| � � n�|dkrn| j
| _n�|dkr�| j �t d dd �� | jd �|dg� | j| _n�|d
kr�| j �t d d d �� | jd �d dg� | j| _nF|t
k�r| j �t d d
d �� | j| _n | jd �|dg� | j| _dS )NTr& r/ rt rs )�'�"rM rH r# �#invalid-character-in-attribute-namer$ r_ r` r2 z#expected-attribute-name-but-got-eof)r r: r rd r
r r; �attributeNameStater r^ r} r' r r r rp r! r! r" r| K s<
�
�
�
z&HTMLTokenizer.beforeAttributeNameStatec C s� | j �� }d}d}|dkr&| j| _�n.|tkr\| jd d d || j �td� 7 < d}n�|dkrjd}n�|tkr|| j| _n�|dkr�| j | _n�|d kr�| j
� t d
d d �� | jd d d d
7 < d}n�|dk�r | j
� t d
dd �� | jd d d |7 < d}nH|t
k�r6| j
� t d
dd �� | j| _n | jd d d |7 < d}|�r�| jd d d �t�| jd d d< | jd d d� D ]>\}}| jd d d |k�r�| j
� t d
dd �� �qҐq�|�r�| �� dS )NTFrM r& rJ r rt rs r_ r# r` r$ r2 �r� r� rH r� zeof-in-attribute-namezduplicate-attribute)r r: �beforeAttributeValueStater r
r rd r �afterAttributeNameStater} r' r; r r r r\ r r^ )r r& ZleavingThisStateZ emitTokenrY �_r! r! r" r� i s^
�
�
�
�
� �
z HTMLTokenizer.attributeNameStatec C sD | j �� }|tkr$| j �td� �n |dkr8| j| _�n|dkrJ| �� n�|tkrp| jd � |dg� | j
| _n�|dkr�| j | _n�|dkr�| j � t
d d d
�� | jd � d dg� | j
| _n�|d kr�| j � t
d d
d
�� | jd � |dg� | j
| _nF|tk�r$| j � t
d dd
�� | j| _n | jd � |dg� | j
| _dS )NTrM rt r& r/ rs r_ r# r` r$ r2 r� z&invalid-character-after-attribute-namez expected-end-of-tag-but-got-eof)r r: r rd r� r r^ r
r r; r� r} r' r r r rp r! r! r" r� � s@
�
�
�
z%HTMLTokenizer.afterAttributeNameStatec C sh | j �� }|tkr$| j �td� �n@|dkr8| j| _�n,|dkrX| j| _| j �|� �n |dkrj| j| _n�|dkr�| j �
t d dd�� | � � n�|d kr�| j �
t d d
d�� | j
d d d
d7 < | j| _n�|dk�r| j �
t d dd�� | j
d d d
|7 < | j| _nL|tk�rB| j �
t d dd�� | j| _n"| j
d d d
|7 < | j| _dS )NTr� rG r� rt r# z.expected-attribute-value-but-got-right-bracketr$ r_ r` r& rJ r r2 )rM rH �`z"equals-in-unquoted-attribute-valuez$expected-attribute-value-but-got-eof)r r: r rd � attributeValueDoubleQuotedStater �attributeValueUnQuotedStaterA � attributeValueSingleQuotedStater' r; r r^ r r r rp r! r! r" r� � sF
�
�
�
�
z'HTMLTokenizer.beforeAttributeValueStatec C s� | j �� }|dkr | j| _n�|dkr0| �d� n�|dkrj| j�td dd�� | jd d d d
7 < nN|t kr�| j�td d d�� | j
| _n&| jd d d || j � d � 7 < d
S )Nr� rG r_ r# r` r$ r& rJ r r2 z#eof-in-attribute-value-double-quote)r� rG r_ T� r r: �afterAttributeValueStater rX r' r; r r r r rd rp r! r! r" r� � s&
� �
�z-HTMLTokenizer.attributeValueDoubleQuotedStatec C s� | j �� }|dkr | j| _n�|dkr0| �d� n�|dkrj| j�td dd�� | jd d d d
7 < nN|t kr�| j�td d d�� | j
| _n&| jd d d || j � d � 7 < d
S )Nr� rG r_ r# r` r$ r& rJ r r2 z#eof-in-attribute-value-single-quote)r� rG r_ Tr� rp r! r! r" r� � s&
� �
�z-HTMLTokenizer.attributeValueSingleQuotedStatec C s | j �� }|tkr | j| _n�|dkr0| �d� n�|dkrB| �� n�|dkr|| j�t d dd�� | j
d d d |7 < n�|d
kr�| j�t d d d�� | j
d d d d 7 < nV|t kr�| j�t d d
d�� | j | _n.| j
d d d || j �
td�tB � 7 < dS )NrG rt )r� r� rM rH r� r# z0unexpected-character-in-unquoted-attribute-valuer$ r& rJ r r_ r` r2 z eof-in-attribute-value-no-quotes)rG rt r� r� rM rH r� r_ T)r r: r r| r rX r^ r' r; r r r r rd r> rp r! r! r" r� s4
� � �
�
z)HTMLTokenizer.attributeValueUnQuotedStatec C s� | j �� }|tkr | j| _n�|dkr.| �� np|dkr@| j| _n^|tkrt| j� t
d dd�� | j � |� | j | _n*| j� t
d dd�� | j � |� | j| _dS )Nrt rs r# z$unexpected-EOF-after-attribute-valuer$ z*unexpected-character-after-attribute-valueT)
r r: r r| r r^ r} r r' r; r rA r rp r! r! r" r� s&
�
� z&HTMLTokenizer.afterAttributeValueStatec C s� | j �� }|dkr&d| jd< | �� n^|tkrZ| j�td dd�� | j �|� | j | _
n*| j�td dd�� | j �|� | j | _
dS )Nrt Tr[ r# z#unexpected-EOF-after-solidus-in-tagr$ z)unexpected-character-after-solidus-in-tag) r r: r r^ r r' r; r rA r r r| rp r! r! r" r} 4 s
�
� z&HTMLTokenizer.selfClosingStartTagStatec C sD | j �d�}|�dd�}| j�td |d�� | j �� | j| _dS )Nrt r_ r2 �Commentr$ T) r rd �replacer' r; r r: r r rp r! r! r" ry F s �
z HTMLTokenizer.bogusCommentStatec C s� | j �� g}|d dkrR|�| j �� � |d dkrPtd dd�| _| j| _dS n�|d dkr�d}dD ](}|�| j �� � |d |krfd } q�qf|r�td
dd d dd �| _| j| _dS n�|d d k�rD| jd k �rD| jj j
�rD| jj j
d j | jj j k�rDd}d
D ].}|�| j �� � |d |k�rd } �q2�q|�rD| j
| _dS | j�td dd�� |�rt| j �|�� � �qZ| j| _dS )NrJ r� r� r/ r$ T)�d�D))�o�O�rD �C��t�T��y�Y��p�P��e�EFZDoctype)r% rY �publicId�systemId�correct�[)r� r� �Ar� r� r� r# zexpected-dashes-or-doctype)r r: r; r r �commentStartStater � doctypeStater ZtreeZ openElements� namespaceZdefaultNamespace�cdataSectionStater' rA r) ry )r rC �matched�expectedr! r! r" rv U s\ ����
�z(HTMLTokenizer.markupDeclarationOpenStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� | j�| j� | j| _nP|t kr�| j�td d
d�� | j�| j� | j| _n| jd |7 < | j
| _d S ) Nr� r_ r# r` r$ r& r2 rt �incorrect-comment�eof-in-commentT) r r: �commentStartDashStater r' r; r r r r � commentStaterp r! r! r" r� � s.
� �
�
z HTMLTokenizer.commentStartStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� | j�| j� | j| _nT|t kr�| j�td d
d�� | j�| j� | j| _n | jd d| 7 < | j
| _d S ) Nr� r_ r# r` r$ r& � -�rt r� r� T) r r: �commentEndStater r' r; r r r r r� rp r! r! r" r� � s.
� �
�
z#HTMLTokenizer.commentStartDashStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < nT|tkr�| j�td dd�� | j�| j� | j | _n | jd || j �
d � 7 < d
S ) Nr� r_ r# r` r$ r& r2 r� )r� r_ T) r r: �commentEndDashStater r' r; r r r r rd rp r! r! r" r� � s$
� �
�zHTMLTokenizer.commentStatec C s� | j �� }|dkr | j| _n�|dkrV| j�td dd�� | jd d7 < | j| _nT|t kr�| j�td dd�� | j�| j� | j
| _n | jd d| 7 < | j| _d S )
Nr� r_ r# r` r$ r& r� zeof-in-comment-end-dashT) r r: r� r r' r; r r r� r r rp r! r! r" r� � s$
�
�
z!HTMLTokenizer.commentEndDashStatec C s, | j �� }|dkr*| j�| j� | j| _n�|dkrd| j�td dd�� | jd d7 < | j| _n�|dkr�| j�td d d�� | j | _n�|d
kr�| j�td d d�� | jd |7 < nj|t
kr�| j�td d d�� | j�| j� | j| _n4| j�td d
d�� | jd d| 7 < | j| _dS )Nrt r_ r# r` r$ r& u --�rr z,unexpected-bang-after-double-dash-in-commentr� z,unexpected-dash-after-double-dash-in-commentzeof-in-comment-double-dashzunexpected-char-in-commentz--T) r r: r' r; r r r r r� �commentEndBangStater rp r! r! r" r� � s@
�
�
� �
�z HTMLTokenizer.commentEndStatec C s� | j �� }|dkr*| j�| j� | j| _n�|dkrN| jd d7 < | j| _n�|dkr�| j�td dd�� | jd d 7 < | j | _nT|t
kr�| j�td d
d�� | j�| j� | j| _n | jd d| 7 < | j | _d S ) Nrt r� r& z--!r_ r# r` r$ u --!�z eof-in-comment-end-bang-stateT) r r: r' r; r r r r� r r� r rp r! r! r" r� � s,
�
�
z!HTMLTokenizer.commentEndBangStatec C s� | j �� }|tkr | j| _nj|tkr\| j�td dd�� d| j d< | j�| j � | j
| _n*| j�td dd�� | j � |� | j| _dS )Nr# �!expected-doctype-name-but-got-eofr$ Fr� zneed-space-after-doctypeT) r r: r �beforeDoctypeNameStater r r' r; r r r rA rp r! r! r" r� s
�
� zHTMLTokenizer.doctypeStatec C s� | j �� }|tkrn�|dkrT| j�td dd�� d| jd< | j�| j� | j| _n�|dkr�| j�td dd�� d | jd
< | j | _nR|t
kr�| j�td d d�� d| jd< | j�| j� | j| _n|| jd
< | j | _d S )
Nrt r# z+expected-doctype-name-but-got-right-bracketr$ Fr� r_ r` r2 rY r� T) r r: r r' r; r r r r �doctypeNameStater rp r! r! r" r� s4
�
�
�
z$HTMLTokenizer.beforeDoctypeNameStatec C s | j �� }|tkr2| jd �t�| jd< | j| _n�|dkrh| jd �t�| jd< | j� | j� | j
| _n�|dkr�| j� t d dd�� | jd d7 < | j | _nh|t
kr�| j� t d dd�� d | jd
< | jd �t�| jd< | j� | j� | j
| _n| jd |7 < d S ) NrY rt r_ r# r` r$ r2 zeof-in-doctype-nameFr� T)r r: r r r\ r �afterDoctypeNameStater r' r; r r r� r rp r! r! r" r� 6 s0
�
�
z HTMLTokenizer.doctypeNameStatec C sH | j �� }|tkr�n.|dkr8| j�| j� | j| _�n |tkr�d| jd< | j � |� | j�t
d dd�� | j�| j� | j| _n�|dkr�d}d D ] }| j �� }||kr�d} q�q�|r�| j | _dS nD|d
k�r
d}d D ] }| j �� }||kr�d} q�q�|�r
| j | _dS | j � |� | j�t
d d d
|id�� d| jd< | j
| _dS )Nrt Fr� r# �eof-in-doctyper$ r� T))�u�U)�b�B)�l�L)�i�Ir� ��s�S)r� r� r� r� )�m�Mz*expected-space-or-right-bracket-in-doctyper& r1 )r r: r r' r; r r r r rA r � afterDoctypePublicKeywordState� afterDoctypeSystemKeywordState�bogusDoctypeState)r r& r� r� r! r! r" r� O sT
�
�
z#HTMLTokenizer.afterDoctypeNameStatec C s� | j �� }|tkr | j| _n�|dkrP| j�td dd�� | j �|� | j| _nT|t kr�| j�td dd�� d| j
d< | j�| j
� | j | _n| j �|� | j| _dS � N)r� r� r# �unexpected-char-in-doctyper$ r� Fr� T) r r: r �"beforeDoctypePublicIdentifierStater r' r; r rA r r r rp r! r! r" r� � s&
�
�
z,HTMLTokenizer.afterDoctypePublicKeywordStatec C s� | j �� }|tkrn�|dkr0d| jd< | j| _n�|dkrLd| jd< | j| _n�|dkr�| j�t d dd�� d | jd
< | j�| j� | j
| _nh|t kr�| j�t d d d�� d | jd
< | j�| j� | j
| _n(| j�t d d d�� d | jd
< | j | _d
S )Nr� r/ r� r� rt r# �unexpected-end-of-doctyper$ Fr� r� r� T)
r r: r r �(doctypePublicIdentifierDoubleQuotedStater �(doctypePublicIdentifierSingleQuotedStater' r; r r r r� rp r! r! r" r� � s:
�
�
�
z0HTMLTokenizer.beforeDoctypePublicIdentifierStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d
| jd < | j�| j� | j| _nR|t kr�| j�td d d�� d
| jd < | j�| j� | j| _n| jd |7 < d
S )Nr� r_ r# r` r$ r� r2 rt r� Fr� r� T�
r r: �!afterDoctypePublicIdentifierStater r' r; r r r r rp r! r! r" r� � s0
� �
�
z6HTMLTokenizer.doctypePublicIdentifierDoubleQuotedStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d
| jd < | j�| j� | j| _nR|t kr�| j�td d d�� d
| jd < | j�| j� | j| _n| jd |7 < d
S )Nr� r_ r# r` r$ r� r2 rt r� Fr� r� Tr� rp r! r! r" r� � s0
� �
�
z6HTMLTokenizer.doctypePublicIdentifierSingleQuotedStatec C s | j �� }|tkr | j| _n�|dkr<| j�| j� | j| _n�|dkrn| j�t d dd�� d| jd< | j
| _n�|dkr�| j�t d dd�� d| jd< | j | _nh|t kr�| j�t d d d�� d
| jd < | j�| j� | j| _n(| j�t d dd�� d
| jd < | j
| _d S )
Nrt r� r# r� r$ r/ r� r� r� Fr� T)r r: r �-betweenDoctypePublicAndSystemIdentifiersStater r' r; r r r �(doctypeSystemIdentifierDoubleQuotedState�(doctypeSystemIdentifierSingleQuotedStater r� rp r! r! r" r� � s>
�
�
�
�
z/HTMLTokenizer.afterDoctypePublicIdentifierStatec C s� | j �� }|tkrn�|dkr4| j�| j� | j| _n�|dkrPd| jd< | j| _n�|dkrld| jd< | j | _nh|t
kr�| j�t d dd�� d | jd
< | j�| j� | j| _n(| j�t d d d�� d | jd
< | j | _d S )
Nrt r� r/ r� r� r# r� r$ Fr� r� T)
r r: r r' r; r r r r� r� r r r� rp r! r! r" r� s2
�
�
z;HTMLTokenizer.betweenDoctypePublicAndSystemIdentifiersStatec C s� | j �� }|tkr | j| _n�|dkrP| j�td dd�� | j �|� | j| _nT|t kr�| j�td dd�� d| j
d< | j�| j
� | j | _n| j �|� | j| _dS r� ) r r: r �"beforeDoctypeSystemIdentifierStater r' r; r rA r r r rp r! r! r" r� s&
�
�
z,HTMLTokenizer.afterDoctypeSystemKeywordStatec C s� | j �� }|tkrn�|dkr0d| jd< | j| _n�|dkrLd| jd< | j| _n�|dkr�| j�t d dd�� d | jd
< | j�| j� | j
| _nh|t kr�| j�t d d d�� d | jd
< | j�| j� | j
| _n(| j�t d dd�� d | jd
< | j | _d S )
Nr� r/ r� r� rt r# r� r$ Fr� r� T)
r r: r r r� r r� r' r; r r r r� rp r! r! r" r� / s:
�
�
�
z0HTMLTokenizer.beforeDoctypeSystemIdentifierStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d
| jd < | j�| j� | j| _nR|t kr�| j�td d d�� d
| jd < | j�| j� | j| _n| jd |7 < d
S )Nr� r_ r# r` r$ r� r2 rt r� Fr� r� T�
r r: �!afterDoctypeSystemIdentifierStater r' r; r r r r rp r! r! r" r� L s0
� �
�
z6HTMLTokenizer.doctypeSystemIdentifierDoubleQuotedStatec C s� | j �� }|dkr | j| _n�|dkrN| j�td dd�� | jd d7 < n�|dkr�| j�td d d�� d
| jd < | j�| j� | j| _nR|t kr�| j�td d d�� d
| jd < | j�| j� | j| _n| jd |7 < d
S )Nr� r_ r# r` r$ r� r2 rt r� Fr� r� Tr� rp r! r! r" r� d s0
� �
�
z6HTMLTokenizer.doctypeSystemIdentifierSingleQuotedStatec C s� | j �� }|tkrn~|dkr4| j�| j� | j| _n^|tkrt| j�t d dd�� d| jd< | j�| j� | j| _n | j�t d dd�� | j
| _dS ) Nrt r# r� r$ Fr� r� T) r r: r r' r; r r r r r r� rp r! r! r" r� | s$
�
�z/HTMLTokenizer.afterDoctypeSystemIdentifierStatec C sZ | j �� }|dkr*| j�| j� | j| _n,|tkrV| j �|� | j�| j� | j| _n dS )Nrt T) r r: r' r; r r r r rA rp r! r! r" r� � s
z HTMLTokenizer.bogusDoctypeStatec C s� g }|� | j�d�� |� | j�d�� | j�� }|tkr>q�q|dksJt�|d dd � dkrv|d d d� |d< q�q|� |� qd�|�}|�d�}|dkr�t|�D ]}| j � t
d d
d �� q�|� dd �}|r�| j � t
d
|d �� | j | _
dS )N�]rt rJ �����z]]r/ r_ r r# r` r$ r2 rO T)r; r rd r: r �AssertionErrorr= �count�ranger' r r� r r )r r& r: Z nullCountr� r! r! r" r� � s2
�
�z HTMLTokenizer.cdataSectionState)N)NF)N�__name__�
__module__� __qualname__�__doc__r r, rF rW rX r^ r rb rj rh rm ro rq rc rw rx ri r� r� rl r� r� rn r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r| r� r� r� r� r� r� r� r} ry rv r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� �
__classcell__r! r! r r" r s� H
P #
6 "- 3 r N)Z
__future__r r r Zpip._vendor.sixr r? � collectionsr Z constantsr r r
r r r
r r r r Z _inputstreamr Z_trier rP �objectr r! r! r! r" � s