ï¿½ ï¿½ï¿½^iï¿½0ï¿½ï¿½ï¿½dZddlZddlZddlZddlmZddlmZmZm Z ddl mZddlm Z ddlmZdd lmZGd ï¿½deï¿½ï¿½ZdS)a Module containing the UniversalDetector detector class, which is the primary class a user of ``chardet`` should use. :author: Mark Pilgrim (initial port to Python) :author: Shy Shalom (original C code) :author: Dan Blanchard (major refactoring for 3.0) :author: Ian Cordasco ï¿½Nï¿½)ï¿½CharSetGroupProber)ï¿½ InputStateï¿½LanguageFilterï¿½ProbingState)ï¿½EscCharSetProber)ï¿½Latin1Prober)ï¿½MBCSGroupProber)ï¿½SBCSGroupProberc ï¿½ï¿½eZdZdZdZejdï¿½ï¿½Zejdï¿½ï¿½Zejdï¿½ï¿½Z dddd d ddd dï¿½Z ejfdï¿½Z dï¿½Zdï¿½Zdï¿½ZdS)ï¿½UniversalDetectoraq The ``UniversalDetector`` class underlies the ``chardet.detect`` function and coordinates all of the different charset probers. To get a ``dict`` containing an encoding and its confidence, you can simply run: .. code:: u = UniversalDetector() u.feed(some_bytes) u.close() detected = u.result gï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½?s[ï¿½-ï¿½]s(|~{)s[ï¿½-ï¿½]zWindows-1252zWindows-1250zWindows-1251zWindows-1256zWindows-1253zWindows-1255zWindows-1254zWindows-1257)z iso-8859-1z iso-8859-2z iso-8859-5z iso-8859-6z iso-8859-7z iso-8859-8z iso-8859-9ziso-8859-13cï¿½ï¿½ï¿½d|_g|_d|_d|_d|_d|_d|_||_tj tï¿½ï¿½|_d|_|ï¿½ ï¿½ï¿½dS)N)ï¿½_esc_charset_proberï¿½_charset_probersï¿½resultï¿½doneï¿½ _got_dataï¿½_input_stateï¿½ _last_charï¿½lang_filterï¿½loggingï¿½ getLoggerï¿½__name__ï¿½loggerï¿½_has_win_bytesï¿½reset)ï¿½selfrs ï¿½ï¿½/builddir/build/BUILDROOT/alt-python311-pip-21.3.1-4.el8.x86_64/opt/alt/python311/lib/python3.11/site-packages/pip/_vendor/chardet/universaldetector.pyï¿½__init__zUniversalDetector.__init__Qsiï¿½ï¿½#'ï¿½ï¿½ ï¿½ "ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ ï¿½ï¿½ï¿½ï¿½ ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½&ï¿½ï¿½ï¿½ï¿½'ï¿½ï¿½1ï¿½1ï¿½ï¿½ï¿½"ï¿½ï¿½ï¿½ï¿½ ï¿½ ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½cï¿½ï¿½ï¿½ddddï¿½|_d|_d|_d|_tj|_d|_|jr|jï¿½ ï¿½ï¿½|j D]}|ï¿½ ï¿½ï¿½ï¿½dS)zï¿½ Reset the UniversalDetector and all of its probers back to their initial states. This is called by ``__init__``, so you only need to call this directly in between analyses of different documents. Nï¿½ï¿½ï¿½encodingï¿½ confidenceï¿½languageFr )rrrrrï¿½ PURE_ASCIIrrrrr)rï¿½probers rrzUniversalDetector.reset^sï¿½ï¿½ï¿½$(ï¿½sï¿½ï¿½Mï¿½Mï¿½ï¿½ï¿½ï¿½ï¿½ ï¿½ï¿½ï¿½ï¿½#ï¿½ï¿½ï¿½&ï¿½1ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½#ï¿½ -ï¿½ï¿½$ï¿½*ï¿½*ï¿½,ï¿½,ï¿½,ï¿½ï¿½+ï¿½ ï¿½ ï¿½Fï¿½ï¿½Lï¿½Lï¿½Nï¿½Nï¿½Nï¿½Nï¿½ ï¿½ r cï¿½ï¿½|jrdSt|ï¿½ï¿½sdSt|tï¿½ï¿½st|ï¿½ï¿½}|jsï¿½|ï¿½tjï¿½ï¿½rddddï¿½|_nï¿½|ï¿½tj tj fï¿½ï¿½rddddï¿½|_nx|ï¿½dï¿½ï¿½rddddï¿½|_nW|ï¿½d ï¿½ï¿½rd dddï¿½|_n6|ï¿½tjtjfï¿½ï¿½rddddï¿½|_d|_|jd ï¿½ d|_dS|j tjkrt|jï¿½|ï¿½ï¿½rtj|_ nH|j tjkr3|jï¿½|j|zï¿½ï¿½rtj|_ |ddï¿½|_|j tjkrï¿½|jst/|jï¿½ï¿½|_|jï¿½|ï¿½ï¿½t4jkr?|jj|jï¿½ï¿½ï¿½|jjdï¿½|_d|_dSdS|j tjkï¿½r|jsztA|jï¿½ï¿½g|_|jtBj"zr&|jï¿½#tIï¿½ï¿½ï¿½ï¿½|jï¿½#tKï¿½ï¿½ï¿½ï¿½|jD]U}|ï¿½|ï¿½ï¿½t4jkr0|j|ï¿½ï¿½ï¿½|jdï¿½|_d|_nï¿½V|j&ï¿½|ï¿½ï¿½rd|_'dSdSdS)aï¿½ Takes a chunk of a document and feeds it through all of the relevant charset probers. After calling ``feed``, you can check the value of the ``done`` attribute to see if you need to continue feeding the ``UniversalDetector`` more data, or if it has made a prediction (in the ``result`` attribute). .. note:: You should always call ``close`` when you're done feeding in your document if ``done`` is not already ``True``. Nz UTF-8-SIGï¿½ï¿½?ï¿½r#zUTF-32sï¿½ï¿½zX-ISO-10646-UCS-4-3412sï¿½ï¿½zX-ISO-10646-UCS-4-2143zUTF-16Tr$ï¿½ï¿½ï¿½ï¿½ï¿½)(rï¿½lenï¿½ isinstanceï¿½ bytearrayrï¿½ startswithï¿½codecsï¿½BOM_UTF8rï¿½BOM_UTF32_LEï¿½BOM_UTF32_BEï¿½BOM_LEï¿½BOM_BErrr'ï¿½HIGH_BYTE_DETECTORï¿½searchï¿½ HIGH_BYTEï¿½ESC_DETECTORrï¿½ ESC_ASCIIrrrï¿½feedrï¿½FOUND_ITï¿½charset_nameï¿½get_confidencer&rr rï¿½NON_CJKï¿½appendrr ï¿½WIN_BYTE_DETECTORr)rï¿½byte_strr(s rr<zUniversalDetector.feedosï¿½ï¿½ï¿½ï¿½9ï¿½ ï¿½ï¿½Fï¿½ï¿½8ï¿½}ï¿½}ï¿½ ï¿½ï¿½Fï¿½ï¿½(ï¿½Iï¿½.ï¿½.ï¿½ +ï¿½ ï¿½ï¿½*ï¿½*ï¿½Hï¿½ï¿½~ï¿½" ï¿½ï¿½"ï¿½"ï¿½6ï¿½?ï¿½3ï¿½3ï¿½ /ï¿½+6ï¿½-0ï¿½+-ï¿½/ï¿½/ï¿½ï¿½ï¿½ï¿½ï¿½$ï¿½$ï¿½fï¿½&9ï¿½&,ï¿½&9ï¿½&;ï¿½<ï¿½<ï¿½ /ï¿½,4ï¿½-0ï¿½+-ï¿½/ï¿½/ï¿½ï¿½ï¿½ï¿½ï¿½$ï¿½$ï¿½%8ï¿½9ï¿½9ï¿½ /ï¿½+Cï¿½-0ï¿½+-ï¿½/ï¿½/ï¿½ï¿½ï¿½ï¿½ï¿½$ï¿½$ï¿½%8ï¿½9ï¿½9ï¿½ /ï¿½+Cï¿½-0ï¿½+-ï¿½/ï¿½/ï¿½ï¿½ï¿½ï¿½ï¿½$ï¿½$ï¿½fï¿½mï¿½Vï¿½]ï¿½%Cï¿½Dï¿½Dï¿½ /ï¿½,4ï¿½-0ï¿½+-ï¿½/ï¿½/ï¿½ï¿½ï¿½"ï¿½Dï¿½Nï¿½ï¿½{ï¿½:ï¿½&ï¿½2ï¿½ ï¿½ï¿½ ï¿½ï¿½ï¿½ï¿½ï¿½ ï¿½ 5ï¿½5ï¿½5ï¿½ï¿½&ï¿½-ï¿½-ï¿½hï¿½7ï¿½7ï¿½ 9ï¿½$.ï¿½$8ï¿½ï¿½!ï¿½!ï¿½ï¿½"ï¿½jï¿½&;ï¿½;ï¿½;ï¿½ï¿½%ï¿½,ï¿½,ï¿½Tï¿½_ï¿½xï¿½-Gï¿½Hï¿½Hï¿½<ï¿½$.ï¿½$8ï¿½ï¿½!ï¿½"ï¿½2ï¿½3ï¿½3ï¿½-ï¿½ï¿½ï¿½ï¿½ï¿½ ï¿½ 4ï¿½4ï¿½4ï¿½ï¿½+ï¿½ Nï¿½+;ï¿½Dï¿½ï¿½lowerr0rï¿½ISO_WIN_MAPï¿½getr&ï¿½getEffectiveLevelrï¿½DEBUGr.rï¿½probers) rï¿½prober_confidenceï¿½max_prober_confidenceï¿½ max_proberr(r>ï¿½lower_charset_namer%ï¿½group_probers rï¿½closezUniversalDetector.closeï¿½sï¿½ï¿½ï¿½ï¿½9ï¿½ ï¿½ï¿½;ï¿½ï¿½ï¿½ï¿½ ï¿½ï¿½~ï¿½! @ï¿½ï¿½Kï¿½ï¿½ï¿½1ï¿½2ï¿½2ï¿½2ï¿½2ï¿½ï¿½ ï¿½*ï¿½"7ï¿½ 7ï¿½ 7ï¿½'.ï¿½),ï¿½')ï¿½+ï¿½+ï¿½Dï¿½Kï¿½Kï¿½ ï¿½ ï¿½*ï¿½"6ï¿½ 6ï¿½ 6ï¿½ $ï¿½ï¿½$'ï¿½!ï¿½ï¿½Jï¿½ï¿½/ï¿½ (ï¿½ (ï¿½ï¿½ï¿½ï¿½ï¿½$*ï¿½$9ï¿½$9ï¿½$;ï¿½$;ï¿½!ï¿½$ï¿½'<ï¿½<ï¿½<ï¿½,=ï¿½)ï¿½!'ï¿½Jï¿½ï¿½ï¿½ @ï¿½4ï¿½tï¿½7Mï¿½Mï¿½Mï¿½)ï¿½6ï¿½ï¿½%/ï¿½%<ï¿½%Bï¿½%Bï¿½%Dï¿½%Dï¿½"ï¿½'ï¿½6ï¿½6ï¿½8ï¿½8ï¿½ ï¿½&ï¿½0ï¿½0ï¿½ï¿½<ï¿½<ï¿½Jï¿½ï¿½*ï¿½Jï¿½'+ï¿½'7ï¿½';ï¿½';ï¿½ï¿½@ï¿½@ï¿½ï¿½ï¿½ ï¿½;ï¿½(ï¿½(ï¿½*ï¿½*ï¿½gï¿½mï¿½;ï¿½;ï¿½ï¿½{ï¿½:ï¿½&ï¿½.ï¿½ï¿½ï¿½!ï¿½!ï¿½"Dï¿½Eï¿½Eï¿½Eï¿½$(ï¿½$9ï¿½ Iï¿½ Iï¿½Lï¿½'ï¿½!ï¿½ ï¿½!ï¿½,ï¿½0Bï¿½Cï¿½Cï¿½ Iï¿½&2ï¿½&:ï¿½Gï¿½Gï¿½Fï¿½ ï¿½Kï¿½-ï¿½-ï¿½.Eï¿½.4ï¿½.Aï¿½.4ï¿½oï¿½.4ï¿½.Cï¿½.Cï¿½.Eï¿½.Eï¿½Gï¿½Gï¿½Gï¿½Gï¿½Gï¿½ï¿½ï¿½)ï¿½)ï¿½*Aï¿½*6ï¿½*Cï¿½*6ï¿½*?ï¿½*6ï¿½*Eï¿½*Eï¿½*Gï¿½*Gï¿½Iï¿½Iï¿½Iï¿½Iï¿½ï¿½{ï¿½r N)rï¿½ __module__ï¿½__qualname__ï¿½__doc__rGï¿½reï¿½compiler7r:rBrIrï¿½ALLrrr<rSï¿½r rr r 3sï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ ï¿½ï¿½#ï¿½ï¿½ï¿½Nï¿½3ï¿½3ï¿½ï¿½ï¿½2ï¿½:ï¿½lï¿½+ï¿½+ï¿½Lï¿½"ï¿½ï¿½ ï¿½>ï¿½2ï¿½2ï¿½ï¿½!/ï¿½!/ï¿½!/ï¿½!/ï¿½!/ï¿½!/ï¿½!/ï¿½"0ï¿½2ï¿½2ï¿½Kï¿½$2ï¿½#5ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½"k+ï¿½k+ï¿½k+ï¿½ZBï¿½Bï¿½Bï¿½Bï¿½Br r )rVr1rrWï¿½charsetgroupproberrï¿½enumsrrrï¿½ escproberrï¿½latin1proberr ï¿½mbcsgroupproberr ï¿½sbcsgroupproberrï¿½objectr rZr rï¿½rbsï¿½ï¿½ï¿½8ï¿½ï¿½ï¿½ ï¿½ ï¿½ ï¿½ï¿½ï¿½ï¿½ï¿½ ï¿½ ï¿½ ï¿½ ï¿½2ï¿½2ï¿½2ï¿½2ï¿½2ï¿½2ï¿½;ï¿½;ï¿½;ï¿½;ï¿½;ï¿½;ï¿½;ï¿½;ï¿½;ï¿½;ï¿½'ï¿½'ï¿½'ï¿½'ï¿½'ï¿½'ï¿½&ï¿½&ï¿½&ï¿½&ï¿½&ï¿½&ï¿½,ï¿½,ï¿½,ï¿½,ï¿½,ï¿½,ï¿½,ï¿½,ï¿½,ï¿½,ï¿½,ï¿½,ï¿½kï¿½kï¿½kï¿½kï¿½kï¿½ï¿½kï¿½kï¿½kï¿½kï¿½kr