|
Server : Apache System : Linux server.mata-lashes.com 3.10.0-1160.90.1.el7.x86_64 #1 SMP Thu May 4 15:21:22 UTC 2023 x86_64 User : matalashes ( 1004) PHP Version : 8.1.29 Disable Function : NONE Directory : /usr/local/lib/python3.6/site-packages/charset_normalizer/__pycache__/ |
Upload File : |
3
G�Kd\$ � @ s, y d dl ZW n ek
r( d dlZY nX d dlZd dlZd dlmZ d dlmZ d dl m
Z
d dl m Z d dl
mZmZmZmZmZ d dlmZ dd lmZmZmZmZmZmZ e
ed
�e e d �d d
��Z e
ed
�e e d �dd��Z e
ed
�e ee d �dd��Z e
ed
�e e d �dd��Z!e e d �dd�Z"e
ed
�e e d �dd��Z#e
ed
�e e d �dd��Z$e
ed
�e e d �dd��Z%e
ed
�e e d �d d ��Z&e
ed
�e e d �d d ��Z'e e d �d d!�Z(e
ed
�e e d �d"d#��Z)e
ed
�e e d �d$d%��Z*e
ed
�e e d �d&d'��Z+e
ed
�e e d �d(d)��Z,e
ed
�e e d �d*d+��Z-e
e.e�d
�e e d,�d-d.��Z/dNe0e1ee d0�d1d2�Z2e
d3d
�e e d4�d5d6��Z3e0eee e0f d7�d8d9�Z4e e d:�d;d<�Z5dOe e e d>�d?d@�Z6e ee dA�dBdC�Z7e e e8dD�dEdF�Z9e e e dD�dGdH�Z:dIej;dJfe e1e ddK�dLdM�Z<dS )P� N)�IncrementalDecoder)�aliases)� lru_cache)�findall)�List�Optional�Set�Tuple�Union)�MultibyteIncrementalDecoder� )�ENCODING_MARKS�IANA_SUPPORTED_SIMILAR� RE_POSSIBLE_ENCODING_INDICATION�UNICODE_RANGES_COMBINED� UNICODE_SECONDARY_RANGE_KEYWORD�UTF8_MAXIMAL_ALLOCATION)�maxsize)� character�returnc C sT yt j| �}W n tk
r" dS X d|kpRd|kpRd|kpRd|kpRd|kpRd|kS )NFz
WITH GRAVEz
WITH ACUTEz WITH CEDILLAzWITH DIAERESISzWITH CIRCUMFLEXz
WITH TILDE)� unicodedata�name�
ValueError)r � description� r �F/tmp/pip-build-8nxjc3nm/charset-normalizer/charset_normalizer/utils.py�is_accentuated s r c C s. t j| �}|s| S |jd�}tt|d d��S )N� r � )r �
decomposition�split�chr�int)r Z
decomposed�codesr r r �
remove_accent* s
r$ c C s. t | �}x tj� D ]\}}||kr|S qW dS )zK
Retrieve the Unicode range official name from a single character.
N)�ordr �items)r Z
character_ord�
range_nameZ ord_ranger r r �
unicode_range5 s
r( c C s, yt j| �}W n tk
r" dS X d|kS )NFZLATIN)r r r )r r r r r �is_latinC s
r) c C s( y| j d� W n tk
r" dS X dS )N�asciiFT)�encode�UnicodeEncodeError)r r r r �is_asciiL s
r- c C s2 t j| �}d|krdS t| �}|d kr*dS d|kS )N�PTFZ Punctuation)r �categoryr( )r �character_category�character_ranger r r �is_punctuationT s
r2 c C s: t j| �}d|ksd|kr dS t| �}|d kr2dS d|kS )N�S�NTFZForms)r r/ r( )r r0 r1 r r r � is_symbolc s
r5 c C s t | �}|d krdS d|kS )NFZ Emoticons)r( )r r1 r r r � is_emoticonr s r6 c C s& | j � s| d krdS tj| �}d|kS )
N� ｜�+�,�;�<�>T�Z> r9 r; r7 r8 r< r: )�isspacer r/ )r r0 r r r � is_separator| s
r? c C s | j � | j� kS )N)�islower�isupper)r r r r �is_case_variable� s rB c C s t j| �}|dkS )N�Co)r r/ )r r0 r r r �is_private_use_only� s
rD c C s, yt j| �}W n tk
r" dS X d|kS )NF�CJK)r r r )r �character_namer r r �is_cjk� s
rG c C s, yt j| �}W n tk
r" dS X d|kS )NFZHIRAGANA)r r r )r rF r r r � is_hiragana� s
rH c C s, yt j| �}W n tk
r" dS X d|kS )NFZKATAKANA)r r r )r rF r r r � is_katakana� s
rI c C s, yt j| �}W n tk
r" dS X d|kS )NFZHANGUL)r r r )r rF r r r � is_hangul� s
rJ c C s, yt j| �}W n tk
r" dS X d|kS )NFZTHAI)r r r )r rF r r r �is_thai� s
rK )r' r c s t � fdd�tD ��S )Nc 3 s | ]}|� kV qd S )Nr )�.0�keyword)r' r r � <genexpr>� s z-is_unicode_range_secondary.<locals>.<genexpr>)�anyr )r' r )r' r �is_unicode_range_secondary� s rP � )�sequence� search_zoner c C s� t | t�st�t| �}tt| dt||�� jddd��}t|�dkrHdS xJ|D ]B}|j� j dd�}x,t
j � D ] \}}||kr�|S ||krl|S qlW qNW dS )zW
Extract using ASCII-only decoder any specified encoding in the first n-bytes.
Nr* �ignore)�errorsr �-�_) �
isinstance�bytes� TypeError�lenr r �min�decode�lower�replacer r& )rR rS Zseq_len�resultsZspecified_encoding�encoding_alias�
encoding_ianar r r �any_specified_encoding� s
rc � )r r c
C s | d kp t tjd
j| ��jt�S ) zQ
Verify is a specific encoding is a multi byte one based on it IANA name
�utf_8� utf_8_sig�utf_16� utf_16_be� utf_16_le�utf_32� utf_32_le� utf_32_be�utf_7z encodings.{}> re rj ri rf rk rh rl rm rg )�
issubclass� importlib�
import_module�formatr r )r r r r �is_multi_byte_encoding� s rr )rR r c C sJ xDt D ]<}t | }t|t�r"|g}x |D ]}| j|�r(||fS q(W qW dS )z9
Identify and extract SIG/BOM in given sequence.
N� )Nrs )r
rX rY �
startswith)rR �
iana_encodingZmarks�markr r r �identify_sig_or_bom� s
rw )ru r c C s | dkS )Nrg rj > rj rg r )ru r r r �should_strip_sig_or_bom s rx T)�cp_name�strictr c C sL | j � jdd�} x$tj� D ]\}}| ||gkr|S qW |rHtdj| ���| S )NrV rW z Unable to retrieve IANA for '{}')r^ r_ r r& r rq )ry rz ra rb r r r � iana_name s r{ )�decoded_sequencer c C s8 t � }x(| D ] }t|�}|d kr"q |j|� q W t|�S )N)�setr( �add�list)r| �rangesr r1 r r r �
range_scan s
r� )� iana_name_a� iana_name_br c C s� t | �st |�rdS tjdj| ��j}tjdj|��j}|dd�}|dd�}d}x6td�D ]*}t|g�}|j|�|j|�krZ|d7 }qZW |d S ) Ng z encodings.{}rT )rU r � r � )rr ro rp rq r �rangerY r] ) r� r� Z decoder_aZ decoder_bZid_aZid_bZcharacter_match_count�iZ
to_be_decodedr r r �
cp_similarity+ s
r� c C s | t ko|t | kS )z�
Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using
the function cp_similarity.
)r )r� r� r r r �
is_cp_similar@ s r� Zcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s)r �level�
format_stringr c C s: t j| �}|j|� t j� }|jt j|�� |j|� d S )N)�logging� getLogger�setLevel�
StreamHandler� setFormatter� Formatter�
addHandler)r r� r� �logger�handlerr r r �set_logging_handlerK s
r� )rQ )T)=Z unicodedata2r � ImportErrorro r� �codecsr Zencodings.aliasesr � functoolsr �rer Ztypingr r r r r
Z_multibytecodecr Zconstantr
r r r r r �str�boolr r$ r( r) r- r2 r5 r6 r? rB rD rG rH rI rJ rK r[ rP rY r" rc rr rw rx r{ r� �floatr� r� �INFOr� r r r r �<module> sr