Parser used for parsing HTTP response and inspecting parsed result to find links or detect blocking.
Hierarchy
- AbstractHttpParser
Index
Methods
find_links
- find_links(parsed_content, selector, attribute): Iterable[str]
Find all links in result using selector.
Parameters
parsed_content: TParseResult
Parsed HTTP response. Result of
parsemethod.selector: str
String used to define matching pattern for finding links.
attribute: str
Which node attribute to extract the links from.
Returns Iterable[str]
Iterable of strings that contain found links.
is_blocked
- is_blocked(parsed_content): BlockedInfo
Detect if blocked and return BlockedInfo with additional information.
Default implementation that expects
is_matching_selectorabstract method to be implemented. Override this method if your parser has different way of blockage detection.Parameters
parsed_content: TParseResult
Parsed HTTP response. Result of
parsemethod.
Returns BlockedInfo
BlockedInfoobject that contains non-empty string description of reason if blockage was detected. Empty string in reason signifies no blockage detected.
is_matching_selector
- is_matching_selector(parsed_content, selector): bool
Find if selector has match in parsed content.
Parameters
parsed_content: TParseResult
Parsed HTTP response. Result of
parsemethod.selector: str
String used to define matching pattern.
Returns bool
True if selector has match in parsed content.
parse
- async parse(response): TParseResult
Parse HTTP response.
Parameters
response: HttpResponse
HTTP response to be parsed.
Returns TParseResult
Parsed HTTP response.
parse_text
- async parse_text(text): TParseResult
Parse text containing html.
Parameters
text: str
String containing html.
Returns TParseResult
Parsed text.
select
- async select(parsed_content, selector): Sequence[TSelectResult]
Use css selector to select page element and return it.
Parameters
parsed_content: TParseResult
Content where the page element will be located.
selector: str
Css selector used to locate desired html element.
Returns Sequence[TSelectResult]
Selected element.