}
# Match single node by CSS selector:
print(repr(tree.document.query_selector('body > main p:last-child')))
# >>>
# Match multiple nodes by CSS selector:
print(repr(tree.body.query_selector_all('main *')))
# >>> {
, , , }
# Check whether there is any element matching this CSS selector:
print(tree.body.matches('.bar'))
# >>> True
:class:`.DOMCollection` objects are iterable, indexable, and slicable. The size of a collection can be checked with ``len()``. If a slice is requested, the returned object will be another :class:`.DOMCollection`:
.. code-block:: python
coll = tree.body.query_selector_all('main *')
# First element
print(repr(coll[0]))
# >>>
# Last element
print(repr(coll[-1]))
# >>>
# First two elements
print(repr(coll[:2]))
# >>> {, }
:class:`.DOMCollection` objects have the same DOM methods for selecting objects as :class:`.DOMNode` objects. This can be used for efficiently matching elements in the subtree(s) of the previously selected elements. The selection methods behave just like their :class:`.DOMNode` counterparts and return either a single :class:`.DOMNode` or another :class:`.DOMCollection`:
.. code-block:: python
coll = tree.body.get_elements_by_class_name('dom')
# Only matches within the subtrees of elements in `coll`:
print(repr(coll.get_elements_by_class_name('bar')))
# >>> {}
.. _parse-html-attributes:
Attributes
^^^^^^^^^^
Attributes of element nodes can be accessed either via :meth:`.DOMNode.getattr` or by dict-like access:
.. code-block:: python
meta = tree.head.query_selector('meta[charset]')
if meta is not None:
print(meta.getattr('charset'))
# >>> utf-8
# Or:
print(meta['charset'])
# >>> utf-8
The dict access method will raise a :exc:`KeyError` exception if the attribute does not exist.
The ``id`` and ``class`` attributes of an element are also available through the :attr:`~.DOMNode.id` and :attr:`~.DOMNode.class_name` or :attr:`~.DOMNode.class_list` properties:
.. code-block:: python
p = tree.body.get_element_by_id('b')
print(p.id)
# >>> b
span = p.query_selector('span')
print(span.class_name)
# >>> bar baz
print(span.class_list)
# >>> ['bar', 'baz']
A list of existing attributes on an element is provided by its :attr:`~.DOMNode.attrs` property:
.. code-block:: python
a = tree.create_element('div')
a.id = 'a-id'
a.class_name = 'a-class'
a['href'] = 'https://example.com'
print(a.attrs)
# >>> ['id', 'class', 'href']
.. _parse-html-text-serialization:
HTML and Text Serialization
^^^^^^^^^^^^^^^^^^^^^^^^^^^
All :class:`.DOMNode` objects have a :attr:`~.DOMNode.text` and :attr:`~.DOMNode.html` property for accessing their plaintext or HTML serialization:
.. code-block:: python
print(tree.body.get_element_by_id('a').text)
# >>> Hello world!
print(tree.body.get_element_by_id('a').html)
# >>> Hello world!
Alternatively, you can also simply cast a :class:`.DOMNode` to ``str``, which is equivalent to :attr:`.DOMNode.html`:
.. code-block:: python
print(tree.body.get_element_by_id('a'))
# >>> Hello world!
For extracting specifically the text contents of the document's ```` element, there is also the :attr:`.HTMLTree.title` property:
.. code-block:: python
# Example page
print(tree.title)
.. _parse-html-traversal:
DOM Tree Traversal
------------------
The DOM subtree of any node can be traversed in pre-order by iterating over a :class:`.DOMNode` instance. Different types of nodes can be distinguished by their :attr:`~.DOMNode.type` property.
.. code-block:: python
from resiliparse.parse.html import NodeType
root = tree.body.get_element_by_id('a')
tag_names = [e.tag for e in root]
tag_names_elements_only = [e.tag for e in root if e.type == NodeType.ELEMENT]
print(tag_names)
# >>> ['p', '#text', 'span', '#text', '#text']
print(tag_names_elements_only)
# >>> ['p', 'span']
To iterate only the immediate children of a node, loop over its :attr:`~.DOMNode.child_nodes` property instead of the node itself:
.. code-block:: python
for e in tree.body.get_element_by_id('foo').child_nodes:
if e.type == NodeType.ELEMENT:
print(e.text)
Output:
::
Hello DOM!
Hello world!
In addition, any :class:`.DOMNode` object also has the following properties:
* :attr:`~.DOMNode.first_child`
* :attr:`~.DOMNode.last_child`
* :attr:`~.DOMNode.prev`
* :attr:`~.DOMNode.next`
* :attr:`~.DOMNode.parent`
These can be used for traversing in a custom order or with custom logic, though the callback-based traversal mechanism described in the next section :ref:`parse-html-traversal-advanced` is usually more convenient.
.. _parse-html-traversal-advanced:
Advanced Traversal
^^^^^^^^^^^^^^^^^^
Besides the simple iterable interface, Resiliparse also supports a more advanced and flexible callback-based method for traversing the DOM tree. The :func:`.traverse_dom` helper function accepts a :class:`.DOMNode` and a callback function that is called for each individual DOM node with a :class:`.DOMContext` context object as its only parameter.
The following example prints the values of all text nodes:
.. code-block:: python
from resiliparse.parse.html import *
def start_cb(ctx: DOMContext):
if ctx.node.type == NodeType.TEXT and ctx.node.value.strip():
print(ctx.node.value.strip(), end=' ')
traverse_dom(tree.body, start_cb)
Output:
::
Hello world ! Hello DOM !
In addition to the start element callback, you can also specify an end element callback that will be invoked every time the DOM tree traverser encounters and element's end tag (i.e., every time the tree traverser goes up one node level):
The following example prints all start and end tags without their textual contents or attributes:
.. code-block:: python
def start_cb(ctx: DOMContext):
if ctx.node.type == NodeType.ELEMENT:
print(f'<{ctx.node.tag}>', end='')
def end_cb(ctx: DOMContext):
if ctx.node.type == NodeType.ELEMENT:
print(f'</{ctx.node.tag}>', end='')
traverse_dom(tree.body, start_cb, end_cb)
Output:
.. code-block:: html
<body><main><p><span></span></p><p><span></span></p></main></body>
Besides a reference to the current node, the context objects also keeps track of the traversal depth, so the following is possible:
.. code-block:: python
def start_cb(ctx: DOMContext):
if ctx.node.type == NodeType.ELEMENT:
print(f'{" " * ctx.depth}<{ctx.node.tag}>')
if ctx.node.type == NodeType.TEXT and ctx.node.value.strip():
print(f'{" " * ctx.depth}{ctx.node.text.strip()}')
def end_cb(ctx: DOMContext):
if ctx.node.type == NodeType.ELEMENT:
print(f'{" " * ctx.depth}</{ctx.node.tag}>')
traverse_dom(tree.body, start_cb, end_cb)
Output:
.. code-block:: html
<body>
<main>
<p>
Hello
<span>
world
</span>
!
</p>
<p>
Hello
<span>
DOM
</span>
!
</p>
</main>
</body>
The context object is the same object throughout the whole traversal process, so besides the node ``node`` and ``depth`` attributes, it can be mutated arbitrarily to maintain your own state. If you need to, you can also pass a pre-initialized context object to :func:`.traverse_dom`. The following example converts the HTML body into nested Python lists:
.. code-block:: python
def start_cb(ctx: DOMContext):
if ctx.node.type == NodeType.ELEMENT:
t = (ctx.node.tag, [])
ctx.list_stack_current[-1].append(t)
ctx.list_stack_current.append(t[1])
elif ctx.node.type == NodeType.TEXT:
txt = ctx.node.value.strip()
if txt:
ctx.list_stack_current[-1].append(txt)
def end_cb(ctx: DOMContext):
if ctx.node.type == NodeType.ELEMENT:
ctx.list_stack_current.pop()
ctx = DOMContext()
ctx.list_stack = []
ctx.list_stack_current = [ctx.list_stack]
traverse_dom(tree.body, start_cb, end_cb, ctx)
print(ctx.list_stack)
Output:
.. code-block:: python
[('body', [('main', [('p', ['Hello', ('span', ['world']), '!']), ('p', ['Hello', ('span', ['DOM']), '!'])])])]
.. _parse-html-manipulate:
DOM Tree Manipulation
---------------------
Resiliparse supports DOM manipulation and the creation of new nodes with a basic set of well-known DOM functions.
.. warning::
A :class:`.DOMNode` object is valid only for as long as its parent tree has not been modified or deallocated. Thus, **DO NOT** use existing instances after any sort of DOM tree manipulation! Doing so may result in Python crashes or (worse) security vulnerabilities due to dangling pointers (*use after free*). This is a `known Lexbor limitation <https://github.com/lexbor/lexbor/issues/132>`__ for which there is no workaround at the moment.
Elements
^^^^^^^^
In the following is an example of how you can create new DOM elements and text nodes and insert them into the tree:
.. code-block:: python
# Create a new <div> element node
new_element = tree.create_element('p')
# Create a new text node
new_text = tree.create_text_node('Hello Resiliparse!')
# Insert nodes into DOM tree
main_element = tree.body.query_selector('main')
main_element.append_child(new_element)
new_element.append_child(new_text)
print(main_element)
Output:
.. code-block:: html
<main id="foo">
<p id="a">Hello <span class="bar">world</span>!</p>
<p id="b" class="dom">Hello <span class="bar baz">DOM</span>!</p>
<p>Hello Resiliparse!</p></main>
In addition to :meth:`~.DOMNode.append_child`, nodes also provide :meth:`~.DOMNode.insert_before` for inserting a child node before another child instead of appending it at the end, and :meth:`~.DOMNode.replace_child` for replacing an existing child node in the tree with another.
Use :meth:`~.DOMNode.remove_child` to remove a node from the tree:
.. code-block:: python
main_element.remove_child(new_element)
To fully delete a node, use :meth:`~.DOMNode.decompose()` on the node itself. This will remove it from the tree (if not already done) and delete the node and its entire subtree recursively:
.. code-block:: python
new_element.decompose()
# From here on, this element and all elements in its subtree are invalid!!!
Attributes
^^^^^^^^^^
Attributes can be added or modified via :meth:`~.DOMNode.setattr` or by assigning directly to its dict entry:
.. code-block:: python
element = tree.create_element('img')
element['src'] = 'https://example.com/foo.png'
element.setattr('alt', 'Foo')
print(element)
# >>> <img src="https://example.com/foo.png" alt="Foo">
For ``id`` and ``class`` attributes, you can also use :attr:`~.DOMNode.id` and :attr:`~.DOMNode.class_name` or :attr:`~.DOMNode.class_list`:
.. code-block:: python
element = tree.create_element('div')
element.id = 'my-id'
element.class_name = 'class-a'
element.class_list.add('class-b')
print(element)
# >>> <div id="my-id" class="class-a class-b"></div>
print(element.class_list)
# >>> ['class-a', 'class-b']
element.class_list.remove('class-a')
print(element)
# >>> <div id="my-id" class="class-b"></div>
Inner HTML and Inner Text
^^^^^^^^^^^^^^^^^^^^^^^^^
An easier, but less efficient way of manipulating the DOM is to assign a string directly to either its :attr:`~.DOMNode.html` or :attr:`~.DOMNode.text` property. This will replace the inner HTML or inner text of these nodes with the new value:
.. code-block:: python
main_element.html = '<p>New inner HTML content</p>'
print(main_element)
# >>> <main id="foo"><p>New HTML content</p></main>
main_element.text = '<p>New inner text content</p>'
print(main_element)
# >>> <main id="foo"><p>New inner text content</p></main>
.. _parse-html-benchmark:
Benchmarks
----------
The :ref:`resiliparse-cli` parser comes with a small HTML parser benchmarking tool that can measure the parsing engine's performance and compare it to other Python HTML parsing libraries. Supported third-party libraries are `Selectolax <https://github.com/rushter/selectolax>`__ (both the old MyHTML and the new Lexbor engine) and `BeautifulSoup4 <https://www.crummy.com/software/BeautifulSoup/bs4/doc/>`__ (lxml engine only, which is the fastest BS4 backend).
Here are the results of extracting the titles from all web pages in an uncompressed 42,015-document WARC file on a Ryzen Threadripper 2920X machine:
.. code-block:: console
$ resiliparse html benchmark CC-MAIN-*.warc
HTML parser benchmark <title> extraction:
=========================================
Resiliparse (Lexbor): 42015 documents in 36.55s (1149.56 documents/s)
Selectolax (Lexbor): 42015 documents in 37.46s (1121.52 documents/s)
Selectolax (MyHTML): 42015 documents in 53.82s (780.72 documents/s)
BeautifulSoup4 (lxml): 42015 documents in 874.40s (48.05 documents/s)
Not surprisingly, the two parsers based on the Lexbor engine perform almost identically, whereas lxml is by far the slowest by a factor of 24x.