Shadow DOM in Selenium

November 22, 2021

Shadow DOM in Selenium

Shadow DOM in Selenium

I’ve seen numerous bugs reported for how Chrome v96 has changed the shadow root return values for Selenium. This is a feature, not a bug! Here’s how to work with the shadow DOM in Selenium 3 and 4.

TL/DR

To access Shadow DOM elements in Selenium 4 with Chromium browsers (Microsoft Edge and Google Chrome) version 96 or greater, use the new shadow root method:

WebElement shadowHost = driver.findElement(By.cssSelector("#shadow_host"));
SearchContext shadowRoot = shadowHost.getShadowRoot();
WebElement shadowContent = shadowRoot.findElement(By.cssSelector("#shadow_content"));

See full example on GitHub

shadow_host = @driver.find_element(css: '#shadow_host')
shadow_root = shadow_host.shadow_root
shadow_content = shadow_root.find_element(css: '#shadow_content')

See full example on GitHub

shadow_host = driver.find_element(By.CSS_SELECTOR, '#shadow_host')
shadow_root = shadow_host.shadow_root
shadow_content = shadow_root.find_element(By.CSS_SELECTOR, '#shadow_content')

See full example on GitHub

var shadowHost = _driver.FindElement(By.CssSelector("#shadow_host"));
var shadowRoot = shadowHost.GetShadowRoot();
var shadowContent = shadowRoot.FindElement(By.CssSelector("#shadow_content"));

See full example on GitHub

What happened in v96 is that Chromium has made its shadow root values compliant with the updated W3C WebDriver specification, which now includes definitions getting an element’s shadow root and locating elements in a shadow root. Microsoft Edge will support this functionality in v96, and we can expect Firefox to add support soon. Additionally, the Selenium team is working to get support added to WebKit, so eventually we’ll see this in Safari.

The way people are used to accessing shadow DOM elements is with JavaScript. If you are already working with shadow roots in Chrome, Edge or Safari, most likely your code looks like this:

WebElement shadowHost = driver.findElement(By.cssSelector("#shadow_host"));
JavascriptExecutor jsDriver = (JavascriptExecutor) driver;

WebElement shadowRoot = (WebElement) jsDriver.executeScript("return arguments[0].shadowRoot", shadowHost);
WebElement shadowContent = shadowRoot.findElement(By.cssSelector("#shadow_content"));

See full example on GitHub

shadow_host = @driver.find_element(css: '#shadow_host')
shadow_root = @driver.execute_script('return arguments[0].shadowRoot', shadow_host)
shadow_content = shadow_root.find_element(css: '#shadow_content')

See full example on GitHub

shadow_host = driver.find_element_by_css_selector('#shadow_host')
shadow_root = driver.execute_script('return arguments[0].shadowRoot', shadow_host)
shadow_content = shadow_root.find_element_by_css_selector('#shadow_content')

See full example on GitHub

var shadowHost = _driver.FindElement(By.CssSelector("#shadow_host"));
var js = ((IJavaScriptExecutor)_driver);

var shadowRoot = (IWebElement)js.ExecuteScript("return arguments[0].shadowRoot", shadowHost);
var shadowContent = shadowRoot.FindElement(By.CssSelector("#shadow_content"));

See full example on GitHub

That code works for Chromium browsers before v96 and Safari in both Selenium 3 and Selenium 4.

What’s happening is that Selenium looks at return values from script execution commands and if it detects an element, it automatically converts it. With Chromium v96, the JavaScript returns something that is not identifiable as an element. There are two issues with this— conversion and casting. Only Java implemented a conversion to a shadow root object in 4.0, so the other languages were broken until 4.1. The second issue is casting, which only applies to the strictly typed Java and .NET. Here’s what we see in each language:

In Java, the return value is a Map, which will not cast to WebElement, so you’ll get this error in either Selenium 3 or Selenium 4:

java.lang.ClassCastException: class com.google.common.collect.Maps$TransformedEntriesMap cannot be cast to
class org.openqa.selenium.WebElement (com.google.common.collect.Maps$TransformedEntriesMap and
org.openqa.selenium.WebElement are in unnamed module of loader 'app')

In Selenium 3 and Selenium 4.0, you’ll get an error like:
```
NoMethodError: undefined method `find_element' for #<Hash:0x00007fdc69997800>
```
Because Ruby doesn’t have strict typing, the code works correctly in Selenium 4.1.
In Selenium 3 and 4.0, Python gives this error:
```
AttributeError: 'dict' object has no attribute 'find_element_by_css_selector'
```
In Selenium 4.1, the code would work, except a ShadowRoot only uses the newest find_element() methods, so you need to update to use the new By class.
```
AttributeError: 'ShadowRoot' object has no attribute 'find_element_by_css_selector'
```

In Selenium 3, and 4.0, you get this error:

System.InvalidCastException: Unable to cast object of type 'System.Collections.Generic.Dictionary`2[System.String,System.Object]' to type 'OpenQA.Selenium.IWebElement'.

In Selenium 4.1, you’ll see this error:

System.InvalidCastException: Unable to cast object of type 'OpenQA.Selenium.ShadowRoot' to type 'OpenQA.Selenium.IWebElement'.

If you’re using Selenium 4, the quick fix for the casting errors is to use the correct interface. This is not the recommended approach, but it has the advantage of being backward compatible with older versions of Chromium and working in other browsers (e.g., Safari):

java
c#

```
SearchContext shadowRoot = (SearchContext) jsDriver.executeScript("return arguments[0].shadowRoot", shadowHost);
```
Note that the ShadowRoot class itself is kept as package-private because we want people to code against the SearchContext API directly. This change is backwards compatible with what you’ve been doing, since WebElement extends SearchContext.

See full example on GitHub
```
var shadowRoot = (ISearchContext)js.ExecuteScript("return arguments[0].shadowRoot", shadowHost);
```
This change is backwards compatible with what you’ve been doing, since IWebElement is a subclass of ISearchContext.

See full example on GitHub

The especially tricky issue here is how to deal with a shadow root in Chromium v96 while still using Selenium 3. If you are using Selenium 3, seriously, please update to Selenium 4. Issues like this highlight why it is worth investing time to upgrade. You can still get a WebElement using JavaScript in Chromium v96 using Selenium 3, it’s just really hacky:

WebElement shadow_host = driver.findElement(By.cssSelector("#shadow_host"));

Object shadowRoot = ((JavascriptExecutor) driver).executeScript("return arguments[0].shadowRoot", shadow_host);
String id = (String) ((Map<String, Object>) shadowRoot).get("shadow-6066-11e4-a52e-4f735466cecf");
RemoteWebElement shadowRootElement = new RemoteWebElement();
shadowRootElement.setParent((RemoteWebDriver) driver);
shadowRootElement.setId(id);

WebElement shadowContent = shadowRootElement.findElement(By.cssSelector("#shadow_content"));

See full example on GitHub

shadow_host = @driver.find_element(css: '#shadow_host')

shadow_root_hash = @driver.execute_script('return arguments[0].shadowRoot', shadow_host)
shadow_root_id = shadow_root_hash['shadow-6066-11e4-a52e-4f735466cecf']
shadow_root = Selenium::WebDriver::Element.new(@driver.send(:bridge), shadow_root_id)

shadow_content = shadow_root.find_element(css: '#shadow_content')

See full example on GitHub

shadow_host = driver.find_element_by_css_selector('#shadow_host')

shadow_root_dict = driver.execute_script('return arguments[0].shadowRoot', shadow_host)
shadow_root_id = shadow_root_dict['shadow-6066-11e4-a52e-4f735466cecf']
shadow_root = WebElement(driver, shadow_root_id, w3c=True)

shadow_content = shadow_root.find_element_by_css_selector('#shadow_content')

See full example on GitHub

var shadowHost = _driver.FindElement(By.CssSelector("#shadow_host"));
var js = ((IJavaScriptExecutor)_driver);

var shadowRoot = (Dictionary<string, object>)js.ExecuteScript("return arguments[0].shadowRoot", shadowHost);
var id = (string)shadowRoot["shadow-6066-11e4-a52e-4f735466cecf"];
var shadowRootElement = new RemoteWebElement((RemoteWebDriver)_driver, id);

var shadowContent = shadowRootElement.FindElement(By.CssSelector("#shadow_content"));

See full example on GitHub

So far, I haven’t mentioned Firefox— that’s because it is special. Until Firefox implements W3C-compliant shadow root support (presumably coming very soon), you have to get the shadow DOM elements from the executed script directly with the children property, then loop through the elements to find the one you want to work with:

WebElement shadowHost = driver.findElement(By.cssSelector("#shadow_host"));
JavascriptExecutor jsDriver = (JavascriptExecutor) driver;

List<WebElement> children = (List<WebElement>) jsDriver.executeScript("return arguments[0].shadowRoot.children", shadowHost);

WebElement shadowContent = null;
for (WebElement element : children) {
    if (element.getAttribute("id").equals("shadow_content")) {
        shadowContent = element;
        break;
    }
}

See full example on GitHub

  shadow_host = @driver.find_element(css: '#shadow_host')
children = @driver.execute_script('return arguments[0].shadowRoot.children', shadow_host)

shadow_content = children.first { |child| child.attribute('id') == 'shadow_content' }

See full example on GitHub

shadow_host = driver.find_element(By.CSS_SELECTOR, '#shadow_host')
children = driver.execute_script('return arguments[0].shadowRoot.children', shadow_host)

shadow_content = next(child for child in children if child.get_attribute('id') == 'shadow_content')

See full example on GitHub

var shadowHost = _driver.FindElement(By.CssSelector("#shadow_host"));
var js = ((IJavaScriptExecutor)_driver);

var children = (IEnumerable<IWebElement>)js.ExecuteScript("return arguments[0].shadowRoot.children", shadowHost);

IWebElement shadowContent = null;
foreach (IWebElement element in children) {
    if (element.GetAttribute("id").Equals("shadow_content")) {
        shadowContent = element;
        break;
    }
}

See full example on GitHub

Here’s a complete working example of how you should be working with shadow DOM elements in Selenium 4.1+

JUnit 5 Test:

@Test
public void recommendedCode() {
    WebDriverManager.chromedriver().setup();
    driver = new ChromeDriver();

    driver.get("http://watir.com/examples/shadow_dom.html");

    WebElement shadowHost = driver.findElement(By.cssSelector("#shadow_host"));
    SearchContext shadowRoot = shadowHost.getShadowRoot();
    WebElement shadowContent = shadowRoot.findElement(By.cssSelector("#shadow_content"));

    Assertions.assertEquals("some text", shadowContent.getText());
}

See full example on GitHub

RSpec Test:

it 'recommended code' do
  @driver = Selenium::WebDriver.for :chrome

  @driver.get('http://watir.com/examples/shadow_dom.html')

  shadow_host = @driver.find_element(css: '#shadow_host')
  shadow_root = shadow_host.shadow_root
  shadow_content = shadow_root.find_element(css: '#shadow_content')

  expect(shadow_content.text).to eq 'some text'
end

See full example on GitHub

PyTest test:

def test_recommended_code():
    driver = Chrome()

    driver.get('http://watir.com/examples/shadow_dom.html')

    shadow_host = driver.find_element(By.CSS_SELECTOR, '#shadow_host')
    shadow_root = shadow_host.shadow_root
    shadow_content = shadow_root.find_element(By.CSS_SELECTOR, '#shadow_content')

    assert shadow_content.text == 'some text'

    driver.quit()

See full example on GitHub

MS Test:

[TestMethod]
public void RecommendedCode()
{
    new DriverManager().SetUpDriver(new ChromeConfig());
    _driver = new ChromeDriver();

    _driver.Navigate().GoToUrl("http://watir.com/examples/shadow_dom.html");

    var shadowHost = _driver.FindElement(By.CssSelector("#shadow_host"));
    var shadowRoot = shadowHost.GetShadowRoot();
    var shadowContent = shadowRoot.FindElement(By.CssSelector("#shadow_content"));

    Assert.AreEqual("some text", shadowContent.Text);
}

See full example on GitHub

Selenium 4 with Chromium 96 provides a much cleaner API for working with Shadow DOM elements without needing to use JavaScript. Please update your code to take advantage of this.

Follow me if you found this article interesting,
or answer one of these questions in the comments or on Twitter: