Recently I’ve got a question:

How do I get pass the dynamic “load more” button using a Python web scraper?

Often we encounter dynamic web content with dynamic web elements like the following url.

Obviously (looking at the page-source) one can see the site is driven by JS React library. Python scraper does not make website to run its on-site JavaScript.
Moreover, when one presses “load more”, the HTTP POST request is generated by site’s JS with


One picture is better than thousand words:
Is it possible to make a similar request to spoof target server? No it’s not. See the Same Origin Policy here.

The solution is to drive a browser to perform all the actions.

Selenium comes to rescue

See the code with the FF browser driven by Selenium in Python:

Alternative ways to scroll down to the btn, provided btn is defined in the driver scope:

If the element is not present in the driver’s scope, then there comes Expected Condition:

If an element is a ‘stale’, not operating, seems the site’s JS restricts the number of usages of that element. My suggestion would be to reload the page and make another search (with other input data) and try over that element. If JS limits number of clicks on the button (usages) – we can’t do much…