Monkey (headless) testing with {shinytest2}
Most people will use {shinytest2} with the plug and play
record_test() app, which is very convenient if you are not
familiar with JavaScript. Under the hood, record_test()
generates an R script composed of a series of directed instructions that
manipulates the app to automate testing on CI/CD environments.
Monkey testing, a type of testing where random inputs are
used to test the behavior of an app, is widely used by web developers to
check application robustness, particularly in apps with a large number
of inputs. The goal is ultimately to try to break the app by triggering
unexpected combinations. Most available libraries are JS-based such as
gremlins.js,
traditionally combined with JS-based global testing libraries like Puppeteer, but can work with
{shinytest2} as well.
In this vignette we’ll provide a more thorough overview of the
AppDriver R6 class (extending on the concepts covered in
the Testing in depth article), which allows
the developer to programmatically control the app. We’ll see how we can
seamlessly benefit from gremlins.js with only few lines of code.
Initialize the driver
We consider a simple app composed of a slider and a plot output:
ui <- fluidPage(
sliderInput("obs", "Number of observations:",
min = 0, max = 1000, value = 500
),
plotOutput("distPlot")
)
# Server logic
server <- function(input, output) {
output$distPlot <- renderPlot({
hist(rnorm(input$obs))
})
}
# Complete app with UI and server components
shinyApp(ui, server)The driver may be initialized with:
headless_app <- AppDriver$new(
app_dir = "<PATH_TO_APP>",
name = "monkey-test",
shiny_args = list(port = 3515)
)Note the shiny_args slot allowing you to pass custom
options to shiny::runApp() such as the port, which might be
useful if your organization restricts port number.
load_timeout defaults to 10s and 20s locally and during
CI/CD, respectively. Therefore, if your app takes longer to launch, you
can change this value. Keep in mind that an app taking more than 20s to
launch is generally under-optimized and would require specific care such
as profiling and refactoring.
AppDriver starts a Chrome-based headless browser. If you need
specific flags
that are not available by default in {shinytest2}, you can
pass them before instantiating the driver:
chromote::set_chrome_args(
c(
chromote::default_chrome_args(),
# Custom flags: see https://peter.sh/experiments/chromium-command-line-switches/
)
)Some flags are considered by default, particularly
--no-sandbox, which is applied only on CI/CD, as Chrome
won’t start without it.
If you run this script locally, you may add view = TRUE
to open the Chrome Devtools, which will significantly ease the testing
calibration. I highly recommend creating the test protocol locally and
then moving to CI/CD later when all bugs are fixed.
In the below figure, the application is shown on the left side panel. The top-right side panel shows the DOM elements (default) and the bottom-right side panel displays the JavaScript console output.
Injecting gremlins.js
The next steps consist of injecting the gremlins.js dependency in the DOM so that we can unleash the horde.
Easy way
The easiest way to inject gremlins.js is to call:
headless_app$run_js("
let s = document.createElement('script');
s.src = 'https://unpkg.com/gremlins.js';
document.body.appendChild(s);
")This creates a <script> tag pointing to the
correct Content Delivery Network (CDN), an optimized server to store
libraries, and inserts it at the end of the body.
To test whether everything worked well, we can dump the DOM and look
for the scripts. We can find gremlins.js by calling
typeof window.gremlins, which returns an object:
You may instead see undefined returned. This is
generally because the JS code is blocked by the network. If this is the
case, consider injecting gremlins.js locally, as explained in the next
section.
Local way
You can store and serve a local copy of the gremlins.js script with
shiny::addResourcePath(), assuming gremlins.js
is in inst/js/gremlins.min.js. You can accomplish this by
adding the following code to the app.R file:
We can subsequently inject the gremlins in the DOM and check whether everything worked as expected:
Unleash the horde
A bit about gremlins.js
The workflow is rather simple:
- We create the horde with
gremlins.createHorde(). - We run the monkey test with
horde.unleash();.
createHorde() accepts many species of gremlins capable
of handling various events such as clicks, touch, form filling,
scrolling, typing, and more, as described in the gremlins.js documentation.
We don’t recommend using the scroller, which sometimes
crashes
the Chrome instance.
If your plots rely on random elements, such as rnorm, it
is best practice to set up a seed using set.seed(). By
default, all species will attack in random order with a delay of 10 ms
between each event. You can also control the attack strategy to fine
tune the global behavior. If you want more control over what your
gremlins species should be doing, you can define a custom species.