Kotlin & Metaprogramming: Using Automata for Smarter UI Testing
Using State Machines to map complex relations between UI Nodes and Functions
In an epic exploration of Kotlin, the next frontier in modern metaprogramming, we have TornadoFX-Suite, a small project that analyzes uploaded TornadoFX code which in turn detects relevant UI elements and writes UI tests.
TornadoFX-Suite works — barely. If a user looked up documentation and followed along with the instructions, then sure, TornadoFX-Suite works.
If it barely works, then it doesn’t work, does it
Sigh…. yeahhhhhh. It is my hopes that in the next year, this can turn into something more serious than a research project with lots of data points and growth in framework configurations. Code analysis, which is achieved through Abstract Syntax Tree parsing, will become stronger as TornadoFX-Suite keeps scanning more projects. This project has made significant strides in research with promising discoveries intended on furthering Kotlin programming, and I’ve been really happy with the progress.
Let’s take a look at what we got now
The UI hierarchal breakdown seen in the previous image is code analysis for a this basic TornadoFX project:
In code analysis, we pull out the control nodes we expect users may interact with (TextFields, Buttons, Forms, etc) and write tests for those controls with less-than-elegant string concatenation, scripting and dynamic ID attachment at runtime.
Here’s some of how the first generated-testing looks:
So far, we have individual interactions for every detected control detected.
Yep. These are tests alright. But they’re not very useful tests. They may be interacting with individual nodes, but our code analysis is not yet smart enough to consider useful permutations of interactions nor what to expect out of our interactions. We can’t just blindly create a permutation of interactions without knowing what to expect.
Hot Dog or Not a Hot Dog: Making smarter tests
In a previous article, I talk about how we can use metaprogramming to write UI tests, but the reality is, these tests are not very smart. Our FxRobots can click on buttons and write in textfields — but in the Editor fragment, we have models tied to both our form and influencing the behavior of our form. In this example, the save button in the Editor dialog is not enabled until the model has been dirtied (the contents of the textfields are changed).
Metaprogramming isn’t self-learning or smart; but that doesn’t mean that we can’t formalize a system for tested views in a way that metaprogramming can formalize these concepts into logic that can be generated for better testing!
One benefit of formalizing specifications for a UI system is that formal ideas and clear structures leads us to being able to create a system metaprogramming can work with. Another more explicit benefit of formal specifications allows us to check a UI Node’s properties; are any transitions ill-specified? Is it possible to get locked out of interactions?
This is where finite state machines (FSM) comes into play. In an FSM, there is have a finite set of states, a set of transitions between states, and a set of events which triggers transitions. Each transition is tied to an event, so if the event happens, you may or may not change your state.
There’s quite a bit more that goes into automata, but we can think of this set of interacting states kind of similar to the elegant mechanics of the famous Tinker Toy computer built in 1978 in a deterministic sense, in which an automation cannot be in more than one state at a time.
Creating our own abstraction of finite automata may involve nondeterministic controls, where the automation may be several states at once, which can allow us to “program” solutions using a higher-level language.
Before we can figure out how metaprogramming might create an FSM from any class we test, let’s start with analyzing what a
Ground Rules
We will need to define our participants as well as the interactions among the participants.
We wish to edit some information within a view table for our cat-sitter schedule. Upon clicking a client’s information, an editor pops up with the nodes, which stores information like its location, properties, and functions associated with the node; namely, the functions written in methods that may affect the state of the node.
The state of the node is dependent on whether its properties changes. This information can be represented in a directed graph, which preserves the hierarchy of the nodes.
We are concerned with only a few controls: three textfields and a button which a user may interact with. However, our button is only enabled when the model is dirty. This means that asides from our node hierarchy representing the UI, we have a model that is attached to the contents of the form. Our participants for this form, then, is:
- The user interacting with the form
- The model attached to our
textfields - Our
button, which is dependent on the model’s state
In our modeling, we can take the user out of our theoretical state machine — the user is will be our robots, interacting through permutations of interactions, looking for dead states — states in which interactions lock causing our application to crash. Our robot is meant to be a bit of a “wild card” for creating inputs of testing validation, but using an FSM can help narrow down some of those automations into slightly more-useful test cases.
We must, in a way, create our own tinker toy machine that holds those dependencies together, as an abstracted version of our application. In order to create a specified FSM for a view, we must ask our metaprogramming for the following information:
- What are the available sets of states for relevant control nodes?
- What is the starting state of our control nodes?
- What functions affect our control nodes? These functions will act as transitions in our finite state machine.
- What are the possible interactions, or inputs, available for our states?
- What view models are tied to certain data controls? This is responsible for weaving our system together.
But how would we use these items in a state machine via metaprogramming if we had this information? Let us further our Editor fragment and come up with a psuedo-code Kotlin FSM that might help illuminate how FSM can help write smarter tests.
A Kotlin FSM Psuedo-Implementation
With any complicated problem, it is imperative to draw out the problem first before trying to code it.
Our model is tied to our textfields, so anytime the contents of any textfield has been altered, then our model is dirtied, and we may expect our button to be enabled.
The lines represent the transition that causes the state to change or change back, which comes from the user interaction; our “wildcard” robots.
Kotlin as a language actually has allowed for significantly easier implementation of our FSM, compared to implementations done in Java or even a simplified Sketch.Systems creating a hierarchal chart.
Implementing a set of states and their transistors
Our set of states can be managed by Kotlin enum classes:
enum class Toggle {
Enable, Disable;
fun isButtonEnabled(): Boolean {
return when (this) {
Enable -> true
Disable -> false
}
}
fun enableOnDirty(model: ModelState): Toggle {
return when (model) {
ModelState.Dirty -> Enable
ModelState.NotDirty -> Disable
}
}
}
enum class ModelState {
Dirty, NotDirty;
fun initModelStateTransition(): ModelState {
return when (this) {
Dirty -> NotDirty
NotDirty -> Dirty
}
}
}I like this in particular because we can use our states in our enums as self-documenting code, and the states and the transitions themselves are not intertwined with the context of the interactors themselves.
Implementing our protocol machine and their starting states
The behavior of the finite state machines themselves are not necessarily tied to our interactors, but there is no need to sweat over that — again, our robots will be serving as a way to interaction with our detected UI nodes.
Kotlin seems to allow the ability to separate these concerns into clear boundaries without affecting the functionality of the FSM we wish to implement, thanks to sealed classes. The sealed classes represents our node hierarchy of relevant controls. The top sealed class is tied to our model and our individual controls listen for an indication the textfield, which is tied to the model, has changed.
You’ll notice that we only have listeners for inputs in order to create our state interdependencies. By no means is this a final version of state machines I wish to use, and it’s always worth exploring more examples of UI state machines in regards to testing — but information has come to light from our initial explorations?
What can we do with this information and how can we tie it to writing tests?
Given that our metaprogramming has given us:
- our UI controls
- models/scopes tied to a TornadoFX view
- knowledge of what functions affect certain control properties with mapping
- a composite sketch of where those affected functions go stored in a hierarchy of a directed graph
We can actually do a few things here: the function mapping and composite sketch may give us an idea of what actions can be expected from certain nodes. If the interaction of a control is found to affect the property of another control in the mapping, then the first can act as an interactor participant while the expected state change in the other node is a state we have located worth keeping track of.
More importantly, it may be possible for us to take the necessary information analyzed and generated and use Kotlin scripting to create and spurn off our own modified DFA at runtime, and record the results of the expected states to help us write smarter tests.
All that work to just get to a couple charts! However, these charts can help us guess expected behavior and write tests going from this:
@Test
fun testButton() {
clickOn(textfield).write("Something")
clickOn(button)
}To tests with actual, educated guesses on expectations:
@Test
fun testButton() {
// FSM transition table tells us the model is not dirty
assertTrue(button.isDisabled)
clickOn(textfield).write("Something") // FSM transition table tells us the model is dirty
assertTrue(!button.isDisabled)
clickOn(button)
}
That’s it for now. There’s a whole lot of information out there on finite state machines and plenty of distinction between the different types. While there is a heavily academic flavor to this blurb, the main idea was to practice formalizing a system just enough to try out a pragmatic version it real life. I’m not so worried about getting the semantics of the use case or the verbiage, although I’m happy for corrections as needed.
The project can be checked out here:
As always, the project continues to evolve every day.
If you’re interested in reading up more on finite state machines, I’ve compiled a list of resources and notes on Github. Until next time!
