Metaprogramming, Machine Learning, and Automated Testing: Back to the drawing board
Hello friends — this is an update on a project I started a over a year ago when I asked “what is good UI testing?”. Problem was, I had no idea how to write good tests and a lot answers I’ve received were broad in range and technologies. I thought, maybe I can use metaprogramming to help me answer that. I also thought that once I start writing tests, perhaps I can use analytics and data science to help me find trends that might suggest what DOES make good testing. Since then, I’ve made a lot of personal progress on my own technical growth as I continue this wild goose chase into the unknown. Well actually, I kind of hit a wall.
The idea is to create a testing suite in which you can upload your project, it scans your code, detect UI controls, and write tests for your views. I’ve made some solid progress on TornadoFX-Suite: a project that is at the moment, in shambles. I write to reorganize my thoughts and efforts about this project — like many projects that are exploratory, it is easy to go down the wrong path (or a path that is not quite ready).
I spent 2 months this year attempting to write a full AST parser for Kotlin using Kastree to no avail. I started writing individual cases for parsing elements in Kotlin and the more I wrote, the more “special cases” I found. Once I got to observable properties, I just kind of
And if you look at my project, what you see is a code representation of the above reaction. It’s a complete shit show, to be frank.
Why am I rage-quitting on AST parsing for compositional breakdown?
I should clarify — I’m rage-quitting for the time being. I would love to create a complete compositional breakdown for Kotlin code with an AST parser, but I found myself writing cases the properties of Kotlin collections each as a special case. I have started to see a pattern in some of these collections, such as Hashmaps and ArrayLists:
Unfortunately, I’ve had little luck finding a comprehensive list of types I will encounter, and after weeks of digging through Kotlin source code, I’m afraid that I simply don’t have the resources or the time to create a full AST parser past the Kotlin compiler wrapping labeled as Nodes types with Kastree. The goal of this project is to create automated testing, and writing an AST parser is simply a means for being able to create automated testing. I need something quicker than this poke-and-prod strategy — although my intention is to come back one day and finish this side project. I think there’s a lot to be said about patterns in Kotlin types described by Kastree Nodes.
AST parsing for UI detection is a go. AST parsing for compositional breakdown is currently a distraction. I need a new game plan. Let’s start with what I’ve learned so far.
Rule of Three — never create “abstraction layers” before you have three independent cases
Honestly, this is why my project is 3 months behind. A while back, I wrote a blurb on how metaprogramming might be able to use an abstracted representation of UI at runtime to test whether it is functional or not. This idea brought me down the wrong road, and frankly, this solution is not useful for developers. Hence, this idea is not useful for my question — what make a good UI test?
This project is not ready for this level of thinking. I need a concrete idea, and I need machine learning to give me concrete results. Let’s revisit my last check point. TornadoFX-Suite scans the project, detects UI inputs, and generates test stubs for those UI items.
I have successfully used AST parsing in TornadoFX to detect the UI controls we care about. I’ve successfully used Kotlin scripting to write my test stubs. I now need to find a way to create a basic breakdown of my classes as well as the methods and members in them. I went down one path writing a full AST parser with Kastree only for it to become real tedious. I’m not ready to throw it away just yet, but now Kotlin 1.3 is out, I’m looking into scripting capabilities for more than just writing the tests — another strategy could be to load the files I read and compile them at run time, then use reflection in Kotlin to grab the members and methods I need. For anyone that might be interested in Kotlin scripting and reflection, a Kotlin Slack user by the name of NikkyAI wrote a wonderful template project for using BasicJvmScriptingHost.
This solution also sounds terrible, at least in terms of performance and processing. This entire project is just full of bad ideas. Still, it will be easier to think of better solutions once there’s a working one. So here are my tangible goals of 3 working independent cases I must aim for:
- Create a basic composite breakdown of the classes I analyze. I don’t care how I get there.
- Use Tensorflow’s Java API to start collecting and analyzing data to questions such as “what tests pass?”, “what tests fails?”, “what is the nature of the failing tests?”, “how complicated are the functions that I try to test?”.
- Solidify and generalize my basic groundwork for detecting UI controls. What I have is works, but it needs to act as an independent module from the project. Same goes for compositional breakdown and the future machine learning portion.
In the meantime
I’m also putting myself through a crash course on data science and machine learning. I’ve compiled a list of (free) intensive learning material with the guidance of data scientists/colleagues/friends in the Kotlin community as well as coworkers who are QA automation engineers. This has been intensive and not very enjoyable, but it is important. Since much of data science and machine learning is done and taught in Python, I’ve taken the time to relearn python and work with hands-on projects analyzing data sets with scripting. If you’re interested in my unqualified and probably unpopular opinion of research models and statistics, you can find it here.