Just another post of TestNG bag of tricks

In this post I’ll talk about using integration tests and how to leverage them in testing big batch of input. In my case it was a big batch of documents that are set out for scanning.

Photo by Annie Spratt on Unsplash

It evolves around an mature technology, that had a fair share of problems running on newer build frameworks. However having been able to run it, I was happy 👏. Hopefully this will be useful for future-me 👴 and anyone else trying to solve similar problem.

So for anyone interested, as I’ve mentioned, it’ll involve a sample of

• Handling big (quantity wise) data input in integration tests
• ..using TestNG testing framework
• ..with AssertJ assertions framework
• ..through Gradle build system
• ..built with Kotlin

Disclosure #1. I honestly didn’t expect for the post to be so big. By reading from top to bottom you probably would get a rough idea how everything evolved better and its easier to trace the end result. But certainly you can jump through topics as well, as it is just a set of tips and tricks. All in all, just in case, I’m providing a table of content. Believe me, I was not expecting on this Sunday to write a blog post that requires a TOC 🤦

1. Problem — case I was solving
2. Solution — overall how to leverage integration tests
3. Regular tests — How to assemble TestNG and run on Gradle build system
4. Reports — How to enrich reports with screenshots
5. Grouping — Grouping tests with more verticals
6. Scaling 1 — Test parametrize
7. Group assertions — Using SoftAssert
8. Scaling 2 — Smarter use of parametrization 2
9. Scaling 3 — Smarter use of parametrization 3

Disclosure #2. I’m quite sure, there won’t be a lot of people having to solve these kind of challenges, however this is similar what QA automation people would be facing and 🤞, that may come in handy as a design solution, a sample template how to start or just in general give an idea what to build on their own.

Demo of generated reports

As always, if you like to get to the chase — sample repository with all the cases I’ve been taking about.

Problem

Right. So here I go.

We are developing a system that would include document inspection. To put it simply — looking for concrete values in PDF / images. To put it more simply — document 👉 values.

As we already have a (kind of) solution that is working for us, now we need to know if it’ll work in the real world. That means, we have to run it on a bunch of documents to see what kind of results we are getting. Also to know if by changing anything we are making it better or worse. Depending on how big the input we have, the bigger confidence we have in our solution. To put it short, book case scenario for integration tests.

The challenge is quite trivial in itself and the solution is straightforward.

Now the real question is how 🔧.

Solution

Here are goals, what I’m trying to achieve. The tools I’ll be using, should (at least partially) check these ’check-boxes’

• Handle a big input base
• Generate results in report
• Generate results as statistics in various ways
• Report should also have some indication what went wrong
• Test should be easily maintainable. (As we have so much input, having to change tests 500 times is not an option.)

That is quite a list to achieve 📝. Might be, that regular tests may not cut it.

However that is where we start for now.

Iteration #1. Regular tests

Testable class

Just before we begin anything, we need a class that we will run tests on. As you have probably guessed I can’t disclose real mechanism we are using due to privacy reasons. And that anyway would distract the topic of testing solutions I’m proposing in this topic. So for sanity reasons we will be testing this complex class 🤓.

/**
 * Imagined document processor
 */
class DocumentProcessor {

    /**
     * Processes document and outputs read value in it
     * @param imagePath document path in file system
     * @return document read value or empty
     */
    fun readDocument(imagePath: String): String {
        return "1234.5"
    }
}

Our class will be located in {rootProject}/app/src/main/kotlin/lt/markmerkk/processor/DocumentProcessor.kt

Testing class

Ok, on to testing now.

Enable TestNG framework in Gradle. It’s actually quite simple.

1. Include library dependency.

We will include in {rootProject}/app/build.gradle.kts

testImplementation("org.testng:testng:7.4.0")
testImplementation("org.testng:reportng:1.2.2")

I’m including report library as well, however it’s optional. Everything I’ll be presenting, does not need it. Here’s more info anyway.

2. Configure Gradle to use TestNG instead of regular JUnit

• Include at the end of file {rootProject}/app/build.gradle.kts
• Commented out lines defines properties to generate reports using ReportNG. So ignore this, unless you want to try customize your reports.
• Our reports will be generated in {rootProject}/app/test-reports1 directory

tasks.withType(Test::class) {
    ignoreFailures = true
    useTestNG {
        // systemProperties.put("org.uncommons.reportng.stylesheet", "${projectDir}/resources/hudsonesque.css")
        // systemProperties.put("org.uncommons.reportng.escape-output", false)
        // systemProperties.put("org.uncommons.reportng.frames", true)
        outputDirectory = File(project.projectDir, "/test-reports1")
        useDefaultListeners = true
        // options {
        //     listeners.add("org.uncommons.reportng.HTMLReporter")
        //     listeners.add("org.uncommons.reportng.JUnitXMLReporter")
        // }
    }
    testLogging.showStandardStreams = true
    testLogging.exceptionFormat = org.gradle.api.tasks.testing.logging.TestExceptionFormat.FULL
}

3. Write initial test

• We are creating our initial test in {rootProject}/app/src/test/lt.markmerkk.processor/DocumentProcessorTest.kt

class DocumentProcessorTest {
    @Test
    fun valid() {
        // Assemble
        val docProcessor = DocumentProcessor()
        val imageFile = TestUtils.projectResAsFile("/image1.png")

        // Act
        val result = docProcessor.readDocument(imageFile)

        // Assert
        assertThat(imageFile.exists()).isTrue()
        assertThat(result).isEqualTo("111.0")
    }
}

• Good thing to mention, make sure you’re using TestNG imports. Otherwise you may be using JUnit framework or something similar.

import org.testng.annotations.Test

Simple, right ? We run it, we get output.

Reports

1. Console output

java.lang.AssertionError:
Expecting:
 <"1234.5">
to be equal to:
 <"111.0">
but was not.

     <...>

===============================================
Default Suite
Total tests run: 1, Passes: 0, Failures: 1, Skips: 0
===============================================

2. HTML

• Html reports can be found in {rootProject}/app/test-reports1/index.html

Troubleshooting

1. Using Gradle

Make sure you’re running tests through Gradle.

• To be sure to use gradle, use this command to run tests (while being in {rootProject}/).

./gradlew :app:test

• If you’re in IDE, it should work out of the box, just by running regular tests.

2. Using ’File’

If you’re planning to pick up files from test environment, this class may be useful for you. I’m including a bit of utility functions in the sample as well.

• One method for reading images from {rootProject}/app/test-input-images/image1.png
• Another from test resources {rootProject}/app/test/resources/image1.png

object TestUtils {

    fun projectResAsFile(testSource: String): File {
        return File("test-input-images", testSource)
    }

    fun resAsFile(testSource: String): File {
        return File(this.javaClass.getResource(testSource).file)
    }
}

Iteration #2. Reports with screenshots

Ok, time to tackle the problem of adding as much information as possible to the reports. The biggest source in my case, was adding a screenshot and attaching an intermediate data around it. For example — display document and draw rectangles which parts of documents were scanned.

This is one of the reasons why I have picked up TestNG. I’ve found samples, where people would add screenshots to their reports.

So the code. It’s quite trivial actually. There is a class called Reporter, that appends custom report information. That is exactly what we will be using. With a bit of extra flavor of course 🍲

Reporting custom messages

What is cool, that whenever you’re reporting additional content to reports, it takes in and renders html tags as well. We could leverage that to make the report more readable.

fun reportMessage(message: String) {
    Reporter.log("<p>$message</p>")
}

Reporting custom screenshots

To report with custom images, we will re-use the same leverage to report HTML tag with <img> tag and copied over image resource to display.

1. Copy over image to reports directory

Because the input image might be too big for the report, we will scale it down as well, so it’ll be easier to read, and report won’t have a size of triple A game.

private fun createReportFile(imageFile: File): File {
    val inputImage = readImageFromFile(imageFile.absolutePath)
    val inputDimen = Dimension(inputImage.width, inputImage.height)
    val targetDimen = Dimension(1024, 1024)
    val scaleDimen = calcScaledDimension(inputDimen, targetDimen)
    val resizeImage = resize(inputImage, scaleDimen.width, scaleDimen.height)
    val targetDir = File("test-reports1/screenshots").apply {
        if (!exists())
            mkdirs()
    }
    val targetFile = storeImageToFile(
        resizeImage,
        targetDir,
        "${System.currentTimeMillis()}-${imageFile.nameWithoutExtension}"
    )
    return targetFile
}

2. Display image in report

Also adding a bit more spice. As we may have more than one image attached to report, we bundle up everything to HTML table as well, so it will format properly when scrolling down, and if you are interested in taking a look at screenshots for a concrete test, you would scroll horizontally.

fun reportImages(imageFiles: List<File>) {
    Reporter.log(wrapImageFilesAsTable(imageFiles))
}

private fun wrapImageFilesAsTable(imageFiles: List<File>): String {
    val reportImages = imageFiles
        .map { createReportFile(it) }
    val reportTDImageNames = reportImages
        .map { "<td>${it.nameWithoutExtension}</td>" }
    val reportTDImages = reportImages
        .map { "<td><img src='screenshots/${it.name}'/></td>" }
    val sb = StringBuilder()
        .append("<table border='1'>")
        .append("<tr>")
        .append(reportTDImageNames.joinToString("\n"))
        .append("</tr>")
        .append("<tr>")
        .append(reportTDImages.joinToString("\n"))
        .append("</tr>")
        .append("</table>")
    return sb.toString()
}

3. Test

Here’s how the test looks like

class DocumentProcessorImageReportTest {
    @Test
    fun testValid1() {
        // Assemble
        val docProcessor = DocumentProcessor()
        val imageFile = TestUtils.projectResAsFile("/image1.png")

        // Act
        val result = docProcessor.readDocument(imageFile)

        // Assert
        ReportUtils.reportMessage("Custom message1")
        ReportUtils.reportImages(listOf(imageFile, imageFile, imageFile))
        assertThat(result).isEqualTo("111.0")
    }
}

4. Report

Iteration #3. Grouping test types

Another issue that I’ve faced, is when I run tests, I would like to have some specific type of documents tested.

So for instance, I would like to run tests:

• Only on scanned documents
• Only on documents that use photos as input
• Only on scanning only specific fields

TestNG out of the box has Suits and test groups.

1. Use a class that would contain all group names

Class with all groups

object TestConsts {
    const val GROUP_PDF_DETAILS = "test-pdf-details"
    const val GROUP_PDF_TEMPLATE = "test-pdf-template"
}

2. Bind a test

Bind group to a test class or method

@Test(groups = [TestConsts.GROUP_PDF_DETAILS])

• This can be used on method as well.
• Same class / method may use multiple groups.

3. Running

To run a group test there are multiple ways of doing it. I’ve used one that would involve running through Gradle and can be provided from external resource.

Modify Gradle config in {rootProject}/app/build.gradle.kts, and create new Task

tasks.register("testTemplates", Test::class) {
    var testGroup = ""
    doFirst {
        val argKeyGroup = "testGroup"
        testGroup = if (project.hasProperty(argKeyGroup)) {
            project.properties.getValue(argKeyGroup).toString()
        } else {
            throw IllegalArgumentException("No 'testGroup' specified")
        }
        println("Using groups: ${testGroup}")
        useTestNG {
            testLogging.showStandardStreams = true
            includeGroups = setOf(testGroup)
            outputDirectory = File(project.projectDir, "/test-reports1")
            useDefaultListeners = true
            includeGroups = setOf(testGroup)
        }
    }

    group = "verification"
    ignoreFailures = true
}

To use it, run this command

./gradlew -PtestGroup=test-pdf-template :app:testTemplates

Normally, whenever you’re developing, you’ll be running only one test / document. But whenever you’re running grouped tests, it’s nice to have a command that is executable from anywhere, especially Continous integration system (or your machine)🕵️‍️.

Iteration #4. Scaling 1

When I’ve ran test on 10 documents it’s not an issue. If I run it on 50 documents it starts to get tedious, though it’s still doable. Right now I’m in second week of testing and I already have over 500 tests. So regular solution simply does not scale.

However TestNG (as most other test frameworks) have an ability to use parametrized tests which automates most of what I’m trying to achieve.

1. Test input template

Let’s create a class that would contain test input and its expected result. In my case its file path and the result I’m expecting after reading it.

data class TestInputTemplateImage1(
    val path: String,
    val expectTemplateType: String
)

2. Input values

Now we need to generate input values

companion object {
    fun generateTemplates(): List<TestInputTemplateImage1> {
        val inputPdfPath = "/image1.png"
        return listOf(
            TestInputTemplateImage1(
                path = inputPdfPath,
                expectTemplateType = "1234.0"
            ),
            TestInputTemplateImage1(
                path = inputPdfPath,
                expectTemplateType = "1234.5"
            ),
        )
    }
}

3. Data provider

Providing these values for the test itself

@DataProvider(name = "input1")
fun pdfs(): Array<TestInputTemplateImage1> {
    return generateTemplates().toTypedArray()
}

4. Test method

And the method itself that will be running multiple times with different data

@Test(dataProvider = "input1")
fun valid(testInput: TestInputTemplateImage1) {
    // Assemble
    val docProcessor = DocumentProcessor()
    val imageFile = TestUtils.projectResAsFile(testInput.path)

    // Act
    val result = docProcessor.readDocument(imageFile)

    // Assert
    assertThat(result).isEqualTo(testInput.expectTemplateType)
}

Basically instead of binding it to test values each, we bind it with provided data.

5. Results

Running same test multiple times and its report

Iteration #5. Group assertions

Whenever running your regular tests and you do assertions, on the first failed assertion it would just break the test. Normally there is no reason to run further inspection. This works well when we are working with unit tests. However when running integration tests, we are looking for gathering mostly information of mechanism, instead of providing strict rules. Remember my case? I’m using it to test AI results.

This is where SoftAssert comes into play. It lets you do assertions without breaking the test.

1. Regular test

Write test regularly as you would do

class DocumentProcessorSoftAssertTest {

    @Test
    fun valid() {
        // Assemble
        val docProcessor = DocumentProcessor()
        val imageFile = TestUtils.projectResAsFile("/image1.png")

        // Act
        val result = docProcessor.readDocument(imageFile)

        // Assert
        assertThat(imageFile.exists()).isTrue()
        assertThat(result).isEqualTo("111.0")
    }

}

2. Using SoftAssert

But instead of doing regular assert, we will use SoftAssertions. I’m using assertions from AssertJ, however TestNG has their own counterpart.

• We initialize soft assert
• And instead of using regular assert, we use soft assert instance to do assertions
• So, using SA, the tests do not break, until we use sa.assertAll()

@Test
fun valid() {
    // Assemble
    val sa = SoftAssertions()
    val docProcessor = DocumentProcessor()
    val imageFile = TestUtils.projectResAsFile("/image1.png")

    // Act
    val result = docProcessor.readDocument(imageFile)

    // Assert
    sa.assertThat(imageFile.exists()).isTrue()
    sa.assertThat(result).isEqualTo("111.0")
    sa.assertAll()
}

3. Results

Using this kind of strategy we gather all possible evaluations, and still provide the end result to the test.

org.assertj.core.api.SoftAssertionError:
The following 2 assertions failed:
1)
Expecting:
 <"1234.5">
to be equal to:
 <"111.0">
but was not.
at DocumentProcessorSoftAssertTest.valid(DocumentProcessorSoftAssertTest.java:23)
2)
Expecting:
 <"1234.5">
to be equal to:
 <"444.0">
but was not.
at DocumentProcessorSoftAssertTest.valid(DocumentProcessorSoftAssertTest.java:24)

Iteration #6. Scaling 2

In the state we have our tests right now, whenever we change something in the tests itself, we still have to go through all our tests and change it manually. This might come at quite a cost if you have a bunch of grouped tests like me. Remember test grouping by its type ? Even if we have parametrized tests, we still have a bunch of classes running all kinds of different tests by groups.

To fix it, why not to move the assertions to parametrized data model as well?

This way, whenever I need to customize assertions for overall statistics, I just need to modify the assertions I’m running in the data to apply to all test cases.

1. Test template input

Lets create a data class once more that we will be providing to our tests.

• We create a class that we did before in Scaling1
• However, we expand on functionality, to provide a method that would do all necessary assetions as well
• Parametrized class for data input, output input and assertions

data class TestInputTemplateImage2(
    val path: String,
    val expectTemplateType: String
) {
    fun assertThat(sa: SoftAssertions, result: String) {
        sa.assertThat(result).isEqualTo(expectTemplateType)
    }
}

2. Generate input data

Once again, we create possible input values

companion object {
    fun generateTemplates(): List<TestInputTemplateImage2> {
        val inputPdfPath = "/image1.png"
        return listOf(
            TestInputTemplateImage2(
                path = inputPdfPath,
                expectTemplateType = "1234.0"
            ),
            TestInputTemplateImage2(
                path = inputPdfPath,
                expectTemplateType = "1234.5"
            ),
        )
    }
}

3. Provide data

Provide those values in tests

@DataProvider(name = "input1")
fun pdfs(): Array<TestInputTemplateImage2> {
    return generateTemplates().toTypedArray()
}

4. Running tests

Do the actual test

• The difference is now, that we take the input values, pass it through mechanism we are testing, but instead of doing assertions, we ask the parametrized input to do assertion itself.
• After doing assertions in input, we do sa.assertAll() to break the tests if needed to be.

@Test(dataProvider = "input1")
fun valid(testInput: TestInputTemplateImage2) {
    // Assemble
    val sa = SoftAssertions()
    val docProcessor = DocumentProcessor()
    val imageFile = TestUtils.projectResAsFile(testInput.path)

    // Act
    val result = docProcessor.readDocument(imageFile)

    // Assert
    testInput.assertThat(sa, result)
    sa.assertAll()
}

Now you would ask, why you would go deeper one level, if we have to assert only a string value, there is nothing more to test, just a concrete value, right ?

That is right in basic case. Although this would not be the case, if we would be resulting an object, that would have multiple values that would change over time. In other words, a custom object instead of simple String

It’s a bit hard to explain in words, but I’m quite sure if you’ll follow repository sample I’ve created, it should clear things up ¯\(ツ)_/¯.

Iteration #7. Scaling 3

Oh dear god, there is a 3rd one, right ? 🤦. Well, I did expand this even a bit more, because my case in the future most definitely would change over time, so we have to make it more defensive against those changes.

As this turned out to be such a long post, and I’m quite sure I’m the only one who got here in this section (and probably my wife, that is doing the double check on my grammar), so I’ll try to make this quick 🚀

1. Test input template

Instead of providing concrete values to expect assert, pass in sealed classes (more like Kotlin feature) you are expecting

For ex.: I was expecting totally different document types, so I created expected document with entirely different object and different assertions.

data class TestInputTemplateImage3(
    val path: String,
    val expectObject: TestExpectTemplateDetail
) {
    fun assertThat(sa: SoftAssertions, result: String) {
        expectObject.assertThat(sa, result)
    }
}

As you can see, we provide a new object, that would do the assertions deeper down the level

2. Possible expectations

In our case, as we run the mechanism, we have 3 different expectation we are looking for

• Expectation 1: Output is empty
• Expectation 2: Output is a concrete number
• Expectation 3: Output is a concrete text

Now based on expectation, we do way different assertions

sealed class TestExpectTemplateDetail {
    abstract fun assertThat(
        sa: SoftAssertions,
        data: String,
    )
}

class TestExpectTemplateEmpty() : TestExpectTemplateDetail() {
    override fun assertThat(
        sa: SoftAssertions,
        data: String,
    ) {
        sa.assertThat(data.isEmpty()).isTrue
    }
}

class TestExpectTemplateNumbers(
    val templateNumbers: Double,
) : TestExpectTemplateDetail() {
    override fun assertThat(
        sa: SoftAssertions,
        data: String,
    ) {
        val dataAsNum = data.toDoubleOrNull()
        sa.assertThat(dataAsNum).isEqualTo(templateNumbers, Offset.offset(0.1))
    }
}

class TestExpectTemplateText(
    val templateText: String,
) : TestExpectTemplateDetail() {

    override fun assertThat(
        sa: SoftAssertions,
        data: String,
    ) {
        sa.assertThat(data).isEqualTo(templateText)
    }
}

3. Generate input data

Once again, we generate input data

companion object {
    fun generateTemplates(): List<TestInputTemplateImage3> {
        val inputPdfPath = "/image1.png"
        return listOf(
            TestInputTemplateImage3(
                path = inputPdfPath,
                expectObject = TestExpectTemplateEmpty()
            ),
            TestInputTemplateImage3(
                path = inputPdfPath,
                expectObject = TestExpectTemplateNumbers(1234.0)
            ),
            TestInputTemplateImage3(
                path = inputPdfPath,
                expectObject = TestExpectTemplateText("1234.0")
            ),
        )
    }
}

4. Running tests

We provide the data and run the tests the same, as we did before.

@DataProvider(name = "input1")
fun pdfs(): Array<TestInputTemplateImage3> {
    return generateTemplates().toTypedArray()
}

@Test(dataProvider = "input1")
fun valid(testInput: TestInputTemplateImage3) {
    // Assemble
    val sa = SoftAssertions()
    val docProcessor = DocumentProcessor()
    val imageFile = TestUtils.projectResAsFile(testInput.path)

    // Act
    val result = docProcessor.readDocument(imageFile)

    // Assert
    testInput.assertThat(sa, result)
    sa.assertAll()
}

You may argue, why this complexity. I agree, normally you may not even go that far.

However in my case this really helped out to distinct what I’m trying to do. Do inspections in so varied way, that now I could extract statistics how my mechanism is behaving in various verticals. That is what gives me confidence, that we could use it!

All in all, I found this really cool and simple concepts, hopefully it gives someone else ideas to their problems.

Iteration #n. Future.

Okay, this is where I’m at right now. I’m quite happy for the solution I have right now. I have reports that I can extract numbers, I can inspect results, I can trace-back why some of the documents do not work.

However the tests takes ages to run (around 30–40mins for 500 docs). It would be a good idea to dockerize it and to run it on remote machine. Though that is probably another topic (maybe future post, eh ?).

Anyway, my hopes this was useful for you if you’re working of anything similar.

Oh right, and if you have missed it, here’s the link to all the examples!

Cheers!