Just another post of TestNG bag of tricks

Photo by Annie Spratt on Unsplash
  • Handling big (quantity wise) data input in integration tests
  • ..using TestNG testing framework
  • ..with AssertJ assertions framework
  • ..through Gradle build system
  • ..built with Kotlin
  1. Problem — case I was solving
  2. Solution — overall how to leverage integration tests
  3. Regular tests — How to assemble TestNG and run on Gradle build system
  4. Reports — How to enrich reports with screenshots
  5. Grouping — Grouping tests with more verticals
  6. Scaling 1 — Test parametrize
  7. Group assertions — Using SoftAssert
  8. Scaling 2 — Smarter use of parametrization 2
  9. Scaling 3 — Smarter use of parametrization 3
Demo of generated reports

Problem

Right. So here I go.

Solution

Here are goals, what I’m trying to achieve. The tools I’ll be using, should (at least partially) check these ’check-boxes’

  • Handle a big input base
  • Generate results in report
  • Generate results as statistics in various ways
  • Report should also have some indication what went wrong
  • Test should be easily maintainable. (As we have so much input, having to change tests 500 times is not an option.)

Iteration #1. Regular tests

Testable class

Just before we begin anything, we need a class that we will run tests on. As you have probably guessed I can’t disclose real mechanism we are using due to privacy reasons. And that anyway would distract the topic of testing solutions I’m proposing in this topic. So for sanity reasons we will be testing this complex class 🤓.

/**
* Imagined document processor
*/
class DocumentProcessor {

/**
* Processes document and outputs read value in it
* @param imagePath document path in file system
* @return document read value or empty
*/
fun readDocument(imagePath: String): String {
return "1234.5"
}
}

Testing class

Ok, on to testing now.

1. Include library dependency.

We will include in {rootProject}/app/build.gradle.kts

testImplementation("org.testng:testng:7.4.0")
testImplementation("org.testng:reportng:1.2.2")

2. Configure Gradle to use TestNG instead of regular JUnit

  • Include at the end of file {rootProject}/app/build.gradle.kts
  • Commented out lines defines properties to generate reports using ReportNG. So ignore this, unless you want to try customize your reports.
  • Our reports will be generated in {rootProject}/app/test-reports1 directory
tasks.withType(Test::class) {
ignoreFailures = true
useTestNG {
// systemProperties.put("org.uncommons.reportng.stylesheet", "${projectDir}/resources/hudsonesque.css")
// systemProperties.put("org.uncommons.reportng.escape-output", false)
// systemProperties.put("org.uncommons.reportng.frames", true)
outputDirectory = File(project.projectDir, "/test-reports1")
useDefaultListeners = true
// options {
// listeners.add("org.uncommons.reportng.HTMLReporter")
// listeners.add("org.uncommons.reportng.JUnitXMLReporter")
// }
}
testLogging.showStandardStreams = true
testLogging.exceptionFormat = org.gradle.api.tasks.testing.logging.TestExceptionFormat.FULL
}

3. Write initial test

  • We are creating our initial test in {rootProject}/app/src/test/lt.markmerkk.processor/DocumentProcessorTest.kt
class DocumentProcessorTest {
@Test
fun valid() {
// Assemble
val docProcessor = DocumentProcessor()
val imageFile = TestUtils.projectResAsFile("/image1.png")

// Act
val result = docProcessor.readDocument(imageFile)

// Assert
assertThat(imageFile.exists()).isTrue()
assertThat(result).isEqualTo("111.0")
}
}
  • Good thing to mention, make sure you’re using TestNG imports. Otherwise you may be using JUnit framework or something similar.
import org.testng.annotations.Test

Reports

1. Console output

java.lang.AssertionError:
Expecting:
<"1234.5">
to be equal to:
<"111.0">
but was not.

<...>

===============================================
Default Suite
Total tests run: 1, Passes: 0, Failures: 1, Skips: 0
===============================================

2. HTML

  • Html reports can be found in {rootProject}/app/test-reports1/index.html

Troubleshooting

1. Using Gradle

Make sure you’re running tests through Gradle.

  • To be sure to use gradle, use this command to run tests (while being in {rootProject}/).
./gradlew :app:test
  • If you’re in IDE, it should work out of the box, just by running regular tests.

2. Using ’File’

If you’re planning to pick up files from test environment, this class may be useful for you. I’m including a bit of utility functions in the sample as well.

  • One method for reading images from {rootProject}/app/test-input-images/image1.png
  • Another from test resources {rootProject}/app/test/resources/image1.png
object TestUtils {

fun projectResAsFile(testSource: String): File {
return File("test-input-images", testSource)
}

fun resAsFile(testSource: String): File {
return File(this.javaClass.getResource(testSource).file)
}
}

Iteration #2. Reports with screenshots

Ok, time to tackle the problem of adding as much information as possible to the reports. The biggest source in my case, was adding a screenshot and attaching an intermediate data around it. For example — display document and draw rectangles which parts of documents were scanned.

Reporting custom messages

What is cool, that whenever you’re reporting additional content to reports, it takes in and renders html tags as well. We could leverage that to make the report more readable.

fun reportMessage(message: String) {
Reporter.log("<p>$message</p>")
}

Reporting custom screenshots

To report with custom images, we will re-use the same leverage to report HTML tag with <img> tag and copied over image resource to display.

1. Copy over image to reports directory

Because the input image might be too big for the report, we will scale it down as well, so it’ll be easier to read, and report won’t have a size of triple A game.

private fun createReportFile(imageFile: File): File {
val inputImage = readImageFromFile(imageFile.absolutePath)
val inputDimen = Dimension(inputImage.width, inputImage.height)
val targetDimen = Dimension(1024, 1024)
val scaleDimen = calcScaledDimension(inputDimen, targetDimen)
val resizeImage = resize(inputImage, scaleDimen.width, scaleDimen.height)
val targetDir = File("test-reports1/screenshots").apply {
if (!exists())
mkdirs()
}
val targetFile = storeImageToFile(
resizeImage,
targetDir,
"${System.currentTimeMillis()}-${imageFile.nameWithoutExtension}"
)
return targetFile
}

2. Display image in report

Also adding a bit more spice. As we may have more than one image attached to report, we bundle up everything to HTML table as well, so it will format properly when scrolling down, and if you are interested in taking a look at screenshots for a concrete test, you would scroll horizontally.

fun reportImages(imageFiles: List<File>) {
Reporter.log(wrapImageFilesAsTable(imageFiles))
}

private fun wrapImageFilesAsTable(imageFiles: List<File>): String {
val reportImages = imageFiles
.map { createReportFile(it) }
val reportTDImageNames = reportImages
.map { "<td>${it.nameWithoutExtension}</td>" }
val reportTDImages = reportImages
.map { "<td><img src='screenshots/${it.name}'/></td>" }
val sb = StringBuilder()
.append("<table border='1'>")
.append("<tr>")
.append(reportTDImageNames.joinToString("\n"))
.append("</tr>")
.append("<tr>")
.append(reportTDImages.joinToString("\n"))
.append("</tr>")
.append("</table>")
return sb.toString()
}

3. Test

Here’s how the test looks like

class DocumentProcessorImageReportTest {
@Test
fun testValid1() {
// Assemble
val docProcessor = DocumentProcessor()
val imageFile = TestUtils.projectResAsFile("/image1.png")

// Act
val result = docProcessor.readDocument(imageFile)

// Assert
ReportUtils.reportMessage("Custom message1")
ReportUtils.reportImages(listOf(imageFile, imageFile, imageFile))
assertThat(result).isEqualTo("111.0")
}
}

4. Report

Iteration #3. Grouping test types

Another issue that I’ve faced, is when I run tests, I would like to have some specific type of documents tested.

  • Only on scanned documents
  • Only on documents that use photos as input
  • Only on scanning only specific fields

1. Use a class that would contain all group names

Class with all groups

object TestConsts {
const val GROUP_PDF_DETAILS = "test-pdf-details"
const val GROUP_PDF_TEMPLATE = "test-pdf-template"
}

2. Bind a test

Bind group to a test class or method

@Test(groups = [TestConsts.GROUP_PDF_DETAILS])
  • This can be used on method as well.
  • Same class / method may use multiple groups.

3. Running

To run a group test there are multiple ways of doing it. I’ve used one that would involve running through Gradle and can be provided from external resource.

tasks.register("testTemplates", Test::class) {
var testGroup = ""
doFirst {
val argKeyGroup = "testGroup"
testGroup = if (project.hasProperty(argKeyGroup)) {
project.properties.getValue(argKeyGroup).toString()
} else {
throw IllegalArgumentException("No 'testGroup' specified")
}
println("Using groups: ${testGroup}")
useTestNG {
testLogging.showStandardStreams = true
includeGroups = setOf(testGroup)
outputDirectory = File(project.projectDir, "/test-reports1")
useDefaultListeners = true
includeGroups = setOf(testGroup)
}
}

group = "verification"
ignoreFailures = true
}
./gradlew -PtestGroup=test-pdf-template :app:testTemplates

Iteration #4. Scaling 1

When I’ve ran test on 10 documents it’s not an issue. If I run it on 50 documents it starts to get tedious, though it’s still doable. Right now I’m in second week of testing and I already have over 500 tests. So regular solution simply does not scale.

1. Test input template

Let’s create a class that would contain test input and its expected result. In my case its file path and the result I’m expecting after reading it.

data class TestInputTemplateImage1(
val path: String,
val expectTemplateType: String
)

2. Input values

Now we need to generate input values

companion object {
fun generateTemplates(): List<TestInputTemplateImage1> {
val inputPdfPath = "/image1.png"
return listOf(
TestInputTemplateImage1(
path = inputPdfPath,
expectTemplateType = "1234.0"
),
TestInputTemplateImage1(
path = inputPdfPath,
expectTemplateType = "1234.5"
),
)
}
}

3. Data provider

Providing these values for the test itself

@DataProvider(name = "input1")
fun pdfs(): Array<TestInputTemplateImage1> {
return generateTemplates().toTypedArray()
}

4. Test method

And the method itself that will be running multiple times with different data

@Test(dataProvider = "input1")
fun valid(testInput: TestInputTemplateImage1) {
// Assemble
val docProcessor = DocumentProcessor()
val imageFile = TestUtils.projectResAsFile(testInput.path)

// Act
val result = docProcessor.readDocument(imageFile)

// Assert
assertThat(result).isEqualTo(testInput.expectTemplateType)
}

5. Results

Running same test multiple times and its report

Iteration #5. Group assertions

Whenever running your regular tests and you do assertions, on the first failed assertion it would just break the test. Normally there is no reason to run further inspection. This works well when we are working with unit tests. However when running integration tests, we are looking for gathering mostly information of mechanism, instead of providing strict rules. Remember my case? I’m using it to test AI results.

1. Regular test

Write test regularly as you would do

class DocumentProcessorSoftAssertTest {

@Test
fun valid() {
// Assemble
val docProcessor = DocumentProcessor()
val imageFile = TestUtils.projectResAsFile("/image1.png")

// Act
val result = docProcessor.readDocument(imageFile)

// Assert
assertThat(imageFile.exists()).isTrue()
assertThat(result).isEqualTo("111.0")
}

}

2. Using SoftAssert

But instead of doing regular assert, we will use SoftAssertions. I’m using assertions from AssertJ, however TestNG has their own counterpart.

  • We initialize soft assert
  • And instead of using regular assert, we use soft assert instance to do assertions
  • So, using SA, the tests do not break, until we use sa.assertAll()
@Test
fun valid() {
// Assemble
val sa = SoftAssertions()
val docProcessor = DocumentProcessor()
val imageFile = TestUtils.projectResAsFile("/image1.png")

// Act
val result = docProcessor.readDocument(imageFile)

// Assert
sa.assertThat(imageFile.exists()).isTrue()
sa.assertThat(result).isEqualTo("111.0")
sa.assertAll()
}

3. Results

Using this kind of strategy we gather all possible evaluations, and still provide the end result to the test.

org.assertj.core.api.SoftAssertionError:
The following 2 assertions failed:
1)
Expecting:
<"1234.5">
to be equal to:
<"111.0">
but was not.
at DocumentProcessorSoftAssertTest.valid(DocumentProcessorSoftAssertTest.java:23)
2)
Expecting:
<"1234.5">
to be equal to:
<"444.0">
but was not.
at DocumentProcessorSoftAssertTest.valid(DocumentProcessorSoftAssertTest.java:24)

Iteration #6. Scaling 2

In the state we have our tests right now, whenever we change something in the tests itself, we still have to go through all our tests and change it manually. This might come at quite a cost if you have a bunch of grouped tests like me. Remember test grouping by its type ? Even if we have parametrized tests, we still have a bunch of classes running all kinds of different tests by groups.

1. Test template input

Lets create a data class once more that we will be providing to our tests.

  • We create a class that we did before in Scaling1
  • However, we expand on functionality, to provide a method that would do all necessary assetions as well
  • Parametrized class for data input, output input and assertions
data class TestInputTemplateImage2(
val path: String,
val expectTemplateType: String
) {
fun assertThat(sa: SoftAssertions, result: String) {
sa.assertThat(result).isEqualTo(expectTemplateType)
}
}

2. Generate input data

Once again, we create possible input values

companion object {
fun generateTemplates(): List<TestInputTemplateImage2> {
val inputPdfPath = "/image1.png"
return listOf(
TestInputTemplateImage2(
path = inputPdfPath,
expectTemplateType = "1234.0"
),
TestInputTemplateImage2(
path = inputPdfPath,
expectTemplateType = "1234.5"
),
)
}
}

3. Provide data

Provide those values in tests

@DataProvider(name = "input1")
fun pdfs(): Array<TestInputTemplateImage2> {
return generateTemplates().toTypedArray()
}

4. Running tests

Do the actual test

  • The difference is now, that we take the input values, pass it through mechanism we are testing, but instead of doing assertions, we ask the parametrized input to do assertion itself.
  • After doing assertions in input, we do sa.assertAll() to break the tests if needed to be.
@Test(dataProvider = "input1")
fun valid(testInput: TestInputTemplateImage2) {
// Assemble
val sa = SoftAssertions()
val docProcessor = DocumentProcessor()
val imageFile = TestUtils.projectResAsFile(testInput.path)

// Act
val result = docProcessor.readDocument(imageFile)

// Assert
testInput.assertThat(sa, result)
sa.assertAll()
}

Iteration #7. Scaling 3

Oh dear god, there is a 3rd one, right ? 🤦. Well, I did expand this even a bit more, because my case in the future most definitely would change over time, so we have to make it more defensive against those changes.

1. Test input template

Instead of providing concrete values to expect assert, pass in sealed classes (more like Kotlin feature) you are expecting

data class TestInputTemplateImage3(
val path: String,
val expectObject: TestExpectTemplateDetail
) {
fun assertThat(sa: SoftAssertions, result: String) {
expectObject.assertThat(sa, result)
}
}

2. Possible expectations

In our case, as we run the mechanism, we have 3 different expectation we are looking for

  • Expectation 1: Output is empty
  • Expectation 2: Output is a concrete number
  • Expectation 3: Output is a concrete text
sealed class TestExpectTemplateDetail {
abstract fun assertThat(
sa: SoftAssertions,
data: String,
)
}

class TestExpectTemplateEmpty() : TestExpectTemplateDetail() {
override fun assertThat(
sa: SoftAssertions,
data: String,
) {
sa.assertThat(data.isEmpty()).isTrue
}
}

class TestExpectTemplateNumbers(
val templateNumbers: Double,
) : TestExpectTemplateDetail() {
override fun assertThat(
sa: SoftAssertions,
data: String,
) {
val dataAsNum = data.toDoubleOrNull()
sa.assertThat(dataAsNum).isEqualTo(templateNumbers, Offset.offset(0.1))
}
}

class TestExpectTemplateText(
val templateText: String,
) : TestExpectTemplateDetail() {

override fun assertThat(
sa: SoftAssertions,
data: String,
) {
sa.assertThat(data).isEqualTo(templateText)
}
}

3. Generate input data

Once again, we generate input data

companion object {
fun generateTemplates(): List<TestInputTemplateImage3> {
val inputPdfPath = "/image1.png"
return listOf(
TestInputTemplateImage3(
path = inputPdfPath,
expectObject = TestExpectTemplateEmpty()
),
TestInputTemplateImage3(
path = inputPdfPath,
expectObject = TestExpectTemplateNumbers(1234.0)
),
TestInputTemplateImage3(
path = inputPdfPath,
expectObject = TestExpectTemplateText("1234.0")
),
)
}
}

4. Running tests

We provide the data and run the tests the same, as we did before.

@DataProvider(name = "input1")
fun pdfs(): Array<TestInputTemplateImage3> {
return generateTemplates().toTypedArray()
}

@Test(dataProvider = "input1")
fun valid(testInput: TestInputTemplateImage3) {
// Assemble
val sa = SoftAssertions()
val docProcessor = DocumentProcessor()
val imageFile = TestUtils.projectResAsFile(testInput.path)

// Act
val result = docProcessor.readDocument(imageFile)

// Assert
testInput.assertThat(sa, result)
sa.assertAll()
}

Iteration #n. Future.

Okay, this is where I’m at right now. I’m quite happy for the solution I have right now. I have reports that I can extract numbers, I can inspect results, I can trace-back why some of the documents do not work.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store