Hot questions for Using Cucumber in integration testing

Top Java Programmings / Cucumber / integration testing

Question:

What would be the best way to capture screenshots after each step when running integration tests?

Tests are written in Java using Selenium(3.0.1) and Cucumber(1.2.4).

Code for taking a screenshot after a test is below, but I need a screenshot after each method annotated with @Given, @When, @Then.

@After
public void after(Scenario scenario){
    final byte[] screenshot = driver.getScreenshotAs(OutputType.BYTES);
    scenario.embed(screenshot, "image/png");
}

Thank you for any hints.


Answer:

Can this post help you?

Embedding screenshots in Cucumber JVM

Question:

I am trying to use different properties with cucumber to run integration tests for my spring boot app. I can get it to run if I use the default properties loaded in the main Application class. However when I specify @TestPropertySource to this different location of config properties, it still uses the main Application class's application context. So its the application Context at test execution is the same as it would be when the app would be running on a server. I do not want this.

This is a work related web app using the Spring Boot 1.5.18, Cucumber 4.2.2.

The directory structure is src/main/java for all my java classes and packages, src/main/resources with the application.properties and some others, a root level folder with environment logging and security properties. Then I have src/test/java with my cucumber code and src/test/resources with my modified application.properties file i'd like to use at test execution. I'd also like to specify a different environment, security, logging config property files for testing as well.

This is my ApplicationTest.Java class where I am trying to use the different property sources.

@RunWith(SpringRunner.class)
@SpringBootTest(classes = Application.class, webEnvironment = 
SpringBootTest.WebEnvironment.RANDOM_PORT)
@TestPropertySource( value = 
{"file:${env.dir}/deploy/acceptance/config/env.properties",
    "file:${env.dir}/deploy/acceptance/config/AppConfig.properties",
    "file:${env.dir}/deploy/acceptance/security/auth.properties", 
"classpath:application-test.properties"})
public abstract class ApplicationTest {
    public ApplicationTest() {
}

This is my Cucumber4Test.Java class

@RunWith(Cucumber.class)
@CucumberOptions(features = "src/test/resources", 
    plugin = {"pretty", "html:target/cucumber", 
"junit:target/reports/cucumber.json"}, glue = { "com.test.packages.cucumber" 
}, monochrome = true)

public class CucumberTest {

}

I am not sure if I am missing anything in those classes as I followed a tutorial. But as I said, if I do not set any property sources in my ApplicationTest class, and run the CucumberTest.java as a junit in eclipse, or run mvn clean install, mvn test etc, the cucumber tests will execute as expected.

I've scoured the questions here trying a bunch of things and nothing seems to work for me. Override @PropertySource with @TestPropertySource in Spring Boot

Load different application.yml in SpringBoot Test

EDIT: The reason I believe @TestPropertySource isn't working is due to the following: property source precedence in Spring. When I load up cucumber in src/test/java, it loads those properties I've specified, but then it starts the app in the src/main/java folder. Here its loading the default properties in Application.java. Spring documentation says whichever properties are loaded last take precedence so my TestPropertySource is getting overwritten when the app starts.

My working solution: I wanted to get cucumber to run in Jenkins in a separate job from our build and deploy pipeline. But couldn't find a way around my work's standard for path and directory structure for configs and properties. So what I did:

1) Add properties i needed into the classpath in src/test/resources.

2) Now this is a bit hacky, but a second Application.java in src/test/java with the @Propertysources reflecting the properties I wanted to use.

3) In jenkins I execute a pre-build step before I run a mvn test. This shell just moves the src/test/java/package/with/Application.java into src/main/java/package/with/Application.java. This overwrites the usual Application.java class with the different properties.

4) run mvn test

5) profit


Answer:

This works.

Using the default application.

package cucumber.examples.spring.txn;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.servlet.config.annotation.EnableWebMvc;

@SpringBootApplication
@EnableWebMvc
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

}

And some application.properties

key=App

Then running with:

package cucumber.examples.spring.txn;

import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
public class RunCukesTest {
}

and using this test context configuration

package cucumber.examples.spring.txn;

import cucumber.api.java.Before;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.TestPropertySource;

@SpringBootTest
@AutoConfigureMockMvc
public class CucumberContextConfiguration  {

    @Value("${key}")
    private String value;

    @Before
    public void setup_cucumber_spring_context(){
        // Dummy method so cucumber will recognize this class as glue
        // and use its context configuration.

        System.out.println("Property was " + value);
    }
}

Will print Property was App.

Adding @TestPropertySource("classpath:test.properties") to the CucumberContextConfiguration and creating a test.properties file containing

key=Test

Will print Property was Test.

Question:

I found myself last week having to start thinking about how to refactor an old application that only contains unit tests. My first idea was to add some component test scenarios with Cucumber to get familiarised with the business logic and to ensure I don't break anything with my changes. But at that point I had a conversation with one of the architects in the company I work for that made me wonder whether it was worth it and what was actually the code I had to actually test.

This application has many different types of endpoints: rest endpoints to be called from and to call to, Oracle stored procedures and JMS topics and queues. It's deployed in a war file to a Tomcat server and the connection factory to the broker and the datasource to the database are configured in the server and fetched using JNDI.

My first idea was to load the whole application inside an embedded Jetty, pointing to the real web.xml so everything is loaded as it would be loaded from a production environment but then mocking the connection factory and the datasource. By doing that, all the connectivity logic to the infrastructure where the application is deployed would be tested. Thinking about the hexagonal architecture, this seems like too much effort having in mind that those are only ports which logic should only be about transforming received data into application data. Shouldn't this just be unit tested?

My next idea was to just mock the stored procedures and load the Spring XMLs in my test without any web server, which makes it easier to mock classes. For this I would be using libraries like Spring MockMvc for the rest endpoints and Mockrunner for JMS. But again, this approach would still test some adapters and complicate the test as the result of the tests would be XML and JSON payloads. The transformations done in this application are quite heavy where the same message type could contain different versions of a class (each message could contain many complex object that implement several interfaces).

So right now I was thinking that maybe the best approach would be to just create my tests from the entry point to the application, the services called from the adapters, and verify that the services responsible to send messages to the broker or to call other REST endpoints are actually invoked. Then just ensure there are proper unit tests for the endpoints and verify everything works once deployed by just providing some smoke tests that are executed in a real environment. This would test the connectivity logic and the business logic would be tested in isolation, without caring if a new adapter is added or one is removed.

Is this approach correct? Would I be leaving something without testing this way? Or is it still too much and I should just trust the unit tests?

Thanks.


Answer:

Your application and environment sound quite complicated. I would definitely want integration tests. I'd test the app outside-in as follows:

  • Write a smoke-test suite that runs against the application in the actual production environment. Cucumber would be a good tool to use. That suite should only do things that are safe in production, and should be as small as possible while giving you confidence that the application is correctly installed and configured and that its integrations with other systems are working.

  • Write an acceptance test suite that runs against the entire application in a test environment. Cucumber would be a good choice here too.

    I would expect the acceptance-test environment to include a Tomcat server with test versions of all services that exist in your production Tomcat and a database with a schema, stored procedure, etc. identical to production (but not production data). Handle external dependencies that you don't own by stubbing and mocking, by using a record/replay library such as Betamax and/or by implementing test versions of them yourself. Acceptance tests should be free to do anything to data, and they shouldn't have to worry about availability of services that you don't own.

    Write enough acceptance tests to both describe the app's major use cases and to test all of the important interactions between the parts of the application (both subsystems and classes). That is, use your acceptance tests as integration tests. I find that there is very little conflict between the goals of acceptance and integration tests. Don't write any more acceptance tests than you need for specification and integration coverage, however, as they're relatively slow.

  • Unit-test each class that does anything interesting whatsoever, leaving out only classes that are fully tested by your acceptance tests. Since you're already integration-testing, your unit tests can be true unit tests which stubb or mock their dependencies. (Although there's nothing wrong with letting a unit-tested class use real dependencies that are simple enough to not cause issues in the unit tests).

Measure code coverage to ensure that the combination of acceptance and unit tests tests all your code.

Question:

I am using cucumber for integration tests, and I need some launchers to run them. Launchers are pretty similar, the only difference should be tags. Lets say, I need three common tags and one unique for every launcher. Can I create a parent launcher contains all common cucumber options needed, and then extend it tags adding a unique tag in every child? How can I do that? Are @CucumberOptions inheritable?


Answer:

@CucumberOptions are not inheritable in v5. There was some support for inheritance in older versions but not in a sensible way.

Question:

i'm getting really disappointing with an error throw when i try to launch my unit tests with Cucumber.

I get this error :

java.lang.NoClassDefFoundError: javax/enterprise/context/spi/Contextual
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.getConstructor(Class.java:1825)
at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:104)
at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:86)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader.createUnfilteredTest(JUnit4TestLoader.java:84)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader.createTest(JUnit4TestLoader.java:70)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader.loadTests(JUnit4TestLoader.java:43)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:444)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: java.lang.ClassNotFoundException: javax.enterprise.context.spi.Contextual
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 17 more

I already tried to add this lib to my classpath, But this error is still throw with another missing lib, and another etc ...

I've done searches, and nothing of all this libs are required by Cucumber. This is my test class.

@RunWith(Runner.class)
@CucumberOptions(monochrome = true, features = { "src/test-integration/resources/scenarii" }, glue = {
        "info.scandi.fusion.cucumber" }, plugin = { "pretty" })
public class AllTests {

}

The paths seems to be correct. And as you can see my steps definition are defined into a jar that i have imported in my project.

My error seems to be related to this stackoverflow question. But i'm not using any root cucumber package.

Any help would be really appreciate.


Answer:

This is not a Cucumber error. You are missing a dependency that is needed runtime.

My approach for trouble shooting this problem would be to remove as much as possible of the Cucumber configuration until the error disappeared.

I would only keep @RunWith(Cucumber.class) and then see what happened.

Looking at your JUnit class, I notice that you are using a runner called Runner.class I don't know what runner this is, but I know that it isn't the JUnit Cucumber runner.

A good starting point for getting started with Cucumber is https://github.com/cucumber/cucumber-java-skeleton