Hot questions for Using Cucumber in qaf

Question:

we already have automated testcases using cucumber-gherkin feature files with WebDriver. Right now we are able to execute feature files parallel, however we are not able to execute parallel at scenario/scenario outline level. By this way we want to minimize the execution time and utilize the cloud service at maximum level.

I read this post, which claims

one can run gherkin as QAF scenario so it will have features like run configuration, reporting, parallel execution, step listener

.

Before migrating, I want to know if anybody has tried or know any limitation as our test bed is very large.


Answer:

Yes, with QMetry Automation Framework you can execute parallel at scenario/scenario outline level with without any change in your feature files.

If you have used any cucumber hooks it will not work. For that you can provide TestNG listener to achieve the same goal. You will get more listener support from TestNG and QAF listeners which may not be achievable with cucumber hooks.

Another difference, which in fact not a limitation, is For execution instead of --tag you need to use TestNG groups filter and provide your tags as group. For Example:

<groups>
    <run>
        <exclude name="@MyTag" />
    </run>
</groups>

Go ahead and give it try without any fear. For quick evaluation, you can download example and check. While using cucumber you will get many more features with QAF for execution as well as for new development of new test case like databean, data-providers, custom-components etc...

Question:

I am trying to see, if possible, to have a more elegant way to handle calling nTh numbers from a Cucumber Scenario Outline that correlates to an excel spreadsheet row(nth).

Currently I am using iteration numbers to define the row # of the excel spread sheet to pull the data from. I wanted to see if it was possible to use cucumber with excel in a more elegant way than the below example with the scenario outline.

Some background:

  • Each iteration needs to be its own scenario. Hence why I'm not using a simple for loop with row.count.
  • I am fully aware of scenario outline as a way to do data tables but my company wants to see a POF that we can integrate large data sets through excel.
  • Current setup works for small data sets but when we get into the large excel spreadsheets I do not want to type out nth numbers on the outline

Cucumber code:

Feature: User is using an excel spreadsheet with cucumber driving it

  Scenario Outline: Data Driven with excel and data sets

   When I am on the amps mainscreen
    Then I input username and passwords with excel row"<row_index>" dataset

    Examples:
    | row_index  |
    | 1          |
    | 2          |
    | 3          |
    | 4          |

Step File:

//Excel Steps
@When("^I am on the amps mainscreen$")
public void i_am_on_the_amps_mainscreen()  {
    System.out.println("Im loading");
}
//Excel Steps
@Then("^I input username and passwords with excel row\"([^\"]*)\" dataset$")
public void i_input_username_and_passwords_with_excel_row_dataset(int rownum)    throws IOException {
    login.readExcel(rownum);
}

Actual Code:

 public void readExcel (int row) throws IOException{

    File src=new File("src/test/resources/username.xlsx");
    FileInputStream fis=new FileInputStream(src);
    XSSFWorkbook srcBook= new XSSFWorkbook(fis);
    XSSFSheet sourceSheet = srcBook.getSheetAt(0);

    XSSFRow sourceRow = sourceSheet.getRow(row);
    XSSFCell username=sourceRow.getCell(0);
    XSSFCell password=sourceRow.getCell(1);
    String userExcel = username.getStringCellValue();
    String pwExcel = password.getStringCellValue();
    System.out.println("The username is" +userExcel);
    System.out.println("The password is" +pwExcel);
    log.info("The username on " +row + " is: "+userExcel);
    log.info("The password on "+row+ " is: "+pwExcel);
    driver.findElement(txtbox_username).sendKeys(userExcel);
    driver.findElement(txtbox_password).sendKeys(pwExcel);
    driver.findElement(btn_logon).click();

}

Answer:

You can use QMetry Automation Framework with gherkin factory. It supports test data provided outside feature file for example excel, xml, json, csv, or database. you can provide datafile for examples like:

Examples:{'datafile':'resources/testdata.xls'}

Here is the example you can check.

Question:

Good afternoon. Is it possible to implement the work of QAF with JUnit? For example, i want to use qaf-gherkin in my project, but it's build on JUnit + Cucumber. As I see, there is a similar question, but there is no description about project and on which test framework it was based.


Answer:

QAF built upon TestNG. If you are using Junit for Java you can run using TestNG by setting junit="true" in configuration file. However for migrating cucumber to QAF it should not impact current runner is either Junit or TestNG until you are using any specific features of Junit. All you need to start using QAF following documentation. You don't need to write or use additional java class to run your tests because it is taken care by QAF. All you need to do is create xml configuration file for to run your BDD

Because QAF provides all testing needs together you don't need to relay on multiple frameworks (junit and cucumber in your case).

Question:

I am trying to implement an automated evidence capture system for each cucumber scenario that is executed. As part of that I need to use the first scenario @tag as part of the evidence folder name.

However, I have found that the getSourceTagNames() method on a cucumber scenario returns the list in a random order, compared to the order in which the tags were written. So simply calling scenario.getSourceTagNames().get(0) will not consistently give me the tag I need.

e.g Scenario: Example @tagINeed @secondTag @thirdTag

getSourceTagNames() may return {@thirdTag, @tagINeed, @secondTag}

Is there a way to ensure the list returned is in the same order as the order in which it was written?


Answer:

What you are looking for can be achieved using gherkin with qaf, you will be able to access tags in order they defined for example:

@tagINeed @secondTag @thirdTag
Scenario: Example 

Java code:

scenario.getGroups()[0];//will return "@tagINeed"

If you will use BDD2 syntax instead of gherkin, you can have meta-data with scenario. For example:

@TestID:ABC-123
@Evidence:tagINeed @firstTag @secondTag
Scenario: Example 

Java code:

scenario.getMetaData().get("Evidence");//will return "tagINeed"
scenario.getGroups()[0];//will return "firstTag"

Question:

I'm trying to figure out how I can automatically update test case results for test cases in Rally through Cucumber automation scripts. I want to be able to run my test scripts, which will then update the test case results in Rally automatically to Pass or Fail.

Is there any way to do that with Cucumber? I'm using Cucumber along with TestNG and Rest Assured.


Answer:

If you are using TestNG's QAF extension for BDD it provides a way to integrate your test results with test management tool by providing TestCaseResultUpdator. In your test case or scenario you need to provide test case id from test management tool and call api to update test result for that test case. QAF supports gherkin but gherking doesn't support custom meta-data. You can use BDD2 which is super set of gherkin and your scenario may look like below:

@smoke @RallyId:TC-12345
Scenario:  A scenario

    Given step represents a precondition to an event
    When step represents the occurrence of the event
    Then step represents the outcome of the event

In above example assume that RallyId represents test case id in test management tool. You can use it while implementing result updator.

package example;
...
public class RallyResultUpdator implements TestCaseResultUpdator{

   @Override
   public String getToolName() {
    return "Rally";
   }

   /**
    * This method will be called by result updator after completion of each testcase/scenario.
    * @param params
    *            tescase/scenario meta-data including method parameters if any
    * @param result
    *            test case result
    * @param details
    *            run details
    * @return
    */

   @Override
   public boolean updateResult(Map<String, ? extends Object> metadata,
        TestCaseRunResult result, String details) {

    String tcid = metadata.get("RallyId");
    // Provide test management tool specific implemeneation/method calls

    return true;
   }
}

Register your updator as below:

result.updator=example.RallyResultUpdator

Result updator will automatically called by qaf when testcase completed and will run in separate thread so your test execution need not to wait.

Question:

Currently we are triggering our smoke test from testng.xml where we have two different sceanrios to be validated.

Our requirement is that if one scenario fails(@Test1), other should not execute(@Test2). How can I achieve this in QAF, Testng - Cucumber set up ?

    <groups>
        <run>
            <include name="@Test1" />
            <include name="@Test2" />
        </run>
    </groups>
    <classes>
        <class
            name="com.qmetry.qaf.automation.step.client.gherkin.GherkinScenarioFactory" />
    </classes>
</test>

Answer:

One of the way is by implementing method invocation listener. In after method you can set a flag and in before method you can check flag and skip test depending on value of flag. For example:

package com.qmetry.qaf.example.test;
...
public class StopRunListener implements IInvokedMethodListener {
   private static boolean hasFailure=false;

    @Override
    public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
        String[] groups = testResult.getMethod().getGroups();

        if(hasFailure && Arrays.asList(groups).contains("Test2")) {
            throw new SkipException("Stop execution due to failure");
        }
    }

    @Override
    public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
        String[] groups = testResult.getMethod().getGroups();

        if(!testResult.isSuccess() && Arrays.asList(groups).contains("Test1")) {
            hasFailure=true;
        }

    }

}

Add listener in your XML configuration file

    <listeners>
       <listener class-name="com.qmetry.qaf.example.test.StopRunListener" />
    </listeners>
    <groups>
        <run>
            <include name="@Test1" />
            <include name="@Test2" />
        </run>
    </groups>
    <classes>
        <class
            name="com.qmetry.qaf.automation.step.client.gherkin.GherkinScenarioFactory" />
    </classes>

With Gherkin syntax you can't specify dependency or other meta-data. You can use BDD2 syntax available with qaf-2.1.14 and set dependency of group test1 on test2. It will make sure that test from group test2 get executed after group test1. But it will not skip the test if one of the test in depending group is fail. That you can achieve using listener like provided in above example.

For example:

#meta-data on feature will be assigned to all scenario in feature file
@Test1
Feature: A feature is a collection of scenarios

@Test2
@dependsOnGroups:Test1
Feature: A feature is a collection of scenarios

XML config will be:

    <listeners>
       <listener class-name="com.qmetry.qaf.example.test.StopRunListener" />
    </listeners>
    <groups>
        <run>
            <include name="Test1" /> <!-- don't add @ in group for BDD or BDD2 -->
            <include name="Test2" />
        </run>
    </groups>
    <classes>
        <class
            name="com.qmetry.qaf.automation.step.client.text.BDDTestFactory2" />
    </classes>