Unit 6: Test Automation and Management/Reporting Frameworks

CSE376 — Automated Testing 11 min read

I. Orientation — Build Management and Test Execution

Maven and TestNG are complementary tools in automated testing. Maven manages a Java project’s build lifecycle, dependencies, plugins, and reports, while TestNG defines, organizes, executes, and reports automated test cases. Together with Selenium, they support repeatable, maintainable, and team-friendly test automation.

Build management: Maven standardizes compiling, testing, packaging, and dependency retrieval through pom.xml.
Test organization: TestNG groups test methods into suites, supports priorities and dependencies, and provides lifecycle annotations.
Automation principle: A test should be repeatable, independently understandable, and executable from a command line or continuous integration server.
Reporting convention: Test results should identify passed, failed, skipped, duration, and failure details.
Project convention: Java source code normally belongs in src/main/java, while automated tests belong in src/test/java.

II. Maven — Project and Build Management

Maven is an open-source build automation and project management tool primarily used for Java applications. Its central configuration file, pom.xml, describes the project and controls its build lifecycle.

A. Introduction to Maven

Maven uses convention over configuration: standard directory structures and lifecycle phases reduce the amount of project-specific setup.

Meaning: Maven comes from the Yiddish word for “accumulator of knowledge”; technically, it coordinates project construction and management.
Core file: pom.xml means Project Object Model and contains coordinates, dependencies, plugins, and build settings.
Execution model: Commands such as mvn test invoke a predefined sequence of lifecycle phases.
Typical structure: src/main/java stores application code; src/test/java stores test code; target stores generated output.

B. Benefits and Features of Maven

Maven improves consistency by giving projects a standard build process and automatically resolving external libraries.

Dependency management: Declaring org.seleniumhq.selenium:selenium-java:4.x allows Maven to download Selenium and its transitive dependencies.
Repeatability: The same command, such as mvn clean test, can run locally or in Jenkins with equivalent lifecycle behavior.
Convention: Standard paths allow tools and developers to locate source, test, and generated files predictably.
Extensibility: Plugins add functions such as compilation, packaging, code analysis, and report generation.
Repository support: Maven retrieves artifacts from local repositories, remote repositories, and Maven Central.

C. Activities Managed by Maven

Maven manages the activities required to move source code from development to a tested deliverable.

Compilation: The compiler plugin converts .java files into .class files.
Testing: The Surefire plugin runs unit and TestNG tests during the test phase.
Packaging: package creates an artifact such as a .jar file.
Installation: install places the artifact in the developer’s local Maven repository.
Deployment: deploy transfers the artifact to a remote repository for team or release use.
Cleaning: clean removes the target directory and previous build output.

D. Architecture of Maven

Maven architecture combines a project object model, lifecycle, plugins, and repositories.

POM: Defines project identity and build instructions.
Lifecycle: Standard phases include validate, compile, test, package, verify, install, and deploy.
Plugins: Maven delegates work to goals, such as compiler:compile or surefire:test.
Repositories:
- Local repository: Usually ~/.m2/repository; caches downloaded artifacts.
- Remote repository: Stores artifacts on a server.
- Central repository: Public default source for many Java libraries.
Coordinates: groupId, artifactId, and version uniquely identify a Maven artifact.

E. Installing/Configuring Maven

Maven requires a compatible Java Development Kit and environment configuration.

Prerequisite: Install a JDK and verify it with java -version.
Installation: Download Maven, extract it, and add its bin directory to PATH.
Verification: Run mvn -version; the output should show Maven, Java, and operating-system details.
Configuration: Maven settings may be placed in ~/.m2/settings.xml for mirrors, proxies, credentials, or repository rules.
Build condition: JAVA_HOME should point to the JDK installation rather than only a Java runtime.

F. Creating Maven Project

A Maven project can be generated from an archetype or created using an IDE.

Coordinates: For example, com.example is the groupId, selenium-tests is the artifactId, and 1.0-SNAPSHOT is the version.
Standard folders: Create src/main/java and src/test/java; Maven recognizes them without additional path configuration.
Command example:
BASH
```
  mvn archetype:generate
```
Build output: Compiled classes, reports, and packaged files are written under target.
Naming rule: Test classes commonly use names such as LoginTest.java, which makes their purpose clear and supports test discovery.

G. Understanding POM.xml File

The POM is an XML document that describes how Maven identifies and builds a project.

Project identity: The combination below identifies the artifact:

XML

  <groupId>com.example</groupId>
  <artifactId>selenium-tests</artifactId>
  <version>1.0-SNAPSHOT</version>

Packaging: <packaging>jar</packaging> selects JAR packaging; jar is the default.
Properties: A property such as <maven.compiler.source>17</maven.compiler.source> centralizes the Java version.
Dependencies: The <dependencies> section lists libraries required by application or test code.
Build section: <build> configures plugins, test execution, source directories, and reporting behavior.

H. Adding Dependencies to POM.xml

Dependencies should be declared with a scope that matches how the library is used.

Test scope: TestNG is normally available only while compiling and running tests:

XML

  <dependency>
    <groupId>org.testng</groupId>
    <artifactId>testng</artifactId>
    <version>7.10.2</version>
    <scope>test</scope>
  </dependency>

Selenium dependency: selenium-java supplies Selenium WebDriver APIs and related modules.
Version control: Explicit versions prevent accidental changes caused by repository updates.
Transitive dependencies: Maven automatically downloads libraries required by a declared dependency.
Conflict handling: mvn dependency:tree displays resolved dependencies and helps locate incompatible versions.

III. TestNG — Test Execution and Reporting Framework

TestNG is a Java testing framework inspired by JUnit but designed to support broader test configuration, grouping, parallel execution, and reporting. It is especially useful for Selenium suites containing many related browser tests.

A. Introduction to TestNG

TestNG uses annotations and an XML suite model to define how test methods are prepared and executed.

Test marker: @Test identifies an executable test method.
Lifecycle: @BeforeMethod runs before each test method; @AfterMethod runs afterward.
Suite model: testng.xml can select classes, packages, groups, and execution settings.
Execution result: TestNG records passed, failed, and skipped methods with timing and exception information.

B. TestNG vs JUnit

Both frameworks execute Java tests, but TestNG provides features that are convenient for larger integration and Selenium suites.

Annotations:
1. TestNG: Uses @BeforeSuite, @BeforeTest, @BeforeClass, and @BeforeMethod.
2. JUnit: Uses lifecycle annotations such as @BeforeAll, @BeforeEach, and framework-specific equivalents.
Organization: TestNG directly supports groups, priorities, dependencies, data providers, and suite XML configuration.
Assertions: Both support assertions, although the exact assertion classes and lifecycle APIs differ by version.
Parallelism: TestNG can configure parallel suites, tests, classes, or methods through testng.xml.
Selection: Groups such as smoke and regression allow TestNG to run selected categories.

C. Downloading and Installing TestNG

TestNG can be installed through Maven or an IDE plugin.

Maven method: Add the TestNG dependency to pom.xml; Maven downloads it during mvn test.
IDE method: Install the TestNG plugin from the IDE’s marketplace when direct IDE execution is required.
Verification: A class importing org.testng.annotations.Test confirms that the library is available to the project.
Compatibility: Select a TestNG version compatible with the project’s Java version and build plugins.
Recommended practice: Keep TestNG in the build file so every developer and CI server receives the same version.

D. Creating Test Cases Using TestNG Annotations

A TestNG test case is a Java method marked with @Test, usually supported by setup and cleanup methods.

Basic test:

JAVA

  import org.testng.annotations.Test;

  public class LoginTest {
      @Test
      public void validLogin() {
          System.out.println("Login validated");
      }
  }

Setup: @BeforeMethod can create a WebDriver before each test.
Cleanup: @AfterMethod should call driver.quit() so browser processes do not remain active.
Isolation: Each test should establish its own required state rather than depending on execution order.
Naming: A method name such as invalidPasswordShowsError communicates the behavior under test.

E. Creating Reports Using TestNG

TestNG creates execution information automatically and supports listener-based custom reporting.

Default output: Results are commonly written to test-output.
Recorded data: Reports include test names, status, duration, exceptions, and method-level details.
Listeners: Implementations of ITestListener can react to events such as test success or failure.
Screenshots: Selenium listeners can capture a screenshot when a WebDriver test fails.
Custom reports: ExtentReports or Allure can consume test events for richer dashboards, while TestNG remains the execution engine.

F. Understanding Annotations of TestNG

Annotations express test lifecycle, ordering, grouping, and data behavior.

Lifecycle order: Suite setup may run before test setup, class setup, method setup, the test, method cleanup, class cleanup, and suite cleanup.
Configuration: @BeforeSuite runs once before the suite; @BeforeClass runs before the first method in a class.
Grouping: @Test(groups = "smoke") assigns a test to a named group.
Dependency: @Test(dependsOnMethods = "createUser") prevents execution when the prerequisite fails.
Data: @DataProvider supplies multiple input sets to one test method.

G. Need of TestNG in Selenium

TestNG supplies structure that Selenium WebDriver alone does not provide.

Browser lifecycle: Configuration annotations consistently start and close browsers around tests.
Regression selection: Groups can run only smoke, sanity, or regression tests.
Parallel execution: Independent browser tests can run concurrently, reducing suite time.
Failure handling: Failed methods and stack traces identify the WebDriver action or assertion that broke.
Parameterized testing: A data provider can test multiple usernames and passwords without duplicating test code.

H. Running the Test

Tests can be launched from an IDE, Maven, or a TestNG suite file.

Maven command:
BASH
```
  mvn clean test
```
IDE execution: Run a class, method, or testng.xml as a TestNG test.
Suite XML: The file can list test classes and define parallel mode or thread count.
Exit status: Maven normally returns a nonzero status when tests fail, allowing CI to reject the build.
Repeatability: Browser versions, drivers, environment URLs, and credentials should be controlled through configuration.

I. Checking Reports Created by TestNG

Generated reports should be inspected for both outcome and diagnostic detail.

Main location: Open test-output/index.html after a TestNG run.
Status meaning: Passed indicates successful completion; failed indicates an exception or assertion failure; skipped indicates that execution was bypassed.
Failure detail: Read the assertion message and stack trace to identify the expected and actual values.
Timing: Long-running tests can expose slow page loads, waits, or environmental problems.
Test hygiene: A report containing many skipped tests may indicate a failed dependency or incorrect suite selection.

J. Generating HTML Reports

HTML reporting converts raw test events into a navigable summary for developers and managers.

Default report: TestNG’s HTML output includes suite and method summaries under test-output.
Maven integration: The Surefire Report Plugin can transform test results into browsable Maven reports.
Report command:
BASH
```
  mvn surefire-report:report
```
Useful content: A good report shows pass/fail totals, execution time, failed assertions, and links to details.
CI use: Publish the generated HTML directory as a build artifact so results remain available after the job finishes.

K. Annotations Used in TestNG

The most frequently used annotations correspond to test execution stages and test metadata.

@Test: Marks a test method or class; attributes include priority, groups, and dependsOnMethods.
@BeforeMethod / @AfterMethod: Run around every test method, making them suitable for browser setup and cleanup.
@BeforeClass / @AfterClass: Run around methods in one class.
@BeforeSuite / @AfterSuite: Run once for the complete suite.
@DataProvider: Supplies named rows of test data to a test method.

L. Validating Tests with Assertions

Assertions compare actual application behavior with an expected result and determine whether a test is correct.

Hard assertion: Assert.assertEquals(actual, expected) throws immediately when values differ.
Boolean assertion: Assert.assertTrue(driver.getTitle().contains("Dashboard")) verifies a condition.
Null assertion: Assert.assertNotNull(element) checks that an object was located successfully.
Message value: Assert.assertEquals(actual, expected, "Unexpected page title") improves failure diagnosis.
Soft assertions: SoftAssert collects multiple failures and requires softAssert.assertAll() to report them; forgetting this call can incorrectly produce a passing test.

M. Creating Multiple Tests

Multiple tests should represent distinct behaviors while sharing only carefully controlled setup.

Separate methods: validLogin, invalidLogin, and logout can be independent @Test methods.
Separate classes: Organize tests by feature, such as LoginTest and CheckoutTest.
Group selection: @Test(groups = {"smoke", "login"}) permits execution by either category.
Isolation rule: Tests should not rely on data created by a previous test unless dependency is explicit.
Parallel concern: Shared static drivers or mutable test data can cause race conditions when tests run concurrently.

N. Prioritizing Tests

Priorities influence execution order when tests are otherwise eligible to run.

Syntax: @Test(priority = 1) normally runs before @Test(priority = 2).
Default: Tests without a priority generally have priority 0.
Negative values: priority = -1 runs before priority 0.

Example: