Unit 5: Performance Testing Tools

I. Orientation — Principles of Performance Testing

Performance testing evaluates how a system behaves under a defined workload. It measures responsiveness, throughput, stability, scalability, and resource consumption rather than merely checking functional correctness.

• Workload model: Represents realistic users, requests, data, pacing, concurrency, and test duration.
• Load testing: Measures behavior under expected and peak operational demand.
• Stress testing: Increases demand beyond capacity to identify limits and failure behavior.
• Spike testing: Applies sudden workload changes to evaluate recovery and elasticity.
• Endurance testing: Runs for an extended period to expose memory leaks, resource exhaustion, or degradation.
• Scalability testing: Determines whether additional resources produce proportional capacity gains.
• Core measurements:
  • Response time: Time between sending a request and receiving its complete response, usually in milliseconds.
  • Latency: Time until the first response byte is received.
  • Throughput: Completed requests or transactions per unit of time.
  • Error rate: Percentage of unsuccessful samples.
  • Percentile: Response-time boundary met by a stated proportion of requests, such as the 95th percentile.
• Repeatability: A valid test controls software versions, test data, environment, workload, and monitoring conditions.
• Non-functional validation: Results are compared with measurable criteria, such as “95% of requests complete within 800 ms.”

II. Performance Testing Tool Selection — Licensing and Capability

A. Open-Source versus Commercial Performance Testing Tools

Tool selection balances cost, protocol support, usability, scalability, reporting, integration, and vendor assistance.

1. Open-source tools:
   • Examples: Apache JMeter, Gatling, k6, and Locust provide source-accessible or community editions.
   • Cost: Licensing expense is generally low, although infrastructure, maintenance, scripting, and training still cost money.
   • Extensibility: JMeter supports Java-based plugins, custom samplers, scripting, and command-line automation.
   • Support model: Assistance commonly comes from documentation, community forums, and third-party providers.
2. Commercial tools:
   • Examples: OpenText LoadRunner Professional and Tricentis NeoLoad offer enterprise-oriented capabilities.
   • Protocol coverage: Commercial suites may support specialized enterprise protocols and recording technologies.
   • Operational features: Centralized test management, hosted load generation, analytics, and vendor support may be included.
   • Trade-off: Faster organizational adoption may be offset by license cost and vendor dependence.
   • Selection criterion: A proof of concept should confirm protocol compatibility, maximum load, scripting effort, CI integration, and reporting before adoption.

III. JMeter Foundations — Purpose, Benefits, and Setup

A. Introduction to JMeter

Apache JMeter is a Java-based open-source application for generating load and measuring the performance of servers, services, and applications.

• Origin: It is an Apache Software Foundation project initially developed for web application testing.
• Supported targets: Common samplers cover HTTP/HTTPS, JDBC, FTP, LDAP, TCP, Java, JMS, and mail protocols.
• Operating model: Virtual users execute sampler requests according to a tree-structured test plan.
• Scope: JMeter operates mainly at the protocol level; it does not render pages or execute browser JavaScript like a real browser.
• Execution modes: GUI mode supports plan construction and debugging, while non-GUI mode supports reliable load generation.

B. Advantages of JMeter

JMeter is valuable because it combines broad protocol support with automation and extensibility.

• Portability: The Java implementation runs on Windows, Linux, and macOS with a compatible Java runtime.
• Cost and access: Apache licensing permits use and modification without per-user load-testing licenses.
• Parameterization: CSV files, variables, functions, and extractors support varied requests and correlated sessions.
• Automation: Command-line execution integrates with CI systems such as Jenkins or GitHub Actions.
• Extensibility: Plugins and JSR223 scripts can add listeners, thread groups, functions, or custom behavior.
• Limitations: Large GUI listeners consume memory, and JMeter does not reproduce browser rendering, layout, or client-side performance.

C. Download and Installation Process of JMeter

Installation consists of preparing Java, obtaining the official binary archive, and verifying execution.

• Prerequisite: Install a Java version supported by the selected JMeter release and confirm it with java -version.
• Download: Obtain the binary archive from the official Apache JMeter download page and verify its published checksum.
• Installation: Extract the archive; JMeter does not require a traditional installer.
• Startup:
  • Windows: Run bin\jmeter.bat.
  • Linux/macOS: Run bin/jmeter.
• Verification: Confirm command-line availability with:

jmeter --version

• Environment: JAVA_HOME may be configured when Java cannot be located automatically.

D. Understanding JMeter GUI

The GUI represents a test as a hierarchical tree and provides editors for the selected component.

• Test tree: The left pane displays the Test Plan, Thread Groups, samplers, controllers, and supporting elements.
• Configuration pane: The right pane exposes fields belonging to the selected tree node.
• Toolbar: Icons provide open, save, start, stop, shutdown, and clear-result operations.
• Status area: Running-thread counts and warning indicators help monitor execution.
• Usage rule: The GUI should be used for design and debugging, not substantial load generation.

IV. JMeter Test Structure — Components and Configuration

A. Elements of JMeter

JMeter elements cooperate to define users, requests, logic, validation, and results.

• Thread Group: Defines virtual-user threads, ramp-up time, iterations, duration, and scheduling.
• Sampler: Sends a request, such as an HTTP Request or JDBC Request.
• Listener: Collects or displays results through components such as Summary Report or View Results Tree.
• Timer: Delays requests to model user pacing or regulate throughput.
• Assertion: Checks whether a sampled response satisfies an expected condition.
• Controller: Determines execution order, repetition, branching, or grouping.
• Pre/Post-Processor: Modifies a request before execution or extracts data after its response.

B. Configuration Elements of JMeter

Configuration elements supply defaults, reusable data, and runtime settings to samplers within their scope.

• HTTP Request Defaults: Centralizes protocol, host, port, path, timeout, or implementation settings.
• HTTP Header Manager: Adds headers such as Content-Type: application/json or authorization tokens.
• HTTP Cookie Manager: Stores and returns cookies, enabling session behavior.
• CSV Data Set Config: Reads parameter values such as usernames from a CSV file.
• JDBC Connection Configuration: Creates a named database connection pool for JDBC samplers.
• Scoping: An element normally affects samplers beneath or within its relevant branch, so tree placement changes behavior.

C. Test Plan and Workbench

The Test Plan is the executable root of a JMeter script, whereas WorkBench was historically a temporary construction area.

• Test Plan: Stores user variables and contains the executable Thread Groups and test components.
• Execution hierarchy: Elements run according to their tree position, controller logic, and scope.
• Workbench history: Older JMeter versions provided WorkBench for temporary HTTP proxy and non-test components.
• Current practice: WorkBench was removed from modern JMeter; recording and support elements are added directly under the Test Plan.

D. Adding elements

Elements are added through the tree’s context menu and positioned according to their required scope.

• Procedure: Right-click a parent node, select Add, choose a category, and select the element.
• Valid placement: Add an HTTP Request beneath a Thread Group and a Response Assertion beneath the sampler it validates.
• Organization: Use descriptive names such as Login Transaction instead of retaining generic labels.
• Reordering: Drag elements or use cut-and-paste operations while preserving intended parent-child relationships.

E. Elements Configuration

Element configuration converts a tree structure into a reproducible workload.

• Thread settings: With 100 threads and a 50-second ramp-up, JMeter starts users at an average rate of two per second.
• Sampler fields: Configure method, endpoint, parameters, body, redirects, connection timeout, and response timeout.
• Variables: Reference a value using ${variableName}, such as ${baseUrl}.
• Validation: Use View Results Tree only during debugging to inspect the request, response, headers, and assertion outcome.
• Isolation: First verify one user and one iteration before increasing concurrency.

V. Test Plan Lifecycle — Creation, Execution, and Reporting

A. Creation of Performance Test Plan using JMeter

A performance test plan translates workload requirements into executable users, transactions, data, timing, and acceptance criteria.

• Objective: Define target throughput, concurrency, duration, response-time percentiles, and acceptable error rate.
• Model: Add a Thread Group, configuration elements, controllers, timers, samplers, extractors, and assertions.
• Transactions: Group business actions with Transaction Controllers, such as login, search, and checkout.
• Data: Parameterize accounts and inputs so concurrent users do not repeatedly submit identical values.
• Monitoring: Correlate JMeter results with server CPU, memory, network, database, and application metrics.
• Baseline: Run a small validation test before the full load to detect script or environment errors.

B. Creating and Running JMX file

A JMX file is JMeter’s XML representation of the complete test-plan tree.

• Creation: Build the plan in the GUI and save it with the .jmx extension.
• GUI run: Select the Start command for a low-load validation execution.
• Non-GUI run:

jmeter -n -t checkout.jmx -l results.jtl

• Option meanings: -n selects non-GUI mode, -t names the JMX plan, and -l stores sample results.
• Runtime properties: A value can be supplied with -Jusers=100 and referenced as ${__P(users,1)}.

C. Saving a Test Plan

Saving preserves component configuration and tree structure for repeatable execution.

• File format: Save the root plan as a .jmx file through File > Save Test Plan As.
• Dependencies: Keep required CSV data, certificates, scripts, and plugins under controlled paths.
• Version control: Commit JMX files with related test data and configuration while excluding generated result files.
• Security: Do not embed production passwords, tokens, or private keys in the plan.

D. Creating Test Reports from running test plan

JMeter can generate an HTML dashboard from JTL sample results after or during a non-GUI execution.

• Direct generation:

jmeter -n -t checkout.jmx -l results.jtl -e -o report

• Option meanings: -e generates the dashboard after execution, while -o specifies an empty output directory.
• Existing results:

jmeter -g results.jtl -o report

• Report measures: Inspect sample count, errors, throughput, response-time distribution, percentiles, and time-series graphs.
• Interpretation: A low average can conceal slow requests; percentiles and error details should be evaluated together.

VI. Workload Behavior — Timing, Validation, Logic, and Data Flow

A. Understanding JMeter Timers

Timers introduce pacing or regulate request rates so virtual users do not send requests unrealistically fast.

• Constant Timer: Adds the same delay, such as 2,000 ms, before each affected sampler.
• Uniform Random Timer: Adds variable delay to avoid synchronized request patterns.
• Constant Throughput Timer: Attempts to maintain a specified sample rate, typically expressed in samples per minute.
• Scope: A timer applies to samplers in its scope; multiple applicable timer delays are combined.
• Accuracy: Timers model protocol-level pacing, so workload validation must compare achieved throughput with the target.

B. Using Assertions in JMeter

Assertions determine whether a technically completed response is functionally acceptable.

• Response Assertion: Checks response text, headers, code, or message using string or pattern matching.
• Duration Assertion: Fails a sample exceeding a defined response time, such as 1,000 ms.
• Size Assertion: Verifies that response data meets expected size constraints.
• JSON JMESPath Assertion: Validates values extracted from JSON responses.
• Failure effect: An assertion marks the sample unsuccessful but does not necessarily stop the thread unless error-handling behavior does so.
• Efficiency: Validate decisive content, such as status code 200 and "orderStatus":"CONFIRMED", without excessively scanning large bodies.

C. Using Controllers in JMeter

Controllers organize samplers and determine their execution sequence or frequency.

• Loop Controller: Repeats its children a specified number of times.
• If Controller: Executes children when its evaluated condition is true.
• Once Only Controller: Runs its children once per thread, commonly for login initialization.
• Transaction Controller: Measures several requests as one business transaction.
• Random Controller: Selects one child path to represent varied user behavior.
• Critical distinction: Logical Controllers alter flow; they do not themselves send requests.

D. Processors in JMeter

Processors prepare requests and extract response data for subsequent operations.

1. Pre-Processors:
   • Purpose: Execute immediately before a sampler within scope.
   • Example: A JSR223 PreProcessor can calculate a timestamp or construct a signature.
2. Post-Processors:
   • Purpose: Execute after a sampler and support correlation.
   • Example: A JSON Extractor can capture access_token into ${token} for a later Authorization header.
   • Ordering: Placement and scope determine which sampler’s data a processor can access.
   • Scripting choice: Groovy through JSR223 is generally appropriate for custom logic because compiled-script caching improves repeated execution.

VII. Distributed Load Generation — Coordinated JMeter Execution

A. Distributed Testing with JMeter

Distributed testing coordinates one JMeter controller with multiple remote engines when one machine cannot generate the required load.

• Architecture: The controller starts the test and gathers results; remote engines execute the JMX workload.
• Setup: Install compatible Java and identical JMeter versions and plugins on every node.
• Connectivity: Configure reachable addresses and required Java RMI ports, then start jmeter-server on each engine.
• Execution:

jmeter -n -t test.jmx -R load1,load2 -l results.jtl

• Load calculation: Thread counts apply per engine; 200 threads on three engines produce up to 600 concurrent JMeter threads.
• File distribution: JMeter does not automatically distribute every external CSV, script, certificate, or plugin; dependencies must exist on each engine.
• Clock and network control: Synchronize system clocks and place generators near the target network to make timestamps and latency meaningful.
• Result efficiency: Return only necessary sample data because sending detailed responses to the controller can become a bottleneck.
• Operational caution: The controller and engines require adequate CPU, memory, file descriptors, and network capacity so the load infrastructure does not limit the test.