Unit 2: Bug Reporting

I. Orientation

Automated testing is valuable only when detected failures are converted into clear, actionable defect reports. A bug report records the difference between expected and actual behavior, while the broader bug-management process controls investigation, prioritization, correction, verification, and closure. Effective reporting depends on reproducibility, evidence, impact assessment, and communication between testers, developers, and project managers.

• Governing principle: A bug is a deviation from an expected requirement, specification, design, or user-visible behavior.
• Expected result: The behavior defined by requirements, acceptance criteria, test oracles, interface contracts, or documented design.
• Actual result: The behavior observed during execution, including incorrect output, crashes, warnings, performance degradation, or data corruption.
• Reproducibility: A defect should be repeatable under stated conditions, although intermittent bugs still require careful evidence collection.
• Traceability: Every report should connect to a test case, build, requirement, commit, environment, or user workflow.
• Prioritization: Severity describes technical or user impact; priority describes how urgently the team should address the defect.
• Lifecycle control: A report changes state as it moves from discovery through analysis, repair, verification, and closure.

II. Getting Your Bugs Fixed — From Observation to Resolution

A. Getting Your Bugs Fixed

Getting a bug fixed requires more than identifying that something failed; the report must give the responsible team enough information to reproduce, understand, and verify the problem.

• State the failure precisely: Replace “login is broken” with “submitting valid credentials on build 2.4.1 returns HTTP 500 instead of opening the dashboard.”
• Separate facts from conclusions: Record the observed response, such as a stack trace or incorrect value, before claiming that a particular database query is responsible.
• Provide reproducible steps: Number actions in execution order:
  1. Open /login.
  2. Enter a registered email address.
  3. Enter a valid password.
  4. Select Sign in.
• Include expected and actual results: Expected behavior might be “the dashboard loads within 2 seconds”; actual behavior might be “the page remains blank and the browser console reports a JavaScript exception.”
• Attach evidence: Useful evidence includes screenshots, screen recordings, console logs, HTTP traces, test output, timestamps, and minimal input data.
• Identify the affected context: Include operating system, browser, device, application version, test data, configuration flags, and relevant service versions.
• Explain business impact: A payment failure affecting every customer is more urgent than a cosmetic alignment error in an internal report.

III. Isolating and Reproducing Bugs — Reducing the Failure

A. Isolating and Reproducing Bugs

Isolating and reproducing bugs means reducing a complex failure to the smallest set of conditions that consistently triggers it.

• Control variables: Keep the application build, input data, environment, and execution sequence fixed while changing one factor at a time.
• Minimize the input: If a parser fails on a 10 MB file, reduce it to the smallest file that still produces the failure. This helps developers identify the responsible token or record.
• Use a reproduction rate: Record results such as “fails 8 of 10 runs” rather than simply calling the defect intermittent.
• Compare environments: Run the same test on a local machine, CI runner, and staging server. A failure occurring only in CI may indicate a timing, dependency, permission, or configuration issue.
• Capture timing information: Concurrency defects often depend on order. Log timestamps, thread identifiers, request IDs, and event sequences.
• Distinguish symptom from cause: A timeout may be caused by a slow database query, an unavailable service, a deadlock, or an overly short client timeout.
• Use automated reproduction: A regression test can preserve the failure:

arrange: create account with status = "pending"
act:    call activate_account(account_id)
assert: response.status == 200
        account.status == "active"

Here, arrange, act, and assert identify setup, operation, and expected verification.

IV. Not All Bugs Are Created Equal — Evaluating Impact

A. Not All Bugs Are Created Equal

Not all bugs are created equal because defects differ in consequence, frequency, visibility, and urgency.

• Severity: Measures the effect on the system or user.
  • Critical: Data loss, security compromise, payment corruption, or total service outage.
  • High: A major feature is unusable, such as registration failing for all new users.
  • Medium: A workaround exists, but normal operation is impaired.
  • Low: Minor visual, wording, or usability defect.
• Priority: Specifies the order in which work should be scheduled. A low-severity typo on a legal document may receive high priority before release.
• Frequency: A crash occurring on every execution is generally more urgent than one occurring once in 1,000 executions, although rare financial or security failures may still be critical.
• Scope: A defect affecting one account differs from one affecting every tenant or all supported browsers.
• Detectability: Hidden corruption may be more dangerous than an obvious error because users may continue operating with invalid data.
• Risk-based decision: Teams commonly combine impact and likelihood. A useful model is:

Risk = impact × likelihood

Impact represents the consequence of failure, while likelihood represents the probability or frequency of occurrence. This model supports prioritization but does not replace engineering judgment.

V. A Bug’s Life Cycle — States and Transitions

A. A Bug’s Life Cycle

A bug’s life cycle describes the controlled states through which a defect report moves from discovery to final disposition.

• New: The tester has submitted the report, but it has not yet been reviewed.
• Triaged: The team has assessed validity, severity, priority, ownership, and target release.
• Assigned: A developer or team is responsible for investigation and correction.
• In progress: Analysis or implementation work has begun.
• Resolved: The developer has made a change and records a resolution such as fixed, duplicate, invalid, or cannot reproduce.
• Ready for verification: The repaired build is available for retesting.
• Verified: The tester confirms that the original failure no longer occurs and that related behavior remains correct.
• Closed: The report is complete and requires no further action.
• Reopened: The defect still occurs, or the fix introduced a regression.

A typical transition is:

New -> Triaged -> Assigned -> In progress -> Resolved
    -> Ready for verification -> Verified -> Closed

The exact states vary by organization, but transitions should be controlled so that a report cannot be closed without evidence of verification.

VI. Bug-Tracking Systems — Managing Defect Information

A. Bug-Tracking Systems

A bug-tracking system is a shared repository and workflow engine for recording, organizing, discussing, and reporting defects.

• Core record: A bug usually contains an identifier, title, description, steps, expected result, actual result, severity, priority, status, owner, environment, and timestamps.
• Searchability: Structured fields allow queries such as “all open critical bugs assigned to the payment team for release 5.0.”
• History: The system records status changes, comments, field edits, attachments, and responsible users, creating an audit trail.
• Relationships: Reports can be linked as duplicates, regressions, dependencies, blockers, or child tasks.
• Notifications: Assignment and status changes can trigger email, chat, or dashboard notifications.
• Metrics: Teams can calculate open-defect count, average resolution time, reopen rate, and defects by release.
• Workflow enforcement: Permissions and required fields reduce incomplete reports and prevent unauthorized closure.
• Integration: Modern systems may connect to source control, CI pipelines, test-management tools, and deployment systems through APIs or webhooks.

VII. Manual Bug Reporting and Tracking — Human-Centered Investigation

A. Manual Bug Reporting and Tracking

Manual bug reporting and tracking relies on a tester or user to observe a problem, document it, submit it, and follow its progress.

• Manual discovery: Exploratory testing may reveal an unexpected result that scripted tests do not cover, such as a confusing navigation path.
• Human context: A tester can describe usability concerns, suspicious behavior, or business consequences that automated output may not express clearly.
• Reporting sequence: The tester normally reproduces the issue, checks for duplicates, completes the report, assigns an initial severity, and submits supporting evidence.
• Tracking responsibility: The reporter monitors developer questions, retests the fix, and reopens the issue if the failure remains.
• Strength: Human observation is flexible and useful for novel, visual, contextual, and exploratory defects.
• Limitation: Manual reports can vary in detail, be delayed, omit exact environment data, or be affected by memory and interpretation.
• Concrete practice: A report should identify “Chrome 124 on Ubuntu 22.04, build a81f3c, viewport 1440x900,” rather than merely stating “browser test failed.”

VIII. Automated Bug Reporting and Tracking — Evidence at Execution Time

A. Automated Bug Reporting and Tracking

Automated bug reporting and tracking connects test execution with defect creation or update, allowing failures to retain precise machine-generated evidence.

• Automatic capture: A failed CI test can record the test name, assertion, stack trace, commit SHA, build number, duration, and environment variables.
• Failure example: An assertion may report:

Expected: 201
Received: 500
Test: POST /orders with valid cart
Commit: 7f3a2d1

201 is the expected HTTP “Created” status, while 500 indicates an internal server error.

• Deduplication: Systems can group failures by normalized stack trace, test identifier, error message, or signature to avoid creating hundreds of duplicate reports.
• Automatic routing: Ownership rules can assign failures to the team responsible for a service, module, or source path.
• Integration: CI tools can create or update tickets when a pipeline fails, and can attach logs or link directly to the failing build.
• Strength: Automation improves speed, consistency, and traceability for repeatable regressions.
• Limitation: A failing assertion is not automatically a confirmed product bug; the test, environment, dependency, or test data may be defective.
• Human review: Automated reports should enter triage rather than bypass it, especially when transient infrastructure failures are common.

IX. Introduction to Bugzilla — An Open-Source Defect Tracker

A. Introduction to Bugzilla

Bugzilla is an open-source web-based bug-tracking system originally developed for managing Mozilla project defects. It organizes reports through searchable fields, workflow states, comments, attachments, and notifications.

• Bug identity: Each report receives a unique numeric ID, allowing references such as “Bug 1842” in commits, test results, and discussions.
• Classification: Products and components categorize ownership; for example, product WebApp may contain component Authentication.
• Important fields: Common fields include summary, description, severity, priority, version, operating system, platform, assignee, status, and resolution.
• Workflow: A report may begin as UNCONFIRMED or NEW, move to ASSIGNED, then become RESOLVED; verification may lead to VERIFIED and finally CLOSED.
• Resolution values: Typical resolutions include FIXED, DUPLICATE, INVALID, WONTFIX, and WORKSFORME, each representing a different disposition.
• Dependencies: A bug can block another bug, such as a database migration defect blocking a release-level deployment task.
• Attachments and comments: Logs, patches, screenshots, and developer discussions remain associated with the report.
• Queries and reports: Saved searches can display open high-severity defects for a release, while dashboards show trends such as newly filed versus resolved bugs.
• Good Bugzilla practice: Use a concise summary, exact reproduction steps, build information, expected and actual results, and a minimal attachment. Link duplicates rather than creating parallel discussions.