Unit 3: Disassembly and Malware Debugging - Subjective Questions

Static code analysis is the examination of a suspicious executable without running it. The analyst studies the file structure, machine instructions, strings, metadata, and imported functions to infer the malware's behavior.

Objectives:

• Identify the executable format, architecture, and compiler characteristics.
• Discover malicious capabilities such as persistence, data theft, process injection, or network communication.
• Locate important functions and control-flow paths.
• Extract indicators of compromise, including domains, file names, registry paths, and mutex names.

Advantages:

• The malware is not executed, reducing the risk of accidental infection.
• Every instruction and possible execution path can potentially be examined.
• It can reveal functionality that activates only under special conditions.
• It supports the creation of signatures and detection rules.

Limitations:

• Packed or encrypted code may not be visible.
• Indirect calls and obfuscated control flow are difficult to interpret.
• Static analysis does not directly show runtime values or environmental interactions.
• Large binaries may require considerable time and expertise.

Therefore, static analysis is generally combined with controlled dynamic analysis and debugging for a complete understanding of malware.

A systematic static-analysis procedure includes the following stages:

1. Preserve the sample:

   • Work on a copy in an isolated environment.
   • Calculate cryptographic hashes such as MD5, SHA-1, and SHA-256 for identification.

2. Identify the file type:

   • Verify whether the file is a Portable Executable, script, document, or archive.
   • Determine whether it is a 32-bit or 64-bit binary.

3. Inspect metadata:

   • Examine file size, timestamps, digital signatures, compiler information, and version resources.
   • Treat timestamps cautiously because malware can modify them.

4. Examine strings:

   • Search ASCII and Unicode strings for URLs, commands, registry paths, error messages, file names, and encoded data.

5. Analyze the PE structure:

   • Review headers, sections, entry point, section permissions, imports, exports, and resources.
   • Look for abnormal section names, high entropy, and mismatched raw and virtual sizes.

6. Inspect imports and exports:

   • Use imported Windows API functions to infer possible behavior.

7. Disassemble the code:

   • Identify functions, loops, branches, API calls, and important cross-references.
   • Construct an approximate control-flow graph.

8. Document findings:

   • Record indicators, suspected capabilities, uncertain conclusions, and locations requiring debugging.

This staged approach moves from inexpensive file inspection to detailed instruction-level analysis.

The Portable Executable format organizes Windows executables and libraries. Important components include:

• DOS header: Begins with the MZ signature and contains a pointer to the PE header.
• PE signature: Usually represented as PE followed by two null bytes; it confirms the PE structure.
• COFF file header: Describes the target architecture, number of sections, timestamp, and file characteristics.
• Optional header: Contains essential loading information, including:
  • Address of Entry Point
  • Image Base
  • Section Alignment
  • File Alignment
  • Size of Image
  • Subsystem
  • Data directory locations
• Section table: Describes sections such as .text, .data, .rdata, .rsrc, and .reloc, together with their sizes, offsets, and permissions.
• Import directory: Lists DLLs and functions required by the program.
• Export directory: Lists functions made available to other modules.
• Resource directory: May contain icons, dialogs, configuration data, or embedded payloads.
• Relocation directory: Holds address adjustments required when the image is not loaded at its preferred base.
• Thread-local storage directory: May contain callbacks that execute before the normal entry point.

Analysts use these fields to locate executable code, detect packing, identify APIs, find hidden resources, and determine the binary's initial execution path.

The three tools provide different representations and methods of examining a binary:

• Disassembler:

  • Converts machine-code bytes into assembly instructions.
  • Does not normally execute the program.
  • Shows low-level operations, registers, branches, calls, and memory references.
  • It is useful for precise static analysis and for locating suspicious code.

• Decompiler:

  • Attempts to translate machine code into high-level pseudocode resembling C.
  • Reconstructs variables, loops, conditions, and function arguments heuristically.
  • It improves readability but may produce inaccurate types, names, or control structures.

• Debugger:

  • Executes the binary under analyst control.
  • Supports breakpoints, single stepping, register inspection, memory examination, and runtime modification.
  • It is useful for observing decoded data, resolved APIs, unpacked code, and actual execution decisions.

A disassembler gives an instruction-level static view, a decompiler gives an approximate high-level view, and a debugger provides runtime evidence. Malware analysts often use all three together.

During disassembly, an analyst interprets instructions according to their effects on processor state.

Instructions:

• MOV transfers data between registers and memory.
• LEA calculates an address and may also be used for arithmetic.
• PUSH and POP add or remove values from the stack.
• CALL transfers control to a function and saves a return address.
• RET returns control to the caller.
• CMP and TEST update flags for a later conditional jump.
• JMP, JZ, JNZ, and related instructions control execution flow.

Registers:

• General-purpose registers hold values, addresses, arguments, or intermediate results.
• On x86, common registers include EAX, EBX, ECX, EDX, ESI, and EDI.
• On x64, their extended forms include RAX, RBX, RCX, and RDX.
• ESP or RSP points to the stack top, while EBP or RBP may act as a stack-frame base.
• EIP or RIP identifies the current instruction location.

Flags:

• The Zero Flag, Carry Flag, Sign Flag, and Overflow Flag influence conditional branches.

Stack:

• Stores return addresses, local variables, saved registers, and sometimes function arguments.
• Tracking stack changes helps reconstruct function calls and parameters.

Understanding these elements allows the analyst to convert low-level instructions into meaningful program behavior.

A control-flow graph (CFG) is a representation of the possible order in which instructions or groups of instructions can execute.

• A basic block is a sequence of instructions with one entry point and one exit point. Execution enters at the first instruction and continues without branching until the end of the block.
• A conditional branch creates two or more possible paths based on a condition.
• An unconditional branch transfers execution directly to another location.
• A loop appears as an edge returning to an earlier basic block.
• A function call transfers control to another function and normally returns to the following instruction.
• A cross-reference shows where an address, function, string, or data item is used.

CFGs help analysts:

• Identify decision-making logic and repeated operations.
• Locate validation checks, decryption loops, and error paths.
• Distinguish important execution paths from irrelevant library code.
• Detect abnormal control-flow patterns introduced by obfuscation.
• Trace how suspicious strings or API calls are reached.

However, indirect jumps, exception-based control flow, and code generated at runtime can prevent a static tool from building a complete CFG.

Imported Windows API functions provide clues about the services that a program may use. Analysts group related APIs to form behavioral hypotheses.

Examples:

• File operations: CreateFile, ReadFile, WriteFile, and DeleteFile may indicate file collection, modification, or destruction.
• Registry operations: RegOpenKeyEx, RegSetValueEx, and RegCreateKeyEx may indicate configuration changes or persistence.
• Process execution: CreateProcess, ShellExecute, and WinExec may launch commands or additional payloads.
• Process injection: OpenProcess, VirtualAllocEx, WriteProcessMemory, and CreateRemoteThread together strongly suggest remote-process injection.
• Networking: InternetOpen, InternetConnect, HttpSendRequest, socket, connect, and send may indicate command-and-control communication.
• Dynamic resolution: LoadLibrary and GetProcAddress may be used to load APIs at runtime.
• Persistence: CreateService, task-scheduling APIs, or registry modification APIs may establish automatic execution.
• Anti-analysis: IsDebuggerPresent, CheckRemoteDebuggerPresent, and timing APIs may be used to detect analysis.

An imported function alone does not prove malicious behavior. Its arguments, surrounding instructions, execution path, and combination with other APIs must also be examined.

The Import Address Table (IAT) is a table in a PE image that holds the runtime addresses of imported functions.

Operation:

1. The PE import directory identifies required DLLs and imported function names or ordinals.
2. When the program is loaded, the Windows loader loads the required DLLs.
3. The loader resolves each imported function to its actual memory address.
4. The resolved addresses are written into the IAT.
5. Program code calls an imported API indirectly through the corresponding IAT entry.

A disassembler parses the PE import structures and assigns symbolic names to IAT locations. Thus, an indirect instruction such as CALL [address] may be displayed as a call to KERNEL32.CreateFileW rather than as an unknown memory reference.

Importance in malware analysis:

• It makes assembly listings easier to understand.
• It provides an initial summary of likely behavior.
• Cross-references to an IAT entry reveal all locations that call a particular API.
• IAT anomalies can indicate packing, manual API resolution, or import-table tampering.

Packed malware may have a very small import table containing only functions such as LoadLibrary and GetProcAddress, because the remaining APIs are resolved dynamically.

Static imports are recorded in the PE import directory before execution. The Windows loader resolves them automatically and places their addresses in the IAT. Static imports are easy for analysis tools to enumerate.

Dynamic API resolution occurs while the program is running. A typical sequence is:

• Load or obtain a handle to a DLL.
• Locate a desired function using GetProcAddress or a custom export-table parser.
• Store the returned address in memory.
• Invoke the function through an indirect CALL instruction.

Malware may dynamically resolve APIs to:

• Hide suspicious capabilities from basic import-table inspection.
• Reduce the number of visible imports.
• Support different Windows versions.
• Delay loading a library until a particular feature is needed.
• Make static disassembly and automated classification more difficult.

Advanced malware may store hashed API names rather than clear-text names. It can enumerate loaded modules, parse DLL export directories, hash each exported name, and compare the result with a stored hash.

Analysts identify dynamic resolution by locating LoadLibrary, GetProcAddress, export-table parsing loops, API-name strings, hash constants, and indirect calls through writable memory.

A calling convention defines how a function receives arguments, returns a value, preserves registers, and removes arguments from the stack.

Important aspects include:

• The order in which arguments are passed.
• Whether arguments are passed through the stack or registers.
• Whether the caller or callee cleans the stack.
• Which registers must be preserved.
• Where the return value is stored.

Common 32-bit conventions:

• cdecl: Arguments are generally pushed right to left, and the caller cleans the stack.
• stdcall: Arguments are generally pushed right to left, and the callee cleans the stack. Many 32-bit Windows APIs use this convention.
• fastcall: Some arguments are passed in registers.
• thiscall: Commonly used for C++ member functions, with the object pointer passed specially.

Windows x64 convention:

• The first four integer or pointer arguments are normally passed in RCX, RDX, R8, and R9.
• Additional arguments are placed on the stack.
• The return value is commonly placed in RAX.

Recognizing the convention allows an analyst to reconstruct API arguments, distinguish functions, interpret stack frames, and determine the meaning of return values. Incorrect assumptions may lead to a false interpretation of the malware's behavior.

Debugging is the controlled execution of a program so that its runtime state and behavior can be observed or modified.

General concepts:

• Breakpoint: Pauses execution at a selected instruction, function, or event.
• Single stepping: Executes one instruction or one source-level operation at a time.
• Step into: Enters a called function so that its instructions can be examined.
• Step over: Executes a called function without tracing each internal instruction.
• Step out: Continues execution until the current function returns.
• Register inspection: Shows instruction pointers, stack pointers, argument registers, flags, and computed values.
• Memory inspection: Displays code, stack data, heap objects, strings, buffers, and loaded modules.
• Execution control: Includes running, pausing, restarting, terminating, and changing the next instruction.
• Patching: Temporarily changes instructions or data in memory to test a hypothesis.

In malware analysis, debugging can reveal decrypted strings, dynamically resolved APIs, unpacked code, network parameters, hidden branches, and payloads generated only at runtime. It must be performed in a properly isolated environment.

Software breakpoints:

• Replace the target instruction byte with a trap instruction, commonly INT 3 on x86 and x64.
• They are easy to create and many can be used.
• They modify code memory and can therefore be detected by integrity checks.
• They may be unsuitable for read-only, self-modifying, or frequently rewritten code.

Hardware breakpoints:

• Use processor debug registers to monitor a specific address.
• They can break on execution, memory reads, or memory writes, depending on processor support.
• They do not alter the target instruction bytes.
• Only a small number are available at one time, commonly four address slots on x86-family processors.
• Malware may detect or clear debug-register values.

Memory breakpoints:

• Often rely on page permissions or guard pages to detect access to a memory region.
• They are useful for monitoring buffers, unpacked code, or large areas.
• They may produce many exceptions because protection applies at page granularity.
• Debuggers implement them differently, so behavior and performance may vary.

The breakpoint type should be selected according to whether the analyst needs to monitor code execution, a specific data access, or an entire memory region.

• Step into: Executes the current instruction and enters a called function when the instruction is a call. It is used when the internal behavior of the function is important.

• Step over: Executes the entire called function and pauses at the instruction following the call. It is useful for skipping trusted library functions or unimportant routines.

• Step out: Continues execution until the current function finishes and returns to its caller. It is useful when the analyst has entered a function that is not relevant.

• Run until return: Runs until a return instruction is reached or executed. Depending on the debugger, it may behave similarly to step out but can be based directly on detecting a RET instruction.

In malware debugging, these controls improve efficiency. The analyst can step into suspicious decoding or injection functions, step over standard APIs, and step out of irrelevant runtime-library routines. Care is required because exceptions, callbacks, tail calls, or manipulated return addresses may cause execution to deviate from the expected path.

When execution pauses near a Windows API call, the analyst should perform the following tasks:

1. Identify the architecture and calling convention:

   • In 32-bit code, parameters are commonly found on the stack.
   • In Windows x64 code, the first four parameters are commonly in RCX, RDX, R8, and R9.

2. Inspect arguments before the call:

   • Follow pointer arguments in the memory view.
   • Decode strings as ASCII or Unicode.
   • Interpret constants as access rights, flags, sizes, handles, or protection values.

3. Inspect the stack:

   • Locate the return address, additional arguments, and local variables.
   • Confirm that stack alignment and argument order match the expected convention.

4. Execute or step over the API:

   • Observe the return value, commonly in EAX or RAX.
   • Check whether the API succeeded and whether it produced an output buffer or handle.

5. Inspect side effects:

   • Examine created files, allocated memory, modified buffers, registry changes, or new threads.

For example, before WriteProcessMemory, the analyst can inspect the target process handle, destination address, source buffer, and size. This reveals exactly what data the malware attempts to place in another process.

Windows uses exceptions to report events such as invalid memory access, division by zero, illegal instructions, guard-page access, and breakpoint traps. Structured Exception Handling (SEH) allows a program to install handlers for such events.

Malware may use exceptions for legitimate error handling, but it can also use them to:

• Transfer control to hidden code paths.
• Obfuscate normal program flow.
• Detect whether a debugger changes exception delivery.
• Execute code after intentionally causing an access violation.
• Implement unpacking or anti-disassembly mechanisms.

Debuggers usually distinguish between:

• First-chance exception: The debugger is notified before the program's handler receives the exception.
• Second-chance exception: The debugger is notified again when no program handler has processed it; the process may then terminate.

An analyst should inspect the exception type, faulting instruction, handler address, stack state, and whether the exception is expected. Automatically ignoring all exceptions may skip important control flow, while stopping on every exception may create excessive noise. Exception settings should therefore be adjusted according to observed behavior.

Common anti-debugging techniques include:

• Calling APIs such as IsDebuggerPresent or CheckRemoteDebuggerPresent.
• Querying process information for debugging-related values.
• Inspecting process-environment structures for debugger indicators.
• Checking processor debug registers for hardware breakpoints.
• Measuring execution time with timers to detect pauses caused by stepping.
• Searching for breakpoint bytes or modified code.
• Detecting debugger windows, process names, drivers, or artifacts.
• Using unusual exceptions and checking how they are handled.
• Terminating, sleeping, or changing behavior when analysis is detected.

Recognition methods:

• Search imports and strings for debugging-related APIs or tool names.
• Identify timing comparisons and suspicious conditional branches.
• Observe exits or behavior changes that occur only under a debugger.
• Compare execution inside and outside the debugger in an isolated laboratory.

General handling methods:

• Patch or redirect the conditional branch used by the check.
• Modify the relevant return value or process-state field.
• Use hardware breakpoints when software breakpoints are checked.
• Configure exception handling carefully.
• Use debugger-hiding or instrumentation mechanisms where legally and operationally appropriate.

Every bypass should be documented because modifying execution can alter later behavior and produce misleading conclusions.

A packer compresses, encrypts, or transforms the original program and adds a small unpacking stub. When executed, the stub reconstructs the original code in memory and transfers control to it.

Effects on static disassembly:

• Most original instructions and strings are hidden.
• The import table may contain only a few loader-related APIs.
• Sections may have high entropy or unusual permissions.
• The visible entry point belongs to the unpacking stub rather than the original program.
• Disassembly may contain misleading instructions or data interpreted as code.

Debugging approach:

1. Begin execution at the packed entry point.
2. Observe memory-allocation and memory-protection APIs.
3. Monitor writes to executable memory using suitable breakpoints.
4. Identify loops that decrypt, decompress, or copy code.
5. Watch for a jump or call from the unpacking stub into a newly written code region.
6. Verify that the destination contains stable code, meaningful functions, strings, and API references.
7. Treat the destination as a candidate original entry point (OEP).
8. Dump the reconstructed image from memory and rebuild imports if necessary.
9. Reanalyze the dumped image statically.

The OEP is not merely any jump into another region; it should represent the transition from unpacking logic to the reconstructed original program.

Static analysis:

• Examines the binary without executing it.
• Reveals instructions, embedded data, imports, resources, and possible execution paths.
• Is safer and allows broad code coverage.
• Is weakened by packing, encryption, obfuscation, indirect calls, and runtime-generated code.

Debugger-based dynamic analysis:

• Executes the binary under controlled conditions.
• Reveals actual parameter values, decrypted data, runtime API addresses, and selected execution paths.
• Supports breakpoints, memory monitoring, and runtime patching.
• May miss dormant paths and can be affected by anti-debugging or environmental conditions.

Benefits of combining both approaches:

• Static analysis identifies suspicious locations where breakpoints should be placed.
• Debugging validates whether a suspected code path is actually executed.
• Runtime values clarify ambiguous static instructions.
• Unpacked memory can be dumped and returned to static analysis.
• Static cross-references explain how a value observed at runtime is later used.
• Conflicting observations can be investigated rather than accepted as fact.

Thus, static analysis provides breadth, while debugging provides concrete runtime evidence. Iterating between them produces a more complete and defensible behavioral description.

A safe debugging workflow may be organized as follows:

1. Prepare an isolated laboratory:

   • Use a disposable virtual machine with snapshots.
   • Isolate or carefully simulate networking.
   • Disable shared folders, clipboard integration, and unnecessary host connections.

2. Preserve and identify the sample:

   • Record cryptographic hashes and file metadata.
   • Work only on a copy of the original sample.

3. Perform preliminary static analysis:

   • Determine architecture, PE structure, imports, strings, entry point, and packing indicators.
   • Form hypotheses about likely behavior.

4. Configure monitoring:

   • Prepare process, file-system, registry, and network observation tools.
   • Establish a clean baseline before execution.

5. Set strategic breakpoints:

   • Place breakpoints at the entry point and important APIs related to memory allocation, process creation, injection, persistence, or networking.

6. Execute incrementally:

   • Inspect registers, stack arguments, buffers, and return values.
   • Record each important transition and avoid uncontrolled execution where possible.

7. Handle evasion:

   • Identify anti-debugging, timing checks, delayed execution, and environment checks.
   • Document any patches or state modifications.

8. Capture artifacts:

   • Save decoded strings, configuration data, memory dumps, payloads, API traces, and indicators of compromise.

9. Restore and validate:

   • Revert the virtual machine to a known-clean snapshot.
   • Repeat important observations when necessary to confirm reliability.

The workflow should maintain safety, reproducibility, and a clear distinction between observed facts and analyst inferences.

The API sequence is strongly associated with remote-process injection:

1. OpenProcess obtains a handle to a target process.
2. VirtualAllocEx allocates memory inside that process.
3. WriteProcessMemory copies a payload into the allocated region.
4. CreateRemoteThread starts execution in the target process, often at the address of the copied payload.

Debugger-based verification:

• Break on OpenProcess and inspect the process identifier and requested access rights.
• Determine which process is being targeted.
• Break on VirtualAllocEx and record the requested size and protection flags.
• After the call, record the returned remote-memory address.
• Break on WriteProcessMemory and inspect:
  • Target process handle
  • Destination address
  • Local source buffer
  • Number of bytes to write
• Dump or disassemble the source buffer to determine whether it contains shellcode, a PE image, configuration data, or another structure.
• Break on CreateRemoteThread and inspect the start address and parameter.
• Verify whether the start address lies within the allocated remote region or points to another loader routine.
• Observe the resulting process and thread behavior using system-monitoring tools.

Although the sequence is a strong indicator, the final conclusion must be based on actual arguments and runtime effects because legitimate security or management software may use similar APIs.