Back to Subject

Unit 2 - Notes

INT347 8 min read

Unit 2: N8N Foundations and Core Building Blocks

1. N8N Architecture and Deployment

N8N Architecture Overview

n8n (nodemation) is an extendable, workflow automation tool built on Node.js. It operates on a "fair-code" model, meaning its source code is open, but commercial usage is restricted under certain conditions.

  • Event-Driven Engine: Workflows in n8n are triggered by events (e.g., an incoming HTTP request, a scheduled time, or an event in a third-party app).
  • Execution Engine: Processes data item-by-item or in batches, passing structured JSON data sequentially from one node to the next.
  • Pluggable Architecture: Integrations (nodes) are standalone modules that interface with the core execution engine, making it easy to create custom nodes.

Cloud vs. Self-Hosted Deployment

  • n8n Cloud:
    • Managed Infrastructure: Hosted directly by n8n. No server setup, maintenance, or security patching is required.
    • Ease of Use: Instant setup, automatic updates, and built-in scaling.
    • Limitations: Recurring subscription costs, strict limits on the number of workflow executions per month, and less control over deep backend configurations or custom environment variables.
  • Self-Hosted (On-Premises):
    • Complete Control: Installed via Docker, npm, or cloud providers (AWS, DigitalOcean). You own the data, ensuring strict compliance (GDPR, HIPAA).
    • Cost-Effective: Free to use under the fair-code license for internal company use; you only pay for the server hosting the instance.
    • Customization: Ability to install community nodes, add custom npm packages to the Code node, and connect directly to internal/local databases securely.
    • Limitations: Requires technical expertise for deployment, securing, updating, and scaling.

2. The N8N Interface

Canvas

The visual editor where workflows are designed. It provides a drag-and-drop environment with an infinite grid. Users can zoom, pan, and organize their automation logic visually.

Nodes

The foundational building blocks of any n8n workflow. Each node represents a specific action (e.g., fetching data, transforming text, sending an email). A node takes input data, performs a specific operation, and outputs data.

Connections

The visual wires linking nodes together. They represent the execution path and the flow of data. Data flows exclusively from the right side (output) of one node to the left side (input) of the next.

3. Data Flow Mechanics and Execution

Data Flow Mechanics

n8n processes data internally as an array of JSON objects called Items.

  • Even if a node processes a single piece of data, n8n treats it as an array containing one object: [ { "json": { "key": "value" } } ]
  • When an input contains multiple items (e.g., an array of 5 user records), the subsequent node will typically execute its operation 5 times—once for each item.

Node Connection Patterns

  • Linear: Node A outputs to Node B, which outputs to Node C.
  • Branching: A single output connects to multiple nodes (e.g., Node A connects to Node B and Node C). Data is duplicated and sent down both paths simultaneously.
  • Merging: Multiple branches converge into a single node (using the Merge node) to combine datasets.

Execution Order

  • Workflows execute from left to right.
  • Execution is triggered by a starting node (Trigger) and follows the connection paths.
  • In branching scenarios, n8n executes one branch fully (or until a wait/pause state) before executing the next branch, though visually it represents parallel processing.

Debugging Techniques

  • Executions Tab: A built-in logging system that shows the history of all workflow runs, displaying success/failure status, execution time, and the exact data passed between nodes at every step.
  • Pinning Data: Allows you to freeze the output data of a node. Useful when building workflows so you don't have to repeatedly call a live API (saving API limits) while testing downstream nodes.
  • Error Nodes: Utilizing the "Error Trigger" workflow to catch global workflow failures, or adjusting node settings to "Continue On Fail" for localized error handling.
  • Console Logging: Using the console.log() command inside a Code node to output data directly to the server terminal/Docker logs for deep inspection.

4. Node Types

Triggers

Trigger nodes initiate the workflow. A workflow must have at least one trigger to run automatically.

  • Manual Trigger: Activated by clicking the "Execute Workflow" button in the canvas. Used primarily for testing and development.
  • Schedule Trigger: Built-in cron/timer functionality. Executes workflows at specific intervals (e.g., every 5 minutes) or specific times (e.g., every Monday at 9:00 AM).
  • Webhook Trigger: Listens for incoming HTTP requests (GET, POST, etc.) from external systems. Provides a unique URL. Excellent for real-time, event-driven automation (e.g., receiving a payload when a user submits a form on a website).

Regular Nodes (Action Nodes)

Perform tasks such as fetching data from an API, creating a record in a CRM, or formatting a string.

Output Nodes

Nodes designed specifically to push data to an end destination (e.g., sending a Slack message, writing to a Google Sheet, returning a Webhook response).

5. Core Logic Nodes

Core nodes manipulate the flow of data rather than interacting with external apps.

  • Set Node:
    • Used to write, update, or delete fields in the JSON data passing through it.
    • Can append new variables (e.g., status = "processed") or be configured to "Keep Only Set" fields, effectively filtering out unnecessary data before passing it to the next step.
  • IF Node:
    • Splits the workflow into a True and False branch based on boolean logic (e.g., IF user.age >= 18).
  • Switch Node:
    • An advanced routing node that directs data down multiple paths based on the value of a specific field (e.g., routing support tickets based on priority: High, Medium, Low).
  • Merge Node:
    • Combines data from multiple branches.
    • Modes include: Append (stacking arrays), Multiplex, and "Keep key matches" (similar to a SQL JOIN, merging two different datasets based on a common ID).

6. HTTP and API Integrations

HTTP Request Node

The most versatile node in n8n, used to interact with any RESTful API, even if n8n doesn't have a dedicated integration node for that app.

  • REST API Methods Supported:
    • GET: Retrieve data from a server.
    • POST: Send new data to a server (e.g., creating a user).
    • PUT/PATCH: Update existing data on a server.
    • DELETE: Remove data from a server.

Authentication Mechanisms

n8n securely stores API keys and passwords in a dedicated "Credentials" vault.

  • API Key: Passed either in the HTTP Header (e.g., x-api-key: your_key_here) or as a Query Parameter in the URL.
  • Bearer Token: A standard HTTP authentication scheme. Uses the Authorization header with the format: Bearer <token>.
  • OAuth2: n8n natively handles the OAuth2 flow (redirecting to login, obtaining the authorization code, and automatically refreshing access tokens).

7. Data Transformation

JSON Data Operations

Data in n8n is strictly JSON. Navigating JSON trees (e.g., accessing nested objects like user.address.city) is a fundamental skill.

Expressions

n8n uses an expression language (similar to JavaScript template literals) to dynamically map data from previous nodes into the current node.

  • Syntax: {{ expression }}
  • Example: {{ $json.firstName }} dynamically inserts the firstName value from the incoming JSON item.
  • Cross-node referencing: {{ $('NodeName').item.json.field }} accesses data from a specific upstream node.

Code Node

When standard nodes cannot achieve the required transformation, the Code node allows writing custom JavaScript (and in newer versions, Python).

  • Scripts iterate over the input array.
  • Example of mapping data in JavaScript:
    JAVASCRIPT
        // Add a new field 'fullName' to every incoming item
    for (const item of $input.all()) {
      item.json.fullName = item.json.firstName + ' ' + item.json.lastName;
    }
    return $input.all();

8. Advanced Integrations

Multi-API Integration

Workflows often chain multiple APIs together.

  • Pattern: Webhook receives an order ID (API 1) -> HTTP Request fetches customer data from CRM using the ID (API 2) -> Slack Node sends a formatted message (API 3).
  • Proper use of the Merge node and Expressions is critical here to ensure data from API 1 is successfully combined with the payload from API 2.

Database Integration (PostgreSQL)

n8n can connect directly to SQL and NoSQL databases. The PostgreSQL node supports native CRUD operations:

  • Insert: Maps incoming JSON fields to database columns to create new rows.
  • Update: Modifies existing rows based on a matching key (e.g., updating a user's email where user_id = 123).
  • Select (Query): Executes custom SQL queries (e.g., SELECT * FROM users WHERE active = true) and outputs the results as an array of JSON items for n8n to process.
  • Delete: Removes records based on specific criteria.

File Processing

Handling non-text data (images, PDFs, CSVs) using binary data management.

  • Binary Data Concept: n8n separates data into json (for text/numbers) and binary (for files).
  • Reading/Writing: Files can be read from local disk (if self-hosted) or downloaded via HTTP Request nodes.
  • Conversion: Tools like the "Spreadsheet File" node can convert a binary CSV or Excel file into a JSON array, allowing n8n to process the rows individually. Conversely, JSON data can be compiled back into a binary file for upload to FTP or Cloud Storage.
Previous
Unit 1
Next
Unit 3