Connections

Connections Overview

In MadConnect, Connections represent the bridge between a data source and a destination, using preconfigured connectors to define how and what data is transferred. While connectors define the integration capabilities with a platform, connections are the actual instances where data flow happens.

This page provides a client-facing overview of what connections are, how they function, and how to configure and manage them in the MadConnect platform.

What Is a Connection?

A connection is the specific pairing of a source connector and a destination connector configured to transfer a defined dataset according to a platform schema. It contains the metadata, credentials, and rules necessary for executing that transfer securely and accurately.

For example:

  • A connection might sync hashed emails from Snowflake to Meta Custom Audiences.

  • Another might push server-side conversion events from BigQuery to Google Ads.

  • A third might pull campaign reporting from The Trade Desk into Snowflake.

Key Components of a Connection

  • Source Connector: The platform and configuration from which data will be pulled (e.g., Snowflake, S3, Redshift).

  • Destination Connector: The platform to which data will be delivered (e.g., Meta, Google, TTD).

  • Schema Mapping: Field-level mapping between your data source and the expected schema of the destination platform.

  • Authentication: Credentials and API keys required to authorize access to both ends.

  • Transfer Mode: Defines how data is delivered (e.g., full replace, incremental add, daily sync).

  • Scheduling and Logging: Optional timing controls and full visibility into job status, errors, and delivery results.

Creating a Connection in MadConnect

To create a new connection:

  1. Go to My Connections and click Create Connection

  2. Select your Source and Destination platforms

  3. Choose the relevant Connectors

  4. Input authentication details if not already configured

  5. Define schema field mapping (manual or guided wizard)

  6. Add optional metadata or transfer preferences

  7. Save and Activate the connection

Once active, you can initiate the transfer manually or schedule it as needed.

Incremental Loads and Marker Columns in MadConnect

MadConnect supports both manual and scheduled sync modes. To make these syncs efficient, the platform uses a marker column to determine which records or files are new since the last successful transfer.

Marker Columns

A marker column is a field in your source that MadConnect uses as a cursor to detect new or updated data. Supported formats include:

  • Datetime string (e.g., 2025-08-20 14:35:00)

  • Unix timestamp (e.g., 1754463600)

  • Incremental number (e.g., load_id = 123)

When you configure a connection, you specify the marker column in the Marker Column field. MadConnect stores the last successful marker value and uses it to filter data for the next sync.

How Incremental Loads Work

Table-based Sources

For tables (e.g., in Snowflake), MadConnect pulls only the rows where the marker column value is greater than the last stored value.

  • Example: If load_id is set as the marker column:

    • First sync pulls all records with load_id = 1.

    • On the next sync, only rows with load_id > 1 are processed.

File-based Sources

For storage systems (e.g., S3 buckets), MadConnect uses the file’s modified timestamp as the marker.

  • After a successful transfer, MadConnect stores the highest modified timestamp.

  • On the next run, only files with a later modified timestamp are included.

Sync Modes

Manual Sync

  • Triggered on demand by the user.

  • Uses the last stored marker value to determine which records/files to pull.

  • Best for testing, troubleshooting, or one-time loads.

  • Limitation: Not automated — you must remember to run it.

Scheduled Sync

  • Runs automatically on the schedule you set (e.g., every 24 hours).

  • Uses the last stored marker value from the previous run.

  • Best for continuous pipelines and production workloads.

  • Limitation: If marker column values are not strictly increasing or consistent, records may be skipped or duplicated.

Benefits of Incremental Loads

  • Efficiency: Only new/updated data is processed, reducing load times and costs.

  • Reliability: Keeps transfers consistent and avoids re-processing old data.

  • Flexibility: Works with multiple formats (datetime, timestamp, numeric IDs).

Key Considerations

  • The marker column must have monotonic behavior (values always increase or strictly represent newer data).

  • Backfills or out-of-order data may require a full reload.

  • A failed transfer does not advance the marker — data will be retried on the next run.

⚡ Tip: For most use cases, we recommend a numeric load_id or a reliable timestamp column as the marker field.

Managing Connections

  • View your connections in the Active or In Progress tabs

  • Use the Reports tab to monitor transfer history, delivery status, and failures

  • Edit, pause, or remove connections at any time

  • Re-authenticate if credentials expire

Why Connections Matter

  • Visibility: Know exactly how your data is flowing across platforms

  • Reusability: Reconfigure existing connections as your campaign strategy evolves

  • Reliability: Connections inherit the schema validation and retry logic from MadConnect’s core framework

Need help configuring your first connection? See Initiate First Transfer or reach out to [email protected].

PreviousConnectorsNextDeployment Options

Last updated