Skip to main content

Marketo

This page contains the setup guide and reference information for the Marketo source connector.

Prerequisites​

  • (Optional) Whitelist Airbyte's IP address if needed
  • An API-only Marketo User Role
  • An Airbyte Marketo API-only user
  • A Marketo API Custom Service
  • Marketo Client ID & Client Secret
  • Marketo Base URL

Setup guide​

Step 1: Set up Marketo​

Step 1.1: (Optional) whitelist Airbyte's IP address​

If you don't have IP Restriction enabled in Marketo, skip this step.

If you have IP Restriction enabled in Marketo, you'll need to whitelist the IP address of the machine running your Airbyte instance. To obtain your IP address, run curl ifconfig.io from the node running Airbyte. You might need to enlist an engineer to help with this. Copy the IP address returned and keep it on hand.

Once you have the IP address, whitelist it by following the Marketo documentation for allowlisting IP addresses for API based access.

Step 1.2: Create an API-only Marketo User Role​

Follow the Marketo documentation for creating an API-only Marketo User Role.

Step 1.3: Create an Airbyte Marketo API-only user​

Follow the Marketo documentation to create an API only user

Step 1.4: Create a Marketo API custom service​

Follow the Marketo documentation for creating a custom service for use with a REST API.

Make sure to follow the "Credentials for API Access" section in the Marketo docs to generate a Client ID and Client Secret. Once generated, copy those credentials and keep them handy for use in the Airbyte UI later.

Step 1.5: Obtain your Endpoint and Identity URLs provided by Marketo​

Follow the Marketo documentation for obtaining your base URL. Specifically, copy your Endpoint without "/rest" and keep them handy for use in the Airbyte UI.

We're almost there! Armed with your Endpoint & Identity URLs and your Client ID and Secret, head over to the Airbyte UI to setup Marketo as a source.

Step 2: Set up the Marketo connector in Airbyte​

For Airbyte Cloud:

  1. Log into your Airbyte Cloud account.
  2. In the left navigation bar, click Sources. In the top-right corner, click +new source.
  3. On the Set up the source page, enter the name for the Marketo connector and select Marketo from the Source type dropdown.
  4. Enter the start date, domain URL, client ID and secret
  5. Submit the form

For Airbyte Open Source:

  1. Navigate to the Airbyte Open Source dashboard
  2. Set the name for your source
  3. Enter the start date
  4. Enter the domain URL
  5. Enter client ID and secret
  6. Click Set up source

Supported sync modes​

The Marketo source connector supports the following sync modes:

  • Full Refresh | Overwrite
  • Full Refresh | Append
  • Incremental | Append
  • Incremental | Deduped

Supported Streams​

This connector can be used to sync the following tables from Marketo:

  • Activities_X where X is an activity type contains information about lead activities of the type X. For example, activities_send_email contains information about lead activities related to the activity type send_email. See the Marketo docs for a detailed explanation of what each column means.
  • Activity types Contains metadata about activity types. See the Marketo docs for a detailed explanation of columns.
  • Campaigns: Contains info about your Marketo campaigns.
  • Leads: Contains info about your Marketo leads.
caution

Available fields are limited by what is presented in the static schema.

  • Lists: Contains info about your Marketo static lists.
  • Programs: Contains info about your Marketo programs.
  • Segmentations: Contains info about your Marketo programs.

Performance considerations​

By default, Marketo caps all accounts to 50,000 API calls per day.

By default, this connector caps itself to 40,000 API calls per day. But you can also customize the maximum number of API calls this source connector makes per day to Marketo (which may be helpful if you have for example other applications which are also hitting the Marketo API). If this source connector reaches the maximum number you configured, it will not replicate any data until the next day.

If the 50,000 limit is too stringent, contact Marketo support for a quota increase.

Data type map​

Integration TypeAirbyte TypeNotes
arrayarrayprimitive arrays are converted into arrays of the types described in this table
int, longnumber
objectobject
stringstring``
NamespacesNo

Build instructions​

Build your own connector image​

This connector is built using our dynamic built process. The base image used to build it is defined within the metadata.yaml file under the connectorBuildOptions. The build logic is defined using Dagger here. It does not rely on a Dockerfile.

If you would like to patch our connector and build your own a simple approach would be:

  1. Create your own Dockerfile based on the latest version of the connector image.
FROM airbyte/source-marketo:latest

COPY . ./airbyte/integration_code
RUN pip install ./airbyte/integration_code

# The entrypoint and default env vars are already set in the base image
# ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
# ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

Please use this as an example. This is not optimized.

  1. Build your image:
docker build -t airbyte/source-marketo:dev .
# Running the spec command against your patched connector
docker run airbyte/source-marketo:dev spec

Customizing our build process​

When contributing on our connector you might need to customize the build process to add a system dependency or set an env var. You can customize our build process by adding a build_customization.py module to your connector. This module should contain a pre_connector_install and post_connector_install async function that will mutate the base image and the connector container respectively. It will be imported at runtime by our build process and the functions will be called if they exist.

Here is an example of a build_customization.py module:

from __future__ import annotations

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    # Feel free to check the dagger documentation for more information on the Container object and its methods.
    # https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/
    from dagger import Container


async def pre_connector_install(base_image_container: Container) -> Container:
    return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")

async def post_connector_install(connector_container: Container) -> Container:
    return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")

Changelog​

VersionDatePull RequestSubject
1.2.22023-10-1331377Use our base image and remove Dockerfile
1.2.12023-09-1830533Fix json_schema for stream Leads
1.2.02023-06-2627726License Update: Elv2
1.1.02023-04-1823956Add Segmentations Stream
1.0.42023-04-2525481Minor fix for bug caused by <= producing additional API call when there is a single date slice
1.0.32023-02-1322938Specified date formatting in specification
1.0.22023-02-0122203Handle Null cursor values
1.0.12023-01-3122015Set AvailabilityStrategy for streams explicitly to None
1.0.02023-01-2521790Fix activities_* stream schemas
0.1.122023-01-1920973Fix encoding error (note: this change is not in version 1.0.0, but is in later versions
0.1.112022-09-3017445Do not use temporary files for memory optimization
0.1.102022-09-3017445Optimize memory consumption
0.1.92022-09-2817304Migrate to per-stream sate.
0.1.72022-08-2315817Improved unit test coverage
0.1.62022-08-2115824Fix semi incremental streams: do not ignore start date, make one api call instead of multiple
0.1.52022-08-1615683Retry failed creation of a job instead of skipping it
0.1.42022-06-2013930Process failing creation of export jobs
0.1.32021-12-108429Updated titles and descriptions
0.1.22021-12-038483Improve field conversion to conform schema
0.1.12021-11-290000Fix timestamp value format issue
0.1.02021-09-065863Release Marketo CDK Connector
Was this page helpful?