Skip to main content

Google Analytics (Universal Analytics)

This page contains the setup guide and reference information for the Google Analytics (Universal Analytics) source connector.

This connector supports Universal Analytics properties through the Reporting API v4.

caution

The Google Analytics (Universal Analytics) connector will be deprecated soon.

Google is phasing out Universal Analytics in favor of Google Analytics 4 (GA4). In consequence, we are deprecating the Google Analytics (Universal Analytics) connector and recommend that you migrate to the Google Analytics 4 (GA4) connector as soon as possible to ensure your syncs are not affected.

Due to this deprecation, we will not be accepting new contributions for this source.

For more information, see "Universal Analytics is going away".

note

Google Analytics Universal Analytics (UA) connector, uses the older version of Google Analytics, which has been the standard for tracking website and app user behavior since 2012.

Google Analytics 4 (GA4) connector is the latest version of Google Analytics, which was introduced in 2020. It offers a new data model that emphasizes events and user properties, rather than pageviews and sessions. This new model allows for more flexible and customizable reporting, as well as more accurate measurement of user behavior across devices and platforms.

Prerequisites​

A Google Cloud account with Viewer permissions and Google Analytics Reporting API and Google Analytics API enabled.

Setup guide​

For Airbyte Cloud:

To set up Google Analytics as a source in Airbyte Cloud:

  1. Log into your Airbyte Cloud account.
  2. In the left navigation bar, click Sources. In the top-right corner, click + New source.
  3. On the Set up the source page, select Google Analytics from the Source type dropdown.
  4. For Name, enter a name for the Google Analytics connector.
  5. Authenticate your Google account via OAuth or Service Account Key Authentication.
    • To authenticate your Google account via OAuth, click Sign in with Google and complete the authentication workflow.
    • To authenticate your Google account via Service Account Key Authentication, enter your Google Cloud service account key in JSON format. Make sure the Service Account has the Project Viewer permission.
  6. Enter the Replication Start Date in YYYY-MM-DD format. The data added on and after this date will be replicated. If this field is blank, Airbyte will replicate all data.
  7. Enter the View ID for the Google Analytics View you want to fetch data from.
  8. Leave Data request time increment in days (Optional) blank or set to 1. For faster syncs, set this value to more than 1 but that might result in the Google Analytics API returning sampled data, potentially causing inaccuracies in the returned results. The maximum allowed value is 364.

For Airbyte Open Source:

To set up Google Analytics as a source in Airbyte Open Source:

  1. Go to the Airbyte UI and click Sources and then click + New source.
  2. On the Set up the source page, select Google Analytics from the Source type dropdown.
  3. Enter a name for the Google Analytics connector.
  4. Authenticate your Google account via OAuth or Service Account Key Authentication:
  5. Enter the Replication Start Date in YYYY-MM-DD format. The data added on and after this date will be replicated. If this field is blank, Airbyte will replicate all data.
  6. Enter the View ID for the Google Analytics View you want to fetch data from.
  7. Optionally, enter a JSON object as a string in the Custom Reports field. For details, refer to Requesting custom reports
  8. Leave Data request time increment in days (Optional) blank or set to 1. For faster syncs, set this value to more than 1 but that might result in the Google Analytics API returning sampled data, potentially causing inaccuracies in the returned results. The maximum allowed value is 364.

Supported sync modes​

The Google Analytics source connector supports the following sync modes:

caution

You need to add the service account email address on the account level, not the property level. Otherwise, an 403 error will be returned.

Supported streams​

The Google Analytics (Universal Analytics) source connector can sync the following tables:

Stream nameSchema
website_overview{"ga_date":"2021-02-11","ga_users":1,"ga_newUsers":0,"ga_sessions":9,"ga_sessionsPerUser":9.0,"ga_avgSessionDuration":28.77777777777778,"ga_pageviews":63,"ga_pageviewsPerSession":7.0,"ga_avgTimeOnPage":4.685185185185185,"ga_bounceRate":0.0,"ga_exitRate":14.285714285714285,"view_id":"211669975"}
traffic_sources{"ga_date":"2021-02-11","ga_source":"(direct)","ga_medium":"(none)","ga_socialNetwork":"(not set)","ga_users":1,"ga_newUsers":0,"ga_sessions":9,"ga_sessionsPerUser":9.0,"ga_avgSessionDuration":28.77777777777778,"ga_pageviews":63,"ga_pageviewsPerSession":7.0,"ga_avgTimeOnPage":4.685185185185185,"ga_bounceRate":0.0,"ga_exitRate":14.285714285714285,"view_id":"211669975"}
pages{"ga_date":"2021-02-11","ga_hostname":"mydemo.com","ga_pagePath":"/home5","ga_pageviews":63,"ga_uniquePageviews":9,"ga_avgTimeOnPage":4.685185185185185,"ga_entrances":9,"ga_entranceRate":14.285714285714285,"ga_bounceRate":0.0,"ga_exits":9,"ga_exitRate":14.285714285714285,"view_id":"211669975"}
locations{"ga_date":"2021-02-11","ga_continent":"Americas","ga_subContinent":"Northern America","ga_country":"United States","ga_region":"Iowa","ga_metro":"Des Moines-Ames IA","ga_city":"Des Moines","ga_users":1,"ga_newUsers":0,"ga_sessions":1,"ga_sessionsPerUser":1.0,"ga_avgSessionDuration":29.0,"ga_pageviews":7,"ga_pageviewsPerSession":7.0,"ga_avgTimeOnPage":4.666666666666667,"ga_bounceRate":0.0,"ga_exitRate":14.285714285714285,"view_id":"211669975"}
monthly_active_users{"ga_date":"2021-02-11","ga_30dayUsers":1,"view_id":"211669975"}
four_weekly_active_users{"ga_date":"2021-02-11","ga_28dayUsers":1,"view_id":"211669975"}
two_weekly_active_users{"ga_date":"2021-02-11","ga_14dayUsers":1,"view_id":"211669975"}
weekly_active_users{"ga_date":"2021-02-11","ga_7dayUsers":1,"view_id":"211669975"}
daily_active_users{"ga_date":"2021-02-11","ga_1dayUsers":1,"view_id":"211669975"}
devices{"ga_date":"2021-02-11","ga_deviceCategory":"desktop","ga_operatingSystem":"Macintosh","ga_browser":"Chrome","ga_users":1,"ga_newUsers":0,"ga_sessions":9,"ga_sessionsPerUser":9.0,"ga_avgSessionDuration":28.77777777777778,"ga_pageviews":63,"ga_pageviewsPerSession":7.0,"ga_avgTimeOnPage":4.685185185185185,"ga_bounceRate":0.0,"ga_exitRate":14.285714285714285,"view_id":"211669975"}
Any custom reportsSee below for details.

Reach out to us on Slack or create an issue if you need to send custom Google Analytics report data with Airbyte.

Rate Limits and Performance Considerations (Airbyte Open-Source)​

Analytics Reporting API v4

  • Number of requests per day per project: 50,000
  • Number of requests per view (profile) per day: 10,000 (cannot be increased)
  • Number of requests per 100 seconds per project: 2,000
  • Number of requests per 100 seconds per user per project: 100 (can be increased in Google API Console to 1,000).

The Google Analytics connector should not run into the "requests per 100 seconds" limitation under normal usage. Create an issue if you see any rate limit issues that are not automatically retried successfully and try increasing the window_in_days value.

Sampled data in reports​

If you are not on the Google Analytics 360 tier, the Google Analytics API may return sampled data if the amount of data in your Google Analytics account exceeds Google's pre-determined compute thresholds. This means the data returned in the report is an estimate which may have some inaccuracy. This Google page provides a comprehensive overview of how Google applies sampling to your data.

In order to minimize the chances of sampling being applied to your data, Airbyte makes data requests to Google in one day increments (the smallest allowed date increment). This reduces the amount of data the Google API processes per request, thus minimizing the chances of sampling being applied. The downside of requesting data in one day increments is that it increases the time it takes to export your Google Analytics data. If sampling is not a concern, you can override this behavior by setting the optional window_in_day parameter to specify the number of days to look back and avoid sampling. When sampling occurs, a warning is logged to the sync log.

Data processing latency​

According to the Google Analytics API documentation, all report data may continue to be updated 48 hours after it appears in the Google Analytics API. This means if you request the same report twice within 48 hours of that data being sent to Google Analytics, the report data might be different across the two requests. This happens when Google Analytics is still processing all events it received.

When this occurs, the returned data will set the flag isDataGolden to false. As mentioned in the Google Analytics API docs, the isDataGolden flag indicates if [data] is golden or not. Data is golden when the exact same request [for a report] will not produce any new results if asked at a later point in time.

To address this issue, the connector adds a lookback window of 2 days to ensure any previously synced non-golden data is re-synced with its potential updates. For example: If your last sync occurred 5 days ago and a sync is initiated today, the connector will attempt to sync data from 7 days ago up to the latest data available.

To determine whether data is finished processing or not, the isDataGolden flag is exposed and should be used.

Requesting Custom Reports​

To replicate Google Analytics Custom Reports using this connector, input a JSON object as a string in the Custom Reports field when setting up the connector. The JSON is an array of objects where each object has the following schema:

{"name": string, "dimensions": [string], "metrics": [string]}

Here is an example input "Custom Reports" field:

[{"name": "new_users_per_day", "dimensions": ["ga:date","ga:country","ga:region"], "metrics": ["ga:newUsers"]}, {"name": "users_per_city", "dimensions": ["ga:city"], "metrics": ["ga:users"]}]

To create a list of dimensions, you can use default Google Analytics dimensions (listed below) or custom dimensions if you have some defined. Each report can contain no more than 7 dimensions, and they must all be unique. The default Google Analytics dimensions are:

  • ga:browser
  • ga:city
  • ga:continent
  • ga:country
  • ga:date
  • ga:deviceCategory
  • ga:hostname
  • ga:medium
  • ga:metro
  • ga:operatingSystem
  • ga:pagePath
  • ga:region
  • ga:socialNetwork
  • ga:source
  • ga:subContinent

To create a list of metrics, use a default Google Analytics metric (values from the list below) or custom metrics if you have defined them. A custom report can contain no more than 10 unique metrics. The default available Google Analytics metrics are:

  • ga:14dayUsers
  • ga:1dayUsers
  • ga:28dayUsers
  • ga:30dayUsers
  • ga:7dayUsers
  • ga:avgSessionDuration
  • ga:avgTimeOnPage
  • ga:bounceRate
  • ga:entranceRate
  • ga:entrances
  • ga:exitRate
  • ga:exits
  • ga:newUsers
  • ga:pageviews
  • ga:pageviewsPerSession
  • ga:sessions
  • ga:sessionsPerUser
  • ga:uniquePageviews
  • ga:users

Incremental sync is supported only if you add ga:date dimension to your custom report.

Build instructions​

Build your own connector image​

This connector is built using our dynamic built process. The base image used to build it is defined within the metadata.yaml file under the connectorBuildOptions. The build logic is defined using Dagger here. It does not rely on a Dockerfile.

If you would like to patch our connector and build your own a simple approach would be:

  1. Create your own Dockerfile based on the latest version of the connector image.
FROM airbyte/source-google-analytics-v4:latest

COPY . ./airbyte/integration_code
RUN pip install ./airbyte/integration_code

# The entrypoint and default env vars are already set in the base image
# ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
# ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

Please use this as an example. This is not optimized.

  1. Build your image:
docker build -t airbyte/source-google-analytics-v4:dev .
# Running the spec command against your patched connector
docker run airbyte/source-google-analytics-v4:dev spec

Customizing our build process​

When contributing on our connector you might need to customize the build process to add a system dependency or set an env var. You can customize our build process by adding a build_customization.py module to your connector. This module should contain a pre_connector_install and post_connector_install async function that will mutate the base image and the connector container respectively. It will be imported at runtime by our build process and the functions will be called if they exist.

Here is an example of a build_customization.py module:

from __future__ import annotations

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    # Feel free to check the dagger documentation for more information on the Container object and its methods.
    # https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/
    from dagger import Container


async def pre_connector_install(base_image_container: Container) -> Container:
    return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")

async def post_connector_install(connector_container: Container) -> Container:
    return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")

Changelog​

VersionDatePull RequestSubject
0.2.22023-10-1331377Use our base image and remove Dockerfile
0.2.12023-07-1128149Specify date format to support datepicker in UI
0.2.02023-06-2627738License Update: Elv2
0.1.362023-04-1322223Fix custom report with Segments dimensions
0.1.352023-05-3126885Remove authSpecification from spec in favour of advancedAuth
0.1.342023-01-2722006Set AvailabilityStrategy for streams explicitly to None
0.1.332022-12-2320858Fix check connection
0.1.322022-11-0418965Fix for discovery stage, when custom_reports are provided with single stream as dict
0.1.312022-10-3018670Add Custom Reports schema validation on check connection
0.1.302022-10-1317943Fix pagination
0.1.292022-10-1217905Handle exceeded daily quota gracefully
0.1.282022-09-2416920Added segments and filters to custom reports
0.1.272022-10-0717717Improve CHECK by using ga:hits metric.
0.1.262022-09-2817326Migrate to per-stream states.
0.1.252022-07-2715087Fix documentationUrl
0.1.242022-07-2615042Update additionalProperties field to true from schemas
0.1.232022-07-2214949Add handle request daily quota error
0.1.222022-06-3014298Specify integer type for ga:dateHourMinute dimension
0.1.212022-04-3012500Improve input configuration copy
0.1.202022-04-2812426Expose isDataGOlden field and always resync data two days back to make sure it is golden
0.1.192022-04-1912150Minor changes to documentation
0.1.182022-04-0711803Improved documentation
0.1.172022-03-3111512Improved Unit and Acceptance tests coverage, fixed read with abnormally large state values
0.1.162022-01-269480Reintroduce window_in_days and log warning when sampling occurs
0.1.152021-12-289165Update titles and descriptions
0.1.142021-12-098656Fix date format in schemas
0.1.132021-12-098676Fix window_in_days validation issue
0.1.122021-12-038175Fix validation of unknown metric(s) or dimension(s) error
0.1.112021-11-308264Corrected date range
0.1.102021-11-198087Support start_date before the account has any data
0.1.92021-10-277410Add check for correct permission for requested view_id
0.1.82021-10-137020Add intermediary auth config support
0.1.72021-10-076414Declare OAuth parameters in Google sources
0.1.62021-09-276459Update OAuth Spec File
0.1.32021-09-216357Fix OAuth workflow parameters
0.1.22021-09-206306Support of Airbyte OAuth initialization flow
0.1.12021-08-255655Corrected validation of empty custom report
0.1.02021-08-105290Initial Release
Was this page helpful?