Mixpanel
This page contains the setup guide and reference information for the Mixpanel source connector.
Prerequisites
To set up the Mixpanel source connector, you'll need a Mixpanel Service Account and it's Project ID, the Project Timezone, and the Project region (US or EU).
Set up the Mixpanel connector in Airbyte
- Log into your Airbyte Cloud or navigate to the Airbyte Open Source dashboard.
- Click Sources and then click + New source.
- On the Set up the source page, select Mixpanel from the Source type dropdown.
- Enter the name for the Mixpanel connector.
- For Authentication, select Service Account from the dropdown and enter the Mixpanel Service Account secret.
- For Project ID, enter the Mixpanel Project ID.
- For Attribution Window, enter the number of days for the length of the attribution window.
- For Project Timezone, enter the timezone for your Mixpanel project.
- For Start Date, enter the date in YYYY-MM-DD format. The data added on and after this date will be replicated. If left blank, the connector will replicate data from up to one year ago by default.
- For End Date, enter the date in YYYY-MM-DD format.
- For Region, enter the region for your Mixpanel project.
- For Date slicing window, enter the number of days to slice through data. If you encounter RAM usage issues due to a huge amount of data in each window, try using a lower value for this parameter.
- Click Set up source.
Supported sync modes
The Mixpanel source connector supports the following sync modes:
Note: Incremental sync returns duplicated (old records) for the state date due to API filter limitation, which is granular to the whole day only.
Supported Streams
- Export (Incremental)
- Engage (Incremental)
- Funnels (Incremental)
- Revenue (Incremental)
- Annotations (Full table)
- Cohorts (Incremental)
- Cohort Members (Incremental)
Performance considerations
Syncing huge date windows may take longer due to Mixpanel's low API rate-limits (60 reqs per hour).
CHANGELOG
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 1.0.1 | 2023-10-13 | 31377 | Use our base image and remove Dockerfile |
| 1.0.0 | 2023-09-27 | 30025 | Fix type of datetime field in engage stream; fix primary key for export stream. |
| 0.1.41 | 2023-09-26 | 30149 | Change config schema; set checkpointing interval; add suggested streams; add casting datetime fields. |
| 0.1.40 | 2022-09-20 | 30090 | Handle 400 error when the credentials become expired |
| 0.1.39 | 2023-09-15 | 30469 | Add default primary key distinct_id to Export stream |
| 0.1.38 | 2023-08-31 | 30028 | Handle gracefully project timezone mismatch |
| 0.1.37 | 2023-07-20 | 27932 | Fix spec: change start/end date format to date |
| 0.1.36 | 2023-06-27 | 27752 | Partially revert version 0.1.32; Use exponential backoff; |
| 0.1.35 | 2023-06-12 | 27252 | Add should_retry False for 402 error |
| 0.1.34 | 2023-05-15 | 21837 | Add "insert_id" field to "export" stream schema |
| 0.1.33 | 2023-04-25 | 25543 | Set should_retry for 104 error in stream export |
| 0.1.32 | 2023-04-11 | 25056 | Set HttpAvailabilityStrategy, add exponential backoff, streams export and annotations add undeclared fields |
| 0.1.31 | 2023-02-13 | 22936 | Specified date formatting in specification |
| 0.1.30 | 2023-01-27 | 22017 | Set AvailabilityStrategy for streams explicitly to None |
| 0.1.29 | 2022-11-02 | 18846 | For "export" stream make line parsing more robust |
| 0.1.28 | 2022-10-06 | 17699 | Fix discover step issue cursor field None |
| 0.1.27 | 2022-09-29 | 17415 | Disable stream "cohort_members" on discover if not access |
| 0.1.26 | 2022-09-28 | 17304 | Migrate to per-stream states. |
| 0.1.25 | 2022-09-27 | 17145 | Disable streams "export", "engage" on discover if not access |
| 0.1.24 | 2022-09-26 | 16915 | Added Service Accounts support |
| 0.1.23 | 2022-09-18 | 16843 | Add stream=True for export stream |
| 0.1.22 | 2022-09-15 | 16770 | Use "Retry-After" header for backoff |
| 0.1.21 | 2022-09-11 | 16191 | Improved connector's input configuration validation |
| 0.1.20 | 2022-08-22 | 15091 | Improve export stream cursor support |
| 0.1.19 | 2022-08-18 | 15739 | Update titile and description for Project Secret field |
| 0.1.18 | 2022-07-21 | 14924 | Remove additionalProperties field from schemas and specs |
| 0.1.17 | 2022-06-01 | 12801 | Acceptance tests fix, fixing some bugs for beta release |
| 0.1.16 | 2022-05-30 | 12801 | Add end_date parameter |
| 0.1.15 | 2022-05-04 | 12482 | Update input configuration copy |
| 0.1.14 | 2022-05-02 | 11501 | Improve incremental sync method to streams |
| 0.1.13 | 2022-04-27 | 12335 | Adding fixtures to mock time.sleep for connectors that explicitly sleep |
| 0.1.12 | 2022-03-31 | 11633 | Increase unit test coverage |
| 0.1.11 | 2022-04-04 | 11318 | Change Response Reading |
| 0.1.10 | 2022-03-31 | 11227 | Fix cohort id always null in the cohort_members stream |
| 0.1.9 | 2021-12-07 | 8429 | Updated titles and descriptions |
| 0.1.7 | 2021-12-01 | 8381 | Increased performance for discovery stage during connector setup |
| 0.1.6 | 2021-11-25 | 8256 | Deleted date_window_size and fix schemas date type issue |
| 0.1.5 | 2021-11-10 | 7451 | Support start_date older than 1 year |
| 0.1.4 | 2021-11-08 | 7499 | Remove base-python dependencies |
| 0.1.3 | 2021-10-30 | 7505 | Guarantee that standard and custom mixpanel properties in the Engage stream are written as strings |
| 0.1.2 | 2021-11-02 | 7439 | Added delay for all streams to match API limitation of requests rate |
| 0.1.1 | 2021-09-16 | 6075 | Added option to select project region |
Build instructions
Build your own connector image
This connector is built using our dynamic built process.
The base image used to build it is defined within the metadata.yaml file under the connectorBuildOptions.
The build logic is defined using Dagger here.
It does not rely on a Dockerfile.
If you would like to patch our connector and build your own a simple approach would be:
- Create your own Dockerfile based on the latest version of the connector image.
FROM airbyte/source-mixpanel:latest
COPY . ./airbyte/integration_code
RUN pip install ./airbyte/integration_code
# The entrypoint and default env vars are already set in the base image
# ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
# ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
Please use this as an example. This is not optimized.
- Build your image:
docker build -t airbyte/source-mixpanel:dev .
# Running the spec command against your patched connector
docker run airbyte/source-mixpanel:dev spec
Customizing our build process
When contributing on our connector you might need to customize the build process to add a system dependency or set an env var.
You can customize our build process by adding a build_customization.py module to your connector.
This module should contain a pre_connector_install and post_connector_install async function that will mutate the base image and the connector container respectively.
It will be imported at runtime by our build process and the functions will be called if they exist.
Here is an example of a build_customization.py module:
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
# Feel free to check the dagger documentation for more information on the Container object and its methods.
# https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/
from dagger import Container
async def pre_connector_install(base_image_container: Container) -> Container:
return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")
async def post_connector_install(connector_container: Container) -> Container:
return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")
| 0.1.0 | 2021-07-06 | 3698 | Created CDK native mixpanel connector |