Skip to main content

GitLab

This page contains the setup guide and reference information for the Gitlab Source connector.

Prerequisites​

  • Gitlab instance or an account at Gitlab
  • Start date (Optional)
  • GitLab Groups (Optional)
  • GitLab Projects (Optional)

For Airbyte Cloud:

For Airbyte Open Source:

Setup guide​

Step 1: Set up GitLab​

Create a GitLab Account or set up a local instance of GitLab.

Airbyte Open Source additional setup steps

Log into GitLab and then generate a personal access token. Your token should have the read_api scope, that Grants read access to the API, including all groups and projects, the container registry, and the package registry.

Step 2: Set up the GitLab connector in Airbyte​

For Airbyte Cloud:

  1. Log into your Airbyte Cloud account.
  2. In the left navigation bar, click Sources. In the top-right corner, click + new source.
  3. On the source setup page, select GitLab from the Source type dropdown and enter a name for this connector.
  4. Click Authenticate your GitLab account by selecting Oauth or Personal Access Token for Authentication.
  5. Log in and Authorize to the GitLab account.
  6. API URL - The URL to access you self-hosted GitLab instance or gitlab.com (default).
  7. Start date (Optional) - The date from which you'd like to replicate data for streams.
  8. Groups (Optional) - List of GitLab group IDs, e.g. airbytehq for single group, airbytehq another-repo for multiple groups.
  9. Projects (Optional) - List of GitLab projects to pull data for, e.g. airbytehq/airbyte.
  10. Click Set up source.

Note: You can specify either Group IDs or Project IDs in the source configuration. If both fields are blank, the connector will retrieve a list of all the groups that are accessible to the configured token and ingest as normal.

For Airbyte Open Source:

  1. Authenticate with Personal Access Token.

Supported sync modes​

The Gitlab Source connector supports the following sync modes:

Supported Streams​

This connector outputs the following streams:

Additional information​

GitLab source works with GitLab API v4. It can also work with self-hosted GitLab API v4.

Performance considerations​

Gitlab has the rate limits, but the Gitlab connector should not run into Gitlab API limitations under normal usage. Please create an issue if you see any rate limit issues that are not automatically retried successfully.

Build instructions​

Build your own connector image​

This connector is built using our dynamic built process. The base image used to build it is defined within the metadata.yaml file under the connectorBuildOptions. The build logic is defined using Dagger here. It does not rely on a Dockerfile.

If you would like to patch our connector and build your own a simple approach would be:

  1. Create your own Dockerfile based on the latest version of the connector image.
FROM airbyte/source-gitlab:latest

COPY . ./airbyte/integration_code
RUN pip install ./airbyte/integration_code

# The entrypoint and default env vars are already set in the base image
# ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
# ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

Please use this as an example. This is not optimized.

  1. Build your image:
docker build -t airbyte/source-gitlab:dev .
# Running the spec command against your patched connector
docker run airbyte/source-gitlab:dev spec

Customizing our build process​

When contributing on our connector you might need to customize the build process to add a system dependency or set an env var. You can customize our build process by adding a build_customization.py module to your connector. This module should contain a pre_connector_install and post_connector_install async function that will mutate the base image and the connector container respectively. It will be imported at runtime by our build process and the functions will be called if they exist.

Here is an example of a build_customization.py module:

from __future__ import annotations

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    # Feel free to check the dagger documentation for more information on the Container object and its methods.
    # https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/
    from dagger import Container


async def pre_connector_install(base_image_container: Container) -> Container:
    return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")

async def post_connector_install(connector_container: Container) -> Container:
    return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")

Changelog​

VersionDatePull RequestSubject
1.8.22023-10-1331377Use our base image and remove Dockerfile
1.8.12023-10-1231375Mark start_date as optional, migrate groups and projects to array
1.8.02023-10-1231339Add undeclared fields to streams schemas, validate date/date-time format in stream schemas
1.7.12023-10-1031210Added expired access_token handling, while checking the connection
1.7.02023-08-0827869Add Deployments stream
1.6.02023-06-3027869Add shared_runners_setting field to groups
1.5.12023-06-2427679Fix formatting
1.5.02023-06-1527392Make API URL an optional parameter in spec.
1.4.22023-06-1527346Partially revert changes made in version 1.0.4, disallow http calls in cloud.
1.4.12023-06-1327351Fix OAuth token expiry date.
1.4.02023-06-1227234Skip stream slices on 403/404 errors, do not fail syncs.
1.3.12023-06-0827147Improve connectivity check for connections with no projects/groups
1.3.02023-06-0827150Update stream schemas
1.2.12023-06-0226947New field name added to Pipelines and PipelinesExtended stream schema
1.2.02023-05-1722293Preserve data in records with flattened keys
1.1.12023-05-2326422Fix error 404 Repository Not Found when syncing project with Repository feature disabled
1.1.02023-05-1025948Introduce two new fields in the Projects stream schema
1.0.42023-04-2021373Accept api_url with or without scheme
1.0.32023-02-1422992Specified date formatting in specification
1.0.22023-01-2722001Set AvailabilityStrategy for streams explicitly to None
1.0.12023-01-2321713Fix missing data issue
1.0.02022-12-057506Add OAuth2.0 authentication option
0.1.122022-12-1520542Revert HttpAvailability changes, run on cdk 0.15.0
0.1.112022-12-1420479Use HttpAvailabilityStrategy + add unit tests
0.1.102022-12-1220384Fetch groups along with their subgroups
0.1.92022-12-1120348Fix 403 error when syncing EpicIssues stream
0.1.82022-12-0220023Fix duplicated records issue for Projects stream
0.1.72022-12-0119986Fix GroupMilestones stream schema
0.1.62022-06-2313252Add GroupIssueBoards stream
0.1.52022-05-0211907Fix null projects param and container_expiration_policy
0.1.42022-03-2311140Ingest All Accessible Groups if not Specified in Config
0.1.32021-12-218991Update connector fields title/description
0.1.22021-10-187108Allow all domains to be used as api_url
0.1.12021-10-126932Fix pattern field in spec file, remove unused fields from config files, use cache from CDK
0.1.02021-07-064174Initial Release
Was this page helpful?