Skip to main content

Discover

The discover method of the Airbyte Protocol returns an AirbyteCatalog: an object which declares all the streams output by a connector and their schemas. It also declares the sync modes supported by the stream (full refresh or incremental). See the beginner's guide to the catalog for more information.

Run a discover command:

poetry run source-survey-monkey-demo discover --config secrets/config.json

The command should succeed, but the schema will be wrong:

{
  "type": "CATALOG",
  "catalog": {
    "streams": [
      {
        "name": "surveys",
        "json_schema": {
          "$schema": "http://json-schema.org/draft-07/schema#",
          "type": "object",
          "properties": {
            "id": { "type": ["null", "string"] },
            "name": { "type": ["null", "string"] },
            "signup_date": { "type": ["null", "string"], "format": "date-time" }
          }
        },
        "supported_sync_modes": ["full_refresh"],
        "source_defined_primary_key": [["id"]]
      }
    ]
  }
}

We'll need to replace the schema with a json_schema representation of the records yielded by the stream.

The easiest way to extract the schema from a HTTP response is to use the Connector Builder. You can also paste the schema below, which was generated by the Connector Builder:

{
  "$schema": "http://json-schema.org/schema#",
  "properties": {
    "analyze_url": {
      "type": "string"
    },
    "collect_stats": {
      "properties": {
        "status": {
          "properties": {
            "open": {
              "type": "number"
            }
          },
          "type": "object"
        },
        "total_count": {
          "type": "number"
        },
        "type": {
          "properties": {
            "weblink": {
              "type": "number"
            }
          },
          "type": "object"
        }
      },
      "type": "object"
    },
    "date_created": {
      "type": "string"
    },
    "date_modified": {
      "type": "string"
    },
    "href": {
      "type": "string"
    },
    "id": {
      "type": "string"
    },
    "language": {
      "type": "string"
    },
    "nickname": {
      "type": "string"
    },
    "preview": {
      "type": "string"
    },
    "question_count": {
      "type": "number"
    },
    "response_count": {
      "type": "number"
    },
    "title": {
      "type": "string"
    }
  },
  "type": "object"
}
info

If the connector you're building has a dynamic schema, you'll need to overwrite the AbstractSource::streams.

The three connector operations work as expected. You can now upload your connector to your Airbyte instance.In the next section, we'll add the connector to our local Airbyte instance.