Structured Properties - DataHub OpenAPI v2 Guide

This guides walks through the process of creating and using a Structured Property using the v2 version of the DataHub OpenAPI implementation. Note that this refers to DataHub's OpenAPI version and not the version of OpenAPI itself.

Requirements:

• curl
• jq

Structured Property Definition

Before a structured property can be added to an entity it must first be defined. Here is an example structured property being created against a local quickstart instance.

Create Property Definition

Example Request:

curl -X 'POST' -v \
  'http://localhost:8080/openapi/v2/entity/structuredProperty/urn%3Ali%3AstructuredProperty%3Amy.test.MyProperty01/propertyDefinition' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "qualifiedName": "my.test.MyProperty01",
    "displayName": "MyProperty01",
    "valueType": "urn:li:dataType:datahub.string",
    "allowedValues": [
        {
            "value": {"string": "foo"},
            "description": "test foo value"
        },
        {
            "value": {"string": "bar"},
            "description": "test bar value"
        }
    ],
    "cardinality": "SINGLE",
    "entityTypes": [
        "urn:li:entityType:datahub.dataset"
    ],
    "description": "test description"
}' | jq

Read Property Definition

Example Request:

curl -X 'GET' -v \
  'http://localhost:8080/openapi/v2/entity/structuredProperty/urn%3Ali%3AstructuredProperty%3Amy.test.MyProperty01/propertyDefinition' \
  -H 'accept: application/json' | jq

Example Response:

{
  "value": {
    "allowedValues": [
      {
        "value": {
          "string": "foo"
        },
        "description": "test foo value"
      },
      {
        "value": {
          "string": "bar"
        },
        "description": "test bar value"
      }
    ],
    "qualifiedName": "my.test.MyProperty01",
    "displayName": "MyProperty01",
    "valueType": "urn:li:dataType:datahub.string",
    "description": "test description",
    "entityTypes": [
      "urn:li:entityType:datahub.dataset"
    ],
    "cardinality": "SINGLE"
  }
}

Delete Property Definition

There are two types of deletion present in DataHub: hard and soft delete. As of the current release only the soft delete is supported for Structured Properties. See the subsections below for more details.

Soft Delete

A soft deleted Structured Property does not remove any underlying data on the Structured Property entity or the Structured Property's values written to other entities. The soft delete is 100% reversible with zero data loss. When a Structured Property is soft deleted, a few operations are not available.

Structured Property Soft Delete Effects:

• Entities with a soft deleted Structured Property value will not return the soft deleted properties
• Updates to a soft deleted Structured Property's definition are denied
• Adding a soft deleted Structured Property's value to an entity is denied
• Search filters using a soft deleted Structured Property will be denied

The following command will soft delete the test property MyProperty01 created in this guide by writing to the status aspect.

curl -X 'POST' \
  'http://localhost:8080/openapi/v2/entity/structuredProperty/urn%3Ali%3AstructuredProperty%3Amy.test.MyProperty01/status?systemMetadata=false' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
"removed": true
}' | jq

Removing the soft delete from the Structured Property can be done by either hard deleting the status aspect or changing the removed boolean to `false.

curl -X 'POST' \
  'http://localhost:8080/openapi/v2/entity/structuredProperty/urn%3Ali%3AstructuredProperty%3Amy.test.MyProperty01/status?systemMetadata=false' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
"removed": false
}' | jq

Hard Delete

⚠ Not Implemented ⚠

Applying Structured Properties

Structured Properties can now be added to entities which have the structuredProperties as aspect. In the following example we'll attach and remove properties to an example dataset entity with urn urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD).

Set Structured Property Values

This will set/replace all structured properties on the entity. See PATCH operations to add/remove a single property.

curl -X 'POST' -v \
  'http://localhost:8080/openapi/v2/entity/dataset/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Ahive%2CSampleHiveDataset%2CPROD%29/structuredProperties' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "properties": [
    {
      "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty01",
      "values": [
        {"string": "foo"}
      ]
    }
  ]
}' | jq

Patch Structured Property Value

For this example, we'll extend create a second structured property and apply both properties to the same dataset used previously. After this your system should include both my.test.MyProperty01 and my.test.MyProperty02.

curl -X 'POST' -v \
  'http://localhost:8080/openapi/v2/entity/structuredProperty/urn%3Ali%3AstructuredProperty%3Amy.test.MyProperty02/propertyDefinition' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "qualifiedName": "my.test.MyProperty02",
    "displayName": "MyProperty02",
    "valueType": "urn:li:dataType:datahub.string",
    "allowedValues": [
        {
            "value": {"string": "foo2"},
            "description": "test foo2 value"
        },
        {
            "value": {"string": "bar2"},
            "description": "test bar2 value"
        }
    ],
    "cardinality": "SINGLE",
    "entityTypes": [
        "urn:li:entityType:datahub.dataset"
    ]
}' | jq

This command will attach one of each of the two properties to our test dataset urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD).

curl -X 'POST' -v \
  'http://localhost:8080/openapi/v2/entity/dataset/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Ahive%2CSampleHiveDataset%2CPROD%29/structuredProperties' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "properties": [
    {
      "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty01",
      "values": [
        {"string": "foo"}
      ]
    },
    {
      "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty02",
      "values": [
        {"string": "bar2"}
      ]
    }
  ]
}' | jq

Remove Structured Property Value

The expected state of our test dataset include 2 structured properties. We'd like to remove the first one and preserve the second property.

curl -X 'PATCH' -v \
  'http://localhost:8080/openapi/v2/entity/dataset/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Ahive%2CSampleHiveDataset%2CPROD%29/structuredProperties' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json-patch+json' \
  -d '{
        "patch": [
            {
                "op": "remove",
                "path": "/properties/urn:li:structuredProperty:my.test.MyProperty01"
            }
        ],
        "arrayPrimaryKeys": {
            "properties": [
                "propertyUrn"
            ]
        }
      }' | jq

The response will show that the expected property has been removed.

{
  "urn": "urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)",
  "aspects": {
    "structuredProperties": {
      "value": {
        "properties": [
          {
            "values": [
              {
                "string": "bar2"
              }
            ],
            "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty02"
          }
        ]
      }
    }
  }
}

Add Structured Property Value

In this example, we'll add the property back with a different value, preserving the existing property.

curl -X 'PATCH' -v \
  'http://localhost:8080/openapi/v2/entity/dataset/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Ahive%2CSampleHiveDataset%2CPROD%29/structuredProperties' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json-patch+json' \
  -d '{
        "patch": [
            {
                "op": "add",
                "path": "/properties/urn:li:structuredProperty:my.test.MyProperty01",
                "value": {
                    "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty01",
                    "values": [
                        {
                            "string": "bar"
                        }
                    ]
                }
            }
        ],
        "arrayPrimaryKeys": {
            "properties": [
                "propertyUrn"
            ]
        }
    }' | jq

The response shows that the property was re-added with the new value bar instead of the previous value foo.

{
    "urn": "urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)",
    "aspects": {
        "structuredProperties": {
            "value": {
                "properties": [
                    {
                        "values": [
                            {
                                "string": "bar2"
                            }
                        ],
                        "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty02"
                    },
                    {
                        "values": [
                            {
                                "string": "bar"
                            }
                        ],
                        "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty01"
                    }
                ]
            }
        }
    }
}