|
| 1 | +--- |
| 2 | +title: Migrate data between Common Data Service environments using the dataflows OData connector |
| 3 | +author: denisem-msft |
| 4 | +ms.reviewer: "nabuthuk" |
| 5 | +description: Migrate data between Common Data Service environments using dataflows OData connector. |
| 6 | +ms.date: 05/05/2020 |
| 7 | +ms.service: powerapps |
| 8 | +ms.topic: "article" |
| 9 | +ms.author: demora |
| 10 | +search.app: |
| 11 | + - PowerApps |
| 12 | +--- |
| 13 | + |
| 14 | + |
| 15 | + |
| 16 | +# Migrate data between Common Data Service environments using the dataflows OData connector |
| 17 | + |
| 18 | +Common Data Service [Web API](/powerapps/developer/common-data-service/webapi/overview) works with any technology that supports OData and OAuth. There are many options available to move data in and out of Common Data Service. OData connector is one of the dataflows, which is designed to support migration and synchronization of large datasets in Common Data Service. |
| 19 | + |
| 20 | +In this article, we walk you through how to migrate data between Common Data Service environments using the dataflows OData connector. |
| 21 | + |
| 22 | +## Prerequisites |
| 23 | + |
| 24 | +- System Administrator or System Customizer security role permission on both the source and target environments. |
| 25 | + |
| 26 | +- Power Apps, Power Automate, or Common Data Service license (per app or per user). |
| 27 | + |
| 28 | +- Two Common Data Service [environments with database](/power-platform/admin/create-environment#create-an-environment-with-a-database). |
| 29 | + |
| 30 | +## Scenarios |
| 31 | + |
| 32 | + - A one-time cross-environment or cross-tenant migration is needed (for example, geo-migration). |
| 33 | + |
| 34 | + - A developer needs to update an app that is being used in production. Test data is needed in their development environment to easily build out changes. |
| 35 | + |
| 36 | +## Step 1: Plan out the dataflow |
| 37 | + |
| 38 | +1. Identify the source and target environments. |
| 39 | + |
| 40 | + - The **source environment** is where the data is migrated from. |
| 41 | + |
| 42 | + - The **target environment** is where the data is migrated to. |
| 43 | + |
| 44 | +1. Ensure that the entities are already defined in the target environment. Ideally both environments should have the same entities defined with the same solution. |
| 45 | + |
| 46 | +1. When importing relationships, multiple dataflows are required. |
| 47 | + |
| 48 | + Entities that are one (parent/independent) to many (children/dependent) require separate dataflows. Configure the parent dataflow to run before any child entities, since the data in the parent needs to be loaded first to correctly map to the fields in the corresponding child entities. |
| 49 | + |
| 50 | +## Step 2: Get the OData endpoint |
| 51 | + |
| 52 | +Common Data Service provides an OData endpoint that does not require additional configuration to authenticate with the dataflows' connector. It is relatively easy to connect to the source environment. |
| 53 | + |
| 54 | +This article will walk through how to set up a new dataflow with the OData connector. For information on connecting to all data sources supported by dataflows, see [Create and use dataflows](https://docs.microsoft.com/powerapps/maker/common-data-service/create-and-use-dataflows). |
| 55 | + |
| 56 | + |
| 57 | +From the **source** environment, get the [OData endpoint](https://docs.microsoft.com/powerapps/developer/common-data-service/view-download-developer-resources) for that environment: |
| 58 | + |
| 59 | +1. Sign in to [Power Apps](https://make.powerapps.com). |
| 60 | + |
| 61 | +1. Select the required source environment from the upper-right corner. |
| 62 | + |
| 63 | +1. Select the **Settings** (gear) icon in the upper-right corner, and then select **Advanced Settings**. |
| 64 | + |
| 65 | +1. On the **Settings** page, select the drop-down arrow next to **Settings**, and select **Customizations**. |
| 66 | + |
| 67 | +1. On the **Customizations** page, select **Developer Resources**. |
| 68 | + |
| 69 | +1. Copy the **Service Root URL** to Notepad. |
| 70 | + |
| 71 | + > [!div class="mx-imgBorder"] |
| 72 | + >  |
| 73 | + |
| 74 | +## Step 3: Create a new OData dataflow |
| 75 | + |
| 76 | +In the **target** environment, create a new dataflow with the OData connector. |
| 77 | + |
| 78 | +1. Sign in to [Power Apps](https://make.powerapps.com). |
| 79 | + |
| 80 | +1. Select the required target environment from the upper-right corner. |
| 81 | + |
| 82 | +1. In the left navigation pane, expand the **Data** menu, and then select **Dataflows**. |
| 83 | + |
| 84 | +1. Select **New dataflow** to create a new dataflow. Provide a meaningful name for the dataflow. Select **Create**. |
| 85 | + > [!div class="mx-imgBorder"] |
| 86 | + >  |
| 87 | +
|
| 88 | +1. Select the **OData** connector. |
| 89 | + |
| 90 | + > [!div class="mx-imgBorder"] |
| 91 | + >  |
| 92 | +
|
| 93 | +1. In the **Connection settings** dialog box, type the field values: |
| 94 | + |
| 95 | + > [!div class="mx-imgBorder"] |
| 96 | + >  |
| 97 | +
|
| 98 | + |
| 99 | + | Field | Description | |
| 100 | + |--|--| |
| 101 | + | URL | Provide the Service Root URL in the URL field of the connection settings. | |
| 102 | + | Connection | Create new connection. This will be automatically chosen if you have not made an OData connection in dataflows before. | |
| 103 | + | Connection name | Optionally, rename the connection name, but a value is automatically populated. | | |
| 104 | + | On-premises data gateway | None. An on-premises data gateway is not needed for connections to this cloud service. | |
| 105 | + | Authentication kind | Organizational account. Select **Sign in** to open the sign-in dialog that authenticates the account associated with the connection. | |
| 106 | + |
| 107 | + > [!IMPORTANT] |
| 108 | + > Disable pop-up and cookies blockers in your browser in order to configure the Azure AD authentication. This is similar to the fact that you are using the Common Data Service OData endpoint or any other OAuth-based authentication data source. |
| 109 | + |
| 110 | +1. Select **Next** in the lower right. |
| 111 | + |
| 112 | +## Step 4: Select and transform data with Power Query |
| 113 | + |
| 114 | +Use Power Query to select the tables and also transform data as per your requirement. |
| 115 | + |
| 116 | +First, select the entities that need to be transferred. You can browse all entities in the source environment and preview some of the data in each entity. |
| 117 | + |
| 118 | +> [!div class="mx-imgBorder"] |
| 119 | +>  |
| 120 | +
|
| 121 | +1. Select one or multiple entities as needed, and then select **Transform data**. |
| 122 | + |
| 123 | + > [!NOTE] |
| 124 | + > When importing relationships, remember that the parent entity dataflow needs to be imported before the child ones. The data for the child dataflow will require data to be in the parent entity for it to correctly map, otherwise it might throw an error. |
| 125 | + |
| 126 | +1. In the **Power Query - Edit queries** window, you can transform the query before import. |
| 127 | + |
| 128 | + - If you are only migrating data, there should not be a need to modify anything here. |
| 129 | + |
| 130 | + - Reducing the number of unnecessary columns will improve the dataflow performance for larger data sets. |
| 131 | + |
| 132 | + > [!TIP] |
| 133 | + > You can go back to choose more tables in the **Get data** ribbon option for the same OData connector. |
| 134 | +
|
| 135 | +1. Select **Next** in the lower right. |
| 136 | + |
| 137 | +## Step 5: Configure target environment settings |
| 138 | + |
| 139 | +This section describes how to define the target environment settings. |
| 140 | + |
| 141 | +### Step 5.1: Map entities |
| 142 | + |
| 143 | +For each entity chosen, select the behavior for importing that entity in these settings and then select **Next**. |
| 144 | + |
| 145 | +> [!div class="mx-imgBorder"] |
| 146 | +>  |
| 147 | +
|
| 148 | +- **Load to existing entity (recommended)** |
| 149 | + |
| 150 | + - The dataflow syncs data from the source environment's entity to the target environment, and the same entity schema is already defined in the target environment. |
| 151 | + |
| 152 | + - Ideally, use the same solution in both target and source environments to make data transfer seamless. Another advantage to having a predefined entity is more control over which solution the entity is defined in and the prefix. |
| 153 | + |
| 154 | + - Choose **Delete rows that no longer exist in the query output**. This ensures that the relationships will map correctly because it maintains the values for the lookups. |
| 155 | + |
| 156 | + - If the schema is identical in both source and target tables, you can select **Auto map** to quickly map the fields. |
| 157 | + |
| 158 | + - Requires a key configuration in the target environment (as the unique identifier fields are not available to modify). |
| 159 | + |
| 160 | +- **Load to new entity (not recommended)** |
| 161 | + |
| 162 | + - Ideally there should be an entity predefined in the target environment from the same solution import as the source environment. However, there are cases where this might not be feasible, so this is an option if there is no existing entity to load to. |
| 163 | + |
| 164 | + - It creates a new custom entity in the target environment's default solution. |
| 165 | + |
| 166 | +- There is an option to **Do not load**, but do not include entities in the dataflow that are not being loaded. You can select **Back** from this menu to return to the Power Query menu and remove the entities that are not needed. |
| 167 | + |
| 168 | +### Step 5.2: Refresh settings |
| 169 | + |
| 170 | +Select **Refresh manually** since this is a one-time migration and then select **Create**. |
| 171 | + |
| 172 | +## Step 6: Run the dataflow |
| 173 | + |
| 174 | +The initial dataflow load initiates when you select **Create**. |
| 175 | + |
| 176 | +> [!div class="mx-imgBorder"] |
| 177 | +>  |
| 178 | +
|
| 179 | +You can manually initiate a dataflow by selecting **(...)** in the dataflows list. Make sure to run dependent dataflows after the parent flows have completed. |
| 180 | + |
| 181 | +> [!div class="mx-imgBorder"] |
| 182 | +>  |
| 183 | +
|
| 184 | +## Tips |
| 185 | + |
| 186 | +- Try out one entity first to walk through the steps, then build out all the dataflows. |
| 187 | + |
| 188 | +- If there are more entities that contain larger amounts of data, consider configuring multiple separate dataflows for individual entities. |
| 189 | + |
| 190 | +- One-to-many relationships will require separate dataflows for each entity. Configure and run the parent (aka one, or independently) entity dataflow before the child entity. |
| 191 | + |
| 192 | +- If there are errors with the dataflow refresh, you can view the refresh history in the **(...)** menu in the dataflows list and download each refresh log. |
| 193 | + |
| 194 | +## Limitations |
| 195 | + |
| 196 | +- Many-to-many relationship data imports are not supported. |
| 197 | + |
| 198 | +- Parent dataflows must be manually configured to run before child dataflows. |
0 commit comments