You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: powerapps-docs/maker/common-data-service/export-to-data-lake-data-adf.md
+63-64Lines changed: 63 additions & 64 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,7 +22,7 @@ search.app:
22
22
23
23
# Ingest exported Common Data Service data with Azure Data Factory
24
24
25
-
After exporting data from Common Data Service to Azure Data Lake Storage Gen2 with the export to data lake service, you can use Azure Data Factory to create dataflows, transform your data, and run analysis.
25
+
After exporting data from Common Data Service to Azure Data Lake Storage Gen2 with the Export to Data Lake service, you can use Azure Data Factory to create dataflows, transform your data, and run analysis.
26
26
27
27
This article shows you how to perform the following tasks:
28
28
@@ -32,33 +32,34 @@ This article shows you how to perform the following tasks:
32
32
33
33
3. Set the Data Lake Storage Gen2 storage account with the Common Data Service data as a *sink* in a Data Factory dataflow.
34
34
35
-
4. Run your data flow by creating a pipeline.
35
+
4. Run your dataflow by creating a pipeline.
36
36
37
37
## Prerequisites
38
-
This section describes the prerequisites necessary to ingest export to Azure data lake data with Data Factory.
38
+
39
+
This section describes the prerequisites necessary to ingest exported Common Data Service data with Data Factory.
39
40
40
41
### Azure roles
41
42
42
-
The user account that is used to sign into Azure must be a member of the
43
+
The user account that's used to sign in to Azure must be a member of the
43
44
*contributor* or *owner* role, or an *administrator* of the Azure subscription.
44
-
To view the permissions that you have in the subscription, go to the [Azure portal](https://portal.azure.com/), select your user name in the upper-right corner, select **...**, and then select **My permissions**. If you have access to multiple subscriptions, select the appropriate subscription. To create and manage child resources for Data Factory in the Azure portal - including datasets, linked services, pipelines, triggers, and integration runtimes - you must belong to the *Data Factory Contributor* role at the resource group level or above.
45
+
To view the permissions that you have in the subscription, go to the [Azure portal](https://portal.azure.com/), select your username in the upper-right corner, select **...**, and then select **My permissions**. If you have access to multiple subscriptions, select the appropriate one. To create and manage child resources for Data Factory in the Azure portal—including datasets, linked services, pipelines, triggers, and integration runtimes—you must belong to the *Data Factory Contributor* role at the resource group level or above.
45
46
46
47
### Export to data lake
47
48
48
-
This article assumes that you have already exported Common Data Service data using the [Export to data lake service](export-to-data-lake.md).
49
+
This article assumes that you've already exported Common Data Service data by using the [Export to Data Lake service](export-to-data-lake.md).
49
50
50
51
In this example, account entity data is exported to the data lake.
51
52
52
53
## Generate the manifest.json from the model.json
53
54
54
-
1.Access [this GitHub repository](https://github.com/t-sazaki/ConvertModelJsonToManifestOriginal) and download the repository to your computer.
55
+
1.Go to [this GitHub repository](https://github.com/t-sazaki/ConvertModelJsonToManifestOriginal) and download it to your computer.
55
56
56
-
2.Navigate to ConvertModelJsonToManifest-master/ConvertModelJsonToManifest-master/ConvertModelJsonToManifest.sln.
57
+
2.Go to ConvertModelJsonToManifest-master/ConvertModelJsonToManifest-master/ConvertModelJsonToManifest.sln.
57
58
58
-
3. Right-click the file and open it in Visual Studio. If you do not have Visual Studio, you can follow this article to install it: [Install Visual Studio](/visualstudio/install/install-visual-studio?view=vs-2019)
59
+
3. Right-click to select the file, and then open it in Visual Studio. If you don't have Visual Studio, you can follow this article to install it: [Install Visual Studio](/visualstudio/install/install-visual-studio?view=vs-2019).
59
60
60
-
4.Access **Project** > **Manage NuGet Packages…** and ensure that the
61
-
following NuGet Packages are installed:
61
+
4.Go to **Project** > **Manage NuGet Packages**, and ensure that the
62
+
following NuGet packages are installed:
62
63
63
64
- Microsoft.CommonDataModel.ObjectModel
64
65
@@ -70,122 +71,120 @@ In this example, account entity data is exported to the data lake.
70
71
71
72
- NLog
72
73
73
-
5. If you are missing the Common Data Model packages or they are unavailable, you can add them by following these steps:
74
+
5. If you're missing the Common Data Model packages or they're unavailable, you can add them by following these steps:
74
75
75
-
a. Select the **gear** to access package settings.
b. Replace *your-folder-name* with the folder containing the **model.json** file. Navigate to your storage account **Overview** > **Storage Explorer** > **Containers** and select the correct folder name.
95
-
95
+
1. Replace **your-folder-name** with the folder containing the model.json file. Go to your storage account **Overview** > **Storage Explorer** > **Containers**, and then select the correct folder name.
96
+
![Replace your folder name](media/replace-your-folder-name.png"Replace your folder name")
96
97
97
-
c. Replace the access key with the access key for this storage account.Navigate to your storage account and on the left panel under **Settings** select **Access Keys**. Then select**Copy** to copy the access key and replace it in the code.
98
+
1. Replace the access key with the access key for this storage account. Go to your storage account, and on the left panel under **Settings**, select **Access Keys**. Select**Copy** to copy the access key and replace it in the code.
98
99
99
-
9. Optionally, change the name of the manifest file as indicated in the code
100
-
comments.
100
+
9. Optionally, you can change the name of the manifest file as indicated in the code comments.
101
101
102
-
10. Run the code and refresh your storage container to find the new manifest,
103
-
entity, resolved entity, and config files.
102
+
10. Run the code, and refresh your storage container to find the new manifest, entity, resolved entity, and config files.
104
103
105
-
## Set the Data Lake Storage Gen2 storage account as a source
104
+
## Set the Data Lake Storage Gen2 storage account as a source
106
105
107
-
1. Open [Azure Data Factory](https://ms-adf.azure.com/home?factory=%2Fsubscriptions%2Fd410b7d3-02af-45c8-895e-dc27c5b35342%2FresourceGroups%2Fsama%2Fproviders%2FMicrosoft.DataFactory%2Ffactories%2Fadfathena) and select **Create data flow**.
106
+
1. Open [Azure Data Factory](https://ms-adf.azure.com/home?factory=%2Fsubscriptions%2Fd410b7d3-02af-45c8-895e-dc27c5b35342%2FresourceGroups%2Fsama%2Fproviders%2FMicrosoft.DataFactory%2Ffactories%2Fadfathena), and then select **Create data flow**.
108
107
109
-
2. Turn on **Data flow debug** mode. This may take up to ten minutes, but you
108
+
2. Turn on **Data flow debug** mode. This might take up to 10 minutes, but you
4. Under **Source Settings**configure the following options:
117
+
4. Under **Source settings**, do the following<!--Suggested. It's "configure the following options" here and "select the following options" in the next procedure, but these are a combination of entering and selecting.-->:
119
118
120
119
-**Output stream name**: Enter the name you want.
121
120
-**Source type**: Select **Common Data Model**.
122
-
-**Linked Service**: Select the storage account from the dropdown menu and then link a new service by providing you subscription details and leaving all default configurations.
121
+
-**Linked Service**: Select the storage account from the drop-down menu, and then link a new service by providing your subscription details and leaving all default configurations.
123
122
-**Sampling**: If you want to use all your data, select **Disable**.
124
123
125
-
5. Under **Source options** configure the following options:
124
+
5. Under **Source options**, do the following:
126
125
127
126
-**Metadata format**: Select **Manifest**.
128
-
-**Root Location**: Set the first box (container) to the container name and the second box (folder path) to**/**.
129
-
-**Manifest file**: Leave the first box **Entity path** blank and set the second box (Manifest name (default)) to the first part of the manifest file name, such as *test.manifest.cdm.json***/***test*).
127
+
-**Root ___location**: In the first box (**Container**), enter the container name. In the second box (**Folder path**), enter**/**.
128
+
-**Manifest file**: Leave the first box (**Entity path**) blank, and in the second box (**Manifest name (default)**), enter the first part of the manifest file name, such as *test.manifest.cdm.json***/***test*).
130
129
131
-
130
+
![Source options, part one](media/source-options.png"Source options, part one")
132
131
133
132
-**Schema linked service**: Select the same storage container as the source settings.
134
-
-**Container**: Enter the container name.
133
+
-**Container**: Enter the container name.
135
134
-**Corpus folder**: Leave blank.
136
-
-**Entity**: Enter in */entityRes.cdm.json/entity* format and replace *entity* with the desired entity name, such as account.
135
+
-**Entity**: Enter text in the format **/*entity*Res.cdm.json/*entity***, replacing *entity* with the entity name you want, such as account.
137
136
138
-
137
+
![Source options, part two](media/source-options-two.png"Source options, part two")
139
138
140
139
## Set the Data Lake Storage Gen2 storage account
141
140
142
-
After setting the exported Common Data Service data in the Data Lake Storage Gen2 storage account as a source in the Data Factory dataflow, there are many possibilities with how to transform your data. For more information, see [Azure Data Factory](/azure/data-factory/introduction).
141
+
After setting the exported Common Data Service data in the Data Lake Storage Gen2 storage account as a source in the Data Factory dataflow, there are many possibilities for transforming your data. More information: [Azure Data Factory](/azure/data-factory/introduction)
143
142
144
-
Ultimately, you must set a sink for your dataflow. Follow the
145
-
below instruction to set the Data Lake Storage Gen2 storage account with your export to data lake data as your sink.
143
+
Ultimately, you must set a sink for your dataflow. Follow these instructions to set the Data Lake Storage Gen2 storage account with the data exported by the Export to Data Lake service as your sink.
146
144
147
-
1. Select **+**on the bottom right corner. Then, search for and select **Sink**.
145
+
1. Select **+**in the lower-right corner, and then search for and select **Sink**.
148
146
149
-
2. On the **Sink** tab, select the following options:
147
+
2. On the **Sink** tab, do the following:
150
148
151
149
-**Output stream name**: Enter the name you want, such as *Sink1*.
152
150
-**Incoming stream**: Select the source name you want.
153
151
-**Sink type**: Select **Common Data Model**.
154
-
-**Linked service**: Select your ADLS Gen2 storage container with your export to data lake data.
152
+
-**Linked service**: Select your Data Lake Storage Gen2 storage container that has the data you exported by using the Export to Data Lake service.
155
153
156
-
154
+
![Configure the Sink tab](media/configure-sink.png"Configure the Sink tab")
157
155
158
-
3. On the **Settings** tab, select the following options:
156
+
3. On the **Settings** tab, do the following:
159
157
160
158
-**Schema linked service**: Select the final destination storage container.
161
-
-**Container**: Set the container name.
159
+
-**Container**: Enter the container name.
162
160
-**Corpus folder**: Enter **/**
163
-
-**Entity**: Enter in */entityRes.cdm.json/entity* format and replace *entity* with the desired entity name, such as account.
161
+
-**Entity**: Enter text in the format **/*entity*Res.cdm.json/*entity***, replacing *entity* with the entity name you want, such as account.
![Configure the sink Settings tab, part one](media/configure-settings.png"Configure the sink Settings tab, part one")
166
164
167
-
-**Root Location**: Set the first box (container) to the container name and the second box (folder path) to */*.
168
-
-**Manifest file**: Leave the first box (entity path) blank and set the second box (manifest name (default)) to the first part of the manifest file name, such as *test.manifest.cdm.json / test*.
165
+
-**Root Location**: In the first box (**Container**), enter the container name. In the second box (**Folder path**), enter **/**.
166
+
-**Manifest file**: Leave the first box (**Entity path**) blank, and in the second box (**Manifest name (default)**), enter the first part of the manifest file name, such as *test.manifest.cdm.json / test*.
169
167
-**Format type**: Select your file format preference.
170
168
171
-
169
+
![Configure the sink Settings tab, part two](media/configure-settings-two.png"Configure the sink Settings tab, part two")
172
170
173
171
## Run your dataflow
174
172
175
-
1.On the left navigation pane under **Factory Resources**, select **+**, and then select **Pipeline** to create a new pipeline.
173
+
1.In the left pane under **Factory Resources**, select **+**, and then select **Pipeline**.
176
174
177
-
175
+
![Create a new pipeline](media/create-pipeline.png"Create a new pipeline")
178
176
179
177
2. Under **Activities**, select **Move & Transform**, and then drag **Data flow** to the workspace.
180
178
181
-
3. Select **Use existing data flow** and select the name of the data flow
179
+
3. Select **Use existing data flow**, and then select the dataflow that you
182
180
created in the previous steps.
183
181
184
182
4. Select **Debug** from the command bar.
185
183
186
-
5. Let the data flow run until the bottom view shows completion. This may take a few minutes.
184
+
5. Let the dataflow run until the bottom view shows that is has been completed. This might take a few minutes.
187
185
188
-
6.Navigate the final destination storage container and find the transformed entity data file.
186
+
6.Go to the final destination storage container, and find the transformed entity data file.
189
187
190
188
### See also
191
-
[Analyze Common Data Service data in Azure Data Lake Storage Gen2 with Power BI](export-to-data-lake-data-powerbi.md)
189
+
190
+
[Analyze Common Data Service data in Azure Data Lake Storage Gen2 with Power BI](export-to-data-lake-data-powerbi.md)
0 commit comments