Merge pull request MicrosoftDocs#2176 from sama-zaki/live

Updating Ingest exported Microsoft Dataverse data with Azure Data Factory Documentation

Author: Mattp123

powerapps-docs/maker/data-platform/configure-rank.png

@@ -2,7 +2,7 @@
 title: "Ingest Microsoft Dataverse data with Azure Data Factory | MicrosoftDocs"
 description: Learn how to use Azure Data Factory to create dataflows, transform, and run analysis on Dataverse data
 ms.custom: ""
-ms.date: 07/29/2020
+ms.date: 03/22/2021
 ms.reviewer: "matp"
 author: sabinn-msft
 ms.service: powerapps
@@ -30,9 +30,9 @@ After exporting data from Dataverse to Azure Data Lake Storage Gen2 with the Exp
 
 This article shows you how to perform the following tasks: 
 
-1.  Generate a manifest.json from the existing model.json in the Data Lake Storage Gen2 storage account that holds the exported data.
+1.  Set the Data Lake Storage Gen2 storage account with the Dataverse data as a *source* in a Data Factory dataflow.
 
-2.  Set the Data Lake Storage Gen2 storage account with the Dataverse data as a *source* in a Data Factory dataflow.
+2.  Transform the Dataverse data in Data Factory with a dataflow.
 
 3.  Set the Data Lake Storage Gen2 storage account with the Dataverse data as a *sink* in a Data Factory dataflow.
 
@@ -41,78 +41,27 @@ This article shows you how to perform the following tasks:
 ## Prerequisites
 
 This section describes the prerequisites necessary to ingest exported Dataverse data with Data Factory.
- 
+
 ### Azure roles
 
 The user account that's used to sign in to Azure must be a member of the
 *contributor* or *owner* role, or an *administrator* of the Azure subscription.
 To view the permissions that you have in the subscription, go to the [Azure portal](https://portal.azure.com/), select your username in the upper-right corner, select **...**, and then select **My permissions**. If you have access to multiple subscriptions, select the appropriate one. To create and manage child resources for Data Factory in the Azure portal—including datasets, linked services, pipelines, triggers, and integration runtimes—you must belong to the *Data Factory Contributor* role at the resource group level or above.
 
 ### Export to data lake
-
-This article assumes that you've already exported Dataverse data by using the [Export to Data Lake service](export-to-data-lake.md).
+This guide assumes that you've already exported Dataverse data by using the [Export to Data Lake service](export-to-data-lake.md).
 
 In this example, account table data is exported to the data lake.
 
-## Generate the manifest.json from the model.json
-
-1.  Go to [this GitHub repository](https://github.com/t-sazaki/ConvertModelJsonToManifestOriginal) and download it to your computer.
-
-2.  Go to ConvertModelJsonToManifest-master/ConvertModelJsonToManifest-master/ConvertModelJsonToManifest.sln.
-
-3.  Right-click to select the file, and then open it in Visual Studio. If you don't have Visual Studio, you can follow this article to install it: [Install Visual Studio](/visualstudio/install/install-visual-studio?view=vs-2019&preserve-view=true).
-
-4.  Go to **Project** > **Manage NuGet Packages**, and ensure that the
-    following NuGet packages are installed:
-
-    -   Microsoft.CommonDataModel.ObjectModel
-
-    -   Microsoft.CommonDataModel.ObjectModel.Adapter.Adls
-
-    -   Microsoft.IdtableModel.Clients.ActiveDirectory
-
-    -   Newtonsoft.Json
-
-    -   NLog
-
-5. If you're missing the Common Data Model packages or they're unavailable, you can add them by following these steps:
-
-    1. Select the gear icon to access package settings.
-       ![Package settings gear icon](media/package-settings-gear.png "Package settings gear icon")
-
-    2. Select **+** in the pop-up window to add a new package source. 
-       ![Add a new package](media/add-new-package.png "Add a new package")
-
-    3.  Configure the new package source, and then select **OK**:
+### Azure Data Factory
 
-        1.  For **Name**, enter **CDM**.
-
-        2.  For **Source**, enter **https[]()://commondatamodel.pkgs.visualstudio.com/_packaging/CDM/nuget/v3/index.json**.
-
-    4.  Make sure that the package source is set to **All**.
-
-8.  In Program.cs, fill in the storage container information on line 26, as indicated here:
-
-    1. Replace **your-storage-account** by substituting the name of your storage account.
-       ![Your storage account substitution](media/your-storage-account.png "Your storage account substitution")
-
-    1. Replace **your-folder-name** with the folder containing the model.json file. Go to your storage account **Overview** > **Storage Explorer** > **Containers**, and then select the correct folder name. 
-      ![Replace your folder name](media/replace-your-folder-name.png "Replace your folder name")
-
-    1.  Replace the access key with the access key for this storage account. Go to your storage account, and on the left panel under **Settings**, select **Access Keys**. Select **Copy** to copy the access key and replace it in the code.
-
-9.  Optionally, you can change the name of the manifest file as indicated in the code comments.
-
-10.  Run the code, and refresh your storage container to find the new manifest, table, resolved table, and config files.
-
-> [!NOTE]
-> If there are changes made to the metadata of the table, you must delete the generated files from the Data Lake and regenerate an updated manifest file by running the code again. It is recommended that you maintain the same name of the manifest file, so there is no need to update any Azure Data Factory dataflows or pipelines.
+This guide assumes that you've already created a data factory under the same subscription and resource group as the storage account containing the exported Dataverse data.
 
 ## Set the Data Lake Storage Gen2 storage account as a source
 
-1.  Open [Azure Data Factory](https://ms-adf.azure.com/home?factory=%2Fsubscriptions%2Fd410b7d3-02af-45c8-895e-dc27c5b35342%2FresourceGroups%2Fsama%2Fproviders%2FMicrosoft.DataFactory%2Ffactories%2Fadfathena), and then select **Create data flow**. 
+1.  Open [Azure Data Factory](https://ms-adf.azure.com/en-us/datafactories) and select the data factory that is on the same subscription and resource group as the storage account containing your exported Dataverse data. Then select **Create data flow** from the home page. 
 
-2.  Turn on **Data flow debug** mode. This might take up to 10 minutes, but you
+2.  Turn on **Data flow debug** mode and select your preferred time to live. This might take up to 10 minutes, but you
     can proceed with the following steps.
 
     ![Dataflow debug mode](media/data-flow-debug.png "Dataflow debug mode")
@@ -121,7 +70,7 @@ In this example, account table data is exported to the data lake.
 
     ![Add source](media/add-source.png "Add source")
 
-4.  Under **Source settings**, do the following<!--Suggested. It's "configure the following options" here and "select the following options" in the next procedure, but these are a combination of entering and selecting.-->:
+4.  Under **Source settings**, do the following:
 
     - **Output stream name**: Enter the name you want. 
     - **Source type**: Select **Common Data Model**.
@@ -130,50 +79,60 @@ In this example, account table data is exported to the data lake.
 
 5.  Under **Source options**, do the following:
 
-    - **Metadata format**: Select **Manifest**. 
-    - **Root ___location**: In the first box (**Container**), enter the container name. In the second box (**Folder path**), enter **/**. 
-    - **Manifest file**: Leave the first box (**table path**) blank, and in the second box (**Manifest name (default)**), enter the first part of the manifest file name, such as *test.manifest.cdm.json* **/** *test*).
+    - **Metadata format**: Select **Model.json**. 
+    - **Root ___location**: Enter the container name in the first box (**Container**) or **Browse** for the container name and select **OK**.
+    - **Entity**: Enter the table name or **Browse** for the table.
+
+  ![Source options](media/source-options.png "Source options")
+  
+6.  Check the **Projection** tab to ensure that your schema has been imported sucessfully. If you do not see any columns, select **Schema options** and check the **Infer drifted column types** option. Configure the formatting options to match your data set then select **Apply**.
 
-       ![Source options, part one](media/source-options.png "Source options, part one")
+7. You can view your data in the **Data preview** tab to ensure the Source creation was complete and accurate.
 
-    - **Schema linked service**: Select the same storage container as the source settings.
-    - **Container**: Enter the container name.
-    - **Corpus folder**: Leave blank.
-    - **table**: Enter text in the format **/*table*Res.cdm.json/*table***, replacing *table* with the table name you want, such as account.
+## Transform your Dataverse data
+After setting the exported Dataverse data in the Data Lake Storage Gen2 storage account as a source in the Data Factory dataflow, there are many possibilities for transforming your data. More information: [Azure Data Factory](/azure/data-factory/introduction)
 
-       ![Source options, part two](media/source-options-two.png "Source options, part two")
+Follow these instructions to create a rank for the each row by the *revenue* of the account.
 
-## Set the Data Lake Storage Gen2 storage account 
+1. Select **+** in the lower-right corner of the previous transformation, and then search for and select **Rank**.
 
-After setting the exported Dataverse data in the Data Lake Storage Gen2 storage account as a source in the Data Factory dataflow, there are many possibilities for transforming your data. More information: [Azure Data Factory](/azure/data-factory/introduction)
+2. On the **Rank settings** tab, do the following:
+     - **Output stream name**: Enter the name you want, such as *Rank1*.
+     - **Incoming Stream**: Select the source name you want. In this case, the source name from the previous step.
+     - **Options**: Leave the options unchecked.
+     - **Rank column**: Enter the name of the rank column generated.
+     - **Sort conditions**: Select the *revenue* column and sorty by *Descending* order.
 
-Ultimately, you must set a sink for your dataflow. Follow these instructions to set the Data Lake Storage Gen2 storage account with the data exported by the Export to Data Lake service as your sink.
+       ![Configure the Rank settings tab](media/configure-rank.png "Configure the Rank settings tab")
 
-1.  Select **+** in the lower-right corner, and then search for and select **Sink**.
+3. You can view your data in the **data preview** tab where you will find the new *revenueRank* column at the right-most position.
+
+## Set the Data Lake Storage Gen2 storage account as a sink
+Ultimately, you must set a sink for your dataflow. Follow these instructions to place your transformed data as a Delimited Text file in the Data Lake.
+
+1.  Select **+** in the lower-right corner of the previous transformation, and then search for and select **Sink**.
 
 2.  On the **Sink** tab, do the following:
 
     - **Output stream name**: Enter the name you want, such as *Sink1*.
-    - **Incoming stream**: Select the source name you want. 
-    - **Sink type**: Select **Common Data Model**. 
+    - **Incoming stream**: Select the source name you want. In this case, the source name from the previous step.
+    - **Sink type**: Select **DelimitedText**. 
     - **Linked service**: Select your Data Lake Storage Gen2 storage container that has the data you exported by using the Export to Data Lake service.
 
       ![Configure the Sink tab](media/configure-sink.png "Configure the Sink tab")
 
 3. On the **Settings** tab, do the following:
 
-    - **Schema linked service**: Select the final destination storage container. 
-    - **Container**: Enter the container name. 
-    - **Corpus folder**: Enter **/** 
-    - **table**: Enter text in the format **/*table*Res.cdm.json/*table***, replacing *table* with the table name you want, such as account.
-
-      ![Configure the sink Settings tab, part one](media/configure-settings.png "Configure the sink Settings tab, part one")
+    - **Folder path**: Enter the container name in the first box (**File system**) or **Browse** for the container name and select **OK**.
+    - **File name option**: Select **output to single file**.
+    - **Output to single file**: Enter a file name, such as *ADFOutput*
+    - Leave all other default settings.
 
-    - **Root Location**: In the first box (**Container**), enter the container name. In the second box (**Folder path**), enter **/**. 
-    - **Manifest file**: Leave the first box (**table path**) blank, and in the second box (**Manifest name (default)**), enter the first part of the manifest file name, such as *test.manifest.cdm.json / test*.
-    - **Format type**: Select your file format preference.
+      ![Configure the sink Settings tab](media/configure-settings.png "Configure the sink Settings tab")
+      
+3. On the **Optimize** tab, set the **Partition option** to **Single partition**.
 
-      ![Configure the sink Settings tab, part two](media/configure-settings-two.png "Configure the sink Settings tab, part two")
+4. You can view your data in the **data preview** tab. 
 
 ## Run your dataflow
 
@@ -197,4 +156,4 @@ Ultimately, you must set a sink for your dataflow. Follow these instructions to
 [Analyze Dataverse data in Azure Data Lake Storage Gen2 with Power BI](export-to-data-lake-data-powerbi.md)
 
 
-[!INCLUDE[footer-include](../../includes/footer-banner.md)]
+[!INCLUDE[footer-include](../../includes/footer-banner.md)]