Merge pull request #3226 from MicrosoftDocs/matp-export-data-lake

Matp export data lake

Author: Mattp123

powerapps-docs/maker/TOC.yml

@@ -1712,7 +1712,13 @@
   - name: Import translated entity and field text
     href: ./common-data-service/import-translated-entity-field-text.md
   - name: Export to data lake
-    href: ./common-data-service/export-to-data-lake.md
+    items: 
+    - name: Export to Azure Data Lake Storage Gen2
+      href: ./common-data-service/export-to-data-lake.md
+    - name: Ingest exported data with Azure Data Factory
+      href: ./common-data-service/export-to-data-lake-data-adf.md
+    - name: Analyze exported data with Power BI
+      href: ./common-data-service/export-to-data-lake-data-powerbi.md    
   - name: Security in Common Data Service
     href: /power-platform/admin/wp-security
   - name: API limits overview

powerapps-docs/maker/common-data-service/export-to-data-lake-data-adf.md

@@ -0,0 +1,190 @@
+---
+title: "Ingest Common Data Service data with Azure Data Factory | MicrosoftDocs"
+ms.custom: ""
+ms.date: 07/29/2020
+ms.reviewer: "matp"
+author: sabinn-msft
+ms.service: powerapps
+ms.suite: ""
+ms.tgt_pltfrm: ""
+ms.topic: "article"
+applies_to: 
+  - "powerapps"
+ms.assetid: 
+ms.author: "matp"
+manager: "kvivek"
+search.audienceType: 
+  - maker
+search.app: 
+  - PowerApps
+  - D365CE
+---
+
+# Ingest exported Common Data Service data with Azure Data Factory
+
+After exporting data from Common Data Service to Azure Data Lake Storage Gen2 with the Export to Data Lake service, you can use Azure Data Factory to create dataflows, transform your data, and run analysis.
+
+This article shows you how to perform the following tasks: 
+
+1.  Generate a manifest.json from the existing model.json in the Data Lake Storage Gen2 storage account that holds the exported data.
+
+2.  Set the Data Lake Storage Gen2 storage account with the Common Data Service data as a *source* in a Data Factory dataflow.
+
+3.  Set the Data Lake Storage Gen2 storage account with the Common Data Service data as a *sink* in a Data Factory dataflow.
+
+4.  Run your dataflow by creating a pipeline.
+
+## Prerequisites
+
+This section describes the prerequisites necessary to ingest exported Common Data Service data with Data Factory.
+ 
+### Azure roles
+
+The user account that's used to sign in to Azure must be a member of the
+*contributor* or *owner* role, or an *administrator* of the Azure subscription.
+To view the permissions that you have in the subscription, go to the [Azure portal](https://portal.azure.com/), select your username in the upper-right corner, select **...**, and then select **My permissions**. If you have access to multiple subscriptions, select the appropriate one. To create and manage child resources for Data Factory in the Azure portal—including datasets, linked services, pipelines, triggers, and integration runtimes—you must belong to the *Data Factory Contributor* role at the resource group level or above.
+
+### Export to data lake
+
+This article assumes that you've already exported Common Data Service data by using the [Export to Data Lake service](export-to-data-lake.md).
+
+In this example, account entity data is exported to the data lake.
+
+## Generate the manifest.json from the model.json
+
+1.  Go to [this GitHub repository](https://github.com/t-sazaki/ConvertModelJsonToManifestOriginal) and download it to your computer.
+
+2.  Go to ConvertModelJsonToManifest-master/ConvertModelJsonToManifest-master/ConvertModelJsonToManifest.sln.
+
+3.  Right-click to select the file, and then open it in Visual Studio. If you don't have Visual Studio, you can follow this article to install it: [Install Visual Studio](/visualstudio/install/install-visual-studio?view=vs-2019).
+
+4.  Go to **Project** > **Manage NuGet Packages**, and ensure that the
+    following NuGet packages are installed:
+
+    -   Microsoft.CommonDataModel.ObjectModel
+
+    -   Microsoft.CommonDataModel.ObjectModel.Adapter.Adls
+
+    -   Microsoft.IdentityModel.Clients.ActiveDirectory
+
+    -   Newtonsoft.Json
+
+    -   NLog
+
+5. If you're missing the Common Data Model packages or they're unavailable, you can add them by following these steps:
+
+    1. Select the gear icon to access package settings.
+       ![Package settings gear icon](media/package-settings-gear.png "Package settings gear icon")
+
+    2. Select **+** in the pop-up window to add a new package source. 
+       ![Add a new package](media/add-new-package.png "Add a new package")
+
+6.  Configure the new package source, and then select **OK**:
+
+    1.  For **Name**, enter **CDM**.
+
+    2.  For **Source**, enter **https[]()://commondatamodel.pkgs.visualstudio.com/_packaging/CDM/nuget/v3/index.json**.
+
+7.  Make sure that the package source is set to **All**.
+
+8.  In Program.cs, fill in the storage container information on line 26, as indicated here:
+
+    1. Replace <span><b>your-storage-account.dfs.core.windows.net</b></span> by substituting the name of your storage account.
+       ![Your storage account substitution](media/your-storage-account.png "Your storage account substitution")
+
+    1. Replace **your-folder-name** with the folder containing the model.json file. Go to your storage account **Overview** > **Storage Explorer** > **Containers**, and then select the correct folder name. 
+      ![Replace your folder name](media/replace-your-folder-name.png "Replace your folder name")
+
+    1.  Replace the access key with the access key for this storage account. Go to your storage account, and on the left panel under **Settings**, select **Access Keys**. Select **Copy** to copy the access key and replace it in the code.
+
+9.  Optionally, you can change the name of the manifest file as indicated in the code comments.
+
+10.  Run the code, and refresh your storage container to find the new manifest, entity, resolved entity, and config files.
+
+## Set the Data Lake Storage Gen2 storage account as a source
+
+1.  Open [Azure Data Factory](https://ms-adf.azure.com/home?factory=%2Fsubscriptions%2Fd410b7d3-02af-45c8-895e-dc27c5b35342%2FresourceGroups%2Fsama%2Fproviders%2FMicrosoft.DataFactory%2Ffactories%2Fadfathena), and then select **Create data flow**. 
+
+2.  Turn on **Data flow debug** mode. This might take up to 10 minutes, but you
+    can proceed with the following steps.
+
+    ![Dataflow debug mode](media/data-flow-debug.png "Dataflow debug mode")
+
+3.  Select **Add Source.**
+
+    ![Add source](media/add-source.png "Add source")
+
+4.  Under **Source settings**, do the following<!--Suggested. It's "configure the following options" here and "select the following options" in the next procedure, but these are a combination of entering and selecting.-->:
+
+    - **Output stream name**: Enter the name you want. 
+    - **Source type**: Select **Common Data Model**.
+    - **Linked Service**: Select the storage account from the drop-down menu, and then link a new service by providing your subscription details and leaving all default configurations.
+    - **Sampling**: If you want to use all your data, select **Disable**.
+
+5.  Under **Source options**, do the following:
+
+    - **Metadata format**: Select **Manifest**. 
+    - **Root ___location**: In the first box (**Container**), enter the container name. In the second box (**Folder path**), enter **/**. 
+    - **Manifest file**: Leave the first box (**Entity path**) blank, and in the second box (**Manifest name (default)**), enter the first part of the manifest file name, such as *test.manifest.cdm.json* **/** *test*).
+
+       ![Source options, part one](media/source-options.png "Source options, part one")
+
+    - **Schema linked service**: Select the same storage container as the source settings.
+    - **Container**: Enter the container name.
+    - **Corpus folder**: Leave blank.
+    - **Entity**: Enter text in the format **/*entity*Res.cdm.json/*entity***, replacing *entity* with the entity name you want, such as account.
+
+       ![Source options, part two](media/source-options-two.png "Source options, part two")
+
+## Set the Data Lake Storage Gen2 storage account 
+
+After setting the exported Common Data Service data in the Data Lake Storage Gen2 storage account as a source in the Data Factory dataflow, there are many possibilities for transforming your data. More information: [Azure Data Factory](/azure/data-factory/introduction)
+
+Ultimately, you must set a sink for your dataflow. Follow these instructions to set the Data Lake Storage Gen2 storage account with the data exported by the Export to Data Lake service as your sink.
+
+1.  Select **+** in the lower-right corner, and then search for and select **Sink**.
+
+2.  On the **Sink** tab, do the following:
+
+    - **Output stream name**: Enter the name you want, such as *Sink1*.
+    - **Incoming stream**: Select the source name you want. 
+    - **Sink type**: Select **Common Data Model**. 
+    - **Linked service**: Select your Data Lake Storage Gen2 storage container that has the data you exported by using the Export to Data Lake service.
+
+      ![Configure the Sink tab](media/configure-sink.png "Configure the Sink tab")
+
+3. On the **Settings** tab, do the following:
+
+    - **Schema linked service**: Select the final destination storage container. 
+    - **Container**: Enter the container name. 
+    - **Corpus folder**: Enter **/** 
+    - **Entity**: Enter text in the format **/*entity*Res.cdm.json/*entity***, replacing *entity* with the entity name you want, such as account.
+
+      ![Configure the sink Settings tab, part one](media/configure-settings.png "Configure the sink Settings tab, part one")
+
+    - **Root Location**: In the first box (**Container**), enter the container name. In the second box (**Folder path**), enter **/**. 
+    - **Manifest file**: Leave the first box (**Entity path**) blank, and in the second box (**Manifest name (default)**), enter the first part of the manifest file name, such as *test.manifest.cdm.json / test*.
+    - **Format type**: Select your file format preference.
+
+      ![Configure the sink Settings tab, part two](media/configure-settings-two.png "Configure the sink Settings tab, part two")
+
+## Run your dataflow
+
+1.  In the left pane under **Factory Resources**, select **+**, and then select **Pipeline**.
+
+     ![Create a new pipeline](media/create-pipeline.png "Create a new pipeline")
+
+2.  Under **Activities**, select **Move & Transform**, and then drag **Data flow** to the workspace.
+
+3.  Select **Use existing data flow**, and then select the dataflow that you
+    created in the previous steps.
+
+4.  Select **Debug** from the command bar.
+
+5.  Let the dataflow run until the bottom view shows that is has been completed. This might take a few minutes.
+
+6.  Go to the final destination storage container, and find the transformed entity data file.
+
+### See also
+
+[Analyze Common Data Service data in Azure Data Lake Storage Gen2 with Power BI](export-to-data-lake-data-powerbi.md)

powerapps-docs/maker/common-data-service/export-to-data-lake-data-powerbi.md

@@ -0,0 +1,86 @@
+---
+title: "Analyze Common Data Service exported to Azure Data Lake Storage Gen2 data with Power BI | MicrosoftDocs"
+ms.custom: ""
+ms.date: 07/29/2020
+ms.reviewer: "matp"
+author: sabinn-msft
+ms.service: powerapps
+ms.suite: ""
+ms.tgt_pltfrm: ""
+ms.topic: "article"
+applies_to: 
+  - "powerapps"
+ms.assetid: 
+ms.author: "matp"
+manager: "kvivek"
+search.audienceType: 
+  - maker
+search.app: 
+  - PowerApps
+  - D365CE
+---
+
+# Analyze Common Data Service data in Azure Data Lake Storage Gen2 with Power BI
+
+After exporting data from Common Data Service to Azure Data Lake Storage Gen2
+with the Export to Data Lake service, you can use Power BI to create business
+reports and analytics. This can be useful for sales managers and sales associates to refine and build additional reports and dashboards in Power BI.
+
+This article shows you how to perform the following tasks: 
+
+1. Connect the Data Lake Storage Gen2 storage container containing the exported Common Data Service data to Power BI.
+
+2. Create a report in Power BI that graphs account owners and their respective total account revenue.
+
+## Prerequisites
+
+This section describes the prerequisites necessary to consume Common Data Service data with Power BI by using the Export to Data Lake service.
+-  **Power BI Desktop**. [Get it now](https://powerbi.microsoft.com/downloads/)
+
+-  **Export to data lake:** This guide assumes that you have already exported data from Common Data Service by using the [Export to Data Lake service](export-to-data-lake.md). In this example, account entity data is exported to the data lake and will generate a report by using the account entity fields.
+
+## Connect the Data Lake Storage Gen2 storage to Power BI Desktop
+
+1. Open Power BI Desktop, and sign in.
+
+2. Select **Get data** > **More**.
+
+3. On the **Get Data** page, select **Azure** > **Azure Data Lake Storage Gen2** > **Connect.**
+
+4.  Enter the storage container name in the format<br>*https://accountname.dfs.core.windows.net/containername/*
+
+5. Find the storage account name by going to **Azure** > **Storage accounts**, and then select the Data Lake Storage Gen2 account that's populated with the exported Common Data Services data.
+
+6. Find the container name by going to **Storage Explorer (preview)** > **Containers**.
+
+    ![Find the storage container name](media/find-container-name.png "Find the storage container name")
+
+7. Select **CDM Folder View (Beta)**, and then select **OK**.
+
+8. If prompted to sign in, do so, and then select **Access key** from the left side bar.
+
+9. Find the access key by going back to the storage account setting and selecting **Access keys** from the left pane. Copy the first key.
+
+    ![Copy the access key](media/copy-access-key.png "Copy the access key")
+
+10. Expand **cdm**, select the **account** data, and then select **Load**.
+
+    ![Load account data](media/load-account-data.png "Load account data")
+
+## Create a Power BI report with account revenue by account owner
+
+1. On the **Visualizations** menu, select **Pie Chart**.
+
+    ![Pie chart](media/pie-chart.png "Pie chart")
+
+2. On the **Fields** menu, search for and select the fields **ownerid** and **revenue**. The fields will appear on the **Visualizations** menu under **Legend** and **Values**, respectively.
+
+    ![Search for and select the revenue field](media/select-fields.png "Search for and select the revenue field")
+
+   A pie chart appears that separates each account owner by color and displays the total revenue of accounts.
+
+    ![Colorful pie chart showing the total revenue and owner of ten individual accounts](media/account-data-pie-chart.png "Colorful pie chart showing the total revenue and owner of ten individual accounts")
+
+### See also
+
+[Export entity data to Azure Data Lake Storage Gen2](export-to-data-lake.md)