Live publish for 04 June 2024.

Author: bishalgoswami

powerapps-docs/developer/data-platform/TOC.yml

@@ -120,6 +120,20 @@
         - name: Sample code
           displayName: FetchXml, Sample code
           href: fetchxml/sample.md
+    - name: Query using wildcard characters
+      href: wildcard-characters.md
+    - name: Query throttling
+      href: query-throttling.md
+    - name: Query Hierarchical data
+      href: query-hierarchical-data.md
+    - name: Quick find
+      href: quick-find.md
+    - name: Saved Queries
+      href: saved-queries.md
+    - name: "Use SQL to query data"
+      href: dataverse-sql-query.md
+    - name: "Dataverse SQL"
+      href: how-dataverse-sql-differs-from-transact-sql.md
     - name: Bulk Operation messages
       displayName: CreateMultiple, UpdateMultiple, UpsertMultiple
       href: bulk-operations.md
@@ -145,32 +159,18 @@
               href: create-custom-api-solution.md
             - name: CustomAPI Tables
               href: custom-api-tables.md
+            - name: Use open types with custom APIs
+              href: use-open-types.md
             - name: "Sample: IsSystemAdmin custom API plug-in"
               href: org-service/samples/issystemadmin-customapi-sample-plugin.md
             - Name: "Sample: ExportDataUsingFetchXmlToAnnotation custom API plug-in"
               href: org-service/samples/export-data-fetchxml-annotation-custom-api-sample.md
         - name: Use Custom Process Actions with code
           href: workflow-custom-actions.md
-    - name: Use open types with custom APIs
-      href: use-open-types.md
     - name: Background operations (preview)
       href: background-operations.md
     - name: Use optional parameters
       href: optional-parameters.md
-    - name: Quick find
-      href: quick-find.md
-    - name: "Use SQL to query data"
-      href: dataverse-sql-query.md
-    - name: "Dataverse SQL"
-      href: how-dataverse-sql-differs-from-transact-sql.md
-    - name: Saved Queries
-      href: saved-queries.md
-    - name: Query using wildcard characters
-      href: wildcard-characters.md
-    - name: Query throttling
-      href: query-throttling.md
-    - name: Query Hierarchical data
-      href: query-hierarchical-data.md
     - name: Files and images overview
       href: files-images-overview.md
       items:

powerapps-docs/developer/data-platform/fetchxml/optimize-performance.md

@@ -20,6 +20,8 @@ contributors:
 
 This article describes ways you can optimize performance when retrieving data using FetchXml.
 
+[!INCLUDE [cc-query-antipatterns](../includes/cc-query-antipatterns.md)]
+
 ## Late Materialize query
 
 If you select many lookup and computed columns, and you're experiencing performance issues, you can try setting the [fetch element](reference/fetch.md) boolean `latematerialize` attribute. Behind the scenes, this setting breaks the query into smaller parts and reassembles the results before returning them to you.

powerapps-docs/developer/data-platform/includes/cc-query-antipatterns.md

@@ -0,0 +1,88 @@
+## Patterns to avoid
+
+Composing optimized queries for Dataverse is vital to ensure applications provide a fast, responsive, and reliable experience. This section describes patterns to avoid and concepts to understand when composing queries for standard tables using the `RetrieveMultiple` message, or messages that have a parameter that inherits from the [QueryBase class](/dotnet/api/microsoft.xrm.sdk.query.querybase). The guidance here might not apply for [Elastic tables](../elastic-tables.md) or when using [Dataverse Search](../search/overview.md).
+
+
+### Minimize the number of selected columns
+
+Don't include columns you don't need in your query. Queries that return all columns or include a large number of columns can encounter performance issues due to the size of the dataset or complexity of the query.
+
+This practice is especially true for *logical columns*. A logical column contains values that are stored in different database tables. The [AttributeMetadata.IsLogical property](/dotnet/api/microsoft.xrm.sdk.metadata.attributemetadata.islogical) tells you whether a column is a logical column. Queries that contain many logical columns are slower because Dataverse needs to combine the data from other database tables.
+
+
+### Avoid leading wild cards in filter conditions
+
+Queries that use conditions with leading wild cards (either explicitly, or implicitly with an operator like `ends-with`) can lead to bad performance. Dataverse can't take advantage of database indexes when a query using leading wild cards, which forces SQL to scan the entire table. Table scans can happen even if there are other non-leading wild card queries that limit the result set.
+
+The following example is a FetchXml [condition element](../fetchxml/reference/condition.md) that uses a leading wild card:
+
+```xml
+<condition attribute='accountnumber'
+   operator='like'
+   value='%234' />
+```
+
+The following example is a [QueryExpression](xref:Microsoft.Xrm.Sdk.Query.QueryExpression) [ConditionExpression](xref:Microsoft.Xrm.Sdk.Query.ConditionExpression) that uses a leading wild card:
+
+```csharp
+new ConditionExpression("accountnumber", ConditionOperator.Like, "%234")
+```
+
+When queries time out and this pattern is detected, Dataverse returns a unique error to help identify which queries are using this pattern:
+
+> Name: `LeadingWildcardCauseTimeout`<br />
+> Code: `0x80048573`<br />
+> Number: `-2147187341`<br />
+> Message: `The database operation timed out; this may be due to a leading wildcard value being used in a filter condition. Please consider removing filter conditions on leading wildcard values, as these filter conditions are expensive and may cause timeouts.`
+
+Dataverse heavily throttles leading wild card queries that are identified as a risk to the health of the org to help prevent outages. [Learn more about query throttling](../query-throttling.md)
+
+If you find yourself using leading wild card queries, look into these options:
+
+- Use [Dataverse search](../search/overview.md) instead.
+- Change your data model to help people avoid needing leading wild cards.
+
+[Learn more about using wildcard characters in conditions for string values](../wildcard-characters.md)
+
+
+### Avoid using formula or calculated columns in filter conditions
+
+[Formula and calculated column](../calculated-rollup-attributes.md#formula-and-calculated-columns) values are calculated in real-time when they're retrieved. Queries that use filters on these columns force Dataverse to calculate the value for each possible record that can be returned so the filter can be applied. Queries are slower because Dataverse can't improve the performance of these queries using SQL.
+
+When queries time out and this pattern is detected, Dataverse returns a unique error to help identify which queries are using this pattern:
+
+> Name: `ComputedColumnCauseTimeout`<br />
+> Code: `0x80048574`<br />
+> Number: `-2147187340`<br />
+> Message: `The database operation timed out; this may be due to a computed column being used in a filter condition. Please consider removing filter conditions on computed columns, as these filter conditions are expensive and may cause timeouts.`
+
+To help prevent outages, Dataverse applies throttles on queries that have filters on calculated columns that are identified as a risk to the health of the environment. [Learn more about query throttling](../query-throttling.md)
+
+
+### Avoid ordering by choice columns
+
+When you order query results using a choice column, the results are sorted using the localized label for each choice option. Ordering by the number value stored in the database wouldn't provide a good experience in your application. You should know that ordering on choice columns requires more compute resources to join and sort the rows by the localized label value. This extra work makes the query slower. If possible, try to avoid ordering results by choice column values.
+
+### Avoid ordering by columns in related tables
+
+Ordering by columns on related tables makes the query slower because of the added complexity.
+
+Ordering by related tables should only be done when needed to as described here:
+
+- [Order rows using FetchXml](../fetchxml/order-rows.md)
+- [Order rows using QueryExpression](../org-service/queryexpression/order-rows.md)
+
+### Avoid using conditions on large text columns
+
+Dataverse has two types of columns that can store large strings of text:
+
+- [StringAttributeMetadata](/dotnet/api/microsoft.xrm.sdk.metadata.stringattributemetadata) can store up to 4,000 characters.
+- [MemoAttributeMetadata](/dotnet/api/microsoft.xrm.sdk.metadata.memoattributemetadata) can store a higher number.
+
+The limit for both of these columns is specified using the `MaxLength` property.
+
+You can use conditions on string columns that have a `MaxLength` configured for fewer than 850 characters.
+
+All memo columns or string columns with a `MaxLength` greater than 850 are defined in Dataverse as large text columns. Large text columns are too large to effectively index, which leads to bad performance when included in a filter condition.
+
+Dataverse search is a better choice to query data in these kinds of columns.

powerapps-docs/developer/data-platform/org-service/queryexpression/optimize-performance.md

@@ -16,6 +16,8 @@ contributors:
 
 This article describes ways you can optimize performance when retrieving data using [QueryExpression](/dotnet/api/microsoft.xrm.sdk.query.queryexpression).
 
+[!INCLUDE [cc-query-antipatterns](../../includes/cc-query-antipatterns.md)]
+
 ## Query Hints
 
 > [!IMPORTANT]

powerapps-docs/developer/data-platform/query-throttling.md

@@ -1,87 +1,58 @@
 ---
-title: "Query throttling (Microsoft Dataverse)| Microsoft Docs"
+title: "Query throttling (Microsoft Dataverse)"
 description: "Read about how the server can throttle a query to reduce system performance impact and what you can do about it."
-ms.custom: ""
-ms.date: 04/29/2021
-
-ms.suite: ""
-ms.tgt_pltfrm: ""
-ms.topic: "article"
+ms.date: 06/04/2024
+ms.topic: article
 applies_to: 
   - "Dynamics 365 (online)"
-ms.assetid: fc3ade34-9c4e-4c33-88a4-aa3842c5eee1
-caps.latest.revision: 78
-author: "pnghub"
+author: pnghub
 ms.subservice: dataverse-developer
-ms.author: "gned"
-ms.reviewer: "pehecke"
+ms.author: gned
+ms.reviewer: pehecke
 search.audienceType: 
   - developer
 ---
 
 # Query throttling
 
-If a specific query creates a disproportional load on the database storing
-Microsoft Dataverse data, it can starve the database of resources and negatively impact
-performance of all data operations. When this happens Dataverse can start
-throttling that particular query which would allow all other scenarios to
-perform normally.
+If a specific query creates a disproportional load on the database storing Microsoft Dataverse data, it can starve the database of resources and negatively impact performance of all data operations. When this happens, Dataverse starts throttling that particular query to allow all other scenarios to perform normally.
 
-The primary way in which *query throttling* is different from 
-[Service protection API limits](api-limits.md) is
-query throttling targets a specific query that causes a performance degradation while
-leaving the rest of the traffic unaffected. If the throttled query
-originates from a non-interactive application, throttling is likely to not be
-noticeable for end-users. If the query originates from an interactive
-application it will affect users that exercise that particular scenario.
+The primary way in which *query throttling* is different from [Service protection API limits](api-limits.md) is
+query throttling targets a specific query that causes a performance degradation while leaving the rest of the traffic unaffected. If the throttled query originates from a non-interactive application, throttling is likely to not be noticeable for end-users. If the query originates from an interactive application, it affects users that exercise that particular scenario.
 
 ## Query throttling behavior
 
-Throttling can manifest in two ways:
+Throttling can manifest in three ways:
 
-- A delay is introduced before each execution of the query, making the
-    scenario that uses it slower
+- A delay is introduced before each execution of the query, making the scenario that uses it slower
+- Some fraction of attempts to execute the query are failing with any of the following errors:
 
-- Some fraction of attempts to execute the query are failing with a dedicated
-    error:
+|Error code|Hex code|Message|
+|---|---|---|
+|`-2147187388`|`0x80048544`| `This query cannot be executed because it conflicts with query throttling.`|
+|`-2147187132`|`0x80048644`| `This query cannot be executed because it conflicts with Query Throttling; the query uses a leading wildcard value in a filter condition, which will cause the query to be throttled more aggressively.`|
+|`-2147186876`|`0x80048744`| `This query cannot be executed because it conflicts with Query Throttling; the query uses a computed column in a filter condition, which will cause the query to be throttled more aggressively.` |
 
-| **Error code** | **Hex code** | **Message**                                                                                                                    |
-|----------------|--------------|--------------------------------------------------------------------------------------------------------------------------------|
-| \-2147187388   | 0x80048544   | This query cannot be executed because it conflicts with query throttling. Please refer to [Query throttling](query-throttling.md) |
+For more information about more aggressively throttled query patterns like leading wild cards can be found in [Optimize performance using FetchXml](fetchxml/optimize-performance.md) and [Optimize performance using QueryExpression](org-service/queryexpression/optimize-performance.md)
 
 ## Common causes
 
-Most of the situations when query throttling is necessary fall into one of the
-two broad categories:
+Most of the situations when query throttling is necessary fall into one of these two broad categories:
 
-- Some query in a common interactive scenario, for example a saved query used
-    in a grid or a query executed by a plug-in, is inefficient and requires a lot
-    of database resources for each execution
+- Some query in a common interactive scenario, for example a saved query used in a grid or a query executed by a plug-in, is inefficient and requires a lot of database resources for each execution
 
-- An automated operation, for example data integration involving moving a
-    large amount of data into or out of Dataverse, executes a query at a very
-    high rate which consumes a lot of database resources in aggregate even if
-    each execution is cheap
+- An automated operation, for example data integration involving moving a large amount of data into or out of Dataverse, executes a query at a high rate that consumes a lot of database resources in aggregate even if each execution is less expensive
 
 ## How to avoid query throttling
 
-Query throttling depends on the query and the scenario where it is executed but there are some common guidelines:
-
-- For slow low-frequency queries, typically used in interactive
-    applications, the query structure needs to be changed to make it more
-    efficient
-
-    - Some common guidelines for improving the query performance can be found
-        in [Optimize performance using FetchXml](fetchxml/optimize-performance.md)
-
-- For non-interactive applications the common ways to reduce the database load
-    are:
+Query throttling depends on the query and the scenario where it's executed but there are some common guidelines:
 
-    - If [ExecuteMultiple](xref:Microsoft.Xrm.Sdk.Messages.ExecuteMultipleRequest) (or another batching mechanism) is used, the batch
-        size can be reduced
+- For slow low-frequency queries, typically used in interactive applications, the query structure needs to be changed to make it more efficient
 
-    - If the application uses concurrency, the number of threads can be reduced
+  - Some common guidelines for improving the query performance can be found in [Optimize performance using FetchXml](fetchxml/optimize-performance.md)
 
-    - A delay between requests can be introduced to reduce the request rate
-        when neither batching nor concurrency are used
+- For non-interactive applications the common ways to reduce the database load are:
 
+  - When using [ExecuteMultiple](xref:Microsoft.Xrm.Sdk.Messages.ExecuteMultipleRequest) (or another batching mechanism), reduce the size of the batch
+  - If the application is multi-threaded, reduce the number of concurrent threads
+  - If you aren't using batching or concurrent requests, add a delay between requests to reduce the request rate