You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[DOCS] Amends data frame analytics overview and adds resources section (elastic#1726)
* Amends data frame analytics overview.
* Adds metadata to How DFA works page.
* Renames Concepts to Advanced concepts.
* Adds DFA at scale link to Advanced concepts.
* Adds Resources section.
* Changes link on DFA main page.
These are iterative stages, meaning that after evaluating each step, you might
63
+
need to make adjustments before you move further.
64
+
65
+
[discrete]
66
+
[[define-problem]]
67
+
=== Define the problem
68
+
69
+
It’s important to take a moment and think about where {ml} can be most
70
+
impactful. Consider what type of data you have available and what value it
71
+
holds. The better you know the data, the quicker you will be able to create {ml}
72
+
models that generate useful insights. What kinds of patterns do you want to
73
+
discover in your data? What type of value do you want to predict: a category, or
74
+
a numerical value? The answers help you choose the type of analysis that fits
75
+
your use case.
76
+
77
+
After you identify the problem, consider which of the {ml-features} are most
78
+
likely to help you solve it. Supervised learning requires a data set that
79
+
contains known values that the model can be trained on. Unsupervised learning –
80
+
like {anomaly-detect} or {oldetection} – does not have this requirement.
81
+
82
+
{stack} provides the following types of supervised learning:
83
+
84
+
* {regression}: predicts **continuous, numerical values** like the response time
85
+
of a web request.
86
+
* {classification}: predicts **discrete, categorical values** like whether a
87
+
https://www.elastic.co/blog/machine-learning-in-cybersecurity-training-supervised-models-to-detect-dga-activity[DNS request originates from a malicious or benign ___domain].
88
+
89
+
90
+
[discrete]
91
+
[[prepare-transform-data]]
92
+
=== Prepare and transform data
93
+
94
+
You have defined the problem and selected an appropriate type of analysis. The
95
+
next step is to produce a high-quality data set in {es} with a clear
96
+
relationship to your training objectives. If your data is not already in {es},
97
+
this is the stage where you develop your data pipeline. If you want to learn
98
+
more about how to ingest data into {es}, refer to the
99
+
{ref}/ingest.html[Ingest node documentation].
100
+
101
+
{regression-cap} and {classification} are supervised {ml} techniques, therefore
102
+
you must supply a labelled data set for training. This is often called the
103
+
"ground truth". The training process uses this information to identify
104
+
relationships among the various characteristics of the data and the predicted
105
+
value. It also plays a critical role in model evaluation.
106
+
107
+
An important requirement is a data set that is large enough to train a model.
108
+
For example, if you would like to train a {classification} model that decides
109
+
whether an email is a spam or not, you need a labelled data set that contains
110
+
enough data points from each possible category to train the model. What counts
111
+
as "enough" depends on various factors like the complexity of the problem or
112
+
the {ml} solution you have chosen. There is no exact number that fits every
113
+
use case; deciding how much data is acceptable is rather a heuristic process
114
+
that might involve iterative trials.
115
+
116
+
Before you train the model, consider preprocessing the data. In practice, the
117
+
type of preprocessing depends on the nature of the data set. Preprocessing can
118
+
include, but is not limited to, mitigating redundancy, reducing biases, applying
119
+
standards and/or conventions, data normalization, and so on.
120
+
121
+
{regression-cap} and {classification} require specifically structured source
122
+
data: a two dimensional tabular data structure. For this reason, you might need
123
+
to {ref}/transforms.html[{transform}] your data to create a {dataframe} which
124
+
can be used as the source for these types of {dfanalytics}.
125
+
126
+
[discrete]
127
+
[[train-test-iterate]]
128
+
=== Train, test, iterate
129
+
130
+
After your data is prepared and transformed into the right format, it is time to
131
+
train the model. Training is an iterative process — every iteration is followed
132
+
by an evaluation to see how the model performs.
133
+
134
+
The first step is defining the features – the relevant fields in the data set –
135
+
that will be used for training the model. By default, all the fields with
136
+
supported types are included in {regression} and {classification} automatically.
137
+
However, you can optionally exclude irrelevant fields from the process. Doing so
138
+
makes a large data set more manageable, reducing the computing resources and
139
+
time required for training.
140
+
141
+
Next you must define how to split your data into a training and a test set. The
142
+
test set won’t be used to train the model; it is used to evaluate how the model
143
+
performs. There is no optimal percentage that fits all use cases, it depends on
144
+
the amount of data and the time you have to train. For large data sets, you may
145
+
want to start with a low training percent to complete an end-to-end iteration in
146
+
a short time.
147
+
148
+
During the training process, the training data is fed through the learning
149
+
algorithm. The model predicts the value and compares it to the ground truth then
150
+
the model is fine-tuned to make the predictions more accurate.
151
+
152
+
Once the model is trained, you can evaluate how well it predicts previously
153
+
unseen data with the model generalization error. There are further
154
+
evaluation types for both {regression} and {classification} analysis which
155
+
provide metrics about training performance. When you are satisfied with the
156
+
results, you are ready to deploy the model. Otherwise, you may want to adjust
157
+
the training configuration or consider alternative ways to preprocess and
158
+
represent your data.
159
+
160
+
[discrete]
161
+
[[deploy-model]]
162
+
=== Deploy model
163
+
164
+
You have trained the model and are satisfied with the performance. The last step
165
+
is to deploy your trained model and start using it on new data.
166
+
167
+
The Elastic {ml} feature called {infer} enables you to make predictions for new
168
+
data either by using it as a processor in an ingest pipeline, in a continuous
169
+
{transform} or as an aggregation at search time. When new data comes into your
170
+
ingest pipeline or you run a search on your data with an {infer} aggregation,
171
+
the model is used to infer against the data and make predictions on it.
172
+
173
+
[discrete]
174
+
[[next-steps]]
175
+
=== Next steps
176
+
177
+
* Read more about how to {ref}/transforms.html[transform you data] into an
178
+
entity-centric index.
179
+
* Consult the documentation to learn more about <<dfa-regression,regression>>
180
+
and <<dfa-classification,classification>>.
181
+
* Learn how to <<ml-dfanalytics-evaluate,evaluate>> regression and
182
+
classification models.
183
+
* Find out how to deploy your model by using <<ml-inference,inference>>.
0 commit comments